Files
PaperClipAI/server/src/services/delivery-work-notifications.ts
T
DottaandClaude Opus 5.5 47a2c82d35 fix(heartbeat): reach the previous server's live runs first after a release swap (#21450)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - During a release swap, the outgoing server stops and the incoming
server takes over the live runs on remote Runners
> - The incoming server reached those runs late: it waited up to 60 s
for the outgoing server's controller lease to lapse, re-verified each
run's whole log from object storage, and deferred a reattach when the
agent directory was busy. Queued starts competed with the reattaches for
the same resources.
> - On hosted staging, the later a live run was reached, the more likely
it was to fail. Runs reached in under 150 s never failed; runs reached
after 420 s almost always failed.
> - This pull request hands live runs over at the start of shutdown,
makes a reattach skip work that the same run already did, and holds
queued starts while handed-over runs are being reached, with a 180 s
cap.
> - The benefit is that live runs are reached within seconds of a swap
instead of minutes, so they continue instead of failing.

## Linked Issues or Issue Description

**What happened?**

Hosted staging, release swaps on 2026-11-13 (06:05Z to 09:45Z). For each
run that crossed a swap we measured the gap from the last Runner output
to the moment the new server recorded the Runner's process identity:

- gap under 150 s: 0 of 9 runs failed
- gap over 420 s: 14 of 15 runs failed
- median gap: 180 s for runs that resumed, 470 s for runs that failed

In one traced run the incoming server waited on
`remote_controller_lease_live` three times (09:14:21, 09:15:18,
09:16:17), claimed at 09:16:59, and spent 85 s in `task.prepare`. Of
that, `heartbeat.prepare.run_output` took 58.7 s, because it re-read the
run's whole log from object storage. A control run spent 23 s in
`task.prepare`. Queued starts used the same preparation path, the same 8
object-I/O slots and the same database pool, and nothing gave a reattach
priority. The first-attempt failures were `PRP command session.snapshot
timed out` and `codex_history_read_failed`. The retry then failed with
`runner_harness_state_mismatch: resumed_not_suspended`,
`runner_remote_process_identity_unavailable` or
`provider_turn_lost_on_restore`.

**Expected behavior**

Every live run is reached within about 30 s of the swap. Queued work
starts after that, or after a bounded wait.

**Steps to reproduce**

1. Run a two-server release swap with live remote runs and queued runs.
2. Observe `native.recovery.transition` "waiting" events with
`remote_controller_lease_live`, then the claim, then a long
`heartbeat.prepare.run_output` span.
3. The new tests in `reattach-phase.test.ts` and
`heartbeat-native-early-hand-off.test.ts` model this.

**Paperclip version or commit**

`master` at 426e8a35dc.

## What Changed

- Early hand-off at shutdown (`heartbeat/recovery.ts`,
`server/src/index.ts`): in deploy-overlap mode, the outgoing server
releases the remote controller lease of its live runs at SIGTERM
(`recoveryState` `awaiting_runner_reattach`). A run another server
already took over is left alone. The incoming server no longer waits for
the lease to lapse.
- Reattach fast path:
- `heartbeat/run-output.ts`, `native-runtime/native-run-log-export.ts`:
a reattach starts without waiting for the existing log to be
re-verified.
- `heartbeat/agent-directory-busy.ts`, `heartbeat.ts`: a reattach that
meets a busy agent directory waits in process (at most 30 s) instead of
being deferred to a full re-preparation. An ordinary start is still
deferred at once, with no added wait.
- Reattach phase lifecycle (`heartbeat/reattach-phase.ts`): the watch is
a worker on the shared delivery coordinator, queue `reattach-phase`.
- Trigger: one startup listing of a previous server's live runs; then
the hand-over notification (`notifyReattachHandOver`, sent in the
hand-over transaction); the earliest lease deadline it watches (a killed
owner); and a 2 s re-check only while handed-over runs are still
pending, which ends with the phase (cap 180 s). The bounded re-check
exists because a run can also end on another path that sends no
notification to this queue.
- Source of truth: `native_run_finalizations` and `heartbeat_runs` rows.
- Quiet condition: nothing pending and no watched lease, so no deadline
and no SQL. It stops with the coordinator at shutdown.
- Reached hook for every driver (`native-session-executor.ts`): a
reattached run counts as reached when the native session runtime hands
the executor its recovered or opened session (`onSession`). Every driver
reaches that point only after the Runner answered this server's attach
and recovery commands, including an adopted `RunnerdExternalDriver`
Runner, which records no new process identity. The identity hook stays
as an earlier signal.
- Claim wake: when the reattach-phase worker finds runs that just became
claimable (hand-over notification or a killed owner's lease deadline),
it starts this server's restart-recovery claim for them at once. A
validation cell showed the claim otherwise waited about 21.5 s after the
hand-over.
- `recovery/assigned-work-sweep.ts`: the `deferred_reattach_phase`
return now sits before every action path of `act()`, except starting an
overdue saved-work recovery (it resumes an existing run).
- The in-transaction admission re-check of a queued claim
(`confirmRunAdmissionStillOpen`) also honours the reattach phase. A
phase that begins during a claim returns the run to the queue through
the existing rollback.
- A reattach whose log re-verification runs in the background awaits it
before the log is finalized (`settleDeferredLogRecovery`, both
finalization sites in `run-completion.ts`).
- Reattach phase (`heartbeat/reattach-phase.ts`,
`heartbeat/run-admission.ts`): `isReattachPhaseActive()` is true while
runs that a previous controller handed over (lease released or observed
expired, or claimed by this server's restart recovery) are claimable but
not yet reached. A run is reached when its process identity is recorded.
Authentication is logged as a mark. The cap (180 s,
`PAPERCLIP_REATTACH_PHASE_CAP_MS`) counts from the moment the runs
became claimable. Later batches join or re-arm the phase. Runs reached
after 150 s are counted and logged.
- Gates on the phase:
- `claimQueuedRun`, start-next, resume-queued and dispatch in
`heartbeat/queue.ts`, `run-lifecycle.ts` and
`native-runner-execution-driver.ts` leave queued runs queued.
- `execution-recovery-resolution.ts`: one line, the first line of
`deliverReconciledExecutions`.
- `recovery/assigned-work-sweep.ts`: one line, outcome
`deferred_reattach_phase`.
- `heartbeat/recovery.ts`: a one-line claim hook
(`noteReattachRunsClaimed`) after restart recovery claims runs.
  - The owner of those three files agreed to these one-line gates.
- `tests/restart-chaos/daytona-image.ts`:
`PAPERCLIP_CHAOS_TEST_ACCEPT_IMAGE_CONTENT_ID` lets a local wave run a
controller-only branch against an existing Daytona image. It accepts a
content-ID mismatch only for the exact content ID of the named image,
and it prints a TEST OVERRIDE warning. This is harness-only.

## Verification

Red on master, green on this branch:

- `heartbeat/reattach-phase.test.ts` (27 tests). Five of the six
two-server scenarios were red on the earlier phase timing:
- `does not hold queued starts while the previous owner is alive and not
handing over`
- `closes the gate when the previous owner hands its runs over, and
counts reach latency from the hand-over`
- `runs the cap from the hand-over, so a wedged run ends the phase
without the overlap eating it`
- `begins when a killed owner's lease is observed expired, without any
hand-over`
- `re-arms for a later batch, including runs the previous owner started
after discovery`
- `closes the gate at once when this server claims a run between two
lease reads, dated from the hand-over`
- plus `claimQueuedRun leaves a queued run queued and unstarted, in the
admission gate's shape`, `is the same gate the assigned-work sweep and
reconciled continuation delivery read`, `ends at the cap, counted from
when the runs became claimable, so one wedged Runner cannot hold run
starts`
- `heartbeat-native-early-hand-off.test.ts`: `releases the remote
controller lease of %i live runs at once` (1, 20, 40 runs; 40 runs in 56
ms), `leaves a run another server already took over alone`, `holds the
incoming server's queued starts only from the hand-over, not while the
outgoing server still drives its runs`
- `heartbeat/run-output.test.ts`: `lets a reattach start before its
existing log is re-verified`
- `heartbeat/agent-directory-busy.test.ts`: `defers an ordinary start at
once, with no wait added to the run-start path`, `lets a reattach wait
its turn in process instead of a deferred full re-preparation`, `still
defers a reattach once its bounded wait runs out`
- `tests/restart-chaos/daytona-image.test.ts`: `accepts a mismatch only
for the exact image content ID named by the test override`
- Review fixes (red before the fix, green after):
`reattach-phase.test.ts` `starts the claim of handed-over runs within a
second of the hand-over notification`; `assigned-work-invariant.test.ts`
`takes no action while handed-over live runs are being reached, and acts
once the phase ends`; `heartbeat-native-early-hand-off.test.ts` `never
lets the outgoing server claim back a run it handed off, while the
incoming server may take it over`; `reattach-phase.test.ts` `retries a
failed first listing instead of going quiet, and still finds the
previous owner's runs`, `keeps a hand-over notification whose listing
failed until a listing succeeds`, `moves the cap earlier when a claimed
run turns out to have been handed over before its claim`;
`queue.test.ts` `returns the claim to the queue when a reattach phase
begins while the claim is being checked`; `run-completion.test.ts`
`finalizes the run log in completeRun|failRun only after a reattach's
deferred log recovery settles`. `native-session-executor.test.ts` `ends
the reattach phase once a reattached %s session is driven by this server
again, without the cap or manual settling` over codex_app_server,
opencode_server, acpx_runtime, muse_external, grok_bot_mcp and
openai_dot_mcp (6 of 6 red without the hook; the test drives the
executor with a mocked session runtime, not each real driver). Lifecycle
tests for the new worker: `does no database work and keeps no deadline
on an instance with nothing to reattach`, `waits for the hand-over
notification or the lease deadline instead of polling while the previous
owner drives its runs`, `begins at a killed owner's lease deadline,
without any hand-over or notification`, `stops with the delivery
coordinator at shutdown and ignores a later hand-over`.

Local runs on this branch: `tsc --noEmit` is clean for `server`. Passing
on this branch: `reattach-phase.test.ts`,
`heartbeat-native-early-hand-off.test.ts`, `run-output.test.ts` (53
tests together), `agent-directory-busy.test.ts`,
`heartbeat-native-restart-suspension.test.ts` (115 of 115, alone), the
new cases in `queue.test.ts` and `run-completion.test.ts`. Under load 30
to 96 on this shared machine, the full `queue.test.ts` and
`recovery-wake-herd.test.ts` runs stalled in their embedded-Postgres
setup; CI supplies them.

Daytona results that exist (local servers with Daytona sandboxes,
release-swap cells). They ran on a combined branch with this PR, #21462,
#21442 and #21362 (merged), so they cannot be credited to this PR alone:

- G5 baseline (master d1d71a7755): SIGTERM to PRP authentication 21.5 to
38 s at swap 1.
- G5c (combined branch 588c6ba267): SIGTERM to authentication 6.2 to 7.7
s at swap 1. All runs that crossed swap 2 were adopted on the same
Runner process. The staging chain (`provider recovery timed out after
900000ms`, then `resumed_not_suspended`) did not recur.
- Caveat: G5c ran before the phase timing fix in this PR. In G5c the
phase opened at process start, its 180 s cap ran out during the overlap,
and swap 2 ended `cap_reached` with 0 runs reached, 1 s before the lease
claims. The timing fix (the phase begins when the runs become claimable)
has not run on Daytona.
- G6c (12 live runs and 20 queued, two graceful swaps, no injection,
combined branch 588c6ba267): 0 failed runs at either swap; 11 of 11
survivors at both.
- Swap 1: hand-off of 11 runs in 291 ms; claims 0.3 to 2.9 s after
hand-off; authentication 7.6 to 12.5 s and identity 8.3 to 13.3 s after
SIGTERM. The 30 s target was met.
- Swap 2: hand-off took 8,562 ms and `hot_restart_prepare` timed out at
19 s, so the outgoing server drained gracefully. The incoming server did
not claim on the hand-over; it claimed only when the outgoing server
exited, 33 s after SIGTERM. Authentication was 46 to 54 s and identity
47 to 57 s after SIGTERM. The target was missed. Why the claims waited
for the exit is not explained.
- The host load was 49 to 70 from other work. 7 of 32 tasks timed out in
the queue or budget after the swaps; we suspect the overload.

## Risks

- Product trade-off: while handed-over live runs are being reached,
queued starts wait, up to the 180 s cap. On an instance with no
handed-over runs, nothing waits.
- Not proven: two-server behavior with the phase timing fix and the
notification-driven watch (no cell has run them); warm standby becoming
active; runs that the outgoing server starts after discovery.
- Early hand-off at SIGTERM orphans a server-side exec tool that is in
flight. With #21442 the next controller settles it as interrupted with
an unknown outcome; it is never run again.
- Swap 2 in G6c: the incoming server claimed only at the outgoing
server's exit. That is unexplained.
- Cross-session ownership: the gates in
`execution-recovery-resolution.ts` and
`recovery/assigned-work-sweep.ts`, and the claim hook in
`heartbeat/recovery.ts`, are in files that the restart-recovery work
owns. They are one-line gates agreed with that owner. That owner said
the gate did not need to be inside the claim transaction; that was
before the phase could begin during a claim. After review, the
in-transaction re-check in `heartbeat/queue.ts` also reads the phase,
through the existing rollback. #21427 (open) also edits
`execution-recovery-resolution.ts`, `heartbeat/queue.ts` and
`recovery/assigned-work-sweep.ts`; whichever merges second needs a
rebase. `native-runtime/native-restart-recovery.ts` (claim logic, owned
by the restart-recovery work): a 3-line guard in
`claimNativeRestartRecoveries` returns a run this process handed off at
shutdown as `blocked` / `handed_off_to_next_controller` before takeover
is evaluated, so the outgoing server never claims back a run it handed
off. #21478 (merged) added a sweep path that starts an overdue
saved-work recovery; it is deliberately not gated by the reattach phase,
because it resumes an existing run (reattach-like work), which the phase
makes room for rather than competes with.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 18:55:31 -06:00

177 lines
8.3 KiB
TypeScript

import { signalDatabaseWork, subscribeDatabaseWork } from "@paperclipai/db/work-signals";
/** Topics for existing durable queues; register intent before their writes. */
export const DELIVERY_QUEUES = {
mcpClientMirror: "mcp-client-mirror",
mcpIngressHandoffs: "mcp-ingress-handoffs",
mcpRevocationIndex: "mcp-revocation-index",
nativeRuntimeResponses: "native-runtime-responses",
nativeCancellation: "native-cancellation",
deploymentNotifications: "deployment-notifications",
liveInvalidations: "live-invalidations",
nativeProcessSessions: "native-process-sessions",
nativeQuestionCancellations: "native-question-cancellations",
activeExecutionRecovery: "active-execution-recovery",
ownershipInvariantReport: "ownership-invariant-report",
settledNativeProcesses: "settled-native-processes",
orphanActiveLeases: "orphan-active-leases",
legacyShutdownSettlement: "legacy-shutdown-settlement",
strandedAgentStatus: "stranded-agent-status",
activeRunSilence: "active-run-silence",
deferredExecutionReceipts: "deferred-execution-receipts",
taskRetryReceiptSettlement: "task-retry-receipt-settlement",
nativeStatusWakeIntents: "native-status-wake-intents",
orphanBlockers: "orphan-blockers",
activeRecoveryActions: "active-recovery-actions",
strandedAssignments: "stranded-assignments",
assignedWorkInvariant: "assigned-work-invariant",
resolvedDependencies: "resolved-dependencies",
taskWatchdogs: "task-watchdogs",
staleIssueLocks: "stale-issue-locks",
scheduledRetryDispatch: "scheduled-retry-dispatch",
queuedRunDispatch: "queued-run-dispatch",
runStartHandover: "run-start-handover",
reattachPhase: "reattach-phase",
statusCardEvaluation: "status-card-evaluation",
runEvidenceMigration: "run-evidence-migration",
providerTraceRetention: "provider-trace-retention",
nativeRunnerEvidenceRetention: "native-runner-evidence-retention",
skillVersionRetention: "skill-version-retention",
externalObjectRefresh: "external-object-refresh",
databaseBackups: "database-backups",
importSpoolCleanup: "import-spool-cleanup",
externalConversationSettlement: "external-conversation-settlement",
nativeRecoveryNotices: "native-recovery-notices",
nativeEnvironmentSettlement: "native-environment-settlement",
nativeRunProjection: "native-run-projection",
nativeResultCommit: "native-result-commit",
nativeWorkspaceFinalization: "native-workspace-finalization",
retainedNativeSessionCleanup: "retained-native-session-cleanup",
nativeStatusReassessment: "native-status-reassessment",
nativeReviewResponse: "native-review-response",
nativeResponsePresentation: "native-response-presentation",
nativePendingEffects: "native-pending-effects",
unsafeWorkspaceExportRecovery: "unsafe-workspace-export-recovery",
automaticCompletionReviews: "automatic-completion-reviews",
obsoletePolicyReviews: "obsolete-policy-reviews",
cancelledWorkspacePreservation: "cancelled-workspace-preservation",
instructionCollection: "instruction-collection",
directoryCleanup: "directory-cleanup",
instructionCommit: "instruction-commit",
scheduledRetryCancellation: "scheduled-retry-cancellation",
executionFinalization: "execution-finalization",
workspaceRecovery: "workspace-recovery",
providerLimitResume: "provider-limit-resume",
executionReplacement: "execution-replacement",
executionContinuation: "execution-continuation",
executionStatus: "execution-status",
executionDisposition: "execution-disposition",
environmentCleanup: "environment-cleanup",
approvalQueueSeeding: "approval-queue-seeding",
attentionRetention: "attention-retention",
approvalRetention: "approval-retention",
joinRequestRetention: "join-request-retention",
decisionTargets: "decision-targets",
decisionEffectRecovery: "decision-effect-recovery",
decisionExpiry: "decision-expiry",
decisionContinuation: "decision-continuation",
decisionArchiveDelivery: "decision-archive-delivery",
toolConnectionHealth: "tool-connection-health",
toolReviewProjection: "tool-review-projection",
toolReviewInterruption: "tool-review-interruption",
toolReviewRecovery: "tool-review-recovery",
toolReviewExpiry: "tool-review-expiry",
gatewayCredentialExpiry: "gateway-credential-expiry",
secretProposalExpiry: "secret-proposal-expiry",
customImageExpiry: "custom-image-expiry",
historicalCleanup: "historical-cleanup",
adapterLoginCleanup: "adapter-login-cleanup",
setupTokenCleanup: "setup-token-cleanup",
localLoginCleanup: "local-login-cleanup",
budgetEnforcement: "budget-enforcement",
accounting: "cost-accounting",
taskDrainExpiry: "task-drain-expiry",
controllerInstanceLease: "controller-instance-lease",
agentLifecycle: "agent-lifecycle",
chatBoardMessages: "chat-board-messages",
chatTaskStarts: "chat-task-starts",
chatReceiptReactions: "chat-receipt-reactions",
chatSessionStops: "chat-session-stops",
chatProviderEffects: "chat-provider-effects",
chatGitHubWebhookRecovery: "chat-github-webhook-recovery",
chatGitHubIngress: "chat-github-ingress",
chatGitHubPublications: "chat-github-publications",
chatGitHubReviewProjection: "chat-github-review-projection",
chatGitHubReviewChecks: "chat-github-review-checks",
chatGitHubTerminalResponses: "chat-github-terminal-responses",
chatGitHubAcknowledgements: "chat-github-acknowledgements",
chatFailedRunRetries: "chat-failed-run-retries",
chatTelegramMaintenance: "chat-telegram-maintenance",
chatReceivers: "chat-receivers",
voiceSessionRecovery: "voice-session-recovery",
voiceReports: "voice-call-reports",
voiceInboundRetention: "voice-inbound-retention",
voiceInboundCleanup: "voice-inbound-cleanup",
voiceReplies: "voice-reply-delivery",
chatTeamsConsentExpiry: "chat-teams-consent-expiry",
chatTeamsAttemptRecovery: "chat-teams-attempt-recovery",
chatCommandRegistration: "chat-command-registration",
chatTelegramSubscriptions: "chat-telegram-subscriptions",
chatSessionSyncs: "chat-session-syncs",
chatVerification: "chat-verification",
chatInbound: "chat-inbound",
chatPublications: "chat-publications",
chatNativeProgress: "chat-native-progress",
chatInboundWakeNotices: "chat-inbound-wake-notices",
chatInteractionContinuations: "chat-interaction-continuations",
chatTerminalConfirmations: "chat-terminal-confirmations",
chatFailedRetryNotices: "chat-failed-retry-notices",
chatRunMilestones: "chat-run-milestones",
chatReceipts: "chat-slack-receipts",
browser: "browser-use",
email: "email-channels",
museMaintenance: "muse-maintenance",
mergedPullRequestConfirmations: "merged-pull-request-confirmations",
mcpEvents: "mcp-events",
mcpTasks: "mcp-tasks",
dotEvents: "dot-events",
grokDispatch: "grok-dispatch",
skillSourceSync: "skill-source-sync",
skillPackages: "skill-packages",
runtimeAssetRetention: "runtime-asset-retention",
fastResponse: "fast-responses",
feedback: "feedback-exports",
chatCompletion: "chat-completions",
connection: "connection-continuations",
connectionEvents: "connection-events",
connectionContinuity: "connection-continuity",
gitRestorePoints: "git-restore-points",
remoteWorkspaceSnapshots: "remote-workspace-snapshots",
nativeWorkspaceSyncRetention: "native-workspace-sync-retention",
question: "question-responses",
pluginSchedules: "plugin-schedules",
pluginLifecycle: "plugin-resource-lifecycle",
toolAction: "tool-action-receipts",
terminalWorkspaces: "terminal-workspaces",
routineExecutionStatus: "routine-execution-status",
routineSchedules: "routine-schedules",
issueMonitors: "issue-monitors",
agentTimers: "agent-timers",
caseHandoffs: "case-handoffs",
nativeHarnessBackupExpiration: "native-harness-backup-expiration",
nativeCheckpointExpiration: "native-checkpoint-expiration",
} as const;
export type DeliveryQueue = typeof DELIVERY_QUEUES[keyof typeof DELIVERY_QUEUES];
export async function notifyDeliveryWork(transaction: object, queue: DeliveryQueue, hint?: string): Promise<void> {
await signalDatabaseWork(transaction, queue, hint);
}
// Public delivery notification observers only see settled work. The coordinator
// uses the lower-level lifecycle subscription to also fence in-flight writes.
export function subscribeDeliveryWork(owner: object, queue: DeliveryQueue, wake: () => void): () => void {
return subscribeDatabaseWork(owner, queue, event => {
if (event === "settled") wake();
});
}