Files
PaperClipAI/server/src/__tests__/live-recovery-work.test.ts
T
DottaandClaude Opus 5.5 340157d1d6 perf(recovery): scope run lifecycle recovery hints to the run's task graph (#21552)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server runs issue recovery queues in the background. Two of
them, `stranded-assignments` and `assigned-work-invariant`, find
assigned tasks that have no live run and restart them.
> - Every run dispatch and every run state change sent a recovery hint
for the whole company. On each hint these queues reload every eligible
open task of the company and check each one again, at most every 30
seconds per queue.
> - With live runs the hint comes all the time, so the queues never
stop. Their cost is (run activity) x (open-task backlog). On a hosted
instance with about 1,400 open tasks and 5 live runs, the server main
thread stayed at 87-92% of a core.
> - A run event can only change the tasks of its own task graph. This
pull request makes run and task producers name that graph, and keeps one
slow full pass as a backstop.
> - The benefit is that a run event costs a few checks, whatever the
size of the backlog, and a missed hint still delays recovery by at most
five minutes.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/enhancement.yml`.

**What existing behavior does this improve?**

The issue recovery queues in `server/src/services/recovery/work.ts`
(`stranded-assignments`, `assigned-work-invariant`,
`resolved-dependencies`, `stale-issue-locks`) and the producers that
wake them on run dispatch, run start, run state change, run finalization
and accounting settlement.

**Subsystem affected**

Server background recovery (`server/src/services/recovery/`,
`server/src/services/scoped-deadline-work.ts`, the notification helpers
in `server/src/services/*-notifications.ts`).

**Current behavior**

`notifyRunDispatch`, `notifyNativeStatusReassessment` and
`notifyExternalConversationSettlement` sent `notifyIssueRecovery(tx, {
companyId })`. Most run lifecycle paths call one of them, often two or
three in the same transaction (for example `notifyIssueReviewState`).
Some other run paths sent the company hint directly: the wake status
update, the execution-control deadline writes, legacy run
terminalization, scheduled retries, accounting settlement, task checkout
and task document writes. Each company hint makes each task queue reload
all eligible open tasks of the company, 100 per page, and check each
task again. A company hint is reconstructed at most every 30 seconds per
queue, so one run event per 30 seconds is enough to keep a full pass
running all the time. `issue-recovery-deadlines.test.ts` pinned this: a
company hint re-checks every eligible task.

**Proposed behavior**

Design. For each task queue, a run event of run R on task T can change:

| Queue | What the check reads | What one run event can change |
| --- | --- | --- |
| `stranded-assignments` | T's own runs, wakes, handoffs, questions,
recovery actions, goal, assignee eligibility | T (its runs). The parent
of T and the tasks T blocks when the event also writes T's status. |
| `assigned-work-invariant` | the same plus open blockers and open
children | T, the parent of T (open children), the tasks T blocks (open
blockers) |
| `resolved-dependencies` | blocked tasks whose blockers are all done |
the tasks T blocks |
| `stale-issue-locks` | tasks whose checkout or execution run is R | T |

The assignee's agent status (`running`, `idle`, `error`) does not change
whether the agent is invokable
(`packages/shared/src/agent-eligibility.ts`), and a freed concurrency
slot starts the agent's next queued run through the
`queued-run-dispatch` queue, which keeps its company hint. So the
agent's other tasks are not in the set.

- `notifyIssueGraphRecovery(tx, { companyId, runId?, issueId? })` sends
`{ companyId, graphRunId?, graphIssueId? }` to the four task queues. It
runs no query in the producer transaction.
- When a task queue loads that hint, it resolves the graph with one
bounded, indexed query (`resolveIssueGraph`): the task of the run
(`issue_id`, `native_issue_id`, or the saved `issueId`/`taskId`,
validated and cast so the primary key is used), its parent, and the
tasks it blocks. The four queues share the result for 2 seconds. With no
task or more than 100 tasks, the queue turns the hint into the plain
company hint, so task-less runs coalesce under its 30-second gate
instead of each starting its own full pass. The lookup runs after
commit, so it sees the written state. Graph hints do not wait for the
30-second reconstruction gate (they load at most 100 tasks); hints for
the same run coalesce while dirty, and the lookup is shared across the
four queues, so it runs about once per pass instead of once per producer
call. An earlier round ran the lookup in every producer transaction: 34
lookups/s at 1,500 open tasks and 70/s at 5,000.
- The three company-level queues (`orphan-blockers`,
`active-recovery-actions`, `task-watchdogs`) keep the company hint. They
hold one entry per company or per watchdog, not per open task.
- `notifyRunDispatch`, `notifyScheduledRetryDispatch`,
`notifyNativeStatusReassessment` and
`notifyExternalConversationSettlement` use it when the scope names a run
or a task. A scope with only a company keeps the company hint.
- Run paths that knew the run or the task but sent only the company now
pass it. This includes three `writeExecutionControl` deadline writes,
legacy run terminalization (`notifyExecutionRunWork`) and native safe
replacement. Those helpers also forward the scope to
`notifyLiveRunRecovery` and `notifyLiveResourceRecovery`, so their hints
to the live-run queues now carry the run id.
- Recovery action changes (`issue-recovery-actions.ts`: create,
supersede, update, exhaust, resolve) name their source task. A recovery
action is a fact of its source task for these queues. The company-level
`active-recovery-actions` queue still gets the company hint. In the
fixture, failing runs create and resolve recovery actions every few
seconds, and that alone kept a full pass running.
- `writeExecutionControl` and `notifyExecutionRunWork` send the
attention retention hint for a task through
`notifyIssueAttentionRetentionWork` (the task, its parents and the tasks
it blocks), a superset of the single-task hint they sent before. A
recovery action of a blocker can change the blocker attention of its
dependents, so the narrower recovery action scope needs this.
- `promoteDeferredWorkIntents` scopes `lock_released` with a task to
that task, and `accounting_settled` to the run when the company has no
active hard-stop budget policy. Only a reached hard stop can refuse work
because of accounting (`evaluateAdmissionInTransaction`), so without
such a policy the settlement of one run cannot admit another task. The
other hold releases keep the company hint.

Backstop. Each scoped hint also sends `{ companyId, backstop: true }` to
the same four queues. Nothing else arms it. The queue runs it as a full
company pass, but not before 5 minutes after the last full pass of that
queue for that company or of the startup pass. It is a scheduler
deadline (`nextWakeAt` reports it), but it holds no idle intent and does
not mark the queue unreconciled. So:

- Startup still reconstructs every company.
- A failed scoped load retries with the existing back-off (30 seconds to
5 minutes). The backstop is pending in any case.
- A graph with no task or more than 100 tasks loads the whole company at
once.
- A task that a scoped hint missed is checked at most
`ISSUE_RECOVERY_BACKSTOP_MS` (5 minutes) after the event. While runs
continue, each queue runs at most one full pass per company per 5
minutes. With no activity, nothing is scheduled.

Why arming on activity only cannot leave a stranded task undetected when
the instance then goes quiet. A task becomes stranded only through a
write or through a run that stops without a write:

1. The write names the task's graph (run and task producers). The scoped
pass checks the task at once.
2. The write sends a company hint (pause, budget, agent and company
configuration, tree holds, the producers listed below). The company pass
checks it within the 30-second gate, as before.
3. The write is a run event whose graph does not contain the task. That
same event armed the backstop. The backstop deadline stays scheduled
while the instance is quiet and checks the task at most 5 minutes later.
4. A run stops without any write (process lost, controller gone). The
existing deadline paths find it: `stale-issue-locks` waits for the
stored controller lease, and `active-run-silence` and
`orphan-active-leases` act on their own deadlines. Each of them
terminalizes the run, which is a run write and falls under 1 or 3. The
new test `restarts a task whose run vanished without a final event once
its lease lapses` covers this with the production backstop.
5. The process restarts or a quiet instance is woken by a restart. The
startup pass reconstructs every company.
6. A check that found a task waiting (grace, back-off, monitor) returns
its own deadline. That entry keeps its deadline whatever hints arrive.

Five minutes is the existing upper bound of recovery back-off in this
area: the maximum retry delay of a failed scoped-deadline check
(`scoped-deadline-work.ts`), and `DEFERRED_WAKE_PENDING_BOUND_MS`, the
time `assigned-work-sweep.ts` already trusts a deferred wake before it
calls a task stranded.

Producers that keep the company hint, and why. They can change
eligibility of tasks outside one task graph, or I could not show that
their effect stays in the graph. None of them fires once per run event
in the fixture.

- Eligibility for many tasks: company status (`companies.ts`), budgets
(`budgets.ts`), agent lifecycle and configuration (`agent-lifecycle.ts`,
`agent-configuration.ts`), instance settings (`instance-settings.ts`,
`*`), tree holds (`issue-tree-control.ts`), company deletion and import
(`company-deletion.ts`, `company-portability.ts`), and the hold releases
`pause_lifted`, `tree_hold_released`, `scheduling_resumed` and
`reconciler_backstop`. `accounting_settled` also keeps it when the
company has an active hard-stop budget policy.
- Task writes that can move a task between graphs or change eligibility:
task creation and a task update that changes the parent in `issues.ts`
(other task updates and checkout releases now name the task), document
delete and restore paths in `documents.ts`, task watchdogs
(`task-watchdogs.ts`), system and workspace recovery
(`system-recovery.ts`, `task-retry.ts`,
`workspace-recovery-capture-policy.ts`,
`workspace-recovery-stop-proof.ts`).
- Conversation and chat paths: `agent-conversations.ts`, the daily-cap
release in `heartbeat/queue.ts`, the Telegram attachment failure,
`chat-attachment-reuse.ts`, `native-runner-file-handoff.ts`,
`notifyChatRunMilestones` without a task or run (company authority
change), and `notifyNativeStatusReassessment` from `access.ts`.
- `writeExecutionControl` and `notifyExecutionRunWork` callers that pass
only a company.

**Reason and benefit**

Background work must not grow with the backlog of open tasks. The checks
a run event causes now depend on its task graph, not on the company.

## What Changed

- `server/src/services/issue-recovery-notifications.ts`:
`notifyIssueGraphRecovery`, `resolveIssueGraph`, the scope matcher and
decoder, `ISSUE_RECOVERY_MAX_ISSUE_IDS` (100) and
`ISSUE_RECOVERY_BACKSTOP_MS` (5 minutes).
- `server/src/services/recovery/work.ts`: the task queues resolve a
graph hint when they load it (shared for 2 seconds), remember the last
full pass per company, and gate the backstop hint on it.
`registerIssueRecoveryWork` takes an optional `backstopMs`.
- `server/src/services/recovery/assigned-work-sweep.ts`: `listSuspects`
accepts `issueIds`.
- `server/src/services/scoped-deadline-work.ts`: an optional
`backstop(scope)` that returns the earliest time of a backstop hint. A
pending backstop hint keeps its deadline but does not count as pending
work or as unreconciled.
- Producers: `issue-recovery-actions.ts`,
`execution-control-notifications.ts`, `run-dispatch-notifications.ts`,
`native-status-reassessment-notifications.ts`,
`external-conversation-settlement-notifications.ts`,
`work-intent-outbox.ts`, `run-cost-accounting.ts`,
`heartbeat/run-lifecycle.ts`, `heartbeat/runtime-execution.ts`,
`heartbeat/retries.ts`, `heartbeat.ts`, `legacy-execution-recovery.ts`,
`issues.ts` (checkout), `documents.ts` (task document writes),
`modules/wake-queue/adapters/postgres.ts` (recovery runs for a task),
`heartbeat/queue.ts` (wake enqueue), `recovery/service.ts`,
`native-runtime/native-provider-capacity-retry.ts`,
`native-runtime/native-safe-replacement.ts`,
`modules/run-dispatch/adapters/workspace-busy-wait.ts` (scheduled
retries).
- `bounds subsidiary scans for100 assignment candidates`: a resolved
recovery action now adds one scoped load per task queue (bound 8 to 11),
and the coalescing check sends one company hint before the burst of 100,
because the first full pass after the resolve is now a backstop pass
with its own key.
- `checks only blocked tasks whose blockers are done`: uses the shared
recording harness and runs out the backstop before its quiet check,
because the blocker's status change is now a graph hint (its 300
dependents overflow to a company load).
- Tests. `issue-recovery.test-fixture.ts` and
`live-recovery-work.test.ts` register the queues with `backstopMs: 0`,
so their quiet checks keep the earlier timing. The new tests use the
production value.

## Verification

New tests. I saw each red test fail before the change.

- `issue-recovery-deadlines.test.ts` `re-checks only the tasks a run
lifecycle event can affect, whatever the backlog` (red before): a run of
the agent on a task with a parent and a dependent, 1,000 backlog tasks
of another agent, and a second open task of the same agent. Seven real
producers (`notifyNativeStatusReassessment`, `notifyExecutionRunWork`,
`notifyRunDispatch`, `promoteDeferredWorkIntents` lock release and
accounting settlement, recovery action create and resolve) check exactly
the 3 tasks of the graph. The recovery action steps were red before the
third commit. With 4,000 more backlog tasks the event issues the same
number of statements. Before the change it checked 1,004 tasks.
- `resolves a run's task graph through index lookups, not the backlog`
(red before the second commit, from a Greptile finding): with 2,000
backlog tasks the graph lookup read 2,005 issue rows
(`pg_stat_xact_user_tables`) because it compared `id::text`. It now
validates and casts the saved id, reads fewer than 50 rows, and returns
exactly the run's graph.
- `finds a stranded task outside the affected set within the backstop
bound` (red before): a task inserted without a hint is not checked by
the run event, and is checked at most 5 minutes after the last full
pass. After 20 quiet minutes, a run event checks a new unrelated task
within 5 seconds, because the last full pass is older than 5 minutes.
Then the queues are quiet.
- `keeps the company pass for hold releases that can admit any task`
(guard, passes before and after): accounting settlement with a hard-stop
policy and a lifted pause still check every task.
- `falls back to the company pass when a run affects more tasks than a
hint names` (guard): a run task that blocks 100 more tasks checks the
whole company.
- `limits an ordinary comment to its issue among one thousand open
assignments`: the old final assertion (a company hint checks all 1,000
tasks) stays, and the test now also runs out the backstop and ends
quiet.
- `issue-recovery-assignments.test.ts`, real heartbeat, real producers,
production backstop (guards; they also pass with the old company hint):
a task whose run failed, a task whose run the system cancelled, a run
that vanished without a final event and is found when its controller
lease lapses, and an agent's queued task that starts when its previous
run ends. Each one recovers within 15 seconds, well before the backstop.
- `scoped-deadline-work.test.ts` `keeps a backstop hint's deadline
without holding the worker busy` (red before).

Suites, each file alone, in `server/` (issue recovery, live recovery,
recovery action and attention suites against dedicated local `_test`
databases): `issue-recovery-deadlines` (13/13),
`issue-recovery-assignments` (7/7), `issue-recovery-work-lifecycle`
(14/14), `scoped-deadline-work` (24/24), `live-recovery-work` (51/51),
`assigned-work-invariant` (32/32), `assigned-work-sweep-round-trips`
(2/2), `saved-work-recovery-overdue` (4/4),
`issue-recovery-dependencies` (5/5), `issue-recovery-conversations`
(3/3), `issue-recovery-actions` (112/112), `attention-retention-work`
(55/55). `recovery-deadline-acceptance` skips without its external
database. `pnpm check:module-boundaries` passes.

Local benchmark. The real server ran against PostgreSQL with
`log_statement=all` and instrumented counters (not committed). Fixture:
1 company, 60 agents, open tasks at 150 / 1,500 / 5,000 (52% in review,
39% blocked behind open blockers, 3% in progress, 2% to do), with
process agents that fail at once so recovery re-dispatches runs all the
time. The tables are analyzed after loading. Each window is 180 s,
starting 150 s after the server is ready, so each "after" window
contains one backstop pass. Base is `master` at 54c3616ad9; after is
this branch at dcd170e29f. The last commit (graph hints without the
30-second gate, task updates and checkout releases scoped) is covered by
the tests below but not re-measured. The machine was shared (load
average 9-16, one base window at 45). Run churn differs between runs, so
the table gives counts per run event (runs created in the window).

| Open tasks | Runs created (/s) base → after | Stranded checks per run
base → after | Assigned-work checks per run base → after |
`agent_task_sessions` reads (/s, 4 shapes, stranded check) | Statements
per run event base → after |
| --- | --- | --- | --- | --- | --- |
| 150 | 0.08 → 0.14 | 45.1 → 6.2 | 2.00 → 0.96 | 3.75 → 1.27 | 2,260 →
1,257 |
| 1,500 | 1.09 → 1.00 | 22.0 → 3.9 | 1.24 → 1.53 | 26.5 → 7.2 | 1,313 →
1,060 |
| 5,000 | 1.12 → 1.99 | 20.4 → 1.6 | 4.10 → 0.91 | 29.3 → 8.1 | 1,400 →
829 |

New statements: the task graph lookup (`with seed(id) ...`) at 0.8 / 9.3
/ 18.5 per second, and scoped loads of the task queues (`select "id",
"company_id" from "issues" ...` grew from 0.7 / 6.6 / 7.8 to 2.9 / 20.6
/ 34.4 per second, each a lookup of at most 100 ids). Stale-lock checks
grew from 0 / 50 / 52 to 22 / 190 / 445 per window: each run event now
checks the locks of its own task at once instead of waiting for the next
full pass. In an earlier round the graph lookup ran in every producer
transaction: 34/s at 1,500 and 70/s at 5,000.

With no activity the server issues no statements before and after this
change (quiet checks in `issue-recovery-deadlines`,
`issue-recovery-assignments`, `live-recovery-work`).

## Risks

- A task outside the graph of the run whose state the event changes is
now found by the backstop, at most 5 minutes later, instead of within 30
seconds. One known case: from #21427, `assigned-work-invariant` counts a
held recovery wake as pending while its agent has queued recovery runs.
When the last one starts, the continuation queue delivers the held wake;
the invariant check of that task waits for the backstop.
- A producer that names a task but changes something wider would now
miss the wider effect. Producers that change eligibility for many tasks
(pause, budget, agent and company configuration, tree holds) still send
the company hint.
- The graph is resolved after commit, from the written state. A parent
change is not a run event and keeps its company hint.
- The live-recovery and issue-recovery suites opt out of the backstop
deadline with `backstopMs: 0`.
- The execution-control deadline writes and legacy run terminalization
now pass the run id. `writeExecutionControl` also forwards that scope to
the live-run queues (`active-run-silence`, `native-process-sessions` and
others), so those hints now name the run instead of the company. Those
queues already handle run-scoped hints from native finalization.
- Test-only option: `registerIssueRecoveryWork(..., { backstopMs })`.
The shared recovery fixtures pass `0` so their quiet checks keep the
earlier timing; the new tests use the production value.
- No schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 01:29:22 -06:00

1115 lines
85 KiB
TypeScript

import { currentProcessSpaceHash } from "../services/controller-process-identity.js";
import { randomUUID } from "node:crypto";
import os from "node:os";
import { spawn } from "node:child_process";
import { once } from "node:events";
import { setImmediate as immediate, setTimeout as realDelay } from "node:timers/promises";
import { and, eq, sql } from "drizzle-orm";
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
import { agents, authUsers, companies, companyMemberships, createDb, closeLivenessDb, closeCheckpointDb, livenessDb,
heartbeatRuns, heartbeatRunEvents, nativeRunFinalizations, environmentLeases, issueThreadInteractions, applyPendingMigrations,
issueRecoveryActions, issues, activityLog, agentWakeupRequests, reconcileDatabaseWork } from "@paperclipai/db";
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js";
import { heartbeatService } from "../services/heartbeat.js";
import { notifyLiveRunRecovery, notifyLiveResourceRecovery, notifyStrandedAgentStatus } from "../services/live-run-recovery-notifications.js";
import { notifyExecutionRunWork } from "../services/execution-control-notifications.js";
import { appendHeartbeatRunEvent } from "../services/heartbeat-run-events.js";
import { writeEnvironmentCleanup } from "../services/environment-cleanup-notifications.js";
import { prepareSandboxStopAndRetain, settleStopOnlyCleanup, hasConfirmedSandboxStopAndRetain } from "../services/sandbox-stop-and-retain.js";
import { beginLegacyShutdownWorkspaceSettlement } from "../services/legacy-shutdown-workspace-settlement.js";
import { renewLegacyControllerLease, legacyControllerBootId } from "../services/legacy-controller-lease.js";
import { idleWorkSnapshot } from "../services/task-admission.js";
import { createHeartbeatRunOutput } from "../services/heartbeat/run-output.js";
import { recoveryService } from "../services/recovery/service.js";
import { instanceSettingsService } from "../services/instance-settings.js";
import { persistActivity } from "../services/activity-log.js";
import { accountRunCost } from "../services/run-cost-accounting.js";
import { persistHeartbeatRunProcessMetadata, createHeartbeatLifecycle } from "../services/heartbeat/run-lifecycle.js";
import { createAgentLifecycle, configureAgentLifecycle } from "../services/agent-lifecycle.js";
import { issueTreeControlService } from "../services/issue-tree-control.js";
import { terminalizeLegacyExecution } from "../services/legacy-execution-recovery.js";
import { issueService } from "../services/issues.js";
import { nativeRetryCancellationEligible, pendingFailedNativeRetryCancellationCondition } from "../services/native-runtime/native-cancellation-request.js";
import { requestNativeQuestionRunCancellation } from "../services/native-runtime/native-question-bridge.js";
import { readNativeLocalProcessStop, releaseSettledNativeRunProcess, settleNativeRunProcess } from "../services/native-local-process-stop.js";
import { DELIVERY_QUEUES } from "../services/delivery-work-notifications.js";
import { renewNativeSessionExecutionLease } from "../services/native-runtime/native-execution-lease.js";
import { buildNativeExecutionInput } from "../services/native-runtime/native-execution-input.js";
import { nativeRuntimeContextFixture } from "../services/native-runtime/runtime-context.test-fixture.js";
import * as nativeStopEvidence from "../services/acknowledged-native-stop.js";
import { remoteTerminationReceipt } from "../services/remote-execution-termination.js";
import { subscribeCompanyLiveEvents } from "../services/live-events.js";
import type { LiveEvent } from "@paperclipai/shared";
import { logger } from "../middleware/logger.js";
vi.setConfig({ testTimeout: 90_000 });
describe("residual recovery production matrix", () => {
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined;
let db: ReturnType<typeof createDb>;
let heartbeat: ReturnType<typeof heartbeatService>;
let coordinator: ReturnType<typeof createDeliveryWorkCoordinator>;
let draining = false, standby = false;
const errors: unknown[] = [], passes = new Set<Promise<unknown>>();
const deadlines = new Map<string, () => number | null>();
const timeline: unknown[] = [];
const extraStops: Array<() => Promise<void>> = [];
beforeAll(async () => {
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_LIVE_RECOVERY_TEST_DATABASE_URL");
if (!external) temporary = await startEmbeddedPostgresTestDatabase("live-recovery-");
else await applyPendingMigrations(external);
db = createDb(external ?? temporary!.connectionString, { maxConnections: 6, idleTimeoutSeconds: 0, maxLifetimeSeconds: 0 });
await Promise.all(Array.from({ length: 6 }, () => db.execute(sql`select pg_sleep(0.01)`)));
});
beforeEach(async () => {
await db.execute(sql`truncate companies cascade`); errors.length = 0; deadlines.clear(); timeline.length = 0; draining = false; standby = false;
heartbeat = heartbeatService(db);
coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => !standby, canReconcile: () => !standby,
onError: error => errors.push(error) });
const register = coordinator.register.bind(coordinator);
coordinator.register = (queue, task) => { deadlines.set(queue, () => task.nextRunAt?.() ?? null); return register(queue, { ...task, onSettled: hints => { timeline.push(["settled", queue, Date.now(), [...hints ?? []]]); task.onSettled?.(hints); }, run: signal => {
timeline.push(["pass", queue, Date.now()]);
const pass = Promise.resolve(task.run(signal)); passes.add(pass);
void pass.then(() => { timeline.push(["finished", queue, Date.now()]); passes.delete(pass); }, () => passes.delete(pass)); return pass;
} }); };
});
afterEach(async () => {
for (const stop of extraStops.splice(0)) await stop();
await coordinator.stop(); await heartbeat.drainActiveRunExecutions();
vi.useRealTimers(); vi.restoreAllMocks(); expect(errors).toEqual([]);
});
afterAll(async () => {
if (db) {
await closeLivenessDb(db, 1); await closeCheckpointDb(db, 1);
}
await db?.$client.end({ timeout: 1 }); await temporary?.cleanup();
});
async function fixture() {
const companyId = randomUUID(), actorId = "live-recovery-user";
await db.insert(authUsers).values({ id: actorId, name: "Recovery owner", email: `${actorId}@example.test`,
createdAt: new Date(), updatedAt: new Date() }).onConflictDoNothing();
await db.insert(companies).values({ id: companyId, name: "Live recovery", issuePrefix: `LR${companyId.slice(0, 6)}`,
requireBoardApprovalForNewAgents: false, defaultResponsibleUserId: actorId });
await db.insert(companyMemberships).values({ companyId, principalType: "user", principalId: actorId,
status: "active", membershipRole: "owner" });
const [agent] = await db.insert(agents).values({ companyId, name: "Recovery process", status: "idle", adapterType: "process",
adapterConfig: { command: process.execPath, args: ["-e", "console.log('live-recovery-proof')"], cwd: process.cwd() },
runtimeConfig: { heartbeat: { enabled: true, intervalSec: 0, wakeOnDemand: true } } }).returning();
const [issue] = await db.insert(issues).values({ companyId, title: "Recovered work", status: "done", assigneeAgentId: agent.id }).returning();
return { companyId, agent, issue, actorId };
}
async function start(realClock = false) {
if (!realClock && !vi.isFakeTimers()) vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"] });
const admitted = () => !draining && !standby;
await heartbeat.registerResidualRecoveryWork(coordinator, admitted).ready;
await heartbeat.registerActiveRunSilenceWork(coordinator, admitted).ready;
await heartbeat.registerIssueStateRecoveryWork(coordinator, admitted, { backstopMs: 0 }).ready;
await heartbeat.registerDeferredReceiptWork(coordinator, admitted).ready;
await heartbeat.registerRunDispatchWork(coordinator, admitted).ready;
await heartbeat.registerScheduledRetryCancellationWork(coordinator, admitted).ready;
}
async function advance(ms: number) {
if (!vi.isFakeTimers()) vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"] });
const until = Date.now() + ms;
while (Date.now() < until) {
await vi.advanceTimersByTimeAsync(Math.min(500, until - Date.now()));
await Promise.allSettled([...passes]); await immediate();
}
}
async function quiet() {
await heartbeat.drainActiveRunExecutions();
expect(await db.select().from(heartbeatRuns).where(sql`${heartbeatRuns.status} in ('queued', 'running')`)).toHaveLength(0);
const quietAt = Date.now();
if (vi.isFakeTimers()) await advance(31_000);
else { await realDelay(31_000); await Promise.allSettled([...passes]); }
console.info("quiet-drain", JSON.stringify({ quietAt, now: Date.now(), active: idleWorkSnapshot().active, next: [...deadlines].map(([q, at]) => [q, at()]), passes: Object.fromEntries([...deadlines.keys()].map(queue => [queue, timeline.filter(event => Array.isArray(event) && event[0] === "pass" && event[1] === queue).length])), settled: timeline.filter(event => Array.isArray(event) && event[0] === "settled") }));
await vi.waitFor(() => { expect(coordinator.nextWakeAt(), JSON.stringify([...deadlines].map(([name, at]) => [name, at()]))).toBeNull(); expect(idleWorkSnapshot().active).toBe(0); });
const queries: string[] = [], prior = db.$client.options.debug;
db.$client.options.debug = (_connection, query) => { queries.push(query); };
try { if (!vi.isFakeTimers()) vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"] }); await vi.advanceTimersByTimeAsync(20 * 60_000); await immediate(); expect(queries).toEqual([]); }
finally { db.$client.options.debug = prior; }
}
async function getRun(id: string) { return (await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, id)))[0]; }
async function getLease(id: string) { return (await db.select().from(environmentLeases).where(eq(environmentLeases.id, id)))[0]; }
it("leaves configured idle and completed native history without a timer or SQL", async () => {
const f = await fixture();
const runs = await db.insert(heartbeatRuns).values(Array.from({ length: 150 }, () => ({ companyId: f.companyId,
agentId: f.agent.id, status: "failed", runtimeMode: "native", nativeIssueId: f.issue.id, finishedAt: new Date() }))).returning();
await db.insert(nativeRunFinalizations).values(runs.map(run => ({ companyId: f.companyId, issueId: f.issue.id, runId: run.id, phase: "committed" })));
await start(); await quiet();
});
it("catches up native resultless retry debt through its real claim owner without replaying missing state", async () => {
const f = await fixture(); await db.update(issues).set({ status: "todo" }).where(eq(issues.id, f.issue.id));
const [run] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agent.id,
status: "failed", runtimeMode: "native", nativeIssueId: f.issue.id, contextSnapshot: { issueId: f.issue.id },
finishedAt: new Date(), errorCode: "original_native_failure" }).returning();
await db.insert(nativeRunFinalizations).values({ companyId: f.companyId, issueId: f.issue.id, runId: run.id,
phase: "retryable_failure", nextAttemptAt: new Date(Date.now() - 1_000) });
await start();
await vi.waitFor(async () => {
expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, run.id)))[0])
.toMatchObject({ phase: "terminal_failure", nextAttemptAt: null, leaseOwner: null });
});
expect((await getRun(run.id)).errorCode).toBe("original_native_failure");
expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issue.id)))
.toEqual([expect.objectContaining({ ownerType: "board", status: "active" })]);
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId))).toHaveLength(1);
await quiet();
});
it.each(["scoped remote retry", "startup"] as const)("reconstructs only the retry deadlines owed by %s", async mode => {
const selected = await fixture(), other = await fixture();
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"] });
const now = Date.now(), otherAt = now + 30_000, selectedAt = now + 60_000;
const seed = async (f: Awaited<ReturnType<typeof fixture>>, nextAttemptAt: number, remote = false) => {
await db.update(issues).set({ status: "todo" }).where(eq(issues.id, f.issue.id));
const [run] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agent.id,
status: "failed", runtimeMode: "native", nativeIssueId: f.issue.id, contextSnapshot: { issueId: f.issue.id },
finishedAt: new Date(), errorCode: "saved_native_failure",
runnerProfileJson: remote ? { nativeWorkspaceSync: { schema: "paperclip.native-workspace-sync/v1",
state: "prepared", descriptorSha256: "a".repeat(64), baselineSha256: "b".repeat(64),
workspaceId: randomUUID(), leaseId: randomUUID(), providerLeaseId: "remote-retry-fixture", remoteCwd: "/workspace" } } : null,
}).returning();
await db.insert(nativeRunFinalizations).values({ companyId: f.companyId, issueId: f.issue.id, runId: run.id,
phase: "retryable_failure", nextAttemptAt: new Date(nextAttemptAt) });
return run;
};
const selectedRun = await seed(selected, selectedAt, true), otherRun = await seed(other, otherAt);
await heartbeat.recoverNativeRunsAfterRestart(new Date(now), mode === "startup" ? undefined : [selectedRun.id]);
const queries: string[] = [], prior = db.$client.options.debug;
if (mode === "scoped remote retry") {
db.$client.options.debug = (_connection, query) => { queries.push(query); };
try { await advance(30_001); await immediate(); expect(queries).toEqual([]); }
finally { db.$client.options.debug = prior; }
expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, otherRun.id)))[0])
.toMatchObject({ phase: "retryable_failure", nextAttemptAt: new Date(otherAt) });
} else {
await advance(30_001);
await vi.waitFor(async () => expect((await db.select().from(nativeRunFinalizations)
.where(eq(nativeRunFinalizations.runId, otherRun.id)))[0].phase).toBe("terminal_failure"));
}
await advance(selectedAt - Date.now() + 1);
await vi.waitFor(async () => expect((await db.select().from(nativeRunFinalizations)
.where(eq(nativeRunFinalizations.runId, selectedRun.id)))[0].phase).toBe("terminal_failure"));
expect((await getRun(selectedRun.id)).errorCode).toBe("saved_native_failure");
expect(await db.select().from(heartbeatRuns)).toHaveLength(2);
await heartbeat.drainActiveRunExecutions();
expect(coordinator.nextWakeAt()).toBeNull();
});
it("consumes the durable native question cancellation from ordinary task completion", async () => {
const f = await fixture(); await db.update(issues).set({ status: "in_progress" }).where(eq(issues.id, f.issue.id));
const [run] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agent.id, status: "running",
runtimeMode: "native", nativeIssueId: f.issue.id, contextSnapshot: { issueId: f.issue.id }, nativePhase: "awaiting_input" }).returning();
await db.insert(nativeRunFinalizations).values({ companyId: f.companyId, issueId: f.issue.id, runId: run.id,
phase: "observed", recoveryState: "awaiting_evidence", leaseExpiresAt: new Date(Date.now() + 60_000) });
await db.insert(issueThreadInteractions).values({ companyId: f.companyId, issueId: f.issue.id, kind: "ask_user_questions",
sourceRunId: run.id, status: "pending", continuationPolicy: "none",
idempotencyKey: `paperclip-runner-question:${run.id}:question-1`, payload: {
version: 1, questions: [{ id: "continue", prompt: "Continue?", selectionMode: "single", required: true, allowOther: false, options: [{ id: "yes", label: "Yes" }] }],
questionSet: { schema: "paperclip.question_set.v1", questions: [{ id: "continue", prompt: "Continue?", required: true, answerMode: "single_select", options: [{ id: "yes", label: "Yes" }] }] }, runtimeRequestId: "question-1",
} });
await start(); await issueService(db).update(f.issue.id, { status: "done" });
await vi.waitFor(async () => expect((await getRun(run.id)).status).toBe("cancelled"), { timeout: 15_000 });
expect((await getRun(run.id)).resultJson).toMatchObject({ cancelledByIssueStatus: "done", cancelledIssueId: f.issue.id });
await quiet();
});
it.each([false, true])("finishes a question cancellation whose run fails into a pending native retry before consumption (partial Stop fault: %s)", async partialStopFault => {
const f = await fixture();
let stopReceipt: Record<string, unknown> | undefined;
const [run] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agent.id,
status: "running", runtimeMode: "native", nativeIssueId: f.issue.id, nativePhase: "awaiting_input",
contextSnapshot: { issueId: f.issue.id } }).returning();
await db.insert(nativeRunFinalizations).values({ companyId: f.companyId, issueId: f.issue.id,
runId: run.id, phase: "observed", recoveryState: "awaiting_evidence" });
if (partialStopFault) {
await db.execute(sql`create sequence question_cancel_failure_once`);
await db.execute(sql.raw(`create function question_cancel_failure_once() returns trigger language plpgsql as $$
begin if new.id = '${run.id}'::uuid and new.status = 'cancelled' and nextval('question_cancel_failure_once') = 1
then raise exception 'fixture question cancellation commit unavailable'; end if; return new; end $$`));
await db.execute(sql`create trigger question_cancel_failure_once before update on heartbeat_runs for each row execute function question_cancel_failure_once()`);
extraStops.push(async () => {
await db.execute(sql`drop trigger if exists question_cancel_failure_once on heartbeat_runs`);
await db.execute(sql`drop function if exists question_cancel_failure_once()`);
await db.execute(sql`drop sequence if exists question_cancel_failure_once`);
});
}
const marker = { version: 1, issueId: f.issue.id, kind: "issue_terminal", issueStatus: "done" };
const history = await db.insert(heartbeatRuns).values(Array.from({ length: 25 }, () => ({
companyId: f.companyId, agentId: f.agent.id, runtimeMode: "native", nativeIssueId: f.issue.id,
status: "failed", finishedAt: new Date(), nativePhase: "terminal_failure",
contextSnapshot: { nativeQuestionCancellation: marker }, resultJson: { nativeCancellation: {
schema: "paperclip.native-cancellation.v1", runId: randomUUID(), companyId: f.companyId, issueId: f.issue.id,
scope: "run", intentId: randomUUID(), intentAuditId: randomUUID(), dispatched: false, dispatchState: "pending",
} },
}))).returning();
await db.insert(nativeRunFinalizations).values(history.map(row => ({ companyId: f.companyId, issueId: f.issue.id,
runId: row.id, phase: "terminal_failure", failureCode: "native_retry_cancelled", recoveryState: "blocked" })));
draining = true; await start();
expect(await db.transaction(tx => requestNativeQuestionRunCancellation(tx, { companyId: f.companyId,
issueId: f.issue.id, sourceRunId: run.id, idempotencyKey: `paperclip-runner-question:${run.id}:raced-question`,
payload: { questionSet: {}, runtimeRequestId: "raced-question" },
}, { kind: "issue_terminal", issueStatus: "done" }))).toBe(run.id);
expect((await getRun(run.id)).contextSnapshot?.nativeQuestionCancellation).toMatchObject(marker);
await db.transaction(async tx => {
await notifyExecutionRunWork(tx, { companyId: f.companyId, runId: run.id, agentId: f.agent.id });
await tx.update(heartbeatRuns).set({ status: "failed", nativePhase: "retryable_failure", finishedAt: new Date() })
.where(eq(heartbeatRuns.id, run.id));
await tx.update(nativeRunFinalizations).set({ phase: "retryable_failure", recoveryState: "resuming_session",
nextAttemptAt: new Date(), leaseOwner: null, leaseExpiresAt: null }).where(eq(nativeRunFinalizations.runId, run.id));
});
draining = false; await advance(1_000);
if (partialStopFault) {
await vi.waitFor(() => expect(errors).toHaveLength(1));
expect(errors.shift()).toBeInstanceOf(AggregateError);
expect((await getRun(run.id)).status).toBe("failed");
const saved = await getRun(run.id);
stopReceipt = saved.resultJson?.nativeCancellation as Record<string, unknown>;
const [coordinator] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, run.id));
expect(coordinator).toMatchObject({ phase: "terminal_failure", failureCode: "native_retry_cancelled", resultId: null });
expect(nativeRetryCancellationEligible({ runId: run.id, companyId: f.companyId, issueId: f.issue.id,
resultJson: saved.resultJson }, coordinator)).toBe(true);
const [debt] = await db.select({ pending: pendingFailedNativeRetryCancellationCondition() }).from(heartbeatRuns)
.where(eq(heartbeatRuns.id, run.id));
expect(debt.pending).toBe(true);
}
await advance(30_000);
await vi.waitFor(async () => expect((await getRun(run.id)).status).toBe("cancelled"));
expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, run.id)))[0])
.toMatchObject({ phase: "terminal_failure", failureCode: "native_retry_cancelled", nextAttemptAt: null });
if (stopReceipt) expect((await getRun(run.id)).resultJson?.nativeCancellation).toMatchObject({
intentId: stopReceipt.intentId, intentAuditId: stopReceipt.intentAuditId,
acknowledgementAuditId: stopReceipt.acknowledgementAuditId,
});
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId))).toHaveLength(26);
expect(await db.select().from(heartbeatRunEvents).where(eq(heartbeatRunEvents.eventType, "native.process_start_requested"))).toEqual([]);
expect((await db.select().from(heartbeatRuns).where(sql`${heartbeatRuns.id} <> ${run.id}`)).every(row => row.status === "failed")).toBe(true);
await quiet();
});
it("preserves a fresh foreign controller until exact expiry and admits one process retry across two owners", async () => {
const f = await fixture(), expiry = new Date(Date.now() + 3_000);
const [run] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agent.id, status: "running",
runtimeMode: "legacy", responsibleUserId: f.actorId, controllerBootId: randomUUID(), controllerLeaseExpiresAt: expiry,
executionStage: "preparing", updatedAt: new Date(Date.now() - 6 * 60_000), startedAt: new Date(Date.now() - 6 * 60_000) }).returning();
await start(true);
const other = heartbeatService(db), competing = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) });
await other.registerResidualRecoveryWork(competing, () => true).ready; extraStops.push(() => competing.stop());
expect((await getRun(run.id)).status).toBe("running"); expect(coordinator.nextWakeAt()).toBeLessThanOrEqual(expiry.getTime());
await vi.waitFor(async () => expect((await getRun(run.id)).status).toBe("failed"), { timeout: 10_000 });
let retryId: string;
await vi.waitFor(async () => {
const retries = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, run.id));
expect(retries).toHaveLength(1); retryId = retries[0].id;
await heartbeat.drainActiveRunExecutions(); await other.drainActiveRunExecutions();
expect((await getRun(retryId)).status).toBe("succeeded");
expect((await getRun(retryId)).stdoutExcerpt).toContain("live-recovery-proof");
}, { timeout: 45_000 });
await competing.stop(); extraStops.pop(); await quiet();
});
it("recovers a fresh run from a prior boot at lease expiry instead of waiting for the stale-age threshold", async () => {
// RC-27: a Retry admitted just before a kill records a controller lease but
// no process. The prior boot's expired lease is the ownership proof.
const f = await fixture(), expiry = new Date(Date.now() + 3_000);
const [run] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agent.id, status: "running",
runtimeMode: "legacy", responsibleUserId: f.actorId, controllerBootId: randomUUID(), controllerLeaseExpiresAt: expiry,
executionStage: "preparing", updatedAt: new Date(), startedAt: new Date() }).returning();
await start(true);
expect((await getRun(run.id)).status).toBe("running");
await vi.waitFor(async () => expect((await getRun(run.id)).status).toBe("failed"), { timeout: 15_000 });
expect(Date.now() - expiry.getTime()).toBeLessThan(15_000);
expect(await getRun(run.id)).toMatchObject({ errorCode: "process_lost",
resultJson: expect.objectContaining({ executionRecovery: { kind: "bootstrap", providerWorkStarted: false } }) });
await vi.waitFor(async () => {
const retries = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, run.id));
expect(retries).toHaveLength(1);
await heartbeat.drainActiveRunExecutions();
expect((await getRun(retries[0].id)).status).toBe("succeeded");
}, { timeout: 45_000 });
await quiet();
});
it("keeps the age guard when the prior boot dispatched between the scan and the revocation", async () => {
// Stage the race: the scanned row says `preparing`, and the row committed
// with the revocation says `dispatching` (the old controller's last renewal).
const f = await fixture(), expiry = new Date(Date.now() + 2_000);
const [run] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agent.id, status: "running",
runtimeMode: "legacy", responsibleUserId: f.actorId, controllerBootId: randomUUID(), controllerLeaseExpiresAt: expiry,
executionStage: "preparing", updatedAt: new Date(), startedAt: new Date() }).returning();
await db.execute(sql.raw(`create or replace function rc27_dispatch_on_revoke() returns trigger language plpgsql as $$
begin if new.id = '${run.id}' and new.controller_boot_id is distinct from old.controller_boot_id
then new.execution_stage := 'dispatching'; end if; return new; end $$`));
await db.execute(sql`create trigger rc27_dispatch_on_revoke before update on heartbeat_runs
for each row execute function rc27_dispatch_on_revoke()`);
extraStops.push(async () => {
await db.execute(sql`drop trigger if exists rc27_dispatch_on_revoke on heartbeat_runs`);
await db.execute(sql`drop function if exists rc27_dispatch_on_revoke()`);
});
await start(true);
await vi.waitFor(async () => expect((await getRun(run.id)).controllerBootId).not.toBe(run.controllerBootId), { timeout: 15_000 });
await realDelay(5_000); await Promise.allSettled([...passes]);
expect(await getRun(run.id)).toMatchObject({ status: "running", executionStage: "dispatching", errorCode: null });
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, run.id))).toEqual([]);
});
// A recorded process is judged by the shared prover, not by age: a live one
// is held (a proven-dead one settles at once, recovery-stale-issue-lock-sweep).
it("keeps the stale-age guard for this boot or a dispatched adapter, and holds a live recorded process", async () => {
const f = await fixture(), now = Date.now(), expired = new Date(now - 1_000);
const [sameBoot] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agent.id, status: "running",
runtimeMode: "legacy", responsibleUserId: f.actorId, controllerBootId: legacyControllerBootId, controllerLeaseExpiresAt: expired,
executionStage: "preparing", updatedAt: new Date(now), startedAt: new Date(now) }).returning();
const [withProcess] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agent.id, status: "running",
runtimeMode: "legacy", responsibleUserId: f.actorId, controllerBootId: randomUUID(), controllerLeaseExpiresAt: expired,
executionStage: "dispatching", processPid: process.pid, processHostHash: currentProcessSpaceHash(),
updatedAt: new Date(now), startedAt: new Date(now) }).returning();
// A dispatched adapter may hold remote work without a local process.
const [dispatched] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agent.id, status: "running",
runtimeMode: "legacy", responsibleUserId: f.actorId, controllerBootId: randomUUID(), controllerLeaseExpiresAt: expired,
executionStage: "dispatching", updatedAt: new Date(now), startedAt: new Date(now) }).returning();
await start(); await advance(4 * 60_000);
expect((await getRun(sameBoot.id)).status).toBe("running");
expect((await getRun(withProcess.id)).status).toBe("running");
expect((await getRun(dispatched.id)).status).toBe("running");
});
it.each(["confirmed", "already_retired", "missing_terminal", "unproven_terminal", "ack_failure", "late_terminal"] as const)(
"settles an API-only Stop after deferred controller retirement without inventing proof: %s", async mode => {
const f = await fixture(), runId = randomUUID(), sessionId = randomUUID(), runnerId = randomUUID();
const statusEvents: LiveEvent[] = [];
const unsubscribe = subscribeCompanyLiveEvents(f.companyId, event => {
if (event.type === "heartbeat.run.status" && event.payload.runId === runId) statusEvents.push(event);
});
extraStops.push(async () => { unsubscribe(); });
const child = spawn(process.execPath, ["-e", "setInterval(() => {}, 1000)"], { detached: true, stdio: "ignore" });
await once(child, "spawn");
const stopChild = async () => {
if (child.exitCode !== null || child.signalCode !== null) return;
const exited = once(child, "exit"); child.kill("SIGTERM"); await exited;
};
extraStops.push(stopChild);
const recordedAt = new Date().toISOString();
const nativeExecutionInput = buildNativeExecutionInput({ companyId: f.companyId, agentId: f.agent.id, runId,
provider: "cursor_cloud", model: "explicit-model", cursorCloud: { schema: "paperclip.cursor-cloud.config.v1",
repos: [{ url: "https://github.com/example/qualification" }] },
issue: { id: f.issue.id, identifier: "API-1", title: f.issue.title, description: null, workMode: "standard" },
taskPrompt: "Stop the foreground command", normalizedSessionId: sessionId,
workspace: { id: randomUUID(), cwd: "/private/controller", repoUrl: null, repoRef: null, branchName: null },
completionContract: { id: randomUUID(), sha256: `sha256:${"a".repeat(64)}`, schemaVersion: "paperclip.run-result.v1",
contract: { revision: "1", objective: "Stop the foreground command", criteria: [{ id: "stop", requirement: "Stop the foreground command" }] } },
runtimeContext: nativeRuntimeContextFixture(),
});
await db.insert(heartbeatRuns).values({ id: runId, companyId: f.companyId, agentId: f.agent.id,
runtimeMode: "native", driverKind: "cursor_cloud_api", nativeIssueId: f.issue.id, nativeSessionId: sessionId,
runnerInstanceId: runnerId, status: "cancelled", finishedAt: new Date(),
runnerProfileJson: { backend: "cursor_cloud_api", nativeExecutionInput, sessionCheckpoint: { providerSessionId: "cloud-agent" },
executionPlacement: { kind: "provider_api", provider: "cursor_cloud", controller: "paperclip" } },
resultJson: { executionCancellation: { state: "requested", requestedAt: recordedAt },
cancelledByActorType: "user", cancelledByUserId: f.actorId, nativeCancellation: {
schema: "paperclip.native-cancellation.v1", companyId: f.companyId, runId, issueId: f.issue.id,
scope: "run", reasonCode: "cancellation_run_only", dispatchState: "acknowledged", dispatched: true,
effects: ["release_run_resources"], recordedAt, acknowledgedAt: recordedAt,
intentAuditId: randomUUID(), acknowledgementAuditId: randomUUID() } } });
await persistHeartbeatRunProcessMetadata(db, runId, { pid: child.pid!, processGroupId: child.pid!, startedAt: recordedAt, targetKind: "local" });
await db.insert(nativeRunFinalizations).values({ companyId: f.companyId, issueId: f.issue.id, runId, phase: "committed" });
const appendTerminal = async (target: typeof db = db) => {
const terminal = { schema: "paperclip.prp.event.v1", schemaVersion: 1, eventType: "turn.interrupted",
sourceKind: "runner", sourceInstanceId: runnerId, sourceEventId: `${runnerId}:${runId}:1`, sourceSeq: 1,
runId, turnId: "stopped-turn", normalizedSessionId: sessionId, emittedAt: new Date().toISOString(), priority: 1,
payload: { status: "interrupted", error: null, ...(mode === "unproven_terminal" ? { providerTerminalObserved: false } : {}) } };
await appendHeartbeatRunEvent(target, { companyId: f.companyId, agentId: f.agent.id, runId,
eventType: terminal.eventType, payload: { prpEvent: terminal }, nativeSource: {
sourceInstanceId: runnerId, sourceEventId: terminal.sourceEventId, sourceSeq: 1,
protocolSchemaVersion: 1, canonicalPayload: terminal } });
};
if (mode !== "missing_terminal" && mode !== "late_terminal") await appendTerminal();
expect(await db.select().from(environmentLeases).where(eq(environmentLeases.heartbeatRunId, runId))).toEqual([]);
expect(await settleNativeRunProcess(db, runId)).toBe("pending");
expect((await getRun(runId)).resultJson?.executionCancellation).toMatchObject({ state: "requested" });
expect(statusEvents).toEqual([]);
if (mode === "already_retired") { await stopChild(); expect(await releaseSettledNativeRunProcess(db, runId)).toBe("released"); }
await start();
if (mode !== "already_retired") {
await vi.waitFor(() => expect(deadlines.get(DELIVERY_QUEUES.settledNativeProcesses)?.()).toBeGreaterThan(Date.now()));
expect((await getRun(runId)).processPid).toBe(child.pid);
await stopChild();
}
const failure = new Error("synthetic Stop proof read failure after controller retirement");
if (mode === "ack_failure") vi.spyOn(nativeStopEvidence, "acknowledgedNativeApiStopExecutionHasStopped").mockRejectedValueOnce(failure);
await advance(31_000);
if (mode === "ack_failure") {
expect((await getRun(runId)).processPid).toBeNull();
expect((await getRun(runId)).resultJson?.executionCancellation).toMatchObject({ state: "requested" });
expect(errors).toHaveLength(1);
expect(errors[0]).toBeInstanceOf(AggregateError);
expect((errors[0] as AggregateError).errors).toEqual([failure]); errors.length = 0;
expect(statusEvents).toEqual([]);
await advance(31_000);
}
const stopped = await getRun(runId);
expect(stopped).toMatchObject({ processPid: null, processGroupId: null, processStartedAt: null });
expect(await readNativeLocalProcessStop(db, f.companyId, runId)).toMatchObject({ processPid: child.pid, processGroupId: child.pid });
expect(stopped.resultJson?.executionCancellation).toMatchObject(["missing_terminal", "unproven_terminal", "late_terminal"].includes(mode)
? { state: "requested" } : { state: "acknowledged", proof: "provider_api_stop_receipt" });
expect(statusEvents).toHaveLength(["missing_terminal", "unproven_terminal", "late_terminal"].includes(mode) ? 0 : 1);
await quiet();
if (mode === "late_terminal") {
const rollback = new Error("rollback the uncommitted terminal receipt");
await expect(db.transaction(async tx => { await appendTerminal(tx as unknown as typeof db); throw rollback; })).rejects.toBe(rollback);
await advance(1_000);
expect((await getRun(runId)).resultJson?.executionCancellation).toMatchObject({ state: "requested" });
expect(statusEvents).toEqual([]);
await quiet();
let commit!: () => void, entered!: () => void;
const gate = new Promise<void>(resolve => { commit = resolve; });
const recorded = new Promise<void>(resolve => { entered = resolve; });
const transaction = db.transaction(async tx => { await appendTerminal(tx as unknown as typeof db); entered(); await gate; });
await recorded;
try {
await vi.advanceTimersByTimeAsync(500); await immediate();
expect((await getRun(runId)).resultJson?.executionCancellation).toMatchObject({ state: "requested" });
expect(statusEvents).toEqual([]);
expect(await db.select().from(heartbeatRunEvents).where(and(eq(heartbeatRunEvents.runId, runId),
eq(heartbeatRunEvents.eventType, "turn.interrupted")))).toEqual([]);
} finally { commit(); await transaction; }
await vi.waitFor(async () => expect((await getRun(runId)).resultJson?.executionCancellation)
.toMatchObject({ state: "acknowledged", proof: "provider_api_stop_receipt" }));
await quiet();
expect(statusEvents).toHaveLength(1);
expect(statusEvents[0]).toMatchObject({ companyId: f.companyId, type: "heartbeat.run.status",
payload: { runId, agentId: f.agent.id, status: "cancelled" } });
expect(statusEvents[0]!.payload).not.toHaveProperty("resultJson");
}
},
);
it("releases a settled native process only after matching host and birth evidence arrives", async () => {
const f = await fixture(), startedAt = new Date();
const child = spawn(process.execPath, ["-e", "console.log('identity-proof')"], { stdio: "ignore" });
const pid = child.pid!; await once(child, "exit");
const [run] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agent.id, runtimeMode: "native",
nativeIssueId: f.issue.id, status: "failed", finishedAt: new Date(), processPid: pid, processStartedAt: startedAt,
runnerProfileJson: { sessionCheckpoint: { sessionId: "preserved-native-session" } } }).returning();
await db.insert(nativeRunFinalizations).values({ companyId: f.companyId, issueId: f.issue.id, runId: run.id, phase: "committed" });
await appendHeartbeatRunEvent(db, { companyId: f.companyId, agentId: f.agent.id, runId: run.id,
eventType: "native.process_identity_recorded", payload: { targetKind: "local", hostname: os.hostname(),
processPid: pid, processGroupId: null, processStartedAt: new Date(startedAt.getTime() - 1).toISOString() } });
await start(); expect((await getRun(run.id)).processPid).toBe(pid);
await appendHeartbeatRunEvent(db, { companyId: f.companyId, agentId: f.agent.id, runId: run.id,
eventType: "native.process_identity_recorded", payload: { targetKind: "local", hostname: os.hostname(),
processPid: pid, processGroupId: null, processStartedAt: startedAt.toISOString() } });
await vi.waitFor(async () => expect((await getRun(run.id)).processPid).toBeNull(), { timeout: 40_000 });
expect(await db.select().from(heartbeatRunEvents).where(and(eq(heartbeatRunEvents.runId, run.id),
eq(heartbeatRunEvents.eventType, "native.local_process_stopped")))).toHaveLength(1);
await quiet();
});
it("excludes unverifiable local birth history until actual process identity arrives", async () => {
const f = await fixture(), startedAt = new Date();
const child = spawn(process.execPath, ["-e", "console.log('legacy-identity-proof')"], { stdio: "ignore" });
const pid = child.pid!; await once(child, "exit");
const [run] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agent.id,
runtimeMode: "native", nativeIssueId: f.issue.id, status: "failed", finishedAt: new Date(), processPid: pid,
runnerProfileJson: { sessionCheckpoint: { sessionId: "legacy-unverifiable-session" } } }).returning();
await db.insert(nativeRunFinalizations).values({ companyId: f.companyId, issueId: f.issue.id, runId: run.id, phase: "committed" });
const checks: string[] = [], prior = db.$client.options.debug;
db.$client.options.debug = (_connection, query) => {
if (query.includes("sessionCheckpoint") && query.includes("for update")) checks.push(query);
};
try {
await start();
expect(checks).toEqual([]);
await db.transaction(tx => notifyLiveRunRecovery(tx, { companyId: f.companyId }));
await advance(31_000); expect(checks).toEqual([]);
} finally { db.$client.options.debug = prior; }
expect(await getRun(run.id)).toMatchObject({ processPid: pid, processStartedAt: null });
await quiet();
await persistHeartbeatRunProcessMetadata(db, run.id, { pid, processGroupId: null, startedAt: startedAt.toISOString(), targetKind: "local" });
await advance(1_000);
await vi.waitFor(async () => expect((await getRun(run.id)).processPid).toBeNull());
expect(await db.select().from(heartbeatRunEvents).where(and(eq(heartbeatRunEvents.runId, run.id),
eq(heartbeatRunEvents.eventType, "native.local_process_stopped")))).toHaveLength(1);
await quiet();
});
it("preserves legacy null-birth recovery with exact server identity and an absent local PID group", async () => {
const f = await fixture();
const child = spawn(process.execPath, ["-e", "console.log('legacy-group-proof')"], { detached: true, stdio: "ignore" });
const pid = child.pid!; await once(child, "exit");
const [run] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agent.id,
runtimeMode: "native", nativeIssueId: f.issue.id, status: "failed", finishedAt: new Date(),
processPid: pid, processGroupId: pid, runnerProfileJson: { sessionCheckpoint: { sessionId: "legacy-exact-session" } } }).returning();
await db.insert(nativeRunFinalizations).values({ companyId: f.companyId, issueId: f.issue.id, runId: run.id, phase: "committed" });
await appendHeartbeatRunEvent(db, { companyId: f.companyId, agentId: f.agent.id, runId: run.id,
eventType: "native.process_identity_recorded", payload: { targetKind: "local", hostname: os.hostname(),
processPid: pid, processGroupId: pid } });
await start();
expect(await getRun(run.id)).toMatchObject({ processPid: null, processGroupId: null, processStartedAt: null });
expect(await readNativeLocalProcessStop(db, f.companyId, run.id)).toEqual({ processPid: pid, processGroupId: pid });
await quiet();
});
it("releases remote process identity without local birth after the exact provider stop receipt", async () => {
const f = await fixture();
const [run] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agent.id,
runtimeMode: "native", nativeIssueId: f.issue.id, status: "failed", finishedAt: new Date(), processPid: 12345,
runnerProfileJson: { sessionCheckpoint: { sessionId: "remote-no-local-birth" } } }).returning();
await db.insert(nativeRunFinalizations).values({ companyId: f.companyId, issueId: f.issue.id, runId: run.id, phase: "committed" });
const [lease] = await db.insert(environmentLeases).values({ companyId: f.companyId, heartbeatRunId: run.id,
provider: "daytona", providerLeaseId: randomUUID(), status: "released", releasedAt: new Date() }).returning();
await start(); expect((await getRun(run.id)).processPid).toBe(12345);
const receipt = remoteTerminationReceipt(lease, { providerLeaseId: lease.providerLeaseId, state: "stopped" });
expect(receipt).toBeDefined();
await writeEnvironmentCleanup(db, tx => tx.update(environmentLeases).set({ cleanupStatus: "success",
metadata: { remoteExecutionTermination: receipt } }).where(eq(environmentLeases.id, lease.id)),
{ companyId: f.companyId, runId: run.id, leaseId: lease.id });
await advance(1_000);
await vi.waitFor(async () => expect((await getRun(run.id)).processPid).toBeNull());
expect((await getRun(run.id)).processStartedAt).toBeNull();
expect(await db.select().from(heartbeatRunEvents).where(and(eq(heartbeatRunEvents.runId, run.id),
eq(heartbeatRunEvents.eventType, "native.local_process_stopped")))).toEqual([]);
await quiet();
});
it("renews without waking recovery and re-reads the moved controller deadline at the old one", async () => {
const f = await fixture(), old = new Date(Date.now() - 6 * 60_000), expiry = new Date(Date.now() + 20_000);
const [run] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agent.id,
runtimeMode: "legacy", status: "running", controllerBootId: legacyControllerBootId,
controllerLeaseExpiresAt: expiry, executionStage: "preparing", startedAt: old, updatedAt: old,
lastOutputAt: new Date() }).returning();
await start(); expect(deadlines.get("active-execution-recovery")!()).toBe(expiry.getTime());
const before = timeline.length;
expect(await renewLegacyControllerLease(db, run)).toBe(true);
const moved = (await getRun(run.id)).controllerLeaseExpiresAt!;
expect(moved.getTime()).toBeGreaterThan(expiry.getTime());
await advance(1_000);
expect(timeline.slice(before).filter(event => Array.isArray(event) && event[0] === "pass")
.map(event => (event as unknown[])[1])).toEqual([]);
await advance(expiry.getTime() - Date.now());
expect((await getRun(run.id)).status).toBe("running");
expect(deadlines.get("active-execution-recovery")!()).toBe(moved.getTime());
await heartbeat.cancelRun(run.id, "Operator cancelled the controller renewal fixture");
await quiet();
});
it("renews a native execution lease without scheduling recovery and still finds the run at the end of its last lease", async () => {
// Another server owns the run. Its renewals must not make this server's
// recovery re-check the healthy run, and when they stop, recovery acts at
// the expiry of the last renewed lease, as it would have before.
const f = await fixture(), old = new Date(Date.now() - 6 * 60_000);
const owner = { bootId: randomUUID(), pid: process.pid + 100_000, processStartedAt: old };
const [run] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agent.id,
status: "running", runtimeMode: "native", nativeIssueId: f.issue.id, contextSnapshot: { issueId: f.issue.id },
startedAt: old, updatedAt: old }).returning();
await db.insert(nativeRunFinalizations).values({ companyId: f.companyId, issueId: f.issue.id, runId: run.id,
phase: "observed", attempt: 1, leaseOwner: "other-server", controllerBootId: owner.bootId, controllerPid: owner.pid,
controllerProcessStartedAt: old, leaseExpiresAt: sql`clock_timestamp() + interval '3 seconds'` });
const savedExpiry = async () => (await db.select().from(nativeRunFinalizations)
.where(eq(nativeRunFinalizations.runId, run.id)))[0].leaseExpiresAt!.getTime();
const expiry = await savedExpiry();
await start(true);
await vi.waitFor(() => expect(deadlines.get("active-execution-recovery")!()).toBe(expiry));
await Promise.allSettled([...passes]);
const before = timeline.length;
// Renew through this client so a recovery signal, if any, reaches the
// registered owners in-process, as it reaches peers through notifications.
try {
await renewNativeSessionExecutionLease({ db, runId: run.id, companyId: f.companyId, issueId: f.issue.id,
leaseOwner: "other-server", attempt: 1, controller: owner, leaseTtlMs: 6_000 });
} finally {
// Only this test uses the private liveness pool; end it while timers are real.
await livenessDb(db).$client.end({ timeout: 1 });
}
const moved = await savedExpiry();
expect(moved).toBeGreaterThan(expiry);
await realDelay(1_000);
expect(timeline.slice(before).filter(event => Array.isArray(event) && event[0] === "pass")).toEqual([]);
// At the old deadline recovery re-reads the row and moves to the new one.
await realDelay(Math.max(0, expiry - Date.now() + 500));
await Promise.allSettled([...passes]);
expect(await getRun(run.id)).toMatchObject({ status: "running", errorCode: null });
expect(deadlines.get("active-execution-recovery")!()).toBe(moved);
// The owner stopped renewing: recovery acts at the end of the last lease.
const acted = async () => {
const current = await getRun(run.id);
const [coordinator] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, run.id));
return current.status !== "running" || current.errorCode !== null || coordinator.recoveryState !== null
|| coordinator.leaseOwner !== "other-server";
};
expect(await acted()).toBe(false);
await vi.waitFor(async () => expect(await acted()).toBe(true), { timeout: 15_000 });
expect(Date.now()).toBeGreaterThanOrEqual(moved);
expect(Date.now() - moved).toBeLessThan(10_000);
await Promise.allSettled([...passes]);
await db.transaction(async tx => {
await notifyExecutionRunWork(tx, { companyId: f.companyId, runId: run.id, agentId: f.agent.id });
await tx.update(heartbeatRuns).set({ status: "cancelled", finishedAt: new Date() }).where(eq(heartbeatRuns.id, run.id));
await tx.update(nativeRunFinalizations).set({ phase: "terminal_failure", recoveryState: "blocked", leaseExpiresAt: null })
.where(eq(nativeRunFinalizations.runId, run.id));
});
await quiet();
});
it("recovers orphan active leases after restart, including a deleted heartbeat reference, without starving behind 100 guarded rows", async () => {
const f = await fixture(), foreign = await fixture(), resource = randomUUID(), old = new Date(Date.now() - 6 * 60_000);
await db.insert(environmentLeases).values({ companyId: foreign.companyId, status: "retained", provider: "fixture", providerLeaseId: resource, updatedAt: old });
const guarded = await db.insert(environmentLeases).values(Array.from({ length: 105 }, () => ({ companyId: f.companyId,
status: "active", provider: "fixture", providerLeaseId: resource, updatedAt: old }))).returning();
const [deleted] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agent.id, status: "succeeded", finishedAt: old }).returning();
const [orphan] = await db.insert(environmentLeases).values({ companyId: f.companyId, heartbeatRunId: deleted.id,
status: "active", provider: "local", updatedAt: old }).returning();
await db.delete(heartbeatRuns).where(eq(heartbeatRuns.id, deleted.id));
expect((await getLease(orphan.id)).heartbeatRunId).toBeNull(); await start();
await vi.waitFor(async () => expect((await getLease(orphan.id)).status).toBe("pending_cleanup"), { timeout: 15_000 });
expect(await db.select().from(environmentLeases).where(and(eq(environmentLeases.companyId, f.companyId),
eq(environmentLeases.status, "pending_cleanup")))).toHaveLength(1);
for (const row of guarded) expect((await getLease(row.id)).status).toBe("active");
await quiet();
});
it("reactivates an orphan when a different company's physical resource owner releases it", async () => {
const f = await fixture(), other = await fixture(), old = new Date(Date.now() - 6 * 60_000), resource = randomUUID();
const [run] = await db.insert(heartbeatRuns).values({ companyId: other.companyId, agentId: other.agent.id, status: "succeeded", finishedAt: old }).returning();
const [owner] = await db.insert(environmentLeases).values({ companyId: other.companyId, heartbeatRunId: run.id,
status: "retained", provider: "fixture", providerLeaseId: resource, updatedAt: old }).returning();
const [orphan] = await db.insert(environmentLeases).values({ companyId: f.companyId, status: "active",
provider: "fixture", providerLeaseId: resource, updatedAt: old }).returning();
await start(); expect((await getLease(orphan.id)).status).toBe("active");
await writeEnvironmentCleanup(db, tx => tx.update(environmentLeases).set({ status: "released", releasedAt: new Date() })
.where(eq(environmentLeases.id, owner.id)), { companyId: other.companyId, runId: run.id, leaseId: owner.id });
await vi.waitFor(async () => expect((await getLease(orphan.id)).status).toBe("pending_cleanup"), { timeout: 40_000 });
await quiet();
});
async function shutdownSource(released: boolean, cleanupExpiry?: number) {
const f = await fixture(), old = new Date(Date.now() - 6 * 60_000);
const [run] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agent.id, issueId: f.issue.id,
status: "failed", runtimeMode: "legacy", contextSnapshot: { issueId: f.issue.id }, finishedAt: old }).returning();
let [lease] = await db.insert(environmentLeases).values({ companyId: f.companyId, heartbeatRunId: run.id,
issueId: f.issue.id, status: "active", provider: "fixture", providerLeaseId: randomUUID(), acquiredAt: old, updatedAt: old,
metadata: { driver: "sandbox", sandboxProviderPlugin: true, pluginId: "shutdown-fixture" } }).returning();
await beginLegacyShutdownWorkspaceSettlement(db, run, new Date(Date.now() - 1000));
if (released) {
const prepared = await prepareSandboxStopAndRetain(db, await getLease(lease.id));
expect(prepared).toBeTruthy();
const settled = await settleStopOnlyCleanup(db, prepared!, { attemptId: String(prepared!.metadata?.pendingCleanupAttemptId),
receipt: { providerLeaseId: lease.providerLeaseId, state: "stopped" } });
expect(settled).toBeTruthy(); lease = settled!;
expect(hasConfirmedSandboxStopAndRetain(lease)).toBe(true);
} else {
[lease] = await db.update(environmentLeases).set({ status: "pending_cleanup", cleanupStatus: "failed",
metadata: sql`${environmentLeases.metadata} || ${JSON.stringify({ pendingCleanupInFlight: true,
...(cleanupExpiry === undefined ? {} : { pendingCleanupLeaseExpiresAtMs: cleanupExpiry }) })}::jsonb`,
}).where(eq(environmentLeases.id, lease.id)).returning();
}
return { ...f, run, lease };
}
it.each(["competing allocation lease", "competing allocation run", "cleanup in flight"] as const)("parks unchanged shutdown refusal for %s until its owning commit", async gate => {
const competingGate = gate !== "cleanup in flight";
const f = await shutdownSource(competingGate);
const [otherRun] = gate === "competing allocation run" ? await db.insert(heartbeatRuns).values({
companyId: f.companyId, agentId: f.agent.id, status: "failed", runtimeMode: "legacy", finishedAt: new Date(),
}).returning() : [];
const [competing] = competingGate ? await db.insert(environmentLeases).values({ companyId: f.companyId,
heartbeatRunId: otherRun?.id,
status: "pending_cleanup", provider: f.lease.provider, providerLeaseId: f.lease.providerLeaseId,
acquiredAt: new Date(f.lease.acquiredAt.getTime() - 1000) }).returning() : [];
await start();
expect((await getLease(f.lease.id)).metadata?.legacyShutdownWorkspaceSettlement).toMatchObject({ state: "pending" });
await quiet();
const releaseGate = async (database: typeof db) => writeEnvironmentCleanup(database, tx => tx.update(environmentLeases).set(
competing ? { status: "released", releasedAt: new Date() }
: { metadata: sql`${environmentLeases.metadata} || '{"pendingCleanupInFlight":false}'::jsonb` },
).where(eq(environmentLeases.id, competing?.id ?? f.lease.id)), { companyId: f.companyId,
...(otherRun ? { runId: otherRun.id } : { leaseId: competing?.id ?? f.lease.id }) });
await expect(db.transaction(async tx => {
await releaseGate(tx as unknown as typeof db);
throw new Error("roll back shutdown gate release");
})).rejects.toThrow("roll back shutdown gate release");
await reconcileDatabaseWork(db, "legacy-shutdown-settlement"); await Promise.allSettled([...passes]); await immediate();
expect((await getLease(f.lease.id)).metadata?.legacyShutdownWorkspaceSettlement).toMatchObject({ state: "pending" });
await quiet();
await releaseGate(db);
await vi.waitFor(async () => {
await Promise.allSettled([...passes]);
const current = await getLease(f.lease.id);
expect(current.metadata?.legacyShutdownWorkspaceSettlement).toMatchObject({ state: "expired" });
expect(current.leasePolicy).toBe("retain_on_failure");
expect(current.metadata?.workspaceRestoreRecovery).toMatchObject({ runId: f.run.id });
expect(current.providerLeaseId).toBe(f.lease.providerLeaseId);
}, { timeout: 15_000 });
expect(await db.select().from(issueRecoveryActions).where(and(eq(issueRecoveryActions.companyId, f.companyId),
eq(issueRecoveryActions.fingerprint, `legacy-execution:${f.run.id}`)))).toHaveLength(1);
await reconcileDatabaseWork(db, "legacy-shutdown-settlement"); await Promise.allSettled([...passes]); await immediate();
await quiet();
});
it("retains the actual cleanup claim deadline after shutdown expiry and honors a moved expiry", async () => {
const initial = Date.now() + 1500, f = await shutdownSource(false, initial);
await start();
expect(deadlines.get("legacy-shutdown-settlement")!()).toBe(initial);
const moved = Date.now() + 4000;
await writeEnvironmentCleanup(db, tx => tx.update(environmentLeases).set({
metadata: sql`jsonb_set(${environmentLeases.metadata}, '{pendingCleanupLeaseExpiresAtMs}', ${JSON.stringify(moved)}::jsonb)`,
}).where(eq(environmentLeases.id, f.lease.id)), { companyId: f.companyId, leaseId: f.lease.id });
await advance(initial - Date.now());
expect((await getLease(f.lease.id)).metadata?.legacyShutdownWorkspaceSettlement).toMatchObject({ state: "pending" });
expect(deadlines.get("legacy-shutdown-settlement")!()).toBe(moved);
await advance(moved - Date.now() - 1);
expect((await getLease(f.lease.id)).metadata?.legacyShutdownWorkspaceSettlement).toMatchObject({ state: "pending" });
await advance(1);
await vi.waitFor(async () => expect((await getLease(f.lease.id)).metadata?.legacyShutdownWorkspaceSettlement)
.toMatchObject({ state: "expired" }));
expect((await getLease(f.lease.id)).metadata?.workspaceRestoreRecovery).toMatchObject({ runId: f.run.id });
await quiet();
});
it("expires the exact shutdown settlement marker into preserved recovery instead of inferring file export", async () => {
const f = await fixture(), old = new Date(Date.now() - 6 * 60_000);
const [run] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agent.id, issueId: f.issue.id,
status: "failed", runtimeMode: "legacy", contextSnapshot: { issueId: f.issue.id }, finishedAt: old }).returning();
const [lease] = await db.insert(environmentLeases).values({ companyId: f.companyId, heartbeatRunId: run.id,
issueId: f.issue.id, status: "active", provider: "fixture", providerLeaseId: randomUUID(), updatedAt: old,
metadata: { driver: "sandbox" } }).returning();
await beginLegacyShutdownWorkspaceSettlement(db, run, new Date(Date.now() + 1_500)); await start();
expect((await getLease(lease.id)).metadata?.legacyShutdownWorkspaceSettlement).toMatchObject({ state: "pending" });
await vi.waitFor(async () => expect((await getLease(lease.id)).metadata?.legacyShutdownWorkspaceSettlement)
.toMatchObject({ state: "expired" }), { timeout: 10_000 });
expect((await getLease(lease.id)).leasePolicy).toBe("retain_on_failure");
expect((await getLease(lease.id)).metadata?.workspaceRestoreRecovery).toBeTruthy();
expect((await getLease(lease.id)).providerLeaseId).toBe(lease.providerLeaseId);
expect((await getLease(lease.id)).metadata).toMatchObject({ workspaceRestoreRecovery: { schema: "paperclip.workspace-restore-recovery.v1", runId: run.id } });
await quiet();
});
it("settles stranded agent status at its saved grace", async () => {
const f = await fixture(), old = new Date(Date.now() - 6 * 60_000);
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agent.id, status: "succeeded", finishedAt: old });
await db.transaction(async tx => {
await notifyStrandedAgentStatus(tx, { companyId: f.companyId, agentId: f.agent.id });
await tx.update(agents).set({ status: "running", updatedAt: old }).where(eq(agents.id, f.agent.id));
});
await start(); await vi.waitFor(async () => expect((await db.select().from(agents).where(eq(agents.id, f.agent.id)))[0].status).toBe("idle"));
await quiet();
});
it("retains only skipped agent lock work until a lock-only transaction releases it", async () => {
const f = await fixture(), old = new Date(Date.now() - 6 * 60_000);
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agent.id, status: "succeeded", finishedAt: old });
await db.update(agents).set({ status: "running", updatedAt: old }).where(eq(agents.id, f.agent.id));
let release!: () => void, locked!: () => void;
const held = new Promise<void>(resolve => { release = resolve; });
const ready = new Promise<void>(resolve => { locked = resolve; });
const lock = db.transaction(async tx => {
await tx.select({ id: agents.id }).from(agents).where(eq(agents.id, f.agent.id)).for("update");
locked(); await held;
});
try {
await ready; await start();
expect(deadlines.get("stranded-agent-status")!()).toBe(Date.now() + 30_000);
expect((await db.select().from(agents).where(eq(agents.id, f.agent.id)))[0].status).toBe("running");
} finally { release(); await lock; }
await advance(29_999);
expect((await db.select().from(agents).where(eq(agents.id, f.agent.id)))[0].status).toBe("running");
await advance(1);
await vi.waitFor(async () => expect((await db.select().from(agents).where(eq(agents.id, f.agent.id)))[0].status).toBe("idle"));
expect(await db.select().from(activityLog).where(and(eq(activityLog.entityId, f.agent.id),
eq(activityLog.action, "agent.status_reconciled")))).toHaveLength(1);
await quiet();
});
it("moves the silence threshold on actual output and folds same-run completion only at strict snooze expiry", async () => {
const f = await fixture(), now = new Date(), old = new Date(now.getTime() - 10 * 60_000);
await db.update(issues).set({ status: "in_progress" }).where(eq(issues.id, f.issue.id));
await db.update(agents).set({ status: "running" }).where(eq(agents.id, f.agent.id));
const [run] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agent.id,
status: "running", runtimeMode: "legacy", contextSnapshot: { issueId: f.issue.id },
responsibleUserId: f.actorId, startedAt: old, processStartedAt: old, updatedAt: old,
controllerBootId: randomUUID(), controllerLeaseExpiresAt: new Date(now.getTime() + 60 * 60_000) }).returning();
const output = createHeartbeatRunOutput(db, { run, runId: run.id, issueId: f.issue.id,
runLogStore: { begin: async () => ({ store: "local_file", logRef: "fixture-output.ndjson" }), append: async (_handle, event) => Buffer.byteLength(event.chunk) },
identityRedactor: { chunk: (_stream, chunk) => chunk, redact: value => value }, secretKeys: new Set(),
appendRunEvent: async (_run, event) => { await appendHeartbeatRunEvent(db, { companyId: f.companyId, agentId: f.agent.id, runId: run.id, ...event }); },
getCurrentUserRedactionOptions: async () => ({ enabled: false }) });
await (await output.begin(run)).onLog("stdout", "fresh actual output");
const progressed = await getRun(run.id), snoozedUntil = new Date(progressed.lastOutputAt!.getTime() + 5 * 60_000 + 2_000);
expect(progressed.lastOutputSeq).toBe(1); expect(progressed.lastOutputAt!.getTime()).toBeGreaterThan(old.getTime());
await issueService(db).update(f.issue.id, { status: "done" });
await persistActivity(db, { companyId: f.companyId, actorType: "agent", actorId: f.agent.id, agentId: f.agent.id,
runId: run.id, entityType: "issue", entityId: f.issue.id, action: "issue.updated", details: { status: "done" } });
await recoveryService(db, { enqueueWakeup: heartbeat.wakeup }).recordWatchdogDecision({ runId: run.id,
actor: { type: "board" }, decision: "snooze", snoozedUntil, reason: "Wait for the exact saved deadline" });
await start();
await advance(snoozedUntil.getTime() - Date.now() - 1);
expect((await getRun(run.id)).status).toBe("running");
await advance(1);
await vi.waitFor(async () => expect((await getRun(run.id)).status).toBe("succeeded"));
expect((await getRun(run.id)).resultJson).toMatchObject({ sourceResolvedWatchdogFold: { sourceIssueStatus: "done", sameRunEvidenceKind: "activity" } });
await quiet();
});
it("re-reads a concurrently moved stranded-agent grace before settlement", async () => {
const f = await fixture(), initialAt = new Date();
await db.update(agents).set({ status: "running", updatedAt: initialAt }).where(eq(agents.id, f.agent.id));
await start(); await advance(60_000);
const moved = new Date();
await db.update(agents).set({ updatedAt: moved }).where(eq(agents.id, f.agent.id));
await advance(initialAt.getTime() + 5 * 60_000 - Date.now());
expect((await db.select().from(agents).where(eq(agents.id, f.agent.id)))[0].status).toBe("running");
await advance(moved.getTime() + 5 * 60_000 - Date.now());
await vi.waitFor(async () => expect((await db.select().from(agents).where(eq(agents.id, f.agent.id)))[0].status).toBe("idle"));
await quiet();
});
it("reaches a silent run behind more than 100 suppressed live runs and leaves only real controller deadlines", async () => {
const f = await fixture(), now = new Date(), old = new Date(now.getTime() - 20 * 60_000);
await db.update(issues).set({ status: "blocked" }).where(eq(issues.id, f.issue.id));
const expiry = new Date(now.getTime() + 60 * 60_000);
const suppressed = await db.insert(heartbeatRuns).values(Array.from({ length: 105 }, () => ({ companyId: f.companyId,
agentId: f.agent.id, status: "running", runtimeMode: "legacy", contextSnapshot: { issueId: f.issue.id },
startedAt: old, updatedAt: old, controllerBootId: randomUUID(), controllerLeaseExpiresAt: expiry }))).returning();
const [source] = await db.insert(issues).values({ companyId: f.companyId, status: "done", title: "Actual completed source", assigneeAgentId: f.agent.id }).returning();
const [target] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agent.id,
status: "running", runtimeMode: "legacy", contextSnapshot: { issueId: source.id },
startedAt: new Date(old.getTime() + 60_000), updatedAt: old,
controllerBootId: randomUUID(), controllerLeaseExpiresAt: expiry }).returning();
await db.insert(activityLog).values({ companyId: f.companyId, actorType: "agent", actorId: f.agent.id,
agentId: f.agent.id, runId: target.id, entityType: "issue", entityId: source.id, action: "issue.updated", details: { status: "done" } });
const broad = await recoveryService(db, { enqueueWakeup: heartbeat.wakeup }).scanSilentActiveRuns({ companyId: f.companyId });
expect(broad.scanned).toBe(100); expect((await getRun(target.id)).status).toBe("running");
await start(); await advance(31_000);
await vi.waitFor(async () => expect((await getRun(target.id)).status).toBe("succeeded"));
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.status, "running"))).toHaveLength(suppressed.length);
expect(deadlines.get("active-run-silence")!()).toBeNull();
const [agentState] = await db.select().from(agents).where(eq(agents.id, f.agent.id));
const grace = agentState.updatedAt.getTime() + 5 * 60_000;
expect(agentState.status).toBe("running");
expect(deadlines.get("stranded-agent-status")!()).toBe(grace);
await advance(grace - Date.now() + 1);
expect(deadlines.get("stranded-agent-status")!()).toBe(expiry.getTime());
expect(coordinator.nextWakeAt(), JSON.stringify([...deadlines].map(([name, at]) => [name, at()]))).toBe(expiry.getTime());
const queries: string[] = [], prior = db.$client.options.debug;
db.$client.options.debug = (_connection, query) => { queries.push(query); };
try { await advance(20 * 60_000); expect(queries).toEqual([]); }
finally { db.$client.options.debug = prior; }
});
it("reconciles nested rollback and dispatches only the outer committed orphan producer", async () => {
const f = await fixture(), old = new Date(Date.now() - 6 * 60_000), rolledBack = randomUUID();
await start(); await quiet();
await expect(db.transaction(async outer => {
await outer.transaction(async inner => {
await notifyLiveResourceRecovery(inner, { companyId: f.companyId, leaseId: rolledBack });
await inner.insert(environmentLeases).values({ id: rolledBack, companyId: f.companyId, status: "active", provider: "local", updatedAt: old });
});
throw new Error("abort outer producer");
})).rejects.toThrow("abort outer producer");
await advance(31_000); expect(await getLease(rolledBack)).toBeUndefined(); await quiet();
const committed = randomUUID();
await db.transaction(async outer => {
await outer.transaction(async inner => {
await notifyLiveResourceRecovery(inner, { companyId: f.companyId, leaseId: committed });
await inner.insert(environmentLeases).values({ id: committed, companyId: f.companyId, status: "active", provider: "local", updatedAt: old });
});
expect(await getLease(committed)).toBeUndefined();
});
await advance(1_000);
await vi.waitFor(async () => expect((await getLease(committed)).status).toBe("pending_cleanup"));
await quiet();
});
it("retains and re-reads a moved native controller deadline while awaiting recovery evidence", async () => {
const f = await fixture(), old = new Date(Date.now() - 6 * 60_000), expiry = new Date(Date.now() + 3_000);
const [run] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agent.id,
status: "running", runtimeMode: "native", nativeIssueId: f.issue.id, contextSnapshot: { issueId: f.issue.id },
startedAt: old, updatedAt: old }).returning();
await db.insert(nativeRunFinalizations).values({ companyId: f.companyId, issueId: f.issue.id, runId: run.id,
phase: "observed", recoveryState: "awaiting_evidence", controllerBootId: randomUUID(), controllerPid: process.pid + 100_000,
controllerProcessStartedAt: old, leaseExpiresAt: expiry });
await start(); expect(deadlines.get("active-execution-recovery")!()).toBe(expiry.getTime());
const moved = new Date(expiry.getTime() + 10_000);
await db.update(nativeRunFinalizations).set({ leaseExpiresAt: moved }).where(eq(nativeRunFinalizations.runId, run.id));
await advance(expiry.getTime() - Date.now());
expect((await getRun(run.id)).status).toBe("running"); expect((await getRun(run.id)).errorCode).toBeNull();
expect(deadlines.get("active-execution-recovery")!()).toBe(moved.getTime());
await db.transaction(async tx => {
await notifyExecutionRunWork(tx, { companyId: f.companyId, runId: run.id, agentId: f.agent.id });
await tx.update(heartbeatRuns).set({ status: "cancelled", finishedAt: new Date() }).where(eq(heartbeatRuns.id, run.id));
await tx.update(nativeRunFinalizations).set({ phase: "terminal_failure", recoveryState: "blocked", leaseExpiresAt: null })
.where(eq(nativeRunFinalizations.runId, run.id));
});
await quiet();
});
async function preparedOrphan() {
const f = await fixture(), old = new Date(Date.now() - 6 * 60_000);
const [run] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agent.id,
status: "failed", runtimeMode: "native", nativeIssueId: f.issue.id, issueId: f.issue.id,
finishedAt: old, updatedAt: old }).returning();
const [lease] = await db.insert(environmentLeases).values({ companyId: f.companyId,
heartbeatRunId: run.id, issueId: f.issue.id, status: "active", provider: "daytona",
providerLeaseId: `sandbox-${run.id}`, metadata: { driver: "sandbox", remoteCwd: "/workspace" }, updatedAt: old }).returning();
await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeWorkspaceSync: {
schema: "paperclip.native-workspace-sync/v3", state: "prepared", descriptorSha256: "a".repeat(64),
baselineSha256: "b".repeat(64), finalHostSha256: null, workspaceId: randomUUID(), leaseId: lease.id,
providerLeaseId: lease.providerLeaseId, remoteCwd: "/workspace", resourceDisposition: "destroy",
} } }).where(eq(heartbeatRuns.id, run.id));
await db.insert(nativeRunFinalizations).values({ companyId: f.companyId, issueId: f.issue.id,
runId: run.id, phase: "terminal_failure", recoveryState: "blocked" });
return { f, run, lease };
}
it.each(["task lock", "statement timeout"])("retries workspace preservation after real PostgreSQL %s without a producer mutation", async kind => {
const { f, run, lease } = await preparedOrphan();
const warnings = vi.spyOn(logger, "warn");
let retryAt = 0;
if (kind === "statement timeout") {
await db.execute(sql`create sequence live_preservation_slow_once`);
await db.execute(sql.raw(`create function live_preservation_slow_once() returns trigger language plpgsql as $$
begin if new.id = '${run.id}'::uuid and new.result_json->>'workspaceRestoreFailure' = 'restore_failed'
and nextval('live_preservation_slow_once') = 1 then perform pg_sleep(16); end if; return new; end $$`));
await db.execute(sql`create trigger live_preservation_slow_once before update on heartbeat_runs for each row execute function live_preservation_slow_once()`);
extraStops.push(async () => {
await db.execute(sql`drop trigger if exists live_preservation_slow_once on heartbeat_runs`);
await db.execute(sql`drop function if exists live_preservation_slow_once()`);
await db.execute(sql`drop sequence if exists live_preservation_slow_once`);
});
}
let release!: () => void, locked!: () => void;
const held = new Promise<void>(resolve => { release = resolve; });
const ready = new Promise<void>(resolve => { locked = resolve; });
const lock = kind === "task lock" ? db.transaction(async tx => {
await tx.select({ id: issues.id }).from(issues).where(eq(issues.id, f.issue.id)).for("update");
locked(); await held;
}) : Promise.resolve();
if (kind !== "task lock") locked();
try {
await ready; await start();
expect((await getRun(run.id)).resultJson).toBeNull();
expect((await getLease(lease.id)).status).toBe("active");
expect(warnings).toHaveBeenCalledWith(expect.objectContaining({ runId: run.id,
code: kind === "task lock" ? "55P03" : "57014" }), expect.any(String));
retryAt = deadlines.get("orphan-active-leases")!()!;
expect(retryAt).toBeGreaterThan(Date.now());
expect(retryAt).toBeLessThanOrEqual(Date.now() + 30_000);
if (kind === "task lock") expect(retryAt).toBe(Date.now() + 30_000);
} finally { release(); await lock; }
await advance(retryAt - Date.now() - 1); expect((await getRun(run.id)).resultJson).toBeNull();
await advance(1);
await vi.waitFor(async () => expect((await getRun(run.id)).resultJson).toMatchObject({
workspaceRestoreFailure: "restore_failed", workspaceRestoreRecovery: { leaseIds: [lease.id] },
}));
expect(await db.select().from(activityLog).where(and(eq(activityLog.runId, run.id),
eq(activityLog.action, "issue.workspace_recovery_required")))).toHaveLength(1);
expect(await getLease(lease.id)).toMatchObject({ status: "pending_cleanup", leasePolicy: "retain_on_failure" });
await quiet();
expect(await db.select().from(activityLog).where(and(eq(activityLog.runId, run.id),
eq(activityLog.action, "issue.workspace_recovery_required")))).toHaveLength(1);
});
it.each(["company", "claim", "manual"])("parks workspace preservation authority gates without timeout retries: %s", async gate => {
const { f, run, lease } = await preparedOrphan();
if (gate === "company") {
const other = await fixture();
await db.update(environmentLeases).set({ companyId: other.companyId }).where(eq(environmentLeases.id, lease.id));
} else if (gate === "claim") {
await db.update(nativeRunFinalizations).set({ leaseOwner: "another-workspace-owner",
leaseExpiresAt: new Date(Date.now() + 60_000) }).where(eq(nativeRunFinalizations.runId, run.id));
} else {
await db.update(nativeRunFinalizations).set({ phase: "observed", recoveryState: "blocked" })
.where(eq(nativeRunFinalizations.runId, run.id));
}
await start();
expect((await getRun(run.id)).resultJson).toBeNull();
expect((await getLease(lease.id)).status).toBe("active");
expect(deadlines.get("orphan-active-leases")!()).toBeNull();
expect(await db.select().from(activityLog).where(eq(activityLog.action, "issue.workspace_recovery_required"))).toEqual([]);
await quiet();
});
it("retains failed orphan work with bounded backoff after a real PostgreSQL write failure", async () => {
const f = await fixture(), old = new Date(Date.now() - 6 * 60_000);
const [lease] = await db.insert(environmentLeases).values({ companyId: f.companyId, status: "active", provider: "local", updatedAt: old }).returning();
await db.execute(sql`create sequence live_recovery_failure_once`);
await db.execute(sql.raw(`create function live_recovery_failure_once() returns trigger language plpgsql as $$
begin if new.id = '${lease.id}'::uuid and new.status = 'pending_cleanup' and nextval('live_recovery_failure_once') = 1
then raise exception 'fixture cleanup persistence unavailable'; end if; return new; end $$`));
await db.execute(sql`create trigger live_recovery_failure_once before update on environment_leases for each row execute function live_recovery_failure_once()`);
extraStops.push(async () => {
await db.execute(sql`drop trigger if exists live_recovery_failure_once on environment_leases`);
await db.execute(sql`drop function if exists live_recovery_failure_once()`);
await db.execute(sql`drop sequence if exists live_recovery_failure_once`);
});
await start();
expect(errors).toHaveLength(1); expect(errors.shift()).toBeInstanceOf(AggregateError);
expect((await getLease(lease.id)).status).toBe("active");
await advance(29_000); expect((await getLease(lease.id)).status).toBe("active");
await advance(1_000);
await vi.waitFor(async () => expect((await getLease(lease.id)).status).toBe("pending_cleanup"));
await quiet();
});
it("parks suppressed worktree recovery without probes and reconstructs it on the committed feature change", async () => {
const f = await fixture(), old = new Date(Date.now() - 6 * 60_000);
const runtimeEnv = { PAPERCLIP_IN_WORKTREE: "true", PAPERCLIP_INSTANCE_ID: "live-recovery-worktree" };
const settings = instanceSettingsService(db, { runtimeEnv });
await settings.updateExperimental({ enableWorktreeRunExecution: false });
heartbeat = heartbeatService(db, { runtimeEnv });
const [lease] = await db.insert(environmentLeases).values({ companyId: f.companyId, status: "active", provider: "local", updatedAt: old }).returning();
await start(); expect((await getLease(lease.id)).status).toBe("active"); await quiet();
try {
await settings.updateExperimental({ enableWorktreeRunExecution: true });
await advance(1_000);
await vi.waitFor(async () => expect((await getLease(lease.id)).status).toBe("pending_cleanup"));
} finally { await settings.updateExperimental({ enableWorktreeRunExecution: false }); }
await quiet();
});
it.each(["agent", "hold"] as const)("revalidates parked native session work on ordinary %s eligibility", async gate => {
const f = await fixture(), tree = issueTreeControlService(db);
const lifecycleWorker = configureAgentLifecycle(db, { requiredPluginIds: async () => [], runHost: async () => "complete", runPlugin: async () => "complete" });
extraStops.push(() => lifecycleWorker.stop());
await db.update(issues).set({ status: "in_progress" }).where(eq(issues.id, f.issue.id));
const actor = { actorType: "user" as const, actorId: f.actorId, userId: f.actorId };
const hold = gate === "hold" ? await tree.createHold(f.companyId, f.issue.id, { mode: "pause", actor }) : null;
if (gate === "agent") { await createAgentLifecycle(db).pauseAgent(f.agent.id); await lifecycleWorker.process(f.agent.id); }
const [run] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agent.id,
status: "failed", runtimeMode: "native", nativeIssueId: f.issue.id, contextSnapshot: { issueId: f.issue.id },
finishedAt: new Date(), errorCode: "saved_native_failure" }).returning();
await db.update(issues).set({ executionRunId: run.id }).where(eq(issues.id, f.issue.id));
await db.insert(nativeRunFinalizations).values({ companyId: f.companyId, issueId: f.issue.id, runId: run.id,
phase: "retryable_failure", nextAttemptAt: new Date(Date.now() - 1_000) });
await start(); await quiet();
expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, run.id)))[0].phase).toBe("retryable_failure");
const before = timeline.filter(event => Array.isArray(event) && event[0] === "finished" && event[1] === "native-process-sessions").length;
if (gate === "agent") { await createAgentLifecycle(db).resumeAgent(f.agent.id); await lifecycleWorker.process(f.agent.id); }
else await tree.releaseHold(f.companyId, f.issue.id, hold!.hold.id, { actor, reason: "Resume native recovery" });
await advance(1_000);
const deadline = performance.now() + 5_000;
while (timeline.filter(event => Array.isArray(event) && event[0] === "finished" && event[1] === "native-process-sessions").length <= before && performance.now() < deadline)
await realDelay(5);
expect(timeline.filter(event => Array.isArray(event) && event[0] === "finished" && event[1] === "native-process-sessions").length).toBeGreaterThan(before);
const expectedPhase = gate === "agent" ? "retryable_failure" : "terminal_failure";
expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, run.id)))[0].phase).toBe(expectedPhase);
if (gate === "agent") expect((await db.select().from(issues).where(eq(issues.id, f.issue.id)))[0].status).toBe("blocked");
expect((await getRun(run.id)).errorCode).toBe("saved_native_failure");
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId))).toHaveLength(1);
await quiet();
});
it("discovers an active lease from the ordinary terminal run commit before any lease write", async () => {
const f = await fixture(), old = new Date(Date.now() - 6 * 60_000);
const [run] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agent.id,
status: "running", runtimeMode: "legacy", startedAt: new Date(), updatedAt: new Date(),
controllerBootId: legacyControllerBootId, controllerLeaseExpiresAt: new Date(Date.now() + 60 * 60_000) }).returning();
const [lease] = await db.insert(environmentLeases).values({ companyId: f.companyId, heartbeatRunId: run.id,
status: "active", provider: "local", updatedAt: old }).returning();
await start(); expect(deadlines.get("orphan-active-leases")!()).toBeNull();
const ended = await terminalizeLegacyExecution({ db, run, status: "failed", patch: { finishedAt: new Date(), errorCode: "owner_process_failure" } });
expect(ended?.status).toBe("failed"); expect((await getLease(lease.id)).status).toBe("active");
await advance(1_000);
await vi.waitFor(async () => expect((await getLease(lease.id)).status).toBe("pending_cleanup"));
await quiet();
});
it.each(["accounting", "wake", "assignment_cancel"] as const)("settles parked agent status when its last %s receipt commits", async receipt => {
const f = await fixture(), old = new Date(Date.now() - 6 * 60_000);
await db.update(agents).set({ status: "running", updatedAt: old }).where(eq(agents.id, f.agent.id));
const [run] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agent.id,
status: "succeeded", finishedAt: old, costAccountingPending: receipt === "accounting",
usageJson: { accountingProviderWorkStarted: false } }).returning();
const [wake] = receipt !== "accounting" ? await db.insert(agentWakeupRequests).values({ companyId: f.companyId,
agentId: f.agent.id, runId: run.id, status: "claimed", source: "automation", reason: "fixture-last-receipt",
requestedByActorType: receipt === "assignment_cancel" ? "user" : "system", requestedByActorId: f.actorId, payload: { issueId: f.issue.id } }).returning() : [];
await start(); await quiet();
expect((await db.select().from(agents).where(eq(agents.id, f.agent.id)))[0].status).toBe("running");
if (receipt === "accounting") expect(await accountRunCost(db, run.id)).toBe(true);
else if (receipt === "assignment_cancel") await issueService(db).update(f.issue.id, { assigneeAgentId: null, assigneeUserId: f.actorId });
else await createHeartbeatLifecycle(db, {} as Parameters<typeof createHeartbeatLifecycle>[1])
.setWakeupStatus(wake.id, "completed", { finishedAt: new Date() });
await advance(1_000);
await vi.waitFor(async () => expect((await db.select().from(agents).where(eq(agents.id, f.agent.id)))[0].status).toBe("idle"));
await quiet();
});
it("retains first producer debt during drain and warm standby, then resumes without duplicate cleanup", async () => {
const f = await fixture(), old = new Date(Date.now() - 6 * 60_000);
draining = true; await start();
const [lease] = await writeEnvironmentCleanup(db, tx => tx.insert(environmentLeases).values({ companyId: f.companyId,
status: "active", provider: "local", updatedAt: old }).returning(), { companyId: f.companyId });
await advance(1_000); await vi.waitFor(() => expect(idleWorkSnapshot().active).toBe(0));
expect((await getLease(lease.id)).status).toBe("active");
standby = true; draining = false; await advance(60_000); expect((await getLease(lease.id)).status).toBe("active");
standby = false; await advance(30_000);
await vi.waitFor(async () => expect((await getLease(lease.id)).status).toBe("pending_cleanup")); await quiet();
});
});