mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-21 19:02:37 +01:00
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent runs are interrupted when a server restarts, a deploy swaps
servers, a controller lease is lost, or a sandbox Runner cannot be
reached again
> - Exact recovery (resume the same session, replace, retry) refuses in
many ways, and each refusal leaves the task blocked or idle for a person
> - On staging, 577 of 1,000 runs in 11 hours failed or were
interrupted, across 25 error codes. About 900 earlier PRs each fixed one
refusal code, and the rate did not converge
> - This pull request adds one universal fallback: when an unfinished
run cannot be resumed exactly, for any reason that is not the person's,
the task continues in a fresh run from its last saved workspace and its
conversation
> - The benefit is that a new failure code next week falls back without
code changes, and tasks stop waiting on a person for infrastructure
faults
## Linked Issues or Issue Description
**What happened?**
After a server restart or deploy swap, many runs end with codes such as
`native_continuation_requires_reconciliation`
(`runner_remote_process_identity_unavailable`), `controller_lease_lost`,
`process_lost`, `server_shutdown_interrupted` and
`native_session_interrupted` (including `runner_harness_state_mismatch:
resumed_not_suspended` / `resumed_unavailable`). The disposition worker
then settles the recovery hold as "blocked, no replay". The task stays
Blocked until a person presses Retry or reconciles the run.
**Expected behavior**
A task waits on a person only for reasons that are the person's: an
approval, a question, a credential to share, a budget stop, a provider
usage limit, or a Stop they asked for. For every other failure, the
system starts a new run for the same task from the last verified saved
workspace, without a person.
**Steps to reproduce**
1. Start a task on a sandbox agent.
2. Restart or swap the server while the run is in progress, so the new
server cannot re-reach the Runner.
3. See that the run fails with
`native_continuation_requires_reconciliation` and the task stays Blocked
with no successor run.
**Paperclip version or commit**
`master` at 9e910ff9ae.
## What Changed
- New module `server/src/services/fresh-continuation-fallback.ts`. One
entry point, `ensureFreshContinuation` / `continueFreshInTransaction`,
takes a run that ended without finishing and guarantees one of:
something already follows it (a newer run, a linked replacement, an
operator decision), exact recovery is still under way (checked again
later), the cause is the person's, or one fresh continuation is
recorded.
- Classification, not an allowlist: `PERSON_OWNED_FAILURE_CODES` plus
task-level waits (pending approval, pending question, paused or
unavailable agent, paused company, review in progress, a Stop that a
person asked for, a Blocked status that another decision owns). Every
other code, including unknown future codes, falls back.
- Idempotent per source run. The task row lock serializes callers on
every server; the recorded `freshContinuation.sourceRunId` makes later
calls a no-op.
- Isolation, never waiting on proof. A local process still running on
this host is settled by its own owner first. If the old local writer is
proven gone, the task workspace is kept. Otherwise the continuation gets
a new isolated workspace (`fresh_isolated_workspace`), so two runs never
write one directory. The decision records the stop proof honestly
(`providerStopProof: isolated`).
- Claim admission no longer waits on the cleanup of a predecessor that a
fresh continuation replaced (`heartbeat/queue.ts`, predecessor loop).
Without this, the successor stayed queued behind the unreachable lease,
which is the staging failure itself.
- Only reconciliation holds that this decision owns are settled.
Workspace restore holds that workspace recovery still owns, and resolved
history, are untouched.
- `continuation_user_authorization_missing` is the system's failure (a
Retry's authorization was not carried forward), not the person's.
- Safety: the old run keeps its fencing. When it still owns its remote
workspace epoch, the epoch is abandoned
(`abandonUnverifiedRemoteWorkspaceEpoch`), so the old writer can never
publish and the next run restores the last committed save into a new
root on a new allocation. The unverified files stay in the abandoned
root, unapplied. The native coordinator records `replacementDenied:
fresh_continuation`, so no exact replay also starts.
- The replaced run loses its write authority. The fallback records
`freshContinuationSupersededAt` on the old run. The agent-JWT middleware
(`loadAgentRunAuthentication` reads one more field in its existing row
read) and `assertAgentRunWriteAllowed` refuse every write of the old
token with `agent_run_superseded`. Before this change, a failed or
interrupted run's token could still write. So an agent that is still
alive in an unreachable sandbox cannot act on the task next to its
continuation.
- The opening context (`recoveryOutcomes` in the execution continuation)
says that the previous attempt was cut off and lists the actions with
unverified outcomes, so the agent checks before it repeats a side
effect. No copy asks a person to inspect or repair a sandbox, workspace
or run. The hold text written at run end no longer says "Inspect the
original failure and reconcile".
- Delivery reuses the existing path: the settled hold gets an
`executionReconciliation` decision and `continuationDelivery: pending`,
and `deliverReconciledExecutions` starts the run (fresh session,
`retryOfRunId` = source).
- Budget per task, not per restart: at most 6 fresh continuations per
task per rolling hour (the server-restart lane's cap). At the cap the
hold stays with the system and is checked again when the window has
room. The failure allowance is never charged.
- Call sites:
- `settleUnrecoverableExecutions` (execution-recovery-resolution.ts):
every hold whose exact recovery is ruled out now goes through the
fallback before it would block the task. Interrupted preparations keep
their own replay lane first.
- `continueHeldExecutions` (same worker): picks up holds saved by older
releases (settled-but-blocked no-replay holds, finished or exhausted
saved-work recoveries) for the task's latest unfinished run.
- Wake admission (heartbeat/queue.ts): an ordinary system continuation
wake (sweep, monitor, timer) that a hold would defer now calls the
fallback in the same transaction and becomes the fresh continuation
itself. This is the path the idle-task sweep uses. A person's Retry,
message or queued send keeps its exact admission; the disposition worker
settles the hold, and the saved input then continues.
- `executionControlSchedule.disposition` wakes the worker for held tasks
and for the next due recheck.
### Human gate versus system uncertainty
One documented rule (`isHumanGate` plus `personOwnedTaskWait`) decides
what the fallback never touches. The rule goes by cause, never by owner
type:
- Human gates: a pending approval, a pending question, a user-owned
hold, an owner-gone attestation, a recorded explicit continuation, an
operator reconciliation, a person's Stop.
- System uncertainty: a no-replay record ("outcome unknown after an
interruption"), a hold that is `board`-owned only by default, an
unverifiable controller process space or host.
For uncertainty the fallback starts a fresh run, never a replay. The
interrupted turn is not re-sent. No tool call or write is re-executed.
Governed actions go through their approval gate again. The earlier
`automaticRecovery` decision is kept on the hold as
`automaticRecoveryHistory`, beside the attributed
`freshContinuationSettlement` and an activity entry.
### Restart recovery refused for an unverifiable controller
The case is `controller_process_space_unverified`, its remote variant,
or `controller_host_unrecorded` / `controller_host_unprovable`. Its
lease has lapsed; a live, renewing lease is still waited for (server
overlap). After `NATIVE_NAMESPACE_UNVERIFIED_WAIT_MS` (2 minutes) the
run ends (failed, `native_restart_recovery_blocked`). That frees the
task's execution lock and the agent's slot, and a system-owned hold lets
the fallback continue the task. Recording the stops is optional. The "a
board operator must record both stops" text is replaced.
Board/operator attention sites in native restart recovery:
- `surfaceNativeControllerNamespaceAttention`: copy replaced; the run
now ends after the bounded wait.
- `requestNativeControllerStopConfirmation`
(native-controller-host-recovery.ts): copy says confirmation is
optional; the same bounded end applies.
- The blocked restart-recovery hold (`nextAction` "Inspect the preserved
checkpoint and reconcile…"): copy replaced; the fallback settles the
hold by cause.
- `owner-gone-attestation.ts` (generic resource records from older
releases): unchanged. It is not a run-recovery path, and it is listed as
a follow-up.
### Every wait and its bound
| Wait | Owner that rechecks | Worst-case total before the fresh
continuation |
| --- | --- | --- |
| `exact_recovery_pending` (execution settling, native finalizer,
controller lease, interrupted-preparation lane, saved-work recovery,
restore hold, retained-source fallback, live local process) |
Disposition worker, woken at each deadline; rechecks at 1, 2, 3 and 4
minutes, then every 4 | About 10 minutes from the first pending check.
Up to 15 while the owner shows progress (a new event, output, lease
renewal or coordinator update since the last check). |
| Saved-work (workspace) recovery that is `scheduled`,
`awaiting_evidence`, or `running` without progress | Held-run scan, now
selected and scheduled like the other holds | About 10 minutes; 15 at
most |
| Saved-work recovery that is `running` and advancing (revision,
attempt, state or hold update since the last check) | Held-run scan |
Its own 30-minute budget |
| Unverifiable lapsed controller in restart recovery | Restart recovery
(`retryAt`) | 2 minutes, then the run ends and the fallback continues it
within one disposition pass |
| Hourly cap (`backoff`) | Disposition worker, woken at the deadline |
Until the rolling hour frees a slot, at most 1 hour. This is the
deliberate restart-lane bound, not a stuck state. |
| Person-owned or dependency wait | The person's own action wakes the
task; rechecked hourly | Not bounded: it is the person's |
Restart-recovery classifications (`claimNativeRestartRecoveries`, which
`recoverNativeRunsAfterRestart` calls):
| Classification | Who rechecks, when | Bound |
| --- | --- | --- |
| Claims: `reattach_existing_runner`, `reattach_remote_runner`,
`resume_dead_runner`, `bootstrap_incomplete` | The claiming server
resumes exactly | Exact resume; no fallback |
| `already_finalized` | Nobody; it is terminal | None needed |
| `blocked` | The run is terminal and its hold goes to the fallback
(disposition worker) | Fallback bounds above |
| `awaiting_evidence` / `current_controller_preparing` | This process's
own executor | Ends with that executor |
| `awaiting_evidence` / `recovery_rows_busy` | Restart dispatch after 1
second | Self-resolving: the row lock clears |
| `awaiting_evidence` / a live lease (`controller_lease_held`,
`controller_lease_grace`, `controller_still_alive`) | Restart dispatch
at the lease-renew interval | Never ended while the old controller
renews its lease |
| `awaiting_evidence` / unverifiable lapsed controller (process space or
host) | Restart dispatch every 2 minutes | Ends 2 minutes after the
lease lapsed |
| `awaiting_evidence` / any other reason (startup cleanup, finalizer or
cleanup owner, recovery failure, routine busy, future reasons) | Restart
dispatch at the owner's `retryAt`, or within 2 minutes when it has none
| Ends 2 minutes after the lease lapsed, or after 10 minutes, whichever
comes first. It never ends while a lease is live. |
Ending a run sets it failed (`native_restart_recovery_blocked`),
releases the task lock and the agent's slot, and records a system-owned
hold. The fallback then continues the task once, isolated.
No wait needs longer. Saved-work recovery and retained-source checks
keep their own attempt budgets. After 10 to 15 minutes the fallback
continues anyway in a new root, without the unverified files, which stay
recorded and unapplied.
### When the fallback preempts saved-work recovery
The unverified files are not destroyed. They stay where recovery found
them: in the abandoned remote epoch's root, or in the old local
workspace that the task is no longer bound to. They are recorded and
never applied. Settling the hold records the decision on it, so the
retained-workspace worker stops scheduling it (its query requires no
`executionReconciliation` unless recovery already succeeded). If an
attempt already in flight finishes later, it cannot publish into the
fresh run's workspace. For a remote workspace, the epoch it targets is
abandoned and the writer is cleared. For a local one, the task now
points at a new workspace. Whenever a wait is preempted, the
continuation is isolated: a fenced remote epoch or a new local
workspace.
### What people see
The task's interruption presentation shows "working" while the primary
fresh-continuation record is pending and has not started (coalesced
holds never count, and they follow the primary's final state), and a
timed wait with no Retry while a check is scheduled. The Retry banner
appears only when the system has given up or the cause is the person's.
Holds settled together with the primary record are `coalesced`; they no
longer show "closed or reassigned". A pending continuation is never a
second run: if a person's Retry or saved input started a run after it
was recorded, delivery links to that run instead. A task that finished
before delivery gets no run.
### Readers of what the fallback writes
- Recheck deadline (`freshContinuationCheck`):
- `unresolvedExecutionDispositions` and
`heldExecutionsAwaitingFreshContinuation` read it to skip work that is
not due.
- `nextFreshContinuationCheckAt` and
`executionControlSchedule.disposition` read it to schedule the worker.
- Tests: the cap deadline is scheduled, and a cancelled task goes quiet.
- Isolation decision:
- The task's workspace binding is cleared. Every path that starts the
next run (system continuation, saved input, Retry, sweep wake) resolves
the workspace from that binding at run start (heartbeat.ts
`issueContext.executionWorkspaceId`).
- `freshContinuationIsolation` in the run context is read by
`readFreshWorkspaceReadiness` for the shared-directory case.
- `freshContinuationSupersededAt` on the old run is read by the auth
middleware, `assertAgentRunWriteAllowed` and the claim gate.
## Verification
- `pnpm vitest run src/services/fresh-continuation-fallback.test.ts`
(server): every test passes (it covers the cases below).
- The continuation starts (leaves the queue) while the unreachable old
lease is `active` or `pending_cleanup`. This was red before the
claim-admission fix.
- Isolation: a writer on another host gets a new isolated workspace; a
process alive on this host waits for its owner; a proven-gone writer
keeps the workspace.
- Owned holds only: a restore hold and resolved history are untouched.
- The recheck schedule goes quiet when the task is cancelled.
- `continuation_user_authorization_missing` continues under the task's
responsible user.
- One test per top-five staging code (native and legacy hold shapes):
the task gets exactly one fresh successor run from saved state.
- `runner_harness_state_mismatch` (`resumed_not_suspended`,
`resumed_unavailable`) falls back.
- An unknown code falls back.
- Two concurrent callers plus the disposition worker create one
successor.
- Old-saved/new-resumed: a settled-but-blocked hold from the previous
release, with its coordinator in `awaiting_runner_reattach`, continues
on the first pass.
- An ordinary system continuation wake is admitted as the fresh
continuation.
- Fencing: after the continuation is recorded, the old run's token is
refused for writes (`agent_run_superseded`) and the continuation's token
is not. The old run's remote workspace epoch is abandoned: the writer is
cleared, the location and provider root are cleared, and the epoch
advances. So the next preparation restores the last save into a new root
(`prepareRemoteWorkspace`: `sameAllocation` is false when the location
is null, server/src/services/remote-workspaces.ts:437-470).
- The opening context names the cut-off run and an unverified tool
action.
- Person-owned causes (pending approval, credential not shared, budget
stop) are not continued.
- The hourly cap backs off without blocking, then continues.
- Red/green: with the wiring reverted to `master`, 9 of these tests fail
(the task stays Blocked with no successor).
- Affected suites pass locally: `execution-recovery-dead-controller`,
`completed-turn-cleanup-hold-settle`, `execution-control-work`,
`heartbeat-process-recovery`, `issue-recovery-actions`,
`native-safe-replacement`, `native-failed-workspace-recovery`,
`task-retry`, `runner-lost-mid-turn-continuation`,
`workspace-recovery-fallback`, `explicit-native-continuation`,
wake-queue `postgres`, `agent-conversations`, `native-startup-retry`,
`governed-wait-runner-loss`, `native-preclaim-continuation`.
- `pnpm verify --explain` selects the full suite (the wake queue is
shared), so CI supplies full coverage.
Existing expectations changed because they asserted the "wait for a
person" behavior this PR replaces:
- `issue-monitor-scheduler.test.ts`: "explains held monitor dispatch
without replaying uncertain execution" is now "continues a held task in
a fresh run when its monitor is due". A due monitor on a task held by a
`process_lost` hold now starts one fresh run instead of deferring.
- `agent-run-authentication.test.ts`: the auth row also reads
`freshContinuationSupersededAt` (still under 512 bytes), and a
superseded run is refused.
- `assigned-work-invariant.test.ts` (#21139): "with the real wake queue,
a hold that defers the continuation leaves a violation…" is now "…a hold
reached by the sweep's wake becomes one pending fresh continuation". The
sweep's wake now yields one pending fresh continuation, and the next
sweep adds no attempt.
- `heartbeat-native-restart-suspension.test.ts` and
`native-runner-restart-recovery.integration.test.ts`: awaiting-evidence
dispositions now carry a `retryAt`, because every wait is rechecked. A
lapsed unverifiable controller no longer waits forever.
- `native-remote-retry-admission.test.ts`: the blocked-hold text is now
"This run stopped before it finished. Your saved work is kept." The old
text asked people to reconcile.
Holdless legacy runs: in the 1,000 staging runs, 0 tasks end on
`controller_lease_lost` or `process_lost` (every one has a later run).
12 end on `server_shutdown_interrupted` with no retry scheduled (attempt
0). On staging now, 9 of those 12 sit `in_progress` with no run and no
hold, 1 is done and 2 have a live run. Those 9 belong to the idle-task
sweep (#21139).
## Risks
- Speed: nothing is added to the normal path before `turn.start`. The
fallback runs only on the disposition worker after exact recovery is
ruled out, or at wake admission when a hold already defers the wake. The
auth middleware reads one more JSON field in the row read it already
does.
- Behavior shift: tasks that today stay Blocked for a person after an
infrastructure failure now continue automatically. A side effect with an
unknown outcome is not replayed; the new run is told about it and must
check first.
- After deploy, the disposition worker continues tasks already blocked
by these codes, at most 25 per pass and at most 6 per task per hour. No
migration. No manual Retry is needed.
- The old executor is not proven stopped before the successor starts.
Fencing is: its API writes are refused, its workspace epoch is
abandoned, the native `replacementDenied` marker is set, and the
successor has a separate run identity.
Known gaps (follow-ups):
- No quarantine attachment: unverified files stay unapplied in the
abandoned workspace root (recorded on the workspace in
`remoteWorkspaceAbandonedEpochs`). They are not read into a task
attachment, because the sandbox is usually unreachable in these
failures.
- Agent Chat conversations are excluded at wake admission. They keep
their owner-bound continuation.
- Under a `reuse_by_environment` lease policy the new allocation can be
on the same provider sandbox. It always uses a new root and a new epoch,
and the old run cannot publish or write.
- Tasks left `in_progress` with no run and no hold (for example after
`server_shutdown_interrupted` with no retry scheduled) belong to the
idle-task sweep. That sweep's wake reaches this fallback when a hold
exists.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
830 lines
62 KiB
TypeScript
830 lines
62 KiB
TypeScript
import { randomUUID } from "node:crypto";
|
|
import { resolve } from "node:path";
|
|
import { and, eq, sql } from "drizzle-orm";
|
|
import { agents, companies, completionContracts, databaseWorkPending, environmentLeases, reconcileDatabaseWork, heartbeatRunEvents, heartbeatRuns, issues, issueRelations,
|
|
issueTreeHolds, issueTreeHoldMembers, issueRecoveryActions, nativeRunFinalizations, nativeRunResults, nativeRunnerCheckpoints } from "@paperclipai/db";
|
|
import { expect, it, vi } from "vitest";
|
|
import type { NativeExecutionInputV1 } from "@paperclipai/paperclip-runner";
|
|
|
|
const provider = vi.hoisted(() => ({ execute: vi.fn() }));
|
|
vi.mock("../../vendor/paperclip-runner/index.js", async importOriginal => ({
|
|
...await importOriginal<typeof import("../../vendor/paperclip-runner/index.js")>(),
|
|
executeNativeSession: provider.execute,
|
|
}));
|
|
import { DurablePrpControlPlane } from "../../vendor/paperclip-runner/index.js";
|
|
import { describeEmbeddedPostgres, nativeRestartFixture, remotePreparedWorkspace } from "../../__tests__/helpers/native-restart.js";
|
|
import { nativeRunnerCheckpointWriter } from "./native-runner-checkpoint.js";
|
|
import { DELIVERY_QUEUES, subscribeDeliveryWork } from "../delivery-work-notifications.js";
|
|
import { claimNativeSessionResumptions } from "./native-finalization-reconciler.js";
|
|
import { cancelNativeSession, executePaperclipNativeSession } from "./native-session-executor.js";
|
|
import { buildNativeCompletionContract } from "./completion-contracts.js";
|
|
import { claimNativeRestartRecoveries, currentNativeControllerIdentity, NATIVE_RESTART_RECOVERY_FAILED } from "./native-restart-recovery.js";
|
|
import { readSystemRecovery, SYSTEM_RECOVERY_MAX_RETRIES } from "../system-recovery-state.js";
|
|
import { recoveryService } from "../recovery/service.js";
|
|
import { issueService } from "../issues.js";
|
|
import { hasPendingRunCleanup } from "../execution-blocker.js";
|
|
import { createPostgresWakeQueueAdapter } from "../../modules/wake-queue/adapters/postgres.js";
|
|
import { nativeManualAdmissionHoldsExecution } from "./native-runner-ownership.js";
|
|
import { remoteTerminationReceipt } from "../remote-execution-termination.js";
|
|
import { releaseSettledNativeRunProcess } from "../native-local-process-stop.js";
|
|
import { createHeartbeatQueue } from "../heartbeat/queue.js";
|
|
import { createHeartbeatQueueTestDependencies } from "../../__tests__/helpers/heartbeat-queue.js";
|
|
import { instanceSettingsService } from "../instance-settings.js";
|
|
import { claimCancellationRequest } from "./native-cancellation-request.js";
|
|
|
|
describeEmbeddedPostgres("remote native retry execution admission", () => {
|
|
const environment = nativeRestartFixture();
|
|
|
|
async function failedRemoteRun(storage: "classic" | "remote" | "plugin") {
|
|
const { companyId, agentId, runtimeRoot } = environment.current;
|
|
const fixture = await environment.seedRun(`REMOTE-RETRY-${storage}-${randomUUID()}`);
|
|
await fixture.db.update(agents).set({ status: "active" }).where(eq(agents.id, agentId));
|
|
const controller = { bootId: randomUUID(), generation: 1, leaseOwner: randomUUID() };
|
|
await fixture.db.update(nativeRunFinalizations).set({
|
|
controllerBootId: controller.bootId, controllerGeneration: controller.generation,
|
|
leaseOwner: controller.leaseOwner, leaseExpiresAt: new Date(Date.now() + 60_000),
|
|
}).where(eq(nativeRunFinalizations.runId, fixture.runId));
|
|
const authority = new DurablePrpControlPlane({
|
|
stateDirectory: resolve(runtimeRoot, fixture.runId),
|
|
identity: { runId: fixture.runId, normalizedSessionId: fixture.native.normalizedSessionId,
|
|
runnerInstanceId: fixture.native.runnerInstanceId, environmentLeaseId: fixture.native.environmentLeaseId,
|
|
turnId: fixture.native.turnId, itemId: fixture.native.itemId },
|
|
expectedRunnerVersion: "retry-admission-fixture", expectedRunnerDigest: `sha256:${"a".repeat(64)}`,
|
|
});
|
|
authority.setCheckpointPersistence(nativeRunnerCheckpointWriter(fixture.db, {
|
|
companyId, issueId: fixture.issueId, runId: fixture.runId, authorityRunId: fixture.runId,
|
|
...controller,
|
|
}));
|
|
await authority.flushCheckpoint();
|
|
await authority.stop();
|
|
const classic = remotePreparedWorkspace();
|
|
const remote = { schema: "paperclip.remote-workspace-run.v1", workspaceId: classic.workspaceId,
|
|
leaseId: classic.leaseId, providerLeaseId: classic.providerLeaseId, remoteCwd: classic.remoteCwd,
|
|
epoch: 1, state: "prepared", snapshotId: null };
|
|
const checkpoint = { backendKind: "acpx", driverKind: "acpx_runtime", sessionId: "provider-session",
|
|
providerSessionId: "provider-session", activeTurnId: "retained-turn", semanticResult: null,
|
|
pendingRuntimeRequests: [], terminalTurns: [], identity: { companyId, agentId,
|
|
issueId: fixture.issueId, runId: fixture.runId, sessionId: fixture.native.normalizedSessionId } };
|
|
const [pluginLease] = storage === "plugin" ? await fixture.db.insert(environmentLeases).values({ companyId,
|
|
issueId: fixture.issueId, heartbeatRunId: fixture.runId, provider: "daytona", providerLeaseId: randomUUID(),
|
|
status: "active", leasePolicy: "ephemeral", metadata: { runnerTaskLease: true },
|
|
}).returning({ id: environmentLeases.id }) : [];
|
|
await fixture.db.update(heartbeatRuns).set({ status: "failed", finishedAt: new Date(),
|
|
processPid: 1386, processStartedAt: new Date("2026-11-01T12:00:00Z"),
|
|
nativePhase: "retryable_failure", errorCode: "native_session_interrupted",
|
|
runnerProfileJson: { ...fixture.preparedProfile, nativeExecutionInput: { binding: {
|
|
companyId, agentId, issueId: fixture.issueId, runId: fixture.runId } }, sessionCheckpoint: checkpoint,
|
|
...(storage === "plugin" ? { nativeWorkspaceSync: null, remoteWorkspace: null }
|
|
: storage === "classic" ? { nativeWorkspaceSync: classic } : { remoteWorkspace: remote }) },
|
|
}).where(eq(heartbeatRuns.id, fixture.runId));
|
|
await fixture.db.update(nativeRunFinalizations).set({ phase: "retryable_failure", attempt: 1,
|
|
leaseOwner: null, leaseExpiresAt: null, recoveryState: "resuming_session",
|
|
failureCode: "native_session_interrupted", nextAttemptAt: new Date(0),
|
|
failureDetail: { recoveryMode: "exact_checkpoint_resume", providerSessionEstablished: true,
|
|
checkpointExists: true, providerEventsExist: true },
|
|
}).where(eq(nativeRunFinalizations.runId, fixture.runId));
|
|
await fixture.db.update(issues).set({ executionRunId: null, checkoutRunId: null })
|
|
.where(eq(issues.id, fixture.issueId));
|
|
return { ...fixture, companyId, agentId, checkpoint, pluginLeaseId: pluginLease?.id };
|
|
}
|
|
|
|
async function manuallyHeldRemoteRun() {
|
|
const f = await failedRemoteRun("remote");
|
|
await f.db.update(issues).set({ executionRunId: f.runId, checkoutRunId: f.runId }).where(eq(issues.id, f.issueId));
|
|
await f.db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, issueId: f.issueId,
|
|
status: "succeeded", createdAt: new Date(Date.now() + 1_000) });
|
|
await claimNativeRestartRecoveries({ db: f.db, controller: environment.current.successor,
|
|
restartKind: "hard", runIds: [f.runId] });
|
|
return f;
|
|
}
|
|
|
|
it.each(["background sweep", "terminal cleanup", "pending cleanup", "queue release", "agent release"] as const)
|
|
("preserves a manual native execution hold through %s", async entry => {
|
|
const f = await manuallyHeldRemoteRun();
|
|
const [before] = await f.db.select().from(issues).where(eq(issues.id, f.issueId));
|
|
if (entry === "background sweep") expect(await recoveryService(f.db, { enqueueWakeup: vi.fn() })
|
|
.sweepStaleIssueLocks({ companyId: f.companyId, issueId: f.issueId })).toMatchObject({ cleared: 0 });
|
|
if (entry === "terminal cleanup") {
|
|
await expect(issueService(f.db).assertCheckoutOwner(f.issueId, f.agentId, randomUUID()))
|
|
.rejects.toMatchObject({ status: 409 });
|
|
}
|
|
if (entry === "agent release") await expect(issueService(f.db).release(f.issueId, f.agentId, randomUUID()))
|
|
.rejects.toMatchObject({ status: 409 });
|
|
if (entry === "pending cleanup") expect(await hasPendingRunCleanup(f.db, f.companyId, f.runId)).toBe(true);
|
|
if (entry === "queue release") await createPostgresWakeQueueAdapter(f.db, {
|
|
resolveResponsibleUserId: async () => "operator", getRoutineEnv: async () => ({ routineId: null, env: null, responsibleUserId: null }),
|
|
resolveSessionBeforeForWakeup: async () => null,
|
|
}).withIssueExecutionLock({ companyId: f.companyId, runId: f.runId, now: new Date() },
|
|
async () => ({ outcome: { kind: "released" }, postCommitEffects: [] }));
|
|
expect((await f.db.select().from(issues).where(eq(issues.id, f.issueId)))[0]).toEqual(before);
|
|
expect((await f.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0])
|
|
.toMatchObject({ processPid: 1386, runnerProfileJson: { sessionCheckpoint: f.checkpoint } });
|
|
});
|
|
|
|
it.each(["execution", "checkout"] as const)("preserves a manual hold referenced only by the %s binding", async pointer => {
|
|
const f = await manuallyHeldRemoteRun();
|
|
await f.db.update(issues).set(pointer === "execution" ? { checkoutRunId: null } : { executionRunId: null })
|
|
.where(eq(issues.id, f.issueId));
|
|
expect(await recoveryService(f.db, { enqueueWakeup: vi.fn() }).sweepStaleIssueLocks({ companyId: f.companyId, issueId: f.issueId }))
|
|
.toMatchObject({ cleared: 0 });
|
|
expect((await f.db.select().from(issues).where(eq(issues.id, f.issueId)))[0])
|
|
.toMatchObject(pointer === "execution" ? { executionRunId: f.runId, checkoutRunId: null } : { executionRunId: null, checkoutRunId: f.runId });
|
|
});
|
|
|
|
it("rechecks a newly committed manual hold under the issue lock before background cleanup", async () => {
|
|
const f = await manuallyHeldRemoteRun();
|
|
const [hold] = await f.db.select({ failureCode: nativeRunFinalizations.failureCode,
|
|
failureDetail: nativeRunFinalizations.failureDetail }).from(nativeRunFinalizations)
|
|
.where(eq(nativeRunFinalizations.runId, f.runId));
|
|
await f.db.update(nativeRunFinalizations).set({ failureCode: null, failureDetail: null })
|
|
.where(eq(nativeRunFinalizations.runId, f.runId));
|
|
const beforeStaleLockClear = vi.fn(async (issueId: string) => {
|
|
if (issueId === f.issueId) await f.db.update(nativeRunFinalizations).set(hold)
|
|
.where(eq(nativeRunFinalizations.runId, f.runId));
|
|
});
|
|
expect(await recoveryService(f.db, { enqueueWakeup: vi.fn(), beforeStaleLockClear })
|
|
.sweepStaleIssueLocks({ companyId: f.companyId, issueId: f.issueId })).toMatchObject({ cleared: 0 });
|
|
expect(beforeStaleLockClear).toHaveBeenCalledExactlyOnceWith(f.issueId);
|
|
expect((await f.db.select().from(issues).where(eq(issues.id, f.issueId)))[0])
|
|
.toMatchObject({ executionRunId: f.runId, checkoutRunId: f.runId });
|
|
});
|
|
|
|
it("defers a new wake behind the manual owner without clearing it or dispatching", async () => {
|
|
const f = await manuallyHeldRemoteRun();
|
|
await f.db.delete(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issueId));
|
|
const [agent] = await f.db.select().from(agents).where(eq(agents.id, f.agentId));
|
|
const dependencies = createHeartbeatQueueTestDependencies(f.db);
|
|
dependencies.getAgent.mockResolvedValue(agent);
|
|
dependencies.instanceSettings.getExperimental.mockImplementation(() => instanceSettingsService(f.db).getExperimental());
|
|
dependencies.wakeQueue.admitWakeBehindIssueExecution.mockResolvedValue({ kind: "deferred" });
|
|
await createHeartbeatQueue(f.db, dependencies).enqueueWakeup(f.agentId, { source: "assignment", reason: "issue_assigned",
|
|
requestedByActorType: "system", payload: { issueId: f.issueId }, contextSnapshot: { issueId: f.issueId } });
|
|
expect(dependencies.executeRun).not.toHaveBeenCalled();
|
|
expect(dependencies.wakeQueue.admitWakeBehindIssueExecution).toHaveBeenCalledOnce();
|
|
expect((await f.db.select().from(issues).where(eq(issues.id, f.issueId)))[0])
|
|
.toMatchObject({ executionRunId: f.runId, checkoutRunId: f.runId });
|
|
});
|
|
|
|
it("retains the hold until every exact remote lease is confirmed stopped, then permits ordinary checkout", async () => {
|
|
const f = await manuallyHeldRemoteRun();
|
|
const scope = { companyId: f.companyId, runId: f.runId, issueId: f.issueId };
|
|
expect(await nativeManualAdmissionHoldsExecution(f.db, scope)).toBe(true);
|
|
expect(await nativeManualAdmissionHoldsExecution(f.db, { ...scope, companyId: randomUUID() })).toBe(false);
|
|
expect(await nativeManualAdmissionHoldsExecution(f.db, { ...scope, runId: randomUUID() })).toBe(false);
|
|
expect(await nativeManualAdmissionHoldsExecution(f.db, { ...scope, issueId: randomUUID() })).toBe(false);
|
|
const leases = await f.db.insert(environmentLeases).values([1, 2].map(index => ({ companyId: f.companyId,
|
|
issueId: f.issueId, heartbeatRunId: f.runId, provider: "daytona", providerLeaseId: `fixture-${index}-${randomUUID()}`,
|
|
status: "released", releasedAt: new Date(), cleanupStatus: "success", metadata: {},
|
|
}))).returning();
|
|
expect(await nativeManualAdmissionHoldsExecution(f.db, scope)).toBe(true);
|
|
for (let index = 0; index < leases.length; index += 1) {
|
|
const lease = leases[index];
|
|
await f.db.update(environmentLeases).set({ metadata: { remoteExecutionTermination: remoteTerminationReceipt(lease,
|
|
{ providerLeaseId: lease.providerLeaseId, state: "stopped" }) } }).where(eq(environmentLeases.id, lease.id));
|
|
expect(await nativeManualAdmissionHoldsExecution(f.db, scope)).toBe(index < leases.length - 1);
|
|
}
|
|
expect(await releaseSettledNativeRunProcess(f.db, f.runId)).toBe("released");
|
|
expect(await hasPendingRunCleanup(f.db, f.companyId, f.runId)).toBe(false);
|
|
expect(await recoveryService(f.db, { enqueueWakeup: vi.fn() }).sweepStaleIssueLocks({ companyId: f.companyId, issueId: f.issueId }))
|
|
.toMatchObject({ cleared: 1 });
|
|
const actorRunId = randomUUID();
|
|
await f.db.insert(heartbeatRuns).values({ id: actorRunId, companyId: f.companyId, agentId: f.agentId,
|
|
status: "running", invocationSource: "manual", contextSnapshot: { issueId: f.issueId } });
|
|
expect(await issueService(f.db).assertCheckoutOwner(f.issueId, f.agentId, actorRunId))
|
|
.toMatchObject({ executionRunId: actorRunId, checkoutRunId: actorRunId });
|
|
});
|
|
|
|
it.each(["ordinary terminal", "accepted result"] as const)("does not preserve unrelated %s ownership", async state => {
|
|
const f = await manuallyHeldRemoteRun();
|
|
if (state === "ordinary terminal") await f.db.update(nativeRunFinalizations).set({ failureDetail: null })
|
|
.where(eq(nativeRunFinalizations.runId, f.runId));
|
|
else {
|
|
const [run] = await f.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
|
|
const [result] = await f.db.insert(nativeRunResults).values({ companyId: f.companyId, issueId: f.issueId,
|
|
runId: f.runId, completionContractId: run.completionContractId!, serverFingerprint: randomUUID(),
|
|
canonicalSha256: "a".repeat(64), schemaStatus: "accepted", resultJson: {} }).returning();
|
|
await f.db.update(nativeRunFinalizations).set({ resultId: result.id }).where(eq(nativeRunFinalizations.runId, f.runId));
|
|
}
|
|
expect(await nativeManualAdmissionHoldsExecution(f.db, { companyId: f.companyId, runId: f.runId, issueId: f.issueId })).toBe(false);
|
|
expect(await recoveryService(f.db, { enqueueWakeup: vi.fn() }).sweepStaleIssueLocks({ companyId: f.companyId, issueId: f.issueId }))
|
|
.toMatchObject({ cleared: 1 });
|
|
});
|
|
|
|
it("admits an explicit correlated Stop for the preserved manual execution", async () => {
|
|
const f = await manuallyHeldRemoteRun();
|
|
const requestId = randomUUID();
|
|
expect(await claimCancellationRequest(f.db, f.runId, f.companyId, requestId, "operator"))
|
|
.toMatchObject({ resultJson: { startupCancellation: { cancellationRequestId: requestId, retryCancellation: true } } });
|
|
const stop = await cancelNativeSession(f.runId, "Operator Stop", { db: f.db, scope: "run", cancellationRequestId: requestId });
|
|
expect(stop).toMatchObject({ dispatched: false, auditId: expect.any(String) });
|
|
expect((await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)))[0])
|
|
.toMatchObject({ phase: "terminal_failure", failureCode: "native_retry_cancelled" });
|
|
expect(await nativeManualAdmissionHoldsExecution(f.db, { companyId: f.companyId, runId: f.runId, issueId: f.issueId })).toBe(true);
|
|
expect(await recoveryService(f.db, { enqueueWakeup: vi.fn() }).sweepStaleIssueLocks({ companyId: f.companyId, issueId: f.issueId }))
|
|
.toMatchObject({ cleared: 0 });
|
|
expect(await cancelNativeSession(f.runId, "Repeat Operator Stop", { db: f.db, scope: "run", cancellationRequestId: requestId }))
|
|
.toMatchObject({ auditId: stop.auditId });
|
|
await f.db.update(nativeRunFinalizations).set({ failureCode: "native_restart_recovery_blocked" })
|
|
.where(eq(nativeRunFinalizations.runId, f.runId));
|
|
expect(await cancelNativeSession(f.runId, "Resume recorded Stop", { db: f.db, scope: "run", cancellationRequestId: requestId }))
|
|
.toMatchObject({ auditId: stop.auditId });
|
|
expect((await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)))[0])
|
|
.toMatchObject({ failureCode: "native_retry_cancelled" });
|
|
expect((await f.db.select().from(issues).where(eq(issues.id, f.issueId)))[0])
|
|
.toMatchObject({ executionRunId: f.runId, checkoutRunId: f.runId });
|
|
});
|
|
|
|
it.each(["classic", "remote", "plugin"] as const)("reclaims the exact failed %s execution before remote inspection", async storage => {
|
|
const fixture = await failedRemoteRun(storage);
|
|
expect(await claimNativeSessionResumptions({ db: fixture.db, runnerInstanceId: "ordinary-resumer",
|
|
runIds: [fixture.runId] })).toEqual([]);
|
|
const [claim] = await claimNativeRestartRecoveries({ db: fixture.db,
|
|
controller: environment.current.successor, restartKind: "hard", runIds: [fixture.runId] });
|
|
expect(claim).toMatchObject({ kind: "reattach_remote_runner", runId: fixture.runId, providerAttempt: 2 });
|
|
expect((await fixture.db.select().from(issues).where(eq(issues.id, fixture.issueId)))[0])
|
|
.toMatchObject({ executionRunId: fixture.runId, status: "in_progress", assigneeAgentId: fixture.agentId });
|
|
expect((await fixture.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId)))[0])
|
|
.toMatchObject({ status: "running", processPid: 1386,
|
|
runnerProfileJson: { sessionCheckpoint: fixture.checkpoint } });
|
|
expect((await fixture.db.select().from(nativeRunFinalizations)
|
|
.where(eq(nativeRunFinalizations.runId, fixture.runId)))[0])
|
|
.toMatchObject({ attempt: 2, phase: "observed", recoveryState: "awaiting_runner_reattach" });
|
|
});
|
|
|
|
it.each(["company", "run", "issue"] as const)("does not use a plugin lease from another %s", async scope => {
|
|
const f = await failedRemoteRun("plugin");
|
|
if (!f.pluginLeaseId) throw new Error("Expected a plugin allocation");
|
|
const foreignId = randomUUID();
|
|
if (scope === "company") {
|
|
await f.db.insert(companies).values({ id: foreignId, name: "Other company", issuePrefix: foreignId.slice(0, 8) });
|
|
await f.db.update(environmentLeases).set({ companyId: foreignId }).where(eq(environmentLeases.id, f.pluginLeaseId));
|
|
} else if (scope === "run") {
|
|
await f.db.insert(heartbeatRuns).values({ id: foreignId, companyId: f.companyId, agentId: f.agentId, status: "failed" });
|
|
await f.db.update(environmentLeases).set({ heartbeatRunId: foreignId }).where(eq(environmentLeases.id, f.pluginLeaseId));
|
|
} else {
|
|
await f.db.insert(issues).values({ id: foreignId, companyId: f.companyId, title: "Other task" });
|
|
await f.db.update(environmentLeases).set({ issueId: foreignId }).where(eq(environmentLeases.id, f.pluginLeaseId));
|
|
}
|
|
const onClaim = vi.fn();
|
|
expect(await claimNativeRestartRecoveries({ db: f.db, controller: environment.current.successor,
|
|
restartKind: "hard", runIds: [f.runId], onClaim }))
|
|
.toMatchObject([{ kind: "blocked", runId: f.runId, reason: "issue_execution_lock_changed" }]);
|
|
expect(onClaim).not.toHaveBeenCalled();
|
|
expect((await f.db.select().from(issues).where(eq(issues.id, f.issueId)))[0])
|
|
.toMatchObject({ executionRunId: null, checkoutRunId: null });
|
|
expect((await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)))[0])
|
|
.toMatchObject({ phase: "retryable_failure", attempt: 1 });
|
|
});
|
|
|
|
it.each((["remote", "plugin"] as const).flatMap(storage => (["done", "cancelled", "blocked", "paused-agent", "pending-agent-approval", "reassigned",
|
|
"dependency", "tree-pause", "native-stop", "operator-stop", "execution-owner", "checkout-owner",
|
|
"queued-input", "newer-completed-input", "wrong-checkpoint", "missing-authority"] as const).map(gate => [storage, gate] as const)))
|
|
("preserves a %s retry held by %s", async (storage, gate) => {
|
|
const f = await failedRemoteRun(storage);
|
|
const { db } = f;
|
|
if (["done", "cancelled", "blocked"].includes(gate)) {
|
|
await db.update(issues).set({ status: gate }).where(eq(issues.id, f.issueId));
|
|
} else if (gate === "paused-agent" || gate === "pending-agent-approval") {
|
|
await db.update(agents).set({ status: gate === "paused-agent" ? "paused" : "pending_approval" }).where(eq(agents.id, f.agentId));
|
|
} else if (gate === "reassigned") {
|
|
const nextAgent = randomUUID();
|
|
await db.insert(agents).values({ id: nextAgent, companyId: f.companyId, name: "Next owner", status: "active" });
|
|
await db.update(issues).set({ assigneeAgentId: nextAgent }).where(eq(issues.id, f.issueId));
|
|
} else if (gate === "dependency") {
|
|
const dependency = randomUUID();
|
|
await db.insert(issues).values({ id: dependency, companyId: f.companyId, title: "Required input", status: "todo" });
|
|
await db.insert(issueRelations).values({ companyId: f.companyId, issueId: dependency, relatedIssueId: f.issueId, type: "blocks" });
|
|
} else if (gate === "tree-pause") {
|
|
const holdId = randomUUID();
|
|
await db.insert(issueTreeHolds).values({ id: holdId, companyId: f.companyId, rootIssueId: f.issueId, mode: "pause" });
|
|
await db.insert(issueTreeHoldMembers).values({ holdId, companyId: f.companyId, issueId: f.issueId,
|
|
issueTitle: "Paused work", issueStatus: "in_progress" });
|
|
} else if (gate === "native-stop" || gate === "operator-stop") {
|
|
await db.update(heartbeatRuns).set({ resultJson: gate === "native-stop"
|
|
? { nativeCancellation: { scope: "run", dispatchState: "pending" } }
|
|
: { cancellation: { source: "operator", expected: true, initiator: { type: "user" },
|
|
reason: "Stop", recordedAt: new Date().toISOString() } } }).where(eq(heartbeatRuns.id, f.runId));
|
|
} else if (["execution-owner", "checkout-owner", "queued-input", "newer-completed-input"].includes(gate)) {
|
|
const nextRun = randomUUID();
|
|
await db.insert(heartbeatRuns).values({ id: nextRun, companyId: f.companyId, agentId: f.agentId,
|
|
issueId: f.issueId, contextSnapshot: { issueId: f.issueId, wakeReason: "issue_comment" },
|
|
invocationSource: "manual", createdAt: new Date(Date.now() + 1_000),
|
|
status: gate === "queued-input" ? "queued" : gate === "newer-completed-input" ? "succeeded" : "failed" });
|
|
if (gate === "execution-owner" || gate === "checkout-owner") await db.update(issues)
|
|
.set(gate === "execution-owner" ? { executionRunId: nextRun } : { checkoutRunId: nextRun }).where(eq(issues.id, f.issueId));
|
|
} else if (gate === "wrong-checkpoint") {
|
|
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
|
|
await db.update(heartbeatRuns).set({ runnerProfileJson: { ...run.runnerProfileJson,
|
|
sessionCheckpoint: { ...f.checkpoint, identity: { ...f.checkpoint.identity, runId: randomUUID() } },
|
|
} }).where(eq(heartbeatRuns.id, f.runId));
|
|
} else if (gate === "missing-authority") {
|
|
await db.delete(nativeRunnerCheckpoints).where(and(eq(nativeRunnerCheckpoints.companyId, f.companyId),
|
|
eq(nativeRunnerCheckpoints.runId, f.runId)));
|
|
}
|
|
const beforeTask = (await db.select().from(issues).where(eq(issues.id, f.issueId)))[0];
|
|
const beforeRun = (await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0];
|
|
const beforeOwner = (await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)))[0];
|
|
const dispositions = await claimNativeRestartRecoveries({ db,
|
|
controller: environment.current.successor, restartKind: "hard", runIds: [f.runId] });
|
|
expect(dispositions.every(value => value.kind === "blocked")).toBe(true);
|
|
expect((await db.select().from(issues).where(eq(issues.id, f.issueId)))[0]).toEqual(beforeTask);
|
|
const [afterRun] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
|
|
const [afterOwner] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId));
|
|
if (["done", "cancelled", "newer-completed-input", "execution-owner", "checkout-owner", "queued-input"].includes(gate)) {
|
|
expect(afterOwner).toMatchObject({ phase: "terminal_failure", attempt: 1, nextAttemptAt: null,
|
|
failureDetail: { systemRecovery: { state: "exhausted" } } });
|
|
expect(afterRun).toMatchObject({ status: "failed", processPid: beforeRun.processPid,
|
|
runnerProfileJson: beforeRun.runnerProfileJson });
|
|
return;
|
|
}
|
|
expect(afterRun).toEqual(beforeRun);
|
|
expect(afterOwner.failureDetail).toMatchObject(beforeOwner.failureDetail!);
|
|
expect({ ...afterOwner, nextAttemptAt: beforeOwner.nextAttemptAt, updatedAt: beforeOwner.updatedAt,
|
|
recoveryHistory: beforeOwner.recoveryHistory, failureDetail: beforeOwner.failureDetail }).toEqual(beforeOwner);
|
|
if (gate === "execution-owner") expect(afterOwner).toEqual(beforeOwner);
|
|
else expect(afterOwner.nextAttemptAt!.getTime()).toBeGreaterThan(beforeOwner.nextAttemptAt!.getTime());
|
|
});
|
|
|
|
it.each(["remote", "plugin"] as const)("waits for the exact old %s controller lease and keeps future retries quiet", async storage => {
|
|
const f = await failedRemoteRun(storage);
|
|
const now = new Date(), deadline = new Date(now.getTime() + 60_000);
|
|
await f.db.update(nativeRunFinalizations).set({ nextAttemptAt: deadline })
|
|
.where(eq(nativeRunFinalizations.runId, f.runId));
|
|
expect(await claimNativeRestartRecoveries({ db: f.db, controller: environment.current.successor,
|
|
restartKind: "hard", now, runIds: [f.runId] })).toEqual([]);
|
|
await f.db.update(nativeRunFinalizations).set({ nextAttemptAt: new Date(0), leaseOwner: "old-owner", leaseExpiresAt: deadline })
|
|
.where(eq(nativeRunFinalizations.runId, f.runId));
|
|
expect(await claimNativeRestartRecoveries({ db: f.db, controller: environment.current.successor,
|
|
restartKind: "hard", now, runIds: [f.runId] })).toMatchObject([{ kind: "awaiting_evidence", reason: "remote_controller_lease_live" }]);
|
|
expect((await f.db.select().from(issues).where(eq(issues.id, f.issueId)))[0].executionRunId).toBeNull();
|
|
expect((await f.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0].status).toBe("failed");
|
|
});
|
|
|
|
it.each(["remote", "plugin"] as const)("grants two competing controllers only one original %s execution", async storage => {
|
|
const f = await failedRemoteRun(storage);
|
|
const claims = (await Promise.all([environment.current.successor,
|
|
{ ...environment.current.successor, bootId: randomUUID() }].map(controller =>
|
|
claimNativeRestartRecoveries({ db: f.db, controller, restartKind: "hard", runIds: [f.runId] })))).flat();
|
|
expect(claims.filter(claim => claim.kind === "reattach_remote_runner")).toHaveLength(1);
|
|
expect((await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)))[0])
|
|
.toMatchObject({ controllerGeneration: 2, attempt: 2 });
|
|
expect(await f.db.select().from(heartbeatRunEvents).where(and(eq(heartbeatRunEvents.runId, f.runId),
|
|
eq(heartbeatRunEvents.eventType, "native.recovery.execution_reclaimed")))).toHaveLength(1);
|
|
});
|
|
|
|
it.each(["remote", "plugin"] as const)("rolls back the %s task, owner, run, and recovery event together", async storage => {
|
|
const f = await failedRemoteRun(storage);
|
|
await expect(f.db.transaction(async tx => {
|
|
const claims = await claimNativeRestartRecoveries({ db: tx as unknown as typeof f.db,
|
|
controller: environment.current.successor, restartKind: "hard", runIds: [f.runId] });
|
|
expect(claims).toMatchObject([{ kind: "reattach_remote_runner" }]);
|
|
throw new Error("rollback admission");
|
|
})).rejects.toThrow("rollback admission");
|
|
expect((await f.db.select().from(issues).where(eq(issues.id, f.issueId)))[0].executionRunId).toBeNull();
|
|
expect((await f.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0].status).toBe("failed");
|
|
expect((await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)))[0])
|
|
.toMatchObject({ phase: "retryable_failure", controllerGeneration: 1, leaseOwner: null, attempt: 1 });
|
|
expect(await f.db.select().from(heartbeatRunEvents).where(and(eq(heartbeatRunEvents.runId, f.runId),
|
|
eq(heartbeatRunEvents.eventType, "native.recovery.execution_reclaimed")))).toEqual([]);
|
|
});
|
|
|
|
it.each([
|
|
["succeeded", "2026-11-01T12:00:00.000600Z", 1, true],
|
|
["succeeded", "2026-11-01T12:00:00.000000Z", 1, true],
|
|
["failed", "2026-11-01T12:00:00.000000Z", 1, true],
|
|
["failed", "2026-11-01T12:00:00.000600Z", 1, true],
|
|
["succeeded", "2026-11-01T12:00:00.000599Z", 1, false],
|
|
["failed", "2026-11-01T12:00:00.000601Z", -1, true],
|
|
["failed", "2026-11-01T12:00:00.000600Z", -1, false],
|
|
] as const)("uses canonical source order for %s input at %s with ID offset %s", async (status, timestamp, offset, superseded) => {
|
|
const f = await failedRemoteRun("remote");
|
|
const idHex = (BigInt(`0x${f.runId.replaceAll("-", "")}`) + BigInt(offset)).toString(16).padStart(32, "0");
|
|
const nextId = `${idHex.slice(0, 8)}-${idHex.slice(8, 12)}-${idHex.slice(12, 16)}-${idHex.slice(16, 20)}-${idHex.slice(20)}`;
|
|
const sourceTimestamp = timestamp.endsWith(".000000Z") ? timestamp : "2026-11-01T12:00:00.000600Z";
|
|
await f.db.update(heartbeatRuns).set({ createdAt: sql`${sourceTimestamp}::timestamptz` })
|
|
.where(eq(heartbeatRuns.id, f.runId));
|
|
await f.db.insert(heartbeatRuns).values({ id: nextId, companyId: f.companyId, agentId: f.agentId,
|
|
issueId: f.issueId, invocationSource: "manual", status, createdAt: sql`${timestamp}::timestamptz` });
|
|
const [disposition] = await claimNativeRestartRecoveries({ db: f.db,
|
|
controller: environment.current.successor, restartKind: "hard", runIds: [f.runId] });
|
|
expect(disposition).toMatchObject(superseded
|
|
? { kind: "blocked", reason: "native_retry_source_superseded" }
|
|
: { kind: "reattach_remote_runner", providerAttempt: 2 });
|
|
expect((await f.db.select().from(issues).where(eq(issues.id, f.issueId)))[0].executionRunId)
|
|
.toBe(superseded ? null : f.runId);
|
|
});
|
|
|
|
it.each((["remote", "plugin"] as const).flatMap(storage => (["released", "retained"] as const).map(lock => [storage, lock] as const)))
|
|
("exhausts the existing attempt budget after repeated %s snapshot failures with a %s task lock", async (storage, lock) => {
|
|
const f = await failedRemoteRun(storage);
|
|
if (lock === "retained") await f.db.update(issues).set({ executionRunId: f.runId }).where(eq(issues.id, f.issueId));
|
|
const [issue] = await f.db.select().from(issues).where(eq(issues.id, f.issueId));
|
|
const [contract] = await f.db.select().from(completionContracts).where(eq(completionContracts.issueId, f.issueId));
|
|
const execution: NativeExecutionInputV1 = {
|
|
schema: "paperclip.native-execution-input.v1",
|
|
binding: { companyId: f.companyId, runId: f.runId, issueId: f.issueId, agentId: f.agentId, executionWorkspaceId: f.runId },
|
|
provider: { kind: "codex", model: "gpt-5.6-luna" },
|
|
task: { identifier: issue.identifier!, title: issue.title, description: null, prompt: "Continue retained work.", workMode: "standard" },
|
|
workspace: { cwd: environment.current.runtimeRoot, repoUrl: null, repoRef: null, branchName: null },
|
|
session: { normalizedSessionId: f.native.normalizedSessionId, driverKind: "codex_app_server", protocolVersion: 1,
|
|
lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null } },
|
|
completionContract: { id: contract.id, sha256: contract.canonicalSha256, schemaVersion: "paperclip.completion-contract.v1",
|
|
contract: buildNativeCompletionContract(issue, { revision: contract.revision }) },
|
|
interactionResponses: [], credentialBindings: [],
|
|
};
|
|
await f.db.update(heartbeatRuns).set({ runnerProfileJson: sql`coalesce(${heartbeatRuns.runnerProfileJson}, '{}'::jsonb)
|
|
|| ${JSON.stringify({ nativeExecutionInput: execution })}::jsonb` }).where(eq(heartbeatRuns.id, f.runId));
|
|
provider.execute.mockReset().mockRejectedValue(new Error("PRP command session.snapshot failed: original provider unavailable"));
|
|
const controller = await currentNativeControllerIdentity();
|
|
for (const attempt of [2, 3]) {
|
|
await f.db.update(nativeRunFinalizations).set({ nextAttemptAt: new Date(0) }).where(eq(nativeRunFinalizations.runId, f.runId));
|
|
const [claim] = await claimNativeRestartRecoveries({ db: f.db, controller, restartKind: "hard", runIds: [f.runId] });
|
|
expect(claim).toMatchObject({ kind: "reattach_remote_runner", providerAttempt: attempt });
|
|
if (claim.kind !== "reattach_remote_runner") throw new Error("Expected exact remote recovery claim");
|
|
await expect(executePaperclipNativeSession({ db: f.db, execution, runnerInstanceId: f.native.runnerInstanceId,
|
|
leaseOwner: claim.leaseOwner, restartRecovery: claim,
|
|
backend: { descriptor: async () => { throw new Error("Unexpected descriptor call"); },
|
|
openSession: async () => { throw new Error("Unexpected session launch"); } },
|
|
})).rejects.toThrow("PRP command session.snapshot failed");
|
|
const [owner] = await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId));
|
|
expect(owner).toMatchObject({ attempt, phase: attempt === 3 ? "terminal_failure" : "retryable_failure",
|
|
failureCode: attempt === 3 ? "native_session_retry_exhausted" : "native_session_interrupted",
|
|
recoveryState: attempt === 3 ? "blocked" : "resuming_session", leaseOwner: null });
|
|
expect(owner.nextAttemptAt === null).toBe(attempt === 3);
|
|
if (lock === "released") await f.db.update(issues).set({ executionRunId: null }).where(eq(issues.id, f.issueId));
|
|
}
|
|
expect(provider.execute).toHaveBeenCalledTimes(2);
|
|
expect(await claimNativeRestartRecoveries({ db: f.db, controller, restartKind: "hard", runIds: [f.runId] })).toEqual([]);
|
|
expect((await f.db.select().from(issues).where(eq(issues.id, f.issueId)))[0]).toMatchObject({ status: "blocked", executionRunId: null });
|
|
expect((await f.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0])
|
|
.toMatchObject({ status: "failed", nativePhase: "terminal_failure", errorCode: "native_session_interrupted",
|
|
processPid: 1386, runnerProfileJson: { sessionCheckpoint: f.checkpoint } });
|
|
});
|
|
|
|
it.each((["remote", "plugin"] as const).flatMap(storage => (["released", "retained"] as const).map(lock => [storage, lock] as const)))
|
|
("does not admit another failed %s retry when the persisted provider budget is already exhausted with a %s task lock", async (storage, lock) => {
|
|
const f = await failedRemoteRun(storage);
|
|
if (lock === "retained") await f.db.update(issues).set({ executionRunId: f.runId }).where(eq(issues.id, f.issueId));
|
|
await f.db.update(nativeRunFinalizations).set({ attempt: 3 }).where(eq(nativeRunFinalizations.runId, f.runId));
|
|
expect(await claimNativeRestartRecoveries({ db: f.db, controller: environment.current.successor,
|
|
restartKind: "hard", runIds: [f.runId] })).toMatchObject([{ kind: "blocked", reason: "execution_recovery_budget_exhausted" }]);
|
|
expect((await f.db.select().from(issues).where(eq(issues.id, f.issueId)))[0].executionRunId).toBeNull();
|
|
expect((await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)))[0])
|
|
.toMatchObject({ phase: "terminal_failure", attempt: 3, leaseOwner: null, nextAttemptAt: null,
|
|
failureCode: "native_restart_recovery_blocked", failureDetail: { reason: "execution_recovery_budget_exhausted" } });
|
|
expect((await f.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0])
|
|
.toMatchObject({ status: "failed", nativePhase: "terminal_failure", processPid: 1386,
|
|
runnerProfileJson: { sessionCheckpoint: f.checkpoint } });
|
|
expect(await f.db.select().from(issueRecoveryActions).where(and(eq(issueRecoveryActions.sourceIssueId, f.issueId),
|
|
eq(issueRecoveryActions.kind, "active_run_watchdog"))))
|
|
.toMatchObject([{ ownerType: "board", evidence: { runId: f.runId, reason: "execution_recovery_budget_exhausted" } }]);
|
|
expect(await claimNativeRestartRecoveries({ db: f.db, controller: environment.current.successor,
|
|
restartKind: "hard", runIds: [f.runId] })).toEqual([]);
|
|
});
|
|
|
|
it("persists a blocked retry deadline and records only changed hold reasons", async () => {
|
|
const f = await failedRemoteRun("remote");
|
|
await f.db.update(agents).set({ status: "paused" }).where(eq(agents.id, f.agentId));
|
|
const now = new Date();
|
|
const attempt = (at: Date) => claimNativeRestartRecoveries({ db: f.db, controller: environment.current.successor,
|
|
restartKind: "hard", runIds: [f.runId], now: at });
|
|
const [first] = await attempt(now);
|
|
expect(first).toMatchObject({ kind: "blocked", reason: "native_retry_admission_ineligible", transitionUnchanged: false });
|
|
if (!("retryAt" in first) || !first.retryAt) throw new Error("missing first retry deadline");
|
|
expect(first.retryAt.getTime() - now.getTime()).toBeGreaterThanOrEqual(24_000);
|
|
expect(first.retryAt.getTime() - now.getTime()).toBeLessThanOrEqual(36_000);
|
|
expect(await attempt(new Date(first.retryAt.getTime() - 1))).toEqual([]);
|
|
const [second] = await attempt(first.retryAt);
|
|
expect(second).toMatchObject({ kind: "blocked", transitionUnchanged: true });
|
|
if (!("retryAt" in second) || !second.retryAt) throw new Error("missing second retry deadline");
|
|
expect(second.retryAt.getTime() - first.retryAt.getTime()).toBeGreaterThanOrEqual(48_000);
|
|
let [owner] = await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId));
|
|
expect(owner).toMatchObject({ phase: "retryable_failure", attempt: 1,
|
|
nextAttemptAt: second.retryAt, recoveryHistory: [{ reason: "native_retry_admission_ineligible" }] });
|
|
await f.db.update(agents).set({ status: "active" }).where(eq(agents.id, f.agentId));
|
|
await f.db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, issueId: f.issueId,
|
|
status: "succeeded", createdAt: new Date(now.getTime() + 1_000) });
|
|
expect(await attempt(second.retryAt)).toMatchObject([{ kind: "blocked", reason: "native_retry_source_superseded" }]);
|
|
[owner] = await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId));
|
|
expect(owner.recoveryHistory).toMatchObject([{ reason: "native_retry_admission_ineligible" }, { reason: "native_retry_source_superseded" }]);
|
|
});
|
|
|
|
it("registers recovery work intent before writing a blocked retry deadline", async () => {
|
|
const f = await failedRemoteRun("remote");
|
|
await f.db.update(agents).set({ status: "paused" }).where(eq(agents.id, f.agentId));
|
|
const transaction = f.db.transaction.bind(f.db);
|
|
let writes = 0;
|
|
const interception = vi.spyOn(f.db, "transaction").mockImplementation((async (callback, options) => transaction(async tx => {
|
|
const update = tx.update.bind(tx);
|
|
const updates = vi.spyOn(tx, "update").mockImplementation((table => {
|
|
if (Object.is(table, nativeRunFinalizations)) {
|
|
writes += 1;
|
|
expect(databaseWorkPending(f.db, DELIVERY_QUEUES.nativeProcessSessions)).toBe(true);
|
|
}
|
|
return update(table);
|
|
}) as typeof tx.update);
|
|
try { return await callback(tx); } finally { updates.mockRestore(); }
|
|
}, options)) as typeof f.db.transaction);
|
|
try {
|
|
expect(await claimNativeRestartRecoveries({ db: f.db, controller: environment.current.successor,
|
|
restartKind: "hard", runIds: [f.runId] })).toMatchObject([{ kind: "blocked" }]);
|
|
expect(writes).toBe(1);
|
|
} finally { interception.mockRestore(); }
|
|
});
|
|
|
|
it.each(["done", "cancelled", "superseded"] as const)("settles a permanently %s remote retry without recurring work", async gate => {
|
|
for (const lock of ["released", "retained"]) {
|
|
const f = await failedRemoteRun("remote");
|
|
if (lock === "retained") await f.db.update(issues).set({ executionRunId: f.runId, checkoutRunId: f.runId })
|
|
.where(eq(issues.id, f.issueId));
|
|
if (gate === "superseded") await f.db.insert(heartbeatRuns).values({ companyId: f.companyId,
|
|
agentId: f.agentId, issueId: f.issueId, status: "succeeded", createdAt: new Date(Date.now() + 1_000) });
|
|
else await f.db.update(issues).set({ status: gate }).where(eq(issues.id, f.issueId));
|
|
const onClaim = vi.fn();
|
|
const environmentWake = vi.fn(), stopWatchingEnvironment = subscribeDeliveryWork(f.db, DELIVERY_QUEUES.nativeEnvironmentSettlement, environmentWake);
|
|
const [disposition] = await claimNativeRestartRecoveries({ db: f.db, controller: environment.current.successor,
|
|
restartKind: "hard", runIds: [f.runId], onClaim });
|
|
expect(disposition).toMatchObject({ kind: "blocked" });
|
|
expect(disposition).not.toHaveProperty("retryAt");
|
|
expect(onClaim).not.toHaveBeenCalled();
|
|
expect(environmentWake).not.toHaveBeenCalled();
|
|
stopWatchingEnvironment();
|
|
expect((await f.db.select().from(issues).where(eq(issues.id, f.issueId)))[0]).toMatchObject({
|
|
executionRunId: lock === "retained" ? f.runId : null, checkoutRunId: lock === "retained" ? f.runId : null });
|
|
expect((await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)))[0])
|
|
.toMatchObject({ phase: "terminal_failure", nextAttemptAt: null, attempt: 1 });
|
|
expect((await f.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0])
|
|
.toMatchObject({ processPid: 1386, runnerProfileJson: { sessionCheckpoint: f.checkpoint } });
|
|
const wake = vi.fn(), unsubscribe = subscribeDeliveryWork(f.db, DELIVERY_QUEUES.nativeProcessSessions, wake);
|
|
const transaction = vi.spyOn(f.db, "transaction");
|
|
try {
|
|
for (const elapsed of [30_000, 60_000, 3_600_000]) expect(await claimNativeRestartRecoveries({ db: f.db,
|
|
controller: environment.current.successor, restartKind: "hard", runIds: [f.runId],
|
|
now: new Date(Date.now() + elapsed), onClaim })).toEqual([]);
|
|
expect(transaction).not.toHaveBeenCalled();
|
|
expect(wake).not.toHaveBeenCalled();
|
|
} finally { transaction.mockRestore(); unsubscribe(); }
|
|
}
|
|
});
|
|
|
|
it.each(["superseded", "released"] as const)("settles a %s bootstrap retry that lost its task lock instead of re-checking forever", async lock => {
|
|
const f = await failedRemoteRun("remote");
|
|
// A bootstrap retry follows a pre-provider failure: no checkpoint, provider session or provider events.
|
|
const [run] = await f.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
|
|
const { sessionCheckpoint: _checkpoint, ...profile } = run.runnerProfileJson ?? {};
|
|
await f.db.update(heartbeatRuns).set({ errorCode: "provider_session_open_rejected", runnerProfileJson: profile })
|
|
.where(eq(heartbeatRuns.id, f.runId));
|
|
await f.db.delete(nativeRunnerCheckpoints)
|
|
.where(and(eq(nativeRunnerCheckpoints.companyId, f.companyId), eq(nativeRunnerCheckpoints.runId, f.runId)));
|
|
await f.db.update(nativeRunFinalizations).set({ failureCode: "provider_session_open_rejected", failureDetail: {
|
|
recoveryMode: "bootstrap_retry", providerSessionEstablished: false, checkpointExists: false, providerEventsExist: false,
|
|
} }).where(eq(nativeRunFinalizations.runId, f.runId));
|
|
if (lock === "superseded") {
|
|
const [newer] = await f.db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, issueId: f.issueId,
|
|
status: "running", createdAt: new Date(Date.now() + 1_000) }).returning({ id: heartbeatRuns.id });
|
|
await f.db.update(issues).set({ executionRunId: newer.id }).where(eq(issues.id, f.issueId));
|
|
}
|
|
const onClaim = vi.fn();
|
|
const attempt = (now: Date) => claimNativeRestartRecoveries({ db: f.db, controller: environment.current.successor,
|
|
restartKind: "hard", runIds: [f.runId], now, onClaim });
|
|
let now = new Date();
|
|
let passes = 0;
|
|
for (; passes <= SYSTEM_RECOVERY_MAX_RETRIES + 1; passes++) {
|
|
const [disposition] = await attempt(now);
|
|
expect(disposition).toMatchObject({ kind: "blocked" });
|
|
const [coordinator] = await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId));
|
|
if (coordinator.phase === "terminal_failure") break;
|
|
if (!("retryAt" in disposition) || !disposition.retryAt) throw new Error("a deferred bootstrap retry must persist its deadline");
|
|
expect(coordinator.nextAttemptAt).toEqual(disposition.retryAt);
|
|
now = disposition.retryAt;
|
|
}
|
|
expect(passes).toBe(lock === "superseded" ? 0 : SYSTEM_RECOVERY_MAX_RETRIES);
|
|
expect(onClaim).not.toHaveBeenCalled();
|
|
expect((await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)))[0])
|
|
.toMatchObject({ phase: "terminal_failure", nextAttemptAt: null, attempt: 1, failureCode: "native_restart_recovery_blocked" });
|
|
expect(await f.db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issueId)))
|
|
.toMatchObject([{ ownerType: "board", wakePolicy: null }]);
|
|
const transaction = vi.spyOn(f.db, "transaction");
|
|
try {
|
|
for (const elapsed of [30_000, 3_600_000]) expect(await attempt(new Date(now.getTime() + elapsed))).toEqual([]);
|
|
expect(transaction).not.toHaveBeenCalled();
|
|
} finally { transaction.mockRestore(); }
|
|
});
|
|
|
|
it.each(["paused", "Stop", "missing-authority"] as const)("bounds %s holds and preserves the provider and own task lock for manual reconciliation", async gate => {
|
|
const f = await failedRemoteRun("remote");
|
|
if (gate === "paused") await f.db.update(agents).set({ status: "paused" }).where(eq(agents.id, f.agentId));
|
|
if (gate === "Stop") await f.db.update(heartbeatRuns).set({ resultJson: {
|
|
nativeCancellation: { scope: "run", dispatchState: "pending" },
|
|
} }).where(eq(heartbeatRuns.id, f.runId));
|
|
if (gate === "missing-authority") await f.db.delete(nativeRunnerCheckpoints)
|
|
.where(and(eq(nativeRunnerCheckpoints.companyId, f.companyId), eq(nativeRunnerCheckpoints.runId, f.runId)));
|
|
await f.db.update(issues).set({ executionRunId: f.runId }).where(eq(issues.id, f.issueId));
|
|
await f.db.update(nativeRunFinalizations).set({ attempt: 3 }).where(eq(nativeRunFinalizations.runId, f.runId));
|
|
let now = new Date();
|
|
const onClaim = vi.fn();
|
|
const environmentWake = vi.fn(), stopWatchingEnvironment = subscribeDeliveryWork(f.db, DELIVERY_QUEUES.nativeEnvironmentSettlement, environmentWake);
|
|
for (let index = 0; index <= SYSTEM_RECOVERY_MAX_RETRIES; index += 1) {
|
|
const [disposition] = await claimNativeRestartRecoveries({ db: f.db, controller: environment.current.successor,
|
|
restartKind: "hard", runIds: [f.runId], now, onClaim });
|
|
const [owner] = await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId));
|
|
const policy = readSystemRecovery(owner.failureDetail);
|
|
expect(policy).not.toBeNull();
|
|
expect(owner.attempt).toBe(3);
|
|
if (index < SYSTEM_RECOVERY_MAX_RETRIES) {
|
|
expect(owner.phase).toBe("retryable_failure");
|
|
expect(policy).toMatchObject({ state: "awaiting_evidence" });
|
|
expect(disposition).toHaveProperty("retryAt", owner.nextAttemptAt);
|
|
now = owner.nextAttemptAt!;
|
|
} else {
|
|
expect(owner).toMatchObject({ phase: "terminal_failure", nextAttemptAt: null });
|
|
expect(policy).toMatchObject({ state: "exhausted" });
|
|
expect(disposition).not.toHaveProperty("retryAt");
|
|
}
|
|
}
|
|
expect(onClaim).not.toHaveBeenCalled();
|
|
expect(environmentWake).not.toHaveBeenCalled();
|
|
stopWatchingEnvironment();
|
|
expect((await f.db.select().from(issues).where(eq(issues.id, f.issueId)))[0]).toMatchObject({ executionRunId: f.runId });
|
|
expect((await f.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0])
|
|
.toMatchObject({ processPid: 1386, runnerProfileJson: { sessionCheckpoint: f.checkpoint } });
|
|
expect(await f.db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issueId)))
|
|
.toMatchObject([{ ownerType: "board", wakePolicy: null,
|
|
evidence: { automaticAdmissionExhausted: true, executorStopEvidence: "unverified", providerPid: 1386, checkpointPreserved: true },
|
|
nextAction: "This run stopped before it finished. Your saved work is kept." }]);
|
|
expect(await claimNativeRestartRecoveries({ db: f.db, controller: environment.current.successor,
|
|
restartKind: "hard", runIds: [f.runId], now: new Date(now.getTime() + 3_600_000) })).toEqual([]);
|
|
});
|
|
|
|
it("settles the old admission policy while preserving a newer active execution and its binding", async () => {
|
|
const f = await failedRemoteRun("remote");
|
|
const newerRunId = randomUUID();
|
|
await f.db.insert(heartbeatRuns).values({ id: newerRunId, companyId: f.companyId, agentId: f.agentId,
|
|
issueId: f.issueId, status: "running", invocationSource: "manual", createdAt: new Date(Date.now() + 1_000) });
|
|
await f.db.update(issues).set({ executionRunId: newerRunId, checkoutRunId: newerRunId }).where(eq(issues.id, f.issueId));
|
|
const [task] = await f.db.select().from(issues).where(eq(issues.id, f.issueId));
|
|
const [newer] = await f.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, newerRunId));
|
|
const environmentWake = vi.fn(), unsubscribe = subscribeDeliveryWork(f.db, DELIVERY_QUEUES.nativeEnvironmentSettlement, environmentWake);
|
|
const onClaim = vi.fn();
|
|
try {
|
|
expect(await claimNativeRestartRecoveries({ db: f.db, controller: environment.current.successor,
|
|
restartKind: "hard", runIds: [f.runId], onClaim }))
|
|
.toMatchObject([{ kind: "blocked", reason: "native_retry_source_superseded" }]);
|
|
expect(onClaim).not.toHaveBeenCalled();
|
|
expect(environmentWake).not.toHaveBeenCalled();
|
|
expect((await f.db.select().from(issues).where(eq(issues.id, f.issueId)))[0]).toEqual(task);
|
|
expect((await f.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, newerRunId)))[0]).toEqual(newer);
|
|
expect((await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)))[0])
|
|
.toMatchObject({ phase: "terminal_failure", nextAttemptAt: null, failureDetail: { systemRecovery: { state: "exhausted" } } });
|
|
expect((await f.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0])
|
|
.toMatchObject({ processPid: 1386, runnerProfileJson: { sessionCheckpoint: f.checkpoint } });
|
|
const transactions = vi.spyOn(f.db, "transaction");
|
|
try {
|
|
expect(await claimNativeRestartRecoveries({ db: f.db, controller: environment.current.successor,
|
|
restartKind: "hard", runIds: [f.runId], now: new Date(Date.now() + 3_600_000) })).toEqual([]);
|
|
expect(transactions).not.toHaveBeenCalled();
|
|
} finally { transactions.mockRestore(); }
|
|
} finally { unsubscribe(); }
|
|
});
|
|
|
|
it("resumes the original provider when eligibility returns before admission backoff exhausts", async () => {
|
|
const f = await failedRemoteRun("remote");
|
|
await f.db.update(agents).set({ status: "paused" }).where(eq(agents.id, f.agentId));
|
|
await claimNativeRestartRecoveries({ db: f.db, controller: environment.current.successor,
|
|
restartKind: "hard", runIds: [f.runId] });
|
|
const [owner] = await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId));
|
|
await f.db.update(agents).set({ status: "active" }).where(eq(agents.id, f.agentId));
|
|
expect(await claimNativeRestartRecoveries({ db: f.db, controller: environment.current.successor,
|
|
restartKind: "hard", runIds: [f.runId], now: owner.nextAttemptAt! }))
|
|
.toMatchObject([{ kind: "reattach_remote_runner", providerAttempt: 2, runId: f.runId }]);
|
|
expect((await f.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0])
|
|
.toMatchObject({ status: "running", processPid: 1386, runnerProfileJson: { sessionCheckpoint: f.checkpoint } });
|
|
expect((await f.db.select().from(issues).where(eq(issues.id, f.issueId)))[0].executionRunId).toBe(f.runId);
|
|
});
|
|
|
|
it("does not settle permanently ineligible work while the original controller is active", async () => {
|
|
const f = await failedRemoteRun("remote");
|
|
await f.db.update(issues).set({ status: "done" }).where(eq(issues.id, f.issueId));
|
|
const [before] = await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId));
|
|
const wake = vi.fn(), unsubscribe = subscribeDeliveryWork(f.db, DELIVERY_QUEUES.nativeProcessSessions, wake);
|
|
try {
|
|
expect(await claimNativeRestartRecoveries({ db: f.db, controller: environment.current.successor,
|
|
restartKind: "hard", runIds: [f.runId], liveRunExecutions: new Set([f.runId]) }))
|
|
.toMatchObject([{ kind: "awaiting_evidence", reason: "current_controller_preparing" }]);
|
|
expect((await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)))[0]).toEqual(before);
|
|
expect(wake).not.toHaveBeenCalled();
|
|
expect(await f.db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issueId))).toEqual([]);
|
|
} finally { unsubscribe(); }
|
|
});
|
|
|
|
it("keeps a retry deadline that changed after discovery read-only and quiet", async () => {
|
|
const f = await failedRemoteRun("remote");
|
|
await f.db.update(agents).set({ status: "paused" }).where(eq(agents.id, f.agentId));
|
|
const deadline = new Date(Date.now() + 60_000);
|
|
const transaction = f.db.transaction.bind(f.db);
|
|
const interception = vi.spyOn(f.db, "transaction").mockImplementationOnce((async (callback, options) => {
|
|
await f.db.update(nativeRunFinalizations).set({ nextAttemptAt: deadline }).where(eq(nativeRunFinalizations.runId, f.runId));
|
|
return transaction(callback, options);
|
|
}) as typeof f.db.transaction);
|
|
const wake = vi.fn(), unsubscribe = subscribeDeliveryWork(f.db, DELIVERY_QUEUES.nativeProcessSessions, wake);
|
|
try {
|
|
expect(await claimNativeRestartRecoveries({ db: f.db, controller: environment.current.successor,
|
|
restartKind: "hard", runIds: [f.runId] })).toMatchObject([{ kind: "blocked", retryAt: deadline, transitionUnchanged: true }]);
|
|
const [owner] = await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId));
|
|
expect(owner).toMatchObject({ nextAttemptAt: deadline, recoveryHistory: [], phase: "retryable_failure", attempt: 1 });
|
|
expect(wake).not.toHaveBeenCalled();
|
|
} finally { interception.mockRestore(); unsubscribe(); }
|
|
});
|
|
|
|
it("rolls back work intent when the locked retry deadline update matches no row", async () => {
|
|
const f = await failedRemoteRun("remote");
|
|
await f.db.update(agents).set({ status: "paused" }).where(eq(agents.id, f.agentId));
|
|
const name = `retry_noop_${f.runId.replaceAll("-", "")}`;
|
|
await f.db.execute(sql.raw(`create function ${name}() returns trigger language plpgsql as $$begin return null; end$$`));
|
|
await f.db.execute(sql.raw(`create trigger ${name} before update on native_run_finalizations for each row
|
|
when (old.run_id = '${f.runId}'::uuid) execute function ${name}()`));
|
|
const [before] = await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId));
|
|
try {
|
|
const now = new Date();
|
|
await expect(claimNativeRestartRecoveries({ db: f.db, controller: environment.current.successor,
|
|
restartKind: "hard", runIds: [f.runId], now })).resolves.toEqual([{ kind: "awaiting_evidence",
|
|
runId: f.runId, reason: NATIVE_RESTART_RECOVERY_FAILED, retryAt: new Date(now.getTime() + 30_000) }]);
|
|
expect((await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)))[0]).toEqual(before);
|
|
await reconcileDatabaseWork(f.db, DELIVERY_QUEUES.nativeProcessSessions);
|
|
expect(databaseWorkPending(f.db, DELIVERY_QUEUES.nativeProcessSessions)).toBe(false);
|
|
} finally {
|
|
await f.db.execute(sql.raw(`drop trigger ${name} on native_run_finalizations`));
|
|
await f.db.execute(sql.raw(`drop function ${name}()`));
|
|
}
|
|
});
|
|
|
|
it("does not publish or log a deferred transition when an ineligible retry still has a live lease", async () => {
|
|
const f = await failedRemoteRun("remote");
|
|
const now = new Date(), leaseDeadline = new Date(now.getTime() + 60_000);
|
|
await f.db.update(agents).set({ status: "paused" }).where(eq(agents.id, f.agentId));
|
|
await f.db.update(nativeRunFinalizations).set({ leaseOwner: "live-controller", leaseExpiresAt: leaseDeadline })
|
|
.where(eq(nativeRunFinalizations.runId, f.runId));
|
|
const [before] = await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId));
|
|
const wake = vi.fn(), unsubscribe = subscribeDeliveryWork(f.db, DELIVERY_QUEUES.nativeProcessSessions, wake);
|
|
try {
|
|
for (let index = 0; index < 2; index += 1) {
|
|
expect(await claimNativeRestartRecoveries({ db: f.db, controller: environment.current.successor,
|
|
restartKind: "hard", runIds: [f.runId], now })).toMatchObject([{ kind: "blocked",
|
|
retryAt: leaseDeadline, transitionUnchanged: true }]);
|
|
expect((await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)))[0]).toEqual(before);
|
|
expect(wake).not.toHaveBeenCalled();
|
|
}
|
|
} finally { unsubscribe(); }
|
|
});
|
|
|
|
it.each(["released", "retained"] as const)("keeps exhausted %s retries fenced behind live owners, Stops and newer input", async lock => {
|
|
for (const gate of ["live-owner", "Stop", "newer-input", "missing-authority"]) {
|
|
const f = await failedRemoteRun("remote");
|
|
if (lock === "retained") await f.db.update(issues).set({ executionRunId: f.runId }).where(eq(issues.id, f.issueId));
|
|
await f.db.update(nativeRunFinalizations).set({ attempt: 3,
|
|
...(gate === "live-owner" ? { leaseOwner: "live-controller", leaseExpiresAt: new Date(Date.now() + 60_000) } : {}),
|
|
}).where(eq(nativeRunFinalizations.runId, f.runId));
|
|
if (gate === "Stop") await f.db.update(heartbeatRuns).set({ resultJson: {
|
|
nativeCancellation: { scope: "run", dispatchState: "pending" },
|
|
} }).where(eq(heartbeatRuns.id, f.runId));
|
|
if (gate === "newer-input") await f.db.insert(heartbeatRuns).values({ companyId: f.companyId,
|
|
agentId: f.agentId, issueId: f.issueId, status: "succeeded", createdAt: new Date(Date.now() + 1_000) });
|
|
if (gate === "missing-authority") await f.db.delete(nativeRunnerCheckpoints)
|
|
.where(and(eq(nativeRunnerCheckpoints.companyId, f.companyId), eq(nativeRunnerCheckpoints.runId, f.runId)));
|
|
const onClaim = vi.fn();
|
|
expect((await claimNativeRestartRecoveries({ db: f.db, controller: environment.current.successor,
|
|
restartKind: "hard", runIds: [f.runId], onClaim }))[0].kind).toBe(gate === "live-owner" ? "awaiting_evidence" : "blocked");
|
|
expect(onClaim).not.toHaveBeenCalled();
|
|
expect((await f.db.select().from(issues).where(eq(issues.id, f.issueId)))[0].executionRunId).toBe(lock === "retained" ? f.runId : null);
|
|
expect((await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)))[0])
|
|
.toMatchObject({ phase: gate === "newer-input" ? "terminal_failure" : "retryable_failure", attempt: 3 });
|
|
const actions = await f.db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issueId));
|
|
if (gate === "newer-input") expect(actions).toMatchObject([{ ownerType: "board", wakePolicy: null }]);
|
|
else expect(actions).toEqual([]);
|
|
}
|
|
});
|
|
|
|
it("preserves the charged attempt when replacing an active remote controller", async () => {
|
|
const f = await failedRemoteRun("remote");
|
|
await f.db.update(issues).set({ executionRunId: f.runId }).where(eq(issues.id, f.issueId));
|
|
await f.db.update(heartbeatRuns).set({ status: "running", nativePhase: "observed" }).where(eq(heartbeatRuns.id, f.runId));
|
|
await f.db.update(nativeRunFinalizations).set({ attempt: 2, phase: "observed", failureCode: null, failureDetail: null })
|
|
.where(eq(nativeRunFinalizations.runId, f.runId));
|
|
expect(await claimNativeRestartRecoveries({ db: f.db, controller: environment.current.successor,
|
|
restartKind: "hard", runIds: [f.runId] })).toMatchObject([{ kind: "reattach_remote_runner", providerAttempt: 2 }]);
|
|
expect((await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)))[0])
|
|
.toMatchObject({ phase: "observed", attempt: 2, controllerGeneration: 2 });
|
|
});
|
|
|
|
it("ignores another company's newer input when reclaiming this company's execution", async () => {
|
|
const f = await failedRemoteRun("remote");
|
|
const foreignCompany = randomUUID(), foreignAgent = randomUUID();
|
|
await f.db.insert(companies).values({ id: foreignCompany, name: "Other company", issuePrefix: "FOREIGN" });
|
|
await f.db.insert(agents).values({ id: foreignAgent, companyId: foreignCompany, name: "Other agent", status: "active" });
|
|
await f.db.insert(heartbeatRuns).values({ companyId: foreignCompany, agentId: foreignAgent,
|
|
status: "queued", contextSnapshot: { issueId: f.issueId }, createdAt: new Date(Date.now() + 1_000) });
|
|
expect(await claimNativeRestartRecoveries({ db: f.db, controller: environment.current.successor,
|
|
restartKind: "hard", runIds: [f.runId] })).toMatchObject([{ kind: "reattach_remote_runner" }]);
|
|
});
|
|
});
|