mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-08 11:23:40 +01:00
fix(recovery): release a recovered workspace hold for the new assignee after reassignment (#19184)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - When a board user reassigns a task in the middle of a run, Paperclip
stops the old run and saves its sandbox files (copy-back).
> - If copy-back fails at the stop, Paperclip records a
file-preservation hold for the stopped run. The hold blocks all
execution on the task until the files are recovered.
> - The new assignee's wakes were already queued. The final dispatch
gate saw the hold and cancelled them.
> - Automatic recovery then saved the files. But the hold stayed
effective for the new assignee: explicit continuation accepts only the
stopped run's own agent, so no message, Retry or stranded-work sweep
could start the new assignee.
> - The task stayed `in_progress` with no run, no wake and no active
recovery.
> - This pull request releases a recovered file-preservation hold for a
different current owner.
> - The benefit is that a reassigned task continues with its new owner
after its files are safe, with no manual action.
## Linked Issues or Issue Description
**What happened?**
A board user reassigned a running native task from agent A to agent B
and posted a comment. A's run ended `cancelled` / `issue_reassigned`.
Copy-back was still in progress, so B's first run was cancelled
`workspace_busy`. Paperclip then recorded a resolved no-replay hold
(`native_continuation_requires_reconciliation`, fingerprint
`native-workspace:<A run>`). The final dispatch gate cancelled B's two
queued runs with `execution_reconciliation_required` ("Recovering saved
work…"). Automatic recovery then saved the files
(`workspaceRecovery.status = recovered`, `workspaceRepairRequired:
false`). The blocker still said "Saved work recovered. Send a message to
continue." with `canContinue: false`. The task stayed `in_progress` with
no run for hours. Two hosted reproductions had the same shape. A sibling
reassignment without a copy-back failure recovered normally.
**Expected behavior**
After the saved-file receipt verifies, the new assignee continues. The
hold must still block every executor while the files are not recovered.
The stopped run's own agent keeps its no-replay hold.
**Steps to reproduce**
1. Start a native run on a task with a retained sandbox.
2. Reassign the task to another agent while the run executes, so that
copy-back records a `workspaceRestoreRecovery` hold.
3. Let automatic workspace recovery finish.
4. Before this change, `getExecutionBlocker` still returns the hold for
the new assignee, and the stranded-work sweep queues nothing.
**Paperclip version or commit**
`b9fbbf8d30` (also seen on `818d2bb6`).
**Deployment mode**
Hosted, PostgreSQL. The test uses embedded PostgreSQL.
Related open pull requests (checked; none covers this path): #19177
(board reconciliation of sibling holds), #19169 (disposition repairs
behind a cancelled blocker), #19143 (park a workspace-busy wait), #19149
(interrupt stop identity).
## What Changed
- New `restoreHoldReleasedForCurrentOwnerPredicate`
(`server/src/services/execution-blocker.ts`) is a SQL predicate. It
matches a file-preservation hold (`workspaceRestoreRecovery`, not
`restore_unsafe_archive`) when all of these are true:
- the stopped run is terminal and finished;
- the task has an agent assignee, and that agent is not the stopped
run's agent;
- the stopped run's agent is not the current review participant;
- `completedWorkspaceRecoveryCondition` is true for the hold: completed
system recovery, `workspaceRecovery.status = recovered`, succeeded
preservation operations bound to this hold, and the `workspace_finalize`
receipt;
- every saved operation is a copyback into the task workspace
(`workspace_recovery`). A snapshot-only save keeps the hold, because the
new owner's ordinary turn does not attach snapshots.
- The predicate is wrapped in `coalesce(..., false)`. Its negation
therefore cannot hide an unrelated hold through NULL JSON reads.
- The proof is the durable completed-recovery receipt. It does not read
live workspace authority (epoch, writer, snapshot), so reuse of the
workspace by the new owner cannot re-arm the hold.
- `getExecutionBlocker` excludes matching holds from both the
file-preservation query and the general blocker query. There is no scan
limit and no extra query per hold.
- `admitExplicitNativeContinuation` excludes the same holds from its
source checks. A released hold therefore cannot reject a new owner's
message with "owner changed".
- `completedWorkspaceRecoveryCondition` also accepts a SQL expression
for the run id. Its behavior is unchanged.
- The hold row, its `automaticRecovery.replay = "blocked"` evidence and
its status are not changed. If the task returns to the stopped run's
agent, the hold applies again.
- `doc/execution-semantics.md` describes the rule next to the existing
"Reassignment does not discard the stopped run's files" sentence.
## Verification
New test in `server/src/services/workspace-recovery.test.ts`: `hands a
reassigned task to its new assignee once the previous owner's files are
recovered`. It uses the real retained-workspace recovery, embedded
PostgreSQL and the real stranded-work sweep. It builds the incident
shape: a cancelled `issue_reassigned` source run, a resolved no-replay
hold, a new assignee, and that assignee's gate-cancelled run.
- Before recovery, the blocker holds the new owner
(`workspaceRepairRequired: true`).
- After recovery, the blocker is `null`. The sweep creates exactly one
pending turn for the new owner. That turn is not a retry of the stopped
run. A second sweep adds no turn.
- The hold row keeps `status: resolved`, `outcome: blocked` and `replay:
blocked`.
- Live repair evidence on the stopped run is then changed, so
`hasVerifiedWorkspaceRepair` returns false (as after a remote reuse).
The blocker stays `null`.
- After a reassignment back to the stopped run's agent, the hold applies
again.
- An unrelated reconciliation hold for the same stopped run, without
file evidence, still blocks the new owner.
- New test `keeps a snapshot-only recovery held for a new assignee`: the
snapshot-only save does not release the hold. This test fails against
the first revision of this pull request.
Red before the fix (test kept, `execution-blocker.ts` restored from
`master`):
```
cd server && npx vitest run src/services/workspace-recovery.test.ts -t "hands a reassigned task"
AssertionError: expected { …(13) } to be null (blocker: "Saved work recovered. Send a message to continue.", canContinue: false)
```
With only the blocker assertion removed, the sweep also failed:
`expected [] to have a length of 1 but got +0`.
Green after the fix: 1 passed.
Related suites, all pass (991 tests): `workspace-recovery`,
`execution-blocker-read`, `legacy-explicit-continuation-recovery`,
`workspace-restore-reconciliation`, `native-failed-workspace-recovery`,
`workspace-restore-repair-route`, `execution-projection-recovery`,
`heartbeat-undispatched-execution-wait`,
`issue-update-comment-wakeup-routes`,
`heartbeat-observed-workspace-preservation`,
`native-workspace-reconstruction`, `explicit-native-continuation`.
Not run: a live run on a deployed instance. The hosted incident tasks
are not recovered by this pull request until it is deployed.
## Risks
- A new owner can start after the files are recovered, while the stopped
run's action outcomes stay unknown. This matches the existing rule for
holds without a workspace: Paperclip does not create those holds for a
reassigned task. The new owner gets a fresh turn, not a replay.
- The hold still blocks every executor until the exact receipt verifies,
so there is no second executor on the workspace during copy-back.
- Other execution holds (non-workspace holds recorded before a
reassignment) keep their current effect. This change does not alter
them.
- A reassignment after a snapshot-only save still needs a board action,
as before. That path is not changed here.
- The release is one correlated `exists` in the existing blocker
queries. Its subqueries use primary keys and the run and issue indexes.
## Model Used
- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, with tool use
and code execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
f7a4c5678f
commit
46a50a3bdd
5 files changed
+148
-6
No files matched your search
@@ -2063,6 +2063,21 @@ ordinary Retry can continue. Finalized workspaces, mismatched sources and
|
||||
accepted results keep their existing settlement paths. Reassignment does not
|
||||
discard the stopped run's files or grant authority over the new assignee.
|
||||
|
||||
When the task was reassigned, that file-preservation hold binds the new owner
|
||||
only until automatic recovery records a completed receipt for that hold. Every
|
||||
saved operation in the receipt must be a copyback into the task workspace. The
|
||||
receipt is durable, so later reuse of the workspace by the new owner cannot
|
||||
apply the hold again. A new owner's turn is a fresh assignment, not a replay of
|
||||
the stopped run. After the receipt, admission, the final dispatch gate and
|
||||
stranded-work recovery no longer apply the hold to that owner. Wakes that the
|
||||
gate cancelled during the copy-back therefore get one bounded continuation.
|
||||
|
||||
The hold row and its no-replay evidence stay unchanged. The hold still applies
|
||||
when the task returns to the stopped run's agent. It also applies when that agent
|
||||
is the current review participant. Before the receipt, the hold prevents a second
|
||||
executor on the workspace. A snapshot-only save does not release the hold,
|
||||
because the task workspace does not contain the saved files.
|
||||
|
||||
The deadline is a backstop, not the normal failure path. When native task
|
||||
preparation fails after a fresh controller claim and before any provider turn,
|
||||
the controller stops renewal and releases only that exact claim. Ordinary
|
||||
|
||||
@@ -4,7 +4,7 @@ import { preDispatchExecutionWaitRecoveryActionPredicate } from "./pre-dispatch-
|
||||
import { and, desc, eq, gt, inArray, not, or, sql } from "drizzle-orm";
|
||||
import { conversationRecoveryActionPredicate, getConversationOwnershipBlocker } from "./conversation-continuation.js";
|
||||
import { z } from "zod";
|
||||
import { agentWakeupRequests, chatConversations, chatEndpoints, environmentLeases, heartbeatRuns, issueComments, issues, issueRecoveryActions, nativeRunFinalizations, toolConnections, type Db } from "@paperclipai/db";
|
||||
import { agentWakeupRequests, chatConversations, chatEndpoints, environmentLeases, heartbeatRuns, issueComments, issues, issueRecoveryActions, nativeRunFinalizations, toolConnections, workspaceOperations, type Db } from "@paperclipai/db";
|
||||
import { canContinueCancelledRun, readRunCancellation } from "./run-cancellation.js";
|
||||
import { canRetryStoppedRun } from "./cancelled-native-startup.js";
|
||||
import { queuedCommentIdsFromWakePayload } from "./issue-queued-comment-queue.js";
|
||||
@@ -13,7 +13,7 @@ import { hasRequiredWorkspaceRecovery, LEGACY_WORKSPACE_RECOVERY_SCHEMA, nativeR
|
||||
import { unreconciledNativeExportPredicate } from "./native-runtime/native-workspace-reconstruction.js";
|
||||
import { hasRemoteTerminationReceipt } from "./remote-execution-termination.js";
|
||||
import { readNativeWorkspaceSyncReference } from "./native-runtime/native-workspace-sync.js";
|
||||
import { hasVerifiedWorkspaceRepair, workspaceRecoveryDisplay } from "./workspace-recovery-receipt.js";
|
||||
import { completedWorkspaceRecoveryCondition, hasVerifiedWorkspaceRepair, workspaceRecoveryDisplay } from "./workspace-recovery-receipt.js";
|
||||
import { nativeManualAdmissionHoldsExecution } from "./native-runtime/native-runner-ownership.js";
|
||||
|
||||
const blockerContextKeys = ["issueId", "taskId", "source", "connectionIntentResolution",
|
||||
@@ -126,6 +126,45 @@ export function nativeExecutionRecoveryActionPredicate() {
|
||||
and ${heartbeatRuns.runtimeMode} = 'native')`;
|
||||
}
|
||||
|
||||
/**
|
||||
* A file-preservation hold binds a newer task owner only until automatic
|
||||
* recovery saved the stopped run's files. That owner's turn is a fresh
|
||||
* assignment, not a replay: the stopped run's unknown outcomes stay with its
|
||||
* own agent, which keeps the hold. The proof is the durable completed-recovery
|
||||
* receipt bound to this action, so later reuse of the workspace by the new
|
||||
* owner (new epoch, writer or snapshot) cannot re-arm the hold.
|
||||
*/
|
||||
export function restoreHoldReleasedForCurrentOwnerPredicate() {
|
||||
const sourceRunId = sql`coalesce(${issueRecoveryActions.evidence}->>'runId', ${issueRecoveryActions.evidence}->>'sourceRunId')`;
|
||||
const operationIds = sql`case when jsonb_typeof(${issueRecoveryActions.evidence}#>'{systemRecovery,preservationOperationIds}') = 'array'
|
||||
then ${issueRecoveryActions.evidence}#>'{systemRecovery,preservationOperationIds}' else '[]'::jsonb end`;
|
||||
// JSON reads are NULL for other holds. Callers negate this predicate, so it
|
||||
// must be strictly true or false: NOT NULL would hide unrelated blockers.
|
||||
return sql`coalesce((${and(
|
||||
sql`${issueRecoveryActions.evidence}->'workspaceRestoreRecovery'->>'schema' = ${LEGACY_WORKSPACE_RECOVERY_SCHEMA}`,
|
||||
sql`coalesce(${issueRecoveryActions.evidence}->>'workspaceRestoreFailure', '') <> 'restore_unsafe_archive'`,
|
||||
sql`exists (select 1 from ${heartbeatRuns} stopped
|
||||
join ${issues} task on task.company_id = stopped.company_id and task.id = ${issueRecoveryActions.sourceIssueId}
|
||||
where stopped.company_id = ${issueRecoveryActions.companyId}
|
||||
and stopped.id::text = ${sourceRunId}
|
||||
and stopped.status in ('failed', 'timed_out', 'interrupted', 'cancelled')
|
||||
and stopped.finished_at is not null
|
||||
and task.assignee_agent_id is not null
|
||||
and task.assignee_agent_id <> stopped.agent_id
|
||||
and not coalesce(task.status = 'in_review'
|
||||
and task.execution_state->>'status' = 'pending'
|
||||
and task.execution_state->'currentParticipant'->>'type' = 'agent'
|
||||
and task.execution_state->'currentParticipant'->>'agentId' = stopped.agent_id::text, false))`,
|
||||
completedWorkspaceRecoveryCondition(sourceRunId),
|
||||
// Copyback put the files back into the task workspace that the new owner
|
||||
// reuses. A snapshot-only save needs an explicit fresh-workspace request
|
||||
// to attach it, so that hold stays until such a request or a repair.
|
||||
sql`not exists (select 1 from jsonb_array_elements_text(${operationIds}) saved(id)
|
||||
join ${workspaceOperations} op on op.id::text = saved.id
|
||||
where op.company_id = ${issueRecoveryActions.companyId} and op.phase <> 'workspace_recovery')`,
|
||||
)}), false)`;
|
||||
}
|
||||
|
||||
export async function getExecutionBlocker(db: Db, companyId: string, issueId: string, options?: { conversationResetCommentId?: string | null }): Promise<ExecutionBlocker | null> {
|
||||
const [conversation] = await db.select({ agentId: issues.conversationAgentId,
|
||||
boundaryId: issues.conversationBoundaryCommentId, status: issues.status,
|
||||
@@ -139,6 +178,7 @@ export async function getExecutionBlocker(db: Db, companyId: string, issueId: st
|
||||
eq(issueRecoveryActions.companyId, companyId),
|
||||
eq(issueRecoveryActions.sourceIssueId, issueId),
|
||||
executionBlockerPredicate(),
|
||||
not(restoreHoldReleasedForCurrentOwnerPredicate()),
|
||||
or(
|
||||
unreconciledNativeExportPredicate(),
|
||||
sql`${issueRecoveryActions.evidence}->>'workspaceRestoreFailure' = 'restore_unsafe_archive'`,
|
||||
@@ -166,6 +206,7 @@ export async function getExecutionBlocker(db: Db, companyId: string, issueId: st
|
||||
eq(issueRecoveryActions.companyId, companyId),
|
||||
eq(issueRecoveryActions.sourceIssueId, issueId),
|
||||
executionBlockerPredicate(),
|
||||
not(restoreHoldReleasedForCurrentOwnerPredicate()),
|
||||
boundary ? or(gt(issueRecoveryActions.createdAt, boundary.createdAt), nativeExecutionRecoveryActionPredicate()) : undefined,
|
||||
)).orderBy(desc(issueRecoveryActions.updatedAt), desc(issueRecoveryActions.id)).limit(1);
|
||||
if (!action) return null;
|
||||
|
||||
@@ -19,13 +19,13 @@ import { hasNativeLocalProcessStop, hasHistoricalSuspendedNativeSession, recorde
|
||||
import { completeTerminatedRemoteNativeSessionCleanup } from "../vendor/paperclip-runner/index.js";
|
||||
import { hasRemoteTerminationReceipt, remoteLeaseCleanupScope } from "./remote-execution-termination.js";
|
||||
import { z } from "zod";
|
||||
import { and, desc, eq, gt, inArray, isNull, ne, or, sql } from "drizzle-orm";
|
||||
import { and, desc, eq, gt, inArray, isNull, ne, not, or, sql } from "drizzle-orm";
|
||||
import {
|
||||
agents, agentWakeupRequests, approvals, issueApprovals, issueThreadInteractions,
|
||||
environmentLeases, heartbeatRuns, issueComments, issueRecoveryActions,
|
||||
issues, nativeRunFinalizations, nativeRunResults, type Db,
|
||||
} from "@paperclipai/db";
|
||||
import { executionBlockerPredicate, getExecutionBlocker, hasPendingRunCleanup, nativeExecutionRecoveryActionPredicate } from "./execution-blocker.js";
|
||||
import { executionBlockerPredicate, getExecutionBlocker, hasPendingRunCleanup, nativeExecutionRecoveryActionPredicate, restoreHoldReleasedForCurrentOwnerPredicate } from "./execution-blocker.js";
|
||||
import { buildExecutionContinuation } from "./execution-continuation.js";
|
||||
import { adapterExecutionControls } from "./adapter-execution-control.js";
|
||||
import { persistActivity } from "./activity-log.js";
|
||||
@@ -227,6 +227,7 @@ export async function admitExplicitNativeContinuation(input: {
|
||||
eq(issueRecoveryActions.companyId, companyId), eq(issueRecoveryActions.sourceIssueId, issueId),
|
||||
boundary ? or(gt(issueRecoveryActions.createdAt, boundary.createdAt), nativeExecutionRecoveryActionPredicate(),
|
||||
blocker?.recoveryActionId ? eq(issueRecoveryActions.id, blocker.recoveryActionId) : undefined) : undefined,
|
||||
not(restoreHoldReleasedForCurrentOwnerPredicate()),
|
||||
or(executionBlockerPredicate(), and(
|
||||
inArray(issueRecoveryActions.cause, ["native_workspace_sync_out_unrecoverable", "native_workspace_sync_out_retry_exhausted"]),
|
||||
sql`${issueRecoveryActions.evidence}#>>'{workspaceRecovery,status}' = 'recovered'`,
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import { and, desc, eq, sql, type AnyColumn } from "drizzle-orm";
|
||||
import { and, desc, eq, sql, type AnyColumn, type SQL } from "drizzle-orm";
|
||||
import { executionWorkspaces, executionWorkspaceSnapshots, heartbeatRuns, issueRecoveryActions, workspaceOperations, type Db } from "@paperclipai/db";
|
||||
import { readRemoteWorkspaceAuthority, remoteWorkspaceRunReferenceSchema, type ExecutionBlocker } from "@paperclipai/shared";
|
||||
import { z } from "zod";
|
||||
@@ -25,7 +25,7 @@ export function readWorkspaceRecovery(evidence: Record<string, unknown> | null)
|
||||
return parsed.success ? parsed.data : null;
|
||||
}
|
||||
|
||||
export function completedWorkspaceRecoveryCondition(runId: string | AnyColumn) {
|
||||
export function completedWorkspaceRecoveryCondition(runId: string | AnyColumn | SQL) {
|
||||
const operationIds = sql`case when jsonb_typeof(${issueRecoveryActions.evidence}#>'{systemRecovery,preservationOperationIds}') = 'array'
|
||||
then ${issueRecoveryActions.evidence}#>'{systemRecovery,preservationOperationIds}' else '[]'::jsonb end`;
|
||||
return and(
|
||||
|
||||
@@ -2970,6 +2970,91 @@ describe("automatic retained workspace recovery", () => {
|
||||
expect(wake.payload?.taskRetry).toMatchObject({ strategy: "continue" });
|
||||
}, 30_000);
|
||||
|
||||
it("hands a reassigned task to its new assignee once the previous owner's files are recovered", async () => {
|
||||
// Shape of a mid-run reassignment: the old owner's run was cancelled for
|
||||
// the reassignment, its copyback was retained, and its no-replay hold was
|
||||
// recorded after the new assignee's queued wakes had been admitted.
|
||||
const f = await seed();
|
||||
const previousOwnerId = (await run(f.runId)).agentId;
|
||||
await db.update(heartbeatRuns).set({ status: "cancelled", errorCode: "issue_reassigned" }).where(eq(heartbeatRuns.id, f.runId));
|
||||
const held = await action(f.actionId);
|
||||
await db.update(issueRecoveryActions).set({ status: "resolved", outcome: "blocked", resolvedAt: new Date(),
|
||||
fingerprint: `native-workspace:${f.runId}`, evidence: { ...held.evidence, continuationDelivery: "invalidated",
|
||||
automaticRecovery: { policy: "preserve_without_replay_v1", runId: f.runId, replay: "blocked", actionOutcome: "unknown" } },
|
||||
}).where(eq(issueRecoveryActions.id, f.actionId));
|
||||
const newOwnerId = randomUUID();
|
||||
await db.insert(agents).values({ id: newOwnerId, companyId: f.companyId, name: "Reviewer", adapterType: "paperclip_runner",
|
||||
status: "idle", runtimeConfig: { heartbeat: { maxConcurrentRuns: 1 } } });
|
||||
await db.update(issues).set({ assigneeAgentId: newOwnerId, status: "in_progress" }).where(eq(issues.id, f.issueId));
|
||||
// The new owner's queued wake reached the final dispatch gate after the hold appeared.
|
||||
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: newOwnerId, runtimeMode: "legacy",
|
||||
status: "cancelled", finishedAt: new Date(), invocationSource: "automation", errorCode: "execution_reconciliation_required",
|
||||
contextSnapshot: { issueId: f.issueId }, resultJson: { stopReason: "execution_reconciliation_required",
|
||||
timeoutSource: "stale_queued_run_gate", executionWait: { issueId: f.issueId, recoveryActionId: f.actionId } } });
|
||||
|
||||
// No second executor may touch the workspace while its files are unrecovered.
|
||||
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toMatchObject({ runId: f.runId, workspaceRepairRequired: true });
|
||||
|
||||
await recoverRetainedWorkspace(f.input);
|
||||
expect(await hasVerifiedWorkspaceRepair(db, await run(f.runId))).toBe(true);
|
||||
// The recovered hold no longer binds the board-chosen new owner...
|
||||
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toBeNull();
|
||||
// A different reconciliation hold for the same stopped run, without file
|
||||
// preservation evidence, still blocks the new owner.
|
||||
const [other] = await db.insert(issueRecoveryActions).values({ companyId: f.companyId, sourceIssueId: f.issueId,
|
||||
kind: "active_run_watchdog", ownerType: "board", cause: "native_continuation_requires_reconciliation",
|
||||
fingerprint: `other:${f.runId}`, nextAction: "Reconcile the stopped run.", evidence: { runId: f.runId } }).returning();
|
||||
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toMatchObject({ recoveryActionId: other.id });
|
||||
await db.delete(issueRecoveryActions).where(eq(issueRecoveryActions.id, other.id));
|
||||
// ...and the stranded-work sweep gives that owner a durable next action.
|
||||
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: newOwnerId, status: "running" });
|
||||
const pending = () => db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId),
|
||||
inArray(heartbeatRuns.status, ["queued", "scheduled_retry"])));
|
||||
await heartbeatService(db).reconcileStrandedAssignedIssues({ companyId: f.companyId, issueId: f.issueId });
|
||||
const successors = await pending();
|
||||
expect(successors).toHaveLength(1);
|
||||
expect(successors[0]).toMatchObject({ agentId: newOwnerId, contextSnapshot: { issueId: f.issueId } });
|
||||
// A fresh turn for the new owner, never a replay of the stopped run.
|
||||
expect(successors[0].retryOfRunId).not.toBe(f.runId);
|
||||
expect(successors[0].contextSnapshot?.previousRunId).not.toBe(f.runId);
|
||||
// Recovery stays bounded: a later sweep observes the pending turn instead of adding another.
|
||||
await heartbeatService(db).reconcileStrandedAssignedIssues({ companyId: f.companyId, issueId: f.issueId });
|
||||
expect((await pending()).map(item => item.id)).toEqual([successors[0].id]);
|
||||
// The release rests on the durable completed-recovery receipt. Later reuse
|
||||
// of the workspace by the new owner changes live workspace evidence, so the
|
||||
// live repair check stops verifying; the hold must not re-arm for that owner.
|
||||
const stopped = await run(f.runId);
|
||||
const sync = stopped.runnerProfileJson?.nativeWorkspaceSync as Record<string, unknown>;
|
||||
await db.update(heartbeatRuns).set({ runnerProfileJson: { ...stopped.runnerProfileJson,
|
||||
nativeWorkspaceSync: { ...sync, finalHostSha256: "f".repeat(64) } } }).where(eq(heartbeatRuns.id, f.runId));
|
||||
expect(await hasVerifiedWorkspaceRepair(db, await run(f.runId))).toBe(false);
|
||||
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toBeNull();
|
||||
await db.update(heartbeatRuns).set({ runnerProfileJson: stopped.runnerProfileJson }).where(eq(heartbeatRuns.id, f.runId));
|
||||
// The audit row and its no-replay hold stay intact for the previous owner.
|
||||
expect(await action(f.actionId)).toMatchObject({ status: "resolved", outcome: "blocked",
|
||||
evidence: { automaticRecovery: { replay: "blocked" } } });
|
||||
await db.update(heartbeatRuns).set({ status: "cancelled", finishedAt: new Date() }).where(eq(heartbeatRuns.id, successors[0].id));
|
||||
await db.update(issues).set({ assigneeAgentId: previousOwnerId }).where(eq(issues.id, f.issueId));
|
||||
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toMatchObject({ runId: f.runId, workspaceRepairRequired: false });
|
||||
}, 30_000);
|
||||
|
||||
it("keeps a snapshot-only recovery held for a new assignee because the task workspace lacks the saved files", async () => {
|
||||
const f = await seed("native", false);
|
||||
await db.update(heartbeatRuns).set({ status: "cancelled", errorCode: "issue_reassigned" }).where(eq(heartbeatRuns.id, f.runId));
|
||||
const held = await action(f.actionId);
|
||||
await db.update(issueRecoveryActions).set({ status: "resolved", outcome: "blocked", resolvedAt: new Date(),
|
||||
evidence: { ...held.evidence, automaticRecovery: { policy: "preserve_without_replay_v1", runId: f.runId, replay: "blocked" } },
|
||||
}).where(eq(issueRecoveryActions.id, f.actionId));
|
||||
const newOwnerId = randomUUID();
|
||||
await db.insert(agents).values({ id: newOwnerId, companyId: f.companyId, name: "Reviewer", adapterType: "paperclip_runner", status: "idle" });
|
||||
await db.update(issues).set({ assigneeAgentId: newOwnerId, status: "in_progress" }).where(eq(issues.id, f.issueId));
|
||||
await recoverRetainedWorkspace(f.input);
|
||||
expect(readSystemRecovery((await action(f.actionId)).evidence)?.state).toBe("complete");
|
||||
expect(await db.select().from(workspaceOperations).where(and(eq(workspaceOperations.heartbeatRunId, f.runId),
|
||||
eq(workspaceOperations.phase, "workspace_recovery_snapshot"), eq(workspaceOperations.status, "succeeded")))).toHaveLength(1);
|
||||
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toMatchObject({ recoveryActionId: f.actionId, runId: f.runId });
|
||||
}, 30_000);
|
||||
|
||||
it("projects active preservation without waiting input as working, and stale observations as failed", async () => {
|
||||
const f = await seed();
|
||||
const [issue] = await db.select().from(issues).where(eq(issues.id, f.issueId));
|
||||
|
||||
Reference in new issue
Block a user