mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-13 07:12:00 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native run on a remote sandbox can finish with an accepted result, and then fail to save its workspace. Saved-work recovery saves the files again from the retained sandbox, with no provider turn. > - #20758 made a save that finds the sandbox disk full stop at once as `sandbox_disk_full` and hand the task to the board with a Retry. > - When the user frees space and presses Retry, the save now succeeds. But the Retry then started a new agent turn (`strategy: continue`). That turn repeats work that already finished and spends provider budget again. The accepted result was not applied (`replacementDenied: explicit_user_continuation`). > - The decision for this case: when a durable record proves that the run's result was already accepted, Retry applies that saved result and starts no new turn. > - This pull request makes Retry admission wait for the run's saved-work holds and then applies the accepted result through the existing preserved-result finalization. > - The benefit is that a Retry after a failed save finishes the task from the work that was already done, with no repeated provider work or spend. ## Linked Issues or Issue Description Refs #20758 (full-disk save rests as `sandbox_disk_full` with a Retry), Refs #20810 (chaos fault DF1 for a full disk on the final workspace save) **What happened?** A native run had an accepted, succeeded result. Its workspace save failed because the sandbox disk was full, and the task rested at `sandbox_disk_full` with a Retry. The user freed space and pressed Retry. The save succeeded, and then Retry admission started a new run with `strategy: continue`. The accepted result stayed unapplied (the finalization coordinator stayed in `terminal_failure`). Two paths caused this: 1. Retry admission let the Retry through as soon as the saved files were verified. That can happen before preserved-result finalization runs: the recovery worker dispatches the Retry right after one hold settles, while the hold that Retry added for the same run is still scheduled. On master this is deterministic in that order, and a race otherwise. 2. Preserved-result finalization reads the newest saved-work hold of the run. While that hold was still scheduled, it declined, and nothing held the Retry back. **Expected behavior** After the save succeeds, and a durable record proves that the run's result was already accepted, Retry applies that result. The run and task finish from it. No new `turn.start` and no new provider spend. If the save fails again with a full disk, the task keeps the `sandbox_disk_full` rest and a working Retry. If there is no proven accepted result, Retry keeps the previous behaviour and starts one run. **Steps to reproduce** 1. Give a native run an accepted, succeeded result whose copy-back failed (`native_workspace_sync_out_retry_exhausted`). 2. Make the saved-work recovery save fail with `remote_workspace_sandbox_disk_full`. The hold goes to the board. 3. Free space. Press Retry. Let recovery save the files, then dispatch the Retry before preserved-result finalization runs. 4. On master, a new run starts with `previousRunId` set to the source run, and the accepted result is not applied. ## What Changed - `preserved-result-finalization.ts`: the eligibility checks move into one read (`readPreservedResult`). They are unchanged: an accepted, succeeded result of this run, a completed saved-work recovery with a verified receipt, no Stop, no newer run, no current execution, and no applied generation. A new read-only `canResumePreservedResultFinalization` uses the same read without locks. `resumePreservedResultFinalization` keeps the finalizer's lock order (coordinator first) and takes the issue and run rows with NOWAIT, so it cannot deadlock with the finalizer, owner-stop confirmation or Retry dispatch; a busy row is retried briefly, and the last attempt waits for its locks. It also writes an activity entry, `issue.saved_result_finalization_resumed`, when it applies a result. - `explicit-native-continuation.ts`: Retry admission holds an accepted Retry (`evaluate`) of a run whose saved files are verified and whose accepted result is still held, with the wait `finalizing_saved_result`. It holds while a saved-work hold of that run is still `scheduled`, `running` or `awaiting_evidence` and the result is otherwise applicable (no Stop, newer run, other owner or current execution), or once `canResumePreservedResultFinalization` is true. In every other case admission is unchanged. - `heartbeat/run-control.ts` (`resumeTaskRetry`): after dispatch, if admission held the Retry with `finalizing_saved_result`, it calls `resumePreservedResultFinalization`. That coalesces the Retry into the source run with `strategy: finalize_result` and re-arms finalization. This is one key read after admission; the resumption runs only for a held Retry. The recovery worker already calls the same function after each saved hold, so either path finishes the work. - Tests: `workspace-recovery-disk-full.test.ts` now proves the new Retry behaviour (red on master), a dispatch that lands while another hold of the run is still saving (red on master), a Retry while the disk is still full, and three negative cases that keep one new run. `workspace-recovery-preparation-gates.test.ts` (s12 and RC-50b) and `explicit-native-continuation.test.ts` (copy-back + `task_retry`) encoded the old "new run" result for the same situation. They now expect the saved result to be applied. A fixture helper `recoverRunHolds` runs the recovery steps for every hold of a run, as the worker does. - `ui/src/components/WorkspaceExportRecovery.tsx`: at the `sandbox_disk_full` rest, the retained-file panel no longer shows its own "Recover saved files" button and no longer hides the task's Retry. Before, the API said `canRetry: true` and the note said "Retry saves the files again", but the board showed only "Recover saved files" (seen in the DF1 chaos wave). The panel now says to free space and use Retry. A new UI test covers it. `pnpm check:token-gates` is clean. - `doc/execution-semantics.md`: documents the Retry behaviour and its conditions. ## Verification Red/green (`cd server && npx vitest run src/services/workspace-recovery-disk-full.test.ts`): - On master source with the new tests: 3 failed, 3 passed (the 3 negative cases pass on master, as they should). Each failure shows a second heartbeat run (the new agent turn): `expected [ …(2) ] to deeply equal [ Array(1) ]`. - On this branch: 7 passed. A fourth negative case (an operator Stop while a hold is still settling keeps one new run) was added after review; it also passes on master. - Guard removal 1: with the `canResumePreservedResultFinalization` check replaced by `true`, the negative test "an operator Stop recorded on the run" fails (`expected [] to have a length of 1 but got +0`: the Retry waits forever instead of starting its one run). - Guard removal 2: with the "hold still saving" check removed, the test "holds a Retry dispatched while another saved-work hold of the run is still settling" fails (a second run starts). Related suites pass on this branch: `workspace-recovery-disk-full`, `workspace-recovery-preparation-gates`, `workspace-recovery-admission`, `native-exhausted-finalization-retry`, `explicit-native-continuation` (641 tests). On the first revision, `task-retry`, `workspace-recovery-retry-copyback`, `workspace-recovery-warm-handoff`, `preserved-native-response-wait.integration`, `deferred-execution-wait` and `deferred-execution-receipt-work` passed (317 tests). Other retry and recovery suites were not run locally; required CI `general_tests` covers them. After the review fixes, the five related suites above passed again (642 tests). `npx tsc --noEmit -p server/tsconfig.json` is clean. `pnpm verify --explain` selects a broad plan (exit 2) because `doc/execution-semantics.md` is a global CI input. Required CI supplies that coverage. ## Risks - Behaviour change: an accepted Retry of a terminal run with an accepted, succeeded result and verified saved files now applies that result and starts no run. Before, it started one new run. This is the intended change. It also covers the s12 and RC-50b recovery-handoff cases, which used to start a new run. - A held Retry waits while any saved-work hold of the run is still saving. The recovery worker drives those holds to `complete` or `exhausted`. When a hold ends `exhausted` or the records do not prove the result, the wait ends and the Retry starts one run, as before. A hold that stays in `awaiting_evidence` keeps the Retry waiting with "Paperclip is finishing the saved result." - Upgrade-safe: no persisted shape changes. The checks read existing records (`native_run_results`, `native_run_finalizations`, the recovery action evidence, workspace receipts). A record written by an older server that lacks any required fact keeps the old behaviour. An older server that reads the new state sees a coalesced Retry and a re-armed finalization, which it already handles. - No new work before `turn.start` on the normal path. Admission adds the eligibility read only when it has already proven an accepted result and verified saved files for a Retry of that run. `resumeTaskRetry` adds one key read after admission. - Lock order: `resumePreservedResultFinalization` keeps the finalizer's order (coordinator, then issue, then run). Because it can now run beside Retry dispatch (which locks the issue first), it takes the issue and run rows with NOWAIT. A busy row ends the attempt, and it retries up to 9 times at 100 ms. The 10th attempt waits for its locks, as this resumption always did, so a recovered result is never left unapplied. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file editing, test execution) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>
3195 lines
250 KiB
TypeScript
3195 lines
250 KiB
TypeScript
import { legacyControllerBootId } from "./legacy-controller-lease.js";
|
|
import { execFile, spawn } from "node:child_process";
|
|
import { once } from "node:events";
|
|
import os from "node:os";
|
|
import { createSystemRecovery, failSystemRecovery, SystemRecoveryFailure } from "./system-recovery-state.js";
|
|
import { instanceSettingsService } from "./instance-settings.js";
|
|
import { createPostgresWakeQueueAdapter } from "../modules/wake-queue/adapters/postgres.js";
|
|
import { createPostgresRunDispatchAdapter } from "../modules/run-dispatch/adapters/postgres.js";
|
|
import { issueTreeControlService } from "./issue-tree-control.js";
|
|
import { createReleaseIssueExecution } from "../modules/wake-queue/application/use-cases.js";
|
|
import * as nativeExecutor from "./native-runtime/native-session-executor.js";
|
|
import { appendHeartbeatRunEvent } from "./heartbeat-run-events.js";
|
|
import { recordNativeLocalProcessStop, hasNativeLocalProcessStop, PRIOR_OWNER_FENCE_REFUSED, PROCESS_IDENTITY_RECORDED, PROCESS_START_REQUESTED } from "./native-local-process-stop.js";
|
|
import { remoteTerminationReceipt } from "./remote-execution-termination.js";
|
|
import { createHash, randomBytes, randomUUID } from "node:crypto";
|
|
import { mkdtemp, mkdir, writeFile, rm, utimes } from "node:fs/promises";
|
|
import { tmpdir } from "node:os";
|
|
import { join } from "node:path";
|
|
import { and, eq, sql } from "drizzle-orm";
|
|
import { beforeAll, afterAll, describe, it, expect, vi } from "vitest";
|
|
import {
|
|
applyPendingMigrations, approvals, issueApprovals, issueThreadInteractions, chatConversations, chatEndpoints, toolApplications, toolConnections,
|
|
agentWakeupRequests, agents, companies, createDb, heartbeatRunEvents, heartbeatRuns, issueComments, issueRecoveryActions,
|
|
issues, nativeRunFinalizations, nativeRunResults, completionContracts, environmentLeases, environments, issueRelations, issueTreeHolds, issueTreeHoldMembers, workspaceOperations, type Db,
|
|
} from "@paperclipai/db";
|
|
import { startEmbeddedPostgresTestDatabase, getEmbeddedPostgresTestSupport } from "../__tests__/helpers/embedded-postgres.js";
|
|
import { resolveExternalTestDatabaseUrl } from "../__tests__/helpers/external-test-database.js";
|
|
import { admitExplicitNativeContinuation } from "./explicit-native-continuation.js";
|
|
import { resumePreservedResultFinalization } from "./preserved-result-finalization.js";
|
|
import { adapterExecutionControls, createAdapterExecutionControl } from "./adapter-execution-control.js";
|
|
import { CONVERSATION_CONTINUATION_POLICY } from "./conversation-continuation.js";
|
|
import { buildExecutionContinuation } from "./execution-continuation.js";
|
|
import { verifiedWorkspaceRepairReceipt } from "./workspace-recovery-receipt.js";
|
|
import { heartbeatService, persistHeartbeatRunProcessMetadata, type HeartbeatEnvironmentRuntime } from "./heartbeat.js";
|
|
import { getExecutionBlocker } from "./execution-blocker.js";
|
|
import { createDurableChatWakeupRequest } from "./durable-chat-wakeup.js";
|
|
import { recordNativeFinalizationFailure } from "./native-runtime/native-run-finalizer.js";
|
|
const externalUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_CONTINUATION_TEST_DATABASE_URL");
|
|
const support = externalUrl ? { supported: true } : await getEmbeddedPostgresTestSupport();
|
|
(support.supported ? describe : describe.skip)("explicit native conversation continuation", () => {
|
|
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
|
let db: ReturnType<typeof createDb>;
|
|
beforeAll(async () => {
|
|
if (externalUrl) {
|
|
await applyPendingMigrations(externalUrl);
|
|
database = { connectionString: externalUrl, cleanup: async () => {} };
|
|
} else database = await startEmbeddedPostgresTestDatabase("explicit-native-message-");
|
|
db = createDb(database.connectionString);
|
|
}, 90_000);
|
|
afterAll(async () => { await db?.$client.end(); await database?.cleanup(); });
|
|
async function seed() {
|
|
const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID();
|
|
const sourceRunId = randomUUID(), successorRunId = randomUUID();
|
|
const commentId: string = randomUUID();
|
|
await db.insert(companies).values({ id: companyId, name: "Explicit turn", defaultResponsibleUserId: "board", issuePrefix: `E${companyId.replaceAll("-", "")}` });
|
|
await db.insert(agents).values({ id: agentId, companyId, name: "Native", role: "engineer", adapterType: "paperclip_runner", status: "idle", runtimeConfig: { heartbeat: { maxConcurrentRuns: 1 } } });
|
|
await db.insert(issues).values({ id: issueId, companyId, title: "Deploy", status: "blocked", assigneeAgentId: agentId });
|
|
await db.insert(heartbeatRuns).values({ id: sourceRunId, companyId, agentId,
|
|
nativeIssueId: issueId, runtimeMode: "native", status: "failed", processPid: 999999999,
|
|
contextSnapshot: { issueId }, finishedAt: new Date("2026-09-11T10:00:00Z") });
|
|
await db.insert(nativeRunFinalizations).values({ runId: sourceRunId, companyId, issueId,
|
|
phase: "terminal_failure", attempt: 3, failureDetail: { replacementDenied: "uncertain_external_action" } });
|
|
await db.insert(issueRecoveryActions).values({ companyId, sourceIssueId: issueId,
|
|
kind: "active_run_watchdog", cause: "uncertain_external_action", fingerprint: sourceRunId,
|
|
status: "resolved", outcome: "blocked", nextAction: "Automatic recovery stopped.",
|
|
evidence: { runId: sourceRunId, automaticRecovery: { replay: "blocked", actionOutcome: "unknown" } } });
|
|
await db.insert(issueComments).values({ id: commentId, companyId, issueId, authorType: "user",
|
|
authorUserId: "board", body: "What happened?", createdAt: new Date("2026-09-11T11:00:00Z") });
|
|
return { companyId, issueId, agentId, sourceRunId, commentId, successorRunId,
|
|
actorType: "user", actorId: "board", reason: "issue_commented" };
|
|
}
|
|
type Fixture = Awaited<ReturnType<typeof seed>>;
|
|
async function readAdmission(f: Fixture, full: boolean) {
|
|
const reads: Array<Array<Record<string, unknown>>> = [];
|
|
const recorded = new Proxy(db, { get(target, key) {
|
|
if (key !== "select") return Reflect.get(target, key);
|
|
return (...fields: Parameters<Db["select"]>) => {
|
|
const builder = target.select(...fields), from = builder.from.bind(builder);
|
|
builder.from = ((...args: Parameters<typeof from>) => {
|
|
const admissionRead = args[0] === heartbeatRuns && Object.hasOwn(fields[0] ?? {}, "runnerInstanceId");
|
|
const query = admissionRead && full ? target.select().from(heartbeatRuns) : from(...args);
|
|
const execute = query.execute.bind(query);
|
|
if (args[0] === heartbeatRuns) query.execute = async (...executeArgs: Parameters<typeof execute>) => {
|
|
const rows = await execute(...executeArgs), statement = query.toSQL().sql;
|
|
if (statement.includes('"id"::text') || statement.endsWith("for update")) reads.push(rows);
|
|
return rows;
|
|
};
|
|
return query;
|
|
}) as typeof builder.from;
|
|
return builder;
|
|
};
|
|
}});
|
|
const blocked: string[] = [];
|
|
const result = await admitExplicitNativeContinuation({ ...f, db: recorded, dryRun: true,
|
|
onBlocked: reason => blocked.push(reason) });
|
|
return { result, blocked, reads };
|
|
}
|
|
const PRIOR_OWNER_REFUSAL = "runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate; "
|
|
+ "retirement_refused=journal_commands_unsettled(turn.start.failed)";
|
|
it.each([
|
|
{ shape: "refused by the cleanup quarantine before launch", errorCode: "native_session_cleanup_quarantined", launched: false,
|
|
comment: "local_cleanup", retry: "cleanup_quarantined" },
|
|
{ shape: "quarantined after it requested a launch", errorCode: "native_session_cleanup_quarantined", launched: true,
|
|
comment: "process_identity_missing", retry: "process_identity_missing" },
|
|
// w-20261110T050818-s20386: a run that failed before it asked for a runner
|
|
// (here a failed preparation query) never had a process to stop.
|
|
{ shape: "failed before it requested a launch", errorCode: "adapter_failed", launched: false,
|
|
comment: null, retry: null },
|
|
{ shape: "failed after a runner event without a process record", errorCode: "adapter_failed", launched: false,
|
|
runnerEvent: true, comment: "process_identity_missing", retry: "process_identity_missing" },
|
|
{ shape: "failed before launch after its native session was prepared", errorCode: "adapter_failed", launched: false,
|
|
prepared: true, comment: "process_identity_missing", retry: "process_identity_missing" },
|
|
{ shape: "failed without a process record after native selection", errorCode: "adapter_failed", launched: false,
|
|
coordinator: true, comment: "process_identity_missing", retry: "process_identity_missing" },
|
|
// w-20261110T033426-s108/t03: a restart cut off the previous run's turn
|
|
// start, so its saved runner state could not be proven finished. The next
|
|
// run was refused at that runner-state fence before it asked for a runner,
|
|
// and Retry then waited forever for a stop record of a process that never
|
|
// existed.
|
|
{ shape: "was refused at the prior-owner runner-state fence before launch", errorCode: "adapter_failed", launched: false,
|
|
prepared: true, fenced: true, error: PRIOR_OWNER_REFUSAL, comment: null, retry: null },
|
|
{ shape: "was refused at the prior-owner runner-state fence with a coordinator", errorCode: "adapter_failed", launched: false,
|
|
prepared: true, coordinator: true, fenced: true, error: PRIOR_OWNER_REFUSAL, comment: null, retry: null },
|
|
{ shape: "was refused at the prior-owner runner-state fence after it requested a launch", errorCode: "adapter_failed", launched: true,
|
|
prepared: true, fenced: true, error: PRIOR_OWNER_REFUSAL, comment: "process_identity_missing", retry: "process_identity_missing" },
|
|
{ shape: "was refused at the prior-owner runner-state fence after a runner event", errorCode: "adapter_failed", launched: false,
|
|
runnerEvent: true, prepared: true, fenced: true, error: PRIOR_OWNER_REFUSAL, comment: "process_identity_missing", retry: "process_identity_missing" },
|
|
// The fence's own server-authored event is the proof, never the error text.
|
|
{ shape: "carries the prior-owner refusal text without the fence event", errorCode: "adapter_failed", launched: false,
|
|
prepared: true, error: PRIOR_OWNER_REFUSAL, comment: "process_identity_missing", retry: "process_identity_missing" },
|
|
{ shape: "failed its own runner-state authority check", errorCode: "adapter_failed", launched: false,
|
|
prepared: true, error: "runner_state_identity_mismatch: authority_indeterminate",
|
|
comment: "process_identity_missing", retry: "process_identity_missing" },
|
|
{ shape: "timed out before it requested a launch", status: "timed_out", errorCode: "timeout", launched: false,
|
|
comment: null, retry: null },
|
|
{ shape: "timed out after a runner event without a process record", status: "timed_out", errorCode: "timeout", launched: false,
|
|
runnerEvent: true, comment: "process_identity_missing", retry: "process_identity_missing" },
|
|
{ shape: "was interrupted without a process record after native selection", status: "interrupted", errorCode: "process_lost", launched: false,
|
|
coordinator: true, comment: "process_identity_missing", retry: "process_identity_missing" },
|
|
] as Array<{ shape: string; status?: "timed_out" | "interrupted"; errorCode: string; launched: boolean; runnerEvent?: boolean; prepared?: boolean; coordinator?: boolean;
|
|
error?: string; fenced?: boolean; comment: string | null; retry: string | null }>)("routes input after a run $shape to the gate that can clear it", async ({
|
|
status, errorCode, launched, runnerEvent, prepared, coordinator, error, fenced, comment, retry }) => {
|
|
// s7532/t04: a Retry's run failed at the quarantine fence before it asked
|
|
// for a runner. Input then waited for a stop record of a process that
|
|
// never existed ("process_identity_missing", Retry still offered) instead
|
|
// of the quarantine's own gate, which names the action that clears it.
|
|
const f = await seed();
|
|
await db.update(heartbeatRuns).set({ processPid: null, errorCode, ...(status ? { status } : {}), ...(error ? { error } : {}) })
|
|
.where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
// A run that failed before native selection has no native issue and no
|
|
// coordinator. A coordinator row requires the run's native issue (foreign
|
|
// key), so the coordinator rows keep it and are held by that evidence.
|
|
if (errorCode !== "native_session_cleanup_quarantined" && !coordinator) {
|
|
await db.delete(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.sourceRunId));
|
|
await db.update(heartbeatRuns).set({ nativeIssueId: null }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
}
|
|
if (launched) await db.insert(heartbeatRunEvents).values({ companyId: f.companyId, runId: f.sourceRunId, agentId: f.agentId,
|
|
seq: 1, eventType: "native.process_start_requested", stream: "system", level: "info", message: "launch requested",
|
|
payload: { hostname: "another-host" } });
|
|
if (runnerEvent) await db.insert(heartbeatRunEvents).values({ companyId: f.companyId, runId: f.sourceRunId, agentId: f.agentId,
|
|
seq: 1, eventType: "session.started", stream: "system", level: "info", message: "runner event",
|
|
sourceEventId: "runner-event-1", payload: {} });
|
|
if (fenced) await db.insert(heartbeatRunEvents).values({ companyId: f.companyId, runId: f.sourceRunId, agentId: f.agentId,
|
|
seq: 2, eventType: PRIOR_OWNER_FENCE_REFUSED, stream: "system", level: "warn", message: "fence refused before launch",
|
|
payload: { reason: "prior_owner_terminal_state_indeterminate" } });
|
|
if (prepared) await db.update(heartbeatRuns).set({ nativeIssueId: f.issueId, nativeSessionId: randomUUID(), runnerInstanceId: randomUUID() })
|
|
.where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
const { result, blocked } = await readAdmission(f, false);
|
|
if (comment === null) expect(result).not.toBeNull();
|
|
else expect(result).toBeNull();
|
|
expect(blocked).toEqual(comment === null ? [] : [comment]);
|
|
const retryBlocked: string[] = [];
|
|
const retried = await admitExplicitNativeContinuation({ ...f, db, dryRun: true, reason: "retry_failed_run",
|
|
failedRunId: f.sourceRunId, onBlocked: reason => retryBlocked.push(reason) });
|
|
if (retry === null) expect(retried).not.toBeNull();
|
|
else expect(retried).toBeNull();
|
|
expect(retryBlocked).toEqual(retry === null ? [] : [retry]);
|
|
});
|
|
|
|
it("keeps admission and the complete locked session profile while omitting prompt data", async () => {
|
|
const f = await seed();
|
|
const payload = randomBytes(128 * 1024).toString("base64");
|
|
const profile = { adapterDispatch: { adapterType: "paperclip_runner", adapterConfig: { instructions: payload } },
|
|
nativeExecutionInput: { task: { description: payload } }, sessionCheckpoint: { providerSessionId: "retained" },
|
|
nativeToolContractFingerprint: "retained-fingerprint" };
|
|
await db.update(heartbeatRuns).set({ contextSnapshot: { issueId: f.issueId, prompt: payload },
|
|
runnerProfileJson: profile }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
const baseline = await readAdmission(f, true), projected = await readAdmission(f, false);
|
|
expect(projected.result).toEqual(baseline.result);
|
|
expect(projected.result).toMatchObject({ previousRunId: f.sourceRunId, commentId: f.commentId });
|
|
expect(projected.blocked).toEqual(baseline.blocked);
|
|
expect(projected.reads).toHaveLength(2);
|
|
expect(projected.reads.map(rows => rows[0].contextSnapshot)).toEqual([{ issueId: f.issueId }, { issueId: f.issueId }]);
|
|
expect(projected.reads[0][0].runnerProfileJson).toEqual({ adapterDispatch: { adapterType: "paperclip_runner" } });
|
|
expect(projected.reads[1][0].runnerProfileJson).toEqual(profile);
|
|
expect(Buffer.byteLength(JSON.stringify(projected.reads))).toBeLessThan(Buffer.byteLength(JSON.stringify(baseline.reads)) / 2);
|
|
const [stored] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
expect(stored.contextSnapshot).toEqual({ issueId: f.issueId, prompt: payload });
|
|
expect(stored.runnerProfileJson).toEqual(profile);
|
|
});
|
|
it.each([null, [], "scalar", 42, {}, { issueId: 42, wakeCommentId: false, wakeCommentIds: [null, 42],
|
|
source: null, adapterDispatch: { adapterType: 42 }, nativeWorkspaceSync: [] }].map(value => ({ value })))(
|
|
"preserves a controller hold with missing or malformed JSON ($value)", async ({ value }) => {
|
|
const f = await seed();
|
|
await db.update(heartbeatRuns).set({ contextSnapshot: value as Record<string, unknown> | null,
|
|
runnerProfileJson: value as Record<string, unknown> | null }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
await db.update(nativeRunFinalizations).set({ leaseOwner: "still-owned" }).where(eq(nativeRunFinalizations.runId, f.sourceRunId));
|
|
const baseline = await readAdmission(f, true), projected = await readAdmission(f, false);
|
|
expect(projected.result).toEqual(baseline.result);
|
|
expect(projected.blocked).toEqual(baseline.blocked);
|
|
expect(projected.blocked).toEqual(["controller_settling"]);
|
|
expect(projected.reads).toHaveLength(2);
|
|
expect(projected.reads[1][0].runnerProfileJson).toEqual(baseline.reads[1][0].runnerProfileJson);
|
|
},
|
|
);
|
|
it.each(["new", "saved"])("delivers a reassignment's saved user question once after the native run stopped (%s)", async origin => {
|
|
const f = await seed();
|
|
await db.update(issues).set({ status: "in_review" }).where(eq(issues.id, f.issueId));
|
|
const [source] = await db.update(heartbeatRuns).set({ status: "cancelled", errorCode: "issue_reassigned",
|
|
processPid: null, processGroupId: null,
|
|
}).where(eq(heartbeatRuns.id, f.sourceRunId)).returning();
|
|
await db.update(issueRecoveryActions).set({ cause: "native_continuation_requires_reconciliation" })
|
|
.where(eq(issueRecoveryActions.sourceIssueId, f.issueId));
|
|
const lease = { id: randomUUID(), companyId: f.companyId, heartbeatRunId: source.id,
|
|
provider: "daytona", providerLeaseId: "reviewed-task-sandbox" };
|
|
await db.insert(environmentLeases).values({ ...lease, status: "expired", leasePolicy: "ephemeral",
|
|
releasedAt: new Date(), cleanupStatus: "success", metadata: {
|
|
remoteExecutionTermination: remoteTerminationReceipt(lease, { providerLeaseId: lease.providerLeaseId, state: "destroyed" }),
|
|
},
|
|
});
|
|
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" });
|
|
const heartbeat = heartbeatService(db);
|
|
if (origin === "new") {
|
|
await heartbeat.wakeup(f.agentId, { source: "assignment", triggerDetail: "system", reason: "issue_assigned",
|
|
requestedByActorType: "user", requestedByActorId: f.actorId,
|
|
payload: { issueId: f.issueId, commentId: f.commentId, mutation: "update" },
|
|
contextSnapshot: { source: "issue.update", issueId: f.issueId, wakeCommentId: f.commentId },
|
|
});
|
|
} else {
|
|
await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: f.agentId,
|
|
source: "assignment", triggerDetail: "system", reason: "issue_assigned", status: "deferred_issue_execution",
|
|
requestedByActorType: "user", requestedByActorId: f.actorId, payload: {
|
|
issueId: f.issueId, commentId: f.commentId, executionWait: { reason: "execution_recovery" },
|
|
_paperclipWakeContext: { source: "issue.update", issueId: f.issueId, wakeReason: "issue_assigned",
|
|
wakeCommentId: f.commentId, wakeCommentIds: [f.commentId] },
|
|
},
|
|
});
|
|
}
|
|
const [receipt] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.companyId));
|
|
if (receipt.status === "deferred_issue_execution") {
|
|
await db.update(agentWakeupRequests).set({ updatedAt: new Date(0) }).where(eq(agentWakeupRequests.id, receipt.id));
|
|
await Promise.all([heartbeat.resumeExecutionWaitComments(), heartbeat.resumeExecutionWaitComments()]);
|
|
}
|
|
const [delivered] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, receipt.id));
|
|
expect(delivered).toMatchObject({ status: origin === "new" ? "queued" : "coalesced", runId: expect.any(String) });
|
|
const successors = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId),
|
|
eq(heartbeatRuns.status, "queued")));
|
|
expect(successors).toHaveLength(1);
|
|
expect(delivered.runId).toBe(successors[0].id);
|
|
expect(successors[0]).toMatchObject({ agentId: f.agentId, contextSnapshot: {
|
|
forceFreshSession: true, previousRunId: source.id, wakeCommentIds: [f.commentId],
|
|
} });
|
|
await heartbeat.resumeRemoteStopComments(source);
|
|
expect(await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId),
|
|
eq(heartbeatRuns.status, "queued")))).toHaveLength(1);
|
|
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toBeNull();
|
|
const [original] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, source.id));
|
|
expect(original).toMatchObject({ status: "cancelled", errorCode: "issue_reassigned" });
|
|
});
|
|
it("keeps a bare human assignment behind the execution hold", async () => {
|
|
const f = await seed();
|
|
expect(await admitExplicitNativeContinuation({ ...f, db, reason: "issue_assigned", commentId: null })).toBeNull();
|
|
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).not.toBeNull();
|
|
expect(await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId),
|
|
eq(heartbeatRuns.status, "queued")))).toHaveLength(0);
|
|
});
|
|
it("automatically admits a saved historical local message once after isolated fallback becomes eligible", async () => {
|
|
const f = await seed();
|
|
const processStartedAt = new Date();
|
|
const child = execFile(process.execPath, ["-e", "process.exit(0)"]);
|
|
await new Promise(resolve => child.on("exit", resolve));
|
|
await db.delete(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.sourceRunId));
|
|
const [source] = await db.update(heartbeatRuns).set({
|
|
runtimeMode: "legacy", nativeIssueId: null, issueId: f.issueId,
|
|
processPid: child.pid, processGroupId: null, processStartedAt,
|
|
runnerProfileJson: { adapterDispatch: { adapterType: "codex_local" } },
|
|
resultJson: { workspaceRestoreFailure: "restore_failed" },
|
|
}).where(eq(heartbeatRuns.id, f.sourceRunId)).returning();
|
|
await appendHeartbeatRunEvent(db, { companyId: f.companyId, runId: source.id, agentId: f.agentId,
|
|
eventType: "native.process_identity_recorded", stream: "system", level: "info",
|
|
message: "Test child exited", payload: { targetKind: "local", hostname: os.hostname(),
|
|
processPid: child.pid, processGroupId: null, processStartedAt: processStartedAt.toISOString() } });
|
|
await db.update(issueRecoveryActions).set({ status: "active", ownerType: "system",
|
|
cause: "legacy_execution_requires_reconciliation", returnOwnerAgentId: f.agentId,
|
|
evidence: { runId: source.id, systemRecovery: failSystemRecovery(createSystemRecovery(source.id),
|
|
new SystemRecoveryFailure("historical_source_missing", "use_alternative_strategy")) },
|
|
}).where(eq(issueRecoveryActions.sourceIssueId, f.issueId));
|
|
// Keep execution queued so this admission test does not invoke a provider.
|
|
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" });
|
|
const [wake] = await db.insert(agentWakeupRequests).values({
|
|
companyId: f.companyId, agentId: f.agentId, source: "on_demand", reason: "issue_commented",
|
|
requestedByActorType: "user", requestedByActorId: "board", status: "deferred_issue_execution",
|
|
payload: { issueId: f.issueId, commentId: f.commentId, _paperclipWakeContext: {
|
|
issueId: f.issueId, wakeReason: "issue_commented", wakeCommentId: f.commentId, wakeCommentIds: [f.commentId],
|
|
} },
|
|
}).returning();
|
|
const service = heartbeatService(db);
|
|
await service.resumeRemoteStopComments(source);
|
|
await service.resumeRemoteStopComments(source);
|
|
const [receipt] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, wake.id));
|
|
expect(receipt).toMatchObject({ status: "coalesced" });
|
|
const runs = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId),
|
|
sql`${heartbeatRuns.contextSnapshot}->>'wakeCommentId' = ${f.commentId}`));
|
|
expect(runs).toHaveLength(1);
|
|
expect(runs[0].contextSnapshot?.explicitUserContinuation).toMatchObject({
|
|
readiness: { mode: "fresh_isolated_workspace", retryIntentId: wake.id, sourceRunId: source.id },
|
|
});
|
|
}, 30_000);
|
|
it.each(["handoff", "foreign_task", "running_source", "different_owner", "mention", "interaction", "chat"])("adopts former-owner comments only during an authorized handoff (%s)", async kind => {
|
|
const f = await seed(), nextAgentId = randomUUID(), queueId = randomUUID();
|
|
await db.delete(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issueId));
|
|
await db.delete(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.sourceRunId));
|
|
await db.insert(agents).values({ id: nextAgentId, companyId: f.companyId, name: "Replacement", role: "engineer", adapterType: "paperclip_runner", runtimeConfig: { heartbeat: { maxConcurrentRuns: 1 } } });
|
|
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: nextAgentId, status: "running" });
|
|
await db.update(issues).set({ status: "in_progress", assigneeAgentId: kind === "different_owner" ? f.agentId : nextAgentId }).where(eq(issues.id, f.issueId));
|
|
await db.update(heartbeatRuns).set({ status: kind === "running_source" ? "running" : "cancelled", errorCode: "issue_reassigned",
|
|
nativeIssueId: kind === "foreign_task" ? null : f.issueId,
|
|
contextSnapshot: { issueId: kind === "foreign_task" ? randomUUID() : f.issueId } }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
const secondId = randomUUID();
|
|
await db.insert(issueComments).values({ id: secondId, companyId: f.companyId, issueId: f.issueId, authorType: "user", authorUserId: "second-user", body: "Preserve the existing draft." });
|
|
await db.insert(agentWakeupRequests).values({ id: queueId, companyId: f.companyId, agentId: f.agentId,
|
|
source: "automation", reason: "issue_execution_deferred", status: "deferred_issue_execution",
|
|
requestedByActorType: "user", requestedByActorId: "board", idempotencyKey: kind === "chat" ? "chat-inbound:handoff-test" : null,
|
|
payload: { issueId: f.issueId, commentId: secondId, _paperclipWakeContext: { issueId: f.issueId,
|
|
wakeReason: kind === "mention" ? "issue_comment_mentioned" : "issue_commented", wakeCommentIds: [f.commentId, secondId],
|
|
...(kind === "interaction" ? { interactionId: randomUUID(), wakeReason: "connection_intent.resolved" } : {}),
|
|
} },
|
|
});
|
|
await heartbeatService(db).wakeup(nextAgentId, { source: "assignment", triggerDetail: "system", reason: "issue_assigned",
|
|
requestedByActorType: "user", requestedByActorId: "board", payload: { issueId: f.issueId, interruptedRunId: f.sourceRunId },
|
|
contextSnapshot: { issueId: f.issueId, interruptedRunId: f.sourceRunId } });
|
|
const [receipt] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, queueId));
|
|
if (kind === "handoff") {
|
|
expect(receipt.status).toBe("coalesced");
|
|
const [successor] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, receipt.runId!));
|
|
expect(successor).toMatchObject({ agentId: nextAgentId, status: "queued", contextSnapshot: { wakeCommentIds: [f.commentId, secondId] } });
|
|
expect(receipt.requestedByActorId).toBe("board");
|
|
const [comment] = await db.select().from(issueComments).where(eq(issueComments.id, secondId));
|
|
expect(comment.authorUserId).toBe("second-user");
|
|
} else expect(receipt.status).toBe("deferred_issue_execution");
|
|
});
|
|
it.each(["ready", "unacknowledged", "pause", "resolved_recovery", "controller", "process_running", "identity_missing", "remote_pending", "remote_stopped", "first_delivered", "last_delivered", "mixed_authors"])("retains pre-Stop input and delivers once only after a later authorized message (%s)", async gate => {
|
|
const f = await seed();
|
|
if (gate !== "resolved_recovery") await db.delete(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issueId));
|
|
await db.update(issues).set({ status: "in_progress" }).where(eq(issues.id, f.issueId));
|
|
await db.update(nativeRunFinalizations).set({ phase: "terminal_failure", failureDetail: null, leaseOwner: gate === "controller" ? "still-cleaning" : null })
|
|
.where(eq(nativeRunFinalizations.runId, f.sourceRunId));
|
|
await db.update(issueComments).set({ createdAt: new Date("2026-09-11T09:00:00Z") })
|
|
.where(eq(issueComments.id, f.commentId));
|
|
const [source] = await db.update(heartbeatRuns).set({ status: "cancelled",
|
|
processPid: gate === "process_running" ? process.pid : gate === "identity_missing" ? null : 999999999, resultJson: {
|
|
cancelledByActorType: "user", cancelledByUserId: "board", nativeCancellation: {
|
|
schema: "paperclip.native-cancellation.v1", runId: f.sourceRunId, companyId: f.companyId,
|
|
issueId: f.issueId, scope: "run", reasonCode: "cancellation_run_only", dispatched: true,
|
|
dispatchState: gate === "unacknowledged" ? "requested" : "acknowledged",
|
|
intentAuditId: randomUUID(), acknowledgementAuditId: randomUUID(),
|
|
},
|
|
} }).where(eq(heartbeatRuns.id, f.sourceRunId)).returning();
|
|
// Occupy the agent slot: admission is real, but no provider should launch.
|
|
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" });
|
|
if (gate === "pause") {
|
|
const holdId = randomUUID();
|
|
await db.insert(issueTreeHolds).values({ id: holdId, companyId: f.companyId, rootIssueId: f.issueId, mode: "pause", status: "active" });
|
|
await db.insert(issueTreeHoldMembers).values({ companyId: f.companyId, holdId, issueId: f.issueId, depth: 0, issueTitle: "Deploy", issueStatus: "in_progress" });
|
|
}
|
|
let queuedIds = [f.commentId];
|
|
let expectedIds = queuedIds;
|
|
if (["first_delivered", "last_delivered", "mixed_authors"].includes(gate)) {
|
|
const secondId = randomUUID();
|
|
await db.insert(issueComments).values({ id: secondId, companyId: f.companyId, issueId: f.issueId,
|
|
authorType: "user", authorUserId: gate === "mixed_authors" ? "second-author" : "board",
|
|
body: "Keep the earlier direction too.", createdAt: new Date("2026-09-11T09:01:00Z") });
|
|
queuedIds = [f.commentId, secondId];
|
|
expectedIds = gate === "first_delivered" ? [secondId] : gate === "last_delivered" ? [f.commentId] : queuedIds;
|
|
if (gate !== "mixed_authors") await db.update(heartbeatRuns).set({ startedAt: new Date("2026-09-11T09:02:00Z"),
|
|
contextSnapshot: { issueId: f.issueId, wakeCommentIds: gate === "first_delivered" ? [f.commentId] : [secondId] },
|
|
}).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
}
|
|
const queueId = randomUUID();
|
|
await db.insert(agentWakeupRequests).values({ id: queueId, companyId: f.companyId, agentId: f.agentId,
|
|
source: "automation", triggerDetail: "system", reason: "issue_execution_deferred", status: "deferred_issue_execution",
|
|
requestedByActorType: "user", requestedByActorId: "board", payload: {
|
|
issueId: f.issueId, commentId: queuedIds.at(-1), _paperclipWakeContext: { issueId: f.issueId, wakeReason: "issue_commented", wakeCommentId: queuedIds.at(-1), wakeCommentIds: queuedIds },
|
|
},
|
|
});
|
|
if (gate.startsWith("remote_")) {
|
|
const identity = { id: randomUUID(), companyId: f.companyId, heartbeatRunId: f.sourceRunId,
|
|
provider: "daytona", providerLeaseId: "owned-sandbox" };
|
|
await db.insert(environmentLeases).values({ ...identity, status: gate === "remote_stopped" ? "released" : "active",
|
|
leasePolicy: "ephemeral", releasedAt: gate === "remote_stopped" ? new Date() : null,
|
|
cleanupStatus: gate === "remote_stopped" ? "success" : null,
|
|
metadata: gate === "remote_stopped" ? { remoteExecutionTermination: remoteTerminationReceipt(identity,
|
|
{ providerLeaseId: identity.providerLeaseId, state: "stopped" }) } : {},
|
|
});
|
|
}
|
|
const heartbeat = heartbeatService(db);
|
|
await heartbeat.resumeRemoteStopComments(source);
|
|
await heartbeat.resumeRemoteStopComments(source);
|
|
expect(await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId), eq(heartbeatRuns.status, "queued")))).toHaveLength(0);
|
|
expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, queueId)))[0])
|
|
.toMatchObject({ status: "deferred_issue_execution", runId: null });
|
|
const resumeId = randomUUID();
|
|
await db.insert(issueComments).values({ id: resumeId, companyId: f.companyId, issueId: f.issueId,
|
|
authorType: "user", authorUserId: "board", body: "Resume with my earlier direction.", createdAt: new Date("2026-09-11T11:00:00Z") });
|
|
queuedIds = [...queuedIds, resumeId];
|
|
expectedIds = [...expectedIds, resumeId];
|
|
await db.update(agentWakeupRequests).set({ payload: {
|
|
issueId: f.issueId, commentId: resumeId, _paperclipWakeContext: { issueId: f.issueId, wakeReason: "issue_commented",
|
|
wakeCommentId: resumeId, wakeCommentIds: queuedIds },
|
|
} }).where(eq(agentWakeupRequests.id, queueId));
|
|
await heartbeat.resumeRemoteStopComments(source);
|
|
await heartbeat.resumeRemoteStopComments(source);
|
|
const successors = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId), eq(heartbeatRuns.status, "queued")));
|
|
expect(successors).toHaveLength(["ready", "remote_stopped", "first_delivered", "last_delivered", "mixed_authors", "resolved_recovery"].includes(gate) ? 1 : 0);
|
|
const [wake] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, queueId));
|
|
if (["ready", "remote_stopped", "first_delivered", "last_delivered", "mixed_authors", "resolved_recovery"].includes(gate)) {
|
|
expect(wake).toMatchObject({ status: "coalesced", runId: successors[0].id, requestedByActorId: "board" });
|
|
expect(successors[0].contextSnapshot).toMatchObject({ wakeCommentIds: expectedIds });
|
|
if (gate === "resolved_recovery") expect(successors[0].contextSnapshot).toMatchObject({ forceFreshSession: true, previousRunId: f.sourceRunId });
|
|
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toBeNull();
|
|
} else expect(wake.status).toBe("deferred_issue_execution");
|
|
});
|
|
it.each(["pending", "failed", "historical", "shared", "retained"])("an explicit queued interrupt retries only its stopped sandbox, without granting automatic retries (%s)", async scenario => {
|
|
const fails = scenario === "failed";
|
|
const protectedLease = scenario === "shared" || scenario === "retained";
|
|
const f = await seed(), other = await seed();
|
|
await db.update(agents).set({ adapterType: "claude_local" }).where(eq(agents.id, f.agentId));
|
|
await db.delete(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.sourceRunId));
|
|
await db.update(heartbeatRuns).set({ runtimeMode: "legacy", nativeIssueId: null, processPid: null })
|
|
.where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
await db.update(issueRecoveryActions).set({ cause: "legacy_execution_requires_reconciliation" })
|
|
.where(eq(issueRecoveryActions.sourceIssueId, f.issueId));
|
|
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" });
|
|
const queueId = randomUUID();
|
|
await db.insert(agentWakeupRequests).values({ id: queueId, companyId: f.companyId, agentId: f.agentId,
|
|
source: "automation", reason: "issue_commented", status: "deferred_issue_execution",
|
|
requestedByActorType: "system", payload: { issueId: f.issueId, commentId: f.commentId,
|
|
_paperclipWakeContext: { wakeCommentIds: [f.commentId] },
|
|
queuedCommentInterrupt: { actorId: "board", requestedAt: new Date().toISOString() } },
|
|
});
|
|
const identities = [f, other].map(fixture => ({ id: randomUUID(), companyId: fixture.companyId,
|
|
heartbeatRunId: fixture.sourceRunId, provider: "daytona", providerLeaseId: fixture.sourceRunId }));
|
|
for (const identity of identities) await db.insert(environmentLeases).values({ ...identity,
|
|
status: "pending_cleanup", leasePolicy: "ephemeral", releasedAt: new Date(), cleanupStatus: "failed",
|
|
metadata: { pendingCleanupRetryAttempts: 5, pendingCleanupRetryCapWarned: true } });
|
|
if (scenario === "historical" || protectedLease) await db.update(environmentLeases).set({
|
|
status: "failed", cleanupStatus: "success",
|
|
}).where(eq(environmentLeases.id, identities[0].id));
|
|
if (scenario === "shared") await db.update(environmentLeases).set({
|
|
providerLeaseId: identities[0].providerLeaseId, status: "active", releasedAt: null,
|
|
}).where(eq(environmentLeases.id, identities[1].id));
|
|
if (scenario === "retained") await db.update(environmentLeases).set({
|
|
status: "retained", leasePolicy: "retain_on_failure",
|
|
}).where(eq(environmentLeases.id, identities[0].id));
|
|
const attempted: string[] = [];
|
|
const heartbeat = heartbeatService(db, { environmentRuntime: {
|
|
isPendingCleanupWorkerReady: async () => true,
|
|
retryPendingSandboxTeardown: async ({ lease }: { lease: { id: string; providerLeaseId: string } }) => {
|
|
attempted.push(lease.id);
|
|
if (fails) throw new Error("Provider unavailable");
|
|
return { providerLeaseId: lease.providerLeaseId, state: "destroyed" };
|
|
},
|
|
} as unknown as HeartbeatEnvironmentRuntime });
|
|
try {
|
|
await heartbeat.resumeQueuedCommentInterrupt(f.companyId, queueId);
|
|
expect(attempted).toEqual([]);
|
|
await heartbeat.resumeQueuedCommentInterrupt(f.companyId, queueId, { retryCleanup: true });
|
|
expect(attempted).toEqual(protectedLease ? [] : [identities[0].id]);
|
|
await heartbeat.resumeQueuedCommentInterrupt(f.companyId, queueId);
|
|
expect(attempted).toHaveLength(protectedLease ? 0 : 1);
|
|
const [queue] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, queueId));
|
|
expect(queue.status).toBe(fails || protectedLease ? "deferred_issue_execution" : "coalesced");
|
|
expect(Boolean(queue.runId)).toBe(!fails && !protectedLease);
|
|
const [untouched] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, identities[1].id));
|
|
expect(untouched).toMatchObject({ status: scenario === "shared" ? "active" : "pending_cleanup", metadata: { pendingCleanupRetryAttempts: 5 } });
|
|
} finally {
|
|
for (const identity of identities) await db.delete(environmentLeases).where(eq(environmentLeases.id, identity.id));
|
|
}
|
|
});
|
|
it.each(["fresh", "predates_stop", "malformed", "wrong_actor", "provider_alive", "identity_missing", "no_click"])("requires fresh explicit Send intent for a stopped native queue: %s", async (kind) => {
|
|
const f = await seed(), queueId = randomUUID();
|
|
await db.update(heartbeatRuns).set({ status: "cancelled", errorCode: "cancelled",
|
|
processPid: kind === "provider_alive" ? process.pid : kind === "identity_missing" ? null : 999999999,
|
|
}).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
await db.update(issueComments).set({ authorUserId: "original-author", createdAt: new Date("2026-09-11T09:00:00Z") })
|
|
.where(eq(issueComments.id, f.commentId));
|
|
await db.insert(agentWakeupRequests).values({ id: queueId, companyId: f.companyId, agentId: f.agentId,
|
|
source: "automation", reason: "issue_execution_deferred", status: "deferred_issue_execution",
|
|
requestedByActorType: "system", requestedByActorId: "heartbeat",
|
|
payload: { issueId: f.issueId, _paperclipWakeContext: { wakeCommentIds: [f.commentId] },
|
|
...(kind === "no_click" ? {} : { queuedCommentInterrupt: {
|
|
actorId: kind === "wrong_actor" ? "someone-else" : "board",
|
|
requestedAt: kind === "malformed" ? "invalid" : kind === "predates_stop" ? "2026-09-11T09:30:00Z" : "2026-09-11T11:00:00Z",
|
|
} }),
|
|
},
|
|
});
|
|
const blocked: string[] = [];
|
|
const result = await db.transaction(async tx => admitExplicitNativeContinuation({ ...f, db: tx as unknown as typeof db,
|
|
queuedCommentInterruptId: queueId, dryRun: true, onBlocked: reason => blocked.push(reason) }));
|
|
if (kind === "fresh") expect(result, JSON.stringify(blocked)).toMatchObject({ previousRunId: f.sourceRunId, commentId: f.commentId });
|
|
else expect(result).toBeNull();
|
|
expect((await db.select().from(issueComments).where(eq(issueComments.id, f.commentId)))[0].authorUserId).toBe("original-author");
|
|
});
|
|
it.each(["clean", "pending", "live_process", "controller"])("checks the latest native Stop despite an older resolved recovery action: %s", async kind => {
|
|
const f = await seed(), latestId = randomUUID(), queueId = randomUUID();
|
|
await db.update(heartbeatRuns).set({ createdAt: new Date("2026-09-11T08:00:00Z") }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
await db.insert(heartbeatRuns).values({ id: latestId, companyId: f.companyId, agentId: f.agentId, nativeIssueId: f.issueId,
|
|
runtimeMode: "native", status: "cancelled", errorCode: "cancelled", processPid: kind === "live_process" ? process.pid : 999999999,
|
|
contextSnapshot: { issueId: f.issueId }, createdAt: new Date("2026-09-11T10:15:00Z"), finishedAt: new Date("2026-09-11T10:30:00Z") });
|
|
await db.insert(nativeRunFinalizations).values({ runId: latestId, companyId: f.companyId, issueId: f.issueId,
|
|
phase: "terminal_failure", leaseOwner: kind === "controller" ? "active-controller" : null });
|
|
const lease = { id: randomUUID(), companyId: f.companyId, heartbeatRunId: latestId, provider: kind === "live_process" ? "local" : "daytona", providerLeaseId: randomUUID() };
|
|
await db.insert(environmentLeases).values({ ...lease, status: kind === "pending" ? "pending_cleanup" : "released", releasedAt: new Date(),
|
|
cleanupStatus: kind === "pending" ? "failed" : "success", metadata: { remoteExecutionTermination: remoteTerminationReceipt(lease, { providerLeaseId: lease.providerLeaseId, state: "destroyed" }) } });
|
|
await db.insert(agentWakeupRequests).values({ id: queueId, companyId: f.companyId, agentId: f.agentId, source: "on_demand", reason: "issue_commented",
|
|
status: "deferred_issue_execution", payload: { issueId: f.issueId, _paperclipWakeContext: { wakeCommentIds: [f.commentId] },
|
|
queuedCommentInterrupt: { actorId: "board", requestedAt: "2026-09-11T11:30:00Z" } } });
|
|
const result = await db.transaction(tx => admitExplicitNativeContinuation({ ...f, db: tx as unknown as typeof db, queuedCommentInterruptId: queueId, dryRun: true }));
|
|
if (kind === "clean") expect(result).toMatchObject({ previousRunId: latestId });
|
|
else expect(result).toBeNull();
|
|
await db.delete(environmentLeases).where(eq(environmentLeases.id, lease.id));
|
|
});
|
|
|
|
it.each(["clean", "pending", "missing_receipt", "wrong_resource", "controller", "workspace_owner", "cleanup_in_flight"])("checks native Stop cleanup without a recovery action: %s", async kind => {
|
|
const f = await seed(), queueId = randomUUID();
|
|
await db.delete(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issueId));
|
|
await db.update(heartbeatRuns).set({ status: "cancelled", errorCode: "cancelled" }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
if (kind === "workspace_owner") await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeWorkspaceFinalizationOwner: "other-controller" } }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
if (kind === "controller") await db.update(nativeRunFinalizations).set({ leaseOwner: "other-controller" }).where(eq(nativeRunFinalizations.runId, f.sourceRunId));
|
|
const lease = { id: randomUUID(), companyId: f.companyId, heartbeatRunId: f.sourceRunId,
|
|
provider: "daytona", providerLeaseId: randomUUID() };
|
|
await db.insert(environmentLeases).values({ ...lease, status: kind === "pending" ? "pending_cleanup" : "released",
|
|
releasedAt: new Date(), cleanupStatus: kind === "pending" ? "failed" : "success", metadata: kind === "missing_receipt" ? {} : {
|
|
...(kind === "cleanup_in_flight" ? { pendingCleanupInFlight: true } : {}),
|
|
remoteExecutionTermination: remoteTerminationReceipt(lease, {
|
|
providerLeaseId: kind === "wrong_resource" ? randomUUID() : lease.providerLeaseId, state: "destroyed",
|
|
}),
|
|
} });
|
|
await db.insert(agentWakeupRequests).values({ id: queueId, companyId: f.companyId, agentId: f.agentId,
|
|
source: "on_demand", reason: "issue_commented", status: "deferred_issue_execution",
|
|
payload: { issueId: f.issueId, _paperclipWakeContext: { wakeCommentIds: [f.commentId] },
|
|
queuedCommentInterrupt: { actorId: "board", requestedAt: "2026-09-11T11:30:00Z" } },
|
|
});
|
|
const blocked: string[] = [];
|
|
const result = await db.transaction(async tx => admitExplicitNativeContinuation({ ...f, db: tx as unknown as typeof db,
|
|
queuedCommentInterruptId: queueId, dryRun: true, onBlocked: reason => blocked.push(reason) }));
|
|
if (kind === "clean") expect(result).toMatchObject({ previousRunId: f.sourceRunId });
|
|
else { expect(result).toBeNull(); expect(blocked).not.toEqual([]); }
|
|
await db.delete(environmentLeases).where(eq(environmentLeases.id, lease.id));
|
|
});
|
|
it.each([true, false])("keeps a native Send behind cleanup through public admission without a recovery action: %s", async pending => {
|
|
const f = await seed(), queueId = randomUUID();
|
|
await db.delete(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issueId));
|
|
await db.update(heartbeatRuns).set({ status: "cancelled", errorCode: "cancelled" }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
const lease = { id: randomUUID(), companyId: f.companyId, heartbeatRunId: f.sourceRunId, provider: "daytona", providerLeaseId: randomUUID() };
|
|
await db.insert(environmentLeases).values({ ...lease, status: pending ? "pending_cleanup" : "released", releasedAt: new Date(),
|
|
cleanupStatus: pending ? "failed" : "success", metadata: { remoteExecutionTermination: remoteTerminationReceipt(lease, { providerLeaseId: lease.providerLeaseId, state: "destroyed" }) } });
|
|
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running", contextSnapshot: { issueId: randomUUID() } });
|
|
await db.insert(agentWakeupRequests).values({ id: queueId, companyId: f.companyId, agentId: f.agentId,
|
|
source: "on_demand", reason: "issue_commented", status: "deferred_issue_execution",
|
|
requestedByActorType: "user", requestedByActorId: "board",
|
|
payload: { issueId: f.issueId, _paperclipWakeContext: { wakeCommentIds: [f.commentId] },
|
|
queuedCommentInterrupt: { actorId: "board", requestedAt: new Date().toISOString() } },
|
|
});
|
|
await heartbeatService(db).resumeQueuedCommentInterrupt(f.companyId, queueId);
|
|
const [receipt] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, queueId));
|
|
if (pending) expect(receipt).toMatchObject({ status: "deferred_issue_execution", runId: null, payload: { executionWait: { reason: "remote_cleanup" } } });
|
|
else {
|
|
expect(receipt.status).toBe("coalesced");
|
|
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, receipt.runId!)))[0].contextSnapshot)
|
|
.toMatchObject({ previousRunId: f.sourceRunId });
|
|
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, receipt.runId!));
|
|
const envelope = await buildExecutionContinuation({ db, companyId: f.companyId, issueId: f.issueId, agentId: f.agentId,
|
|
runId: run.id, context: run.contextSnapshot!, summary: null, exposeLowTrustRaw: false });
|
|
expect(envelope.interruptedRunId).toBe(f.sourceRunId);
|
|
expect(envelope.originCommentIds).toContain(f.commentId);
|
|
await db.insert(issueRecoveryActions).values({ companyId: f.companyId, sourceIssueId: f.issueId,
|
|
kind: "active_run_watchdog", cause: "native_continuation_requires_reconciliation", fingerprint: randomUUID(),
|
|
status: "resolved", outcome: "recovered", evidence: { runId: f.sourceRunId }, nextAction: "Recovery is complete." });
|
|
const afterRecovery = await buildExecutionContinuation({ db, companyId: f.companyId, issueId: f.issueId, agentId: f.agentId,
|
|
runId: run.id, context: run.contextSnapshot!, summary: null, exposeLowTrustRaw: false });
|
|
expect(afterRecovery.originCommentIds).toContain(f.commentId);
|
|
await expect(buildExecutionContinuation({ db, companyId: f.companyId, issueId: f.issueId, agentId: f.agentId,
|
|
runId: randomUUID(), context: run.contextSnapshot!, summary: null, exposeLowTrustRaw: false }))
|
|
.rejects.toThrow("continuation_user_authorization_missing");
|
|
await db.update(agentWakeupRequests).set({ payload: { ...receipt.payload,
|
|
queuedCommentInterrupt: { ...(receipt.payload!.queuedCommentInterrupt as Record<string, unknown>), actorId: "someone-else" },
|
|
} }).where(eq(agentWakeupRequests.id, queueId));
|
|
await expect(buildExecutionContinuation({ db, companyId: f.companyId, issueId: f.issueId, agentId: f.agentId,
|
|
runId: run.id, context: run.contextSnapshot!, summary: null, exposeLowTrustRaw: false }))
|
|
.rejects.toThrow("continuation_user_authorization_missing");
|
|
}
|
|
await db.delete(environmentLeases).where(eq(environmentLeases.id, lease.id));
|
|
});
|
|
it.each(["ask_user_questions", "request_confirmation"] as const)("sends a saved native %s response after a fresh click", async kind => {
|
|
const f = await seed(), queueId = randomUUID();
|
|
const [interaction] = await db.insert(issueThreadInteractions).values({ companyId: f.companyId, issueId: f.issueId,
|
|
createdByAgentId: f.agentId, sourceRunId: f.sourceRunId, kind, status: kind === "ask_user_questions" ? "answered" : "accepted",
|
|
resolvedByUserId: "board", resolvedAt: new Date("2026-09-11T09:00:00Z"),
|
|
payload: kind === "ask_user_questions" ? { version: 1, questions: [{ id: "q1", prompt: "Continue?", selectionMode: "single", options: [{ id: "yes", label: "Yes" }] }] }
|
|
: { version: 1, prompt: "Continue?" },
|
|
result: kind === "ask_user_questions" ? { version: 1, answers: [{ questionId: "q1", optionIds: ["yes"] }] }
|
|
: { version: 1, outcome: "accepted" },
|
|
}).returning();
|
|
await db.insert(agentWakeupRequests).values({ id: queueId, companyId: f.companyId, agentId: f.agentId,
|
|
source: "on_demand", reason: "issue_commented", status: "deferred_issue_execution",
|
|
payload: { issueId: f.issueId, mutation: "interaction", interactionId: interaction.id, interactionStatus: interaction.status,
|
|
queuedCommentInterrupt: { actorId: "board", requestedAt: "2026-09-11T11:30:00Z" } },
|
|
});
|
|
const result = await db.transaction(async tx => admitExplicitNativeContinuation({ ...f, db: tx as unknown as typeof db,
|
|
commentId: null, queuedCommentInterruptId: queueId }));
|
|
expect(result).toMatchObject({ previousRunId: f.sourceRunId });
|
|
const context = { issueId: f.issueId, interactionId: interaction.id, mutation: "interaction", interactionKind: kind,
|
|
interactionStatus: interaction.status, explicitUserContinuation: result };
|
|
await db.insert(heartbeatRuns).values({ id: f.successorRunId, companyId: f.companyId, agentId: f.agentId,
|
|
runtimeMode: "native", nativeIssueId: f.issueId, status: "queued", contextSnapshot: context });
|
|
await db.update(agentWakeupRequests).set({ status: "coalesced", runId: f.successorRunId }).where(eq(agentWakeupRequests.id, queueId));
|
|
const envelope = await buildExecutionContinuation({ db, companyId: f.companyId, issueId: f.issueId, agentId: f.agentId,
|
|
runId: f.successorRunId, context, summary: null, exposeLowTrustRaw: false });
|
|
expect(envelope.interruptedRunId).toBe(f.sourceRunId);
|
|
expect((await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.id, interaction.id)))[0])
|
|
.toMatchObject({ result: interaction.result, resolvedByUserId: "board" });
|
|
});
|
|
it.each(["matching", "wrong_queue", "wrong_actor", "wrong_target", "missing_receipt", "ordinary_response", "comment"])("accepts only the exact live native queue Interrupt receipt: %s", async kind => {
|
|
const f = await seed(), queueId = randomUUID();
|
|
const [interaction] = kind === "comment" ? [] : await db.insert(issueThreadInteractions).values({ companyId: f.companyId, issueId: f.issueId,
|
|
sourceRunId: f.sourceRunId, createdByAgentId: f.agentId, kind: "request_confirmation", status: "accepted",
|
|
resolvedByUserId: "board", resolvedAt: new Date("2026-09-11T09:00:00Z"),
|
|
payload: { version: 1, prompt: "Approve this plan?" }, result: { version: 1, outcome: "accepted" },
|
|
}).returning();
|
|
await db.update(heartbeatRuns).set({ status: "cancelled", resultJson: kind === "missing_receipt" ? null : {
|
|
operatorInterrupted: true, interruptionSource: "issue_comment_interrupt", interruptedIssueId: f.issueId,
|
|
interruptedByActorType: "user", interruptedByActorId: kind === "wrong_actor" ? "someone-else" : "board",
|
|
queuedCommentInterruptQueueId: kind === "wrong_queue" ? randomUUID() : queueId,
|
|
} }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
await db.insert(agentWakeupRequests).values({ id: queueId, companyId: f.companyId, agentId: f.agentId,
|
|
source: "on_demand", reason: "issue_commented", status: "deferred_issue_execution",
|
|
payload: { issueId: f.issueId,
|
|
...(kind === "comment" ? { commentId: f.commentId } : {
|
|
mutation: "interaction", interactionId: interaction!.id, interactionStatus: "accepted" }),
|
|
_paperclipWakeContext: kind === "comment" ? { wakeCommentIds: [f.commentId] }
|
|
: { forceFreshSession: kind !== "ordinary_response" },
|
|
queuedCommentInterrupt: { actorId: "board", requestedAt: "2026-09-11T09:30:00Z",
|
|
targetRunId: kind === "wrong_target" ? randomUUID() : f.sourceRunId },
|
|
},
|
|
});
|
|
const result = await db.transaction(tx => admitExplicitNativeContinuation({ ...f, db: tx as unknown as typeof db,
|
|
commentId: kind === "comment" ? f.commentId : null, queuedCommentInterruptId: queueId, dryRun: true }));
|
|
if (["matching", "ordinary_response", "comment"].includes(kind)) expect(result).toMatchObject({ previousRunId: f.sourceRunId });
|
|
else expect(result).toBeNull();
|
|
});
|
|
|
|
it.each(["valid", "workspace_hold", "wrong_author"])("admits only a verified chat context reset while keeping workspace holds: %s", async scenario => {
|
|
const workspaceHold = scenario === "workspace_hold";
|
|
const f = await seed(), settings = instanceSettingsService(db), previous = await settings.getExperimental();
|
|
await settings.updateExperimental({ enableAgentChat: true });
|
|
try {
|
|
await db.update(issues).set({ conversationAgentId: f.agentId, conversationUserId: "board", conversationState: "active", status: "in_progress" }).where(eq(issues.id, f.issueId));
|
|
await db.update(issueComments).set({ body: "/new" }).where(eq(issueComments.id, f.commentId));
|
|
await db.update(heartbeatRuns).set({ processPid: null }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" });
|
|
if (workspaceHold) await db.update(issueRecoveryActions).set({
|
|
evidence: { runId: f.sourceRunId, workspaceRestoreFailure: "restore_unsafe_archive", automaticRecovery: { replay: "blocked" } },
|
|
}).where(eq(issueRecoveryActions.sourceIssueId, f.issueId));
|
|
const blocker = await getExecutionBlocker(db, f.companyId, f.issueId, { conversationResetCommentId: f.commentId });
|
|
if (workspaceHold) expect(blocker).not.toBeNull(); else expect(blocker).toBeNull();
|
|
const wake = await heartbeatService(db).wakeup(f.agentId, { source: "on_demand", triggerDetail: "manual", reason: "issue_commented",
|
|
requestedByActorType: "user", requestedByActorId: scenario === "wrong_author" ? "another-user" : "board", payload: { issueId: f.issueId, commentId: f.commentId },
|
|
contextSnapshot: { source: "issue.comment", issueId: f.issueId, wakeCommentId: f.commentId },
|
|
});
|
|
if (workspaceHold || scenario === "wrong_author") expect(wake).toBeNull(); else expect(wake).not.toBeNull();
|
|
expect((await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issueId)))[0].evidence)
|
|
.not.toHaveProperty("explicitUserContinuation");
|
|
} finally { await settings.updateExperimental(previous); }
|
|
});
|
|
|
|
it("a durable queue interrupt authorizes older legacy messages but still requires the provider to stop", async () => {
|
|
const f = await seed();
|
|
await db.update(agents).set({ adapterType: "claude_local" }).where(eq(agents.id, f.agentId));
|
|
await db.delete(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.sourceRunId));
|
|
await db.update(heartbeatRuns).set({ runtimeMode: "legacy", nativeIssueId: null,
|
|
processPid: process.pid,
|
|
}).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
await db.update(issueRecoveryActions).set({ cause: "legacy_execution_requires_reconciliation" })
|
|
.where(eq(issueRecoveryActions.sourceIssueId, f.issueId));
|
|
await db.update(issueComments).set({ authorUserId: "original-author", createdAt: new Date("2026-09-11T09:00:00Z") })
|
|
.where(eq(issueComments.id, f.commentId));
|
|
const queueId = randomUUID();
|
|
await db.insert(agentWakeupRequests).values({ id: queueId, companyId: f.companyId, agentId: f.agentId,
|
|
source: "on_demand", reason: "issue_commented", status: "deferred_issue_execution",
|
|
requestedByActorType: "user", requestedByActorId: "original-author",
|
|
payload: { issueId: f.issueId, _paperclipWakeContext: { wakeCommentIds: [f.commentId] },
|
|
queuedCommentInterrupt: { actorId: "board", requestedAt: new Date().toISOString() } },
|
|
});
|
|
const attempt = (queue = queueId) => db.transaction(async tx => {
|
|
await tx.select().from(issues).where(eq(issues.id, f.issueId)).for("update");
|
|
return admitExplicitNativeContinuation({ ...f, db: tx as unknown as typeof db,
|
|
queuedCommentInterruptId: queue, dryRun: true });
|
|
});
|
|
expect(await attempt()).toBeNull();
|
|
await db.update(heartbeatRuns).set({ processPid: 999999999 }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
expect(await attempt(randomUUID())).toBeNull();
|
|
expect(await attempt()).toMatchObject({ previousRunId: f.sourceRunId, commentId: f.commentId });
|
|
await db.update(agentWakeupRequests).set({ status: "cancelled" }).where(eq(agentWakeupRequests.id, queueId));
|
|
expect(await attempt()).toBeNull();
|
|
});
|
|
it("dispatches another user's queued legacy message using the consumed board interrupt receipt", async () => {
|
|
const f = await seed();
|
|
await db.update(agents).set({ adapterType: "claude_local" }).where(eq(agents.id, f.agentId));
|
|
await db.delete(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.sourceRunId));
|
|
await db.update(heartbeatRuns).set({ runtimeMode: "legacy", nativeIssueId: null })
|
|
.where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
await db.update(issueRecoveryActions).set({ cause: "legacy_execution_requires_reconciliation" })
|
|
.where(eq(issueRecoveryActions.sourceIssueId, f.issueId));
|
|
await db.update(issueComments).set({ authorUserId: "original-author", createdAt: new Date("2026-09-11T09:00:00Z") })
|
|
.where(eq(issueComments.id, f.commentId));
|
|
// Hold adapter startup so the test can exercise the real dispatch envelope
|
|
// deterministically, without invoking a provider.
|
|
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" });
|
|
const queueId = randomUUID();
|
|
await db.insert(agentWakeupRequests).values({ id: queueId, companyId: f.companyId, agentId: f.agentId,
|
|
source: "automation", reason: "issue_commented", status: "deferred_issue_execution",
|
|
requestedByActorType: "system", payload: { issueId: f.issueId, commentId: f.commentId,
|
|
_paperclipWakeContext: { wakeCommentIds: [f.commentId] },
|
|
queuedCommentInterrupt: { actorId: "board", requestedAt: new Date().toISOString() } },
|
|
});
|
|
await heartbeatService(db).resumeQueuedCommentInterrupt(f.companyId, queueId);
|
|
const [receipt] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, queueId));
|
|
expect(receipt.status).toBe("coalesced");
|
|
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, receipt.runId!));
|
|
const dispatch = (runId = run.id) => buildExecutionContinuation({ db, companyId: f.companyId,
|
|
issueId: f.issueId, agentId: f.agentId, runId, context: run.contextSnapshot!,
|
|
summary: null, exposeLowTrustRaw: false });
|
|
const envelope = await dispatch();
|
|
expect(envelope.interruptedRunId).toBe(f.sourceRunId);
|
|
expect(envelope.originCommentIds).toContain(f.commentId);
|
|
expect(envelope.messages).toEqual(expect.arrayContaining([expect.objectContaining({ id: f.commentId, body: "What happened?" })]));
|
|
await expect(dispatch(randomUUID())).rejects.toThrow("continuation_user_authorization_missing");
|
|
for (const patch of [
|
|
{ status: "cancelled" }, { runId: f.sourceRunId },
|
|
{ payload: { ...receipt.payload, issueId: randomUUID() } },
|
|
{ payload: { ...receipt.payload, queuedCommentInterrupt: { actorId: "someone-else" } } },
|
|
{ payload: { ...receipt.payload, _paperclipWakeContext: { wakeCommentIds: [] }, commentId: undefined } },
|
|
]) {
|
|
await db.update(agentWakeupRequests).set(patch).where(eq(agentWakeupRequests.id, queueId));
|
|
await expect(dispatch()).rejects.toThrow("continuation_user_authorization_missing");
|
|
await db.update(agentWakeupRequests).set({ status: receipt.status, runId: receipt.runId, payload: receipt.payload })
|
|
.where(eq(agentWakeupRequests.id, queueId));
|
|
}
|
|
const [action] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issueId));
|
|
await db.update(issueRecoveryActions).set({ evidence: { ...action.evidence,
|
|
explicitUserContinuation: { ...(action.evidence.explicitUserContinuation as Record<string, unknown>),
|
|
queuedCommentInterruptId: "malformed-historical-receipt" },
|
|
} }).where(eq(issueRecoveryActions.id, action.id));
|
|
await expect(dispatch()).rejects.toThrow("continuation_user_authorization_missing");
|
|
});
|
|
it.each(["valid", "stale_retry_context", "wrong_actor", "wrong_run", "wrong_company", "discarded"])("verifies another author's queued Interrupt at execution setup: %s", async kind => {
|
|
const f = await seed();
|
|
await db.update(agents).set({ adapterType: "claude_local" }).where(eq(agents.id, f.agentId));
|
|
await db.delete(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.sourceRunId));
|
|
await db.update(heartbeatRuns).set({ runtimeMode: "legacy", nativeIssueId: null })
|
|
.where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
await db.update(issueRecoveryActions).set({ cause: "legacy_execution_requires_reconciliation" })
|
|
.where(eq(issueRecoveryActions.sourceIssueId, f.issueId));
|
|
await db.update(issueComments).set({ authorUserId: "original-author", createdAt: new Date("2026-09-11T09:00:00Z") })
|
|
.where(eq(issueComments.id, f.commentId));
|
|
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId,
|
|
status: "running", contextSnapshot: { issueId: randomUUID() } });
|
|
const queueId = randomUUID();
|
|
const payload = { issueId: f.issueId, _paperclipWakeContext: { wakeCommentIds: [f.commentId] },
|
|
queuedCommentInterrupt: { actorId: "board", requestedAt: new Date().toISOString() } };
|
|
await db.insert(agentWakeupRequests).values({ id: queueId, companyId: f.companyId, agentId: f.agentId,
|
|
source: "automation", reason: "issue_commented", status: "deferred_issue_execution",
|
|
requestedByActorType: "system", payload });
|
|
await heartbeatService(db).resumeQueuedCommentInterrupt(f.companyId, queueId);
|
|
const [wake] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, queueId));
|
|
expect(wake.status).toBe("coalesced");
|
|
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, wake.runId!));
|
|
if (kind === "wrong_actor") await db.update(agentWakeupRequests).set({
|
|
payload: { ...payload, queuedCommentInterrupt: { ...payload.queuedCommentInterrupt, actorId: "forged" } },
|
|
}).where(eq(agentWakeupRequests.id, queueId));
|
|
if (kind === "wrong_run") await db.update(agentWakeupRequests).set({ runId: f.sourceRunId })
|
|
.where(eq(agentWakeupRequests.id, queueId));
|
|
if (kind === "wrong_company") await db.update(agentWakeupRequests).set({ companyId: (await seed()).companyId })
|
|
.where(eq(agentWakeupRequests.id, queueId));
|
|
if (kind === "discarded") await db.update(issueComments).set({ deletedAt: new Date() })
|
|
.where(eq(issueComments.id, f.commentId));
|
|
const result = buildExecutionContinuation({ db, companyId: f.companyId, issueId: f.issueId,
|
|
agentId: f.agentId, runId: run.id, context: {
|
|
...run.contextSnapshot, ...(kind === "stale_retry_context" ? { retryOfRunId: randomUUID() } : {}),
|
|
}, summary: null, exposeLowTrustRaw: false });
|
|
if (kind === "valid" || kind === "stale_retry_context") await expect(result).resolves.toMatchObject({ interruptedRunId: f.sourceRunId });
|
|
else await expect(result).rejects.toThrow("continuation_user_authorization_missing");
|
|
});
|
|
|
|
it.each(["valid", "provider_cancel", "pending_tool", "unknown_cancel", "wrong_actor", "consumed", "discarded", "operator_stop", "already_delivered", "earlier_delivered", "unstarted_cancelled", "unstarted_cancelled_metadata", "foreign_queue"])("validates automatic saved-message delivery: %s", async kind => {
|
|
const f = await seed();
|
|
await db.update(agents).set({ adapterType: "claude_local" }).where(eq(agents.id, f.agentId));
|
|
await db.delete(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.sourceRunId));
|
|
await db.update(heartbeatRuns).set({ runtimeMode: "legacy", nativeIssueId: null,
|
|
status: ["operator_stop", "provider_cancel", "pending_tool", "unknown_cancel"].includes(kind) ? "cancelled" : "failed",
|
|
startedAt: new Date("2026-09-11T08:00:00Z"),
|
|
...(["provider_cancel", "pending_tool"].includes(kind) ? { resultJson: {
|
|
acpToolInventoryComplete: true, acpPendingToolCount: kind === "pending_tool" ? 1 : 0,
|
|
cancellation: { source: "provider", expected: false, initiator: { type: "provider" },
|
|
reason: "Provider cancelled execution", recordedAt: new Date().toISOString() },
|
|
} } : {}),
|
|
contextSnapshot: { issueId: f.issueId, ...(kind === "already_delivered" ? { wakeCommentIds: [f.commentId] } : {}) },
|
|
}).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
await db.update(issueRecoveryActions).set({ cause: "legacy_execution_requires_reconciliation" })
|
|
.where(eq(issueRecoveryActions.sourceIssueId, f.issueId));
|
|
await db.update(issueComments).set({ createdAt: new Date("2026-09-11T09:00:00Z"),
|
|
...(kind === "discarded" ? { deletedAt: new Date() } : {}),
|
|
}).where(eq(issueComments.id, f.commentId));
|
|
const queueId = randomUUID();
|
|
await db.insert(agentWakeupRequests).values({ id: queueId, companyId: f.companyId, agentId: f.agentId,
|
|
source: "automation", reason: "issue_commented", status: kind === "consumed" ? "coalesced" : "deferred_issue_execution",
|
|
requestedByActorType: "system", payload: { issueId: kind === "foreign_queue" ? randomUUID() : f.issueId,
|
|
_paperclipWakeContext: { wakeCommentIds: [f.commentId] } },
|
|
});
|
|
if (kind.startsWith("unstarted_cancelled")) {
|
|
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId,
|
|
status: "cancelled", runtimeMode: "legacy", errorCode: "agent_paused", finishedAt: new Date(),
|
|
...(kind === "unstarted_cancelled_metadata" ? { nativeIssueId: f.issueId, processPid: 999999999, processGroupId: 999999999 } : {}),
|
|
contextSnapshot: { issueId: f.issueId, wakeCommentIds: [f.commentId] },
|
|
});
|
|
}
|
|
if (kind === "earlier_delivered") {
|
|
const earlierId = randomUUID();
|
|
await db.insert(issueComments).values({ id: earlierId, companyId: f.companyId, issueId: f.issueId,
|
|
authorType: "user", authorUserId: f.actorId, body: "Already handled" });
|
|
await db.update(heartbeatRuns).set({ contextSnapshot: { issueId: f.issueId, wakeCommentIds: [earlierId] } })
|
|
.where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
await db.update(agentWakeupRequests).set({ payload: { issueId: f.issueId,
|
|
_paperclipWakeContext: { wakeCommentIds: [earlierId, f.commentId] } } })
|
|
.where(eq(agentWakeupRequests.id, queueId));
|
|
}
|
|
const result = await db.transaction(async tx => {
|
|
await tx.select().from(issues).where(eq(issues.id, f.issueId)).for("update");
|
|
return admitExplicitNativeContinuation({ ...f, actorId: kind === "wrong_actor" ? "someone-else" : f.actorId,
|
|
db: tx as unknown as typeof db, queuedCommentRequestId: queueId, dryRun: true });
|
|
});
|
|
if (kind === "valid" || kind === "provider_cancel" || kind.startsWith("unstarted_cancelled")) expect(result).toMatchObject({ previousRunId: f.sourceRunId, commentId: f.commentId });
|
|
else expect(result).toBeNull();
|
|
});
|
|
|
|
const admit = (f: Fixture, dryRun = false) => db.transaction(async tx => {
|
|
await tx.select().from(issues).where(eq(issues.id, f.issueId)).for("update");
|
|
const result = await admitExplicitNativeContinuation({ ...f, dryRun, db: tx as unknown as typeof db });
|
|
if (result && !dryRun) await tx.insert(heartbeatRuns).values({ id: f.successorRunId, companyId: f.companyId,
|
|
agentId: f.agentId, status: "queued", contextSnapshot: { issueId: f.issueId, previousRunId: result.previousRunId, forceFreshSession: true } });
|
|
return result;
|
|
});
|
|
|
|
it.each(["other_user", "same_user", "unaddressed", "tool_access"])("scopes connection waits to the user starting the fresh turn (%s)", async audience => {
|
|
const f = await seed();
|
|
await db.update(heartbeatRuns).set({ status: "cancelled", errorCode: "slack_session_stopped" })
|
|
.where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
const [interaction] = await db.insert(issueThreadInteractions).values({
|
|
companyId: f.companyId, issueId: f.issueId, sourceRunId: f.sourceRunId,
|
|
createdByAgentId: f.agentId, kind: "connection_intent", status: "pending",
|
|
addresseeUserId: ["other_user", "tool_access"].includes(audience) ? "other-user" : audience === "same_user" ? "board" : null,
|
|
effectiveResolverPolicy: "human_only",
|
|
payload: { version: 1, ...(audience === "tool_access" ? {} : { purpose: "ai" as const }), serviceSlug: "openai", serviceName: "OpenAI",
|
|
requestingAgentId: f.agentId, requestingAgentName: "Native", phase: "requested" },
|
|
}).returning();
|
|
expect(await admit(f)).toEqual(audience === "other_user"
|
|
? { previousRunId: f.sourceRunId, commentId: f.commentId } : null);
|
|
// Starting another user's turn grants no connection authority and retains the card.
|
|
expect((await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.id, interaction.id)))[0])
|
|
.toMatchObject({ status: "pending", result: null, addresseeUserId: interaction.addresseeUserId });
|
|
const blocker = await getExecutionBlocker(db, f.companyId, f.issueId);
|
|
if (audience === "other_user") expect(blocker).toBeNull();
|
|
else expect(blocker).not.toBeNull();
|
|
});
|
|
|
|
it.each(["fresh", "before_stop", "delivered", "unknown_stop_time", "wrong_actor", "approval"])(
|
|
"resumes an already-saved follow-up after native Slack Stop (%s)", async gate => {
|
|
const f = await seed(), queueId = randomUUID();
|
|
await db.update(heartbeatRuns).set({ status: "cancelled", errorCode: "slack_session_stopped",
|
|
startedAt: new Date("2026-09-11T09:30:00Z"), finishedAt: gate === "unknown_stop_time" ? null : new Date("2026-09-11T10:00:00Z"),
|
|
contextSnapshot: { issueId: f.issueId, ...(gate === "delivered" ? { wakeCommentId: f.commentId } : {}) },
|
|
// Pre-fix Slack Stops recorded a system cancellation, with no operator identity.
|
|
resultJson: { cancellation: { source: "control_plane", expected: true, initiator: { type: "system" },
|
|
reason: "Stopped from the bound Slack agent session", recordedAt: "2026-09-11T10:00:00Z" } },
|
|
}).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
if (gate === "before_stop") await db.update(issueComments).set({ createdAt: new Date("2026-09-11T09:00:00Z") })
|
|
.where(eq(issueComments.id, f.commentId));
|
|
await db.insert(issueThreadInteractions).values({
|
|
companyId: f.companyId, issueId: f.issueId, sourceRunId: f.sourceRunId,
|
|
createdByAgentId: f.agentId, kind: "connection_intent", status: "pending", addresseeUserId: "other-user",
|
|
effectiveResolverPolicy: "human_only", payload: { version: 1, purpose: "ai", serviceSlug: "openai",
|
|
serviceName: "OpenAI", requestingAgentId: f.agentId, requestingAgentName: "Native", phase: "requested" },
|
|
});
|
|
if (gate === "approval") {
|
|
const [approval] = await db.insert(approvals).values({ companyId: f.companyId, type: "hire_agent", status: "pending", payload: {} }).returning();
|
|
await db.insert(issueApprovals).values({ companyId: f.companyId, issueId: f.issueId, approvalId: approval.id });
|
|
}
|
|
await db.insert(agentWakeupRequests).values({ id: queueId, companyId: f.companyId, agentId: f.agentId,
|
|
source: "automation", reason: "issue_execution_deferred", status: "deferred_issue_execution",
|
|
requestedByActorType: "user", requestedByActorId: gate === "wrong_actor" ? "other-user" : "board",
|
|
payload: { issueId: f.issueId, commentId: f.commentId, _paperclipWakeContext: {
|
|
issueId: f.issueId, wakeReason: "issue_commented", wakeCommentId: f.commentId, wakeCommentIds: [f.commentId] } },
|
|
});
|
|
const result = await db.transaction(async tx => {
|
|
await tx.select().from(issues).where(eq(issues.id, f.issueId)).for("update");
|
|
return admitExplicitNativeContinuation({ ...f, actorId: gate === "wrong_actor" ? "other-user" : f.actorId,
|
|
db: tx as unknown as typeof db, queuedCommentRequestId: queueId, dryRun: true });
|
|
});
|
|
expect(result).toEqual(gate === "fresh" ? { previousRunId: f.sourceRunId, commentId: f.commentId } : null);
|
|
},
|
|
);
|
|
|
|
async function seedTimedOutExplicitTurn() {
|
|
const f = await seed();
|
|
const explicitUserContinuation = await admit(f);
|
|
expect(explicitUserContinuation).not.toBeNull();
|
|
const [receipt] = await db.select().from(issueRecoveryActions)
|
|
.where(eq(issueRecoveryActions.sourceIssueId, f.issueId));
|
|
await db.update(agents).set({ adapterType: "claude_local" }).where(eq(agents.id, f.agentId));
|
|
const now = new Date();
|
|
const [parent] = await db.update(heartbeatRuns).set({ runtimeMode: "legacy", status: "timed_out",
|
|
errorCode: "timeout", startedAt: now, finishedAt: now,
|
|
resultJson: { conversationContinuation: CONVERSATION_CONTINUATION_POLICY },
|
|
contextSnapshot: { issueId: f.issueId, wakeReason: "issue_commented", wakeCommentId: f.commentId,
|
|
previousRunId: f.sourceRunId, explicitUserContinuation },
|
|
}).where(eq(heartbeatRuns.id, f.successorRunId)).returning();
|
|
await db.update(issues).set({ status: "in_progress", executionRunId: parent.id })
|
|
.where(eq(issues.id, f.issueId));
|
|
return { ...f, parent, receipt, now };
|
|
}
|
|
const dispatchExplicitRetry = (f: Fixture, run: typeof heartbeatRuns.$inferSelect) => buildExecutionContinuation({
|
|
db, companyId: f.companyId, issueId: f.issueId, agentId: f.agentId, runId: run.id,
|
|
context: run.contextSnapshot!, summary: null, exposeLowTrustRaw: false,
|
|
});
|
|
|
|
it("re-admits an explicit user turn's timeout retry with its own durable authorization", async () => {
|
|
const f = await seedTimedOutExplicitTurn();
|
|
await expect(dispatchExplicitRetry(f, f.parent)).resolves.toMatchObject({ interruptedRunId: f.sourceRunId });
|
|
const heartbeat = heartbeatService(db);
|
|
const [scheduled, competing] = await Promise.all([
|
|
heartbeat.scheduleBoundedRetry(f.parent.id, { now: f.now, delayMs: 1000 }),
|
|
heartbeat.scheduleBoundedRetry(f.parent.id, { now: f.now, delayMs: 1000 }),
|
|
]);
|
|
expect(scheduled.outcome).toBe("scheduled");
|
|
if (scheduled.outcome !== "scheduled") return;
|
|
expect(competing).toMatchObject({ outcome: "scheduled", run: { id: scheduled.run.id } });
|
|
expect(scheduled.run.contextSnapshot?.explicitUserContinuation).toEqual({
|
|
previousRunId: f.parent.id, commentId: f.commentId,
|
|
});
|
|
// Dispatch validates the new run, not a borrowed parent receipt.
|
|
await expect(dispatchExplicitRetry(f, scheduled.run)).resolves.toMatchObject({ interruptedRunId: f.parent.id });
|
|
const receipts = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issueId));
|
|
expect(receipts).toHaveLength(2);
|
|
expect(receipts.find(row => row.id === f.receipt.id)?.evidence).toEqual(f.receipt.evidence);
|
|
expect(receipts.find(row => row.id !== f.receipt.id)?.evidence.explicitUserContinuation).toMatchObject({
|
|
runId: scheduled.run.id, previousRunId: f.parent.id, actorId: "board", commentId: f.commentId,
|
|
automaticRetry: { sourceRunId: f.parent.id, sourceAuthorizationId: f.receipt.id },
|
|
});
|
|
expect(await heartbeat.scheduleBoundedRetry(f.parent.id, { now: f.now })).toMatchObject({
|
|
outcome: "scheduled", run: { id: scheduled.run.id },
|
|
});
|
|
expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issueId))).toHaveLength(2);
|
|
// Removing the new receipt reproduces the original dispatch failure.
|
|
await db.delete(issueRecoveryActions).where(eq(issueRecoveryActions.id, receipts.find(row => row.id !== f.receipt.id)!.id));
|
|
await expect(dispatchExplicitRetry(f, scheduled.run)).rejects.toThrow("continuation_user_authorization_missing");
|
|
});
|
|
|
|
it("waits for the parent adapter controller even when no environment lease remains", async () => {
|
|
const f = await seedTimedOutExplicitTurn();
|
|
const heartbeat = heartbeatService(db);
|
|
const control = createAdapterExecutionControl();
|
|
adapterExecutionControls.set(f.parent.id, control);
|
|
try {
|
|
expect(await heartbeat.scheduleBoundedRetry(f.parent.id)).toMatchObject({ outcome: "not_scheduled" });
|
|
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, f.parent.id))).toHaveLength(0);
|
|
expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issueId))).toHaveLength(1);
|
|
} finally {
|
|
adapterExecutionControls.delete(f.parent.id);
|
|
control.finish();
|
|
}
|
|
expect(await heartbeat.scheduleBoundedRetry(f.parent.id)).toMatchObject({ outcome: "scheduled" });
|
|
});
|
|
|
|
it("binds each bounded explicit retry to its immediate parent without replacing prior receipts", async () => {
|
|
const f = await seedTimedOutExplicitTurn();
|
|
const heartbeat = heartbeatService(db);
|
|
const first = await heartbeat.scheduleBoundedRetry(f.parent.id, { now: f.now });
|
|
expect(first.outcome).toBe("scheduled");
|
|
if (first.outcome !== "scheduled") return;
|
|
const nextNow = new Date(f.now.getTime() + 2000);
|
|
await db.update(heartbeatRuns).set({ status: "timed_out", errorCode: "timeout", startedAt: f.now,
|
|
finishedAt: nextNow }).where(eq(heartbeatRuns.id, first.run.id));
|
|
const second = await heartbeat.scheduleBoundedRetry(first.run.id, { now: nextNow });
|
|
expect(second.outcome).toBe("scheduled");
|
|
if (second.outcome !== "scheduled") return;
|
|
expect(second.run).toMatchObject({ retryOfRunId: first.run.id, scheduledRetryAttempt: 2 });
|
|
await expect(dispatchExplicitRetry(f, second.run)).resolves.toMatchObject({ interruptedRunId: first.run.id });
|
|
const receipts = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issueId));
|
|
expect(receipts).toHaveLength(3);
|
|
expect(receipts.find(row => row.id === f.receipt.id)?.evidence).toEqual(f.receipt.evidence);
|
|
await db.update(heartbeatRuns).set({ status: "timed_out", errorCode: "timeout", startedAt: nextNow,
|
|
finishedAt: nextNow }).where(eq(heartbeatRuns.id, second.run.id));
|
|
expect(await heartbeat.scheduleBoundedRetry(second.run.id, { now: nextNow })).toMatchObject({ outcome: "retry_exhausted" });
|
|
expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issueId))).toHaveLength(3);
|
|
});
|
|
|
|
it("keeps sub-millisecond source ordering when authorizing a timeout retry", async () => {
|
|
const f = await seedTimedOutExplicitTurn();
|
|
await db.execute(sql`update heartbeat_runs set created_at = ${f.now.toISOString()}::timestamptz +
|
|
case when id = ${f.sourceRunId} then interval '100 microseconds' else interval '500 microseconds' end
|
|
where company_id = ${f.companyId} and id in (${f.sourceRunId}, ${f.parent.id})`);
|
|
const result = await heartbeatService(db).scheduleBoundedRetry(f.parent.id, { now: f.now });
|
|
expect(result.outcome).toBe("scheduled");
|
|
});
|
|
|
|
it.each(["max_turns_continuation", "interaction_continuation_infra_retry", "workspace_busy"])(
|
|
"does not lend explicit user authority to the separate %s retry contract", async retryReason => {
|
|
const f = await seedTimedOutExplicitTurn();
|
|
const result = await heartbeatService(db).scheduleBoundedRetry(f.parent.id, { now: f.now, retryReason });
|
|
expect(result.outcome).not.toBe("scheduled");
|
|
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, f.parent.id))).toHaveLength(0);
|
|
},
|
|
);
|
|
|
|
it.each(["deleted", "edited", "same_millisecond_edit", "empty", "actor", "run_authored", "legacy_receipt", "receipt_run", "receipt_company", "source", "task", "reassigned", "done", "superseded", "native_superseding", "interrupted_intent"])(
|
|
"refuses to re-admit an automatic explicit retry after %s changes", async kind => {
|
|
const f = await seedTimedOutExplicitTurn();
|
|
if (["deleted", "edited", "same_millisecond_edit", "empty", "actor", "run_authored"].includes(kind)) await db.update(issueComments).set({
|
|
...(kind === "deleted" ? { deletedAt: new Date() } : {}),
|
|
...(kind === "edited" ? { body: "Different instructions", updatedAt: new Date(f.now.getTime() + 1000) } : {}),
|
|
...(kind === "same_millisecond_edit" ? { body: "Different instructions within the recorded millisecond" } : {}),
|
|
...(kind === "empty" ? { body: " " } : {}),
|
|
...(kind === "actor" ? { authorUserId: "another-user" } : {}),
|
|
...(kind === "run_authored" ? { createdByRunId: f.parent.id } : {}),
|
|
}).where(eq(issueComments.id, f.commentId));
|
|
if (["receipt_run", "interrupted_intent"].includes(kind)) await db.update(issueRecoveryActions).set({ evidence: {
|
|
...f.receipt.evidence, explicitUserContinuation: {
|
|
...(f.receipt.evidence.explicitUserContinuation as Record<string, unknown>),
|
|
...(kind === "receipt_run" ? { runId: randomUUID() } : { queuedCommentInterruptId: randomUUID() }),
|
|
},
|
|
} }).where(eq(issueRecoveryActions.id, f.receipt.id));
|
|
if (kind === "receipt_company") await db.update(issueRecoveryActions).set({ companyId: (await seed()).companyId })
|
|
.where(eq(issueRecoveryActions.id, f.receipt.id));
|
|
if (kind === "legacy_receipt") {
|
|
const { commentBodyHash: _bodyHash, commentUpdatedAt: _updatedAt, ...legacy } =
|
|
f.receipt.evidence.explicitUserContinuation as Record<string, unknown>;
|
|
await db.update(issueRecoveryActions).set({ evidence: { ...f.receipt.evidence,
|
|
explicitUserContinuation: legacy } }).where(eq(issueRecoveryActions.id, f.receipt.id));
|
|
}
|
|
if (kind === "source" || kind === "task") await db.update(heartbeatRuns).set({ contextSnapshot: {
|
|
...f.parent.contextSnapshot,
|
|
...(kind === "task" ? { issueId: randomUUID() } : { explicitUserContinuation: { previousRunId: randomUUID(), commentId: f.commentId } }),
|
|
} }).where(eq(heartbeatRuns.id, f.parent.id));
|
|
if (kind === "reassigned" || kind === "done") await db.update(issues).set(kind === "done"
|
|
? { status: "done" } : { assigneeAgentId: null }).where(eq(issues.id, f.issueId));
|
|
if (kind === "superseded") await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId,
|
|
status: "succeeded", contextSnapshot: { issueId: f.issueId }, createdAt: new Date(f.now.getTime() + 1000) });
|
|
if (kind === "native_superseding") await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId,
|
|
status: "running", runtimeMode: "native", nativeIssueId: f.issueId, contextSnapshot: {} });
|
|
const result = await heartbeatService(db).scheduleBoundedRetry(f.parent.id, { now: f.now });
|
|
expect(result.outcome).not.toBe("scheduled");
|
|
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, f.parent.id))).toHaveLength(0);
|
|
expect(await db.select().from(issueRecoveryActions).where(and(eq(issueRecoveryActions.sourceIssueId, f.issueId),
|
|
eq(issueRecoveryActions.cause, "explicit_user_continuation_retry")))).toHaveLength(0);
|
|
},
|
|
);
|
|
|
|
it("keeps the user's continuation while it waits out a storage outage", async () => {
|
|
const f = await seedTimedOutExplicitTurn();
|
|
await db.update(heartbeatRuns).set({ status: "failed", errorCode: "storage_unavailable" }).where(eq(heartbeatRuns.id, f.parent.id));
|
|
const scheduled = await heartbeatService(db).scheduleBoundedRetry(f.parent.id, { now: f.now });
|
|
expect(scheduled.outcome).toBe("scheduled");
|
|
const [successor] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, f.parent.id));
|
|
expect(successor).toMatchObject({ scheduledRetryReason: "storage_unavailable_wait",
|
|
contextSnapshot: { explicitUserContinuation: { previousRunId: f.parent.id } } });
|
|
await expect(dispatchExplicitRetry(f, successor!)).resolves.toBeTruthy();
|
|
});
|
|
|
|
it("binds the initial user-message admission before any automatic retry", async () => {
|
|
const f = await seedTimedOutExplicitTurn();
|
|
await expect(dispatchExplicitRetry(f, f.parent)).resolves.toMatchObject({ interruptedRunId: f.sourceRunId });
|
|
await db.update(issueComments).set({ body: "Changed after the original admission" })
|
|
.where(eq(issueComments.id, f.commentId));
|
|
await expect(dispatchExplicitRetry(f, f.parent)).rejects.toThrow("continuation_user_authorization_missing");
|
|
});
|
|
|
|
it.each(["deleted", "edited", "same_millisecond_edit", "actor", "receipt_run", "receipt_source", "retry_parent", "execution_owner"])(
|
|
"revalidates the successor's exact authorization at dispatch: %s", async kind => {
|
|
const f = await seedTimedOutExplicitTurn();
|
|
const scheduled = await heartbeatService(db).scheduleBoundedRetry(f.parent.id, { now: f.now });
|
|
expect(scheduled.outcome).toBe("scheduled");
|
|
if (scheduled.outcome !== "scheduled") return;
|
|
if (["deleted", "edited", "same_millisecond_edit", "actor"].includes(kind)) await db.update(issueComments).set({
|
|
...(kind === "deleted" ? { deletedAt: new Date() } : {}),
|
|
...(kind === "edited" ? { body: "Changed after scheduling", updatedAt: new Date(f.now.getTime() + 1000) } : {}),
|
|
...(kind === "same_millisecond_edit" ? { body: "Changed within the recorded millisecond" } : {}),
|
|
...(kind === "actor" ? { authorUserId: "another-user" } : {}),
|
|
}).where(eq(issueComments.id, f.commentId));
|
|
if (kind === "receipt_run" || kind === "receipt_source") {
|
|
const [receipt] = await db.select().from(issueRecoveryActions).where(and(
|
|
eq(issueRecoveryActions.sourceIssueId, f.issueId), eq(issueRecoveryActions.cause, "explicit_user_continuation_retry")));
|
|
await db.update(issueRecoveryActions).set({ evidence: { ...receipt.evidence, explicitUserContinuation: {
|
|
...(receipt.evidence.explicitUserContinuation as Record<string, unknown>),
|
|
...(kind === "receipt_run" ? { runId: f.parent.id } : { previousRunId: f.sourceRunId }),
|
|
} } }).where(eq(issueRecoveryActions.id, receipt.id));
|
|
}
|
|
if (kind === "retry_parent") await db.update(heartbeatRuns).set({ retryOfRunId: f.sourceRunId })
|
|
.where(eq(heartbeatRuns.id, scheduled.run.id));
|
|
if (kind === "execution_owner") await db.update(issues).set({ executionRunId: f.parent.id })
|
|
.where(eq(issues.id, f.issueId));
|
|
await expect(dispatchExplicitRetry(f, scheduled.run)).rejects.toThrow("continuation_user_authorization_missing");
|
|
},
|
|
);
|
|
|
|
it.each(["active", "removed", "unavailable", "paused", "disabled"])("shows usable recovery guidance for external chat cancellations: %s", async kind => {
|
|
const f = await seed();
|
|
await db.delete(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.sourceRunId));
|
|
await db.update(heartbeatRuns).set({ runtimeMode: "legacy", status: "cancelled", nativeIssueId: null,
|
|
startedAt: new Date("2026-09-11T08:00:00Z"), resultJson: {
|
|
acpToolInventoryComplete: true, acpPendingToolCount: 0,
|
|
cancellation: { source: "provider", expected: false, initiator: { type: "provider" },
|
|
reason: "Provider cancelled execution", recordedAt: new Date().toISOString() },
|
|
},
|
|
}).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
await db.update(issueRecoveryActions).set({ cause: "legacy_execution_requires_reconciliation" })
|
|
.where(eq(issueRecoveryActions.sourceIssueId, f.issueId));
|
|
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toMatchObject({ canContinue: true });
|
|
const applicationId = randomUUID(), connectionId = randomUUID(), endpointId = randomUUID();
|
|
await db.insert(toolApplications).values({ id: applicationId, companyId: f.companyId, name: "Slack", type: "chat" });
|
|
await db.insert(toolConnections).values({ id: connectionId, companyId: f.companyId, applicationId,
|
|
name: "Slack", uid: connectionId, transport: "chat_sdk", connectionPurpose: "channel", status: "active", enabled: kind !== "disabled" });
|
|
await db.insert(chatEndpoints).values({ id: endpointId, companyId: f.companyId, connectionId,
|
|
provider: "slack", publicId: endpointId, assignedAgentId: f.agentId,
|
|
status: kind === "removed" ? "archived" : kind === "paused" ? "paused" : "active" });
|
|
await db.insert(chatConversations).values({ companyId: f.companyId, endpointId, issueId: f.issueId,
|
|
externalConversationId: "channel", externalLabel: "#recovery",
|
|
state: kind === "removed" ? "endpoint_removed" : kind === "unavailable" ? "unavailable" : "active" });
|
|
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toMatchObject({
|
|
runId: f.sourceRunId, canContinue: false,
|
|
nextAction: kind === "active" ? "Send a new chat message to continue this conversation."
|
|
: kind === "removed" ? "This chat connection was removed. Inspect the stopped run and create a new task to continue the work."
|
|
: "This chat connection is unavailable. Restore access in Apps or create a new task to continue the work.",
|
|
});
|
|
});
|
|
|
|
it("delivers saved messages once after provider cancellation, preserving source history", async () => {
|
|
const f = await seed();
|
|
await db.update(agents).set({ adapterType: "claude_local" }).where(eq(agents.id, f.agentId));
|
|
// Hold agent capacity so the test observes durable admission without launching a provider.
|
|
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" });
|
|
await db.delete(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.sourceRunId));
|
|
const resultJson = { status: "cancelled", acpToolInventoryComplete: true, acpPendingToolCount: 0,
|
|
cancellation: { source: "provider", expected: false, initiator: { type: "provider" },
|
|
reason: "Provider cancelled execution", recordedAt: new Date().toISOString() },
|
|
completedAction: "preserve-this-record" };
|
|
await db.update(heartbeatRuns).set({ runtimeMode: "legacy", status: "cancelled", nativeIssueId: null,
|
|
startedAt: new Date("2026-09-11T08:00:00Z"), resultJson,
|
|
}).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
await db.update(issueRecoveryActions).set({ cause: "legacy_execution_requires_reconciliation" })
|
|
.where(eq(issueRecoveryActions.sourceIssueId, f.issueId));
|
|
const queueId = randomUUID();
|
|
await db.insert(agentWakeupRequests).values({ id: queueId, companyId: f.companyId, agentId: f.agentId,
|
|
source: "automation", reason: "issue_commented", status: "deferred_issue_execution",
|
|
requestedByActorType: "user", requestedByActorId: f.actorId, updatedAt: new Date(0),
|
|
payload: { issueId: f.issueId, commentId: f.commentId, executionWait: { reason: "execution_recovery" },
|
|
_paperclipWakeContext: { issueId: f.issueId, wakeReason: "issue_commented", wakeCommentIds: [f.commentId], wakeCommentId: f.commentId } },
|
|
});
|
|
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toMatchObject({ savedMessageCount: 1, canContinue: true });
|
|
await Promise.all([heartbeatService(db).resumeExecutionWaitComments(), heartbeatService(db).resumeExecutionWaitComments()]);
|
|
await heartbeatService(db).resumeExecutionWaitComments();
|
|
const runs = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId), eq(heartbeatRuns.status, "queued")));
|
|
expect(runs).toHaveLength(1);
|
|
expect(runs[0].contextSnapshot).toMatchObject({ forceFreshSession: true, previousRunId: f.sourceRunId, wakeCommentIds: [f.commentId] });
|
|
const [queue] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, queueId));
|
|
expect(queue).toMatchObject({ status: "coalesced", runId: runs[0].id });
|
|
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toBeNull();
|
|
const [source] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
expect(source.resultJson).toEqual(resultJson);
|
|
});
|
|
|
|
// Rows saved by earlier releases carry operator-facing text. The task
|
|
// notice shows plain wording instead; nothing asks an administrator to act.
|
|
it("shows plain wording for a cleanup refusal saved with the earlier operator text", async () => {
|
|
const f = await seed();
|
|
await db.update(heartbeatRuns).set({ errorCode: "native_session_cleanup_quarantined",
|
|
error: "native_session_cleanup_quarantined: prior session cleanup requires operator recovery; verify its retained process ownership and checkpoint before a controlled restart. Clearing a task session does not resolve this quarantine." })
|
|
.where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
const notice = await getExecutionBlocker(db, f.companyId, f.issueId);
|
|
expect(notice).toMatchObject({ runId: f.sourceRunId, canRetry: false });
|
|
expect(notice?.runError).toMatch(/still shutting down/);
|
|
expect(notice?.runError).not.toMatch(/operator|admin|controlled restart/i);
|
|
});
|
|
|
|
it.each(["verified", "unproven", "changed", "dry_run", "retry", "duplicate"])(
|
|
"requires exact local cleanup for a new turn after worker loss (%s)", async mode => {
|
|
const f = await seed();
|
|
await db.update(heartbeatRuns).set({ errorCode: "native_session_cleanup_quarantined" }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
const retire = vi.fn(() => mode !== "changed");
|
|
const verify = vi.spyOn(nativeExecutor, "verifyStoppedNativeSessionForContinuation").mockResolvedValue(
|
|
mode === "unproven" ? null : { evidence: { runId: f.sourceRunId, schema: "paperclip.stopped_native_conversation.v1" }, retire });
|
|
try {
|
|
const result = mode === "retry"
|
|
? await db.transaction(tx => admitExplicitNativeContinuation({ ...f, db: tx as unknown as typeof db,
|
|
reason: "retry_failed_run", failedRunId: f.sourceRunId }))
|
|
: await admit(f, mode === "dry_run");
|
|
// A native Retry starts a fresh conversation, so the same verified stop
|
|
// admits it; it never resumes the interrupted turn.
|
|
expect(Boolean(result)).toBe(["verified", "dry_run", "duplicate", "retry"].includes(mode));
|
|
expect(retire).toHaveBeenCalledTimes(["verified", "changed", "duplicate", "retry"].includes(mode) ? 1 : 0);
|
|
const actions = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issueId));
|
|
expect(Boolean(actions[0].evidence.explicitUserContinuation)).toBe(["verified", "duplicate", "retry"].includes(mode));
|
|
const events = await db.select().from(heartbeatRunEvents).where(and(eq(heartbeatRunEvents.runId, f.sourceRunId),
|
|
eq(heartbeatRunEvents.eventType, "native.stopped_conversation_verified")));
|
|
expect(events).toHaveLength(["verified", "duplicate", "retry"].includes(mode) ? 1 : 0);
|
|
if (mode === "duplicate") {
|
|
expect(await admit(f)).toBeNull();
|
|
expect(retire).toHaveBeenCalledTimes(1);
|
|
}
|
|
} finally { verify.mockRestore(); }
|
|
});
|
|
|
|
it.each(["absent", "live_runner", "saved_state", "dry_run", "foreign_host", "unbound_launch"] as const)(
|
|
"admits a Retry after a launch whose runner identity was never recorded only when no runner exists (%s)", async kind => {
|
|
// RC-42: the server was killed after requesting the runner launch and
|
|
// before recording its process identity, so there is no PID to verify.
|
|
const f = await seed();
|
|
const stateBase = await mkdtemp(join(tmpdir(), "unidentified-runner-"));
|
|
const previous = process.env.PAPERCLIP_RUNNER_STATE_DIR;
|
|
process.env.PAPERCLIP_RUNNER_STATE_DIR = stateBase;
|
|
let child: ReturnType<typeof spawn> | undefined;
|
|
try {
|
|
const nativeSessionId = randomUUID();
|
|
const execution = {
|
|
schema: "paperclip.native-execution-input.v1", provider: { kind: "codex", model: null },
|
|
binding: { companyId: f.companyId, issueId: f.issueId, agentId: f.agentId, runId: f.sourceRunId, executionWorkspaceId: "workspace" },
|
|
task: { identifier: "TEST", title: "Continue", description: null, prompt: "Continue", workMode: "standard" },
|
|
workspace: { cwd: stateBase, repoUrl: null, repoRef: null, branchName: null },
|
|
session: { normalizedSessionId: nativeSessionId, driverKind: "codex_app_server", protocolVersion: 1, lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null } },
|
|
completionContract: { id: "contract", sha256: "sha", schemaVersion: "paperclip.completion-contract.v1",
|
|
contract: { revision: "1", objective: "Continue", criteria: [{ id: "objective", requirement: "Continue" }] } },
|
|
interactionResponses: [], credentialBindings: [],
|
|
};
|
|
await db.update(heartbeatRuns).set({ processPid: null, processGroupId: null, nativeSessionId: null, runnerInstanceId: null,
|
|
errorCode: "adapter_failed", runnerProfileJson: { nativeExecutionInput: execution } }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
// Only the launching host can inspect its process table.
|
|
await appendHeartbeatRunEvent(db, { companyId: f.companyId, runId: f.sourceRunId, agentId: f.agentId, eventType: PROCESS_START_REQUESTED,
|
|
...(kind === "unbound_launch" ? {} : { payload: { hostname: kind === "foreign_host" ? `other-${os.hostname()}` : os.hostname() } }) });
|
|
if (kind === "live_runner") {
|
|
child = spawn(process.execPath, ["-e", "setInterval(() => {}, 1000)", "--", "--run-id", f.sourceRunId], { stdio: "ignore" });
|
|
await once(child, "spawn");
|
|
}
|
|
if (kind === "saved_state") {
|
|
const canonical = (value: unknown): string => value && typeof value === "object" && !Array.isArray(value)
|
|
? `{${Object.entries(value).sort(([a], [b]) => a.localeCompare(b)).map(([key, entry]) => `${JSON.stringify(key)}:${canonical(entry)}`).join(",")}}`
|
|
: JSON.stringify(value);
|
|
const root = join(stateBase, createHash("sha256").update(canonical({
|
|
schema: "paperclip.native-session-scope.v2", companyId: f.companyId, agentId: f.agentId,
|
|
workspace: { kind: "managed", executionWorkspaceId: "workspace" },
|
|
provider: { driverKind: "codex_app_server", identity: { kind: "codex" } }, normalizedSessionId: nativeSessionId,
|
|
})).digest("hex"));
|
|
await mkdir(join(root, "control-plane"), { recursive: true });
|
|
await writeFile(join(root, "control-plane/control-plane-state.json"), JSON.stringify({
|
|
schema: "paperclip.runner.durable.control-plane-state.v1", identity: { runId: f.sourceRunId } }));
|
|
}
|
|
const admitted = await admit(f, kind === "dry_run");
|
|
if (kind === "absent" || kind === "dry_run") expect(admitted).toMatchObject({ previousRunId: f.sourceRunId });
|
|
else expect(admitted).toBeNull();
|
|
// Only a real admission records the stop; a later launch invalidates it.
|
|
expect(await hasNativeLocalProcessStop(db, f.companyId, f.sourceRunId)).toBe(kind === "absent");
|
|
} finally {
|
|
if (child) { const exited = once(child, "exit"); child.kill("SIGKILL"); await exited; }
|
|
if (previous === undefined) delete process.env.PAPERCLIP_RUNNER_STATE_DIR;
|
|
else process.env.PAPERCLIP_RUNNER_STATE_DIR = previous;
|
|
await rm(stateBase, { recursive: true, force: true });
|
|
}
|
|
},
|
|
);
|
|
|
|
it.each(["proven", "coarse_before_request", "coarse_same_second", "state_after_request", "lineage_after_request", "previous_release",
|
|
"earlier_runner_unproven", "live_runner"] as const)(
|
|
"starts a Retry after restarts cut off a relaunch whose earlier runners were proven stopped (%s)", async kind => {
|
|
const f = await seed();
|
|
const stateBase = await mkdtemp(join(tmpdir(), "relaunched-runner-"));
|
|
const previous = process.env.PAPERCLIP_RUNNER_STATE_DIR;
|
|
process.env.PAPERCLIP_RUNNER_STATE_DIR = stateBase;
|
|
let child: ReturnType<typeof spawn> | undefined;
|
|
try {
|
|
const nativeSessionId = randomUUID();
|
|
const execution = {
|
|
schema: "paperclip.native-execution-input.v1", provider: { kind: "codex", model: null },
|
|
binding: { companyId: f.companyId, issueId: f.issueId, agentId: f.agentId, runId: f.sourceRunId, executionWorkspaceId: "workspace" },
|
|
task: { identifier: "TEST", title: "Continue", description: null, prompt: "Continue", workMode: "standard" },
|
|
workspace: { cwd: stateBase, repoUrl: null, repoRef: null, branchName: null },
|
|
session: { normalizedSessionId: nativeSessionId, driverKind: "codex_app_server", protocolVersion: 1, lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null } },
|
|
completionContract: { id: "contract", sha256: "sha", schemaVersion: "paperclip.completion-contract.v1",
|
|
contract: { revision: "1", objective: "Continue", criteria: [{ id: "objective", requirement: "Continue" }] } },
|
|
interactionResponses: [], credentialBindings: [],
|
|
};
|
|
await db.update(heartbeatRuns).set({ processPid: null, processGroupId: null, nativeSessionId: null, runnerInstanceId: null,
|
|
errorCode: "adapter_failed", runnerProfileJson: { nativeExecutionInput: execution } }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
const event = (eventType: string, payload: Record<string, unknown>) => appendHeartbeatRunEvent(db, {
|
|
companyId: f.companyId, runId: f.sourceRunId, agentId: f.agentId, eventType, payload });
|
|
const host = os.hostname(), deadPid = 999999998;
|
|
await event(PROCESS_START_REQUESTED, { hostname: host, providerAttempt: 1 });
|
|
await event(PROCESS_IDENTITY_RECORDED, { hostname: host, targetKind: "local", processPid: deadPid, processGroupId: deadPid,
|
|
processStartedAt: "2026-11-10T11:13:34.000Z" });
|
|
if (kind !== "earlier_runner_unproven") await event("native.local_process_stopped",
|
|
{ processPid: deadPid, processGroupId: deadPid, processStartedAt: "2026-11-10T11:13:34.000Z" });
|
|
const canonical = (value: unknown): string => value && typeof value === "object" && !Array.isArray(value)
|
|
? `{${Object.entries(value).sort(([a], [b]) => a.localeCompare(b)).map(([key, entry]) => `${JSON.stringify(key)}:${canonical(entry)}`).join(",")}}`
|
|
: JSON.stringify(value);
|
|
const root = join(stateBase, createHash("sha256").update(canonical({
|
|
schema: "paperclip.native-session-scope.v2", companyId: f.companyId, agentId: f.agentId,
|
|
workspace: { kind: "managed", executionWorkspaceId: "workspace" },
|
|
provider: { driverKind: "codex_app_server", identity: { kind: "codex" } }, normalizedSessionId: nativeSessionId,
|
|
})).digest("hex"));
|
|
await mkdir(join(root, "control-plane"), { recursive: true });
|
|
await mkdir(join(root, "runner"), { recursive: true });
|
|
await mkdir(join(root, "runner/process-lineage"), { recursive: true });
|
|
await writeFile(join(root, "control-plane/control-plane-state.json"), JSON.stringify({
|
|
schema: "paperclip.runner.durable.control-plane-state.v1", identity: { runId: f.sourceRunId } }));
|
|
await writeFile(join(root, "runner/runner-state.json"), JSON.stringify({ schema: "paperclip.runner.durable.state.v1" }));
|
|
await writeFile(join(root, `runner/process-lineage/writer-${deadPid}.json`), JSON.stringify({ schema: "paperclip.runner.process-lineage-writer.v1" }));
|
|
const requestedAt = new Date(Math.floor(Date.now() / 1000) * 1000 - 59_500);
|
|
const earlier = new Date(requestedAt.getTime() - 30_000), later = new Date(requestedAt.getTime() + 30_250);
|
|
const stateTimes: Partial<Record<typeof kind, Date>> = { state_after_request: later,
|
|
coarse_same_second: new Date(requestedAt.getTime() - 500), coarse_before_request: new Date(requestedAt.getTime() - 3_500) };
|
|
const stateTime = stateTimes[kind] ?? earlier;
|
|
await utimes(join(root, "runner/runner-state.json"), earlier, stateTime);
|
|
await utimes(join(root, `runner/process-lineage/writer-${deadPid}.json`), earlier, kind === "lineage_after_request" ? later : earlier);
|
|
await event(PROCESS_START_REQUESTED, { hostname: host, providerAttempt: 1,
|
|
...(kind === "previous_release" ? {} : { requestedAt: requestedAt.toISOString() }) });
|
|
if (kind === "live_runner") {
|
|
child = spawn(process.execPath, ["-e", "setInterval(() => {}, 1000)", "--", "--run-id", f.sourceRunId], { stdio: "ignore" });
|
|
await once(child, "spawn");
|
|
}
|
|
const blocked: string[] = [];
|
|
const admitted = await db.transaction(async tx => {
|
|
await tx.select().from(issues).where(eq(issues.id, f.issueId)).for("update");
|
|
return admitExplicitNativeContinuation({ ...f, db: tx as unknown as typeof db, reason: "retry_failed_run",
|
|
failedRunId: f.sourceRunId, onBlocked: reason => blocked.push(reason) });
|
|
});
|
|
if (kind === "proven" || kind === "coarse_before_request") {
|
|
expect(admitted).toMatchObject({ previousRunId: f.sourceRunId, failedRunId: f.sourceRunId });
|
|
expect(blocked).toEqual([]);
|
|
} else {
|
|
expect(admitted).toBeNull();
|
|
expect(blocked).toEqual(["process_identity_missing"]);
|
|
}
|
|
expect(await hasNativeLocalProcessStop(db, f.companyId, f.sourceRunId)).toBe(kind === "proven" || kind === "coarse_before_request");
|
|
} finally {
|
|
if (child) { const exited = once(child, "exit"); child.kill("SIGKILL"); await exited; }
|
|
if (previous === undefined) delete process.env.PAPERCLIP_RUNNER_STATE_DIR;
|
|
else process.env.PAPERCLIP_RUNNER_STATE_DIR = previous;
|
|
await rm(stateBase, { recursive: true, force: true });
|
|
}
|
|
},
|
|
);
|
|
|
|
it.each(["suspended", "ready", "wrong_run", "wrong_thread", "active_provider", "pending_tool", "pending_output", "missing_state", "new_launch"])(
|
|
"recovers a historical run without process metadata only from exact suspended state (%s)", async kind => {
|
|
const f = await seed();
|
|
const stateBase = await mkdtemp(join(tmpdir(), "historical-native-followup-"));
|
|
const previous = process.env.PAPERCLIP_RUNNER_STATE_DIR;
|
|
process.env.PAPERCLIP_RUNNER_STATE_DIR = stateBase;
|
|
try {
|
|
const nativeSessionId = randomUUID(), runnerInstanceId = randomUUID();
|
|
const execution = {
|
|
schema: "paperclip.native-execution-input.v1", provider: { kind: "codex", model: null },
|
|
binding: { companyId: f.companyId, issueId: f.issueId, agentId: f.agentId, runId: f.sourceRunId, executionWorkspaceId: "workspace" },
|
|
task: { identifier: "TEST", title: "Continue", description: null, prompt: "Continue", workMode: "standard" },
|
|
workspace: { cwd: stateBase, repoUrl: null, repoRef: null, branchName: null },
|
|
session: { normalizedSessionId: nativeSessionId, driverKind: "codex_app_server", protocolVersion: 1, lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null } },
|
|
completionContract: { id: "contract", sha256: "sha", schemaVersion: "paperclip.completion-contract.v1",
|
|
contract: { revision: "1", objective: "Continue", criteria: [{ id: "objective", requirement: "Continue" }] } },
|
|
interactionResponses: [], credentialBindings: [],
|
|
};
|
|
await db.update(heartbeatRuns).set({ processPid: null, nativeSessionId, runnerInstanceId,
|
|
errorCode: "native_runner_process_exited", runnerProfileJson: { nativeExecutionInput: execution,
|
|
sessionCheckpoint: { sessionId: "exact-thread", providerSessionId: "backend-account" } },
|
|
}).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
const canonical = (value: unknown): string => value && typeof value === "object" && !Array.isArray(value)
|
|
? `{${Object.entries(value).sort(([a], [b]) => a.localeCompare(b)).map(([key, entry]) => `${JSON.stringify(key)}:${canonical(entry)}`).join(",")}}`
|
|
: JSON.stringify(value);
|
|
const root = join(stateBase, createHash("sha256").update(canonical({
|
|
schema: "paperclip.native-session-scope.v2", companyId: f.companyId, agentId: f.agentId,
|
|
workspace: { kind: "managed", executionWorkspaceId: "workspace" },
|
|
provider: { driverKind: "codex_app_server", identity: { kind: "codex" } }, normalizedSessionId: nativeSessionId,
|
|
})).digest("hex"));
|
|
if (kind !== "missing_state") {
|
|
await mkdir(join(root, "control-plane"), { recursive: true });
|
|
await mkdir(join(root, "runner"), { recursive: true });
|
|
const identity = { runId: kind === "wrong_run" ? randomUUID() : f.sourceRunId, runnerInstanceId,
|
|
normalizedSessionId: nativeSessionId, environmentLeaseId: "workspace" };
|
|
await writeFile(join(root, "control-plane/control-plane-state.json"), JSON.stringify({ schema: "paperclip.runner.durable.control-plane-state.v1", identity }));
|
|
await writeFile(join(root, "runner/runner-state.json"), JSON.stringify({ schema: "paperclip.runner.durable.state.v1",
|
|
...identity, lifecycle: kind === "ready" ? "ready" : "suspended", outbox: kind === "pending_output" ? [{}] : [] }));
|
|
await writeFile(join(root, "runner/codex-provider-state.json"), JSON.stringify({
|
|
schema: "paperclip.runner.codex-provider-state.v1", lifecycle: "prepared",
|
|
threadId: kind === "wrong_thread" ? "another-thread" : "exact-thread", providerSessionId: "backend-account",
|
|
activeProviderTurnId: kind === "active_provider" ? "unfinished-turn" : null, ambiguousTurnStartPending: false,
|
|
config: { provider: "codex", driver: "codex_app_server" }, pendingEvents: [], queuedEvents: [],
|
|
toolBridge: { pending: kind === "pending_tool" ? { call: {} } : {} }, activeProviderResultFingerprint: null,
|
|
}));
|
|
}
|
|
if (kind === "new_launch") await appendHeartbeatRunEvent(db, { companyId: f.companyId, runId: f.sourceRunId,
|
|
agentId: f.agentId, eventType: PROCESS_START_REQUESTED });
|
|
if (kind !== "suspended") {
|
|
expect(await admit(f, true)).toBeNull();
|
|
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).not.toBeNull();
|
|
return;
|
|
}
|
|
expect(await admit(f, true)).toMatchObject({ previousRunId: f.sourceRunId });
|
|
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).not.toBeNull();
|
|
// Exercise the real message admission path while keeping the provider slot occupied.
|
|
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" });
|
|
const heartbeat = heartbeatService(db);
|
|
for (let n = 0; n < 2; n++) await heartbeat.wakeup(f.agentId, { source: "automation", triggerDetail: "system",
|
|
reason: "issue_commented", requestedByActorType: "user", requestedByActorId: "board",
|
|
payload: { issueId: f.issueId, commentId: f.commentId },
|
|
contextSnapshot: { issueId: f.issueId, wakeCommentId: f.commentId } });
|
|
const successors = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId), eq(heartbeatRuns.status, "queued")));
|
|
expect(successors).toHaveLength(1);
|
|
expect(successors[0].contextSnapshot).toMatchObject({ previousRunId: f.sourceRunId, forceFreshSession: true, wakeCommentId: f.commentId });
|
|
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toBeNull();
|
|
const [coordinator] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.sourceRunId));
|
|
expect(coordinator).toMatchObject({ phase: "terminal_failure", attempt: 3 });
|
|
const [action] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issueId));
|
|
expect(action.evidence.automaticRecovery).toMatchObject({ actionOutcome: "unknown", replay: "explicit_user_continuation" });
|
|
expect(await hasNativeLocalProcessStop(db, f.companyId, f.sourceRunId)).toBe(false);
|
|
} finally {
|
|
if (previous === undefined) delete process.env.PAPERCLIP_RUNNER_STATE_DIR;
|
|
else process.env.PAPERCLIP_RUNNER_STATE_DIR = previous;
|
|
await rm(stateBase, { recursive: true, force: true });
|
|
}
|
|
},
|
|
);
|
|
async function seedCancelledStartup() {
|
|
const f = await seed();
|
|
await db.update(heartbeatRuns).set({ status: "cancelled", processPid: null,
|
|
startedAt: new Date("2026-09-11T09:59:59Z"),
|
|
runtimeModeResolvedAt: new Date("2026-09-11T10:00:01Z"),
|
|
controllerBootId: randomUUID(), controllerLeaseExpiresAt: new Date("2026-09-11T10:01:00Z"),
|
|
}).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
await db.update(nativeRunFinalizations).set({ phase: "observed", attempt: 0,
|
|
failureDetail: null,
|
|
}).where(eq(nativeRunFinalizations.runId, f.sourceRunId));
|
|
await db.insert(environmentLeases).values({ companyId: f.companyId, heartbeatRunId: f.sourceRunId,
|
|
provider: "local", status: "released", releasedAt: new Date("2026-09-11T10:00:02Z"),
|
|
cleanupStatus: "succeeded", leasePolicy: "ephemeral" });
|
|
return f;
|
|
}
|
|
|
|
it("settles a cancelled unclaimed coordinator after restart and admits one user successor", async () => {
|
|
const f = await seedCancelledStartup();
|
|
expect(await admit(f, true)).toMatchObject({ previousRunId: f.sourceRunId });
|
|
expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.sourceRunId)))[0].phase).toBe("observed");
|
|
const results = await Promise.all([admit(f), admit(f)]);
|
|
expect(results.filter(Boolean)).toHaveLength(1);
|
|
const [coordinator] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.sourceRunId));
|
|
expect(coordinator).toMatchObject({ phase: "terminal_failure", attempt: 0,
|
|
failureCode: "native_startup_cancelled", failureDetail: { replacementDenied: "explicit_user_continuation" } });
|
|
const [action] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issueId));
|
|
expect(action.evidence.automaticRecovery).toMatchObject({ actionOutcome: "unknown", replay: "explicit_user_continuation" });
|
|
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toBeNull();
|
|
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.sourceRunId)))[0].status).toBe("cancelled");
|
|
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.successorRunId))).toHaveLength(1);
|
|
});
|
|
|
|
it("continues native-runner preparation cancelled before runtime selection", async () => {
|
|
const f = await seedCancelledStartup();
|
|
await db.delete(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.sourceRunId));
|
|
await db.update(heartbeatRuns).set({ runtimeMode: "legacy", runtimeModeResolvedAt: null, nativeIssueId: null,
|
|
runnerProfileJson: { adapterDispatch: { adapterType: "paperclip_runner" } },
|
|
resultJson: { startupCancellation: { beforeNativeSelection: true }, startupPreparationSettledAt: new Date().toISOString() },
|
|
}).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
expect(await admit(f)).toMatchObject({ previousRunId: f.sourceRunId });
|
|
});
|
|
|
|
async function seedHistoricalCancelledPreparation(retainedReceipt = false) {
|
|
const f = await seedCancelledStartup();
|
|
await db.delete(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.sourceRunId));
|
|
await db.delete(environmentLeases).where(eq(environmentLeases.heartbeatRunId, f.sourceRunId));
|
|
await db.update(heartbeatRuns).set({ runtimeMode: "legacy", runtimeModeResolvedAt: null,
|
|
nativeIssueId: null, nativeSessionId: null, executionStage: "preparing", errorCode: "operator_interrupted",
|
|
runnerProfileJson: { adapterDispatch: { adapterType: "paperclip_runner" } },
|
|
resultJson: retainedReceipt ? { operatorInterrupted: true, stopReason: "cancelled",
|
|
startupCancellation: { beforeNativeSelection: true, requestedAt: "2026-09-11T10:00:00Z" } } : null,
|
|
}).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
await db.update(issueRecoveryActions).set({ cause: "legacy_execution_requires_reconciliation" })
|
|
.where(eq(issueRecoveryActions.sourceIssueId, f.issueId));
|
|
return f;
|
|
}
|
|
|
|
async function seedCancelledReviewWait() {
|
|
const f = await seedHistoricalCancelledPreparation();
|
|
await db.update(heartbeatRuns).set({ startedAt: null, executionStage: null,
|
|
controllerBootId: null, controllerLeaseExpiresAt: null, runnerProfileJson: null,
|
|
errorCode: "issue_continuation_waiting_on_review",
|
|
resultJson: { stopReason: "issue_continuation_waiting_on_review", timeoutSource: "stale_queued_run_gate" },
|
|
}).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
return f;
|
|
}
|
|
|
|
async function seedCancelledWorkspaceWaitRetry() {
|
|
const f = await seed(), parentRunId = randomUUID();
|
|
await db.delete(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.sourceRunId));
|
|
await db.insert(heartbeatRuns).values({ id: parentRunId, companyId: f.companyId, agentId: f.agentId,
|
|
scopeKind: "issue", issueId: f.issueId,
|
|
runtimeMode: "legacy", status: "cancelled", executionStage: "settled", errorCode: "workspace_busy",
|
|
createdAt: new Date("2026-09-11T09:57:00Z"), startedAt: new Date("2026-09-11T09:57:00Z"),
|
|
finishedAt: new Date("2026-09-11T09:58:00Z"), contextSnapshot: { issueId: f.issueId },
|
|
runnerProfileJson: { adapterDispatch: { adapterType: "paperclip_runner" }, chatControlRecoveryAdmission: {} },
|
|
usageJson: { ledgerScope: { issueId: f.issueId, projectId: null, billingCode: null } },
|
|
resultJson: { executionRecovery: { kind: "workspace_wait", providerWorkStarted: false } } });
|
|
await db.update(heartbeatRuns).set({ runtimeMode: "legacy", nativeIssueId: null, nativeSessionId: null,
|
|
scopeKind: "issue", issueId: f.issueId, retryOfRunId: parentRunId, scheduledRetryReason: "workspace_busy",
|
|
status: "cancelled", errorCode: "cancelled", startedAt: null, processPid: null,
|
|
createdAt: new Date("2026-09-11T09:59:00Z"), runnerProfileJson: null,
|
|
contextSnapshot: { issueId: f.issueId, source: "issue.comment", retryReason: "workspace_busy", retryOfRunId: parentRunId },
|
|
resultJson: { stopReason: "cancelled", cancellation: { reason: "Cancelled by control plane", source: "control_plane",
|
|
expected: true, initiator: { type: "system" }, recordedAt: "2026-09-11T10:00:00Z" },
|
|
startupCancellation: { errorCode: "cancelled", requestedAt: "2026-09-11T10:00:00Z", beforeNativeSelection: false } },
|
|
}).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
await db.update(issueRecoveryActions).set({ cause: "legacy_execution_requires_reconciliation" })
|
|
.where(eq(issueRecoveryActions.sourceIssueId, f.issueId));
|
|
return { ...f, parentRunId };
|
|
}
|
|
|
|
it("admits a fresh direction after an unclaimed workspace retry was cancelled", async () => {
|
|
const f = await seedCancelledWorkspaceWaitRetry();
|
|
expect(await admit(f)).toMatchObject({ previousRunId: f.sourceRunId, commentId: f.commentId });
|
|
const [source] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
expect(source).toMatchObject({ status: "cancelled", startedAt: null, processPid: null, runnerProfileJson: null });
|
|
const [hold] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issueId));
|
|
expect(hold.evidence.automaticRecovery).toMatchObject({ replay: "explicit_user_continuation", actionOutcome: "unknown" });
|
|
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toBeNull();
|
|
expect(await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId),
|
|
eq(heartbeatRuns.status, "queued")))).toHaveLength(1);
|
|
});
|
|
|
|
async function seedHistoricalPausedWorkspaceRetry(parentHold = false, retryNow = false) {
|
|
const f = await seedCancelledWorkspaceWaitRetry();
|
|
const dueAt = new Date("2026-09-11T10:00:00Z");
|
|
const pauseRootIssueId = parentHold ? randomUUID() : f.issueId;
|
|
if (parentHold) {
|
|
await db.insert(issues).values({ id: pauseRootIssueId, companyId: f.companyId,
|
|
title: "Shared workspace parent", status: "in_progress", assigneeAgentId: f.agentId });
|
|
await db.update(issues).set({ parentId: pauseRootIssueId }).where(eq(issues.id, f.issueId));
|
|
}
|
|
const [pause] = await db.insert(issueTreeHolds).values({ companyId: f.companyId,
|
|
rootIssueId: pauseRootIssueId, mode: "pause", status: "active", createdAt: new Date(dueAt.getTime() - 1_000),
|
|
releasePolicy: { strategy: "manual" } }).returning();
|
|
await db.insert(issueTreeHoldMembers).values({ companyId: f.companyId, holdId: pause.id,
|
|
issueId: f.issueId, parentIssueId: parentHold ? pauseRootIssueId : null, depth: parentHold ? 1 : 0,
|
|
issueTitle: "Paused workspace retry", issueStatus: "blocked", assigneeAgentId: f.agentId });
|
|
await db.update(heartbeatRuns).set({ status: "scheduled_retry", errorCode: null,
|
|
finishedAt: null, resultJson: null, scheduledRetryAt: dueAt, scheduledRetryAttempt: 1,
|
|
}).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
if (retryNow) {
|
|
expect(await heartbeatService(db).retryScheduledRetryNow({ issueId: f.issueId,
|
|
actor: { actorType: "user", actorId: "board" }, now: dueAt })).toMatchObject({ outcome: "gate_suppressed" });
|
|
const events = await db.select().from(heartbeatRunEvents).where(eq(heartbeatRunEvents.runId, f.sourceRunId))
|
|
.orderBy(heartbeatRunEvents.seq);
|
|
expect(events).toMatchObject([
|
|
{ seq: 1, eventType: "lifecycle", stream: "system", level: "info", message: "Scheduled retry was requested to run now" },
|
|
{ seq: 2, eventType: "lifecycle", stream: "system", level: "warn",
|
|
message: "Scheduled retry suppressed because the issue is held by an active subtree pause hold" },
|
|
]);
|
|
} else {
|
|
expect(await createPostgresRunDispatchAdapter(db).promoteOrCancelDueRetry({
|
|
companyId: f.companyId, runId: f.sourceRunId, now: dueAt,
|
|
})).toMatchObject({ outcome: "gate_suppressed", errorCode: "issue_paused" });
|
|
}
|
|
await db.update(heartbeatRuns).set({ resultJson: null }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
await issueTreeControlService(db).releaseHold(f.companyId, pauseRootIssueId, pause.id,
|
|
{ actor: { actorType: "user", actorId: "board" } });
|
|
return { ...f, pauseId: pause.id, dueAt };
|
|
}
|
|
|
|
it.each([
|
|
{ parentHold: false, retryNow: false }, { parentHold: true, retryNow: false },
|
|
{ parentHold: false, retryNow: true }, { parentHold: true, retryNow: true },
|
|
])("resumes saved input after a historical pause suppressed an unclaimed workspace retry (parent hold: $parentHold, retry now: $retryNow)", async ({ parentHold, retryNow }) => {
|
|
const f = await seedHistoricalPausedWorkspaceRetry(parentHold, retryNow);
|
|
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" });
|
|
const [wake] = await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: f.agentId,
|
|
source: "on_demand", reason: "issue_commented", requestedByActorType: "user", requestedByActorId: "board",
|
|
status: "deferred_issue_execution", updatedAt: new Date(0), payload: { issueId: f.issueId, commentId: f.commentId,
|
|
executionWait: { reason: "execution_recovery" },
|
|
_paperclipWakeContext: { issueId: f.issueId, wakeCommentId: f.commentId, wakeCommentIds: [f.commentId] } },
|
|
}).returning();
|
|
const heartbeat = heartbeatService(db);
|
|
await Promise.all([heartbeat.resumeExecutionWaitComments(), heartbeat.resumeExecutionWaitComments()]);
|
|
const successors = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId),
|
|
eq(heartbeatRuns.status, "queued")));
|
|
expect(successors).toHaveLength(1);
|
|
expect(successors[0].contextSnapshot).toMatchObject({ forceFreshSession: true, wakeCommentIds: [f.commentId] });
|
|
expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, wake.id)))[0])
|
|
.toMatchObject({ status: "coalesced", runId: successors[0].id });
|
|
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.sourceRunId)))[0])
|
|
.toMatchObject({ status: "cancelled", startedAt: null, errorCode: "issue_paused", resultJson: null });
|
|
});
|
|
|
|
it("holds a Retry Now pause history that also contains provider execution evidence", async () => {
|
|
const f = await seedHistoricalPausedWorkspaceRetry(false, true);
|
|
await appendHeartbeatRunEvent(db, { companyId: f.companyId, agentId: f.agentId, runId: f.sourceRunId,
|
|
eventType: "turn.started", stream: "system", level: "info" });
|
|
expect(await admit(f)).toBeNull();
|
|
expect(await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId),
|
|
eq(heartbeatRuns.status, "queued")))).toHaveLength(0);
|
|
});
|
|
|
|
it.each(["missing_event", "wrong_event", "missing_member", "active_pause", "late_pause", "early_release",
|
|
"started", "provider_event", "lease", "operator_stop", "parent_executed", "retry_binding"])(
|
|
"holds a historical pause-suppressed workspace retry with %s evidence", async kind => {
|
|
const f = await seedHistoricalPausedWorkspaceRetry();
|
|
if (kind === "missing_event") await db.delete(heartbeatRunEvents).where(eq(heartbeatRunEvents.runId, f.sourceRunId));
|
|
if (kind === "wrong_event") await db.update(heartbeatRunEvents).set({ payload: { holdId: f.pauseId,
|
|
rootIssueId: f.issueId, issueId: randomUUID(), scheduledRetryReason: "workspace_busy" } })
|
|
.where(eq(heartbeatRunEvents.runId, f.sourceRunId));
|
|
if (kind === "missing_member") await db.delete(issueTreeHoldMembers).where(eq(issueTreeHoldMembers.holdId, f.pauseId));
|
|
if (kind === "active_pause") await db.update(issueTreeHolds).set({ status: "active", releasedAt: null })
|
|
.where(eq(issueTreeHolds.id, f.pauseId));
|
|
if (kind === "late_pause") await db.update(issueTreeHolds).set({ createdAt: new Date(f.dueAt.getTime() + 1) })
|
|
.where(eq(issueTreeHolds.id, f.pauseId));
|
|
if (kind === "early_release") await db.update(issueTreeHolds).set({ releasedAt: new Date(f.dueAt.getTime() - 1) })
|
|
.where(eq(issueTreeHolds.id, f.pauseId));
|
|
if (kind === "started") await db.update(heartbeatRuns).set({ startedAt: f.dueAt }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
if (kind === "provider_event") await appendHeartbeatRunEvent(db, { companyId: f.companyId,
|
|
agentId: f.agentId, runId: f.sourceRunId, eventType: "turn.started", stream: "system", level: "info" });
|
|
if (kind === "lease") await db.insert(environmentLeases).values({ companyId: f.companyId,
|
|
heartbeatRunId: f.sourceRunId, provider: "local", leasePolicy: "ephemeral", status: "released",
|
|
cleanupStatus: "success", releasedAt: new Date() });
|
|
if (kind === "operator_stop") await db.update(heartbeatRuns).set({ errorCode: "operator_interrupted",
|
|
resultJson: { stopReason: "cancelled", cancellation: { source: "operator", expected: true,
|
|
initiator: { type: "user", userId: "board" }, reason: "Stopped", recordedAt: f.dueAt.toISOString() } } })
|
|
.where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
if (kind === "parent_executed") await db.update(heartbeatRuns).set({ resultJson: {
|
|
executionRecovery: { kind: "workspace_wait", providerWorkStarted: true } } }).where(eq(heartbeatRuns.id, f.parentRunId));
|
|
if (kind === "retry_binding") await db.update(heartbeatRuns).set({ retryOfRunId: null }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
expect(await admit(f)).toBeNull();
|
|
});
|
|
|
|
it.each(["message", "retry"])("admits an unclaimed workspace retry through the heartbeat %s path", async kind => {
|
|
const f = await seedCancelledWorkspaceWaitRetry();
|
|
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" });
|
|
const successor = await heartbeatService(db).wakeup(f.agentId, { source: "on_demand", triggerDetail: "manual",
|
|
reason: kind === "retry" ? "retry_failed_run" : "issue_commented",
|
|
...(kind === "retry" ? { failedRunId: f.sourceRunId } : {}), requestedByActorType: "user", requestedByActorId: "board",
|
|
payload: { issueId: f.issueId, ...(kind === "message" ? { commentId: f.commentId } : {}) },
|
|
contextSnapshot: { issueId: f.issueId, ...(kind === "message" ? { wakeCommentId: f.commentId } : {}) },
|
|
});
|
|
expect(successor).toMatchObject({ status: "queued", contextSnapshot: { forceFreshSession: true,
|
|
previousRunId: f.sourceRunId, explicitUserContinuation: { commentId: kind === "message" ? f.commentId : null } } });
|
|
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toBeNull();
|
|
});
|
|
|
|
it("resumes saved input after an unclaimed workspace retry exactly once", async () => {
|
|
const f = await seedCancelledWorkspaceWaitRetry();
|
|
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" });
|
|
const [wake] = await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: f.agentId,
|
|
source: "on_demand", reason: "issue_commented", requestedByActorType: "user", requestedByActorId: "board",
|
|
status: "deferred_issue_execution", updatedAt: new Date(0), payload: { issueId: f.issueId, commentId: f.commentId,
|
|
executionWait: { reason: "execution_recovery" },
|
|
_paperclipWakeContext: { issueId: f.issueId, wakeCommentId: f.commentId, wakeCommentIds: [f.commentId] } },
|
|
}).returning();
|
|
const heartbeat = heartbeatService(db);
|
|
await Promise.all([heartbeat.resumeExecutionWaitComments(), heartbeat.resumeExecutionWaitComments()]);
|
|
const successors = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId), eq(heartbeatRuns.status, "queued")));
|
|
expect(successors).toHaveLength(1);
|
|
expect(successors[0].contextSnapshot).toMatchObject({ forceFreshSession: true, wakeCommentIds: [f.commentId] });
|
|
expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, wake.id)))[0])
|
|
.toMatchObject({ status: "coalesced", runId: successors[0].id });
|
|
});
|
|
|
|
it.each(["started", "lease", "launch", "foreign_parent", "parent_executed", "parent_reason", "parent_receipt", "profile", "owner", "operator_stop", "output",
|
|
"parent_lease", "parent_launch", "parent_native", "retry_binding", "parent_output", "parent_profile", "parent_later", "restore"])(
|
|
"keeps cancelled workspace retries held with contradictory %s evidence", async kind => {
|
|
const f = await seedCancelledWorkspaceWaitRetry();
|
|
if (kind === "started") await db.update(heartbeatRuns).set({ startedAt: new Date("2026-09-11T09:59:30Z") }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
if (kind === "lease") await db.insert(environmentLeases).values({ companyId: f.companyId, heartbeatRunId: f.sourceRunId,
|
|
provider: "local", leasePolicy: "ephemeral", status: "released", cleanupStatus: "success", releasedAt: new Date() });
|
|
if (kind === "launch") await db.insert(heartbeatRunEvents).values({ companyId: f.companyId,
|
|
agentId: f.agentId, runId: f.sourceRunId, seq: 1, eventType: PROCESS_START_REQUESTED, payload: {} });
|
|
if (kind === "foreign_parent") {
|
|
const foreign = await seed();
|
|
await db.update(heartbeatRuns).set({ contextSnapshot: { issueId: f.issueId, retryReason: "workspace_busy",
|
|
retryOfRunId: foreign.sourceRunId } }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
}
|
|
if (kind === "parent_executed" || kind === "parent_receipt") await db.update(heartbeatRuns).set({ resultJson:
|
|
kind === "parent_receipt" ? {} : { executionRecovery: { kind: "workspace_wait", providerWorkStarted: true } } })
|
|
.where(eq(heartbeatRuns.id, f.parentRunId));
|
|
if (kind === "parent_reason") await db.update(heartbeatRuns).set({ errorCode: "cancelled" }).where(eq(heartbeatRuns.id, f.parentRunId));
|
|
if (kind === "profile") await db.update(heartbeatRuns).set({ runnerProfileJson: { adapterDispatch: { adapterType: "paperclip_runner" } } })
|
|
.where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
if (kind === "owner") await db.update(heartbeatRuns).set({ controllerBootId: randomUUID() }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
if (kind === "operator_stop") await db.update(heartbeatRuns).set({ resultJson: { stopReason: "cancelled",
|
|
cancellation: { reason: "Stopped", source: "operator", expected: true, initiator: { type: "user", userId: "board" },
|
|
recordedAt: "2026-09-11T10:00:00Z" } } }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
if (kind === "output") await db.update(heartbeatRuns).set({ lastOutputSeq: 1 }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
if (kind === "parent_lease") await db.insert(environmentLeases).values({ companyId: f.companyId, heartbeatRunId: f.parentRunId,
|
|
provider: "local", leasePolicy: "ephemeral", status: "released", cleanupStatus: "success", releasedAt: new Date() });
|
|
if (kind === "parent_launch") await db.insert(heartbeatRunEvents).values({ companyId: f.companyId,
|
|
agentId: f.agentId, runId: f.parentRunId, seq: 1, eventType: PROCESS_START_REQUESTED, payload: {} });
|
|
if (kind === "parent_native") await db.update(heartbeatRuns).set({ nativeIssueId: f.issueId }).where(eq(heartbeatRuns.id, f.parentRunId));
|
|
if (kind === "retry_binding") await db.update(heartbeatRuns).set({ retryOfRunId: null }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
if (kind === "parent_output") await db.update(heartbeatRuns).set({ lastOutputSeq: 1 }).where(eq(heartbeatRuns.id, f.parentRunId));
|
|
if (kind === "parent_profile") await db.update(heartbeatRuns).set({ runnerProfileJson: {
|
|
adapterDispatch: { adapterType: "paperclip_runner" }, remoteWorkspace: {} } }).where(eq(heartbeatRuns.id, f.parentRunId));
|
|
if (kind === "parent_later") await db.update(heartbeatRuns).set({ finishedAt: new Date("2026-09-11T10:01:00Z") }).where(eq(heartbeatRuns.id, f.parentRunId));
|
|
if (kind === "restore") await db.update(heartbeatRuns).set({ resultJson: { stopReason: "cancelled",
|
|
workspaceRestoreFailure: "restore_unsafe_archive" } }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
expect(await admit(f)).toBeNull();
|
|
if (kind === "lease" || kind === "parent_lease") await db.delete(environmentLeases)
|
|
.where(eq(environmentLeases.heartbeatRunId, kind === "lease" ? f.sourceRunId : f.parentRunId));
|
|
},
|
|
);
|
|
|
|
it("keeps provider diagnostics out of admission evidence and preserves absent results", async () => {
|
|
const f = await seedCancelledReviewWait();
|
|
const heartbeat = heartbeatService(db);
|
|
await db.update(heartbeatRuns).set({ resultJson: {
|
|
stopReason: "issue_continuation_waiting_on_review", timeoutSource: "stale_queued_run_gate",
|
|
summary: "private provider output", providerPayload: "large provider output".repeat(10000),
|
|
} }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
const projected = await heartbeat.getRun(f.sourceRunId, { includeExecutionEvidence: true });
|
|
expect(projected?.resultJson).toMatchObject({ stopReason: "issue_continuation_waiting_on_review", timeoutSource: "stale_queued_run_gate" });
|
|
expect(projected?.resultJson).not.toHaveProperty("summary");
|
|
expect(projected?.resultJson).not.toHaveProperty("providerPayload");
|
|
await db.update(heartbeatRuns).set({ resultJson: null }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
expect((await heartbeat.getRun(f.sourceRunId, { includeExecutionEvidence: true }))?.resultJson).toBeNull();
|
|
await db.update(heartbeatRuns).set({ resultJson: { unrecognizedReceipt: true } }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
expect((await heartbeat.getRun(f.sourceRunId, { includeExecutionEvidence: true }))?.resultJson).not.toBeNull();
|
|
});
|
|
|
|
it.each([false, true].flatMap(ascii => ["message", "retry"].map(kind => ({ ascii, kind }))))(
|
|
"recovers a pre-dispatch review wait through an explicit $kind (SQL_ASCII: $ascii)", async ({ ascii, kind }) => {
|
|
const f = await seedCancelledReviewWait();
|
|
const heartbeat = heartbeatService(db);
|
|
if (ascii) {
|
|
const encoding = vi.spyOn(db, "execute").mockResolvedValueOnce([{ server_encoding: "SQL_ASCII" }] as never);
|
|
try { expect((await heartbeat.getRun(f.sourceRunId))?.resultJson).toBeNull(); }
|
|
finally { encoding.mockRestore(); }
|
|
}
|
|
const notice = await getExecutionBlocker(db, f.companyId, f.issueId);
|
|
expect(notice).toMatchObject({ canRetry: true, runError: "Waiting for review; this continuation never started." });
|
|
expect(notice?.nextAction).not.toContain("Inspect the run before sending a new message");
|
|
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" });
|
|
const successor = await heartbeat.wakeup(f.agentId, { source: kind === "retry" ? "on_demand" : "automation", triggerDetail: "manual",
|
|
reason: kind === "retry" ? "retry_failed_run" : "issue_commented",
|
|
...(kind === "retry" ? { failedRunId: f.sourceRunId } : {}), requestedByActorType: "user", requestedByActorId: "board",
|
|
payload: { issueId: f.issueId, ...(kind === "message" ? { commentId: f.commentId } : {}) },
|
|
contextSnapshot: { issueId: f.issueId, ...(kind === "message" ? { wakeCommentId: f.commentId } : {}) },
|
|
});
|
|
expect(successor).toMatchObject({ status: "queued", contextSnapshot: { forceFreshSession: true,
|
|
previousRunId: f.sourceRunId, explicitUserContinuation: { commentId: kind === "message" ? f.commentId : null } } });
|
|
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toBeNull();
|
|
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.sourceRunId)))[0].status).toBe("cancelled");
|
|
});
|
|
|
|
it.each([false, true])("reconsiders saved input after a pre-dispatch review wait exactly once (SQL_ASCII: %s)", async ascii => {
|
|
const f = await seedCancelledReviewWait();
|
|
const heartbeat = heartbeatService(db);
|
|
if (ascii) {
|
|
const encoding = vi.spyOn(db, "execute").mockResolvedValueOnce([{ server_encoding: "SQL_ASCII" }] as never);
|
|
try { expect((await heartbeat.getRun(f.sourceRunId))?.resultJson).toBeNull(); }
|
|
finally { encoding.mockRestore(); }
|
|
}
|
|
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" });
|
|
const leaseId = randomUUID();
|
|
await db.insert(environmentLeases).values({ id: leaseId, companyId: f.companyId,
|
|
heartbeatRunId: f.sourceRunId, provider: "local", status: "pending_cleanup", cleanupStatus: "failed" });
|
|
await heartbeat.wakeup(f.agentId, { source: "automation", reason: "issue_commented",
|
|
requestedByActorType: "user", requestedByActorId: "board", payload: { issueId: f.issueId, commentId: f.commentId },
|
|
contextSnapshot: { issueId: f.issueId, wakeCommentId: f.commentId } });
|
|
const [waiting] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.companyId));
|
|
expect(waiting.status).toBe("deferred_issue_execution");
|
|
await db.update(environmentLeases).set({ status: "released", releasedAt: new Date(), cleanupStatus: "succeeded" })
|
|
.where(eq(environmentLeases.id, leaseId));
|
|
await db.update(agentWakeupRequests).set({ updatedAt: new Date(0) }).where(eq(agentWakeupRequests.id, waiting.id));
|
|
await Promise.all([heartbeat.resumeExecutionWaitComments(), heartbeat.resumeExecutionWaitComments()]);
|
|
const successors = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId), eq(heartbeatRuns.status, "queued")));
|
|
expect(successors).toHaveLength(1);
|
|
expect(successors[0].contextSnapshot).toMatchObject({ wakeCommentIds: [f.commentId],
|
|
explicitUserContinuation: { commentId: f.commentId }, forceFreshSession: true });
|
|
expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, waiting.id)))[0])
|
|
.toMatchObject({ status: "coalesced", runId: successors[0].id });
|
|
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toBeNull();
|
|
});
|
|
|
|
it.each(["launch", "provider", "cleanup"])("retains a pre-dispatch review wait with contradictory %s evidence", async kind => {
|
|
const f = await seedCancelledReviewWait();
|
|
if (kind === "cleanup") await db.insert(environmentLeases).values({ companyId: f.companyId,
|
|
heartbeatRunId: f.sourceRunId, provider: "local", status: "pending_cleanup", cleanupStatus: "failed" });
|
|
else await db.insert(heartbeatRunEvents).values({ companyId: f.companyId, runId: f.sourceRunId,
|
|
agentId: f.agentId, seq: 1, eventType: kind === "launch" ? PROCESS_START_REQUESTED : "provider.event",
|
|
stream: "system", level: "info", message: "Contradictory execution evidence" });
|
|
try {
|
|
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toMatchObject({ canRetry: false });
|
|
expect(await admit(f)).toBeNull();
|
|
} finally {
|
|
await db.delete(environmentLeases).where(eq(environmentLeases.heartbeatRunId, f.sourceRunId));
|
|
}
|
|
});
|
|
|
|
it.each([false, true].flatMap(receipt => ["message", "retry"].map(kind => ({ receipt, kind }))))(
|
|
"recovers historical native preparation through an explicit $kind (receipt: $receipt)", async ({ receipt, kind }) => {
|
|
const f = await seedHistoricalCancelledPreparation(receipt);
|
|
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toMatchObject({ canRetry: true });
|
|
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" });
|
|
const successor = await heartbeatService(db).wakeup(f.agentId, {
|
|
source: "on_demand", triggerDetail: "manual", reason: kind === "retry" ? "retry_failed_run" : "issue_commented",
|
|
...(kind === "retry" ? { failedRunId: f.sourceRunId } : {}), requestedByActorType: "user", requestedByActorId: "board",
|
|
payload: { issueId: f.issueId, ...(kind === "message" ? { commentId: f.commentId } : {}) },
|
|
contextSnapshot: { issueId: f.issueId, ...(kind === "message" ? { wakeCommentId: f.commentId } : {}) },
|
|
});
|
|
expect(successor).toMatchObject({ status: "queued", contextSnapshot: { forceFreshSession: true,
|
|
previousRunId: f.sourceRunId, explicitUserContinuation: { commentId: kind === "message" ? f.commentId : null } } });
|
|
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toBeNull();
|
|
const [source] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
expect(source).toMatchObject({ status: "cancelled", runtimeMode: "legacy",
|
|
resultJson: receipt ? { startupCancellation: { beforeNativeSelection: true } } : null });
|
|
});
|
|
|
|
it.each([false, true].flatMap(receipt => ["fresh", "earlier_delivered", "last_delivered"].map(kind => ({ receipt, kind }))))(
|
|
"resumes saved input after historical native preparation expires across restart exactly once ($kind, receipt: $receipt)", async ({ receipt, kind }) => {
|
|
const f = await seedHistoricalCancelledPreparation(receipt);
|
|
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" });
|
|
await db.update(heartbeatRuns).set({ controllerLeaseExpiresAt: new Date(Date.now() + 60_000) })
|
|
.where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
await heartbeatService(db).wakeup(f.agentId, { source: "automation", reason: "issue_commented",
|
|
requestedByActorType: "user", requestedByActorId: "board", payload: { issueId: f.issueId, commentId: f.commentId },
|
|
contextSnapshot: { issueId: f.issueId, wakeCommentId: f.commentId } });
|
|
const [waiting] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.companyId));
|
|
expect(waiting.status).toBe("deferred_issue_execution");
|
|
if (kind !== "fresh") {
|
|
const deliveredId = randomUUID();
|
|
await db.insert(issueComments).values({ id: deliveredId, companyId: f.companyId, issueId: f.issueId,
|
|
authorType: "user", authorUserId: "another-author", body: "Already handled",
|
|
createdAt: new Date(kind === "earlier_delivered" ? "2026-09-11T10:30:00Z" : "2026-09-11T11:30:00Z") });
|
|
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "succeeded",
|
|
startedAt: new Date("2026-09-11T12:00:00Z"), finishedAt: new Date("2026-09-11T12:01:00Z"),
|
|
contextSnapshot: { issueId: f.issueId, wakeCommentIds: [deliveredId] } });
|
|
await db.update(agentWakeupRequests).set({ payload: { ...waiting.payload,
|
|
_paperclipWakeContext: { wakeCommentIds: kind === "earlier_delivered" ? [deliveredId, f.commentId] : [f.commentId, deliveredId] } },
|
|
}).where(eq(agentWakeupRequests.id, waiting.id));
|
|
}
|
|
await db.update(heartbeatRuns).set({ controllerLeaseExpiresAt: new Date(0) })
|
|
.where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
await db.update(agentWakeupRequests).set({ updatedAt: new Date(0) }).where(eq(agentWakeupRequests.id, waiting.id));
|
|
await Promise.all([heartbeatService(db).resumeExecutionWaitComments(), heartbeatService(db).resumeExecutionWaitComments()]);
|
|
const successors = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId), eq(heartbeatRuns.status, "queued")));
|
|
expect(successors).toHaveLength(1);
|
|
expect(successors[0].contextSnapshot).toMatchObject({ wakeCommentIds: [f.commentId],
|
|
explicitUserContinuation: { commentId: f.commentId }, forceFreshSession: true });
|
|
const [queueReceipt] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, waiting.id));
|
|
expect(queueReceipt).toMatchObject({ status: "coalesced", runId: successors[0].id });
|
|
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toBeNull();
|
|
});
|
|
|
|
it.each([false, true].flatMap(receipt => ["resolved", "dispatching", "adapter", "missing_boot", "current_boot", "missing_lease", "live_lease", "receipt", "launch", "invoked", "coordinator", "cleanup", "remote"].map(kind => ({ receipt, kind }))))(
|
|
"holds historical native preparation with contradictory or incomplete $kind evidence (receipt: $receipt)", async ({ receipt, kind }) => {
|
|
const f = await seedHistoricalCancelledPreparation(receipt);
|
|
const patch = kind === "resolved" ? { runtimeModeResolvedAt: new Date() }
|
|
: kind === "dispatching" ? { executionStage: "dispatching" }
|
|
: kind === "adapter" ? { runnerProfileJson: { adapterDispatch: { adapterType: "process" } } }
|
|
: kind === "missing_boot" ? { controllerBootId: null }
|
|
: kind === "current_boot" ? { controllerBootId: legacyControllerBootId }
|
|
: kind === "missing_lease" ? { controllerLeaseExpiresAt: null }
|
|
: kind === "live_lease" ? { controllerLeaseExpiresAt: new Date(Date.now() + 60_000) }
|
|
: kind === "receipt" ? { resultJson: { startupCancellation: { beforeNativeSelection: false } } } : {};
|
|
if (Object.keys(patch).length) await db.update(heartbeatRuns).set(patch).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
if (kind === "launch" || kind === "invoked") await db.insert(heartbeatRunEvents).values({ companyId: f.companyId,
|
|
agentId: f.agentId, runId: f.sourceRunId, seq: 1,
|
|
eventType: kind === "launch" ? PROCESS_START_REQUESTED : "adapter.invoke", payload: { adapterType: "paperclip_runner" } });
|
|
if (kind === "coordinator") {
|
|
await db.update(heartbeatRuns).set({ nativeIssueId: f.issueId }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
await db.insert(nativeRunFinalizations).values({ companyId: f.companyId,
|
|
runId: f.sourceRunId, issueId: f.issueId, phase: "observed", attempt: 0 });
|
|
}
|
|
if (kind === "cleanup" || kind === "remote") await db.insert(environmentLeases).values({ companyId: f.companyId,
|
|
heartbeatRunId: f.sourceRunId, provider: kind === "remote" ? "daytona" : "local",
|
|
providerLeaseId: kind === "remote" ? "unverified" : null, leasePolicy: "ephemeral",
|
|
status: "pending_cleanup", cleanupStatus: "failed" });
|
|
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toMatchObject({ canRetry: false });
|
|
expect(await admit(f)).toBeNull();
|
|
// Later tests exercise the global cleanup sweep with their own retry
|
|
// counters. Do not leave this negative fixture as another cleanup target.
|
|
await db.delete(environmentLeases).where(eq(environmentLeases.heartbeatRunId, f.sourceRunId));
|
|
},
|
|
);
|
|
|
|
it.each(["attempt", "generation", "controller", "lease", "process", "launch", "provider", "cleanup", "remote", "preparing", "closed", "reassigned"])(
|
|
"retains cancellation safeguards with %s evidence", async kind => {
|
|
const f = await seedCancelledStartup();
|
|
if (kind === "attempt") await db.update(nativeRunFinalizations).set({ attempt: 1 }).where(eq(nativeRunFinalizations.runId, f.sourceRunId));
|
|
if (kind === "generation") await db.update(nativeRunFinalizations).set({ controllerGeneration: 1 }).where(eq(nativeRunFinalizations.runId, f.sourceRunId));
|
|
if (kind === "controller") await db.update(nativeRunFinalizations).set({ controllerBootId: "old-owner" }).where(eq(nativeRunFinalizations.runId, f.sourceRunId));
|
|
if (kind === "lease") await db.update(nativeRunFinalizations).set({ leaseOwner: "owner", leaseExpiresAt: new Date(Date.now() + 60000) }).where(eq(nativeRunFinalizations.runId, f.sourceRunId));
|
|
if (kind === "process") await db.update(heartbeatRuns).set({ processPid: process.pid }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
if (kind === "launch" || kind === "provider") await db.insert(heartbeatRunEvents).values({ companyId: f.companyId,
|
|
agentId: f.agentId, runId: f.sourceRunId, seq: 1,
|
|
eventType: kind === "launch" ? PROCESS_START_REQUESTED : "provider.event",
|
|
...(kind === "provider" ? { sourceEventId: "provider-1", sourceInstanceId: "provider", sourceSeq: 1, protocolSchemaVersion: 1, canonicalPayloadHash: "hash" } : {}),
|
|
});
|
|
if (kind === "cleanup") await db.update(environmentLeases).set({ status: "pending_cleanup", cleanupStatus: "failed" }).where(eq(environmentLeases.heartbeatRunId, f.sourceRunId));
|
|
if (kind === "remote") await db.update(environmentLeases).set({ provider: "daytona", providerLeaseId: "unverified" }).where(eq(environmentLeases.heartbeatRunId, f.sourceRunId));
|
|
if (kind === "preparing") await db.update(heartbeatRuns).set({ controllerLeaseExpiresAt: new Date(Date.now() + 60000) }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
if (kind === "closed") await db.update(issues).set({ status: "done" }).where(eq(issues.id, f.issueId));
|
|
if (kind === "reassigned") await db.update(issues).set({ assigneeAgentId: null }).where(eq(issues.id, f.issueId));
|
|
expect(await admit(f)).toBeNull();
|
|
expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.sourceRunId)))[0].phase).toBe("observed");
|
|
await db.delete(environmentLeases).where(eq(environmentLeases.heartbeatRunId, f.sourceRunId));
|
|
},
|
|
);
|
|
|
|
it("preserves local stop proof after process metadata is cleared and invalidates it on another launch", async () => {
|
|
const f = await seed();
|
|
const [source] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
await db.transaction(async tx => {
|
|
expect(await recordNativeLocalProcessStop(tx as unknown as typeof db, source)).toBe(true);
|
|
await tx.update(heartbeatRuns).set({ processPid: null }).where(eq(heartbeatRuns.id, source.id));
|
|
});
|
|
expect(await hasNativeLocalProcessStop(db, f.companyId, source.id)).toBe(true);
|
|
expect(await hasNativeLocalProcessStop(db, randomUUID(), source.id)).toBe(false);
|
|
expect(await admit(f, true)).toMatchObject({ previousRunId: source.id });
|
|
await persistHeartbeatRunProcessMetadata(db, source.id, { pid: 999999999, processGroupId: null, startedAt: new Date().toISOString(), targetKind: "local" });
|
|
await db.update(heartbeatRuns).set({ processPid: null }).where(eq(heartbeatRuns.id, source.id));
|
|
expect(await admit(f, true)).toBeNull();
|
|
expect(await db.transaction(tx => recordNativeLocalProcessStop(tx as unknown as typeof db, source))).toBe(true);
|
|
await appendHeartbeatRunEvent(db, { companyId: f.companyId, runId: source.id, agentId: f.agentId,
|
|
eventType: PROCESS_START_REQUESTED });
|
|
// No PID was stored for the new launch, as when the server dies after spawn.
|
|
expect(await admit(f, true)).toBeNull();
|
|
});
|
|
|
|
it.each(["live", "remote", "provider_event"])("does not accept invalid local stop proof: %s", async kind => {
|
|
const f = await seed();
|
|
if (kind === "remote") {
|
|
const [environment] = await db.insert(environments).values({ name: "Remote stop", driver: "sandbox" }).returning();
|
|
await db.insert(environmentLeases).values({ companyId: f.companyId, heartbeatRunId: f.sourceRunId,
|
|
environmentId: environment.id, provider: "daytona", status: "active", leasePolicy: "ephemeral" });
|
|
}
|
|
if (kind === "live") await db.update(heartbeatRuns).set({ processPid: process.pid }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
const [source] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
if (kind === "provider_event") {
|
|
await db.insert(heartbeatRunEvents).values({ companyId: f.companyId, runId: source.id, agentId: f.agentId,
|
|
eventType: "native.local_process_stopped", seq: 1, sourceEventId: randomUUID() });
|
|
} else expect(await recordNativeLocalProcessStop(db, source)).toBe(false);
|
|
expect(await hasNativeLocalProcessStop(db, f.companyId, source.id)).toBe(false);
|
|
});
|
|
|
|
it.each(["stopped_process", "cancelled_startup"])("resumes saved local messages after restart exactly once: %s", async kind => {
|
|
const f = kind === "cancelled_startup" ? await seedCancelledStartup() : await seed();
|
|
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" });
|
|
await db.update(heartbeatRuns).set({ processPid: process.pid }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
// A prior cancelled admission is also held, but cannot select the native
|
|
// retry source. The newest blocker must win just as it does on Send.
|
|
const oldAdmissionId = randomUUID();
|
|
await db.insert(heartbeatRuns).values({ id: oldAdmissionId, companyId: f.companyId, agentId: f.agentId,
|
|
status: "cancelled", errorCode: "execution_reconciliation_required", contextSnapshot: { issueId: f.issueId },
|
|
finishedAt: new Date("2026-09-11T09:00:00Z") });
|
|
await db.update(issueRecoveryActions).set({ updatedAt: new Date(0), evidence: { runId: oldAdmissionId,
|
|
automaticRecovery: { replay: "blocked", actionOutcome: "unknown" } } }).where(eq(issueRecoveryActions.sourceIssueId, f.issueId));
|
|
await db.insert(issueRecoveryActions).values({ companyId: f.companyId, sourceIssueId: f.issueId,
|
|
kind: "active_run_watchdog", cause: "uncertain_external_action", fingerprint: randomUUID(), status: "active",
|
|
nextAction: "Waiting for the current run.", evidence: { runId: f.sourceRunId } });
|
|
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toMatchObject({ runId: f.sourceRunId });
|
|
await heartbeatService(db).wakeup(f.agentId, { source: "automation", triggerDetail: "system", reason: "issue_commented",
|
|
requestedByActorType: "user", requestedByActorId: "board", payload: { issueId: f.issueId, commentId: f.commentId },
|
|
contextSnapshot: { issueId: f.issueId, wakeCommentId: f.commentId } });
|
|
const [waiting] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.companyId));
|
|
expect(waiting.payload?.executionWait).toMatchObject({ reason: kind === "cancelled_startup" ? "controller_settling" : "process_running" });
|
|
const makeDue = () => db.update(agentWakeupRequests).set({ updatedAt: new Date(0) }).where(eq(agentWakeupRequests.id, waiting.id));
|
|
await makeDue();
|
|
await heartbeatService(db).resumeExecutionWaitComments();
|
|
expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.companyId))).toHaveLength(1);
|
|
await db.update(heartbeatRuns).set({ processPid: 999999999 }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
const [stopped] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
await db.transaction(async tx => {
|
|
await recordNativeLocalProcessStop(tx as unknown as typeof db, stopped);
|
|
await tx.update(heartbeatRuns).set({ processPid: null }).where(eq(heartbeatRuns.id, stopped.id));
|
|
});
|
|
const holdId = randomUUID();
|
|
await db.insert(issueTreeHolds).values({ id: holdId, companyId: f.companyId, rootIssueId: f.issueId, mode: "pause", status: "active" });
|
|
await db.insert(issueTreeHoldMembers).values({ companyId: f.companyId, holdId, issueId: f.issueId, depth: 0, issueTitle: "Deploy", issueStatus: "blocked" });
|
|
for (let attempt = 0; attempt < 2; attempt++) {
|
|
await makeDue();
|
|
await heartbeatService(db).resumeExecutionWaitComments();
|
|
}
|
|
expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.companyId))).toHaveLength(1);
|
|
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).not.toBeNull();
|
|
const [paused] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, waiting.id));
|
|
expect(paused.payload?.executionWait).toMatchObject({ reason: "issue_tree_hold_active" });
|
|
await db.update(issueTreeHolds).set({ status: "released" }).where(eq(issueTreeHolds.id, holdId));
|
|
await db.update(agents).set({ runtimeConfig: { heartbeat: { maxConcurrentRuns: 1, maxDailyRuns: 0 } } }).where(eq(agents.id, f.agentId));
|
|
for (let attempt = 0; attempt < 2; attempt++) {
|
|
await makeDue();
|
|
await heartbeatService(db).resumeExecutionWaitComments();
|
|
}
|
|
expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.companyId))).toHaveLength(1);
|
|
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).not.toBeNull();
|
|
await db.update(agents).set({ runtimeConfig: { heartbeat: { maxConcurrentRuns: 1 } } }).where(eq(agents.id, f.agentId));
|
|
await makeDue();
|
|
await Promise.all([heartbeatService(db).resumeExecutionWaitComments(), heartbeatService(db).resumeExecutionWaitComments()]);
|
|
const runs = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId), eq(heartbeatRuns.status, "queued")));
|
|
expect(runs).toHaveLength(1);
|
|
expect(runs[0].contextSnapshot).toMatchObject({ forceFreshSession: true, previousRunId: f.sourceRunId,
|
|
explicitUserContinuation: { commentId: f.commentId } });
|
|
const [adopted] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, waiting.id));
|
|
expect(adopted).toMatchObject({ status: "coalesced", runId: runs[0].id });
|
|
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toBeNull();
|
|
});
|
|
|
|
|
|
it.each(["issue_commented", "retry_failed_run"])("does not bypass an unsafe restore hold through %s", async reason => {
|
|
const f = await seed();
|
|
await db.update(agents).set({ adapterType: "grok_local" }).where(eq(agents.id, f.agentId));
|
|
await db.update(heartbeatRuns).set({ runtimeMode: "legacy", processPid: null,
|
|
errorCode: "workspace_restore_failed", resultJson: { workspaceRestoreFailure: "restore_unsafe_archive", conversationContinuation: "continue_conversation_v1" },
|
|
}).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
await db.delete(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.sourceRunId));
|
|
await db.update(issueRecoveryActions).set({ cause: "legacy_execution_requires_reconciliation" }).where(eq(issueRecoveryActions.sourceIssueId, f.issueId));
|
|
const blocked = vi.fn();
|
|
expect(await db.transaction(tx => admitExplicitNativeContinuation({ ...f, reason,
|
|
commentId: reason === "issue_commented" ? f.commentId : null,
|
|
failedRunId: reason === "retry_failed_run" ? f.sourceRunId : null,
|
|
onBlocked: blocked, db: tx as unknown as typeof db,
|
|
}))).toBeNull();
|
|
expect(blocked).toHaveBeenCalledWith("workspace_repair_required", expect.any(String));
|
|
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toMatchObject({ runId: f.sourceRunId });
|
|
});
|
|
|
|
it.each(["issue_commented", "retry_failed_run"])("continues a legacy Daytona run lost before adapter.invoke: %s", async reason => {
|
|
const f = await seed();
|
|
await db.update(agents).set({ adapterType: "claude_local" }).where(eq(agents.id, f.agentId));
|
|
await db.update(heartbeatRuns).set({ runtimeMode: "legacy", processPid: null,
|
|
errorCode: "process_lost" }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
await db.delete(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.sourceRunId));
|
|
await db.update(issueRecoveryActions).set({ cause: "legacy_execution_requires_reconciliation" })
|
|
.where(eq(issueRecoveryActions.sourceIssueId, f.issueId));
|
|
const [environment] = await db.insert(environments).values({ name: `Daytona startup ${f.sourceRunId}`, driver: "sandbox" }).returning();
|
|
const identity = { id: randomUUID(), companyId: f.companyId, heartbeatRunId: f.sourceRunId,
|
|
provider: "daytona", providerLeaseId: "startup-sandbox" };
|
|
await db.insert(environmentLeases).values({ ...identity, environmentId: environment.id,
|
|
status: "released", leasePolicy: "ephemeral", releasedAt: new Date(), cleanupStatus: "success",
|
|
metadata: { remoteExecutionTermination: remoteTerminationReceipt(identity,
|
|
{ providerLeaseId: identity.providerLeaseId, state: "destroyed" }) } });
|
|
const result = await db.transaction(tx => admitExplicitNativeContinuation({ ...f, reason,
|
|
commentId: reason === "issue_commented" ? f.commentId : null,
|
|
failedRunId: reason === "retry_failed_run" ? f.sourceRunId : null,
|
|
db: tx as unknown as typeof db }));
|
|
expect(result).toMatchObject({ previousRunId: f.sourceRunId });
|
|
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toBeNull();
|
|
const [source] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
expect(source.resultJson).toBeNull();
|
|
});
|
|
|
|
it.each(["claim", "invocation"])("does not convert a known process run after switching the agent to Claude: %s", async evidence => {
|
|
const f = await seed();
|
|
await db.update(agents).set({ adapterType: "claude_local" }).where(eq(agents.id, f.agentId));
|
|
await db.update(heartbeatRuns).set({ runtimeMode: "legacy", errorCode: "process_lost",
|
|
runnerProfileJson: evidence === "claim" ? { adapterDispatch: { adapterType: "process" } } : null,
|
|
}).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
if (evidence === "invocation") await db.insert(heartbeatRunEvents).values({ companyId: f.companyId,
|
|
runId: f.sourceRunId, agentId: f.agentId, seq: 1, eventType: "adapter.invoke", payload: { adapterType: "process" } });
|
|
await db.update(issueRecoveryActions).set({ cause: "legacy_execution_requires_reconciliation" })
|
|
.where(eq(issueRecoveryActions.sourceIssueId, f.issueId));
|
|
expect(await admit(f)).toBeNull();
|
|
expect(await admitExplicitNativeContinuation({ ...f, db, reason: "retry_failed_run",
|
|
commentId: null, failedRunId: f.sourceRunId })).toBeNull();
|
|
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).not.toBeNull();
|
|
});
|
|
|
|
it("keeps failed remote cleanup blocked even after the lease release timestamp is recorded", async () => {
|
|
const f = await seed();
|
|
await db.update(heartbeatRuns).set({ runtimeMode: "legacy", processPid: null,
|
|
resultJson: { conversationContinuation: "continue_conversation_v1" } }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
const [environment] = await db.insert(environments).values({ name: `Cleanup ${f.sourceRunId}`, driver: "sandbox" }).returning();
|
|
await db.insert(environmentLeases).values({ companyId: f.companyId, heartbeatRunId: f.sourceRunId,
|
|
environmentId: environment.id, provider: "daytona", providerLeaseId: "still-running",
|
|
status: "pending_cleanup", releasedAt: new Date(), cleanupStatus: "failed", leasePolicy: "ephemeral" });
|
|
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toMatchObject({ cause: "execution_owner_active" });
|
|
await db.delete(environmentLeases).where(eq(environmentLeases.heartbeatRunId, f.sourceRunId));
|
|
});
|
|
|
|
it("retries exhausted cleanup only for the selected failed run and adopts concurrent Retry clicks", async () => {
|
|
const f = await seed(), other = await seed();
|
|
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" });
|
|
const identities = [f, other].map(fixture => ({ id: randomUUID(), companyId: fixture.companyId,
|
|
heartbeatRunId: fixture.sourceRunId, provider: "daytona", providerLeaseId: fixture.sourceRunId }));
|
|
for (const identity of identities) await db.insert(environmentLeases).values({ ...identity,
|
|
status: "pending_cleanup", leasePolicy: "ephemeral", releasedAt: new Date(), cleanupStatus: "failed",
|
|
metadata: { pendingCleanupRetryAttempts: 5, pendingCleanupRetryCapWarned: true } });
|
|
const destroyed: string[] = [];
|
|
let readyCount = 0;
|
|
let bothReady!: () => void;
|
|
const ready = new Promise<void>(resolve => { bothReady = resolve; });
|
|
const heartbeat = heartbeatService(db, { environmentRuntime: {
|
|
isPendingCleanupWorkerReady: async () => { if (++readyCount === 2) bothReady(); await ready; return true; },
|
|
retryPendingSandboxTeardown: async ({ lease }: { lease: { id: string; providerLeaseId: string } }) => {
|
|
destroyed.push(lease.id);
|
|
return { providerLeaseId: lease.providerLeaseId, state: "destroyed" };
|
|
},
|
|
} as unknown as HeartbeatEnvironmentRuntime });
|
|
const request = { source: "on_demand" as const, triggerDetail: "manual" as const,
|
|
reason: "retry_failed_run", failedRunId: f.sourceRunId,
|
|
requestedByActorType: "user" as const, requestedByActorId: "board", payload: { issueId: f.issueId } };
|
|
try {
|
|
const [first, second] = await Promise.all([heartbeat.wakeup(f.agentId, request), heartbeat.wakeup(f.agentId, request)]);
|
|
// A losing cleanup claim can still see the hold until the winner finishes;
|
|
// a subsequent click adopts the already admitted successor.
|
|
const successor = first ?? second;
|
|
expect(successor?.id).toBeTruthy();
|
|
expect((await heartbeat.wakeup(f.agentId, request))?.id).toBe(successor?.id);
|
|
expect(destroyed).toEqual([identities[0].id]);
|
|
const [untouched] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, identities[1].id));
|
|
expect(untouched).toMatchObject({ status: "pending_cleanup", metadata: { pendingCleanupRetryAttempts: 5 } });
|
|
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toBeNull();
|
|
} finally {
|
|
for (const identity of identities) await db.delete(environmentLeases).where(eq(environmentLeases.id, identity.id));
|
|
}
|
|
});
|
|
|
|
it("allows a later user cleanup attempt after transient failure without resetting automatic retries", async () => {
|
|
const f = await seed();
|
|
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" });
|
|
const identity = { id: randomUUID(), companyId: f.companyId, heartbeatRunId: f.sourceRunId,
|
|
provider: "daytona", providerLeaseId: f.sourceRunId };
|
|
await db.insert(environmentLeases).values({ ...identity, status: "pending_cleanup", leasePolicy: "ephemeral",
|
|
releasedAt: new Date(), cleanupStatus: "failed", metadata: { pendingCleanupRetryAttempts: 5 } });
|
|
let attempts = 0;
|
|
const heartbeat = heartbeatService(db, { environmentRuntime: {
|
|
retryPendingSandboxTeardown: async () => {
|
|
if (++attempts < 3) throw new Error("provider temporarily unavailable");
|
|
return { providerLeaseId: identity.providerLeaseId, state: "destroyed" };
|
|
},
|
|
} as unknown as HeartbeatEnvironmentRuntime });
|
|
const request = { source: "on_demand" as const, triggerDetail: "manual" as const,
|
|
reason: "retry_failed_run", failedRunId: f.sourceRunId, requestedByActorType: "user" as const,
|
|
requestedByActorId: "board", payload: { issueId: f.issueId } };
|
|
try {
|
|
expect(await heartbeat.wakeup(f.agentId, request)).toBeNull();
|
|
expect(attempts).toBe(1);
|
|
expect(await heartbeat.wakeup(f.agentId, request)).toBeNull();
|
|
expect(attempts).toBe(2);
|
|
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).not.toBeNull();
|
|
await heartbeat.sweepPendingCleanupLeases();
|
|
expect(attempts).toBe(2);
|
|
const successor = await heartbeat.wakeup(f.agentId, request);
|
|
expect(attempts).toBe(3);
|
|
expect(successor?.retryOfRunId).toBe(f.sourceRunId);
|
|
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toBeNull();
|
|
expect((await heartbeat.wakeup(f.agentId, request))?.id).toBe(successor?.id);
|
|
expect(attempts).toBe(3);
|
|
} finally {
|
|
await db.delete(environmentLeases).where(eq(environmentLeases.id, identity.id));
|
|
}
|
|
});
|
|
|
|
it("queues one exact Retry with fresh history and adopts repeated clicks", async () => {
|
|
const f = await seed();
|
|
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" });
|
|
const service = heartbeatService(db);
|
|
const request = { source: "on_demand" as const, triggerDetail: "manual" as const,
|
|
reason: "retry_failed_run", failedRunId: f.sourceRunId,
|
|
requestedByActorType: "user" as const, requestedByActorId: "board", payload: { issueId: f.issueId } };
|
|
const [first, second] = await Promise.all([service.wakeup(f.agentId, request), service.wakeup(f.agentId, request)]);
|
|
expect(first?.id).toBeTruthy();
|
|
expect(second?.id).toBe(first?.id);
|
|
expect(first).toMatchObject({ retryOfRunId: f.sourceRunId,
|
|
contextSnapshot: { previousRunId: f.sourceRunId, forceFreshSession: true } });
|
|
const envelope = await buildExecutionContinuation({ db, companyId: f.companyId, issueId: f.issueId,
|
|
agentId: f.agentId, runId: first!.id, context: first!.contextSnapshot!, summary: null, exposeLowTrustRaw: false });
|
|
expect(envelope.interruptedRunId).toBe(f.sourceRunId);
|
|
await expect(buildExecutionContinuation({ db, companyId: f.companyId, issueId: f.issueId,
|
|
agentId: f.agentId, runId: randomUUID(), context: first!.contextSnapshot!, summary: null, exposeLowTrustRaw: false }))
|
|
.rejects.toThrow("continuation_user_authorization_missing");
|
|
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toBeNull();
|
|
});
|
|
|
|
it.each([true, false])("acknowledges a legacy remote Stop only after confirmed lease cleanup: %s", async confirmed => {
|
|
const f = await seed();
|
|
await db.update(agents).set({ adapterType: "claude_local" }).where(eq(agents.id, f.agentId));
|
|
await db.update(heartbeatRuns).set({ runtimeMode: "legacy", status: "cancelled", processPid: null,
|
|
resultJson: { executionCancellation: { state: "requested" } },
|
|
}).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
await db.insert(heartbeatRunEvents).values({ companyId: f.companyId, runId: f.sourceRunId,
|
|
agentId: f.agentId, seq: 1, eventType: "adapter.invoke", payload: { adapterType: "claude_local" } });
|
|
await db.update(issueRecoveryActions).set({ cause: "legacy_execution_requires_reconciliation" })
|
|
.where(eq(issueRecoveryActions.sourceIssueId, f.issueId));
|
|
const [environment] = await db.insert(environments).values({ name: `Remote ${f.sourceRunId}`, driver: "sandbox" }).returning();
|
|
const identity = { id: randomUUID(), companyId: f.companyId, heartbeatRunId: f.sourceRunId,
|
|
provider: "daytona", providerLeaseId: "sandbox-legacy" };
|
|
await db.insert(environmentLeases).values({ ...identity, environmentId: environment.id,
|
|
status: "expired", leasePolicy: "ephemeral", releasedAt: new Date(), cleanupStatus: "success",
|
|
metadata: confirmed ? { remoteExecutionTermination: remoteTerminationReceipt(identity,
|
|
{ providerLeaseId: identity.providerLeaseId, state: "destroyed" }) } : {},
|
|
});
|
|
await heartbeatService(db).releaseEnvironmentLeasesForRun({ runId: f.sourceRunId,
|
|
companyId: f.companyId, agentId: f.agentId, status: "cancelled" });
|
|
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
expect(run.resultJson?.executionCancellation).toMatchObject({ state: confirmed ? "acknowledged" : "requested" });
|
|
if (confirmed) expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toBeNull();
|
|
else expect(await getExecutionBlocker(db, f.companyId, f.issueId)).not.toBeNull();
|
|
});
|
|
|
|
it.each(["stopped", "destroyed", "missing", "wrong_lease", "cleanup_failed", "active"])(
|
|
"admits a remote native predecessor only with confirmed termination: %s", async kind => {
|
|
const f = await seed();
|
|
// This PID exists on the control-plane host. It must never be used to
|
|
// infer liveness of the identically numbered remote process.
|
|
await db.update(heartbeatRuns).set({ processPid: process.pid }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
const [environment] = await db.insert(environments).values({ name: `Remote ${f.sourceRunId}`, driver: "sandbox" }).returning();
|
|
const identity = { id: randomUUID(), companyId: f.companyId, heartbeatRunId: f.sourceRunId,
|
|
provider: "daytona", providerLeaseId: "sandbox-1" };
|
|
const proof = remoteTerminationReceipt(identity, { providerLeaseId: "sandbox-1",
|
|
state: kind === "destroyed" ? "destroyed" : "stopped" });
|
|
await db.insert(environmentLeases).values({ ...identity, environmentId: environment.id,
|
|
issueId: f.issueId, status: kind === "active" ? "active" : "released", leasePolicy: "ephemeral",
|
|
releasedAt: kind === "active" ? null : new Date(), cleanupStatus: kind === "cleanup_failed" ? "failed" : "success",
|
|
metadata: kind === "missing" ? {} : { remoteExecutionTermination:
|
|
kind === "wrong_lease" ? { ...proof, providerLeaseId: "other-sandbox" } : proof },
|
|
});
|
|
const result = await admit(f);
|
|
if (["stopped", "destroyed"].includes(kind)) expect(result).toMatchObject({ previousRunId: f.sourceRunId });
|
|
else expect(result).toBeNull();
|
|
},
|
|
);
|
|
|
|
it.each([
|
|
{ runtime: "native", retry: false }, { runtime: "native", retry: true },
|
|
{ runtime: "legacy", retry: false }, { runtime: "legacy", retry: true },
|
|
{ runtime: "legacy_startup", retry: false }, { runtime: "legacy_startup", retry: true },
|
|
{ runtime: "native", retry: false, reason: "issue_assigned" },
|
|
{ runtime: "legacy", retry: false, reason: "issue_assigned" },
|
|
])("resumes a user message after confirmed cleanup: %j", async ({ runtime, retry, reason = "issue_commented" }) => {
|
|
const f = await seed();
|
|
if (runtime.startsWith("legacy")) {
|
|
await db.update(agents).set({ adapterType: "claude_local" }).where(eq(agents.id, f.agentId));
|
|
await db.update(heartbeatRuns).set({ runtimeMode: "legacy", status: "cancelled", processPid: null,
|
|
resultJson: { executionCancellation: { state: "requested" } } }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
if (runtime === "legacy_startup") {
|
|
await db.update(heartbeatRuns).set({ status: "failed", resultJson: null, errorCode: "process_lost" })
|
|
.where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
} else await db.insert(heartbeatRunEvents).values({ companyId: f.companyId, runId: f.sourceRunId,
|
|
agentId: f.agentId, seq: 1, eventType: "adapter.invoke", payload: { adapterType: "claude_local" } });
|
|
await db.update(issueRecoveryActions).set({ cause: "legacy_execution_requires_reconciliation" })
|
|
.where(eq(issueRecoveryActions.sourceIssueId, f.issueId));
|
|
}
|
|
// Keep the successor queued so this test never starts an actual provider.
|
|
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" });
|
|
const [environment] = await db.insert(environments).values({ name: `pending-${f.sourceRunId}`, driver: "sandbox" }).returning();
|
|
const identity = { id: randomUUID(), companyId: f.companyId, heartbeatRunId: f.sourceRunId,
|
|
provider: "daytona", providerLeaseId: "pending-sandbox" };
|
|
await db.insert(environmentLeases).values({ ...identity, environmentId: environment.id, status: "active", leasePolicy: "ephemeral" });
|
|
const heartbeat = heartbeatService(db, retry ? { environmentRuntime: {
|
|
retryPendingSandboxTeardown: async () => ({ providerLeaseId: identity.providerLeaseId, state: "destroyed" }),
|
|
} as unknown as HeartbeatEnvironmentRuntime } : {});
|
|
await heartbeat.wakeup(f.agentId, { source: "automation", triggerDetail: "system", reason,
|
|
requestedByActorType: "user", requestedByActorId: "board", payload: { issueId: f.issueId, commentId: f.commentId },
|
|
contextSnapshot: { issueId: f.issueId, wakeCommentId: f.commentId } });
|
|
const [source] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
await heartbeat.resumeRemoteStopComments(source);
|
|
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).not.toBeNull();
|
|
if (retry) {
|
|
await db.update(environmentLeases).set({ status: "pending_cleanup", cleanupStatus: "failed" })
|
|
.where(eq(environmentLeases.id, identity.id));
|
|
expect(await heartbeat.sweepPendingCleanupLeases()).toMatchObject({ destroyed: 1 });
|
|
const [lease] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, identity.id));
|
|
expect(lease.metadata?.remoteExecutionTermination).toMatchObject({ runId: f.sourceRunId, state: "destroyed" });
|
|
} else {
|
|
await db.update(environmentLeases).set({ status: "released", releasedAt: new Date(), cleanupStatus: "success",
|
|
metadata: { remoteExecutionTermination: remoteTerminationReceipt(identity,
|
|
{ providerLeaseId: identity.providerLeaseId, state: "stopped" }) } }).where(eq(environmentLeases.id, identity.id));
|
|
await heartbeat.releaseEnvironmentLeasesForRun({ runId: source.id, companyId: source.companyId,
|
|
agentId: source.agentId, status: source.status });
|
|
}
|
|
await heartbeat.resumeRemoteStopComments(source);
|
|
await heartbeat.resumeRemoteStopComments(source);
|
|
const runs = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId), eq(heartbeatRuns.status, "queued")));
|
|
expect(runs).toHaveLength(1);
|
|
if (runtime !== "legacy") expect(runs[0].contextSnapshot).toMatchObject({ forceFreshSession: true, previousRunId: f.sourceRunId,
|
|
explicitUserContinuation: { commentId: f.commentId } });
|
|
else expect(runs[0].contextSnapshot).toMatchObject({ wakeCommentId: f.commentId });
|
|
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toBeNull();
|
|
});
|
|
|
|
it("queues the actual user wake with a fresh session and retained source context", async () => {
|
|
const f = await seed();
|
|
// Occupy this agent's only slot so this admission test never starts a provider.
|
|
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" });
|
|
const wake = await heartbeatService(db).wakeup(f.agentId, {
|
|
source: "automation", triggerDetail: "system", reason: "issue_commented",
|
|
requestedByActorType: "user", requestedByActorId: "board",
|
|
payload: { issueId: f.issueId, commentId: f.commentId },
|
|
contextSnapshot: { issueId: f.issueId, wakeCommentId: f.commentId },
|
|
});
|
|
expect(wake).toMatchObject({ status: "queued", retryOfRunId: null,
|
|
contextSnapshot: { forceFreshSession: true, previousRunId: f.sourceRunId,
|
|
explicitUserContinuation: { previousRunId: f.sourceRunId, commentId: f.commentId } } });
|
|
const [action] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issueId));
|
|
expect(action.evidence.explicitUserContinuation).toMatchObject({ runId: wake!.id });
|
|
const envelope = await buildExecutionContinuation({ db, companyId: f.companyId, issueId: f.issueId,
|
|
agentId: f.agentId, context: wake!.contextSnapshot!, summary: "Deployment completed.", exposeLowTrustRaw: false });
|
|
expect(envelope.interruptedRunId).toBe(f.sourceRunId);
|
|
expect(envelope.objective).toBe("What happened?");
|
|
expect(envelope.completedWork).toBe("Deployment completed.");
|
|
});
|
|
|
|
it.each(["pause", "dependency", "state"])("keeps the existing %s gate on the actual user wake", async gate => {
|
|
const f = await seed();
|
|
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" });
|
|
if (gate === "pause") {
|
|
const holdId = randomUUID();
|
|
await db.insert(issueTreeHolds).values({ id: holdId, companyId: f.companyId, rootIssueId: f.issueId, mode: "pause", status: "active" });
|
|
await db.insert(issueTreeHoldMembers).values({ companyId: f.companyId, holdId, issueId: f.issueId, depth: 0, issueTitle: "Deploy", issueStatus: "blocked" });
|
|
} else if (gate === "dependency") {
|
|
const blockerId = randomUUID();
|
|
await db.insert(issues).values({ id: blockerId, companyId: f.companyId, title: "Required approval", status: "todo" });
|
|
await db.insert(issueRelations).values({ companyId: f.companyId, issueId: blockerId, relatedIssueId: f.issueId, type: "blocks" });
|
|
}
|
|
const wake = await heartbeatService(db).wakeup(f.agentId, {
|
|
source: "automation", triggerDetail: "system", reason: "issue_commented", requestedByActorType: "user", requestedByActorId: "board",
|
|
...(gate === "state" ? { issueStateGuard: { statuses: ["todo"], assigneeAgentId: f.agentId } } : {}),
|
|
payload: { issueId: f.issueId, commentId: f.commentId }, contextSnapshot: { issueId: f.issueId, wakeCommentId: f.commentId },
|
|
});
|
|
const [action] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issueId));
|
|
if (gate === "pause" || gate === "state") {
|
|
expect(wake).toBeNull();
|
|
expect(action.evidence.explicitUserContinuation).toBeUndefined();
|
|
} else {
|
|
expect(wake).toMatchObject({ contextSnapshot: { dependencyBlockedInteraction: true, unresolvedBlockerCount: 1 } });
|
|
expect(await db.select().from(issueRelations).where(eq(issueRelations.companyId, f.companyId))).toHaveLength(1);
|
|
}
|
|
});
|
|
|
|
it("preflights eligibility without retiring the hold or creating a successor", async () => {
|
|
const f = await seed();
|
|
expect(await admit(f, true)).toMatchObject({ previousRunId: f.sourceRunId });
|
|
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).not.toBeNull();
|
|
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.successorRunId))).toHaveLength(0);
|
|
const [action] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issueId));
|
|
expect(action.evidence.explicitUserContinuation).toBeUndefined();
|
|
});
|
|
it("retains the message receipt without a phantom run when ownership is still live", async () => {
|
|
const f = await seed();
|
|
await db.update(heartbeatRuns).set({ processPid: process.pid }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
const wake = await heartbeatService(db).wakeup(f.agentId, {
|
|
source: "automation", triggerDetail: "system", reason: "issue_commented",
|
|
requestedByActorType: "user", requestedByActorId: "board",
|
|
payload: { issueId: f.issueId, commentId: f.commentId },
|
|
contextSnapshot: { issueId: f.issueId, wakeCommentId: f.commentId },
|
|
});
|
|
expect(wake).toBeNull();
|
|
const [receipt] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.companyId));
|
|
expect(receipt).toMatchObject({ status: "deferred_issue_execution", requestedByActorId: "board", runId: null });
|
|
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId))).toHaveLength(1);
|
|
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).not.toBeNull();
|
|
});
|
|
it("lets a new human message continue after exhausted recovery without certifying old actions", async () => {
|
|
const f = await seed();
|
|
expect(await admit(f)).toEqual({ previousRunId: f.sourceRunId, commentId: f.commentId });
|
|
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toBeNull();
|
|
const [action] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issueId));
|
|
expect(action.evidence).toMatchObject({ automaticRecovery: { actionOutcome: "unknown" }, explicitUserContinuation: { runId: f.successorRunId, commentId: f.commentId } });
|
|
const [source] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
expect(source.status).toBe("failed");
|
|
const [coordinator] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.sourceRunId));
|
|
expect(coordinator.attempt).toBe(3);
|
|
expect(coordinator.failureDetail?.replacementDenied).toBe("explicit_user_continuation");
|
|
});
|
|
|
|
it.each(["ready", "live_process", "pause", "delivered", "mixed_authors", "last_delivered", "operator_stop"])(
|
|
"automatically delivers an unconsumed native message sent before the failed run stopped, respecting %s", async gate => {
|
|
const f = await seed();
|
|
await db.update(issueComments).set({ createdAt: new Date("2026-09-11T09:59:00Z") }).where(eq(issueComments.id, f.commentId));
|
|
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" });
|
|
let queuedIds = [f.commentId];
|
|
let expectedIds = queuedIds;
|
|
if (["mixed_authors", "last_delivered"].includes(gate)) {
|
|
const secondId = randomUUID();
|
|
await db.insert(issueComments).values({ id: secondId, companyId: f.companyId, issueId: f.issueId,
|
|
authorType: "user", authorUserId: "second-author", body: "Also preserve the existing work.",
|
|
createdAt: new Date("2026-09-11T09:59:01Z") });
|
|
queuedIds = [f.commentId, secondId];
|
|
expectedIds = gate === "last_delivered" ? [f.commentId] : queuedIds;
|
|
if (gate === "last_delivered") await db.update(heartbeatRuns).set({
|
|
contextSnapshot: { issueId: f.issueId, wakeCommentIds: [secondId] },
|
|
}).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
}
|
|
if (gate === "live_process") await db.update(heartbeatRuns).set({ processPid: process.pid }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
if (gate === "operator_stop") await db.update(heartbeatRuns).set({ status: "cancelled", errorCode: "operator_interrupted" }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
if (gate === "delivered") await db.update(heartbeatRuns).set({ contextSnapshot: { issueId: f.issueId,
|
|
wakeCommentIds: queuedIds } }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
if (gate === "pause") {
|
|
const holdId = randomUUID();
|
|
await db.insert(issueTreeHolds).values({ id: holdId, companyId: f.companyId, rootIssueId: f.issueId, mode: "pause", status: "active" });
|
|
await db.insert(issueTreeHoldMembers).values({ companyId: f.companyId, holdId, issueId: f.issueId, depth: 0, issueTitle: "Deploy", issueStatus: "blocked" });
|
|
}
|
|
const queueId = randomUUID();
|
|
await db.insert(agentWakeupRequests).values({ id: queueId, companyId: f.companyId, agentId: f.agentId,
|
|
source: "automation", status: "deferred_issue_execution", reason: "issue_commented",
|
|
requestedByActorType: "user", requestedByActorId: "board", updatedAt: new Date(0),
|
|
payload: { issueId: f.issueId, commentId: queuedIds.at(-1), executionWait: { reason: "execution_recovery" },
|
|
_paperclipWakeContext: { issueId: f.issueId, wakeReason: "issue_commented", wakeCommentIds: queuedIds, wakeCommentId: queuedIds.at(-1) } },
|
|
});
|
|
const service = heartbeatService(db);
|
|
await service.resumeExecutionWaitComments();
|
|
await service.resumeExecutionWaitComments();
|
|
const [receipt] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, queueId));
|
|
if (!["ready", "mixed_authors", "last_delivered"].includes(gate)) {
|
|
expect(receipt.status).toBe("deferred_issue_execution");
|
|
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).not.toBeNull();
|
|
expect(await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId), eq(heartbeatRuns.status, "queued")))).toHaveLength(0);
|
|
return;
|
|
}
|
|
expect(receipt.status).toBe("coalesced");
|
|
const [successor] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, receipt.runId!));
|
|
expect(successor.contextSnapshot).toMatchObject({ wakeCommentIds: expectedIds,
|
|
explicitUserContinuation: { previousRunId: f.sourceRunId, commentId: expectedIds.at(-1) } });
|
|
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toBeNull();
|
|
const [prior] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
expect(prior.status).toBe("failed");
|
|
});
|
|
|
|
it.each(["native", "before_selection"])("offers and admits an exact Retry for native preparation cancelled before provider startup: %s", async phase => {
|
|
const f = await seedCancelledStartup();
|
|
if (phase === "before_selection") {
|
|
await db.delete(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.sourceRunId));
|
|
await db.update(heartbeatRuns).set({ runtimeMode: "legacy", runtimeModeResolvedAt: null, nativeIssueId: null,
|
|
runnerProfileJson: { adapterDispatch: { adapterType: "paperclip_runner" } },
|
|
resultJson: { startupCancellation: { beforeNativeSelection: true }, startupPreparationSettledAt: new Date().toISOString() },
|
|
}).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
}
|
|
const blocker = await getExecutionBlocker(db, f.companyId, f.issueId);
|
|
expect(blocker).toMatchObject({ runId: f.sourceRunId, canRetry: true });
|
|
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" });
|
|
const service = heartbeatService(db);
|
|
const successor = await service.wakeup(f.agentId, { source: "on_demand", triggerDetail: "manual", reason: "retry_failed_run",
|
|
failedRunId: f.sourceRunId, requestedByActorType: "user", requestedByActorId: "board", payload: { issueId: f.issueId } });
|
|
expect(successor).toMatchObject({ status: "queued", retryOfRunId: f.sourceRunId });
|
|
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toBeNull();
|
|
});
|
|
it.each(["failed", "succeeded_result", "rejected_result", "still_committing", "controller", "successor", "live_process"])(
|
|
"requires a fully committed failed result and verified stop for a new turn (%s)", async scenario => {
|
|
const f = await seed(), contractId = randomUUID(), resultId = randomUUID();
|
|
await db.insert(completionContracts).values({ id: contractId, companyId: f.companyId, issueId: f.issueId,
|
|
revision: 1, schemaVersion: "paperclip.completion-contract.v1", policyVersion: "test",
|
|
risk: "standard", completionAuthority: "server_arbiter", incompleteCriteriaPolicy: "preserve_non_terminal",
|
|
contractJson: {}, canonicalSha256: contractId, createdByActorType: "system", createdByActorId: "test" });
|
|
await db.update(heartbeatRuns).set({ completionContractId: contractId,
|
|
...(scenario === "live_process" ? { processPid: process.pid } : {}),
|
|
}).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
await db.insert(nativeRunResults).values({ id: resultId, companyId: f.companyId, issueId: f.issueId,
|
|
runId: f.sourceRunId, completionContractId: contractId, serverFingerprint: resultId,
|
|
schemaStatus: scenario === "rejected_result" ? "rejected" : "accepted", canonicalSha256: resultId,
|
|
resultJson: { terminal: { runTerminalState: scenario === "succeeded_result" ? "succeeded" : "failed" } } });
|
|
await db.update(nativeRunFinalizations).set({ resultId,
|
|
phase: scenario === "still_committing" ? "result_accepted" : "committed",
|
|
leaseOwner: scenario === "controller" ? "active-controller" : null,
|
|
failureDetail: scenario === "successor" ? { successorRunId: randomUUID() } : null,
|
|
}).where(eq(nativeRunFinalizations.runId, f.sourceRunId));
|
|
const result = await admit(f);
|
|
if (scenario === "failed") {
|
|
expect(result).toEqual({ previousRunId: f.sourceRunId, commentId: f.commentId });
|
|
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toBeNull();
|
|
const [saved] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.sourceRunId));
|
|
expect(saved).toMatchObject({ phase: "committed", resultId });
|
|
} else {
|
|
expect(result).toBeNull();
|
|
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).not.toBeNull();
|
|
}
|
|
});
|
|
async function seedResultBeforeTerminalFailure(f: Fixture) {
|
|
const contractId = randomUUID(), resultId = randomUUID();
|
|
await db.insert(completionContracts).values({ id: contractId, companyId: f.companyId, issueId: f.issueId,
|
|
revision: 1, schemaVersion: "paperclip.completion-contract.v1", policyVersion: "test",
|
|
risk: "standard", completionAuthority: "server_arbiter", incompleteCriteriaPolicy: "preserve_non_terminal",
|
|
contractJson: {}, canonicalSha256: contractId, createdByActorType: "system", createdByActorId: "test" });
|
|
await db.update(heartbeatRuns).set({ completionContractId: contractId, errorCode: "provider_transport_failed" })
|
|
.where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
// Dickens returned a result before checkpoint/suspension failed. Accepting
|
|
// that result did not commit its disposition or make the failed run live.
|
|
await db.insert(nativeRunResults).values({ id: resultId, companyId: f.companyId, issueId: f.issueId,
|
|
runId: f.sourceRunId, completionContractId: contractId, serverFingerprint: resultId,
|
|
schemaStatus: "accepted", canonicalSha256: resultId,
|
|
resultJson: { terminal: { runTerminalState: "succeeded", reportedWorkDisposition: "blocked" } } });
|
|
await db.update(nativeRunFinalizations).set({ resultId, failureCode: "native_session_retry_exhausted" })
|
|
.where(eq(nativeRunFinalizations.runId, f.sourceRunId));
|
|
await db.update(issueRecoveryActions).set({ status: "active", cause: "native_session_retry_exhausted",
|
|
evidence: { runId: f.sourceRunId, coordinatorAttempt: 3, sourceFailureCode: "provider_transport_failed" },
|
|
}).where(eq(issueRecoveryActions.sourceIssueId, f.issueId));
|
|
return resultId;
|
|
}
|
|
it.each(["stopped", "controller", "successor", "still_committing", "live_process", "identity_missing"])(
|
|
"continues after terminal failure with a retained result only when execution stopped (%s)", async scenario => {
|
|
const f = await seed(), resultId = await seedResultBeforeTerminalFailure(f);
|
|
await db.update(nativeRunFinalizations).set({
|
|
phase: scenario === "still_committing" ? "result_accepted" : "terminal_failure",
|
|
leaseOwner: scenario === "controller" ? "active-controller" : null,
|
|
failureDetail: scenario === "successor" ? { successorRunId: randomUUID() } : null,
|
|
}).where(eq(nativeRunFinalizations.runId, f.sourceRunId));
|
|
if (scenario === "live_process" || scenario === "identity_missing") {
|
|
await db.update(heartbeatRuns).set({ processPid: scenario === "live_process" ? process.pid : null })
|
|
.where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
}
|
|
const result = await admit(f);
|
|
expect(result).toEqual(scenario === "stopped" ? { previousRunId: f.sourceRunId, commentId: f.commentId } : null);
|
|
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toEqual(scenario === "stopped" ? null : expect.anything());
|
|
const [coordinator] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.sourceRunId));
|
|
expect(coordinator).toMatchObject({ resultId, attempt: 3,
|
|
phase: scenario === "still_committing" ? "result_accepted" : "terminal_failure" });
|
|
const [source] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
expect(source).toMatchObject({ status: "failed", errorCode: "provider_transport_failed" });
|
|
const [savedResult] = await db.select().from(nativeRunResults).where(eq(nativeRunResults.id, resultId));
|
|
expect(savedResult.resultJson).toEqual({ terminal: { runTerminalState: "succeeded", reportedWorkDisposition: "blocked" } });
|
|
});
|
|
it.each(["task", "chat", "chat_with_next_message"])("delivers a queued %s message exactly once after a retained-result failure and delayed remote stop", async kind => {
|
|
const f = await seed(), resultId = await seedResultBeforeTerminalFailure(f);
|
|
const settings = instanceSettingsService(db);
|
|
const experimental = await settings.getExperimental();
|
|
await settings.updateExperimental({ enableAgentChat: true });
|
|
try {
|
|
if (kind !== "task") await db.update(issues).set({ conversationAgentId: f.agentId,
|
|
conversationUserId: "board", conversationState: "active",
|
|
}).where(eq(issues.id, f.issueId));
|
|
// Occupy the agent slot so admission is real without launching a provider.
|
|
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" });
|
|
const lease = { id: randomUUID(), companyId: f.companyId, heartbeatRunId: f.sourceRunId,
|
|
provider: "daytona", providerLeaseId: "dickens-stopped-sandbox" };
|
|
await db.insert(environmentLeases).values({ ...lease, status: "active", leasePolicy: "ephemeral" });
|
|
const heartbeat = heartbeatService(db);
|
|
const wake = await heartbeat.wakeup(f.agentId, {
|
|
source: "on_demand", triggerDetail: "manual", reason: "issue_commented",
|
|
requestedByActorType: "user", requestedByActorId: "board",
|
|
payload: { issueId: f.issueId, commentId: f.commentId },
|
|
contextSnapshot: { source: "issue.comment", issueId: f.issueId, wakeCommentId: f.commentId },
|
|
});
|
|
expect(wake).toBeNull();
|
|
const [waiting] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.companyId));
|
|
expect(waiting).toMatchObject({ status: "deferred_issue_execution", runId: null,
|
|
payload: { executionWait: { reason: "remote_cleanup" } } });
|
|
let nextCommentId: string | null = null;
|
|
let nextWakeId: string | null = null;
|
|
if (kind === "chat_with_next_message") {
|
|
nextCommentId = randomUUID();
|
|
nextWakeId = randomUUID();
|
|
await db.insert(issueComments).values({ id: nextCommentId, companyId: f.companyId, issueId: f.issueId,
|
|
authorType: "user", authorUserId: "board", body: "A separate follow-up." });
|
|
await db.insert(agentWakeupRequests).values({ id: nextWakeId, companyId: f.companyId, agentId: f.agentId,
|
|
source: "on_demand", triggerDetail: "manual", reason: "issue_commented", status: "deferred_issue_execution",
|
|
requestedByActorType: "user", requestedByActorId: "board", payload: {
|
|
issueId: f.issueId, commentId: nextCommentId,
|
|
_paperclipWakeContext: { source: "issue.comment", issueId: f.issueId, wakeCommentId: nextCommentId, wakeReason: "issue_commented" },
|
|
},
|
|
});
|
|
}
|
|
const [source] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
await heartbeat.resumeRemoteStopComments(source);
|
|
expect(await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId), eq(heartbeatRuns.status, "queued")))).toHaveLength(0);
|
|
await db.update(environmentLeases).set({ status: "released", releasedAt: new Date(), cleanupStatus: "success",
|
|
metadata: { remoteExecutionTermination: remoteTerminationReceipt(lease, { providerLeaseId: lease.providerLeaseId, state: "stopped" }) },
|
|
}).where(eq(environmentLeases.id, lease.id));
|
|
// Cleanup and periodic recovery can race to dispatch the same receipt.
|
|
await Promise.all([heartbeat.resumeRemoteStopComments(source), heartbeat.resumeRemoteStopComments(source)]);
|
|
const successors = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId), eq(heartbeatRuns.status, "queued")));
|
|
expect(successors).toHaveLength(1);
|
|
expect(successors[0].contextSnapshot).toMatchObject({ forceFreshSession: true, previousRunId: f.sourceRunId, wakeCommentId: f.commentId });
|
|
if (nextWakeId) {
|
|
expect(successors[0].contextSnapshot?.wakeCommentIds).not.toContain(nextCommentId);
|
|
const [nextWake] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, nextWakeId));
|
|
expect(nextWake.status).toBe("deferred_issue_execution");
|
|
}
|
|
// Complete the admitted turn and run the same transactional queue drain
|
|
// used by finalization. A stale receipt must not create another turn.
|
|
await db.update(heartbeatRuns).set({ status: "succeeded", startedAt: new Date(), finishedAt: new Date() })
|
|
.where(eq(heartbeatRuns.id, successors[0].id));
|
|
await db.update(issues).set({ executionRunId: successors[0].id, status: "in_progress" })
|
|
.where(eq(issues.id, f.issueId));
|
|
const release = createReleaseIssueExecution({
|
|
issueLock: createPostgresWakeQueueAdapter(db, {
|
|
resolveResponsibleUserId: async () => "board",
|
|
getRoutineEnv: async () => ({ routineId: null, env: null, responsibleUserId: null }),
|
|
resolveSessionBeforeForWakeup: async () => null,
|
|
}),
|
|
recovery: { escalateStrandedAssignedIssue: async () => {}, escalateStrandedRecoveryIssueInPlace: async () => {} },
|
|
});
|
|
const drained = await release({ companyId: f.companyId, runId: successors[0].id, now: new Date(), suppressImmediateRecovery: true });
|
|
expect(drained.outcome.kind).toBe(nextWakeId ? "promoted" : "released");
|
|
await heartbeat.resumeRemoteStopComments(source);
|
|
await release({ companyId: f.companyId, runId: successors[0].id, now: new Date(), suppressImmediateRecovery: true });
|
|
const afterDrain = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId), eq(heartbeatRuns.status, "queued")));
|
|
expect(afterDrain).toHaveLength(nextWakeId ? 1 : 0);
|
|
if (nextWakeId) {
|
|
expect(afterDrain[0]).toMatchObject({ wakeupRequestId: nextWakeId, contextSnapshot: { wakeCommentId: nextCommentId } });
|
|
expect(afterDrain[0].contextSnapshot?.wakeCommentIds ?? []).not.toContain(f.commentId);
|
|
}
|
|
const [delivered] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, waiting.id));
|
|
expect(delivered).toMatchObject({ status: "coalesced", runId: successors[0].id });
|
|
const [coordinator] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.sourceRunId));
|
|
expect(coordinator).toMatchObject({ phase: "terminal_failure", resultId, attempt: 3 });
|
|
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toBeNull();
|
|
} finally {
|
|
await settings.updateExperimental({ enableAgentChat: experimental.enableAgentChat });
|
|
}
|
|
});
|
|
async function recoveredCopybackFixture(origin: "message" | "queued_message" | "task_retry") {
|
|
const f = await seed(), resultId = await seedResultBeforeTerminalFailure(f);
|
|
const requestId = randomUUID(), operationId = randomUUID();
|
|
const [action] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issueId));
|
|
const lease = { id: randomUUID(), companyId: f.companyId, heartbeatRunId: f.sourceRunId,
|
|
provider: "daytona", providerLeaseId: randomUUID() };
|
|
const reference = { schema: "paperclip.native-workspace-sync/v3", state: "finalized", workspaceId: randomUUID(),
|
|
leaseId: lease.id, providerLeaseId: lease.providerLeaseId, remoteCwd: "/workspaces/task",
|
|
descriptorSha256: "12".repeat(32), baselineSha256: "34".repeat(32), finalHostSha256: "56".repeat(32),
|
|
resourceDisposition: "destroy" };
|
|
const recovery = { ...createSystemRecovery(f.sourceRunId, origin === "task_retry" ? requestId : null),
|
|
state: "complete" as const, preservationOperationIds: [operationId] };
|
|
await db.update(issueRecoveryActions).set({ status: "cancelled", cause: "native_workspace_sync_out_retry_exhausted",
|
|
evidence: { runId: f.sourceRunId, workspaceRecovery: { schema: "paperclip.workspace-recovery.v1",
|
|
runId: f.sourceRunId, status: "recovered", stage: "saving_receipts", attempt: 1, requestIds: [] },
|
|
systemRecovery: recovery, automaticRecovery: { replay: "blocked", actionOutcome: "unknown" } },
|
|
}).where(eq(issueRecoveryActions.id, action.id));
|
|
await db.update(heartbeatRuns).set({ processPid: null, createdAt: new Date("2020-01-01T09:00:00Z"),
|
|
finishedAt: new Date("2020-01-01T10:00:00Z"), runnerProfileJson: { nativeWorkspaceSync: reference } })
|
|
.where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
await db.update(issueComments).set({ createdAt: new Date("2020-01-01T13:00:00Z") }).where(eq(issueComments.id, f.commentId));
|
|
await db.insert(environmentLeases).values({ ...lease, status: "expired", leasePolicy: "ephemeral",
|
|
releasedAt: new Date(), cleanupStatus: "success", metadata: {
|
|
remoteExecutionTermination: remoteTerminationReceipt(lease, { providerLeaseId: lease.providerLeaseId, state: "destroyed" }),
|
|
},
|
|
});
|
|
await db.insert(workspaceOperations).values({ id: operationId, companyId: f.companyId, issueId: f.issueId,
|
|
heartbeatRunId: f.sourceRunId, phase: "workspace_recovery", status: "succeeded", metadata: {
|
|
recoveryActionId: action.id, leaseId: lease.id, providerLeaseId: lease.providerLeaseId,
|
|
descriptorSha256: reference.descriptorSha256, finalHostSha256: reference.finalHostSha256,
|
|
},
|
|
});
|
|
if (origin !== "message") await db.insert(agentWakeupRequests).values({ id: requestId, companyId: f.companyId, agentId: f.agentId,
|
|
source: "on_demand", triggerDetail: "manual", status: "deferred_issue_execution",
|
|
reason: origin === "task_retry" ? "task_retry" : "issue_commented", requestedByActorType: "user", requestedByActorId: f.actorId,
|
|
payload: { issueId: f.issueId, ...(origin === "task_retry" ? { taskRetry: { schema: "paperclip.task-retry.v1",
|
|
generationId: requestId, sourceRunId: f.sourceRunId, actorId: f.actorId, actorSource: "session", strategy: "evaluate" } }
|
|
: { commentId: f.commentId, executionWait: { reason: "finalizing_saved_result" },
|
|
_paperclipWakeContext: { source: "issue.comment", issueId: f.issueId, wakeReason: "issue_commented",
|
|
wakeCommentId: f.commentId, wakeCommentIds: [f.commentId] } }) },
|
|
});
|
|
return { ...f, resultId, action, lease, reference, operationId, requestId, recovery,
|
|
...(origin === "task_retry" ? { taskRetryRequestId: requestId }
|
|
: origin === "queued_message" ? { queuedCommentRequestId: requestId } : {}) };
|
|
}
|
|
async function committedQueuedSendFixture() {
|
|
const f = await recoveredCopybackFixture("queued_message");
|
|
await db.update(nativeRunFinalizations).set({ phase: "committed", leaseOwner: null })
|
|
.where(eq(nativeRunFinalizations.runId, f.sourceRunId));
|
|
await db.update(heartbeatRuns).set({ status: "succeeded" }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
await db.update(issues).set({ status: "done", completedAt: new Date("2020-01-01T10:00:00Z") }).where(eq(issues.id, f.issueId));
|
|
await db.update(issueRecoveryActions).set({ status: "resolved", outcome: "recovered" })
|
|
.where(eq(issueRecoveryActions.id, f.action.id));
|
|
await db.update(issueComments).set({ body: "Show the unread count as a number",
|
|
createdAt: new Date("2020-01-01T09:30:00Z") }).where(eq(issueComments.id, f.commentId));
|
|
const [waiting] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, f.requestId));
|
|
await db.update(agentWakeupRequests).set({ payload: { ...waiting.payload,
|
|
queuedCommentInterrupt: { actorId: f.actorId, requestedAt: new Date().toISOString() },
|
|
} }).where(eq(agentWakeupRequests.id, f.requestId));
|
|
return f;
|
|
}
|
|
it("sends a saved follow-up once after its native source commits and the task is done", async () => {
|
|
const f = await committedQueuedSendFixture();
|
|
const [sourceBefore] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
const [coordinatorBefore] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.sourceRunId));
|
|
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" });
|
|
const heartbeat = heartbeatService(db);
|
|
await heartbeat.resumeQueuedCommentInterrupt(f.companyId, f.requestId);
|
|
await heartbeat.resumeQueuedCommentInterrupt(f.companyId, f.requestId);
|
|
const [receipt] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, f.requestId));
|
|
expect(receipt).toMatchObject({ status: "coalesced", runId: expect.any(String) });
|
|
const successors = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId),
|
|
eq(heartbeatRuns.issueId, f.issueId), sql`${heartbeatRuns.id} <> ${f.sourceRunId}`));
|
|
expect(successors).toHaveLength(1);
|
|
expect(successors[0]).toMatchObject({ id: receipt.runId, status: "queued",
|
|
contextSnapshot: { wakeCommentIds: [f.commentId] } });
|
|
const envelope = await buildExecutionContinuation({ db, companyId: f.companyId, issueId: f.issueId,
|
|
agentId: f.agentId, runId: successors[0].id, context: successors[0].contextSnapshot!,
|
|
summary: null, exposeLowTrustRaw: false });
|
|
expect(envelope.messages).toEqual(expect.arrayContaining([expect.objectContaining({ id: f.commentId,
|
|
body: "Show the unread count as a number" })]));
|
|
expect((await db.select().from(issues).where(eq(issues.id, f.issueId)))[0]).toMatchObject({ status: "todo" });
|
|
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.sourceRunId)))[0]).toEqual(sourceBefore);
|
|
expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.sourceRunId)))[0])
|
|
.toEqual(coordinatorBefore);
|
|
});
|
|
it.each(["failed", "cancelled"] as const)("sends a saved follow-up once after a %s native source and a task marked done", async status => {
|
|
const f = await committedQueuedSendFixture();
|
|
await db.update(heartbeatRuns).set({ status }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
await db.update(nativeRunFinalizations).set({ phase: "terminal_failure" })
|
|
.where(eq(nativeRunFinalizations.runId, f.sourceRunId));
|
|
const [sourceBefore] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" });
|
|
const heartbeat = heartbeatService(db);
|
|
await heartbeat.resumeQueuedCommentInterrupt(f.companyId, f.requestId);
|
|
await heartbeat.resumeQueuedCommentInterrupt(f.companyId, f.requestId);
|
|
const [receipt] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, f.requestId));
|
|
expect(receipt).toMatchObject({ status: "coalesced", runId: expect.any(String) });
|
|
const successors = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId),
|
|
eq(heartbeatRuns.issueId, f.issueId), sql`${heartbeatRuns.id} <> ${f.sourceRunId}`));
|
|
expect(successors).toHaveLength(1);
|
|
const envelope = await buildExecutionContinuation({ db, companyId: f.companyId, issueId: f.issueId,
|
|
agentId: f.agentId, runId: successors[0].id, context: successors[0].contextSnapshot!,
|
|
summary: null, exposeLowTrustRaw: false });
|
|
expect(envelope.messages).toEqual(expect.arrayContaining([expect.objectContaining({ id: f.commentId,
|
|
body: "Show the unread count as a number" })]));
|
|
expect((await db.select().from(issues).where(eq(issues.id, f.issueId)))[0]).toMatchObject({ status: "todo" });
|
|
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.sourceRunId)))[0]).toEqual(sourceBefore);
|
|
});
|
|
it("blocks explicit Send on a done task before dispatch when its isolated worktree has no source", async () => {
|
|
const f = await committedQueuedSendFixture();
|
|
const settings = instanceSettingsService(db), experimental = await settings.getExperimental();
|
|
await settings.updateExperimental({ enableIsolatedWorkspaces: true });
|
|
try {
|
|
await db.update(issues).set({ executionWorkspaceSettings: { mode: "isolated_workspace",
|
|
workspaceStrategy: { type: "git_worktree" } }, projectId: null, projectWorkspaceId: null,
|
|
executionWorkspaceId: null }).where(eq(issues.id, f.issueId));
|
|
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" });
|
|
await heartbeatService(db).resumeQueuedCommentInterrupt(f.companyId, f.requestId);
|
|
expect(await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId),
|
|
eq(heartbeatRuns.issueId, f.issueId), sql`${heartbeatRuns.id} <> ${f.sourceRunId}`))).toHaveLength(0);
|
|
expect((await db.select().from(issues).where(eq(issues.id, f.issueId)))[0])
|
|
.toMatchObject({ status: "blocked", completedAt: null, executionState: null });
|
|
expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, f.requestId)))[0])
|
|
.toMatchObject({ status: "deferred_issue_execution", runId: null });
|
|
expect(await db.select().from(agentWakeupRequests).where(and(eq(agentWakeupRequests.companyId, f.companyId),
|
|
eq(agentWakeupRequests.reason, "workspace_worktree_requires_project"))))
|
|
.toEqual([expect.objectContaining({ status: "skipped" })]);
|
|
expect(await db.select().from(issueComments).where(eq(issueComments.issueId, f.issueId)))
|
|
.toEqual(expect.arrayContaining([expect.objectContaining({ body: expect.stringContaining("workspace_worktree_requires_project") })]));
|
|
} finally { await settings.updateExperimental(experimental); }
|
|
});
|
|
|
|
it.each(["pending lease", "failed lease", "workspace owner", "coordinator owner", "uncommitted result", "missing result"] as const)(
|
|
"keeps explicit Send on a done task pending until its committed successful predecessor settles: %s", async fault => {
|
|
const f = await committedQueuedSendFixture();
|
|
const [lease] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id));
|
|
const [source] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
const [coordinator] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.sourceRunId));
|
|
if (fault === "pending lease") await db.update(environmentLeases).set({ status: "pending_cleanup", releasedAt: null,
|
|
cleanupStatus: null, metadata: { ...lease.metadata, pendingCleanupInFlight: true } }).where(eq(environmentLeases.id, lease.id));
|
|
if (fault === "failed lease") await db.update(environmentLeases).set({ cleanupStatus: "failed" }).where(eq(environmentLeases.id, lease.id));
|
|
if (fault === "workspace owner") await db.update(heartbeatRuns).set({ runnerProfileJson: {
|
|
...source.runnerProfileJson, nativeWorkspaceFinalizationOwner: { token: randomUUID() },
|
|
} }).where(eq(heartbeatRuns.id, source.id));
|
|
if (fault === "coordinator owner") await db.update(nativeRunFinalizations).set({ leaseOwner: "live-controller" })
|
|
.where(eq(nativeRunFinalizations.runId, f.sourceRunId));
|
|
if (fault === "uncommitted result") await db.update(nativeRunFinalizations).set({ phase: "result_accepted" })
|
|
.where(eq(nativeRunFinalizations.runId, f.sourceRunId));
|
|
if (fault === "missing result") await db.update(nativeRunFinalizations).set({ resultId: null })
|
|
.where(eq(nativeRunFinalizations.runId, f.sourceRunId));
|
|
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" });
|
|
const heartbeat = heartbeatService(db);
|
|
await heartbeat.resumeQueuedCommentInterrupt(f.companyId, f.requestId);
|
|
expect(await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId),
|
|
eq(heartbeatRuns.issueId, f.issueId), sql`${heartbeatRuns.id} <> ${f.sourceRunId}`))).toHaveLength(0);
|
|
expect((await db.select().from(issues).where(eq(issues.id, f.issueId)))[0].status).toBe("done");
|
|
expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, f.requestId)))[0])
|
|
.toMatchObject({ status: "deferred_issue_execution", runId: null });
|
|
await db.update(environmentLeases).set({ status: lease.status, releasedAt: lease.releasedAt,
|
|
cleanupStatus: lease.cleanupStatus, metadata: lease.metadata }).where(eq(environmentLeases.id, lease.id));
|
|
await db.update(heartbeatRuns).set({ runnerProfileJson: source.runnerProfileJson }).where(eq(heartbeatRuns.id, source.id));
|
|
await db.update(nativeRunFinalizations).set({ phase: coordinator.phase, resultId: coordinator.resultId,
|
|
leaseOwner: coordinator.leaseOwner }).where(eq(nativeRunFinalizations.runId, f.sourceRunId));
|
|
await heartbeat.resumeQueuedCommentInterrupt(f.companyId, f.requestId);
|
|
await heartbeat.resumeQueuedCommentInterrupt(f.companyId, f.requestId);
|
|
expect(await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId),
|
|
eq(heartbeatRuns.issueId, f.issueId), sql`${heartbeatRuns.id} <> ${f.sourceRunId}`))).toHaveLength(1);
|
|
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.sourceRunId)))[0]).toEqual(source);
|
|
expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.sourceRunId)))[0]).toEqual(coordinator);
|
|
});
|
|
|
|
it.each(["approval", "question"] as const)("keeps explicit Send on a done task behind its pending %s", async gate => {
|
|
const f = await committedQueuedSendFixture();
|
|
if (gate === "question") await db.insert(issueThreadInteractions).values({ companyId: f.companyId,
|
|
issueId: f.issueId, kind: "ask_user_questions", status: "pending", payload: { version: 1, questions: [] } });
|
|
else {
|
|
const approvalId = randomUUID();
|
|
await db.insert(approvals).values({ id: approvalId, companyId: f.companyId, type: "hire_agent", status: "pending", payload: {} });
|
|
await db.insert(issueApprovals).values({ companyId: f.companyId, issueId: f.issueId, approvalId });
|
|
}
|
|
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" });
|
|
await heartbeatService(db).resumeQueuedCommentInterrupt(f.companyId, f.requestId);
|
|
expect(await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId),
|
|
eq(heartbeatRuns.issueId, f.issueId), sql`${heartbeatRuns.id} <> ${f.sourceRunId}`))).toHaveLength(0);
|
|
expect((await db.select().from(issues).where(eq(issues.id, f.issueId)))[0].status).toBe("done");
|
|
expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, f.requestId)))[0])
|
|
.toMatchObject({ status: "deferred_issue_execution", runId: null });
|
|
});
|
|
|
|
// A Retry asks for the accepted result, which now has its saved files. It is
|
|
// applied by preserved-result finalization, never replaced by a new turn.
|
|
it("holds a Retry of a terminal retained result after verified copy-back until its accepted result is applied", async () => {
|
|
const f = await recoveredCopybackFixture("task_retry");
|
|
const [before] = await db.select().from(nativeRunResults).where(eq(nativeRunResults.id, f.resultId));
|
|
const waits: string[] = [];
|
|
expect(await db.transaction(async tx => {
|
|
await tx.select().from(issues).where(eq(issues.id, f.issueId)).for("update");
|
|
return admitExplicitNativeContinuation({ ...f, db: tx as unknown as typeof db, onBlocked: reason => { waits.push(reason); } });
|
|
})).toBeNull();
|
|
expect(waits).toEqual(["finalizing_saved_result"]);
|
|
expect(await resumePreservedResultFinalization(db, f.companyId, f.sourceRunId)).toBe(true);
|
|
expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, f.requestId)))[0])
|
|
.toMatchObject({ status: "coalesced", runId: f.sourceRunId, payload: { taskRetry: { strategy: "finalize_result" } } });
|
|
expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.sourceRunId)))[0])
|
|
.toMatchObject({ phase: "workspace_finalizing", resultId: f.resultId });
|
|
expect((await db.select().from(nativeRunResults).where(eq(nativeRunResults.id, f.resultId)))[0]).toEqual(before);
|
|
expect(await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId),
|
|
sql`${heartbeatRuns.id} <> ${f.sourceRunId}`))).toEqual([]);
|
|
});
|
|
|
|
it.each(["message", "queued_message"] as const)("continues a terminal retained result after verified copy-back: %s", async origin => {
|
|
const f = await recoveredCopybackFixture(origin);
|
|
const [before] = await db.select().from(nativeRunResults).where(eq(nativeRunResults.id, f.resultId));
|
|
const [source] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
expect(await verifiedWorkspaceRepairReceipt(db, source)).toBe(f.operationId);
|
|
const result = await admit(f);
|
|
expect(result).toMatchObject({ previousRunId: f.sourceRunId, commentId: f.commentId });
|
|
expect((await db.select().from(nativeRunResults).where(eq(nativeRunResults.id, f.resultId)))[0]).toEqual(before);
|
|
const [action] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.action.id));
|
|
expect(action.evidence.automaticRecovery).toMatchObject({ actionOutcome: "unknown", replay: "explicit_user_continuation" });
|
|
expect(action.evidence.systemRecovery).toEqual(f.recovery);
|
|
expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.sourceRunId)))[0])
|
|
.toMatchObject({ phase: "terminal_failure", resultId: f.resultId });
|
|
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toBeNull();
|
|
});
|
|
|
|
it.each(["missing", "descriptor", "files", "lease", "provider", "source", "issue", "unfinished", "unrecovered",
|
|
"committing", "controller", "remote_stop", "active_run"] as const)("keeps retained-result continuation held without exact settled copy-back: %s", async fault => {
|
|
const f = await recoveredCopybackFixture("queued_message");
|
|
const [before] = await db.select().from(nativeRunResults).where(eq(nativeRunResults.id, f.resultId));
|
|
const [operation] = await db.select().from(workspaceOperations).where(eq(workspaceOperations.id, f.operationId));
|
|
if (fault === "missing") await db.delete(workspaceOperations).where(eq(workspaceOperations.id, f.operationId));
|
|
const changed = fault === "descriptor" ? { descriptorSha256: "78".repeat(32) }
|
|
: fault === "files" ? { finalHostSha256: "78".repeat(32) }
|
|
: fault === "lease" ? { leaseId: randomUUID() }
|
|
: fault === "provider" ? { providerLeaseId: randomUUID() } : null;
|
|
if (changed) await db.update(workspaceOperations).set({ metadata: { ...operation.metadata, ...changed } })
|
|
.where(eq(workspaceOperations.id, f.operationId));
|
|
if (fault === "source") {
|
|
const [other] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "failed" }).returning();
|
|
await db.update(workspaceOperations).set({ heartbeatRunId: other.id }).where(eq(workspaceOperations.id, f.operationId));
|
|
}
|
|
if (fault === "issue") {
|
|
const [other] = await db.insert(issues).values({ companyId: f.companyId, title: "Other task" }).returning();
|
|
await db.update(workspaceOperations).set({ issueId: other.id }).where(eq(workspaceOperations.id, f.operationId));
|
|
}
|
|
if (fault === "unfinished") await db.update(workspaceOperations).set({ status: "running" }).where(eq(workspaceOperations.id, f.operationId));
|
|
if (fault === "unrecovered") await db.update(issueRecoveryActions).set({ evidence: { ...f.action.evidence,
|
|
runId: f.sourceRunId, systemRecovery: f.recovery, workspaceRecovery: { schema: "paperclip.workspace-recovery.v1",
|
|
runId: f.sourceRunId, status: "blocked", stage: "restoring_files", attempt: 1, requestIds: [] },
|
|
} }).where(eq(issueRecoveryActions.id, f.action.id));
|
|
if (fault === "committing") await db.update(nativeRunFinalizations).set({ phase: "result_accepted" })
|
|
.where(eq(nativeRunFinalizations.runId, f.sourceRunId));
|
|
if (fault === "controller") await db.update(nativeRunFinalizations).set({ leaseOwner: "live-controller" })
|
|
.where(eq(nativeRunFinalizations.runId, f.sourceRunId));
|
|
if (fault === "remote_stop") await db.update(environmentLeases).set({ metadata: {} }).where(eq(environmentLeases.id, f.lease.id));
|
|
if (fault === "active_run") await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId,
|
|
nativeIssueId: f.issueId, runtimeMode: "native", status: "running" });
|
|
expect(await admit(f)).toBeNull();
|
|
expect((await db.select().from(nativeRunResults).where(eq(nativeRunResults.id, f.resultId)))[0]).toEqual(before);
|
|
expect((await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.action.id)))[0].evidence)
|
|
.not.toHaveProperty("explicitUserContinuation");
|
|
});
|
|
|
|
it("delivers the saved message once after terminal result copy-back, without replaying its accepted result", async () => {
|
|
const f = await recoveredCopybackFixture("queued_message");
|
|
const failedRunId = randomUUID();
|
|
await db.insert(heartbeatRuns).values({ id: failedRunId, companyId: f.companyId, agentId: f.agentId,
|
|
nativeIssueId: f.issueId, runtimeMode: "native", status: "failed", errorCode: "native_provider_terminal_failed",
|
|
createdAt: new Date("2020-01-01T11:00:00Z"), finishedAt: new Date("2020-01-01T12:00:00Z"), contextSnapshot: { issueId: f.issueId } });
|
|
await db.insert(nativeRunFinalizations).values({ companyId: f.companyId, issueId: f.issueId, runId: failedRunId,
|
|
phase: "terminal_failure" });
|
|
await db.insert(issueRecoveryActions).values({ companyId: f.companyId, sourceIssueId: f.issueId,
|
|
kind: "active_run_watchdog", cause: "native_continuation_requires_reconciliation", fingerprint: failedRunId,
|
|
status: "resolved", outcome: "blocked", nextAction: "Continue from recorded work without replay.",
|
|
evidence: { runId: failedRunId, automaticRecovery: { replay: "blocked", actionOutcome: "unknown" } } });
|
|
const lease = { id: randomUUID(), companyId: f.companyId, heartbeatRunId: failedRunId,
|
|
provider: "daytona", providerLeaseId: randomUUID() };
|
|
await db.insert(environmentLeases).values({ ...lease, status: "expired", leasePolicy: "ephemeral",
|
|
releasedAt: new Date(), cleanupStatus: "success", metadata: {
|
|
remoteExecutionTermination: remoteTerminationReceipt(lease, { providerLeaseId: lease.providerLeaseId, state: "destroyed" }),
|
|
} });
|
|
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" });
|
|
await db.update(agentWakeupRequests).set({ updatedAt: new Date(0) }).where(eq(agentWakeupRequests.id, f.requestId));
|
|
const heartbeat = heartbeatService(db);
|
|
await Promise.all([heartbeat.resumeExecutionWaitComments(), heartbeat.resumeExecutionWaitComments()]);
|
|
const [delivered] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, f.requestId));
|
|
expect(delivered).toMatchObject({ status: "coalesced", runId: expect.any(String) });
|
|
const successors = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId), eq(heartbeatRuns.status, "queued")));
|
|
expect(successors).toHaveLength(1);
|
|
expect(successors[0]).toMatchObject({ id: delivered.runId, contextSnapshot: {
|
|
previousRunId: failedRunId, forceFreshSession: true, wakeCommentIds: [f.commentId],
|
|
} });
|
|
await heartbeat.resumeExecutionWaitComments();
|
|
expect(await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId), eq(heartbeatRuns.status, "queued"))))
|
|
.toHaveLength(1);
|
|
expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.sourceRunId)))[0])
|
|
.toMatchObject({ phase: "terminal_failure", resultId: f.resultId });
|
|
});
|
|
|
|
it("defers an ordinary follow-up while a recovered export's successful result is still finalizing", async () => {
|
|
const f = await seed(), resultId = await seedResultBeforeTerminalFailure(f);
|
|
await db.update(nativeRunFinalizations).set({ phase: "result_accepted", failureCode: null, failureDetail: null })
|
|
.where(eq(nativeRunFinalizations.runId, f.sourceRunId));
|
|
await db.update(issueRecoveryActions).set({ status: "resolved", outcome: "recovered", cause: "native_workspace_sync_out_retry_exhausted",
|
|
evidence: { runId: f.sourceRunId, workspaceRecovery: { status: "recovered" }, systemRecovery: createSystemRecovery(f.sourceRunId) },
|
|
}).where(eq(issueRecoveryActions.sourceIssueId, f.issueId));
|
|
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toBeNull();
|
|
const blocked = vi.fn();
|
|
expect(await db.transaction(tx => admitExplicitNativeContinuation({ ...f, db: tx as unknown as typeof db, onBlocked: blocked, dryRun: true }))).toBeNull();
|
|
expect(blocked).toHaveBeenCalledWith("finalizing_saved_result", expect.any(String));
|
|
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" });
|
|
const wake = await heartbeatService(db).wakeup(f.agentId, { source: "on_demand", triggerDetail: "manual", reason: "issue_commented",
|
|
requestedByActorType: "user", requestedByActorId: "board", payload: { issueId: f.issueId, commentId: f.commentId },
|
|
contextSnapshot: { source: "issue.comment", issueId: f.issueId, wakeCommentId: f.commentId },
|
|
});
|
|
expect(wake).toBeNull();
|
|
const successors = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId), eq(heartbeatRuns.nativeIssueId, f.issueId), eq(heartbeatRuns.status, "queued")));
|
|
expect(successors).toHaveLength(0);
|
|
const [waiting] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.companyId));
|
|
expect(waiting).toMatchObject({ status: "deferred_issue_execution", runId: null, payload: { executionWait: { reason: "finalizing_saved_result" } } });
|
|
expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.sourceRunId)))[0]).toMatchObject({ phase: "result_accepted", resultId });
|
|
});
|
|
|
|
it.each(["completed", "after_reset", "after_reset_workspace_hold"])("admits fresh messages after historical execution work is cleared: %s", async scenario => {
|
|
const f = await seed(), settings = instanceSettingsService(db), previous = await settings.getExperimental();
|
|
try {
|
|
if (scenario === "completed") {
|
|
await seedResultBeforeTerminalFailure(f);
|
|
await db.update(nativeRunFinalizations).set({ phase: "committed", leaseOwner: null }).where(eq(nativeRunFinalizations.runId, f.sourceRunId));
|
|
await db.update(heartbeatRuns).set({ status: "succeeded" }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
await db.update(issueRecoveryActions).set({ status: "resolved", outcome: "recovered", cause: "native_workspace_sync_out_retry_exhausted",
|
|
evidence: { runId: f.sourceRunId, workspaceRecovery: { status: "recovered" }, systemRecovery: createSystemRecovery(f.sourceRunId) },
|
|
}).where(eq(issueRecoveryActions.sourceIssueId, f.issueId));
|
|
} else {
|
|
await settings.updateExperimental({ enableAgentChat: true });
|
|
const boundaryId = randomUUID();
|
|
await db.insert(issueComments).values({ id: boundaryId, companyId: f.companyId, issueId: f.issueId, authorType: "user", authorUserId: "board", body: "/new", createdAt: new Date("2026-09-11T10:30:00Z") });
|
|
await db.update(issues).set({ conversationAgentId: f.agentId, conversationUserId: "board", conversationState: "active", conversationBoundaryCommentId: boundaryId }).where(eq(issues.id, f.issueId));
|
|
await db.update(issueRecoveryActions).set({ createdAt: new Date("2026-09-11T09:00:00Z") }).where(eq(issueRecoveryActions.sourceIssueId, f.issueId));
|
|
const [source] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
expect(await db.transaction(tx => recordNativeLocalProcessStop(tx as unknown as typeof db, source))).toBe(true);
|
|
await db.update(heartbeatRuns).set({ processPid: null }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
if (scenario === "after_reset_workspace_hold") {
|
|
await db.update(heartbeatRuns).set({ processPid: 999999999, resultJson: { workspaceRestoreFailure: "restore_unsafe_archive" } }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
await db.update(issueRecoveryActions).set({ evidence: { runId: f.sourceRunId, workspaceRestoreFailure: "restore_unsafe_archive", automaticRecovery: { replay: "blocked" } } }).where(eq(issueRecoveryActions.sourceIssueId, f.issueId));
|
|
}
|
|
}
|
|
const blocker = await getExecutionBlocker(db, f.companyId, f.issueId);
|
|
if (scenario === "completed") expect(blocker).toBeNull(); else expect(blocker).not.toBeNull();
|
|
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" });
|
|
const wake = await heartbeatService(db).wakeup(f.agentId, { source: "on_demand", triggerDetail: "manual", reason: "issue_commented",
|
|
requestedByActorType: "user", requestedByActorId: "board", payload: { issueId: f.issueId, commentId: f.commentId },
|
|
contextSnapshot: { source: "issue.comment", issueId: f.issueId, wakeCommentId: f.commentId },
|
|
});
|
|
if (scenario === "after_reset_workspace_hold") expect(wake).toBeNull();
|
|
else { expect(wake).not.toBeNull(); expect(wake).toMatchObject({ status: "queued" }); expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toBeNull(); }
|
|
} finally { await settings.updateExperimental(previous); }
|
|
});
|
|
|
|
it.each(["live_process", "missing_identity", "quarantined", "verified", "remote_cleanup", "controller"])(
|
|
"keeps native stop and cleanup checks after an applied chat reset: %s", async scenario => {
|
|
const f = await seed(), settings = instanceSettingsService(db), previous = await settings.getExperimental();
|
|
const retire = vi.fn(() => true);
|
|
const verify = vi.spyOn(nativeExecutor, "verifyStoppedNativeSessionForContinuation").mockResolvedValue(
|
|
scenario === "verified" ? { evidence: { runId: f.sourceRunId, schema: "paperclip.stopped_native_conversation.v1" }, retire } : null);
|
|
try {
|
|
await settings.updateExperimental({ enableAgentChat: true });
|
|
const boundaryId = randomUUID();
|
|
await db.insert(issueComments).values({ id: boundaryId, companyId: f.companyId, issueId: f.issueId, authorType: "user", authorUserId: "board", body: "/new", createdAt: new Date("2026-09-11T10:30:00Z") });
|
|
await db.update(issues).set({ conversationAgentId: f.agentId, conversationUserId: "board", conversationState: "active", conversationBoundaryCommentId: boundaryId }).where(eq(issues.id, f.issueId));
|
|
await db.update(issueRecoveryActions).set({ createdAt: new Date("2026-09-11T09:00:00Z") }).where(eq(issueRecoveryActions.sourceIssueId, f.issueId));
|
|
await db.update(heartbeatRuns).set({ createdAt: new Date("2026-09-11T09:00:00Z"),
|
|
processPid: scenario === "live_process" ? process.pid : scenario === "missing_identity" ? null : 999999999,
|
|
errorCode: "native_session_cleanup_quarantined",
|
|
}).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
if (scenario === "remote_cleanup") await db.insert(environmentLeases).values({ companyId: f.companyId, heartbeatRunId: f.sourceRunId, provider: "daytona", providerLeaseId: "old-chat-environment", status: "active", leasePolicy: "ephemeral" });
|
|
if (scenario === "controller") await db.update(nativeRunFinalizations).set({ leaseOwner: "old-controller" }).where(eq(nativeRunFinalizations.runId, f.sourceRunId));
|
|
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" });
|
|
const wake = await heartbeatService(db).wakeup(f.agentId, { source: "on_demand", triggerDetail: "manual", reason: "issue_commented",
|
|
requestedByActorType: "user", requestedByActorId: "board", payload: { issueId: f.issueId, commentId: f.commentId },
|
|
contextSnapshot: { source: "issue.comment", issueId: f.issueId, wakeCommentId: f.commentId },
|
|
});
|
|
if (scenario === "verified") {
|
|
expect(wake).toMatchObject({ status: "queued" });
|
|
expect(retire).toHaveBeenCalledOnce();
|
|
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toBeNull();
|
|
} else {
|
|
expect(wake).toBeNull();
|
|
const [saved] = await db.select().from(agentWakeupRequests).where(and(eq(agentWakeupRequests.companyId, f.companyId), eq(agentWakeupRequests.reason, "issue_commented")));
|
|
expect(saved).toMatchObject({ status: "deferred_issue_execution", runId: null });
|
|
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).not.toBeNull();
|
|
expect(retire).not.toHaveBeenCalled();
|
|
}
|
|
} finally { verify.mockRestore(); await settings.updateExperimental(previous); }
|
|
});
|
|
|
|
it.each(["live", "stopped"])("checks the native predecessor after reset even without a recovery record: %s", async state => {
|
|
const f = await seed(), queueId = randomUUID(), boundaryId = randomUUID();
|
|
await db.delete(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issueId));
|
|
await db.update(heartbeatRuns).set({ createdAt: new Date("2026-09-11T09:00:00Z"), processPid: state === "live" ? process.pid : 999999999 }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
await db.insert(issueComments).values({ id: boundaryId, companyId: f.companyId, issueId: f.issueId, authorType: "user", authorUserId: "board", body: "/new", createdAt: new Date("2026-09-11T10:30:00Z") });
|
|
await db.update(issues).set({ conversationAgentId: f.agentId, conversationUserId: "board", conversationState: "active", conversationBoundaryCommentId: boundaryId }).where(eq(issues.id, f.issueId));
|
|
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, issueId: f.issueId, runtimeMode: "legacy", status: "succeeded", createdAt: new Date("2026-09-11T10:30:01Z"), finishedAt: new Date("2026-09-11T10:30:02Z"), contextSnapshot: { issueId: f.issueId, wakeCommentId: boundaryId } });
|
|
await db.insert(agentWakeupRequests).values({ id: queueId, companyId: f.companyId, agentId: f.agentId, source: "on_demand", reason: "issue_commented", status: "deferred_issue_execution",
|
|
payload: { issueId: f.issueId, _paperclipWakeContext: { wakeCommentIds: [f.commentId] }, queuedCommentInterrupt: { actorId: "board", requestedAt: "2026-09-11T11:30:00Z" } } });
|
|
const blocked: string[] = [];
|
|
const continuation = await db.transaction(tx => admitExplicitNativeContinuation({ ...f, db: tx as unknown as typeof db, queuedCommentInterruptId: queueId, onBlocked: reason => blocked.push(reason) }));
|
|
if (state === "live") { expect(continuation).toBeNull(); expect(blocked).toEqual(["process_running"]); }
|
|
else { expect(continuation).toMatchObject({ previousRunId: f.sourceRunId }); expect(blocked).toEqual([]); }
|
|
});
|
|
|
|
it("dispatches a fresh native Send after a committed failed result without a recovery action", async () => {
|
|
const f = await seed(), resultId = await seedResultBeforeTerminalFailure(f), queueId = randomUUID();
|
|
await db.delete(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issueId));
|
|
await db.update(nativeRunResults).set({ resultJson: { terminal: { runTerminalState: "failed" } } }).where(eq(nativeRunResults.id, resultId));
|
|
await db.update(nativeRunFinalizations).set({ phase: "committed", failureDetail: null, leaseOwner: null }).where(eq(nativeRunFinalizations.runId, f.sourceRunId));
|
|
await db.insert(agentWakeupRequests).values({ id: queueId, companyId: f.companyId, agentId: f.agentId, source: "on_demand", reason: "issue_commented", status: "deferred_issue_execution",
|
|
payload: { issueId: f.issueId, _paperclipWakeContext: { wakeCommentIds: [f.commentId] }, queuedCommentInterrupt: { actorId: "board", requestedAt: "2026-09-11T11:30:00Z" } } });
|
|
const continuation = await db.transaction(tx => admitExplicitNativeContinuation({ ...f, db: tx as unknown as typeof db, queuedCommentInterruptId: queueId }));
|
|
expect(continuation).toMatchObject({ previousRunId: f.sourceRunId });
|
|
const context = { issueId: f.issueId, previousRunId: f.sourceRunId, wakeCommentId: f.commentId, wakeCommentIds: [f.commentId], explicitUserContinuation: continuation };
|
|
await db.insert(heartbeatRuns).values({ id: f.successorRunId, companyId: f.companyId, agentId: f.agentId, runtimeMode: "native", nativeIssueId: f.issueId, status: "queued", contextSnapshot: context });
|
|
await db.update(agentWakeupRequests).set({ status: "coalesced", runId: f.successorRunId }).where(eq(agentWakeupRequests.id, queueId));
|
|
const envelope = await buildExecutionContinuation({ db, companyId: f.companyId, issueId: f.issueId, agentId: f.agentId, runId: f.successorRunId, context, summary: null, exposeLowTrustRaw: false });
|
|
expect(envelope.interruptedRunId).toBe(f.sourceRunId);
|
|
expect(envelope.originCommentIds).toContain(f.commentId);
|
|
});
|
|
|
|
it("keeps exhausted workspace export on its own repair path after the sandbox stops", async () => {
|
|
const f = await seed(), resultId = await seedResultBeforeTerminalFailure(f);
|
|
await db.delete(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issueId));
|
|
await db.update(nativeRunFinalizations).set({ phase: "result_accepted", attempt: 1, failureCode: null, failureDetail: null })
|
|
.where(eq(nativeRunFinalizations.runId, f.sourceRunId));
|
|
await db.update(heartbeatRuns).set({ status: "running", nativePhase: "result_accepted", finishedAt: null })
|
|
.where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
const lease = { id: randomUUID(), companyId: f.companyId, heartbeatRunId: f.sourceRunId,
|
|
provider: "daytona", providerLeaseId: "pending-workspace-export" };
|
|
await db.insert(environmentLeases).values({ ...lease, status: "active", leasePolicy: "ephemeral" });
|
|
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
await db.update(heartbeatRuns).set({ runnerProfileJson: { ...run.runnerProfileJson,
|
|
nativeWorkspaceSync: { schema: "paperclip.native-workspace-sync/v1", state: "prepared",
|
|
descriptorSha256: "a".repeat(64), baselineSha256: "b".repeat(64), finalHostSha256: null,
|
|
workspaceId: randomUUID(), leaseId: lease.id, providerLeaseId: lease.providerLeaseId,
|
|
remoteCwd: "/work", resourceDisposition: "destroy" },
|
|
} }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
for (let attempt = 0; attempt < 3; attempt++) {
|
|
await recordNativeFinalizationFailure({ db, runId: f.sourceRunId,
|
|
error: new Error("native_workspace_sync_out_failed"), failureScope: "workspace", projectRunStatus: true });
|
|
}
|
|
const [retainedLease] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, lease.id));
|
|
expect(retainedLease).toMatchObject({ status: "pending_cleanup", metadata: {
|
|
nativeWorkspaceExportResume: { runId: f.sourceRunId, leaseId: lease.id, resultId },
|
|
} });
|
|
await db.update(environmentLeases).set({ status: "released", releasedAt: new Date(), cleanupStatus: "success",
|
|
metadata: { ...retainedLease.metadata,
|
|
remoteExecutionTermination: remoteTerminationReceipt(lease, { providerLeaseId: lease.providerLeaseId, state: "stopped" }) },
|
|
}).where(eq(environmentLeases.id, lease.id));
|
|
const exportState = () => Promise.all([
|
|
db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.sourceRunId)),
|
|
db.select().from(nativeRunResults).where(eq(nativeRunResults.id, resultId)),
|
|
db.select().from(environmentLeases).where(eq(environmentLeases.id, lease.id)),
|
|
db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issueId)),
|
|
]);
|
|
const before = await exportState();
|
|
// Export repair is intentionally excluded before conversation stop checks.
|
|
// A new message must leave the accepted result, retention, and repair intact.
|
|
await db.update(issueComments).set({ createdAt: new Date(Date.now() + 1000) }).where(eq(issueComments.id, f.commentId));
|
|
expect(await admit(f)).toBeNull();
|
|
expect(await exportState()).toEqual(before);
|
|
const [coordinator] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.sourceRunId));
|
|
expect(coordinator).toMatchObject({ phase: "terminal_failure", resultId, attempt: 1,
|
|
failureCode: "native_workspace_sync_out_retry_exhausted", failureDetail: { workspaceFinalizeAttempt: 3 } });
|
|
const [action] = await db.select().from(issueRecoveryActions).where(and(
|
|
eq(issueRecoveryActions.sourceIssueId, f.issueId), eq(issueRecoveryActions.cause, "native_workspace_sync_out_retry_exhausted"),
|
|
));
|
|
expect(action).toMatchObject({ status: "active", cause: "native_workspace_sync_out_retry_exhausted" });
|
|
});
|
|
it.each(["linked", "revoked", "unlinked", "wrong_author"])(
|
|
"admits a fresh Slack message after failure only with current linked-user authority (%s)", async scenario => {
|
|
const f = await seed();
|
|
// Keep execution queued so this tests real admission without launching a provider.
|
|
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" });
|
|
const authorize = vi.fn(async () => { if (scenario === "revoked") throw new Error("chat_authorization_revoked"); });
|
|
const request = createDurableChatWakeupRequest({ id: randomUUID(), ...f,
|
|
requestedByActorType: scenario === "unlinked" ? "system" : "user",
|
|
requestedByActorId: scenario === "wrong_author" ? "another-user" : "board", requestedAt: new Date(), authorize });
|
|
const wake = () => heartbeatService(db).wakeup(f.agentId, {
|
|
source: "assignment", triggerDetail: "system", reason: "External chat message received",
|
|
requestedByActorType: request.requestedByActorType, requestedByActorId: request.requestedByActorId,
|
|
payload: { issueId: f.issueId, wakeCommentId: f.commentId, mutation: "chat_message_received" },
|
|
contextSnapshot: { issueId: f.issueId, wakeCommentId: f.commentId, source: "chat:slack" },
|
|
durableChatRequest: request,
|
|
});
|
|
if (scenario === "revoked") await expect(wake()).rejects.toThrow("chat_authorization_revoked");
|
|
else await wake();
|
|
expect(authorize).toHaveBeenCalled();
|
|
const successors = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId), eq(heartbeatRuns.status, "queued")));
|
|
expect(successors).toHaveLength(scenario === "linked" ? 1 : 0);
|
|
if (scenario === "linked") {
|
|
expect(successors[0].contextSnapshot).toMatchObject({ forceFreshSession: true, previousRunId: f.sourceRunId });
|
|
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toBeNull();
|
|
} else expect(await getExecutionBlocker(db, f.companyId, f.issueId)).not.toBeNull();
|
|
});
|
|
it.each(["issue_commented", "issue_assigned"].flatMap(reason =>
|
|
["live_process", "missing_process", "lease", "coordinator", "successor", "agent_message", "system_message", "old_comment", "wrong_author", "run_authored", "reassigned", "automatic", "mention", "approval", "question", "malformed_comment", "missing_comment", "foreign_comment", "legacy_owner"].map(kind => ({ reason, kind })),
|
|
))("keeps the hold for $kind ($reason)", async ({ reason, kind }) => {
|
|
const f = await seed();
|
|
f.reason = reason;
|
|
if (kind === "legacy_owner") await db.insert(heartbeatRuns).values({ companyId: f.companyId,
|
|
agentId: f.agentId, status: "failed", runtimeMode: "legacy", processPid: process.pid,
|
|
contextSnapshot: { issueId: f.issueId }, resultJson: { conversationContinuation: "continue_conversation_v1" } });
|
|
if (kind === "live_process") await db.update(heartbeatRuns).set({ processPid: process.pid }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
if (kind === "missing_process") await db.update(heartbeatRuns).set({ processPid: null }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
if (kind === "coordinator") await db.update(nativeRunFinalizations).set({ leaseOwner: "active-controller" }).where(eq(nativeRunFinalizations.runId, f.sourceRunId));
|
|
if (kind === "successor") await db.update(nativeRunFinalizations).set({ failureDetail: { successorRunId: randomUUID() } }).where(eq(nativeRunFinalizations.runId, f.sourceRunId));
|
|
if (kind === "lease") {
|
|
const [environment] = await db.select().from(environments).where(eq(environments.driver, "local"));
|
|
const environmentId = environment.id;
|
|
await db.insert(environmentLeases).values({ companyId: f.companyId, environmentId, heartbeatRunId: f.sourceRunId, issueId: f.issueId, status: "active", leasePolicy: "ephemeral", provider: "local" });
|
|
}
|
|
if (kind === "question") await db.insert(issueThreadInteractions).values({
|
|
companyId: f.companyId, issueId: f.issueId, kind: "ask_user_questions", status: "pending", payload: { version: 1, questions: [] },
|
|
});
|
|
if (kind === "approval") {
|
|
const approvalId = randomUUID();
|
|
await db.insert(approvals).values({ id: approvalId, companyId: f.companyId, type: "hire_agent", status: "pending", payload: {} });
|
|
await db.insert(issueApprovals).values({ companyId: f.companyId, issueId: f.issueId, approvalId });
|
|
}
|
|
if (kind === "malformed_comment") f.commentId = "not-a-uuid";
|
|
if (kind === "missing_comment") f.commentId = randomUUID();
|
|
if (kind === "foreign_comment") f.commentId = (await seed()).commentId;
|
|
if (kind === "agent_message") f.actorType = "agent";
|
|
if (kind === "system_message") f.actorType = "system";
|
|
if (kind === "automatic") f.reason = "issue_continuation_needed";
|
|
if (kind === "mention") f.reason = "issue_comment_mentioned";
|
|
if (kind === "wrong_author") f.actorId = "someone-else";
|
|
if (kind === "run_authored") await db.update(issueComments).set({ createdByRunId: f.sourceRunId }).where(eq(issueComments.id, f.commentId));
|
|
if (kind === "old_comment") await db.update(issueComments).set({ createdAt: new Date("2026-09-11T09:00:00Z") }).where(eq(issueComments.id, f.commentId));
|
|
if (kind === "reassigned") await db.update(issues).set({ assigneeAgentId: null }).where(eq(issues.id, f.issueId));
|
|
expect(await admit(f)).toBeNull();
|
|
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).not.toBeNull();
|
|
});
|
|
it.each(["foreign_source", "missing_authorization", "nonterminal_source"])("rejects unverified interruption context: %s", async kind => {
|
|
const f = await seed();
|
|
let previousRunId: string = f.sourceRunId;
|
|
if (kind === "foreign_source") previousRunId = (await seed()).sourceRunId;
|
|
if (kind === "nonterminal_source") {
|
|
await admit(f);
|
|
await db.update(heartbeatRuns).set({ status: "running" }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
}
|
|
await expect(buildExecutionContinuation({ db, companyId: f.companyId, issueId: f.issueId,
|
|
agentId: f.agentId, context: { previousRunId: f.sourceRunId,
|
|
explicitUserContinuation: { previousRunId, commentId: f.commentId } },
|
|
summary: null, exposeLowTrustRaw: false })).rejects.toThrow("continuation_user_authorization_missing");
|
|
});
|
|
it("keeps one new turn under concurrent delivery of the same message", async () => {
|
|
const f = await seed();
|
|
const results = await Promise.all([admit(f), admit(f)]);
|
|
expect(results.filter(Boolean)).toHaveLength(1);
|
|
expect(await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId), eq(heartbeatRuns.status, "queued")))).toHaveLength(1);
|
|
});
|
|
it("retains the source incident through prior rejected message admissions", async () => {
|
|
const f = await seed(), rejectedRunId = randomUUID();
|
|
await db.insert(heartbeatRuns).values({ id: rejectedRunId, companyId: f.companyId, agentId: f.agentId,
|
|
status: "cancelled", errorCode: "execution_reconciliation_required", contextSnapshot: { issueId: f.issueId },
|
|
finishedAt: new Date("2026-09-11T10:30:00Z") });
|
|
await db.insert(issueRecoveryActions).values({ companyId: f.companyId, sourceIssueId: f.issueId,
|
|
kind: "active_run_watchdog", cause: "legacy_execution_requires_reconciliation", fingerprint: rejectedRunId,
|
|
status: "resolved", outcome: "blocked", nextAction: "Could not start", evidence: { runId: rejectedRunId, automaticRecovery: { replay: "blocked" } } });
|
|
expect(await admit(f)).toMatchObject({ previousRunId: f.sourceRunId });
|
|
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toBeNull();
|
|
});
|
|
});
|