mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-21 02:43:25 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Paperclip Runner and the shared `adapter-utils` engine run ACP agents through the embedded ACPX library > - Paperclip pins ACPX 0.13.1 and keeps a large patch (`patches/acpx@0.13.1.patch`) on it > - ACPX 0.22.0 adds host hooks (`onAcpMessage`, `assertActive`, `processLifecycle`, prompt-response `_meta`, `spawnAgent`-friendly launch) that make much of that patch unnecessary > - A large patch is expensive to port and hides behavior that belongs to Paperclip > - This pull request upgrades to ACPX 0.22.0 and moves most patched behavior into Runner and `adapter-utils` > - The benefit is a patch that is 484 lines instead of 1,334, with the moved behavior covered by Paperclip tests ## Linked Issues or Issue Description Refs #20675, #14430, #15075, #17942, #9980 **Proposed enhancement** Upgrade the embedded ACPX library from 0.13.1 to 0.22.0. Keep only the patch behavior that stock 0.22 cannot express. **Motivation** The 0.13.1 patch held behavior that Paperclip owns, for example usage receipts, Copilot policy, model checks and the agent environment. Each ACPX upgrade had to port all of it. ACPX 0.22 gives hooks that let Paperclip do this work itself. **Current behavior** Runner and `adapter-utils` depend on 1,334 lines of patched ACPX 0.13.1 code. **Expected behavior** Runner and `adapter-utils` use ACPX 0.22.0. The patch keeps only: extension methods, Cursor subagent routing, event `_meta` and the message boundary chunk, the protocol guard hook, `spawnAgent`, and `requestExtension`. ## What Changed - ACPX 0.13.1 → 0.22.0 in Runner, `adapter-utils`, the server, the Daytona runner images and the qualified profiles (`acpxVersion`). `@agentclientprotocol/sdk` moves to 1.9.0. - New `patches/acpx@0.22.0.patch` (484 lines). The 0.12.0 and 0.13.1 patches are removed. - Delivery receipts: Runner confirms a permission reply when `onAcpMessage` sees the exact outbound reply. Question and extension replies count as delivered when Runner hands them back to ACPX. New `interaction-delivery.ts`. - Copilot policy: the existing policy function runs from `onAcpMessage`. A violation stops the agent, cancels later permission requests and fails the next prompt through `assertActive`. - Model selection: stock 0.22 validation, plus a sidecar gate (`createAcpxEffectiveModelGate`) that refuses a prompt if the agent reports another model. Grok binds its model when the session opens. Antigravity adds its concrete model to the model list ACPX reads. - Typed session failures: Runner and `adapter-utils` read them from the turn result `_meta` and fail the turn. They advertise the capability themselves. - Pi and Cursor usage receipts: a collector (`prompt-receipts.ts`) binds them to the turn's own `session/prompt` reply. - Goal capability: Runner reads it from the `initialize` reply. It keeps it under `paperclip.goalCapability` in the saved agent capabilities `_meta` and restores it when a saved session loads. - Agent environment: Runner and `adapter-utils` set the agent environment in `spawnAgent`. ACP terminals keep their current environment. `adapter-utils` blocks inheritance of `XAI_API_KEY` and refuses to start if the server has an `ACPX_AUTH_*` variable. - `adapter-utils` spawn hooks (`spawnCwd`, stderr, `onAgentSpawn`): a new `stockAcpRuntimeOptions` maps them to stock `spawnAgent` and `processLifecycle`. - Windows: ACPX 0.22 rejects raw command strings on Windows. `adapter-utils` now passes an argv array there. - Grok rules: a Runner guard (`grok-session-rules.ts`) writes `session/new` `_meta.rules`. - Turn-dispatch journal (#20675): ACPX 0.22 ignores errors from `onAcpMessage`. The adapter keeps a failed `dispatched` write and throws it from the turn's `assertActive`. ACPX calls `assertActive` before the prompt write, so the write still stops. The real-ACPX test now checks this route. `docs/durable-recovery.md` is updated. - Plan events: the sidecar reads 0.22 `status` events tagged `plan`. - CI: the backup-checkpoint lane now installs through the shared `setup-ci-dependencies` action, like PR CI, so it resolves a stale `pnpm-lock.yaml` in place. It runs on any root `package.json` change, and pull requests never commit the lockfile, so its own frozen install failed on dependency changes. - Tests that checked removed patch internals are removed or rewritten: the response-delivery, Cursor-usage and Pi-receipt package contracts, and the goal-persistence packaging test. New unit tests cover the moved behavior. ## Verification Local runs on macOS (darwin-arm64), Node 26, on the rebased head: - `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/drivers/acpx src/cli`: 2,000 passed. - `tsc -p packages/paperclip-runner/tsconfig.json` and `tsconfig.test.json`: no errors, except the existing `@paperclipai/paperclip-eval-kernel` build gap in `workflow-harness.ts`. - Runner package contracts (`node --test test/acpx-*-package-contract.test.mjs test/acpx-sidecar-contract.test.mjs`), `scripts/build-inputs.test.mjs`, `scripts/contracts/contract-fingerprints.test.mjs`, `scripts/acpx-patch-packaging.test.mjs`: all pass. `node scripts/generate-acpx-sidecar-contract.mjs --check` passes. - `cargo test -p paperclip-runner-core --test acpx_park_preparation --test acpx_park_readiness --test acpx_warm_attachment --test native_provider_backend`: 86 passed. `cargo fmt --check` passes. - `adapter-utils`: `tsc` clean. `vitest run src/acpx-engine src/mcp-isolation.integration.test.ts`: 457 passed (one Gemini flag test failed once under load and passes when run alone). - Adapters: `claude-local` and `codex-local` `acp.quota.test.ts` (23 and 39 passed), `grok-local` (149 passed). - Server: `scripts/prepare-runner-vendor.test.mjs`, `native-session-executor.test.ts`, `quarantined-task-retry.test.ts` (1,416 passed). - `cargo test -p paperclip-runner-core --lib -- previous_acpx_release acpx_upgrade unrestorable_saved_sessions`: 3 passed. `node --test test/acpx-session-carry-forward-package-contract.test.mjs`: passes. - `patch -p1 --forward` on a clean `acpx-0.22.0.tgz` gives output identical to the port. - Under full-suite load, some Runner live and recovery tests time out. Each one passes when run alone. Full regression is left to CI. Not run: live provider runs (they need provider credentials), the opt-in Kimi live peer test, the Gemini filesystem smoke test, and Windows. ## Risks - Previous-release story: sessions saved by the previous release (ACPX 0.13.1) carry forward. ACPX 0.22 loads the session records 0.13.1 wrote. Runner moves a saved descriptor that names 0.13.1 (`CARRIED_ACPX_RELEASES`) to 0.22.0, and the sidecar keeps 0.13.1 as the ACPX value in the session identity, so the saved session key and identity record still match. The existing launch-profile rebind still decides which sessions resume: a settled session, or the same run's interrupted turn through lost-turn recovery. A session saved by any other ACPX release is still refused as `provider_state_incompatible`. `rules/acpx-profile-identity.mdc` and `docs/durable-recovery.md` now describe this. - Proof: `acpx-session-carry-forward-package-contract.test.mjs` resumes a session record written by the real ACPX 0.13.1 with its Paperclip patch. 0.22 loads the same provider session (no `session/new`), runs a turn and keeps the earlier turn. Rust tests cover a settled 0.13.1 session that the next run attaches to, and a turn interrupted by the upgrade. A recovery-identity test checks that the session key and identity record match across the versions. - `pnpm-lock.yaml` is not committed. CI resolves it for each run, and the lockfile refresh workflow updates it after merge. - Behavior change on the server: `adapter-utils` refuses to create an ACPX runtime if the server environment has an `ACPX_AUTH_*` variable. Before, ACPX ignored such variables. Stock 0.22 would use them to authenticate agents with server credentials. - Delivery confirmation is now Runner-owned. A permission reply is confirmed by the exact outbound message. Question and extension replies are confirmed when Runner hands them to ACPX, not when the pipe accepts them. - Users can select only models that the agent advertises. Stock 0.22 rejects other model IDs (Claude is exempt). - ACP terminals keep the sidecar or server environment, as before. This existing exposure is not changed by this pull request. - The `adapter-utils` Gemini version probe now runs with the server environment plus the run variables, not the run variables alone. - The pack contract digest and profile `acpxVersion` change, so Daytona needs a new runner image. - Grok launch flags (`--model`, `--reasoning-effort`) were checked on 1.0.13 but session open needs a login, so Grok keeps the existing `_meta` path. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file editing, subagents). Extended reasoning was used. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
585 lines
45 KiB
TypeScript
585 lines
45 KiB
TypeScript
import { randomUUID } from "node:crypto";
|
|
import { mkdtemp, mkdir, readFile, rm, writeFile } from "node:fs/promises";
|
|
import { hostname, tmpdir } from "node:os";
|
|
import { join } from "node:path";
|
|
import { and, eq } from "drizzle-orm";
|
|
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
|
|
import {
|
|
agents, agentWakeupRequests, companies, createDb, environmentLeases, executionWorkspaces, heartbeatRunEvents,
|
|
heartbeatRuns, issueRecoveryActions, issues, nativeRunFinalizations, workspaceOperations,
|
|
} from "@paperclipai/db";
|
|
import { startEmbeddedPostgresTestDatabase } from "../__tests__/helpers/embedded-postgres.js";
|
|
import { executeNativeSession, NativeSessionCloseUnrecoverableError, nativeSessionCleanupBlock, onNativeSessionCleanupSettled, parseNativeExecutionInput } from "../vendor/paperclip-runner/index.js";
|
|
import type { NativeSession } from "../vendor/paperclip-runner/index.js";
|
|
import { appendHeartbeatRunEvent } from "./heartbeat-run-events.js";
|
|
import { heartbeatService } from "./heartbeat.js";
|
|
import { getExecutionBlocker } from "./execution-blocker.js";
|
|
import { readFreshWorkspaceReadiness } from "./fresh-workspace-continuation.js";
|
|
import { instanceSettingsService } from "./instance-settings.js";
|
|
import { PROCESS_IDENTITY_RECORDED, PROCESS_START_REQUESTED } from "./native-local-process-stop.js";
|
|
import { currentProcessStartedAtMs } from "./controller-process-identity.js";
|
|
import { nativeSha256 } from "./native-runtime/canonical.js";
|
|
import { retireStoppedSessionCleanupGates, scopedRunnerdStateRoot, verifyStoppedNativeSessionForAutomaticReplacement, verifyStoppedNativeSessionForContinuation } from "./native-runtime/native-session-executor.js";
|
|
import { reconcileSafeNativeReplacements } from "./native-runtime/native-safe-replacement.js";
|
|
import { requestTaskRetry } from "./task-retry.js";
|
|
import { DELIVERY_QUEUES, subscribeDeliveryWork } from "./delivery-work-notifications.js";
|
|
import { registerNativeCleanupReleaseWake } from "./native-runtime/native-cleanup-release-wake.js";
|
|
|
|
describe("task Retry after inherited native cleanup quarantine", () => {
|
|
let postgres: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
|
let db: ReturnType<typeof createDb>;
|
|
beforeAll(async () => {
|
|
postgres = await startEmbeddedPostgresTestDatabase("quarantined-task-retry-");
|
|
db = createDb(postgres.connectionString);
|
|
await instanceSettingsService(db).updateExperimental({ enableNativeRunner: true });
|
|
}, 90_000);
|
|
afterAll(async () => { await db?.$client.end(); await postgres?.cleanup(); });
|
|
|
|
async function seed(base: string, transient = false, provider: "codex" | "acpx" = "codex") {
|
|
const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID();
|
|
const ownerId = randomUUID(), sourceId = randomUUID(), nativeSessionId = randomUUID(), runnerInstanceId = randomUUID();
|
|
const workspaceId = randomUUID();
|
|
const execution = (runId: string) => parseNativeExecutionInput({
|
|
schema: "paperclip.native-execution-input.v1",
|
|
provider: provider === "acpx" ? { kind: "acpx", agent: "claude", model: "claude-sonnet-5", permissionPolicy: "interactive",
|
|
profile: { driverKind: "acpx_runtime", protocolVersion: 1, acpxVersion: "0.22.0", agent: "claude", agentProfileVersion: 1,
|
|
agentServerPackage: "@zed-industries/claude-agent-acp", agentServerVersion: "1", agentRuntimePackage: null,
|
|
agentRuntimeVersion: null, commandDigest: "fixture" } } : { kind: "codex", model: null },
|
|
binding: { companyId, agentId, issueId, runId, executionWorkspaceId: transient ? runId : workspaceId },
|
|
task: { identifier: "TEST", title: "Continue saved work", description: null, prompt: "Continue saved work", workMode: "standard" },
|
|
workspace: { cwd: base, repoUrl: null, repoRef: null, branchName: null },
|
|
session: { normalizedSessionId: nativeSessionId, driverKind: provider === "acpx" ? "acpx_runtime" : "codex_app_server", protocolVersion: 1,
|
|
lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null } },
|
|
completionContract: { id: "contract", sha256: "sha", schemaVersion: "paperclip.completion-contract.v1",
|
|
contract: { revision: "1", objective: "Continue saved work", criteria: [{ id: "done", requirement: "Save the result" }] } },
|
|
interactionResponses: [], credentialBindings: [],
|
|
});
|
|
const processStartedAt = new Date(Date.now() - 120_000);
|
|
const processIdentity = { processPid: 99_999_999, processGroupId: 99_999_999, processStartedAt };
|
|
await db.insert(companies).values({ id: companyId, name: "Quarantined Retry", defaultResponsibleUserId: "board",
|
|
issuePrefix: `Q${companyId.replaceAll("-", "")}` });
|
|
await db.insert(agents).values({ id: agentId, companyId, name: "Native worker", adapterType: "paperclip_runner", status: "idle",
|
|
runtimeConfig: { heartbeat: { maxConcurrentRuns: 1 } } });
|
|
await db.insert(issues).values({ id: issueId, companyId, title: "Continue saved work", status: "blocked", assigneeAgentId: agentId });
|
|
if (!transient) {
|
|
await db.insert(executionWorkspaces).values({ id: workspaceId, companyId, sourceIssueId: issueId,
|
|
mode: "shared_workspace", strategyType: "project_primary", name: "Saved work", cwd: base });
|
|
await db.update(issues).set({ executionWorkspaceId: workspaceId }).where(eq(issues.id, issueId));
|
|
}
|
|
const resultJson = { savedAnswer: "Existing work", recoveredExecutionFailure: { errorCode: "provider_transport_failed" } };
|
|
for (const runId of [ownerId, sourceId]) {
|
|
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, issueId, nativeIssueId: issueId,
|
|
runtimeMode: "native", nativeSessionId, runnerInstanceId, ...processIdentity,
|
|
status: runId === ownerId ? "succeeded" : "failed", resultJson: runId === ownerId ? resultJson : null,
|
|
errorCode: runId === sourceId ? "native_session_cleanup_quarantined" : null,
|
|
createdAt: new Date(Date.now() - (runId === ownerId ? 110_000 : 60_000)),
|
|
startedAt: new Date(Date.now() - (runId === ownerId ? 109_000 : 59_000)),
|
|
finishedAt: new Date(Date.now() - (runId === ownerId ? 100_000 : 50_000)),
|
|
contextSnapshot: { issueId }, runnerProfileJson: { nativeExecutionInput: execution(runId),
|
|
nativeToolContractFingerprint: `sha256:${"1".repeat(64)}`, sessionCheckpoint: { sessionId: "retained-provider-thread" } },
|
|
});
|
|
await db.insert(nativeRunFinalizations).values({ companyId, issueId, runId,
|
|
phase: runId === ownerId ? "committed" : "terminal_failure" });
|
|
await db.insert(environmentLeases).values({ companyId, issueId, heartbeatRunId: runId,
|
|
provider: "local", status: "failed", leasePolicy: "ephemeral", releasedAt: new Date(Date.now() - 40_000) });
|
|
}
|
|
await appendHeartbeatRunEvent(db, { companyId, runId: ownerId, agentId, eventType: PROCESS_IDENTITY_RECORDED,
|
|
payload: { ...processIdentity, processStartedAt: processStartedAt.toISOString(), targetKind: "local", hostname: hostname() } });
|
|
await db.insert(issueRecoveryActions).values({ companyId, sourceIssueId: issueId, kind: "active_run_watchdog",
|
|
cause: "native_session_cleanup_quarantined", fingerprint: sourceId, status: "resolved", outcome: "blocked",
|
|
nextAction: "Automatic recovery stopped.", evidence: { runId: sourceId,
|
|
automaticRecovery: { replay: "blocked", actionOutcome: "unknown" } } });
|
|
const identity = { runId: ownerId, runnerInstanceId, normalizedSessionId: nativeSessionId,
|
|
environmentLeaseId: transient ? ownerId : workspaceId, turnId: `turn-${ownerId}`, itemId: `item-${ownerId}` };
|
|
const event = { schema: "paperclip.prp.event.v1", schemaVersion: 1, sourceKind: "runner", sourceSeq: 5,
|
|
sourceEventId: "provider-open", sourceInstanceId: runnerInstanceId, runId: ownerId,
|
|
normalizedSessionId: nativeSessionId, turnId: identity.turnId, itemId: identity.itemId,
|
|
priority: 0, emittedAt: processStartedAt.toISOString(), eventType: "session.resumed",
|
|
payload: { providerSessionId: "retained-provider-thread", processId: 99_999_998 } };
|
|
const normalized = { ...event, sourceSeq: 1, sourceEventId: `${runnerInstanceId}:${ownerId}:1`, turnId: undefined, itemId: undefined,
|
|
payload: { providerSessionId: "retained-provider-thread", driverSessionId: "retained-provider-thread" } };
|
|
await appendHeartbeatRunEvent(db, { companyId, runId: ownerId, agentId, eventType: event.eventType,
|
|
payload: { prpEvent: normalized }, nativeSource: { sourceInstanceId: runnerInstanceId,
|
|
sourceEventId: normalized.sourceEventId, sourceSeq: 1, protocolSchemaVersion: 1, canonicalPayload: normalized } });
|
|
const root = provider === "acpx" ? scopedRunnerdStateRoot(execution(ownerId)) : join(base, nativeSha256({ schema: "paperclip.native-session-scope.v2", companyId, agentId,
|
|
workspace: transient ? { kind: "transient", cwd: base, repoUrl: null, repoRef: null, branchName: null }
|
|
: { kind: "managed", executionWorkspaceId: workspaceId },
|
|
provider: { driverKind: "codex_app_server", identity: { kind: "codex" } }, normalizedSessionId: nativeSessionId }));
|
|
const files: Record<string, Record<string, unknown>> = {
|
|
"control-plane/control-plane-state.json": { schema: "paperclip.runner.durable.control-plane-state.v1", identity,
|
|
committedEvents: [{ envelope: { payload: event } }], commands: [{ status: "pending", payload: { commandType: "tool.result" } }] },
|
|
"runner/runner-state.json": { schema: "paperclip.runner.durable.state.v1", ...identity, lifecycle: "suspended", outbox: [] },
|
|
...(provider === "acpx" ? {
|
|
"runner/acpx-provider-state.json": { schema: "paperclip.runner.acpx-provider-state.v3", lifecycle: "session_open",
|
|
startupAttempt: null, pendingEvents: [], queuedEvents: [] },
|
|
} : {
|
|
"runner/codex-provider-state.json": { schema: "paperclip.runner.codex-provider-state.v1", lifecycle: "session_open",
|
|
threadId: "retained-provider-thread", startupAttempt: null, pendingEvents: [], queuedEvents: [] },
|
|
}),
|
|
};
|
|
await mkdir(join(root, "control-plane"), { recursive: true });
|
|
await mkdir(join(root, "runner"), { recursive: true });
|
|
for (const [path, value] of Object.entries(files)) await writeFile(join(root, path), JSON.stringify(value));
|
|
await writeFile(join(base, "saved-work.txt"), "Keep the prior result and uncertain action history.");
|
|
return { companyId, agentId, issueId, workspaceId, ownerId, sourceId, runnerInstanceId, resultJson, root, files };
|
|
}
|
|
|
|
it.each(["preserved", "projectless", "changed_binding", "changed_cwd", "pending_operation", "active_lease", "closed_workspace", "remote_workspace", "foreign_receipt"])(
|
|
"admits one fresh successor through canonical Retry without replaying the retained session (%s)", async mode => {
|
|
const base = await mkdtemp(join(tmpdir(), "quarantined-retry-"));
|
|
const previous = process.env.PAPERCLIP_RUNNER_STATE_DIR;
|
|
process.env.PAPERCLIP_RUNNER_STATE_DIR = base;
|
|
try {
|
|
const f = await seed(base, mode === "projectless");
|
|
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" });
|
|
const input = { db, companyId: f.companyId, issueId: f.issueId, actorId: "board", actorSource: "local_implicit", observedRunId: f.sourceId };
|
|
const receipt = await requestTaskRetry({ ...input, idempotencyKey: randomUUID() });
|
|
const duplicate = await requestTaskRetry({ ...input, idempotencyKey: randomUUID() });
|
|
expect(duplicate.requestId).toBe(receipt.requestId);
|
|
const heartbeat = heartbeatService(db);
|
|
await Promise.all([heartbeat.resumeTaskRetry(f.companyId, receipt.requestId!), heartbeat.resumeTaskRetry(f.companyId, receipt.requestId!)]);
|
|
const successors = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId), eq(heartbeatRuns.status, "queued")));
|
|
const [state] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, receipt.requestId!));
|
|
expect(successors, JSON.stringify(state.payload)).toHaveLength(1);
|
|
expect(successors[0].contextSnapshot).toMatchObject({ previousRunId: f.sourceId, forceFreshSession: true, taskRetryRequestId: receipt.requestId });
|
|
if (mode === "changed_binding") await db.update(issues).set({ executionWorkspaceId: null }).where(eq(issues.id, f.issueId));
|
|
if (mode === "changed_cwd") await db.update(executionWorkspaces).set({ cwd: join(base, "another-directory") }).where(eq(executionWorkspaces.id, f.workspaceId));
|
|
if (mode === "closed_workspace") await db.update(executionWorkspaces).set({ status: "closed" }).where(eq(executionWorkspaces.id, f.workspaceId));
|
|
if (mode === "remote_workspace") await db.update(executionWorkspaces).set({ providerType: "plugin" }).where(eq(executionWorkspaces.id, f.workspaceId));
|
|
if (mode === "pending_operation") await db.insert(workspaceOperations).values({ companyId: f.companyId,
|
|
executionWorkspaceId: f.workspaceId, heartbeatRunId: f.ownerId, issueId: f.issueId, phase: "workspace_restore", status: "running" });
|
|
if (mode === "active_lease") await db.insert(environmentLeases).values({ companyId: f.companyId,
|
|
executionWorkspaceId: f.workspaceId, heartbeatRunId: f.ownerId, issueId: f.issueId, provider: "local", status: "active", leasePolicy: "ephemeral" });
|
|
if (mode === "foreign_receipt") {
|
|
const [proof] = await db.select().from(heartbeatRunEvents).where(and(eq(heartbeatRunEvents.runId, f.sourceId),
|
|
eq(heartbeatRunEvents.eventType, "native.stopped_conversation_verified")));
|
|
await db.update(heartbeatRunEvents).set({ payload: { ...proof.payload, successorRunId: randomUUID() } })
|
|
.where(eq(heartbeatRunEvents.id, proof.id));
|
|
expect(await readFreshWorkspaceReadiness(db, successors[0])).toMatchObject({ mode: "fresh_isolated_workspace" });
|
|
} else if (mode === "projectless") {
|
|
expect(await readFreshWorkspaceReadiness(db, successors[0])).toMatchObject({ mode: "fresh_isolated_workspace",
|
|
retryIntentId: receipt.requestId, sourceRunId: f.sourceId });
|
|
} else if (mode === "preserved") {
|
|
expect(await readFreshWorkspaceReadiness(db, successors[0])).toMatchObject({ mode: "resume_stopped_native_workspace",
|
|
retryIntentId: receipt.requestId, sourceRunId: f.sourceId, workspaceId: f.workspaceId, cwd: base });
|
|
} else await expect(readFreshWorkspaceReadiness(db, successors[0])).rejects.toThrow(/Verified native continuation workspace/);
|
|
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toBeNull();
|
|
const [wake] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, receipt.requestId!));
|
|
expect(wake).toMatchObject({ status: "coalesced", runId: successors[0].id });
|
|
const [owner] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.ownerId));
|
|
expect(owner.resultJson).toEqual(f.resultJson);
|
|
const [action] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issueId));
|
|
expect(action.evidence.automaticRecovery).toMatchObject({ actionOutcome: "unknown", replay: "explicit_user_continuation" });
|
|
const proof = await db.select().from(heartbeatRunEvents).where(and(eq(heartbeatRunEvents.runId, f.sourceId), eq(heartbeatRunEvents.eventType, "native.stopped_conversation_verified")));
|
|
expect(proof).toHaveLength(1);
|
|
expect(proof[0].payload).toMatchObject({ runId: f.sourceId, ownerRunId: f.ownerId });
|
|
for (const [path, value] of Object.entries(f.files)) expect(await readFile(join(f.root, path), "utf8")).toBe(JSON.stringify(value));
|
|
expect(await readFile(join(base, "saved-work.txt"), "utf8")).toContain("Keep the prior result");
|
|
} finally {
|
|
if (previous === undefined) delete process.env.PAPERCLIP_RUNNER_STATE_DIR; else process.env.PAPERCLIP_RUNNER_STATE_DIR = previous;
|
|
await rm(base, { recursive: true, force: true });
|
|
}
|
|
});
|
|
|
|
it.each(["provider_alive", "worker_alive", "owner_running", "controller_owned", "owner_remote", "owner_unreleased",
|
|
"owner_new_launch", "source_new_launch", "foreign_host", "foreign_workspace", "foreign_cwd", "foreign_runner", "missing_provider_receipt",
|
|
"pending_provider_launch", "changed_process_birth", "changed_journal", "normalized_receipt_identity", "provider_identity_conflict"])("keeps quarantine for %s", async mode => {
|
|
const base = await mkdtemp(join(tmpdir(), "quarantined-retry-fence-"));
|
|
const previous = process.env.PAPERCLIP_RUNNER_STATE_DIR;
|
|
process.env.PAPERCLIP_RUNNER_STATE_DIR = base;
|
|
try {
|
|
const f = await seed(base);
|
|
if (mode === "provider_alive" || mode === "pending_provider_launch") {
|
|
await writeFile(join(f.root, "runner/codex-provider-state.json"), JSON.stringify({
|
|
...f.files["runner/codex-provider-state.json"],
|
|
...(mode === "provider_alive" ? { processId: process.pid } : { startupAttempt: { phase: "intent" } }),
|
|
}));
|
|
}
|
|
if (mode === "worker_alive") {
|
|
// This test process, with its real start time: the shared prover then
|
|
// sees the same live worker, not a reused PID.
|
|
const startedAt = new Date((await currentProcessStartedAtMs())!);
|
|
await db.update(heartbeatRuns).set({ processPid: process.pid, processGroupId: process.pid, processStartedAt: startedAt })
|
|
.where(eq(heartbeatRuns.companyId, f.companyId));
|
|
const [owner] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.ownerId));
|
|
await appendHeartbeatRunEvent(db, { companyId: f.companyId, runId: f.ownerId, agentId: f.agentId,
|
|
eventType: PROCESS_IDENTITY_RECORDED, payload: { targetKind: "local", hostname: hostname(),
|
|
processPid: process.pid, processGroupId: process.pid, processStartedAt: owner.processStartedAt!.toISOString() } });
|
|
}
|
|
if (mode === "owner_running") await db.update(heartbeatRuns).set({ status: "running", finishedAt: null }).where(eq(heartbeatRuns.id, f.ownerId));
|
|
if (mode === "controller_owned") await db.update(nativeRunFinalizations).set({ leaseOwner: "other-controller" }).where(eq(nativeRunFinalizations.runId, f.ownerId));
|
|
if (mode === "owner_remote" || mode === "owner_unreleased") await db.update(environmentLeases).set(
|
|
mode === "owner_remote" ? { provider: "daytona" } : { releasedAt: null },
|
|
).where(eq(environmentLeases.heartbeatRunId, f.ownerId));
|
|
if (mode === "owner_new_launch" || mode === "source_new_launch") await appendHeartbeatRunEvent(db, {
|
|
companyId: f.companyId, runId: mode === "owner_new_launch" ? f.ownerId : f.sourceId, agentId: f.agentId, eventType: PROCESS_START_REQUESTED,
|
|
});
|
|
if (mode === "foreign_host") {
|
|
const [identity] = await db.select().from(heartbeatRunEvents).where(and(eq(heartbeatRunEvents.runId, f.ownerId),
|
|
eq(heartbeatRunEvents.eventType, PROCESS_IDENTITY_RECORDED)));
|
|
await db.update(heartbeatRunEvents).set({ payload: { ...identity.payload, hostname: "another-host" } })
|
|
.where(eq(heartbeatRunEvents.id, identity.id));
|
|
}
|
|
if (mode === "changed_process_birth") await db.update(heartbeatRuns).set({ processStartedAt: new Date() }).where(eq(heartbeatRuns.id, f.sourceId));
|
|
if (mode === "foreign_runner") await db.update(heartbeatRuns).set({ runnerInstanceId: randomUUID() }).where(eq(heartbeatRuns.id, f.ownerId));
|
|
if (mode === "foreign_workspace" || mode === "foreign_cwd") {
|
|
const [owner] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.ownerId));
|
|
const input = parseNativeExecutionInput(owner.runnerProfileJson!.nativeExecutionInput);
|
|
await db.update(heartbeatRuns).set({ runnerProfileJson: { ...owner.runnerProfileJson,
|
|
nativeExecutionInput: { ...input,
|
|
...(mode === "foreign_workspace" ? { binding: { ...input.binding, executionWorkspaceId: "another-workspace" } }
|
|
: { workspace: { ...input.workspace, cwd: join(base, "another-directory") } }),
|
|
},
|
|
} }).where(eq(heartbeatRuns.id, f.ownerId));
|
|
}
|
|
if (mode === "missing_provider_receipt") await db.delete(heartbeatRunEvents).where(and(
|
|
eq(heartbeatRunEvents.runId, f.ownerId), eq(heartbeatRunEvents.eventType, "session.resumed")));
|
|
if (mode === "normalized_receipt_identity") await db.update(heartbeatRunEvents).set({ sourceSeq: 2 }).where(and(
|
|
eq(heartbeatRunEvents.runId, f.ownerId), eq(heartbeatRunEvents.eventType, "session.resumed")));
|
|
if (mode === "provider_identity_conflict") {
|
|
const [receipt] = await db.select().from(heartbeatRunEvents).where(and(eq(heartbeatRunEvents.runId, f.ownerId),
|
|
eq(heartbeatRunEvents.eventType, "session.resumed")));
|
|
const event = receipt.payload!.prpEvent as Record<string, unknown>;
|
|
const changed = { ...event, payload: { providerSessionId: "another-thread", driverSessionId: "another-thread" } };
|
|
await db.update(heartbeatRunEvents).set({ payload: { prpEvent: changed }, sourcePayloadSha256: nativeSha256(changed) })
|
|
.where(eq(heartbeatRunEvents.id, receipt.id));
|
|
}
|
|
const [source] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.sourceId));
|
|
const proof = await verifyStoppedNativeSessionForContinuation(db, source);
|
|
if (mode === "changed_journal") {
|
|
expect(proof).not.toBeNull();
|
|
await writeFile(join(f.root, "runner/runner-state.json"), "{}");
|
|
expect(proof!.retire()).toBe(false);
|
|
} else expect(proof).toBeNull();
|
|
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).not.toBeNull();
|
|
} finally {
|
|
if (previous === undefined) delete process.env.PAPERCLIP_RUNNER_STATE_DIR; else process.env.PAPERCLIP_RUNNER_STATE_DIR = previous;
|
|
await rm(base, { recursive: true, force: true });
|
|
}
|
|
});
|
|
|
|
it.each(["stopped", "source_launch_requested", "owner_running"])(
|
|
"certifies a quarantined source that never launched from the previous owner's stop (%s)", async mode => {
|
|
// s7532/t04: the quarantine fence refused the source before it asked for
|
|
// a runner, so it has no process record at all. Only the owner can prove
|
|
// the retained session stopped.
|
|
const base = await mkdtemp(join(tmpdir(), "quarantined-never-launched-"));
|
|
const previous = process.env.PAPERCLIP_RUNNER_STATE_DIR;
|
|
process.env.PAPERCLIP_RUNNER_STATE_DIR = base;
|
|
try {
|
|
const f = await seed(base);
|
|
await db.update(heartbeatRuns).set({ processPid: null, processGroupId: null, processStartedAt: null })
|
|
.where(eq(heartbeatRuns.id, f.sourceId));
|
|
if (mode === "source_launch_requested") await appendHeartbeatRunEvent(db, {
|
|
companyId: f.companyId, runId: f.sourceId, agentId: f.agentId, eventType: PROCESS_START_REQUESTED });
|
|
if (mode === "owner_running") await db.update(heartbeatRuns).set({ status: "running", finishedAt: null }).where(eq(heartbeatRuns.id, f.ownerId));
|
|
const [source] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.sourceId));
|
|
const proof = await verifyStoppedNativeSessionForContinuation(db, source);
|
|
if (mode === "stopped") expect(proof).not.toBeNull();
|
|
else expect(proof).toBeNull();
|
|
} finally {
|
|
if (previous === undefined) delete process.env.PAPERCLIP_RUNNER_STATE_DIR; else process.env.PAPERCLIP_RUNNER_STATE_DIR = previous;
|
|
await rm(base, { recursive: true, force: true });
|
|
}
|
|
});
|
|
|
|
it.each([
|
|
{ provider: "codex", mode: "stopped" }, { provider: "codex", mode: "owner_running" },
|
|
{ provider: "acpx", mode: "stopped" }, { provider: "acpx", mode: "provider_alive" },
|
|
] as const)(
|
|
"recovers a quarantined $provider source that never launched with no click once its owner is proven stopped ($mode)", async ({ provider, mode }) => {
|
|
const base = await mkdtemp(join(tmpdir(), "quarantined-auto-continuation-"));
|
|
const previous = process.env.PAPERCLIP_RUNNER_STATE_DIR;
|
|
process.env.PAPERCLIP_RUNNER_STATE_DIR = base;
|
|
try {
|
|
const f = await seed(base, false, provider);
|
|
await db.update(heartbeatRuns).set({ processPid: null, processGroupId: null, processStartedAt: null })
|
|
.where(eq(heartbeatRuns.id, f.sourceId));
|
|
if (provider === "acpx") await writeFile(join(f.root, "runner", "acpx-provider-state.json"), JSON.stringify({
|
|
...f.files["runner/acpx-provider-state.json"],
|
|
identity: { agentPid: mode === "provider_alive" ? process.pid : 99_999_997 } }));
|
|
if (mode === "owner_running") await db.update(heartbeatRuns).set({ status: "running", finishedAt: null }).where(eq(heartbeatRuns.id, f.ownerId));
|
|
await db.update(nativeRunFinalizations).set({ failureCode: "native_session_cleanup_quarantined", attempt: 1,
|
|
failureDetail: { recoverable: false, recoveryMode: "exact_checkpoint_resume", originalFailureCode: "native_session_cleanup_quarantined",
|
|
providerEventsExist: false, checkpointExists: true, providerSessionEstablished: true } })
|
|
.where(eq(nativeRunFinalizations.runId, f.sourceId));
|
|
const [task] = await db.select().from(issues).where(eq(issues.id, f.issueId));
|
|
const [hold] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issueId));
|
|
await db.update(issueRecoveryActions).set({ evidence: { ...hold!.evidence,
|
|
nativeFailureBlock: { runId: f.sourceId, statusVersion: task!.statusVersion } } })
|
|
.where(eq(issueRecoveryActions.id, hold!.id));
|
|
const [before] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.sourceId));
|
|
await reconcileSafeNativeReplacements(db, new Date(), {
|
|
verifyStoppedSession: run => verifyStoppedNativeSessionForAutomaticReplacement(db, run) });
|
|
const [after] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.sourceId));
|
|
const successorRunId = after?.failureDetail?.successorRunId;
|
|
if (mode !== "stopped") {
|
|
expect(successorRunId).toBeUndefined();
|
|
return;
|
|
}
|
|
expect(typeof successorRunId).toBe("string");
|
|
const [successor] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, String(successorRunId)));
|
|
expect(successor?.contextSnapshot).toMatchObject({ forceFreshSession: true });
|
|
expect(successor?.scheduledRetryAttempt).toBe(before!.attempt);
|
|
} finally {
|
|
if (previous === undefined) delete process.env.PAPERCLIP_RUNNER_STATE_DIR; else process.env.PAPERCLIP_RUNNER_STATE_DIR = previous;
|
|
await rm(base, { recursive: true, force: true });
|
|
}
|
|
});
|
|
|
|
it("holds a Retry while the task's own provider session close is still retained, then starts it", async () => {
|
|
const base = await mkdtemp(join(tmpdir(), "retained-close-retry-"));
|
|
const previous = process.env.PAPERCLIP_RUNNER_STATE_DIR;
|
|
process.env.PAPERCLIP_RUNNER_STATE_DIR = base;
|
|
let releaseClose = () => {};
|
|
try {
|
|
const f = await seed(base);
|
|
// The source failed for another reason; only the runner's in-process
|
|
// admission gate still holds this task's provider session close.
|
|
await db.update(heartbeatRuns).set({ errorCode: "provider_transport_failed" }).where(eq(heartbeatRuns.id, f.sourceId));
|
|
const settled = vi.fn();
|
|
const unsubscribe = onNativeSessionCleanupSettled(settled);
|
|
// The production release wake, observed on the delivery queue it signals.
|
|
const wakeFailure = vi.fn();
|
|
const unregisterWake = registerNativeCleanupReleaseWake(db, wakeFailure);
|
|
const deferredWake = vi.fn();
|
|
const unsubscribeDeferred = subscribeDeliveryWork(db, DELIVERY_QUEUES.deferredExecutionReceipts, deferredWake);
|
|
const pendingClose = new Promise<void>(resolve => { releaseClose = resolve; });
|
|
const identity = { runId: randomUUID(), sessionId: "retained-close-session", companyId: f.companyId, issueId: f.issueId, agentId: f.agentId };
|
|
const capabilities = { resume: true, typedEvents: true, steering: false, interruption: false, structuredResult: true };
|
|
const session: NativeSession = {
|
|
identity: () => identity, capabilities: async () => capabilities,
|
|
async *events() { throw new Error("retained close transport stopped"); },
|
|
startTurn: async () => ({ turnId: "retained-close-turn" }), result: async () => null,
|
|
async snapshot() { return { backendKind: "mock", sessionId: "retained-close-driver", identity,
|
|
providerSessionId: "retained-close-provider", cursor: null, activeTurnId: null, pendingRuntimeRequests: [], lineage: [] }; },
|
|
close: () => pendingClose,
|
|
};
|
|
await expect(executeNativeSession({
|
|
input: parseNativeExecutionInput({ schema: "paperclip.native-execution-input.v1", provider: { kind: "codex", model: null },
|
|
binding: { companyId: f.companyId, agentId: f.agentId, issueId: f.issueId, runId: identity.runId, executionWorkspaceId: f.workspaceId },
|
|
task: { identifier: "TEST", title: "Continue saved work", description: null, prompt: "Continue saved work", workMode: "standard" },
|
|
workspace: { cwd: base, repoUrl: null, repoRef: null, branchName: null },
|
|
session: { normalizedSessionId: identity.sessionId, driverKind: "codex_app_server", protocolVersion: 1,
|
|
lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null } },
|
|
completionContract: { id: "contract", sha256: "sha", schemaVersion: "paperclip.completion-contract.v1",
|
|
contract: { revision: "1", objective: "Continue saved work", criteria: [{ id: "done", requirement: "Save the result" }] } },
|
|
interactionResponses: [], credentialBindings: [] }),
|
|
backend: { descriptor: async () => ({ kind: "runner", name: "retained-close-backend", version: "1", capabilities }),
|
|
openSession: async () => session },
|
|
controlPlane: { openRun: async () => {}, checkpointSession: async () => {},
|
|
appendEvent: async () => ({ cursor: 0, highestContiguousSourceSeq: 0, disposition: "committed" }),
|
|
replayEvents: async () => ({ events: [], highestContiguousSourceSeq: 0 }), completeRun: async () => {} },
|
|
runnerInstanceId: f.runnerInstanceId, controlPlaneInstanceId: "control",
|
|
})).rejects.toThrow("retained close transport stopped");
|
|
expect(nativeSessionCleanupBlock({ companyId: f.companyId, issueId: f.issueId })).toBe("pending");
|
|
expect(nativeSessionCleanupBlock({ companyId: f.companyId, issueId: randomUUID() })).toBeNull();
|
|
|
|
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" });
|
|
const receipt = await requestTaskRetry({ db, companyId: f.companyId, issueId: f.issueId, actorId: "board",
|
|
actorSource: "local_implicit", observedRunId: f.sourceId, idempotencyKey: randomUUID() });
|
|
const heartbeat = heartbeatService(db);
|
|
await heartbeat.resumeTaskRetry(f.companyId, receipt.requestId!);
|
|
const queued = () => db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId), eq(heartbeatRuns.status, "queued")));
|
|
// No successor is dispatched that the runner would refuse at once.
|
|
expect(await queued()).toHaveLength(0);
|
|
const [held] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, receipt.requestId!));
|
|
expect(held).toMatchObject({ status: "deferred_issue_execution", runId: null });
|
|
expect(held.payload).toMatchObject({ executionWait: { reason: "local_cleanup" } });
|
|
|
|
// The close settles by itself; its release wakes the held Retry, which then starts.
|
|
releaseClose();
|
|
await vi.waitFor(() => expect(settled).toHaveBeenCalledWith({ companyId: f.companyId, issueIds: [f.issueId] }));
|
|
await vi.waitFor(() => expect(deferredWake).toHaveBeenCalled());
|
|
expect(wakeFailure).not.toHaveBeenCalled();
|
|
expect(nativeSessionCleanupBlock({ companyId: f.companyId, issueId: f.issueId })).toBeNull();
|
|
unsubscribe();
|
|
unregisterWake();
|
|
unsubscribeDeferred();
|
|
// The deferred-execution worker runs this pass on that wake.
|
|
await heartbeat.resumeTaskRetry(f.companyId, receipt.requestId!);
|
|
expect(await queued()).toHaveLength(1);
|
|
} finally {
|
|
releaseClose();
|
|
if (previous === undefined) delete process.env.PAPERCLIP_RUNNER_STATE_DIR; else process.env.PAPERCLIP_RUNNER_STATE_DIR = previous;
|
|
await rm(base, { recursive: true, force: true });
|
|
}
|
|
});
|
|
|
|
// s9107 t04/t05: the server paused (SIGSTOP) after a turn completed, its
|
|
// runner checkpoint owner lapsed, and the session close could not prove the
|
|
// runner suspended, leaving an operator-required gate in this process.
|
|
async function quarantineOwnClose(f: { companyId: string; agentId: string; issueId: string; sourceId: string;
|
|
workspaceId: string; runnerInstanceId: string }, base: string, runId = f.sourceId) {
|
|
const identity = { runId, sessionId: "own-close-session", companyId: f.companyId, issueId: f.issueId, agentId: f.agentId };
|
|
const capabilities = { resume: true, typedEvents: true, steering: false, interruption: false, structuredResult: true };
|
|
const session: NativeSession = {
|
|
identity: () => identity, capabilities: async () => capabilities,
|
|
async *events() { throw new Error("native_runner_checkpoint_owner_lost"); },
|
|
startTurn: async () => ({ turnId: "own-close-turn" }), result: async () => null,
|
|
async snapshot() { return { backendKind: "mock", sessionId: "own-close-driver", identity,
|
|
providerSessionId: "own-close-provider", cursor: null, activeTurnId: null, pendingRuntimeRequests: [], lineage: [] }; },
|
|
close: async () => { throw new NativeSessionCloseUnrecoverableError({ runnerSuspended: false, providerDrained: true }); },
|
|
};
|
|
await expect(executeNativeSession({
|
|
input: parseNativeExecutionInput({ schema: "paperclip.native-execution-input.v1", provider: { kind: "codex", model: null },
|
|
binding: { companyId: f.companyId, agentId: f.agentId, issueId: f.issueId, runId, executionWorkspaceId: f.workspaceId },
|
|
task: { identifier: "TEST", title: "Continue saved work", description: null, prompt: "Continue saved work", workMode: "standard" },
|
|
workspace: { cwd: base, repoUrl: null, repoRef: null, branchName: null },
|
|
session: { normalizedSessionId: identity.sessionId, driverKind: "codex_app_server", protocolVersion: 1,
|
|
lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null } },
|
|
completionContract: { id: "contract", sha256: "sha", schemaVersion: "paperclip.completion-contract.v1",
|
|
contract: { revision: "1", objective: "Continue saved work", criteria: [{ id: "done", requirement: "Save the result" }] } },
|
|
interactionResponses: [], credentialBindings: [] }),
|
|
backend: { descriptor: async () => ({ kind: "runner", name: "own-close-backend", version: "1", capabilities }),
|
|
openSession: async () => session },
|
|
controlPlane: { openRun: async () => {}, checkpointSession: async () => {},
|
|
appendEvent: async () => ({ cursor: 0, highestContiguousSourceSeq: 0, disposition: "committed" }),
|
|
replayEvents: async () => ({ events: [], highestContiguousSourceSeq: 0 }), completeRun: async () => {} },
|
|
runnerInstanceId: f.runnerInstanceId, controlPlaneInstanceId: "control",
|
|
})).rejects.toThrow();
|
|
await vi.waitFor(() => expect(nativeSessionCleanupBlock({ companyId: f.companyId, issueId: f.issueId })).toBe("operator_required"));
|
|
}
|
|
|
|
// An ACPX session open names no provider PID; its sidecar reports one separately (s9107 t04).
|
|
async function recordOwnStop(f: { companyId: string; agentId: string; sourceId: string }, providerPid: number,
|
|
sidecar?: { processGroupId: number | null }, options: { noIdentity?: boolean; startedPid?: boolean } = {}) {
|
|
const processStartedAt = new Date(Date.now() - 90_000).toISOString();
|
|
await appendHeartbeatRunEvent(db, { companyId: f.companyId, runId: f.sourceId, agentId: f.agentId, eventType: PROCESS_IDENTITY_RECORDED,
|
|
payload: { processPid: 99_999_996, processGroupId: 99_999_996, processStartedAt, targetKind: "local", hostname: hostname() } });
|
|
// The real Codex projection names no PID in session opens; the checkpoint does.
|
|
await appendHeartbeatRunEvent(db, { companyId: f.companyId, runId: f.sourceId, agentId: f.agentId,
|
|
eventType: "session.started", payload: { prpEvent: { payload: { providerSessionId: "own-provider", driverSessionId: "own-provider",
|
|
...(options.startedPid ? { started: { pid: providerPid } } : {}) } } } });
|
|
await appendHeartbeatRunEvent(db, { companyId: f.companyId, runId: f.sourceId, agentId: f.agentId,
|
|
eventType: "session.reconciled", payload: { prpEvent: { payload: { providerSessionId: "own-provider", turnCount: 1,
|
|
activeTurnId: null, terminalTurnId: "own-turn" } } } });
|
|
if (!sidecar && !options.noIdentity && !options.startedPid) {
|
|
const [row] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.sourceId));
|
|
await db.update(heartbeatRuns).set({ runnerProfileJson: { ...row.runnerProfileJson,
|
|
sessionCheckpoint: { ...(row.runnerProfileJson as { sessionCheckpoint: object }).sessionCheckpoint, process: { providerPid } } } })
|
|
.where(eq(heartbeatRuns.id, f.sourceId));
|
|
}
|
|
if (sidecar) await appendHeartbeatRunEvent(db, { companyId: f.companyId, runId: f.sourceId, agentId: f.agentId,
|
|
eventType: "harness.diagnostic", payload: { prpEvent: { payload: { code: "acpx_process",
|
|
details: { pid: providerPid, role: "sidecar", processGroupId: sidecar.processGroupId } } } } });
|
|
await appendHeartbeatRunEvent(db, { companyId: f.companyId, runId: f.sourceId, agentId: f.agentId, eventType: "native.local_process_stopped",
|
|
payload: { processPid: 99_999_996, processGroupId: 99_999_996, processStartedAt } });
|
|
}
|
|
|
|
it.each(["stopped", "provider_alive", "no_stop", "no_provider_identity", "started_pid", "acpx_sidecar", "acpx_sidecar_own_group"] as const)(
|
|
"retires a run's own failed-close gate once its runner and providers are proven stopped (%s)", async mode => {
|
|
const base = await mkdtemp(join(tmpdir(), "own-close-gate-"));
|
|
const previous = process.env.PAPERCLIP_RUNNER_STATE_DIR;
|
|
process.env.PAPERCLIP_RUNNER_STATE_DIR = base;
|
|
try {
|
|
const f = await seed(base);
|
|
await quarantineOwnClose(f, base);
|
|
if (mode !== "no_stop") await recordOwnStop(f, mode === "provider_alive" ? process.pid : 99_999_995,
|
|
mode === "acpx_sidecar" ? { processGroupId: null } : mode === "acpx_sidecar_own_group" ? { processGroupId: 99_999_995 } : undefined,
|
|
{ noIdentity: mode === "no_provider_identity", startedPid: mode === "started_pid" });
|
|
const retires = mode === "stopped" || mode === "started_pid" || mode === "acpx_sidecar";
|
|
const [source] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.sourceId));
|
|
expect(await retireStoppedSessionCleanupGates(db, { companyId: f.companyId, issueId: f.issueId, run: source })).toBe(retires);
|
|
expect(nativeSessionCleanupBlock({ companyId: f.companyId, issueId: f.issueId }))
|
|
.toBe(retires ? null : "operator_required");
|
|
} finally {
|
|
if (previous === undefined) delete process.env.PAPERCLIP_RUNNER_STATE_DIR; else process.env.PAPERCLIP_RUNNER_STATE_DIR = previous;
|
|
await rm(base, { recursive: true, force: true });
|
|
}
|
|
});
|
|
|
|
it.each(["stopped", "provider_alive"] as const)(
|
|
"starts a task Retry for a run's own quarantine proven stopped by its own process record (%s)", async mode => {
|
|
const base = await mkdtemp(join(tmpdir(), "own-quarantine-retry-"));
|
|
const previous = process.env.PAPERCLIP_RUNNER_STATE_DIR;
|
|
process.env.PAPERCLIP_RUNNER_STATE_DIR = base;
|
|
try {
|
|
const f = await seed(base);
|
|
// The source launched its own runner, so the retained owner's inventory
|
|
// cannot stand in for it; only its own stop record can.
|
|
await recordOwnStop(f, mode === "provider_alive" ? process.pid : 99_999_995);
|
|
await db.update(heartbeatRuns).set({ processPid: null, processGroupId: null, processStartedAt: null })
|
|
.where(eq(heartbeatRuns.id, f.sourceId));
|
|
const [source] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.sourceId));
|
|
expect(await verifyStoppedNativeSessionForContinuation(db, source)).toBeNull();
|
|
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" });
|
|
const receipt = await requestTaskRetry({ db, companyId: f.companyId, issueId: f.issueId, actorId: "board",
|
|
actorSource: "local_implicit", observedRunId: f.sourceId, idempotencyKey: randomUUID() });
|
|
await heartbeatService(db).resumeTaskRetry(f.companyId, receipt.requestId!);
|
|
const queued = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId), eq(heartbeatRuns.status, "queued")));
|
|
const [state] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, receipt.requestId!));
|
|
if (mode === "provider_alive") {
|
|
expect(queued).toHaveLength(0);
|
|
return;
|
|
}
|
|
expect(queued, JSON.stringify(state.payload)).toHaveLength(1);
|
|
expect(queued[0].contextSnapshot).toMatchObject({ previousRunId: f.sourceId, forceFreshSession: true });
|
|
} finally {
|
|
if (previous === undefined) delete process.env.PAPERCLIP_RUNNER_STATE_DIR; else process.env.PAPERCLIP_RUNNER_STATE_DIR = previous;
|
|
await rm(base, { recursive: true, force: true });
|
|
}
|
|
});
|
|
|
|
it.each(["stopped", "runner_alive"] as const)(
|
|
"retires a finished predecessor's failed-close gate before a follow-up session (%s)", async mode => {
|
|
// s9107 rerun t05: the run succeeded, its close kept an operator-required
|
|
// gate, and the follow-up run for a new message was refused at admission.
|
|
const base = await mkdtemp(join(tmpdir(), "predecessor-close-gate-"));
|
|
const previous = process.env.PAPERCLIP_RUNNER_STATE_DIR;
|
|
process.env.PAPERCLIP_RUNNER_STATE_DIR = base;
|
|
try {
|
|
const f = await seed(base);
|
|
await quarantineOwnClose(f, base);
|
|
const [source] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.sourceId));
|
|
// No stop record: the predecessor's recorded launch identity proves its stop.
|
|
const pid = mode === "runner_alive" ? process.pid : source.processPid!;
|
|
const startedAt = mode === "runner_alive" ? new Date((await currentProcessStartedAtMs())!) : source.processStartedAt!;
|
|
await db.update(heartbeatRuns).set({ status: "succeeded", errorCode: null, processPid: pid, processGroupId: pid, processStartedAt: startedAt })
|
|
.where(eq(heartbeatRuns.id, f.sourceId));
|
|
await appendHeartbeatRunEvent(db, { companyId: f.companyId, runId: f.sourceId, agentId: f.agentId, eventType: PROCESS_IDENTITY_RECORDED,
|
|
payload: { processPid: pid, processGroupId: pid, processStartedAt: startedAt.toISOString(),
|
|
targetKind: "local", hostname: hostname() } });
|
|
await appendHeartbeatRunEvent(db, { companyId: f.companyId, runId: f.sourceId, agentId: f.agentId, eventType: "harness.diagnostic",
|
|
payload: { prpEvent: { payload: { code: "acpx_process", details: { pid: 99_999_994, role: "sidecar", processGroupId: null } } } } });
|
|
await appendHeartbeatRunEvent(db, { companyId: f.companyId, runId: f.sourceId, agentId: f.agentId, eventType: "session.started",
|
|
payload: { prpEvent: { payload: { providerSessionId: "own-provider" } } } });
|
|
expect(await retireStoppedSessionCleanupGates(db, { companyId: f.companyId, issueId: f.issueId })).toBe(mode === "stopped");
|
|
expect(nativeSessionCleanupBlock({ companyId: f.companyId, issueId: f.issueId }))
|
|
.toBe(mode === "stopped" ? null : "operator_required");
|
|
} finally {
|
|
if (previous === undefined) delete process.env.PAPERCLIP_RUNNER_STATE_DIR; else process.env.PAPERCLIP_RUNNER_STATE_DIR = previous;
|
|
await rm(base, { recursive: true, force: true });
|
|
}
|
|
});
|
|
|
|
it("starts a task Retry for a gated follow-up that never launched once its predecessor is proven stopped", async () => {
|
|
// s9107 rerun t05: the gate is the finished owner's; the refused source
|
|
// launched nothing. The queue previews admission before committing it, so
|
|
// the preview must leave the gate for the committing call to retire.
|
|
const base = await mkdtemp(join(tmpdir(), "gated-follow-up-retry-"));
|
|
const previous = process.env.PAPERCLIP_RUNNER_STATE_DIR;
|
|
process.env.PAPERCLIP_RUNNER_STATE_DIR = base;
|
|
try {
|
|
const f = await seed(base);
|
|
await quarantineOwnClose(f, base, f.ownerId);
|
|
// The retained-state inventory proof no longer applies to this saved state.
|
|
await writeFile(join(f.root, "runner", "codex-provider-state.json"), JSON.stringify({
|
|
...f.files["runner/codex-provider-state.json"], lifecycle: "ready" }));
|
|
const [source] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.sourceId));
|
|
expect(await verifyStoppedNativeSessionForContinuation(db, source)).toBeNull();
|
|
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" });
|
|
const receipt = await requestTaskRetry({ db, companyId: f.companyId, issueId: f.issueId, actorId: "board",
|
|
actorSource: "local_implicit", observedRunId: f.sourceId, idempotencyKey: randomUUID() });
|
|
await heartbeatService(db).resumeTaskRetry(f.companyId, receipt.requestId!);
|
|
const queued = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId), eq(heartbeatRuns.status, "queued")));
|
|
const [state] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, receipt.requestId!));
|
|
expect(queued, JSON.stringify(state.payload)).toHaveLength(1);
|
|
expect(nativeSessionCleanupBlock({ companyId: f.companyId, issueId: f.issueId })).toBeNull();
|
|
} finally {
|
|
if (previous === undefined) delete process.env.PAPERCLIP_RUNNER_STATE_DIR; else process.env.PAPERCLIP_RUNNER_STATE_DIR = previous;
|
|
await rm(base, { recursive: true, force: true });
|
|
}
|
|
});
|
|
});
|