mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 21:34:48 +01:00
## Thinking Path > - Paperclip delegates remote Runner execution to environment plugins. > - A plugin can move a session to another task allocation. > - The Core controller can lose an ownership reply or restart before the Runner starts. > - The source and target need a durable record of the same request. > - This change binds restart and attachment to that request and the original unlaunched task lease. ## Linked Issues or Issue Description **What existing behavior does this improve?** Recovery of plugin Runner sessions on replacement compute. **Current behavior** Core cannot replay a session ownership request from saved lease state during startup. **Proposed behavior** Record the operation on both leases and check the matched ownership receipt before attaching. Preserve the original pre-admission lease across controller restart. Managed environment configuration is in [#21065](https://github.com/paperclipai/paperclip/pull/21065). Related warm sandbox cleanup is in [#20065](https://github.com/paperclipai/paperclip/pull/20065). This layer preserves the same-sandbox fallback from [#21001](https://github.com/paperclipai/paperclip/pull/21001). Workspace placement follows in [#20743](https://github.com/paperclipai/paperclip/pull/20743). ## What Changed - Add the optional `sessionHandoffVersion` capability and the ownership task operation. - Save matching ownership claims and fence source checkpoint access. Replay a pending request on the original target. - Restore a portable session using the durable claim and the current credentials for the same account. Start a fresh conversation when the provider reports an unavailable checkpoint, or when Core's persisted compatibility decision selects a different session after transfer. The latter requires an unlaunched target; subsequent recovery must match the fresh task's saved identity. Both paths retain the workspace transfer and source fencing. - Reuse the original unlaunched task lease after hard or graceful controller restart. After a move has attached, validate the surviving Runner through the saved claim rather than requiring its logical lease ID to equal the new physical lease. - Keep older retained leases without a saved account identity on the existing resume path. - Apply the existing supported MCP-only compatibility rule to warm recovery and predecessor accounting. Persist the capability discovered at admission and reuse it after restart without repeating discovery. - Keep predecessor-retirement fallback scoped to the same physical sandbox. A refusal after a portable move preserves the session instead of requesting a replacement that the plugin driver cannot enforce. ## Verification - On `1eb527be27`, server typecheck (including Runner build and SDK dependencies) passed. - Independent final verification passed 168 tests across portable source recovery, session resume, controller recovery, and task authority; two existing tests were skipped. The full portable source integration suite passed 64 tests. - The new integration follows confirmed ownership through an incompatible instruction change, persisted fresh admission, target-only prepare/auth/submit without the source checkpoint, and reattachment after controller replacement. It also rejects unpersisted session selection, ambiguous target leases, and incorrect saved identities. - Before this final change, independent verification passed 1,667 tests across executor, session resume, recovery usage, remote warm recovery, runtime selection, and runtime execution; the environment lifecycle suite passed 208 tests. - Existing regressions cover previous-release account metadata, supported and unsupported MCP changes, provider/instruction incompatibility, and admission capability reuse without another discovery call. - On `51d7eec373`, all four S3 multipart throttle tests passed independently. CI exposed an existing timing-sensitive test twice; its 20 ms overlap assumption now uses synchronized retries and an event-loop yield. Production upload code is unchanged. Temporary concurrency caps of one and eight both fail the retained exact-four assertion. - Exact SDK source-pin checks passed for both stack heads. The upper #20743 code patch remains unchanged; its documentation preserves both layers' behavior. ## Risks - The handoff protocol is opt-in. Existing plugin leases without portable ownership metadata keep the previous release's resume path. Live attachment still requires end-to-end qualification. An unverified claim blocks startup. Git workspaces prepared on a portable allocation also need the workspace placement layer. ## Model Used - OpenAI Codex GPT-6 with GPT-6.1 Sol agents for implementation and independent review. ## Checklist - [x] Thinking path and model are included. - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] The issue is described using the enhancement template. - [x] Local checks passed and new behavior has tests. - [x] All Paperclip CI gates are green. - [x] Greptile is 5/5 with no open findings. --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
1087 lines
77 KiB
TypeScript
1087 lines
77 KiB
TypeScript
import { randomUUID } from "node:crypto";
|
|
import { mkdir, mkdtemp, readFile, rm, stat, writeFile } from "node:fs/promises";
|
|
import { execFile as execFileCallback } from "node:child_process";
|
|
import { promisify } from "node:util";
|
|
import { tmpdir } from "node:os";
|
|
import { join } from "node:path";
|
|
import { activityLog, agents, authUsers, companyMemberships, companySkillRepositorySnapshots, principalPermissionGrants, environments, environmentLeases, companies, completionContracts, costEvents, createDb, applyPendingMigrations, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, executionWorkspaces, heartbeatRuns, issues, nativeRunFinalizations, workspaceOperations, type Db } from "@paperclipai/db";
|
|
import { accountRunCost } from "../run-cost-accounting.js";
|
|
import { eq, sql } from "drizzle-orm";
|
|
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
|
|
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
|
|
import { resolveExternalTestDatabaseUrl } from "../../__tests__/helpers/external-test-database.js";
|
|
import { CHAT_CONTROL_RECOVERY_ADMISSION_KEY } from "../chat-control-recovery-stop.js";
|
|
import { buildNativeRuntimeContext, materializeSelectedSkills, rehydratePinnedSkillAssets } from "../native-runtime/runtime-context.js";
|
|
import { applyConnectorSkills } from "../connector-runtime.js";
|
|
import * as storageModule from "../../storage/index.js";
|
|
import { createStorageService } from "../../storage/service.js";
|
|
import { createLocalDiskStorageProvider } from "../../storage/local-disk-provider.js";
|
|
import { readRuntimeSkillPreservation } from "../native-runtime/runtime-skill-preservation-reference.js";
|
|
import { hasLocalRuntimeSkillObligations } from "../native-runtime/runtime-skill-preservation.js";
|
|
import * as runtimeSkillPreservation from "../native-runtime/runtime-skill-preservation.js";
|
|
import { removeRuntimeAssetTree } from "../native-runtime/runtime-asset-retention.js";
|
|
import { nativeRuntimeContextFixture } from "../native-runtime/runtime-context.test-fixture.js";
|
|
import { prepareNativeWorkspaceSync } from "../native-runtime/native-workspace-sync.js";
|
|
import { resolveHeartbeatNativeRuntimeMode } from "../native-runtime/runtime-mode.js";
|
|
import { claimNativeRestartRecoveries } from "../native-runtime/native-restart-recovery.js";
|
|
import { buildNativeModelEnvelope, canonicalNativeRuntimeContextDigest } from "../../vendor/paperclip-runner/index.js";
|
|
import * as runner from "../../vendor/paperclip-runner/index.js";
|
|
import { prepareAndSelectHeartbeatRuntime, selectHeartbeatRuntime, type HeartbeatRuntimeSelectionInput } from "./runtime-selection.js";
|
|
import { resolveRunnerEnvironmentForRun, resolveWarmRunnerIdleTimeoutMsForRun } from "../runner-environment-lifecycle.js";
|
|
import { reserveRunBudget } from "../budget-reservations.js";
|
|
import { legacyControllerClaim } from "../legacy-controller-lease.js";
|
|
import { agentInstructionWorkingCopyService } from "../agent-instruction-working-copies.js";
|
|
import { createHeartbeatInstructionWorkingCopy } from "./instruction-working-copy.js";
|
|
import { environmentService } from "../environments.js";
|
|
|
|
// Keep the real contract, session, and runtime-selection transactions. Replace only
|
|
// filesystem materialization and sandbox synchronization at their external boundaries.
|
|
vi.mock("../native-runtime/runtime-context.js", async (original) => ({
|
|
...await original<typeof import("../native-runtime/runtime-context.js")>(),
|
|
buildNativeRuntimeContext: vi.fn(async () => nativeRuntimeContextFixture()),
|
|
}));
|
|
vi.mock("../native-runtime/native-workspace-sync.js", async (original) => ({
|
|
...await original<typeof import("../native-runtime/native-workspace-sync.js")>(),
|
|
prepareNativeWorkspaceSync: vi.fn(async () => null),
|
|
}));
|
|
|
|
const externalDatabaseUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_RUNTIME_SELECTION_TEST_DATABASE_URL");
|
|
const support = externalDatabaseUrl ? { supported: true } : await getEmbeddedPostgresTestSupport();
|
|
describe.skipIf(!support.supported)("heartbeat runtime selection boundary", () => {
|
|
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
|
let db: Db;
|
|
let instructionRoot: string;
|
|
beforeAll(async () => {
|
|
instructionRoot = await mkdtemp(join(tmpdir(), "paperclip-selection-instructions-"));
|
|
await writeFile(join(instructionRoot, "AGENTS.md"), "Complete the assigned task.\n");
|
|
if (externalDatabaseUrl) await applyPendingMigrations(externalDatabaseUrl);
|
|
database = externalDatabaseUrl ? { connectionString: externalDatabaseUrl, cleanup: async () => {} }
|
|
: await startEmbeddedPostgresTestDatabase("paperclip-runtime-selection");
|
|
db = createDb(database.connectionString);
|
|
await db.execute(sql`set client_min_messages = warning`);
|
|
}, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS);
|
|
afterAll(async () => { await db?.$client.end(); await database?.cleanup(); if (instructionRoot) await rm(instructionRoot, { recursive: true, force: true }); });
|
|
afterEach(() => vi.unstubAllEnvs());
|
|
beforeEach(async () => {
|
|
await db.execute(sql`truncate table companies restart identity cascade`);
|
|
vi.clearAllMocks();
|
|
vi.mocked(prepareNativeWorkspaceSync).mockResolvedValue(null);
|
|
vi.mocked(buildNativeRuntimeContext).mockImplementation(async () => {
|
|
const context = nativeRuntimeContextFixture();
|
|
context.instructions.bundle.rootPath = instructionRoot;
|
|
return { ...context, aggregateDigest: canonicalNativeRuntimeContextDigest(context) };
|
|
});
|
|
});
|
|
|
|
async function fixture(adapterType = "paperclip_runner") {
|
|
const [company] = await db.insert(companies).values({ name: "Runtime selection", issuePrefix: "SEL" }).returning();
|
|
const [agent] = await db.insert(agents).values({ companyId: company.id, name: "Selecting agent", status: "running", adapterType, adapterConfig: { provider: "codex", model: "gpt-5.6-luna" } }).returning();
|
|
const [issue] = await db.insert(issues).values({ companyId: company.id, title: "Complete this task", description: "Preserve the requested work", assigneeAgentId: agent.id }).returning();
|
|
const [run] = await db.insert(heartbeatRuns).values({ companyId: company.id, agentId: agent.id, issueId: issue.id, scopeKind: "issue", status: "running" }).returning();
|
|
const now = new Date();
|
|
const input: HeartbeatRuntimeSelectionInput = {
|
|
run, agent,
|
|
nativeRuntimeResolution: resolveHeartbeatNativeRuntimeMode({ persisted: run, enabled: true, runtimeConfig: {}, adapterConfig: agent.adapterConfig, agent, issue, target: null, workspaceId: null }),
|
|
task: { issueRef: issue, issueContext: null, context: { paperclipTaskMarkdown: "Complete this task" }, executionContinuation: null, safeWakeComments: [], safeWakeCommentContext: null, paperclipWakePayload: null, acceptedPlanContinuationWake: false },
|
|
session: { persistedNativeExecutionInput: null, persistedRunnerProfile: {}, taskResumeSessionId: null, isFailedChatRunRetry: false, getFreshSessionHandoff: vi.fn(async () => "Earlier context") },
|
|
workspace: { persistedExecutionWorkspace: null, executionWorkspace: { cwd: "/tmp/paperclip-runtime-selection", repoUrl: null, repoRef: null, branchName: null }, executionTarget: null, isDotRun: false, projectRepositoryPaths: [],
|
|
activeEnvironmentLease: { lease: { id: randomUUID(), companyId: company.id, environmentId: null, executionWorkspaceId: null, issueId: issue.id, heartbeatRunId: run.id, status: "active", leasePolicy: "ephemeral", provider: null, providerLeaseId: null, acquiredAt: now, lastUsedAt: now, expiresAt: null, releasedAt: null, failureReason: null, cleanupStatus: null, metadata: null, createdAt: now, updatedAt: now } },
|
|
},
|
|
config: { runtimeConfig: {}, runtimeSkillEntries: [], runScopedMentionedSkillKeys: [], instructionCopy: null, agentIdentity: undefined, issueAssigneeOverrides: null, resolvedConfig: {}, managedAiRuntime: undefined },
|
|
recovery: {}, trace: { providerTraceRequested: false, providerTraceCapture: null },
|
|
issuesSvc: { addComment: vi.fn() }, runtimeEnv: {}, onAdapterEvent: vi.fn(async () => {}),
|
|
onNativeLifecycleSelected: vi.fn(), onProviderResourceDisposition: vi.fn(), onNativeOwnershipHeld: vi.fn(), stopControllerLease: vi.fn(),
|
|
};
|
|
return { input, issue, run, agent };
|
|
}
|
|
const read = (id: string) => db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, id)).then(rows => rows[0]!);
|
|
const coordinators = (id: string) => db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, id));
|
|
const sandbox = { kind: "remote" as const, transport: "sandbox" as const, remoteCwd: "/workspace", reusableLeaseConfigured: false };
|
|
|
|
it.each(["uninspected", "confirmed", "replaced"])("selects an independent native session after %s disk-only recovery", async placement => {
|
|
const { input, run, issue } = await fixture();
|
|
const priorSession = randomUUID();
|
|
const lease = input.workspace.activeEnvironmentLease!.lease;
|
|
const description = { kind: "description" as const, taskId: randomUUID(), runner: { version: "test", digest: `sha256:${"a".repeat(64)}` } };
|
|
const previousDescription = placement === "replaced" ? { ...description, taskId: randomUUID() } : description;
|
|
const pending = { ...lease, providerLeaseId: description.taskId, metadata: { portableTaskPreviousLeaseId: randomUUID(),
|
|
portableTaskPredecessor: { operation: { identity: { normalizedSessionId: priorSession } } },
|
|
} };
|
|
const ready = { ...pending.metadata, portableTaskOwnershipReady: true, portableTaskCheckpointUnavailable: true };
|
|
await db.insert(environmentLeases).values(placement === "confirmed" ? { ...pending, metadata: ready } : pending);
|
|
input.workspace.activeEnvironmentLease!.lease = pending;
|
|
if (placement !== "uninspected") input.workspace.activeEnvironmentLease!.pluginTaskDescription = previousDescription;
|
|
const taskRpc = vi.fn(async ({ kind }: { kind: string }) => {
|
|
expect(kind).toBe("describe");
|
|
await db.update(environmentLeases).set({
|
|
metadata: ready,
|
|
}).where(eq(environmentLeases.id, lease.id));
|
|
return description;
|
|
});
|
|
input.workspace.executionTarget = { ...sandbox, leaseId: lease.id, runnerTask: taskRpc,
|
|
reusableLeaseConfigured: true, effectiveCapabilities: { reusableLeases: true },
|
|
runnerLifecyclePolicy: { mode: "warm", idleTimeoutMs: 300_000 } } as never;
|
|
input.session.taskResumeSessionId = priorSession;
|
|
const network = vi.spyOn(globalThis, "fetch").mockRejectedValue(new Error("Cloud is unavailable"));
|
|
try {
|
|
const selected = await selectHeartbeatRuntime(db, input);
|
|
expect(selected.selected).toBe(true);
|
|
if (!selected.selected || !selected.nativeExecution) throw new Error("Expected native selection");
|
|
const saved = await read(run.id);
|
|
expect(saved.nativeSessionId).toBeTruthy();
|
|
expect(saved.nativeSessionId).not.toBe(priorSession);
|
|
expect(await selected.getNativeFreshSessionHandoff?.()).toBe("Earlier context");
|
|
expect((await db.select().from(issues).where(eq(issues.id, issue.id)))[0]).toEqual(issue);
|
|
expect(selected.pluginTaskDescription).toEqual(description);
|
|
expect(taskRpc).toHaveBeenCalledTimes(placement === "confirmed" ? 0 : 1);
|
|
expect(network).not.toHaveBeenCalled();
|
|
|
|
// A lost reply after committing selection must allow the same replacement run to retry.
|
|
input.run = saved;
|
|
input.session.persistedRunnerProfile = saved.runnerProfileJson ?? {};
|
|
input.session.persistedNativeExecutionInput = selected.nativeExecution;
|
|
await expect(selectHeartbeatRuntime(db, input)).resolves.toMatchObject({
|
|
selected: true, nativeExecution: selected.nativeExecution,
|
|
});
|
|
input.run = { ...saved, nativeSessionId: priorSession };
|
|
await expect(selectHeartbeatRuntime(db, input)).rejects.toThrow("native_portable_reseed_session_already_attached");
|
|
} finally {
|
|
network.mockRestore();
|
|
}
|
|
});
|
|
|
|
it("rejects invalid local Runner configuration before native authority and retains the preparation owner", async () => {
|
|
const f = await fixture();
|
|
await claimPreparation(f);
|
|
vi.stubEnv("PAPERCLIP_RUNNER_BINARY", "relative-invalid-dot-retry-controller");
|
|
f.input.runtimeEnv = { PAPERCLIP_RUNNER_BINARY: join(tmpdir(), "ignored-injected-runnerd") };
|
|
const before = await read(f.run.id);
|
|
await expect(selectHeartbeatRuntime(db, f.input)).rejects.toThrow("PAPERCLIP_RUNNER_BINARY must be an absolute path");
|
|
expect(await read(f.run.id)).toEqual(before);
|
|
expect(await coordinators(f.run.id)).toEqual([]);
|
|
expect(await db.select().from(completionContracts)).toHaveLength(0);
|
|
expect(f.input.stopControllerLease).not.toHaveBeenCalled();
|
|
expect(buildNativeRuntimeContext).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it("ignores an injected runtime Runner value that the local backend does not consume", async () => {
|
|
const f = await fixture();
|
|
vi.stubEnv("PAPERCLIP_RUNNER_BINARY", join(tmpdir(), "configuration-only-controller-runnerd"));
|
|
f.input.runtimeEnv = { PAPERCLIP_RUNNER_BINARY: "relative-ignored-injected-runnerd" };
|
|
expect((await selectHeartbeatRuntime(db, f.input)).selected).toBe(true);
|
|
expect((await read(f.run.id)).runtimeMode).toBe("native");
|
|
expect(await coordinators(f.run.id)).toHaveLength(1);
|
|
});
|
|
|
|
it("does not apply local Runner configuration to remote native selection", async () => {
|
|
const f = await fixture();
|
|
vi.stubEnv("PAPERCLIP_RUNNER_BINARY", "relative-unused-remote-controller");
|
|
f.input.workspace.executionTarget = sandbox;
|
|
expect((await selectHeartbeatRuntime(db, f.input)).selected).toBe(true);
|
|
expect((await read(f.run.id)).runtimeMode).toBe("native");
|
|
expect(await coordinators(f.run.id)).toHaveLength(1);
|
|
});
|
|
|
|
it("preserves immutable native recovery when current local Runner configuration is invalid", async () => {
|
|
const f = await fixture();
|
|
const selected = await selectHeartbeatRuntime(db, f.input);
|
|
if (!selected.selected || !selected.nativeExecution) throw new Error("Expected native selection");
|
|
const saved = await read(f.run.id);
|
|
f.input.run = saved;
|
|
f.input.session.persistedRunnerProfile = saved.runnerProfileJson ?? {};
|
|
f.input.session.persistedNativeExecutionInput = selected.nativeExecution;
|
|
vi.stubEnv("PAPERCLIP_RUNNER_BINARY", "relative-unused-recovery-controller");
|
|
const resumed = await selectHeartbeatRuntime(db, f.input);
|
|
expect(resumed).toMatchObject({ selected: true, nativeExecution: selected.nativeExecution, nativeRunnerInstanceId: selected.nativeRunnerInstanceId });
|
|
expect((await read(f.run.id)).runnerProfileJson).toEqual(saved.runnerProfileJson);
|
|
expect(await coordinators(f.run.id)).toHaveLength(1);
|
|
});
|
|
|
|
async function connectorFixture(durable = true) {
|
|
const f = await fixture(), { input } = f;
|
|
const home = await mkdtemp(join(tmpdir(), "selection-connector-assets-"));
|
|
vi.stubEnv("PAPERCLIP_HOME", home);
|
|
const config = await applyConnectorSkills({}, [], [{ key: "slack", label: "Slack", skillKey: "paperclipai/paperclip/slack",
|
|
resources: [{ id: "synthetic-channel", label: "Fixture channel", connectionId: randomUUID() }],
|
|
tools: [{ name: "fixture_read", description: "Read synthetic messages", inputSchema: { type: "object" } }] }]);
|
|
input.config.runtimeConfig = config;
|
|
input.config.runtimeSkillEntries = config.paperclipRuntimeSkills;
|
|
vi.mocked(buildNativeRuntimeContext).mockImplementation(async () => {
|
|
const context = nativeRuntimeContextFixture();
|
|
context.instructions.bundle.rootPath = instructionRoot;
|
|
context.skills = await materializeSelectedSkills(config, config.paperclipRuntimeSkills, true);
|
|
return { ...context, aggregateDigest: canonicalNativeRuntimeContextDigest(context) };
|
|
});
|
|
const provider = createLocalDiskStorageProvider(join(home, "objects"));
|
|
const storage = createStorageService(durable ? { ...provider, id: "s3" } : provider);
|
|
const storageSpy = vi.spyOn(storageModule, "getStorageService").mockReturnValue(storage);
|
|
return { ...f, home, storage, cleanup: async () => { storageSpy.mockRestore(); await removeRuntimeAssetTree(home); } };
|
|
}
|
|
|
|
it("publishes a dynamic connector skill receipt with admission and rehydrates its exact bytes on a cold host", async () => {
|
|
const { input, run, agent, home, storage, cleanup } = await connectorFixture();
|
|
let admitting = true;
|
|
const getObject = storage.getObject.bind(storage);
|
|
const readback = vi.spyOn(storage, "getObject").mockImplementation(async (...args) => {
|
|
if (admitting) {
|
|
expect((await read(run.id)).runnerProfileJson ?? {}).not.toHaveProperty("nativeExecutionInput");
|
|
expect(await coordinators(run.id)).toEqual([]);
|
|
}
|
|
return getObject(...args);
|
|
});
|
|
try {
|
|
const selected = await selectHeartbeatRuntime(db, input);
|
|
admitting = false;
|
|
if (!selected.selected || !selected.nativeExecution || !("runtimeContext" in selected.nativeExecution)) throw new Error("Expected native selection");
|
|
const saved = await read(run.id);
|
|
expect(readRuntimeSkillPreservation(saved)).not.toBeNull();
|
|
expect(await hasLocalRuntimeSkillObligations(db)).toBe(false);
|
|
const original = selected.nativeExecution.runtimeContext.skills[0]!;
|
|
expect(original.versionId).toBeNull();
|
|
const files = await Promise.all(["SKILL.md", "TOOLS.json"].map(async name => ({ name, bytes: await readFile(join(original.bundle.rootPath, name)) })));
|
|
vi.stubEnv("PAPERCLIP_HOME", join(home, "cold"));
|
|
const recovered = await rehydratePinnedSkillAssets(db, { companyId: agent.companyId, agentId: agent.id, runId: run.id }, selected.nativeExecution.runtimeContext);
|
|
expect(recovered.skills[0]!.versionId).toBeNull();
|
|
expect(recovered.skills[0]!.bundle.digest).toBe(original.bundle.digest);
|
|
expect(recovered.skills[0]!.bundle.rootPath).not.toBe(original.bundle.rootPath);
|
|
for (const file of files) {
|
|
expect(await readFile(join(recovered.skills[0]!.bundle.rootPath, file.name))).toEqual(file.bytes);
|
|
expect((await stat(join(recovered.skills[0]!.bundle.rootPath, file.name))).mode & 0o777).toBe(0o444);
|
|
}
|
|
readback.mockRestore();
|
|
await removeRuntimeAssetTree(original.bundle.rootPath);
|
|
const outage = vi.spyOn(storage, "getObject").mockRejectedValue(new Error("Synthetic storage outage"));
|
|
try {
|
|
const staged = await runtimeSkillPreservation.stageNativeRuntimeSkillAssets(db, { companyId: agent.companyId, agentId: agent.id,
|
|
runId: run.id, issueId: selected.nativeExecution.binding.issueId, execution: selected.nativeExecution }, storage);
|
|
expect(staged?.receipt).toEqual(readRuntimeSkillPreservation(saved));
|
|
expect(outage).not.toHaveBeenCalled();
|
|
} finally { outage.mockRestore(); }
|
|
expect(await coordinators(run.id)).toMatchObject([{ phase: "observed", leaseOwner: null }]);
|
|
} finally { readback.mockRestore(); await cleanup(); }
|
|
});
|
|
|
|
it.each(["storage failure", "cancelled", "changed binding", "retired package"])("keeps dynamic skill admission unpublished after %s", async failure => {
|
|
const { input, run, agent, storage, cleanup } = await connectorFixture();
|
|
const stage = vi.spyOn(runtimeSkillPreservation, "stageNativeRuntimeSkillAssets");
|
|
const putFile = storage.putFile.bind(storage);
|
|
const put = ["storage failure", "cancelled"].includes(failure) ? vi.spyOn(storage, "putFile").mockImplementation(async data => {
|
|
const result = await putFile(data);
|
|
if (failure === "storage failure") throw new Error("Synthetic storage failure");
|
|
await db.update(heartbeatRuns).set({ status: "cancelled" }).where(eq(heartbeatRuns.id, run.id));
|
|
return result;
|
|
}) : null;
|
|
input.beforeNativeRuntimeSelection = async () => {
|
|
if (failure === "changed binding") {
|
|
const execution = structuredClone(stage.mock.calls[0]![1].execution);
|
|
execution.binding.agentId = randomUUID();
|
|
await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeExecutionInput: execution } }).where(eq(heartbeatRuns.id, run.id));
|
|
}
|
|
if (failure === "retired package") await db.update(companySkillRepositorySnapshots).set({ storage: null })
|
|
.where(eq(companySkillRepositorySnapshots.companyId, agent.companyId));
|
|
};
|
|
try {
|
|
if (failure === "cancelled") expect(await selectHeartbeatRuntime(db, input)).toEqual({ selected: false });
|
|
else await expect(selectHeartbeatRuntime(db, input)).rejects.toThrow(failure === "storage failure" ? "Synthetic storage failure"
|
|
: failure === "changed binding" ? "binding_changed" : "package_not_durable");
|
|
const saved = await read(run.id);
|
|
expect(saved.runnerProfileJson ?? {}).not.toHaveProperty("nativeRuntimeSkillPreservation");
|
|
if (failure !== "changed binding") expect(saved.runnerProfileJson ?? {}).not.toHaveProperty("nativeExecutionInput");
|
|
else expect((saved.runnerProfileJson!.nativeExecutionInput as { binding: { agentId: string } }).binding.agentId).not.toBe(agent.id);
|
|
expect(await coordinators(run.id)).toEqual([]);
|
|
expect(prepareNativeWorkspaceSync).not.toHaveBeenCalled();
|
|
} finally { put?.mockRestore(); stage.mockRestore(); await cleanup(); }
|
|
});
|
|
|
|
it("keeps volume-backed local connector admission independent of object storage", async () => {
|
|
const { input, run, storage, cleanup } = await connectorFixture(false);
|
|
const put = vi.spyOn(storage, "putFile").mockRejectedValue(new Error("Local mode must not upload"));
|
|
try {
|
|
expect((await selectHeartbeatRuntime(db, input)).selected).toBe(true);
|
|
expect((await read(run.id)).runnerProfileJson).not.toHaveProperty("nativeRuntimeSkillPreservation");
|
|
expect(put).not.toHaveBeenCalled();
|
|
expect(await hasLocalRuntimeSkillObligations(db)).toBe(true);
|
|
} finally { put.mockRestore(); await cleanup(); }
|
|
});
|
|
|
|
it("preserves preparation failure and skips runtime selection", async () => {
|
|
const { input, run } = await fixture();
|
|
const initiating = new Error("Synthetic preparation failure");
|
|
const phases: string[] = [];
|
|
await expect(prepareAndSelectHeartbeatRuntime(db, input, {
|
|
measure: async (phase, work) => { phases.push(phase); return work(); },
|
|
prepareInstructions: async () => { throw initiating; },
|
|
})).rejects.toBe(initiating);
|
|
expect(phases).toEqual(["instructions"]);
|
|
expect(await coordinators(run.id)).toEqual([]);
|
|
expect(buildNativeRuntimeContext).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it.each([false, true])("checkpoints the real instruction copy under the selected agent-default warm policy (changed: %s)", async changed => {
|
|
const { input, issue, agent, run } = await fixture();
|
|
const userId = randomUUID(), environmentId = randomUUID(), leaseId = randomUUID();
|
|
vi.stubEnv("PAPERCLIP_HOME", instructionRoot);
|
|
const { resolveManagedInstructionsRoot } = await import("../agent-instructions.js");
|
|
const canonical = resolveManagedInstructionsRoot(agent), remoteCwd = join(instructionRoot, randomUUID(), "task");
|
|
await mkdir(canonical, { recursive: true }); await mkdir(remoteCwd, { recursive: true });
|
|
await writeFile(join(canonical, "AGENTS.md"), "Synthetic instructions for this local fixture only.\n");
|
|
const execute = promisify(execFileCallback);
|
|
await execute("git", ["init", remoteCwd]);
|
|
const target = { kind: "remote" as const, transport: "sandbox" as const, environmentId, leaseId, remoteCwd,
|
|
runnerLifecyclePolicy: null, reusableLeaseConfigured: true, effectiveCapabilities: { reusableLeases: true } as never,
|
|
runner: { execute: async (command: { command: string; args?: string[]; cwd?: string; env?: Record<string,string>; stdin?: string; timeoutMs?: number }) => {
|
|
const args = [...(command.args ?? [])], env = { PATH: process.env.PATH, ...command.env };
|
|
if (command.stdin != null && (args[0] === "-c" || args[0] === "-lc")) {
|
|
Object.assign(env, { PAPERCLIP_TEST_STDIN: command.stdin });
|
|
args[1] = `printf '%s' "$PAPERCLIP_TEST_STDIN" | (${args[1]})`;
|
|
}
|
|
const result = await execute(command.command, args, { cwd: command.cwd, env, timeout: command.timeoutMs, maxBuffer: 32 * 1024 * 1024 });
|
|
return { exitCode: 0, signal: null, timedOut: false, ...result, pid: null, startedAt: new Date().toISOString() };
|
|
} } };
|
|
await db.insert(authUsers).values({ id: userId, name: "Fixture editor", email: `${userId}@example.test`, createdAt: new Date(), updatedAt: new Date() });
|
|
await db.insert(companyMemberships).values([{ companyId: agent.companyId, principalType: "user", principalId: userId, membershipRole: "operator" },
|
|
{ companyId: agent.companyId, principalType: "agent", principalId: agent.id, membershipRole: "member" }]);
|
|
await db.insert(principalPermissionGrants).values({ companyId: agent.companyId, principalType: "user", principalId: userId, permissionKey: "agents:configure", scope: { agentIds: [agent.id] } });
|
|
await db.insert(environments).values({ id: environmentId, name: `Loopback sandbox ${environmentId}`, driver: "sandbox" });
|
|
const [lease] = await db.insert(environmentLeases).values({ id: leaseId, companyId: agent.companyId, environmentId, heartbeatRunId: run.id,
|
|
provider: "fixture", providerLeaseId: "local-loopback", leasePolicy: "reuse_by_environment" }).returning();
|
|
input.agent = { ...agent, adapterConfig: { ...agent.adapterConfig, lifecycleMode: "warm", idleTimeoutMs: 900_000,
|
|
instructionsBundleMode: "managed", instructionsRootPath: canonical, instructionsEntryFile: "AGENTS.md" } };
|
|
await db.update(agents).set({ adapterConfig: input.agent.adapterConfig }).where(eq(agents.id, agent.id));
|
|
await db.update(heartbeatRuns).set({ responsibleUserId: userId }).where(eq(heartbeatRuns.id, run.id));
|
|
input.run = await read(run.id);
|
|
input.workspace.executionTarget = target;
|
|
input.workspace.executionWorkspace.cwd = remoteCwd;
|
|
input.workspace.activeEnvironmentLease = { lease: (await environmentService(db).getLeaseById(lease.id))! };
|
|
input.nativeRuntimeResolution = resolveHeartbeatNativeRuntimeMode({ persisted: input.run, enabled: true, runtimeConfig: {}, adapterConfig: input.agent.adapterConfig,
|
|
agent: input.agent, issue, target, workspaceId: null });
|
|
expect(input.nativeRuntimeResolution.kind).toBe("native");
|
|
const { agentInstructionsBundleMode } = await import("../agent-instructions.js");
|
|
expect(agentInstructionsBundleMode(input.agent)).toBe("managed");
|
|
const copies = agentInstructionWorkingCopyService(db);
|
|
const instructionEvents = vi.fn(async () => {});
|
|
const controller = createHeartbeatInstructionWorkingCopy({ db, instructionCopies: copies, readRun: () => input.run, agent: input.agent,
|
|
context: input.task.context, session: { taskSession: null, taskSessionForRun: null, sessionConfigMetadata: {} as never },
|
|
workspace: input.workspace, appendRunEvent: instructionEvents });
|
|
const phases: string[] = [];
|
|
const selected = await prepareAndSelectHeartbeatRuntime(db, input, {
|
|
measure: async (phase, work) => { phases.push(phase); return work(); },
|
|
prepareInstructions: async nativeLifecyclePolicy => {
|
|
const runtimeConfig = await controller.prepare({ runtimeConfig: {}, nativeRuntimeResolution: input.nativeRuntimeResolution, hasInstructionFilesystem: true, nativeLifecyclePolicy });
|
|
return { runtimeConfig, instructionCopy: controller.instructionCopy };
|
|
},
|
|
});
|
|
expect(phases).toEqual(["instructions", "runtime_selection"]);
|
|
expect(instructionEvents.mock.calls).toEqual([]);
|
|
expect(controller.instructionCopy).not.toBeNull();
|
|
if (!selected.selected || !selected.nativeExecution) throw new Error("Expected native selection");
|
|
expect(selected.nativeExecution.session.lifecyclePolicy).toEqual({ mode: "warm", idleTimeoutMs: 900_000 });
|
|
expect((await copies.get(agent.companyId, run.id))?.receipt?.warm).toBe(true);
|
|
if (changed) await writeFile(join(controller.instructionCopy!.executionRoot, "memory.txt"), "Synthetic changed memory.\n");
|
|
expect(await controller.nativeInstructionWorkingCopy()!.checkpointWarm!()).toBe(true);
|
|
expect((await copies.get(agent.companyId, run.id))?.state).toBe("warm_saved");
|
|
expect(instructionEvents).toHaveBeenCalledWith(expect.anything(), expect.objectContaining({ eventType: "instruction_save",
|
|
payload: expect.objectContaining({ state: changed ? "saved" : "unchanged", contract: "agent_files", errorCode: null }) }));
|
|
if (changed) expect(await readFile(join(canonical, "memory.txt"), "utf8")).toBe("Synthetic changed memory.\n");
|
|
}, 45_000);
|
|
|
|
it.each([true, false])("records which earlier run a follow-up's process identity came from (prior identity: %s)", async priorIdentity => {
|
|
const { input, run, agent, issue } = await fixture();
|
|
const sessionId = randomUUID();
|
|
const prior = await selectHeartbeatRuntime(db, input);
|
|
if (!prior.selected || !prior.nativeExecution) throw new Error("Expected prior native selection");
|
|
const priorInput = { ...prior.nativeExecution, session: { ...prior.nativeExecution.session, normalizedSessionId: sessionId } };
|
|
const priorProfile = (await read(run.id)).runnerProfileJson;
|
|
const processStartedAt = new Date(Date.now() - 60_000);
|
|
await db.update(heartbeatRuns).set({ status: "succeeded", nativeSessionId: sessionId,
|
|
...(priorIdentity ? { processPid: 11_356, processGroupId: 11_356, processStartedAt } : {}),
|
|
runnerProfileJson: { ...priorProfile, nativeExecutionInput: priorInput, sessionCheckpoint: {
|
|
backendKind: "runner", driverKind: "codex_app_server", sessionId: "retained-provider-thread",
|
|
providerSessionId: "retained-provider-thread", identity: { ...priorInput.binding, sessionId },
|
|
} },
|
|
}).where(eq(heartbeatRuns.id, run.id));
|
|
const [nextRun] = await db.insert(heartbeatRuns).values({ companyId: agent.companyId, agentId: agent.id,
|
|
issueId: issue.id, scopeKind: "issue", status: "running", nativeSessionId: sessionId,
|
|
createdAt: new Date(run.createdAt.getTime() + 1000),
|
|
}).returning();
|
|
input.run = nextRun!;
|
|
const selected = await selectHeartbeatRuntime(db, input);
|
|
if (!selected.selected) throw new Error("Expected follow-up native selection");
|
|
const saved = await read(nextRun!.id);
|
|
if (priorIdentity) {
|
|
expect(saved).toMatchObject({ processPid: 11_356, processGroupId: 11_356 });
|
|
expect(saved.runnerProfileJson?.inheritedProcessIdentityRunId).toBe(run.id);
|
|
// Settled-process cleanup clears the predecessor before the follow-up's
|
|
// accounting runs; the follow-up was refused before it launched.
|
|
await db.update(heartbeatRuns).set({ processPid: null, processGroupId: null, processStartedAt: null })
|
|
.where(eq(heartbeatRuns.id, run.id));
|
|
const sourceId = randomUUID();
|
|
await db.update(heartbeatRuns).set({ status: "failed", finishedAt: new Date(), costAccountingPending: true,
|
|
errorCode: "native_session_cleanup_quarantined",
|
|
usageJson: { accountingReceiptSourceId: sourceId, accountingReceiptSequence: 0, accountingReceiptReady: false,
|
|
accountingReceiptUnavailable: true, accountingReceiptSourceFences: {},
|
|
ledgerScope: { issueId: issue.id, projectId: null, billingCode: null } } })
|
|
.where(eq(heartbeatRuns.id, nextRun!.id));
|
|
await db.update(nativeRunFinalizations).set({ phase: "terminal_failure", failureCode: "native_session_cleanup_quarantined" })
|
|
.where(eq(nativeRunFinalizations.runId, nextRun!.id));
|
|
expect(await accountRunCost(db, nextRun!.id)).toBe(true);
|
|
expect(await db.select().from(costEvents).where(eq(costEvents.heartbeatRunId, nextRun!.id))).toEqual([]);
|
|
} else {
|
|
expect(saved.processPid).toBeNull();
|
|
expect(saved.runnerProfileJson?.inheritedProcessIdentityRunId).toBeUndefined();
|
|
}
|
|
});
|
|
|
|
it.each(["matching", "different-session", "absent"])("persists the migrated logical cwd only for the receipt's exact session: %s", async receipt => {
|
|
const sameSession = receipt === "matching";
|
|
const { input, run, agent, issue } = await fixture();
|
|
const sessionId = randomUUID(), operationId = randomUUID();
|
|
const logicalCwd = "/paperclip/legacy/task-workspace";
|
|
const marker = { schema: "paperclip.native-workspace-migration.v1", operationId };
|
|
const [workspace] = await db.insert(executionWorkspaces).values({
|
|
companyId: agent.companyId, sourceIssueId: issue.id, name: "Migrated workspace",
|
|
mode: "shared_workspace", strategyType: "project_primary", cwd: "/tmp/ephemeral/task-workspace",
|
|
providerType: "cloud_sandbox", metadata: receipt === "absent" ? {} : { nativeWorkspaceMigration: marker },
|
|
}).returning();
|
|
await db.insert(workspaceOperations).values({
|
|
id: operationId, companyId: agent.companyId, executionWorkspaceId: workspace!.id,
|
|
phase: "remote_workspace_migration", status: "succeeded", metadata: {
|
|
schema: "paperclip.remote-workspace-migration.v1", snapshotId: randomUUID(),
|
|
nativeSessionContinuity: { schema: marker.schema, remoteCwd: sandbox.remoteCwd, sessions: [{
|
|
sourceRunId: randomUUID(), agentId: agent.id, issueId: issue.id,
|
|
sessionId: sameSession ? sessionId : randomUUID(), logicalCwd,
|
|
}] },
|
|
},
|
|
});
|
|
input.workspace.persistedExecutionWorkspace = workspace!;
|
|
input.workspace.executionWorkspace.cwd = logicalCwd;
|
|
input.workspace.executionTarget = sandbox;
|
|
const prior = await selectHeartbeatRuntime(db, input);
|
|
if (!prior.selected || !prior.nativeExecution) throw new Error("Expected prior native selection");
|
|
const priorInput = { ...prior.nativeExecution, session: { ...prior.nativeExecution.session, normalizedSessionId: sessionId } };
|
|
const priorProfile = (await read(run.id)).runnerProfileJson;
|
|
await db.update(heartbeatRuns).set({ status: "succeeded", nativeSessionId: sessionId,
|
|
runnerProfileJson: { ...priorProfile, nativeExecutionInput: priorInput, sessionCheckpoint: {
|
|
backendKind: "runner", driverKind: "codex_app_server", sessionId: "retained-provider-thread",
|
|
providerSessionId: "retained-provider-thread", identity: { ...priorInput.binding, sessionId },
|
|
} },
|
|
}).where(eq(heartbeatRuns.id, run.id));
|
|
const [nextRun] = await db.insert(heartbeatRuns).values({ companyId: agent.companyId, agentId: agent.id,
|
|
issueId: issue.id, scopeKind: "issue", status: "running", nativeSessionId: sessionId,
|
|
createdAt: new Date(run.createdAt.getTime() + 1000),
|
|
}).returning();
|
|
input.run = nextRun!;
|
|
input.workspace.executionWorkspace.cwd = workspace!.cwd!;
|
|
const selected = await selectHeartbeatRuntime(db, input);
|
|
if (!selected.selected || !selected.nativeExecution) throw new Error("Expected migrated native selection");
|
|
expect(selected.nativeExecution.session.normalizedSessionId).toBe(sessionId);
|
|
expect(selected.nativeExecution.workspace.cwd).toBe(sameSession ? logicalCwd : workspace!.cwd);
|
|
expect((await read(nextRun!.id)).runnerProfileJson?.nativeExecutionInput).toEqual(selected.nativeExecution);
|
|
expect((await read(nextRun!.id)).nativeSessionId).toBe(selected.nativeExecution.session.normalizedSessionId);
|
|
});
|
|
|
|
it.each([false, true])("persists the Cursor provider API placement and completion authority without a Computer lease (ephemeral: %s)", async ephemeral => {
|
|
vi.stubEnv("PAPERCLIP_EPHEMERAL_SERVER", String(ephemeral));
|
|
const { input, run, issue } = await fixture();
|
|
vi.stubEnv("PAPERCLIP_RUNNER_BINARY", "relative-unused-api-controller");
|
|
input.agent = { ...input.agent, adapterConfig: { provider: "cursor_cloud", model: "caller-model", repoUrl: "https://github.com/example/qualification" } };
|
|
await db.update(agents).set({ adapterConfig: input.agent.adapterConfig }).where(eq(agents.id, input.agent.id));
|
|
input.workspace.activeEnvironmentLease = null;
|
|
input.nativeRuntimeResolution = resolveHeartbeatNativeRuntimeMode({ persisted: run, enabled: true, runtimeConfig: {}, adapterConfig: input.agent.adapterConfig,
|
|
agent: input.agent, issue, target: null, workspaceId: null });
|
|
const result = await selectHeartbeatRuntime(db, input);
|
|
expect(result.selected).toBe(true);
|
|
if (!result.selected) throw new Error("Expected native Cursor selection");
|
|
expect(result.nativeExecution?.provider).toMatchObject({ kind: "cursor_cloud", model: "caller-model" });
|
|
expect(await read(run.id)).toMatchObject({ runtimeMode: "native", driverKind: "cursor_cloud_api", runnerProfileJson: {
|
|
backend: "cursor_cloud_api", executionPlacement: { kind: "provider_api", provider: "cursor_cloud", controller: "paperclip" },
|
|
nativeExecutionInput: result.nativeExecution,
|
|
} });
|
|
expect(await db.select().from(completionContracts)).toHaveLength(1);
|
|
expect(await coordinators(run.id)).toHaveLength(1);
|
|
expect(prepareNativeWorkspaceSync).not.toHaveBeenCalled();
|
|
expect(input.stopControllerLease).toHaveBeenCalledOnce();
|
|
});
|
|
|
|
it.each(["cursor_cloud_api", null] as const)("restores a historical Cursor workspace binding without allocating a Computer (driver: %s)", async driverKind => {
|
|
const { input, run, issue } = await fixture();
|
|
input.agent = { ...input.agent, adapterConfig: { provider: "cursor_cloud", model: "caller-model", repoUrl: "https://github.com/example/qualification" } };
|
|
await db.update(agents).set({ adapterConfig: input.agent.adapterConfig }).where(eq(agents.id, input.agent.id));
|
|
const [workspace] = await db.insert(executionWorkspaces).values({ companyId: input.agent.companyId, sourceIssueId: issue.id,
|
|
name: "Previously admitted Cursor workspace", mode: "shared_workspace", strategyType: "project_primary", cwd: "/previous-cursor-workspace" }).returning();
|
|
input.workspace.persistedExecutionWorkspace = workspace;
|
|
input.workspace.activeEnvironmentLease = null;
|
|
input.nativeRuntimeResolution = resolveHeartbeatNativeRuntimeMode({ persisted: run, enabled: true, runtimeConfig: {}, adapterConfig: input.agent.adapterConfig,
|
|
agent: input.agent, issue, target: null, workspaceId: workspace.id });
|
|
const selected = await selectHeartbeatRuntime(db, input);
|
|
if (!selected.selected || !selected.nativeExecution) throw new Error("Expected native Cursor selection");
|
|
expect(selected.nativeExecution.binding.executionWorkspaceId).toBe(workspace.id);
|
|
const saved = await read(run.id);
|
|
const historicalProfile: Record<string, unknown> = { ...saved.runnerProfileJson, sessionCheckpoint: { retained: "Cursor provider checkpoint" } };
|
|
delete historicalProfile.executionPlacement;
|
|
await db.update(heartbeatRuns).set({ runnerProfileJson: historicalProfile, driverKind }).where(eq(heartbeatRuns.id, run.id));
|
|
input.run = await read(run.id);
|
|
input.nativeRuntimeResolution = resolveHeartbeatNativeRuntimeMode({ persisted: input.run, enabled: true, runtimeConfig: {}, adapterConfig: input.agent.adapterConfig,
|
|
agent: input.agent, issue, target: null, workspaceId: null });
|
|
input.session.persistedRunnerProfile = historicalProfile;
|
|
input.session.persistedNativeExecutionInput = selected.nativeExecution;
|
|
input.workspace.persistedExecutionWorkspace = null;
|
|
input.workspace.executionWorkspace.cwd = "/private-cursor-controller";
|
|
vi.mocked(buildNativeRuntimeContext).mockClear();
|
|
const resumed = await selectHeartbeatRuntime(db, input);
|
|
expect(resumed).toMatchObject({ selected: true, nativeExecution: selected.nativeExecution, nativeRunnerInstanceId: selected.nativeRunnerInstanceId });
|
|
expect(await read(run.id)).toMatchObject({ driverKind: "cursor_cloud_api" });
|
|
expect((await read(run.id)).runnerProfileJson).toMatchObject({ ...historicalProfile,
|
|
executionPlacement: { kind: "provider_api", provider: "cursor_cloud", controller: "paperclip" } });
|
|
expect(await db.select().from(executionWorkspaces)).toEqual([workspace]);
|
|
expect(await db.select().from(completionContracts)).toHaveLength(1);
|
|
expect(await coordinators(run.id)).toHaveLength(1);
|
|
expect(prepareNativeWorkspaceSync).not.toHaveBeenCalled();
|
|
expect(buildNativeRuntimeContext).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it("persists and admits a standing-chat response contract without requiring the unavailable action", async () => {
|
|
const { input, issue, agent, run } = await fixture();
|
|
const [conversation] = await db.update(issues).set({ title: "Chat with CEO", description: null,
|
|
conversationAgentId: agent.id, conversationUserId: "runtime-test-operator", conversationState: "active" })
|
|
.where(eq(issues.id, issue.id)).returning();
|
|
input.task.issueRef = conversation!;
|
|
input.task.context.conversationMode = true;
|
|
const request = "Restart this instance.";
|
|
input.task.safeWakeComments = [{ id: randomUUID(), body: request }];
|
|
const selected = await selectHeartbeatRuntime(db, input);
|
|
if (!selected.selected || !selected.nativeExecution) throw new Error("Expected native selection");
|
|
const contract = selected.nativeExecution.completionContract.contract;
|
|
expect(contract.objective).toMatch(/current conversation response/i);
|
|
expect(contract.criteria).toEqual([{ id: "objective", requirement: expect.stringContaining(request) }]);
|
|
expect(contract.criteria[0]!.requirement).not.toBe(request);
|
|
const [persisted] = await db.select().from(completionContracts).where(eq(completionContracts.issueId, issue.id));
|
|
expect(persisted!.contractJson).toEqual(contract);
|
|
expect(selected.nativeExecution.completionContract.sha256).toBe(persisted!.canonicalSha256);
|
|
expect((await read(run.id)).runnerProfileJson?.nativeExecutionInput).toEqual(selected.nativeExecution);
|
|
expect(buildNativeModelEnvelope(selected.nativeExecution).completionContract).toEqual(contract);
|
|
});
|
|
|
|
it("keeps recovery out of live preparation before the workspace descriptor exists", async () => {
|
|
const { input, run, issue } = await fixture();
|
|
input.workspace.executionTarget = sandbox;
|
|
await db.update(issues).set({ executionRunId: run.id }).where(eq(issues.id, issue.id));
|
|
const liveRunExecutions = new Set([run.id]);
|
|
const recoveryInput = { db, restartKind: "hard" as const, runIds: [run.id], liveRunExecutions };
|
|
vi.mocked(prepareNativeWorkspaceSync).mockImplementationOnce(async () => {
|
|
const saved = await read(run.id);
|
|
expect(saved.runnerProfileJson).toHaveProperty("nativeExecutionInput");
|
|
expect(saved.runnerProfileJson).not.toHaveProperty("nativeWorkspaceSync");
|
|
const before = await coordinators(run.id);
|
|
expect(await claimNativeRestartRecoveries(recoveryInput)).toEqual([
|
|
{ kind: "awaiting_evidence", runId: run.id, reason: "current_controller_preparing" },
|
|
]);
|
|
expect(await coordinators(run.id)).toEqual(before);
|
|
expect((await read(run.id)).status).toBe("running");
|
|
return null;
|
|
});
|
|
expect((await selectHeartbeatRuntime(db, input)).selected).toBe(true);
|
|
liveRunExecutions.delete(run.id);
|
|
expect(await claimNativeRestartRecoveries(recoveryInput)).toEqual([
|
|
expect.objectContaining({ kind: "bootstrap_incomplete", runId: run.id, controllerGeneration: 1 }),
|
|
]);
|
|
});
|
|
|
|
it("persists the native input, completion contract, and coordinator before transferring cleanup ownership", async () => {
|
|
const { input, run } = await fixture();
|
|
input.workspace.onCheckpoint = vi.fn(async () => {});
|
|
input.workspace.onPhase = async (_name, work) => work();
|
|
vi.mocked(prepareNativeWorkspaceSync).mockImplementationOnce(async args => {
|
|
expect(await read(run.id)).toMatchObject({ runtimeMode: "native", nativeIssueId: input.task.issueRef!.id });
|
|
expect(await coordinators(run.id)).toHaveLength(1);
|
|
expect(input.stopControllerLease).toHaveBeenCalledOnce();
|
|
expect(args).toMatchObject({ runId: run.id, workspaceId: run.id, workspaceLocalDir: input.workspace.executionWorkspace.cwd });
|
|
expect(args.onCheckpoint).toBe(input.workspace.onCheckpoint);
|
|
expect(args.onPhase).toBe(input.workspace.onPhase);
|
|
return null;
|
|
});
|
|
const selected = await selectHeartbeatRuntime(db, input);
|
|
expect(selected.selected).toBe(true);
|
|
if (!selected.selected) throw new Error("Expected selection");
|
|
const saved = await read(run.id);
|
|
expect(saved).toMatchObject({ runtimeMode: "native", driverKind: "codex_app_server", nativePhase: "observed", nativeSessionId: expect.any(String), runnerInstanceId: selected.nativeRunnerInstanceId,
|
|
runnerProfileJson: { nativeExecutionInput: selected.nativeExecution, recoveryEventInventoryVersion: 1, nativePreparationFencingVersion: 1 } });
|
|
expect(await db.select().from(completionContracts)).toHaveLength(1);
|
|
expect(selected.getNativeFreshSessionHandoff).toBe(input.session.getFreshSessionHandoff);
|
|
expect(input.onNativeOwnershipHeld).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it("persists the Copilot task model without accepting task provider or permission changes", async () => {
|
|
const { input, run, issue } = await fixture();
|
|
const adapterConfig = { provider: "acpx", acpxAgent: "copilot", model: "gpt-5.6-luna", acpxPermissionMode: "approve-reads" };
|
|
input.agent = { ...input.agent, adapterConfig };
|
|
await db.update(agents).set({ adapterConfig }).where(eq(agents.id, input.agent.id));
|
|
input.nativeRuntimeResolution = resolveHeartbeatNativeRuntimeMode({
|
|
persisted: input.run, enabled: true, runtimeConfig: {}, adapterConfig,
|
|
agent: input.agent, issue, target: null, workspaceId: null,
|
|
});
|
|
input.config.issueAssigneeOverrides = { adapterConfig: {
|
|
model: "gpt-6-luna", provider: "codex", acpxAgent: "pi", acpxPermissionMode: "approve-all",
|
|
} };
|
|
const selected = await selectHeartbeatRuntime(db, input);
|
|
if (!selected.selected || !selected.nativeExecution) throw new Error("Expected native Copilot selection");
|
|
const expected = { kind: "acpx", agent: "copilot", model: "gpt-6-luna", permissionMode: "approve-reads" };
|
|
expect(selected.nativeExecution.provider).toMatchObject(expected);
|
|
expect((await read(run.id)).runnerProfileJson).toMatchObject({ nativeExecutionInput: { provider: expected } });
|
|
});
|
|
|
|
it("persists configured GitHub instruction skills without admitting provider-mentioned skills", async () => {
|
|
const { input, issue, run } = await fixture();
|
|
input.config.runScopedMentionedSkillKeys = ["company/test/configured"];
|
|
input.task.context.paperclipWake = {
|
|
reason: "External chat message received", externalChatProvider: "github", checkedOutByHarness: true,
|
|
issue: { id: issue.id, workMode: "standard" },
|
|
comments: [{ id: "review-comment", body: "Please use /provider-mentioned" }],
|
|
commentIds: ["review-comment"], latestCommentId: "review-comment",
|
|
commentWindow: { requestedCount: 1, includedCount: 1, missingCount: 0 }, fallbackFetchNeeded: false,
|
|
};
|
|
vi.mocked(buildNativeRuntimeContext).mockImplementationOnce(async () => {
|
|
const context = nativeRuntimeContextFixture();
|
|
context.instructions.bundle.rootPath = instructionRoot;
|
|
context.skills = ["configured", "provider-mentioned"].map(name => ({
|
|
key: `company/test/${name}`, runtimeName: name, versionId: null,
|
|
bundle: { ...context.instructions.bundle },
|
|
}));
|
|
return { ...context, aggregateDigest: canonicalNativeRuntimeContextDigest(context) };
|
|
});
|
|
expect(await selectHeartbeatRuntime(db, input)).toMatchObject({
|
|
selected: true, nativeExecution: { task: { description: "Configured GitHub instruction skills:\n/configured" } },
|
|
});
|
|
expect((await read(run.id)).runnerProfileJson).toMatchObject({
|
|
nativeExecutionInput: { task: { description: "Configured GitHub instruction skills:\n/configured" } },
|
|
});
|
|
});
|
|
|
|
it("does not upgrade an older preparation from supplied runtime profile data", async () => {
|
|
const { input, run } = await fixture();
|
|
const selected = await selectHeartbeatRuntime(db, input);
|
|
if (!selected.selected || !selected.nativeExecution || input.nativeRuntimeResolution.kind !== "native") {
|
|
throw new Error("Expected native selection");
|
|
}
|
|
const saved = await read(run.id);
|
|
const olderProfile = { ...saved.runnerProfileJson };
|
|
delete olderProfile.nativePreparationFencingVersion;
|
|
await db.update(heartbeatRuns).set({ runnerProfileJson: olderProfile }).where(eq(heartbeatRuns.id, run.id));
|
|
input.run = await read(run.id);
|
|
input.session.persistedNativeExecutionInput = selected.nativeExecution;
|
|
input.session.persistedRunnerProfile = { ...olderProfile, nativePreparationFencingVersion: 1 };
|
|
Object.assign(input.nativeRuntimeResolution.profile, { nativePreparationFencingVersion: 1 });
|
|
expect((await selectHeartbeatRuntime(db, input)).selected).toBe(true);
|
|
expect((await read(run.id)).runnerProfileJson).not.toHaveProperty("nativePreparationFencingVersion");
|
|
});
|
|
|
|
it("restores the immutable admitted input and saved checkpoint when newer context arrives", async () => {
|
|
using discovery = vi.spyOn(runner, "describeRunnerdNativeSessionBackend");
|
|
const { input, run } = await fixture();
|
|
const selected = await selectHeartbeatRuntime(db, input);
|
|
if (!selected.selected || !selected.nativeExecution) throw new Error("Expected native selection");
|
|
expect(selected.toolRefreshOnResume).toBe(true);
|
|
expect(discovery).toHaveBeenCalledTimes(1);
|
|
const saved = await read(run.id);
|
|
expect(saved.runnerProfileJson?.nativeToolRefreshOnResume).toBe(true);
|
|
const profile = { ...saved.runnerProfileJson, sessionCheckpoint: { retained: "checkpoint" } };
|
|
await db.update(heartbeatRuns).set({ runnerProfileJson: profile }).where(eq(heartbeatRuns.id, run.id));
|
|
input.run = await read(run.id);
|
|
input.session.persistedRunnerProfile = profile;
|
|
input.session.persistedNativeExecutionInput = selected.nativeExecution;
|
|
input.task.context.paperclipTaskMarkdown = "A newer unrelated request";
|
|
input.task.safeWakeComments = [{ id: randomUUID(), body: "New direction belongs to a separate wake" }];
|
|
vi.mocked(buildNativeRuntimeContext).mockClear();
|
|
const resumed = await selectHeartbeatRuntime(db, input);
|
|
expect(resumed).toMatchObject({ selected: true, nativeExecution: selected.nativeExecution, nativeRunnerInstanceId: selected.nativeRunnerInstanceId,
|
|
toolRefreshOnResume: true });
|
|
expect((await read(run.id)).runnerProfileJson).toMatchObject(profile);
|
|
expect(await db.select().from(completionContracts)).toHaveLength(1);
|
|
expect(await coordinators(run.id)).toHaveLength(1);
|
|
expect(buildNativeRuntimeContext).not.toHaveBeenCalled();
|
|
expect(discovery).toHaveBeenCalledTimes(1);
|
|
});
|
|
|
|
it("rejects an admitted input bound to another run without replacing the saved runtime", async () => {
|
|
const { input, run } = await fixture();
|
|
const selected = await selectHeartbeatRuntime(db, input);
|
|
if (!selected.selected || !selected.nativeExecution) throw new Error("Expected native selection");
|
|
const saved = await read(run.id);
|
|
input.run = saved;
|
|
input.session.persistedNativeExecutionInput = { ...selected.nativeExecution, binding: { ...selected.nativeExecution.binding, runId: randomUUID() } };
|
|
await expect(selectHeartbeatRuntime(db, input)).rejects.toThrow("native_execution_input_persisted_binding_mismatch");
|
|
expect((await read(run.id)).runnerProfileJson).toEqual(saved.runnerProfileJson);
|
|
});
|
|
|
|
it("rejects resume after skill access revocation without rewriting the admitted revision", async () => {
|
|
const { input, run } = await fixture();
|
|
const key = "company/test/private-skill", versionId = randomUUID();
|
|
await db.update(heartbeatRuns).set({ contextSnapshot: { skillVersions: { [key]: versionId } } }).where(eq(heartbeatRuns.id, run.id));
|
|
input.run = await read(run.id);
|
|
vi.mocked(buildNativeRuntimeContext).mockImplementationOnce(async () => {
|
|
const context = nativeRuntimeContextFixture();
|
|
context.instructions.bundle.rootPath = instructionRoot;
|
|
context.skills = [{ key, runtimeName: "private-skill", versionId, bundle: { ...context.instructions.bundle } }];
|
|
return { ...context, aggregateDigest: canonicalNativeRuntimeContextDigest(context) };
|
|
});
|
|
const selected = await selectHeartbeatRuntime(db, input);
|
|
if (!selected.selected || !selected.nativeExecution) throw new Error("Expected native selection");
|
|
const saved = await read(run.id);
|
|
input.run = saved;
|
|
input.session.persistedNativeExecutionInput = selected.nativeExecution;
|
|
input.config.runtimeSkillEntries = [];
|
|
await expect(selectHeartbeatRuntime(db, input)).rejects.toThrow("Project skill access changed");
|
|
expect((await read(run.id)).runnerProfileJson).toEqual(saved.runnerProfileJson);
|
|
expect((await read(run.id)).contextSnapshot?.skillVersions).toEqual({ [key]: versionId });
|
|
});
|
|
|
|
it.each(["exact", "fresh", "bootstrap", "dead-runner", "expired", "foreign-owner", "missing-audit"] as const)(
|
|
"admits only exact pending Stop cleanup at runtime selection (%s)", async mode => {
|
|
const { input, run, issue } = await fixture();
|
|
const initial = await selectHeartbeatRuntime(db, input);
|
|
if (!initial.selected || !initial.nativeExecution) throw new Error("Expected original native selection");
|
|
await reserveRunBudget(db, run.companyId, run.id, null);
|
|
const saved = await read(run.id), leaseOwner = `stop:${randomUUID()}`, intentAuditId = randomUUID();
|
|
input.run = saved;
|
|
input.session.persistedNativeExecutionInput = initial.nativeExecution;
|
|
input.session.persistedRunnerProfile = saved.runnerProfileJson ?? {};
|
|
await db.insert(activityLog).values({ id: intentAuditId, companyId: run.companyId, actorType: "system", actorId: "native-session-cancellation",
|
|
action: "native.cancellation_intent_recorded", entityType: "heartbeat_run", entityId: run.id, runId: run.id, agentId: run.agentId });
|
|
await db.update(nativeRunFinalizations).set({ leaseOwner: mode === "foreign-owner" ? "another-owner" : leaseOwner,
|
|
leaseExpiresAt: new Date(Date.now() + (mode === "expired" ? -1 : 60_000)), controllerGeneration: 2, attempt: 1,
|
|
recoveryState: mode === "bootstrap" ? "bootstrap_incomplete" : mode === "dead-runner" ? "resuming_session" : "awaiting_runner_reattach",
|
|
recoveryHistory: [{ disposition: "reattach_remote_runner", controllerGeneration: 2, providerAttempt: 1 }],
|
|
}).where(eq(nativeRunFinalizations.runId, run.id));
|
|
await db.update(heartbeatRuns).set({ resultJson: {
|
|
startupCancellation: { beforeNativeSelection: false, errorCode: "cancelled", requestedAt: new Date().toISOString() },
|
|
executionCancellation: { state: "requested" },
|
|
nativeCancellation: { schema: "paperclip.native-cancellation.v1", companyId: run.companyId, runId: run.id, issueId: issue.id,
|
|
scope: "run", reasonCode: "cancellation_run_only", intentId: `native-cancellation:${randomUUID()}`,
|
|
intentAuditId: mode === "missing-audit" ? randomUUID() : intentAuditId, dispatchState: "pending", dispatched: false,
|
|
effects: ["release_run_resources"] },
|
|
} }).where(eq(heartbeatRuns.id, run.id));
|
|
input.recovery = mode === "fresh" ? {} : { nativeLeaseOwner: leaseOwner, nativeRestartRecovery: {
|
|
kind: "reattach_remote_runner", runId: run.id, leaseOwner, controllerGeneration: 2, providerAttempt: 1,
|
|
restartKind: "hard", recoveryRequestId: null, remote: { providerLeaseId: "existing-sandbox", remoteCwd: "/workspace" },
|
|
} };
|
|
const selected = await selectHeartbeatRuntime(db, input);
|
|
expect(selected.selected).toBe(mode === "exact");
|
|
expect(await coordinators(run.id)).toHaveLength(1);
|
|
expect((await read(run.id)).nativeSessionId).toBe(saved.nativeSessionId);
|
|
expect((await read(run.id)).runnerProfileJson?.nativeExecutionInput).toEqual(initial.nativeExecution);
|
|
});
|
|
|
|
it.each(["cancelled", "startup_cancellation"])("does not create a native coordinator when %s wins preparation", async winner => {
|
|
const { input, run } = await fixture();
|
|
input.beforeNativeRuntimeSelection = async () => {
|
|
await db.update(heartbeatRuns).set(winner === "cancelled" ? { status: "cancelled" } : { resultJson: { startupCancellation: { beforeNativeSelection: true } } }).where(eq(heartbeatRuns.id, run.id));
|
|
};
|
|
expect(await selectHeartbeatRuntime(db, input)).toEqual({ selected: false });
|
|
expect(await read(run.id)).toMatchObject({ runtimeMode: "legacy", nativeSessionId: null, completionContractId: null });
|
|
expect(await coordinators(run.id)).toHaveLength(0);
|
|
expect(input.stopControllerLease).not.toHaveBeenCalled();
|
|
expect(prepareNativeWorkspaceSync).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it("reports a lost controller lease before returning without native selection", async () => {
|
|
const { input, run } = await fixture();
|
|
await db.update(heartbeatRuns).set({ controllerBootId: randomUUID(), controllerLeaseExpiresAt: new Date(Date.now() + 60_000) }).where(eq(heartbeatRuns.id, run.id));
|
|
expect(await selectHeartbeatRuntime(db, input)).toEqual({ selected: false });
|
|
expect(input.onNativeOwnershipHeld).toHaveBeenCalledOnce();
|
|
expect(input.stopControllerLease).not.toHaveBeenCalled();
|
|
expect(await coordinators(run.id)).toHaveLength(0);
|
|
});
|
|
|
|
it("rejects a competing persisted legacy selection atomically", async () => {
|
|
const { input, run } = await fixture();
|
|
input.beforeNativeRuntimeSelection = async () => {
|
|
await db.update(heartbeatRuns).set({ runtimeModeResolvedAt: new Date(), runtimeMode: "legacy" }).where(eq(heartbeatRuns.id, run.id));
|
|
};
|
|
await expect(selectHeartbeatRuntime(db, input)).rejects.toThrow("native_runtime_mode_conflict");
|
|
expect((await read(run.id)).runtimeMode).toBe("legacy");
|
|
expect(await coordinators(run.id)).toHaveLength(0);
|
|
expect(input.stopControllerLease).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it("reports sandbox teardown policy even when a later preparation step throws", async () => {
|
|
const { input, run } = await fixture();
|
|
input.workspace.executionTarget = sandbox;
|
|
input.beforeNativeRuntimeSelection = async () => { throw new Error("selection interrupted"); };
|
|
await expect(selectHeartbeatRuntime(db, input)).rejects.toThrow("selection interrupted");
|
|
expect(input.onNativeLifecycleSelected).toHaveBeenCalledWith({ provider: "codex", harness: "codex_app_server", lifecycleMode: "per_turn", sandboxResource: "destroy_after_turn" });
|
|
expect(input.onProviderResourceDisposition).toHaveBeenCalledWith("destroy");
|
|
expect(input.stopControllerLease).not.toHaveBeenCalled();
|
|
expect(await coordinators(run.id)).toHaveLength(0);
|
|
});
|
|
|
|
it("retains native admission and teardown state if workspace sync preparation fails", async () => {
|
|
const { input, run } = await fixture();
|
|
input.workspace.executionTarget = sandbox;
|
|
vi.mocked(prepareNativeWorkspaceSync).mockRejectedValueOnce(new Error("sync preparation failed"));
|
|
await expect(selectHeartbeatRuntime(db, input)).rejects.toThrow("sync preparation failed");
|
|
expect(input.onProviderResourceDisposition).toHaveBeenCalledWith("destroy");
|
|
expect(input.stopControllerLease).toHaveBeenCalledOnce();
|
|
expect((await read(run.id)).runtimeMode).toBe("native");
|
|
expect(await coordinators(run.id)).toHaveLength(1);
|
|
});
|
|
|
|
// The remote-sandbox warm default follows the acquired lease: it never turns a
|
|
// run that works per-turn into a refused warm run.
|
|
it.each([
|
|
["a reusable lease", { reusableLeaseConfigured: true, effectiveCapabilities: { reusableLeases: true } }, "warm", "keep_running"],
|
|
["reuse configured on a provider without reusable leases", { reusableLeaseConfigured: true, effectiveCapabilities: { reusableLeases: false } }, "per_turn", "destroy_after_turn"],
|
|
["an ephemeral lease", {}, "per_turn", "destroy_after_turn"],
|
|
] as const)("defaults an agent without a lifecycle on %s", async (_name, lease, lifecycleMode, sandboxResource) => {
|
|
const { input } = await fixture();
|
|
const { lifecycleMode: _unset, ...adapterConfig } = (input.agent.adapterConfig ?? {}) as Record<string, unknown>;
|
|
input.agent.adapterConfig = adapterConfig;
|
|
input.workspace.executionTarget = { ...sandbox, ...lease } as never;
|
|
input.beforeNativeRuntimeSelection = async () => { throw new Error("selection interrupted"); };
|
|
await expect(selectHeartbeatRuntime(db, input)).rejects.toThrow("selection interrupted");
|
|
expect(input.onNativeLifecycleSelected).toHaveBeenCalledWith({ provider: "codex", harness: "codex_app_server", lifecycleMode, sandboxResource });
|
|
});
|
|
|
|
// The environment view (lease) and the native policy (runner) must make the
|
|
// same remote-sandbox decision for every harness, including a provider-less
|
|
// Codex agent and ACPX without an agent (Claude).
|
|
it.each([
|
|
[{ provider: "codex", model: "gpt-5.6-luna" }, "warm"],
|
|
[{ model: "gpt-5.6-luna" }, "warm"],
|
|
[{ provider: "opencode", model: "openai/gpt-5" }, "warm"],
|
|
[{ provider: "acpx", acpxAgent: "claude", model: "claude-sonnet-4-6" }, "warm"],
|
|
[{ provider: "acpx", model: "claude-sonnet-4-6" }, "warm"],
|
|
[{ provider: "acpx", acpxAgent: "copilot", model: "gpt-5" }, "per_turn"],
|
|
] as const)("agrees with the environment view's lease for %j", async (adapterConfig, mode) => {
|
|
const { input, run, agent, issue } = await fixture();
|
|
input.agent.adapterConfig = adapterConfig;
|
|
input.nativeRuntimeResolution = resolveHeartbeatNativeRuntimeMode({ persisted: run, enabled: true, runtimeConfig: {}, adapterConfig, agent, issue, target: null, workspaceId: null });
|
|
// An inheriting environment with its own idle window: the provider and the runner must use the same one.
|
|
const environment = { driver: "sandbox" as const, config: { provider: "daytona", runnerIdleTimeoutMs: 900_000 } as Record<string, unknown> };
|
|
const view = resolveRunnerEnvironmentForRun(environment, "paperclip_runner", adapterConfig, undefined, undefined, { reusableLeaseSupported: true });
|
|
expect(view.config.reuseLease === true).toBe(mode === "warm");
|
|
const forwardedIdleMs = resolveWarmRunnerIdleTimeoutMsForRun(environment, "paperclip_runner", adapterConfig, undefined, { reusableLeaseSupported: true });
|
|
input.workspace.executionTarget = { ...sandbox, reusableLeaseConfigured: view.config.reuseLease === true, effectiveCapabilities: { reusableLeases: true },
|
|
runnerIdleTimeoutMs: 900_000 } as never;
|
|
const stop = new Error("policy captured");
|
|
const policies: unknown[] = [];
|
|
await expect(prepareAndSelectHeartbeatRuntime(db, input, {
|
|
measure: async (_phase, work) => work(),
|
|
prepareInstructions: async (policy) => { policies.push(policy); throw stop; },
|
|
})).rejects.toBe(stop);
|
|
expect(policies).toEqual([mode === "warm" ? { mode, idleTimeoutMs: forwardedIdleMs } : { mode, idleTimeoutMs: null }]);
|
|
expect(forwardedIdleMs).toBe(mode === "warm" ? 900_000 : undefined);
|
|
});
|
|
|
|
it("keeps a plugin task's sandbox target out of the remote default", async () => {
|
|
const { input } = await fixture();
|
|
const { lifecycleMode: _unset, ...adapterConfig } = (input.agent.adapterConfig ?? {}) as Record<string, unknown>;
|
|
input.agent.adapterConfig = adapterConfig;
|
|
input.workspace.executionTarget = { ...sandbox, reusableLeaseConfigured: true, effectiveCapabilities: { reusableLeases: true }, runnerTask: vi.fn() } as never;
|
|
const stop = new Error("policy captured");
|
|
const policies: unknown[] = [];
|
|
await expect(prepareAndSelectHeartbeatRuntime(db, input, {
|
|
measure: async (_phase, work) => work(),
|
|
prepareInstructions: async (policy) => { policies.push(policy?.mode); throw stop; },
|
|
})).rejects.toBe(stop);
|
|
expect(policies).toEqual(["per_turn"]);
|
|
});
|
|
|
|
it("rejects a warm native sandbox that cannot reuse its lease", async () => {
|
|
const { input, run } = await fixture();
|
|
input.workspace.executionTarget = { ...sandbox, runnerLifecyclePolicy: { mode: "warm", idleTimeoutMs: 300_000 } };
|
|
await expect(selectHeartbeatRuntime(db, input)).rejects.toThrow("runner_warm_environment_requires_reusable_lease");
|
|
expect(input.onNativeLifecycleSelected).not.toHaveBeenCalled();
|
|
expect(await coordinators(run.id)).toHaveLength(0);
|
|
});
|
|
|
|
it("rejects resolved legacy execution on an ephemeral server before runtime admission", async () => {
|
|
const { input, run } = await fixture("process");
|
|
vi.stubEnv("PAPERCLIP_EPHEMERAL_SERVER", "true");
|
|
expect(input.nativeRuntimeResolution.kind).toBe("legacy");
|
|
await expect(selectHeartbeatRuntime(db, input)).rejects.toThrow("native Paperclip Runner");
|
|
expect((await read(run.id)).runtimeMode).toBe("legacy");
|
|
expect(await coordinators(run.id)).toHaveLength(0);
|
|
expect(input.onNativeLifecycleSelected).not.toHaveBeenCalled();
|
|
expect(prepareNativeWorkspaceSync).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it.each(["warm", "per_turn"] as const)("preserves resolved %s policy on an ephemeral server", async mode => {
|
|
const { input, run } = await fixture();
|
|
vi.stubEnv("PAPERCLIP_EPHEMERAL_SERVER", "true");
|
|
// Deliberately disagree with the resolved provider policy in both cases.
|
|
input.agent.adapterConfig = { ...input.agent.adapterConfig, ...(mode === "per_turn" ? { lifecycleMode: "warm" } : {}) };
|
|
input.workspace.executionTarget = { ...sandbox, reusableLeaseConfigured: true,
|
|
effectiveCapabilities: { reusableLeases: true } as never,
|
|
runnerLifecyclePolicy: mode === "warm" ? { mode, idleTimeoutMs: 300_000 } : { mode, idleTimeoutMs: null } };
|
|
const selected = await selectHeartbeatRuntime(db, input);
|
|
expect(selected.selected && selected.nativeExecution?.session.lifecyclePolicy.mode).toBe(mode);
|
|
expect((await read(run.id)).runnerProfileJson).toMatchObject({ nativeExecutionInput: { session: { lifecyclePolicy: { mode } } } });
|
|
expect(await coordinators(run.id)).toHaveLength(1);
|
|
});
|
|
|
|
it.each([
|
|
{ provider: "acpx", acpxAgent: "claude", model: "claude-opus-5-5", credentialProvider: "anthropic" as const, method: "subscription" as const },
|
|
{ provider: "acpx", acpxAgent: "grok", model: "grok-4.7", credentialProvider: "xai" as const, method: "subscription" as const },
|
|
{ provider: "opencode", model: "openrouter/deepseek/deepseek-v4-flash-0731", credentialProvider: "openrouter" as const, method: "api_key" as const },
|
|
])("admits managed $provider/$acpxAgent on an ephemeral server without changing credential lifecycle", async profile => {
|
|
vi.stubEnv("PAPERCLIP_EPHEMERAL_SERVER", "true");
|
|
const { input, run, issue } = await fixture();
|
|
const { credentialProvider, method, ...providerConfig } = profile;
|
|
const adapterConfig = { ...providerConfig, lifecycleMode: "warm" };
|
|
input.agent = { ...input.agent, adapterConfig };
|
|
await db.update(agents).set({ adapterConfig }).where(eq(agents.id, input.agent.id));
|
|
input.workspace.executionTarget = { ...sandbox, reusableLeaseConfigured: true,
|
|
effectiveCapabilities: { reusableLeases: true } as never, runnerLifecyclePolicy: { mode: "warm", idleTimeoutMs: 300_000 } };
|
|
input.nativeRuntimeResolution = resolveHeartbeatNativeRuntimeMode({ persisted: run, enabled: true, runtimeConfig: {}, adapterConfig,
|
|
agent: input.agent, issue, target: input.workspace.executionTarget, workspaceId: null });
|
|
expect(input.nativeRuntimeResolution.kind).toBe("native");
|
|
const attribution = { connectionId: randomUUID(), grantId: randomUUID(), provider: credentialProvider, method,
|
|
mode: "responsible_user" as const, responsibleUserId: "task-owner" };
|
|
input.config.managedAiRuntime = { sessionIdentity: "account:epoch-1", identity: "account:generation-1",
|
|
config: { env: {}, managedAiConnection: { ...attribution, identity: "account:generation-1", sessionIdentity: "account:epoch-1" } },
|
|
attribution, accountName: "Saved account", accountOwnerUserId: "task-owner", home: instructionRoot, cleanup: async () => {} };
|
|
const selected = await selectHeartbeatRuntime(db, input);
|
|
if (!selected.selected || !selected.nativeExecution) throw new Error("Expected native selection");
|
|
expect(selected.nativeExecution.provider.kind).toBe(profile.provider);
|
|
expect(selected.nativeExecution.session.lifecyclePolicy).toEqual({ mode: "per_turn", idleTimeoutMs: null });
|
|
expect((await read(run.id)).runnerProfileJson).toMatchObject({
|
|
nativeExecutionInput: { session: { lifecyclePolicy: { mode: "per_turn" } } },
|
|
});
|
|
expect(await coordinators(run.id)).toHaveLength(1);
|
|
});
|
|
|
|
it.each([
|
|
{ provider: "github" as const, method: "api_key" as const, expected: "warm" },
|
|
{ provider: "github" as const, method: "subscription" as const, expected: "per_turn" },
|
|
{ provider: "openai" as const, method: "api_key" as const, expected: "per_turn" },
|
|
])("persists $expected Copilot lifecycle for a managed $provider/$method account", async ({ provider, method, expected }) => {
|
|
const { input, run, issue } = await fixture();
|
|
const adapterConfig = { provider: "acpx", acpxAgent: "copilot", model: "gpt-5.6-luna", lifecycleMode: "warm", idleTimeoutMs: 300_000 };
|
|
input.agent = { ...input.agent, adapterConfig };
|
|
await db.update(agents).set({ adapterConfig }).where(eq(agents.id, input.agent.id));
|
|
input.nativeRuntimeResolution = resolveHeartbeatNativeRuntimeMode({
|
|
persisted: input.run, enabled: true, runtimeConfig: {}, adapterConfig,
|
|
agent: input.agent, issue, target: null, workspaceId: null,
|
|
});
|
|
const attribution = { connectionId: randomUUID(), grantId: randomUUID(), provider, method, mode: "responsible_user" as const, responsibleUserId: "task-owner" };
|
|
input.config.managedAiRuntime = {
|
|
sessionIdentity: "fixture-grant:task-owner:epoch-1",
|
|
config: { env: {}, managedAiConnection: { ...attribution, identity: "fixture-grant:task-owner:generation-1", sessionIdentity: "fixture-grant:task-owner:epoch-1" } },
|
|
attribution,
|
|
accountName: "Saved account",
|
|
accountOwnerUserId: "task-owner",
|
|
identity: "fixture-grant:task-owner:generation-1",
|
|
home: instructionRoot,
|
|
cleanup: async () => {},
|
|
};
|
|
const selected = await selectHeartbeatRuntime(db, input);
|
|
if (!selected.selected || !selected.nativeExecution) throw new Error("Expected native Copilot selection");
|
|
expect(selected.nativeExecution.session.lifecyclePolicy.mode).toBe(expected);
|
|
expect((await read(run.id)).runnerProfileJson).toMatchObject({
|
|
nativeExecutionInput: { session: { lifecyclePolicy: { mode: expected } } },
|
|
});
|
|
expect(await coordinators(run.id)).toHaveLength(1);
|
|
});
|
|
|
|
it.each(["api_key", "subscription"] as const)("keeps managed Claude %s task execution warm", async method => {
|
|
const { input, run, issue } = await fixture();
|
|
const adapterConfig = { provider: "acpx", acpxAgent: "claude", model: "claude-sonnet-4-6", lifecycleMode: "warm" };
|
|
input.agent = { ...input.agent, adapterConfig };
|
|
await db.update(agents).set({ adapterConfig }).where(eq(agents.id, input.agent.id));
|
|
input.nativeRuntimeResolution = resolveHeartbeatNativeRuntimeMode({ persisted: input.run, enabled: true, runtimeConfig: {}, adapterConfig,
|
|
agent: input.agent, issue, target: null, workspaceId: null });
|
|
input.workspace.executionTarget = { ...sandbox, runnerTask: vi.fn(), reusableLeaseConfigured: true,
|
|
effectiveCapabilities: { reusableLeases: true } as never, runnerLifecyclePolicy: { mode: "warm", idleTimeoutMs: 300_000 } };
|
|
const attribution = { connectionId: randomUUID(), grantId: randomUUID(), provider: "anthropic" as const, method,
|
|
mode: "responsible_user" as const, responsibleUserId: "task-owner" };
|
|
input.config.managedAiRuntime = { sessionIdentity: "account:epoch-1", identity: "account:generation-1",
|
|
config: { env: {}, managedAiConnection: { ...attribution, identity: "account:generation-1", sessionIdentity: "account:epoch-1" } },
|
|
attribution, accountName: "Saved account", accountOwnerUserId: "task-owner", home: instructionRoot, cleanup: async () => {} };
|
|
const selected = await selectHeartbeatRuntime(db, input);
|
|
if (!selected.selected || !selected.nativeExecution) throw new Error("Expected native Claude selection");
|
|
expect(selected.nativeExecution.session.lifecyclePolicy).toEqual({ mode: "warm", idleTimeoutMs: 300_000 });
|
|
expect((await read(run.id)).runnerProfileJson).toMatchObject({
|
|
nativeResourceDisposition: "keep_running",
|
|
nativeExecutionInput: { session: { lifecyclePolicy: { mode: "warm" } } },
|
|
});
|
|
});
|
|
|
|
it("preserves only server-owned admission and dispatch evidence when recording legacy selection", async () => {
|
|
const { input, run } = await fixture("process");
|
|
const admission = { version: 1, kind: "fixture" }, dispatch = { entered: true };
|
|
await db.update(heartbeatRuns).set({ runnerProfileJson: { [CHAT_CONTROL_RECOVERY_ADMISSION_KEY]: admission, adapterDispatch: dispatch, unrelated: "drop" } }).where(eq(heartbeatRuns.id, run.id));
|
|
input.trace = { providerTraceRequested: true, providerTraceCapture: { metadata: { id: "trace-id" } } };
|
|
expect(await selectHeartbeatRuntime(db, input)).toMatchObject({ selected: true, nativeExecution: null, nativeWorkspaceSync: null });
|
|
expect((await read(run.id)).runnerProfileJson).toEqual({ [CHAT_CONTROL_RECOVERY_ADMISSION_KEY]: admission, adapterDispatch: dispatch, providerTrace: { mode: "raw", traceId: "trace-id", maxBytes: 64 * 1024 * 1024 } });
|
|
expect(await coordinators(run.id)).toHaveLength(0);
|
|
expect(input.stopControllerLease).not.toHaveBeenCalled();
|
|
expect(buildNativeRuntimeContext).not.toHaveBeenCalled();
|
|
});
|
|
async function claimPreparation(f: Awaited<ReturnType<typeof fixture>>) {
|
|
await db.update(heartbeatRuns).set(legacyControllerClaim("legacy")).where(eq(heartbeatRuns.id, f.run.id));
|
|
f.input.run = await read(f.run.id);
|
|
expect(f.input.run.controllerBootId).toBeTruthy();
|
|
expect(f.input.run.controllerLeaseExpiresAt!.getTime()).toBeGreaterThan(Date.now());
|
|
expect(f.input.run.executionStage).toBe("preparing");
|
|
}
|
|
|
|
it.each(["local", "remote"] as const)("retires actual preparation ownership atomically at native admission: %s", async target => {
|
|
const f = await fixture();
|
|
await claimPreparation(f);
|
|
if (target === "remote") f.input.workspace.executionTarget = sandbox;
|
|
vi.mocked(prepareNativeWorkspaceSync).mockImplementationOnce(async () => {
|
|
const saved = await read(f.run.id);
|
|
expect(saved).toMatchObject({ runtimeMode: "native", controllerBootId: null, controllerLeaseExpiresAt: null, executionStage: null });
|
|
expect(await coordinators(f.run.id)).toHaveLength(1);
|
|
expect(f.input.stopControllerLease).toHaveBeenCalledOnce();
|
|
return null;
|
|
});
|
|
expect((await selectHeartbeatRuntime(db, f.input)).selected).toBe(true);
|
|
expect(await read(f.run.id)).toMatchObject({ runtimeMode: "native", controllerBootId: null, controllerLeaseExpiresAt: null, executionStage: null });
|
|
});
|
|
|
|
it.each(["foreign_boot", "expired", "cancelled"] as const)("keeps preparation ownership when native admission is denied: %s", async denied => {
|
|
const f = await fixture();
|
|
await claimPreparation(f);
|
|
const patch = denied === "foreign_boot" ? { controllerBootId: randomUUID() } : denied === "expired" ? { controllerLeaseExpiresAt: new Date(0) } : { status: "cancelled" as const };
|
|
await db.update(heartbeatRuns).set(patch).where(eq(heartbeatRuns.id, f.run.id));
|
|
const before = await read(f.run.id);
|
|
expect(await selectHeartbeatRuntime(db, f.input)).toEqual({ selected: false });
|
|
const after = await read(f.run.id);
|
|
expect(after).toMatchObject({ runtimeMode: "legacy", controllerBootId: before.controllerBootId, controllerLeaseExpiresAt: before.controllerLeaseExpiresAt, executionStage: before.executionStage });
|
|
expect(await coordinators(f.run.id)).toHaveLength(0);
|
|
expect(f.input.stopControllerLease).not.toHaveBeenCalled();
|
|
expect(prepareNativeWorkspaceSync).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it("rolls the preparation handoff back if the native coordinator cannot commit", async () => {
|
|
const f = await fixture();
|
|
await claimPreparation(f);
|
|
const before = await read(f.run.id);
|
|
await db.execute(sql`CREATE FUNCTION reject_test_native_coordinator() RETURNS trigger LANGUAGE plpgsql AS $$ BEGIN RAISE EXCEPTION 'controlled_native_coordinator_rejected'; END $$`);
|
|
await db.execute(sql`CREATE TRIGGER reject_test_native_coordinator BEFORE INSERT ON native_run_finalizations FOR EACH ROW EXECUTE FUNCTION reject_test_native_coordinator()`);
|
|
try {
|
|
await expect(selectHeartbeatRuntime(db, f.input)).rejects.toMatchObject({ cause: { code: "P0001", message: "controlled_native_coordinator_rejected" } });
|
|
expect(await read(f.run.id)).toMatchObject({ runtimeMode: "legacy", controllerBootId: before.controllerBootId, controllerLeaseExpiresAt: before.controllerLeaseExpiresAt, executionStage: before.executionStage, nativeSessionId: null });
|
|
expect(await coordinators(f.run.id)).toHaveLength(0);
|
|
expect(f.input.stopControllerLease).not.toHaveBeenCalled();
|
|
expect(prepareNativeWorkspaceSync).not.toHaveBeenCalled();
|
|
} finally {
|
|
await db.execute(sql`DROP TRIGGER reject_test_native_coordinator ON native_run_finalizations`);
|
|
await db.execute(sql`DROP FUNCTION reject_test_native_coordinator()`);
|
|
}
|
|
});
|
|
|
|
it("keeps committed native authority after a later workspace synchronization failure", async () => {
|
|
const f = await fixture();
|
|
await claimPreparation(f);
|
|
f.input.workspace.executionTarget = sandbox;
|
|
vi.mocked(prepareNativeWorkspaceSync).mockRejectedValueOnce(new Error("controlled_workspace_sync_failure"));
|
|
await expect(selectHeartbeatRuntime(db, f.input)).rejects.toThrow("controlled_workspace_sync_failure");
|
|
expect(await read(f.run.id)).toMatchObject({ runtimeMode: "native", controllerBootId: null, controllerLeaseExpiresAt: null, executionStage: null });
|
|
expect(await coordinators(f.run.id)).toHaveLength(1);
|
|
expect(f.input.stopControllerLease).toHaveBeenCalledOnce();
|
|
});
|
|
|
|
});
|