mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-11 21:59:38 +01:00
## Thinking Path > - Paperclip helps people manage AI agents at work. > - Cursor Cloud hosts execution behind its API. > - Product setup still expected a Paperclip Computer. > - Managed-only tenants therefore rejected valid setup. > - This admits the validated API-owned controller. > - Authority, Stop proof, budgets and cleanup remain enforced. ## Linked Issues or Issue Description Refs #18068 and #18069; both dependencies have merged. **What happened?** Managed-only setup still requested a Computer. Live tests exposed missing Cursor Stop proof and duplicate replay. **Expected behavior** Use Cursor’s API controller. Verify provider/controller termination and replay each log identity once. **Steps to reproduce** Enable native Runner/managed-only. Configure Cursor model/repository/key without a Computer, then Test, finish setup, run and Stop. ## What Changed - Derive API-only placement from validated native Cursor configuration; reject placement/command/workspace/home overrides and keep Kubernetes-only execution closed. - Skip Computers/checkout/workspace realization. Keep scoped tools, completion, private staging, budgets, leases and recovery. Persist placement; reject contradictions. - Align Cloud cards, Test, Save and Finish setup; hold unready drafts. Legacy keeps its Computer, credential and policy checks. - Verify Stop from exact API ownership, audited intent, provider-terminal proof, no Computer/process ownership and controller retirement. Recheck immediate/deferred retirement; wake existing recovery on durable evidence. - Clear warnings when the actual stopped target later has authoritative confirmation. Keep missing/other-run negatives and the existing predicate. - Publish safe status refresh after full-proof ACK. Retain bounded native/physical cancellation history; add no poller or weaker proof path. ## Verification Core merged as `3910352c923a`; Product merged as `27d35314be8e`. Public branch `5f86eb66b081` has tree `67344151d303`. Its exact-head Greptile summary is 5/5; the fresh audit found zero unresolved threads and complete pagination. The controlled workspace retry and required summary passed. All 59 current-head contexts are terminal and clean. Prior green-36 evidence keeps its identity. [Current CI](https://github.com/paperclipai/paperclip/actions/runs/41680511731) first failed workspaces-a 1/2: the CLI company-import/export beforeAll 30-second runtime-info/child-PID/HTTP readiness wait expired. All 431 UI files/4,714 tests and 285 other CLI tests passed (one skip). The cause remains UNKNOWN. Child output records startup recovery complete, but buffered CI prefixes do not establish its relation to the waiter deadline; runtime-file/PID/last-health state was not captured. Both unchanged local probes were RED. The second observed dependency preflight still pending at the deadline; observer overhead was not quantified. Neither proves the CI cause. After all functional jobs finished, only the queued metadata summary was cancelled; the other 48 records stayed unchanged. Attempt 1 retains 46 successes, the original failure, one skip and the cancelled summary. One same-head debug retry was admitted at 18:42 UTC on November 3. Its workspace job passed at 19:23 UTC on the same checkout: UI 431 files/4,714 tests and CLI 33 files/286 tests, including company import/export. Test caching was disabled. The new `ci / verify` summary passed; workflow attempt 2 completed SUCCESS. Required `ci / e2e` also passed. The final check at 19:53 UTC on November 3 found all 59 current-head contexts terminal and clean: zero pending and zero failing. The 47 inherited completed/skipped rows are not new executions. No code, assertion or timeout changed; success does not explain the original failure or erase the local REDs. CI checkout is `1ea9d8608ced`/tree `78cb1ec13d59` (parents `2e37c73c05fe`, 5f86); both workflows use 2e37. Its 20 extra paths add low-memory build/Rust task-memory. Scripts cohere; later matrix changes only labels, with no newer debug-cache steps. Static audit found no blocker. Supported API arguments omit the memory flag: no default monitor/instruction changes. Flagged external launchers can change Cursor instructions/digests and remain unqualified. The merge added 41 master-identical paths, retained the green-36 inbox hunk and found UI fixtures matching upstream. Startup/backup, recovery projection, sleep diagnostics and binary backup change production. Static audit found eight Stop/ACK owner blobs and 37 sleep-work SQL templates unchanged, with sleep fences/single-flight accounting intact. Exact precommit tree: 416/416 across eight server files, binary backup 1/1, server types PASS; repeated scopes are not summed. The deferred-retirement P1 reproduced three real-DB/owned-process failures; missing proof stayed unverified. Recovery passed 47 process cases/33 proof negatives. Warning repair binds actual targets to same-run proof. Validation passed 729 scoped UI cases, a separate 359-case thread run, 111 backend cases (84 CAS, 6 assembled, 21 activity/privacy), types/tokens. Proof validators/predicate are unchanged; no new poller or increased timeout. The d8d3 attempt stays FAILED: 45 passes, two workspaces-a failures, derived verify failure and one skip. Both shards shared three failures: two successful mocks omitted the account guard’s required userId. The two-line fixture repair changed no guard, assertion or timeout. Three failures reproduced; 54 access/auth tests then passed, including account-change/denial negatives, plus UI types/tokens. The 69285 attempt stays FAILED: 49 jobs, 46 passed, serialized shard 9 failed, its verify aggregate failed and one selection job skipped. The shard passed 575/576 cases. Its aborted-snapshot case expected nine computes but saw eight: a final HTTP 200 coalesced while seven ordinary reads had not proved the aborted compute settled. Earlier CI and the unmodified local baseline passed with a miss; no local RED is claimed. The repair awaits the snapshot, allows exactly one extra GET if coalesced, then requires a miss and retains nine computes/all HTTP429 assertions. No production code, hook, loop, sleep or timeout changed. Six inbox and seven cache/admission cases passed locally, plus server types; 30 unrelated cases were filtered. Green 36 passed 576/576 without taking the extra-GET branch; local runs also missed that schedule. Current CI event merge 1ea9 now passed 576/576 and naturally exercised the former failure: coalesced (count 2, 1ms), then exactly one fresh miss (count 1, 16ms), with strict nine-compute/HTTP429 assertions intact. No instrumentation or retry was added. Earlier compatibility passed scoped remote/inbox, routine, API-placement and UI tests, plus types/migrations/snapshots/maps/tokens. The 89-path adoption restored fleet routing without workflow overrides or added skips. Each result retains its source identity. Live qualification remains actual `9af1a5de42a2`: unchanged Artifact grader PASSED 17/17 and active-Stop grader PASSED 21/21. All three provider test resources were deleted canonically with GET404; post-delete download returned the exact 27 bytes. Histories stayed stable (Artifact 121 and Stop 67 canonical identities), with 21 company runs, zero active work and no accounting/cancellation/reservation debt or integrity findings. Saved settings reopened; no-prompt Test/Linux admission passed with `claude-sonnet-4-5`. No new Save is claimed. The first Stop sampling case stays FAILED on a 10.373-second gap. Unchanged retry kept all 24 samples: 35.076 seconds quiet, maximum gap 4.584 seconds, plus a separate CANCELLED/artifact-absent read after 180 seconds. Initial grading rejected locator spelling before functional checks; metadata changed only after same-file/inode/byte proof, with raw evidence/oracles unchanged. Original Question/Plan/reconnect and historical deleted-artifact results keep their identities. Git trees verified 11 history-only conflicts and six earlier mainline paths. Hosted 9af to the public BRANCH differs in 222 paths; 13,500 other mode/blob leaves match. Runner runtime implementations and six observer/driver pins match 9af on that branch only; inherited `drivers/acpx/runtime-host.test.ts` differs. No all-Runner/src, CI/latest-master runtime or image equality is claimed. Later static audits through `8ab89f41e499` found no concrete API-only blocker. That base adds completed-history sleep proofs and an MCP cleanup migration; protected Stop/ACK owners are unchanged. These later changes are outside this CI and have no live qualification here. Host duplicate-card and later recovery/UI repairs retain separate proof, not deployed 9af identity. Direct-key coverage requires absent aiConnection; API-only placement skips inherited remote-workspace disk-full paths, and config-read gates match their parent. Other retained failures: stale fixture 132/133; E8 48/135 with 87 reset-hook failures; Mac 17 SQL failures/skips; title/startup/custody/Stop/deletion unknown causes and bounded recoveries; superseded warning/50 duplicate identities; pre-repair ACK/Core CI failures. Fixture RED/GREEN passed 7/7 browser and 121/121 shutdown tests with a failing broken-unsubscribe control; stable WSP wrapper passed 128 after physical teardown failure. No full local-suite pass is inferred. All paid attempts remain recorded; vendor dollar cost is UNKNOWN. ## Risks Experimental: Cursor independently acknowledges neither the effective model nor instruction digest before the first prompt. Explicit values are submitted and behavior checked live; unpriced provider dollar cost stays UNKNOWN. The placement exception applies only to validated native Cursor API execution. Other local work remains denied under managed-only policy. Missing/stale Stop or cleanup proof stays pending; damaged archives remain failed. Explicit Legacy keeps its existing policy owners. ## Model Used OpenAI Codex/GPT-6 family: tools, execution, browser checks, delegated review. API ID/context window unavailable. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used and the limits of available model metadata - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have linked public issues or described the issue in this PR - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch describes the change and contains no internal ticket ID - [x] I have run the scoped local tests reported above; retained failures are stated separately - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation - [x] I have considered and documented the risks - [x] All required CI gates pass on the current head - [x] Fresh Greptile is 5/5 with no unresolved findings on the current head - [x] I will address all reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
1874 lines
140 KiB
TypeScript
1874 lines
140 KiB
TypeScript
import { execFile as execFileCallback } from "node:child_process";
|
|
import { randomUUID } from "node:crypto";
|
|
import { mkdtemp, mkdir, readFile, readdir, rm, writeFile, access, rename, symlink } from "node:fs/promises";
|
|
import { tmpdir } from "node:os";
|
|
import { join } from "node:path";
|
|
import { pathToFileURL } from "node:url";
|
|
import { promisify } from "node:util";
|
|
import { captureDirectorySnapshot, directorySnapshotSha256, serializeDirectorySnapshot } from "@paperclipai/adapter-utils/workspace-restore-merge";
|
|
import type { AdapterExecutionTarget } from "@paperclipai/adapter-utils/execution-target";
|
|
import { agentWakeupRequests, heartbeatRunEvents, companyObservability, issueRecoveryActions, agents, authUsers, companyMemberships, companies, createDb, applyPendingMigrations, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, environmentLeases, environments, executionWorkspaces, executionWorkspaceRepositoryAccess, heartbeatRuns, issues, issueAccessGrants, nativeRunFinalizations, plugins, projects, projectWorkspaces, workspaceOperations, type Db } from "@paperclipai/db";
|
|
import { eq, sql } from "drizzle-orm";
|
|
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
|
|
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
|
|
import { resolveExternalTestDatabaseUrl } from "../../__tests__/helpers/external-test-database.js";
|
|
import { heartbeatService } from "../heartbeat.js";
|
|
import { registerServerAdapter, getServerAdapter, type AdapterExecutionContext } from "../../adapters/index.js";
|
|
import { environmentRunOrchestrator } from "../environment-run-orchestrator.js";
|
|
import { environmentService } from "../environments.js";
|
|
import { executionWorkspaceService, toExecutionWorkspace } from "../execution-workspaces.js";
|
|
import { instanceSettingsService } from "../instance-settings.js";
|
|
import { workspaceOperationService } from "../workspace-operations.js";
|
|
import { nativeWorkspaceSyncInternals, prepareNativeWorkspaceSync } from "../native-runtime/native-workspace-sync.js";
|
|
import { createHeartbeatWorkspaceResolver, prepareProjectRepositoryWorkspaces, resolveExecutionWorkspaceReuseRequestForIssue } from "./workspaces.js";
|
|
import { prepareHeartbeatWorkspace, type HeartbeatWorkspacePreparationInput } from "./workspace-preparation.js";
|
|
import { buildNativeExecutionInput } from "../native-runtime/native-execution-input.js";
|
|
import { nativeRuntimeContextFixture } from "../native-runtime/runtime-context.test-fixture.js";
|
|
import * as managedRepositoryCheckout from "../managed-repository-checkout.js";
|
|
import * as freshWorkspaceSeed from "../fresh-workspace-seed.js";
|
|
import { requestTaskRetry } from "../task-retry.js";
|
|
import { readFreshWorkspaceReadiness } from "../fresh-workspace-continuation.js";
|
|
import { createSystemRecovery } from "../system-recovery-state.js";
|
|
import { isolatedRemoteSourcePath } from "../isolated-remote-workspace-source.js";
|
|
import { cleanupExecutionWorkspaceArtifacts } from "../workspace-runtime.js";
|
|
import { buildProjectMentionHref, readRemoteWorkspaceAuthority } from "@paperclipai/shared";
|
|
import { migrateRemoteWorkspace } from "../remote-workspace-migration.js";
|
|
import { trace } from "@opentelemetry/api";
|
|
import { effectiveProjectWorkspacePolicy } from "../project-workspace-provider.js";
|
|
import { resolveExecutionWorkspaceMode, buildExecutionWorkspaceAdapterConfig } from "../execution-workspace-policy.js";
|
|
import { createLocalDiskStorageProvider } from "../../storage/local-disk-provider.js";
|
|
import { createStorageService } from "../../storage/service.js";
|
|
import { InMemorySpanExporter, SimpleSpanProcessor, TracerProvider } from "@opentelemetry/sdk-trace";
|
|
import { selectHeartbeatEnvironment } from "./environment-selection.js";
|
|
import { assertNativeSessionPreparationAuthority } from "../native-runtime/native-execution-lease.js";
|
|
import { currentNativeControllerIdentity } from "../native-runtime/native-restart-recovery.js";
|
|
import { projectService } from "../projects.js";
|
|
import { WORKSPACE_BUSY_HOLDER_STALE_AFTER_MS, WorkspaceBusyDeferral } from "./retries.js";
|
|
|
|
const execFile = promisify(execFileCallback);
|
|
const externalDatabaseUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_WORKSPACE_PREPARATION_TEST_DATABASE_URL");
|
|
const support = externalDatabaseUrl ? { supported: true } : await getEmbeddedPostgresTestSupport();
|
|
|
|
describe.skipIf(!support.supported && !externalDatabaseUrl)("heartbeat workspace preparation boundary", () => {
|
|
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
|
let db: Db;
|
|
let home: string;
|
|
const persistenceKeys = ["PAPERCLIP_EPHEMERAL_SERVER", "DATABASE_URL", "PAPERCLIP_STORAGE_PROVIDER", "PAPERCLIP_SECRETS_MASTER_KEY", "PAPERCLIP_SECRETS_MASTER_KEY_FILE"];
|
|
let persistenceBefore = new Map<string, string | undefined>();
|
|
const configureRemotePersistence = () => Object.assign(process.env, { DATABASE_URL: database.connectionString,
|
|
PAPERCLIP_STORAGE_PROVIDER: "s3", PAPERCLIP_SECRETS_MASTER_KEY: "remote-workspace-preparation-fixture" });
|
|
|
|
beforeAll(async () => {
|
|
home = await mkdtemp(join(tmpdir(), "paperclip-workspace-preparation-"));
|
|
vi.stubEnv("PAPERCLIP_HOME", home);
|
|
const external = externalDatabaseUrl;
|
|
if (external) await applyPendingMigrations(external);
|
|
database = external ? { connectionString: external, cleanup: async () => {} }
|
|
: await startEmbeddedPostgresTestDatabase("paperclip-workspace-preparation");
|
|
db = createDb(database.connectionString);
|
|
await db.execute(sql`set client_min_messages = warning`);
|
|
if (external) await db.execute(sql`truncate table environments restart identity cascade`);
|
|
}, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS);
|
|
afterAll(async () => {
|
|
await db?.$client.end({ timeout: 1 });
|
|
await database?.cleanup();
|
|
vi.unstubAllEnvs();
|
|
if (home) await rm(home, { recursive: true, force: true });
|
|
}, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS);
|
|
beforeEach(async () => {
|
|
persistenceBefore = new Map(persistenceKeys.map(key => [key, process.env[key]]));
|
|
vi.restoreAllMocks();
|
|
await db.execute(sql`truncate table companies, plugins restart identity cascade`);
|
|
await instanceSettingsService(db).updateExperimental({ enableIsolatedWorkspaces: true });
|
|
});
|
|
|
|
afterEach(() => {
|
|
for (const [key, value] of persistenceBefore) {
|
|
if (value === undefined) delete process.env[key]; else process.env[key] = value;
|
|
}
|
|
});
|
|
|
|
async function fixture(isolated = false) {
|
|
const [company] = await db.insert(companies).values({ name: "Workspace preparation", issuePrefix: "WSP" }).returning();
|
|
const [agent] = await db.insert(agents).values({ companyId: company.id, name: "Workspace agent", status: "running", adapterType: "process" }).returning();
|
|
const [project] = await db.insert(projects).values({ companyId: company.id, name: "Source project" }).returning();
|
|
const source = join(home, project.id);
|
|
await mkdir(source);
|
|
await execFile("git", ["init", "--initial-branch=main", source]);
|
|
await execFile("git", ["-C", source, "config", "user.email", "test@example.com"]);
|
|
await execFile("git", ["-C", source, "config", "user.name", "Workspace test"]);
|
|
await writeFile(join(source, "README.md"), "source content\n");
|
|
await execFile("git", ["-C", source, "add", "."]);
|
|
await execFile("git", ["-C", source, "commit", "-m", "Source"]);
|
|
const [projectWorkspace] = await db.insert(projectWorkspaces).values({ companyId: company.id, projectId: project.id, name: "Source", cwd: source, isPrimary: true }).returning();
|
|
const [issue] = await db.insert(issues).values({ companyId: company.id, title: "Prepare this workspace", identifier: "WSP-1", projectId: project.id, projectWorkspaceId: projectWorkspace.id, assigneeAgentId: agent.id }).returning();
|
|
const [run] = await db.insert(heartbeatRuns).values({ companyId: company.id, agentId: agent.id, issueId: issue.id, scopeKind: "issue", status: "running", runtimeMode: "legacy" }).returning();
|
|
const localEnvironment = await environmentService(db).ensureLocalEnvironment(company.id);
|
|
const envOrchestrator = environmentRunOrchestrator(db);
|
|
const resolver = createHeartbeatWorkspaceResolver(db);
|
|
const input: HeartbeatWorkspacePreparationInput = {
|
|
run, agent,
|
|
task: { issueRef: issue, issueId: issue.id, wasIssueAssignee: true, context: { issueId: issue.id, projectId: project.id }, responsibleUserId: null, previousSessionParams: null },
|
|
policy: {
|
|
trustPreset: { kind: "standard", preset: "standard", boundary: null, sourcePresets: {} },
|
|
isolatedWorkspacesEnabled: true,
|
|
effectiveExecutionWorkspaceMode: isolated ? "isolated_workspace" : "shared_workspace",
|
|
requestedExecutionWorkspaceMode: isolated ? "isolated_workspace" : "shared_workspace",
|
|
useIsolatedTaskDirectory: false, nativeChatWorkspaceScope: null,
|
|
projectExecutionWorkspacePolicy: null, issueExecutionWorkspaceSettings: null, environmentExecutionWorkspaceSettings: null,
|
|
executionPolicy: { executionMode: "any" },
|
|
},
|
|
environment: { selectedEnvironmentId: localEnvironment.id, localEnvironment, selectedEnvironmentForConfig: localEnvironment, environmentResolution: { environmentId: localEnvironment.id, source: "default" }, resolvedInstanceSettings: await instanceSettingsService(db).get() },
|
|
config: { mergedConfig: isolated ? { workspaceStrategy: { type: "git_worktree", baseRef: "main", branchTemplate: "test-{{issue.identifier}}" } } : {}, configSnapshot: null, secretManifest: [] },
|
|
reuse: { requestedShouldReuseExisting: false, existingExecutionWorkspace: null, reusableExistingExecutionWorkspace: null, workspaceReuseRequest: resolveExecutionWorkspaceReuseRequestForIssue({ issueExecutionWorkspaceId: null, issueExecutionWorkspacePreference: null, existingExecutionWorkspaceStatus: null }), nativeRecoveryExecutionWorkspaceId: null, persistedNativeExecutionInput: null, isDotRun: false, runOptions: {} },
|
|
services: { envOrchestrator, executionWorkspacesSvc: executionWorkspaceService(db), workspaceOperationsSvc: workspaceOperationService(db), ...resolver, appendRunEvent: vi.fn() },
|
|
controllerLease: { assertOwned: vi.fn(async () => {}) }, nativeRunnerPreparationSpans: [],
|
|
};
|
|
return { input, source, issue, run, envOrchestrator };
|
|
}
|
|
it.each([false, true])("places Cursor Cloud in its provider API without a Computer under managed-only policy (ephemeral: %s)", async ephemeral => {
|
|
vi.stubEnv("PAPERCLIP_EPHEMERAL_SERVER", String(ephemeral));
|
|
const { input, source, issue, run } = await fixture();
|
|
input.agent = { ...input.agent, adapterType: "paperclip_runner", adapterConfig: {
|
|
provider: "cursor_cloud", model: "caller-model", repoUrl: "https://github.com/example/qualification",
|
|
} };
|
|
await db.update(agents).set({ adapterType: input.agent.adapterType, adapterConfig: input.agent.adapterConfig }).where(eq(agents.id, input.agent.id));
|
|
const settingsBefore = await instanceSettingsService(db).get();
|
|
try {
|
|
await instanceSettingsService(db).updateExperimental({ enableManagedSandboxOnly: true, enableNativeRunner: true });
|
|
const selectionInput = {
|
|
run: input.run, agent: input.agent,
|
|
task: { issueRef: issue, issueContext: issue, issueId: issue.id, context: input.task.context,
|
|
executionProjectId: issue.projectId, workspaceProjectContext: { hasWorkspace: true }, boundSourceWorkspace: null,
|
|
hasSelectedWorkspaceSource: true, taskWorkspaceIntent: null, responsibleUserId: null },
|
|
policy: { ...input.policy }, config: { config: input.agent.adapterConfig, sourceWorkspaceConfig: input.agent.adapterConfig, issueAssigneeOverrides: null },
|
|
services: { environmentsSvc: environmentService(db), executionWorkspacesSvc: executionWorkspaceService(db),
|
|
instanceSettings: instanceSettingsService(db), findSharedWorkspaceHolder: async () => null, findExecutionWorkspaceHolder: async () => null },
|
|
};
|
|
const selection = await selectHeartbeatEnvironment(db, selectionInput);
|
|
expect(selection).toMatchObject({ providerApiPlacement: { kind: "provider_api", provider: "cursor_cloud", controller: "paperclip" },
|
|
selectedEnvironmentId: null, selectedEnvironmentForConfig: null, executionPolicy: { managedSandboxOnly: true } });
|
|
input.environment = { ...input.environment, selectedEnvironmentId: selection.selectedEnvironmentId,
|
|
selectedEnvironmentForConfig: selection.selectedEnvironmentForConfig, environmentResolution: selection.environmentResolution,
|
|
resolvedInstanceSettings: selection.resolvedInstanceSettings };
|
|
input.policy.executionPolicy = selection.executionPolicy;
|
|
const acquire = vi.spyOn(input.services.envOrchestrator, "acquireForRun");
|
|
const realize = vi.spyOn(input.services.envOrchestrator, "realizeForRun");
|
|
const resolveWorkspace = vi.spyOn(input.services, "resolveWorkspaceForRun");
|
|
const result = await prepareHeartbeatWorkspace(db, input);
|
|
expect(result).toMatchObject({ selectedEnvironment: null, activeEnvironmentLease: null, executionTarget: null,
|
|
persistedExecutionWorkspace: null, projectRepositoryPaths: [], hostExecutionWorkspaceConfig: {}, skipHostProvision: true });
|
|
expect(result.executionWorkspace.cwd).toContain(`/cursor-cloud-controllers/${input.agent.companyId}/${input.agent.id}/${issue.id}`);
|
|
expect(result.executionWorkspace).toMatchObject({ projectId: null, repoUrl: null, repoRef: null, created: false });
|
|
expect(await readFile(join(source, "README.md"), "utf8")).toBe("source content\n");
|
|
expect(acquire).not.toHaveBeenCalled(); expect(realize).not.toHaveBeenCalled(); expect(resolveWorkspace).not.toHaveBeenCalled();
|
|
expect(await db.select().from(environmentLeases).where(eq(environmentLeases.heartbeatRunId, run.id))).toEqual([]);
|
|
expect(await db.select().from(executionWorkspaces)).toEqual([]);
|
|
expect((await instanceSettingsService(db).getExperimental()).enableManagedSandboxOnly).toBe(true);
|
|
await instanceSettingsService(db).updateGeneral({ executionMode: "kubernetes" });
|
|
await expect(selectHeartbeatEnvironment(db, selectionInput)).rejects.toMatchObject({ status: 422,
|
|
details: { code: "provider_api_kubernetes_execution_unsupported" } });
|
|
} finally {
|
|
await instanceSettingsService(db).updateGeneral({ executionMode: settingsBefore.general.executionMode ?? "any" });
|
|
await instanceSettingsService(db).updateExperimental({
|
|
enableManagedSandboxOnly: settingsBefore.experimental.enableManagedSandboxOnly,
|
|
enableNativeRunner: settingsBefore.experimental.enableNativeRunner,
|
|
});
|
|
}
|
|
});
|
|
it.each(["unconfigured", "operator-disabled"] as const)("auto-enrollment keeps %s native sandboxes on their existing host path", async mode => {
|
|
const { input, source, envOrchestrator } = await fixture();
|
|
input.agent.adapterType = "paperclip_runner";
|
|
input.environment.selectedEnvironmentForConfig = { ...input.environment.localEnvironment,
|
|
driver: "sandbox", config: { provider: "daytona" } };
|
|
input.environment.resolvedInstanceSettings.experimental.remoteWorkspaceStorageEnabled = true;
|
|
if (mode === "operator-disabled") {
|
|
configureRemotePersistence();
|
|
input.environment.resolvedInstanceSettings.experimental.remoteWorkspaceStorageEnabled = false;
|
|
} else {
|
|
Object.assign(process.env, { DATABASE_URL: database.connectionString, PAPERCLIP_STORAGE_PROVIDER: "local_disk" });
|
|
}
|
|
const resolve = vi.spyOn(input.services, "resolveWorkspaceForRun").mockRejectedValue(new Error("host authority retained"));
|
|
const acquire = vi.spyOn(envOrchestrator, "acquireForRun");
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow("host authority retained");
|
|
expect(resolve).toHaveBeenCalledOnce(); expect(acquire).not.toHaveBeenCalled();
|
|
expect(await readFile(join(source, "README.md"), "utf8")).toBe("source content\n");
|
|
expect(await db.select().from(executionWorkspaces)).toHaveLength(0);
|
|
});
|
|
|
|
it("auto-enrollment preserves an existing host workspace even when persistence is ready", async () => {
|
|
const { input, source, issue, envOrchestrator } = await fixture();
|
|
const [workspace] = await db.insert(executionWorkspaces).values({ companyId: issue.companyId,
|
|
sourceIssueId: issue.id, projectId: issue.projectId, projectWorkspaceId: issue.projectWorkspaceId,
|
|
name: "Existing host workspace", cwd: source, providerType: "local_fs", strategyType: "project_primary",
|
|
mode: "shared_workspace", status: "active" }).returning();
|
|
await writeFile(join(source, "unpublished.txt"), "keep existing work\n");
|
|
await db.update(issues).set({ executionWorkspaceId: workspace.id }).where(eq(issues.id, issue.id));
|
|
input.reuse.existingExecutionWorkspace = toExecutionWorkspace(workspace);
|
|
input.reuse.reusableExistingExecutionWorkspace = toExecutionWorkspace(workspace);
|
|
input.reuse.requestedShouldReuseExisting = true;
|
|
input.agent.adapterType = "paperclip_runner";
|
|
input.environment.selectedEnvironmentForConfig = { ...input.environment.localEnvironment,
|
|
driver: "sandbox", config: { provider: "daytona" } };
|
|
input.environment.resolvedInstanceSettings.experimental.remoteWorkspaceStorageEnabled = true;
|
|
configureRemotePersistence();
|
|
const resolve = vi.spyOn(input.services, "resolveWorkspaceForRun").mockRejectedValue(new Error("host authority retained"));
|
|
const acquire = vi.spyOn(envOrchestrator, "acquireForRun");
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow("host authority retained");
|
|
expect(resolve).toHaveBeenCalledOnce(); expect(acquire).not.toHaveBeenCalled();
|
|
expect(await readFile(join(source, "unpublished.txt"), "utf8")).toBe("keep existing work\n");
|
|
expect((await db.select().from(executionWorkspaces))[0]?.metadata?.remoteWorkspace).toBeUndefined();
|
|
});
|
|
|
|
it.each([
|
|
{ kind: "empty", repoUrl: null, branchMode: null, fresh: false, strategyChange: false, ephemeral: true },
|
|
{ kind: "git", repoUrl: "https://github.com/example/remote.git", branchMode: null, fresh: false, strategyChange: false },
|
|
{ kind: "git", repoUrl: "ssh://git@github.com/example/remote.git", branchMode: null, fresh: false, strategyChange: false },
|
|
{ kind: "git", repoUrl: "git@github.com:example/remote.git", branchMode: null, fresh: false, strategyChange: false },
|
|
{ kind: "git", repoUrl: "https://github.com/example/remote.git", branchMode: "existing", fresh: false, strategyChange: false },
|
|
{ kind: "git", repoUrl: "https://github.com/example/remote.git", branchMode: "create", fresh: false, strategyChange: false },
|
|
{ kind: "git", repoUrl: "https://github.com/example/remote.git", branchMode: "create", fresh: true, strategyChange: false },
|
|
{ kind: "git", repoUrl: "https://github.com/example/remote.git", branchMode: "create", fresh: true, strategyChange: true },
|
|
] as const)("reserves a $kind remote workspace ($repoUrl, branch: $branchMode, fresh: $fresh, strategy change: $strategyChange) without materializing a controller checkout", async (scenario) => {
|
|
const { kind, repoUrl, branchMode, fresh, strategyChange } = scenario;
|
|
const { input, source, issue, envOrchestrator } = await fixture();
|
|
const [savedEnvironment] = await db.insert(environments).values({ name: `Remote ${randomUUID()}`,
|
|
driver: "sandbox", status: "active", config: { provider: "daytona" } }).returning();
|
|
const environment = { ...input.environment.localEnvironment, id: savedEnvironment.id,
|
|
driver: "sandbox" as const, config: { provider: "daytona" } };
|
|
await db.update(agents).set({ adapterType: "paperclip_runner" }).where(eq(agents.id, input.agent.id));
|
|
input.agent.adapterType = "paperclip_runner";
|
|
input.environment.selectedEnvironmentForConfig = environment;
|
|
input.environment.selectedEnvironmentId = environment.id;
|
|
expect(input.environment.resolvedInstanceSettings.experimental.remoteWorkspaceStorageEnabled).toBe(true);
|
|
input.policy.requestedExecutionWorkspaceMode = "agent_default";
|
|
input.policy.useIsolatedTaskDirectory = kind === "empty";
|
|
if (branchMode) {
|
|
input.policy.requestedExecutionWorkspaceMode = "isolated_workspace";
|
|
input.policy.effectiveExecutionWorkspaceMode = "isolated_workspace";
|
|
input.config.mergedConfig = strategyChange ? { workspaceStrategy: { type: "project_primary" } } : { workspaceStrategy: { type: "git_worktree", baseRef: "main",
|
|
...(branchMode === "existing" ? { existingBranch: "feature/diverged" } : { branchTemplate: "test-{{issue.identifier}}" }) } };
|
|
}
|
|
if (kind === "git") {
|
|
await db.update(projectWorkspaces).set({ cwd: null, sourceType: "git", repoUrl, repoRef: "main" })
|
|
.where(eq(projectWorkspaces.id, issue.projectWorkspaceId!));
|
|
}
|
|
if (fresh) {
|
|
const sourceRunId = randomUUID(), retryIntentId = randomUUID();
|
|
await db.insert(heartbeatRuns).values({ id: sourceRunId, companyId: issue.companyId, agentId: input.agent.id,
|
|
issueId: issue.id, status: "failed", finishedAt: new Date() });
|
|
await db.insert(agentWakeupRequests).values({ id: retryIntentId, companyId: issue.companyId,
|
|
agentId: input.agent.id, reason: "task_retry", source: "on_demand", status: "coalesced", runId: input.run.id,
|
|
payload: { issueId: issue.id } });
|
|
const readiness = { mode: "fresh_isolated_workspace" as const, retryIntentId, sourceRunId, retainedSourceIds: [] };
|
|
await db.insert(issueRecoveryActions).values({ companyId: issue.companyId, sourceIssueId: issue.id,
|
|
kind: "active_run_watchdog", status: "resolved", ownerType: "system", nextAction: "Synthetic committed fresh attempt",
|
|
cause: "native_continuation_requires_reconciliation", fingerprint: randomUUID(), evidence: { runId: sourceRunId,
|
|
explicitUserContinuation: { runId: input.run.id, readiness }, systemRecovery: {
|
|
...createSystemRecovery(sourceRunId, retryIntentId), state: "exhausted", freshAttemptRunId: input.run.id,
|
|
} } });
|
|
input.task.freshWorkspace = await readFreshWorkspaceReadiness(db, input.run);
|
|
expect(input.task.freshWorkspace).toEqual(readiness);
|
|
}
|
|
await rm(source, { recursive: true });
|
|
const resolve = vi.spyOn(input.services, "resolveWorkspaceForRun").mockRejectedValue(new Error("host source resolver called"));
|
|
const hostSeed = vi.spyOn(freshWorkspaceSeed, "materializeFreshWorkspaceSeed");
|
|
const repositoryPreparation = vi.spyOn(input.services.executionWorkspacesSvc, "prepareTaskRepositoriesForAdmission");
|
|
vi.spyOn(envOrchestrator, "acquireForRun").mockImplementation(async args => {
|
|
expect(args.persistedExecutionWorkspace).toMatchObject({ providerType: "cloud_sandbox" });
|
|
expect(args.persistedExecutionWorkspace).toMatchObject({ cwd: expect.stringMatching(/^\/workspaces\//) });
|
|
throw new Error("reached remote acquisition");
|
|
});
|
|
const configured = { DATABASE_URL: database.connectionString, PAPERCLIP_STORAGE_PROVIDER: "s3",
|
|
PAPERCLIP_SECRETS_MASTER_KEY: "remote-workspace-preparation-fixture",
|
|
PAPERCLIP_EPHEMERAL_SERVER: "ephemeral" in scenario ? "true" : "false" };
|
|
const previous = new Map(Object.keys(configured).map(key => [key, process.env[key]]));
|
|
Object.assign(process.env, configured);
|
|
try {
|
|
// Fresh runs are still marked legacy until the runtime selector owns promotion.
|
|
expect(input.run.runtimeMode).toBe("legacy");
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow("reached remote acquisition");
|
|
expect(resolve).not.toHaveBeenCalled();
|
|
expect(repositoryPreparation).not.toHaveBeenCalled();
|
|
expect(hostSeed).not.toHaveBeenCalled();
|
|
const [workspace] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.sourceIssueId, issue.id));
|
|
expect(workspace.mode).toBe(branchMode ? "isolated_workspace" : "shared_workspace");
|
|
expect(workspace.metadata?.remoteWorkspace).toMatchObject({ source: { kind }, writerRunId: input.run.id });
|
|
if (branchMode) expect(workspace.metadata?.remoteWorkspace).toMatchObject({ source: {
|
|
ref: "main", branchMode,
|
|
...(strategyChange ? {} : { branchName: branchMode === "existing" ? "feature/diverged" : "test-WSP-1" }),
|
|
} });
|
|
await expect(access(workspace.cwd!)).rejects.toThrow();
|
|
if ((fresh || strategyChange) && branchMode === "create") {
|
|
expect(workspace.strategyType).toBe("git_worktree");
|
|
await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date() })
|
|
.where(eq(heartbeatRuns.id, input.run.id));
|
|
const [saved] = await db.update(executionWorkspaces).set({ metadata: {
|
|
...workspace.metadata,
|
|
remoteWorkspace: { ...workspace.metadata!.remoteWorkspace as object, writerRunId: null, dirty: false },
|
|
} }).where(eq(executionWorkspaces.id, workspace.id)).returning();
|
|
const [nextRun] = await db.insert(heartbeatRuns).values({ companyId: issue.companyId,
|
|
agentId: input.agent.id, issueId: issue.id, scopeKind: "issue", status: "running", runtimeMode: "legacy" }).returning();
|
|
input.run = nextRun;
|
|
input.task.freshWorkspace = null;
|
|
input.task.issueRef = { ...input.task.issueRef!, executionWorkspaceId: saved.id, executionWorkspacePreference: "reuse_existing" };
|
|
input.reuse.existingExecutionWorkspace = toExecutionWorkspace(saved);
|
|
input.reuse.reusableExistingExecutionWorkspace = toExecutionWorkspace(saved);
|
|
input.reuse.requestedShouldReuseExisting = true;
|
|
input.reuse.workspaceReuseRequest = resolveExecutionWorkspaceReuseRequestForIssue({
|
|
issueExecutionWorkspaceId: saved.id, issueExecutionWorkspacePreference: "reuse_existing", existingExecutionWorkspaceStatus: "active",
|
|
});
|
|
if (strategyChange) input.config.mergedConfig = { workspaceStrategy: { type: "git_worktree" } };
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow("reached remote acquisition");
|
|
const operations = await db.select().from(workspaceOperations).where(eq(workspaceOperations.heartbeatRunId, nextRun.id));
|
|
const freshness = operations.filter(operation => operation.phase === "workspace_config_freshness");
|
|
if (strategyChange) {
|
|
expect(freshness).toHaveLength(1);
|
|
expect(freshness[0].metadata).toMatchObject({ action: "replace", changedCategories: ["strategy"] });
|
|
} else expect(freshness).toEqual([]);
|
|
const [retained] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, saved.id));
|
|
expect(retained.id).toBe(workspace.id);
|
|
expect(retained.metadata?.configFingerprint).toMatchObject({
|
|
workspaceHash: (saved.metadata?.configFingerprint as { workspaceHash: string }).workspaceHash,
|
|
});
|
|
expect(retained.metadata?.remoteWorkspace).toMatchObject({ writerRunId: nextRun.id });
|
|
}
|
|
} finally {
|
|
for (const [key, value] of previous) {
|
|
if (value === undefined) delete process.env[key]; else process.env[key] = value;
|
|
}
|
|
}
|
|
});
|
|
|
|
async function inheritedRemoteFixture(mode: "isolated_workspace" | "operator_branch") {
|
|
const { input, issue, envOrchestrator } = await fixture();
|
|
configureRemotePersistence();
|
|
process.env.PAPERCLIP_SECRETS_MASTER_KEY = "71".repeat(32);
|
|
input.agent.adapterType = "paperclip_runner";
|
|
input.environment.selectedEnvironmentForConfig = { ...input.environment.localEnvironment,
|
|
driver: "sandbox", config: { provider: "daytona" } };
|
|
input.policy.requestedExecutionWorkspaceMode = mode;
|
|
input.policy.effectiveExecutionWorkspaceMode = mode;
|
|
const [parent] = await db.insert(issues).values({ companyId: issue.companyId, projectId: issue.projectId,
|
|
projectWorkspaceId: issue.projectWorkspaceId, title: "Still running coordinator", assigneeAgentId: input.agent.id }).returning();
|
|
const [writer] = await db.insert(heartbeatRuns).values({ companyId: issue.companyId, agentId: input.agent.id,
|
|
nativeIssueId: parent.id, runtimeMode: "native", status: "running", startedAt: new Date(), lastOutputAt: new Date() }).returning();
|
|
const authority = { schema: "paperclip.remote-workspace.v1" as const, epoch: 3,
|
|
snapshotId: randomUUID(), source: { kind: "empty" as const }, location: null, writerRunId: writer.id, dirty: true };
|
|
const [workspace] = await db.insert(executionWorkspaces).values({ companyId: issue.companyId,
|
|
sourceIssueId: parent.id, projectId: issue.projectId, projectWorkspaceId: issue.projectWorkspaceId,
|
|
name: "Inherited remote workspace", mode, strategyType: "project_primary", providerType: "cloud_sandbox",
|
|
cwd: `/workspaces/${randomUUID()}`, metadata: { remoteWorkspace: authority } }).returning();
|
|
await db.update(issues).set({ executionWorkspaceId: workspace.id, executionWorkspacePreference: "reuse_existing" })
|
|
.where(eq(issues.id, issue.id));
|
|
input.task.issueRef = { ...issue, executionWorkspaceId: workspace.id, executionWorkspacePreference: "reuse_existing" };
|
|
input.reuse.existingExecutionWorkspace = toExecutionWorkspace(workspace);
|
|
input.reuse.reusableExistingExecutionWorkspace = toExecutionWorkspace(workspace);
|
|
input.reuse.requestedShouldReuseExisting = true;
|
|
input.reuse.workspaceReuseRequest = resolveExecutionWorkspaceReuseRequestForIssue({ issueExecutionWorkspaceId: workspace.id,
|
|
issueExecutionWorkspacePreference: "reuse_existing", existingExecutionWorkspaceStatus: "active" });
|
|
return { input, issue, envOrchestrator, parent, writer, workspace, authority };
|
|
}
|
|
|
|
it.each(["isolated_workspace", "operator_branch"] as const)("waits for the live writer of an inherited remote %s at reservation", async mode => {
|
|
const { input, issue, envOrchestrator, parent, writer, workspace, authority } = await inheritedRemoteFixture(mode);
|
|
const acquire = vi.spyOn(envOrchestrator, "acquireForRun").mockRejectedValue(new Error("writer released; acquisition admitted"));
|
|
const readAuthority = async () => readRemoteWorkspaceAuthority((await db.select().from(executionWorkspaces)
|
|
.where(eq(executionWorkspaces.id, workspace.id)))[0].metadata);
|
|
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toMatchObject({
|
|
constructor: WorkspaceBusyDeferral, code: "workspace_busy", executionWorkspaceId: workspace.id,
|
|
holder: { runId: writer.id, issueId: parent.id }, wasIssueAssignee: true,
|
|
});
|
|
expect(acquire).not.toHaveBeenCalled();
|
|
expect(await readAuthority()).toEqual(authority);
|
|
|
|
await db.update(heartbeatRuns).set({ lastOutputAt: new Date(Date.now() - WORKSPACE_BUSY_HOLDER_STALE_AFTER_MS - 1_000) })
|
|
.where(eq(heartbeatRuns.id, writer.id));
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toMatchObject({ status: 409 });
|
|
await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date() }).where(eq(heartbeatRuns.id, writer.id));
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toMatchObject({ status: 409 });
|
|
expect(await readAuthority()).toEqual(authority);
|
|
expect(acquire).not.toHaveBeenCalled();
|
|
|
|
const saved = { ...authority, writerRunId: null, dirty: false };
|
|
await db.update(executionWorkspaces).set({ metadata: { remoteWorkspace: saved } }).where(eq(executionWorkspaces.id, workspace.id));
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow("writer released; acquisition admitted");
|
|
expect(acquire).toHaveBeenCalledOnce();
|
|
expect(await readAuthority()).toEqual({ ...saved, epoch: 4, writerRunId: input.run.id });
|
|
const [bound] = await db.select().from(issues).where(eq(issues.id, issue.id));
|
|
expect(bound.executionWorkspaceId).toBe(workspace.id);
|
|
});
|
|
|
|
it("retains admitted ownership when a remote writer wait races with reassignment", async () => {
|
|
const { input, issue } = await inheritedRemoteFixture("isolated_workspace");
|
|
const [replacement] = await db.insert(agents).values({ companyId: issue.companyId,
|
|
name: "New task owner", adapterType: "process" }).returning();
|
|
input.controllerLease.assertOwned = async () => {
|
|
await db.update(issues).set({ assigneeAgentId: replacement.id }).where(eq(issues.id, issue.id));
|
|
};
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toMatchObject({
|
|
code: "workspace_busy", wasIssueAssignee: true,
|
|
});
|
|
const [current] = await db.select().from(issues).where(eq(issues.id, issue.id));
|
|
expect(current.assigneeAgentId).toBe(replacement.id);
|
|
});
|
|
|
|
it("rejects automatic Core services before remote reservation or acquisition", async () => {
|
|
const { input, envOrchestrator } = await fixture();
|
|
configureRemotePersistence();
|
|
input.agent.adapterType = "paperclip_runner";
|
|
input.environment.selectedEnvironmentForConfig = { ...input.environment.localEnvironment,
|
|
driver: "sandbox", config: { provider: "daytona" } };
|
|
input.environment.resolvedInstanceSettings.experimental.remoteWorkspaceStorageEnabled = true;
|
|
input.policy.useIsolatedTaskDirectory = true;
|
|
input.config.mergedConfig = { workspaceRuntime: { services: [{ name: "web", command: "node server.js" }] } };
|
|
const acquire = vi.spyOn(envOrchestrator, "acquireForRun");
|
|
const resolve = vi.spyOn(input.services, "resolveWorkspaceForRun");
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toMatchObject({
|
|
code: "workspace_validation_failed", resultJson: { reason: "remote_workspace_runtime_services_unsupported" },
|
|
});
|
|
expect(acquire).not.toHaveBeenCalled();
|
|
expect(resolve).not.toHaveBeenCalled();
|
|
expect(await db.select().from(executionWorkspaces)).toHaveLength(0);
|
|
});
|
|
|
|
it.each([false, true])("configured local folders retain host authority only on durable servers (ephemeral=%s)", async ephemeral => {
|
|
const { input, envOrchestrator } = await fixture();
|
|
input.agent.adapterType = "paperclip_runner";
|
|
input.environment.selectedEnvironmentForConfig = { ...input.environment.localEnvironment,
|
|
driver: "sandbox", config: { provider: "daytona" } };
|
|
input.environment.resolvedInstanceSettings.experimental.remoteWorkspaceStorageEnabled = true;
|
|
const acquire = vi.spyOn(envOrchestrator, "acquireForRun");
|
|
configureRemotePersistence();
|
|
const resolve = vi.spyOn(input.services, "resolveWorkspaceForRun").mockRejectedValue(new Error("local folder retained"));
|
|
process.env.PAPERCLIP_EPHEMERAL_SERVER = String(ephemeral);
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow(ephemeral
|
|
? "Volume-free servers require warm Paperclip Runner execution with remote workspace storage." : "local folder retained");
|
|
if (ephemeral) expect(resolve).not.toHaveBeenCalled();
|
|
else expect(resolve).toHaveBeenCalledOnce();
|
|
expect(await db.select().from(executionWorkspaces)).toHaveLength(0);
|
|
expect(acquire).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it.each(["configured-folder", "explicit-task-directory", "enrolled-remote"] as const)(
|
|
"preserves effective configured cwd authority for %s", async scenario => {
|
|
const { input, source, issue, envOrchestrator } = await fixture();
|
|
configureRemotePersistence();
|
|
await writeFile(join(source, "unpublished.txt"), "configured folder work\n");
|
|
await db.update(issues).set({ projectId: null, projectWorkspaceId: null }).where(eq(issues.id, issue.id));
|
|
input.task.issueRef = { ...issue, projectId: null, projectWorkspaceId: null };
|
|
input.task.context = { issueId: issue.id };
|
|
input.config.mergedConfig = { cwd: source };
|
|
await db.update(agents).set({ adapterType: "paperclip_runner" }).where(eq(agents.id, input.agent.id));
|
|
input.agent.adapterType = "paperclip_runner";
|
|
const [savedEnvironment] = await db.insert(environments).values({ name: `Configured cwd ${randomUUID()}`,
|
|
driver: "sandbox", status: "active", config: { provider: "daytona" } }).returning();
|
|
input.environment.selectedEnvironmentForConfig = { ...input.environment.localEnvironment,
|
|
id: savedEnvironment.id, driver: "sandbox", config: { provider: "daytona" } };
|
|
input.environment.selectedEnvironmentId = savedEnvironment.id;
|
|
expect(input.environment.resolvedInstanceSettings.experimental.remoteWorkspaceStorageEnabled).toBe(true);
|
|
if (scenario === "explicit-task-directory") {
|
|
input.task.workspaceSelection = { version: 1, source: "explicit", selection: { kind: "task_directory" } };
|
|
await db.update(issues).set({ workspaceSelection: input.task.workspaceSelection }).where(eq(issues.id, issue.id));
|
|
input.policy.useIsolatedTaskDirectory = true;
|
|
}
|
|
let enrolledId: string | undefined;
|
|
if (scenario === "enrolled-remote") {
|
|
enrolledId = randomUUID();
|
|
const [workspace] = await db.insert(executionWorkspaces).values({ id: enrolledId, companyId: issue.companyId,
|
|
sourceIssueId: issue.id, name: "Existing remote workspace", cwd: `/workspaces/${enrolledId}`,
|
|
providerType: "cloud_sandbox", strategyType: "project_primary", mode: "shared_workspace", status: "active",
|
|
metadata: { remoteWorkspace: { schema: "paperclip.remote-workspace.v1", epoch: 1, snapshotId: null,
|
|
source: { kind: "empty" }, location: null, writerRunId: null, dirty: false } } }).returning();
|
|
await db.update(issues).set({ executionWorkspaceId: workspace.id }).where(eq(issues.id, issue.id));
|
|
input.task.issueRef.executionWorkspaceId = workspace.id;
|
|
input.reuse.existingExecutionWorkspace = toExecutionWorkspace(workspace);
|
|
input.reuse.reusableExistingExecutionWorkspace = toExecutionWorkspace(workspace);
|
|
input.reuse.requestedShouldReuseExisting = true;
|
|
input.environment.resolvedInstanceSettings.experimental.remoteWorkspaceStorageEnabled = false;
|
|
}
|
|
const resolve = vi.spyOn(input.services, "resolveWorkspaceForRun");
|
|
const acquire = vi.spyOn(envOrchestrator, "acquireForRun").mockRejectedValue(new Error("configured cwd acquisition boundary"));
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow("configured cwd acquisition boundary");
|
|
expect(acquire).toHaveBeenCalledOnce();
|
|
const [workspace] = await db.select().from(executionWorkspaces);
|
|
if (scenario === "configured-folder") {
|
|
expect(resolve).toHaveBeenCalledOnce();
|
|
expect(workspace).toMatchObject({ providerType: "local_fs", cwd: source });
|
|
expect(workspace.metadata?.remoteWorkspace).toBeUndefined();
|
|
} else {
|
|
expect(resolve).not.toHaveBeenCalled();
|
|
expect(workspace).toMatchObject({ providerType: "cloud_sandbox" });
|
|
expect(workspace.metadata?.remoteWorkspace).toMatchObject({ source: { kind: "empty" }, writerRunId: input.run.id });
|
|
if (enrolledId) expect(workspace.id).toBe(enrolledId);
|
|
}
|
|
expect(await readFile(join(source, "unpublished.txt"), "utf8")).toBe("configured folder work\n");
|
|
},
|
|
);
|
|
|
|
it.each(["path", "file-url"] as const)("resumes an imported %s Git source from saved authority after its host files disappear", async sourceKind => {
|
|
const { input, source, issue, envOrchestrator } = await fixture();
|
|
configureRemotePersistence();
|
|
process.env.PAPERCLIP_SECRETS_MASTER_KEY = "72".repeat(32);
|
|
const upstream = join(home, `import-upstream-${randomUUID()}`);
|
|
await execFile("git", ["clone", "--bare", source, upstream]);
|
|
const repoUrl = sourceKind === "path" ? upstream : pathToFileURL(upstream).href;
|
|
await execFile("git", ["-C", source, "remote", "add", "origin", repoUrl]);
|
|
await writeFile(join(source, "unsaved.txt"), "imported untracked work\n");
|
|
await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date() }).where(eq(heartbeatRuns.id, input.run.id));
|
|
const [original] = await db.insert(executionWorkspaces).values({ companyId: issue.companyId,
|
|
sourceIssueId: issue.id, projectId: issue.projectId, projectWorkspaceId: issue.projectWorkspaceId,
|
|
name: "Imported Git workspace", cwd: source, providerRef: source, repoUrl, baseRef: "main", branchName: "main",
|
|
providerType: "local_fs", strategyType: "project_primary", mode: "shared_workspace", status: "active" }).returning();
|
|
await db.update(issues).set({ executionWorkspaceId: original.id }).where(eq(issues.id, issue.id));
|
|
const storage = createStorageService(createLocalDiskStorageProvider(join(home, `import-objects-${randomUUID()}`)));
|
|
await expect(migrateRemoteWorkspace({ db, storage, companyId: issue.companyId, workspaceId: original.id, apply: true }))
|
|
.resolves.toMatchObject({ applied: true });
|
|
const [imported] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, original.id));
|
|
const saved = readRemoteWorkspaceAuthority(imported.metadata)!;
|
|
expect(saved).toMatchObject({ source: { kind: "git", repoUrl, ref: "main", branchName: "main" },
|
|
snapshotId: expect.any(String), location: null, dirty: false, writerRunId: null });
|
|
await rm(source, { recursive: true });
|
|
await rm(upstream, { recursive: true });
|
|
|
|
const [next] = await db.insert(heartbeatRuns).values({ companyId: issue.companyId, agentId: input.agent.id,
|
|
issueId: issue.id, scopeKind: "issue", status: "running", runtimeMode: "legacy" }).returning();
|
|
input.run = next;
|
|
await db.update(agents).set({ adapterType: "paperclip_runner" }).where(eq(agents.id, input.agent.id));
|
|
input.agent.adapterType = "paperclip_runner";
|
|
const [environment] = await db.insert(environments).values({ name: `Imported source ${randomUUID()}`,
|
|
driver: "sandbox", status: "active", config: { provider: "daytona" } }).returning();
|
|
input.environment.selectedEnvironmentForConfig = { ...input.environment.localEnvironment,
|
|
id: environment.id, driver: "sandbox", config: { provider: "daytona" } };
|
|
input.environment.selectedEnvironmentId = environment.id;
|
|
input.environment.resolvedInstanceSettings.experimental.remoteWorkspaceStorageEnabled = false;
|
|
input.task.issueRef = { ...issue, executionWorkspaceId: imported.id };
|
|
input.reuse.existingExecutionWorkspace = toExecutionWorkspace(imported);
|
|
input.reuse.reusableExistingExecutionWorkspace = toExecutionWorkspace(imported);
|
|
input.reuse.requestedShouldReuseExisting = true;
|
|
input.reuse.workspaceReuseRequest = resolveExecutionWorkspaceReuseRequestForIssue({ issueExecutionWorkspaceId: imported.id,
|
|
issueExecutionWorkspacePreference: null, existingExecutionWorkspaceStatus: imported.status });
|
|
const resolve = vi.spyOn(input.services, "resolveWorkspaceForRun");
|
|
const hostSeed = vi.spyOn(freshWorkspaceSeed, "materializeFreshWorkspaceSeed");
|
|
const acquire = vi.spyOn(envOrchestrator, "acquireForRun").mockImplementation(async args => {
|
|
expect(args.persistedExecutionWorkspace).toMatchObject({ id: imported.id, providerType: "cloud_sandbox" });
|
|
const [reserved] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, imported.id));
|
|
expect(readRemoteWorkspaceAuthority(reserved.metadata)).toMatchObject({
|
|
snapshotId: saved.snapshotId, source: saved.source,
|
|
});
|
|
throw new Error("imported saved authority reached remote acquisition");
|
|
});
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow("imported saved authority reached remote acquisition");
|
|
expect(acquire).toHaveBeenCalledOnce();
|
|
expect(resolve).not.toHaveBeenCalled();
|
|
expect(hostSeed).not.toHaveBeenCalled();
|
|
const [after] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, imported.id));
|
|
expect(readRemoteWorkspaceAuthority(after.metadata)).toMatchObject({ snapshotId: saved.snapshotId, source: saved.source });
|
|
await expect(access(source)).rejects.toThrow();
|
|
await expect(access(upstream)).rejects.toThrow();
|
|
});
|
|
|
|
it.each(["file:///tmp/repository", "/tmp/repository", "../repository", "C:\\work\\repository", "C:repository"])(
|
|
"auto-enrollment leaves local repository source %s on the host authority", async repoUrl => {
|
|
const { input, issue, envOrchestrator } = await fixture();
|
|
input.agent.adapterType = "paperclip_runner";
|
|
input.environment.selectedEnvironmentForConfig = { ...input.environment.localEnvironment,
|
|
driver: "sandbox", config: { provider: "daytona" } };
|
|
input.environment.resolvedInstanceSettings.experimental.remoteWorkspaceStorageEnabled = true;
|
|
await db.update(projectWorkspaces).set({ cwd: null, sourceType: "git", repoUrl })
|
|
.where(eq(projectWorkspaces.id, issue.projectWorkspaceId!));
|
|
const acquire = vi.spyOn(envOrchestrator, "acquireForRun");
|
|
configureRemotePersistence();
|
|
const resolve = vi.spyOn(input.services, "resolveWorkspaceForRun").mockRejectedValue(new Error("local repository retained"));
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow("local repository retained");
|
|
expect(resolve).toHaveBeenCalledOnce();
|
|
expect(acquire).not.toHaveBeenCalled();
|
|
expect(await db.select().from(executionWorkspaces)).toHaveLength(0);
|
|
},
|
|
);
|
|
|
|
it.each(["https://", "ftp://example.com/repo", "javascript://invalid", "https://example.com/repo\ninvalid"])(
|
|
"rejects malformed or unsupported repository source %s without a host fallback", async repoUrl => {
|
|
const { input, issue, envOrchestrator } = await fixture();
|
|
configureRemotePersistence();
|
|
input.agent.adapterType = "paperclip_runner";
|
|
input.environment.selectedEnvironmentForConfig = { ...input.environment.localEnvironment,
|
|
driver: "sandbox", config: { provider: "daytona" } };
|
|
await db.update(projectWorkspaces).set({ cwd: null, sourceType: "git", repoUrl })
|
|
.where(eq(projectWorkspaces.id, issue.projectWorkspaceId!));
|
|
const resolve = vi.spyOn(input.services, "resolveWorkspaceForRun");
|
|
const acquire = vi.spyOn(envOrchestrator, "acquireForRun");
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toMatchObject({
|
|
code: "workspace_validation_failed", resultJson: { reason: "remote_workspace_source_invalid" },
|
|
});
|
|
expect(resolve).not.toHaveBeenCalled(); expect(acquire).not.toHaveBeenCalled();
|
|
expect(await db.select().from(executionWorkspaces)).toHaveLength(0);
|
|
},
|
|
);
|
|
|
|
it("does not turn an unauthorized configured-source selection into a host fallback", async () => {
|
|
const { input, envOrchestrator } = await fixture();
|
|
configureRemotePersistence();
|
|
input.agent.adapterType = "paperclip_runner";
|
|
input.environment.selectedEnvironmentForConfig = { ...input.environment.localEnvironment,
|
|
driver: "sandbox", config: { provider: "daytona" } };
|
|
input.task.workspaceSelection = { version: 1, source: "explicit",
|
|
selection: { kind: "configured_source", projectWorkspaceId: randomUUID(), mode: "shared" } };
|
|
const resolve = vi.spyOn(input.services, "resolveWorkspaceForRun");
|
|
const acquire = vi.spyOn(envOrchestrator, "acquireForRun");
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toMatchObject({ status: 403 });
|
|
expect(resolve).not.toHaveBeenCalled(); expect(acquire).not.toHaveBeenCalled();
|
|
expect(await db.select().from(executionWorkspaces)).toHaveLength(0);
|
|
});
|
|
|
|
it.each(["source", "effective-policy"] as const)("keeps plugin repository ownership via %s out of automatic Core enrollment", async ownership => {
|
|
const { input, issue, envOrchestrator } = await fixture();
|
|
configureRemotePersistence();
|
|
input.agent.adapterType = "paperclip_runner";
|
|
input.environment.selectedEnvironmentForConfig = { ...input.environment.localEnvironment,
|
|
driver: "sandbox", config: { provider: "daytona" } };
|
|
if (ownership === "effective-policy") input.policy.projectExecutionWorkspacePolicy = {
|
|
enabled: true, storageProviderPluginId: randomUUID(), defaultMode: "isolated_workspace",
|
|
};
|
|
await db.update(projectWorkspaces).set({ cwd: null, sourceType: ownership === "source" ? "remote_managed" : "git", repoUrl: "https://github.com/example/managed.git" })
|
|
.where(eq(projectWorkspaces.id, issue.projectWorkspaceId!));
|
|
const resolve = vi.spyOn(input.services, "resolveWorkspaceForRun").mockRejectedValue(new Error("managed ownership retained"));
|
|
const acquire = vi.spyOn(envOrchestrator, "acquireForRun");
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow("managed ownership retained");
|
|
expect(resolve).toHaveBeenCalledOnce(); expect(acquire).not.toHaveBeenCalled();
|
|
expect(await db.select().from(executionWorkspaces)).toHaveLength(0);
|
|
});
|
|
|
|
it.each(["shared_workspace", "agent_default"] as const)("rejects concurrent shared writers resolved from %s before resolving a remote source", async requestedMode => {
|
|
const { input, envOrchestrator } = await fixture();
|
|
configureRemotePersistence();
|
|
input.policy.useIsolatedTaskDirectory = true;
|
|
input.agent.adapterType = "paperclip_runner";
|
|
input.environment.selectedEnvironmentForConfig = { ...input.environment.localEnvironment,
|
|
driver: "sandbox", config: { provider: "daytona" } };
|
|
input.environment.resolvedInstanceSettings.experimental.remoteWorkspaceStorageEnabled = true;
|
|
input.policy.requestedExecutionWorkspaceMode = requestedMode;
|
|
input.policy.issueExecutionWorkspaceSettings = { sharedWorkspaceConcurrency: "allow" };
|
|
const acquire = vi.spyOn(envOrchestrator, "acquireForRun");
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow("one active writer");
|
|
expect(acquire).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it.each([false, true])("holds execution while the selected workspace is migrating (remote: %s)", async remote => {
|
|
const { input, source, issue, envOrchestrator } = await fixture();
|
|
const [workspace] = await db.insert(executionWorkspaces).values({ companyId: issue.companyId,
|
|
sourceIssueId: issue.id, projectId: issue.projectId, projectWorkspaceId: issue.projectWorkspaceId,
|
|
name: "Migrating workspace", cwd: source, providerType: "local_fs", strategyType: "project_primary",
|
|
mode: "shared_workspace", status: "active", metadata: { remoteWorkspaceMigration: { state: "running" } } }).returning();
|
|
await db.update(issues).set({ executionWorkspaceId: workspace.id }).where(eq(issues.id, issue.id));
|
|
// The admission snapshot may predate the operator starting the import.
|
|
input.reuse.existingExecutionWorkspace = toExecutionWorkspace({ ...workspace, metadata: {} });
|
|
if (remote) {
|
|
input.agent.adapterType = "paperclip_runner";
|
|
input.environment.selectedEnvironmentForConfig = { ...input.environment.localEnvironment,
|
|
driver: "sandbox", config: { provider: "daytona" } };
|
|
input.environment.resolvedInstanceSettings.experimental.remoteWorkspaceStorageEnabled = true;
|
|
}
|
|
const resolve = vi.spyOn(input.services, "resolveWorkspaceForRun");
|
|
const acquire = vi.spyOn(envOrchestrator, "acquireForRun");
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toMatchObject({
|
|
code: "workspace_validation_failed", resultJson: { reason: "remote_workspace_migration_in_progress" },
|
|
});
|
|
expect(resolve).not.toHaveBeenCalled();
|
|
expect(acquire).not.toHaveBeenCalled();
|
|
expect(await readFile(join(source, "README.md"), "utf8")).toBe("source content\n");
|
|
});
|
|
|
|
it("reserves private remote task files for an organizational project without a configured source", async () => {
|
|
const { input, issue, source, envOrchestrator } = await fixture();
|
|
await db.update(issues).set({ projectWorkspaceId: null }).where(eq(issues.id, issue.id));
|
|
await db.delete(projectWorkspaces).where(eq(projectWorkspaces.projectId, issue.projectId!));
|
|
input.task.issueRef = await readIssue(issue.id);
|
|
const project = (await projectService(db).getById(issue.projectId!))!;
|
|
expect(project.workspaces).toEqual([]);
|
|
expect(project.codebase).toMatchObject({ workspaceId: null, repoUrl: null, localFolder: null, origin: "managed_checkout" });
|
|
await mkdir(project.codebase.managedFolder, { recursive: true });
|
|
const sentinel = join(project.codebase.managedFolder, "unselected-project-files.txt");
|
|
await writeFile(sentinel, "Unselected project files\n");
|
|
configureRemotePersistence();
|
|
input.agent.adapterType = "paperclip_runner";
|
|
input.policy.requestedExecutionWorkspaceMode = "agent_default";
|
|
input.environment.selectedEnvironmentForConfig = { ...input.environment.localEnvironment,
|
|
driver: "sandbox", config: { provider: "daytona" } };
|
|
const resolveHost = vi.spyOn(input.services, "resolveWorkspaceForRun");
|
|
const acquire = vi.spyOn(envOrchestrator, "acquireForRun").mockImplementation(async args => {
|
|
expect(args.persistedExecutionWorkspace).toMatchObject({ companyId: issue.companyId, sourceIssueId: issue.id,
|
|
projectId: null, projectWorkspaceId: null, providerType: "cloud_sandbox", repoUrl: null,
|
|
metadata: { remoteWorkspace: { source: { kind: "empty" }, writerRunId: input.run.id } } });
|
|
throw new Error("private task workspace reached acquisition");
|
|
});
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow("private task workspace reached acquisition");
|
|
expect(acquire).toHaveBeenCalledOnce();
|
|
expect(resolveHost).not.toHaveBeenCalled();
|
|
const [workspace] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.sourceIssueId, issue.id));
|
|
expect(workspace).toMatchObject({ companyId: issue.companyId, sourceIssueId: issue.id,
|
|
projectId: null, projectWorkspaceId: null, providerType: "cloud_sandbox", repoUrl: null });
|
|
expect(workspace.cwd).toBe(`/workspaces/${workspace.id}`);
|
|
expect((await readIssue(issue.id))).toMatchObject({ projectId: issue.projectId, executionWorkspaceId: workspace.id });
|
|
await expect(access(workspace.cwd!)).rejects.toThrow();
|
|
expect(await readFile(sentinel, "utf8")).toBe("Unselected project files\n");
|
|
expect(await readFile(join(source, "README.md"), "utf8")).toBe("source content\n");
|
|
});
|
|
|
|
it("reserves editable task copies for referenced projects before acquiring the remote environment", async () => {
|
|
const { input, source, issue, envOrchestrator } = await fixture();
|
|
const previous = new Map(["PAPERCLIP_MULTI_PROJECT_WORKSPACE_SYNC", "PAPERCLIP_MULTI_PROJECT_WORKSPACE_SYNC_REMOTE"]
|
|
.map(key => [key, process.env[key]]));
|
|
vi.stubEnv("PAPERCLIP_MULTI_PROJECT_WORKSPACE_SYNC", "true");
|
|
vi.stubEnv("PAPERCLIP_MULTI_PROJECT_WORKSPACE_SYNC_REMOTE", "true");
|
|
configureRemotePersistence();
|
|
input.agent.adapterType = "paperclip_runner";
|
|
input.environment.selectedEnvironmentForConfig = { ...input.environment.localEnvironment,
|
|
driver: "sandbox", config: { provider: "daytona" } };
|
|
input.environment.resolvedInstanceSettings.experimental.remoteWorkspaceStorageEnabled = true;
|
|
await db.update(projectWorkspaces).set({ cwd: null, sourceType: "git", repoUrl: "https://github.com/example/remote.git" })
|
|
.where(eq(projectWorkspaces.id, issue.projectWorkspaceId!));
|
|
const [reference] = await db.insert(projects).values({ companyId: issue.companyId, name: "Referenced project" }).returning();
|
|
const [repository] = await db.insert(projectWorkspaces).values({ companyId: issue.companyId, projectId: reference.id,
|
|
name: "Referenced repository", sourceType: "git_repo", repoUrl: "https://github.com/example/reference.git" }).returning();
|
|
await db.update(issues).set({ description: `[project](${buildProjectMentionHref(reference.id)})` }).where(eq(issues.id, issue.id));
|
|
const acquire = vi.spyOn(envOrchestrator, "acquireForRun").mockImplementation(async args => {
|
|
expect(args.persistedExecutionWorkspace).toMatchObject({ companyId: issue.companyId, sourceIssueId: issue.id,
|
|
metadata: { remoteWorkspace: { source: { kind: "git", repoUrl: "https://github.com/example/remote.git", repositories: [
|
|
{ id: repository.id, projectId: reference.id, projectWorkspaceId: repository.id,
|
|
repoUrl: "https://github.com/example/reference.git" },
|
|
] } } } });
|
|
const [stored] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, args.persistedExecutionWorkspace!.id));
|
|
expect(readRemoteWorkspaceAuthority(stored.metadata)?.source.repositories?.[0]?.readOnly).not.toBe(true);
|
|
throw new Error("editable project copies reached acquisition");
|
|
});
|
|
try {
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow("editable project copies reached acquisition");
|
|
expect(acquire).toHaveBeenCalledOnce();
|
|
expect(await db.select().from(executionWorkspaces)).toHaveLength(1);
|
|
expect(await readdir(source)).not.toContain(".paperclip-repositories");
|
|
} finally {
|
|
for (const [key, value] of previous) {
|
|
if (value === undefined) delete process.env[key]; else process.env[key] = value;
|
|
}
|
|
}
|
|
});
|
|
|
|
it("fences recovered native preparation when its controller claim changes or expires", async () => {
|
|
const { input, issue, envOrchestrator } = await fixture();
|
|
const controller = await currentNativeControllerIdentity();
|
|
const leaseOwner = `recovery:${randomUUID()}`;
|
|
await db.update(heartbeatRuns).set({ runtimeMode: "native", nativeIssueId: issue.id }).where(eq(heartbeatRuns.id, input.run.id));
|
|
input.run.runtimeMode = "native";
|
|
configureRemotePersistence();
|
|
input.policy.useIsolatedTaskDirectory = true;
|
|
input.agent.adapterType = "paperclip_runner";
|
|
input.environment.selectedEnvironmentForConfig = { ...input.environment.localEnvironment,
|
|
driver: "sandbox", config: { provider: "daytona" } };
|
|
input.environment.resolvedInstanceSettings.experimental.remoteWorkspaceStorageEnabled = true;
|
|
input.reuse.runOptions = { nativeLeaseOwner: leaseOwner };
|
|
await db.insert(nativeRunFinalizations).values({ runId: input.run.id, companyId: issue.companyId,
|
|
issueId: issue.id, phase: "preparing", attempt: 2, controllerGeneration: 3, leaseOwner,
|
|
leaseExpiresAt: new Date(Date.now() + 60_000), controllerBootId: controller.bootId,
|
|
controllerPid: controller.pid, controllerProcessStartedAt: controller.processStartedAt });
|
|
const authority = { db, companyId: issue.companyId, runId: input.run.id, leaseOwner,
|
|
controllerGeneration: 3, providerAttempt: 2 };
|
|
await expect(assertNativeSessionPreparationAuthority(authority)).resolves.toBeUndefined();
|
|
await expect(assertNativeSessionPreparationAuthority({ ...authority, controllerGeneration: 2 }))
|
|
.rejects.toThrow("native_session_lease_lost");
|
|
await db.update(nativeRunFinalizations).set({ leaseOwner: "successor-controller" })
|
|
.where(eq(nativeRunFinalizations.runId, input.run.id));
|
|
const resolve = vi.spyOn(input.services, "resolveWorkspaceForRun");
|
|
const acquire = vi.spyOn(envOrchestrator, "acquireForRun");
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow("native_session_lease_lost");
|
|
expect(resolve).not.toHaveBeenCalled();
|
|
expect(acquire).not.toHaveBeenCalled();
|
|
expect(await db.select().from(executionWorkspaces)).toHaveLength(0);
|
|
await db.update(nativeRunFinalizations).set({ leaseOwner, leaseExpiresAt: new Date(Date.now() - 1_000) })
|
|
.where(eq(nativeRunFinalizations.runId, input.run.id));
|
|
await expect(assertNativeSessionPreparationAuthority(authority)).rejects.toThrow("native_session_lease_lost");
|
|
await db.update(nativeRunFinalizations).set({ leaseExpiresAt: new Date(Date.now() + 60_000) })
|
|
.where(eq(nativeRunFinalizations.runId, input.run.id));
|
|
await db.update(heartbeatRuns).set({ status: "cancelled" }).where(eq(heartbeatRuns.id, input.run.id));
|
|
await expect(assertNativeSessionPreparationAuthority(authority)).rejects.toThrow("native_session_lease_lost");
|
|
});
|
|
|
|
it.each([
|
|
{ inherited: false, mode: "isolated_workspace" as const, fresh: false },
|
|
{ inherited: false, mode: "isolated_workspace" as const, fresh: true },
|
|
{ inherited: true, mode: "isolated_workspace" as const, fresh: false },
|
|
{ inherited: true, mode: "operator_branch" as const, fresh: false },
|
|
])("prepares managed storage before acquisition (inherited: $inherited, mode: $mode, fresh: $fresh) without touching a Core checkout", async ({ inherited, mode: defaultMode, fresh }) => {
|
|
const { input, source, issue } = await fixture(true);
|
|
const pluginKey = `test.managed-storage-${inherited}-${defaultMode}-${fresh}`;
|
|
const [plugin] = await db.insert(plugins).values({ pluginKey, packageName: "test-storage", version: "1.0.0", apiVersion: 1, status: "ready", categories: ["workspace"],
|
|
manifestJson: { id: pluginKey, apiVersion: 1, version: "1.0.0", displayName: "Storage", description: "Storage", author: "Test", categories: ["workspace"],
|
|
workspaceProvider: { freshWorkspaceVersion: 1, modes: ["isolated_workspace", "operator_branch"], strategies: ["git_worktree"], ...(inherited ? { defaultForProjects: { mode: defaultMode, strategy: "git_worktree" as const } } : {}) }, capabilities: ["workspaces.manage", "events.subscribe"], resourceLifecycle: ["project"], entrypoints: { worker: "./worker.js" } } }).returning();
|
|
const explicitPolicy = { enabled: true, storageProviderPluginId: plugin.id, defaultMode: "isolated_workspace" as const,
|
|
workspaceStrategy: { type: "git_worktree" as const, baseRef: "main", branchTemplate: "test-{{issue.identifier}}" } };
|
|
if (!inherited) await db.update(projects).set({ executionWorkspacePolicy: explicitPolicy }).where(eq(projects.id, issue.projectId!));
|
|
{
|
|
const policy = await effectiveProjectWorkspacePolicy(db, input.agent.companyId, inherited ? null : explicitPolicy, true);
|
|
const mode = resolveExecutionWorkspaceMode({ projectPolicy: policy, issueSettings: null, legacyUseProjectWorkspace: null });
|
|
input.policy.projectExecutionWorkspacePolicy = policy;
|
|
input.policy.requestedExecutionWorkspaceMode = mode;
|
|
input.policy.effectiveExecutionWorkspaceMode = mode;
|
|
input.config.mergedConfig = buildExecutionWorkspaceAdapterConfig({ agentConfig: input.config.mergedConfig, projectPolicy: policy, issueSettings: null, mode, legacyUseProjectWorkspace: null });
|
|
}
|
|
await db.update(projectWorkspaces).set({ cwd: null, sourceType: "remote_managed", repoUrl: "https://github.com/example/repo.git" }).where(eq(projectWorkspaces.id, issue.projectWorkspaceId!));
|
|
await rm(source, { recursive: true });
|
|
if (fresh) input.task.freshWorkspace = { mode: "fresh_isolated_workspace", retryIntentId: input.run.id, sourceRunId: input.run.id, retainedSourceIds: [] };
|
|
const sequence: string[] = [];
|
|
input.services.workspaceWorkers = { getWorker: () => ({ supportedMethods: ["workspaceOperation"] }), call: async (_: string, __: string, params: any) => {
|
|
sequence.push("prepare");
|
|
return { kind: "prepared", binding: { pluginId: plugin.id, projectId: issue.projectId, executionWorkspaceId: params.executionWorkspaceId,
|
|
projectWorkspaceId: issue.projectWorkspaceId, sourceRevision: params.operation.source.revision, workspaceRef: "provider:checkout" },
|
|
branchName: params.operation.branchName, baseSha: "a".repeat(40), created: true, branchCreatedByRuntime: true };
|
|
} } as any;
|
|
const acquire = input.services.envOrchestrator.acquireForRun;
|
|
vi.spyOn(input.services.envOrchestrator, "acquireForRun").mockImplementation(async args => {
|
|
sequence.push("acquire"); expect(args.persistedExecutionWorkspace).toMatchObject({ providerType: "plugin", cwd: null });
|
|
return acquire(args);
|
|
});
|
|
vi.spyOn(input.services.envOrchestrator, "realizeForRun").mockImplementation(async args => {
|
|
expect(args.executionWorkspace.providerBinding?.workspaceRef).toBe("provider:checkout");
|
|
throw new Error("reached execution handoff");
|
|
});
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow("reached execution handoff");
|
|
expect(sequence).toEqual(["prepare", "acquire"]);
|
|
await expect(access(source)).rejects.toThrow();
|
|
const [task] = await db.select().from(issues).where(eq(issues.id, issue.id));
|
|
const [workspace] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, task.executionWorkspaceId!));
|
|
expect(workspace).toMatchObject({ providerType: "plugin", cwd: null, mode: defaultMode, strategyType: "git_worktree", branchName: fresh ? null : inherited ? "WSP-1-prepare-this-workspace" : "test-WSP-1" });
|
|
});
|
|
async function installDefault() {
|
|
const [plugin] = await db.insert(plugins).values({ pluginKey: "test.default-storage", packageName: "test-storage", version: "1.0.0", status: "ready",
|
|
manifestJson: { id: "test.default-storage", apiVersion: 1, version: "1.0.0", displayName: "Default storage", description: "Storage", author: "Test", categories: ["workspace"],
|
|
workspaceProvider: { modes: ["isolated_workspace"], strategies: ["git_worktree"], defaultForProjects: { mode: "isolated_workspace", strategy: "git_worktree" } },
|
|
capabilities: ["workspaces.manage", "events.subscribe"], resourceLifecycle: ["project"], entrypoints: { worker: "./worker.js" } } }).returning();
|
|
return plugin;
|
|
}
|
|
it("keeps source-free project tasks outside installed provider preparation", async () => {
|
|
const { input, issue } = await fixture();
|
|
await installDefault();
|
|
await db.update(issues).set({ projectWorkspaceId: null }).where(eq(issues.id, issue.id));
|
|
await db.delete(projectWorkspaces).where(eq(projectWorkspaces.id, issue.projectWorkspaceId!));
|
|
input.task.issueRef = { ...issue, projectWorkspaceId: null };
|
|
input.policy.projectExecutionWorkspacePolicy = await effectiveProjectWorkspacePolicy(db, input.agent.companyId, null, false);
|
|
const call = vi.fn();
|
|
input.services.workspaceWorkers = { getWorker: () => ({ supportedMethods: ["workspaceOperation"] }), call } as any;
|
|
const prepared = await prepareHeartbeatWorkspace(db, input);
|
|
expect(prepared.executionWorkspace.cwd).toBeTruthy();
|
|
expect(prepared.persistedExecutionWorkspace?.providerType).not.toBe("plugin");
|
|
expect(call).not.toHaveBeenCalled();
|
|
});
|
|
it("pins a resolved Core choice through source selection when a default appears during setup", async () => {
|
|
const { input, source } = await fixture();
|
|
expect(input.policy.projectExecutionWorkspacePolicy).toBeNull();
|
|
await installDefault();
|
|
const call = vi.fn();
|
|
input.services.workspaceWorkers = { getWorker: () => ({ supportedMethods: ["workspaceOperation"] }), call } as any;
|
|
const prepared = await prepareHeartbeatWorkspace(db, input);
|
|
expect(prepared.executionWorkspace.cwd).toBe(source);
|
|
expect(prepared.persistedExecutionWorkspace?.providerType).not.toBe("plugin");
|
|
expect(call).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it.each([
|
|
{ project: false, savedSource: false },
|
|
{ project: true, savedSource: true },
|
|
{ project: true, savedSource: false },
|
|
])("keeps a fresh task reservation stable through repeated preparation (project: $project, saved source: $savedSource)", async ({ project, savedSource }) => {
|
|
const { input, issue, envOrchestrator } = await fixture();
|
|
if (!project) await db.update(issues).set({ projectId: null, projectWorkspaceId: null }).where(eq(issues.id, issue.id));
|
|
else if (!savedSource) {
|
|
await db.update(issues).set({ projectWorkspaceId: null }).where(eq(issues.id, issue.id));
|
|
input.task.issueRef = { ...input.task.issueRef!, projectWorkspaceId: null };
|
|
}
|
|
const pluginKey = "test.fresh-task";
|
|
await db.insert(plugins).values({ pluginKey, packageName: "test-tasks", version: "1.0.0", apiVersion: 1,
|
|
status: "ready", categories: ["workspace"], manifestJson: { id: pluginKey, apiVersion: 1, version: "1.0.0",
|
|
displayName: "Tasks", description: "Tasks", author: "Test", categories: ["workspace"],
|
|
capabilities: ["environment.drivers.register"], entrypoints: { worker: "./worker.js" },
|
|
environmentDrivers: [{ driverKey: "tasks", displayName: "Task VM", configSchema: { type: "object" }, supportsTasks: true, freshWorkspaceVersion: 1 }] } });
|
|
input.agent.adapterType = "paperclip_runner";
|
|
if (!project) {
|
|
input.task.issueRef = { ...input.task.issueRef!, projectId: null, projectWorkspaceId: null };
|
|
input.task.context = { issueId: issue.id };
|
|
}
|
|
input.task.freshWorkspace = { mode: "fresh_isolated_workspace", retryIntentId: input.run.id, sourceRunId: input.run.id, retainedSourceIds: [] };
|
|
input.policy.requestedExecutionWorkspaceMode = input.policy.effectiveExecutionWorkspaceMode = "isolated_workspace";
|
|
input.config.mergedConfig = { workspaceStrategy: { type: "project_primary" } };
|
|
input.environment.selectedEnvironmentForConfig = { ...input.environment.localEnvironment, driver: "plugin",
|
|
config: { pluginKey, driverKey: "tasks" } };
|
|
const stopped = new Error("Reached projectless environment handoff");
|
|
const acquire = vi.spyOn(envOrchestrator, "acquireForRun").mockRejectedValue(stopped);
|
|
const read = vi.spyOn(input.services.executionWorkspacesSvc, "getStoredById");
|
|
for (let attempt = 0; attempt < 2; attempt++) await expect(prepareHeartbeatWorkspace(db, input)).rejects.toBe(stopped);
|
|
expect(read).toHaveBeenCalledWith(input.agent.companyId, input.run.id);
|
|
expect(acquire).toHaveBeenCalledTimes(2);
|
|
for (const [args] of acquire.mock.calls) expect(args.persistedExecutionWorkspace).toMatchObject({ id: input.run.id,
|
|
providerType: "adapter_managed", projectId: project ? issue.projectId : null,
|
|
projectWorkspaceId: project ? issue.projectWorkspaceId : null,
|
|
metadata: { runnerTaskFreshWorkspace: { attemptId: input.run.id } } });
|
|
const rows = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.companyId, input.agent.companyId));
|
|
expect(rows).toHaveLength(1);
|
|
const [task] = await db.select().from(issues).where(eq(issues.id, issue.id));
|
|
expect(task.projectWorkspaceId).toBe(project ? issue.projectWorkspaceId : null);
|
|
});
|
|
it.each(["implicit", "task", "agent"] as const)("keeps provider directory selection after a fresh task succeeds (strategy: %s)", async strategySource => {
|
|
const { input, issue, envOrchestrator } = await fixture();
|
|
await db.update(issues).set({ projectId: null, projectWorkspaceId: null }).where(eq(issues.id, issue.id));
|
|
input.task.issueRef = { ...issue, projectId: null, projectWorkspaceId: null };
|
|
input.task.context = { issueId: issue.id };
|
|
const pluginKey = "test.fresh-followup";
|
|
await db.insert(plugins).values({ pluginKey, packageName: "test-tasks", version: "1.0.0", apiVersion: 1,
|
|
status: "ready", categories: ["workspace"], manifestJson: { id: pluginKey, apiVersion: 1, version: "1.0.0",
|
|
displayName: "Tasks", description: "Tasks", author: "Test", categories: ["workspace"],
|
|
capabilities: ["environment.drivers.register"], entrypoints: { worker: "./worker.js" },
|
|
environmentDrivers: [{ driverKey: "tasks", displayName: "Task VM", configSchema: { type: "object" }, supportsTasks: true, freshWorkspaceVersion: 1 }] } });
|
|
const environment = await environmentService(db).create({ name: `Task computer ${issue.id}`, driver: "plugin", status: "active", envVars: {}, config: { pluginKey, driverKey: "tasks" } });
|
|
input.agent = { ...input.agent, adapterType: "paperclip_runner", defaultEnvironmentId: environment.id };
|
|
input.environment.selectedEnvironmentForConfig = environment;
|
|
input.policy.requestedExecutionWorkspaceMode = input.policy.effectiveExecutionWorkspaceMode = "isolated_workspace";
|
|
input.task.freshWorkspace = { mode: "fresh_isolated_workspace", retryIntentId: input.run.id, sourceRunId: input.run.id, retainedSourceIds: [] };
|
|
const stopped = new Error("Provider directory prepared");
|
|
vi.spyOn(envOrchestrator, "acquireForRun").mockRejectedValue(stopped);
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toBe(stopped);
|
|
const [task] = await db.select().from(issues).where(eq(issues.id, issue.id));
|
|
expect(task.executionWorkspaceSettings).toEqual({ mode: "isolated_workspace" });
|
|
const settings = { mode: "isolated_workspace" as const, ...(strategySource === "task" ? { workspaceStrategy: { type: "git_worktree" as const } } : {}) };
|
|
const agentConfig = strategySource === "agent" ? { workspaceStrategy: { type: "git_worktree" as const } } : {};
|
|
const selection = await selectHeartbeatEnvironment(db, {
|
|
run: input.run, agent: input.agent,
|
|
task: { issueRef: task, issueContext: task, issueId: issue.id, context: input.task.context,
|
|
executionProjectId: null, workspaceProjectContext: null, boundSourceWorkspace: null,
|
|
hasSelectedWorkspaceSource: false, taskWorkspaceIntent: null, responsibleUserId: null },
|
|
policy: { ...input.policy, issueExecutionWorkspaceSettings: settings },
|
|
config: { config: agentConfig, sourceWorkspaceConfig: buildExecutionWorkspaceAdapterConfig({ agentConfig, projectPolicy: null,
|
|
issueSettings: settings, mode: "isolated_workspace", legacyUseProjectWorkspace: null }), issueAssigneeOverrides: null },
|
|
services: { environmentsSvc: environmentService(db), executionWorkspacesSvc: executionWorkspaceService(db),
|
|
instanceSettings: instanceSettingsService(db), findSharedWorkspaceHolder: async () => null, findExecutionWorkspaceHolder: async () => null },
|
|
});
|
|
expect(selection.useIsolatedTaskDirectory).toBe(strategySource === "implicit");
|
|
expect(selection.reusableExistingExecutionWorkspace?.id).toBe(task.executionWorkspaceId);
|
|
});
|
|
|
|
const readIssue = (id: string) => db.select().from(issues).where(eq(issues.id, id)).then(rows => rows[0]!);
|
|
const readRun = (id: string) => db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, id)).then(rows => rows[0]!);
|
|
const leases = (id: string) => db.select().from(environmentLeases).where(eq(environmentLeases.heartbeatRunId, id));
|
|
|
|
it("keeps plugin-owned storage untouched when a fresh attempt provider is unavailable", async () => {
|
|
const { input, source, issue } = await fixture(true);
|
|
await db.update(projects).set({ executionWorkspacePolicy: {
|
|
enabled: true, storageProviderPluginId: "bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb",
|
|
} }).where(eq(projects.id, issue.projectId!));
|
|
input.policy.projectExecutionWorkspacePolicy = { enabled: true, storageProviderPluginId: "bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb" };
|
|
input.task.freshWorkspace = { mode: "fresh_isolated_workspace", retryIntentId: input.run.id,
|
|
sourceRunId: input.run.id, retainedSourceIds: [] };
|
|
const call = vi.fn();
|
|
input.services.workspaceWorkers = { getWorker: () => ({ supportedMethods: ["workspaceOperation"] }), call } as any;
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow("workspace storage plugin is unavailable");
|
|
expect(call).not.toHaveBeenCalled();
|
|
expect(await leases(input.run.id)).toHaveLength(0);
|
|
expect((await readIssue(issue.id)).executionWorkspaceId).toBeNull();
|
|
expect(await readFile(join(source, "README.md"), "utf8")).toBe("source content\n");
|
|
});
|
|
|
|
it("a fresh retry creates independent files and Git storage instead of reopening a configured task branch", async () => {
|
|
const { input, source, issue } = await fixture(true);
|
|
const original = await prepareHeartbeatWorkspace(db, input);
|
|
await writeFile(join(original.executionWorkspace.cwd, "unpublished.txt"), "keep the old work\n");
|
|
const [nextRun] = await db.insert(heartbeatRuns).values({ companyId: input.run.companyId,
|
|
agentId: input.agent.id, issueId: issue.id, scopeKind: "issue", status: "running", runtimeMode: "legacy" }).returning();
|
|
input.task.freshWorkspace = { mode: "fresh_isolated_workspace", retryIntentId: nextRun.id,
|
|
sourceRunId: input.run.id, retainedSourceIds: [] };
|
|
input.run = nextRun;
|
|
input.task.issueRef = { ...issue, executionWorkspaceId: null, executionWorkspacePreference: "create_new" };
|
|
const replacement = await prepareHeartbeatWorkspace(db, input);
|
|
expect(replacement.persistedExecutionWorkspace!.id).not.toBe(original.persistedExecutionWorkspace!.id);
|
|
expect(replacement.executionWorkspace.cwd).not.toBe(original.executionWorkspace.cwd);
|
|
expect(replacement.executionWorkspace.cwd).not.toBe(source);
|
|
expect(await readFile(join(replacement.executionWorkspace.cwd, "README.md"), "utf8")).toBe("source content\n");
|
|
expect(await readFile(join(original.executionWorkspace.cwd, "unpublished.txt"), "utf8")).toBe("keep the old work\n");
|
|
const common = await execFile("git", ["-C", replacement.executionWorkspace.cwd, "rev-parse", "--path-format=absolute", "--git-common-dir"]);
|
|
expect(common.stdout.trim()).toBe(join(replacement.executionWorkspace.cwd, ".git"));
|
|
await expect(access(join(replacement.executionWorkspace.cwd, "unpublished.txt"))).rejects.toThrow();
|
|
expect((await readIssue(issue.id)).executionWorkspaceId).toBe(replacement.persistedExecutionWorkspace!.id);
|
|
}, 30_000);
|
|
|
|
it.each(["historical_first", "recovery_first"] as const)(
|
|
"prepares the admitted fresh directory with sibling holds in either order: %s", async order => {
|
|
const { input, source, issue } = await fixture(true);
|
|
const original = await prepareHeartbeatWorkspace(db, input);
|
|
await writeFile(join(original.executionWorkspace.cwd, "unpublished.txt"), "keep the old work\n");
|
|
const sourceRunId = input.run.id, retryIntentId = randomUUID();
|
|
const [nextRun] = await db.insert(heartbeatRuns).values({ companyId: input.run.companyId,
|
|
agentId: input.agent.id, issueId: issue.id, scopeKind: "issue", status: "running", runtimeMode: "legacy",
|
|
contextSnapshot: { issueId: issue.id, taskRetryRequestId: retryIntentId } }).returning();
|
|
await db.insert(agentWakeupRequests).values({ id: retryIntentId, companyId: input.run.companyId,
|
|
agentId: input.agent.id, reason: "task_retry", source: "on_demand", status: "coalesced", runId: nextRun.id,
|
|
payload: { issueId: issue.id } });
|
|
const readiness = { mode: "fresh_isolated_workspace" as const, retryIntentId, sourceRunId, retainedSourceIds: [] };
|
|
const evidence = { runId: sourceRunId, explicitUserContinuation: { runId: nextRun.id, readiness } };
|
|
const hold = { companyId: input.run.companyId, sourceIssueId: issue.id, kind: "active_run_watchdog",
|
|
status: "resolved", ownerType: "system", nextAction: "Preserve the original source outcome." };
|
|
const historical = { ...hold, cause: "native_session_retry_exhausted", fingerprint: randomUUID(), evidence };
|
|
const recovery = { ...hold, cause: "native_continuation_requires_reconciliation", fingerprint: randomUUID(),
|
|
evidence: { ...evidence, systemRecovery: { ...createSystemRecovery(sourceRunId, retryIntentId),
|
|
state: "exhausted", freshAttemptRunId: nextRun.id } } };
|
|
await db.insert(issueRecoveryActions).values(order === "historical_first" ? [historical, recovery] : [recovery, historical]);
|
|
// Use the same committed-receipt lookup as run preparation, not a
|
|
// hand-built freshWorkspace input. The historical hold has no generation.
|
|
input.task.freshWorkspace = await readFreshWorkspaceReadiness(db, nextRun);
|
|
expect(input.task.freshWorkspace).toEqual(readiness);
|
|
input.run = nextRun;
|
|
input.task.issueRef = { ...issue, executionWorkspaceId: null, executionWorkspacePreference: "create_new" };
|
|
const replacement = await prepareHeartbeatWorkspace(db, input);
|
|
expect(replacement.executionWorkspace.cwd).not.toBe(original.executionWorkspace.cwd);
|
|
expect(replacement.executionWorkspace.cwd).not.toBe(source);
|
|
expect(await readFile(join(replacement.executionWorkspace.cwd, "README.md"), "utf8")).toBe("source content\n");
|
|
await expect(access(join(replacement.executionWorkspace.cwd, "unpublished.txt"))).rejects.toThrow();
|
|
expect(await readFile(join(original.executionWorkspace.cwd, "unpublished.txt"), "utf8")).toBe("keep the old work\n");
|
|
const common = await execFile("git", ["-C", replacement.executionWorkspace.cwd, "rev-parse", "--path-format=absolute", "--git-common-dir"]);
|
|
expect(common.stdout.trim()).toBe(join(replacement.executionWorkspace.cwd, ".git"));
|
|
expect((await readIssue(issue.id)).executionWorkspaceId).toBe(replacement.persistedExecutionWorkspace!.id);
|
|
}, 30_000,
|
|
);
|
|
|
|
it("starts a fresh clean task directory without treating it as a Git source", async () => {
|
|
const { input, source, issue } = await fixture();
|
|
input.policy.useIsolatedTaskDirectory = true;
|
|
input.policy.effectiveExecutionWorkspaceMode = "isolated_workspace";
|
|
input.policy.requestedExecutionWorkspaceMode = "isolated_workspace";
|
|
input.task.freshWorkspace = { mode: "fresh_isolated_workspace", retryIntentId: input.run.id,
|
|
sourceRunId: input.run.id, retainedSourceIds: [] };
|
|
const prepared = await prepareHeartbeatWorkspace(db, input);
|
|
expect(prepared.executionWorkspace.cwd).toContain("/isolated-workspace-attempts/");
|
|
expect(prepared.executionWorkspace.cwd).toContain(`/${input.run.id}`);
|
|
expect(prepared.executionWorkspace.cwd).not.toBe(source);
|
|
await expect(access(join(prepared.executionWorkspace.cwd, "README.md"))).rejects.toThrow();
|
|
expect(await readFile(join(source, "README.md"), "utf8")).toBe("source content\n");
|
|
expect((await readIssue(issue.id)).executionWorkspaceId).toBe(prepared.persistedExecutionWorkspace!.id);
|
|
});
|
|
|
|
it.each(["reuse", "explicit_git", "foreign_path", "missing_path", "symlink_path", "binding_changed", "canonical_retry", "canonical_warning", "canonical_unknown_warning", "canonical_provider_warning", "canonical_fresh", "canonical_lease", "canonical_result", "canonical_controller", "canonical_output", "canonical_measurement", "canonical_repaired_git"] as const)("recomputes ordinary-turn policy for a saved source-free fresh directory: %s", async scenario => {
|
|
const { input, issue } = await fixture();
|
|
const userId = `directory-owner-${randomUUID()}`;
|
|
await db.insert(authUsers).values({ id: userId, name: "Directory owner", email: `${userId}@example.test`, createdAt: new Date(), updatedAt: new Date() });
|
|
await db.insert(companyMemberships).values({ companyId: input.agent.companyId, principalId: userId, principalType: "user", membershipRole: "owner", status: "active" });
|
|
await db.update(companies).set({ defaultResponsibleUserId: userId }).where(eq(companies.id, input.agent.companyId));
|
|
await db.update(issues).set({ projectId: null, projectWorkspaceId: null, status: "in_progress", responsibleUserId: userId }).where(eq(issues.id, issue.id));
|
|
input.task.issueRef = await readIssue(issue.id);
|
|
input.task.context = { issueId: issue.id };
|
|
input.policy.useIsolatedTaskDirectory = true;
|
|
input.policy.effectiveExecutionWorkspaceMode = "isolated_workspace";
|
|
input.policy.requestedExecutionWorkspaceMode = "isolated_workspace";
|
|
input.config.mergedConfig = { workspaceStrategy: { type: "git_worktree" } };
|
|
input.task.freshWorkspace = { mode: "fresh_isolated_workspace", retryIntentId: input.run.id,
|
|
sourceRunId: input.run.id, retainedSourceIds: [] };
|
|
const original = await prepareHeartbeatWorkspace(db, input);
|
|
const workspace = original.persistedExecutionWorkspace!;
|
|
await writeFile(join(workspace.cwd!, "saved-review.txt"), "completed snapshot review\n");
|
|
await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date() }).where(eq(heartbeatRuns.id, input.run.id));
|
|
await db.update(environmentLeases).set({ status: "expired", cleanupStatus: "success", releasedAt: new Date() }).where(eq(environmentLeases.heartbeatRunId, input.run.id));
|
|
const adapterType = "process";
|
|
const originalAdapter = getServerAdapter(adapterType);
|
|
let preservedCwd = workspace.cwd!;
|
|
if (scenario === "explicit_git" || scenario.startsWith("canonical_")) await db.update(issues).set({ executionWorkspaceSettings: { mode: "isolated_workspace", workspaceStrategy: { type: "git_worktree" } } }).where(eq(issues.id, issue.id));
|
|
if (scenario === "foreign_path") {
|
|
const outside = join(home, "unowned-directory"); await mkdir(outside);
|
|
await db.update(executionWorkspaces).set({ cwd: outside }).where(eq(executionWorkspaces.id, workspace.id));
|
|
}
|
|
if (scenario === "missing_path" || scenario === "symlink_path") {
|
|
preservedCwd = join(home, `retained-files-${issue.id}`); await rename(workspace.cwd!, preservedCwd);
|
|
if (scenario === "symlink_path") await symlink(preservedCwd, workspace.cwd!);
|
|
}
|
|
if (scenario === "binding_changed") await db.update(issues).set({ executionWorkspaceId: null }).where(eq(issues.id, issue.id));
|
|
const freshExpected = ["canonical_unknown_warning", "canonical_provider_warning", "canonical_fresh", "canonical_lease", "canonical_result", "canonical_controller", "canonical_output", "canonical_measurement"].includes(scenario);
|
|
const executed = vi.fn(async (execution: AdapterExecutionContext) => {
|
|
const [lease] = await leases(execution.runId);
|
|
if (freshExpected) {
|
|
expect(lease.executionWorkspaceId).not.toBe(workspace.id);
|
|
await db.update(issues).set({ status: "done" }).where(eq(issues.id, issue.id));
|
|
return { exitCode: 0, signal: null, timedOut: false, summary: "Fresh workspace required" };
|
|
}
|
|
if (scenario !== "canonical_repaired_git") expect(lease.executionWorkspaceId).toBe(workspace.id);
|
|
const activeWorkspace = await input.services.executionWorkspacesSvc.getStoredById(input.agent.companyId, lease.executionWorkspaceId!);
|
|
if (scenario === "canonical_repaired_git") {
|
|
expect(execution.config.workspaceStrategy).toMatchObject({ type: "git_worktree" });
|
|
expect((await execFile("git", ["-C", activeWorkspace!.cwd!, "rev-parse", "--is-inside-work-tree"])).stdout.trim()).toBe("true");
|
|
}
|
|
else expect(activeWorkspace?.cwd).toBe(workspace.cwd);
|
|
expect(await readFile(join(activeWorkspace!.cwd!, "saved-review.txt"), "utf8")).toBe("completed snapshot review\n");
|
|
await db.update(issues).set({ status: "done" }).where(eq(issues.id, issue.id));
|
|
return { exitCode: 0, signal: null, timedOut: false, summary: "Retained review confirmed" };
|
|
});
|
|
registerServerAdapter({ ...originalAdapter, type: adapterType, supportsLocalAgentJwt: false, execute: executed,
|
|
testEnvironment: async () => ({ adapterType, status: "pass", checks: [], testedAt: new Date().toISOString() }) });
|
|
await db.update(agents).set({ adapterType, status: "idle", adapterConfig: { workspaceStrategy: { type: "git_worktree" } } }).where(eq(agents.id, input.agent.id));
|
|
if (scenario.includes("warning")) await db.insert(companyObservability).values({ companyId: input.agent.companyId, revision: randomUUID(), enabled: true,
|
|
destination: { provider: "otlp", endpointBaseUrl: "https://collector.example/otel" },
|
|
capture: { userPrompts: false, assistantResponses: false, toolDetails: false, toolContent: false } });
|
|
const heartbeat = heartbeatService(db);
|
|
try {
|
|
const next = await heartbeat.wakeup(input.agent.id, { source: "on_demand", triggerDetail: "manual", reason: "manual",
|
|
requestedByActorType: "user", requestedByActorId: userId, payload: { issueId: issue.id }, contextSnapshot: { issueId: issue.id } });
|
|
await heartbeat.drainActiveRunExecutions();
|
|
let finalRunId = next?.id;
|
|
if (scenario.startsWith("canonical_")) {
|
|
expect(await readRun(next!.id)).toMatchObject({ status: "failed", errorCode: "workspace_validation_failed" });
|
|
expect(executed).not.toHaveBeenCalled();
|
|
if (scenario.includes("warning")) {
|
|
const warning = (await db.select().from(heartbeatRunEvents).where(eq(heartbeatRunEvents.runId, next!.id))).find(event => event.seq === 1)!;
|
|
expect(warning).toMatchObject({ eventType: "agent_observability.unavailable", payload: { reason: "unsupported_harness" } });
|
|
if (scenario === "canonical_unknown_warning") await db.update(heartbeatRunEvents).set({ payload: { ...warning.payload, reason: "unknown" } }).where(eq(heartbeatRunEvents.id, warning.id));
|
|
if (scenario === "canonical_provider_warning") await db.update(heartbeatRunEvents).set({ sourcePayloadSha256: "a".repeat(64), protocolSchemaVersion: 1 }).where(eq(heartbeatRunEvents.id, warning.id));
|
|
}
|
|
if (scenario === "canonical_lease") await db.insert(environmentLeases).values({ companyId: input.agent.companyId, issueId: issue.id, heartbeatRunId: next!.id, provider: "local_fs", status: "expired", cleanupStatus: "success" });
|
|
if (scenario === "canonical_result") { const failed = await readRun(next!.id); await db.update(heartbeatRuns).set({ resultJson: { ...failed.resultJson, acceptedResult: { outcome: "unknown" } } }).where(eq(heartbeatRuns.id, next!.id)); }
|
|
if (scenario === "canonical_controller") await db.update(heartbeatRuns).set({ controllerLeaseExpiresAt: new Date(Date.now() + 60_000) }).where(eq(heartbeatRuns.id, next!.id));
|
|
if (scenario === "canonical_output") await db.update(heartbeatRuns).set({ lastOutputSeq: 1 }).where(eq(heartbeatRuns.id, next!.id));
|
|
if (scenario === "canonical_measurement") { const events = await db.select().from(heartbeatRunEvents).where(eq(heartbeatRunEvents.runId, next!.id)); const measured = events.find(event => event.eventType === "lifecycle")!; await db.update(heartbeatRunEvents).set({ payload: { ...measured.payload, durationMs: -1 } }).where(eq(heartbeatRunEvents.id, measured.id)); }
|
|
if (scenario === "canonical_repaired_git") {
|
|
await execFile("git", ["init", "--initial-branch=main", workspace.cwd!]);
|
|
await execFile("git", ["-C", workspace.cwd!, "config", "user.email", "test@example.com"]);
|
|
await execFile("git", ["-C", workspace.cwd!, "config", "user.name", "Directory repair"]);
|
|
await execFile("git", ["-C", workspace.cwd!, "add", "."]);
|
|
await execFile("git", ["-C", workspace.cwd!, "commit", "-m", "Retain saved review"]);
|
|
} else await db.update(issues).set({ executionWorkspaceSettings: { mode: "isolated_workspace" } }).where(eq(issues.id, issue.id));
|
|
const intent = await requestTaskRetry({ db, companyId: input.agent.companyId, issueId: issue.id,
|
|
actorId: userId, actorSource: "local_implicit", observedRunId: next!.id, idempotencyKey: randomUUID() });
|
|
expect(intent.requestId).toBeTruthy();
|
|
if (scenario === "canonical_fresh") await db.update(agentWakeupRequests).set({ payload: sql`jsonb_set(${agentWakeupRequests.payload}, '{taskRetry,strategy}', '"fresh_isolated_workspace"'::jsonb)` }).where(eq(agentWakeupRequests.id, intent.requestId!));
|
|
await heartbeat.resumeTaskRetry(input.agent.companyId, intent.requestId!);
|
|
await heartbeat.drainActiveRunExecutions();
|
|
const [receipt] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, intent.requestId!));
|
|
expect(receipt.runId).toBeTruthy();
|
|
finalRunId = receipt.runId!;
|
|
}
|
|
const continued = scenario === "reuse" || scenario.startsWith("canonical_");
|
|
if (scenario === "binding_changed") {
|
|
expect(next).toBeNull();
|
|
const [receipt] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.agentId, input.agent.id));
|
|
expect(receipt).toMatchObject({ status: "skipped", reason: "workspace_worktree_requires_project", runId: null });
|
|
} else {
|
|
expect(next).not.toBeNull();
|
|
expect(await readRun(finalRunId!)).toMatchObject(continued
|
|
? { status: "succeeded", errorCode: null } : { status: "failed", errorCode: "workspace_validation_failed" });
|
|
}
|
|
expect(executed).toHaveBeenCalledTimes(continued ? 1 : 0);
|
|
if (freshExpected) expect((await readIssue(issue.id)).executionWorkspaceId).not.toBe(workspace.id);
|
|
else if (scenario !== "canonical_repaired_git") expect((await readIssue(issue.id)).executionWorkspaceId).toBe(scenario === "binding_changed" ? null : workspace.id);
|
|
expect(await readFile(join(preservedCwd, "saved-review.txt"), "utf8")).toBe("completed snapshot review\n");
|
|
if (scenario === "missing_path") await expect(access(workspace.cwd!)).rejects.toThrow();
|
|
} finally { await heartbeat.drainActiveRunExecutions(); registerServerAdapter(originalAdapter); }
|
|
}, 30_000);
|
|
|
|
it("reuses a successful fresh task attempt for the next ordinary user turn", async () => {
|
|
const { input, issue } = await fixture();
|
|
input.policy.useIsolatedTaskDirectory = true;
|
|
input.policy.effectiveExecutionWorkspaceMode = "isolated_workspace";
|
|
input.policy.requestedExecutionWorkspaceMode = "isolated_workspace";
|
|
input.task.freshWorkspace = { mode: "fresh_isolated_workspace", retryIntentId: input.run.id,
|
|
sourceRunId: input.run.id, retainedSourceIds: [] };
|
|
const original = await prepareHeartbeatWorkspace(db, input);
|
|
const workspace = original.persistedExecutionWorkspace!;
|
|
await writeFile(join(workspace.cwd!, "saved-review.txt"), "completed review history\n");
|
|
await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date() })
|
|
.where(eq(heartbeatRuns.id, input.run.id));
|
|
await reuse(input, workspace);
|
|
input.task.freshWorkspace = null;
|
|
const next = await prepareHeartbeatWorkspace(db, input);
|
|
expect(next.persistedExecutionWorkspace!.id).toBe(workspace.id);
|
|
expect(next.executionWorkspace.cwd).toBe(workspace.cwd);
|
|
expect(await readFile(join(next.executionWorkspace.cwd, "saved-review.txt"), "utf8"))
|
|
.toBe("completed review history\n");
|
|
expect((await readIssue(issue.id)).executionWorkspaceId).toBe(workspace.id);
|
|
await reuse(input, workspace);
|
|
input.task.freshWorkspace = { mode: "fresh_isolated_workspace", retryIntentId: input.run.id,
|
|
sourceRunId: input.run.id, retainedSourceIds: [] };
|
|
input.reuse.existingExecutionWorkspace = null;
|
|
input.reuse.reusableExistingExecutionWorkspace = null;
|
|
input.reuse.requestedShouldReuseExisting = false;
|
|
input.reuse.workspaceReuseRequest = resolveExecutionWorkspaceReuseRequestForIssue({
|
|
issueExecutionWorkspaceId: null, issueExecutionWorkspacePreference: "create_new", existingExecutionWorkspaceStatus: null,
|
|
});
|
|
const replacement = await prepareHeartbeatWorkspace(db, input);
|
|
expect(replacement.executionWorkspace.cwd).not.toBe(workspace.cwd);
|
|
await expect(access(join(replacement.executionWorkspace.cwd, "saved-review.txt"))).rejects.toThrow();
|
|
expect(await readFile(join(workspace.cwd!, "saved-review.txt"), "utf8")).toBe("completed review history\n");
|
|
});
|
|
|
|
it.each(["task_binding", "workspace_row", "foreign_issue", "foreign_company", "closed_workspace"])(
|
|
"rejects a retained task attempt after its %s changes", async change => {
|
|
const { input, issue, source } = await fixture();
|
|
input.policy.useIsolatedTaskDirectory = true;
|
|
input.policy.effectiveExecutionWorkspaceMode = "isolated_workspace";
|
|
input.policy.requestedExecutionWorkspaceMode = "isolated_workspace";
|
|
input.task.freshWorkspace = { mode: "fresh_isolated_workspace", retryIntentId: input.run.id,
|
|
sourceRunId: input.run.id, retainedSourceIds: [] };
|
|
const original = await prepareHeartbeatWorkspace(db, input);
|
|
const workspace = original.persistedExecutionWorkspace!;
|
|
await writeFile(join(workspace.cwd!, "saved-review.txt"), "keep saved work\n");
|
|
await reuse(input, workspace);
|
|
input.task.freshWorkspace = null;
|
|
if (change === "task_binding") await db.update(issues).set({ executionWorkspaceId: null }).where(eq(issues.id, issue.id));
|
|
if (change === "workspace_row") await db.update(executionWorkspaces).set({ cwd: source }).where(eq(executionWorkspaces.id, workspace.id));
|
|
if (change === "foreign_issue") await db.update(executionWorkspaces).set({ sourceIssueId: null }).where(eq(executionWorkspaces.id, workspace.id));
|
|
if (change === "foreign_company") {
|
|
const [other] = await db.insert(companies).values({ name: "Other workspace company", issuePrefix: "OTH" }).returning();
|
|
await db.update(executionWorkspaces).set({ companyId: other.id }).where(eq(executionWorkspaces.id, workspace.id));
|
|
}
|
|
if (change === "closed_workspace") await db.update(executionWorkspaces).set({ status: "archived" }).where(eq(executionWorkspaces.id, workspace.id));
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toMatchObject({
|
|
resultJson: { workspaceValidation: { reason: "isolated_task_directory_binding_mismatch", issueId: issue.id } },
|
|
});
|
|
expect(await readFile(join(workspace.cwd!, "saved-review.txt"), "utf8")).toBe("keep saved work\n");
|
|
},
|
|
);
|
|
|
|
async function reuse(input: HeartbeatWorkspacePreparationInput, workspace: NonNullable<Awaited<ReturnType<typeof prepareHeartbeatWorkspace>>["persistedExecutionWorkspace"]>) {
|
|
input.reuse.requestedShouldReuseExisting = true;
|
|
input.reuse.existingExecutionWorkspace = workspace;
|
|
input.reuse.reusableExistingExecutionWorkspace = workspace;
|
|
input.reuse.workspaceReuseRequest = resolveExecutionWorkspaceReuseRequestForIssue({ issueExecutionWorkspaceId: workspace.id, issueExecutionWorkspacePreference: "reuse_existing", existingExecutionWorkspaceStatus: workspace.status });
|
|
input.task.issueRef = { ...input.task.issueRef!, executionWorkspaceId: workspace.id, executionWorkspacePreference: "reuse_existing" };
|
|
const [run] = await db.insert(heartbeatRuns).values({ companyId: input.run.companyId, agentId: input.agent.id, issueId: input.task.issueId, scopeKind: "issue", status: "running", runtimeMode: "legacy" }).returning();
|
|
input.run = run;
|
|
}
|
|
|
|
async function persistRecoveryInput(input: HeartbeatWorkspacePreparationInput, workspaceId: string, cwd: string) {
|
|
const issue = await readIssue(input.task.issueId!);
|
|
const admitted = buildNativeExecutionInput({ companyId: input.agent.companyId, agentId: input.agent.id,
|
|
runId: input.run.id, provider: "codex",
|
|
issue: { id: issue.id, identifier: issue.identifier, title: issue.title, description: null, workMode: "standard" },
|
|
taskPrompt: "Continue admitted task", normalizedSessionId: null,
|
|
workspace: { id: workspaceId, cwd, repoUrl: null, repoRef: null, branchName: null },
|
|
completionContract: { id: input.run.id, sha256: `sha256:${"a".repeat(64)}`, schemaVersion: "paperclip.run-result.v1",
|
|
contract: { revision: "1", objective: "Continue admitted task", criteria: [{ id: "objective", requirement: "Finish" }] } },
|
|
runtimeContext: nativeRuntimeContextFixture(),
|
|
});
|
|
input.run.runtimeMode = "native";
|
|
input.run.nativeIssueId = issue.id;
|
|
input.run.runnerProfileJson = { nativeExecutionInput: admitted };
|
|
input.reuse.persistedNativeExecutionInput = admitted;
|
|
await db.update(heartbeatRuns).set({ runtimeMode: "native", nativeIssueId: issue.id,
|
|
runnerProfileJson: input.run.runnerProfileJson }).where(eq(heartbeatRuns.id, input.run.id));
|
|
return admitted;
|
|
}
|
|
|
|
it.each(["muse", "openai_dot", "grok_bot"] as const)("uses private %s controller storage without rebinding task files or preparing repositories", async provider => {
|
|
const { input, issue } = await fixture();
|
|
input.agent.adapterType = "paperclip_runner";
|
|
input.config.mergedConfig = { provider, cwd: "/task/source/must-not-be-read", workspaceStrategy: { type: "git_worktree" } };
|
|
input.reuse.isDotRun = true;
|
|
const resolve = vi.spyOn(input.services, "resolveWorkspaceForRun").mockRejectedValue(new Error("Task files must not be adopted"));
|
|
const bind = vi.spyOn(input.services.executionWorkspacesSvc, "bindTaskWorkspace").mockRejectedValue(new Error("Task binding must not change"));
|
|
const repos = vi.spyOn(input.services.executionWorkspacesSvc, "prepareTaskRepositoriesForAdmission").mockRejectedValue(new Error("Repositories must not be prepared"));
|
|
try {
|
|
const result = await prepareHeartbeatWorkspace(db, input);
|
|
expect(result.executionWorkspace.cwd).toContain(provider === "muse" ? "muse-controllers" : "dot-controllers");
|
|
expect(result.persistedExecutionWorkspace).toBeNull();
|
|
expect(result.projectRepositoryPaths).toEqual([]);
|
|
expect((await readIssue(issue.id)).executionWorkspaceId).toBeNull();
|
|
expect(resolve).not.toHaveBeenCalled();
|
|
expect(bind).not.toHaveBeenCalled();
|
|
expect(repos).not.toHaveBeenCalled();
|
|
const binding = { companyId: input.agent.companyId, agentId: input.agent.id, bindingId: input.run.id,
|
|
bindingGeneration: 1, acceptByUnixMs: Date.now() + 60_000, expiresAtUnixMs: Date.now() + 600_000 };
|
|
const admitted = buildNativeExecutionInput({ companyId: input.agent.companyId, agentId: input.agent.id,
|
|
runId: input.run.id, provider, ...(provider === "muse" ? { museBinding: binding } : { dotBinding: binding }),
|
|
issue: { id: issue.id, identifier: null, title: issue.title, description: null, workMode: "standard" },
|
|
taskPrompt: "Preserve the admitted controller", normalizedSessionId: input.run.id,
|
|
workspace: { id: input.run.id, cwd: result.executionWorkspace.cwd, repoUrl: null, repoRef: null, branchName: null },
|
|
completionContract: { id: input.run.id, sha256: `sha256:${"a".repeat(64)}`, schemaVersion: "paperclip.run-result.v1",
|
|
contract: { revision: "1", objective: "Preserve the controller", criteria: [{ id: "objective", requirement: "Preserve the original checkpoint path" }] } },
|
|
runtimeContext: nativeRuntimeContextFixture(),
|
|
});
|
|
input.reuse.persistedNativeExecutionInput = admitted;
|
|
input.run.runtimeMode = "native";
|
|
input.run.runnerProfileJson = { nativeExecutionInput: admitted };
|
|
input.config.mergedConfig = { provider: "codex" };
|
|
const recovered = await prepareHeartbeatWorkspace(db, input);
|
|
expect(recovered.executionWorkspace.cwd).toBe(result.executionWorkspace.cwd);
|
|
expect(recovered.persistedExecutionWorkspace).toBeNull();
|
|
expect(bind).not.toHaveBeenCalled();
|
|
expect(repos).not.toHaveBeenCalled();
|
|
|
|
} finally { resolve.mockRestore(); bind.mockRestore(); repos.mockRestore(); }
|
|
});
|
|
|
|
it("persists the workspace and issue binding before acquiring its environment", async () => {
|
|
const { input, source, issue, envOrchestrator } = await fixture();
|
|
input.config.managedAiCredentialIdentity = "account:generation";
|
|
const acquire = envOrchestrator.acquireForRun;
|
|
vi.spyOn(envOrchestrator, "acquireForRun").mockImplementation(async args => {
|
|
expect((await readIssue(issue.id)).executionWorkspaceId).toBe(args.persistedExecutionWorkspace?.id);
|
|
expect((await readRun(input.run.id)).contextSnapshot?.executionWorkspaceId).toBe(args.persistedExecutionWorkspace?.id);
|
|
expect(args.managedAiCredentialIdentity).toBe("account:generation");
|
|
return acquire(args);
|
|
});
|
|
const result = await prepareHeartbeatWorkspace(db, input);
|
|
expect(result.executionWorkspace.cwd).toBe(source);
|
|
expect(result.persistedExecutionWorkspace).toMatchObject({ companyId: issue.companyId, projectId: issue.projectId, projectWorkspaceId: issue.projectWorkspaceId, mode: "shared_workspace" });
|
|
expect(await leases(input.run.id)).toMatchObject([{ status: "active", executionWorkspaceId: result.persistedExecutionWorkspace!.id }]);
|
|
expect(input.nativeRunnerPreparationSpans.map(span => span.name)).toEqual(["environment.acquire", "environment.workspace.realize"]);
|
|
expect(input.nativeRunnerPreparationSpans.every(span => span.parentName === "task.provider_session")).toBe(true);
|
|
expect(input.controllerLease.assertOwned).toHaveBeenCalledTimes(2);
|
|
});
|
|
|
|
it("records workspace steps around real persistence before environment acquisition", async () => {
|
|
const { input, source, issue, envOrchestrator } = await fixture();
|
|
const endpoint = process.env.OTEL_EXPORTER_OTLP_ENDPOINT;
|
|
vi.stubEnv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://collector.test");
|
|
const exporter = new InMemorySpanExporter();
|
|
const provider = new TracerProvider({ spanProcessors: [new SimpleSpanProcessor({ exporter })] });
|
|
expect(trace.setGlobalTracerProvider(provider)).toBe(true);
|
|
const acquire = envOrchestrator.acquireForRun;
|
|
vi.spyOn(envOrchestrator, "acquireForRun").mockImplementation(async args => {
|
|
const names = exporter.getFinishedSpans().map(span => span.name);
|
|
expect(names).toContain("heartbeat.prepare.workspace_record_create");
|
|
expect(names).toContain("heartbeat.prepare.workspace_binding");
|
|
expect((await readIssue(issue.id)).executionWorkspaceId).toBe(args.persistedExecutionWorkspace?.id);
|
|
return acquire(args);
|
|
});
|
|
try {
|
|
const result = await prepareHeartbeatWorkspace(db, input);
|
|
expect(result.executionWorkspace.cwd).toBe(source);
|
|
await provider.forceFlush();
|
|
const spans = exporter.getFinishedSpans();
|
|
expect(spans.map(span => span.name)).toEqual(expect.arrayContaining([
|
|
"heartbeat.prepare.workspace_source_resolution",
|
|
"heartbeat.prepare.workspace_provision_policy",
|
|
"heartbeat.prepare.workspace_base_validation",
|
|
"heartbeat.prepare.workspace_provision",
|
|
"heartbeat.prepare.workspace_operation_binding",
|
|
"heartbeat.prepare.workspace_config_recording",
|
|
"heartbeat.prepare.workspace_repository_admission",
|
|
"heartbeat.prepare.workspace_project_repositories",
|
|
]));
|
|
for (const span of spans) {
|
|
expect(Object.keys(span.attributes)).toEqual(["paperclip.task.run.run_id"]);
|
|
expect(JSON.stringify(span.attributes)).not.toContain(source);
|
|
}
|
|
} finally {
|
|
vi.stubEnv("OTEL_EXPORTER_OTLP_ENDPOINT", endpoint);
|
|
trace.disable();
|
|
await provider.shutdown();
|
|
}
|
|
});
|
|
|
|
it("prepares, reuses, and reopens a selected source containing an inaccessible private worktree", async () => {
|
|
const { input, source, issue } = await fixture(true);
|
|
vi.stubEnv("PAPERCLIP_ISSUE_PRIVACY_MODE", "enforce");
|
|
const [privateIssue] = await db.insert(issues).values({ companyId: issue.companyId,
|
|
title: "Private historical task", visibility: "private", projectId: issue.projectId }).returning();
|
|
const privateCwd = join(source, "private-work");
|
|
await execFile("git", ["-C", source, "worktree", "add", "-b", "historical-private", privateCwd, "main"]);
|
|
await writeFile(join(privateCwd, "confidential.txt"), "Shared OS storage is not task authorization");
|
|
const [privateWorkspace] = await db.insert(executionWorkspaces).values({ companyId: issue.companyId,
|
|
name: "Private workspace", sourceIssueId: privateIssue.id, cwd: privateCwd,
|
|
mode: "isolated_workspace", strategyType: "git_worktree", providerType: "git_worktree",
|
|
providerRef: privateCwd, branchName: "historical-private", status: "idle" }).returning();
|
|
await db.update(issues).set({ executionWorkspaceId: privateWorkspace.id }).where(eq(issues.id, privateIssue.id));
|
|
const clone = vi.spyOn(managedRepositoryCheckout, "materializeManagedProjectWorkspace");
|
|
const result = await prepareHeartbeatWorkspace(db, input);
|
|
expect(result.executionWorkspace.baseCwd).toBe(source);
|
|
expect(result.persistedExecutionWorkspace!.metadata?.isolatedRemoteSource).toBeUndefined();
|
|
expect(await readFile(join(result.executionWorkspace.cwd, "README.md"), "utf8")).toBe("source content\n");
|
|
expect(clone).not.toHaveBeenCalled();
|
|
await expect(access(isolatedRemoteSourcePath(issue.companyId, issue.id))).rejects.toMatchObject({ code: "ENOENT" });
|
|
await writeFile(join(result.executionWorkspace.cwd, "unpublished-work.txt"), "Keep current task edits\n");
|
|
await reuse(input, result.persistedExecutionWorkspace!);
|
|
const resumed = await prepareHeartbeatWorkspace(db, input);
|
|
expect(resumed.persistedExecutionWorkspace!.id).toBe(result.persistedExecutionWorkspace!.id);
|
|
expect(await readFile(join(resumed.executionWorkspace.cwd, "unpublished-work.txt"), "utf8")).toBe("Keep current task edits\n");
|
|
const workspace = resumed.persistedExecutionWorkspace!;
|
|
const cleanup = await cleanupExecutionWorkspaceArtifacts({ workspace,
|
|
projectWorkspace: { cwd: source, cleanupCommand: null }, runCleanupCommands: false });
|
|
expect(cleanup.warnings).toEqual([]);
|
|
await db.update(executionWorkspaces).set({ status: "archived" }).where(eq(executionWorkspaces.id, workspace.id));
|
|
const reopened = await executionWorkspaceService(db).reopenClosedIsolatedExecutionWorkspaceForIssue({ workspaceId: workspace.id,
|
|
issue: { id: issue.id, companyId: issue.companyId, projectId: issue.projectId },
|
|
actor: { agentId: input.agent.id, actorType: "agent" },
|
|
authorizationActor: { type: "agent", source: "agent_jwt", agentId: input.agent.id, companyId: issue.companyId, runId: input.run.id },
|
|
});
|
|
expect(reopened).toMatchObject({ ok: true });
|
|
expect(await readFile(join(workspace.cwd!, "README.md"), "utf8")).toBe("source content\n");
|
|
expect(await readFile(join(privateCwd, "confidential.txt"), "utf8")).toContain("OS storage");
|
|
});
|
|
|
|
it("cleans up a new local worktree after workspace persistence fails", async () => {
|
|
const { input, source, issue } = await fixture(true);
|
|
vi.spyOn(input.services.executionWorkspacesSvc, "create").mockRejectedValueOnce(new Error("Workspace persistence failed"));
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow("Workspace persistence failed");
|
|
const worktrees = (await execFile("git", ["-C", source, "worktree", "list", "--porcelain"])).stdout;
|
|
expect(worktrees.match(/^worktree /gm)).toHaveLength(1);
|
|
await expect(execFile("git", ["-C", source, "rev-parse", "--verify", `refs/heads/test-${issue.identifier}`])).rejects.toBeDefined();
|
|
expect((await readIssue(issue.id)).executionWorkspaceId).toBeNull();
|
|
});
|
|
|
|
it.each(["sandbox", "ssh", "plugin", "local", null] as const)("uses the selected %s placement for controller provisioning", async driver => {
|
|
const { input, source, issue, envOrchestrator } = await fixture(true);
|
|
const marker = join(source, "host-provision-ran");
|
|
const provisionCommand = `printf host > '${marker}'`;
|
|
input.config.mergedConfig = { workspaceStrategy: { type: "git_worktree", provisionCommand, runtimeProvisionCommand: "printf runtime" } };
|
|
const originalConfig = structuredClone(input.config.mergedConfig);
|
|
input.environment.selectedEnvironmentForConfig = driver === null ? null : { ...input.environment.localEnvironment, driver };
|
|
const stopped = new Error("Fixture stops before transport acquisition");
|
|
vi.spyOn(envOrchestrator, "acquireForRun").mockRejectedValue(stopped);
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toBe(stopped);
|
|
const bound = await readIssue(issue.id);
|
|
const [workspace] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, bound.executionWorkspaceId!));
|
|
expect(await readFile(join(workspace.cwd!, "README.md"), "utf8")).toBe("source content\n");
|
|
if (driver === "sandbox" || driver === "ssh" || driver === "plugin") await expect(readFile(marker, "utf8")).rejects.toMatchObject({ code: "ENOENT" });
|
|
else expect(await readFile(marker, "utf8")).toBe("host");
|
|
expect(input.config.mergedConfig).toEqual(originalConfig);
|
|
});
|
|
|
|
it("keeps a plugin workspace ledger across runs without preparing Core repositories", async () => {
|
|
const { input, issue, envOrchestrator, source } = await fixture();
|
|
input.agent.adapterType = "paperclip_runner";
|
|
input.environment.selectedEnvironmentForConfig = { ...input.environment.localEnvironment, driver: "plugin" };
|
|
const guestOnly = join(home, "guest-only", issue.projectId!);
|
|
await db.update(projectWorkspaces).set({ cwd: guestOnly }).where(eq(projectWorkspaces.id, issue.projectWorkspaceId!));
|
|
const copy = vi.spyOn(input.services.executionWorkspacesSvc, "prepareTaskRepositoriesForAdmission").mockRejectedValue(new Error("Core must not prepare repositories"));
|
|
const restoredCopy = vi.spyOn(input.services.executionWorkspacesSvc, "prepareConfiguredRepositoryCopy").mockRejectedValue(new Error("Core must not restore repositories"));
|
|
const stopped = new Error("Plugin realization boundary");
|
|
const realize = vi.spyOn(envOrchestrator, "realizeForRun").mockRejectedValue(stopped);
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toBe(stopped);
|
|
const firstId = (await readIssue(issue.id)).executionWorkspaceId!;
|
|
const [workspace] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, firstId));
|
|
expect(workspace).toMatchObject({ cwd: `/workspaces/${issue.projectWorkspaceId}`, strategyType: "project_primary", providerType: "adapter_managed" });
|
|
await reuse(input, toExecutionWorkspace(workspace));
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toBe(stopped);
|
|
expect((await readIssue(issue.id)).executionWorkspaceId).toBe(firstId);
|
|
expect(realize).toHaveBeenCalledTimes(2);
|
|
expect(copy).not.toHaveBeenCalled(); expect(restoredCopy).not.toHaveBeenCalled();
|
|
await expect(access(guestOnly)).rejects.toMatchObject({ code: "ENOENT" });
|
|
input.environment.selectedEnvironmentForConfig = input.environment.localEnvironment;
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toMatchObject({
|
|
resultJson: { reason: "plugin_workspace_owner_mismatch" },
|
|
});
|
|
expect(realize).toHaveBeenCalledTimes(2);
|
|
input.environment.selectedEnvironmentForConfig = { ...input.environment.localEnvironment, driver: "plugin" };
|
|
const [localWorkspace] = await db.update(executionWorkspaces).set({ providerType: "local_fs", cwd: source, metadata: {} })
|
|
.where(eq(executionWorkspaces.id, firstId)).returning();
|
|
await reuse(input, toExecutionWorkspace(localWorkspace));
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toMatchObject({
|
|
resultJson: { reason: "plugin_workspace_owner_mismatch" },
|
|
});
|
|
expect(realize).toHaveBeenCalledTimes(2);
|
|
expect(await readFile(join(source, "README.md"), "utf8")).toBe("source content\n");
|
|
});
|
|
|
|
it.each(["services", "fresh"] as const)("rejects unsupported %s before plugin workspace preparation or acquisition", async mode => {
|
|
const { input, envOrchestrator } = await fixture();
|
|
input.agent.adapterType = "paperclip_runner";
|
|
input.environment.selectedEnvironmentForConfig = { ...input.environment.localEnvironment, driver: "plugin" };
|
|
if (mode === "services") input.config.mergedConfig.workspaceRuntime = { services: [{ name: "preview", command: "must-not-run-on-core" }] };
|
|
else input.task.freshWorkspace = { mode: "fresh_isolated_workspace", retryIntentId: input.run.id,
|
|
sourceRunId: input.run.id, retainedSourceIds: [] };
|
|
const acquire = vi.spyOn(envOrchestrator, "acquireForRun");
|
|
const resolve = vi.spyOn(input.services, "resolveWorkspaceForRun");
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toMatchObject({
|
|
resultJson: { reason: mode === "services" ? "plugin_workspace_services_unsupported" : "fresh_workspace_storage_unsupported" },
|
|
});
|
|
expect(resolve).not.toHaveBeenCalled();
|
|
expect(acquire).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it("admits the metadata-only agent-default strategy for a plugin workspace", async () => {
|
|
const { input, envOrchestrator } = await fixture();
|
|
input.agent.adapterType = "paperclip_runner";
|
|
input.environment.selectedEnvironmentForConfig = { ...input.environment.localEnvironment, driver: "plugin" };
|
|
input.policy.requestedExecutionWorkspaceMode = "agent_default";
|
|
input.policy.effectiveExecutionWorkspaceMode = "agent_default";
|
|
const stopped = new Error("Plugin agent-default admission reached");
|
|
const realize = vi.spyOn(envOrchestrator, "realizeForRun").mockRejectedValue(stopped);
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toBe(stopped);
|
|
expect(realize).toHaveBeenCalledWith(expect.objectContaining({ executionWorkspace: expect.objectContaining({ strategy: "adapter_managed", cwd: `/workspaces/${input.task.issueId}` }),
|
|
persistedExecutionWorkspace: expect.objectContaining({ strategyType: "adapter_managed", providerType: "adapter_managed" }) }));
|
|
});
|
|
|
|
it("rejects Core Git worktree preparation for a plugin-owned workspace", async () => {
|
|
const { input, envOrchestrator } = await fixture(true);
|
|
input.agent.adapterType = "paperclip_runner";
|
|
input.environment.selectedEnvironmentForConfig = { ...input.environment.localEnvironment, driver: "plugin" };
|
|
const acquire = vi.spyOn(envOrchestrator, "acquireForRun");
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toMatchObject({
|
|
code: "workspace_validation_failed", resultJson: { reason: "plugin_workspace_strategy_unsupported" },
|
|
});
|
|
expect(acquire).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it.each([true, false])("forwards only authorized plugin project IDs without staging (remote enabled: %s)", async enabled => {
|
|
const { input, issue, envOrchestrator } = await fixture();
|
|
vi.stubEnv("PAPERCLIP_MULTI_PROJECT_WORKSPACE_SYNC", "true");
|
|
vi.stubEnv("PAPERCLIP_MULTI_PROJECT_WORKSPACE_SYNC_REMOTE", String(enabled));
|
|
input.agent = { ...input.agent, adapterType: "paperclip_runner" };
|
|
input.environment.selectedEnvironmentForConfig = { ...input.environment.localEnvironment, driver: "plugin" };
|
|
const [readable, privateProject] = await db.insert(projects).values([
|
|
{ companyId: issue.companyId, name: "Readable reference" },
|
|
{ companyId: issue.companyId, name: "Private reference", visibility: "private" },
|
|
]).returning();
|
|
const unavailableCwd = join(home, readable.id, "must-not-stage");
|
|
await db.insert(projectWorkspaces).values({ companyId: issue.companyId, projectId: readable.id, name: "Provider-only storage", cwd: unavailableCwd });
|
|
await db.update(issues).set({ description: [readable, privateProject].map(project => `[project](${buildProjectMentionHref(project.id)})`).join(" ") }).where(eq(issues.id, issue.id));
|
|
const stopped = new Error("Fixture reached plugin realization");
|
|
const acquire=vi.spyOn(envOrchestrator,"acquireForRun");
|
|
const realize = vi.spyOn(envOrchestrator, "realizeForRun").mockRejectedValue(stopped);
|
|
try {
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toBe(stopped);
|
|
expect(realize).toHaveBeenCalledWith(expect.objectContaining({
|
|
runnerTaskProjectIds: enabled ? [issue.projectId, readable.id] : undefined,
|
|
executionWorkspace: expect.objectContaining({ additionalWorkspaces: [] }),
|
|
}));
|
|
expect(acquire).toHaveBeenCalledWith(expect.objectContaining({runnerTaskProjectIds:
|
|
enabled ? [issue.projectId,readable.id] : [issue.projectId]}));
|
|
await expect(access(unavailableCwd)).rejects.toMatchObject({ code: "ENOENT" });
|
|
} finally {
|
|
acquire.mockRestore();
|
|
vi.stubEnv("PAPERCLIP_MULTI_PROJECT_WORKSPACE_SYNC", "true");
|
|
vi.stubEnv("PAPERCLIP_MULTI_PROJECT_WORKSPACE_SYNC_REMOTE", "true");
|
|
}
|
|
});
|
|
|
|
it("checks current repository admission before publishing configured sibling repositories", async () => {
|
|
const { input, source, issue } = await fixture();
|
|
await db.insert(projectWorkspaces).values({ companyId: issue.companyId, projectId: issue.projectId!,
|
|
name: "Sibling repository", repoUrl: pathToFileURL(source).href });
|
|
vi.spyOn(input.services.executionWorkspacesSvc, "prepareTaskRepositoriesForAdmission")
|
|
.mockRejectedValue(new Error("Current run identity is unavailable"));
|
|
const acquire = vi.spyOn(input.services.envOrchestrator, "acquireForRun");
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow("Current run identity is unavailable");
|
|
await expect(access(join(source, ".paperclip-repositories"))).rejects.toMatchObject({ code: "ENOENT" });
|
|
expect(acquire).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it("rechecks current binding after configured clone and before publication", async () => {
|
|
const { input, source, issue } = await fixture();
|
|
await db.insert(projectWorkspaces).values({ companyId: issue.companyId, projectId: issue.projectId!,
|
|
name: "Sibling repository", repoUrl: pathToFileURL(source).href });
|
|
const namespace = join(source, ".paperclip-repositories");
|
|
const guard = vi.spyOn(input.services.executionWorkspacesSvc, "assertTaskRepositoryPublicationAllowed")
|
|
.mockImplementation(async () => {
|
|
expect(await readdir(namespace)).toEqual([]);
|
|
throw new Error("Task binding changed during clone");
|
|
});
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow("Task binding changed during clone");
|
|
expect(guard).toHaveBeenCalledOnce();
|
|
expect(await readdir(namespace)).toEqual([]);
|
|
expect(await leases(input.run.id)).toEqual([]);
|
|
});
|
|
|
|
it("rejects a configured sibling copy when its source disappears before publication", async () => {
|
|
const previousPrivacyMode = process.env.PAPERCLIP_ISSUE_PRIVACY_MODE;
|
|
vi.stubEnv("PAPERCLIP_ISSUE_PRIVACY_MODE", "enforce");
|
|
try {
|
|
const { input, source, issue } = await fixture();
|
|
const sibling = source + "-private-sibling";
|
|
await execFile("git", ["clone", "--no-hardlinks", source, sibling]);
|
|
await writeFile(join(sibling, "private.txt"), "PRIVATE_SIBLING_STAGING_BYTES");
|
|
const [privateIssue] = await db.insert(issues).values({ companyId: issue.companyId, title: "Private sibling owner",
|
|
visibility: "private", responsibleUserId: "private-owner" }).returning();
|
|
const [privateWorkspace] = await db.insert(executionWorkspaces).values({ companyId: issue.companyId,
|
|
name: "Private sibling", projectId: null, sourceIssueId: privateIssue.id, cwd: sibling,
|
|
providerType: "local_fs", strategyType: "project_primary", mode: "shared_workspace", status: "active" }).returning();
|
|
await db.insert(issueAccessGrants).values({ issueId: privateIssue.id, subjectType: "agent", subjectId: input.agent.id, source: "explicit" });
|
|
const [configured] = await db.insert(projectWorkspaces).values({ companyId: issue.companyId, projectId: issue.projectId!,
|
|
name: "Private sibling", cwd: sibling, repoUrl: pathToFileURL(sibling).href }).returning();
|
|
const namespace = join(source, ".paperclip-repositories");
|
|
const originalPrepare = input.services.executionWorkspacesSvc.prepareConfiguredRepositoryCopy;
|
|
const prepare = vi.spyOn(input.services.executionWorkspacesSvc, "prepareConfiguredRepositoryCopy").mockImplementation(async args => {
|
|
const recheck = await originalPrepare(args);
|
|
return async () => {
|
|
const candidates = await readdir(namespace);
|
|
expect(candidates).toHaveLength(1);
|
|
expect(candidates[0]).toContain(".clone-");
|
|
expect(await readFile(join(namespace, candidates[0]!, "private.txt"), "utf8")).toBe("PRIVATE_SIBLING_STAGING_BYTES");
|
|
const retained = await db.select().from(executionWorkspaceRepositoryAccess).where(eq(executionWorkspaceRepositoryAccess.companyId, issue.companyId));
|
|
expect(retained).toEqual([]);
|
|
await rename(sibling, sibling + "-moved");
|
|
await db.update(projectWorkspaces).set({ cwd: sibling + "-moved" }).where(eq(projectWorkspaces.id, configured.id));
|
|
await recheck();
|
|
};
|
|
});
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow(/ENOENT|unavailable|inaccessible|no longer/i);
|
|
expect(prepare).toHaveBeenCalledOnce();
|
|
expect(await readdir(namespace)).toEqual([]);
|
|
expect(await leases(input.run.id)).toEqual([]);
|
|
await db.delete(issueAccessGrants).where(eq(issueAccessGrants.issueId, privateIssue.id));
|
|
|
|
} finally {
|
|
vi.stubEnv("PAPERCLIP_ISSUE_PRIVACY_MODE", previousPrivacyMode);
|
|
}
|
|
});
|
|
|
|
it.each(["file://untrusted-host/private/repository", "file:///private%2Frepository", "file:///private/repository?other", "file:///private/repository#other"])("rejects malformed local repository URL before source access: %s", async repoUrl => {
|
|
const { source, issue } = await fixture();
|
|
const prepareLocalSource = vi.fn();
|
|
await expect(prepareProjectRepositoryWorkspaces({ cwd: source, anchorRepoUrl: null,
|
|
workspaces: [{ id: issue.projectWorkspaceId!, repoUrl, repoRef: null, cwd: null }], prepareLocalSource })).rejects.toThrow();
|
|
expect(prepareLocalSource).not.toHaveBeenCalled();
|
|
expect(await readdir(join(source, ".paperclip-repositories"))).toEqual([]);
|
|
});
|
|
|
|
it("persists and reuses task-owned files without adopting the organizational project as their source", async () => {
|
|
const { input, issue } = await fixture();
|
|
await db.update(issues).set({ projectWorkspaceId: null,
|
|
workspaceSelection: { version: 1, source: "explicit", selection: { kind: "task_directory" } },
|
|
}).where(eq(issues.id, issue.id));
|
|
input.task.issueRef = await readIssue(issue.id);
|
|
const bind = vi.spyOn(input.services.executionWorkspacesSvc, "bindTaskWorkspace");
|
|
const first = await prepareHeartbeatWorkspace(db, input);
|
|
expect(first.persistedExecutionWorkspace).toMatchObject({ projectId: null, projectWorkspaceId: null, sourceIssueId: issue.id });
|
|
expect(bind).toHaveBeenCalled();
|
|
expect((await readIssue(issue.id)).executionWorkspaceId).toBe(first.persistedExecutionWorkspace!.id);
|
|
await writeFile(join(first.executionWorkspace.cwd, "retained.txt"), "task files");
|
|
const [organization] = await db.insert(projects).values({ companyId: issue.companyId, name: "New organization" }).returning();
|
|
await db.update(issues).set({ projectId: organization.id }).where(eq(issues.id, issue.id));
|
|
input.task.issueRef = await readIssue(issue.id);
|
|
await reuse(input, first.persistedExecutionWorkspace!);
|
|
const next = await prepareHeartbeatWorkspace(db, input);
|
|
expect(next.persistedExecutionWorkspace).toMatchObject({ id: first.persistedExecutionWorkspace!.id, projectId: null });
|
|
expect(next.executionWorkspace.cwd).toBe(first.executionWorkspace.cwd);
|
|
expect(await readFile(join(next.executionWorkspace.cwd, "retained.txt"), "utf8")).toBe("task files");
|
|
expect((await readIssue(issue.id)).projectId).toBe(organization.id);
|
|
});
|
|
|
|
it("retains task-directory provenance through shared workspace preparation", async () => {
|
|
const { input, issue, source } = await fixture();
|
|
await db.update(issues).set({ projectId: null, projectWorkspaceId: null,
|
|
workspaceSelection: { version: 1, source: "explicit", selection: { kind: "task_directory" } },
|
|
}).where(eq(issues.id, issue.id));
|
|
input.task.issueRef = await readIssue(issue.id);
|
|
input.task.context = { issueId: issue.id };
|
|
const first = await prepareHeartbeatWorkspace(db, input);
|
|
expect(first.executionWorkspace.source).toBe("task_session");
|
|
expect(first.persistedExecutionWorkspace).toMatchObject({ mode: "shared_workspace", strategyType: "project_primary",
|
|
sourceIssueId: issue.id, projectId: null, projectWorkspaceId: null, repoUrl: null, metadata: { source: "task_session" } });
|
|
await mkdir(join(first.executionWorkspace.cwd, ".git", "info"), { recursive: true });
|
|
await writeFile(join(first.executionWorkspace.cwd, ".git", "info", "exclude"), "/.paperclip-repositories/\n");
|
|
await writeFile(join(first.executionWorkspace.cwd, "original-context.txt"), "Preserve the original task bytes\n");
|
|
await db.update(issues).set({ executionRunId: input.run.id }).where(eq(issues.id, issue.id));
|
|
const taskRepositories = executionWorkspaceService(db);
|
|
const receipt = await taskRepositories.requestTaskRepository({ companyId: issue.companyId, issueId: issue.id,
|
|
actor: { type: "board", source: "local_implicit", userId: "local-board" },
|
|
request: { repository: { kind: "url", url: "https://github.com/public/source" }, requestKey: "retained-task-directory" } });
|
|
const materialize = managedRepositoryCheckout.materializeManagedProjectWorkspace;
|
|
vi.spyOn(managedRepositoryCheckout, "materializeManagedProjectWorkspace").mockImplementation((cwd, options) =>
|
|
materialize(cwd, { ...options, localSource: source }));
|
|
await reuse(input, first.persistedExecutionWorkspace!);
|
|
await db.update(issues).set({ executionRunId: input.run.id }).where(eq(issues.id, issue.id));
|
|
input.task.issueRef = await readIssue(issue.id);
|
|
const next = await prepareHeartbeatWorkspace(db, input);
|
|
expect(next.executionWorkspace.source).toBe("task_session");
|
|
expect(next.persistedExecutionWorkspace!.metadata?.source).toBe("task_session");
|
|
expect(next.persistedExecutionWorkspace!.id).toBe(first.persistedExecutionWorkspace!.id);
|
|
expect(await readFile(join(next.executionWorkspace.cwd, "original-context.txt"), "utf8")).toBe("Preserve the original task bytes\n");
|
|
const [repository] = await taskRepositories.listTaskRepositories(issue.companyId, next.persistedExecutionWorkspace!.id);
|
|
expect(repository).toMatchObject({ id: receipt.operationId, state: "ready" });
|
|
expect(await readFile(join(next.executionWorkspace.cwd, repository.relativePath, "README.md"), "utf8")).toBe("source content\n");
|
|
});
|
|
|
|
it("creates a real isolated worktree and reuses its recorded workspace", async () => {
|
|
const { input, source, issue } = await fixture(true);
|
|
const first = await prepareHeartbeatWorkspace(db, input);
|
|
expect(first.executionWorkspace.cwd).not.toBe(source);
|
|
expect(await readFile(join(first.executionWorkspace.cwd, "README.md"), "utf8")).toBe("source content\n");
|
|
expect(await readIssue(issue.id)).toMatchObject({ executionWorkspaceId: first.persistedExecutionWorkspace!.id, executionWorkspacePreference: "reuse_existing" });
|
|
await writeFile(join(first.executionWorkspace.cwd, "retained.txt"), "keep this work");
|
|
await reuse(input, first.persistedExecutionWorkspace!);
|
|
const next = await prepareHeartbeatWorkspace(db, input);
|
|
expect(next.reusedExecutionWorkspace).toMatchObject({ cwd: first.executionWorkspace.cwd });
|
|
expect(next.persistedExecutionWorkspace!.id).toBe(first.persistedExecutionWorkspace!.id);
|
|
expect(await readFile(join(next.executionWorkspace.cwd, "retained.txt"), "utf8")).toBe("keep this work");
|
|
});
|
|
|
|
it("keeps the admitted recovery workspace when the issue now points elsewhere", async () => {
|
|
const { input, issue } = await fixture(true);
|
|
const first = await prepareHeartbeatWorkspace(db, input);
|
|
await reuse(input, first.persistedExecutionWorkspace!);
|
|
await db.update(issues).set({ executionWorkspaceId: null, projectWorkspaceId: null }).where(eq(issues.id, issue.id));
|
|
input.task.issueRef = { ...input.task.issueRef!, executionWorkspaceId: null, projectWorkspaceId: null };
|
|
input.reuse.nativeRecoveryExecutionWorkspaceId = first.persistedExecutionWorkspace!.id;
|
|
await persistRecoveryInput(input, first.persistedExecutionWorkspace!.id, first.executionWorkspace.cwd);
|
|
const anchor = vi.spyOn(input.services, "resolveReusedGitWorkspaceAnchor");
|
|
const prepareRepositories = vi.spyOn(input.services.executionWorkspacesSvc, "prepareTaskRepositoriesForAdmission");
|
|
const reserveRoot = vi.spyOn(input.services.executionWorkspacesSvc, "assertTaskRepositoryRecoveryBinding");
|
|
const next = await prepareHeartbeatWorkspace(db, input);
|
|
expect(reserveRoot).toHaveBeenCalledWith({ companyId: issue.companyId, issueId: issue.id,
|
|
workspaceId: first.persistedExecutionWorkspace!.id, cwd: first.executionWorkspace.cwd,
|
|
agentId: input.agent.id, runId: input.run.id });
|
|
expect(prepareRepositories).not.toHaveBeenCalled();
|
|
expect(next.persistedExecutionWorkspace!.id).toBe(first.persistedExecutionWorkspace!.id);
|
|
expect(anchor).toHaveBeenCalledWith(expect.objectContaining({ immutableNativeBinding: true, explicitProjectWorkspaceId: first.persistedExecutionWorkspace!.projectWorkspaceId }));
|
|
expect(await readIssue(issue.id)).toMatchObject({ executionWorkspaceId: null, projectWorkspaceId: null });
|
|
});
|
|
|
|
it.each([
|
|
{ transport: "local", descriptor: "absent" },
|
|
{ transport: "ssh", descriptor: "absent" },
|
|
{ transport: "task", descriptor: "absent" },
|
|
{ transport: "daytona", descriptor: "valid" },
|
|
{ transport: "daytona", descriptor: "absent" },
|
|
{ transport: "daytona", descriptor: "corrupt" },
|
|
] as const)("validates the admitted host binding for $transport recovery with $descriptor descriptor", async ({ transport, descriptor }) => {
|
|
const { input, issue } = await fixture(true);
|
|
const first = await prepareHeartbeatWorkspace(db, input);
|
|
await reuse(input, first.persistedExecutionWorkspace!);
|
|
await db.update(issues).set({ executionWorkspaceId: null, projectWorkspaceId: null }).where(eq(issues.id, issue.id));
|
|
input.task.issueRef = { ...input.task.issueRef!, executionWorkspaceId: null, projectWorkspaceId: null };
|
|
input.reuse.nativeRecoveryExecutionWorkspaceId = first.persistedExecutionWorkspace!.id;
|
|
const admittedRoot = join(home, `admitted-${input.run.id}`);
|
|
await mkdir(admittedRoot);
|
|
const remoteCwd = "/remote/admitted-task";
|
|
await persistRecoveryInput(input, first.persistedExecutionWorkspace!.id, transport === "local" ? admittedRoot : remoteCwd);
|
|
const target: AdapterExecutionTarget = transport === "local" ? { kind: "local" }
|
|
: transport === "ssh" ? { kind: "remote", transport: "ssh", remoteCwd,
|
|
spec: { host: "fixture.invalid", port: 22, username: "fixture", remoteWorkspacePath: remoteCwd,
|
|
privateKey: null, knownHosts: null, strictHostKeyChecking: true, remoteCwd } }
|
|
: { kind: "remote", transport: "sandbox", providerKey: "daytona", remoteCwd,
|
|
...(transport === "task" ? { runnerTask: vi.fn() } : {}) };
|
|
const realize = input.services.envOrchestrator.realizeForRun;
|
|
vi.spyOn(input.services.envOrchestrator, "realizeForRun").mockImplementation(async args => ({
|
|
...await realize(args), executionTarget: target,
|
|
}));
|
|
if (descriptor !== "absent") {
|
|
const baseline = await captureDirectorySnapshot(admittedRoot);
|
|
const reference = await nativeWorkspaceSyncInternals.writeDescriptor({
|
|
schema: "paperclip.native-workspace-sync/v1", state: "prepared",
|
|
binding: { companyId: issue.companyId, runId: input.run.id, workspaceId: first.persistedExecutionWorkspace!.id,
|
|
leaseId: "admitted-lease", providerLeaseId: "admitted-provider", localCwd: admittedRoot, remoteCwd },
|
|
baseline: serializeDirectorySnapshot(baseline), baselineSha256: directorySnapshotSha256(baseline),
|
|
gitSnapshot: null, seed: null, createdAt: new Date().toISOString(), finalizedAt: null, finalHostSha256: null, resourceDisposition: null,
|
|
});
|
|
input.run.runnerProfileJson = { ...input.run.runnerProfileJson, nativeWorkspaceSync: reference };
|
|
await db.update(heartbeatRuns).set({ runnerProfileJson: input.run.runnerProfileJson }).where(eq(heartbeatRuns.id, input.run.id));
|
|
if (descriptor === "corrupt") {
|
|
await writeFile(nativeWorkspaceSyncInternals.descriptorPath(input.run.id, reference.descriptorSha256), "{}");
|
|
}
|
|
}
|
|
const reserve = vi.spyOn(input.services.executionWorkspacesSvc, "assertTaskRepositoryRecoveryBinding");
|
|
const prepare = vi.spyOn(input.services.executionWorkspacesSvc, "prepareTaskRepositoriesForAdmission");
|
|
if (transport !== "daytona" || descriptor === "valid") {
|
|
const result = await prepareHeartbeatWorkspace(db, input);
|
|
expect(result.executionTarget).toEqual(target);
|
|
if (transport === "ssh" || transport === "task") expect(reserve).not.toHaveBeenCalled();
|
|
else expect(reserve).toHaveBeenCalledWith({ companyId: issue.companyId, issueId: issue.id,
|
|
workspaceId: first.persistedExecutionWorkspace!.id, cwd: admittedRoot, agentId: input.agent.id, runId: input.run.id });
|
|
if (transport !== "daytona") {
|
|
await expect(prepareNativeWorkspaceSync({ db, runId: input.run.id, companyId: issue.companyId,
|
|
workspaceId: first.persistedExecutionWorkspace!.id, workspaceLocalDir: first.executionWorkspace.cwd,
|
|
target, lease: result.activeEnvironmentLease!.lease })).resolves.toBeNull();
|
|
}
|
|
expect(admittedRoot).not.toBe(first.executionWorkspace.cwd);
|
|
} else {
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow(descriptor === "absent"
|
|
? "native_workspace_sync_descriptor_binding_missing" : "native_workspace_sync_descriptor_digest_mismatch");
|
|
expect(reserve).not.toHaveBeenCalled();
|
|
expect(await leases(input.run.id)).toMatchObject([{ status: "active" }]);
|
|
}
|
|
expect(prepare).not.toHaveBeenCalled();
|
|
expect((await readIssue(issue.id)).executionWorkspaceId).toBeNull();
|
|
});
|
|
|
|
it("removes a newly created worktree when persistence fails and preserves the original error", async () => {
|
|
const { input, source } = await fixture(true);
|
|
const error = new Error("workspace persistence unavailable");
|
|
let createdCwd = "";
|
|
vi.spyOn(input.services.executionWorkspacesSvc, "create").mockImplementation(async args => { createdCwd = args.cwd!; throw error; });
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toBe(error);
|
|
expect(createdCwd).not.toBe("");
|
|
await expect(access(createdCwd)).rejects.toThrow();
|
|
expect(await readFile(join(source, "README.md"), "utf8")).toBe("source content\n");
|
|
expect(await leases(input.run.id)).toHaveLength(0);
|
|
});
|
|
|
|
it("preserves a reused worktree when its metadata update fails", async () => {
|
|
const { input } = await fixture(true);
|
|
const first = await prepareHeartbeatWorkspace(db, input);
|
|
await reuse(input, first.persistedExecutionWorkspace!);
|
|
await writeFile(join(first.executionWorkspace.cwd, "retained.txt"), "unsaved work");
|
|
const error = new Error("workspace update unavailable");
|
|
vi.spyOn(input.services.executionWorkspacesSvc, "update").mockRejectedValueOnce(error);
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toBe(error);
|
|
expect(await readFile(join(first.executionWorkspace.cwd, "retained.txt"), "utf8")).toBe("unsaved work");
|
|
expect(await leases(input.run.id)).toHaveLength(0);
|
|
});
|
|
|
|
it.each(["acquire", "realize", "ownership"] as const)("preserves %s errors and durable lease state for outer cleanup", async failure => {
|
|
const { input, envOrchestrator } = await fixture();
|
|
const error = new Error(`${failure} failed`);
|
|
if (failure === "acquire") vi.spyOn(envOrchestrator, "acquireForRun").mockRejectedValueOnce(error);
|
|
if (failure === "realize") vi.spyOn(envOrchestrator, "realizeForRun").mockRejectedValueOnce(error);
|
|
if (failure === "ownership") vi.mocked(input.controllerLease.assertOwned).mockResolvedValueOnce(undefined).mockRejectedValueOnce(error);
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toBe(error);
|
|
expect(input.nativeRunnerPreparationSpans.at(-1)).toMatchObject({ name: failure === "realize" ? "environment.workspace.realize" : "environment.acquire", outcome: "failed" });
|
|
expect((await readRun(input.run.id)).status).toBe("running");
|
|
expect(await leases(input.run.id)).toHaveLength(failure === "acquire" ? 0 : 1);
|
|
if (failure !== "acquire") {
|
|
expect(await leases(input.run.id)).toMatchObject([{ status: "active" }]);
|
|
// The production executor terminalizes first, then releases by durable run ID.
|
|
await db.update(heartbeatRuns).set({ status: "failed" }).where(eq(heartbeatRuns.id, input.run.id));
|
|
await envOrchestrator.releaseForRun({ heartbeatRunId: input.run.id, companyId: input.run.companyId, agentId: input.agent.id, status: "released" });
|
|
expect(await leases(input.run.id)).toMatchObject([{ status: "released" }]);
|
|
}
|
|
});
|
|
|
|
it("checks the acquired environment allowlist before realization", async () => {
|
|
const { input, envOrchestrator } = await fixture();
|
|
input.policy.executionPolicy = { managedSandboxOnly: true };
|
|
const realize = vi.spyOn(envOrchestrator, "realizeForRun");
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow("forbids local execution");
|
|
expect(realize).not.toHaveBeenCalled();
|
|
expect(await leases(input.run.id)).toMatchObject([{ status: "active" }]);
|
|
});
|
|
|
|
it("rejects low-trust local execution before resolving or provisioning a workspace", async () => {
|
|
const { input } = await fixture(true);
|
|
input.policy.trustPreset = {
|
|
kind: "low_trust_review", preset: "low_trust_review", sourcePresets: {},
|
|
boundary: { mode: "low_trust_review", companyId: input.agent.companyId, rootIssueId: input.task.issueId! },
|
|
};
|
|
const resolve = vi.spyOn(input.services, "resolveWorkspaceForRun");
|
|
const create = vi.spyOn(input.services.executionWorkspacesSvc, "create");
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow("requires a sandbox environment");
|
|
expect(resolve).not.toHaveBeenCalled();
|
|
expect(create).not.toHaveBeenCalled();
|
|
expect(await leases(input.run.id)).toHaveLength(0);
|
|
});
|
|
|
|
it("rejects remote recovery without its retained sync binding before acquiring a lease", async () => {
|
|
const { input, envOrchestrator } = await fixture();
|
|
input.reuse.runOptions.nativeRestartRecovery = {
|
|
kind: "reattach_remote_runner", runId: input.run.id, leaseOwner: "controller",
|
|
controllerGeneration: 1, providerAttempt: 1, restartKind: "hard", recoveryRequestId: null,
|
|
remote: { providerLeaseId: "original-provider-lease", remoteCwd: "/original" },
|
|
};
|
|
const acquire = vi.spyOn(envOrchestrator, "acquireForRun");
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow("native_remote_recovery_lease_mismatch");
|
|
expect(acquire).not.toHaveBeenCalled();
|
|
expect(await leases(input.run.id)).toHaveLength(0);
|
|
expect(input.nativeRunnerPreparationSpans).toMatchObject([{ name: "environment.acquire", outcome: "failed" }]);
|
|
});
|
|
|
|
it("binds a replacement workspace from realization and snapshots the restore lease", async () => {
|
|
const { input, envOrchestrator, issue } = await fixture();
|
|
const realize = envOrchestrator.realizeForRun;
|
|
vi.spyOn(envOrchestrator, "realizeForRun").mockImplementation(async args => {
|
|
const result = await realize(args);
|
|
const workspace = await input.services.executionWorkspacesSvc.create({ companyId: issue.companyId, projectId: issue.projectId!, projectWorkspaceId: issue.projectWorkspaceId, sourceIssueId: issue.id, name: "Realized workspace", mode: "shared_workspace", strategyType: "project_primary", cwd: args.executionWorkspace.cwd });
|
|
return { ...result, persistedExecutionWorkspace: workspace, lease: { ...result.lease, metadata: { remoteCwd: "/original" } } };
|
|
});
|
|
const result = await prepareHeartbeatWorkspace(db, input);
|
|
expect((await readIssue(issue.id)).executionWorkspaceId).toBe(result.persistedExecutionWorkspace!.id);
|
|
expect(result.persistedExecutionWorkspace!.name).toBe("Realized workspace");
|
|
result.activeEnvironmentLease!.lease.metadata!.remoteCwd = "/rebound";
|
|
expect(result.workspaceRestoreSource!.metadata?.remoteCwd).toBe("/original");
|
|
});
|
|
});
|