mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-08 22:13:59 +01:00
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on sandbox environments sync the host workspace into the
sandbox before the provider starts
> - Runtime selection writes the native execution input first. It writes
the workspace sync reference only after the sandbox is seeded, and
before any provider dispatch
> - A server restart between those two writes leaves a run with an input
but no sync reference
> - Restart recovery then fails the run during workspace preparation
with `native_workspace_sync_descriptor_binding_missing`, although the
sandbox never received any work from that run
> - This pull request lets recovery re-admit the host workspace in that
state, so sync preparation starts fresh on the same run
> - The benefit is that a deploy or restart during sandbox preparation
no longer turns queued work into a terminal `setup_failed` run
## Linked Issues or Issue Description
**What happened?**
After a server restart, six native sandbox runs failed with
`setup_failed` and the error
`native_workspace_sync_descriptor_binding_missing`. Each run timeline
shows the same order: run started, `snapshot.git`, `snapshot.baseline`,
then "Server shutdown suspended native controller ownership", then a
restart recovery claim ("Restarting an incomplete native runner
bootstrap" or "Resuming the durable native provider session"), then the
error. No run had a `nativeWorkspaceSync` reference. Each run had a
persisted `nativeExecutionInput`.
**Expected behavior**
When sandbox sync preparation did not finish, restart recovery prepares
the sandbox workspace again from the host workspace and continues the
run.
**Steps to reproduce**
1. Start a native run on a sandbox environment.
2. Restart the server after runtime selection persists
`nativeExecutionInput` but before `prepareNativeWorkspaceSync` persists
its reference.
3. Restart recovery re-enters workspace preparation and fails with
`native_workspace_sync_descriptor_binding_missing`.
**Paperclip version or commit**
Reproduced on `master` at be20d4ad97.
**Relevant logs or output**
`native_workspace_sync_descriptor_binding_missing` thrown from
`server/src/services/heartbeat/workspace-preparation.ts`.
Related: #17838 (preparation recovery ownership). No open PR covers this
guard.
## What Changed
- `server/src/services/heartbeat/workspace-preparation.ts`: remove the
guard that failed sandbox recovery when the run has no workspace sync
reference. The existing fallback admits `executionWorkspace.cwd`, which
is the same host directory that `prepareNativeWorkspaceSync` then uses
as `workspaceLocalDir`. A comment records the ordering invariant that
makes this safe.
- `server/src/services/heartbeat/workspace-preparation.test.ts`: the
Daytona "absent descriptor" recovery case now expects preparation to
succeed and to re-admit the host workspace. The "corrupt descriptor"
case still fails closed with
`native_workspace_sync_descriptor_digest_mismatch`.
## Verification
- Red before the fix: `npx vitest run
src/services/heartbeat/workspace-preparation.test.ts -t "validates the
admitted host binding"` (from `server/`) failed for "daytona recovery
with absent descriptor" with
`native_workspace_sync_descriptor_binding_missing`, the same error as
the failed runs.
- Green after the fix: the full
`src/services/heartbeat/workspace-preparation.test.ts` file passes,
including the local, ssh, runner-task, valid-descriptor and
corrupt-descriptor recovery cases.
- Ordering check (code read): `runtime-selection.ts` persists
`nativeExecutionInput` in the selection transaction, then calls
`prepareNativeWorkspaceSync`, which persists the reference as its last
step. `heartbeat.ts` calls `executeHeartbeatRuntime` only after runtime
selection returns. No code path removes `nativeWorkspaceSync` from
`runnerProfileJson`.
- Not run: a live restart on a sandbox environment. The original staging
runs are not recovered by this PR.
## Risks
- Low risk. The change only affects native recovery on sandbox targets
(not runner-task, ssh, remote-authority or controller-only runs) when
the sync reference is absent.
- A present but invalid descriptor still fails closed.
- If a future change dispatches a provider before the sync reference is
persisted, this path would re-seed over remote work. The comment at the
call site records that invariant.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, browser reads of run records).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
1885 lines
141 KiB
TypeScript
1885 lines
141 KiB
TypeScript
import { execFile as execFileCallback } from "node:child_process";
|
|
import { randomUUID } from "node:crypto";
|
|
import { mkdtemp, mkdir, readFile, readdir, rm, writeFile, access, rename, symlink } from "node:fs/promises";
|
|
import { tmpdir } from "node:os";
|
|
import { join } from "node:path";
|
|
import { pathToFileURL } from "node:url";
|
|
import { promisify } from "node:util";
|
|
import { captureDirectorySnapshot, directorySnapshotSha256, serializeDirectorySnapshot } from "@paperclipai/adapter-utils/workspace-restore-merge";
|
|
import type { AdapterExecutionTarget } from "@paperclipai/adapter-utils/execution-target";
|
|
import { agentWakeupRequests, heartbeatRunEvents, companyObservability, issueRecoveryActions, agents, authUsers, companyMemberships, companies, createDb, applyPendingMigrations, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, environmentLeases, environments, executionWorkspaces, executionWorkspaceRepositoryAccess, heartbeatRuns, issues, issueAccessGrants, nativeRunFinalizations, plugins, projects, projectWorkspaces, workspaceOperations, type Db } from "@paperclipai/db";
|
|
import { eq, sql } from "drizzle-orm";
|
|
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
|
|
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
|
|
import { resolveExternalTestDatabaseUrl } from "../../__tests__/helpers/external-test-database.js";
|
|
import { heartbeatService } from "../heartbeat.js";
|
|
import { registerServerAdapter, getServerAdapter, type AdapterExecutionContext } from "../../adapters/index.js";
|
|
import { environmentRunOrchestrator } from "../environment-run-orchestrator.js";
|
|
import { environmentService } from "../environments.js";
|
|
import { executionWorkspaceService, toExecutionWorkspace } from "../execution-workspaces.js";
|
|
import { instanceSettingsService } from "../instance-settings.js";
|
|
import { workspaceOperationService } from "../workspace-operations.js";
|
|
import { nativeWorkspaceSyncInternals, prepareNativeWorkspaceSync } from "../native-runtime/native-workspace-sync.js";
|
|
import { createHeartbeatWorkspaceResolver, prepareProjectRepositoryWorkspaces, resolveExecutionWorkspaceReuseRequestForIssue } from "./workspaces.js";
|
|
import { prepareHeartbeatWorkspace, type HeartbeatWorkspacePreparationInput } from "./workspace-preparation.js";
|
|
import { buildNativeExecutionInput } from "../native-runtime/native-execution-input.js";
|
|
import { nativeRuntimeContextFixture } from "../native-runtime/runtime-context.test-fixture.js";
|
|
import * as managedRepositoryCheckout from "../managed-repository-checkout.js";
|
|
import * as freshWorkspaceSeed from "../fresh-workspace-seed.js";
|
|
import { requestTaskRetry } from "../task-retry.js";
|
|
import { readFreshWorkspaceReadiness } from "../fresh-workspace-continuation.js";
|
|
import { createSystemRecovery } from "../system-recovery-state.js";
|
|
import { isolatedRemoteSourcePath } from "../isolated-remote-workspace-source.js";
|
|
import { cleanupExecutionWorkspaceArtifacts } from "../workspace-runtime.js";
|
|
import { buildProjectMentionHref, readRemoteWorkspaceAuthority } from "@paperclipai/shared";
|
|
import { migrateRemoteWorkspace } from "../remote-workspace-migration.js";
|
|
import { trace } from "@opentelemetry/api";
|
|
import { effectiveProjectWorkspacePolicy } from "../project-workspace-provider.js";
|
|
import { resolveExecutionWorkspaceMode, buildExecutionWorkspaceAdapterConfig } from "../execution-workspace-policy.js";
|
|
import { createLocalDiskStorageProvider } from "../../storage/local-disk-provider.js";
|
|
import { createStorageService } from "../../storage/service.js";
|
|
import { InMemorySpanExporter, SimpleSpanProcessor, TracerProvider } from "@opentelemetry/sdk-trace";
|
|
import { selectHeartbeatEnvironment } from "./environment-selection.js";
|
|
import { assertNativeSessionPreparationAuthority } from "../native-runtime/native-execution-lease.js";
|
|
import { currentNativeControllerIdentity } from "../native-runtime/native-restart-recovery.js";
|
|
import { projectService } from "../projects.js";
|
|
import { WORKSPACE_BUSY_HOLDER_STALE_AFTER_MS, WorkspaceBusyDeferral } from "./retries.js";
|
|
|
|
const execFile = promisify(execFileCallback);
|
|
const externalDatabaseUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_WORKSPACE_PREPARATION_TEST_DATABASE_URL");
|
|
const support = externalDatabaseUrl ? { supported: true } : await getEmbeddedPostgresTestSupport();
|
|
|
|
describe.skipIf(!support.supported && !externalDatabaseUrl)("heartbeat workspace preparation boundary", () => {
|
|
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
|
let db: Db;
|
|
let home: string;
|
|
const persistenceKeys = ["PAPERCLIP_EPHEMERAL_SERVER", "DATABASE_URL", "PAPERCLIP_STORAGE_PROVIDER", "PAPERCLIP_SECRETS_MASTER_KEY", "PAPERCLIP_SECRETS_MASTER_KEY_FILE"];
|
|
let persistenceBefore = new Map<string, string | undefined>();
|
|
const configureRemotePersistence = () => Object.assign(process.env, { DATABASE_URL: database.connectionString,
|
|
PAPERCLIP_STORAGE_PROVIDER: "s3", PAPERCLIP_SECRETS_MASTER_KEY: "remote-workspace-preparation-fixture" });
|
|
|
|
beforeAll(async () => {
|
|
home = await mkdtemp(join(tmpdir(), "paperclip-workspace-preparation-"));
|
|
vi.stubEnv("PAPERCLIP_HOME", home);
|
|
const external = externalDatabaseUrl;
|
|
if (external) await applyPendingMigrations(external);
|
|
database = external ? { connectionString: external, cleanup: async () => {} }
|
|
: await startEmbeddedPostgresTestDatabase("paperclip-workspace-preparation");
|
|
db = createDb(database.connectionString);
|
|
await db.execute(sql`set client_min_messages = warning`);
|
|
if (external) await db.execute(sql`truncate table environments restart identity cascade`);
|
|
}, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS);
|
|
afterAll(async () => {
|
|
await db?.$client.end({ timeout: 1 });
|
|
await database?.cleanup();
|
|
vi.unstubAllEnvs();
|
|
if (home) await rm(home, { recursive: true, force: true });
|
|
}, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS);
|
|
beforeEach(async () => {
|
|
persistenceBefore = new Map(persistenceKeys.map(key => [key, process.env[key]]));
|
|
vi.restoreAllMocks();
|
|
await db.execute(sql`truncate table companies, plugins restart identity cascade`);
|
|
await instanceSettingsService(db).updateExperimental({ enableIsolatedWorkspaces: true });
|
|
});
|
|
|
|
afterEach(() => {
|
|
for (const [key, value] of persistenceBefore) {
|
|
if (value === undefined) delete process.env[key]; else process.env[key] = value;
|
|
}
|
|
});
|
|
|
|
async function fixture(isolated = false) {
|
|
const [company] = await db.insert(companies).values({ name: "Workspace preparation", issuePrefix: "WSP" }).returning();
|
|
const [agent] = await db.insert(agents).values({ companyId: company.id, name: "Workspace agent", status: "running", adapterType: "process" }).returning();
|
|
const [project] = await db.insert(projects).values({ companyId: company.id, name: "Source project" }).returning();
|
|
const source = join(home, project.id);
|
|
await mkdir(source);
|
|
await execFile("git", ["init", "--initial-branch=main", source]);
|
|
await execFile("git", ["-C", source, "config", "user.email", "test@example.com"]);
|
|
await execFile("git", ["-C", source, "config", "user.name", "Workspace test"]);
|
|
await writeFile(join(source, "README.md"), "source content\n");
|
|
await execFile("git", ["-C", source, "add", "."]);
|
|
await execFile("git", ["-C", source, "commit", "-m", "Source"]);
|
|
const [projectWorkspace] = await db.insert(projectWorkspaces).values({ companyId: company.id, projectId: project.id, name: "Source", cwd: source, isPrimary: true }).returning();
|
|
const [issue] = await db.insert(issues).values({ companyId: company.id, title: "Prepare this workspace", identifier: "WSP-1", projectId: project.id, projectWorkspaceId: projectWorkspace.id, assigneeAgentId: agent.id }).returning();
|
|
const [run] = await db.insert(heartbeatRuns).values({ companyId: company.id, agentId: agent.id, issueId: issue.id, scopeKind: "issue", status: "running", runtimeMode: "legacy" }).returning();
|
|
const localEnvironment = await environmentService(db).ensureLocalEnvironment(company.id);
|
|
const envOrchestrator = environmentRunOrchestrator(db);
|
|
const resolver = createHeartbeatWorkspaceResolver(db);
|
|
const input: HeartbeatWorkspacePreparationInput = {
|
|
run, agent,
|
|
task: { issueRef: issue, issueId: issue.id, wasIssueAssignee: true, context: { issueId: issue.id, projectId: project.id }, responsibleUserId: null, previousSessionParams: null },
|
|
policy: {
|
|
trustPreset: { kind: "standard", preset: "standard", boundary: null, sourcePresets: {} },
|
|
isolatedWorkspacesEnabled: true,
|
|
effectiveExecutionWorkspaceMode: isolated ? "isolated_workspace" : "shared_workspace",
|
|
requestedExecutionWorkspaceMode: isolated ? "isolated_workspace" : "shared_workspace",
|
|
useIsolatedTaskDirectory: false, nativeChatWorkspaceScope: null,
|
|
projectExecutionWorkspacePolicy: null, issueExecutionWorkspaceSettings: null, environmentExecutionWorkspaceSettings: null,
|
|
executionPolicy: { executionMode: "any" },
|
|
},
|
|
environment: { selectedEnvironmentId: localEnvironment.id, localEnvironment, selectedEnvironmentForConfig: localEnvironment, environmentResolution: { environmentId: localEnvironment.id, source: "default" }, resolvedInstanceSettings: await instanceSettingsService(db).get() },
|
|
config: { mergedConfig: isolated ? { workspaceStrategy: { type: "git_worktree", baseRef: "main", branchTemplate: "test-{{issue.identifier}}" } } : {}, configSnapshot: null, secretManifest: [] },
|
|
reuse: { requestedShouldReuseExisting: false, existingExecutionWorkspace: null, reusableExistingExecutionWorkspace: null, workspaceReuseRequest: resolveExecutionWorkspaceReuseRequestForIssue({ issueExecutionWorkspaceId: null, issueExecutionWorkspacePreference: null, existingExecutionWorkspaceStatus: null }), nativeRecoveryExecutionWorkspaceId: null, persistedNativeExecutionInput: null, isDotRun: false, runOptions: {} },
|
|
services: { envOrchestrator, executionWorkspacesSvc: executionWorkspaceService(db), workspaceOperationsSvc: workspaceOperationService(db), ...resolver, appendRunEvent: vi.fn() },
|
|
controllerLease: { assertOwned: vi.fn(async () => {}) }, nativeRunnerPreparationSpans: [],
|
|
};
|
|
return { input, source, issue, run, envOrchestrator };
|
|
}
|
|
it.each([false, true])("places Cursor Cloud in its provider API without a Computer under managed-only policy (ephemeral: %s)", async ephemeral => {
|
|
vi.stubEnv("PAPERCLIP_EPHEMERAL_SERVER", String(ephemeral));
|
|
const { input, source, issue, run } = await fixture();
|
|
input.agent = { ...input.agent, adapterType: "paperclip_runner", adapterConfig: {
|
|
provider: "cursor_cloud", model: "caller-model", repoUrl: "https://github.com/example/qualification",
|
|
} };
|
|
await db.update(agents).set({ adapterType: input.agent.adapterType, adapterConfig: input.agent.adapterConfig }).where(eq(agents.id, input.agent.id));
|
|
const settingsBefore = await instanceSettingsService(db).get();
|
|
try {
|
|
await instanceSettingsService(db).updateExperimental({ enableManagedSandboxOnly: true, enableNativeRunner: true });
|
|
const selectionInput = {
|
|
run: input.run, agent: input.agent,
|
|
task: { issueRef: issue, issueContext: issue, issueId: issue.id, context: input.task.context,
|
|
executionProjectId: issue.projectId, workspaceProjectContext: { hasWorkspace: true }, boundSourceWorkspace: null,
|
|
hasSelectedWorkspaceSource: true, taskWorkspaceIntent: null, responsibleUserId: null },
|
|
policy: { ...input.policy }, config: { config: input.agent.adapterConfig, sourceWorkspaceConfig: input.agent.adapterConfig, issueAssigneeOverrides: null },
|
|
services: { environmentsSvc: environmentService(db), executionWorkspacesSvc: executionWorkspaceService(db),
|
|
instanceSettings: instanceSettingsService(db), findSharedWorkspaceHolder: async () => null, findExecutionWorkspaceHolder: async () => null },
|
|
};
|
|
const selection = await selectHeartbeatEnvironment(db, selectionInput);
|
|
expect(selection).toMatchObject({ providerApiPlacement: { kind: "provider_api", provider: "cursor_cloud", controller: "paperclip" },
|
|
selectedEnvironmentId: null, selectedEnvironmentForConfig: null, executionPolicy: { managedSandboxOnly: true } });
|
|
input.environment = { ...input.environment, selectedEnvironmentId: selection.selectedEnvironmentId,
|
|
selectedEnvironmentForConfig: selection.selectedEnvironmentForConfig, environmentResolution: selection.environmentResolution,
|
|
resolvedInstanceSettings: selection.resolvedInstanceSettings };
|
|
input.policy.executionPolicy = selection.executionPolicy;
|
|
const acquire = vi.spyOn(input.services.envOrchestrator, "acquireForRun");
|
|
const realize = vi.spyOn(input.services.envOrchestrator, "realizeForRun");
|
|
const resolveWorkspace = vi.spyOn(input.services, "resolveWorkspaceForRun");
|
|
const result = await prepareHeartbeatWorkspace(db, input);
|
|
expect(result).toMatchObject({ selectedEnvironment: null, activeEnvironmentLease: null, executionTarget: null,
|
|
persistedExecutionWorkspace: null, projectRepositoryPaths: [], hostExecutionWorkspaceConfig: {}, skipHostProvision: true });
|
|
expect(result.executionWorkspace.cwd).toContain(`/cursor-cloud-controllers/${input.agent.companyId}/${input.agent.id}/${issue.id}`);
|
|
expect(result.executionWorkspace).toMatchObject({ projectId: null, repoUrl: null, repoRef: null, created: false });
|
|
expect(await readFile(join(source, "README.md"), "utf8")).toBe("source content\n");
|
|
expect(acquire).not.toHaveBeenCalled(); expect(realize).not.toHaveBeenCalled(); expect(resolveWorkspace).not.toHaveBeenCalled();
|
|
expect(await db.select().from(environmentLeases).where(eq(environmentLeases.heartbeatRunId, run.id))).toEqual([]);
|
|
expect(await db.select().from(executionWorkspaces)).toEqual([]);
|
|
expect((await instanceSettingsService(db).getExperimental()).enableManagedSandboxOnly).toBe(true);
|
|
await instanceSettingsService(db).updateGeneral({ executionMode: "kubernetes" });
|
|
await expect(selectHeartbeatEnvironment(db, selectionInput)).rejects.toMatchObject({ status: 422,
|
|
details: { code: "provider_api_kubernetes_execution_unsupported" } });
|
|
} finally {
|
|
await instanceSettingsService(db).updateGeneral({ executionMode: settingsBefore.general.executionMode ?? "any" });
|
|
await instanceSettingsService(db).updateExperimental({
|
|
enableManagedSandboxOnly: settingsBefore.experimental.enableManagedSandboxOnly,
|
|
enableNativeRunner: settingsBefore.experimental.enableNativeRunner,
|
|
});
|
|
}
|
|
});
|
|
it.each(["unconfigured", "operator-disabled"] as const)("auto-enrollment keeps %s native sandboxes on their existing host path", async mode => {
|
|
const { input, source, envOrchestrator } = await fixture();
|
|
input.agent.adapterType = "paperclip_runner";
|
|
input.environment.selectedEnvironmentForConfig = { ...input.environment.localEnvironment,
|
|
driver: "sandbox", config: { provider: "daytona" } };
|
|
input.environment.resolvedInstanceSettings.experimental.remoteWorkspaceStorageEnabled = true;
|
|
if (mode === "operator-disabled") {
|
|
configureRemotePersistence();
|
|
input.environment.resolvedInstanceSettings.experimental.remoteWorkspaceStorageEnabled = false;
|
|
} else {
|
|
Object.assign(process.env, { DATABASE_URL: database.connectionString, PAPERCLIP_STORAGE_PROVIDER: "local_disk" });
|
|
}
|
|
const resolve = vi.spyOn(input.services, "resolveWorkspaceForRun").mockRejectedValue(new Error("host authority retained"));
|
|
const acquire = vi.spyOn(envOrchestrator, "acquireForRun");
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow("host authority retained");
|
|
expect(resolve).toHaveBeenCalledOnce(); expect(acquire).not.toHaveBeenCalled();
|
|
expect(await readFile(join(source, "README.md"), "utf8")).toBe("source content\n");
|
|
expect(await db.select().from(executionWorkspaces)).toHaveLength(0);
|
|
});
|
|
|
|
it("auto-enrollment preserves an existing host workspace even when persistence is ready", async () => {
|
|
const { input, source, issue, envOrchestrator } = await fixture();
|
|
const [workspace] = await db.insert(executionWorkspaces).values({ companyId: issue.companyId,
|
|
sourceIssueId: issue.id, projectId: issue.projectId, projectWorkspaceId: issue.projectWorkspaceId,
|
|
name: "Existing host workspace", cwd: source, providerType: "local_fs", strategyType: "project_primary",
|
|
mode: "shared_workspace", status: "active" }).returning();
|
|
await writeFile(join(source, "unpublished.txt"), "keep existing work\n");
|
|
await db.update(issues).set({ executionWorkspaceId: workspace.id }).where(eq(issues.id, issue.id));
|
|
input.reuse.existingExecutionWorkspace = toExecutionWorkspace(workspace);
|
|
input.reuse.reusableExistingExecutionWorkspace = toExecutionWorkspace(workspace);
|
|
input.reuse.requestedShouldReuseExisting = true;
|
|
input.agent.adapterType = "paperclip_runner";
|
|
input.environment.selectedEnvironmentForConfig = { ...input.environment.localEnvironment,
|
|
driver: "sandbox", config: { provider: "daytona" } };
|
|
input.environment.resolvedInstanceSettings.experimental.remoteWorkspaceStorageEnabled = true;
|
|
configureRemotePersistence();
|
|
const resolve = vi.spyOn(input.services, "resolveWorkspaceForRun").mockRejectedValue(new Error("host authority retained"));
|
|
const acquire = vi.spyOn(envOrchestrator, "acquireForRun");
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow("host authority retained");
|
|
expect(resolve).toHaveBeenCalledOnce(); expect(acquire).not.toHaveBeenCalled();
|
|
expect(await readFile(join(source, "unpublished.txt"), "utf8")).toBe("keep existing work\n");
|
|
expect((await db.select().from(executionWorkspaces))[0]?.metadata?.remoteWorkspace).toBeUndefined();
|
|
});
|
|
|
|
it.each([
|
|
{ kind: "empty", repoUrl: null, branchMode: null, fresh: false, strategyChange: false, ephemeral: true },
|
|
{ kind: "git", repoUrl: "https://github.com/example/remote.git", branchMode: null, fresh: false, strategyChange: false },
|
|
{ kind: "git", repoUrl: "ssh://git@github.com/example/remote.git", branchMode: null, fresh: false, strategyChange: false },
|
|
{ kind: "git", repoUrl: "git@github.com:example/remote.git", branchMode: null, fresh: false, strategyChange: false },
|
|
{ kind: "git", repoUrl: "https://github.com/example/remote.git", branchMode: "existing", fresh: false, strategyChange: false },
|
|
{ kind: "git", repoUrl: "https://github.com/example/remote.git", branchMode: "create", fresh: false, strategyChange: false },
|
|
{ kind: "git", repoUrl: "https://github.com/example/remote.git", branchMode: "create", fresh: true, strategyChange: false },
|
|
{ kind: "git", repoUrl: "https://github.com/example/remote.git", branchMode: "create", fresh: true, strategyChange: true },
|
|
] as const)("reserves a $kind remote workspace ($repoUrl, branch: $branchMode, fresh: $fresh, strategy change: $strategyChange) without materializing a controller checkout", async (scenario) => {
|
|
const { kind, repoUrl, branchMode, fresh, strategyChange } = scenario;
|
|
const { input, source, issue, envOrchestrator } = await fixture();
|
|
const [savedEnvironment] = await db.insert(environments).values({ name: `Remote ${randomUUID()}`,
|
|
driver: "sandbox", status: "active", config: { provider: "daytona" } }).returning();
|
|
const environment = { ...input.environment.localEnvironment, id: savedEnvironment.id,
|
|
driver: "sandbox" as const, config: { provider: "daytona" } };
|
|
await db.update(agents).set({ adapterType: "paperclip_runner" }).where(eq(agents.id, input.agent.id));
|
|
input.agent.adapterType = "paperclip_runner";
|
|
input.environment.selectedEnvironmentForConfig = environment;
|
|
input.environment.selectedEnvironmentId = environment.id;
|
|
expect(input.environment.resolvedInstanceSettings.experimental.remoteWorkspaceStorageEnabled).toBe(true);
|
|
input.policy.requestedExecutionWorkspaceMode = "agent_default";
|
|
input.policy.useIsolatedTaskDirectory = kind === "empty";
|
|
if (branchMode) {
|
|
input.policy.requestedExecutionWorkspaceMode = "isolated_workspace";
|
|
input.policy.effectiveExecutionWorkspaceMode = "isolated_workspace";
|
|
input.config.mergedConfig = strategyChange ? { workspaceStrategy: { type: "project_primary" } } : { workspaceStrategy: { type: "git_worktree", baseRef: "main",
|
|
...(branchMode === "existing" ? { existingBranch: "feature/diverged" } : { branchTemplate: "test-{{issue.identifier}}" }) } };
|
|
}
|
|
if (kind === "git") {
|
|
await db.update(projectWorkspaces).set({ cwd: null, sourceType: "git", repoUrl, repoRef: "main" })
|
|
.where(eq(projectWorkspaces.id, issue.projectWorkspaceId!));
|
|
}
|
|
if (fresh) {
|
|
const sourceRunId = randomUUID(), retryIntentId = randomUUID();
|
|
await db.insert(heartbeatRuns).values({ id: sourceRunId, companyId: issue.companyId, agentId: input.agent.id,
|
|
issueId: issue.id, status: "failed", finishedAt: new Date() });
|
|
await db.insert(agentWakeupRequests).values({ id: retryIntentId, companyId: issue.companyId,
|
|
agentId: input.agent.id, reason: "task_retry", source: "on_demand", status: "coalesced", runId: input.run.id,
|
|
payload: { issueId: issue.id } });
|
|
const readiness = { mode: "fresh_isolated_workspace" as const, retryIntentId, sourceRunId, retainedSourceIds: [] };
|
|
await db.insert(issueRecoveryActions).values({ companyId: issue.companyId, sourceIssueId: issue.id,
|
|
kind: "active_run_watchdog", status: "resolved", ownerType: "system", nextAction: "Synthetic committed fresh attempt",
|
|
cause: "native_continuation_requires_reconciliation", fingerprint: randomUUID(), evidence: { runId: sourceRunId,
|
|
explicitUserContinuation: { runId: input.run.id, readiness }, systemRecovery: {
|
|
...createSystemRecovery(sourceRunId, retryIntentId), state: "exhausted", freshAttemptRunId: input.run.id,
|
|
} } });
|
|
input.task.freshWorkspace = await readFreshWorkspaceReadiness(db, input.run);
|
|
expect(input.task.freshWorkspace).toEqual(readiness);
|
|
}
|
|
await rm(source, { recursive: true });
|
|
const resolve = vi.spyOn(input.services, "resolveWorkspaceForRun").mockRejectedValue(new Error("host source resolver called"));
|
|
const hostSeed = vi.spyOn(freshWorkspaceSeed, "materializeFreshWorkspaceSeed");
|
|
const repositoryPreparation = vi.spyOn(input.services.executionWorkspacesSvc, "prepareTaskRepositoriesForAdmission");
|
|
vi.spyOn(envOrchestrator, "acquireForRun").mockImplementation(async args => {
|
|
expect(args.persistedExecutionWorkspace).toMatchObject({ providerType: "cloud_sandbox" });
|
|
expect(args.persistedExecutionWorkspace).toMatchObject({ cwd: expect.stringMatching(/^\/workspaces\//) });
|
|
throw new Error("reached remote acquisition");
|
|
});
|
|
const configured = { DATABASE_URL: database.connectionString, PAPERCLIP_STORAGE_PROVIDER: "s3",
|
|
PAPERCLIP_SECRETS_MASTER_KEY: "remote-workspace-preparation-fixture",
|
|
PAPERCLIP_EPHEMERAL_SERVER: "ephemeral" in scenario ? "true" : "false" };
|
|
const previous = new Map(Object.keys(configured).map(key => [key, process.env[key]]));
|
|
Object.assign(process.env, configured);
|
|
try {
|
|
// Fresh runs are still marked legacy until the runtime selector owns promotion.
|
|
expect(input.run.runtimeMode).toBe("legacy");
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow("reached remote acquisition");
|
|
expect(resolve).not.toHaveBeenCalled();
|
|
expect(repositoryPreparation).not.toHaveBeenCalled();
|
|
expect(hostSeed).not.toHaveBeenCalled();
|
|
const [workspace] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.sourceIssueId, issue.id));
|
|
expect(workspace.mode).toBe(branchMode ? "isolated_workspace" : "shared_workspace");
|
|
expect(workspace.metadata?.remoteWorkspace).toMatchObject({ source: { kind }, writerRunId: input.run.id });
|
|
if (branchMode) expect(workspace.metadata?.remoteWorkspace).toMatchObject({ source: {
|
|
ref: "main", branchMode,
|
|
...(strategyChange ? {} : { branchName: branchMode === "existing" ? "feature/diverged" : "test-WSP-1" }),
|
|
} });
|
|
await expect(access(workspace.cwd!)).rejects.toThrow();
|
|
if ((fresh || strategyChange) && branchMode === "create") {
|
|
expect(workspace.strategyType).toBe("git_worktree");
|
|
await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date() })
|
|
.where(eq(heartbeatRuns.id, input.run.id));
|
|
const [saved] = await db.update(executionWorkspaces).set({ metadata: {
|
|
...workspace.metadata,
|
|
remoteWorkspace: { ...workspace.metadata!.remoteWorkspace as object, writerRunId: null, dirty: false },
|
|
} }).where(eq(executionWorkspaces.id, workspace.id)).returning();
|
|
const [nextRun] = await db.insert(heartbeatRuns).values({ companyId: issue.companyId,
|
|
agentId: input.agent.id, issueId: issue.id, scopeKind: "issue", status: "running", runtimeMode: "legacy" }).returning();
|
|
input.run = nextRun;
|
|
input.task.freshWorkspace = null;
|
|
input.task.issueRef = { ...input.task.issueRef!, executionWorkspaceId: saved.id, executionWorkspacePreference: "reuse_existing" };
|
|
input.reuse.existingExecutionWorkspace = toExecutionWorkspace(saved);
|
|
input.reuse.reusableExistingExecutionWorkspace = toExecutionWorkspace(saved);
|
|
input.reuse.requestedShouldReuseExisting = true;
|
|
input.reuse.workspaceReuseRequest = resolveExecutionWorkspaceReuseRequestForIssue({
|
|
issueExecutionWorkspaceId: saved.id, issueExecutionWorkspacePreference: "reuse_existing", existingExecutionWorkspaceStatus: "active",
|
|
});
|
|
if (strategyChange) input.config.mergedConfig = { workspaceStrategy: { type: "git_worktree" } };
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow("reached remote acquisition");
|
|
const operations = await db.select().from(workspaceOperations).where(eq(workspaceOperations.heartbeatRunId, nextRun.id));
|
|
const freshness = operations.filter(operation => operation.phase === "workspace_config_freshness");
|
|
if (strategyChange) {
|
|
expect(freshness).toHaveLength(1);
|
|
expect(freshness[0].metadata).toMatchObject({ action: "replace", changedCategories: ["strategy"] });
|
|
} else expect(freshness).toEqual([]);
|
|
const [retained] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, saved.id));
|
|
expect(retained.id).toBe(workspace.id);
|
|
expect(retained.metadata?.configFingerprint).toMatchObject({
|
|
workspaceHash: (saved.metadata?.configFingerprint as { workspaceHash: string }).workspaceHash,
|
|
});
|
|
expect(retained.metadata?.remoteWorkspace).toMatchObject({ writerRunId: nextRun.id });
|
|
}
|
|
} finally {
|
|
for (const [key, value] of previous) {
|
|
if (value === undefined) delete process.env[key]; else process.env[key] = value;
|
|
}
|
|
}
|
|
});
|
|
|
|
async function inheritedRemoteFixture(mode: "isolated_workspace" | "operator_branch") {
|
|
const { input, issue, envOrchestrator } = await fixture();
|
|
configureRemotePersistence();
|
|
process.env.PAPERCLIP_SECRETS_MASTER_KEY = "71".repeat(32);
|
|
input.agent.adapterType = "paperclip_runner";
|
|
input.environment.selectedEnvironmentForConfig = { ...input.environment.localEnvironment,
|
|
driver: "sandbox", config: { provider: "daytona" } };
|
|
input.policy.requestedExecutionWorkspaceMode = mode;
|
|
input.policy.effectiveExecutionWorkspaceMode = mode;
|
|
const [parent] = await db.insert(issues).values({ companyId: issue.companyId, projectId: issue.projectId,
|
|
projectWorkspaceId: issue.projectWorkspaceId, title: "Still running coordinator", assigneeAgentId: input.agent.id }).returning();
|
|
const [writer] = await db.insert(heartbeatRuns).values({ companyId: issue.companyId, agentId: input.agent.id,
|
|
nativeIssueId: parent.id, runtimeMode: "native", status: "running", startedAt: new Date(), lastOutputAt: new Date() }).returning();
|
|
const authority = { schema: "paperclip.remote-workspace.v1" as const, epoch: 3,
|
|
snapshotId: randomUUID(), source: { kind: "empty" as const }, location: null, writerRunId: writer.id, dirty: true };
|
|
const [workspace] = await db.insert(executionWorkspaces).values({ companyId: issue.companyId,
|
|
sourceIssueId: parent.id, projectId: issue.projectId, projectWorkspaceId: issue.projectWorkspaceId,
|
|
name: "Inherited remote workspace", mode, strategyType: "project_primary", providerType: "cloud_sandbox",
|
|
cwd: `/workspaces/${randomUUID()}`, metadata: { remoteWorkspace: authority } }).returning();
|
|
await db.update(issues).set({ executionWorkspaceId: workspace.id, executionWorkspacePreference: "reuse_existing" })
|
|
.where(eq(issues.id, issue.id));
|
|
input.task.issueRef = { ...issue, executionWorkspaceId: workspace.id, executionWorkspacePreference: "reuse_existing" };
|
|
input.reuse.existingExecutionWorkspace = toExecutionWorkspace(workspace);
|
|
input.reuse.reusableExistingExecutionWorkspace = toExecutionWorkspace(workspace);
|
|
input.reuse.requestedShouldReuseExisting = true;
|
|
input.reuse.workspaceReuseRequest = resolveExecutionWorkspaceReuseRequestForIssue({ issueExecutionWorkspaceId: workspace.id,
|
|
issueExecutionWorkspacePreference: "reuse_existing", existingExecutionWorkspaceStatus: "active" });
|
|
return { input, issue, envOrchestrator, parent, writer, workspace, authority };
|
|
}
|
|
|
|
it.each(["isolated_workspace", "operator_branch"] as const)("waits for the live writer of an inherited remote %s at reservation", async mode => {
|
|
const { input, issue, envOrchestrator, parent, writer, workspace, authority } = await inheritedRemoteFixture(mode);
|
|
const acquire = vi.spyOn(envOrchestrator, "acquireForRun").mockRejectedValue(new Error("writer released; acquisition admitted"));
|
|
const readAuthority = async () => readRemoteWorkspaceAuthority((await db.select().from(executionWorkspaces)
|
|
.where(eq(executionWorkspaces.id, workspace.id)))[0].metadata);
|
|
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toMatchObject({
|
|
constructor: WorkspaceBusyDeferral, code: "workspace_busy", executionWorkspaceId: workspace.id,
|
|
holder: { runId: writer.id, issueId: parent.id }, wasIssueAssignee: true,
|
|
});
|
|
expect(acquire).not.toHaveBeenCalled();
|
|
expect(await readAuthority()).toEqual(authority);
|
|
|
|
await db.update(heartbeatRuns).set({ lastOutputAt: new Date(Date.now() - WORKSPACE_BUSY_HOLDER_STALE_AFTER_MS - 1_000) })
|
|
.where(eq(heartbeatRuns.id, writer.id));
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toMatchObject({ status: 409 });
|
|
// An ended writer whose save has not settled still holds the workspace:
|
|
// the next run waits from the writer's end, within the same holder cutoff.
|
|
await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date() }).where(eq(heartbeatRuns.id, writer.id));
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toMatchObject({
|
|
constructor: WorkspaceBusyDeferral, code: "workspace_busy", executionWorkspaceId: workspace.id,
|
|
holder: { runId: writer.id, issueId: parent.id, ended: true },
|
|
});
|
|
await db.update(heartbeatRuns).set({ finishedAt: new Date(Date.now() - WORKSPACE_BUSY_HOLDER_STALE_AFTER_MS - 1_000) })
|
|
.where(eq(heartbeatRuns.id, writer.id));
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toMatchObject({ status: 409 });
|
|
expect(await readAuthority()).toEqual(authority);
|
|
expect(acquire).not.toHaveBeenCalled();
|
|
|
|
const saved = { ...authority, writerRunId: null, dirty: false };
|
|
await db.update(executionWorkspaces).set({ metadata: { remoteWorkspace: saved } }).where(eq(executionWorkspaces.id, workspace.id));
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow("writer released; acquisition admitted");
|
|
expect(acquire).toHaveBeenCalledOnce();
|
|
expect(await readAuthority()).toEqual({ ...saved, epoch: 4, writerRunId: input.run.id });
|
|
const [bound] = await db.select().from(issues).where(eq(issues.id, issue.id));
|
|
expect(bound.executionWorkspaceId).toBe(workspace.id);
|
|
});
|
|
|
|
it("retains admitted ownership when a remote writer wait races with reassignment", async () => {
|
|
const { input, issue } = await inheritedRemoteFixture("isolated_workspace");
|
|
const [replacement] = await db.insert(agents).values({ companyId: issue.companyId,
|
|
name: "New task owner", adapterType: "process" }).returning();
|
|
input.controllerLease.assertOwned = async () => {
|
|
await db.update(issues).set({ assigneeAgentId: replacement.id }).where(eq(issues.id, issue.id));
|
|
};
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toMatchObject({
|
|
code: "workspace_busy", wasIssueAssignee: true,
|
|
});
|
|
const [current] = await db.select().from(issues).where(eq(issues.id, issue.id));
|
|
expect(current.assigneeAgentId).toBe(replacement.id);
|
|
});
|
|
|
|
it("rejects automatic Core services before remote reservation or acquisition", async () => {
|
|
const { input, envOrchestrator } = await fixture();
|
|
configureRemotePersistence();
|
|
input.agent.adapterType = "paperclip_runner";
|
|
input.environment.selectedEnvironmentForConfig = { ...input.environment.localEnvironment,
|
|
driver: "sandbox", config: { provider: "daytona" } };
|
|
input.environment.resolvedInstanceSettings.experimental.remoteWorkspaceStorageEnabled = true;
|
|
input.policy.useIsolatedTaskDirectory = true;
|
|
input.config.mergedConfig = { workspaceRuntime: { services: [{ name: "web", command: "node server.js" }] } };
|
|
const acquire = vi.spyOn(envOrchestrator, "acquireForRun");
|
|
const resolve = vi.spyOn(input.services, "resolveWorkspaceForRun");
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toMatchObject({
|
|
code: "workspace_validation_failed", resultJson: { reason: "remote_workspace_runtime_services_unsupported" },
|
|
});
|
|
expect(acquire).not.toHaveBeenCalled();
|
|
expect(resolve).not.toHaveBeenCalled();
|
|
expect(await db.select().from(executionWorkspaces)).toHaveLength(0);
|
|
});
|
|
|
|
it.each([false, true])("configured local folders retain host authority only on durable servers (ephemeral=%s)", async ephemeral => {
|
|
const { input, envOrchestrator } = await fixture();
|
|
input.agent.adapterType = "paperclip_runner";
|
|
input.environment.selectedEnvironmentForConfig = { ...input.environment.localEnvironment,
|
|
driver: "sandbox", config: { provider: "daytona" } };
|
|
input.environment.resolvedInstanceSettings.experimental.remoteWorkspaceStorageEnabled = true;
|
|
const acquire = vi.spyOn(envOrchestrator, "acquireForRun");
|
|
configureRemotePersistence();
|
|
const resolve = vi.spyOn(input.services, "resolveWorkspaceForRun").mockRejectedValue(new Error("local folder retained"));
|
|
process.env.PAPERCLIP_EPHEMERAL_SERVER = String(ephemeral);
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow(ephemeral
|
|
? "Volume-free servers require native Paperclip Runner execution with remote workspace storage." : "local folder retained");
|
|
if (ephemeral) expect(resolve).not.toHaveBeenCalled();
|
|
else expect(resolve).toHaveBeenCalledOnce();
|
|
expect(await db.select().from(executionWorkspaces)).toHaveLength(0);
|
|
expect(acquire).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it.each(["configured-folder", "explicit-task-directory", "enrolled-remote"] as const)(
|
|
"preserves effective configured cwd authority for %s", async scenario => {
|
|
const { input, source, issue, envOrchestrator } = await fixture();
|
|
configureRemotePersistence();
|
|
await writeFile(join(source, "unpublished.txt"), "configured folder work\n");
|
|
await db.update(issues).set({ projectId: null, projectWorkspaceId: null }).where(eq(issues.id, issue.id));
|
|
input.task.issueRef = { ...issue, projectId: null, projectWorkspaceId: null };
|
|
input.task.context = { issueId: issue.id };
|
|
input.config.mergedConfig = { cwd: source };
|
|
await db.update(agents).set({ adapterType: "paperclip_runner" }).where(eq(agents.id, input.agent.id));
|
|
input.agent.adapterType = "paperclip_runner";
|
|
const [savedEnvironment] = await db.insert(environments).values({ name: `Configured cwd ${randomUUID()}`,
|
|
driver: "sandbox", status: "active", config: { provider: "daytona" } }).returning();
|
|
input.environment.selectedEnvironmentForConfig = { ...input.environment.localEnvironment,
|
|
id: savedEnvironment.id, driver: "sandbox", config: { provider: "daytona" } };
|
|
input.environment.selectedEnvironmentId = savedEnvironment.id;
|
|
expect(input.environment.resolvedInstanceSettings.experimental.remoteWorkspaceStorageEnabled).toBe(true);
|
|
if (scenario === "explicit-task-directory") {
|
|
input.task.workspaceSelection = { version: 1, source: "explicit", selection: { kind: "task_directory" } };
|
|
await db.update(issues).set({ workspaceSelection: input.task.workspaceSelection }).where(eq(issues.id, issue.id));
|
|
input.policy.useIsolatedTaskDirectory = true;
|
|
}
|
|
let enrolledId: string | undefined;
|
|
if (scenario === "enrolled-remote") {
|
|
enrolledId = randomUUID();
|
|
const [workspace] = await db.insert(executionWorkspaces).values({ id: enrolledId, companyId: issue.companyId,
|
|
sourceIssueId: issue.id, name: "Existing remote workspace", cwd: `/workspaces/${enrolledId}`,
|
|
providerType: "cloud_sandbox", strategyType: "project_primary", mode: "shared_workspace", status: "active",
|
|
metadata: { remoteWorkspace: { schema: "paperclip.remote-workspace.v1", epoch: 1, snapshotId: null,
|
|
source: { kind: "empty" }, location: null, writerRunId: null, dirty: false } } }).returning();
|
|
await db.update(issues).set({ executionWorkspaceId: workspace.id }).where(eq(issues.id, issue.id));
|
|
input.task.issueRef.executionWorkspaceId = workspace.id;
|
|
input.reuse.existingExecutionWorkspace = toExecutionWorkspace(workspace);
|
|
input.reuse.reusableExistingExecutionWorkspace = toExecutionWorkspace(workspace);
|
|
input.reuse.requestedShouldReuseExisting = true;
|
|
input.environment.resolvedInstanceSettings.experimental.remoteWorkspaceStorageEnabled = false;
|
|
}
|
|
const resolve = vi.spyOn(input.services, "resolveWorkspaceForRun");
|
|
const acquire = vi.spyOn(envOrchestrator, "acquireForRun").mockRejectedValue(new Error("configured cwd acquisition boundary"));
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow("configured cwd acquisition boundary");
|
|
expect(acquire).toHaveBeenCalledOnce();
|
|
const [workspace] = await db.select().from(executionWorkspaces);
|
|
if (scenario === "configured-folder") {
|
|
expect(resolve).toHaveBeenCalledOnce();
|
|
expect(workspace).toMatchObject({ providerType: "local_fs", cwd: source });
|
|
expect(workspace.metadata?.remoteWorkspace).toBeUndefined();
|
|
} else {
|
|
expect(resolve).not.toHaveBeenCalled();
|
|
expect(workspace).toMatchObject({ providerType: "cloud_sandbox" });
|
|
expect(workspace.metadata?.remoteWorkspace).toMatchObject({ source: { kind: "empty" }, writerRunId: input.run.id });
|
|
if (enrolledId) expect(workspace.id).toBe(enrolledId);
|
|
}
|
|
expect(await readFile(join(source, "unpublished.txt"), "utf8")).toBe("configured folder work\n");
|
|
},
|
|
);
|
|
|
|
it.each(["path", "file-url"] as const)("resumes an imported %s Git source from saved authority after its host files disappear", async sourceKind => {
|
|
const { input, source, issue, envOrchestrator } = await fixture();
|
|
configureRemotePersistence();
|
|
process.env.PAPERCLIP_SECRETS_MASTER_KEY = "72".repeat(32);
|
|
const upstream = join(home, `import-upstream-${randomUUID()}`);
|
|
await execFile("git", ["clone", "--bare", source, upstream]);
|
|
const repoUrl = sourceKind === "path" ? upstream : pathToFileURL(upstream).href;
|
|
await execFile("git", ["-C", source, "remote", "add", "origin", repoUrl]);
|
|
await writeFile(join(source, "unsaved.txt"), "imported untracked work\n");
|
|
await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date() }).where(eq(heartbeatRuns.id, input.run.id));
|
|
const [original] = await db.insert(executionWorkspaces).values({ companyId: issue.companyId,
|
|
sourceIssueId: issue.id, projectId: issue.projectId, projectWorkspaceId: issue.projectWorkspaceId,
|
|
name: "Imported Git workspace", cwd: source, providerRef: source, repoUrl, baseRef: "main", branchName: "main",
|
|
providerType: "local_fs", strategyType: "project_primary", mode: "shared_workspace", status: "active" }).returning();
|
|
await db.update(issues).set({ executionWorkspaceId: original.id }).where(eq(issues.id, issue.id));
|
|
const storage = createStorageService(createLocalDiskStorageProvider(join(home, `import-objects-${randomUUID()}`)));
|
|
await expect(migrateRemoteWorkspace({ db, storage, companyId: issue.companyId, workspaceId: original.id, apply: true }))
|
|
.resolves.toMatchObject({ applied: true });
|
|
const [imported] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, original.id));
|
|
const saved = readRemoteWorkspaceAuthority(imported.metadata)!;
|
|
expect(saved).toMatchObject({ source: { kind: "git", repoUrl, ref: "main", branchName: "main" },
|
|
snapshotId: expect.any(String), location: null, dirty: false, writerRunId: null });
|
|
await rm(source, { recursive: true });
|
|
await rm(upstream, { recursive: true });
|
|
|
|
const [next] = await db.insert(heartbeatRuns).values({ companyId: issue.companyId, agentId: input.agent.id,
|
|
issueId: issue.id, scopeKind: "issue", status: "running", runtimeMode: "legacy" }).returning();
|
|
input.run = next;
|
|
await db.update(agents).set({ adapterType: "paperclip_runner" }).where(eq(agents.id, input.agent.id));
|
|
input.agent.adapterType = "paperclip_runner";
|
|
const [environment] = await db.insert(environments).values({ name: `Imported source ${randomUUID()}`,
|
|
driver: "sandbox", status: "active", config: { provider: "daytona" } }).returning();
|
|
input.environment.selectedEnvironmentForConfig = { ...input.environment.localEnvironment,
|
|
id: environment.id, driver: "sandbox", config: { provider: "daytona" } };
|
|
input.environment.selectedEnvironmentId = environment.id;
|
|
input.environment.resolvedInstanceSettings.experimental.remoteWorkspaceStorageEnabled = false;
|
|
input.task.issueRef = { ...issue, executionWorkspaceId: imported.id };
|
|
input.reuse.existingExecutionWorkspace = toExecutionWorkspace(imported);
|
|
input.reuse.reusableExistingExecutionWorkspace = toExecutionWorkspace(imported);
|
|
input.reuse.requestedShouldReuseExisting = true;
|
|
input.reuse.workspaceReuseRequest = resolveExecutionWorkspaceReuseRequestForIssue({ issueExecutionWorkspaceId: imported.id,
|
|
issueExecutionWorkspacePreference: null, existingExecutionWorkspaceStatus: imported.status });
|
|
const resolve = vi.spyOn(input.services, "resolveWorkspaceForRun");
|
|
const hostSeed = vi.spyOn(freshWorkspaceSeed, "materializeFreshWorkspaceSeed");
|
|
const acquire = vi.spyOn(envOrchestrator, "acquireForRun").mockImplementation(async args => {
|
|
expect(args.persistedExecutionWorkspace).toMatchObject({ id: imported.id, providerType: "cloud_sandbox" });
|
|
const [reserved] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, imported.id));
|
|
expect(readRemoteWorkspaceAuthority(reserved.metadata)).toMatchObject({
|
|
snapshotId: saved.snapshotId, source: saved.source,
|
|
});
|
|
throw new Error("imported saved authority reached remote acquisition");
|
|
});
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow("imported saved authority reached remote acquisition");
|
|
expect(acquire).toHaveBeenCalledOnce();
|
|
expect(resolve).not.toHaveBeenCalled();
|
|
expect(hostSeed).not.toHaveBeenCalled();
|
|
const [after] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, imported.id));
|
|
expect(readRemoteWorkspaceAuthority(after.metadata)).toMatchObject({ snapshotId: saved.snapshotId, source: saved.source });
|
|
await expect(access(source)).rejects.toThrow();
|
|
await expect(access(upstream)).rejects.toThrow();
|
|
});
|
|
|
|
it.each(["file:///tmp/repository", "/tmp/repository", "../repository", "C:\\work\\repository", "C:repository"])(
|
|
"auto-enrollment leaves local repository source %s on the host authority", async repoUrl => {
|
|
const { input, issue, envOrchestrator } = await fixture();
|
|
input.agent.adapterType = "paperclip_runner";
|
|
input.environment.selectedEnvironmentForConfig = { ...input.environment.localEnvironment,
|
|
driver: "sandbox", config: { provider: "daytona" } };
|
|
input.environment.resolvedInstanceSettings.experimental.remoteWorkspaceStorageEnabled = true;
|
|
await db.update(projectWorkspaces).set({ cwd: null, sourceType: "git", repoUrl })
|
|
.where(eq(projectWorkspaces.id, issue.projectWorkspaceId!));
|
|
const acquire = vi.spyOn(envOrchestrator, "acquireForRun");
|
|
configureRemotePersistence();
|
|
const resolve = vi.spyOn(input.services, "resolveWorkspaceForRun").mockRejectedValue(new Error("local repository retained"));
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow("local repository retained");
|
|
expect(resolve).toHaveBeenCalledOnce();
|
|
expect(acquire).not.toHaveBeenCalled();
|
|
expect(await db.select().from(executionWorkspaces)).toHaveLength(0);
|
|
},
|
|
);
|
|
|
|
it.each(["https://", "ftp://example.com/repo", "javascript://invalid", "https://example.com/repo\ninvalid"])(
|
|
"rejects malformed or unsupported repository source %s without a host fallback", async repoUrl => {
|
|
const { input, issue, envOrchestrator } = await fixture();
|
|
configureRemotePersistence();
|
|
input.agent.adapterType = "paperclip_runner";
|
|
input.environment.selectedEnvironmentForConfig = { ...input.environment.localEnvironment,
|
|
driver: "sandbox", config: { provider: "daytona" } };
|
|
await db.update(projectWorkspaces).set({ cwd: null, sourceType: "git", repoUrl })
|
|
.where(eq(projectWorkspaces.id, issue.projectWorkspaceId!));
|
|
const resolve = vi.spyOn(input.services, "resolveWorkspaceForRun");
|
|
const acquire = vi.spyOn(envOrchestrator, "acquireForRun");
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toMatchObject({
|
|
code: "workspace_validation_failed", resultJson: { reason: "remote_workspace_source_invalid" },
|
|
});
|
|
expect(resolve).not.toHaveBeenCalled(); expect(acquire).not.toHaveBeenCalled();
|
|
expect(await db.select().from(executionWorkspaces)).toHaveLength(0);
|
|
},
|
|
);
|
|
|
|
it("does not turn an unauthorized configured-source selection into a host fallback", async () => {
|
|
const { input, envOrchestrator } = await fixture();
|
|
configureRemotePersistence();
|
|
input.agent.adapterType = "paperclip_runner";
|
|
input.environment.selectedEnvironmentForConfig = { ...input.environment.localEnvironment,
|
|
driver: "sandbox", config: { provider: "daytona" } };
|
|
input.task.workspaceSelection = { version: 1, source: "explicit",
|
|
selection: { kind: "configured_source", projectWorkspaceId: randomUUID(), mode: "shared" } };
|
|
const resolve = vi.spyOn(input.services, "resolveWorkspaceForRun");
|
|
const acquire = vi.spyOn(envOrchestrator, "acquireForRun");
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toMatchObject({ status: 403 });
|
|
expect(resolve).not.toHaveBeenCalled(); expect(acquire).not.toHaveBeenCalled();
|
|
expect(await db.select().from(executionWorkspaces)).toHaveLength(0);
|
|
});
|
|
|
|
it.each(["source", "effective-policy"] as const)("keeps plugin repository ownership via %s out of automatic Core enrollment", async ownership => {
|
|
const { input, issue, envOrchestrator } = await fixture();
|
|
configureRemotePersistence();
|
|
input.agent.adapterType = "paperclip_runner";
|
|
input.environment.selectedEnvironmentForConfig = { ...input.environment.localEnvironment,
|
|
driver: "sandbox", config: { provider: "daytona" } };
|
|
if (ownership === "effective-policy") input.policy.projectExecutionWorkspacePolicy = {
|
|
enabled: true, storageProviderPluginId: randomUUID(), defaultMode: "isolated_workspace",
|
|
};
|
|
await db.update(projectWorkspaces).set({ cwd: null, sourceType: ownership === "source" ? "remote_managed" : "git", repoUrl: "https://github.com/example/managed.git" })
|
|
.where(eq(projectWorkspaces.id, issue.projectWorkspaceId!));
|
|
const resolve = vi.spyOn(input.services, "resolveWorkspaceForRun").mockRejectedValue(new Error("managed ownership retained"));
|
|
const acquire = vi.spyOn(envOrchestrator, "acquireForRun");
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow("managed ownership retained");
|
|
expect(resolve).toHaveBeenCalledOnce(); expect(acquire).not.toHaveBeenCalled();
|
|
expect(await db.select().from(executionWorkspaces)).toHaveLength(0);
|
|
});
|
|
|
|
it.each(["shared_workspace", "agent_default"] as const)("rejects concurrent shared writers resolved from %s before resolving a remote source", async requestedMode => {
|
|
const { input, envOrchestrator } = await fixture();
|
|
configureRemotePersistence();
|
|
input.policy.useIsolatedTaskDirectory = true;
|
|
input.agent.adapterType = "paperclip_runner";
|
|
input.environment.selectedEnvironmentForConfig = { ...input.environment.localEnvironment,
|
|
driver: "sandbox", config: { provider: "daytona" } };
|
|
input.environment.resolvedInstanceSettings.experimental.remoteWorkspaceStorageEnabled = true;
|
|
input.policy.requestedExecutionWorkspaceMode = requestedMode;
|
|
input.policy.issueExecutionWorkspaceSettings = { sharedWorkspaceConcurrency: "allow" };
|
|
const acquire = vi.spyOn(envOrchestrator, "acquireForRun");
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow("one active writer");
|
|
expect(acquire).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it.each([false, true])("holds execution while the selected workspace is migrating (remote: %s)", async remote => {
|
|
const { input, source, issue, envOrchestrator } = await fixture();
|
|
const [workspace] = await db.insert(executionWorkspaces).values({ companyId: issue.companyId,
|
|
sourceIssueId: issue.id, projectId: issue.projectId, projectWorkspaceId: issue.projectWorkspaceId,
|
|
name: "Migrating workspace", cwd: source, providerType: "local_fs", strategyType: "project_primary",
|
|
mode: "shared_workspace", status: "active", metadata: { remoteWorkspaceMigration: { state: "running" } } }).returning();
|
|
await db.update(issues).set({ executionWorkspaceId: workspace.id }).where(eq(issues.id, issue.id));
|
|
// The admission snapshot may predate the operator starting the import.
|
|
input.reuse.existingExecutionWorkspace = toExecutionWorkspace({ ...workspace, metadata: {} });
|
|
if (remote) {
|
|
input.agent.adapterType = "paperclip_runner";
|
|
input.environment.selectedEnvironmentForConfig = { ...input.environment.localEnvironment,
|
|
driver: "sandbox", config: { provider: "daytona" } };
|
|
input.environment.resolvedInstanceSettings.experimental.remoteWorkspaceStorageEnabled = true;
|
|
}
|
|
const resolve = vi.spyOn(input.services, "resolveWorkspaceForRun");
|
|
const acquire = vi.spyOn(envOrchestrator, "acquireForRun");
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toMatchObject({
|
|
code: "workspace_validation_failed", resultJson: { reason: "remote_workspace_migration_in_progress" },
|
|
});
|
|
expect(resolve).not.toHaveBeenCalled();
|
|
expect(acquire).not.toHaveBeenCalled();
|
|
expect(await readFile(join(source, "README.md"), "utf8")).toBe("source content\n");
|
|
});
|
|
|
|
it("reserves private remote task files for an organizational project without a configured source", async () => {
|
|
const { input, issue, source, envOrchestrator } = await fixture();
|
|
await db.update(issues).set({ projectWorkspaceId: null }).where(eq(issues.id, issue.id));
|
|
await db.delete(projectWorkspaces).where(eq(projectWorkspaces.projectId, issue.projectId!));
|
|
input.task.issueRef = await readIssue(issue.id);
|
|
const project = (await projectService(db).getById(issue.projectId!))!;
|
|
expect(project.workspaces).toEqual([]);
|
|
expect(project.codebase).toMatchObject({ workspaceId: null, repoUrl: null, localFolder: null, origin: "managed_checkout" });
|
|
await mkdir(project.codebase.managedFolder, { recursive: true });
|
|
const sentinel = join(project.codebase.managedFolder, "unselected-project-files.txt");
|
|
await writeFile(sentinel, "Unselected project files\n");
|
|
configureRemotePersistence();
|
|
input.agent.adapterType = "paperclip_runner";
|
|
input.policy.requestedExecutionWorkspaceMode = "agent_default";
|
|
input.environment.selectedEnvironmentForConfig = { ...input.environment.localEnvironment,
|
|
driver: "sandbox", config: { provider: "daytona" } };
|
|
const resolveHost = vi.spyOn(input.services, "resolveWorkspaceForRun");
|
|
const acquire = vi.spyOn(envOrchestrator, "acquireForRun").mockImplementation(async args => {
|
|
expect(args.persistedExecutionWorkspace).toMatchObject({ companyId: issue.companyId, sourceIssueId: issue.id,
|
|
projectId: null, projectWorkspaceId: null, providerType: "cloud_sandbox", repoUrl: null,
|
|
metadata: { remoteWorkspace: { source: { kind: "empty" }, writerRunId: input.run.id } } });
|
|
throw new Error("private task workspace reached acquisition");
|
|
});
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow("private task workspace reached acquisition");
|
|
expect(acquire).toHaveBeenCalledOnce();
|
|
expect(resolveHost).not.toHaveBeenCalled();
|
|
const [workspace] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.sourceIssueId, issue.id));
|
|
expect(workspace).toMatchObject({ companyId: issue.companyId, sourceIssueId: issue.id,
|
|
projectId: null, projectWorkspaceId: null, providerType: "cloud_sandbox", repoUrl: null });
|
|
expect(workspace.cwd).toBe(`/workspaces/${workspace.id}`);
|
|
expect((await readIssue(issue.id))).toMatchObject({ projectId: issue.projectId, executionWorkspaceId: workspace.id });
|
|
await expect(access(workspace.cwd!)).rejects.toThrow();
|
|
expect(await readFile(sentinel, "utf8")).toBe("Unselected project files\n");
|
|
expect(await readFile(join(source, "README.md"), "utf8")).toBe("source content\n");
|
|
});
|
|
|
|
it("reserves editable task copies for referenced projects before acquiring the remote environment", async () => {
|
|
const { input, source, issue, envOrchestrator } = await fixture();
|
|
const previous = new Map(["PAPERCLIP_MULTI_PROJECT_WORKSPACE_SYNC", "PAPERCLIP_MULTI_PROJECT_WORKSPACE_SYNC_REMOTE"]
|
|
.map(key => [key, process.env[key]]));
|
|
vi.stubEnv("PAPERCLIP_MULTI_PROJECT_WORKSPACE_SYNC", "true");
|
|
vi.stubEnv("PAPERCLIP_MULTI_PROJECT_WORKSPACE_SYNC_REMOTE", "true");
|
|
configureRemotePersistence();
|
|
input.agent.adapterType = "paperclip_runner";
|
|
input.environment.selectedEnvironmentForConfig = { ...input.environment.localEnvironment,
|
|
driver: "sandbox", config: { provider: "daytona" } };
|
|
input.environment.resolvedInstanceSettings.experimental.remoteWorkspaceStorageEnabled = true;
|
|
await db.update(projectWorkspaces).set({ cwd: null, sourceType: "git", repoUrl: "https://github.com/example/remote.git" })
|
|
.where(eq(projectWorkspaces.id, issue.projectWorkspaceId!));
|
|
const [reference] = await db.insert(projects).values({ companyId: issue.companyId, name: "Referenced project" }).returning();
|
|
const [repository] = await db.insert(projectWorkspaces).values({ companyId: issue.companyId, projectId: reference.id,
|
|
name: "Referenced repository", sourceType: "git_repo", repoUrl: "https://github.com/example/reference.git" }).returning();
|
|
await db.update(issues).set({ description: `[project](${buildProjectMentionHref(reference.id)})` }).where(eq(issues.id, issue.id));
|
|
const acquire = vi.spyOn(envOrchestrator, "acquireForRun").mockImplementation(async args => {
|
|
expect(args.persistedExecutionWorkspace).toMatchObject({ companyId: issue.companyId, sourceIssueId: issue.id,
|
|
metadata: { remoteWorkspace: { source: { kind: "git", repoUrl: "https://github.com/example/remote.git", repositories: [
|
|
{ id: repository.id, projectId: reference.id, projectWorkspaceId: repository.id,
|
|
repoUrl: "https://github.com/example/reference.git" },
|
|
] } } } });
|
|
const [stored] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, args.persistedExecutionWorkspace!.id));
|
|
expect(readRemoteWorkspaceAuthority(stored.metadata)?.source.repositories?.[0]?.readOnly).not.toBe(true);
|
|
throw new Error("editable project copies reached acquisition");
|
|
});
|
|
try {
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow("editable project copies reached acquisition");
|
|
expect(acquire).toHaveBeenCalledOnce();
|
|
expect(await db.select().from(executionWorkspaces)).toHaveLength(1);
|
|
expect(await readdir(source)).not.toContain(".paperclip-repositories");
|
|
} finally {
|
|
for (const [key, value] of previous) {
|
|
if (value === undefined) delete process.env[key]; else process.env[key] = value;
|
|
}
|
|
}
|
|
});
|
|
|
|
it("fences recovered native preparation when its controller claim changes or expires", async () => {
|
|
const { input, issue, envOrchestrator } = await fixture();
|
|
const controller = await currentNativeControllerIdentity();
|
|
const leaseOwner = `recovery:${randomUUID()}`;
|
|
await db.update(heartbeatRuns).set({ runtimeMode: "native", nativeIssueId: issue.id }).where(eq(heartbeatRuns.id, input.run.id));
|
|
input.run.runtimeMode = "native";
|
|
configureRemotePersistence();
|
|
input.policy.useIsolatedTaskDirectory = true;
|
|
input.agent.adapterType = "paperclip_runner";
|
|
input.environment.selectedEnvironmentForConfig = { ...input.environment.localEnvironment,
|
|
driver: "sandbox", config: { provider: "daytona" } };
|
|
input.environment.resolvedInstanceSettings.experimental.remoteWorkspaceStorageEnabled = true;
|
|
input.reuse.runOptions = { nativeLeaseOwner: leaseOwner };
|
|
await db.insert(nativeRunFinalizations).values({ runId: input.run.id, companyId: issue.companyId,
|
|
issueId: issue.id, phase: "preparing", attempt: 2, controllerGeneration: 3, leaseOwner,
|
|
leaseExpiresAt: new Date(Date.now() + 60_000), controllerBootId: controller.bootId,
|
|
controllerPid: controller.pid, controllerProcessStartedAt: controller.processStartedAt });
|
|
const authority = { db, companyId: issue.companyId, runId: input.run.id, leaseOwner,
|
|
controllerGeneration: 3, providerAttempt: 2 };
|
|
await expect(assertNativeSessionPreparationAuthority(authority)).resolves.toBeUndefined();
|
|
await expect(assertNativeSessionPreparationAuthority({ ...authority, controllerGeneration: 2 }))
|
|
.rejects.toThrow("native_session_lease_lost");
|
|
await db.update(nativeRunFinalizations).set({ leaseOwner: "successor-controller" })
|
|
.where(eq(nativeRunFinalizations.runId, input.run.id));
|
|
const resolve = vi.spyOn(input.services, "resolveWorkspaceForRun");
|
|
const acquire = vi.spyOn(envOrchestrator, "acquireForRun");
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow("native_session_lease_lost");
|
|
expect(resolve).not.toHaveBeenCalled();
|
|
expect(acquire).not.toHaveBeenCalled();
|
|
expect(await db.select().from(executionWorkspaces)).toHaveLength(0);
|
|
await db.update(nativeRunFinalizations).set({ leaseOwner, leaseExpiresAt: new Date(Date.now() - 1_000) })
|
|
.where(eq(nativeRunFinalizations.runId, input.run.id));
|
|
await expect(assertNativeSessionPreparationAuthority(authority)).rejects.toThrow("native_session_lease_lost");
|
|
await db.update(nativeRunFinalizations).set({ leaseExpiresAt: new Date(Date.now() + 60_000) })
|
|
.where(eq(nativeRunFinalizations.runId, input.run.id));
|
|
await db.update(heartbeatRuns).set({ status: "cancelled" }).where(eq(heartbeatRuns.id, input.run.id));
|
|
await expect(assertNativeSessionPreparationAuthority(authority)).rejects.toThrow("native_session_lease_lost");
|
|
});
|
|
|
|
it.each([
|
|
{ fromDefault: false, mode: "isolated_workspace" as const, fresh: false },
|
|
{ fromDefault: false, mode: "isolated_workspace" as const, fresh: true },
|
|
{ fromDefault: true, mode: "isolated_workspace" as const, fresh: false },
|
|
{ fromDefault: true, mode: "operator_branch" as const, fresh: false },
|
|
])("prepares managed storage before acquisition (fromDefault: $fromDefault, mode: $mode, fresh: $fresh) without touching a Core checkout", async ({ fromDefault, mode: defaultMode, fresh }) => {
|
|
const { input, source, issue } = await fixture(true);
|
|
const pluginKey = `test.managed-storage-${fromDefault}-${defaultMode}-${fresh}`;
|
|
const [plugin] = await db.insert(plugins).values({ pluginKey, packageName: "test-storage", version: "1.0.0", apiVersion: 1, status: "ready", categories: ["workspace"],
|
|
manifestJson: { id: pluginKey, apiVersion: 1, version: "1.0.0", displayName: "Storage", description: "Storage", author: "Test", categories: ["workspace"],
|
|
workspaceProvider: { freshWorkspaceVersion: 1, modes: ["isolated_workspace", "operator_branch"], strategies: ["git_worktree"], ...(fromDefault ? { defaultForProjects: { mode: defaultMode, strategy: "git_worktree" as const } } : {}) }, capabilities: ["workspaces.manage", "events.subscribe"], resourceLifecycle: ["project"], entrypoints: { worker: "./worker.js" } } }).returning();
|
|
const savedPolicy = fromDefault ? { enabled: true, storageProviderPluginId: plugin.id } : { enabled: true, storageProviderPluginId: plugin.id, defaultMode: "isolated_workspace" as const,
|
|
workspaceStrategy: { type: "git_worktree" as const, baseRef: "main", branchTemplate: "test-{{issue.identifier}}" } };
|
|
await db.update(projects).set({ executionWorkspacePolicy: savedPolicy }).where(eq(projects.id, issue.projectId!));
|
|
{
|
|
const policy = await effectiveProjectWorkspacePolicy(db, input.agent.companyId, savedPolicy, true);
|
|
const mode = resolveExecutionWorkspaceMode({ projectPolicy: policy, issueSettings: null, legacyUseProjectWorkspace: null });
|
|
input.policy.projectExecutionWorkspacePolicy = policy;
|
|
input.policy.requestedExecutionWorkspaceMode = mode;
|
|
input.policy.effectiveExecutionWorkspaceMode = mode;
|
|
input.config.mergedConfig = buildExecutionWorkspaceAdapterConfig({ agentConfig: input.config.mergedConfig, projectPolicy: policy, issueSettings: null, mode, legacyUseProjectWorkspace: null });
|
|
}
|
|
await db.update(projectWorkspaces).set({ cwd: null, sourceType: "remote_managed", repoUrl: "https://github.com/example/repo.git" }).where(eq(projectWorkspaces.id, issue.projectWorkspaceId!));
|
|
await rm(source, { recursive: true });
|
|
if (fresh) input.task.freshWorkspace = { mode: "fresh_isolated_workspace", retryIntentId: input.run.id, sourceRunId: input.run.id, retainedSourceIds: [] };
|
|
const sequence: string[] = [];
|
|
input.services.workspaceWorkers = { getWorker: () => ({ supportedMethods: ["workspaceOperation"] }), call: async (_: string, __: string, params: any) => {
|
|
sequence.push("prepare");
|
|
return { kind: "prepared", binding: { pluginId: plugin.id, projectId: issue.projectId, executionWorkspaceId: params.executionWorkspaceId,
|
|
projectWorkspaceId: issue.projectWorkspaceId, sourceRevision: params.operation.source.revision, workspaceRef: "provider:checkout" },
|
|
branchName: params.operation.branchName, baseSha: "a".repeat(40), created: true, branchCreatedByRuntime: true };
|
|
} } as any;
|
|
const acquire = input.services.envOrchestrator.acquireForRun;
|
|
vi.spyOn(input.services.envOrchestrator, "acquireForRun").mockImplementation(async args => {
|
|
sequence.push("acquire"); expect(args.persistedExecutionWorkspace).toMatchObject({ providerType: "plugin", cwd: null });
|
|
return acquire(args);
|
|
});
|
|
vi.spyOn(input.services.envOrchestrator, "realizeForRun").mockImplementation(async args => {
|
|
expect(args.executionWorkspace.providerBinding?.workspaceRef).toBe("provider:checkout");
|
|
throw new Error("reached execution handoff");
|
|
});
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow("reached execution handoff");
|
|
expect(sequence).toEqual(["prepare", "acquire"]);
|
|
await expect(access(source)).rejects.toThrow();
|
|
const [task] = await db.select().from(issues).where(eq(issues.id, issue.id));
|
|
const [workspace] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, task.executionWorkspaceId!));
|
|
expect(workspace).toMatchObject({ providerType: "plugin", cwd: null, mode: defaultMode, strategyType: "git_worktree", branchName: fresh ? null : fromDefault ? "WSP-1-prepare-this-workspace" : "test-WSP-1" });
|
|
});
|
|
async function installDefault() {
|
|
const [plugin] = await db.insert(plugins).values({ pluginKey: "test.default-storage", packageName: "test-storage", version: "1.0.0", status: "ready",
|
|
manifestJson: { id: "test.default-storage", apiVersion: 1, version: "1.0.0", displayName: "Default storage", description: "Storage", author: "Test", categories: ["workspace"],
|
|
workspaceProvider: { modes: ["isolated_workspace"], strategies: ["git_worktree"], defaultForProjects: { mode: "isolated_workspace", strategy: "git_worktree" } },
|
|
capabilities: ["workspaces.manage", "events.subscribe"], resourceLifecycle: ["project"], entrypoints: { worker: "./worker.js" } } }).returning();
|
|
return plugin;
|
|
}
|
|
it("keeps source-free project tasks outside installed provider preparation", async () => {
|
|
const { input, issue } = await fixture();
|
|
await installDefault();
|
|
await db.update(issues).set({ projectWorkspaceId: null }).where(eq(issues.id, issue.id));
|
|
await db.delete(projectWorkspaces).where(eq(projectWorkspaces.id, issue.projectWorkspaceId!));
|
|
input.task.issueRef = { ...issue, projectWorkspaceId: null };
|
|
input.policy.projectExecutionWorkspacePolicy = await effectiveProjectWorkspacePolicy(db, input.agent.companyId, null, false);
|
|
const call = vi.fn();
|
|
input.services.workspaceWorkers = { getWorker: () => ({ supportedMethods: ["workspaceOperation"] }), call } as any;
|
|
const prepared = await prepareHeartbeatWorkspace(db, input);
|
|
expect(prepared.executionWorkspace.cwd).toBeTruthy();
|
|
expect(prepared.persistedExecutionWorkspace?.providerType).not.toBe("plugin");
|
|
expect(call).not.toHaveBeenCalled();
|
|
});
|
|
it("pins a resolved Core choice through source selection when a default appears during setup", async () => {
|
|
const { input, source } = await fixture();
|
|
expect(input.policy.projectExecutionWorkspacePolicy).toBeNull();
|
|
await installDefault();
|
|
const call = vi.fn();
|
|
input.services.workspaceWorkers = { getWorker: () => ({ supportedMethods: ["workspaceOperation"] }), call } as any;
|
|
const prepared = await prepareHeartbeatWorkspace(db, input);
|
|
expect(prepared.executionWorkspace.cwd).toBe(source);
|
|
expect(prepared.persistedExecutionWorkspace?.providerType).not.toBe("plugin");
|
|
expect(call).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it.each([
|
|
{ project: false, savedSource: false },
|
|
{ project: true, savedSource: true },
|
|
{ project: true, savedSource: false },
|
|
])("keeps a fresh task reservation stable through repeated preparation (project: $project, saved source: $savedSource)", async ({ project, savedSource }) => {
|
|
const { input, issue, envOrchestrator } = await fixture();
|
|
if (!project) await db.update(issues).set({ projectId: null, projectWorkspaceId: null }).where(eq(issues.id, issue.id));
|
|
else if (!savedSource) {
|
|
await db.update(issues).set({ projectWorkspaceId: null }).where(eq(issues.id, issue.id));
|
|
input.task.issueRef = { ...input.task.issueRef!, projectWorkspaceId: null };
|
|
}
|
|
const pluginKey = "test.fresh-task";
|
|
await db.insert(plugins).values({ pluginKey, packageName: "test-tasks", version: "1.0.0", apiVersion: 1,
|
|
status: "ready", categories: ["workspace"], manifestJson: { id: pluginKey, apiVersion: 1, version: "1.0.0",
|
|
displayName: "Tasks", description: "Tasks", author: "Test", categories: ["workspace"],
|
|
capabilities: ["environment.drivers.register"], entrypoints: { worker: "./worker.js" },
|
|
environmentDrivers: [{ driverKey: "tasks", displayName: "Task VM", configSchema: { type: "object" }, supportsTasks: true, freshWorkspaceVersion: 1 }] } });
|
|
input.agent.adapterType = "paperclip_runner";
|
|
if (!project) {
|
|
input.task.issueRef = { ...input.task.issueRef!, projectId: null, projectWorkspaceId: null };
|
|
input.task.context = { issueId: issue.id };
|
|
}
|
|
input.task.freshWorkspace = { mode: "fresh_isolated_workspace", retryIntentId: input.run.id, sourceRunId: input.run.id, retainedSourceIds: [] };
|
|
input.policy.requestedExecutionWorkspaceMode = input.policy.effectiveExecutionWorkspaceMode = "isolated_workspace";
|
|
input.config.mergedConfig = { workspaceStrategy: { type: "project_primary" } };
|
|
input.environment.selectedEnvironmentForConfig = { ...input.environment.localEnvironment, driver: "plugin",
|
|
config: { pluginKey, driverKey: "tasks" } };
|
|
const stopped = new Error("Reached projectless environment handoff");
|
|
const acquire = vi.spyOn(envOrchestrator, "acquireForRun").mockRejectedValue(stopped);
|
|
const read = vi.spyOn(input.services.executionWorkspacesSvc, "getStoredById");
|
|
for (let attempt = 0; attempt < 2; attempt++) await expect(prepareHeartbeatWorkspace(db, input)).rejects.toBe(stopped);
|
|
expect(read).toHaveBeenCalledWith(input.agent.companyId, input.run.id);
|
|
expect(acquire).toHaveBeenCalledTimes(2);
|
|
for (const [args] of acquire.mock.calls) expect(args.persistedExecutionWorkspace).toMatchObject({ id: input.run.id,
|
|
providerType: "adapter_managed", projectId: project ? issue.projectId : null,
|
|
projectWorkspaceId: project ? issue.projectWorkspaceId : null,
|
|
metadata: { runnerTaskFreshWorkspace: { attemptId: input.run.id } } });
|
|
const rows = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.companyId, input.agent.companyId));
|
|
expect(rows).toHaveLength(1);
|
|
const [task] = await db.select().from(issues).where(eq(issues.id, issue.id));
|
|
expect(task.projectWorkspaceId).toBe(project ? issue.projectWorkspaceId : null);
|
|
});
|
|
it.each(["implicit", "task", "agent"] as const)("keeps provider directory selection after a fresh task succeeds (strategy: %s)", async strategySource => {
|
|
const { input, issue, envOrchestrator } = await fixture();
|
|
await db.update(issues).set({ projectId: null, projectWorkspaceId: null }).where(eq(issues.id, issue.id));
|
|
input.task.issueRef = { ...issue, projectId: null, projectWorkspaceId: null };
|
|
input.task.context = { issueId: issue.id };
|
|
const pluginKey = "test.fresh-followup";
|
|
await db.insert(plugins).values({ pluginKey, packageName: "test-tasks", version: "1.0.0", apiVersion: 1,
|
|
status: "ready", categories: ["workspace"], manifestJson: { id: pluginKey, apiVersion: 1, version: "1.0.0",
|
|
displayName: "Tasks", description: "Tasks", author: "Test", categories: ["workspace"],
|
|
capabilities: ["environment.drivers.register"], entrypoints: { worker: "./worker.js" },
|
|
environmentDrivers: [{ driverKey: "tasks", displayName: "Task VM", configSchema: { type: "object" }, supportsTasks: true, freshWorkspaceVersion: 1 }] } });
|
|
const environment = await environmentService(db).create({ name: `Task computer ${issue.id}`, driver: "plugin", status: "active", envVars: {}, config: { pluginKey, driverKey: "tasks" } });
|
|
input.agent = { ...input.agent, adapterType: "paperclip_runner", defaultEnvironmentId: environment.id };
|
|
input.environment.selectedEnvironmentForConfig = environment;
|
|
input.policy.requestedExecutionWorkspaceMode = input.policy.effectiveExecutionWorkspaceMode = "isolated_workspace";
|
|
input.task.freshWorkspace = { mode: "fresh_isolated_workspace", retryIntentId: input.run.id, sourceRunId: input.run.id, retainedSourceIds: [] };
|
|
const stopped = new Error("Provider directory prepared");
|
|
vi.spyOn(envOrchestrator, "acquireForRun").mockRejectedValue(stopped);
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toBe(stopped);
|
|
const [task] = await db.select().from(issues).where(eq(issues.id, issue.id));
|
|
expect(task.executionWorkspaceSettings).toEqual({ mode: "isolated_workspace" });
|
|
const settings = { mode: "isolated_workspace" as const, ...(strategySource === "task" ? { workspaceStrategy: { type: "git_worktree" as const } } : {}) };
|
|
const agentConfig = strategySource === "agent" ? { workspaceStrategy: { type: "git_worktree" as const } } : {};
|
|
const selection = await selectHeartbeatEnvironment(db, {
|
|
run: input.run, agent: input.agent,
|
|
task: { issueRef: task, issueContext: task, issueId: issue.id, context: input.task.context,
|
|
executionProjectId: null, workspaceProjectContext: null, boundSourceWorkspace: null,
|
|
hasSelectedWorkspaceSource: false, taskWorkspaceIntent: null, responsibleUserId: null },
|
|
policy: { ...input.policy, issueExecutionWorkspaceSettings: settings },
|
|
config: { config: agentConfig, sourceWorkspaceConfig: buildExecutionWorkspaceAdapterConfig({ agentConfig, projectPolicy: null,
|
|
issueSettings: settings, mode: "isolated_workspace", legacyUseProjectWorkspace: null }), issueAssigneeOverrides: null },
|
|
services: { environmentsSvc: environmentService(db), executionWorkspacesSvc: executionWorkspaceService(db),
|
|
instanceSettings: instanceSettingsService(db), findSharedWorkspaceHolder: async () => null, findExecutionWorkspaceHolder: async () => null },
|
|
});
|
|
expect(selection.useIsolatedTaskDirectory).toBe(strategySource === "implicit");
|
|
expect(selection.reusableExistingExecutionWorkspace?.id).toBe(task.executionWorkspaceId);
|
|
});
|
|
|
|
const readIssue = (id: string) => db.select().from(issues).where(eq(issues.id, id)).then(rows => rows[0]!);
|
|
const readRun = (id: string) => db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, id)).then(rows => rows[0]!);
|
|
const leases = (id: string) => db.select().from(environmentLeases).where(eq(environmentLeases.heartbeatRunId, id));
|
|
|
|
it("keeps plugin-owned storage untouched when a fresh attempt provider is unavailable", async () => {
|
|
const { input, source, issue } = await fixture(true);
|
|
await db.update(projects).set({ executionWorkspacePolicy: {
|
|
enabled: true, storageProviderPluginId: "bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb",
|
|
} }).where(eq(projects.id, issue.projectId!));
|
|
input.policy.projectExecutionWorkspacePolicy = { enabled: true, storageProviderPluginId: "bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb" };
|
|
input.task.freshWorkspace = { mode: "fresh_isolated_workspace", retryIntentId: input.run.id,
|
|
sourceRunId: input.run.id, retainedSourceIds: [] };
|
|
const call = vi.fn();
|
|
input.services.workspaceWorkers = { getWorker: () => ({ supportedMethods: ["workspaceOperation"] }), call } as any;
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow("workspace storage plugin is unavailable");
|
|
expect(call).not.toHaveBeenCalled();
|
|
expect(await leases(input.run.id)).toHaveLength(0);
|
|
expect((await readIssue(issue.id)).executionWorkspaceId).toBeNull();
|
|
expect(await readFile(join(source, "README.md"), "utf8")).toBe("source content\n");
|
|
});
|
|
|
|
it("a fresh retry creates independent files and Git storage instead of reopening a configured task branch", async () => {
|
|
const { input, source, issue } = await fixture(true);
|
|
const original = await prepareHeartbeatWorkspace(db, input);
|
|
await writeFile(join(original.executionWorkspace.cwd, "unpublished.txt"), "keep the old work\n");
|
|
const [nextRun] = await db.insert(heartbeatRuns).values({ companyId: input.run.companyId,
|
|
agentId: input.agent.id, issueId: issue.id, scopeKind: "issue", status: "running", runtimeMode: "legacy" }).returning();
|
|
input.task.freshWorkspace = { mode: "fresh_isolated_workspace", retryIntentId: nextRun.id,
|
|
sourceRunId: input.run.id, retainedSourceIds: [] };
|
|
input.run = nextRun;
|
|
input.task.issueRef = { ...issue, executionWorkspaceId: null, executionWorkspacePreference: "create_new" };
|
|
const replacement = await prepareHeartbeatWorkspace(db, input);
|
|
expect(replacement.persistedExecutionWorkspace!.id).not.toBe(original.persistedExecutionWorkspace!.id);
|
|
expect(replacement.executionWorkspace.cwd).not.toBe(original.executionWorkspace.cwd);
|
|
expect(replacement.executionWorkspace.cwd).not.toBe(source);
|
|
expect(await readFile(join(replacement.executionWorkspace.cwd, "README.md"), "utf8")).toBe("source content\n");
|
|
expect(await readFile(join(original.executionWorkspace.cwd, "unpublished.txt"), "utf8")).toBe("keep the old work\n");
|
|
const common = await execFile("git", ["-C", replacement.executionWorkspace.cwd, "rev-parse", "--path-format=absolute", "--git-common-dir"]);
|
|
expect(common.stdout.trim()).toBe(join(replacement.executionWorkspace.cwd, ".git"));
|
|
await expect(access(join(replacement.executionWorkspace.cwd, "unpublished.txt"))).rejects.toThrow();
|
|
expect((await readIssue(issue.id)).executionWorkspaceId).toBe(replacement.persistedExecutionWorkspace!.id);
|
|
}, 30_000);
|
|
|
|
it.each(["historical_first", "recovery_first"] as const)(
|
|
"prepares the admitted fresh directory with sibling holds in either order: %s", async order => {
|
|
const { input, source, issue } = await fixture(true);
|
|
const original = await prepareHeartbeatWorkspace(db, input);
|
|
await writeFile(join(original.executionWorkspace.cwd, "unpublished.txt"), "keep the old work\n");
|
|
const sourceRunId = input.run.id, retryIntentId = randomUUID();
|
|
const [nextRun] = await db.insert(heartbeatRuns).values({ companyId: input.run.companyId,
|
|
agentId: input.agent.id, issueId: issue.id, scopeKind: "issue", status: "running", runtimeMode: "legacy",
|
|
contextSnapshot: { issueId: issue.id, taskRetryRequestId: retryIntentId } }).returning();
|
|
await db.insert(agentWakeupRequests).values({ id: retryIntentId, companyId: input.run.companyId,
|
|
agentId: input.agent.id, reason: "task_retry", source: "on_demand", status: "coalesced", runId: nextRun.id,
|
|
payload: { issueId: issue.id } });
|
|
const readiness = { mode: "fresh_isolated_workspace" as const, retryIntentId, sourceRunId, retainedSourceIds: [] };
|
|
const evidence = { runId: sourceRunId, explicitUserContinuation: { runId: nextRun.id, readiness } };
|
|
const hold = { companyId: input.run.companyId, sourceIssueId: issue.id, kind: "active_run_watchdog",
|
|
status: "resolved", ownerType: "system", nextAction: "Preserve the original source outcome." };
|
|
const historical = { ...hold, cause: "native_session_retry_exhausted", fingerprint: randomUUID(), evidence };
|
|
const recovery = { ...hold, cause: "native_continuation_requires_reconciliation", fingerprint: randomUUID(),
|
|
evidence: { ...evidence, systemRecovery: { ...createSystemRecovery(sourceRunId, retryIntentId),
|
|
state: "exhausted", freshAttemptRunId: nextRun.id } } };
|
|
await db.insert(issueRecoveryActions).values(order === "historical_first" ? [historical, recovery] : [recovery, historical]);
|
|
// Use the same committed-receipt lookup as run preparation, not a
|
|
// hand-built freshWorkspace input. The historical hold has no generation.
|
|
input.task.freshWorkspace = await readFreshWorkspaceReadiness(db, nextRun);
|
|
expect(input.task.freshWorkspace).toEqual(readiness);
|
|
input.run = nextRun;
|
|
input.task.issueRef = { ...issue, executionWorkspaceId: null, executionWorkspacePreference: "create_new" };
|
|
const replacement = await prepareHeartbeatWorkspace(db, input);
|
|
expect(replacement.executionWorkspace.cwd).not.toBe(original.executionWorkspace.cwd);
|
|
expect(replacement.executionWorkspace.cwd).not.toBe(source);
|
|
expect(await readFile(join(replacement.executionWorkspace.cwd, "README.md"), "utf8")).toBe("source content\n");
|
|
await expect(access(join(replacement.executionWorkspace.cwd, "unpublished.txt"))).rejects.toThrow();
|
|
expect(await readFile(join(original.executionWorkspace.cwd, "unpublished.txt"), "utf8")).toBe("keep the old work\n");
|
|
const common = await execFile("git", ["-C", replacement.executionWorkspace.cwd, "rev-parse", "--path-format=absolute", "--git-common-dir"]);
|
|
expect(common.stdout.trim()).toBe(join(replacement.executionWorkspace.cwd, ".git"));
|
|
expect((await readIssue(issue.id)).executionWorkspaceId).toBe(replacement.persistedExecutionWorkspace!.id);
|
|
}, 30_000,
|
|
);
|
|
|
|
it("starts a fresh clean task directory without treating it as a Git source", async () => {
|
|
const { input, source, issue } = await fixture();
|
|
input.policy.useIsolatedTaskDirectory = true;
|
|
input.policy.effectiveExecutionWorkspaceMode = "isolated_workspace";
|
|
input.policy.requestedExecutionWorkspaceMode = "isolated_workspace";
|
|
input.task.freshWorkspace = { mode: "fresh_isolated_workspace", retryIntentId: input.run.id,
|
|
sourceRunId: input.run.id, retainedSourceIds: [] };
|
|
const prepared = await prepareHeartbeatWorkspace(db, input);
|
|
expect(prepared.executionWorkspace.cwd).toContain("/isolated-workspace-attempts/");
|
|
expect(prepared.executionWorkspace.cwd).toContain(`/${input.run.id}`);
|
|
expect(prepared.executionWorkspace.cwd).not.toBe(source);
|
|
await expect(access(join(prepared.executionWorkspace.cwd, "README.md"))).rejects.toThrow();
|
|
expect(await readFile(join(source, "README.md"), "utf8")).toBe("source content\n");
|
|
expect((await readIssue(issue.id)).executionWorkspaceId).toBe(prepared.persistedExecutionWorkspace!.id);
|
|
});
|
|
|
|
it.each(["reuse", "explicit_git", "foreign_path", "missing_path", "symlink_path", "binding_changed", "canonical_retry", "canonical_warning", "canonical_unknown_warning", "canonical_provider_warning", "canonical_fresh", "canonical_lease", "canonical_result", "canonical_controller", "canonical_output", "canonical_measurement", "canonical_repaired_git"] as const)("recomputes ordinary-turn policy for a saved source-free fresh directory: %s", async scenario => {
|
|
const { input, issue } = await fixture();
|
|
const userId = `directory-owner-${randomUUID()}`;
|
|
await db.insert(authUsers).values({ id: userId, name: "Directory owner", email: `${userId}@example.test`, createdAt: new Date(), updatedAt: new Date() });
|
|
await db.insert(companyMemberships).values({ companyId: input.agent.companyId, principalId: userId, principalType: "user", membershipRole: "owner", status: "active" });
|
|
await db.update(companies).set({ defaultResponsibleUserId: userId }).where(eq(companies.id, input.agent.companyId));
|
|
await db.update(issues).set({ projectId: null, projectWorkspaceId: null, status: "in_progress", responsibleUserId: userId }).where(eq(issues.id, issue.id));
|
|
input.task.issueRef = await readIssue(issue.id);
|
|
input.task.context = { issueId: issue.id };
|
|
input.policy.useIsolatedTaskDirectory = true;
|
|
input.policy.effectiveExecutionWorkspaceMode = "isolated_workspace";
|
|
input.policy.requestedExecutionWorkspaceMode = "isolated_workspace";
|
|
input.config.mergedConfig = { workspaceStrategy: { type: "git_worktree" } };
|
|
input.task.freshWorkspace = { mode: "fresh_isolated_workspace", retryIntentId: input.run.id,
|
|
sourceRunId: input.run.id, retainedSourceIds: [] };
|
|
const original = await prepareHeartbeatWorkspace(db, input);
|
|
const workspace = original.persistedExecutionWorkspace!;
|
|
await writeFile(join(workspace.cwd!, "saved-review.txt"), "completed snapshot review\n");
|
|
await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date() }).where(eq(heartbeatRuns.id, input.run.id));
|
|
await db.update(environmentLeases).set({ status: "expired", cleanupStatus: "success", releasedAt: new Date() }).where(eq(environmentLeases.heartbeatRunId, input.run.id));
|
|
const adapterType = "process";
|
|
const originalAdapter = getServerAdapter(adapterType);
|
|
let preservedCwd = workspace.cwd!;
|
|
if (scenario === "explicit_git" || scenario.startsWith("canonical_")) await db.update(issues).set({ executionWorkspaceSettings: { mode: "isolated_workspace", workspaceStrategy: { type: "git_worktree" } } }).where(eq(issues.id, issue.id));
|
|
if (scenario === "foreign_path") {
|
|
const outside = join(home, "unowned-directory"); await mkdir(outside);
|
|
await db.update(executionWorkspaces).set({ cwd: outside }).where(eq(executionWorkspaces.id, workspace.id));
|
|
}
|
|
if (scenario === "missing_path" || scenario === "symlink_path") {
|
|
preservedCwd = join(home, `retained-files-${issue.id}`); await rename(workspace.cwd!, preservedCwd);
|
|
if (scenario === "symlink_path") await symlink(preservedCwd, workspace.cwd!);
|
|
}
|
|
if (scenario === "binding_changed") await db.update(issues).set({ executionWorkspaceId: null }).where(eq(issues.id, issue.id));
|
|
const freshExpected = ["canonical_unknown_warning", "canonical_provider_warning", "canonical_fresh", "canonical_lease", "canonical_result", "canonical_controller", "canonical_output", "canonical_measurement"].includes(scenario);
|
|
const executed = vi.fn(async (execution: AdapterExecutionContext) => {
|
|
const [lease] = await leases(execution.runId);
|
|
if (freshExpected) {
|
|
expect(lease.executionWorkspaceId).not.toBe(workspace.id);
|
|
await db.update(issues).set({ status: "done" }).where(eq(issues.id, issue.id));
|
|
return { exitCode: 0, signal: null, timedOut: false, summary: "Fresh workspace required" };
|
|
}
|
|
if (scenario !== "canonical_repaired_git") expect(lease.executionWorkspaceId).toBe(workspace.id);
|
|
const activeWorkspace = await input.services.executionWorkspacesSvc.getStoredById(input.agent.companyId, lease.executionWorkspaceId!);
|
|
if (scenario === "canonical_repaired_git") {
|
|
expect(execution.config.workspaceStrategy).toMatchObject({ type: "git_worktree" });
|
|
expect((await execFile("git", ["-C", activeWorkspace!.cwd!, "rev-parse", "--is-inside-work-tree"])).stdout.trim()).toBe("true");
|
|
}
|
|
else expect(activeWorkspace?.cwd).toBe(workspace.cwd);
|
|
expect(await readFile(join(activeWorkspace!.cwd!, "saved-review.txt"), "utf8")).toBe("completed snapshot review\n");
|
|
await db.update(issues).set({ status: "done" }).where(eq(issues.id, issue.id));
|
|
return { exitCode: 0, signal: null, timedOut: false, summary: "Retained review confirmed" };
|
|
});
|
|
registerServerAdapter({ ...originalAdapter, type: adapterType, supportsLocalAgentJwt: false, execute: executed,
|
|
testEnvironment: async () => ({ adapterType, status: "pass", checks: [], testedAt: new Date().toISOString() }) });
|
|
await db.update(agents).set({ adapterType, status: "idle", adapterConfig: { workspaceStrategy: { type: "git_worktree" } } }).where(eq(agents.id, input.agent.id));
|
|
if (scenario.includes("warning")) await db.insert(companyObservability).values({ companyId: input.agent.companyId, revision: randomUUID(), enabled: true,
|
|
destination: { provider: "otlp", endpointBaseUrl: "https://collector.example/otel" },
|
|
capture: { userPrompts: false, assistantResponses: false, toolDetails: false, toolContent: false } });
|
|
const heartbeat = heartbeatService(db);
|
|
try {
|
|
const next = await heartbeat.wakeup(input.agent.id, { source: "on_demand", triggerDetail: "manual", reason: "manual",
|
|
requestedByActorType: "user", requestedByActorId: userId, payload: { issueId: issue.id }, contextSnapshot: { issueId: issue.id } });
|
|
await heartbeat.drainActiveRunExecutions();
|
|
let finalRunId = next?.id;
|
|
if (scenario.startsWith("canonical_")) {
|
|
expect(await readRun(next!.id)).toMatchObject({ status: "failed", errorCode: "workspace_validation_failed" });
|
|
expect(executed).not.toHaveBeenCalled();
|
|
if (scenario.includes("warning")) {
|
|
const warning = (await db.select().from(heartbeatRunEvents).where(eq(heartbeatRunEvents.runId, next!.id))).find(event => event.seq === 1)!;
|
|
expect(warning).toMatchObject({ eventType: "agent_observability.unavailable", payload: { reason: "unsupported_harness" } });
|
|
if (scenario === "canonical_unknown_warning") await db.update(heartbeatRunEvents).set({ payload: { ...warning.payload, reason: "unknown" } }).where(eq(heartbeatRunEvents.id, warning.id));
|
|
if (scenario === "canonical_provider_warning") await db.update(heartbeatRunEvents).set({ sourcePayloadSha256: "a".repeat(64), protocolSchemaVersion: 1 }).where(eq(heartbeatRunEvents.id, warning.id));
|
|
}
|
|
if (scenario === "canonical_lease") await db.insert(environmentLeases).values({ companyId: input.agent.companyId, issueId: issue.id, heartbeatRunId: next!.id, provider: "local_fs", status: "expired", cleanupStatus: "success" });
|
|
if (scenario === "canonical_result") { const failed = await readRun(next!.id); await db.update(heartbeatRuns).set({ resultJson: { ...failed.resultJson, acceptedResult: { outcome: "unknown" } } }).where(eq(heartbeatRuns.id, next!.id)); }
|
|
if (scenario === "canonical_controller") await db.update(heartbeatRuns).set({ controllerLeaseExpiresAt: new Date(Date.now() + 60_000) }).where(eq(heartbeatRuns.id, next!.id));
|
|
if (scenario === "canonical_output") await db.update(heartbeatRuns).set({ lastOutputSeq: 1 }).where(eq(heartbeatRuns.id, next!.id));
|
|
if (scenario === "canonical_measurement") { const events = await db.select().from(heartbeatRunEvents).where(eq(heartbeatRunEvents.runId, next!.id)); const measured = events.find(event => event.eventType === "lifecycle")!; await db.update(heartbeatRunEvents).set({ payload: { ...measured.payload, durationMs: -1 } }).where(eq(heartbeatRunEvents.id, measured.id)); }
|
|
if (scenario === "canonical_repaired_git") {
|
|
await execFile("git", ["init", "--initial-branch=main", workspace.cwd!]);
|
|
await execFile("git", ["-C", workspace.cwd!, "config", "user.email", "test@example.com"]);
|
|
await execFile("git", ["-C", workspace.cwd!, "config", "user.name", "Directory repair"]);
|
|
await execFile("git", ["-C", workspace.cwd!, "add", "."]);
|
|
await execFile("git", ["-C", workspace.cwd!, "commit", "-m", "Retain saved review"]);
|
|
} else await db.update(issues).set({ executionWorkspaceSettings: { mode: "isolated_workspace" } }).where(eq(issues.id, issue.id));
|
|
const intent = await requestTaskRetry({ db, companyId: input.agent.companyId, issueId: issue.id,
|
|
actorId: userId, actorSource: "local_implicit", observedRunId: next!.id, idempotencyKey: randomUUID() });
|
|
expect(intent.requestId).toBeTruthy();
|
|
if (scenario === "canonical_fresh") await db.update(agentWakeupRequests).set({ payload: sql`jsonb_set(${agentWakeupRequests.payload}, '{taskRetry,strategy}', '"fresh_isolated_workspace"'::jsonb)` }).where(eq(agentWakeupRequests.id, intent.requestId!));
|
|
await heartbeat.resumeTaskRetry(input.agent.companyId, intent.requestId!);
|
|
await heartbeat.drainActiveRunExecutions();
|
|
const [receipt] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, intent.requestId!));
|
|
expect(receipt.runId).toBeTruthy();
|
|
finalRunId = receipt.runId!;
|
|
}
|
|
const continued = scenario === "reuse" || scenario.startsWith("canonical_");
|
|
if (scenario === "binding_changed") {
|
|
expect(next).toBeNull();
|
|
const [receipt] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.agentId, input.agent.id));
|
|
expect(receipt).toMatchObject({ status: "skipped", reason: "workspace_worktree_requires_project", runId: null });
|
|
} else {
|
|
expect(next).not.toBeNull();
|
|
expect(await readRun(finalRunId!)).toMatchObject(continued
|
|
? { status: "succeeded", errorCode: null } : { status: "failed", errorCode: "workspace_validation_failed" });
|
|
}
|
|
expect(executed).toHaveBeenCalledTimes(continued ? 1 : 0);
|
|
if (freshExpected) expect((await readIssue(issue.id)).executionWorkspaceId).not.toBe(workspace.id);
|
|
else if (scenario !== "canonical_repaired_git") expect((await readIssue(issue.id)).executionWorkspaceId).toBe(scenario === "binding_changed" ? null : workspace.id);
|
|
expect(await readFile(join(preservedCwd, "saved-review.txt"), "utf8")).toBe("completed snapshot review\n");
|
|
if (scenario === "missing_path") await expect(access(workspace.cwd!)).rejects.toThrow();
|
|
} finally { await heartbeat.drainActiveRunExecutions(); registerServerAdapter(originalAdapter); }
|
|
}, 30_000);
|
|
|
|
it("reuses a successful fresh task attempt for the next ordinary user turn", async () => {
|
|
const { input, issue } = await fixture();
|
|
input.policy.useIsolatedTaskDirectory = true;
|
|
input.policy.effectiveExecutionWorkspaceMode = "isolated_workspace";
|
|
input.policy.requestedExecutionWorkspaceMode = "isolated_workspace";
|
|
input.task.freshWorkspace = { mode: "fresh_isolated_workspace", retryIntentId: input.run.id,
|
|
sourceRunId: input.run.id, retainedSourceIds: [] };
|
|
const original = await prepareHeartbeatWorkspace(db, input);
|
|
const workspace = original.persistedExecutionWorkspace!;
|
|
await writeFile(join(workspace.cwd!, "saved-review.txt"), "completed review history\n");
|
|
await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date() })
|
|
.where(eq(heartbeatRuns.id, input.run.id));
|
|
await reuse(input, workspace);
|
|
input.task.freshWorkspace = null;
|
|
const next = await prepareHeartbeatWorkspace(db, input);
|
|
expect(next.persistedExecutionWorkspace!.id).toBe(workspace.id);
|
|
expect(next.executionWorkspace.cwd).toBe(workspace.cwd);
|
|
expect(await readFile(join(next.executionWorkspace.cwd, "saved-review.txt"), "utf8"))
|
|
.toBe("completed review history\n");
|
|
expect((await readIssue(issue.id)).executionWorkspaceId).toBe(workspace.id);
|
|
await reuse(input, workspace);
|
|
input.task.freshWorkspace = { mode: "fresh_isolated_workspace", retryIntentId: input.run.id,
|
|
sourceRunId: input.run.id, retainedSourceIds: [] };
|
|
input.reuse.existingExecutionWorkspace = null;
|
|
input.reuse.reusableExistingExecutionWorkspace = null;
|
|
input.reuse.requestedShouldReuseExisting = false;
|
|
input.reuse.workspaceReuseRequest = resolveExecutionWorkspaceReuseRequestForIssue({
|
|
issueExecutionWorkspaceId: null, issueExecutionWorkspacePreference: "create_new", existingExecutionWorkspaceStatus: null,
|
|
});
|
|
const replacement = await prepareHeartbeatWorkspace(db, input);
|
|
expect(replacement.executionWorkspace.cwd).not.toBe(workspace.cwd);
|
|
await expect(access(join(replacement.executionWorkspace.cwd, "saved-review.txt"))).rejects.toThrow();
|
|
expect(await readFile(join(workspace.cwd!, "saved-review.txt"), "utf8")).toBe("completed review history\n");
|
|
});
|
|
|
|
it.each(["task_binding", "workspace_row", "foreign_issue", "foreign_company", "closed_workspace"])(
|
|
"rejects a retained task attempt after its %s changes", async change => {
|
|
const { input, issue, source } = await fixture();
|
|
input.policy.useIsolatedTaskDirectory = true;
|
|
input.policy.effectiveExecutionWorkspaceMode = "isolated_workspace";
|
|
input.policy.requestedExecutionWorkspaceMode = "isolated_workspace";
|
|
input.task.freshWorkspace = { mode: "fresh_isolated_workspace", retryIntentId: input.run.id,
|
|
sourceRunId: input.run.id, retainedSourceIds: [] };
|
|
const original = await prepareHeartbeatWorkspace(db, input);
|
|
const workspace = original.persistedExecutionWorkspace!;
|
|
await writeFile(join(workspace.cwd!, "saved-review.txt"), "keep saved work\n");
|
|
await reuse(input, workspace);
|
|
input.task.freshWorkspace = null;
|
|
if (change === "task_binding") await db.update(issues).set({ executionWorkspaceId: null }).where(eq(issues.id, issue.id));
|
|
if (change === "workspace_row") await db.update(executionWorkspaces).set({ cwd: source }).where(eq(executionWorkspaces.id, workspace.id));
|
|
if (change === "foreign_issue") await db.update(executionWorkspaces).set({ sourceIssueId: null }).where(eq(executionWorkspaces.id, workspace.id));
|
|
if (change === "foreign_company") {
|
|
const [other] = await db.insert(companies).values({ name: "Other workspace company", issuePrefix: "OTH" }).returning();
|
|
await db.update(executionWorkspaces).set({ companyId: other.id }).where(eq(executionWorkspaces.id, workspace.id));
|
|
}
|
|
if (change === "closed_workspace") await db.update(executionWorkspaces).set({ status: "archived" }).where(eq(executionWorkspaces.id, workspace.id));
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toMatchObject({
|
|
resultJson: { workspaceValidation: { reason: "isolated_task_directory_binding_mismatch", issueId: issue.id } },
|
|
});
|
|
expect(await readFile(join(workspace.cwd!, "saved-review.txt"), "utf8")).toBe("keep saved work\n");
|
|
},
|
|
);
|
|
|
|
async function reuse(input: HeartbeatWorkspacePreparationInput, workspace: NonNullable<Awaited<ReturnType<typeof prepareHeartbeatWorkspace>>["persistedExecutionWorkspace"]>) {
|
|
input.reuse.requestedShouldReuseExisting = true;
|
|
input.reuse.existingExecutionWorkspace = workspace;
|
|
input.reuse.reusableExistingExecutionWorkspace = workspace;
|
|
input.reuse.workspaceReuseRequest = resolveExecutionWorkspaceReuseRequestForIssue({ issueExecutionWorkspaceId: workspace.id, issueExecutionWorkspacePreference: "reuse_existing", existingExecutionWorkspaceStatus: workspace.status });
|
|
input.task.issueRef = { ...input.task.issueRef!, executionWorkspaceId: workspace.id, executionWorkspacePreference: "reuse_existing" };
|
|
const [run] = await db.insert(heartbeatRuns).values({ companyId: input.run.companyId, agentId: input.agent.id, issueId: input.task.issueId, scopeKind: "issue", status: "running", runtimeMode: "legacy" }).returning();
|
|
input.run = run;
|
|
}
|
|
|
|
async function persistRecoveryInput(input: HeartbeatWorkspacePreparationInput, workspaceId: string, cwd: string) {
|
|
const issue = await readIssue(input.task.issueId!);
|
|
const admitted = buildNativeExecutionInput({ companyId: input.agent.companyId, agentId: input.agent.id,
|
|
runId: input.run.id, provider: "codex",
|
|
issue: { id: issue.id, identifier: issue.identifier, title: issue.title, description: null, workMode: "standard" },
|
|
taskPrompt: "Continue admitted task", normalizedSessionId: null,
|
|
workspace: { id: workspaceId, cwd, repoUrl: null, repoRef: null, branchName: null },
|
|
completionContract: { id: input.run.id, sha256: `sha256:${"a".repeat(64)}`, schemaVersion: "paperclip.run-result.v1",
|
|
contract: { revision: "1", objective: "Continue admitted task", criteria: [{ id: "objective", requirement: "Finish" }] } },
|
|
runtimeContext: nativeRuntimeContextFixture(),
|
|
});
|
|
input.run.runtimeMode = "native";
|
|
input.run.nativeIssueId = issue.id;
|
|
input.run.runnerProfileJson = { nativeExecutionInput: admitted };
|
|
input.reuse.persistedNativeExecutionInput = admitted;
|
|
await db.update(heartbeatRuns).set({ runtimeMode: "native", nativeIssueId: issue.id,
|
|
runnerProfileJson: input.run.runnerProfileJson }).where(eq(heartbeatRuns.id, input.run.id));
|
|
return admitted;
|
|
}
|
|
|
|
it.each(["muse", "openai_dot", "grok_bot"] as const)("uses private %s controller storage without rebinding task files or preparing repositories", async provider => {
|
|
const { input, issue } = await fixture();
|
|
input.agent.adapterType = "paperclip_runner";
|
|
input.config.mergedConfig = { provider, cwd: "/task/source/must-not-be-read", workspaceStrategy: { type: "git_worktree" } };
|
|
input.reuse.isDotRun = true;
|
|
const resolve = vi.spyOn(input.services, "resolveWorkspaceForRun").mockRejectedValue(new Error("Task files must not be adopted"));
|
|
const bind = vi.spyOn(input.services.executionWorkspacesSvc, "bindTaskWorkspace").mockRejectedValue(new Error("Task binding must not change"));
|
|
const repos = vi.spyOn(input.services.executionWorkspacesSvc, "prepareTaskRepositoriesForAdmission").mockRejectedValue(new Error("Repositories must not be prepared"));
|
|
try {
|
|
const result = await prepareHeartbeatWorkspace(db, input);
|
|
expect(result.executionWorkspace.cwd).toContain(provider === "muse" ? "muse-controllers" : "dot-controllers");
|
|
expect(result.persistedExecutionWorkspace).toBeNull();
|
|
expect(result.projectRepositoryPaths).toEqual([]);
|
|
expect((await readIssue(issue.id)).executionWorkspaceId).toBeNull();
|
|
expect(resolve).not.toHaveBeenCalled();
|
|
expect(bind).not.toHaveBeenCalled();
|
|
expect(repos).not.toHaveBeenCalled();
|
|
const binding = { companyId: input.agent.companyId, agentId: input.agent.id, bindingId: input.run.id,
|
|
bindingGeneration: 1, acceptByUnixMs: Date.now() + 60_000, expiresAtUnixMs: Date.now() + 600_000 };
|
|
const admitted = buildNativeExecutionInput({ companyId: input.agent.companyId, agentId: input.agent.id,
|
|
runId: input.run.id, provider, ...(provider === "muse" ? { museBinding: binding } : { dotBinding: binding }),
|
|
issue: { id: issue.id, identifier: null, title: issue.title, description: null, workMode: "standard" },
|
|
taskPrompt: "Preserve the admitted controller", normalizedSessionId: input.run.id,
|
|
workspace: { id: input.run.id, cwd: result.executionWorkspace.cwd, repoUrl: null, repoRef: null, branchName: null },
|
|
completionContract: { id: input.run.id, sha256: `sha256:${"a".repeat(64)}`, schemaVersion: "paperclip.run-result.v1",
|
|
contract: { revision: "1", objective: "Preserve the controller", criteria: [{ id: "objective", requirement: "Preserve the original checkpoint path" }] } },
|
|
runtimeContext: nativeRuntimeContextFixture(),
|
|
});
|
|
input.reuse.persistedNativeExecutionInput = admitted;
|
|
input.run.runtimeMode = "native";
|
|
input.run.runnerProfileJson = { nativeExecutionInput: admitted };
|
|
input.config.mergedConfig = { provider: "codex" };
|
|
const recovered = await prepareHeartbeatWorkspace(db, input);
|
|
expect(recovered.executionWorkspace.cwd).toBe(result.executionWorkspace.cwd);
|
|
expect(recovered.persistedExecutionWorkspace).toBeNull();
|
|
expect(bind).not.toHaveBeenCalled();
|
|
expect(repos).not.toHaveBeenCalled();
|
|
|
|
} finally { resolve.mockRestore(); bind.mockRestore(); repos.mockRestore(); }
|
|
});
|
|
|
|
it("persists the workspace and issue binding before acquiring its environment", async () => {
|
|
const { input, source, issue, envOrchestrator } = await fixture();
|
|
input.config.managedAiCredentialIdentity = "account:generation";
|
|
const acquire = envOrchestrator.acquireForRun;
|
|
vi.spyOn(envOrchestrator, "acquireForRun").mockImplementation(async args => {
|
|
expect((await readIssue(issue.id)).executionWorkspaceId).toBe(args.persistedExecutionWorkspace?.id);
|
|
expect((await readRun(input.run.id)).contextSnapshot?.executionWorkspaceId).toBe(args.persistedExecutionWorkspace?.id);
|
|
expect(args.managedAiCredentialIdentity).toBe("account:generation");
|
|
return acquire(args);
|
|
});
|
|
const result = await prepareHeartbeatWorkspace(db, input);
|
|
expect(result.executionWorkspace.cwd).toBe(source);
|
|
expect(result.persistedExecutionWorkspace).toMatchObject({ companyId: issue.companyId, projectId: issue.projectId, projectWorkspaceId: issue.projectWorkspaceId, mode: "shared_workspace" });
|
|
expect(await leases(input.run.id)).toMatchObject([{ status: "active", executionWorkspaceId: result.persistedExecutionWorkspace!.id }]);
|
|
expect(input.nativeRunnerPreparationSpans.map(span => span.name)).toEqual(["environment.acquire", "environment.workspace.realize"]);
|
|
expect(input.nativeRunnerPreparationSpans.every(span => span.parentName === "task.provider_session")).toBe(true);
|
|
expect(input.controllerLease.assertOwned).toHaveBeenCalledTimes(2);
|
|
});
|
|
|
|
it("records workspace steps around real persistence before environment acquisition", async () => {
|
|
const { input, source, issue, envOrchestrator } = await fixture();
|
|
const endpoint = process.env.OTEL_EXPORTER_OTLP_ENDPOINT;
|
|
vi.stubEnv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://collector.test");
|
|
const exporter = new InMemorySpanExporter();
|
|
const provider = new TracerProvider({ spanProcessors: [new SimpleSpanProcessor({ exporter })] });
|
|
expect(trace.setGlobalTracerProvider(provider)).toBe(true);
|
|
const acquire = envOrchestrator.acquireForRun;
|
|
vi.spyOn(envOrchestrator, "acquireForRun").mockImplementation(async args => {
|
|
const names = exporter.getFinishedSpans().map(span => span.name);
|
|
expect(names).toContain("heartbeat.prepare.workspace_record_create");
|
|
expect(names).toContain("heartbeat.prepare.workspace_binding");
|
|
expect((await readIssue(issue.id)).executionWorkspaceId).toBe(args.persistedExecutionWorkspace?.id);
|
|
return acquire(args);
|
|
});
|
|
try {
|
|
const result = await prepareHeartbeatWorkspace(db, input);
|
|
expect(result.executionWorkspace.cwd).toBe(source);
|
|
await provider.forceFlush();
|
|
const spans = exporter.getFinishedSpans();
|
|
expect(spans.map(span => span.name)).toEqual(expect.arrayContaining([
|
|
"heartbeat.prepare.workspace_source_resolution",
|
|
"heartbeat.prepare.workspace_provision_policy",
|
|
"heartbeat.prepare.workspace_base_validation",
|
|
"heartbeat.prepare.workspace_provision",
|
|
"heartbeat.prepare.workspace_operation_binding",
|
|
"heartbeat.prepare.workspace_config_recording",
|
|
"heartbeat.prepare.workspace_repository_admission",
|
|
"heartbeat.prepare.workspace_project_repositories",
|
|
]));
|
|
for (const span of spans) {
|
|
expect(Object.keys(span.attributes)).toEqual(["paperclip.task.run.run_id"]);
|
|
expect(JSON.stringify(span.attributes)).not.toContain(source);
|
|
}
|
|
} finally {
|
|
vi.stubEnv("OTEL_EXPORTER_OTLP_ENDPOINT", endpoint);
|
|
trace.disable();
|
|
await provider.shutdown();
|
|
}
|
|
});
|
|
|
|
it("prepares, reuses, and reopens a selected source containing an inaccessible private worktree", async () => {
|
|
const { input, source, issue } = await fixture(true);
|
|
vi.stubEnv("PAPERCLIP_ISSUE_PRIVACY_MODE", "enforce");
|
|
const [privateIssue] = await db.insert(issues).values({ companyId: issue.companyId,
|
|
title: "Private historical task", visibility: "private", projectId: issue.projectId }).returning();
|
|
const privateCwd = join(source, "private-work");
|
|
await execFile("git", ["-C", source, "worktree", "add", "-b", "historical-private", privateCwd, "main"]);
|
|
await writeFile(join(privateCwd, "confidential.txt"), "Shared OS storage is not task authorization");
|
|
const [privateWorkspace] = await db.insert(executionWorkspaces).values({ companyId: issue.companyId,
|
|
name: "Private workspace", sourceIssueId: privateIssue.id, cwd: privateCwd,
|
|
mode: "isolated_workspace", strategyType: "git_worktree", providerType: "git_worktree",
|
|
providerRef: privateCwd, branchName: "historical-private", status: "idle" }).returning();
|
|
await db.update(issues).set({ executionWorkspaceId: privateWorkspace.id }).where(eq(issues.id, privateIssue.id));
|
|
const clone = vi.spyOn(managedRepositoryCheckout, "materializeManagedProjectWorkspace");
|
|
const result = await prepareHeartbeatWorkspace(db, input);
|
|
expect(result.executionWorkspace.baseCwd).toBe(source);
|
|
expect(result.persistedExecutionWorkspace!.metadata?.isolatedRemoteSource).toBeUndefined();
|
|
expect(await readFile(join(result.executionWorkspace.cwd, "README.md"), "utf8")).toBe("source content\n");
|
|
expect(clone).not.toHaveBeenCalled();
|
|
await expect(access(isolatedRemoteSourcePath(issue.companyId, issue.id))).rejects.toMatchObject({ code: "ENOENT" });
|
|
await writeFile(join(result.executionWorkspace.cwd, "unpublished-work.txt"), "Keep current task edits\n");
|
|
await reuse(input, result.persistedExecutionWorkspace!);
|
|
const resumed = await prepareHeartbeatWorkspace(db, input);
|
|
expect(resumed.persistedExecutionWorkspace!.id).toBe(result.persistedExecutionWorkspace!.id);
|
|
expect(await readFile(join(resumed.executionWorkspace.cwd, "unpublished-work.txt"), "utf8")).toBe("Keep current task edits\n");
|
|
const workspace = resumed.persistedExecutionWorkspace!;
|
|
const cleanup = await cleanupExecutionWorkspaceArtifacts({ workspace,
|
|
projectWorkspace: { cwd: source, cleanupCommand: null }, runCleanupCommands: false });
|
|
expect(cleanup.warnings).toEqual([]);
|
|
await db.update(executionWorkspaces).set({ status: "archived" }).where(eq(executionWorkspaces.id, workspace.id));
|
|
const reopened = await executionWorkspaceService(db).reopenClosedIsolatedExecutionWorkspaceForIssue({ workspaceId: workspace.id,
|
|
issue: { id: issue.id, companyId: issue.companyId, projectId: issue.projectId },
|
|
actor: { agentId: input.agent.id, actorType: "agent" },
|
|
authorizationActor: { type: "agent", source: "agent_jwt", agentId: input.agent.id, companyId: issue.companyId, runId: input.run.id },
|
|
});
|
|
expect(reopened).toMatchObject({ ok: true });
|
|
expect(await readFile(join(workspace.cwd!, "README.md"), "utf8")).toBe("source content\n");
|
|
expect(await readFile(join(privateCwd, "confidential.txt"), "utf8")).toContain("OS storage");
|
|
});
|
|
|
|
it("cleans up a new local worktree after workspace persistence fails", async () => {
|
|
const { input, source, issue } = await fixture(true);
|
|
vi.spyOn(input.services.executionWorkspacesSvc, "create").mockRejectedValueOnce(new Error("Workspace persistence failed"));
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow("Workspace persistence failed");
|
|
const worktrees = (await execFile("git", ["-C", source, "worktree", "list", "--porcelain"])).stdout;
|
|
expect(worktrees.match(/^worktree /gm)).toHaveLength(1);
|
|
await expect(execFile("git", ["-C", source, "rev-parse", "--verify", `refs/heads/test-${issue.identifier}`])).rejects.toBeDefined();
|
|
expect((await readIssue(issue.id)).executionWorkspaceId).toBeNull();
|
|
});
|
|
|
|
it.each(["sandbox", "ssh", "plugin", "local", null] as const)("uses the selected %s placement for controller provisioning", async driver => {
|
|
const { input, source, issue, envOrchestrator } = await fixture(true);
|
|
const marker = join(source, "host-provision-ran");
|
|
const provisionCommand = `printf host > '${marker}'`;
|
|
input.config.mergedConfig = { workspaceStrategy: { type: "git_worktree", provisionCommand, runtimeProvisionCommand: "printf runtime" } };
|
|
const originalConfig = structuredClone(input.config.mergedConfig);
|
|
input.environment.selectedEnvironmentForConfig = driver === null ? null : { ...input.environment.localEnvironment, driver };
|
|
const stopped = new Error("Fixture stops before transport acquisition");
|
|
vi.spyOn(envOrchestrator, "acquireForRun").mockRejectedValue(stopped);
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toBe(stopped);
|
|
const bound = await readIssue(issue.id);
|
|
const [workspace] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, bound.executionWorkspaceId!));
|
|
expect(await readFile(join(workspace.cwd!, "README.md"), "utf8")).toBe("source content\n");
|
|
if (driver === "sandbox" || driver === "ssh" || driver === "plugin") await expect(readFile(marker, "utf8")).rejects.toMatchObject({ code: "ENOENT" });
|
|
else expect(await readFile(marker, "utf8")).toBe("host");
|
|
expect(input.config.mergedConfig).toEqual(originalConfig);
|
|
});
|
|
|
|
it("keeps a plugin workspace ledger across runs without preparing Core repositories", async () => {
|
|
const { input, issue, envOrchestrator, source } = await fixture();
|
|
input.agent.adapterType = "paperclip_runner";
|
|
input.environment.selectedEnvironmentForConfig = { ...input.environment.localEnvironment, driver: "plugin" };
|
|
const guestOnly = join(home, "guest-only", issue.projectId!);
|
|
await db.update(projectWorkspaces).set({ cwd: guestOnly }).where(eq(projectWorkspaces.id, issue.projectWorkspaceId!));
|
|
const copy = vi.spyOn(input.services.executionWorkspacesSvc, "prepareTaskRepositoriesForAdmission").mockRejectedValue(new Error("Core must not prepare repositories"));
|
|
const restoredCopy = vi.spyOn(input.services.executionWorkspacesSvc, "prepareConfiguredRepositoryCopy").mockRejectedValue(new Error("Core must not restore repositories"));
|
|
const stopped = new Error("Plugin realization boundary");
|
|
const realize = vi.spyOn(envOrchestrator, "realizeForRun").mockRejectedValue(stopped);
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toBe(stopped);
|
|
const firstId = (await readIssue(issue.id)).executionWorkspaceId!;
|
|
const [workspace] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, firstId));
|
|
expect(workspace).toMatchObject({ cwd: `/workspaces/${issue.projectWorkspaceId}`, strategyType: "project_primary", providerType: "adapter_managed" });
|
|
await reuse(input, toExecutionWorkspace(workspace));
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toBe(stopped);
|
|
expect((await readIssue(issue.id)).executionWorkspaceId).toBe(firstId);
|
|
expect(realize).toHaveBeenCalledTimes(2);
|
|
expect(copy).not.toHaveBeenCalled(); expect(restoredCopy).not.toHaveBeenCalled();
|
|
await expect(access(guestOnly)).rejects.toMatchObject({ code: "ENOENT" });
|
|
input.environment.selectedEnvironmentForConfig = input.environment.localEnvironment;
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toMatchObject({
|
|
resultJson: { reason: "plugin_workspace_owner_mismatch" },
|
|
});
|
|
expect(realize).toHaveBeenCalledTimes(2);
|
|
input.environment.selectedEnvironmentForConfig = { ...input.environment.localEnvironment, driver: "plugin" };
|
|
const [localWorkspace] = await db.update(executionWorkspaces).set({ providerType: "local_fs", cwd: source, metadata: {} })
|
|
.where(eq(executionWorkspaces.id, firstId)).returning();
|
|
await reuse(input, toExecutionWorkspace(localWorkspace));
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toMatchObject({
|
|
resultJson: { reason: "plugin_workspace_owner_mismatch" },
|
|
});
|
|
expect(realize).toHaveBeenCalledTimes(2);
|
|
expect(await readFile(join(source, "README.md"), "utf8")).toBe("source content\n");
|
|
});
|
|
|
|
it.each(["services", "fresh"] as const)("rejects unsupported %s before plugin workspace preparation or acquisition", async mode => {
|
|
const { input, envOrchestrator } = await fixture();
|
|
input.agent.adapterType = "paperclip_runner";
|
|
input.environment.selectedEnvironmentForConfig = { ...input.environment.localEnvironment, driver: "plugin" };
|
|
if (mode === "services") input.config.mergedConfig.workspaceRuntime = { services: [{ name: "preview", command: "must-not-run-on-core" }] };
|
|
else input.task.freshWorkspace = { mode: "fresh_isolated_workspace", retryIntentId: input.run.id,
|
|
sourceRunId: input.run.id, retainedSourceIds: [] };
|
|
const acquire = vi.spyOn(envOrchestrator, "acquireForRun");
|
|
const resolve = vi.spyOn(input.services, "resolveWorkspaceForRun");
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toMatchObject({
|
|
resultJson: { reason: mode === "services" ? "plugin_workspace_services_unsupported" : "fresh_workspace_storage_unsupported" },
|
|
});
|
|
expect(resolve).not.toHaveBeenCalled();
|
|
expect(acquire).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it("admits the metadata-only agent-default strategy for a plugin workspace", async () => {
|
|
const { input, envOrchestrator } = await fixture();
|
|
input.agent.adapterType = "paperclip_runner";
|
|
input.environment.selectedEnvironmentForConfig = { ...input.environment.localEnvironment, driver: "plugin" };
|
|
input.policy.requestedExecutionWorkspaceMode = "agent_default";
|
|
input.policy.effectiveExecutionWorkspaceMode = "agent_default";
|
|
const stopped = new Error("Plugin agent-default admission reached");
|
|
const realize = vi.spyOn(envOrchestrator, "realizeForRun").mockRejectedValue(stopped);
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toBe(stopped);
|
|
expect(realize).toHaveBeenCalledWith(expect.objectContaining({ executionWorkspace: expect.objectContaining({ strategy: "adapter_managed", cwd: `/workspaces/${input.task.issueId}` }),
|
|
persistedExecutionWorkspace: expect.objectContaining({ strategyType: "adapter_managed", providerType: "adapter_managed" }) }));
|
|
});
|
|
|
|
it("rejects Core Git worktree preparation for a plugin-owned workspace", async () => {
|
|
const { input, envOrchestrator } = await fixture(true);
|
|
input.agent.adapterType = "paperclip_runner";
|
|
input.environment.selectedEnvironmentForConfig = { ...input.environment.localEnvironment, driver: "plugin" };
|
|
const acquire = vi.spyOn(envOrchestrator, "acquireForRun");
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toMatchObject({
|
|
code: "workspace_validation_failed", resultJson: { reason: "plugin_workspace_strategy_unsupported" },
|
|
});
|
|
expect(acquire).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it.each([true, false])("forwards only authorized plugin project IDs without staging (remote enabled: %s)", async enabled => {
|
|
const { input, issue, envOrchestrator } = await fixture();
|
|
vi.stubEnv("PAPERCLIP_MULTI_PROJECT_WORKSPACE_SYNC", "true");
|
|
vi.stubEnv("PAPERCLIP_MULTI_PROJECT_WORKSPACE_SYNC_REMOTE", String(enabled));
|
|
input.agent = { ...input.agent, adapterType: "paperclip_runner" };
|
|
input.environment.selectedEnvironmentForConfig = { ...input.environment.localEnvironment, driver: "plugin" };
|
|
const [readable, privateProject] = await db.insert(projects).values([
|
|
{ companyId: issue.companyId, name: "Readable reference" },
|
|
{ companyId: issue.companyId, name: "Private reference", visibility: "private" },
|
|
]).returning();
|
|
const unavailableCwd = join(home, readable.id, "must-not-stage");
|
|
await db.insert(projectWorkspaces).values({ companyId: issue.companyId, projectId: readable.id, name: "Provider-only storage", cwd: unavailableCwd });
|
|
await db.update(issues).set({ description: [readable, privateProject].map(project => `[project](${buildProjectMentionHref(project.id)})`).join(" ") }).where(eq(issues.id, issue.id));
|
|
const stopped = new Error("Fixture reached plugin realization");
|
|
const acquire=vi.spyOn(envOrchestrator,"acquireForRun");
|
|
const realize = vi.spyOn(envOrchestrator, "realizeForRun").mockRejectedValue(stopped);
|
|
try {
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toBe(stopped);
|
|
expect(realize).toHaveBeenCalledWith(expect.objectContaining({
|
|
runnerTaskProjectIds: enabled ? [issue.projectId, readable.id] : undefined,
|
|
executionWorkspace: expect.objectContaining({ additionalWorkspaces: [] }),
|
|
}));
|
|
expect(acquire).toHaveBeenCalledWith(expect.objectContaining({runnerTaskProjectIds:
|
|
enabled ? [issue.projectId,readable.id] : [issue.projectId]}));
|
|
await expect(access(unavailableCwd)).rejects.toMatchObject({ code: "ENOENT" });
|
|
} finally {
|
|
acquire.mockRestore();
|
|
vi.stubEnv("PAPERCLIP_MULTI_PROJECT_WORKSPACE_SYNC", "true");
|
|
vi.stubEnv("PAPERCLIP_MULTI_PROJECT_WORKSPACE_SYNC_REMOTE", "true");
|
|
}
|
|
});
|
|
|
|
it("checks current repository admission before publishing configured sibling repositories", async () => {
|
|
const { input, source, issue } = await fixture();
|
|
await db.insert(projectWorkspaces).values({ companyId: issue.companyId, projectId: issue.projectId!,
|
|
name: "Sibling repository", repoUrl: pathToFileURL(source).href });
|
|
vi.spyOn(input.services.executionWorkspacesSvc, "prepareTaskRepositoriesForAdmission")
|
|
.mockRejectedValue(new Error("Current run identity is unavailable"));
|
|
const acquire = vi.spyOn(input.services.envOrchestrator, "acquireForRun");
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow("Current run identity is unavailable");
|
|
await expect(access(join(source, ".paperclip-repositories"))).rejects.toMatchObject({ code: "ENOENT" });
|
|
expect(acquire).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it("rechecks current binding after configured clone and before publication", async () => {
|
|
const { input, source, issue } = await fixture();
|
|
await db.insert(projectWorkspaces).values({ companyId: issue.companyId, projectId: issue.projectId!,
|
|
name: "Sibling repository", repoUrl: pathToFileURL(source).href });
|
|
const namespace = join(source, ".paperclip-repositories");
|
|
const guard = vi.spyOn(input.services.executionWorkspacesSvc, "assertTaskRepositoryPublicationAllowed")
|
|
.mockImplementation(async () => {
|
|
expect(await readdir(namespace)).toEqual([]);
|
|
throw new Error("Task binding changed during clone");
|
|
});
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow("Task binding changed during clone");
|
|
expect(guard).toHaveBeenCalledOnce();
|
|
expect(await readdir(namespace)).toEqual([]);
|
|
expect(await leases(input.run.id)).toEqual([]);
|
|
});
|
|
|
|
it("rejects a configured sibling copy when its source disappears before publication", async () => {
|
|
const previousPrivacyMode = process.env.PAPERCLIP_ISSUE_PRIVACY_MODE;
|
|
vi.stubEnv("PAPERCLIP_ISSUE_PRIVACY_MODE", "enforce");
|
|
try {
|
|
const { input, source, issue } = await fixture();
|
|
const sibling = source + "-private-sibling";
|
|
await execFile("git", ["clone", "--no-hardlinks", source, sibling]);
|
|
await writeFile(join(sibling, "private.txt"), "PRIVATE_SIBLING_STAGING_BYTES");
|
|
const [privateIssue] = await db.insert(issues).values({ companyId: issue.companyId, title: "Private sibling owner",
|
|
visibility: "private", responsibleUserId: "private-owner" }).returning();
|
|
const [privateWorkspace] = await db.insert(executionWorkspaces).values({ companyId: issue.companyId,
|
|
name: "Private sibling", projectId: null, sourceIssueId: privateIssue.id, cwd: sibling,
|
|
providerType: "local_fs", strategyType: "project_primary", mode: "shared_workspace", status: "active" }).returning();
|
|
await db.insert(issueAccessGrants).values({ issueId: privateIssue.id, subjectType: "agent", subjectId: input.agent.id, source: "explicit" });
|
|
const [configured] = await db.insert(projectWorkspaces).values({ companyId: issue.companyId, projectId: issue.projectId!,
|
|
name: "Private sibling", cwd: sibling, repoUrl: pathToFileURL(sibling).href }).returning();
|
|
const namespace = join(source, ".paperclip-repositories");
|
|
const originalPrepare = input.services.executionWorkspacesSvc.prepareConfiguredRepositoryCopy;
|
|
const prepare = vi.spyOn(input.services.executionWorkspacesSvc, "prepareConfiguredRepositoryCopy").mockImplementation(async args => {
|
|
const recheck = await originalPrepare(args);
|
|
return async () => {
|
|
const candidates = await readdir(namespace);
|
|
expect(candidates).toHaveLength(1);
|
|
expect(candidates[0]).toContain(".clone-");
|
|
expect(await readFile(join(namespace, candidates[0]!, "private.txt"), "utf8")).toBe("PRIVATE_SIBLING_STAGING_BYTES");
|
|
const retained = await db.select().from(executionWorkspaceRepositoryAccess).where(eq(executionWorkspaceRepositoryAccess.companyId, issue.companyId));
|
|
expect(retained).toEqual([]);
|
|
await rename(sibling, sibling + "-moved");
|
|
await db.update(projectWorkspaces).set({ cwd: sibling + "-moved" }).where(eq(projectWorkspaces.id, configured.id));
|
|
await recheck();
|
|
};
|
|
});
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow(/ENOENT|unavailable|inaccessible|no longer/i);
|
|
expect(prepare).toHaveBeenCalledOnce();
|
|
expect(await readdir(namespace)).toEqual([]);
|
|
expect(await leases(input.run.id)).toEqual([]);
|
|
await db.delete(issueAccessGrants).where(eq(issueAccessGrants.issueId, privateIssue.id));
|
|
|
|
} finally {
|
|
vi.stubEnv("PAPERCLIP_ISSUE_PRIVACY_MODE", previousPrivacyMode);
|
|
}
|
|
});
|
|
|
|
it.each(["file://untrusted-host/private/repository", "file:///private%2Frepository", "file:///private/repository?other", "file:///private/repository#other"])("rejects malformed local repository URL before source access: %s", async repoUrl => {
|
|
const { source, issue } = await fixture();
|
|
const prepareLocalSource = vi.fn();
|
|
await expect(prepareProjectRepositoryWorkspaces({ cwd: source, anchorRepoUrl: null,
|
|
workspaces: [{ id: issue.projectWorkspaceId!, repoUrl, repoRef: null, cwd: null }], prepareLocalSource })).rejects.toThrow();
|
|
expect(prepareLocalSource).not.toHaveBeenCalled();
|
|
expect(await readdir(join(source, ".paperclip-repositories"))).toEqual([]);
|
|
});
|
|
|
|
it("persists and reuses task-owned files without adopting the organizational project as their source", async () => {
|
|
const { input, issue } = await fixture();
|
|
await db.update(issues).set({ projectWorkspaceId: null,
|
|
workspaceSelection: { version: 1, source: "explicit", selection: { kind: "task_directory" } },
|
|
}).where(eq(issues.id, issue.id));
|
|
input.task.issueRef = await readIssue(issue.id);
|
|
const bind = vi.spyOn(input.services.executionWorkspacesSvc, "bindTaskWorkspace");
|
|
const first = await prepareHeartbeatWorkspace(db, input);
|
|
expect(first.persistedExecutionWorkspace).toMatchObject({ projectId: null, projectWorkspaceId: null, sourceIssueId: issue.id });
|
|
expect(bind).toHaveBeenCalled();
|
|
expect((await readIssue(issue.id)).executionWorkspaceId).toBe(first.persistedExecutionWorkspace!.id);
|
|
await writeFile(join(first.executionWorkspace.cwd, "retained.txt"), "task files");
|
|
const [organization] = await db.insert(projects).values({ companyId: issue.companyId, name: "New organization" }).returning();
|
|
await db.update(issues).set({ projectId: organization.id }).where(eq(issues.id, issue.id));
|
|
input.task.issueRef = await readIssue(issue.id);
|
|
await reuse(input, first.persistedExecutionWorkspace!);
|
|
const next = await prepareHeartbeatWorkspace(db, input);
|
|
expect(next.persistedExecutionWorkspace).toMatchObject({ id: first.persistedExecutionWorkspace!.id, projectId: null });
|
|
expect(next.executionWorkspace.cwd).toBe(first.executionWorkspace.cwd);
|
|
expect(await readFile(join(next.executionWorkspace.cwd, "retained.txt"), "utf8")).toBe("task files");
|
|
expect((await readIssue(issue.id)).projectId).toBe(organization.id);
|
|
});
|
|
|
|
it("retains task-directory provenance through shared workspace preparation", async () => {
|
|
const { input, issue, source } = await fixture();
|
|
await db.update(issues).set({ projectId: null, projectWorkspaceId: null,
|
|
workspaceSelection: { version: 1, source: "explicit", selection: { kind: "task_directory" } },
|
|
}).where(eq(issues.id, issue.id));
|
|
input.task.issueRef = await readIssue(issue.id);
|
|
input.task.context = { issueId: issue.id };
|
|
const first = await prepareHeartbeatWorkspace(db, input);
|
|
expect(first.executionWorkspace.source).toBe("task_session");
|
|
expect(first.persistedExecutionWorkspace).toMatchObject({ mode: "shared_workspace", strategyType: "project_primary",
|
|
sourceIssueId: issue.id, projectId: null, projectWorkspaceId: null, repoUrl: null, metadata: { source: "task_session" } });
|
|
await mkdir(join(first.executionWorkspace.cwd, ".git", "info"), { recursive: true });
|
|
await writeFile(join(first.executionWorkspace.cwd, ".git", "info", "exclude"), "/.paperclip-repositories/\n");
|
|
await writeFile(join(first.executionWorkspace.cwd, "original-context.txt"), "Preserve the original task bytes\n");
|
|
await db.update(issues).set({ executionRunId: input.run.id }).where(eq(issues.id, issue.id));
|
|
const taskRepositories = executionWorkspaceService(db);
|
|
const receipt = await taskRepositories.requestTaskRepository({ companyId: issue.companyId, issueId: issue.id,
|
|
actor: { type: "board", source: "local_implicit", userId: "local-board" },
|
|
request: { repository: { kind: "url", url: "https://github.com/public/source" }, requestKey: "retained-task-directory" } });
|
|
const materialize = managedRepositoryCheckout.materializeManagedProjectWorkspace;
|
|
vi.spyOn(managedRepositoryCheckout, "materializeManagedProjectWorkspace").mockImplementation((cwd, options) =>
|
|
materialize(cwd, { ...options, localSource: source }));
|
|
await reuse(input, first.persistedExecutionWorkspace!);
|
|
await db.update(issues).set({ executionRunId: input.run.id }).where(eq(issues.id, issue.id));
|
|
input.task.issueRef = await readIssue(issue.id);
|
|
const next = await prepareHeartbeatWorkspace(db, input);
|
|
expect(next.executionWorkspace.source).toBe("task_session");
|
|
expect(next.persistedExecutionWorkspace!.metadata?.source).toBe("task_session");
|
|
expect(next.persistedExecutionWorkspace!.id).toBe(first.persistedExecutionWorkspace!.id);
|
|
expect(await readFile(join(next.executionWorkspace.cwd, "original-context.txt"), "utf8")).toBe("Preserve the original task bytes\n");
|
|
const [repository] = await taskRepositories.listTaskRepositories(issue.companyId, next.persistedExecutionWorkspace!.id);
|
|
expect(repository).toMatchObject({ id: receipt.operationId, state: "ready" });
|
|
expect(await readFile(join(next.executionWorkspace.cwd, repository.relativePath, "README.md"), "utf8")).toBe("source content\n");
|
|
});
|
|
|
|
it("creates a real isolated worktree and reuses its recorded workspace", async () => {
|
|
const { input, source, issue } = await fixture(true);
|
|
const first = await prepareHeartbeatWorkspace(db, input);
|
|
expect(first.executionWorkspace.cwd).not.toBe(source);
|
|
expect(await readFile(join(first.executionWorkspace.cwd, "README.md"), "utf8")).toBe("source content\n");
|
|
expect(await readIssue(issue.id)).toMatchObject({ executionWorkspaceId: first.persistedExecutionWorkspace!.id, executionWorkspacePreference: "reuse_existing" });
|
|
await writeFile(join(first.executionWorkspace.cwd, "retained.txt"), "keep this work");
|
|
await reuse(input, first.persistedExecutionWorkspace!);
|
|
const next = await prepareHeartbeatWorkspace(db, input);
|
|
expect(next.reusedExecutionWorkspace).toMatchObject({ cwd: first.executionWorkspace.cwd });
|
|
expect(next.persistedExecutionWorkspace!.id).toBe(first.persistedExecutionWorkspace!.id);
|
|
expect(await readFile(join(next.executionWorkspace.cwd, "retained.txt"), "utf8")).toBe("keep this work");
|
|
});
|
|
|
|
it("keeps the admitted recovery workspace when the issue now points elsewhere", async () => {
|
|
const { input, issue } = await fixture(true);
|
|
const first = await prepareHeartbeatWorkspace(db, input);
|
|
await reuse(input, first.persistedExecutionWorkspace!);
|
|
await db.update(issues).set({ executionWorkspaceId: null, projectWorkspaceId: null }).where(eq(issues.id, issue.id));
|
|
input.task.issueRef = { ...input.task.issueRef!, executionWorkspaceId: null, projectWorkspaceId: null };
|
|
input.reuse.nativeRecoveryExecutionWorkspaceId = first.persistedExecutionWorkspace!.id;
|
|
await persistRecoveryInput(input, first.persistedExecutionWorkspace!.id, first.executionWorkspace.cwd);
|
|
const anchor = vi.spyOn(input.services, "resolveReusedGitWorkspaceAnchor");
|
|
const prepareRepositories = vi.spyOn(input.services.executionWorkspacesSvc, "prepareTaskRepositoriesForAdmission");
|
|
const reserveRoot = vi.spyOn(input.services.executionWorkspacesSvc, "assertTaskRepositoryRecoveryBinding");
|
|
const next = await prepareHeartbeatWorkspace(db, input);
|
|
expect(reserveRoot).toHaveBeenCalledWith({ companyId: issue.companyId, issueId: issue.id,
|
|
workspaceId: first.persistedExecutionWorkspace!.id, cwd: first.executionWorkspace.cwd,
|
|
agentId: input.agent.id, runId: input.run.id });
|
|
expect(prepareRepositories).not.toHaveBeenCalled();
|
|
expect(next.persistedExecutionWorkspace!.id).toBe(first.persistedExecutionWorkspace!.id);
|
|
expect(anchor).toHaveBeenCalledWith(expect.objectContaining({ immutableNativeBinding: true, explicitProjectWorkspaceId: first.persistedExecutionWorkspace!.projectWorkspaceId }));
|
|
expect(await readIssue(issue.id)).toMatchObject({ executionWorkspaceId: null, projectWorkspaceId: null });
|
|
});
|
|
|
|
it.each([
|
|
{ transport: "local", descriptor: "absent" },
|
|
{ transport: "ssh", descriptor: "absent" },
|
|
{ transport: "task", descriptor: "absent" },
|
|
{ transport: "daytona", descriptor: "valid" },
|
|
{ transport: "daytona", descriptor: "absent" },
|
|
{ transport: "daytona", descriptor: "corrupt" },
|
|
] as const)("validates the admitted host binding for $transport recovery with $descriptor descriptor", async ({ transport, descriptor }) => {
|
|
const { input, issue } = await fixture(true);
|
|
const first = await prepareHeartbeatWorkspace(db, input);
|
|
await reuse(input, first.persistedExecutionWorkspace!);
|
|
await db.update(issues).set({ executionWorkspaceId: null, projectWorkspaceId: null }).where(eq(issues.id, issue.id));
|
|
input.task.issueRef = { ...input.task.issueRef!, executionWorkspaceId: null, projectWorkspaceId: null };
|
|
input.reuse.nativeRecoveryExecutionWorkspaceId = first.persistedExecutionWorkspace!.id;
|
|
const admittedRoot = join(home, `admitted-${input.run.id}`);
|
|
await mkdir(admittedRoot);
|
|
const remoteCwd = "/remote/admitted-task";
|
|
await persistRecoveryInput(input, first.persistedExecutionWorkspace!.id, transport === "local" ? admittedRoot : remoteCwd);
|
|
const target: AdapterExecutionTarget = transport === "local" ? { kind: "local" }
|
|
: transport === "ssh" ? { kind: "remote", transport: "ssh", remoteCwd,
|
|
spec: { host: "fixture.invalid", port: 22, username: "fixture", remoteWorkspacePath: remoteCwd,
|
|
privateKey: null, knownHosts: null, strictHostKeyChecking: true, remoteCwd } }
|
|
: { kind: "remote", transport: "sandbox", providerKey: "daytona", remoteCwd,
|
|
...(transport === "task" ? { runnerTask: vi.fn() } : {}) };
|
|
const realize = input.services.envOrchestrator.realizeForRun;
|
|
vi.spyOn(input.services.envOrchestrator, "realizeForRun").mockImplementation(async args => ({
|
|
...await realize(args), executionTarget: target,
|
|
}));
|
|
if (descriptor !== "absent") {
|
|
const baseline = await captureDirectorySnapshot(admittedRoot);
|
|
const reference = await nativeWorkspaceSyncInternals.writeDescriptor({
|
|
schema: "paperclip.native-workspace-sync/v1", state: "prepared",
|
|
binding: { companyId: issue.companyId, runId: input.run.id, workspaceId: first.persistedExecutionWorkspace!.id,
|
|
leaseId: "admitted-lease", providerLeaseId: "admitted-provider", localCwd: admittedRoot, remoteCwd },
|
|
baseline: serializeDirectorySnapshot(baseline), baselineSha256: directorySnapshotSha256(baseline),
|
|
gitSnapshot: null, seed: null, createdAt: new Date().toISOString(), finalizedAt: null, finalHostSha256: null, resourceDisposition: null,
|
|
});
|
|
input.run.runnerProfileJson = { ...input.run.runnerProfileJson, nativeWorkspaceSync: reference };
|
|
await db.update(heartbeatRuns).set({ runnerProfileJson: input.run.runnerProfileJson }).where(eq(heartbeatRuns.id, input.run.id));
|
|
if (descriptor === "corrupt") {
|
|
await writeFile(nativeWorkspaceSyncInternals.descriptorPath(input.run.id, reference.descriptorSha256), "{}");
|
|
}
|
|
}
|
|
const reserve = vi.spyOn(input.services.executionWorkspacesSvc, "assertTaskRepositoryRecoveryBinding");
|
|
const prepare = vi.spyOn(input.services.executionWorkspacesSvc, "prepareTaskRepositoriesForAdmission");
|
|
if (descriptor !== "corrupt") {
|
|
const result = await prepareHeartbeatWorkspace(db, input);
|
|
expect(result.executionTarget).toEqual(target);
|
|
if (transport === "ssh" || transport === "task") expect(reserve).not.toHaveBeenCalled();
|
|
else expect(reserve).toHaveBeenCalledWith({ companyId: issue.companyId, issueId: issue.id,
|
|
workspaceId: first.persistedExecutionWorkspace!.id,
|
|
// Without a committed sync descriptor the sandbox never received this
|
|
// run's work, so recovery re-admits the host workspace it will seed from.
|
|
cwd: transport === "daytona" && descriptor === "absent" ? first.executionWorkspace.cwd : admittedRoot,
|
|
agentId: input.agent.id, runId: input.run.id });
|
|
if (transport !== "daytona") {
|
|
await expect(prepareNativeWorkspaceSync({ db, runId: input.run.id, companyId: issue.companyId,
|
|
workspaceId: first.persistedExecutionWorkspace!.id, workspaceLocalDir: first.executionWorkspace.cwd,
|
|
target, lease: result.activeEnvironmentLease!.lease })).resolves.toBeNull();
|
|
}
|
|
expect(admittedRoot).not.toBe(first.executionWorkspace.cwd);
|
|
} else {
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow("native_workspace_sync_descriptor_digest_mismatch");
|
|
expect(reserve).not.toHaveBeenCalled();
|
|
expect(await leases(input.run.id)).toMatchObject([{ status: "active" }]);
|
|
}
|
|
expect(prepare).not.toHaveBeenCalled();
|
|
expect((await readIssue(issue.id)).executionWorkspaceId).toBeNull();
|
|
});
|
|
|
|
it("removes a newly created worktree when persistence fails and preserves the original error", async () => {
|
|
const { input, source } = await fixture(true);
|
|
const error = new Error("workspace persistence unavailable");
|
|
let createdCwd = "";
|
|
vi.spyOn(input.services.executionWorkspacesSvc, "create").mockImplementation(async args => { createdCwd = args.cwd!; throw error; });
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toBe(error);
|
|
expect(createdCwd).not.toBe("");
|
|
await expect(access(createdCwd)).rejects.toThrow();
|
|
expect(await readFile(join(source, "README.md"), "utf8")).toBe("source content\n");
|
|
expect(await leases(input.run.id)).toHaveLength(0);
|
|
});
|
|
|
|
it("preserves a reused worktree when its metadata update fails", async () => {
|
|
const { input } = await fixture(true);
|
|
const first = await prepareHeartbeatWorkspace(db, input);
|
|
await reuse(input, first.persistedExecutionWorkspace!);
|
|
await writeFile(join(first.executionWorkspace.cwd, "retained.txt"), "unsaved work");
|
|
const error = new Error("workspace update unavailable");
|
|
vi.spyOn(input.services.executionWorkspacesSvc, "update").mockRejectedValueOnce(error);
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toBe(error);
|
|
expect(await readFile(join(first.executionWorkspace.cwd, "retained.txt"), "utf8")).toBe("unsaved work");
|
|
expect(await leases(input.run.id)).toHaveLength(0);
|
|
});
|
|
|
|
it.each(["acquire", "realize", "ownership"] as const)("preserves %s errors and durable lease state for outer cleanup", async failure => {
|
|
const { input, envOrchestrator } = await fixture();
|
|
const error = new Error(`${failure} failed`);
|
|
if (failure === "acquire") vi.spyOn(envOrchestrator, "acquireForRun").mockRejectedValueOnce(error);
|
|
if (failure === "realize") vi.spyOn(envOrchestrator, "realizeForRun").mockRejectedValueOnce(error);
|
|
if (failure === "ownership") vi.mocked(input.controllerLease.assertOwned).mockResolvedValueOnce(undefined).mockRejectedValueOnce(error);
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toBe(error);
|
|
expect(input.nativeRunnerPreparationSpans.at(-1)).toMatchObject({ name: failure === "realize" ? "environment.workspace.realize" : "environment.acquire", outcome: "failed" });
|
|
expect((await readRun(input.run.id)).status).toBe("running");
|
|
expect(await leases(input.run.id)).toHaveLength(failure === "acquire" ? 0 : 1);
|
|
if (failure !== "acquire") {
|
|
expect(await leases(input.run.id)).toMatchObject([{ status: "active" }]);
|
|
// The production executor terminalizes first, then releases by durable run ID.
|
|
await db.update(heartbeatRuns).set({ status: "failed" }).where(eq(heartbeatRuns.id, input.run.id));
|
|
await envOrchestrator.releaseForRun({ heartbeatRunId: input.run.id, companyId: input.run.companyId, agentId: input.agent.id, status: "released" });
|
|
expect(await leases(input.run.id)).toMatchObject([{ status: "released" }]);
|
|
}
|
|
});
|
|
|
|
it("checks the acquired environment allowlist before realization", async () => {
|
|
const { input, envOrchestrator } = await fixture();
|
|
input.policy.executionPolicy = { managedSandboxOnly: true };
|
|
const realize = vi.spyOn(envOrchestrator, "realizeForRun");
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow("forbids local execution");
|
|
expect(realize).not.toHaveBeenCalled();
|
|
expect(await leases(input.run.id)).toMatchObject([{ status: "active" }]);
|
|
});
|
|
|
|
it("rejects low-trust local execution before resolving or provisioning a workspace", async () => {
|
|
const { input } = await fixture(true);
|
|
input.policy.trustPreset = {
|
|
kind: "low_trust_review", preset: "low_trust_review", sourcePresets: {},
|
|
boundary: { mode: "low_trust_review", companyId: input.agent.companyId, rootIssueId: input.task.issueId! },
|
|
};
|
|
const resolve = vi.spyOn(input.services, "resolveWorkspaceForRun");
|
|
const create = vi.spyOn(input.services.executionWorkspacesSvc, "create");
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow("requires a sandbox environment");
|
|
expect(resolve).not.toHaveBeenCalled();
|
|
expect(create).not.toHaveBeenCalled();
|
|
expect(await leases(input.run.id)).toHaveLength(0);
|
|
});
|
|
|
|
it("rejects remote recovery without its retained sync binding before acquiring a lease", async () => {
|
|
const { input, envOrchestrator } = await fixture();
|
|
input.reuse.runOptions.nativeRestartRecovery = {
|
|
kind: "reattach_remote_runner", runId: input.run.id, leaseOwner: "controller",
|
|
controllerGeneration: 1, providerAttempt: 1, restartKind: "hard", recoveryRequestId: null,
|
|
remote: { providerLeaseId: "original-provider-lease", remoteCwd: "/original" },
|
|
};
|
|
const acquire = vi.spyOn(envOrchestrator, "acquireForRun");
|
|
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow("native_remote_recovery_lease_mismatch");
|
|
expect(acquire).not.toHaveBeenCalled();
|
|
expect(await leases(input.run.id)).toHaveLength(0);
|
|
expect(input.nativeRunnerPreparationSpans).toMatchObject([{ name: "environment.acquire", outcome: "failed" }]);
|
|
});
|
|
|
|
it("binds a replacement workspace from realization and snapshots the restore lease", async () => {
|
|
const { input, envOrchestrator, issue } = await fixture();
|
|
const realize = envOrchestrator.realizeForRun;
|
|
vi.spyOn(envOrchestrator, "realizeForRun").mockImplementation(async args => {
|
|
const result = await realize(args);
|
|
const workspace = await input.services.executionWorkspacesSvc.create({ companyId: issue.companyId, projectId: issue.projectId!, projectWorkspaceId: issue.projectWorkspaceId, sourceIssueId: issue.id, name: "Realized workspace", mode: "shared_workspace", strategyType: "project_primary", cwd: args.executionWorkspace.cwd });
|
|
return { ...result, persistedExecutionWorkspace: workspace, lease: { ...result.lease, metadata: { remoteCwd: "/original" } } };
|
|
});
|
|
const result = await prepareHeartbeatWorkspace(db, input);
|
|
expect((await readIssue(issue.id)).executionWorkspaceId).toBe(result.persistedExecutionWorkspace!.id);
|
|
expect(result.persistedExecutionWorkspace!.name).toBe("Realized workspace");
|
|
result.activeEnvironmentLease!.lease.metadata!.remoteCwd = "/rebound";
|
|
expect(result.workspaceRestoreSource!.metadata?.remoteCwd).toBe("/original");
|
|
});
|
|
});
|