mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-21 08:22:20 +02:00
refactor(heartbeat): extract workspace preparation (#15807)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The heartbeat service prepares the workspace and dispatches each agent run. > - After the runtime extraction, `heartbeat.ts` still had 6,417 lines. > - Workspace setup occupied one contiguous 857-line block inside `executeRun`. > - This pull request moves that block into one workspace preparation module. > - The executor keeps dispatch gates and terminal cleanup. Reviewers can inspect workspace setup as one phase. ## Linked Issues or Issue Description Refs #15803 and #15578. This continues the heartbeat refactor series. The search found no open duplicate workspace preparation extraction. **What existing behavior does this improve?** It makes heartbeat workspace preparation easier to maintain. It does not change the workspace policy or execution behavior. **Current behavior** `executeRun` contains workspace resolution, freshness decisions, provisioning, persistence, issue binding, environment acquisition, and realization inline. **Proposed behavior** `prepareHeartbeatWorkspace` owns that sequence in `server/src/services/heartbeat/workspace-preparation.ts`. `heartbeat.ts` calls it at the same point and uses its returned workspace, lease, target, and metadata. **Reason and benefit** The main file falls from 6,417 to 5,597 lines. The workspace phase has a typed boundary and direct regression tests. **Breaking changes** None. There are no schema, API, feature-flag, or execution-policy changes. ## What Changed - Extract the 857-line workspace preparation block without changing its operations or their order. - Preserve native recovery bindings, workspace persistence failure cleanup, issue rebinding after realization, and the cloned restore lease. - Keep environment selection, dispatch admission, terminalization, and final lease release in `heartbeat.ts`. - Add 12 boundary tests with disposable PostgreSQL, real Git repositories, and real local environment leases. - Make the existing Stop and plan-only recovery tests await executor completion instead of short polling deadlines. Assert exactly one continuation for the specific plan-only source run. - Keep server-only Node preload hooks out of the deterministic process adapter used by Stop browser acceptance. - Refresh workspace recovery browser acceptance for the current task composer, and give its disposable test-drive an explicit 10-second snapshot deadline. Production snapshot settings stay unchanged. - Document the workspace preparation boundary in `doc/DEVELOPING.md`. ## Verification - AST comparison: all 857 moved lines match the original block. The remaining module body differs only at the extraction call. - `pnpm build`: passed. - `pnpm -r typecheck`: passed. - Focused regression coverage: 651 distinct cases across eight suites passed across targeted runs, including all 12 new boundary tests. The complete 400-test recovery suite passed together on the final commit. - Workspace bootstrap browser acceptance: both journeys passed (4.2 minutes), including preserved source edits, ignored-file exclusion, bounded retries, and persistence across reload. - Stop/queue browser acceptance: both process and native runner journeys passed (2.2 minutes), covering queueing, Stop, subtree pause/cancel, resume, and unrelated-task isolation. - [Full repository CI](https://github.com/paperclipai/paperclip/actions/runs/38062408589): all 47 jobs passed, including the general and serialized test suites, eight browser shards, build, typecheck, native runner checks, and canary dry run. - The local `pnpm test:run` was interrupted to prioritize the recovery regression; the complete repository suite was verified in CI. - Greptile: 5/5 on `c28e698b885cf5fd12f36f7b4052e7a2e6f879d7`, with no open review threads. ## Risks - This code writes durable workspace bindings and acquires environment leases. A changed operation order could break recovery or cleanup after partial setup. The extraction preserves the original statements and exception boundaries. - The new tests cover creation, reuse, immutable recovery bindings, persistence failure, acquisition and realization failure, controller ownership loss, isolation, execution policy, and lease snapshot independence. - Browser acceptance uses real application services with deterministic local workers and a fake Codex protocol provider. It does not qualify live cloud providers. ## Model Used OpenAI GPT-6 through Codex, with code editing, shell execution, and test tools. The exact serving model identifier and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
19e9facc5b
commit
07fee85eaf
9 files changed
+1445
-917
No files matched your search
@@ -1360,6 +1360,14 @@ lifecycle and ownership callbacks update its teardown state immediately, includi
|
||||
when preparation later throws. Public sandbox lifecycle and model fallback helpers
|
||||
remain re-exported by `heartbeat.ts`.
|
||||
|
||||
Workspace preparation is in `server/src/services/heartbeat/workspace-preparation.ts`.
|
||||
`prepareHeartbeatWorkspace` resolves and provisions the workspace, persists its
|
||||
metadata and issue binding, then acquires and realizes the execution environment.
|
||||
It preserves the immutable workspace binding used by native recovery and records
|
||||
preparation spans even when acquisition or realization fails. The executor retains
|
||||
environment selection, dispatch gates, and terminal lease cleanup. Persistence
|
||||
failures still clean newly created workspace artifacts before propagating.
|
||||
|
||||
Runtime execution is in `server/src/services/heartbeat/runtime-execution.ts`.
|
||||
`executeHeartbeatRuntime` dispatches a prepared native session or legacy adapter,
|
||||
captures usage, finalizes workspace operations, and releases gateway tokens and
|
||||
|
||||
@@ -8455,12 +8455,10 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => {
|
||||
.set({ status: "paused" })
|
||||
.where(eq(agents.id, agentId));
|
||||
release();
|
||||
if (next!)
|
||||
await vi.waitFor(async () =>
|
||||
expect((await heartbeat.getRun(next!.id))?.status).not.toBe(
|
||||
"running",
|
||||
),
|
||||
);
|
||||
// Wait for the released executor and its cleanup, not the default
|
||||
// one-second polling window. The next case must not inherit live work.
|
||||
await heartbeat.drainActiveRunExecutions();
|
||||
if (next!) expect((await heartbeat.getRun(next!.id))?.status).not.toBe("running");
|
||||
}
|
||||
},
|
||||
);
|
||||
@@ -12100,7 +12098,9 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => {
|
||||
const legacyReviewBefore = withLegacyReview
|
||||
? await db.select().from(issues).where(eq(issues.id, legacyReviewId))
|
||||
: [];
|
||||
mockAdapterExecute.mockImplementationOnce(async () => {
|
||||
let planOnlyRunId = "";
|
||||
mockAdapterExecute.mockImplementationOnce(async (input?: unknown) => {
|
||||
planOnlyRunId = (input as { runId: string }).runId;
|
||||
if (withLegacyReview) {
|
||||
// These pre-dispatch cancellations used to satisfy both the no-comment
|
||||
// and churn thresholds and suppress an otherwise valid continuation.
|
||||
@@ -12124,17 +12124,23 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => {
|
||||
await heartbeat.reconcileStrandedAssignedIssues();
|
||||
await heartbeat.promoteDueScheduledRetries(new Date(Date.now() + 31_000));
|
||||
await heartbeat.resumeQueuedRuns();
|
||||
// The continuation is produced by completion; wait for that durable boundary.
|
||||
await heartbeat.drainActiveRunExecutions();
|
||||
|
||||
const livenessWake = await waitForValue(async () => {
|
||||
expect(planOnlyRunId).not.toBe("");
|
||||
const livenessWakes = await waitForValue(async () => {
|
||||
const rows = await db
|
||||
.select()
|
||||
.from(agentWakeupRequests)
|
||||
.where(eq(agentWakeupRequests.agentId, agentId));
|
||||
return (
|
||||
rows.find((row) => row.reason === "issue_disposition_repair") ?? null
|
||||
);
|
||||
// Follow-up repairs can already exist. Match this plan-only result,
|
||||
// rather than relying on the database's unordered row order.
|
||||
const matching = rows.filter((row) => row.reason === "issue_disposition_repair"
|
||||
&& row.payload?.retryOfRunId === planOnlyRunId);
|
||||
return matching.length > 0 ? matching : null;
|
||||
});
|
||||
expect(livenessWake).toBeTruthy();
|
||||
expect(livenessWakes).toHaveLength(1);
|
||||
const livenessWake = livenessWakes![0]!;
|
||||
expect(livenessWake?.payload).toMatchObject({
|
||||
issueId,
|
||||
dispositionRepairAttempt: 1,
|
||||
@@ -12143,7 +12149,7 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => {
|
||||
const sourceRunId = (
|
||||
livenessWake?.payload as Record<string, unknown> | null
|
||||
)?.retryOfRunId;
|
||||
expect(sourceRunId).toBeTruthy();
|
||||
expect(sourceRunId).toBe(planOnlyRunId);
|
||||
const sourceRun = await db
|
||||
.select()
|
||||
.from(heartbeatRuns)
|
||||
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,236 @@
|
||||
import { execFile as execFileCallback } from "node:child_process";
|
||||
import { mkdtemp, mkdir, readFile, rm, writeFile, access } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { promisify } from "node:util";
|
||||
import { agents, companies, createDb, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, environmentLeases, heartbeatRuns, issues, projects, projectWorkspaces, type Db } from "@paperclipai/db";
|
||||
import { eq, sql } from "drizzle-orm";
|
||||
import { afterAll, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
|
||||
import { environmentRunOrchestrator } from "../environment-run-orchestrator.js";
|
||||
import { environmentService } from "../environments.js";
|
||||
import { executionWorkspaceService } from "../execution-workspaces.js";
|
||||
import { instanceSettingsService } from "../instance-settings.js";
|
||||
import { issueService } from "../issues.js";
|
||||
import { workspaceOperationService } from "../workspace-operations.js";
|
||||
import { createHeartbeatWorkspaceResolver, resolveExecutionWorkspaceReuseRequestForIssue } from "./workspaces.js";
|
||||
import { prepareHeartbeatWorkspace, type HeartbeatWorkspacePreparationInput } from "./workspace-preparation.js";
|
||||
|
||||
const execFile = promisify(execFileCallback);
|
||||
const support = await getEmbeddedPostgresTestSupport();
|
||||
|
||||
describe.skipIf(!support.supported)("heartbeat workspace preparation boundary", () => {
|
||||
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
||||
let db: Db;
|
||||
let home: string;
|
||||
|
||||
beforeAll(async () => {
|
||||
home = await mkdtemp(join(tmpdir(), "paperclip-workspace-preparation-"));
|
||||
vi.stubEnv("PAPERCLIP_HOME", home);
|
||||
database = await startEmbeddedPostgresTestDatabase("paperclip-workspace-preparation");
|
||||
db = createDb(database.connectionString);
|
||||
await db.execute(sql`set client_min_messages = warning`);
|
||||
}, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS);
|
||||
afterAll(async () => {
|
||||
await db?.$client.end();
|
||||
await database?.cleanup();
|
||||
vi.unstubAllEnvs();
|
||||
if (home) await rm(home, { recursive: true, force: true });
|
||||
});
|
||||
beforeEach(async () => {
|
||||
vi.restoreAllMocks();
|
||||
await db.execute(sql`truncate table companies restart identity cascade`);
|
||||
await instanceSettingsService(db).updateExperimental({ enableIsolatedWorkspaces: true });
|
||||
});
|
||||
|
||||
async function fixture(isolated = false) {
|
||||
const [company] = await db.insert(companies).values({ name: "Workspace preparation", issuePrefix: "WSP" }).returning();
|
||||
const [agent] = await db.insert(agents).values({ companyId: company.id, name: "Workspace agent", status: "running", adapterType: "process" }).returning();
|
||||
const [project] = await db.insert(projects).values({ companyId: company.id, name: "Source project" }).returning();
|
||||
const source = join(home, project.id);
|
||||
await mkdir(source);
|
||||
await execFile("git", ["init", "--initial-branch=main", source]);
|
||||
await execFile("git", ["-C", source, "config", "user.email", "test@example.com"]);
|
||||
await execFile("git", ["-C", source, "config", "user.name", "Workspace test"]);
|
||||
await writeFile(join(source, "README.md"), "source content\n");
|
||||
await execFile("git", ["-C", source, "add", "."]);
|
||||
await execFile("git", ["-C", source, "commit", "-m", "Source"]);
|
||||
const [projectWorkspace] = await db.insert(projectWorkspaces).values({ companyId: company.id, projectId: project.id, name: "Source", cwd: source, isPrimary: true }).returning();
|
||||
const [issue] = await db.insert(issues).values({ companyId: company.id, title: "Prepare this workspace", identifier: "WSP-1", projectId: project.id, projectWorkspaceId: projectWorkspace.id, assigneeAgentId: agent.id }).returning();
|
||||
const [run] = await db.insert(heartbeatRuns).values({ companyId: company.id, agentId: agent.id, issueId: issue.id, scopeKind: "issue", status: "running", runtimeMode: "legacy" }).returning();
|
||||
const localEnvironment = await environmentService(db).ensureLocalEnvironment(company.id);
|
||||
const envOrchestrator = environmentRunOrchestrator(db);
|
||||
const resolver = createHeartbeatWorkspaceResolver(db);
|
||||
const input: HeartbeatWorkspacePreparationInput = {
|
||||
run, agent,
|
||||
task: { issueRef: issue, issueId: issue.id, context: { issueId: issue.id, projectId: project.id }, executionProjectId: project.id, responsibleUserId: null, previousSessionParams: null },
|
||||
policy: {
|
||||
trustPreset: { kind: "standard", preset: "standard", boundary: null, sourcePresets: {} },
|
||||
isolatedWorkspacesEnabled: true,
|
||||
effectiveExecutionWorkspaceMode: isolated ? "isolated_workspace" : "shared_workspace",
|
||||
requestedExecutionWorkspaceMode: isolated ? "isolated_workspace" : "shared_workspace",
|
||||
useIsolatedTaskDirectory: false, nativeChatWorkspaceScope: null,
|
||||
projectExecutionWorkspacePolicy: null, issueExecutionWorkspaceSettings: null, environmentExecutionWorkspaceSettings: null,
|
||||
executionPolicy: { executionMode: "any" },
|
||||
},
|
||||
environment: { selectedEnvironmentId: localEnvironment.id, localEnvironment, selectedEnvironmentForConfig: localEnvironment, environmentResolution: { environmentId: localEnvironment.id, source: "default" }, resolvedInstanceSettings: await instanceSettingsService(db).get() },
|
||||
config: { mergedConfig: isolated ? { workspaceStrategy: { type: "git_worktree", baseRef: "main", branchTemplate: "test-{{issue.identifier}}" } } : {}, configSnapshot: null, secretManifest: [] },
|
||||
reuse: { requestedShouldReuseExisting: false, existingExecutionWorkspace: null, reusableExistingExecutionWorkspace: null, workspaceReuseRequest: resolveExecutionWorkspaceReuseRequestForIssue({ issueExecutionWorkspaceId: null, issueExecutionWorkspacePreference: null, existingExecutionWorkspaceStatus: null }), nativeRecoveryExecutionWorkspaceId: null, persistedNativeExecutionInput: null, isDotRun: false, runOptions: {} },
|
||||
services: { envOrchestrator, executionWorkspacesSvc: executionWorkspaceService(db), workspaceOperationsSvc: workspaceOperationService(db), issuesSvc: issueService(db), ...resolver, appendRunEvent: vi.fn() },
|
||||
controllerLease: { assertOwned: vi.fn(async () => {}) }, nativeRunnerPreparationSpans: [],
|
||||
};
|
||||
return { input, source, issue, run, envOrchestrator };
|
||||
}
|
||||
const readIssue = (id: string) => db.select().from(issues).where(eq(issues.id, id)).then(rows => rows[0]!);
|
||||
const readRun = (id: string) => db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, id)).then(rows => rows[0]!);
|
||||
const leases = (id: string) => db.select().from(environmentLeases).where(eq(environmentLeases.heartbeatRunId, id));
|
||||
|
||||
async function reuse(input: HeartbeatWorkspacePreparationInput, workspace: NonNullable<Awaited<ReturnType<typeof prepareHeartbeatWorkspace>>["persistedExecutionWorkspace"]>) {
|
||||
input.reuse.requestedShouldReuseExisting = true;
|
||||
input.reuse.existingExecutionWorkspace = workspace;
|
||||
input.reuse.reusableExistingExecutionWorkspace = workspace;
|
||||
input.reuse.workspaceReuseRequest = resolveExecutionWorkspaceReuseRequestForIssue({ issueExecutionWorkspaceId: workspace.id, issueExecutionWorkspacePreference: "reuse_existing", existingExecutionWorkspaceStatus: workspace.status });
|
||||
input.task.issueRef = { ...input.task.issueRef!, executionWorkspaceId: workspace.id, executionWorkspacePreference: "reuse_existing" };
|
||||
const [run] = await db.insert(heartbeatRuns).values({ companyId: input.run.companyId, agentId: input.agent.id, issueId: input.task.issueId, scopeKind: "issue", status: "running", runtimeMode: "legacy" }).returning();
|
||||
input.run = run;
|
||||
}
|
||||
|
||||
it("persists the workspace and issue binding before acquiring its environment", async () => {
|
||||
const { input, source, issue, envOrchestrator } = await fixture();
|
||||
const acquire = envOrchestrator.acquireForRun;
|
||||
vi.spyOn(envOrchestrator, "acquireForRun").mockImplementation(async args => {
|
||||
expect((await readIssue(issue.id)).executionWorkspaceId).toBe(args.persistedExecutionWorkspace?.id);
|
||||
expect((await readRun(input.run.id)).contextSnapshot?.executionWorkspaceId).toBe(args.persistedExecutionWorkspace?.id);
|
||||
return acquire(args);
|
||||
});
|
||||
const result = await prepareHeartbeatWorkspace(db, input);
|
||||
expect(result.executionWorkspace.cwd).toBe(source);
|
||||
expect(result.persistedExecutionWorkspace).toMatchObject({ companyId: issue.companyId, projectId: issue.projectId, projectWorkspaceId: issue.projectWorkspaceId, mode: "shared_workspace" });
|
||||
expect(await leases(input.run.id)).toMatchObject([{ status: "active", executionWorkspaceId: result.persistedExecutionWorkspace!.id }]);
|
||||
expect(input.nativeRunnerPreparationSpans.map(span => span.name)).toEqual(["environment.acquire", "environment.workspace.realize"]);
|
||||
expect(input.controllerLease.assertOwned).toHaveBeenCalledTimes(2);
|
||||
});
|
||||
|
||||
it("creates a real isolated worktree and reuses its recorded workspace", async () => {
|
||||
const { input, source, issue } = await fixture(true);
|
||||
const first = await prepareHeartbeatWorkspace(db, input);
|
||||
expect(first.executionWorkspace.cwd).not.toBe(source);
|
||||
expect(await readFile(join(first.executionWorkspace.cwd, "README.md"), "utf8")).toBe("source content\n");
|
||||
expect(await readIssue(issue.id)).toMatchObject({ executionWorkspaceId: first.persistedExecutionWorkspace!.id, executionWorkspacePreference: "reuse_existing" });
|
||||
await writeFile(join(first.executionWorkspace.cwd, "retained.txt"), "keep this work");
|
||||
await reuse(input, first.persistedExecutionWorkspace!);
|
||||
const next = await prepareHeartbeatWorkspace(db, input);
|
||||
expect(next.reusedExecutionWorkspace).toMatchObject({ cwd: first.executionWorkspace.cwd });
|
||||
expect(next.persistedExecutionWorkspace!.id).toBe(first.persistedExecutionWorkspace!.id);
|
||||
expect(await readFile(join(next.executionWorkspace.cwd, "retained.txt"), "utf8")).toBe("keep this work");
|
||||
});
|
||||
|
||||
it("keeps the admitted recovery workspace when the issue now points elsewhere", async () => {
|
||||
const { input, issue } = await fixture(true);
|
||||
const first = await prepareHeartbeatWorkspace(db, input);
|
||||
await reuse(input, first.persistedExecutionWorkspace!);
|
||||
await db.update(issues).set({ executionWorkspaceId: null, projectWorkspaceId: null }).where(eq(issues.id, issue.id));
|
||||
input.task.issueRef = { ...input.task.issueRef!, executionWorkspaceId: null, projectWorkspaceId: null };
|
||||
input.reuse.nativeRecoveryExecutionWorkspaceId = first.persistedExecutionWorkspace!.id;
|
||||
const anchor = vi.spyOn(input.services, "resolveReusedGitWorkspaceAnchor");
|
||||
const next = await prepareHeartbeatWorkspace(db, input);
|
||||
expect(next.persistedExecutionWorkspace!.id).toBe(first.persistedExecutionWorkspace!.id);
|
||||
expect(anchor).toHaveBeenCalledWith(expect.objectContaining({ immutableNativeBinding: true, explicitProjectWorkspaceId: first.persistedExecutionWorkspace!.projectWorkspaceId }));
|
||||
expect(await readIssue(issue.id)).toMatchObject({ executionWorkspaceId: null, projectWorkspaceId: null });
|
||||
});
|
||||
|
||||
it("removes a newly created worktree when persistence fails and preserves the original error", async () => {
|
||||
const { input, source } = await fixture(true);
|
||||
const error = new Error("workspace persistence unavailable");
|
||||
let createdCwd = "";
|
||||
vi.spyOn(input.services.executionWorkspacesSvc, "create").mockImplementation(async args => { createdCwd = args.cwd!; throw error; });
|
||||
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toBe(error);
|
||||
expect(createdCwd).not.toBe("");
|
||||
await expect(access(createdCwd)).rejects.toThrow();
|
||||
expect(await readFile(join(source, "README.md"), "utf8")).toBe("source content\n");
|
||||
expect(await leases(input.run.id)).toHaveLength(0);
|
||||
});
|
||||
|
||||
it("preserves a reused worktree when its metadata update fails", async () => {
|
||||
const { input } = await fixture(true);
|
||||
const first = await prepareHeartbeatWorkspace(db, input);
|
||||
await reuse(input, first.persistedExecutionWorkspace!);
|
||||
await writeFile(join(first.executionWorkspace.cwd, "retained.txt"), "unsaved work");
|
||||
const error = new Error("workspace update unavailable");
|
||||
vi.spyOn(input.services.executionWorkspacesSvc, "update").mockRejectedValueOnce(error);
|
||||
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toBe(error);
|
||||
expect(await readFile(join(first.executionWorkspace.cwd, "retained.txt"), "utf8")).toBe("unsaved work");
|
||||
expect(await leases(input.run.id)).toHaveLength(0);
|
||||
});
|
||||
|
||||
it.each(["acquire", "realize", "ownership"] as const)("preserves %s errors and durable lease state for outer cleanup", async failure => {
|
||||
const { input, envOrchestrator } = await fixture();
|
||||
const error = new Error(`${failure} failed`);
|
||||
if (failure === "acquire") vi.spyOn(envOrchestrator, "acquireForRun").mockRejectedValueOnce(error);
|
||||
if (failure === "realize") vi.spyOn(envOrchestrator, "realizeForRun").mockRejectedValueOnce(error);
|
||||
if (failure === "ownership") vi.mocked(input.controllerLease.assertOwned).mockResolvedValueOnce(undefined).mockRejectedValueOnce(error);
|
||||
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toBe(error);
|
||||
expect(input.nativeRunnerPreparationSpans.at(-1)).toMatchObject({ name: failure === "realize" ? "environment.workspace.realize" : "environment.acquire", outcome: "failed" });
|
||||
expect((await readRun(input.run.id)).status).toBe("running");
|
||||
expect(await leases(input.run.id)).toHaveLength(failure === "acquire" ? 0 : 1);
|
||||
if (failure !== "acquire") {
|
||||
expect(await leases(input.run.id)).toMatchObject([{ status: "active" }]);
|
||||
// The production executor terminalizes first, then releases by durable run ID.
|
||||
await db.update(heartbeatRuns).set({ status: "failed" }).where(eq(heartbeatRuns.id, input.run.id));
|
||||
await envOrchestrator.releaseForRun({ heartbeatRunId: input.run.id, companyId: input.run.companyId, agentId: input.agent.id, status: "released" });
|
||||
expect(await leases(input.run.id)).toMatchObject([{ status: "released" }]);
|
||||
}
|
||||
});
|
||||
|
||||
it("checks the acquired environment allowlist before realization", async () => {
|
||||
const { input, envOrchestrator } = await fixture();
|
||||
input.policy.executionPolicy = { managedSandboxOnly: true };
|
||||
const realize = vi.spyOn(envOrchestrator, "realizeForRun");
|
||||
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow("forbids local execution");
|
||||
expect(realize).not.toHaveBeenCalled();
|
||||
expect(await leases(input.run.id)).toMatchObject([{ status: "active" }]);
|
||||
});
|
||||
|
||||
it("rejects low-trust local execution before resolving or provisioning a workspace", async () => {
|
||||
const { input } = await fixture(true);
|
||||
input.policy.trustPreset = {
|
||||
kind: "low_trust_review", preset: "low_trust_review", sourcePresets: {},
|
||||
boundary: { mode: "low_trust_review", companyId: input.agent.companyId, rootIssueId: input.task.issueId! },
|
||||
};
|
||||
const resolve = vi.spyOn(input.services, "resolveWorkspaceForRun");
|
||||
const create = vi.spyOn(input.services.executionWorkspacesSvc, "create");
|
||||
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow("requires a sandbox environment");
|
||||
expect(resolve).not.toHaveBeenCalled();
|
||||
expect(create).not.toHaveBeenCalled();
|
||||
expect(await leases(input.run.id)).toHaveLength(0);
|
||||
});
|
||||
|
||||
it("rejects remote recovery without its retained sync binding before acquiring a lease", async () => {
|
||||
const { input, envOrchestrator } = await fixture();
|
||||
input.reuse.runOptions.nativeRestartRecovery = {
|
||||
kind: "reattach_remote_runner", runId: input.run.id, leaseOwner: "controller",
|
||||
controllerGeneration: 1, providerAttempt: 1, restartKind: "hard", recoveryRequestId: null,
|
||||
remote: { providerLeaseId: "original-provider-lease", remoteCwd: "/original" },
|
||||
};
|
||||
const acquire = vi.spyOn(envOrchestrator, "acquireForRun");
|
||||
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow("native_remote_recovery_lease_mismatch");
|
||||
expect(acquire).not.toHaveBeenCalled();
|
||||
expect(await leases(input.run.id)).toHaveLength(0);
|
||||
expect(input.nativeRunnerPreparationSpans).toMatchObject([{ name: "environment.acquire", outcome: "failed" }]);
|
||||
});
|
||||
|
||||
it("binds a replacement workspace from realization and snapshots the restore lease", async () => {
|
||||
const { input, envOrchestrator, issue } = await fixture();
|
||||
const realize = envOrchestrator.realizeForRun;
|
||||
vi.spyOn(envOrchestrator, "realizeForRun").mockImplementation(async args => {
|
||||
const result = await realize(args);
|
||||
const workspace = await input.services.executionWorkspacesSvc.create({ companyId: issue.companyId, projectId: issue.projectId!, projectWorkspaceId: issue.projectWorkspaceId, sourceIssueId: issue.id, name: "Realized workspace", mode: "shared_workspace", strategyType: "project_primary", cwd: args.executionWorkspace.cwd });
|
||||
return { ...result, persistedExecutionWorkspace: workspace, lease: { ...result.lease, metadata: { remoteCwd: "/original" } } };
|
||||
});
|
||||
const result = await prepareHeartbeatWorkspace(db, input);
|
||||
expect((await readIssue(issue.id)).executionWorkspaceId).toBe(result.persistedExecutionWorkspace!.id);
|
||||
expect(result.persistedExecutionWorkspace!.name).toBe("Realized workspace");
|
||||
result.activeEnvironmentLease.lease.metadata!.remoteCwd = "/rebound";
|
||||
expect(result.workspaceRestoreSource.metadata?.remoteCwd).toBe("/original");
|
||||
});
|
||||
});
|
||||
File diff suppressed because it is too large.
Load diff
@@ -205,6 +205,9 @@ for (const adapter of ["process", "paperclip_runner"] as const) {
|
||||
"-e",
|
||||
"console.log('stop fixture ready'); setInterval(() => console.log('working'), 200);",
|
||||
],
|
||||
// The child only runs this script; server-only fixture
|
||||
// loaders must not follow NODE_OPTIONS into the adapter.
|
||||
env: { NODE_OPTIONS: "" },
|
||||
graceSec: 1,
|
||||
}
|
||||
: { provider: "codex", model: "gpt-5.1-codex-mini" },
|
||||
|
||||
@@ -9,14 +9,13 @@ timeout only when the test explicitly arms a marker inside its repository.
|
||||
From a development worktree, start the foreground instance:
|
||||
|
||||
```sh
|
||||
BOOTSTRAP_FIXTURE_KEY=not-a-provider-key \
|
||||
PATH="$PWD/tests/e2e/workspace-bootstrap/bin:$PATH" \
|
||||
NODE_ENV=test \
|
||||
node cli/node_modules/tsx/dist/cli.mjs cli/src/index.ts test-drive \
|
||||
--harness codex --api-key-env BOOTSTRAP_FIXTURE_KEY \
|
||||
--company-name 'Workspace Recovery QA' --no-browser
|
||||
node cli/node_modules/tsx/dist/cli.mjs tests/e2e/workspace-bootstrap/launch.ts
|
||||
```
|
||||
|
||||
The launcher creates an isolated test-drive, installs the Git shim, and sets a
|
||||
10-second snapshot deadline after test-drive clears inherited configuration.
|
||||
Production's longer snapshot deadline would outlast the injected failure.
|
||||
|
||||
Set `BOOTSTRAP_REAL_GIT` to the absolute Git executable if it is not
|
||||
`/usr/bin/git`. Do not point this fixture at a normal development or production
|
||||
instance. The seeder checks the loopback host, trusted-local mode, and company
|
||||
|
||||
@@ -0,0 +1,18 @@
|
||||
import path from "node:path";
|
||||
import { prepareTestDriveEnvironment, testDriveCommand } from "../../../cli/src/commands/test-drive.js";
|
||||
|
||||
const options = {
|
||||
harness: "codex",
|
||||
apiKeyEnv: "BOOTSTRAP_FIXTURE_KEY",
|
||||
companyName: "Workspace Recovery QA",
|
||||
browser: false,
|
||||
} as const;
|
||||
|
||||
process.env.BOOTSTRAP_FIXTURE_KEY = "not-a-provider-key";
|
||||
await prepareTestDriveEnvironment(options);
|
||||
process.env.NODE_ENV = "test";
|
||||
// Test-drive clears PAPERCLIP_* settings. Apply the fixture's short deadline
|
||||
// after isolation; production snapshots may legitimately take much longer.
|
||||
process.env.PAPERCLIP_WORKSPACE_GIT_SNAPSHOT_TIMEOUT_MS = "10000";
|
||||
process.env.PATH = `${path.join(import.meta.dirname, "bin")}${path.delimiter}${process.env.PATH ?? ""}`;
|
||||
await testDriveCommand(options);
|
||||
@@ -22,11 +22,11 @@ for (const persistent of [false, true]) {
|
||||
if (await announcement.isVisible()) await announcement.click();
|
||||
await page.getByRole("link", { name: "Tasks", exact: true }).click();
|
||||
await page.getByRole("button", { name: "New Task", exact: true }).last().click();
|
||||
await page.getByPlaceholder("Task title (optional)", { exact: true }).fill(title);
|
||||
await page.getByRole("button", { name: "Assignee", exact: true }).click();
|
||||
await page.getByRole("button", { name: fixture.agentName, exact: true }).click();
|
||||
await page.getByRole("textbox", { name: "editable markdown" }).fill(title);
|
||||
await page.getByRole("button", { name: "Select assignee", exact: true }).click();
|
||||
await page.getByRole("option").filter({ hasText: fixture.agentName }).click();
|
||||
// Let the closing popover unmount before clicking another popover trigger.
|
||||
await expect(page.getByRole("textbox", { name: "Search assignees...", includeHidden: true })).toHaveCount(0);
|
||||
await expect(page.getByRole("searchbox", { name: "Search assignees", includeHidden: true })).toHaveCount(0);
|
||||
// The preceding popover's focus restoration can consume the first click.
|
||||
await expect(async () => {
|
||||
if (!await page.getByRole("textbox", { name: "Search projects..." }).isVisible()) {
|
||||
@@ -36,15 +36,21 @@ for (const persistent of [false, true]) {
|
||||
}).toPass({ timeout: 10_000, intervals: [1_000] });
|
||||
await page.getByRole("button", { name: fixture.projectName, exact: true }).click();
|
||||
await expect(page.getByRole("textbox", { name: "Search projects...", includeHidden: true })).toHaveCount(0);
|
||||
await page.getByRole("button", { name: "Create Task", exact: true }).click();
|
||||
const taskLink = page.getByRole("complementary").getByRole("link", { name: title, exact: true });
|
||||
const created = page.waitForResponse(response =>
|
||||
response.request().method() === "POST"
|
||||
&& new URL(response.url()).pathname === `/api/companies/${fixture.companyId}/issues`,
|
||||
);
|
||||
await page.getByRole("button", { name: "Create task", exact: true }).click();
|
||||
const response = await created;
|
||||
expect(response.ok(), await response.text()).toBeTruthy();
|
||||
const task = await response.json();
|
||||
expect(task.description).toBe(title);
|
||||
const taskLink = page.getByRole("complementary").getByRole("link", { name: task.title, exact: true });
|
||||
await expect(taskLink).toBeVisible();
|
||||
// Follow the UI's actual persisted link, including across task-tab state.
|
||||
const href = await taskLink.getAttribute("href");
|
||||
expect(href).toBeTruthy();
|
||||
await page.goto(new URL(href!, base).href);
|
||||
const tasks = await api(`/companies/${fixture.companyId}/issues`);
|
||||
const task = tasks.find((row: { title: string }) => row.title === title);
|
||||
const runs = async () => (await api(`/companies/${fixture.companyId}/heartbeat-runs`)).filter((row: { agentId: string }) => row.agentId === fixture.agentId);
|
||||
await expect.poll(async () => (await runs()).some((row: { errorCode: string }) => row.errorCode === "workspace_git_scan_timeout"), { timeout: 30_000 }).toBe(true);
|
||||
await expect(page.getByText(/Agent resumes in/)).toBeVisible();
|
||||
|
||||
Reference in new issue
Block a user