fix(ssh): allow CLI agents on volume-free servers (#21957)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - An SSH environment runs an installed harness on a computer the
operator controls.
> - Volume-free Cloud servers require durable task files outside the
controller.
> - The current admission path accepts native sandbox workspaces and
rejects existing SSH CLI agents before execution.
> - The remote workspace owner already provides saved versions and a
single-writer fence.
> - This pull request gives that owner an SSH archive transport and
admits the existing Claude and Codex CLI adapters under its authority.
> - The benefit is SSH task execution without a controller volume or a
new environment type.

## Linked Issues or Issue Description

**What happened?**

A task assigned to an existing SSH Claude Code agent fails on a
volume-free Cloud server. It reports that execution requires native
Paperclip Runner and remote workspace storage. The failure occurs before
provider work starts, even when SSH and the configured AI account are
valid.

**Expected behavior**

An SSH CLI agent can use the configured Paperclip AI account and an
installed harness. Task files remain durable when the Cloud controller
is replaced. The same SSH behavior applies to direct and Tailscale
connections.

**Steps to reproduce**

1. Configure an SSH environment on a volume-free server with external
Postgres, stable encryption, and object storage.
2. Assign a Claude Code or Codex CLI agent to that environment.
3. Give it a task without a project, such as creating `hello_world.txt`.
4. Observe the workspace admission failure before execution.

Related work: #21248 adds agent-selected SSH authentication. #20617 adds
Tailscale network selection. #15120 and #17291 cover SSH process and
lease cleanup. A search of open and closed PRs found no duplicate of
this workspace admission fix.

## What Changed

- Reuse remote workspace preparation, archives, saved versions, and
single-writer admission for SSH targets.
- Add an explicit file-sync capability to the existing SSH command
runner. It streams archive files with bounded deadlines and private
temporary files. Ordinary SSH runners keep their previous capabilities.
- Run the existing CLI adapter in the remote workspace. Bind both the
target and its SSH command specification to that directory.
- Admit a direct SSH adapter on a volume-free server only when its
durable workspace, environment, lease, and path agree. Local filesystem
authority remains rejected.
- Preserve the server-owned remote workspace receipt when selecting the
legacy runtime.
- Reuse the workspace finalization lock for a legacy run only when its
company, workspace, writer, epoch, and lease receipt match. A claimed
token can finish and clear itself after another task reserves a
successfully saved shared workspace. Storage writes still require the
current workspace authority.
- Transfer recovered copies to SSH through the verified workspace
archive format. Sandbox recovery retains its existing directory
transfer.
- Document Node.js, Git, explicit `engine=cli`, and uncertain-process
limitations. No UI, Cloud control-plane configuration, schema, or
deployment change is included.

The remote workspace service owns this change. The shared SSH utility
adds file transport because its existing authentication and stream
cleanup must remain the only owners of SSH credentials. The heartbeat
changes pass the storage authority through existing preparation,
selection, and finalization boundaries. The finalization owner extension
avoids a separate copy-back lock for SSH. Existing native callers keep
their prior admission contract.

The existing SSH utility and heartbeat test files are already large. The
added code stays beside their existing stream lifecycle and shared
fixtures. A larger extraction would widen this compatibility fix.

## Verification

- Red-green tests reproduce SSH rejection and verify admission only with
remote storage authority.
- A real isolated OpenSSH fixture exercises archive capture, a
replacement database client, a saved-workspace restore to another
allocation, and a failed SSH transfer. Failed saves retain the files and
block a competing writer.
- A whole-heartbeat test uses the real SSH transport and Claude CLI
adapter, a synthetic CLI, a selected Paperclip AI account, and fixture
object storage. A follow-up resumes the same provider session and its
retained history. It makes no provider API calls. The synthetic provider
also supplies its task disposition update.
- Existing Claude and Codex SSH adapter tests, workspace preparation,
and native workspace owner recovery are included in focused checks.
- Before the review fixes, clean-checkout verification passed 610 tests
in 11 files. This includes the full workspace preparation, runtime
selection, runtime execution, environment orchestrator,
ephemeral-server, workspace transport, native owner recovery, SSH
integration, SSH command, and Claude/Codex remote execution suites. The
macOS SSH fixture was enabled with
`PAPERCLIP_ENABLE_DARWIN_SSH_ENV_LAB=1`; Vitest used `--maxWorkers=2`.
- An initial run in the main worktree hit a skill-root lookup outside
that checkout. The full selected set passes in the clean checkout. The
final company-binding type correction changes no executable behavior;
all 8 focused selection regressions pass after that correction.
- At the initial head `dd77a4d3cf0`, server and adapter-utils typechecks
passed, and the clean verification checkout matched its tree
`3f90b0681b1004556d1a720ed45fc8d439b04e96`.
- Review regressions first reproduced both a lost ownership token after
shared-workspace handoff and a rejected recovered-directory transfer.
With the fixes, 223 tests in 5 affected suites pass, including real SSH
recovery and native owner-restart, save-concurrency, archive transport,
and recovery-receipt checks. The handoff test also confirms that a stale
owner cannot save again or claim ownership after a new writer takes
over. Server typecheck passes after these fixes. The tested review-fix
head is `c5182f5d9c56e738e683d9f26b226e9a820bba83`.
- [The initial CI server
shard](https://github.com/paperclipai/paperclip/actions/runs/44009944657/job/133794523181)
failed on an assertion still expecting the old native-sandbox-only
error. Its unsupported sandbox adapter remains rejected; the assertion
now matches the supported-remote-environment error. After updating that
assertion, all 115 tests in the workspace lifecycle suite pass. Together
with the 5 affected suites, this gives 338 passing tests for the review
fixes.
- `pnpm verify` returns 2 and requests broad CI coverage for the
shared/runtime change. The full local suite was not substituted for CI.
Final-head
[CI](https://github.com/paperclipai/paperclip/actions/runs/44012991355)
is green on `c5182f5d9c56e738e683d9f26b226e9a820bba83`: 58 checks pass
and 3 are skipped, with no failures or pending checks. Typecheck, build,
end-to-end, and the aggregate verification gate all pass. Greptile is
5/5 on this head, security scan passes, and there are no unresolved
review threads or merge conflicts.

Review-fix checks:

```sh
PAPERCLIP_ENABLE_DARWIN_SSH_ENV_LAB=1 pnpm exec vitest run --maxWorkers=2 \
  server/src/services/ssh-remote-workspace.integration.test.ts \
  server/src/services/native-runtime/native-workspace-owner-restart-recovery.test.ts \
  server/src/services/remote-workspace-transport.test.ts \
  server/src/services/remote-workspace-recovery-receipt-fallback.test.ts \
  server/src/services/remote-workspace-save-concurrency.test.ts
pnpm exec vitest run --maxWorkers=2 server/src/services/remote-workspaces.integration.test.ts
pnpm --filter @paperclipai/server typecheck
```

## Risks

This extends the remote workspace owner to another transport and allows
qualified SSH CLI execution on volume-free servers. It changes remote
file placement to a task directory under `.paperclip-workspaces`. It
does not import an existing controller-local workspace.

The installed harness uses its existing adapter. Set `engine=cli`
explicitly; ACP archive staging does not support these in-place
workspaces. This does not qualify native macOS provider packs or change
an agent's engine or permission settings automatically.

An unknown remote process after SSH loss remains held until it can be
reconciled. This change does not add automatic reconciliation of that
process. SSH computers are never stopped or deleted. Real provider
acceptance and a hosted staging journey have not been run for this
revision.

## Model Used

OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) through the Codex harness, with xhigh
reasoning effort and code tools.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
Devin FoleyandPaperclip authored and GitHub committed 2026-11-17 09:33:11 -08:00
1 parent f61b919701
commit 1d2b669e7a
17 files changed
+597 -43

No files matched your search

+30
View File
@@ -61,6 +61,36 @@ SSH environments display it as on; opening or upgrading an installation does
not rewrite their saved configuration. The same option applies to direct SSH
and Tailscale connections.
## SSH on a volume-free Cloud stack
Cloud SSH execution uses the same durable task workspace storage as Cloud
sandboxes. Claude Code and Codex use their existing SSH adapters with
`engine=cli`. ACP archive staging cannot use an in-place Cloud workspace. An
explicit engine choice preserves the agent's execution and permission settings.
The selected
SSH account supplies the installed harness; the agent's Paperclip AI account
supplies authentication unless the remote authentication option is enabled.
Each task receives a private directory below the configured SSH workspace path,
inside `.paperclip-workspaces`. The task runs in that remote directory. Core
keeps the workspace binding and saved file versions in its database and object
storage. A Cloud restart reconnects to the same remote files. Restoring a saved
workspace to another allocation uses the existing snapshot verification and
single-writer rules. It does not restore private provider credentials from a
task snapshot.
The SSH machine needs Node.js and Git on its login PATH in addition to its agent
harness. The workspace owner uses Node.js for preparation and verified file
archives. SSH access alone does not qualify these dependencies or the selected
AI account.
An existing server-local workspace is not automatically imported. Cloud still
rejects local filesystem authority and unsupported adapters. An uncertain SSH
exit or failed save retains the remote files and the writer hold; it does not
authorize another run to overwrite them. Paperclip does not stop or delete the
SSH computer. This change does not add automatic reconciliation of an unknown
remote provider process after an SSH connection is lost.
The environment keeps the `ssh` driver. Machine selection stores the connector
ID and Tailscale’s stable device ID, and runtime
resolution looks up its current address through that connection’s private
@@ -1,8 +1,9 @@
import { mkdtemp, rm, writeFile } from "node:fs/promises";
import { mkdtemp, readFile, readdir, rm, writeFile } from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { afterEach, describe, expect, it, vi } from "vitest";
import { spawn } from "node:child_process";
import { createHash } from "node:crypto";
vi.mock("node:child_process", async (importOriginal) => {
const actual = await importOriginal<typeof import("node:child_process")>();
@@ -23,14 +24,16 @@ import { createSshCommandManagedRuntimeRunner, runSshCommand } from "./ssh.js";
const roots: string[] = [];
afterEach(async () => {
vi.useRealTimers();
vi.unstubAllEnvs();
for (const root of roots.splice(0)) await rm(root, { recursive: true, force: true });
});
async function runner() {
async function runner(fileSync = false) {
const cwd = await mkdtemp(path.join(os.tmpdir(), "ssh-managed-argv-"));
roots.push(cwd);
return createSshCommandManagedRuntimeRunner({
fileSync,
spec: {
host: "fixture.invalid", username: "fixture", port: 22,
remoteCwd: cwd, remoteWorkspacePath: cwd,
@@ -40,6 +43,59 @@ async function runner() {
}
describe("SSH command-managed argv", () => {
it("opts into bounded streaming archive transfer without changing ordinary SSH runners", async () => {
const plain = await runner();
expect(plain.syncIn).toBeUndefined();
expect(plain.syncOut).toBeUndefined();
const syncing = await runner(true);
const directory = await mkdtemp(path.join(os.tmpdir(), "ssh-archive-transfer-"));
roots.push(directory);
const source = path.join(directory, "source.pcws"), remote = path.join(directory, "quote's remote.pcws"), received = path.join(directory, "received.pcws");
const contents = Buffer.alloc(20 * 1024 * 1024, 0xa5);
const digest = (bytes: Buffer) => createHash("sha256").update(bytes).digest("hex");
await writeFile(source, contents);
await expect(syncing.syncIn!([{ operationId: "upload", files: [{ kind: "file", sourcePath: source, targetPath: remote }] }]))
.resolves.toMatchObject({ operations: [{ filesTransferred: 1, bytesTransferred: contents.length }] });
await expect(syncing.syncOut!([{ operationId: "download", files: [{ kind: "file", sourcePath: remote, targetPath: received }] }]))
.resolves.toMatchObject({ operations: [{ filesTransferred: 1, bytesTransferred: contents.length }] });
expect(digest(await readFile(received))).toBe(digest(contents));
expect((await readdir(directory)).some(name => name.includes(".upload-"))).toBe(false);
await expect(syncing.syncOut!([{ operationId: "overwrite", files: [{ kind: "file", sourcePath: remote, targetPath: received }] }]))
.rejects.toThrow();
expect(digest(await readFile(received))).toBe(digest(contents));
});
it("refuses unsupported directory and post-upload operations before transfer", async () => {
const syncing = await runner(true);
const files = [{ kind: "directory" as const, sourcePath: "/source", targetPath: "/target" }];
const calls = vi.mocked(spawn).mock.calls.length;
await expect(syncing.syncIn!([{ operationId: "directory", files }])).rejects.toThrow("archive files only");
expect(vi.mocked(spawn).mock.calls).toHaveLength(calls);
await expect(syncing.syncOut!([], { minimumTimeoutMs: Infinity })).rejects.toThrow("Invalid SSH transfer timeout");
});
it("terminates a stalled archive upload at its bounded deadline", async () => {
const syncing = await runner(true);
const source = path.join(roots.at(-1)!, "source.pcws");
await writeFile(source, "archive fixture");
const actual = await vi.importActual<typeof import("node:child_process")>("node:child_process");
let started!: () => void;
const ready = new Promise<void>(resolve => { started = resolve; });
let child: ReturnType<typeof actual.spawn> | undefined;
vi.mocked(spawn).mockImplementationOnce((_file, _args, options) => {
child = actual.spawn(process.execPath, ["-e", "setTimeout(() => {}, 60000)"], options);
started();
return child;
});
vi.useFakeTimers();
const transfer = syncing.syncIn!([{ operationId: "stall", files: [{ kind: "file", sourcePath: source, targetPath: source + ".remote" }] }]);
const rejected = expect(transfer).rejects.toThrow("SSH archive upload timed out");
await ready;
await vi.advanceTimersByTimeAsync(300_000);
await rejected;
expect(child?.killed).toBe(true);
});
it.each(["spawn", "command"])("decodes with BSD-compatible flags and preserves stdin through %s", async mode => {
const bin = await mkdtemp(path.join(os.tmpdir(), "ssh-bsd-base64-"));
roots.push(bin);
+49 -3
View File
@@ -6,6 +6,7 @@ import os from "node:os";
import path from "node:path";
import { pipeline, Transform } from "node:stream";
import type { CommandManagedRuntimeRunner } from "./command-managed-runtime.js";
import { MAX_SANDBOX_SYNC_TIMEOUT_MS, type SandboxSyncOperation, type SandboxSyncResult } from "./sandbox-managed-runtime.js";
import {
createUnrelatedHistoryGraftCommit,
GIT_SYNC_COMMIT_IDENTITY_ARGS,
@@ -54,6 +55,7 @@ export function createSshCommandManagedRuntimeRunner(input: {
spec: SshRemoteExecutionSpec;
defaultCwd?: string | null;
maxBufferBytes?: number | null;
fileSync?: boolean;
}): CommandManagedRuntimeRunner {
const defaultCwd = input.defaultCwd?.trim() || input.spec.remoteCwd;
const maxBufferBytes =
@@ -62,6 +64,10 @@ export function createSshCommandManagedRuntimeRunner(input: {
: 1024 * 1024;
return {
...(input.fileSync ? {
syncIn: (operations, options) => transferSshWorkspaceFiles(input.spec, "in", operations, options?.minimumTimeoutMs),
syncOut: (operations, options) => transferSshWorkspaceFiles(input.spec, "out", operations, options?.minimumTimeoutMs),
} satisfies Pick<CommandManagedRuntimeRunner, "syncIn" | "syncOut"> : {}),
execute: async (commandInput): Promise<RunProcessResult> => {
const startedAt = new Date().toISOString();
const command = commandInput.command.trim();
@@ -125,6 +131,37 @@ export function createSshCommandManagedRuntimeRunner(input: {
};
}
async function transferSshWorkspaceFiles(spec: SshRemoteExecutionSpec, direction: "in" | "out",
operations: SandboxSyncOperation[], minimumTimeoutMs = 300_000): Promise<SandboxSyncResult> {
if (!Number.isFinite(minimumTimeoutMs) || minimumTimeoutMs <= 0) throw new Error("Invalid SSH transfer timeout");
const timeoutMs = Math.min(MAX_SANDBOX_SYNC_TIMEOUT_MS, Math.max(300_000, minimumTimeoutMs));
for (const operation of operations) {
if (operation.postUploadCommands?.length || operation.files.some(file => file.kind !== "file")) {
throw new Error("SSH workspace transport supports archive files only");
}
}
const results: SandboxSyncResult["operations"] = [];
for (const operation of operations) {
let bytesTransferred = 0;
for (const file of operation.files) {
if (direction === "in") {
const source = await fs.lstat(file.sourcePath);
if (!source.isFile()) throw new Error("SSH archive source must be a regular file");
const temporary = `${file.targetPath}.upload-${randomUUID()}`;
await streamLocalFileToSsh({ spec, localFile: file.sourcePath, timeoutMs,
remoteScript: `umask 077; trap ${shellQuote(`rm -f ${shellQuote(temporary)}`)} EXIT; cat > ${shellQuote(temporary)} && mv -f ${shellQuote(temporary)} ${shellQuote(file.targetPath)}` });
bytesTransferred += source.size;
} else {
await streamSshToLocalFile({ spec, localFile: file.targetPath, timeoutMs, exclusive: true,
remoteScript: `test -f ${shellQuote(file.sourcePath)} && test ! -L ${shellQuote(file.sourcePath)} && cat ${shellQuote(file.sourcePath)}` });
bytesTransferred += (await fs.stat(file.targetPath)).size;
}
}
results.push({ operationId: operation.operationId, filesTransferred: operation.files.length, bytesTransferred });
}
return { operations: results };
}
export interface SshEnvLabSupport {
supported: boolean;
reason: string | null;
@@ -679,6 +716,7 @@ async function streamLocalFileToSsh(input: {
localFile: string;
remoteScript: string;
progress?: TransferProgress;
timeoutMs?: number;
}): Promise<void> {
const auth = await createSshAuthArgs(input.spec);
const sshArgs = [
@@ -689,6 +727,7 @@ async function streamLocalFileToSsh(input: {
`sh -c ${shellQuote(input.remoteScript)}`,
];
let timer: ReturnType<typeof setTimeout> | undefined;
await new Promise<void>((resolve, reject) => {
const source = createReadStream(input.localFile);
const ssh = spawn("ssh", sshArgs, {
@@ -706,6 +745,8 @@ async function streamLocalFileToSsh(input: {
reject(error);
};
if (input.timeoutMs) timer = setTimeout(() => fail(new Error("SSH archive upload timed out")), input.timeoutMs);
ssh.stderr?.on("data", (chunk) => {
sshStderr += String(chunk);
});
@@ -726,7 +767,7 @@ async function streamLocalFileToSsh(input: {
}
resolve();
});
}).finally(auth.cleanup);
}).finally(async () => { clearTimeout(timer); await auth.cleanup(); });
}
async function streamSshToLocalFile(input: {
@@ -734,6 +775,8 @@ async function streamSshToLocalFile(input: {
remoteScript: string;
localFile: string;
progress?: TransferProgress;
timeoutMs?: number;
exclusive?: boolean;
}): Promise<void> {
const auth = await createSshAuthArgs(input.spec);
const sshArgs = [
@@ -744,11 +787,12 @@ async function streamSshToLocalFile(input: {
`sh -c ${shellQuote(input.remoteScript)}`,
];
let timer: ReturnType<typeof setTimeout> | undefined;
await new Promise<void>((resolve, reject) => {
const ssh = spawn("ssh", sshArgs, {
stdio: ["ignore", "pipe", "pipe"],
});
const sink = createWriteStream(input.localFile, { mode: 0o600 });
const sink = createWriteStream(input.localFile, { mode: 0o600, ...(input.exclusive ? { flags: "wx" } : {}) });
let sshStderr = "";
let settled = false;
@@ -771,6 +815,8 @@ async function streamSshToLocalFile(input: {
reject(error);
};
if (input.timeoutMs) timer = setTimeout(() => fail(new Error("SSH archive download timed out")), input.timeoutMs);
const onStreamFinished = (error: NodeJS.ErrnoException | null) => {
if (error) {
fail(error);
@@ -796,7 +842,7 @@ async function streamSshToLocalFile(input: {
sshClosed = true;
settle();
});
}).finally(auth.cleanup);
}).finally(async () => { clearTimeout(timer); await auth.cleanup(); });
}
async function importGitWorkspaceToSsh(input: {
@@ -294,8 +294,8 @@ describe("environmentRunOrchestrator — realizeForRun", () => {
expect(runtime.acquireRunLease).toHaveBeenCalledWith(expect.objectContaining({ assertCompanyBinding: true }));
});
it("realizes an enrolled remote workspace in place and persists its authoritative remote root", async () => {
const environment = { ...makeEnvironment("sandbox"), id: randomUUID() };
it.each(["sandbox", "ssh"] as const)("realizes an enrolled %s workspace in place and persists its authoritative remote root", async driver => {
const environment = { ...makeEnvironment(driver), id: randomUUID() };
const lease = makeLease({ id: randomUUID(), environmentId: environment.id, provider: "daytona", providerLeaseId: "sandbox-allocation" });
const workspace = makePersistedExecutionWorkspace({ id: randomUUID(), providerType: "cloud_sandbox", metadata: {
remoteWorkspace: { schema: "paperclip.remote-workspace.v1", epoch: 1, source: { kind: "empty" },
@@ -303,7 +303,9 @@ describe("environmentRunOrchestrator — realizeForRun", () => {
} });
const remoteRoot = `/sandbox/.paperclip-workspaces/${workspace.id}`;
const savedLease = { ...lease, metadata: { remoteCwd: remoteRoot } };
const target = { kind: "remote", transport: "sandbox", remoteCwd: "/sandbox", runner: { execute: vi.fn() } };
const target = driver === "ssh"
? { kind: "remote", transport: "ssh", remoteCwd: "/sandbox", spec: { remoteCwd: "/sandbox" } }
: { kind: "remote", transport: "sandbox", remoteCwd: "/sandbox", runner: { execute: vi.fn() } };
mockResolveEnvironmentExecutionTarget.mockResolvedValue(target);
mockGetStoredWorkspace.mockResolvedValue({ ...workspace, cwd: remoteRoot });
mockUpdateLeaseMetadata.mockResolvedValue(savedLease);
@@ -312,7 +314,7 @@ describe("environmentRunOrchestrator — realizeForRun", () => {
} as never);
try {
const runtime = makeMockRuntime();
const input = { ...makeRealizeInput({ environment, lease, persistedExecutionWorkspace: workspace }), adapterType: "paperclip_runner" };
const input = { ...makeRealizeInput({ environment, lease, persistedExecutionWorkspace: workspace }), adapterType: driver === "ssh" ? "claude_local" : "paperclip_runner" };
const result = await environmentRunOrchestrator(mockDb, { environmentRuntime: runtime }).realizeForRun(input);
expect(prepare).toHaveBeenCalledWith(expect.objectContaining({ workspace, target, source: { kind: "empty" } }));
expect(mockBuildWorkspaceRealizationRequest).not.toHaveBeenCalled();
@@ -320,6 +322,9 @@ describe("environmentRunOrchestrator — realizeForRun", () => {
expect(result.executionTarget).toMatchObject({ remoteCwd: remoteRoot,
workspaceRealization: { mode: "in_place", authoritativeRoot: remoteRoot, outboundRestorePaths: [] } });
expect(input.executionWorkspace.cwd).toBe(remoteRoot);
if (result.executionTarget?.kind === "remote" && result.executionTarget.transport === "ssh") {
expect(result.executionTarget.spec.remoteCwd).toBe(remoteRoot);
}
expect(mockUpdateLeaseMetadata).toHaveBeenCalledWith(lease.id, expect.objectContaining({ remoteCwd: remoteRoot }));
expect(result.lease.metadata?.remoteCwd).toBe(remoteRoot);
} finally { prepare.mockRestore(); }
@@ -1,6 +1,6 @@
import { afterEach, expect, it, vi } from "vitest";
import { assertEphemeralServerConfig } from "../services/ephemeral-server-contract.js";
import { assertEphemeralWorkspaceExecution, assertEphemeralNativeRuntime } from "../services/remote-workspaces.js";
import { assertEphemeralWorkspaceExecution, assertEphemeralNativeRuntime, shouldUseRemoteWorkspaces } from "../services/remote-workspaces.js";
import { assertEphemeralAgentInstructions } from "../services/agent-instructions.js";
const config = { databaseUrl: "postgresql://runtime:fixture@ep-example-pooler.neon.test/tenant", storageProvider: "s3" as const };
@@ -41,3 +41,18 @@ it("requires remote workspace authority and native execution independently of pr
expect(() => assertEphemeralWorkspaceExecution({ ...remote, remoteAuthority: false })).not.toThrow();
expect(() => assertEphemeralNativeRuntime(false)).not.toThrow();
});
it.each(["claude_local", "codex_local"])("admits %s over SSH only with durable remote workspace authority", adapterType => {
vi.stubEnv("PAPERCLIP_EPHEMERAL_SERVER", "true");
vi.stubEnv("DATABASE_URL", config.databaseUrl);
vi.stubEnv("PAPERCLIP_STORAGE_PROVIDER", "s3");
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", env.PAPERCLIP_SECRETS_MASTER_KEY);
const remote = { externalConversation: false, adapterType, environmentDriver: "ssh", pluginTaskWorkspace: false, remoteAuthority: true };
expect(shouldUseRemoteWorkspaces({ adapterType, environmentDriver: "ssh", experimentalEnabled: true })).toBe(true);
expect(() => assertEphemeralWorkspaceExecution(remote)).not.toThrow();
expect(() => assertEphemeralWorkspaceExecution({ ...remote, remoteAuthority: false })).toThrow();
expect(() => assertEphemeralWorkspaceExecution({ ...remote, environmentDriver: "local" })).toThrow();
expect(() => assertEphemeralWorkspaceExecution({ ...remote, adapterType: "process" })).toThrow();
expect(() => assertEphemeralNativeRuntime(false, { ...remote, remoteCwd: "/workspace", authorityCwd: "/workspace" })).not.toThrow();
expect(() => assertEphemeralNativeRuntime(false, { ...remote, remoteCwd: "/other", authorityCwd: "/workspace" })).toThrow();
});
@@ -1,6 +1,6 @@
import { RetainedSandboxAbsentError } from "./retained-sandbox-absent-error.js";
import { freshWorkspaceSchema } from "@paperclipai/plugin-sdk";
import { prepareRemoteWorkspace, readRemoteWorkspaceAuthority, remoteWorkspaceRepositoryPath } from "./remote-workspaces.js";
import { prepareRemoteWorkspace, readRemoteWorkspaceAuthority, remoteWorkspaceRepositoryPath, supportsSshRemoteWorkspace } from "./remote-workspaces.js";
/**
* Centralized environment run orchestrator.
*
@@ -559,15 +559,15 @@ export function environmentRunOrchestrator(
};
const remoteAuthority = readRemoteWorkspaceAuthority(persistedExecutionWorkspace?.metadata);
if (remoteAuthority && persistedExecutionWorkspace) {
if (adapterType !== "paperclip_runner" || environment.driver !== "sandbox") {
throw new EnvironmentRunError("unsupported_adapter_environment", "Remote workspace storage requires a qualified native sandbox");
if (!(adapterType === "paperclip_runner" && environment.driver === "sandbox") && !supportsSshRemoteWorkspace(adapterType, environment.driver)) {
throw new EnvironmentRunError("unsupported_adapter_environment", "Remote workspace storage requires a supported remote environment");
}
const target = withAcquisitionReceipt(await resolveEnvironmentExecutionTarget({
db, companyId, adapterType, environment, leaseId: lease.id, leaseMetadata: lease.metadata,
lease, environmentRuntime, duplexObservabilityRecorder: input.duplexObservabilityRecorder ?? null,
}));
if (!target || target.kind !== "remote" || target.transport !== "sandbox" || !target.runner || target.runnerTask) {
throw new EnvironmentRunError("transport_resolution_failed", "Remote workspace storage requires a sandbox file transport");
if (!target || target.kind !== "remote" || (target.transport === "sandbox" && (!target.runner || target.runnerTask))) {
throw new EnvironmentRunError("transport_resolution_failed", "Remote workspace storage requires a remote file transport");
}
const prepared = await prepareRemoteWorkspace({
db, workspace: persistedExecutionWorkspace, runId: heartbeatRunId, lease, target,
@@ -596,6 +596,7 @@ export function environmentRunOrchestrator(
input.executionWorkspace.cwd = prepared.remoteRoot;
const executionTarget: AdapterExecutionTarget = {
...target, remoteCwd: prepared.remoteRoot,
...(target.transport === "ssh" ? { spec: { ...target.spec, remoteCwd: prepared.remoteRoot } } : {}),
workspaceRealization: { mode: "in_place", authoritativeRoot: prepared.remoteRoot,
pathAliases: [], outboundRestorePaths: [] },
};
@@ -1091,6 +1091,8 @@ export async function executeHeartbeatRuntime(db: Db, input: HeartbeatRuntimeExe
if (nativeWorkspaceSync || remoteWorkspaceAuthority) {
const owned = await measureRunPhase("execution.workspace_ownership", () => withNativeWorkspaceFinalizationOwnership({
db, companyId: run.companyId, runId: run.id,
...(remoteWorkspaceAuthority && nativeRuntimeResolution.kind !== "native" && persistedExecutionWorkspace
? { remoteWorkspaceId: persistedExecutionWorkspace.id } : {}),
}, completeWorkspace));
if (!owned.acquired) throw new NativeWorkspaceFinalizationBusyError();
} else {
@@ -942,6 +942,42 @@ describe.skipIf(!support.supported)("heartbeat runtime selection boundary", () =
expect(prepareNativeWorkspaceSync).not.toHaveBeenCalled();
});
it.each(["claude_local", "codex_local"])("keeps %s SSH runtime and its durable workspace receipt on a volume-free server", async adapterType => {
const { input, run, agent, issue } = await fixture(adapterType);
vi.stubEnv("PAPERCLIP_EPHEMERAL_SERVER", "true");
const environmentId = randomUUID(), leaseId = randomUUID(), workspaceId = randomUUID();
const remoteCwd = "/home/fixture/work/.paperclip-workspaces/task";
const reference = { schema: "paperclip.remote-workspace-run.v1", workspaceId, epoch: 1,
leaseId, providerLeaseId: "ssh-fixture", remoteCwd, state: "prepared", snapshotId: null };
const [workspace] = await db.insert(executionWorkspaces).values({ id: workspaceId, companyId: agent.companyId,
name: "SSH task files", mode: "isolated_workspace", strategyType: "project_primary", providerType: "cloud_sandbox", cwd: remoteCwd,
metadata: { remoteWorkspace: { schema: "paperclip.remote-workspace.v1", epoch: 1, source: { kind: "empty" },
writerRunId: run.id, dirty: true, snapshotId: null, location: { environmentId, leaseId, providerLeaseId: "ssh-fixture", remoteCwd } } } }).returning();
await db.update(heartbeatRuns).set({ runnerProfileJson: { remoteWorkspace: reference } }).where(eq(heartbeatRuns.id, run.id));
input.workspace.persistedExecutionWorkspace = workspace;
input.workspace.executionTarget = { kind: "remote", transport: "ssh", environmentId, leaseId, remoteCwd,
workspaceRealization: { mode: "in_place", authoritativeRoot: remoteCwd, pathAliases: [], outboundRestorePaths: [] },
spec: { host: "fixture.invalid", username: "fixture", port: 22, remoteWorkspacePath: "/home/fixture/work", remoteCwd,
useHostAuth: false, privateKey: null, knownHosts: null, strictHostKeyChecking: true } };
input.nativeRuntimeResolution = resolveHeartbeatNativeRuntimeMode({ persisted: run, enabled: true,
runtimeConfig: {}, adapterConfig: agent.adapterConfig, agent, issue, target: input.workspace.executionTarget, workspaceId });
const selected = await selectHeartbeatRuntime(db, input);
expect(selected.selected).toBe(true);
if (!selected.selected) throw new Error("Expected legacy SSH runtime selection");
expect(selected.nativeExecution).toBeNull();
expect((await read(run.id)).runnerProfileJson?.remoteWorkspace).toEqual(reference);
expect(await coordinators(run.id)).toHaveLength(0);
expect(prepareNativeWorkspaceSync).not.toHaveBeenCalled();
input.workspace.executionTarget.spec.remoteCwd = "/other-directory";
await expect(selectHeartbeatRuntime(db, input)).rejects.toThrow("native Paperclip Runner");
input.workspace.executionTarget.spec.remoteCwd = remoteCwd;
input.workspace.persistedExecutionWorkspace.companyId = randomUUID();
await expect(selectHeartbeatRuntime(db, input)).rejects.toThrow("native Paperclip Runner");
input.workspace.persistedExecutionWorkspace.companyId = agent.companyId;
input.workspace.executionTarget.leaseId = randomUUID();
await expect(selectHeartbeatRuntime(db, input)).rejects.toThrow("native Paperclip Runner");
});
it.each(["warm", "per_turn"] as const)("preserves resolved %s policy on an ephemeral server", async mode => {
const { input, run } = await fixture();
vi.stubEnv("PAPERCLIP_EPHEMERAL_SERVER", "true");
@@ -125,7 +125,7 @@ export interface HeartbeatRuntimeSelectionInput {
getFreshSessionHandoff: (() => Promise<string | null>) | undefined;
};
workspace: {
persistedExecutionWorkspace: { id: string; metadata?: Record<string, unknown> | null } | null;
persistedExecutionWorkspace: { id: string; companyId?: string; metadata?: Record<string, unknown> | null } | null;
executionWorkspace: Omit<Parameters<typeof buildNativeExecutionInput>[0]["workspace"], "id">;
executionTarget: Parameters<typeof prepareNativeWorkspaceSync>[0]["target"];
activeEnvironmentLease: (Pick<Parameters<typeof prepareNativeWorkspaceSync>[0], "lease"> & {
@@ -325,7 +325,18 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele
const { runtimeConfig, runtimeSkillEntries, runScopedMentionedSkillKeys, instructionCopy, agentIdentity, issueAssigneeOverrides, resolvedConfig, managedAiRuntime } = input.config;
const runOptions = input.recovery;
const { providerTraceRequested, providerTraceCapture } = input.trace;
if (!isDotRun) assertEphemeralNativeRuntime(nativeRuntimeResolution.kind === "native");
const sshAuthority = readRemoteWorkspaceAuthority(persistedExecutionWorkspace?.metadata);
if (!isDotRun) assertEphemeralNativeRuntime(nativeRuntimeResolution.kind === "native", {
adapterType: agent.adapterType,
environmentDriver: executionTarget?.kind === "remote" ? executionTarget.transport : null,
remoteAuthority: Boolean(sshAuthority?.location && executionTarget?.kind === "remote" && executionTarget.transport === "ssh"
&& persistedExecutionWorkspace?.companyId === agent.companyId && sshAuthority.writerRunId === run.id
&& executionTarget.spec.remoteCwd === executionTarget.remoteCwd
&& sshAuthority.location.environmentId === executionTarget.environmentId
&& sshAuthority.location.leaseId === executionTarget.leaseId),
remoteCwd: executionTarget?.kind === "remote" ? executionTarget.remoteCwd : null,
authorityCwd: sshAuthority?.location?.remoteCwd ?? null,
});
let nativeExecution: NativeExecutionInput | null = null;
let toolRefreshOnResume: boolean | undefined;
let getNativeFreshSessionHandoff: (() => Promise<string | null>) | undefined;
@@ -1184,6 +1195,9 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele
else '{}'::jsonb end)
|| (case when ${heartbeatRuns.runnerProfileJson} ? 'adapterDispatch'
then jsonb_build_object('adapterDispatch', ${heartbeatRuns.runnerProfileJson}->'adapterDispatch')
else '{}'::jsonb end)
|| (case when ${heartbeatRuns.runnerProfileJson}#>>'{remoteWorkspace,schema}' = 'paperclip.remote-workspace-run.v1'
then jsonb_build_object('remoteWorkspace', ${heartbeatRuns.runnerProfileJson}->'remoteWorkspace')
else '{}'::jsonb end) || ${JSON.stringify(providerTraceRequested ? { providerTrace: { mode: "raw", traceId: providerTraceCapture?.metadata.id ?? null, maxBytes: PROVIDER_TRACE_MAX_BYTES } } : {})}::jsonb`,
updatedAt: new Date(),
})
@@ -590,6 +590,31 @@ describe.skipIf(!support.supported && !externalDatabaseUrl)("heartbeat workspace
expect(await db.select().from(executionWorkspaces)).toHaveLength(0);
});
it.each(["claude_local", "codex_local"])("reserves remote storage for volume-free %s SSH tasks before acquiring the machine", async adapterType => {
const { input, issue, envOrchestrator } = await fixture();
configureRemotePersistence();
process.env.PAPERCLIP_EPHEMERAL_SERVER = "true";
input.agent.adapterType = adapterType;
await db.update(agents).set({ adapterType }).where(eq(agents.id, input.agent.id));
await db.update(issues).set({ projectId: null, projectWorkspaceId: null }).where(eq(issues.id, issue.id));
input.task.issueRef = { ...issue, projectId: null, projectWorkspaceId: null };
input.task.context = { issueId: issue.id };
input.policy.useIsolatedTaskDirectory = true;
const environment = await environmentService(db).create(input.agent.companyId, { name: `SSH ${adapterType}`, driver: "ssh", status: "active", envVars: {},
config: { host: "fixture.invalid", port: 22, username: "fixture", remoteWorkspacePath: "/home/fixture/work", useHostAuth: false } });
input.environment.selectedEnvironmentId = environment.id;
input.environment.selectedEnvironmentForConfig = environment;
const resolveHost = vi.spyOn(input.services, "resolveWorkspaceForRun");
const acquire = vi.spyOn(envOrchestrator, "acquireForRun").mockImplementation(async () => { throw new Error("SSH acquisition admitted"); });
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow("SSH acquisition admitted");
expect(acquire).toHaveBeenCalledOnce();
expect(resolveHost).not.toHaveBeenCalled();
const [workspace] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.companyId, input.agent.companyId));
expect(readRemoteWorkspaceAuthority(workspace.metadata)).toMatchObject({ source: { kind: "empty" }, writerRunId: input.run.id });
const [bound] = await db.select().from(issues).where(eq(issues.id, issue.id));
expect(bound.executionWorkspaceId).toBe(workspace.id);
});
it.each([false, true])("configured local folders retain host authority only on durable servers (ephemeral=%s)", async ephemeral => {
const { input, envOrchestrator } = await fixture();
input.agent.adapterType = "paperclip_runner";
@@ -285,6 +285,7 @@ async function prepareHeartbeatWorkspaceLocked(db: Db, input: HeartbeatWorkspace
});
if (!providerApiPlacement) assertEphemeralWorkspaceExecution({ externalConversation: isDotRun,
adapterType: agent.adapterType,
environmentDriver: selectedEnvironmentForConfig?.driver,
pluginTaskWorkspace, remoteAuthority });
const assertRemotePreparationOwned = run.runtimeMode === "native" ? () => assertNativeSessionPreparationAuthority({
db, companyId: agent.companyId, runId: run.id,
@@ -370,7 +371,7 @@ async function prepareHeartbeatWorkspaceLocked(db: Db, input: HeartbeatWorkspace
// Source resolution can retain a configured host folder. Check the
// resolved authority before materializing it or acquiring a sandbox.
if (!providerApiPlacement) assertEphemeralWorkspaceExecution({ externalConversation: isDotRun,
adapterType: agent.adapterType, pluginTaskWorkspace, remoteAuthority });
adapterType: agent.adapterType, environmentDriver: selectedEnvironmentForConfig?.driver, pluginTaskWorkspace, remoteAuthority });
}
else {
remoteSourceState.value = sourceState;
@@ -10,8 +10,8 @@ import { nativeSha256 } from "./canonical.js";
import { NATIVE_WORKSPACE_OWNER_UNVERIFIED, NATIVE_WORKSPACE_OWNER_WAIT_NEXT_ACTION } from "./native-workspace-finalization-recovery-state.js";
import { randomUUID } from "node:crypto";
import os from "node:os";
import { and, eq, inArray, sql } from "drizzle-orm";
import { heartbeatRuns, issues, issueRecoveryActions, nativeRunFinalizations, workspaceOperations, withDedicatedDbConnection, type Db } from "@paperclipai/db";
import { and, eq, inArray, or, sql } from "drizzle-orm";
import { executionWorkspaces, heartbeatRuns, issues, issueRecoveryActions, nativeRunFinalizations, workspaceOperations, withDedicatedDbConnection, type Db } from "@paperclipai/db";
import { currentNativeWorkspaceRecoveryResult } from "./native-workspace-recovery-binding.js";
import { withWorkspaceLockKeepalive } from "./native-workspace-lock-keepalive.js";
import {
@@ -105,7 +105,7 @@ export function nativeWorkspaceFinalizationOwnerCanUpgrade(profile: Record<strin
* There is deliberately no wall-clock timeout that can steal a slow export.
*/
export async function withNativeWorkspaceFinalizationOwnership<T>(
input: { db: Db; companyId: string; runId: string; retainedRepairActionId?: string },
input: { db: Db; companyId: string; runId: string; retainedRepairActionId?: string; remoteWorkspaceId?: string },
action: (ownership: NativeWorkspaceFinalizationOwnership) => Promise<T>,
): Promise<{ acquired: false; reason: NativeWorkspaceFinalizationWaitReason } | { acquired: true; value: T }> {
const identity = await currentNativeControllerIdentity();
@@ -120,9 +120,22 @@ export async function withNativeWorkspaceFinalizationOwnership<T>(
...(deployment ? { exclusiveVolume: deployment.exclusiveVolume } : {}),
...(deployment?.exclusiveVolume && deployment.volumeId ? { volumeId: deployment.volumeId } : {}),
};
const scope = and(eq(heartbeatRuns.id, input.runId), eq(heartbeatRuns.companyId, input.companyId),
input.retainedRepairActionId ? inArray(heartbeatRuns.runtimeMode, ["native", "legacy"]) : eq(heartbeatRuns.runtimeMode, "native"));
const owned = and(scope, sql`${heartbeatRuns.runnerProfileJson}->${OWNER_KEY}->>'token' = ${owner.token}`);
const remoteLegacy = input.remoteWorkspaceId ? and(eq(heartbeatRuns.runtimeMode, "legacy"),
sql`${heartbeatRuns.runnerProfileJson}#>>'{remoteWorkspace,schema}' = 'paperclip.remote-workspace-run.v1'`,
sql`${heartbeatRuns.runnerProfileJson}#>>'{remoteWorkspace,workspaceId}' = ${input.remoteWorkspaceId}`,
sql`exists (select 1 from ${executionWorkspaces} where ${executionWorkspaces.id} = ${input.remoteWorkspaceId}::uuid
and ${executionWorkspaces.companyId} = ${heartbeatRuns.companyId}
and ${executionWorkspaces.metadata}#>>'{remoteWorkspace,schema}' = 'paperclip.remote-workspace.v1'
and (${executionWorkspaces.metadata}#>>'{remoteWorkspace,writerRunId}' = ${heartbeatRuns.id}::text
or (${executionWorkspaces.metadata}#>>'{remoteWorkspace,writerRunId}' is null
and ${heartbeatRuns.runnerProfileJson}#>>'{remoteWorkspace,state}' = 'saved'))
and ${executionWorkspaces.metadata}#>>'{remoteWorkspace,epoch}' = ${heartbeatRuns.runnerProfileJson}#>>'{remoteWorkspace,epoch}'
and ${executionWorkspaces.metadata}#>>'{remoteWorkspace,location,leaseId}' = ${heartbeatRuns.runnerProfileJson}#>>'{remoteWorkspace,leaseId}')`) : undefined;
const runScope = and(eq(heartbeatRuns.id, input.runId), eq(heartbeatRuns.companyId, input.companyId),
input.retainedRepairActionId || input.remoteWorkspaceId ? inArray(heartbeatRuns.runtimeMode, ["native", "legacy"])
: eq(heartbeatRuns.runtimeMode, "native"));
const claimScope = and(runScope, input.retainedRepairActionId ? undefined : or(eq(heartbeatRuns.runtimeMode, "native"), remoteLegacy));
const owned = and(runScope, sql`${heartbeatRuns.runnerProfileJson}->${OWNER_KEY}->>'token' = ${owner.token}`);
const runKey = nativeWorkspaceFinalizationLockKey(input.companyId, input.runId);
let claimed = false;
let actionPromise: Promise<T> | undefined;
@@ -139,7 +152,7 @@ export async function withNativeWorkspaceFinalizationOwnership<T>(
// Reads and durable progress use the ordinary pool, not the lock transaction.
const [run] = await input.db.select({ profile: heartbeatRuns.runnerProfileJson,
issueId: sql<string | null>`coalesce(${heartbeatRuns.nativeIssueId}::text, ${heartbeatRuns.issueId}::text, ${heartbeatRuns.contextSnapshot}->>'issueId')`,
agentId: heartbeatRuns.agentId }).from(heartbeatRuns).where(scope).limit(1);
agentId: heartbeatRuns.agentId }).from(heartbeatRuns).where(claimScope).limit(1);
if (!run) throw new Error("native_workspace_finalization_binding_missing");
const rawPrior = run.profile?.[OWNER_KEY];
const prior = readOwner(rawPrior);
@@ -154,7 +167,7 @@ export async function withNativeWorkspaceFinalizationOwnership<T>(
const [issue] = await db.select().from(issues).where(and(
eq(issues.companyId, input.companyId), eq(issues.id, run.issueId),
)).for("update");
const [currentRun] = await db.select().from(heartbeatRuns).where(scope).for("update");
const [currentRun] = await db.select().from(heartbeatRuns).where(claimScope).for("update");
if (input.retainedRepairActionId) {
const [repair] = await db.select().from(issueRecoveryActions).where(and(
eq(issueRecoveryActions.companyId, input.companyId), eq(issueRecoveryActions.sourceIssueId, run.issueId),
@@ -276,7 +289,7 @@ export async function withNativeWorkspaceFinalizationOwnership<T>(
claimed = true;
const [claimedRun] = await db.update(heartbeatRuns).set({
runnerProfileJson: sql`jsonb_set(coalesce(${heartbeatRuns.runnerProfileJson}, '{}'::jsonb), array[${OWNER_KEY}], ${JSON.stringify(owner)}::jsonb) || ${JSON.stringify(platformProof ? { nativeWorkspaceOwnerStopProof: platformProof } : {})}::jsonb`,
}).where(and(scope, prior
}).where(and(claimScope, prior
? sql`${heartbeatRuns.runnerProfileJson}->${OWNER_KEY} = ${JSON.stringify(rawPrior)}::jsonb`
: sql`${heartbeatRuns.runnerProfileJson}->>${OWNER_KEY} is null`
)).returning({ id: heartbeatRuns.id });
@@ -1,8 +1,7 @@
import { lstat, mkdir, mkdtemp, readFile, realpath, rename, rm, writeFile } from "node:fs/promises";
import path from "node:path";
import os from "node:os";
import type { AdapterSandboxExecutionTarget } from "@paperclipai/adapter-utils/execution-target";
import { hasRemoteRecoveryAttachment, removeRemoteRecoveryAttachment } from "./remote-workspace-transport.js";
import { hasRemoteRecoveryAttachment, removeRemoteRecoveryAttachment, type RemoteWorkspaceTarget } from "./remote-workspace-transport.js";
import { removeTaskDirectoryTree } from "./isolated-task-directory-snapshot.js";
import { LEGACY_WORKSPACE_RECOVERY_MIGRATION } from "./legacy-workspace-recovery-migration.js";
import { and, eq, inArray, sql } from "drizzle-orm";
@@ -225,7 +224,7 @@ function record(value: unknown): Record<string, unknown> {
* provenance. This checks the selected root; it never imports or replaces files. */
export async function retainRemoteRecoveryAttachments(input: {
db: Db; workspace: Pick<typeof executionWorkspaces.$inferSelect, "id" | "companyId" | "sourceIssueId" | "metadata">;
runId: string; remoteRoot: string; target: AdapterSandboxExecutionTarget; assertOwned?: () => Promise<void>;
runId: string; remoteRoot: string; target: RemoteWorkspaceTarget; assertOwned?: () => Promise<void>;
recoveredFiles: Array<{ operationId: string; sourceRunId: string; path: string; provenance: string; retired?: true }>;
retiredOperationIds: ReadonlySet<string>; storage?: StorageService;
}) {
@@ -1,8 +1,10 @@
import { randomUUID } from "node:crypto";
import { createHash, randomUUID } from "node:crypto";
import { lstat, mkdtemp, realpath, rm, stat } from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import type { AdapterExecutionTarget } from "@paperclipai/adapter-utils/execution-target";
import type { CommandManagedRuntimeRunner } from "@paperclipai/adapter-utils/command-managed-runtime";
import { createSshCommandManagedRuntimeRunner } from "@paperclipai/adapter-utils/ssh";
import type { RemoteWorkspaceSource } from "@paperclipai/shared";
import { captureRemoteWorkspaceDirectory, remoteWorkspaceArchiveScript, remoteWorkspaceCaptureFailureMessage, remoteWorkspaceFailureDetail } from "./remote-workspace-snapshot-archive.js";
import { readWorkspaceInventory, readonlyWorkspaceInventoryScript } from "./remote-workspace-readonly-inventory.js";
@@ -13,7 +15,8 @@ import { classifyWorkspaceRestoreFailure } from "@paperclipai/adapter-utils/work
import { MAX_SANDBOX_SYNC_TIMEOUT_MS, type SandboxSyncOperation } from "@paperclipai/adapter-utils/sandbox-managed-runtime";
import { REMOTE_WORKSPACE_CAPTURE_TRANSFER_MISMATCH } from "./remote-workspace-archive-io.js";
export type RemoteWorkspaceTarget = Extract<AdapterExecutionTarget, { transport: "sandbox" }>;
export type RemoteWorkspaceTarget = Extract<AdapterExecutionTarget, { transport: "sandbox" }>
| (Extract<AdapterExecutionTarget, { transport: "ssh" }> & { runner: CommandManagedRuntimeRunner });
async function transferArchive(target: RemoteWorkspaceTarget, direction: "syncIn" | "syncOut",
operations: SandboxSyncOperation[], byteSize: number) {
@@ -30,11 +33,34 @@ const checkoutConfiguredRefScript = String.raw`if(input.ref){const remoteRef=inp
await git(['checkout','--detach',selected.stdout.trim()]);}`;
export function requireRemoteWorkspaceTarget(target: AdapterExecutionTarget | null): RemoteWorkspaceTarget {
if (target?.kind === "remote" && target.transport === "ssh") {
return { ...target, runner: createSshCommandManagedRuntimeRunner({ spec: { ...target.spec, remoteCwd: target.remoteCwd }, fileSync: true }) };
}
if (target?.kind !== "remote" || target.transport !== "sandbox" || !target.runner || target.runnerTask
|| !target.runner.syncIn || !target.runner.syncOut) throw new Error("remote_workspace_transport_unsupported");
return target;
}
export async function syncRemoteWorkspaceRecoveryDirectory(input: {
target: RemoteWorkspaceTarget; sourcePath: string; targetPath: string;
}) {
if (input.target.transport === "sandbox") {
await input.target.runner!.syncIn!([{ operationId: randomUUID(), files: [{
kind: "directory", sourcePath: input.sourcePath, targetPath: input.targetPath,
}] }]);
return;
}
const scratch = await mkdtemp(path.join(os.tmpdir(), "pc-recovery-transfer-"));
try {
const archivePath = path.join(scratch, "recovery.pcws");
const captured = await captureRemoteWorkspaceDirectory({ cwd: input.sourcePath, archivePath });
const restorationIdentity = createHash("sha256").update(JSON.stringify([input.targetPath, captured.manifestDigest])).digest("hex");
const restored = await restoreRemoteWorkspaceDirectory({ target: input.target, remoteCwd: input.targetPath,
archivePath, manifestDigest: captured.manifestDigest, restorationIdentity });
await cleanupRemoteWorkspaceArchive({ target: input.target, remoteCwd: restored.stagingRoot, operationId: restored.operationId });
} finally { await removeTaskDirectoryTree(scratch); }
}
/** The workspace archive is built inside the sandbox, so a full sandbox disk stops the save. */
export const REMOTE_WORKSPACE_SANDBOX_DISK_FULL_CODE = "remote_workspace_sandbox_disk_full";
export const REMOTE_WORKSPACE_SANDBOX_DISK_FULL =
@@ -1802,7 +1802,7 @@ require('node:module').syncBuiltinESMExports();`);
it("preserves enrolled workspaces after disabling enrollment and rejects a legacy writer", async () => {
const reserved = await reserve({ kind: "empty" });
expect(shouldUseRemoteWorkspaces({ adapterType: "paperclip_runner", environmentDriver: "sandbox", experimentalEnabled: false, existingWorkspace: reserved })).toBe(true);
expect(() => shouldUseRemoteWorkspaces({ adapterType: "codex_local", environmentDriver: "sandbox", experimentalEnabled: false, existingWorkspace: reserved })).toThrow("native sandbox");
expect(() => shouldUseRemoteWorkspaces({ adapterType: "codex_local", environmentDriver: "sandbox", experimentalEnabled: false, existingWorkspace: reserved })).toThrow("supported remote execution environment");
});
it("keeps enrollment conditional while enrolled authority fails closed when persistence disappears", async () => {
+21 -10
View File
@@ -33,7 +33,7 @@ import {
captureRemoteWorkspaceArchive, cleanupRemoteWorkspaceArchive, prepareRemoteWorkspaceDirectory, prepareRemoteRepository, remoteWorkspaceRepositoryPath,
RemoteWorkspaceOperationTimeoutError, RemoteWorkspacePreparationTimeoutError, remoteWorkspaceFailureCause, hasRemoteRecoveryAttachment,
requireRemoteWorkspaceTarget, restoreRemoteWorkspaceDirectory, RemoteWorkspaceRestoreError, type RemoteWorkspaceRestoreStep,
type RemoteWorkspaceSavePhaseTiming,
type RemoteWorkspaceSavePhaseTiming, syncRemoteWorkspaceRecoveryDirectory,
} from "./remote-workspace-transport.js";
import {
beginWorkspaceSnapshot, findUnchangedWorkspaceSnapshot, getWorkspaceSnapshot, loadWorkspaceSnapshotArchive, newWorkspaceSnapshotStorageTiming, uploadWorkspaceSnapshot,
@@ -94,25 +94,32 @@ function sameExportLocation(value: unknown, location: NonNullable<RemoteWorkspac
return (["environmentId", "leaseId", "providerLeaseId", "remoteCwd"] as const).every(key => recorded[key] === location[key]);
}
export function supportsSshRemoteWorkspace(adapterType: string, environmentDriver: string | null | undefined) {
return environmentDriver === "ssh" && ["claude_local", "codex_local", "paperclip_runner"].includes(adapterType);
}
export function shouldUseRemoteWorkspaces(input: {
adapterType: string; environmentDriver: string | null; experimentalEnabled: boolean; existingWorkspace?: { metadata?: Record<string, unknown> | null } | null;
}) {
const enrolled = readRemoteWorkspaceAuthority(input.existingWorkspace?.metadata) !== null;
if (enrolled && (input.adapterType !== "paperclip_runner" || input.environmentDriver !== "sandbox")) {
throw conflict("This workspace requires its native sandbox execution environment.", { code: "remote_workspace_execution_required" });
const supported = input.adapterType === "paperclip_runner" && input.environmentDriver === "sandbox"
|| supportsSshRemoteWorkspace(input.adapterType, input.environmentDriver);
if (enrolled && !supported) {
throw conflict("This workspace requires a supported remote execution environment.", { code: "remote_workspace_execution_required" });
}
if (enrolled) return true;
// Existing host authority changes only through the explicit import workflow.
if (input.existingWorkspace || !input.experimentalEnabled || input.adapterType !== "paperclip_runner"
|| input.environmentDriver !== "sandbox") return false;
if (input.existingWorkspace || !input.experimentalEnabled || !supported) return false;
return configuredRemoteWorkspacePersistence() !== null;
}
export function assertEphemeralWorkspaceExecution(input: {
externalConversation: boolean; adapterType: string;
pluginTaskWorkspace: boolean; remoteAuthority: boolean;
environmentDriver?: string | null;
}) {
if (process.env.PAPERCLIP_EPHEMERAL_SERVER !== "true" || input.externalConversation) return;
if (input.remoteAuthority && supportsSshRemoteWorkspace(input.adapterType, input.environmentDriver)) return;
// Runtime selection has not promoted a fresh run yet. Validate storage here;
if (input.adapterType !== "paperclip_runner"
|| (!input.pluginTaskWorkspace && !input.remoteAuthority)) {
@@ -121,8 +128,13 @@ export function assertEphemeralWorkspaceExecution(input: {
}
}
export function assertEphemeralNativeRuntime(native: boolean) {
if (process.env.PAPERCLIP_EPHEMERAL_SERVER === "true" && !native) {
export function assertEphemeralNativeRuntime(native: boolean, ssh?: {
adapterType: string; environmentDriver: string | null; remoteAuthority: boolean;
remoteCwd: string | null; authorityCwd: string | null;
}) {
const durableSsh = ssh?.remoteAuthority && supportsSshRemoteWorkspace(ssh.adapterType, ssh.environmentDriver)
&& Boolean(ssh.remoteCwd) && ssh.remoteCwd === ssh.authorityCwd;
if (process.env.PAPERCLIP_EPHEMERAL_SERVER === "true" && !native && !durableSsh) {
throw unprocessable("Volume-free servers require native Paperclip Runner execution with remote workspace storage.",
{ code: "ephemeral_execution_configuration_required" });
}
@@ -556,9 +568,8 @@ export async function prepareRemoteWorkspace(input: {
const provenance = await readFile(path.join(item.path, "provenance.json"), "utf8");
const exists = await hasRemoteRecoveryAttachment({ target, remoteCwd: remoteRoot, operationId: item.operationId,
provenance });
if (!exists) await target.runner!.syncIn!([{ operationId: randomUUID(), files: [{
kind: "directory", sourcePath: item.path, targetPath: path.posix.join(remoteRoot, relative),
}] }]);
if (!exists) await syncRemoteWorkspaceRecoveryDirectory({ target,
sourcePath: item.path, targetPath: path.posix.join(remoteRoot, relative) });
recoveredFiles.push({ operationId: item.operationId, sourceRunId: item.sourceRunId, path: relative, provenance });
}
} finally { await removeTaskDirectoryTree(scratch); }
@@ -0,0 +1,274 @@
import { randomUUID } from "node:crypto";
import { mkdir, mkdtemp, readFile, realpath, rm, writeFile } from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { and, eq } from "drizzle-orm";
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
import { agents, authUsers, companyMemberships, closeRegisteredClients, companies, createDb, environmentLeases, environments,
executionWorkspaces, heartbeatRuns, issues } from "@paperclipai/db";
import type { AdapterSshExecutionTarget } from "@paperclipai/adapter-utils/execution-target";
import { buildSshEnvLabFixtureConfig, getSshEnvLabSupport, startSshEnvLabFixture,
stopSshEnvLabFixture, type SshEnvLabFixtureState } from "@paperclipai/adapter-utils/ssh";
import type { EnvironmentLease } from "@paperclipai/shared";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../__tests__/helpers/embedded-postgres.js";
import { createLocalDiskStorageProvider } from "../storage/local-disk-provider.js";
import { createStorageService } from "../storage/service.js";
import { prepareRemoteWorkspace, readRemoteWorkspaceAuthority, reserveRemoteWorkspace, saveRemoteWorkspace } from "./remote-workspaces.js";
import { requireRemoteWorkspaceTarget } from "./remote-workspace-transport.js";
import { remoteTerminationReceipt } from "./remote-execution-termination.js";
import { withNativeWorkspaceFinalizationOwnership } from "./native-runtime/native-workspace-finalization-ownership.js";
import { heartbeatService } from "./heartbeat.js";
import { aiConnectionService } from "./ai-connections.js";
import * as storageModule from "../storage/index.js";
import { collectEnvironmentSecretRefs, normalizeEnvironmentConfigForPersistence } from "./environment-config.js";
import { secretService } from "./secrets.js";
import * as runLogModule from "./run-log-store.js";
import { objectLogFixture } from "./heartbeat/__tests__/object-log-fixture.js";
import { issueService } from "./issues.js";
import { getServerAdapter } from "../adapters/index.js";
import * as recoverySnapshots from "./workspace-recovery-snapshot.js";
import { hasRemoteRecoveryAttachment } from "./remote-workspace-transport.js";
const [postgres, ssh] = await Promise.all([getEmbeddedPostgresTestSupport(), getSshEnvLabSupport()]);
describe.skipIf(!postgres.supported || !ssh.supported)("Cloud SSH durable workspace with a real SSH server", () => {
let root: string;
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
let db: ReturnType<typeof createDb>;
let sshd: SshEnvLabFixtureState | undefined;
beforeAll(async () => {
root = await realpath(await mkdtemp(path.join(os.tmpdir(), "cloud-ssh-workspace-")));
database = await startEmbeddedPostgresTestDatabase("cloud-ssh-workspace-");
db = createDb(database.connectionString);
vi.stubEnv("DATABASE_URL", database.connectionString);
vi.stubEnv("PAPERCLIP_HOME", path.join(root, "controller-home"));
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "62".repeat(32));
sshd = await startSshEnvLabFixture({ statePath: path.join(root, "sshd", "state.json") });
}, 60_000);
afterAll(async () => {
if (sshd) await stopSshEnvLabFixture(sshd);
if (database) await closeRegisteredClients(database.connectionString);
await database?.cleanup();
if (root) await rm(root, { recursive: true, force: true });
vi.restoreAllMocks();
vi.unstubAllEnvs();
}, 60_000);
it("keeps Hello World through controller replacement and restores it to a new SSH allocation", async () => {
const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(), runId = randomUUID();
await db.insert(companies).values({ id: companyId, name: "SSH workspace test", issuePrefix: "SSH" });
await db.insert(agents).values({ id: agentId, companyId, name: "SSH Claude", adapterType: "claude_local" });
await db.insert(issues).values({ id: issueId, companyId, title: "Write Hello World", assigneeAgentId: agentId });
const config = await buildSshEnvLabFixtureConfig(sshd!);
const [environment] = await db.insert(environments).values({ driver: "ssh", name: "SSH test", config: {} }).returning();
const storage = { ...createStorageService(createLocalDiskStorageProvider(path.join(root, "objects"))), provider: "s3" as const };
async function acquire(id: string, remoteCwd = config.remoteWorkspacePath, providerLeaseId = "ssh-fixture") {
await db.insert(heartbeatRuns).values({ id, companyId, agentId, issueId, runtimeMode: "legacy", status: "running" });
const [lease] = await db.insert(environmentLeases).values({ companyId, environmentId: environment.id,
heartbeatRunId: id, issueId, provider: "ssh", providerLeaseId, status: "active",
metadata: { driver: "ssh", remoteCwd } }).returning();
const target: AdapterSshExecutionTarget = { kind: "remote", transport: "ssh", environmentId: environment.id,
leaseId: lease.id, remoteCwd, spec: { ...config, remoteCwd, useHostAuth: false } };
return { lease: lease as EnvironmentLease, target };
}
async function stop(id: string, lease: EnvironmentLease) {
await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date() }).where(eq(heartbeatRuns.id, id));
const [stopped] = await db.update(environmentLeases).set({ metadata: { ...lease.metadata,
remoteExecutionTermination: remoteTerminationReceipt(lease, { state: "stopped", providerLeaseId: lease.providerLeaseId }) } })
.where(eq(environmentLeases.id, lease.id)).returning();
return stopped as EnvironmentLease;
}
const first = await acquire(runId);
const reserved = await reserveRemoteWorkspace({ db, storage, companyId, issueId, runId, agentId,
environmentId: environment.id, projectId: null, projectWorkspaceId: null, existingWorkspaceId: null, source: { kind: "empty" }, mode: "shared_workspace" });
const prepared = await prepareRemoteWorkspace({ db, storage, workspace: reserved, runId, ...first });
const firstTarget = { ...first.target, remoteCwd: prepared.remoteRoot, spec: { ...first.target.spec, remoteCwd: prepared.remoteRoot } };
const result = await requireRemoteWorkspaceTarget(firstTarget).runner!.execute({ command: "node", args: ["-e",
"require('fs').writeFileSync('hello_world.txt','Hello World\\n')"], cwd: prepared.remoteRoot, timeoutMs: 10_000 });
expect(result.exitCode).toBe(0);
const stopped = await stop(runId, prepared.lease);
await expect(withNativeWorkspaceFinalizationOwnership({ db, companyId, runId }, async () => {}))
.rejects.toThrow("binding_missing");
await expect(withNativeWorkspaceFinalizationOwnership({ db, companyId, runId, remoteWorkspaceId: randomUUID() }, async () => {}))
.rejects.toThrow("binding_missing");
await expect(withNativeWorkspaceFinalizationOwnership({ db, companyId, runId, remoteWorkspaceId: prepared.workspace.id }, ownership =>
saveRemoteWorkspace({ db, storage, workspace: prepared.workspace, runId, lease: stopped,
target: { ...firstTarget, spec: { ...firstTarget.spec, host: "127.0.0.1", port: 1 } }, assertOwned: ownership.assertHeld })))
.rejects.toThrow();
const [held] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, reserved.id));
expect(readRemoteWorkspaceAuthority(held.metadata)).toMatchObject({ writerRunId: runId, dirty: true,
snapshotId: readRemoteWorkspaceAuthority(prepared.workspace.metadata)!.snapshotId });
expect(await readFile(path.join(prepared.remoteRoot, "hello_world.txt"), "utf8")).toBe("Hello World\n");
const blockedId = randomUUID();
await db.insert(heartbeatRuns).values({ id: blockedId, companyId, agentId, issueId, runtimeMode: "legacy", status: "running" });
await expect(reserveRemoteWorkspace({ db, storage, companyId, issueId, runId: blockedId, agentId,
environmentId: environment.id, projectId: null, projectWorkspaceId: null,
existingWorkspaceId: prepared.workspace.id, source: { kind: "empty" } })).rejects.toMatchObject({ status: 409 });
const peerIssueId = randomUUID(), peerRunId = randomUUID();
await db.insert(issues).values({ id: peerIssueId, companyId, title: "Use the saved shared workspace", assigneeAgentId: agentId,
executionWorkspaceId: prepared.workspace.id });
await db.insert(heartbeatRuns).values({ id: peerRunId, companyId, agentId, issueId: peerIssueId, runtimeMode: "legacy", status: "running" });
const finalized = await withNativeWorkspaceFinalizationOwnership({ db, companyId, runId, remoteWorkspaceId: prepared.workspace.id }, async ownership => {
await saveRemoteWorkspace({ db, storage, workspace: prepared.workspace, runId,
lease: stopped, target: firstTarget, assertOwned: ownership.assertHeld });
const handedOn = await reserveRemoteWorkspace({ db, storage, companyId, issueId: peerIssueId, runId: peerRunId, agentId,
environmentId: environment.id, projectId: null, projectWorkspaceId: null, existingWorkspaceId: prepared.workspace.id, source: { kind: "empty" } });
expect(readRemoteWorkspaceAuthority(handedOn.metadata)).toMatchObject({ epoch: 2, writerRunId: peerRunId });
await ownership.assertHeld();
await expect(saveRemoteWorkspace({ db, storage, workspace: prepared.workspace, runId,
lease: stopped, target: firstTarget, assertOwned: ownership.assertHeld })).rejects.toThrow("authority_changed");
});
expect(finalized.acquired).toBe(true);
const [finishedOwner] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
expect(finishedOwner.runnerProfileJson).not.toHaveProperty("nativeWorkspaceFinalizationOwner");
await expect(withNativeWorkspaceFinalizationOwnership({ db, companyId, runId, remoteWorkspaceId: prepared.workspace.id }, async () => {}))
.rejects.toThrow("binding_missing");
await db.update(heartbeatRuns).set({ status: "cancelled", finishedAt: new Date() }).where(eq(heartbeatRuns.id, peerRunId));
// Replace the controller connection. The saved task and remote files are the authority.
db = createDb(database.connectionString);
const [saved] = await db.select().from(executionWorkspaces).where(and(eq(executionWorkspaces.id, reserved.id), eq(executionWorkspaces.companyId, companyId)));
const snapshotId = readRemoteWorkspaceAuthority(saved.metadata)!.snapshotId;
expect(snapshotId).toBeTruthy();
expect(await readFile(path.join(prepared.remoteRoot, "hello_world.txt"), "utf8")).toBe("Hello World\n");
const followupId = randomUUID();
const followup = await acquire(followupId);
const reused = await reserveRemoteWorkspace({ db, storage, companyId, issueId, runId: followupId, agentId,
environmentId: environment.id, projectId: null, projectWorkspaceId: null, existingWorkspaceId: saved.id, source: { kind: "empty" } });
const continued = await prepareRemoteWorkspace({ db, storage, workspace: reused, runId: followupId, ...followup });
expect(continued.remoteRoot).toBe(prepared.remoteRoot);
await saveRemoteWorkspace({ db, storage, workspace: continued.workspace, runId: followupId,
lease: await stop(followupId, continued.lease), target: followup.target, assertOwned: async () => {} });
const [latest] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, saved.id));
const restoredBase = path.join(root, "replacement-remote");
await mkdir(restoredBase);
const restoreId = randomUUID(), replacement = await acquire(restoreId, restoredBase, "replacement-ssh-allocation");
const restoring = await reserveRemoteWorkspace({ db, storage, companyId, issueId, runId: restoreId, agentId,
environmentId: environment.id, projectId: null, projectWorkspaceId: null, existingWorkspaceId: latest.id, source: { kind: "empty" } });
const restored = await prepareRemoteWorkspace({ db, storage, workspace: restoring, runId: restoreId, ...replacement });
expect(restored.remoteRoot).not.toBe(prepared.remoteRoot);
expect(await readFile(path.join(restored.remoteRoot, "hello_world.txt"), "utf8")).toBe("Hello World\n");
}, 60_000);
it("prepares a verified recovered copy on SSH without directory-sync support", async () => {
const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(), runId = randomUUID(), sourceId = randomUUID(), operationId = randomUUID();
await db.insert(companies).values({ id: companyId, name: "SSH recovery test", issuePrefix: "REC" });
await db.insert(agents).values({ id: agentId, companyId, name: "Recovery worker", adapterType: "claude_local" });
await db.insert(issues).values({ id: issueId, companyId, title: "Recover earlier files", assigneeAgentId: agentId });
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, issueId, runtimeMode: "legacy", status: "running" });
const [environment] = await db.insert(environments).values({ driver: "ssh", name: "SSH recovery", config: {} }).returning();
const config = await buildSshEnvLabFixtureConfig(sshd!);
const [lease] = await db.insert(environmentLeases).values({ companyId, environmentId: environment.id,
heartbeatRunId: runId, issueId, provider: "ssh", providerLeaseId: "ssh-recovery", status: "active" }).returning();
const target: AdapterSshExecutionTarget = { kind: "remote", transport: "ssh", environmentId: environment.id,
leaseId: lease.id, remoteCwd: config.remoteWorkspacePath, spec: { ...config, remoteCwd: config.remoteWorkspacePath } };
const storage = { ...createStorageService(createLocalDiskStorageProvider(path.join(root, "recovery-objects"))), provider: "s3" as const };
const reserved = await reserveRemoteWorkspace({ db, storage, companyId, issueId, runId, agentId,
environmentId: environment.id, projectId: null, projectWorkspaceId: null, existingWorkspaceId: null,
source: { kind: "empty" }, recoverySourceRunIds: [sourceId] });
const provenance = JSON.stringify({ operationId, sourceRunId: sourceId });
// Start at the verified extraction boundary; source authorization and snapshot verification keep their existing owner.
const extract = vi.spyOn(recoverySnapshots, "attachPreservedWorkspaceSnapshots").mockImplementationOnce(async input => {
expect(input.sourceRunIds).toContain(sourceId);
const recovered = path.join(input.cwd, "recovered", operationId);
await mkdir(path.join(recovered, "files", "nested"), { recursive: true });
await writeFile(path.join(recovered, "files", "nested", "preserved.txt"), "Earlier work\n");
await writeFile(path.join(recovered, "provenance.json"), provenance);
return [{ operationId, sourceRunId: sourceId, path: recovered }];
});
try {
const prepared = await prepareRemoteWorkspace({ db, storage, workspace: reserved, runId, lease: lease as EnvironmentLease, target });
const recovered = path.join(prepared.remoteRoot, ".paperclip-recovered", operationId);
expect(await readFile(path.join(recovered, "files", "nested", "preserved.txt"), "utf8")).toBe("Earlier work\n");
expect(await readFile(path.join(recovered, "provenance.json"), "utf8")).toBe(provenance);
expect(await hasRemoteRecoveryAttachment({ target: requireRemoteWorkspaceTarget(target), remoteCwd: prepared.remoteRoot, operationId, provenance })).toBe(true);
await writeFile(path.join(recovered, "files", "nested", "preserved.txt"), "A later edit\n");
await prepareRemoteWorkspace({ db, storage, workspace: prepared.workspace, runId, lease: prepared.lease, target });
expect(await readFile(path.join(recovered, "files", "nested", "preserved.txt"), "utf8")).toBe("A later edit\n");
} finally { extract.mockRestore(); }
}, 60_000);
it("runs an assigned Claude task through heartbeat with the configured Paperclip account", async () => {
vi.stubEnv("PAPERCLIP_EPHEMERAL_SERVER", "true");
vi.stubEnv("PAPERCLIP_STORAGE_PROVIDER", "s3");
vi.stubEnv("PAPERCLIP_AGENT_JWT_SECRET", "synthetic-cloud-ssh-jwt-secret");
vi.stubEnv("AWS_EC2_METADATA_DISABLED", "true");
const storage = { ...createStorageService(createLocalDiskStorageProvider(path.join(root, "heartbeat-objects"))), provider: "s3" as const };
vi.spyOn(storageModule, "getStorageService").mockReturnValue(storage);
vi.spyOn(runLogModule, "getRunLogStore").mockReturnValue(objectLogFixture());
const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(), userId = `ssh-user-${randomUUID()}`;
await db.insert(authUsers).values({ id: userId, name: "SSH test user", email: `${userId}@example.test`, createdAt: new Date(), updatedAt: new Date() });
await db.insert(companies).values({ id: companyId, name: "Cloud SSH task", issuePrefix: "HSSH", defaultResponsibleUserId: userId });
await db.insert(companyMemberships).values({ companyId, principalId: userId, principalType: "user", status: "active", membershipRole: "member" });
const account = await aiConnectionService(db).save(companyId, userId, { provider: "anthropic", method: "api_key",
ownership: "personal", name: "Selected test account", apiKey: "synthetic-selected-ssh-key", agentIds: [], allAgents: true }, "synthetic-selected-ssh-key");
const config = await buildSshEnvLabFixtureConfig(sshd!);
const command = path.join(root, "fake-claude");
await writeFile(command, `#!${process.execPath}
if (process.argv.includes('--version')) { console.log('2.1.0'); process.exit(0); }
if (process.env.ANTHROPIC_API_KEY !== 'synthetic-selected-ssh-key') process.exit(91);
const fs = require('fs'), path = require('path');
const history = path.join(process.env.CLAUDE_CONFIG_DIR, 'projects', 'fixture-history');
const resumed = process.argv.includes('--resume');
if (resumed && !fs.existsSync(history)) process.exit(92);
fs.mkdirSync(path.dirname(history), {recursive:true}); fs.writeFileSync(history, 'provider conversation state');
fs.writeFileSync('continuation.json', JSON.stringify({resumed}));
require('fs').writeFileSync('hello_world.txt', 'Hello World\\n');
console.log(JSON.stringify({type:'result',subtype:'success',is_error:false,result:'Created hello_world.txt',session_id:'12345678-1234-4abc-9def-123456789012',total_cost_usd:0,usage:{input_tokens:1,output_tokens:1}}));
`, { mode: 0o700 });
const storedConfig = await normalizeEnvironmentConfigForPersistence({ db, companyId, environmentName: "Heartbeat SSH",
driver: "ssh", secretProvider: "local_encrypted", config: { ...config, useHostAuth: false } });
const [environment] = await db.insert(environments).values({ driver: "ssh", name: "Heartbeat SSH", config: storedConfig }).returning();
await secretService(db).replaceSecretRefsForInstanceTarget({ targetType: "environment", targetId: environment.id },
await collectEnvironmentSecretRefs({ db, environment: { id: environment.id, driver: "ssh", config: storedConfig } }));
await db.insert(agents).values({ id: agentId, companyId, name: "Claude SSH task", adapterType: "claude_local", status: "idle",
defaultEnvironmentId: environment.id, adapterConfig: { command, engine: "cli", model: "claude-sonnet-4-6", timeoutSec: 20 },
runtimeConfig: { aiConnection: { provider: "anthropic", method: "api_key", mode: "delegated", connectionId: account.connectionId, grantId: account.grantId } } });
await db.insert(issues).values({ id: issueId, companyId, title: "Create hello_world.txt", status: "todo", assigneeAgentId: agentId,
responsibleUserId: userId, issueNumber: 1 });
const adapter = getServerAdapter("claude_local"), execute = adapter.execute;
expect(adapter.type).toBe("claude_local");
vi.spyOn(adapter, "execute").mockImplementation(async input => {
const result = await execute(input);
// The synthetic CLI stands in for the provider's final task update too.
if (result.exitCode === 0) await db.update(issues).set({ status: "in_review" }).where(eq(issues.id, issueId));
return result;
});
const heartbeat = heartbeatService(db);
try {
const run = await heartbeat.wakeup(agentId, { source: "on_demand", triggerDetail: "manual", reason: "manual",
payload: { issueId }, contextSnapshot: { issueId } });
expect(run).toBeTruthy();
await heartbeat.drainActiveRunExecutions();
const finished = await heartbeat.getRun(run!.id);
if (finished?.status !== "succeeded") throw new Error(JSON.stringify({ errorCode: finished?.errorCode,
error: finished?.error, log: await heartbeat.readLog(run!.id, { limitBytes: 12_000 }).catch(() => null) }));
expect(finished).toMatchObject({ status: "succeeded", runtimeMode: "legacy" });
const authority = finished!.runnerProfileJson!.remoteWorkspace as { workspaceId: string; state: string; remoteCwd: string };
expect(authority.state).toBe("saved");
expect(await readFile(path.join(authority.remoteCwd, "hello_world.txt"), "utf8")).toBe("Hello World\n");
const [workspace] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, authority.workspaceId));
expect(readRemoteWorkspaceAuthority(workspace.metadata)).toMatchObject({ writerRunId: null, dirty: false });
const comment = await issueService(db).addComment(issueId, "Confirm the file still exists.", { userId });
const followup = await heartbeat.wakeup(agentId, { source: "on_demand", triggerDetail: "manual", reason: "manual",
payload: { issueId }, contextSnapshot: { issueId, wakeCommentId: comment.id } });
await heartbeat.resumeQueuedRunsForAgent(companyId, agentId);
await heartbeat.drainActiveRunExecutions();
const continued = await heartbeat.getRun(followup!.id);
if (continued?.status !== "succeeded") throw new Error(JSON.stringify({ status: continued?.status,
retryReason: continued?.scheduledRetryReason, retryAt: continued?.scheduledRetryAt, errorCode: continued?.errorCode,
error: continued?.error, result: continued?.resultJson, log: await heartbeat.readLog(followup!.id, { limitBytes: 12_000 }).catch(() => null) }));
expect(continued).toMatchObject({ status: "succeeded" });
expect(JSON.parse(await readFile(path.join(authority.remoteCwd, "continuation.json"), "utf8"))).toEqual({ resumed: true });
} finally {
await heartbeat.drainActiveRunExecutions();
}
}, 60_000);
});