Files
PaperClipAI/packages/adapter-utils/src/ssh.ts
T
Devin FoleyandPaperclip 21bf7ebcee feat(connectors): connect Tailscale networks to SSH environments (#20617)
## Thinking Path

> - Paperclip manages work for AI agents.
> - SSH environments run tasks on computers that users own.
> - A Paperclip server needs network access before it can reach a
private computer.
> - A reusable Tailscale connection belongs in Connectors.
> - Connected machines should become choices in the existing SSH
environment form.
> - This change connects a tailnet and resolves selected machines
through a private network client.
> - Agents retain their choice of harness and AI account.

## Linked Issues or Issue Description

**Subsystem affected**

Connectors, environment settings, SSH execution, company secret storage,
and runtime packaging.

**Problem or motivation**

Entering a Tailscale address in the SSH form does not connect the
Paperclip server to that tailnet. Requiring a separate device key for
every environment also duplicates network setup.

**Proposed solution**

Add a Tailscale Connector with **Connect Tailscale Network**. Use
brokered OAuth consent to enroll one private network client per
connection. Let SSH environments select a connected network and stable
machine identity. Preserve the enrolled node state through the company
vault so a restart does not reuse the one-time token.

**Alternatives considered**

A separate Tailscale environment type would duplicate SSH behavior and
mix network access with harness selection. This change uses the existing
`ssh` driver. Related #15648 adds OAuth client credentials and a health
check; this change uses user consent and supplies outbound task
execution. Related #11743 concerns HTTPS brokering.

**Roadmap alignment**

This extends the existing Connected Apps, environment, and Personal &
Shared AI Accounts capabilities. This PR is stacked on #21248, which
provides SSH account selection independently of Tailscale. The hosted
OAuth broker is a companion control-plane change.

## What Changed

- Preserve Cloud reauthentication in the current browser tab. Explain
canceled or failed consent in the connector and offer a retry of the
same authorized saved connection.
- Record Cloud enrollment starts initiated through the Tailscale
connector in the company activity log.
- Check install-preparation descendant liveness through the kernel
process table in the Runner support test. Keep the direct child’s strict
reaping assertion and reject any non-zombie descendant.
- Align the application image stage contract with the Tailscale client
stage after the upstream Runner-image refactor. Keep the exact stage
list and check every application stage, including `tailscale-client`,
for prohibited Runner compiler commands.
- Group Tailscale consent, discovery, daemon lifecycle, pairing, and
deletion under `server/src/services/connectors/tailscale/`, with their
service tests. Update existing route, runtime, environment, vault, and
company-deletion imports. Keep the existing contracts, authorization,
audit, and transaction owners.
- Handle early stdin closure in the Kubernetes real-shell test helper.
Stop the input stream on `EPIPE`, preserve nonzero child exits, and
reject a zero exit when input was lost. Add two real-process regression
cases. Production transfer code and confinement checks are unchanged.
- Read configured Tailscale executable paths when a client starts, after
supported `.env` loading. Keep explicit command overrides available.
- Discard pending pairing results after switching to Direct SSH or
closing the form. Retain the existing company, connection, machine, and
port checks.
- Store network credentials with an immutable server-owned purpose from
their first vault row. Exclude them from generic catalogs, bindings,
environment projection, and value reads. Use a dedicated audited server
consumer for pinned connector reads. Block unclassified legacy
serialized node state at the same value boundary. Keep SSH private-key
binding separate from connector credentials.
- Gate network queries and setup on verified local-implicit or
instance-admin access. Hide Tailscale catalog rows for other members,
show an admin requirement on direct setup navigation, and avoid device
discovery before management access is known.
- Keep failed-client cleanup in the network manager. A failed request
cannot disconnect a newer retry for the same identity; explicit
connector disconnect and enrollment retirement still stop their
networks.
- Check for readable Ed25519, ECDSA, or RSA host public keys before
pairing edits `authorized_keys`. Print a known-hosts entry for every
available key, or fail before changing authorized keys if none are
available.
- Return self-hosted Cloud enrollment to the Tailscale connector. Save
the unprefixed setup path and let the shared callback add the verified
company prefix. Keep external and unrelated destinations rejected.
- Hide pairing instructions after an SSH port change. Ignore late
key-selection results for the old port. Operators prepare again for
matching known-hosts output.
- Retire connector references before deleting a company or its vault
credential. Stop company daemons after the database commit. Keep failed
secret retirement retryable and prevent late node-state writes from
restoring deleted connections. Discard retired state without blocking
behind the vault deletion lock.
- Reload a mismatched credential before replacing a ready daemon.
Recheck the active entry and manager generation after the read. An older
request cannot stop a newer healthy socket or cross shutdown.
- Pin connector credentials to the exact vault version and SHA-256
digest recorded by trusted enrollment and node-state writes. Ignore
generic secret rotations. Advance both fields with a compare-and-set
guard. Reject unpinned or tampered credentials before network startup.
- Save an explicit Direct SSH selection and clear old Tailscale fields
after config merging. Preserve the operator's entered direct host.
- Route all Tailscale SSH runtime access through the connection service
after SSH-key loading. Pin access to the environment's saved connector
identity and recheck after device discovery. Remove standalone key-based
daemon startup and reject raw enrollment keys or credential-only network
configurations. Align the shared config type and SDK pins.
- Correct two E2E assumptions exposed during connector qualification.
Keep the two-window chat fixture stale until its own mutation. Validate
the original failed run and its permitted write-revocation timestamp in
the legacy recovery browser test.
- Add the Tailscale Connector page, catalog entry, OAuth return flow,
connected-network management, device discovery, and disconnect. Document
all six routes in OpenAPI and the API reference, including their
instance-admin requirement.
- Add company-scoped connection storage in `0480_lazy_lethal_legion`,
trusted credential version/digest fields in `0481_round_pet_avengers`,
and server-owned secret purpose/backfill in
`0482_aberrant_rumiko_fujikawa`. Generate all three migrations through
the maintained command and retain the five newest snapshots.
- Add machine pairing that generates a fresh SSH key per request, stores
its private half in the company vault, and uses the generated public
half in the install command. Do not trust editable saved metadata.
Preserve existing authorized keys, including a file without a final
newline. Keep SSH username, workspace, and trusted host keys explicit.
- Use the existing SSH driver. Resolve the machine's current address
from its stable device ID. Reject disconnected, removed, offline, and
wrong-company selections.
- Hide Network when Direct is the only option. Select a sole connected
tailnet automatically and hide its selector. Remove the requested setup
helper copy. Do not restrict the harness to Claude.
- Run a private userspace Tailscale daemon with a separate socket.
Persist its node state before execution. Restore it after restart, bound
startup, serialize reconnect cleanup, and close it during server
shutdown.
- Guard OAuth completion with a durable attempt marker. Disconnect and
newer authorization invalidate older callbacks. Close replaced daemons
and remove cancelled enrollment secrets without changing newer
connection state.
- Preserve SSH host verification and tailnet-specific session identity.
Honor the configured Tailscale executable path, including spaces and
OpenSSH percent escaping.
- Package checksum-pinned Tailscale clients and their license in the
standard runtime image. Exercise the client build stage in PR CI and
include it in image smoke checks.

## Verification

- Merge status: all CI and image checks pass, and current-head Apex is
5/5 with no unresolved comments. GitHub rejected a direct merge because
the active master ruleset requires code-owner approval for
`.github/workflows/pr-trusted.yml`. The [CODEOWNERS workflow
pattern](https://github.com/paperclipai/paperclip/blob/master/.github/CODEOWNERS#L3)
applies; eligible maintainers already have review requests. Auto-merge
is enabled with the squash method for the verified head. The PR remains
open pending that required approval; no branch policy was bypassed.
- Final revision: `fdb29b2cc366f4945b7f4c3e2b2d25070329a21b`, including
master `999709b0a6dd416af335b5debded13e1de7fcb7a`. [Final CI
verification
passed](https://github.com/paperclipai/paperclip/actions/runs/43518584814/job/132213508320).
All required checks pass, all applicable test/build gates are green, and
the branch is mergeable against master. SSH prerequisite #21248 and
Cloud broker paperclipai/paperclip-cloud#1314 are already merged.
- [Apex reviewed this exact revision at
5/5](https://github.com/paperclipai/paperclip/pull/20617#issuecomment-6776629437).
There are no unresolved review threads. The PR is classified as Critical
impact.
- Final image qualification passes for the [application Runner
path](https://github.com/paperclipai/paperclip/actions/runs/43518584522/job/132206784267),
[amd64 Runner
image](https://github.com/paperclipai/paperclip/actions/runs/43518584544/job/132207296198),
and [arm64 Runner
image](https://github.com/paperclipai/paperclip/actions/runs/43518584544/job/132207296156).
Typechecks, Runner support, native/Rust tests, provider tests,
serialized-server suites, browser shards, and the remaining applicable
CI checks pass. This is source/image CI qualification, not a production
rollout.
- The last master conflict affected only Docker stage declarations and
their boundary test. Both the checksum-pinned Tailscale stage and
upstream `runner-lockfile` guard are retained, including the upstream
`deps` inheritance. The Tailscale client stage is byte-identical. All
180 cases across five Docker/image contract suites pass locally, along
with server and UI typechecks. The maintained 543-input SDK check has no
drift. Runtime and lock pins are unchanged. Secret/PII and whitespace
scans pass.
- Seven OAuth regression cases failed before the fixes. The current-tab
test uses the real handoff helper, checks that the session is stored
before reauthentication navigation, mounts the shared return page,
resumes consent, and verifies handoff cleanup. Callback tests check
trusted saved-connection context, fixed safe outcomes, failure audit
metadata, denied membership and expired-state rejection, JSON API
errors, and no provider text in browser returns. UI retries require a
connection in the current company's catalog. All 80 OAuth
route/service/UI/shared-handoff cases pass; server and UI source/test
typechecks pass.
- [Server shard 4/12 on the prior
refresh](https://github.com/paperclipai/paperclip/actions/runs/43510727031/job/132180301436)
passed 2,784 cases and skipped 21, then failed a reassignment fixture's
`TRUNCATE companies CASCADE` cleanup with PostgreSQL `40P01`. Its
automatic retry found the surviving `REA` prefix. The unchanged test
passes all three cases locally. A controlled reader/cleanup overlap
reproduces the same deadlock, but the exact competing CI query was not
logged. Each case now has a distinct company prefix and identifier; the
owned database is removed at suite end. The three test bodies and
assertions are byte-identical. Eight local reassignment/lock-release
cases pass, and the [final server shard 4/12
passes](https://github.com/paperclipai/paperclip/actions/runs/43518584814/job/132207159392).
Production reassignment logic, deadlines, and retries are unchanged.
- The initial master integration passed 330 cases across 17 suites.
Coverage includes connector state, device discovery, network lifecycle,
pairing, routes, SSH binding and exports, environment persistence, and
connector UI. All three unpublished migrations were regenerated after
master's `0479_short_darkstar`: `0480_lazy_lethal_legion`,
`0481_round_pet_avengers`, and `0482_aberrant_rumiko_fujikawa`. Each SQL
body is byte-identical to its earlier unpublished migration. Generated
snapshots preserve existing master tables and add only the intended
Tailscale table and secret-purpose column. Exactly five snapshots
remain, 0478–0482. Numbering, safety, compatibility, and snapshot-drift
checks pass. The later Docker refresh did not alter this migration
sequence.
- Earlier attempts remain part of the verification record. On `1c020a0`,
Rust cache restoration and server shard 5/12 were interrupted by
shutdown signals, Runner shard 2/3 lost communication with no
retrievable log, and an amd64 provider download failed with npm
`ECONNRESET`. Same-head retries recovered, without establishing the
cause of the communication loss. Later superseded runs also had
cancellations with incomplete logs. These interrupted attempts are not
counted as passes. The final revision has fresh successful CI.
- Local platform limits remain recorded. The Kubernetes suite on macOS
passed 191 cases, failed 12 Linux `/proc/self/fd` transfer cases, and
skipped four Linux-only cases. Unchanged source reproduced the same 12
failures. An initial run also lacked provider dependencies until the
maintained preparation command installed them. Those local attempts are
not green qualification. Final Linux provider CI passes. Local Docker
qualification was unavailable; final image checks passed in CI.
- Before PR separation, a live tailnet/SSH Claude Code task completed on
Ubuntu ARM64 without provider login for the remote user. The selected
Paperclip subscription ran it, wrote the expected file, posted
completion, and marked the task done. File contents and private runtime
cleanup were checked. This is prior live evidence; the final revision
was not live-retested. Other providers have automated coverage only.
Hosted registration for unrelated tailnets and production concurrency
remain qualification limits below.
- The Superagent security and supply-chain reports remain neutral and
inconclusive. Earlier attempts to restart the failed security scan
returned HTTP 404. Supply-chain intelligence lacked coverage for
extracted pinned dependencies and the shared workflow reference. These
reports do not establish successful security qualification. The required
CI and review gates pass; this external coverage gap remains disclosed.

## Risks

- The Superagent security and supply-chain reports are inconclusive. The
security scan re-run request returned HTTP 404, and the supply-chain
scanner lacks coverage for the extracted dependencies and shared
workflow reference. These neutral results are not successful security
qualification.
- The server-owned secret purpose is internal database metadata, not an
HTTP-writable field. The generic secret service owns enforcement so
direct API reads and runtime projection share the same restriction.
Previously copied node state remains valid outside Paperclip until the
enrolled device is revoked in Tailscale; a local disconnect cannot stop
an external copy.
- Tailscale network configurations now require a connected Connector and
selected machine. The unreleased key-only path is removed; any saved
configuration from that path needs Connector setup.
- The connector remains a pilot. The local test used an OAuth app
registered in the consenting user's tailnet. Qualify hosted app
registration for unrelated customer tailnets before general
availability.
- This implementation supports one serving process per instance.
Concurrent replicas need exclusive ownership of the persisted network
identity before production rollout. Preserve the database and company
vault master key across restarts.
- Each machine-pairing preparation creates a new vault secret. Unused
generated keys follow the existing secret deletion lifecycle. A saved
SSH key remains available as an explicit choice.
- Apply additive migrations `0480_lazy_lethal_legion`,
`0481_round_pet_avengers`, and `0482_aberrant_rumiko_fujikawa` before
application rollout. The new credential version and digest columns are
nullable. Existing unpinned connections require fresh OAuth consent; the
migration deliberately does not trust or backfill a generic secret
value. Regenerate or renumber migrations if master advances the sequence
before merge.
- The target needs an SSH server, its chosen harness, and allowed
tailnet access. A reachable network does not grant SSH or provider
authorization. Pairing reads standard `/etc/ssh` host-public-key files;
custom host-key locations require manual known-hosts configuration.
- Roll back the application revision to remove new entry points.
Existing connection secrets and registered Tailscale nodes retain their
separate lifecycle. Do not drop the table or delete node state as part
of an application rollback.
- Production qualification still needs denied-policy, revocation,
cancellation, and deployment-overlap checks. Local Docker and a
production deployment were not performed.

## Model Used

OpenAI Codex, gpt-6.1-sol, xhigh reasoning, with local code execution
and tool use.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used
- [x] I have checked ROADMAP.md and this extends existing connector and
environment features
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have described the issue in-PR using the feature request fields
- [x] I have not referenced internal issues, local URLs, or private
machine identifiers
- [x] My branch name describes the change
- [x] I have run focused tests locally and they pass
- [x] I have added tests for new behavior and failure paths
- [x] I have updated documentation
- [x] I have documented migration and runtime risks
- [x] All Paperclip CI gates are green
- [ ] Required code-owner approval received for the workflow change
- [x] Greptile Apex is 5/5 on the final head with no unresolved comments
- [x] I will address CI and review findings before requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-13 15:15:43 -08:00

2248 lines
75 KiB
TypeScript

import { randomUUID } from "node:crypto";
import { execFile, spawn } from "node:child_process";
import { constants as fsConstants, createReadStream, createWriteStream, promises as fs } from "node:fs";
import net from "node:net";
import os from "node:os";
import path from "node:path";
import { pipeline, Transform } from "node:stream";
import type { CommandManagedRuntimeRunner } from "./command-managed-runtime.js";
import {
createUnrelatedHistoryGraftCommit,
GIT_SYNC_COMMIT_IDENTITY_ARGS,
listProjectRepositoryStagingPaths,
PROJECT_REPOSITORIES_DIR,
PROJECT_REPOSITORY_STAGING_PATTERN,
readSanitizedOriginRemoteUrl,
resetLocalGitIndexToHead,
} from "./git-workspace-sync.js";
import type { RunProcessResult } from "./server-utils.js";
import type { DirectorySnapshot } from "./workspace-restore-merge.js";
import { mergeDirectoryWithBaseline } from "./workspace-restore-merge.js";
import {
createRuntimeProgressReporter,
type RuntimeProgressDirection,
type RuntimeProgressPhase,
type RuntimeProgressSink,
} from "./runtime-progress.js";
const SSH_STDIN_COMMAND = "IFS= read -r paperclip_ssh_command_b64 || exit 125; paperclip_ssh_command=$(printf '%s' \"$paperclip_ssh_command_b64\" | base64 -d) || exit 125; unset paperclip_ssh_command_b64; eval \"$paperclip_ssh_command\"";
export interface SshConnectionConfig {
useHostAuth?: boolean;
tailscaleSocket?: string;
tailscaleIdentity?: string;
host: string;
port: number;
username: string;
remoteWorkspacePath: string;
privateKey: string | null;
knownHosts: string | null;
strictHostKeyChecking: boolean;
}
export interface SshCommandResult {
stdout: string;
stderr: string;
}
export interface SshRemoteExecutionSpec extends SshConnectionConfig {
remoteCwd: string;
}
export function createSshCommandManagedRuntimeRunner(input: {
spec: SshRemoteExecutionSpec;
defaultCwd?: string | null;
maxBufferBytes?: number | null;
}): CommandManagedRuntimeRunner {
const defaultCwd = input.defaultCwd?.trim() || input.spec.remoteCwd;
const maxBufferBytes =
typeof input.maxBufferBytes === "number" && Number.isFinite(input.maxBufferBytes) && input.maxBufferBytes > 0
? Math.trunc(input.maxBufferBytes)
: 1024 * 1024;
return {
execute: async (commandInput): Promise<RunProcessResult> => {
const startedAt = new Date().toISOString();
const command = commandInput.command.trim();
const args = commandInput.args ?? [];
const cwd = commandInput.cwd?.trim() || defaultCwd;
const envEntries = Object.entries(commandInput.env ?? {})
.filter((entry): entry is [string, string] => typeof entry[1] === "string");
for (const [key] of envEntries) {
if (!isValidShellEnvKey(key)) {
throw new Error(`Invalid SSH environment variable key: ${key}`);
}
}
const commandScript = `exec ${[shellQuote(command), ...args.map((arg) => shellQuote(arg))].join(" ")}`;
const remoteCommand = `cd ${shellQuote(cwd)} && ${commandScript}`;
try {
const result = await runSshCommand(input.spec, remoteCommand, {
env: Object.fromEntries(envEntries),
stdin: commandInput.stdin,
timeoutMs: commandInput.timeoutMs,
maxBuffer: maxBufferBytes,
});
if (result.stdout) await commandInput.onLog?.("stdout", result.stdout);
if (result.stderr) await commandInput.onLog?.("stderr", result.stderr);
return {
exitCode: 0,
signal: null,
timedOut: false,
stdout: result.stdout,
stderr: result.stderr,
pid: null,
startedAt,
};
} catch (error) {
const failure = error as {
stdout?: unknown;
stderr?: unknown;
code?: unknown;
signal?: unknown;
killed?: unknown;
};
const stdout = typeof failure.stdout === "string" ? failure.stdout : "";
const stderr = typeof failure.stderr === "string"
? failure.stderr
: error instanceof Error
? error.message
: String(error);
if (stdout) await commandInput.onLog?.("stdout", stdout);
if (stderr) await commandInput.onLog?.("stderr", stderr);
return {
exitCode: typeof failure.code === "number" ? failure.code : null,
signal: typeof failure.signal === "string" ? failure.signal : null,
timedOut: failure.killed === true,
stdout,
stderr,
pid: null,
startedAt,
};
}
},
};
}
export interface SshEnvLabSupport {
supported: boolean;
reason: string | null;
}
export interface SshEnvLabFixtureState {
kind: "ssh_openbsd";
bindHost: string;
host: string;
port: number;
username: string;
rootDir: string;
workspaceDir: string;
statePath: string;
pid: number;
createdAt: string;
clientPrivateKeyPath: string;
clientPublicKeyPath: string;
hostPrivateKeyPath: string;
hostPublicKeyPath: string;
authorizedKeysPath: string;
knownHostsPath: string;
sshdConfigPath: string;
sshdLogPath: string;
}
interface LocalGitWorkspaceSnapshot {
headCommit: string;
branchName: string | null;
deletedPaths: string[];
}
export function shellQuote(value: string) {
return `'${value.replace(/'/g, `'\"'\"'`)}'`;
}
function isValidShellEnvKey(value: string) {
return /^[A-Za-z_][A-Za-z0-9_]*$/.test(value);
}
export function parseSshRemoteExecutionSpec(value: unknown): SshRemoteExecutionSpec | null {
if (!value || typeof value !== "object" || Array.isArray(value)) {
return null;
}
const parsed = value as Record<string, unknown>;
const host = typeof parsed.host === "string" ? parsed.host.trim() : "";
const username = typeof parsed.username === "string" ? parsed.username.trim() : "";
const remoteCwd = typeof parsed.remoteCwd === "string" ? parsed.remoteCwd.trim() : "";
const portValue = typeof parsed.port === "number" ? parsed.port : Number(parsed.port);
if (!host || !username || !remoteCwd || !Number.isInteger(portValue) || portValue < 1 || portValue > 65535) {
return null;
}
if (parsed.useHostAuth !== undefined && typeof parsed.useHostAuth !== "boolean") return null;
if (parsed.tailscaleSocket !== undefined && (typeof parsed.tailscaleSocket !== "string" || !/^\/[A-Za-z0-9_./-]+$/.test(parsed.tailscaleSocket))) return null;
if (parsed.tailscaleIdentity && !parsed.tailscaleSocket) return null;
return {
...(typeof parsed.useHostAuth === "boolean" ? { useHostAuth: parsed.useHostAuth } : {}),
...(typeof parsed.tailscaleSocket === "string" ? { tailscaleSocket: parsed.tailscaleSocket } : {}),
...(typeof parsed.tailscaleIdentity === "string" ? { tailscaleIdentity: parsed.tailscaleIdentity } : {}),
host,
port: portValue,
username,
remoteCwd,
remoteWorkspacePath:
typeof parsed.remoteWorkspacePath === "string" && parsed.remoteWorkspacePath.trim().length > 0
? parsed.remoteWorkspacePath.trim()
: remoteCwd,
privateKey: typeof parsed.privateKey === "string" && parsed.privateKey.length > 0 ? parsed.privateKey : null,
knownHosts: typeof parsed.knownHosts === "string" && parsed.knownHosts.length > 0 ? parsed.knownHosts : null,
strictHostKeyChecking:
typeof parsed.strictHostKeyChecking === "boolean" ? parsed.strictHostKeyChecking : true,
};
}
async function execFileText(
file: string,
args: string[],
options: {
timeout?: number;
maxBuffer?: number;
} = {},
): Promise<SshCommandResult> {
return await new Promise<SshCommandResult>((resolve, reject) => {
execFile(
file,
args,
{
timeout: options.timeout ?? 15_000,
maxBuffer: options.maxBuffer ?? 1024 * 128,
},
(error, stdout, stderr) => {
if (error) {
reject(Object.assign(error, { stdout: stdout ?? "", stderr: stderr ?? "" }));
return;
}
resolve({
stdout: stdout ?? "",
stderr: stderr ?? "",
});
},
);
});
}
async function spawnText(
file: string,
args: string[],
options: {
stdin?: string;
timeout?: number;
maxBuffer?: number;
} = {},
): Promise<SshCommandResult> {
return await new Promise<SshCommandResult>((resolve, reject) => {
const child = spawn(file, args, {
stdio: [options.stdin != null ? "pipe" : "ignore", "pipe", "pipe"],
});
const maxBuffer = options.maxBuffer ?? 1024 * 128;
let stdout = "";
let stderr = "";
let settled = false;
let timedOut = false;
const finishReject = (error: Error & { stdout?: string; stderr?: string; code?: number | null; killed?: boolean }) => {
if (settled) return;
settled = true;
error.stdout = stdout;
error.stderr = stderr;
error.killed = timedOut;
reject(error);
};
const append = (
streamName: "stdout" | "stderr",
chunk: unknown,
) => {
const text = String(chunk);
if (streamName === "stdout") {
stdout += text;
} else {
stderr += text;
}
if (Buffer.byteLength(stdout, "utf8") > maxBuffer || Buffer.byteLength(stderr, "utf8") > maxBuffer) {
child.kill("SIGTERM");
finishReject(Object.assign(new Error(`Process output exceeded maxBuffer of ${maxBuffer} bytes.`), {
code: null,
}));
}
};
let killEscalation: NodeJS.Timeout | null = null;
const timeout = options.timeout && options.timeout > 0
? setTimeout(() => {
timedOut = true;
child.kill("SIGTERM");
// Escalate to SIGKILL after a 5s grace window so a hung remote
// command that ignores SIGTERM cannot keep the child alive
// indefinitely.
killEscalation = setTimeout(() => {
try {
child.kill("SIGKILL");
} catch {
// child may have already exited between the SIGTERM and the
// escalation — that's fine.
}
}, 5_000);
killEscalation.unref?.();
}, options.timeout)
: null;
const clearTimers = () => {
if (timeout) clearTimeout(timeout);
if (killEscalation) clearTimeout(killEscalation);
};
child.stdout?.on("data", (chunk) => {
append("stdout", chunk);
});
child.stderr?.on("data", (chunk) => {
append("stderr", chunk);
});
child.on("error", (error) => {
clearTimers();
finishReject(Object.assign(error, { code: null }));
});
child.on("close", (code, signal) => {
clearTimers();
if (settled) return;
settled = true;
if (code === 0) {
resolve({ stdout, stderr });
return;
}
reject(Object.assign(new Error(stderr.trim() || stdout.trim() || `Process exited with code ${code ?? -1}`), {
stdout,
stderr,
code,
signal,
killed: timedOut,
}));
});
if (options.stdin != null && child.stdin) {
child.stdin.on("error", (error) => {
child.kill("SIGTERM");
finishReject(Object.assign(error, { code: null }));
});
child.stdin.end(options.stdin);
}
});
}
async function runLocalGit(
localDir: string,
args: string[],
options: {
timeout?: number;
maxBuffer?: number;
} = {},
): Promise<SshCommandResult> {
return await execFileText("git", ["-C", localDir, ...args], options);
}
async function commandExists(command: string): Promise<boolean> {
return (await resolveCommandPath(command)) !== null;
}
async function resolveCommandPath(command: string): Promise<string | null> {
try {
const result = await execFileText("sh", ["-c", `command -v ${shellQuote(command)}`], {
timeout: 5_000,
maxBuffer: 8 * 1024,
});
const resolved = result.stdout.trim().split("\n")[0]?.trim() ?? "";
return resolved.length > 0 ? resolved : null;
} catch {
return null;
}
}
async function withTempFile(
prefix: string,
contents: string,
mode: number,
): Promise<{ path: string; cleanup: () => Promise<void> }> {
const dir = await fs.mkdtemp(path.join(os.tmpdir(), prefix));
const filePath = path.join(dir, "payload");
const normalizedContents = contents.endsWith("\n") ? contents : `${contents}\n`;
await fs.writeFile(filePath, normalizedContents, { mode, encoding: "utf8" });
return {
path: filePath,
cleanup: async () => {
await fs.rm(dir, { recursive: true, force: true }).catch(() => undefined);
},
};
}
async function createSshAuthArgs(
config: Pick<SshConnectionConfig, "privateKey" | "knownHosts" | "strictHostKeyChecking" | "tailscaleSocket" | "tailscaleIdentity" | "host">,
): Promise<{ args: string[]; cleanup: () => Promise<void> }> {
const tempFiles: Array<() => Promise<void>> = [];
const sshArgs = [
"-o",
"BatchMode=yes",
"-o",
"ConnectTimeout=10",
"-o",
`StrictHostKeyChecking=${config.strictHostKeyChecking ? "yes" : "no"}`,
];
if (config.tailscaleIdentity && !config.tailscaleSocket) throw new Error("Tailscale connection is unavailable");
if (config.tailscaleSocket) {
if (!net.isIP(config.host) && !/^[A-Za-z0-9][A-Za-z0-9_.-]*$/.test(config.host)) throw new Error("Invalid Tailscale host");
if (!/^\/[A-Za-z0-9_./-]+$/.test(config.tailscaleSocket)) throw new Error("Invalid Tailscale socket path");
const configuredCli = process.env.PAPERCLIP_TAILSCALE_CLI?.trim();
const cli = configuredCli ? shellQuote(configuredCli).replaceAll("%", "%%") : "tailscale";
sshArgs.push("-o", `ProxyCommand=${cli} --socket=${config.tailscaleSocket} nc %h %p`);
}
if (config.strictHostKeyChecking) {
if (config.knownHosts) {
const knownHosts = await withTempFile("paperclip-ssh-known-hosts-", config.knownHosts, 0o600);
tempFiles.push(knownHosts.cleanup);
sshArgs.push("-o", `UserKnownHostsFile=${knownHosts.path}`);
}
} else {
sshArgs.push("-o", "UserKnownHostsFile=/dev/null");
}
if (config.privateKey) {
const privateKey = await withTempFile("paperclip-ssh-key-", config.privateKey, 0o600);
tempFiles.push(privateKey.cleanup);
sshArgs.push("-i", privateKey.path);
}
return {
args: sshArgs,
cleanup: async () => {
await Promise.all(tempFiles.map((cleanup) => cleanup()));
},
};
}
function tarExcludeArgs(exclude: string[] | undefined): string[] {
const combined = ["._*", ...(exclude ?? [])];
return combined.flatMap((entry) => ["--exclude", entry]);
}
function tarSpawnEnv(): NodeJS.ProcessEnv {
return {
...process.env,
// Prevent macOS bsdtar from emitting AppleDouble metadata files like ._README.md.
COPYFILE_DISABLE: "1",
};
}
// Converts a tar `--exclude` pattern into a regexp for the local-size estimate.
// We only need approximate fidelity here (the estimate feeds a clamped percent),
// so we support the literal names and `*`/`?` globs used in practice.
function tarPatternToRegExp(pattern: string): RegExp {
const escaped = pattern
.replace(/[.+^${}()|[\]\\]/g, "\\$&")
.replace(/\*/g, "[^/]*")
.replace(/\?/g, "[^/]");
return new RegExp(`^${escaped}$`);
}
// Walks `localDir` summing regular-file sizes, mirroring tar's `--exclude`
// handling (plus the implicit `._*`) and `followSymlinks` so the to-ssh upload
// can report an estimated total before tar finishes producing the stream.
async function estimateLocalDirSize(input: {
localDir: string;
exclude?: string[];
followSymlinks?: boolean;
}): Promise<number> {
const regexes = ["._*", ...(input.exclude ?? [])].map(tarPatternToRegExp);
const isExcluded = (relPath: string, base: string) =>
regexes.some((regex) => regex.test(relPath) || regex.test(base));
let total = 0;
const walk = async (dir: string, relative: string): Promise<void> => {
const entries = await fs.readdir(dir, { withFileTypes: true }).catch(() => []);
for (const entry of entries) {
const entryRelative = relative ? `${relative}/${entry.name}` : entry.name;
if (isExcluded(entryRelative, entry.name)) continue;
const full = path.join(dir, entry.name);
const stats = await (input.followSymlinks ? fs.stat(full) : fs.lstat(full)).catch(() => null);
if (!stats) continue;
if (stats.isDirectory()) {
await walk(full, entryRelative);
} else if (stats.isFile()) {
total += stats.size;
}
}
};
await walk(input.localDir, "");
return total;
}
// Best-effort remote size probe for the from-ssh restore. `du -sk` is POSIX and
// available on the BSD/Linux remotes we target; it over-counts (block-rounded,
// includes excluded dirs) which keeps the reported percent safely below 100
// until the stream actually closes. Returns null when unavailable so the caller
// falls back to MB-received mode.
async function probeRemoteDirSize(input: {
spec: SshConnectionConfig;
remoteDir: string;
}): Promise<number | null> {
try {
const result = await runSshScript(
input.spec,
`du -sk ${shellQuote(input.remoteDir)} 2>/dev/null | cut -f1`,
{ timeoutMs: 15_000, maxBuffer: 16 * 1024 },
);
const kilobytes = Number.parseInt(result.stdout.trim(), 10);
return Number.isFinite(kilobytes) && kilobytes > 0 ? kilobytes * 1024 : null;
} catch {
return null;
}
}
interface TransferProgress {
// Backpressure-respecting counter to splice into a transport pipe.
counter: Transform;
// Last cumulative byte count observed by the counter.
transferred: () => number;
// Emit the terminal completion line. Idempotent.
finish: () => Promise<void>;
// Emit a terminal failure marker instead of a completion line. Idempotent.
fail: () => Promise<void>;
}
// Wraps a throttled progress reporter behind a counting Transform so transports
// can `source.pipe(progress.counter).pipe(dest)`. When `totalBytes` is a known
// exact size (e.g. a git bundle) the reporter emits an exact percentage. When it
// is an estimate (tar upload / remote probe) we clamp the reported bytes to 99%
// of the estimate so an inaccurate total never shows a premature 100%; `finish`
// then emits the terminal 100% (or, in MB-only mode, the final MB) line.
//
// `totalBytes` may be a promise so an expensive size estimate (a local dir walk
// or a remote `du` probe) runs concurrently with the transfer instead of
// blocking the pipe from opening. Until it resolves the counter reports bytes in
// MB-only mode, then adopts the percentage once the total is known; `finish`
// awaits the estimate so the terminal 100% line is still guaranteed.
function createTransferProgress(input: {
onProgress: RuntimeProgressSink;
phase: RuntimeProgressPhase;
direction: RuntimeProgressDirection;
label?: string;
totalBytes: number | null | Promise<number | null>;
estimated: boolean;
}): TransferProgress {
const reporter = createRuntimeProgressReporter({
sink: input.onProgress,
phase: input.phase,
direction: input.direction,
label: input.label,
target: "ssh",
});
let total: number | null = null;
let cap: number | null = null;
const applyTotal = (value: number | null) => {
total = value != null && value > 0 ? value : null;
cap = total != null && input.estimated ? Math.floor(total * 0.99) : null;
};
const totalReady: Promise<void> =
input.totalBytes != null && typeof (input.totalBytes as Promise<number | null>).then === "function"
? (input.totalBytes as Promise<number | null>).then(applyTotal, () => applyTotal(null))
: (applyTotal(input.totalBytes as number | null), Promise.resolve());
let transferred = 0;
let chain: Promise<void> = Promise.resolve();
const enqueue = (work: () => Promise<void>) => {
chain = chain.then(work).catch(() => undefined);
};
const counter = new Transform({
transform(chunk: Buffer, _encoding, callback) {
transferred += chunk.length;
const reported = cap != null ? Math.min(transferred, cap) : transferred;
const totalSnapshot = total;
enqueue(() => reporter.report(reported, totalSnapshot));
callback(null, chunk);
},
});
return {
counter,
transferred: () => transferred,
finish: async () => {
await chain.catch(() => undefined);
await totalReady.catch(() => undefined);
await reporter.complete(total != null ? total : transferred, total).catch(() => undefined);
},
fail: async () => {
await chain.catch(() => undefined);
await reporter.fail(transferred, total).catch(() => undefined);
},
};
}
async function runSshScript(
config: SshConnectionConfig,
script: string,
options: {
timeoutMs?: number;
maxBuffer?: number;
} = {},
): Promise<SshCommandResult> {
return await runSshCommand(
config,
script,
options,
);
}
async function clearLocalDirectory(
localDir: string,
preserveEntries: string[] = [],
): Promise<void> {
await fs.mkdir(localDir, { recursive: true });
const preserve = new Set(preserveEntries);
const entries = await fs.readdir(localDir);
await Promise.all(
entries
.filter((entry) => !preserve.has(entry))
.map((entry) => fs.rm(path.join(localDir, entry), { recursive: true, force: true })),
);
}
async function copyDirectoryContents(sourceDir: string, targetDir: string): Promise<void> {
await fs.mkdir(targetDir, { recursive: true });
const entries = await fs.readdir(sourceDir);
await Promise.all(entries.map(async (entry) => {
await fs.cp(path.join(sourceDir, entry), path.join(targetDir, entry), {
recursive: true,
force: true,
preserveTimestamps: true,
});
}));
}
async function readLocalGitWorkspaceSnapshot(localDir: string): Promise<LocalGitWorkspaceSnapshot | null> {
try {
const insideWorkTree = await runLocalGit(localDir, ["rev-parse", "--is-inside-work-tree"], {
timeout: 10_000,
maxBuffer: 16 * 1024,
});
if (insideWorkTree.stdout.trim() !== "true") {
return null;
}
const [headCommitResult, branchResult, deletedResult] = await Promise.all([
runLocalGit(localDir, ["rev-parse", "HEAD"], {
timeout: 10_000,
maxBuffer: 16 * 1024,
}),
runLocalGit(localDir, ["rev-parse", "--abbrev-ref", "HEAD"], {
timeout: 10_000,
maxBuffer: 16 * 1024,
}),
runLocalGit(localDir, ["ls-files", "--deleted", "-z"], {
timeout: 10_000,
maxBuffer: 256 * 1024,
}),
]);
const branchName = branchResult.stdout.trim();
return {
headCommit: headCommitResult.stdout.trim(),
branchName: branchName && branchName !== "HEAD" ? branchName : null,
deletedPaths: deletedResult.stdout
.split("\0")
.map((entry) => entry.trim())
.filter(Boolean),
};
} catch {
return null;
}
}
async function streamLocalFileToSsh(input: {
spec: SshConnectionConfig;
localFile: string;
remoteScript: string;
progress?: TransferProgress;
}): Promise<void> {
const auth = await createSshAuthArgs(input.spec);
const sshArgs = [
...auth.args,
"-p",
String(input.spec.port),
`${input.spec.username}@${input.spec.host}`,
`sh -c ${shellQuote(input.remoteScript)}`,
];
await new Promise<void>((resolve, reject) => {
const source = createReadStream(input.localFile);
const ssh = spawn("ssh", sshArgs, {
stdio: ["pipe", "ignore", "pipe"],
});
let sshStderr = "";
let settled = false;
const fail = (error: Error) => {
if (settled) return;
settled = true;
source.destroy();
ssh.kill("SIGTERM");
reject(error);
};
ssh.stderr?.on("data", (chunk) => {
sshStderr += String(chunk);
});
source.on("error", fail);
ssh.on("error", fail);
if (input.progress) {
input.progress.counter.on("error", fail);
source.pipe(input.progress.counter).pipe(ssh.stdin ?? null);
} else {
source.pipe(ssh.stdin ?? null);
}
ssh.on("close", (code) => {
if (settled) return;
settled = true;
if ((code ?? 0) !== 0) {
reject(new Error(sshStderr.trim() || `ssh exited with code ${code ?? -1}`));
return;
}
resolve();
});
}).finally(auth.cleanup);
}
async function streamSshToLocalFile(input: {
spec: SshConnectionConfig;
remoteScript: string;
localFile: string;
progress?: TransferProgress;
}): Promise<void> {
const auth = await createSshAuthArgs(input.spec);
const sshArgs = [
...auth.args,
"-p",
String(input.spec.port),
`${input.spec.username}@${input.spec.host}`,
`sh -c ${shellQuote(input.remoteScript)}`,
];
await new Promise<void>((resolve, reject) => {
const ssh = spawn("ssh", sshArgs, {
stdio: ["ignore", "pipe", "pipe"],
});
const sink = createWriteStream(input.localFile, { mode: 0o600 });
let sshStderr = "";
let settled = false;
let sshClosed = false;
let streamFinished = false;
const settle = () => {
if (settled || !sshClosed || !streamFinished) return;
settled = true;
resolve();
};
const fail = (error: Error) => {
if (settled) return;
settled = true;
ssh.kill("SIGTERM");
ssh.stdout.destroy();
input.progress?.counter.destroy();
sink.destroy();
reject(error);
};
const onStreamFinished = (error: NodeJS.ErrnoException | null) => {
if (error) {
fail(error);
return;
}
streamFinished = true;
settle();
};
if (input.progress) {
pipeline(ssh.stdout, input.progress.counter, sink, onStreamFinished);
} else {
pipeline(ssh.stdout, sink, onStreamFinished);
}
ssh.stderr?.on("data", (chunk) => {
sshStderr += String(chunk);
});
ssh.on("error", fail);
ssh.on("close", (code) => {
if ((code ?? 0) !== 0) {
fail(new Error(sshStderr.trim() || `ssh exited with code ${code ?? -1}`));
return;
}
sshClosed = true;
settle();
});
}).finally(auth.cleanup);
}
async function importGitWorkspaceToSsh(input: {
spec: SshRemoteExecutionSpec;
localDir: string;
remoteDir: string;
snapshot: LocalGitWorkspaceSnapshot;
onProgress?: RuntimeProgressSink;
}): Promise<void> {
const bundleDir = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-ssh-bundle-"));
const bundlePath = path.join(bundleDir, "workspace.bundle");
// Per-import unique ref so concurrent imports against the same local repo
// can't race on `update-ref` between this run's update and bundle create.
const tempRef = `refs/paperclip/ssh-sync/import/${randomUUID()}`;
try {
await runLocalGit(input.localDir, ["update-ref", tempRef, input.snapshot.headCommit], {
timeout: 10_000,
maxBuffer: 16 * 1024,
});
await runLocalGit(input.localDir, ["bundle", "create", bundlePath, tempRef], {
timeout: 60_000,
maxBuffer: 1024 * 1024,
});
const originUrl = await readSanitizedOriginRemoteUrl(input.localDir);
const remoteSetupScript = [
"set -e",
`mkdir -p ${shellQuote(path.posix.join(input.remoteDir, ".paperclip-runtime"))}`,
`tmp_bundle=$(mktemp ${shellQuote(path.posix.join(input.remoteDir, ".paperclip-runtime", "import-XXXXXX.bundle"))})`,
'trap \'rm -f "$tmp_bundle"\' EXIT',
'cat > "$tmp_bundle"',
`if [ ! -d ${shellQuote(path.posix.join(input.remoteDir, ".git"))} ]; then git init ${shellQuote(input.remoteDir)} >/dev/null; fi`,
// Carry the workspace's (credential-scrubbed) origin into the transported
// repo so branches there keep a publishable remote instead of reading as
// remote-less snapshots. set-url covers a reused workspace whose origin
// changed; add covers the fresh-init case. Best-effort under `set -e`.
...(originUrl
? [
`{ git -C ${shellQuote(input.remoteDir)} remote set-url origin ${shellQuote(originUrl)} >/dev/null 2>&1 || git -C ${shellQuote(input.remoteDir)} remote add origin ${shellQuote(originUrl)} >/dev/null 2>&1; } || true`,
]
: []),
`git -C ${shellQuote(input.remoteDir)} fetch --force "$tmp_bundle" '${tempRef}:${tempRef}' >/dev/null`,
input.snapshot.branchName
? `git -C ${shellQuote(input.remoteDir)} checkout --force -B ${shellQuote(input.snapshot.branchName)} ${shellQuote(input.snapshot.headCommit)} >/dev/null`
: `git -C ${shellQuote(input.remoteDir)} -c advice.detachedHead=false checkout --force --detach ${shellQuote(input.snapshot.headCommit)} >/dev/null`,
`git -C ${shellQuote(input.remoteDir)} reset --hard ${shellQuote(input.snapshot.headCommit)} >/dev/null`,
`git -C ${shellQuote(input.remoteDir)} clean -fdx -e .paperclip-runtime >/dev/null`,
// Drop the per-import ref on the remote side too so it can't accumulate.
`git -C ${shellQuote(input.remoteDir)} update-ref -d ${shellQuote(tempRef)} >/dev/null 2>&1 || true`,
].join("\n");
// The git bundle is a real local file of known size, so report an exact
// percentage. No `workspace` label: the "Importing git history" phase is
// already self-describing in the log line.
const progress = input.onProgress
? createTransferProgress({
onProgress: input.onProgress,
phase: "Importing git history",
direction: "to",
totalBytes: (await fs.stat(bundlePath)).size,
estimated: false,
})
: null;
try {
await streamLocalFileToSsh({
spec: input.spec,
localFile: bundlePath,
remoteScript: remoteSetupScript,
progress: progress ?? undefined,
});
await progress?.finish();
} catch (error) {
await progress?.fail();
throw error;
}
} finally {
await runLocalGit(input.localDir, ["update-ref", "-d", tempRef], {
timeout: 10_000,
maxBuffer: 16 * 1024,
}).catch(() => undefined);
await fs.rm(bundleDir, { recursive: true, force: true }).catch(() => undefined);
}
}
async function exportGitWorkspaceFromSsh(input: {
spec: SshRemoteExecutionSpec;
remoteDir: string;
localDir: string;
importedRef?: string;
resetLocalWorkspace?: boolean;
onProgress?: RuntimeProgressSink;
}): Promise<string> {
const bundleDir = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-ssh-bundle-"));
const bundlePath = path.join(bundleDir, "workspace.bundle");
const importedRef = input.importedRef ?? `refs/paperclip/ssh-sync/imported/${randomUUID()}`;
try {
const exportScript = [
"set -e",
`git -C ${shellQuote(input.remoteDir)} update-ref refs/paperclip/ssh-sync/export HEAD`,
`mkdir -p ${shellQuote(path.posix.join(input.remoteDir, ".paperclip-runtime"))}`,
`tmp_bundle=$(mktemp ${shellQuote(path.posix.join(input.remoteDir, ".paperclip-runtime", "export-XXXXXX.bundle"))})`,
'cleanup() { rm -f "$tmp_bundle"; git -C ' + shellQuote(input.remoteDir) + ' update-ref -d refs/paperclip/ssh-sync/export >/dev/null 2>&1 || true; }',
'trap cleanup EXIT',
`git -C ${shellQuote(input.remoteDir)} bundle create "$tmp_bundle" refs/paperclip/ssh-sync/export >/dev/null`,
'cat "$tmp_bundle"',
].join("\n");
// The remote bundle size isn't known before streaming, so report bytes
// received (MB mode) with a terminal completion line.
const progress = input.onProgress
? createTransferProgress({
onProgress: input.onProgress,
phase: "Exporting git history",
direction: "from",
totalBytes: null,
estimated: false,
})
: null;
try {
await streamSshToLocalFile({
spec: input.spec,
remoteScript: exportScript,
localFile: bundlePath,
progress: progress ?? undefined,
});
await progress?.finish();
} catch (error) {
await progress?.fail();
throw error;
}
await runLocalGit(input.localDir, ["fetch", "--force", bundlePath, `refs/paperclip/ssh-sync/export:${importedRef}`], {
timeout: 60_000,
maxBuffer: 1024 * 1024,
});
if (input.resetLocalWorkspace !== false) {
await runLocalGit(input.localDir, ["reset", "--hard", importedRef], {
timeout: 60_000,
maxBuffer: 1024 * 1024,
});
}
const importedHead = await runLocalGit(input.localDir, ["rev-parse", importedRef], {
timeout: 10_000,
maxBuffer: 16 * 1024,
});
return importedHead.stdout.trim();
} finally {
if (input.resetLocalWorkspace !== false) {
await runLocalGit(input.localDir, ["update-ref", "-d", importedRef], {
timeout: 10_000,
maxBuffer: 16 * 1024,
}).catch(() => undefined);
}
await fs.rm(bundleDir, { recursive: true, force: true }).catch(() => undefined);
}
}
async function integrateImportedGitHead(input: {
localDir: string;
importedHead: string;
}): Promise<void> {
const isConcurrentRefUpdateError = (error: unknown) => {
const message = error instanceof Error ? error.message : String(error);
return message.includes("cannot lock ref") && message.includes("expected");
};
for (let attempt = 0; attempt < 5; attempt += 1) {
const snapshot = await readLocalGitWorkspaceSnapshot(input.localDir);
if (!snapshot) return;
const currentHead = snapshot.headCommit;
if (!currentHead || currentHead === input.importedHead) return;
const headRef = snapshot.branchName ? `refs/heads/${snapshot.branchName}` : "HEAD";
// `git merge-base` exits 1 when the commits share no ancestor — the only
// outcome that authorizes the graft fallback below. Every other failure
// (timeout, missing object, repository error) must keep failing the
// integration instead of silently rewriting the tip.
let noCommonAncestor = false;
const mergeBase = await runLocalGit(input.localDir, ["merge-base", currentHead, input.importedHead], {
timeout: 10_000,
maxBuffer: 16 * 1024,
}).catch((error: unknown) => {
noCommonAncestor = (error as { code?: unknown } | null)?.code === 1;
return null;
});
const mergeBaseHead = mergeBase?.stdout.trim() ?? "";
if (mergeBaseHead === input.importedHead) {
return;
}
if (mergeBaseHead === currentHead) {
try {
await runLocalGit(input.localDir, ["update-ref", headRef, input.importedHead, currentHead], {
timeout: 10_000,
maxBuffer: 16 * 1024,
});
return;
} catch (error) {
if (isConcurrentRefUpdateError(error) && attempt < 4) continue;
throw error;
}
}
if (noCommonAncestor) {
// No common ancestor — merging is impossible and failing here would
// discard the imported work. Graft it onto the current head instead;
// see createUnrelatedHistoryGraftCommit.
const graftCommit = await createUnrelatedHistoryGraftCommit({
localDir: input.localDir,
currentHead,
importedHead: input.importedHead,
syncLabel: "Paperclip SSH sync",
});
try {
await runLocalGit(input.localDir, ["update-ref", headRef, graftCommit, currentHead], {
timeout: 10_000,
maxBuffer: 16 * 1024,
});
return;
} catch (error) {
if (isConcurrentRefUpdateError(error) && attempt < 4) continue;
throw error;
}
}
let mergedTree;
try {
mergedTree = await runLocalGit(input.localDir, ["merge-tree", "--write-tree", currentHead, input.importedHead], {
timeout: 60_000,
maxBuffer: 256 * 1024,
});
} catch (error) {
const reason = error instanceof Error ? error.message : String(error);
throw new Error(
`Failed to merge concurrent SSH git histories for ${currentHead.slice(0, 12)} and ${input.importedHead.slice(0, 12)}: ${reason}`,
);
}
const mergedTreeId = mergedTree.stdout.trim().split("\n")[0]?.trim() ?? "";
if (!mergedTreeId) {
throw new Error("Failed to compute a merged git tree for SSH workspace restore.");
}
const mergeCommit = await runLocalGit(
input.localDir,
[
...GIT_SYNC_COMMIT_IDENTITY_ARGS,
"commit-tree",
mergedTreeId,
"-p",
currentHead,
"-p",
input.importedHead,
"-m",
`Paperclip SSH sync merge ${input.importedHead.slice(0, 12)}`,
],
{
timeout: 60_000,
maxBuffer: 64 * 1024,
},
);
try {
await runLocalGit(input.localDir, ["update-ref", headRef, mergeCommit.stdout.trim(), currentHead], {
timeout: 10_000,
maxBuffer: 16 * 1024,
});
return;
} catch (error) {
if (isConcurrentRefUpdateError(error) && attempt < 4) continue;
throw error;
}
}
throw new Error(`Failed to integrate concurrent SSH git history for ${input.importedHead.slice(0, 12)} after multiple retries.`);
}
async function clearRemoteDirectory(input: {
spec: SshConnectionConfig;
remoteDir: string;
preserveEntries?: string[];
}): Promise<void> {
const preservePatterns = (input.preserveEntries ?? [])
.map((entry) => `! -name ${shellQuote(entry)}`)
.join(" ");
const script = [
"set -e",
`mkdir -p ${shellQuote(input.remoteDir)}`,
`find ${shellQuote(input.remoteDir)} -mindepth 1 -maxdepth 1 ${preservePatterns} -exec rm -rf -- {} +`,
].join("\n");
await runSshScript(input.spec, script, {
timeoutMs: 30_000,
maxBuffer: 256 * 1024,
});
}
async function removeDeletedPathsOnSsh(input: {
spec: SshConnectionConfig;
remoteDir: string;
deletedPaths: string[];
}): Promise<void> {
if (input.deletedPaths.length === 0) return;
const quotedPaths = input.deletedPaths.map((entry) => shellQuote(entry)).join(" ");
const script = `cd ${shellQuote(input.remoteDir)} && rm -rf -- ${quotedPaths}`;
await runSshScript(input.spec, script, {
timeoutMs: 30_000,
maxBuffer: 256 * 1024,
});
}
const PROJECT_REPOSITORY_NAME_PATTERN = /^[a-zA-Z0-9_-]+$/;
async function isLocalGitRepositoryRoot(localDir: string): Promise<boolean> {
try {
const toplevel = await runLocalGit(localDir, ["rev-parse", "--show-toplevel"], {
timeout: 10_000,
maxBuffer: 16 * 1024,
});
const [directory, repository] = await Promise.all([
fs.realpath(localDir),
fs.realpath(toplevel.stdout.trim()),
]);
return directory === repository;
} catch {
return false;
}
}
export async function listLocalProjectRepositories(localDir: string): Promise<string[]> {
const root = path.join(localDir, PROJECT_REPOSITORIES_DIR);
const rootStat = await fs.lstat(root).catch((error: NodeJS.ErrnoException) => {
if (error.code === "ENOENT") return null;
throw error;
});
if (!rootStat) return [];
if (!rootStat.isDirectory() || rootStat.isSymbolicLink()) {
throw new Error("Invalid project repositories directory");
}
const repositories: string[] = [];
for (const entry of await fs.readdir(root, { withFileTypes: true })) {
if (entry.isDirectory() && PROJECT_REPOSITORY_STAGING_PATTERN.test(entry.name)) continue;
if (!entry.isDirectory() || !PROJECT_REPOSITORY_NAME_PATTERN.test(entry.name)) {
throw new Error("Invalid project repository directory");
}
const relative = `${PROJECT_REPOSITORIES_DIR}/${entry.name}`;
if (!(await isLocalGitRepositoryRoot(path.join(localDir, relative)))) {
throw new Error(`Project repository is not a Git checkout: ${relative}`);
}
repositories.push(relative);
}
return repositories.sort();
}
async function transportGitWorkspaceToSsh(input: {
spec: SshRemoteExecutionSpec;
localDir: string;
remoteDir: string;
snapshot: LocalGitWorkspaceSnapshot;
exclude?: string[];
onProgress?: RuntimeProgressSink;
progressLabel: string;
}): Promise<void> {
await importGitWorkspaceToSsh({
spec: input.spec,
localDir: input.localDir,
remoteDir: input.remoteDir,
snapshot: input.snapshot,
onProgress: input.onProgress,
});
await syncDirectoryToSsh({
spec: input.spec,
localDir: input.localDir,
remoteDir: input.remoteDir,
exclude: [".git", ".paperclip-runtime", ...(input.exclude ?? [])],
onProgress: input.onProgress,
progressLabel: input.progressLabel,
});
await removeDeletedPathsOnSsh({
spec: input.spec,
remoteDir: input.remoteDir,
deletedPaths: input.snapshot.deletedPaths,
});
}
async function excludeProjectRepositoriesOnSsh(spec: SshConnectionConfig, remoteDir: string): Promise<void> {
const pattern = `/${PROJECT_REPOSITORIES_DIR}/`;
const excludeFile = path.posix.join(remoteDir, ".git", "info", "exclude");
await runSshScript(
spec,
`mkdir -p ${shellQuote(path.posix.dirname(excludeFile))} && ` +
`{ grep -qxF ${shellQuote(pattern)} ${shellQuote(excludeFile)} 2>/dev/null || ` +
`printf '\\n%s\\n' ${shellQuote(pattern)} >> ${shellQuote(excludeFile)}; }`,
{ timeoutMs: 30_000 },
);
}
async function allocateLoopbackPort(host: string): Promise<number> {
return await new Promise<number>((resolve, reject) => {
const server = net.createServer();
server.once("error", reject);
server.listen(0, host, () => {
const address = server.address();
if (!address || typeof address === "string") {
server.close(() => reject(new Error("Failed to allocate a loopback port.")));
return;
}
const { port } = address;
server.close((error) => {
if (error) {
reject(error);
return;
}
resolve(port);
});
});
});
}
async function waitForCondition(
fn: () => Promise<void>,
options: {
timeoutMs?: number;
intervalMs?: number;
} = {},
): Promise<void> {
const timeoutAt = Date.now() + (options.timeoutMs ?? 10_000);
const intervalMs = options.intervalMs ?? 200;
let lastError: unknown = null;
while (Date.now() < timeoutAt) {
try {
await fn();
return;
} catch (error) {
lastError = error;
await new Promise((resolve) => setTimeout(resolve, intervalMs));
}
}
throw lastError instanceof Error
? lastError
: new Error("Timed out waiting for SSH fixture readiness.");
}
async function isPidRunning(pid: number): Promise<boolean> {
try {
process.kill(pid, 0);
return true;
} catch {
return false;
}
}
async function readProcessCommand(pid: number): Promise<string | null> {
for (const format of ["command=", "args="]) {
try {
const result = await execFileText("ps", ["-o", format, "-p", String(pid)], {
timeout: 5_000,
maxBuffer: 16 * 1024,
});
const command = result.stdout.trim();
if (command.length > 0) {
return command;
}
} catch {
continue;
}
}
return null;
}
async function isSshEnvLabFixtureProcess(state: Pick<SshEnvLabFixtureState, "pid" | "sshdConfigPath">): Promise<boolean> {
if (!(await isPidRunning(state.pid))) {
return false;
}
const command = await readProcessCommand(state.pid);
if (!command) {
return false;
}
return command.includes(state.sshdConfigPath);
}
export async function getSshEnvLabSupport(): Promise<SshEnvLabSupport> {
if (process.platform === "darwin" && process.env.PAPERCLIP_ENABLE_DARWIN_SSH_ENV_LAB !== "1") {
return {
supported: false,
reason: "SSH env-lab fixture is disabled on macOS; set PAPERCLIP_ENABLE_DARWIN_SSH_ENV_LAB=1 to opt in.",
};
}
for (const command of ["ssh", "sshd", "ssh-keygen"]) {
if (!(await commandExists(command))) {
return {
supported: false,
reason: `Missing required command: ${command}`,
};
}
}
return {
supported: true,
reason: null,
};
}
export function buildKnownHostsEntry(input: {
host: string;
port: number;
publicKey: string;
}): string {
return `[${input.host}]:${input.port} ${input.publicKey.trim()}`;
}
export async function runSshCommand(
config: SshConnectionConfig,
remoteCommand: string,
options: {
env?: Record<string, string>;
stdin?: string;
timeoutMs?: number;
maxBuffer?: number;
} = {},
): Promise<SshCommandResult> {
let cleanup: () => Promise<void> = () => Promise.resolve();
try {
const auth = await createSshAuthArgs(config);
cleanup = auth.cleanup;
const sshArgs = [...auth.args];
const envEntries = Object.entries(options.env ?? {})
.filter((entry): entry is [string, string] => typeof entry[1] === "string");
for (const [key] of envEntries) {
if (!isValidShellEnvKey(key)) {
throw new Error(`Invalid SSH environment variable key: ${key}`);
}
}
// Mirror buildSshSpawnTarget: source the login profiles first, then run
// `env KEY=VAL cmd` so user-supplied identity overrides win over anything a
// profile re-exports. The SSH target is an operator-configured host, not a
// Paperclip sandbox image, so it can expose `node` or an agent CLI only
// through a login profile; a non-login SSH command would miss that PATH.
// Source `/etc/profile` first so a host that exposes the PATH through
// `/etc/profile.d` scripts still resolves node and the agent CLI.
// The script no longer sources `nvm.sh`; a profile that adds nvm still runs.
// .bash_profile typically sources .bashrc itself; only source .bashrc
// directly when no .bash_profile exists, so a host that adds nvm in
// .bashrc still resolves node without a double-run of the setup.
const envArgs = envEntries.map(([key, value]) => `${key}=${shellQuote(value)}`);
const remoteScript = [
'if [ -f /etc/profile ]; then . /etc/profile >/dev/null 2>&1 || true; fi',
'if [ -f "$HOME/.profile" ]; then . "$HOME/.profile" >/dev/null 2>&1 || true; fi',
'if [ -f "$HOME/.bash_profile" ]; then . "$HOME/.bash_profile" >/dev/null 2>&1 || true; elif [ -f "$HOME/.bashrc" ]; then . "$HOME/.bashrc" >/dev/null 2>&1 || true; fi',
'if [ -f "$HOME/.zprofile" ]; then . "$HOME/.zprofile" >/dev/null 2>&1 || true; fi',
envArgs.length > 0
? `exec env ${envArgs.join(" ")} sh -c ${shellQuote(remoteCommand)}`
: `exec sh -c ${shellQuote(remoteCommand)}`,
].join(" && ");
sshArgs.push(
"-p",
String(config.port),
`${config.username}@${config.host}`,
`sh -c ${shellQuote(SSH_STDIN_COMMAND)}`,
);
return await spawnText("ssh", sshArgs, {
stdin: Buffer.from(remoteScript).toString("base64") + "\n" + (options.stdin ?? ""),
timeout: options.timeoutMs ?? 15_000,
maxBuffer: options.maxBuffer ?? 1024 * 128,
});
} finally {
await cleanup();
}
}
export async function buildSshSpawnTarget(input: {
spec: SshRemoteExecutionSpec;
command: string;
args: string[];
env: Record<string, string>;
}): Promise<{
command: string;
args: string[];
stdinPrefix: string;
cleanup: () => Promise<void>;
}> {
for (const key of Object.keys(input.env)) {
if (!isValidShellEnvKey(key)) {
throw new Error(`Invalid SSH environment variable key: ${key}`);
}
}
const auth = await createSshAuthArgs(input.spec);
const sshArgs = [...auth.args];
const envArgs = Object.entries(input.env)
.filter((entry): entry is [string, string] => typeof entry[1] === "string")
.map(([key, value]) => `${key}=${shellQuote(value)}`);
const remoteCommandParts = [shellQuote(input.command), ...input.args.map((arg) => shellQuote(arg))].join(" ");
// Source the login profiles first, then run `env KEY=VAL cmd` so
// user-supplied identity overrides win over anything a profile re-exports.
// The SSH target is an operator-configured host, not a Paperclip sandbox
// image, so it can expose `node` or an agent CLI only through a login
// profile; a non-login SSH command would miss that PATH. Source
// `/etc/profile` first so a host that exposes the PATH through
// `/etc/profile.d` scripts still resolves node and the agent CLI. The script
// no longer sources `nvm.sh`; a profile that adds nvm still runs.
// .bash_profile typically sources .bashrc itself; only source .bashrc
// directly when no .bash_profile exists, so a host that adds nvm in
// .bashrc still resolves node without a double-run of the setup.
const remoteScript = [
'if [ -f /etc/profile ]; then . /etc/profile >/dev/null 2>&1 || true; fi',
'if [ -f "$HOME/.profile" ]; then . "$HOME/.profile" >/dev/null 2>&1 || true; fi',
'if [ -f "$HOME/.bash_profile" ]; then . "$HOME/.bash_profile" >/dev/null 2>&1 || true; elif [ -f "$HOME/.bashrc" ]; then . "$HOME/.bashrc" >/dev/null 2>&1 || true; fi',
'if [ -f "$HOME/.zprofile" ]; then . "$HOME/.zprofile" >/dev/null 2>&1 || true; fi',
`cd ${shellQuote(input.spec.remoteCwd)}`,
envArgs.length > 0
? `exec env ${envArgs.join(" ")} ${remoteCommandParts}`
: `exec ${remoteCommandParts}`,
].join(" && ");
sshArgs.push(
"-p",
String(input.spec.port),
`${input.spec.username}@${input.spec.host}`,
`sh -c ${shellQuote(SSH_STDIN_COMMAND)}`,
);
return {
command: "ssh",
args: sshArgs,
stdinPrefix: Buffer.from(remoteScript).toString("base64") + "\n",
cleanup: auth.cleanup,
};
}
export async function syncDirectoryToSsh(input: {
spec: SshRemoteExecutionSpec;
localDir: string;
remoteDir: string;
exclude?: string[];
followSymlinks?: boolean;
onProgress?: RuntimeProgressSink;
progressLabel?: string;
}): Promise<void> {
const auth = await createSshAuthArgs(input.spec);
const sshArgs = [
...auth.args,
"-p",
String(input.spec.port),
`${input.spec.username}@${input.spec.host}`,
`sh -c ${shellQuote(`mkdir -p ${shellQuote(input.remoteDir)} && tar -xf - -C ${shellQuote(input.remoteDir)}`)}`,
];
// tar's archive size isn't known until tar finishes, so estimate it from the
// local file sizes and clamp the reported percent to 99% until the pipe closes.
// The estimate walk runs concurrently with the transfer so it never delays the
// pipe from opening on large workspaces.
const progress = input.onProgress
? createTransferProgress({
onProgress: input.onProgress,
phase: "Syncing",
direction: "to",
label: input.progressLabel,
totalBytes: estimateLocalDirSize({
localDir: input.localDir,
exclude: input.exclude,
followSymlinks: input.followSymlinks,
}),
estimated: true,
})
: null;
try {
await new Promise<void>((resolve, reject) => {
const tarArgs = [
...(input.followSymlinks ? ["-h"] : []),
"-C",
input.localDir,
...tarExcludeArgs(input.exclude),
"-cf",
"-",
".",
];
const tar = spawn("tar", tarArgs, {
stdio: ["ignore", "pipe", "pipe"],
env: tarSpawnEnv(),
});
const ssh = spawn("ssh", sshArgs, {
stdio: ["pipe", "ignore", "pipe"],
});
let tarStderr = "";
let sshStderr = "";
let settled = false;
let tarExited = false;
let sshExited = false;
let tarExitCode: number | null = null;
let sshExitCode: number | null = null;
const maybeFinish = () => {
if (settled || !tarExited || !sshExited) {
return;
}
settled = true;
if ((tarExitCode ?? 0) !== 0) {
reject(new Error(tarStderr.trim() || `tar exited with code ${tarExitCode ?? -1}`));
return;
}
if ((sshExitCode ?? 0) !== 0) {
reject(new Error(sshStderr.trim() || `ssh exited with code ${sshExitCode ?? -1}`));
return;
}
resolve();
};
const fail = (error: Error) => {
if (settled) {
return;
}
settled = true;
tar.kill("SIGTERM");
ssh.kill("SIGTERM");
reject(error);
};
if (progress) {
progress.counter.on("error", fail);
tar.stdout?.pipe(progress.counter).pipe(ssh.stdin ?? null);
} else {
tar.stdout?.pipe(ssh.stdin ?? null);
}
tar.stderr?.on("data", (chunk) => {
tarStderr += String(chunk);
});
ssh.stderr?.on("data", (chunk) => {
sshStderr += String(chunk);
});
tar.on("error", fail);
ssh.on("error", fail);
tar.on("close", (code) => {
tarExited = true;
tarExitCode = code;
maybeFinish();
});
ssh.on("close", (code) => {
sshExited = true;
sshExitCode = code;
maybeFinish();
});
}).finally(auth.cleanup);
await progress?.finish();
} catch (error) {
await progress?.fail();
throw error;
}
}
export async function syncDirectoryFromSsh(input: {
spec: SshRemoteExecutionSpec;
remoteDir: string;
localDir: string;
exclude?: string[];
preserveLocalEntries?: string[];
onProgress?: RuntimeProgressSink;
progressLabel?: string;
}): Promise<void> {
const auth = await createSshAuthArgs(input.spec);
const stagingDir = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-ssh-sync-back-"));
const remoteTarScript = [
`cd ${shellQuote(input.remoteDir)}`,
`tar ${[...tarExcludeArgs(input.exclude).map(shellQuote), "-cf", "-", "."].join(" ")}`,
].join(" && ");
const sshArgs = [
...auth.args,
"-p",
String(input.spec.port),
`${input.spec.username}@${input.spec.host}`,
`sh -c ${shellQuote(remoteTarScript)}`,
];
// The remote tar size isn't known locally, so probe the remote directory for
// an estimate (clamped to 99%). The probe runs concurrently with the transfer
// so its round-trip never delays the restore; when it is unavailable we report
// bytes received in MB mode with a terminal completion line.
const progress = input.onProgress
? createTransferProgress({
onProgress: input.onProgress,
phase: "Restoring",
direction: "from",
label: input.progressLabel,
totalBytes: probeRemoteDirSize({ spec: input.spec, remoteDir: input.remoteDir }),
estimated: true,
})
: null;
try {
await new Promise<void>((resolve, reject) => {
const ssh = spawn("ssh", sshArgs, {
stdio: ["ignore", "pipe", "pipe"],
});
const tar = spawn("tar", ["-xf", "-", "-C", stagingDir], {
stdio: ["pipe", "ignore", "pipe"],
env: tarSpawnEnv(),
});
let sshStderr = "";
let tarStderr = "";
let settled = false;
let sshExited = false;
let tarExited = false;
let sshExitCode: number | null = null;
let tarExitCode: number | null = null;
const maybeFinish = () => {
if (settled || !sshExited || !tarExited) return;
settled = true;
if ((sshExitCode ?? 0) !== 0) {
reject(new Error(sshStderr.trim() || `ssh exited with code ${sshExitCode ?? -1}`));
return;
}
if ((tarExitCode ?? 0) !== 0) {
reject(new Error(tarStderr.trim() || `tar exited with code ${tarExitCode ?? -1}`));
return;
}
resolve();
};
const fail = (error: Error) => {
if (settled) return;
settled = true;
ssh.kill("SIGTERM");
tar.kill("SIGTERM");
reject(error);
};
if (progress) {
progress.counter.on("error", fail);
ssh.stdout?.pipe(progress.counter).pipe(tar.stdin ?? null);
} else {
ssh.stdout?.pipe(tar.stdin ?? null);
}
ssh.stderr?.on("data", (chunk) => {
sshStderr += String(chunk);
});
tar.stderr?.on("data", (chunk) => {
tarStderr += String(chunk);
});
ssh.on("error", fail);
tar.on("error", fail);
ssh.on("close", (code) => {
sshExited = true;
sshExitCode = code;
maybeFinish();
});
tar.on("close", (code) => {
tarExited = true;
tarExitCode = code;
maybeFinish();
});
});
await progress?.finish();
await clearLocalDirectory(input.localDir, input.preserveLocalEntries);
await copyDirectoryContents(stagingDir, input.localDir);
} catch (error) {
await progress?.fail();
throw error;
} finally {
await fs.rm(stagingDir, { recursive: true, force: true }).catch(() => undefined);
await auth.cleanup();
}
}
export async function prepareWorkspaceForSshExecution(input: {
spec: SshRemoteExecutionSpec;
localDir: string;
remoteDir?: string;
onProgress?: RuntimeProgressSink;
workspaceFileMode?: "all";
workspaceExclude?: string[];
}): Promise<{ gitBacked: boolean; repositories?: string[]; excludedPaths?: string[] }> {
const remoteDir = input.remoteDir ?? input.spec.remoteCwd;
const gitSnapshot = input.workspaceFileMode === "all" ? null : await readLocalGitWorkspaceSnapshot(input.localDir);
const excludedPaths = await listProjectRepositoryStagingPaths(input.localDir);
const excludedResult = excludedPaths.length > 0 ? { excludedPaths } : {};
if (gitSnapshot) {
const repositories = await listLocalProjectRepositories(input.localDir);
await transportGitWorkspaceToSsh({
spec: input.spec,
localDir: input.localDir,
remoteDir,
snapshot: gitSnapshot,
exclude: repositories.length > 0 ? [PROJECT_REPOSITORIES_DIR] : excludedPaths,
onProgress: input.onProgress,
progressLabel: "workspace",
});
if (repositories.length > 0) {
await excludeProjectRepositoriesOnSsh(input.spec, remoteDir);
}
for (const relative of repositories) {
const localDir = path.join(input.localDir, relative);
const snapshot = await readLocalGitWorkspaceSnapshot(localDir);
if (!snapshot) throw new Error(`Cannot read the Git state of project repository: ${relative}`);
await transportGitWorkspaceToSsh({
spec: input.spec,
localDir,
remoteDir: path.posix.join(remoteDir, relative),
snapshot,
onProgress: input.onProgress,
progressLabel: relative,
});
}
return { gitBacked: true, ...(repositories.length > 0 ? { repositories } : {}), ...excludedResult };
}
await clearRemoteDirectory({
spec: input.spec,
remoteDir,
preserveEntries: [".paperclip-runtime"],
});
await syncDirectoryToSsh({
spec: input.spec,
localDir: input.localDir,
remoteDir,
exclude: [".paperclip-runtime", ...excludedPaths, ...(input.workspaceFileMode === "all" ? input.workspaceExclude ?? [] : [])],
onProgress: input.onProgress,
progressLabel: "workspace",
});
return { gitBacked: false, ...excludedResult };
}
export async function restoreWorkspaceFromSshExecution(input: {
spec: SshRemoteExecutionSpec;
localDir: string;
remoteDir?: string;
baselineSnapshot?: DirectorySnapshot;
restoreGitHistory?: boolean;
onProgress?: RuntimeProgressSink;
repositories?: Array<{ path: string; baselineSnapshot?: DirectorySnapshot }>;
}): Promise<void> {
const remoteDir = input.remoteDir ?? input.spec.remoteCwd;
const repositories = input.repositories ?? [];
for (const repository of repositories) {
if (
path.posix.dirname(repository.path) !== PROJECT_REPOSITORIES_DIR ||
!PROJECT_REPOSITORY_NAME_PATTERN.test(path.posix.basename(repository.path))
) {
throw new Error(`Invalid project repository path: ${repository.path}`);
}
if (input.baselineSnapshot && !repository.baselineSnapshot) {
throw new Error(`Project repository has no workspace baseline: ${repository.path}`);
}
}
if (
input.baselineSnapshot &&
repositories.length > 0 &&
!input.baselineSnapshot.exclude.includes(PROJECT_REPOSITORIES_DIR)
) {
throw new Error(`Workspace baseline must exclude ${PROJECT_REPOSITORIES_DIR} when project repositories are restored separately`);
}
for (const repository of repositories) {
await restoreWorkspaceRootFromSsh({
spec: input.spec,
localDir: path.join(input.localDir, repository.path),
remoteDir: path.posix.join(remoteDir, repository.path),
baselineSnapshot: repository.baselineSnapshot,
restoreGitHistory: input.restoreGitHistory,
onProgress: input.onProgress,
progressLabel: repository.path,
});
}
await restoreWorkspaceRootFromSsh({
spec: input.spec,
localDir: input.localDir,
remoteDir,
baselineSnapshot: input.baselineSnapshot,
restoreGitHistory: input.restoreGitHistory,
onProgress: input.onProgress,
progressLabel: "workspace",
hasProjectRepositories: repositories.length > 0,
});
}
async function restoreWorkspaceRootFromSsh(input: {
spec: SshRemoteExecutionSpec;
localDir: string;
remoteDir: string;
baselineSnapshot?: DirectorySnapshot;
restoreGitHistory?: boolean;
onProgress?: RuntimeProgressSink;
progressLabel: string;
hasProjectRepositories?: boolean;
}): Promise<void> {
const remoteDir = input.remoteDir;
if (input.baselineSnapshot) {
const stagingDir = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-ssh-sync-back-"));
const importedRef = input.restoreGitHistory
? `refs/paperclip/ssh-sync/imported/${randomUUID()}`
: null;
try {
const importedHead = input.restoreGitHistory
? await exportGitWorkspaceFromSsh({
spec: input.spec,
remoteDir,
localDir: input.localDir,
importedRef: importedRef ?? undefined,
resetLocalWorkspace: false,
onProgress: input.onProgress,
})
: null;
await syncDirectoryFromSsh({
spec: input.spec,
remoteDir,
localDir: stagingDir,
exclude: input.baselineSnapshot.exclude,
onProgress: input.onProgress,
progressLabel: input.progressLabel,
});
await mergeDirectoryWithBaseline({
baseline: input.baselineSnapshot,
sourceDir: stagingDir,
targetDir: input.localDir,
// Git history advances via integrateImportedGitHead; the working tree
// still comes from the remote file snapshot so dirty remote edits win.
beforeApply: importedHead
? async () => {
await integrateImportedGitHead({
localDir: input.localDir,
importedHead,
});
}
: undefined,
// integrateImportedGitHead moves the branch with update-ref, which leaves the index
// on the old tip. Reset it to HEAD as the sandbox restore does (#8595).
afterApply: importedHead
? async () => {
await resetLocalGitIndexToHead({ localDir: input.localDir });
}
: undefined,
});
} finally {
if (importedRef) {
await runLocalGit(input.localDir, ["update-ref", "-d", importedRef], {
timeout: 10_000,
maxBuffer: 16 * 1024,
}).catch(() => undefined);
}
await fs.rm(stagingDir, { recursive: true, force: true }).catch(() => undefined);
}
return;
}
const gitSnapshot = await readLocalGitWorkspaceSnapshot(input.localDir);
if (gitSnapshot) {
const projectRepositoryEntries = input.hasProjectRepositories ? [PROJECT_REPOSITORIES_DIR] : [];
await exportGitWorkspaceFromSsh({
spec: input.spec,
remoteDir,
localDir: input.localDir,
onProgress: input.onProgress,
});
await syncDirectoryFromSsh({
spec: input.spec,
remoteDir,
localDir: input.localDir,
exclude: [".git", ".paperclip-runtime", ...projectRepositoryEntries],
preserveLocalEntries: [".git", ...projectRepositoryEntries],
onProgress: input.onProgress,
progressLabel: input.progressLabel,
});
return;
}
await syncDirectoryFromSsh({
spec: input.spec,
remoteDir,
localDir: input.localDir,
exclude: [".paperclip-runtime"],
onProgress: input.onProgress,
progressLabel: input.progressLabel,
});
}
export async function ensureSshWorkspaceReady(
config: SshConnectionConfig,
): Promise<{ remoteCwd: string }> {
const result = await runSshCommand(
config,
`mkdir -p ${shellQuote(config.remoteWorkspacePath)} && cd ${shellQuote(config.remoteWorkspacePath)} && pwd`,
);
return {
remoteCwd: result.stdout.trim(),
};
}
const SSH_ENV_LAB_FIXTURE_PATH_FIELDS = [
"rootDir",
"workspaceDir",
"statePath",
"clientPrivateKeyPath",
"clientPublicKeyPath",
"hostPrivateKeyPath",
"hostPublicKeyPath",
"authorizedKeysPath",
"knownHostsPath",
"sshdConfigPath",
"sshdLogPath",
] as const satisfies readonly (keyof SshEnvLabFixtureState)[];
// True when candidate is an absolute path equal to rootDir or nested under
// it. Used to reject a state file whose paths point outside the fixture
// root it was read from.
function isPathRootedAt(candidate: string, rootDir: string): boolean {
if (!path.isAbsolute(candidate)) return false;
if (candidate === rootDir) return true;
const relative = path.relative(rootDir, candidate);
return (
relative.length > 0 &&
relative !== ".." &&
!relative.startsWith(`..${path.sep}`) &&
!path.isAbsolute(relative)
);
}
// The state file is untrusted input: any local process running as the same
// user can write one. A forged pid or an empty sshdConfigPath would weaken
// isSshEnvLabFixtureProcess's identity check — an empty string is a
// substring of every command line, so it would match any running process.
// Reject a state file that fails this check before any identity check or
// signal runs against it.
function isValidSshEnvLabFixtureState(
raw: SshEnvLabFixtureState,
expectedRootDir: string,
): boolean {
if (!Number.isSafeInteger(raw.pid) || raw.pid <= 0) return false;
if (raw.rootDir !== expectedRootDir) return false;
for (const field of SSH_ENV_LAB_FIXTURE_PATH_FIELDS) {
const value = raw[field];
if (typeof value !== "string" || !isPathRootedAt(value, expectedRootDir)) {
return false;
}
}
const expectedSshdConfigPath = path.join(expectedRootDir, "sshd_config");
if (raw.sshdConfigPath.length === 0 || raw.sshdConfigPath !== expectedSshdConfigPath) {
return false;
}
return true;
}
export async function readSshEnvLabFixtureState(
statePath: string,
): Promise<SshEnvLabFixtureState | null> {
try {
// Resolve a relative statePath against the current working directory
// before use. The state validator below only accepts absolute paths, and
// a relative statePath must resolve to the same absolute directory every
// time a caller reads it, no matter the process working directory.
const resolvedStatePath = path.resolve(statePath);
const raw = JSON.parse(await fs.readFile(resolvedStatePath, "utf8")) as SshEnvLabFixtureState;
if (!raw || raw.kind !== "ssh_openbsd") return null;
if (!isValidSshEnvLabFixtureState(raw, path.dirname(resolvedStatePath))) return null;
return raw;
} catch {
return null;
}
}
async function waitUntilFixtureProcessExits(
state: Pick<SshEnvLabFixtureState, "pid" | "sshdConfigPath">,
timeoutMs: number,
intervalMs = 100,
): Promise<boolean> {
const timeoutAt = Date.now() + timeoutMs;
while (true) {
if (!(await isSshEnvLabFixtureProcess(state))) return true;
if (Date.now() >= timeoutAt) return false;
await new Promise((resolve) => setTimeout(resolve, intervalMs));
}
}
// Sends a signal to a pid and treats an already-dead process as success.
// The identity check that runs before this call is not free: it spawns
// `ps`, which opens a real gap between the check and the signal. If the
// process exits inside that gap, `process.kill` throws ESRCH even though
// the outcome the caller wants (the process is gone) already holds. Any
// other error, such as EPERM for a pid that belongs to another user, must
// still propagate.
function signalFixtureProcess(pid: number, signal: NodeJS.Signals): boolean {
try {
process.kill(pid, signal);
return true;
} catch (error) {
if (error instanceof Error && "code" in error && error.code === "ESRCH") {
return false;
}
throw error;
}
}
// Bounded shutdown escalation shared by every caller that must stop a
// fixture process: send SIGTERM, wait, re-check process identity (the pid
// can be reused in the gap between two signals), then SIGKILL, then wait
// again. Returns true only when the listener is confirmed gone.
async function escalateSshEnvLabFixtureShutdown(
state: Pick<SshEnvLabFixtureState, "pid" | "sshdConfigPath">,
): Promise<boolean> {
if (!(await isSshEnvLabFixtureProcess(state))) return true;
if (!signalFixtureProcess(state.pid, "SIGTERM")) return true;
if (await waitUntilFixtureProcessExits(state, 5_000)) return true;
if (!(await isSshEnvLabFixtureProcess(state))) return true;
if (!signalFixtureProcess(state.pid, "SIGKILL")) return true;
if (await waitUntilFixtureProcessExits(state, 2_000)) return true;
return !(await isSshEnvLabFixtureProcess(state));
}
// Accepts a state path or an already-read state so a caller that already
// holds the fixture state in memory does not have to depend on the state
// file, which a teardown step may have already removed.
export async function stopSshEnvLabFixture(
stateOrPath: string | SshEnvLabFixtureState,
): Promise<boolean> {
const state = typeof stateOrPath === "string"
? await readSshEnvLabFixtureState(stateOrPath)
: stateOrPath;
if (!state) return false;
if (!(await escalateSshEnvLabFixtureShutdown(state))) {
throw new Error(
`SSH env-lab fixture did not stop: pid ${state.pid} on port ${state.port} is still running after SIGKILL.`,
);
}
// Remove the root directory only after the listener process is confirmed
// gone. Removing it earlier would delete the state file the process needs
// for a later stop attempt to find and signal it.
await fs.rm(state.rootDir, { recursive: true, force: true }).catch(() => undefined);
return true;
}
export async function startSshEnvLabFixture(input: {
statePath: string;
bindHost?: string;
host?: string;
// Test-only. Shortens the readiness wait below its 10 second default, so
// a regression test can force the start-failure cleanup path without a
// real 10 second wait.
readinessTimeoutMs?: number;
}): Promise<SshEnvLabFixtureState> {
// Resolve a relative statePath against the current working directory once,
// up front. Every derived path (rootDir and the persisted statePath field)
// must be absolute, so the state validator in readSshEnvLabFixtureState
// accepts the file that this function writes.
const statePath = path.resolve(input.statePath);
const existing = await readSshEnvLabFixtureState(statePath);
if (existing && await isSshEnvLabFixtureProcess(existing)) {
return existing;
}
if (existing) {
await fs.rm(existing.rootDir, { recursive: true, force: true }).catch(() => undefined);
}
const support = await getSshEnvLabSupport();
if (!support.supported) {
throw new Error(`SSH env-lab fixture is unavailable: ${support.reason}`);
}
const sshdPath = await resolveCommandPath("sshd");
if (!sshdPath) {
throw new Error("SSH env-lab fixture is unavailable: missing required command: sshd");
}
const bindHost = input.bindHost ?? "127.0.0.1";
const host = input.host ?? bindHost;
const rootDir = path.dirname(statePath);
await fs.mkdir(rootDir, { recursive: true });
const username = os.userInfo().username;
const port = await allocateLoopbackPort(bindHost);
const workspaceDir = path.join(rootDir, "workspace");
const clientPrivateKeyPath = path.join(rootDir, "client_key");
const clientPublicKeyPath = `${clientPrivateKeyPath}.pub`;
const hostPrivateKeyPath = path.join(rootDir, "host_key");
const hostPublicKeyPath = `${hostPrivateKeyPath}.pub`;
const authorizedKeysPath = path.join(rootDir, "authorized_keys");
const knownHostsPath = path.join(rootDir, "known_hosts");
const sshdConfigPath = path.join(rootDir, "sshd_config");
const sshdLogPath = path.join(rootDir, "sshd.log");
const sshdPidPath = path.join(rootDir, "sshd.pid");
await fs.mkdir(workspaceDir, { recursive: true });
await execFileText("ssh-keygen", ["-q", "-t", "ed25519", "-N", "", "-f", clientPrivateKeyPath], {
timeout: 15_000,
});
await execFileText("ssh-keygen", ["-q", "-t", "ed25519", "-N", "", "-f", hostPrivateKeyPath], {
timeout: 15_000,
});
await fs.copyFile(clientPublicKeyPath, authorizedKeysPath);
const hostPublicKey = (await execFileText("ssh-keygen", ["-y", "-f", hostPrivateKeyPath], {
timeout: 15_000,
})).stdout.trim();
await fs.writeFile(
knownHostsPath,
`${buildKnownHostsEntry({ host, port, publicKey: hostPublicKey })}\n`,
{ mode: 0o600 },
);
await fs.writeFile(
sshdConfigPath,
[
`Port ${port}`,
`ListenAddress ${bindHost}`,
`HostKey ${hostPrivateKeyPath}`,
`PidFile ${sshdPidPath}`,
`AuthorizedKeysFile ${authorizedKeysPath}`,
"PasswordAuthentication no",
"ChallengeResponseAuthentication no",
"KbdInteractiveAuthentication no",
"PubkeyAuthentication yes",
"PermitRootLogin no",
"UsePAM no",
"StrictModes no",
`AllowUsers ${username}`,
"LogLevel VERBOSE",
"PrintMotd no",
"UseDNS no",
"Subsystem sftp internal-sftp",
"",
].join("\n"),
{ mode: 0o600 },
);
const child = spawn(sshdPath, ["-D", "-f", sshdConfigPath, "-E", sshdLogPath], {
detached: true,
stdio: "ignore",
});
child.unref();
const state: SshEnvLabFixtureState = {
kind: "ssh_openbsd",
bindHost,
host,
port,
username,
rootDir,
workspaceDir,
statePath,
pid: child.pid ?? 0,
createdAt: new Date().toISOString(),
clientPrivateKeyPath,
clientPublicKeyPath,
hostPrivateKeyPath,
hostPublicKeyPath,
authorizedKeysPath,
knownHostsPath,
sshdConfigPath,
sshdLogPath,
};
if (!state.pid) {
throw new Error("Failed to start SSH env-lab fixture.");
}
try {
await waitForCondition(async () => {
if (!(await isPidRunning(state.pid))) {
const logOutput = await fs.readFile(sshdLogPath, "utf8").catch(() => "");
throw new Error(logOutput || "SSH env-lab fixture exited before becoming ready.");
}
const config = await buildSshEnvLabFixtureConfig(state);
await ensureSshWorkspaceReady(config);
}, { timeoutMs: input.readinessTimeoutMs ?? 10_000, intervalMs: 250 });
await fs.writeFile(statePath, JSON.stringify(state, null, 2), { mode: 0o600 });
return state;
} catch (error) {
// No state file exists on this path yet, so a later stopSshEnvLabFixture
// call can never find this pid. Escalate and wait for exit here, the
// same way stopSshEnvLabFixture does, before the root directory goes
// away — otherwise a slow-to-exit sshd survives as an orphan with its
// root directory already gone.
const stopped = await escalateSshEnvLabFixtureShutdown(state);
if (stopped) {
await fs.rm(rootDir, { recursive: true, force: true }).catch(() => undefined);
} else {
const survivalNote =
`SSH env-lab fixture pid ${state.pid} on port ${state.port} is still running after SIGKILL. ` +
`Kept ${rootDir} for inspection; no state file exists to target it with a later stop call.`;
if (error instanceof Error) {
error.message = `${error.message}\n${survivalNote}`;
} else {
console.error(survivalNote);
}
}
throw error;
}
}
export async function buildSshEnvLabFixtureConfig(
state: SshEnvLabFixtureState,
): Promise<SshConnectionConfig> {
const [privateKey, knownHosts] = await Promise.all([
fs.readFile(state.clientPrivateKeyPath, "utf8"),
fs.readFile(state.knownHostsPath, "utf8"),
]);
return {
host: state.host,
port: state.port,
username: state.username,
remoteWorkspacePath: state.workspaceDir,
privateKey,
knownHosts,
strictHostKeyChecking: true,
};
}
export async function readSshEnvLabFixtureStatus(statePath: string): Promise<{
running: boolean;
state: SshEnvLabFixtureState | null;
}> {
const state = await readSshEnvLabFixtureState(statePath);
if (!state) {
return { running: false, state: null };
}
return {
running: await isSshEnvLabFixtureProcess(state),
state,
};
}
export async function fileExists(filePath: string): Promise<boolean> {
try {
await fs.access(filePath, fsConstants.F_OK);
return true;
} catch {
return false;
}
}