Files
PaperClipAI/doc/tailscale-environments.md
T
Devin FoleyandPaperclip 1d2b669e7a fix(ssh): allow CLI agents on volume-free servers (#21957)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - An SSH environment runs an installed harness on a computer the
operator controls.
> - Volume-free Cloud servers require durable task files outside the
controller.
> - The current admission path accepts native sandbox workspaces and
rejects existing SSH CLI agents before execution.
> - The remote workspace owner already provides saved versions and a
single-writer fence.
> - This pull request gives that owner an SSH archive transport and
admits the existing Claude and Codex CLI adapters under its authority.
> - The benefit is SSH task execution without a controller volume or a
new environment type.

## Linked Issues or Issue Description

**What happened?**

A task assigned to an existing SSH Claude Code agent fails on a
volume-free Cloud server. It reports that execution requires native
Paperclip Runner and remote workspace storage. The failure occurs before
provider work starts, even when SSH and the configured AI account are
valid.

**Expected behavior**

An SSH CLI agent can use the configured Paperclip AI account and an
installed harness. Task files remain durable when the Cloud controller
is replaced. The same SSH behavior applies to direct and Tailscale
connections.

**Steps to reproduce**

1. Configure an SSH environment on a volume-free server with external
Postgres, stable encryption, and object storage.
2. Assign a Claude Code or Codex CLI agent to that environment.
3. Give it a task without a project, such as creating `hello_world.txt`.
4. Observe the workspace admission failure before execution.

Related work: #21248 adds agent-selected SSH authentication. #20617 adds
Tailscale network selection. #15120 and #17291 cover SSH process and
lease cleanup. A search of open and closed PRs found no duplicate of
this workspace admission fix.

## What Changed

- Reuse remote workspace preparation, archives, saved versions, and
single-writer admission for SSH targets.
- Add an explicit file-sync capability to the existing SSH command
runner. It streams archive files with bounded deadlines and private
temporary files. Ordinary SSH runners keep their previous capabilities.
- Run the existing CLI adapter in the remote workspace. Bind both the
target and its SSH command specification to that directory.
- Admit a direct SSH adapter on a volume-free server only when its
durable workspace, environment, lease, and path agree. Local filesystem
authority remains rejected.
- Preserve the server-owned remote workspace receipt when selecting the
legacy runtime.
- Reuse the workspace finalization lock for a legacy run only when its
company, workspace, writer, epoch, and lease receipt match. A claimed
token can finish and clear itself after another task reserves a
successfully saved shared workspace. Storage writes still require the
current workspace authority.
- Transfer recovered copies to SSH through the verified workspace
archive format. Sandbox recovery retains its existing directory
transfer.
- Document Node.js, Git, explicit `engine=cli`, and uncertain-process
limitations. No UI, Cloud control-plane configuration, schema, or
deployment change is included.

The remote workspace service owns this change. The shared SSH utility
adds file transport because its existing authentication and stream
cleanup must remain the only owners of SSH credentials. The heartbeat
changes pass the storage authority through existing preparation,
selection, and finalization boundaries. The finalization owner extension
avoids a separate copy-back lock for SSH. Existing native callers keep
their prior admission contract.

The existing SSH utility and heartbeat test files are already large. The
added code stays beside their existing stream lifecycle and shared
fixtures. A larger extraction would widen this compatibility fix.

## Verification

- Red-green tests reproduce SSH rejection and verify admission only with
remote storage authority.
- A real isolated OpenSSH fixture exercises archive capture, a
replacement database client, a saved-workspace restore to another
allocation, and a failed SSH transfer. Failed saves retain the files and
block a competing writer.
- A whole-heartbeat test uses the real SSH transport and Claude CLI
adapter, a synthetic CLI, a selected Paperclip AI account, and fixture
object storage. A follow-up resumes the same provider session and its
retained history. It makes no provider API calls. The synthetic provider
also supplies its task disposition update.
- Existing Claude and Codex SSH adapter tests, workspace preparation,
and native workspace owner recovery are included in focused checks.
- Before the review fixes, clean-checkout verification passed 610 tests
in 11 files. This includes the full workspace preparation, runtime
selection, runtime execution, environment orchestrator,
ephemeral-server, workspace transport, native owner recovery, SSH
integration, SSH command, and Claude/Codex remote execution suites. The
macOS SSH fixture was enabled with
`PAPERCLIP_ENABLE_DARWIN_SSH_ENV_LAB=1`; Vitest used `--maxWorkers=2`.
- An initial run in the main worktree hit a skill-root lookup outside
that checkout. The full selected set passes in the clean checkout. The
final company-binding type correction changes no executable behavior;
all 8 focused selection regressions pass after that correction.
- At the initial head `dd77a4d3cf0`, server and adapter-utils typechecks
passed, and the clean verification checkout matched its tree
`3f90b0681b1004556d1a720ed45fc8d439b04e96`.
- Review regressions first reproduced both a lost ownership token after
shared-workspace handoff and a rejected recovered-directory transfer.
With the fixes, 223 tests in 5 affected suites pass, including real SSH
recovery and native owner-restart, save-concurrency, archive transport,
and recovery-receipt checks. The handoff test also confirms that a stale
owner cannot save again or claim ownership after a new writer takes
over. Server typecheck passes after these fixes. The tested review-fix
head is `c5182f5d9c56e738e683d9f26b226e9a820bba83`.
- [The initial CI server
shard](https://github.com/paperclipai/paperclip/actions/runs/44009944657/job/133794523181)
failed on an assertion still expecting the old native-sandbox-only
error. Its unsupported sandbox adapter remains rejected; the assertion
now matches the supported-remote-environment error. After updating that
assertion, all 115 tests in the workspace lifecycle suite pass. Together
with the 5 affected suites, this gives 338 passing tests for the review
fixes.
- `pnpm verify` returns 2 and requests broad CI coverage for the
shared/runtime change. The full local suite was not substituted for CI.
Final-head
[CI](https://github.com/paperclipai/paperclip/actions/runs/44012991355)
is green on `c5182f5d9c56e738e683d9f26b226e9a820bba83`: 58 checks pass
and 3 are skipped, with no failures or pending checks. Typecheck, build,
end-to-end, and the aggregate verification gate all pass. Greptile is
5/5 on this head, security scan passes, and there are no unresolved
review threads or merge conflicts.

Review-fix checks:

```sh
PAPERCLIP_ENABLE_DARWIN_SSH_ENV_LAB=1 pnpm exec vitest run --maxWorkers=2 \
  server/src/services/ssh-remote-workspace.integration.test.ts \
  server/src/services/native-runtime/native-workspace-owner-restart-recovery.test.ts \
  server/src/services/remote-workspace-transport.test.ts \
  server/src/services/remote-workspace-recovery-receipt-fallback.test.ts \
  server/src/services/remote-workspace-save-concurrency.test.ts
pnpm exec vitest run --maxWorkers=2 server/src/services/remote-workspaces.integration.test.ts
pnpm --filter @paperclipai/server typecheck
```

## Risks

This extends the remote workspace owner to another transport and allows
qualified SSH CLI execution on volume-free servers. It changes remote
file placement to a task directory under `.paperclip-workspaces`. It
does not import an existing controller-local workspace.

The installed harness uses its existing adapter. Set `engine=cli`
explicitly; ACP archive staging does not support these in-place
workspaces. This does not qualify native macOS provider packs or change
an agent's engine or permission settings automatically.

An unknown remote process after SSH loss remains held until it can be
reconciled. This change does not add automatic reconciliation of that
process. SSH computers are never stopped or deleted. Real provider
acceptance and a hosted staging journey have not been run for this
revision.

## Model Used

OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) through the Codex harness, with xhigh
reasoning effort and code tools.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-17 09:33:11 -08:00

15 KiB
Raw Blame History

Use Tailscale machines in SSH environments

The Tailscale connector gives Paperclip access to a tailnet. Paperclip joins a separate device to that network, then uses the existing SSH environment driver to run tasks on selected computers. The connector is managed from Connectors and can serve multiple SSH environments in the same company. The computer is not replaced or provisioned by Paperclip.

In Connectors, find Tailscale, open it and click Connect Tailscale Network. After approving consent, click Create SSH environment, or create an SSH environment and select Tailscale under Network. Choose the connection and machine, then configure the SSH user, port, absolute workspace path, private key, and trusted host keys. The Tailscale network option appears once a tailnet is connected. On a Mac with the normal Tailscale application, enable macOS Remote Login. Tailscale SSH server support on macOS requires the open-source tailscaled variant; ordinary OpenSSH over Tailscale works with the standard Mac application.

Device sign-in opens Tailscale in a separate tab and keeps the connector page open. A normal sign-in link remains available if the browser blocks the tab. Approve the Paperclip device in your own tailnet; no OAuth app, API key, Cloud broker credential, or per-instance pilot admission is needed. If your tailnet requires device approval, the connector shows that separate pending state. Sign-in expires after ten minutes and can be cancelled from the connector.

SSH setup hides the Network selector when Direct connection is the only available option. A sole connected tailnet is selected automatically and its connection selector is hidden; multiple connected tailnets remain selectable. Switching an existing environment to Direct connection clears its saved Tailscale machine and credential references and saves the direct host you enter.

After selecting a machine, Prepare machine pairing creates an SSH key in Paperclip’s vault and displays a command containing only the public key. Each preparation generates a fresh key pair. The command uses the public key produced during that request, never mutable metadata from a saved secret. Run it as the target SSH user. The command preserves existing authorized keys, including files without a final newline. Before installing the key, it checks for readable Ed25519, ECDSA, or RSA host public keys in /etc/ssh and stops without changing authorized_keys if none are available. It prints that user's name, home workspace path, and known-hosts entries for every available host key for the form. You can also select an existing SSH private key secret.

Pairing output is tied to the selected company, tailnet, machine, and SSH port. After editing the port, prepare pairing again for matching known-hosts output. A late response for an earlier port cannot replace the selected private key. Switching to Direct connection or closing the form also prevents pending pairing results from changing the selected key.

Save the environment and test it. Select it as an agent’s environment and choose any harness supported by SSH environments. Select the agent's AI account in Paperclip. Runs use that account's subscription or API key, with private credential directories on the target computer, just as sandbox runs do. The computer supplies the installed harness and workspace; its SSH user's provider login is not used for an agent with a configured AI account. Test the chosen harness separately: SSH access does not prove provider authentication.

SSH's Advanced section includes Use authentication configured on the remote machine. New environments leave it off, requiring the agent's Paperclip AI account. Turn it on to use the remote user's provider login instead. Existing SSH environments display it as on; opening or upgrading an installation does not rewrite their saved configuration. The same option applies to direct SSH and Tailscale connections.

SSH on a volume-free Cloud stack

Cloud SSH execution uses the same durable task workspace storage as Cloud sandboxes. Claude Code and Codex use their existing SSH adapters with engine=cli. ACP archive staging cannot use an in-place Cloud workspace. An explicit engine choice preserves the agent's execution and permission settings. The selected SSH account supplies the installed harness; the agent's Paperclip AI account supplies authentication unless the remote authentication option is enabled.

Each task receives a private directory below the configured SSH workspace path, inside .paperclip-workspaces. The task runs in that remote directory. Core keeps the workspace binding and saved file versions in its database and object storage. A Cloud restart reconnects to the same remote files. Restoring a saved workspace to another allocation uses the existing snapshot verification and single-writer rules. It does not restore private provider credentials from a task snapshot.

The SSH machine needs Node.js and Git on its login PATH in addition to its agent harness. The workspace owner uses Node.js for preparation and verified file archives. SSH access alone does not qualify these dependencies or the selected AI account.

An existing server-local workspace is not automatically imported. Cloud still rejects local filesystem authority and unsupported adapters. An uncertain SSH exit or failed save retains the remote files and the writer hold; it does not authorize another run to overwrite them. Paperclip does not stop or delete the SSH computer. This change does not add automatic reconciliation of an unknown remote provider process after an SSH connection is lost.

The environment keeps the ssh driver. Machine selection stores the connector ID and Tailscale’s stable device ID, and runtime resolution looks up its current address through that connection’s private socket. Offline, removed, disconnected, and wrong-company machines fail closed. The host’s existing Tailscale connection is never a fallback.

Device sign-in and credential custody

It is a network connector on the Connectors surface; it does not publish agent tools or enter MCP discovery. Management requires an instance administrator with access to the company. Only the person who started a pending sign-in can retrieve its sign-in link. Existing SSH environment access governs which agents execute there. Cloud and self-hosted instances use the same Tailscale device sign-in. For Cloud owners, the board access summary reports the effective instance-admin rights computed by Cloud authentication while enableOwnerInstanceAdmin is enabled. Disabling that setting removes those rights on the next request.

The instance starts a private userspace Tailscale device and returns its generated https://login.tailscale.com/a/... sign-in address. Pending device state is stored in the configured company vault before the address is returned. The active tailscale up command watches Tailscale state until consent and any required device approval complete, or the ten-minute deadline expires. It saves the registered device state before marking the connection connected. Pending sign-ins can resume from the saved state after a server restart without extending their deadline. There is no background database scan: browser progress requests exist only during the bounded sign-in, and the CLI waits for Tailscale state events.

The previous OAuth callback remains available for already-started pilot flows; existing enrolled devices continue to use their saved state. New connections use device sign-in and do not call the Cloud OAuth broker.

The connection records the exact trusted vault version and a SHA-256 digest of the server-written value. Runtime reads that version and verifies its digest. A generic company secret rotation cannot change the authorized network. Trusted node-state saves advance both fields together, with a compare-and-set guard so an older save cannot replace a newer state. Reads keep using the previous trusted version until the new version is committed. Private node state, digests and enrollment credentials never appear in connection or environment responses.

Enrollment tokens and node state are server-owned secrets with a durable system_purpose classification. Generic company catalogs omit them. Generic bindings, environment injection, and agent value reads reject them, including bindings saved before the restriction. Renaming or rotating a secret cannot remove this classification. Disconnecting or replacing a connection leaves its old credential protected. Only the connector's server consumer can resolve its trusted value; an SSH environment binds its private key, never the node state. Unclassified legacy node-state values are also rejected by generic reads, including state left behind by an earlier connection replacement. Migration 0482_aberrant_rumiko_fujikawa classifies existing connector-referenced secrets. Apply it before running the updated server. If node state was copied before this restriction, revoke that enrolled device in Tailscale; a local disconnect cannot stop a copy outside Paperclip.

Connections created before migration 0481_round_pet_avengers have no trusted version and digest. They fail closed and require a fresh Connect Tailscale Network consent. The migration leaves these fields empty; it does not trust or backfill a generic secret's latest value.

Each connection has a separate userspace daemon and private socket. A stale credential read reloads the trusted value before replacing a ready daemon and rechecks which daemon is active after the reload. It preserves a healthy newer socket and cannot cross a manager shutdown. Failed connection requests leave cleanup to the network manager's exact client; they cannot disconnect a newer retry using the same connector identity. The daemon accepts no subnet routes or DNS changes and blocks incoming connections. Normal shutdown saves node state and removes temporary files. A restart restores the enrolled device; it does not reuse the OAuth token or require consent for every task. Disconnect invalidates pending consent and stops the connection’s local daemon. A callback already exchanging or saving credentials cannot reconnect after disconnect or replace a newer authorization. Completion retains the consumed state's digest as a private attempt marker, with no expiry; disconnect replaces that marker.

Database writes and credential reloads verify ownership without holding a database lock across broker or daemon startup. Cancelled attempts close their daemon and remove their newly created secret without touching a newer connection's state. Disconnect does not remove the target computer from Tailscale.

Company deletion removes connector records before vault secrets in the company transaction. After commit, it stops every daemon owned by those records. A rolled-back deletion leaves the connector and its daemon intact. Deleting a connector's vault secret invalidates pending consent, disconnects its network, and clears the trusted credential reference before removing the secret. If shutdown fails, the disconnected reference remains available for a deletion retry. Late daemon state writes cannot restore a retired connector.

Task runtime resolution obtains its socket from this same guarded connection service after loading the SSH key. It requires the connection to remain connected with the environment's saved credential identity during credential reload, daemon startup, and device discovery. A disconnect or replacement rejects the run before returning a socket; runtime resolution cannot restart the connection from a stale saved secret. Saved network credentials without a connector and machine selection are rejected; all Tailscale environments use the connector's lifecycle.

This implementation currently supports one serving process per instance. Concurrent replicas need a durable connection ownership lease before rollout.

Tailscale identity belongs to the persisted machine key in the saved node state, not the container ID or hostname. A redeploy must retain both the database's encrypted node state and the same vault master key. Cloud already materializes the stable per-stack PAPERCLIP_SECRETS_MASTER_KEY across release rolls. Do not make the connector ephemeral or replace its node state at each start. Losing that state, device removal, or a tailnet reauthentication policy can require fresh consent. The app-owned device has the consenting user's tailnet rights.

Local development

Run the Paperclip instance at http://localhost:3100. No local Cloud broker or Tailscale OAuth app is required. The instance needs tailscale and tailscaled. For a separate local userspace client, install the pinned programs outside the checkout:

GOBIN="$HOME/.local/share/paperclip-tailscale/bin" go install tailscale.com/cmd/tailscale@v1.104.5
GOBIN="$HOME/.local/share/paperclip-tailscale/bin" go install tailscale.com/cmd/tailscaled@v1.104.5

Set PAPERCLIP_TAILSCALE_CLI and PAPERCLIP_TAILSCALED to their absolute paths. These settings can be set in the supported .env files. Each new client reads them at startup, after environment loading.

Verification and rollout gates

Offline tests cover device sign-in without an auth key, saved state and restart, separate company identities, initiator binding, cancellation and expiry, stale completion, trusted credential versions, device approval, popup fallback, and route authorization/CSRF. Existing tests retain legacy broker callback, SSH, pairing, and network lifecycle coverage. These do not prove live consent or remote execution.

Before rollout, qualify two unrelated customer tailnets through ordinary device sign-in, machine discovery, and an agent task using its selected AI account that writes a file on the selected computer, result streaming, cancellation, server restart, machine sleep/reboot, and revocation. A local instance whose public URL is localhost also needs an API path reachable by the remote agent for tasks that call back to Paperclip unless the adapter uses the existing remote callback bridge. Production Cloud uses its public HTTPS stack URL.

The connector's bundled artwork comes from the vendor's https://tailscale.com/favicon.png, copied without modification (SHA-256 680570c9090b604450792f68ceed10beeb33a69c9bccc2a92e398564c7075a5c).

The old managed OAuth profile remains disabled unless explicitly configured for its existing pilot. Its same-tailnet registration restriction does not apply to the new device sign-in path. Do not configure a customer-specific OAuth app or admit a pilot instance to make device sign-in work.

Provider references: