## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Hosted `claude_local` runs execute in a sandbox. The sandbox has a fixed memory limit (4 GiB in the observed sandboxes). > - In that sandbox, every process has the same `oom_score_adj`. When a Bash tool command uses most of the memory, the kernel kills the largest process. That can be the Claude Code process. > - When Claude Code is killed, `claude-agent-acp` reports "The connection to Claude was lost." and the whole run fails. Server logs show `Claude Agent process died: Claude Code process terminated by signal SIGKILL` for each failure. > - A failed tool command is recoverable. A dead agent is not. > - This pull request makes Claude's tool commands the first out-of-memory victims in managed sandboxes. > - The benefit is that a memory-heavy command fails with exit 137, and the agent continues the run. ## Linked Issues or Issue Description Refs #16366 (diagnostic PR for the same failure). Related: #14509 protects the host control plane from local agent processes. That PR does not cover a sandboxed agent and its own tool commands. **What happened?** Hosted `claude_local` runs fail with `acpx_turn_failed` and "The connection to Claude was lost." while one `execute` (Bash) tool call is pending. The pending commands are memory-heavy, for example `tsc --noEmit` with a 3.3 GB heap, a monorepo `pnpm install`, or a type check plus `vitest`. At the same second, the server log shows `Claude Agent process died: Claude Code process terminated by signal SIGKILL`. Over 10 days, all 12 such lines on one staging stack were `SIGKILL`, and they match the failed runs. The sandbox memory cgroup has `memory.max` 4 GiB and `memory.oom.group` 0, so the kernel kills one process. Every process has `oom_score_adj` 0, so the kernel picks the largest one. **Expected behavior** Under memory pressure, the kernel kills the tool command. Claude gets a failed command (exit 137) and continues the run. **Steps to reproduce** 1. Start a `claude_local` run on a sandbox target with a 4 GiB memory limit. 2. Ask the agent to run a memory-heavy command in a large repository, for example a full `tsc --noEmit` and a test run together. 3. The kernel kills the Claude Code process, and the run fails with "The connection to Claude was lost." **Agent adapter(s) involved** Claude Code (`claude_local`, ACP lane and CLI lane, managed sandbox targets). ## What Changed - `claude-config.ts`: add `CLAUDE_TOOL_SHELL_PREFIX_SCRIPT`. Claude Code calls a `CLAUDE_CODE_SHELL_PREFIX` with one argument, the full command string. The script raises its own `oom_score_adj` to 1000 (no privilege is necessary to raise it). Then it runs the command string in the shell that Claude Code selects: `CLAUDE_CODE_SHELL`, then `SHELL`, when either is an executable bash or zsh, else zsh or bash from `PATH` (bash first when `SHELL` names bash). - `buildRemoteClaudeConfigMaterializationCommand` can install the script (umask 077, write to `.partial`, `chmod 700`, `mv`). The install runs last in the existing materialization command, so it adds no sandbox round trip. It prints a marker only when the file is in place. - `materializeRemoteClaudeConfig` returns `toolShellPrefixInstalled`. It is true only when the command exits 0 and prints the marker. - ACP lane (`acp.ts`) and CLI lane (`execute.ts`): set `CLAUDE_CODE_SHELL_PREFIX` to the installed script only when the install is confirmed. A prefix path that does not exist would fail every Bash tool call, so a failed install keeps the previous behavior. An operator `CLAUDE_CODE_SHELL_PREFIX` in the adapter env is kept, and no script is installed. - `docs/adapters/claude-local.md`: add "Sandbox memory limits". ## Verification - `npx vitest run src/server/claude-config.test.ts` in `packages/adapters/claude-local`: 10 pass, 6 new. The new tests install the script with the real materialization command through `/bin/sh` and check: - the file content, mode 0700, the marker, and that no `.partial` file remains; - the command runs with `oom_score_adj` 1000 while the caller keeps its own score (Linux only); - shell selection: a valid bash `SHELL`, an invalid `CLAUDE_CODE_SHELL` that falls back to `SHELL`, and a `SHELL` of `/bin/sh` that falls back to bash from `PATH`; - quoting and the exit status of the command are kept; - an operator prefix wins, and no marker means no install. - `acp.test.ts`: the managed-config test now checks that `CLAUDE_CODE_SHELL_PREFIX` names an executable 0700 file next to the config dir. A new test checks that an operator prefix is kept and no script is installed. Both use the local sandbox runner, which runs the setup commands for real. - `tsc --noEmit -p packages/adapters/claude-local` is clean. - Two tests fail in my sandbox both with and without this change: `uses the same default model in ACP startup and session identity` and `reselects the full assignment and bootstrap guidance after a failed resume`. They also fail on clean `master` in the same environment, so I think they depend on that environment. CI covers them. - Real Claude Code check (Claude Code 2.1.286): with `CLAUDE_CODE_SHELL_PREFIX` set to the script, a Bash tool call printed `oom_score_adj` 1000 and ran in zsh (the selected shell), and its parent was the `claude` process at score 0. Without a prefix file that exists, every Bash tool call fails with exit 127. This is why the env is set only after a confirmed install. - Real OOM check in a 4 GiB Daytona sandbox (`memory.oom.group` 0). A stand-in "agent" process held 1.2 GB at score 0, and a tool command started 14 processes of 250 MB each: - without the prefix, the kernel killed the stand-in agent (SIGKILL, 137; `oom_kill` +1); - through the installed prefix, the stand-in agent stayed alive, and the kernel killed 5 tool processes (`oom_kill` +5). - Assumption that could invalidate the design: Claude Code passes one argument (the command string) to the prefix and honors the variable through `claude-agent-acp`. I checked the argument shape with a logging prefix, and `claude-agent-acp` 0.73.0 spawns the CLI with `{ ...process.env }`. ## Risks - Every Bash tool command and hook in a managed sandbox now runs through one more `sh` process and an `exec`. The cost is a few milliseconds for each command. - If a future Claude Code release changes how it calls `CLAUDE_CODE_SHELL_PREFIX`, Bash tool calls can fail. The tests above and the documented variable limit this risk. Removing the variable restores the old behavior. - The inner shell is not a login shell. Claude Code sources its own shell snapshot inside the command string, so the environment stays the same. If Claude Code runs without a snapshot, it uses a login shell. Only exported variables then pass through to the inner shell. - If the selected shell differs from the one that Claude Code chose (for example, a custom `CLAUDE_CODE_SHELL` that is not bash or zsh), the script uses the same fallback order as Claude Code. - This does not help when the Claude Code process itself uses most of the memory. The local (non-sandbox) lanes do not change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with tool use and code execution. It read the repository and the Claude Code CLI behavior, ran the OOM experiment, edited the code, and ran the tests. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass (two tests fail on clean `master` in my sandbox too; see Verification) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Paperclip <noreply@paperclip.ing>
9.7 KiB
title, summary
| title | summary |
|---|---|
| Claude Code | Claude Code local adapter setup and configuration |
The claude_local adapter runs Anthropic's Claude Code CLI locally. It supports session persistence, skills injection, and structured output parsing.
Quota waits
Claude ACP runs that end with a typed provider-quota error retain the quota
classification and any parsed reset time. Recovery waits until that time, or
uses its existing one-hour quota backoff when no reset time is available.
This includes the Claude bridge's typed “The Claude account has no available
quota.” fallback, which carries no reset timestamp.
The adapter inspects the terminal provider message in memory; the run result
and run log retain only the generic failure message, recovery labels, and reset
timestamp. Context, turn, rate, and configured budget limits are not treated as
subscription quota exhaustion merely because ACP labels them limit.
Prerequisites
- Claude Code CLI installed (
claudecommand available) - Either
ANTHROPIC_API_KEYorCLAUDE_CODE_OAUTH_TOKENin adapter or environment env (or host env), or a Claude Code subscription login available to the execution target
Configuration Fields
| Field | Type | Required | Description |
|---|---|---|---|
cwd |
string | Yes | Working directory for the agent process (absolute path; created automatically if missing when permissions allow) |
model |
string | No | Claude model to use (default: claude-opus-5) |
promptTemplate |
string | No | Prompt used for all runs |
env |
object | No | Environment variables (supports secret refs) |
timeoutSec |
number | No | Process timeout (0 = no timeout) |
graceSec |
number | No | Grace period before force-kill |
maxTurnsPerRun |
number | No | Max agentic turns per heartbeat (defaults to 300) |
dangerouslySkipPermissions |
boolean | No | Skip permission prompts (default: true); required for headless runs where interactive approval is impossible |
On the CLI engine, version and --effort capability probes have a bounded
20-second budget when timeoutSec is zero or unset. A positive run timeout below
20 seconds also limits these probes, with a minimum probe budget of one second.
Default model
An omitted, empty, or whitespace-only model uses Claude Opus 5
(claude-opus-5) on both the CLI and ACP engines. This also applies to existing
agents with an unset model, including agents created through the API and agents
running in sandboxes. No database migration is needed. The editor shows the
Paperclip default and leaves the setting unset until you select a model.
An explicit model takes precedence over ANTHROPIC_MODEL. When only
ANTHROPIC_MODEL is configured, the adapter keeps that override. Bedrock and
Vertex configurations without an explicit model keep their provider-specific
default because those providers use different model IDs. Host environment
settings apply only to local targets when resolving the model.
The default does not change explicitly configured agent models or the separate Paperclip Runner's qualified provider profiles.
Prompt Templates
Templates support {{variable}} substitution:
| Variable | Value |
|---|---|
{{agentId}} |
Agent's ID |
{{companyId}} |
Company ID |
{{runId}} |
Current run ID |
{{agent.name}} |
Agent's name |
{{company.name}} |
Company name |
Session Persistence
The adapter persists Claude Code session IDs between heartbeats. On the next wake, it resumes the existing conversation so the agent retains full context.
Session resume is cwd-aware: if the agent's working directory changed since the last run, a fresh session starts instead.
If resume fails with an unknown session error, the adapter automatically retries with a fresh session.
Poisoned previous_message_id (recovery)
Symptom in logs / issue thread:
API Error: 400 diagnostics.previous_message_id: must be the `id` from a prior /v1/messages response (starts with `msg_`)
What it means: the on-disk Claude Code transcript JSONL for that session contains a malformed (non-msg_-prefixed) previous_message_id. Anthropic's /v1/messages rejects every resume attempt against that transcript with a deterministic 400. Without guards, Paperclip would re-persist the same poisoned session id and the issue is stranded permanently — see RED-976 / RED-978.
What the adapter does automatically:
- Auto-rotate on resume. If a
--resumeattempt returns this 400, the adapter retries once with a fresh session, deletes the poisoned<session>.jsonlfrom the local Claude config dir (best effort), and uses the fresh session id going forward. - Validate-before-persist. A result that carries this 400 never gets its
session_idwritten back to the task session store, even if Claude Code emits one in the result event. The adapter returnssessionId: null,sessionParams: null, anderrorCode: "claude_poisoned_previous_message_id". - Clear-on-error. The adapter sets
clearSession: trueon the result, which causes the heartbeat service to drop any persisted session row for that issue (clearTaskSessions). The next continuation starts from a clean slate.
On-call checklist if you see this in production:
- Confirm
errorCodeisclaude_poisoned_previous_message_idin the run row — that means the guards fired correctly and the issue auto-recovers on the next heartbeat. - If the same issue still loops after one heartbeat, check that
agentTaskSessionsfor that(agentId, taskKey)was cleared. If not, the adapter return value was lost (e.g. a malformed run finalization) — escalate; do not manually edit the row, file a child issue with the run id. - For remote execution targets (sandbox/SSH), the poisoned JSONL is on the remote and the adapter only logs the cleanup intent. The fresh-session retry still succeeds because it uses a new session id, and the server-side
clearSession: trueis authoritative regardless of remote disk state.
Skills Injection
The adapter creates a temporary directory with symlinks to Paperclip skills and passes it via --add-dir. This makes skills discoverable without polluting the agent's working directory.
Remote credential ownership
When no API key or CLAUDE_CODE_OAUTH_TOKEN is configured,
claude_local uses a snapshot-owns-auth topology for managed sandbox execution
targets. When the run uses a sandbox execution target and no explicit
CLAUDE_CONFIG_DIR is configured, Paperclip creates a remote
CLAUDE_CONFIG_DIR under the run's Claude runtime directory. It uploads
sanitized host-side settings such as settings.json and CLAUDE.md, but the
managed seed does not upload host Claude credential files.
After the seed is copied, the remote materialization command checks the
execution target's own $HOME/.claude directory. For each missing credential
file, it copies .credentials.json or credentials.json from that remote home
into the managed CLAUDE_CONFIG_DIR. That means credentials baked into the
sandbox image win for managed remote Claude runs.
Worked example: a sandbox image contains $HOME/.claude/.credentials.json from
its own Claude Code login. Paperclip starts a managed remote claude_local run,
uploads only the sanitized config seed, and sets CLAUDE_CONFIG_DIR to the
remote runtime config path. Because the managed config has no credential file,
the adapter copies the sandbox image's $HOME/.claude/.credentials.json into
that path before invoking Claude. The sandbox snapshot owns the credential for
the run.
This differs from codex_local, where a
Paperclip-managed sandbox run uploads a host-owned CODEX_HOME/auth.json and
therefore shadows any Codex login already present inside the sandbox image.
Sandbox memory limits
A sandbox has a fixed memory limit. When a tool command (for example a type check or a dependency install) uses most of that memory, the kernel kills one process. Without help it can kill the Claude Code process, and the run fails with "The connection to Claude was lost."
For managed sandbox runs, the same materialization step installs a
tool-shell-prefix.sh script in the run's Claude runtime directory and sets
CLAUDE_CODE_SHELL_PREFIX to it. Claude Code runs each Bash tool command and
hook through that script. The script raises the command's oom_score_adj to
1000 and then runs the command in the shell that Claude Code selects. The
kernel then kills the tool command first, and Claude sees a failed command
(exit 137) that it can recover from. Paperclip sets the variable only after the
sandbox confirms the install. An operator CLAUDE_CODE_SHELL_PREFIX in the
adapter env is kept, and no script is installed.
For manual local CLI usage outside heartbeat runs (for example running as claudecoder directly), use:
npx paperclipai agent local-cli claudecoder --company-id <company-id>
This installs Paperclip skills in ~/.claude/skills, creates an agent API key, and prints shell exports to run as that agent.
Environment Test
Use the "Test Environment" button in the UI to validate the adapter config. It checks:
- Claude CLI is installed and accessible
- Working directory is absolute and available (auto-created if missing and permitted)
- API key/auth mode hints (
ANTHROPIC_API_KEYvsCLAUDE_CODE_OAUTH_TOKENvs subscription login) - A live hello probe (
claude --print - --output-format stream-json --verbosewith promptRespond with hello.) to verify CLI readiness
The probe sees the same layered env as a real run: when an environment is
selected, its environment variables (secret refs included) are resolved and
merged under the adapter config's env, so environment-level auth is
reflected in the test result. A secret binding that is missing surfaces as
an environment_env_binding_missing failure instead of a silently passing
probe.