mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-09 07:10:34 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Hosted `claude_local` runs execute in a sandbox. The sandbox has a fixed memory limit (4 GiB in the observed sandboxes). > - In that sandbox, every process has the same `oom_score_adj`. When a Bash tool command uses most of the memory, the kernel kills the largest process. That can be the Claude Code process. > - When Claude Code is killed, `claude-agent-acp` reports "The connection to Claude was lost." and the whole run fails. Server logs show `Claude Agent process died: Claude Code process terminated by signal SIGKILL` for each failure. > - A failed tool command is recoverable. A dead agent is not. > - This pull request makes Claude's tool commands the first out-of-memory victims in managed sandboxes. > - The benefit is that a memory-heavy command fails with exit 137, and the agent continues the run. ## Linked Issues or Issue Description Refs #16366 (diagnostic PR for the same failure). Related: #14509 protects the host control plane from local agent processes. That PR does not cover a sandboxed agent and its own tool commands. **What happened?** Hosted `claude_local` runs fail with `acpx_turn_failed` and "The connection to Claude was lost." while one `execute` (Bash) tool call is pending. The pending commands are memory-heavy, for example `tsc --noEmit` with a 3.3 GB heap, a monorepo `pnpm install`, or a type check plus `vitest`. At the same second, the server log shows `Claude Agent process died: Claude Code process terminated by signal SIGKILL`. Over 10 days, all 12 such lines on one staging stack were `SIGKILL`, and they match the failed runs. The sandbox memory cgroup has `memory.max` 4 GiB and `memory.oom.group` 0, so the kernel kills one process. Every process has `oom_score_adj` 0, so the kernel picks the largest one. **Expected behavior** Under memory pressure, the kernel kills the tool command. Claude gets a failed command (exit 137) and continues the run. **Steps to reproduce** 1. Start a `claude_local` run on a sandbox target with a 4 GiB memory limit. 2. Ask the agent to run a memory-heavy command in a large repository, for example a full `tsc --noEmit` and a test run together. 3. The kernel kills the Claude Code process, and the run fails with "The connection to Claude was lost." **Agent adapter(s) involved** Claude Code (`claude_local`, ACP lane and CLI lane, managed sandbox targets). ## What Changed - `claude-config.ts`: add `CLAUDE_TOOL_SHELL_PREFIX_SCRIPT`. Claude Code calls a `CLAUDE_CODE_SHELL_PREFIX` with one argument, the full command string. The script raises its own `oom_score_adj` to 1000 (no privilege is necessary to raise it). Then it runs the command string in the shell that Claude Code selects: `CLAUDE_CODE_SHELL`, then `SHELL`, when either is an executable bash or zsh, else zsh or bash from `PATH` (bash first when `SHELL` names bash). - `buildRemoteClaudeConfigMaterializationCommand` can install the script (umask 077, write to `.partial`, `chmod 700`, `mv`). The install runs last in the existing materialization command, so it adds no sandbox round trip. It prints a marker only when the file is in place. - `materializeRemoteClaudeConfig` returns `toolShellPrefixInstalled`. It is true only when the command exits 0 and prints the marker. - ACP lane (`acp.ts`) and CLI lane (`execute.ts`): set `CLAUDE_CODE_SHELL_PREFIX` to the installed script only when the install is confirmed. A prefix path that does not exist would fail every Bash tool call, so a failed install keeps the previous behavior. An operator `CLAUDE_CODE_SHELL_PREFIX` in the adapter env is kept, and no script is installed. - `docs/adapters/claude-local.md`: add "Sandbox memory limits". ## Verification - `npx vitest run src/server/claude-config.test.ts` in `packages/adapters/claude-local`: 10 pass, 6 new. The new tests install the script with the real materialization command through `/bin/sh` and check: - the file content, mode 0700, the marker, and that no `.partial` file remains; - the command runs with `oom_score_adj` 1000 while the caller keeps its own score (Linux only); - shell selection: a valid bash `SHELL`, an invalid `CLAUDE_CODE_SHELL` that falls back to `SHELL`, and a `SHELL` of `/bin/sh` that falls back to bash from `PATH`; - quoting and the exit status of the command are kept; - an operator prefix wins, and no marker means no install. - `acp.test.ts`: the managed-config test now checks that `CLAUDE_CODE_SHELL_PREFIX` names an executable 0700 file next to the config dir. A new test checks that an operator prefix is kept and no script is installed. Both use the local sandbox runner, which runs the setup commands for real. - `tsc --noEmit -p packages/adapters/claude-local` is clean. - Two tests fail in my sandbox both with and without this change: `uses the same default model in ACP startup and session identity` and `reselects the full assignment and bootstrap guidance after a failed resume`. They also fail on clean `master` in the same environment, so I think they depend on that environment. CI covers them. - Real Claude Code check (Claude Code 2.1.286): with `CLAUDE_CODE_SHELL_PREFIX` set to the script, a Bash tool call printed `oom_score_adj` 1000 and ran in zsh (the selected shell), and its parent was the `claude` process at score 0. Without a prefix file that exists, every Bash tool call fails with exit 127. This is why the env is set only after a confirmed install. - Real OOM check in a 4 GiB Daytona sandbox (`memory.oom.group` 0). A stand-in "agent" process held 1.2 GB at score 0, and a tool command started 14 processes of 250 MB each: - without the prefix, the kernel killed the stand-in agent (SIGKILL, 137; `oom_kill` +1); - through the installed prefix, the stand-in agent stayed alive, and the kernel killed 5 tool processes (`oom_kill` +5). - Assumption that could invalidate the design: Claude Code passes one argument (the command string) to the prefix and honors the variable through `claude-agent-acp`. I checked the argument shape with a logging prefix, and `claude-agent-acp` 0.73.0 spawns the CLI with `{ ...process.env }`. ## Risks - Every Bash tool command and hook in a managed sandbox now runs through one more `sh` process and an `exec`. The cost is a few milliseconds for each command. - If a future Claude Code release changes how it calls `CLAUDE_CODE_SHELL_PREFIX`, Bash tool calls can fail. The tests above and the documented variable limit this risk. Removing the variable restores the old behavior. - The inner shell is not a login shell. Claude Code sources its own shell snapshot inside the command string, so the environment stays the same. If Claude Code runs without a snapshot, it uses a login shell. Only exported variables then pass through to the inner shell. - If the selected shell differs from the one that Claude Code chose (for example, a custom `CLAUDE_CODE_SHELL` that is not bash or zsh), the script uses the same fallback order as Claude Code. - This does not help when the Claude Code process itself uses most of the memory. The local (non-sandbox) lanes do not change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with tool use and code execution. It read the repository and the Claude Code CLI behavior, ran the OOM experiment, edited the code, and ran the tests. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass (two tests fail on clean `master` in my sandbox too; see Verification) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Paperclip <noreply@paperclip.ing>
175 lines
9.7 KiB
Markdown
175 lines
9.7 KiB
Markdown
---
|
|
title: Claude Code
|
|
summary: Claude Code local adapter setup and configuration
|
|
---
|
|
|
|
The `claude_local` adapter runs Anthropic's Claude Code CLI locally. It supports session persistence, skills injection, and structured output parsing.
|
|
|
|
## Quota waits
|
|
|
|
Claude ACP runs that end with a typed provider-quota error retain the quota
|
|
classification and any parsed reset time. Recovery waits until that time, or
|
|
uses its existing one-hour quota backoff when no reset time is available.
|
|
This includes the Claude bridge's typed “The Claude account has no available
|
|
quota.” fallback, which carries no reset timestamp.
|
|
The adapter inspects the terminal provider message in memory; the run result
|
|
and run log retain only the generic failure message, recovery labels, and reset
|
|
timestamp. Context, turn, rate, and configured budget limits are not treated as
|
|
subscription quota exhaustion merely because ACP labels them `limit`.
|
|
|
|
## Prerequisites
|
|
|
|
- Claude Code CLI installed (`claude` command available)
|
|
- Either `ANTHROPIC_API_KEY` or `CLAUDE_CODE_OAUTH_TOKEN` in adapter or
|
|
environment env (or host env), or a Claude Code subscription login
|
|
available to the execution target
|
|
|
|
## Configuration Fields
|
|
|
|
| Field | Type | Required | Description |
|
|
|-------|------|----------|-------------|
|
|
| `cwd` | string | Yes | Working directory for the agent process (absolute path; created automatically if missing when permissions allow) |
|
|
| `model` | string | No | Claude model to use (default: `claude-opus-5`) |
|
|
| `promptTemplate` | string | No | Prompt used for all runs |
|
|
| `env` | object | No | Environment variables (supports secret refs) |
|
|
| `timeoutSec` | number | No | Process timeout (0 = no timeout) |
|
|
| `graceSec` | number | No | Grace period before force-kill |
|
|
| `maxTurnsPerRun` | number | No | Max agentic turns per heartbeat (defaults to `300`) |
|
|
| `dangerouslySkipPermissions` | boolean | No | Skip permission prompts (default: `true`); required for headless runs where interactive approval is impossible |
|
|
|
|
On the CLI engine, version and `--effort` capability probes have a bounded
|
|
20-second budget when `timeoutSec` is zero or unset. A positive run timeout below
|
|
20 seconds also limits these probes, with a minimum probe budget of one second.
|
|
|
|
## Default model
|
|
|
|
An omitted, empty, or whitespace-only `model` uses Claude Opus 5
|
|
(`claude-opus-5`) on both the CLI and ACP engines. This also applies to existing
|
|
agents with an unset model, including agents created through the API and agents
|
|
running in sandboxes. No database migration is needed. The editor shows the
|
|
Paperclip default and leaves the setting unset until you select a model.
|
|
|
|
An explicit `model` takes precedence over `ANTHROPIC_MODEL`. When only
|
|
`ANTHROPIC_MODEL` is configured, the adapter keeps that override. Bedrock and
|
|
Vertex configurations without an explicit model keep their provider-specific
|
|
default because those providers use different model IDs. Host environment
|
|
settings apply only to local targets when resolving the model.
|
|
|
|
The default does not change explicitly configured agent models or the separate
|
|
Paperclip Runner's qualified provider profiles.
|
|
|
|
## Prompt Templates
|
|
|
|
Templates support `{{variable}}` substitution:
|
|
|
|
| Variable | Value |
|
|
|----------|-------|
|
|
| `{{agentId}}` | Agent's ID |
|
|
| `{{companyId}}` | Company ID |
|
|
| `{{runId}}` | Current run ID |
|
|
| `{{agent.name}}` | Agent's name |
|
|
| `{{company.name}}` | Company name |
|
|
|
|
## Session Persistence
|
|
|
|
The adapter persists Claude Code session IDs between heartbeats. On the next wake, it resumes the existing conversation so the agent retains full context.
|
|
|
|
Session resume is cwd-aware: if the agent's working directory changed since the last run, a fresh session starts instead.
|
|
|
|
If resume fails with an unknown session error, the adapter automatically retries with a fresh session.
|
|
|
|
### Poisoned `previous_message_id` (recovery)
|
|
|
|
Symptom in logs / issue thread:
|
|
|
|
```
|
|
API Error: 400 diagnostics.previous_message_id: must be the `id` from a prior /v1/messages response (starts with `msg_`)
|
|
```
|
|
|
|
What it means: the on-disk Claude Code transcript JSONL for that session contains a malformed (non-`msg_`-prefixed) `previous_message_id`. Anthropic's `/v1/messages` rejects every resume attempt against that transcript with a deterministic 400. Without guards, Paperclip would re-persist the same poisoned session id and the issue is stranded permanently — see [RED-976](../../../) / [RED-978](../../../).
|
|
|
|
What the adapter does automatically:
|
|
|
|
1. **Auto-rotate on resume.** If a `--resume` attempt returns this 400, the adapter retries once with a fresh session, deletes the poisoned `<session>.jsonl` from the local Claude config dir (best effort), and uses the fresh session id going forward.
|
|
2. **Validate-before-persist.** A result that carries this 400 never gets its `session_id` written back to the task session store, even if Claude Code emits one in the result event. The adapter returns `sessionId: null`, `sessionParams: null`, and `errorCode: "claude_poisoned_previous_message_id"`.
|
|
3. **Clear-on-error.** The adapter sets `clearSession: true` on the result, which causes the heartbeat service to drop any persisted session row for that issue (`clearTaskSessions`). The next continuation starts from a clean slate.
|
|
|
|
On-call checklist if you see this in production:
|
|
|
|
- Confirm `errorCode` is `claude_poisoned_previous_message_id` in the run row — that means the guards fired correctly and the issue auto-recovers on the next heartbeat.
|
|
- If the same issue still loops after one heartbeat, check that `agentTaskSessions` for that `(agentId, taskKey)` was cleared. If not, the adapter return value was lost (e.g. a malformed run finalization) — escalate; do **not** manually edit the row, file a child issue with the run id.
|
|
- For remote execution targets (sandbox/SSH), the poisoned JSONL is on the remote and the adapter only logs the cleanup intent. The fresh-session retry still succeeds because it uses a new session id, and the server-side `clearSession: true` is authoritative regardless of remote disk state.
|
|
|
|
## Skills Injection
|
|
|
|
The adapter creates a temporary directory with symlinks to Paperclip skills and passes it via `--add-dir`. This makes skills discoverable without polluting the agent's working directory.
|
|
|
|
## Remote credential ownership
|
|
|
|
When no API key or `CLAUDE_CODE_OAUTH_TOKEN` is configured,
|
|
`claude_local` uses a snapshot-owns-auth topology for managed sandbox execution
|
|
targets. When the run uses a sandbox execution target and no explicit
|
|
`CLAUDE_CONFIG_DIR` is configured, Paperclip creates a remote
|
|
`CLAUDE_CONFIG_DIR` under the run's Claude runtime directory. It uploads
|
|
sanitized host-side settings such as `settings.json` and `CLAUDE.md`, but the
|
|
managed seed does not upload host Claude credential files.
|
|
|
|
After the seed is copied, the remote materialization command checks the
|
|
execution target's own `$HOME/.claude` directory. For each missing credential
|
|
file, it copies `.credentials.json` or `credentials.json` from that remote home
|
|
into the managed `CLAUDE_CONFIG_DIR`. That means credentials baked into the
|
|
sandbox image win for managed remote Claude runs.
|
|
|
|
Worked example: a sandbox image contains `$HOME/.claude/.credentials.json` from
|
|
its own Claude Code login. Paperclip starts a managed remote `claude_local` run,
|
|
uploads only the sanitized config seed, and sets `CLAUDE_CONFIG_DIR` to the
|
|
remote runtime config path. Because the managed config has no credential file,
|
|
the adapter copies the sandbox image's `$HOME/.claude/.credentials.json` into
|
|
that path before invoking Claude. The sandbox snapshot owns the credential for
|
|
the run.
|
|
|
|
This differs from [`codex_local`](/adapters/codex-local), where a
|
|
Paperclip-managed sandbox run uploads a host-owned `CODEX_HOME/auth.json` and
|
|
therefore shadows any Codex login already present inside the sandbox image.
|
|
|
|
## Sandbox memory limits
|
|
|
|
A sandbox has a fixed memory limit. When a tool command (for example a type
|
|
check or a dependency install) uses most of that memory, the kernel kills one
|
|
process. Without help it can kill the Claude Code process, and the run fails
|
|
with "The connection to Claude was lost."
|
|
|
|
For managed sandbox runs, the same materialization step installs a
|
|
`tool-shell-prefix.sh` script in the run's Claude runtime directory and sets
|
|
`CLAUDE_CODE_SHELL_PREFIX` to it. Claude Code runs each Bash tool command and
|
|
hook through that script. The script raises the command's `oom_score_adj` to
|
|
1000 and then runs the command in the shell that Claude Code selects. The
|
|
kernel then kills the tool command first, and Claude sees a failed command
|
|
(exit 137) that it can recover from. Paperclip sets the variable only after the
|
|
sandbox confirms the install. An operator `CLAUDE_CODE_SHELL_PREFIX` in the
|
|
adapter env is kept, and no script is installed.
|
|
|
|
For manual local CLI usage outside heartbeat runs (for example running as `claudecoder` directly), use:
|
|
|
|
```sh
|
|
npx paperclipai agent local-cli claudecoder --company-id <company-id>
|
|
```
|
|
|
|
This installs Paperclip skills in `~/.claude/skills`, creates an agent API key, and prints shell exports to run as that agent.
|
|
|
|
## Environment Test
|
|
|
|
Use the "Test Environment" button in the UI to validate the adapter config. It checks:
|
|
|
|
- Claude CLI is installed and accessible
|
|
- Working directory is absolute and available (auto-created if missing and permitted)
|
|
- API key/auth mode hints (`ANTHROPIC_API_KEY` vs `CLAUDE_CODE_OAUTH_TOKEN` vs subscription login)
|
|
- A live hello probe (`claude --print - --output-format stream-json --verbose` with prompt `Respond with hello.`) to verify CLI readiness
|
|
|
|
The probe sees the same layered env as a real run: when an environment is
|
|
selected, its environment variables (secret refs included) are resolved and
|
|
merged under the adapter config's `env`, so environment-level auth is
|
|
reflected in the test result. A secret binding that is missing surfaces as
|
|
an `environment_env_binding_missing` failure instead of a silently passing
|
|
probe.
|