Devin FoleyandPaperclip
724fa6144d
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The Legacy OpenCode adapter runs an agent and its shell tools in the same sandbox. > - A large build can exceed the sandbox memory limit and trigger a kernel OOM kill. > - The kernel can kill OpenCode before its tool, which loses the active turn. > - OpenCode provides a shell environment hook that can change only the tool subprocess. > - This pull request uses that hook to prefer Bash tool processes as OOM victims. > - The benefit is that OpenCode can report a failed tool and continue the task. ## Linked Issues or Issue Description **What happened?** An OpenCode sandbox run ended with exit 137 after a TypeScript process had also been killed. Its cgroup limit was 4 GiB. It had completed useful tool work and had not reached its execution deadline. The original sandbox was deleted, so its historical OOM counters are unavailable. Memory pressure is the leading explanation, not a proven historical kernel diagnosis. **Expected behavior** When a shell tool exhausts sandbox memory, the kernel should prefer that tool over OpenCode where possible. The adapter must preserve configured shell startup, model, permissions and resource limits. **Steps to reproduce** Use a disposable 4 GiB Linux sandbox with OpenCode. Retain a 1,200 MiB buffer in the agent and run a Bash tool with fourteen 250 MiB workers. Compare the same workload before and after installing the generated hook. Record the cgroup memory events, agent exit and final response. Refs: #16844. That PR covers the same protection for Claude. This PR adds the OpenCode-owned hook. Refs: #17254 for the separate shutdown recovery path. This PR does not change retry policy. ## What Changed - Generate a dependency-free OpenCode `shell.env` plugin and a Bash startup file in the isolated runtime config. - Stage those assets only for sandbox execution. Resolve the startup file relative to the copied plugin. - Set child `oom_score_adj` to `1000` on Linux. Keep the parent environment unchanged and honor existing `BASH_ENV` values, including an empty value. - Keep command execution available if the optional plugin cannot be installed or the process priority cannot be changed. - Materialize linked plugin directories inside the isolated config before writing. Keep the operator directory and its plugins unchanged. - Cover relocation, shell output and exit status, operator settings, local execution, remote staging and failed installation. Update the adapter documentation and verification recipe. ## Verification - Red before the fix: the new runtime regression failed because the sandbox shell plugin was absent. The existing 15 runtime config tests passed. The same regression passes with the fix. - Apex first found a host-write risk through linked plugins directories (4/5). A new regression reproduced the host directory modification, then passed after the installer materialized the copied link. The finding is resolved. Apex reviewed current head `df82a8f51725a197c0ea3ebeb942488509f32b34` at 5/5 with no actionable issues remaining. - The full OpenCode adapter suite passes: 87 source tests. The adapter typecheck and map checks pass. - Real sandbox acceptance used the plugin generated from `df82a8f51725a197c0ea3ebeb942488509f32b34` and OpenCode 1.18.34 in a disposable 4 GiB Daytona sandbox. A controlled local model endpoint issued an actual Bash tool call; no real model credentials were used. The tool reported priority `1000`; the parent reported `0`. - Controlled OOM comparison: without the guard, OpenCode was killed by SIGKILL after 8.95 seconds and the cgroup recorded one OOM kill. With the guard, five workers were killed, OpenCode exited `0` after 21.87 seconds, and its final response was `fixture-complete`. The cgroup counter increased by five. Both attempts used the same allocation and workload. The fixture started with a linked operator plugins directory. The installer preserved that directory, staged a real plugins directory, and passed both the process-priority probe and OOM comparison. The fixture allocation was deleted afterward. - The first live probe was not accepted because the harness could not parse the provider response framing. After correcting the decoder, the unchanged plugin passed the probe and the controlled OOM comparison. Both initial disposable allocations were deleted. After the review fix, a third allocation repeated the probe and OOM comparison with a linked plugins directory. That allocation was also deleted. The initial comparison had the same agent-killed/agent-survived result (8.42 seconds without the guard; 19.42 seconds with it). - Repository `pnpm -r typecheck` and `pnpm build` pass. The full CI matrix passes on `df82a8f51725a197c0ea3ebeb942488509f32b34`, including tests, browser shards, Runner checks, build, policy and Apex. [CI run](https://github.com/paperclipai/paperclip/actions/runs/40043947375). - The first full local test attempt had 15 embedded PostgreSQL startup failures. Direct `initdb --version` and `postgres --version` probes showed missing bundled library symlinks and SIGABRT. Running the bundled symlink hydration script restored both executables. All 15 affected cases pass after that setup repair. No product code changed for it. - Two existing workspace tests assumed that new Git repositories have no `main` branch. They failed under the host default branch setting of `main`, then passed with a per-command `init.defaultBranch=master` setting. The global setting was preserved. The full local test command was restarted with hydrated libraries and that controlled fixture default. It had no reported assertion failures when it was stopped after the complete CI matrix passed. It did not complete locally; full-suite verification comes from CI. The first failed invocation was also stopped. - The original tenant task has not been retried or recovered by this PR. Its board API credential is expired. The original hosted workflow and the separate shutdown recovery remain outside this acceptance result. ## Risks - This is a preference, not a memory reservation. It cannot guarantee that OpenCode survives every OOM condition, including memory growth inside OpenCode itself. - The guard applies to non-interactive Bash tools. Other shells ignore `BASH_ENV`. Existing operator startup settings take precedence. - Older OpenCode releases without the shell environment hook will not apply the preference. No model, permission, deadline, memory quota or retry limit changes. - No schema change or migration. ## Model Used OpenAI GPT-6 through Codex, with repository tools and code execution. The exact deployment variant and context window are not exposed by this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
15 KiB
navigation
| navigation | ||||||
|---|---|---|---|---|---|---|
|
Runs, harnesses, and model accounts
Operators choose an agent harness and model account, inspect execution history and transcripts, and distinguish a working configuration from a successful task result.
Feature ID: runs-adapters.
Implementation: adapter manager, run host, configuration.
Agent access: companion.
Sub-features
availability:runs-adapters/availability— discover installed adapters and available models/accounts.validation:runs-adapters/validation— test an adapter in the selected execution environment and surface authentication failures.runs:runs-adapters/runs— inspect run status, events, output, costs, and the source task.session:runs-adapters/session— preserve or reset execution context through supported lifecycle actions.
How to get to it (user POV)
adapter-setup
Open an agent’s model/adapter configuration or the instance adapter settings page when permitted.
run-history
Open an agent run at /agents/:agentId/runs/:runId from the agent or task activity.
The agent header has one Run now action. Administrators can enable
Capture raw provider traces under the agent configuration’s Debugging section
for future runs. The header shows Raw tracing on while that setting is enabled.
For latency diagnosis with an operator OTLP endpoint, search the hashed run ID
without restricting the root span name. Compare workspace preparation with its
workspace_<step> intervals. Measure controller readiness separately through
controller.startup.<phase>. A recovery pause or deployment drain is not model
execution time. See the tracing contract.
These source-level diagnostics do not establish a live response-time budget.
Inspect native.controller.<phase> for backend construction, assigned-tool
discovery and post-turn cleanup. Compare those intervals with the actual model
turn; do not count a nested discovery interval twice.
Driving it with Playwright and the Paperclip CLI
Preconditions: follow the baseline. Record adapter, model, account, native/legacy runtime, and environment. Capabilities vary; one harness result is not a matrix-wide qualification.
session
For runs-adapters/session, complete a task in a disposable warm sandbox and
retain its saved workspace file. Let the warm session close normally and verify
that the retained lease has a complete, lease-bound harness backup before
injecting sandbox loss. Removing a still-warm sandbox before that checkpoint
exercises the separate acquisition safety gate, which must remain blocked.
Use a board-authorized session reset, then remove only that fixture's inactive
provider sandbox. Send a follow-up that
reads the retained file and appends one new line. Verify a replacement sandbox,
a new provider session, the exact file contents, successful finalization and no
stranded task lock. Send another follow-up and verify ordinary session reuse.
Repeat without resetting the session: its replacement must restore verified history or remain blocked. Missing backup state must not silently start an existing session again. The executor controls cover the bootstrap identity, activity and atomic directory claim; those controls do not establish live provider acceptance.
adapter-setup
Automated: adapter validation routes and model refresh cover server behavior.
Claude native setup uses the existing Anthropic API-key or subscription connection flow. Record the selected account separately from successful execution; unavailable models/authentication and unsupported Legacy-only options must fail visibly.
Claude CLI tests and runs also prepare a compatible private executable when a managed sandbox image's default CLI is too old for the selected model. Verify the command used for the hello test and task, reuse of a compatible private installation, and a blocking failure when installation cannot establish a valid version. Local, SSH and explicit custom commands retain operator management. Cloud onboarding must not offer shell commands the customer cannot run.
Manual: Select the intended adapter and account, refresh model choices, and run its environment test. Save, reload, then execute a small task. Repeat with a deliberately unavailable account and verify the failure remains visible.
For Legacy Codex, assign a service secret such as SLACK_BOT_TOKEN through agent
configuration and run a task that checks its presence without printing its value.
Check both ACP and CLI execution. Remove the binding and repeat on the next run.
The assigned secret must reach shell tools; an unassigned host token and Codex's
own login keys must remain unavailable. Verify the same delivery without an
agent identity. Shell-policy metadata must contain names rather than values.
Choose Codex as the harness; the UI displays its name/icon for both runners. Advanced identifies the automatic default and offers Paperclip Runner or Legacy. New Codex agents default to the native runner on supported macOS or GNU/Linux targets; the shared selection contract and runtime prerequisites define the platform/libc boundary. Windows and Alpine/musl remain Legacy. Claude follows its qualified rollout described above. Disabled adapters, external overrides, and selected-target qualification also affect availability.
Choose OpenCode with the native-runner experimental flag off on a qualified
macOS or GNU/Linux x64 target. Linux ARM64 must retain Legacy because its pinned
OpenCode runtime is not qualified. Advanced should show Paperclip Runner as the
automatic default on qualified targets. Save an inactive draft without a model,
resume it, select an explicit provider/model,
test, finish setup, and reload. Verify the saved provider is opencode. Repeat
with explicit Legacy and an existing Legacy agent; neither should migrate on an
unrelated edit. Check that saved native permission/lifecycle choices survive
resuming setup. Custom commands, arguments, variants, and inherited permissions
must give Legacy remediation when native is selected. A live task with the
selected account is separate proof from these setup and persistence controls.
Repeat with Pi on macOS or GNU/Linux x64 after explicit operator setup of the
pinned runtime (paperclipai runtime setup pi). Linux ARM64 retains Legacy.
Resume an inactive model-free draft, connect an account, choose a provider/model,
and finish setup with the generic native flag off. Verify Pi branding and native
ACPX Pi execution by completing a simple assigned task. Preserve saved native
thinking/permission/lifecycle settings on resume and unrelated edits. Native
thinking supports off, low, high, max; legacy aliases must give remediation,
and returning to Legacy must reject settings it cannot honor.
On a fresh forced-Kubernetes instance without an environment row, inspect discovery
and create with automatic selection. Both must retain Legacy without provisioning;
an explicit Paperclip request must give target-qualification guidance. Begin the
ordinary task to exercise the existing lazy environment path. This live journey
remains separate from the resolver's source controls.
For sandbox defaults, follow the environment probe recipe first. Discovery/creation read company-bound immutable-image evidence without allocating a target. A mutable image or snapshot name does not establish native compatibility. Then test the actual selected runtime with its selected account/model: a legacy CLI or authentication probe alone cannot establish native readiness. For an image-provided Codex harness, verify both final executables after concurrent artifact preparation on an eligible provider. Repeat with a failed runner probe and confirm the harness work has settled before failure returns. Providers without concurrent sync and explicit harness installs remain serial. The real-shell artifact controls cover these source contracts; they do not establish live reply latency. For remote native Codex with managed GitHub access, inspect the controller preparation spans for overlapping GitHub setup and assigned MCP discovery. Provider startup must wait for both. Force either preparation to fail while the other is pending; confirm that the pending operation settles and run-owned access is released before failure returns. Repeat a warm follow-up and retain its actual reply time. The executor controls cover ordering, failure drain and credential projection; they do not qualify a live response-time target. Missing dependencies, failed authentication, invalid models and incompatible custom settings must remain actionable errors without silent runner switching. Saved unchanged execution can still be tested without applying new selection policy, but runtime/account validation must still run.
Verify an automatic Codex hire and an explicit Legacy control through a useful file artifact, follow-up and reload. Preserve Codex reasoning effort and account isolation; retain explicit Legacy permission choices, including false. Reopen an existing agent, make an unrelated edit, and confirm both its persisted runner and next run stay unchanged. Use hiring/import preservation for approval and portability controls.
Test native Codex with a saved subscription whose setup auth file is a managed
link, as well as a regular file and an API-key account. Setup must seed the
isolated probe from the checked account without exposing credentials or changing
its saved runner. With subscription authentication, an unbound configured
CODEX_HOME with linked credentials must fail setup with managed-account or explicit Legacy guidance; it must not claim
readiness when task isolation would omit the file. Dangling or incorrectly owned
credentials must also fail clearly. A selected CODEX_API_KEY or
OPENAI_API_KEY takes precedence over an unrelated subscription file. Test valid,
dangling and invalid subscription links with that selected key; setup must test
the key without reading or refreshing the unrelated account. The private probe copy must be removed after confirmed retirement following success,
failure or cancellation. Unconfirmed shutdown remains a cleanup failure.
Exercise an unavailable local daemon and a preparation failure before probe
handoff. No owned temporary credential copy may remain, and original account
files must be unchanged. A probe whose shutdown is unconfirmed must retain its
recovery state; do not treat it as a preparation-only failure.
Existing setup controls use fake credentials; real account execution remains
a separate acceptance requirement.
run-history
Automated: live run routes and run event sequencing cover run data contracts.
Manual: Start a disposable task, open its running transcript, and follow it to a terminal outcome. Reload and compare task/run IDs, status, output, and cost. Verify a failed run remains distinguishable from a completed run with useful output.
For Legacy OpenCode on a disposable Linux sandbox, run a Bash tool that reads
/proc/self/oom_score_adj and its parent's score. The tool should report 1000;
OpenCode should retain its original score. Repeat with an operator BASH_ENV
and confirm that it takes precedence. Repeat with a linked operator plugins
directory; it must remain unchanged, and the uploaded runtime must contain a
real plugins directory with the guard and existing plugins. Under controlled memory pressure, record
the cgroup OOM counters and verify that a killed tool can return an error while
the agent continues. Local and SSH execution must keep their original shell
environment. See the adapter contract.
These controls do not guarantee that every memory exhaustion preserves the agent.
An explicit runner change starts a configuration revision and invalidates the eligible session. Active runs, restart recovery and historical transcript parsing retain their recorded execution configuration; do not infer their runner from a later edit to the agent. Account/harness branding and execution identity are different observations, both required in the evidence.
For an onboarding child-completion callback, verify actual session/turn events
and the source-correlated reply. A succeeded row alone is insufficient: the
literal ea6c8eb5e16d61111968f633a66748b397b64a3e managed attempt was marked
successful during startup without a provider turn. The recovery correction
preserves the existing validated Done-parent handoff; process-death and ownership
checks remain active. Its PostgreSQL controls are separate from the required
revised-source live onboarding and follow-up proof.
Gotchas
-
Model accounts are managed in Apps and inline agent account controls.
/company/settings/connectionsconfigures decision/fast-response models, not the general model-account library. -
The run log is instance database data; it is not the opt-out first-party Telemetry system or operator-configured OpenTelemetry tracing.
-
Authentication/model discovery does not establish tool, image, steering, or resumption capability parity.
-
Use recovery for interrupted runs and execution environments for local/SSH/sandbox differences.
Native setup retirement joins owned startup before cleanup can be confirmed. Transport ownership controls hold preparation or registration without launching a provider, then check bounded close, rejected duplicate requests, no later launch and retention of unconfirmed state. Readiness observes shutdown; resumed startup cannot reactivate ingress after close. These are lifecycle controls, separate from installed provider proof. The real daemon readiness and recovery controls require their existing native fixtures in hosted CI; passing credential-free controls does not qualify them. A primary setup timeout remains a failure even when teardown also fails.
For remote setup shutdown, verify that each durable-state read respects the
remaining suspension deadline. A late result cannot establish suspension or
queue a replacement command after that deadline. An intentional ingress close
must observe physical socket closure before retiring its connection owner and
releasing the endpoint; an unresponsive peer must not extend the setup budget.
The existing transport and ingress suites exercise these conditions without a
provider. Repeat the actual managed setup and useful task journey separately;
the literal 651a3de64fc943216cc414b64b1443809f9499da Test attempt failed
cleanup before hire and retains its unconfirmed recovery state.