mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-13 07:12:00 +01:00
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents can run in remote sandboxes (Daytona), where each turn needs
the task's workspace and a runner inside the sandbox
> - With a default environment, every remote turn got a brand-new
sandbox: the whole workspace was uploaded again and a new runner
started, even for a follow-up comment
> - In a paperclip-sized workspace that made a follow-up take 30 s to
first token, against 8–10 s with a reusable warm sandbox
> - This pull request makes an agent that never chose a lifecycle run
warm on a sandbox for the measured harnesses (Codex, OpenCode, Claude),
so a follow-up keeps its sandbox, runner and workspace
> - The default applies only where warm can work: a sandbox provider
that keeps reusable leases, and a managed AI account that supports a
warm session. Everywhere else the run is per-turn, exactly as before
> - The benefit is fast remote follow-ups by default, without
re-uploading a large repository every turn
## Linked Issues or Issue Description
**What happened?**
On Daytona, with a fresh paperclip clone as the project workspace, a
follow-up comment took 30 s to its first token. The run log showed
`environment: fresh/created` on every follow-up. It spent 13.3 s in
`workspace_sync.transfer`, a full upload into the new sandbox, and 8–9 s
in `runner.session.resume`.
**Steps to reproduce**
1. Create a Daytona environment with the defaults: lifecycle inherited,
lease reuse off.
2. Run a Codex task in a large repository, then post a follow-up.
3. The follow-up gets a new sandbox, uploads the whole workspace again
and starts a new runner.
**Expected behavior**
A follow-up on the same task reuses the live sandbox, its runner and its
workspace (plan rule 2: no full re-upload per turn).
## What Changed
- **One rule for the default.** `defaultsRemoteSandboxRunToWarm`
(`native-runtime/native-warm-capability.ts`) decides the remote-sandbox
warm default for both the lease (environment view) and the runner
(native session policy), in one vocabulary (runtime backend plus ACPX
agent). It is applied only when no lifecycle was chosen anywhere:
admitted session, environment `runnerLifecycleMode`, or agent
`lifecycleMode`. It holds only when:
1. the harness is measured (Codex, OpenCode, ACPX Claude);
2. the sandbox provider keeps reusable leases (Daytona, CreateOS and
boxd declare them; E2B, Modal and the built-in `fake` do not); and
3. with a managed AI account, that harness supports a managed warm
session (`supportsManagedNativeWarmSession`: Codex).
- **The harness resolves as the runtime resolves it.**
`resolveWarmProviderSelection` uses
`resolvePaperclipRunnerProviderProfile`, so an omitted `provider` is
Codex and ACPX without an agent is Claude, in both decisions. A config
the runtime would refuse gets no default.
- **Lease side** (`runner-environment-lifecycle.ts`,
`environment-run-orchestrator.ts`): for an eligible run only, the
orchestrator asks the provider before acquisition
(`providerSupportsReusableLeases` in `environment-runtime.ts`: the
acquisition rule, one registry lookup for a plugin provider, no
worker-ready wait). With support, the run-scoped view sets `reuseLease:
true` and forwards the warm idle window to the provider's auto-stop. The
shared environment is never rewritten. Every other run resolves exactly
as on master, without the probe.
- **Runner side** (`native-lifecycle.ts`, `runtime-selection.ts`): the
native policy applies the same rule to the acquired lease
(`reusableLeaseConfigured && effectiveCapabilities.reusableLeases`). If
acquisition downgrades the lease anyway, the run falls back to per-turn;
a default never produces
`runner_warm_environment_requires_reusable_lease`.
- **Idle window.** The remote default uses the window the environment
view forwards to the provider: an inheriting environment's
`runnerIdleTimeoutMs`, then the agent's `idleTimeoutMs`, then 5 minutes.
The sandbox execution target now carries the environment's window
(`runnerIdleTimeoutMs`, an optional field in `adapter-utils`'s execution
target) so the runner and the provider auto-stop agree. An explicit warm
choice and the local default are unchanged.
- Unchanged: explicit agent choices, environments that pin a lifecycle,
plugin-task environments, admitted sessions (`persistedPolicy`), and
local behavior.
- `tests/restart-chaos`: the vCPU budget check counts `--daytona-inherit
--lifecycle default` sandboxes as retained, because they now default to
warm.
- `doc/plans/2026-11-08-large-workspace-speed.md`: the measurements, the
ranked waits, and the gate.
## Verification
- Unit and integration tests, one per rule:
- Rule 1 (provider gate): `native-lifecycle.test.ts` (ephemeral lease →
per-turn), `runner-environment-lifecycle.test.ts` (no reusable leases →
environment unchanged, no warm window),
`environment-runtime-reusable-lease-probe.test.ts` (`fake`, an
undeclared plugin, a declared plugin whose worker does not verify the
reuse methods, and a plugin without a running worker read false),
`runtime-selection.test.ts` ("defaults an agent without a lifecycle on
...": reusable → warm/keep_running; reuse configured on an unsupported
provider → per_turn/destroy_after_turn; ephemeral → per_turn),
`environment-run-orchestrator.test.ts` (probe result decides
`reuseLease` and the forwarded window).
- Rule 2 (explicit configuration wins): explicit per-turn agent, pinned
environment, admitted session, configured `idleTimeoutMs`, managed
Claude per-turn, managed Codex warm; the orchestrator does not probe
when a lifecycle is explicit.
- Rule 3 (omitted provider): `{}` resolves to Codex in the environment
view; `runtime-selection.test.ts` "agrees with the environment view's
lease for ..." feeds the environment view's lease into selection for
Codex, a provider-less Codex agent, OpenCode, ACPX Claude, ACPX without
an agent, and Copilot (per-turn), and checks both decisions agree. A
plugin-task target stays out of the default.
- The two suites that failed on the previous head
(`runtime-selection.test.ts`,
`workspace-recovery-preparation-dispatch.test.ts`) pass locally.
Locally, 6 connector-skill cases in `runtime-selection.test.ts` fail
with `ENOENT .claude/skills/slack/SKILL.md` because this checkout is a
nested worktree; they do not touch this change and pass in CI.
- End-to-end check of this default on Daytona (before the gate was
added; Daytona keeps reusable leases, so the gate does not change this
path): a user-default environment (lifecycle inherited, lease reuse
unset, `--daytona-inherit`), a fresh paperclip clone, built from the
image's source revision with this change (branch measure/remote-warm).
| | Before (per-turn sandbox) | After (this change) |
|---|---|---|
| Follow-up, Codex | 30.4 s | 7.6 s |
| Follow-up, Claude | about 30 s | 11.0 s |
| First follow-up after an idle K1, Codex | 45.9 s | 20.3 s |
| First follow-up after an idle K1, Claude | 38.3 s | 18.3 s |
The run-log reuse records show the follow-ups reusing the workspace,
environment, runner and provider session, with no
`workspace_sync.transfer`. First turns still upload the workspace once
(14.8 s and 26.1 s). Every sandbox was cleaned up.
- Daytona restart chaos wave (K1/K2/K4 with a mid-run comment,
inheriting environment, agents with no lifecycle, runner image built
from this branch): wave `w-20261110T171438-s20048`, seed 20048, on
b7f6243d60 with image `paperclip-daytona-runner@sha256:76a33275…6fe29`
(built from this branch by the `image_only` runner workflow; runner
content id `ed503454…a479` matches every later commit here, which change
only server, adapter-utils types and tests). `--daytona-inherit
--lifecycle default --adapters codex,claude --workloads A1,C1 --triggers
time,comment --faults K1,K2,K4 --tasks 6 --concurrency 3`.
- **6/6 tasks pass** (R1–R6 and T), all 6 crossed a restart; cells:
Codex A1 K2, Codex A1 K4, Claude A1 K4, Claude C1 K1, Claude C1 K4,
Codex C1 K2 at the mid-run comment. **No task needed Retry** (R3 n/a on
every task; one Claude task recovered automatically from
`process_lost`). No open signatures, 0 speed breaches.
- The default applied: the R6 sweep found 6 retained reusable leases
(one per task, Daytona keeps them) and no orphan or leaked sandbox; the
environment was deleted cleanly.
- Timings (Daytona, first turns, n=3 each): TTFT p50 Codex 21.9 s,
Claude 31.0 s. Restart recovery p50/p95: Codex 10.5 s / 29.6 s (n=3),
Claude 8.8 s / 23.9 s (n=2). Warm follow-up TTFT has no clean sample in
this wave (both follow-ups were crossed by a fault), so the follow-up
comparison above (30.4 s → 7.6 s Codex, ~30 s → 11.0 s Claude) remains
the speed evidence; no separate per-turn chaos wave was run.
- An earlier attempt, `w-20261110T170357-s20048`, never started tasks:
the first sandbox from the new image stayed `creating` past the
harness's 300 s Codex qualification bound (Daytona building the new
snapshot). It left one sandbox that the harness reported after deleting
the environment; the rerun on the cached snapshot is the wave above.
- The idle-window fix (9a74116082) landed after the wave; it changes
only runs whose inheriting environment sets `runnerIdleTimeoutMs`, which
this wave's environment does not.
## Previous-release story
No persisted state changes shape. A session admitted per-turn before
this change keeps its saved policy (`persistedPolicy` wins) until a new
session starts. A run in flight at the cutover keeps its admitted
lifecycle for recovery (`admittedLifecycleMode` wins in the environment
view).
## Cost and opt-out
- **Cost:** after a turn, the sandbox and its warm runner stay up for
the warm idle window (the environment's `runnerIdleTimeoutMs`, else the
agent's `idleTimeoutMs`, else 5 minutes). Then the controller suspends
the runner, and the provider auto-stop follows the same window. One idle
window per active task.
- **Opt-out:** set the agent's lifecycle to "Turn by turn"
(`lifecycleMode: per_turn`), or set the environment's
`runnerLifecycleMode` to `per_turn`. Either wins over this default.
## Reuse scope, restart and saved work
- **One lineage per sandbox.** The Daytona plugin keys a reusable lease
by the agent plus the execution workspace (or issue). Another task never
gets a sandbox holding a different task's files, and subtasks still get
fresh workspaces from the project base, so they get their own sandbox.
The workspace is adopted only after its stamp matches the host's.
- **After a restart,** a warm sandbox is reattached or replaced through
the existing warm recovery and owner fencing. A sandbox with a live
owner is never claimed twice. The Daytona chaos wave above checks this.
- **No lost work.** Each turn's workspace copyback
(`workspace_finalize`) runs at the end of the turn, before any idle
suspend, as for per-turn runs.
## Risks
- **Cost:** an idle sandbox stays up for the warm idle window (5
minutes) after a turn, then suspends. Only on providers with reusable
leases.
- **Probe cost:** one plugin-registry lookup before acquisition, only
for runs the default can apply to. No wait for a worker: a provider
whose worker is not running yet reads as unsupported, so that run is
per-turn.
- **Known gap (pre-existing, not widened):** with an explicit agent warm
choice on an inheriting environment that sets `runnerIdleTimeoutMs`, the
provider gets the environment's window while the host uses the agent's,
as on master.
- **Restart behavior on warm remote sandboxes** uses the existing warm
recovery paths. The Daytona chaos wave above checks K1, K2 and K4.
- Sessions admitted per-turn before this change stay per-turn until a
new session starts.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file editing, test execution, Daytona measurements).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
141 lines
6.6 KiB
TypeScript
141 lines
6.6 KiB
TypeScript
import { resolvePaperclipRunnerIdleTimeoutMs } from "@paperclipai/adapter-utils";
|
|
import type { Environment } from "@paperclipai/shared";
|
|
import { REMOTE_RUNNER_RECONNECT_GRACE_MS } from "./native-runtime/native-runner-recovery-contract.js";
|
|
import { defaultsRemoteSandboxRunToWarm, resolveWarmProviderSelection } from "./native-runtime/native-warm-capability.js";
|
|
|
|
/** Facts the remote-sandbox warm default needs beyond the configuration. */
|
|
export interface RunnerLifecycleDefaultContext {
|
|
/** The sandbox provider keeps reusable leases (probed before acquisition). */
|
|
reusableLeaseSupported?: boolean;
|
|
/** The run uses a managed AI account. */
|
|
managedAiCredentialIdentity?: string;
|
|
}
|
|
|
|
function explicitLifecycleMode(
|
|
config: Record<string, unknown>,
|
|
adapterConfig: Record<string, unknown>,
|
|
admittedLifecycleMode?: "warm" | "per_turn",
|
|
): "warm" | "per_turn" | undefined {
|
|
if (admittedLifecycleMode) return admittedLifecycleMode;
|
|
if (config.runnerLifecycleMode === "warm" || config.runnerLifecycleMode === "per_turn") return config.runnerLifecycleMode;
|
|
if (adapterConfig.lifecycleMode === "warm" || adapterConfig.lifecycleMode === "per_turn") return adapterConfig.lifecycleMode;
|
|
return undefined;
|
|
}
|
|
|
|
/**
|
|
* The remote-sandbox warm default for a run whose lifecycle nobody chose,
|
|
* through the same rule as the native session policy. The harness resolves
|
|
* exactly as the runtime resolves it (an omitted provider is Codex); a config
|
|
* the runtime would refuse gets no default here.
|
|
*/
|
|
function sandboxRunDefaultsToWarm(
|
|
adapterConfig: Record<string, unknown>,
|
|
context: RunnerLifecycleDefaultContext,
|
|
): boolean {
|
|
const profile = resolveWarmProviderSelection(adapterConfig);
|
|
if (!profile) return false;
|
|
return defaultsRemoteSandboxRunToWarm({
|
|
profile,
|
|
reusableLeaseSupported: context.reusableLeaseSupported === true,
|
|
// The attribution is not known here; a harness that needs it for a managed
|
|
// warm session (Copilot) is not a measured default, so this is exact.
|
|
managedAi: context.managedAiCredentialIdentity ? {} : null,
|
|
});
|
|
}
|
|
|
|
function resolveRunnerLifecycleModeForRun(
|
|
environment: Pick<Environment, "driver" | "config">,
|
|
adapterConfig: Record<string, unknown>,
|
|
admittedLifecycleMode: "warm" | "per_turn" | undefined,
|
|
context: RunnerLifecycleDefaultContext,
|
|
): unknown {
|
|
const config = environment.config ?? {};
|
|
const explicit = explicitLifecycleMode(config, adapterConfig, admittedLifecycleMode);
|
|
if (explicit) return explicit;
|
|
if (environment.driver === "sandbox" && sandboxRunDefaultsToWarm(adapterConfig, context)) return "warm";
|
|
return adapterConfig.lifecycleMode;
|
|
}
|
|
|
|
/**
|
|
* True when only the provider's reusable-lease support is left to decide this
|
|
* run's sandbox lifecycle, so the caller probes it only for these runs.
|
|
*/
|
|
export function runnerLifecycleDefaultNeedsLeaseProbe(
|
|
environment: Pick<Environment, "driver" | "config">,
|
|
adapterType: string,
|
|
adapterConfig: Record<string, unknown> = {},
|
|
admittedLifecycleMode?: "warm" | "per_turn",
|
|
managedAiCredentialIdentity?: string,
|
|
): boolean {
|
|
if (adapterType !== "paperclip_runner" || environment.driver !== "sandbox") return false;
|
|
if (explicitLifecycleMode(environment.config ?? {}, adapterConfig, admittedLifecycleMode)) return false;
|
|
return sandboxRunDefaultsToWarm(adapterConfig, { reusableLeaseSupported: true, managedAiCredentialIdentity });
|
|
}
|
|
|
|
/** Resolve runner retention before acquiring a sandbox, not after it is ephemeral.
|
|
* This is a run-scoped view; never persist it over a shared environment's config.
|
|
* The driver still verifies provider support and chooses the actual lease policy.
|
|
*/
|
|
export function resolveRunnerEnvironmentForRun<T extends Pick<Environment, "driver" | "config">>(
|
|
environment: T,
|
|
adapterType: string,
|
|
adapterConfig: Record<string, unknown> = {},
|
|
admittedLifecycleMode?: "warm" | "per_turn",
|
|
managedAiCredentialIdentity?: string,
|
|
defaultContext: Pick<RunnerLifecycleDefaultContext, "reusableLeaseSupported"> = {},
|
|
): T {
|
|
if (adapterType !== "paperclip_runner" || !["sandbox", "plugin"].includes(environment.driver)) return environment;
|
|
const config = environment.config ?? {};
|
|
const lifecycleMode = resolveRunnerLifecycleModeForRun(environment, adapterConfig, admittedLifecycleMode,
|
|
{ ...defaultContext, managedAiCredentialIdentity });
|
|
// Warm tasks need a managed credential identity; recovery keeps its admitted policy.
|
|
if (environment.driver === "plugin") return { ...environment, config: { ...config,
|
|
runnerLifecycleMode: admittedLifecycleMode ?? (managedAiCredentialIdentity && lifecycleMode !== "per_turn" ? "warm" : "per_turn"),
|
|
runnerIdleTimeoutMs: resolvePaperclipRunnerIdleTimeoutMs(config.runnerIdleTimeoutMs ?? adapterConfig.idleTimeoutMs),
|
|
} };
|
|
if (lifecycleMode !== "warm") return environment;
|
|
return {
|
|
...environment,
|
|
config: {
|
|
...config,
|
|
reuseLease: true,
|
|
},
|
|
};
|
|
}
|
|
|
|
/**
|
|
* The warm runner idle window (ms) a sandbox-provider lease must outlive for
|
|
* this run, or undefined when the run keeps no warm runner on the sandbox. The
|
|
* host forwards it to the provider so a provider idle auto-stop (for example
|
|
* Daytona's `autoStopInterval`) is derived from the same value the host uses
|
|
* for its warm idle timer, instead of an independent default that can match it.
|
|
*/
|
|
export function resolveWarmRunnerIdleTimeoutMsForRun(
|
|
environment: Pick<Environment, "driver" | "config">,
|
|
adapterType: string,
|
|
adapterConfig: Record<string, unknown> = {},
|
|
admittedLifecycleMode?: "warm" | "per_turn",
|
|
defaultContext: RunnerLifecycleDefaultContext = {},
|
|
): number | undefined {
|
|
if (adapterType !== "paperclip_runner" || environment.driver !== "sandbox") return undefined;
|
|
const config = environment.config ?? {};
|
|
if (resolveRunnerLifecycleModeForRun(environment, adapterConfig, admittedLifecycleMode, defaultContext) !== "warm") return undefined;
|
|
return resolvePaperclipRunnerIdleTimeoutMs(config.runnerIdleTimeoutMs ?? adapterConfig.idleTimeoutMs);
|
|
}
|
|
|
|
/**
|
|
* The remote runner reconnect grace (ms) a sandbox-provider lease must outlive
|
|
* for this run, or undefined when the run places no remote runner on a sandbox.
|
|
* A remote runner keeps working while the controller is offline, and the host
|
|
* reattaches it within this grace. The host forwards it to the provider so a
|
|
* provider idle auto-stop (for example Daytona's `autoStopInterval`) cannot stop
|
|
* the sandbox, and kill the runner, during an outage the host still recovers.
|
|
*/
|
|
export function resolveRunnerReconnectGraceMsForRun(
|
|
environment: Pick<Environment, "driver">,
|
|
adapterType: string,
|
|
): number | undefined {
|
|
if (adapterType !== "paperclip_runner" || environment.driver !== "sandbox") return undefined;
|
|
return REMOTE_RUNNER_RECONNECT_GRACE_MS;
|
|
}
|