Jannik/PaperClipAI
1
0 Fork 0
mirror of https://github.com/paperclipai/paperclip.git synced 2026-10-28 12:04:41 +01:00
Code Issues Packages Projects Releases Wiki Activity
Files
3e7c3ee9a97538215454054bcaf3891ecefbd44f
PaperClipAI/maps/feature-map/execution-environments.md
T
DottaandPaperclip 32ec84db2e fix(native): restore plugin checkpoints for same-run retries (#17814) 
## Thinking Path

> - Paperclip manages AI agents and their work.
> - Native sessions retain provider state across execution attempts.
> - A same-run retry can receive a replacement plugin task lease.
> - The executor loaded the old task checkpoint only for crash recovery.
> - An ordinary retry could fail its local authority check before
reading that checkpoint.
> - This change restores the checkpoint through the existing authorized
lease transfer.

## Linked Issues or Issue Description

**What happened?**

A native provider turn ended without its completion result. Paperclip
scheduled an exact-checkpoint retry on the same heartbeat run. The retry
acquired a new plugin task lease, then failed with
`runner_state_identity_mismatch: authority_indeterminate`.

The checkpoint restore branch required a `resume_dead_runner` restart
claim. Ordinary same-run retries have no such claim. A controlled
fixture with a retained remote checkpoint and no local lifecycle receipt
reproduces the error. The original controller file contents were
unavailable, so that part of the incident mechanism remains inferred.

**Expected behavior**

Load the retained plugin checkpoint after the existing policy proves the
lease transfer is safe. Keep the original run, session, and saved
control journal. Reject an unverified predecessor or a changed
checkpoint identity.

**Steps to reproduce**

1. Start a native session through a plugin task environment.
2. End its provider turn without a completion result and retain its
checkpoint.
3. Retry the same run on a replacement task lease without a restart
claim.
4. With no local lifecycle receipt, observe the authority error before
the checkpoint read.

**Paperclip version or commit**

Reproduced against source `507dc70be26c191e372f09e58dc0965e73bf8f57`.

**Deployment mode**

Native Runner with a plugin task environment. The observed provider used
ACPX Claude. The controlled executor fixture uses the shared native
boundary.

Related work: #17439 concerns recovery after controller replacement.
This fix uses the existing checkpoint contract and does not depend on
that PR.

## What Changed

- Restore prior plugin state for an ordinary same-run attempt when its
lease changes.
- Keep the existing company, agent, workspace, retained-state, and
termination checks.
- Extend executor regressions for ordinary retries, mismatched
checkpoints, and missing stop proof.
- Exercise the same-run checkpoint read through the database-backed task
API.
- Document the recovery rule and verification recipes.

## Verification

- Failing before: the ordinary retry reproduces
`authority_indeterminate`. Both new denial cases also fail because the
old branch never reaches the transfer policy.
- Passing after: all six focused checkpoint cases pass. The saved
control journal stays byte-for-byte unchanged. Rejected cases do not
construct a provider backend.
- `pnpm -r typecheck` passes.
- Map and module-boundary checks pass.
- All 858 executor, database lease-authority, plugin task, and ownership
tests pass. `pnpm build` passes.
- The broad local `pnpm test:run` attempt did not complete. It hit a
20-second timeout in the existing historical workspace-recovery HTTP
Retry case. That exact case then passed in isolation on the unchanged
head. The broad local attempt was stopped after the full CI matrix
passed; it is not counted as a local pass.
- Greptile reviewed current head
`5bc11fa978ab87d3d9c5d575fa9666185aaafc2c` at 5/5 with no actionable
findings. The full current-head CI matrix passed on attempt 2: 55 checks
passed, no pending or failed checks, and two expected Storybook jobs
skipped.
- The first broader local attempt could not bind embedded PostgreSQL
inside the sandbox (`EPERM`). It was stopped and restarted with loopback
access. This was a setup failure, separate from the observed red
regression.
- The first CI attempt failed two existing checks: the process-adapter
signoff happy path exhausted its run wait budget, and the workspace
export scheduler did not reach `complete` within 20 seconds. Both tests
pass in isolated local runs on the unchanged head. Both failed shards
passed on their single rerun without code changes. The initial failures
remain recorded. Their timing cause is not proven by the later passes.
- Live acceptance at the fixed revision has not run. The hosted
workspace still serves the earlier release. No deployment or retry of
the original task is claimed.

## Risks

The new checkpoint read runs only for same-run plugin recovery with a
replacement lease. Missing termination proof, changed scope, or a
changed Runner identity still stops execution. The lease resolver owns
these checks. The fix does not replay tools or change retry budgets. A
turn that omitted its completion result still needs the existing bounded
continuation to finish useful work.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository tools, and code
execution. The runtime does not expose a more specific backend model ID
or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-26 18:01:55 -05:00

168 lines
11 KiB
Markdown
Raw Blame History

---
navigation: {"section":"Projects and execution","sectionOrder":40,"order":40}
---
# Execution environments
Operators configure where agents execute, probe readiness, select an environment for work, and inspect remote provisioning or custom-image state when the installed provider supports it.
Feature ID: `execution-environments`.
Implementation: [environment settings](../../ui/src/pages/CompanyEnvironments.tsx), [environment runtime](../../server/src/services/environment-runtime.ts).
Agent access: [companion](../agent-access-map/execution-environments.md).
## Sub-features
- `configuration`: `execution-environments/configuration` — inspect local environments and configure available SSH/sandbox targets.
- `capabilities`: `execution-environments/capabilities` — select only drivers/providers that advertise the required execution capability.
- `probe`: `execution-environments/probe` — test reachability and runtime prerequisites in the actual target.
- `assignment`: `execution-environments/assignment` — bind the intended agent/project/workspace to its execution target.
- `images`: `execution-environments/images` — inspect supported sandbox custom-image preparation and terminal workflows.
## How to get to it (user POV)
### `environment-settings`
Open Settings → Environments, then create/edit a supported target.
The canonical URL is `/company/settings/instance/environments`. Enable
`enableEnvironments` first; keep `enableManagedSandboxOnly` off to inspect local
targets. The catalog is instance-scoped and mutation requires a local operator
or instance administrator.
### `remote-execution`
Choose the environment in agent/project/workspace configuration, then inspect the resulting task/run and workspace.
## Driving it with Playwright and the Paperclip CLI
Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Use an owned disposable target and a supported provider plugin. Record provider, target identity, account, and spending constraints before a live remote run.
### `environment-settings`
Automated: [environment UI](../../ui/src/pages/CompanyEnvironments.test.tsx), [driver configuration UI](../../ui/src/pages/CompanySettings.test.tsx), and [environment service](../../server/src/__tests__/environment-service.test.ts) cover configuration behavior.
Manual: Save a disposable target, reload it, and run its available probe. Verify both success and missing-credential/reachability errors identify the target. Confirm unavailable providers are not offered as working run targets.
For an immutable sandbox image, pin the OCI image with `@sha256`, or select
a Daytona snapshot whose provider ID can be verified, and explicitly probe the saved environment in the intended company. The existing connection
probe allocates a fresh lease and can measure its OS, architecture and libc.
Qualification is recorded only when the selected image/configuration matches
the live lease and cleanup is confirmed. Inspect `platformQualificationRecorded`
and its diagnostic in the probe result: connection success alone does not mean
the image has native runtime qualification.
The measured record is server-owned, company-bound and tied to the effective
configuration and active custom-image identity. Mutable tags, provider defaults
and snapshots without verified provider identities do not qualify. Daytona
qualification creates by provider ID and binds native execution to that same ID;
a deleted ID fails instead of resolving a replacement name. Changing the target,
account, image or active captured template invalidates
the record; normal metadata edits cannot forge or overwrite it, and API reads
omit the private metadata. Reading this evidence for discovery or creation does
not allocate a sandbox. The draft `probe-config` checks are not a saved-image
qualification operation. See the [runtime contract](../../packages/paperclip-runner/README.md)
and [adapter validation recipe](./runs-adapters.md#adapter-setup) for native
binary/provider readiness after platform qualification.
Verification recipe: probe a pinned owned image, record its company/config/image
identity and cleanup result, then change the image or target and verify its old
qualification is not reused. A mutable-image connection can succeed while
qualification remains absent. Assert ordinary agent and non-admin board probe
requests are denied, and environment create/PATCH payloads cannot supply
qualification metadata.
This recipe and its [probe controls](../../server/src/__tests__/environment-probe.test.ts)
are not claims that a live provider or agent-access journey ran for this map update.
The local row has no UI Test button. Its API/CLI probe reports host/cwd
availability only; it does not validate an adapter binary, workspace, or task
execution. Remote-driver probes and a real bounded run need their own evidence.
### `remote-execution`
For `paperclip_runner`, a saved plugin environment can provide per-turn Runner
execution when its manifest and worker expose the Runner task API. The task
contract starts a provider-owned Runner without shell or workspace hooks. Core
resolves the saved lease, connects through PRP, and verifies stop before lease
cleanup. The provider owns the installed agent home and retained session files;
data transfer is outside task execution. See the
[plugin contract](../../doc/plugins/PLUGIN_RUNNER_EXECUTION.md).
The [fake-provider integration](../../server/src/__tests__/plugin-runner-task.integration.test.ts)
checks acquisition, SDK dispatch, scope denial, stop proof and deletion recovery.
Its WebSocket is mocked; it does not qualify a live provider or a real Runner handshake.
For a same-run retry on a replacement plugin task, verify that the retained
checkpoint loads before provider startup. The prior execution must have a stop
receipt, and the checkpoint must match the original session identity. Include a
turn that ends without a completion result; a successful initial turn alone does
not exercise this recovery path. Reject missing stop proof and changed identities.
Automated: [execution target](../../server/src/__tests__/environment-execution-target.test.ts) and [custom images](../../server/src/__tests__/environment-custom-images-service.test.ts) cover contracts; [SSH live test](../../server/src/__tests__/environment-live-ssh.test.ts) requires its own external setup.
The [environment runtime regression](../../server/src/__tests__/environment-runtime.test.ts)
also covers sandbox reuse after a start longer than one minute. Acquisition
uses the configured or provider-declared startup budget for resume and shares
one deadline across retries. Failure preserves the recorded sandbox. This
controlled timing check does not establish live provider acceptance.
Manual: Run a bounded command in the intended target, inspect its output and workspace, and verify execution did not silently fall back to the server host. For custom-image changes, build/probe the disposable image before assigning it. Inspect cleanup and shutdown after the run.
## Gotchas
- A probe is narrower evidence than a complete task with tools, files, and resumption.
- Platform qualification does not authenticate a model account or qualify every
harness; the selected native runtime still needs its own setup and useful-task proof.
- The local option may appear only when editing an existing local environment; availability follows current UI/provider capabilities.
- Workspace service reachability and environment reachability are separate checks.
The task-only setup path uses a Core-issued expiring probe identity and a real,
read-only Runner hello. The plugin checks project authentication before provider startup.
[Setup probe coverage](../../server/src/services/native-runtime/plugin-runner-probe.test.ts)
uses the real Runner and a synthetic Codex provider through loopback PRP; TLS
termination is replaced by the fixture. It does not qualify a live plugin provider.
The plugin prepares execution-host resources and the provider descriptor. Core validates the result, then supplies task identity and PRP launch limits.
### Storybook verification
#### `configuration`
Capability ID: `execution-environments/configuration`.
Scope: ui
Storybook: Local/SSH/sandbox editors; unavailable providers, missing credentials, save pending/error and restricted operator.
Production: Persist target configuration and enforce instance-admin/local-operator mutation boundary; target catalog is instance-scoped.
Sources: Source: [CompanyEnvironments.tsx](../../ui/src/pages/CompanyEnvironments.tsx), [environments.ts](../../server/src/routes/environments.ts). Tests: [CompanyEnvironments.test.tsx](../../ui/src/pages/CompanyEnvironments.test.tsx), [environment-service.test.ts](../../server/src/__tests__/environment-service.test.ts)
#### `capabilities`
Capability ID: `execution-environments/capabilities`.
Scope: ui
Storybook: Supported versus unavailable driver/provider controls and interactive-setup/template-capture limitations.
Production: Verify advertised driver/plugin capabilities match operations; unavailable provider cannot become working target; capability discovery must not allocate sandbox.
Sources: Source: [CompanyEnvironments.tsx](../../ui/src/pages/CompanyEnvironments.tsx), [environments.ts](../../ui/src/api/environments.ts). Tests: [environment-capability-contract.test.ts](../../server/src/__tests__/environment-capability-contract.test.ts), [environment-execution-target-capabilities.test.ts](../../server/src/__tests__/environment-execution-target-capabilities.test.ts), [plugin-environment-driver-sandbox-capabilities.test.ts](../../server/src/__tests__/plugin-environment-driver-sandbox-capabilities.test.ts)
#### `probe`
Capability ID: `execution-environments/probe`.
Scope: ui
Storybook: Remote Test pending, success, reachability/auth failure and platform qualification absent/invalidated notice.
Production: Probe actual owned target; verify selected saved immutable config/image/company qualification plus lease cleanup; local API probe proves host/cwd only and has no UI Test button.
Sources: Source: [CompanyEnvironments.tsx](../../ui/src/pages/CompanyEnvironments.tsx), [environment-probe.ts](../../server/src/services/environment-probe.ts). Tests: [CompanyEnvironments.test.tsx](../../ui/src/pages/CompanyEnvironments.test.tsx), [environment-probe.test.ts](../../server/src/__tests__/environment-probe.test.ts), [environment-live-ssh.test.ts](../../server/src/__tests__/environment-live-ssh.test.ts)
#### `images`
Capability ID: `execution-environments/images`.
Scope: ui
Storybook: Custom-image setup/capture/progress, terminal connect/disconnect/failure, unsupported capability and active-image drift.
Production: Execute owned provider setup/capture/probe/terminal transport with correct saved image identity, authority, cleanup and shutdown; terminal screenshot cannot prove operation.
Sources: Source: [CompanyEnvironments.tsx](../../ui/src/pages/CompanyEnvironments.tsx), [environment-custom-images.ts](../../server/src/services/environment-custom-images.ts). Tests: [CompanyEnvironments.test.tsx](../../ui/src/pages/CompanyEnvironments.test.tsx), [environment-custom-images-service.test.ts](../../server/src/__tests__/environment-custom-images-service.test.ts)
Managed account transfer requires `projectAuthCheckVersion: 1` and a successful
`check_project_auth` result for the exact prepared configuration, provider, and
project set. New preparation removes old validation. Core checks the saved
validation again in the dispatch update.
Reference in new issue View Git Blame Copy Permalink