Commit Graph
3 Commits
Author SHA1 Message Date
Dotta 0c1e7504c0 fix(runner): persist warm Daytona workspaces across turns (#12904)
## Thinking Path

> - Daytona preserves a stopped sandbox filesystem, but deleting or
replacing a sandbox removes its only remote copy.
> - Warm reuse therefore improves latency but cannot be Paperclip's
durability boundary.
> - The host execution workspace must remain authoritative after every
successful turn, while same-run recovery must avoid overwriting
unexported remote work.
> - Result proposal, workspace export/merge, and terminal completion
need a durable, replayable ordering so a crash never starts a duplicate
provider turn.
> - A paid browser acceptance suite must exercise both legacy Codex and
Runner Codex for three real turns on one continuously warm Daytona
sandbox.

## Linked Issues or Issue Description

Refs #12901.

Runner Codex did not previously export successful Daytona workspace
changes back to the authoritative host workspace. That made warm reuse
depend on Daytona's remote filesystem and left deleted/replacement
sandboxes without a reliable reconstruction path. The existing paid
fixture also lacked a focused three-turn continuity case for both Codex
adapters.

## What Changed

- Persist versioned, atomic native workspace-sync descriptors and
durable seeds in `PAPERCLIP_HOME`, without credentials or a database
migration.
- Classify fresh, warm, replacement, and same-run-recovery workspace
preparation explicitly; ambiguous lease/root/digest evidence fails
closed.
- Finalize native workspace export/merge after semantic result proposal
and before run completion, with idempotent replay that never submits a
second provider turn.
- Surface legacy Codex workspace restoration failures instead of masking
them, while preserving an earlier provider error when both fail.
- Keep healthy reusable Daytona leases warm for legacy and native
adapters, stamp finalized workspace generations, and retain existing
cleanup behavior for per-turn or unhealthy leases.
- Preserve Runner Codex's provider process/session across warm turns,
including bounded post-terminal tail draining and exact authority
rotation.
- Add the exact paid `daytona-warm-continuity` matrix:
  - `legacy-codex × daytona × warm-three-turn`
  - `runner-codex × daytona × warm-three-turn`
- Drive all three turns through the browser, verify ordered file
continuity and stable lease/workspace/runtime identities, capture
per-turn timings, and delete the sandbox immediately after assertions.
- Document `pnpm test:e2e:runner -- --suite daytona-warm-continuity`; no
package script was added.

## Verification

- `pnpm typecheck` — passed, including migration safety (no migration
added)
- Focused server/runner Vitest coverage — 144 passed
- `pnpm test:e2e:runner:unit` — 114 passed
- `pnpm test:e2e:runner:typecheck` — passed
- `pnpm --filter @paperclipai/paperclip-runner test:codex` — 66 passed,
1 helper ignored
- `native-session-executor.test.ts` — 139 passed, including safe
fail-closed cleanup after remote runner identity capture failure
- Paid local browser acceptance, exact post-rebase Linux/amd64 runner
binary:
- Runner Codex — passed in 1.7m; 3 runs; lease outcomes `created,
resumed, resumed`; 10/10 matchers; cleanup passed
- Legacy Codex — passed in 2.7m; 3 runs; lease outcomes `created,
resumed, resumed`; 10/10 matchers; cleanup passed
- [Protected paid GitHub Actions
campaign](https://github.com/paperclipai/paperclip/actions/runs/34026735033)
against `7da42a91b95fa7fb2df126668ef7e37afb3b2b9d` — passed 2/2:
- Runner Codex — 3 runs; lease outcomes `created, resumed, resumed`;
evidence and cleanup passed
- Legacy Codex — 3 runs; lease outcomes `created, resumed, resumed`;
evidence and cleanup passed
  - Merge/enforcement, S3 history, and Pages publication jobs passed
- Paid result artifacts were scanned for both provider credentials;
neither secret was present.
- Current PR checks — 31 passed, 1 expected Storybook skip; Greptile
5/5; Superagent security scan passed
- `git diff --check origin/master...HEAD` — passed
- Confirmed no `package.json`, lockfile, migration, or SQL changes.

## Risks

- Workspace synchronization now sits on the terminal-success path, so a
remote export failure deliberately prevents false success. Retryable
state retains its lease/seed; loss of the only unexported remote copy
fails closed.
- Warm provider reuse has strict identity and quiescence checks.
Mismatched or ambiguous evidence blocks reuse rather than risking
concurrent provider work.
- The paid suite incurs Daytona and Codex cost only in the existing
protected scheduled/manual workflow and explicitly destroys its sandbox
after each cell.

## Model Used

OpenAI Codex with GPT-5 agentic reasoning, repository inspection, real
browser E2E execution, Rust/TypeScript test execution, and GitHub
Actions diagnostics.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either linked an existing issue or described the issue
in-PR
- [x] I have not referenced internal/instance-local Paperclip issues or
links
- [x] My branch name contains no internal ticket id
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have documented the dedicated suite invocation without adding a
package script
- [x] I have considered and documented risks above
- [x] All Paperclip CI gates are green on the current revision
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
on the current revision
- [x] I will address all reviewer comments before requesting merge
2026-09-06 08:16:12 -05:00
Dotta 1dceee9a4e fix(runner): persist warm Daytona workspaces (#12901)
## Thinking Path

> - Paperclip manages AI agent work and the execution state for each
task.
> - Remote agents run in sandbox environments such as Daytona.
> - Daytona keeps files while a sandbox is stopped, but deletion removes
those files.
> - Runner Codex did not copy successful remote workspace changes back
to the host workspace.
> - A warm sandbox could therefore hide data loss until Daytona replaced
or deleted the sandbox.
> - This pull request makes the host workspace durable after every
successful turn and keeps verified reusable sandboxes warm.
> - The benefit is reliable multi-turn work across warm reuse, restart,
stop, and sandbox replacement.

## Linked Issues or Issue Description

**What happened?**

A successful native Codex turn in Daytona could leave workspace changes
only in the remote sandbox. A later warm turn appeared to work because
it reused that filesystem. A replacement sandbox could start from stale
host data and lose the successful changes.

**Expected behavior**

Paperclip must merge each successful remote turn into the authoritative
host workspace before it completes the run. A verified warm lease may
reuse its remote files. A replacement lease must reconstruct the exact
durable workspace seed.

**Steps to reproduce**

1. Run Codex in a reusable Daytona environment.
2. Write a file during one successful turn.
3. Replace the Daytona sandbox before the next turn.
4. Observe that the next turn can start without the prior file on the
unpatched code.

Related remote workspace foundation: #10070.

## What Changed

- Added explicit `host_current`, `durable_seed`, and `adopt_remote`
workspace preparation modes.
- Added atomic, versioned native workspace descriptors and seed archives
under `PAPERCLIP_HOME`.
- Added real native sandbox export and three-way host merge before
terminal result completion.
- Added workspace-only recovery after a proposed result. Recovery does
not submit another provider turn or consume the provider retry budget.
- Added fail-closed handling when a sandbox with unexported changes is
gone.
- Kept healthy reusable Daytona sandboxes started for legacy Codex and
Runner Codex.
- Kept the Runner Codex process and provider session across verified
warm turns.
- Added the paid `daytona-warm-continuity` browser suite. It contains
exactly the legacy Codex and Runner Codex cells. Each cell performs
three measured turns.
- Documented `pnpm test:e2e:runner -- --suite daytona-warm-continuity`.
No package script was added.
- Added no database migration. The metadata format is backward
compatible and idempotent.

## Verification

- `pnpm typecheck`
- `pnpm test:e2e:runner:unit` — 114 passed
- Native workspace, finalizer, session, and environment tests — 232
passed
- Daytona provider tests — 150 passed
- Workspace staging and merge tests — 98 passed
- Runner transport tests — 63 passed
- Legacy Codex restore tests — 5 passed
- Rust format and compile checks pass through root typecheck
- The paid Daytona suite was not run locally because the required
Daytona, OpenAI, and immutable image credentials are not present.

## Risks

- The main risk is an incorrect workspace identity or merge after a
crash. Durable descriptors bind the run, workspace, lease, provider
lease, local root, remote root, and baseline digest. Ambiguous evidence
fails closed.
- The host merge may conflict with concurrent host edits. The existing
three-way merge and exclusion rules handle this case and surface
failures.
- A deleted sandbox cannot recover unexported bytes. Paperclip now
blocks with `workspace_sync_out_unrecoverable` instead of reporting
success or rerunning the provider.
- There is no database migration. Descriptor writes and recovery are
atomic and idempotent.

## Model Used

OpenAI Codex with GPT-5. The run used agentic reasoning, repository
inspection, code execution, test execution, Git, and GitHub CLI tools.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-09-05 13:00:57 -05:00
Dotta 0a422fda52 feat(runner): add remote execution substrate (#12638)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Paperclip Runner gives native runs a durable and governed execution
path.
> - The current native path runs on the control-plane host.
> - Remote environments need an authenticated execution-target contract.
> - The contract must not change direct adapters or enable new runtimes
by default.
> - This pull request adds the remote execution substrate and Daytona
ingress.
> - The benefit is a bounded base for later remote runner transport
work.

## Linked Issues or Issue Description

Refs #12616.
Refs #12352.

**Subsystem affected**

Cross-cutting. This change touches runner transport, server
orchestration, plugin contracts, and shared settings.

**Problem or motivation**

Native execution cannot resolve an authenticated runner ingress through
a remote environment. The server also lacks one provider-neutral
contract for remote execution targets.

**Proposed solution**

Add a default-off runner preview ingress capability. Add
transport-neutral runner connectivity. Add remote execution target and
lifecycle handling. Add a Daytona ingress implementation with redacted
credentials.

**Alternatives considered**

A provider-specific server path would duplicate orchestration and
authorization. A public endpoint without an environment contract would
weaken the trust boundary.

**Roadmap alignment**

This work supports the Cloud and Sandbox agents milestone. It also
supports self-healing runs and governed tool access.

## What Changed

- Added execution-target traits for local, SSH, and sandbox
environments.
- Added plugin RPC contracts for runner ingress endpoints.
- Added authenticated Daytona preview ingress.
- Added transport-neutral PRP outbound connections.
- Added remote runner artifact verification and fail-closed provider
selection.
- Added bounded native session resume, cancellation, and lifecycle
recovery.
- Preserved Codex-only selection for fresh experimental runner starts.
- Preserved all direct adapter execution and finalization paths.
- Removed stale Pi provider-pack requirements that security review
rejected.
- Kept the rollout controls off by default.
- Did not change pnpm-lock.yaml, Cargo, database migrations, or GitHub
workflows.

## Verification

- GitHub Actions will run the repository test, typecheck, build,
security, and policy gates.
- Focused tests cover ingress validation, redaction, execution targets,
remote lifecycle, cancellation, resume, and legacy adapter selection.
- Local tests were not run. The requested verification policy uses
GitHub Actions for this series.
- `git diff --check origin/master...HEAD` passes.
- The diff contains 52 files.

## Risks

- Remote execution crosses a trust boundary.
- The implementation validates target capabilities, artifact digests,
provider-pack pins, and connection metadata.
- The feature remains default-off.
- Fresh native selection remains Codex-only.
- Existing direct adapters remain on the legacy path.
- This PR does not yet make remote Codex runnable. The next PR adds the
Rust WSS and TLS transport.

## Model Used

OpenAI Codex with GPT-5.6. The work used high-reasoning agent mode,
repository tools, GitHub tools, and parallel code-audit agents.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with Fixes: / Closes /
Refs OR (b) described the issue in-PR following the relevant issue
template
- [x] I have not referenced internal or instance-local Paperclip issues
or links
- [x] My branch name describes the change and contains no internal
Paperclip ticket id
- [ ] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-09-01 01:29:06 -05:00