Commit Graph
3 Commits
Author SHA1 Message Date
Nicky LeachandClaude Opus 5.5 afe86e9ded fix(runtime): record a plugin task's stop receipt when its stop is proven (#21237)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A plugin environment runs each Paperclip Runner turn as a provider
task, and a follow-up run can continue the same Claude/ACPX conversation
from the saved checkpoint
> - Core authorizes that continuation only when the predecessor run's
remote execution has stopped (`remoteExecutionHasStopped`), so two
Runners never write one checkpoint
> - The predecessor's own settlement already proves the stop: the
provider reports `executionStopped`, and Core then completes the task
> - On a per-turn (ephemeral) lease, Core records the stop receipt only
when it releases the lease, and that release can be held for a long time
(for example by a pending workspace export)
> - In that window, an ordinary follow-up fails with `adapter_failed:
Plugin Runner retained session is not authorized`, even though the stop
is already proven
> - This pull request records the stop receipt when the stop is proven,
and the stop check accepts it before the release
> - The benefit is that follow-ups continue as soon as the predecessor
is proven stopped, and the release gates keep their meaning

## Linked Issues or Issue Description

Fixes #21211

Refs #21214 and #21223. Those cover the creation fence for same-run
recovery, which still requires a released lease.

Production evidence for this change: in both observed failures, the
predecessor was a per-turn plugin run. Its `task.settle` completed with
all provider calls successful. Core then did not release its lease for
30 minutes or more, and no release call was made in that time. The
follow-up failed in checkpoint recovery (`recoverPluginRunnerCheckpoint`
→ `resolvePluginTaskResume`) within seconds of admission.

## What Changed

- `server/src/services/environment-task-runtime.ts`
- After a recorded provider stop, `complete` writes the bound stop
receipt (`remoteTerminationReceipt(..., { state: "stopped" })`) on the
current lease for every lease policy. Before, it did this only for
`reuse_by_environment` leases. Reusable tasks still confirm every owner
and still move retained owners to released.
- New work on a lease (`submit`, `copilot_metadata`, first portable
resume) removes the stop receipt in the same write that clears
`runnerTaskExecutionStopped`. A replacement physical task does not copy
the old receipt.
- `server/src/services/remote-execution-termination.ts`
- New `hasBoundRemoteStopReceipt` checks only the receipt binding
(company, run, lease, provider and physical task).
`hasRemoteTerminationReceipt` uses it and keeps its released-only
semantics. Resource cleanup gates do not change.
- `remoteExecutionHasStopped` and `stoppedRemoteCleanupScopes` accept a
lease that is released with its receipt, or that is still `active` (not
released) with a bound stop receipt. A lease whose release started and
did not settle does not count. After release starts, the release owns
the receipt.
  - New `remoteExecutionHasReleased` gives the release-based answer.
- `server/src/services/environment-run-orchestrator.ts`: same-run plugin
recovery uses `remoteExecutionHasReleased`. The same-run resume check
after it requires released receipts, so its behavior does not change.
- `resolvePluginTaskResume` keeps its checks and their order. Its
refusal now names the first failing check: `Plugin Runner retained
session is not authorized (reason: <code>)`, for example
`predecessor_not_stopped`, `scope_mismatch` or `identity_mismatch`. The
old message stays as the prefix, and no ids go into it. The
stale-availability recheck in `environment-task-runtime.ts` names its
refusal the same way. Production telemetry for #21211 could not tell
which check fired. The next occurrence will show it.
- `doc/execution-semantics.md` and
`doc/testing-native-remote-continuity.md` describe the stop receipt and
when it is recorded.

Caller audit for the changed meaning of `remoteExecutionHasStopped`:

| Caller | Needs | Decision |
| --- | --- | --- |
| `agent-instruction-working-copies.ts` | stop | Accepts. Directory
cleanup still requires a released "destroyed" receipt. |
| `environment-run-orchestrator.ts` | release | Moved to
`remoteExecutionHasReleased`. |
| `plugin-task-resume.ts` | stop | Accepts. This is the fix. |
| `native-runtime/native-runner-ownership.ts` | stop | Accepts.
`hasPendingRunCleanup` still counts the active lease as pending. |
| `native-runtime/preserved-native-result-finalization.ts` | stop |
Accepts. Workspace recovery has its own gate. |
| `heartbeat/case-handoff-lifecycle.ts` | stop | Accepts.
`hasPendingRunCleanup` still holds active leases. |
| `heartbeat/run-control.ts` (3 sites) | stop | Accepts. Stop
acknowledgement and quarantined ownership retirement. Holds and blockers
still apply on the normal path. |
| `heartbeat/runtime-execution.ts` | stop | Accepts. Release errors
still fail it. |

## Verification

From `server/`, after `node scripts/agent-preflight.mjs`:

- `npx vitest run
src/__tests__/environment-task-authority.integration.test.ts`: 10/10
pass. The 2 new tests fail without the fix.
- The plugin task and environment runtime suites
(`environment-task-authority.integration`,
`plugin-runner-task.integration`, `plugin-warm-tasks.integration`,
`environment-task-runtime`, `plugin-task-source-recovery.integration`,
`environment-runtime`): 6 files, 366 pass.
- The audited caller suites (`agent-instruction-working-copies`, case
handoff legacy and native, `environment-run-orchestrator`,
`heartbeat-case-handoff`, `explicit-native-continuation`, `run-control`,
`native-remote-retry-admission`, `preserved-native-response-wait`): 9
files, 806 pass.
- `pnpm --filter @paperclipai/server typecheck`: pass.

The new integration tests cover:

- `complete` writes a bound stop receipt on a still-active ephemeral
lease. `remoteExecutionHasStopped` is true and
`hasRemoteTerminationReceipt` is still false.
- `resolvePluginTaskResume` (inspect and launch) authorizes a
follow-up's new task, and the real host `read_state` checkpoint read
succeeds, while the predecessor's release is held.
- Refusals: no provider stop, a receipt for another provider task, an
unstopped sibling lease, and a release in progress.
- New work on the lease removes the receipt. Release overwrites the
receipt with its own receipt. Warm owners keep their behavior.

I did not add an executor-level test. `native-session-executor.test.ts`
mocks `resolvePluginTaskResume`, so the integration test runs the real
checkpoint read path instead. I did not run the full suite locally. CI
runs it.

## Risks

- `remoteExecutionHasStopped` now returns true earlier for a run whose
leases are still held. Each caller is audited above. Callers that need
released resources use the release-based predicates.
- The stop receipt can now be on an `active` lease. The SQL readers of
`remoteExecutionTermination` (cost accounting, execution blockers,
idle-work proof, orphan settlement) also require a released status.
`pinFailedObservedWorkspace` treats an active ephemeral lease with a
receipt as not pinnable. That path applies only to the remote-workspace
flow and falls back to `preserveNativeFailedWorkspace`.
- Leases that settled before this change have no early receipt. They
continue to wait for their release.
- Nothing compares the refusal message exactly. Matchers use substrings,
and run failure classification does not use the message.
- A follow-up still fails closed if the predecessor's stop was never
proven. This change does not add a recoverable wait.
- The reason the release was held in production is not addressed here.
- No migration and no API change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, sub-agents, Honeycomb trace queries) and extended
thinking. One sub-agent wrote the change. A second session reviewed it
before this pull request.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 20:20:45 -08:00
Nicky LeachandPaperclip a924d2e2c7 Persist portable Runner ownership through controller recovery (#21066)
## Thinking Path

> - Paperclip delegates remote Runner execution to environment plugins.
> - A plugin can move a session to another task allocation.
> - The Core controller can lose an ownership reply or restart before
the Runner starts.
> - The source and target need a durable record of the same request.
> - This change binds restart and attachment to that request and the
original unlaunched task lease.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Recovery of plugin Runner sessions on replacement compute.

**Current behavior**
Core cannot replay a session ownership request from saved lease state
during startup.

**Proposed behavior**
Record the operation on both leases and check the matched ownership
receipt before attaching. Preserve the original pre-admission lease
across controller restart.

Managed environment configuration is in
[#21065](https://github.com/paperclipai/paperclip/pull/21065). Related
warm sandbox cleanup is in
[#20065](https://github.com/paperclipai/paperclip/pull/20065). This
layer preserves the same-sandbox fallback from
[#21001](https://github.com/paperclipai/paperclip/pull/21001). Workspace
placement follows in
[#20743](https://github.com/paperclipai/paperclip/pull/20743).

## What Changed

- Add the optional `sessionHandoffVersion` capability and the ownership
task operation.
- Save matching ownership claims and fence source checkpoint access.
Replay a pending request on the original target.
- Restore a portable session using the durable claim and the current
credentials for the same account. Start a fresh conversation when the
provider reports an unavailable checkpoint, or when Core's persisted
compatibility decision selects a different session after transfer. The
latter requires an unlaunched target; subsequent recovery must match the
fresh task's saved identity. Both paths retain the workspace transfer
and source fencing.
- Reuse the original unlaunched task lease after hard or graceful
controller restart. After a move has attached, validate the surviving
Runner through the saved claim rather than requiring its logical lease
ID to equal the new physical lease.
- Keep older retained leases without a saved account identity on the
existing resume path.
- Apply the existing supported MCP-only compatibility rule to warm
recovery and predecessor accounting. Persist the capability discovered
at admission and reuse it after restart without repeating discovery.
- Keep predecessor-retirement fallback scoped to the same physical
sandbox. A refusal after a portable move preserves the session instead
of requesting a replacement that the plugin driver cannot enforce.

## Verification

- On `1eb527be27`, server typecheck (including Runner build and SDK
dependencies) passed.
- Independent final verification passed 168 tests across portable source
recovery, session resume, controller recovery, and task authority; two
existing tests were skipped. The full portable source integration suite
passed 64 tests.
- The new integration follows confirmed ownership through an
incompatible instruction change, persisted fresh admission, target-only
prepare/auth/submit without the source checkpoint, and reattachment
after controller replacement. It also rejects unpersisted session
selection, ambiguous target leases, and incorrect saved identities.
- Before this final change, independent verification passed 1,667 tests
across executor, session resume, recovery usage, remote warm recovery,
runtime selection, and runtime execution; the environment lifecycle
suite passed 208 tests.
- Existing regressions cover previous-release account metadata,
supported and unsupported MCP changes, provider/instruction
incompatibility, and admission capability reuse without another
discovery call.
- On `51d7eec373`, all four S3 multipart throttle tests passed
independently. CI exposed an existing timing-sensitive test twice; its
20 ms overlap assumption now uses synchronized retries and an event-loop
yield. Production upload code is unchanged. Temporary concurrency caps
of one and eight both fail the retained exact-four assertion.
- Exact SDK source-pin checks passed for both stack heads. The upper
#20743 code patch remains unchanged; its documentation preserves both
layers' behavior.

## Risks

- The handoff protocol is opt-in. Existing plugin leases without
portable ownership metadata keep the previous release's resume path.
Live attachment still requires end-to-end qualification. An unverified
claim blocks startup. Git workspaces prepared on a portable allocation
also need the workspace placement layer.

## Model Used

- OpenAI Codex GPT-6 with GPT-6.1 Sol agents for implementation and
independent review.

## Checklist

- [x] Thinking path and model are included.
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] The issue is described using the enhancement template.
- [x] Local checks passed and new behavior has tests.
- [x] All Paperclip CI gates are green.
- [x] Greptile is 5/5 with no open findings.

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-12 10:43:48 -08:00
Nicky LeachandPaperclip 2506bcdaf4 Admit exact stopped Runner sources under the current run lease (#19134)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native execution keeps provider state and a durable controller
journal.
> - A completed process can stop before the next run attaches.
> - The current run must own recovery of that exact source.
> - An old controller must lose permission to change the consumed source
journal.
> - This pull request adds explicit source admission and protected task
dispatch.
> - The benefit is a narrow recovery boundary that preserves the
original successful result.

## Linked Issues or Issue Description

Native prerequisites #19132 and #19153 are merged into master. This PR
targets master and updates the TypeScript Runner driver contract
alongside Core and SDK admission. This change adds the SDK and Core
admission boundary. Executor selection and fresh source preparation are
a separate change. Related #17916 supplies the workspace export holds
preserved here.

## What Changed

- Plugins advertise `codexSourceRecoveryVersion: 1` for Codex App Server
and `acpxSourceRecoveryVersion: 1` for ACPX. Hosts can qualify either,
both, or neither. Core selects the matching field, because a
stopped-source check for one provider cannot establish the other
provider’s recovery guarantees. ACPX orchestration is tracked separately
in #19196. Use the existing exact source identity and `resumeFrom` under
the authenticated current run. No public recovery mode is added. Require
the claimed source, current account and prepared launch at protected
dispatch.
- Keep the first source binding pinned on the consumer lease, including
after uncertain dispatch. A newly acquired coordinator can refresh only
the same source.
- Claim one stopped successful source under the current run's acquired
native lease. Store the token-free consumer reference in existing lease
metadata. Preserve the source status and result.
- Check the exact six source identity fields, encrypted journal header,
company, task, agent, environment, driver, workspace, account selection,
and current coordinator generation. Keep workspace export and legacy
settlement holds.
- Serialize claims and checkpoint writers in the same coordinator lock
order. Reject stale source writes after consumption. Check the current
consumer before recovered source writes.
- Check source admission before and after protected worker dispatch.
Keep the request and lease owned by the current run while the native
identity remains the source.
- Allow the same owner to prepare B after a fresh source submission,
using the exact consumed-source proof instead of a generic warm-reuse
flag. Recheck that proof after dispatch.
- Preserve ordinary cached adoption and repeated preparation after
launch. The current authenticated owner can differ from the adopted PRP
identity without requesting a new source launch. Keep ordinary suspended
recovery and live task attachment checks. Extract the shared source
resolver to avoid a module cycle, and make ordinary selection honor the
same durable source consumer.

## Verification

- Current head `cac9cf188a` includes merged #19300 and the symmetric
capabilities introduced in `b7e25a64b4`. The task-owner and warm-rebind
fixtures now identify their physical task. All 29 SDK schema cases, 95
Core ownership and source-admission cases, 25 native Codex and ACPX
recovery cases, and the binary metadata check passed on this head. The
SDK and Runner builds and server typecheck passed. Both native providers
reject the other provider’s capability. GitHub CI is green and [standard
Greptile scored this commit
5/5](https://github.com/paperclipai/paperclip/pull/19134#issuecomment-6645574114)
with no open threads.

- Previous head `bda49271a4ae362a15baf3fe6d6d297e25fd2000` is directly
based on master after #19132 and #19153 landed. [GitHub workflow
42072192023](https://github.com/paperclipai/paperclip/actions/runs/42072192023)
passed on its first attempt. All check runs completed successfully or as
intentional workflow skips; Snyk passed. Standard Greptile scored this
exact head 5/5. That review had no unresolved findings.
- Runner and SDK TypeScript builds and server typecheck passed. The
focused SDK, task-owner, task-runtime, PostgreSQL source and authority
suites passed all 161 cases. The production transport-factory getter
regression and 14 native-lane selection checks also passed.
- The Linux native lane executed the new ordinary warm A→B process case.
It kept one Runner/provider process and completed two turns on the same
thread. The eager-getter mutation fails with
`native_runner_source_attachment_pending`; preserving the live getter
passes. The production factory test changes the current owner after
transport creation and checks that same options object.
- PostgreSQL cases cover both concurrent source-writer/claim orders,
exact source submission under the current caller, source-switch denial
after uncertain dispatch, same-source coordinator refresh, stale
same-boot generation, corrupted journal headers, competing consumers,
account changes and export holds.
- The protected dispatch sequence submits A, prepares B and retries B
preparation. It rejects undispatched sources, changed accounts and
foreign target bindings. Source status and accepted result remain
unchanged.
- Ordinary adopted A preparation under current B, repeated ordinary
preparation after launch and ordinary suspended continuation retain
their existing path. Absent or false claims preserve ordinary identity
denial; malformed claims fail closed.
- Earlier development attempts exposed ordinary preparation regressions
and the live-getter omission; the final regressions cover those fixes.
The first embedded PostgreSQL attempt needed missing library aliases
repaired in ignored dependencies. Those attempts are not passing
evidence.
- Historical, superseded CI heads had unchanged UI fake-timer and
Playwright health-response-lifetime failures. Their isolated cases and
scoped retries passed. The response-lifetime cause remains unproven. No
UI/test timeout was changed. The final rebased-head workflow above
passed without a retry.
- Whitespace and added-line secret/PII scans passed. Full workspace
checks ran on GitHub. No live vendor probe or deployment was performed.

## Risks

- Any driver that used the earlier Codex TypeScript option must use
`codexSourceRecoveryVersion`. The new plugin description rejects the
generic spelling. The Launcher’s existing Codex-specific HTTP header can
map to this property when the plugin adopts the new SDK. The change
requires no journal migration or ROM change.

- This is an inactive admission foundation. Executor orchestration is
separate in #19157. Native restore integrity and provider
stopped-source/account validation must be present before executor
activation. The native prerequisite uses ordinary recovery followed by
attachment. The composed executor proofs are in #19157.
- The source claim is durable. A later controller must recover the same
consumer or keep the hold. It cannot select another source after an
ambiguous launch.
- Admission adds bounded database checks around protected startup calls.
Busy issue locks fail closed; other lock waits have a one-second bound.
- No migration, snapshot format, proactive receipt, recurring worker, or
deployment is included. Full workspace checks and platform coverage run
in CI. These tests do not prove vendor history durability or a hosted
rollout.

## Model Used

OpenAI GPT-6.1 Sol (`gpt-6.1-sol`), high reasoning, through Codex with
repository tools and code execution. The runtime does not expose a
context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green on the current head
- [x] Standard Greptile is 5/5 with no unresolved findings on the
current head
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-05 12:55:06 -08:00