mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-13 15:55:46 +01:00
dfd2a6efcab84bc19106a2dbf720eb53cb2282d3
26
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandClaude Opus 5.5
|
c45f54ee74 |
fix(native): stop or release unclosable idle warm Runners on environment delete, and say why a close failed (#20957)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs keep a warm Runner in a Daytona sandbox between turns of
a task, so the next turn starts fast
> - Each idle warm Runner has an in-memory owner in the controller.
Deleting an environment closes these owners before it destroys the
sandboxes
> - In every warm chaos wave today, some owners never closed. The server
logged only `{"failed":2}`. Environment delete returned 409 forever, and
all sandboxes of the wave stayed `started` until their 60-minute TTL
> - On staging, that is leaked sandboxes (cost) and owners that block
the work after them
> - This pull request logs and reports the exact close error, stops an
unsettled idle Runner when its environment is deleted, and releases an
owner whose sandbox is already destroyed
> - The benefit is that environment delete finishes or names the exact
run and error, and every warm chaos wave now grades a clean shutdown
## Linked Issues or Issue Description
**What happened?**
After warm Daytona chaos waves (W2, W2Q, W2C cells, `--faults K0`) on
Codex, Claude and OpenCode, the server logged `idle native session
cleanup failed before controller shutdown {"idleSessions":{"failed":2}}`
with no reason. `DELETE
/api/environments/:id?destroyReusableSandboxLeases=true` returned 409
"warm native runner did not shut down cleanly" on every retry. Eight
sandboxes per wave stayed `started`.
**Root cause.**
`closeIdleWarmNativeSessions`
(`server/src/services/native-runtime/native-session-executor.ts`)
dropped the close error (`catch {}`) and counted it. The owners that
failed are the W2Q owners. A W2Q task parks a question and is answered
by a later run on a fresh sandbox. With the reason logged, a wave shows
two causes for the same owner:
1. The owner's close cannot settle (`native_session_close_unrecoverable:
runner did not durably suspend before checkpoint`, the governed-wait
shape from #20844). Only a successor reset could force a stop
(`discardUnsettledSession`). An idle sweep or environment delete had no
forced stop.
2. The owner's sandbox lease was already destroyed by the
pending-cleanup sweep after the successor moved to a fresh sandbox
(#20863). Every close then fails with `Daytona sandbox lease <id> is no
longer active.`, because the stop RPC has no sandbox to reach. The owner
stays registered for ever.
The Runner close memoizes its rejection, so every later attempt fails at
once with the same error. This is not the "controller sequence must be
contiguous" exit (#20946): no close failure in these waves names a
sequence mismatch.
**Expected behavior**
An idle warm owner with no turn in flight never blocks environment
delete for ever. Every failed close names its run and its error.
**Steps to reproduce**
1. Run `tests/restart-chaos/main.ts --env daytona --daytona-warm
--adapters claude --workloads W2,W2Q,W2C --faults K0 --concurrency 8
--tasks 8`.
2. Read `<wave>/sandbox-sweep.json`: `environmentDeleteBlocked` is set
and the sandboxes are `started`.
## What Changed
- `closeIdleWarmNativeSessions` now returns `failures: [{ runId,
issueId, environmentId, providerLeaseId, reason }]`. The reason is the
close error with its code, its settlement record and its cause. It also
logs `idle warm native sessions did not close` with that list. The
idle-timeout close logs its error too.
- `closeWarmNativeSessionsForEnvironment` takes `forceStopUnsettled` and
`db`. The environment delete route sets both. When the graceful close
fails, Core retires the owner as #20844 does: one confirmed Launcher
`stop`, the credential snapshot, `complete`, then instruction
collection. A failed forced stop is added to the reason.
- When retirement still fails and the owner's sandbox lease has a
`destroyed` termination receipt, Core releases the owner. It does not
contact the sandbox. It logs `warm native session's sandbox was already
destroyed; released its owner` with the close error.
- The environment delete 409 now names each run and its reasons, and
`details.nativeRunnerSessions.failures` lists them.
- Idle timeout, controller restart and tenant sleep do not force a stop.
Their conversation can still be resumed, so they only log the reason.
- Chaos harness: new check `warm-shutdown.ts`, "no leaked sandboxes /
clean warm shutdown". Every warm wave (`--daytona-warm` or `--lifecycle
warm`) runs it. It fails the wave on a blocked delete, on a sandbox left
running after cleanup, or on a warm session that did not close at
environment delete or controller shutdown. It writes
`<wave>/warm-shutdown.json` and a section in `SUMMARY.md`.
- Docs: `doc/plugins/PLUGIN_RUNNER_EXECUTION.md` and the chaos README.
## Verification
- RED on master: `native-session-executor.test.ts -t "stops an idle warm
owner whose close cannot settle"` fails: `expected { closed: 0, busy: 0,
failed: 1 } to match object { closed: 0, failed: 1, failures: [...] }`
(a bare count, no reason). `environment-routes.test.ts -t "names the
warm runner session"` fails: the 409 does not name the run. On the first
fix commit, `-t "sandbox lease is gone"` fails: the lease state is never
read, and the owner stays.
- GREEN: those tests pass. The executor suite, environment routes, run
control, recovery, heartbeat process recovery, instance settings, the
idle-sleep, eviction, release-resume, restart-recovery and
warm-checkpoint integration suites pass (11 files, 2471 tests).
`tests/restart-chaos` unit test `warm-shutdown` passes. Server and
harness `tsc --noEmit` pass.
- Daytona warm waves (Claude, W2/W2Q/W2C, K0, 8 tasks). The runner image
`sha256:e04d9ce4…` matches
|
||
|
Nicky LeachandPaperclip
|
5dfbb04b85 |
Poll asynchronous workspace provider preparation (#20865)
## Thinking Path > - Paperclip starts agent runs through a workspace provider when a project uses remote storage. > - The provider must prepare the workspace before the agent starts. > - One long worker call ties the checkout to its transport. > - Some checkouts take longer than the transport allows. > - This change lets an opted-in provider accept work and report its deadline. > - Core waits for completion through short status calls and stops at the provider deadline. ## Linked Issues or Issue Description **What happened?** A workspace provider can lose its reply when preparation takes longer than a single transport request. This stops run bootstrap before agent execution. **Expected behavior** Core should wait for a provider-confirmed workspace through short requests. The provider's deadline must bound the wait. **Steps to reproduce** Use a workspace provider that takes longer to prepare a checkout than its transport can remain open. Start a run that requires a new checkout. ## What Changed - Add an optional workspace preparation method to the plugin SDK. - Validate pending receipts and keep the original binding checks on completed replies. - Poll only during active preparation. Retry a lost status reply within the advertised deadline. - Preserve the original operation for providers that do not offer asynchronous preparation. - Reject deadline offers that exceed the existing lifecycle lock budget. Honor accepted offers without shortening them. - Checkout verification uses the existing synchronous RPC with a 10-second deadline and a 12-second transport timeout. [Core verification PR #20876](https://github.com/paperclipai/paperclip/pull/20876) ## Verification - All 63 workspace SDK and Core integration tests passed with temporary local PostgreSQL. The tests cover lost status replies, deadline expiry, oversized offers, and synchronous verification with an async-capable worker. - All 58 runner SDK input, evidence, and public type tests passed with the regenerated source pins. The local server typecheck passed. ## Risks - This change activates only when the provider advertises the method. Existing providers keep their current behavior. - Polling uses one-second intervals during active preparation. It stops at completion, failure, or the provider deadline. - No database migration is required. ## Model Used - OpenAI GPT-6 Codex with local code execution and tests. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
566a97b882 |
fix(native): an unretirable predecessor sends the next attempt to a fresh sandbox (#20863)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs on a reused sandbox keep the previous run's warm Runner, and a new run must retire that owner before it starts > - #20844 retires the owner when Paperclip can prove it stopped. Some owners can never be proven stopped: the controller journal drain outlives the close grace, the Runner never confirms suspension, or a parked question leaves a tool input unanswered > - Then the new run failed before any provider work, and the task stayed blocked with no automatic continuation > - The general answer is to leave the old Runner on its sandbox and start the next attempt on a fresh sandbox restored from the saved workspace > - This pull request adds that fallback for every path that retires a predecessor, and settles a parked question's tool input once its Runner is gone > - The benefit is that an unprovable predecessor no longer fails a task that did no work ## Linked Issues or Issue Description **What happened?** An automatic wake started a new run 30 to 60 seconds after a long successful run on the same task, on the same reused sandbox. The new run had to retire the previous run's warm session first. The retirement failed and the run failed before any provider work, with one of: - `Controller journal drain timed out; preserve the session before replacing its writer.` (during instruction preparation, before runtime selection, or in the executor); - `provider_transport_failed: runner did not durably suspend before checkpoint` (during instruction preparation); - `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate; retirement_refused=journal_tool_input_unanswered` (the previous run had parked on a question). The tasks rested `blocked` with a reconciliation hold or with nothing scheduled. One bounded retry reused the same sandbox and ended `execution_finalization_deadline_exceeded`, then its recovery refused with `allocation_ownership_ambiguous`. **Expected behavior** A run that did no provider work does not fail the task because its predecessor cannot be retired. The next attempt drops the reused sandbox and starts on a fresh one restored from the saved workspace. Unsaved work is never dropped: when the workspace or the session backup is not durably saved, the sandbox is kept and today's behavior applies. **Steps to reproduce** 1. Complete a warm native run on a Daytona sandbox. 2. Wake the task again so that the next run must retire the warm owner (configuration change, new session, changed agent files). 3. Make the owner's close fail (journal drain timeout, unconfirmed suspension). 4. The new run fails before provider work and the task is not continued. Refs #20844, #18373. ## What Changed - `native-runtime/predecessor-retirement.ts`: a typed pre-provider error `NativePredecessorRetirementUnprovenError` (`native_predecessor_retirement_unproven`) and the `replaceBeforeReuse` lease marker. - Native executor: every path that retires a predecessor for a new run (instruction-copy claim, instruction-directory reservation, session reset, configuration change) throws the typed error when the close fails. The owner stays registered for its own sandbox, so no run on that sandbox launches over it. A run on a different sandbox forgets that owner instead of replaying its cached rejected close. The executor does not settle this error as a provider failure. - A prior-owner fence that refuses a terminal owner on a resumed sandbox (`prior_owner_terminal_state_indeterminate`) also throws the typed error. - Retirement of a finished remote owner accepts an unanswered tool input when the owner's committed result is a governed-wait park (`response_wake` on a task interaction) and its Runner's sandbox lease ended elsewhere. - Heartbeat (`heartbeat/predecessor-retirement-fallback.ts`): on the typed error with no execution-start evidence and no result, mark this run's reusable lease on that sandbox `replaceBeforeReuse` and log `environment.lease_replacement_requested`. After the run ends, `failRun` hands the prepared workspace on only through the shared unchanged-files proof (`proveRemoteWorkspaceUnchanged`; a provisioning command or any runner command refuses it). With the proof it records `predecessorRetirementFallback`, closes the run's coordinator without a result and re-queues the run in the uncharged `interrupted_preparation` lane (native and legacy runtime modes); a run that answered a task interaction keeps its own wake in that uncharged lane, so the retry still carries the answer. Without the proof the workspace stays dirty, heartbeat runs today's failed-workspace preservation and no retry is scheduled. - Environment runtime: a marked lease is replaced instead of resumed, under the same rule as a failed resume (`reusableSandboxReplacementIsSafe`: run ended, clean workspace with a committed save, verified harness backup). The old lease goes to `pending_cleanup` for the sweep. Otherwise the sandbox is resumed as before. - Chaos: session fault SR1U, the SR1 variant where the retained owner cannot be retired. The server gets a test-only fault (`PAPERCLIP_TEST_WARM_RETIREMENT_FAULT=unconfirmed`, armed only when `NODE_ENV=test`). The oracle allows only the fallback failure and requires the succeeding follow-up on a different sandbox. - Docs: `doc/plugins/PLUGIN_RUNNER_EXECUTION.md`, chaos README. ## Verification - RED on master before this change (with #20844), then GREEN: - `native-session-executor.test.ts -t "predecessor cannot be retired"`: base fails with `Error: Controller journal drain timed out…` instead of the typed error. - `native-session-executor.test.ts -t "unretired predecessor"` (instruction preparation): same base failure. - `native-recovered-owner-followup.test.ts -t "unacknowledged"`: base fails with `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate; retirement_refused=journal_tool_input_unanswered`. - New contract tests: `heartbeat-native-predispatch-retry.test.ts` (claimed native run: lease marked, activity logged, uncharged `interrupted_preparation` retry, no recovery action, task not blocked, retry succeeds) and `environment-runtime-reusable-resume-replacement.test.ts` (a marked lease is replaced without a resume; with unsaved work or no verified backup it resumes as before), `remote-workspace-unchanged-release.test.ts` (an untouched workspace is handed on; a setup-written one stays dirty) and `retries.test.ts` (an answered interaction re-queued in the uncharged lane stays a resolved-interaction continuation with no failure retry spent). - Existing suites pass: the native executor, recovered-owner follow-up, instruction working copy, run completion, run failure cause and resume replacement test files. `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts question-handoff scheduler config` passes. Server and chaos harness `tsc --noEmit` pass. - Not run: a Daytona SR1U wave. It needs a runner image that matches this checkout. ## Risks - A sandbox marked for replacement is resumed when it cannot be replaced safely. That run can meet the same unretired owner again, fail again in the bounded lane, and then rest with Retry, as today. - The replacement session restores the latest verified harness backup. When the old Runner's final checkpoint was never saved, a same-session continuation resumes from that backup. - An unanswered tool input is accepted only for a committed governed-wait park whose Runner's sandbox lease ended on another sandbox. - Previous-release story: the marker and the run record are additive JSON. An older server ignores the marker and resumes the sandbox as before. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, agentic coding with tool use (shell, file edits, test runs) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Nicky LeachandPaperclip
|
b2edea5754 |
fix(workspaces): verify existing worktrees before fresh retries (#20876)
## Thinking Path > - Paperclip manages work for AI agents. > - Retry checks whether the task can keep its existing workspace. > - A provider can complete checkout before Core receives the result. > - Recreating a completed worktree costs time and can change its base commit. > - Core uses the same ownership and writer checks for local and provider workspaces. > - The provider verifies its checkout before Core creates a fresh workspace. ## Linked Issues or Issue Description **What happened?** A retry can allocate a new provider worktree even though the original checkout completed. **Steps to reproduce** Complete workspace checkout through a storage provider. Fail Runner startup or lose the prepare reply. Retry the task. **Expected behavior** Reuse the same worktree when the provider confirms checkout and the prior run can no longer write to it. Related: #20865 addresses timeouts while the original prepare is in progress. This PR checks an earlier checkout when the user retries. ## What Changed - Check the prior task binding and writer state before asking a storage provider to verify a checkout. - Extend the existing prepare operation with an optional verification mode. Providers opt in through their manifest. - Return through normal workspace reuse after verification. Keep the fresh retry fallback when verification cannot establish checkout. ## Verification - Core workspace preparation, plugin workspace ownership, retry continuity, and SDK workspace-provider suites passed locally (213 tests). - Focused cases pass for provider verification, retry through heartbeat dispatch, and real local Git checkout verification with no plugin worker. The heartbeat test confirms that ordinary preparation and adapter dispatch keep the original workspace, branch, and base commit. - Core server typecheck passed before the final audit and timing edits. GitHub CI covers the final revision and broader regressions. ## Risks Provider verification adds one bounded call before a fresh retry. Older providers keep their current behavior. Verification must preserve the workspace, branch, and recorded base commit. The normal execution preparation checks still run before admission. No database migration is needed. ## Model Used OpenAI GPT-6 (Codex), with source inspection and local test execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either linked existing issues or described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal issue details - [x] I have run targeted tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
e6b538efd2 |
fix(native): retire a superseded warm owner instead of failing the new run (#20844)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs keep a warm Runner between turns of the same task, so
the next run can start fast
> - When the next run needs a new provider session (for example after an
agent setting or skill changes), it first retires the previous run's
warm Runner
> - That retirement uses the previous owner's close. When the close
cannot prove checkpoint suspension, the new run failed before any
provider work, with the previous owner's error
`provider_transport_failed: runner did not durably suspend before
checkpoint`
> - A governed-wait park (a saved question or review request) always
leaves such an owner: its question callback stays unsettled by design. A
failed Launcher stop RPC leaves one too
> - Nothing retried the run, and the task rested as "recovery needed"
with no provider work done
> - This pull request lets the new run retire the old owner when that
owner is contained, and forces the containment when the close left it
unproven
> - The benefit is that a fresh run never fails on its predecessor's
close, and the chaos harness now covers this class
## Linked Issues or Issue Description
**What happened?**
A new run on a task whose previous run left a warm Runner retained
failed within two minutes, before any provider turn. The error was
`provider_transport_failed: runner did not durably suspend before
checkpoint`. The run row recorded `executionRecovery: { kind:
"bootstrap", providerWorkStarted: false }`. Two runs of the same agent
on two tasks failed this way 47 seconds apart, after a configuration
change made both start new provider sessions. One predecessor had parked
on a `request_human_input` question; its callback never settled.
**Expected behavior**
A run that has done no provider work does not fail because its
predecessor's Runner could not prove suspension. The new session never
resumes the old checkpoint. Core must only prove that the old owner is
contained, and then start.
**Steps to reproduce**
1. Run a warm native task that parks on a saved question (chaos workload
Q1).
2. After the asking run settles, change the agent's adapter
configuration, so the next run cannot resume the saved session.
3. Answer the question. The follow-up run retires the parked owner and
fails with the error above.
Related: #18373 added the stopped-owner reset path that this change
extends.
## What Changed
- `closeWarmNativeSession` (server native executor): during retirement
for a different normalized session, accept an unrecoverable close when
the old owner is contained. A plugin task owner must show its
`executionStopped: true` receipt and a completed handoff. Without a task
owner, the Runner's own durable suspension
(`settlement.runnerSuspended`) is the proof. Unsettled Core tool
callbacks of the discarded conversation no longer block the reset.
- With a plugin task owner, any close error qualifies once the Launcher
receipt exists. This includes a failed `stop` that
`releaseRunnerProcessOwnership` rethrows, and "Controller journal drain
timed out": the discarded session's journal lives in its own state root.
Owners without a task keep the narrower rule. `--matrix` now rejects
session faults.
- `createPluginRunnerTaskOwner`: add `retireSupersededTask()`. When
close left the receipt unproven, the new run sends one confirmed `stop`
(not resent after a confirmed stop), takes the credential snapshot, and
calls `complete`. It never reads the old state.
- Same-session continuation, a handoff that still fails, other close
errors and failed instruction collection still block replacement.
- Chaos harness: add session fault SR1
(`tests/restart-chaos/session-faults.ts`). With Q1 and `--lifecycle
warm` (or `--daytona-warm`), the board changes
`adapterConfig.idleTimeoutMs` after the asking run settles and before it
answers. `--tasks 2 --concurrency 2` is the two-concurrent-runs variant.
Q1's oracle applies, plus `session_rotated_before_answer` and
`follow_up_new_session`.
- Docs: `doc/plugins/PLUGIN_RUNNER_EXECUTION.md` and the chaos README
describe the new boundary and cell.
## Verification
- RED on master: `native-session-executor.test.ts -t "permanently
rejected plugin close during reset"` fails the `pending-semantic` and
`stop-unconfirmed-at-close` cases with
`NativeSessionCloseUnrecoverableError: provider_transport_failed: runner
did not durably suspend before checkpoint`. GREEN with this change: all
identity-transition cases pass (49).
- Chaos SR1 local, master server code (`--env local --lifecycle warm
--workloads Q1 --faults SR1`, seeds 301 with 2 tasks and 303 with 1
task): all tasks pass, and the follow-up started a new provider session.
The local parked owner closes cleanly, so the local cell does not
reproduce the failure. It needs a Daytona task owner.
- Chaos SR1 local with this change (seed 302, 2 concurrent tasks on one
agent): 2/2 pass, `follow_up_new_session` true for both.
- Not run: the Daytona SR1 variant. No runner image matches this
checkout's runner content ID, and this change does not touch runner
inputs.
- `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts
question-handoff scheduler config` passes. Server and harness `tsc
--noEmit` pass.
## Risks
- The reset no longer waits for unsettled Core tool callbacks of the
discarded session. Their run is already terminal and their provider is
stopped, so no result can reach a provider, and the new session never
replays them. Core-side tool executions keep their own receipts under
the old run.
- A local owner without a task needs the Runner's durable suspension. A
local owner that did not suspend still blocks replacement, as before.
- Previous-release story: this changes no persisted state. Warm owners
live only in controller memory, so a deploy drops them as before.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, agentic coding with
tool use (shell, file edits, test runs) in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
Nicky LeachandPaperclip
|
052c638b51 |
fix(environments): honor stopped-task proof after warm preparation (#20289)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A warm plugin Runner retains its task for later runs. > - A follow-up can find that the retained task has stopped. > - The resume path already asks settle to check status before it sends stop. > - A saved startup marker disabled this order, even after warm preparation had finished. > - This pull request applies the same stop-proof check to all observed-stop releases. > - The next run skips a repeated stop when status proves both execution and required preparation have stopped. ## Linked Issues or Issue Description Fixes: #20224 Refs #20203 ## What Changed - Remove the startup-marker exclusion from the existing status-first path. Keep the existing shared confirmation check. It still requires `preparationStopped: true` for startup work. - Keep stop followed by status when the first status lacks the required proof. Keep the startup drain, credential writeback, complete, and cleanup failure handling. - Add retained-task coverage with the startup metadata that warm preparation saves. Cover a startup cleanup whose first status lacks preparation proof. - Update the plugin task lifecycle contract. ## Verification - Local macOS, source base `3e8744fa91`: from `server`, `pnpm exec vitest run src/__tests__/environment-task-runtime.test.ts src/__tests__/plugin-warm-tasks.integration.test.ts src/__tests__/plugin-task-credential-writeback.test.ts` passed: 3 files, 82 tests. The integration tests use real embedded PostgreSQL and controlled plugin results. - After restoring the fix, the two affected files passed again: 73 tests. - Independent parent verification on commit `19f720ff68`: the two files filtered to the changed startup-proof and retained-task cases passed (4 tests). Independent source review found no actionable issue. - Baseline proof: with only the source condition restored to upstream, the new retained-task test with saved startup metadata fails because it sees an extra `stop`. The case without that metadata passes. The source fix was then restored. - The retained-task regression asserts exactly `environmentResumeLease`, `status`, `complete`, and `environmentAcquireLease`, with both task operations bound to the old task. The startup test also proves that missing preparation proof sends `stop` and a second `status`. - Test setup: `node scripts/agent-preflight.mjs --skip-rust-debug --explain` installed dependencies, then stopped because Cargo is unavailable on this Mac. Initial test collection failed because the plugin SDK output was absent. `pnpm --filter @paperclipai/plugin-sdk build` built that required dependency and the shared package; the targeted test run then passed. - Standard Greptile reviewed commit `19f720ff686058b4dd6b34f27cdfb9c8ff60e50a` at 5/5 with no review threads. [Review](https://github.com/paperclipai/paperclip/pull/20289#issuecomment-6743829614). - Full local checks were not run, as requested. [GitHub PR workflow 42644023868](https://github.com/paperclipai/paperclip/actions/runs/42644023868) passed on final commit `19f720ff686058b4dd6b34f27cdfb9c8ff60e50a`. It supplied broad typecheck, build, and test coverage. The separate Snyk status passed. Optional Storybook checks were skipped. - The change removes one provider stop call on the reported path. The issue's measured eight calls would become seven if other calls stay the same. The earlier five-call target is not claimed. No deployed latency measurement is claimed. ## Risks - A provider that cannot prove preparation stopped still gets stop and status. That fallback can cost one extra status call, as the existing status-first contract already allows. - A status failure keeps cleanup pending and blocks replacement. No termination proof or credential handoff is skipped. - No schema, API, dependency, or timer change. No added work on successful task reuse or ordinary task dispatch. - Live provider acceptance remains unrun. The local evidence checks orchestration and stop-proof handling. ## Model Used - OpenAI Codex, GPT-6 family, with reasoning, repository inspection, shell tools, and code execution. The exact serving model ID and context window are not exposed by this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandClaude Opus 5.5
|
990de1e0bf |
perf(environments): release a stopped plugin task once (#20203)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Plugin environment drivers with `supportsTasks` run agents as provider tasks, and warm runs keep a parked task for the next run > - A follow-up run can find that its parked task already stopped. Core then retires that task before it acquires a new one > - Core sent `stop`, `status`, and `complete` in the settle step. Then the plugin release hook sent `stop`, `status`, and `complete` again for the same task. Every call was on the critical path of the new run > - This pull request removes the repeated calls and keeps every call that does work no other call does > - The benefit is a faster start for a run whose parked task already stopped ## Linked Issues or Issue Description No issue exists. This follows the enhancement template. **What existing behavior does this improve?** The release of a plugin task lease that has already stopped. This occurs when a follow-up run finds that its parked warm task stopped. **Subsystem affected** Server: plugin environment drivers and the plugin Runner task lifecycle. **Current behavior** Core sends `stop`, `status`, and `complete` in the settle step. Then the plugin release hook sends `stop`, `status`, and `complete` again for the same task. Core also sends `stop` after resume already saw that the task stopped. On a hosted test deployment, 8 of 40 follow-up runs took this path. From the resume to the new acquire, these runs spent P50 1014 ms and P90 1080 ms. These 8 runs are the full overall P90 (1275 ms). **Proposed behavior** Core does not call the release hook when settle confirmed the stop and completed the same task. After a resume with no lease, Core sends `status` first, and sends `stop` only when `status` does not confirm the stop. **Reason and benefit** The repeated calls do no new work. Removing them saves about 355 ms, plus the `stop` share of about 187 ms, for each such run. These numbers are modeled from trace timings. **Breaking changes** None for the plugin API. A task driver must release task-scoped resources in its `status` or `complete` handling. The doc now says this. **Additional context** The settle `status` stays. Providers release task storage there, and it records the stop that gates the credential writeback and `complete`. ## What Changed - `settleDispatchedEnvironmentTask` now returns `"completed"` when it confirmed the stop and completed the task in this call, `"stopped"` when the lease was already settled, and `false` otherwise. - The plugin driver release does not call `environmentReleaseLease` when settle returned `"completed"`. In all other cases it calls the hook as before. - When `environmentResumeLease` returns no provider lease, the release sends `status` first. It sends `stop` (and a second `status`) only when that `status` does not confirm `executionStopped: true`. Startup and startup-cleanup leases keep the old `stop` first order. - The cleanup sweep (`retryPendingSandboxTeardown`) and `destroyRunLease` do not change. A settle failure still leaves the lease in `pending_cleanup`, and the sweep runs `stop`, `status`, `complete`, and the release hook. - `doc/plugins/PLUGIN_RUNNER_EXECUTION.md` tells drivers about the new order and that task-scoped resources must be released in `status` or `complete`. ### Dispatch count for a stopped parked task (resume to new acquire) | | Calls | | --- | --- | | Before | resume, stop, status, (credentials), complete, release hook (stop, status, complete), acquire: 9 provider calls (10 with Codex writeback) | | After | resume, status, (credentials), complete, acquire: 4 provider calls (5 with Codex writeback) | The modeled saving is about 540 ms for each such run: about 355 ms for the repeated release, and about 187 ms for the settle `stop` and `status`. This pull request keeps the settle `status`, so the real saving is about 355 ms plus the `stop` share of the 187 ms. These numbers are modeled from trace timings. They are not measured on this branch. ### What covers each removed call - Release hook `stop`, `status`, `complete`: settle already sent `stop` and `status` and checked `executionStopped: true` for the same task id (`lease.providerLeaseId`), then sent `complete`. The skip uses settle's success as its condition, not the code path. - Settle `stop` after a resume with no lease: the next `status` must report `executionStopped: true`. If it does not, Core sends `stop` and `status` as before. ### What each kept call protects - Settle `status`: providers release task storage and cancel preparations here. It also writes `runnerTaskExecutionStopped`, which the credential writeback and the `complete` bookkeeping need. - Credential writeback: Codex can rotate its token. The writeback must land before the new task installs credentials. It still runs inside `complete`, before the new acquire. - Settle `complete`: it retires every run lease that owned the task and clears the dispatch marker. - Release hook when settle did not complete a dispatched task in this call: the provider may still hold the lease. ## Verification - `pnpm --filter @paperclipai/shared --filter @paperclipai/plugin-sdk build` - `pnpm --filter @paperclipai/server typecheck` - `npx vitest run src/__tests__/plugin-warm-tasks.integration.test.ts src/__tests__/plugin-task-credential-writeback.test.ts src/__tests__/environment-task-runtime.test.ts` - Related suites: `environment-runtime`, `plugin-runner-task.integration`, `task-creation-recovery.integration`, `heartbeat-plugin-environment`, `native-startup-cleanup-recovery`, `plugin-environment-driver-ready-recovery`, `environment-task-authority.integration`, `native-workspace-warm-reuse`, `remote-workspaces.integration`, `legacy-workspace-restore-recovery`, `plugin-task-source-recovery.integration`, `managed-plugin-environments`. Locally, all of these pass except `heartbeat-plugin-environment` "inherits the instance default environment across companies…". That test also fails locally with the `origin/master` service files, so this change does not cause it. The existing `plugin-runner-task.integration` same-run replacement test now expects no release hook after a completed settle. New tests: - The stopped parked path sends exactly resume, `status`, `complete`, acquire. The `status` goes to the old task. - A resume with no lease for a task that still runs sends `status`, `stop`, `status`, `complete`. - A failure at `status`, `complete`, or `stop` leaves the lease in `pending_cleanup`, and the sweep then sends `stop`, `status`, `complete`, and the release hook. - With an observed stop, the Codex writeback runs before `complete`. A writeback failure leaves the lease for cleanup, and the retry writes back and completes. - A startup-cleanup lease still sends `stop` first. ## Risks - A third-party task driver that releases resources only in `onEnvironmentReleaseLease` would not get that call after a completed settle. The doc now states the requirement. The cleanup sweep still calls the hook after any failure. - A driver that returns no lease from resume while the task still runs costs one extra `status`. The behavior stays correct. - No migration. No API change for plugins. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Nicky Leach
|
17ad23aafe | fix(workspaces): back off failed terminal cleanup and retire abandoned reservations through remove (#20020) | ||
|
Nicky LeachandClaude Opus 5.5
|
e6c74982fe |
fix(workspaces): report why a workspace provider operation failed (#19362)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A project can store its task workspaces with a workspace storage
provider plugin. Core calls the plugin's `workspaceOperation` method to
prepare, inspect, finalize and remove a workspace.
> - `callWorkspaceProvider` catches every rejection of that call and
throws the same conflict: "Workspace storage operation is incomplete;
retry the same workspace".
> - A provider error, an RPC timeout, a crashed worker and a worker that
never got the call all look the same in the run, the log and the API.
The message tells the user to retry, also when the failure is a fixed
configuration problem that a retry cannot fix.
> - In one diagnosis, three different configuration failures in a row
showed this same message. Finding each cause took trace span counts and
database reads.
> - This pull request lets a provider send a reason from a closed set,
and makes Core tell transport failures apart from provider failures.
> - The benefit is that a failed run says what failed and whether a
retry can help. Provider text and values still never cross into Core.
## Linked Issues or Issue Description
**What happened?**
A workspace provider's `workspaceOperation` failed. The run failed with
"Workspace storage operation is incomplete; retry the same workspace",
`errorCode: setup_failed`, and no reason in the run record or the
`heartbeat execution setup failed` log. Retrying did not help, because
the cause was a provider configuration problem.
**Expected behavior**
Core tells these cases apart: the worker did not receive the call, the
call timed out, the worker stopped during the call, or the provider
reported a failure. For a provider failure, Core shows a bounded reason.
Core tells the user to retry only when a retry can help.
**Steps to reproduce**
1. Install a workspace storage provider plugin whose
`onWorkspaceOperation` throws during `prepare` (for example, because of
missing provider settings).
2. Run an agent task in a project that uses that provider.
3. See the failed run: the error is the generic retry message, and the
run record and logs do not show the cause.
**Paperclip version / commit**
`master` at
|
||
|
Nicky LeachandClaude Opus 5.5
|
f9d366acdf |
fix(runner): require a managed AI connection for plugin Runner-task environments (#19341)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A plugin can provide Runner execution through an environment driver that declares `supportsTasks: true`. Agent computers use this path. > - Core let an agent without a managed AI connection run on that path. The run then used the account that was installed on the plugin's execution host. > - That is not a supported credential source. When the host Runner did not match, the run failed three times, 30 seconds apart, and then blocked the task with a generic error. The error did not name the fix. > - The plugin cannot refuse these runs itself. `describe` has no connection data, and Core replaces plugin error text before the user sees it. > - This pull request makes a managed AI connection a requirement for every plugin Runner-task environment. Core refuses a run without one before it acquires a lease or calls the plugin. > - The benefit is one clear notice that tells the operator to choose an AI connection, and no wasted retries or plugin work. ## Linked Issues or Issue Description Refs #19326. That pull request makes a Runner revision mismatch retry. It noted that runs without a managed AI connection reach the Runner check on a different path, which still blocks after 3 attempts. This pull request removes that path for plugin Runner-task environments. **Problem or motivation** A `paperclip_runner` agent with no managed AI connection can run in a plugin environment that provides Runner tasks. The run uses the account installed on the execution host. If the host Runner does not match, the native session retries 3 times and then blocks the task with "Plugin-provided Runner execution operation unavailable". The operator cannot tell what to fix. **Proposed solution** Refuse these runs in Core before any lease, plugin call, secret resolution, or provider work. Use one stable reason, `ai_connection_required`, for run admission, the setup probe, and agent verification. Show a task notice that names the fix. **Alternatives considered** A refusal in the plugin. It cannot see the connection at `describe`, which runs first. Its error text is also replaced twice before the user sees it. A new structured plugin error code would still not cover `describe`. ## What Changed - Add `server/src/services/plugin-runner-ai-connection.ts`. It holds the `ai_connection_required` code and message, the probe result, and the manifest lookup for a Runner-task driver. The lookup moves here from `agent-runner-selection.ts`, so run admission and runner selection use the same rule. - `heartbeat/execution-configuration.ts`: refuse a new `paperclip_runner` run in a Runner-task environment when it has no managed AI connection. This also applies when `homeMode: "preconfigured"` drops the connection. The run fails as `configuration_incomplete` with reason `ai_connection_required` and an action URL to the agent's runtime settings. The check runs before observability, secrets, the managed runtime, the workspace, and the lease. A resumed native run keeps its admitted contract. - `recovery/stranded-notice.ts`: add an "AI connection required" task notice. - `run-failure-report.ts`: treat this pre-dispatch refusal as an owner action, not an unexpected failure. - `adapters/registry.ts`: the unmanaged probe for a Runner-task target now returns a failed `ai_connection_required` check. It calls no task operation. Managed probes still go through `testManagedEnvironment`. - `agent-harness-verification.ts` and `agent-lifecycle/application/failure.ts`: classify the check as `ai_connection_required` with a fixed message. Verification waits for the configuration to change instead of retrying. - Docs: `PLUGIN_RUNNER_EXECUTION.md` states the requirement. `PLUGIN_ENVIRONMENT_TASKS.md` no longer mentions installed accounts. `agent-config-ui.md` notes the requirement. - Tests: the "plugin-owned account" heartbeat test described the behavior that this change removes. It now checks the assembled heartbeat refusal: no plugin call, no native boundary, no adapter, no workspace files. ## Verification - `npx vitest run` in `server/` for these files. All tests passed: - `src/__tests__/heartbeat-execution-configuration.test.ts`. New cases: refused without a connection, refused for a preconfigured home, managed passes, resumed native input passes, non-task plugin driver passes. The refused cases also check that observability, secrets, and the managed runtime did not start. - `src/__tests__/heartbeat-plugin-environment.test.ts`. One test, "inherits the instance default environment across companies…", also fails on unmodified `master` in my environment. This change does not touch it, because its driver does not declare `supportsTasks`. - `src/adapters/registry.test.ts`, `src/__tests__/agent-harness-verification.test.ts`, `src/modules/agent-lifecycle/__tests__/diagnostics.test.ts`, `src/services/recovery/stranded-notice.test.ts`, and `src/services/__tests__/run-failure-report.test.ts`. - 17 more suites that use plugin Runner-task environments, with 705 tests: `plugin-runner-task.integration`, `plugin-warm-tasks.integration`, `task-creation-recovery.integration`, `native-startup-cleanup-recovery`, `environment-task-authority.integration`, `plugin-task-credential-writeback`, agent-lifecycle `lifecycle`, `setup`, and `environment-plugins`, `workspace-preparation`, `agent-runner-selection`, `agent-harness-verification-provider-api`, `agent-harness-verification-remote`, `plugin-environment-driver-seam`, `environment-task-runtime`, `agent-test-environment-routes`, and `adapter-registry`. - `pnpm typecheck` in `server/` reported no errors in the changed files. Its only errors came from sandbox-provider packages that my local install did not link. - Not run: `plugin-runner-probe.test.ts`, because it needs a Cargo build. This change does not modify `probePluginRunnerEnvironment`. Also not run: the full `pnpm test:run` and `pnpm build`. CI runs them. ## Risks - Behavior change: an agent in a plugin Runner-task environment with no managed AI connection now stops before dispatch. Before, it ran with the host's installed account. The agent's owner must choose an AI connection. This includes agents with `homeMode: "preconfigured"` in these environments. - Runs that were already admitted and resume from saved native input are not refused. They keep the earlier behavior until they finish. - The check reads the plugin manifest on each new run admission for plugin environments. This is the same lookup that runner selection already does. - This pull request and #19326 both edit `stranded-notice.ts` and `failure.ts`. The changes are additive, but whichever merges second needs a small rebase. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (file edits, shell, test runs). Default context window, extended thinking on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Nicky LeachandPaperclip
|
aab87ce707 |
Prove workspace delivery before inspecting cleanup candidates (#18493)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Completed tasks leave workspaces that need safe cleanup. > - Providers without ancestry inspection currently need a recorded merged PR before cleanup can start. > - A squash merge changes commit identity, so direct ancestry cannot prove that delivery. > - This change proves delivery from the finalized HEAD and captured destination before storage inspection. > - Fresh storage checks and a second delivery check still protect removal. > - This reuses the existing worker and receipts. It adds no table or timer. ## Linked Issues or Issue Description Refs #18410. Related: #13400 addresses workspace identity during inspection; this change adds destination-aware delivery discovery. ## What Changed - Resolve direct ancestry or discover an associated merged GitHub PR without a recorded work product. Require exact HEAD, destination repository and branch, and a reachable integration commit. - Use the latest settled finalization receipt as a candidate. Reject missing, failed, pending, conflicting, or changed evidence. - Inspect storage only after positive remote proof for providers that disable ancestry inspection. Recheck HEAD, branch, source identity and delivery before removal intent. - Add an optional provider capability for a specific PR recovery hint and an optional prepared destination receipt. Older providers keep their existing protocol. - Wake cleanup when finalization writes settle or gain a workspace binding. Keep remote-only retries for concrete unresolved delivery. Park missing or unsupported candidate identity. - Reuse the source project's explicit GitHub consent for delivery checks through the existing authorization service. Revalidate the bound repository and consent before removal; missing or revoked consent retains storage without switching identities. Public sources remain anonymous. - Update the contracts, documentation, access map, and targeted regressions. ## Verification - Initial-head GitHub CI passed at `5acd4c818be6f1764f8319f42891f9956c357c26`, including full build and typecheck. Standard Greptile is 5/5 with no open threads. The first Apex review scored 3/5 and found startup latency from remote proof lookups and removal retry identity changes across provider capability changes. Both corrections are pushed in `9d1a38df3d`, with the qualified destination-ref length fix. Red regressions reproduced each issue. Latest targeted checks passed: cleanup/worker 26, provider integration 41, SDK contract 8. Head `9d1a38df3d` passed standard and Apex reviews at 5/5 with no open threads, plus all CI including full build/typecheck. Canary Dry Run initially failed on an npm 404 fetching `rollup-4.67.3.tgz`; the exact tarball subsequently returned HTTP 200, and one targeted retry passed without source changes. Aggregate verification also passed. After #18403 merged, master introduced conflicts in shared workspace cleanup and metadata handling. Resolved them in `7700832536`, preserving remote snapshot cleanup and Git delivery-proof cleanup. Targeted merge verification: 93 tests passed in the first run (59 opt-in skips), then all 16 remote lifecycle tests passed with explicit database opt-in; map checks passed. The isolated database is stopped. The merge head received standard review 5/5. Its CI found two failures in unchanged test cleanup paths: native runtime temporary-directory ENOTEMPTY and agent-chat cancellation lock timeout. Those failures remain recorded and require successful final-head verification. The subsequent project-consent change has targeted coverage: authorization36, resolver33 and cleanup19 tests passed, including revoked consent, anonymous public access and source-binding races. The positive consent test failed before the fix. Current head `c772679e05` has green GitHub CI, including full build/typecheck and both previously failing test jobs, plus standard Greptile 5/5 with no open threads. Neither cleanup failure required changes to its test path. Apex also completed at 5/5 on this exact head (check `124651548532`, summary updated 2026-10-31 20:45 UTC), with no open review threads. The PR remains draft. Earlier failures remain recorded above. - Final focused run on the rebased branch: **182 tests passed across six files**, covering the resolver, delivery cleanup, provider bridge, finalization notifications, existing workspace service and SDK contracts. The disposable test database was stopped afterward. - Targeted resolver tests: 33 passed. These cover ancestry, discovered squash PRs, wrong HEAD/base/repository, removed integration commits, bounded responses, cancellation and credential isolation. - Targeted cleanup integration tests: 17 passed. Provider bridge integration: 36 passed. SDK contract tests: 7 passed. Finalization notification tests: 9 passed. - A negative control that removed the final delivery check failed the race regression as expected. The check was restored. - Read-only live GitHub probe: the new resolver discovered PR #18410 from HEAD `3f5ade4cd42b707040e9a82fdaed343e3d736a09`, without PR hints. It proved integration commit `6484f96c28bfb11a793ce4be6cc5ac15643a5a02` reachable from observed master `5bc029bf6c1db6647e96acea323f3723fd1d3eba`. GitHub also advertised the exact original HEAD at `refs/pull/18410/head`. - `node scripts/check-maps.mjs` passed. Structural checks do not prove deployment behavior. - Local tests are targeted by user request. Full build, typecheck and test suite run in GitHub CI. A standalone server typecheck could not complete because local Runner build declarations were absent; no changed-file diagnostics were reported. - Initial local execution hit a stalled pnpm wrapper, sandbox socket denial and stale generated SDK declarations. Direct installed binaries, the authorized isolated test database and rebuilt SDK resolved those setup failures. No deployment was performed. ## Risks - Destructive cleanup still requires fresh storage safety and lifecycle checks. Remote proof alone never authorizes removal. A persisted removal intent preserves existing recovery semantics. - Legacy workspaces without an unambiguous destination or completion HEAD remain retained. This does not guess the original default branch. - Private repositories use the explicit standing GitHub consent on the bound source project, with current membership, grant, repository access and trust checks. No separate company token is required when that consent is valid. Missing or revoked consent and unavailable or rate-limited access retain storage; the worker does not borrow a task or agent identity. - A known undelivered commit keeps the existing five-minute retry because external GitHub changes have no guaranteed ingress to this lane. This can keep cleanup scheduled. Missing evidence parks instead. - Providers need the new recovery capability to clean a squash merge whose original HEAD is reachable only through its PR ref. Providers without it remain conservative. - The resolver currently supports GitHub.com. Existing local and other-provider ancestry paths stay in place. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository tools and parallel code agents. The exact runtime model ID and context-window size were not exposed to this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky Leach
|
6484f96c28 | fix(workspaces): inspect storage only when needed (#18410) | ||
|
Nicky LeachandPaperclip
|
6174b4d7c8 |
fix(runner): allow session reset after confirmed task retirement (#18373)
## Thinking Path > - Paperclip manages AI agents and their work. > - Native warm sessions retain a Runner owner between turns. > - Closing that owner must stop execution and preserve valid recovery state. > - A failed suspension can leave a rejected close promise after the task has stopped. > - A session reset then retries that same promise and blocks every fresh session. > - This PR separates task retirement proof from checkpoint reuse for a new session. > - A reset can proceed after exact stop proof, completed handoff, and file collection. ## Linked Issues or Issue Description Refs #18011. That change preserves checkpoint authority during warm preparation. This change handles a stopped owner whose checkpoint cannot be reused. Related: #14695 covers managed files retained with warm processes. A search found no duplicate stopped-owner reset fix. **What happened?** A warm Runner can stop and complete its task handoff while close reports `runner did not durably suspend before checkpoint`. The transport caches that rejected close promise. Core retains the warm owner. Each session reset retries the same promise before it can launch a provider. A controller restart clears the in-memory owner and permits recovery. **Expected behavior** A reset to a new normalized session should release the old owner when its exact task has proved that execution stopped and its handoff completed. Core must collect stopped instruction files before it admits the new session. An unconfirmed owner must continue to block replacement. **Steps to reproduce** 1. Complete a warm native run through a plugin task provider. 2. Change configuration so Core retires the warm owner. 3. Let the task stop and complete, but fail durable suspension proof. Close keeps its rejected promise. 4. Reset the session and start another run on the same task environment. 5. Observe the old close error before any new provider work starts. ## What Changed - Keep a lease-bound retirement receipt in the plugin task owner. It requires `executionStopped: true` and successful task completion after credential handoff. - Accept an unrecoverable suspension error only during retirement for a different normalized session, with that exact owner's receipt and settled Core tool results. - Collect stopped instruction files before releasing the warm owner. Preserve old checkpoint evidence. - Keep same-session, active-owner, incomplete-handoff, unrelated-owner, unresolved Core tool results, and other close-error fences. - Report Core tool settlement on the close error after ingress retirement and drain. Default missing proof to false. Expose the error through the server's Runner shim. Add regression tests and document the recovery boundary. ## Verification - The reset regression fails with the new handling disabled and passes with it enabled. - Both affected suites passed: 1,032 tests. The final 15 focused owner/reset cases pass after adding the tool-settlement guard. - The assembled regression uses the real task-owner and backend binding. Its controlled task provider proves stop and completion. Close returns the same rejected promise on every call. - Negative cases cover incomplete handoff, other close errors, same-session continuation, provider change with the same session ID, unrelated receipts, unresolved Core tool results, and failed file collection. Collection can later succeed without replaying stop or reading the old checkpoint. - Five real-process transport cases pass. They cover an insufficient suspension deadline, the cached rejected promise, late tool completion within and beyond the deadline, interruption, and result-persistence failure. Unsettled callbacks produce `semanticToolsSettled: false`. - `pnpm -r typecheck` passed with the repository's pinned Rust toolchain on PATH. After the final contract change, Runner TypeScript build and server `tsc --noEmit` passed. - The local full-suite run was stopped at the requester's direction. Full regression and build coverage are delegated to GitHub CI. - Full GitHub CI passed on `ef0040d80af679ce6d4a130468e15f75678c1f4c`. Standard Greptile reviewed the same commit: 5/5, zero comments. Apex then reviewed that same commit: 5/5, zero comments. Final checks confirm no unresolved review threads and no merge conflicts. - Initial full CI on `00c002fd5a` failed one scheduled-retry route test at 18/20 cancellations. Controlled cancellation latency reproduced the short poll-budget failure. Before the next cycle, master advanced with #18322, which coordinates that test at the first completed batch. The separate local timeout commit was dropped during rebase. The upstream test passes locally. The rebased run passed 46 targeted owner/reset and scheduled-retry route cases. This PR has no scheduled-retry source or test diff. - No deployed provider run or tenant rollout is claimed by these source tests. ## Risks The exception applies only to a new normalized session and an exact completed plugin task. The old checkpoint remains evidence and cannot authorize reuse through this path. The original suspension failure can remain visible on its run. This fix prevents repeated startup failures after a reset; it does not hide unconfirmed cleanup or make every suspension succeed. Missing task receipts and failed file collection continue to block replacement. No schema migration, plugin protocol change, Rust Runner binary change, or new polling is required. Tests use a controlled task boundary; they do not certify every external provider implementation. ## Model Used OpenAI Codex, GPT-6 family, with tool use and local code execution. The exact runtime model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub references) - [x] My branch name describes the change and contains no internal issue or instance details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] Greptile Apex is 5/5 with no unresolved findings - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky Leach
|
fba0ab8162 | fix(runner): preserve GitHub helpers and Codex probe policy (#18212) | ||
|
Nicky Leach
|
e2cfdc7e0c | feat(plugins): expose GitHub credential broker to task providers (#18178) | ||
|
Nicky LeachandPaperclip
|
247de89eae |
fix(runner): preserve warm tasks during recovery (#18169)
## Thinking Path > - Paperclip manages agent work and the lifetime of its execution resources. > - A warm native task can keep one Runner alive between successful turns. > - Foreground completion selects whether that resource stays running or stops. > - Background recovery must use the same selected policy. > - Recovery currently reads that policy only from workspace copy-back records. > - A task without such a record can be stopped immediately after foreground retention. > - This pull request saves the resource policy with the native runtime and permits the retained task to refresh its control connection. ## Linked Issues or Issue Description **What happened?** Background native finalization can stop a successful warm Runner when its run has no Core workspace-sync record. Foreground completion retains that same task. The stopped Runner cannot receive the later drain and suspend commands. Cold resume then correctly rejects its unproven checkpoint. Hosted diagnostics confirmed the recovery reconciler attempted environment release immediately after foreground retention. The provider connection returned HTTP 404 within the same second. A database-backed regression reproduced the stop without a workspace-sync record. A separate authorization gap also prevents the current retained task from refreshing its connection endpoint after the run succeeds. **Expected behavior** Foreground completion and recovery must use the selected resource policy. A successful warm task stays running until its idle shutdown can suspend it. Failed or cancelled work still stops. The current retained task can refresh its existing control route, but cannot prepare or submit new execution. **Steps to reproduce** 1. Select a warm native runtime on a reusable remote execution environment. 2. Complete a turn without a Core workspace-sync record. 3. Let native finalization recovery settle the still-active run lease before foreground retention finishes. 4. Before this change, recovery defaults to `stop_and_retain` and invokes the provider stop path. 5. Separately, request the retained task's connection endpoint after successful completion. Before this change, Core rejects that request. **Paperclip version/commit** The hosted failure used Core |
||
|
DottaandPaperclip
|
09ad9adce2 |
feat(server): adopt running remote agents through durable controller checkpoints (#17439)
## Thinking Path > - Paperclip must keep admitted remote work alive while its main server is replaced. > - Runner owns recovery mechanics and exposes opaque checkpoints through #17097. > - The server owns company scope, encryption, Stop, tool authority and execution ownership. > - A successor must prove current ownership before it adopts the original execution. > - Checkpoint bursts must also leave capacity for normal requests and lease renewal. > - This PR persists and adopts that authority through the existing runtime boundaries. ## Linked Issues or Issue Description **What existing behavior does this improve?** Remote agents retain the original execution during a bounded main-server outage. **Current behavior** A server restart can lose the controller's local state and connection. An unsafe replacement could repeat a turn or an uncertain mutation. Checkpoint writes in the request pool can also delay ordinary requests during an aligned burst. **Proposed behavior** Restore encrypted control records from Postgres and adopt the original runner under a fenced lease. Recover known tool receipts through current tool authority. Give control checkpoints a shared pool capped at two connections per database owner. Depends on #17097. Successors are #17455 and #17456. Qualification and presentation remain separate review slices. ## What Changed - Add company/run-bound encrypted checkpoint records and migration 0394. Keep the SQL idempotent after renumbering against current master. - Recover startup and expired ownership through native runtime and heartbeat. Preserve the sandbox lease, admitted runner artifact, harness configuration, native session and active turn. - Keep Stop, finalization, preparation ownership and stale-writer fencing transactional. Preserve graceful detach and natural crash takeover. - Recover committed tool receipts through the existing authority. Reauthorize reads and hold uncertain mutations. - Bound startup retention with the existing coordinator and preserve zero idle scans. - Share at most two checkpoint connections per database client. Keep application and liveness capacity separate. Close the checkpoint pool during shutdown; late writes cannot reopen it. - Count SQL across both application and checkpoint clients. Prove request/liveness progress, command recovery and failure after pool shutdown with real Postgres. ## Verification Project creation recovery now uses the existing project activity receipt owner. It retains authenticated input normalization, fingerprint checks and current run authority after the receipt lock. An absent receipt returns no result before mutation. The new lost-HTTP-response and no-effect regression uses real API routes and Postgres. The seven transport cases pass locally. This Mac skips all 20 database cases. On current source `5de803c2cd`, Linux [job 122593415890](https://github.com/paperclipai/paperclip/actions/runs/40620576702/job/122593415890) executes all 20 project-tool cases successfully (9.480 seconds), including the committed-project lost-response regression. Its shard passes 1,627 cases with three unrelated skips. Server typecheck passes. The transport regression initially has two failures; a subsequent incorrect expectation about schema defaults was corrected before the final pass. Ordinary execution adds no checkpoint writes or recurring work. Current candidate `5de803c2cd` includes the project-creation receipt recovery fix and master `2cd747d1b6`. Migration 0394 retains the prior idempotent checkpoint DDL. Every master SQL file and journal entry is preserved. Snapshots 0390–0394, migration checks and server typecheck pass. The ordinary warm-turn regression fails on its second turn before `a219ab51ad`. After correction, three completed warm turns select the correct prior checkpoint and preserve the original session. All 288 provider-wiring cases pass. The control plane, external persistence callback, identity rotation and filesystem are real; the backend and database-load seam are scripted. The 13 warm admission/configuration variants also pass after final master integration. This does not qualify an underlying harness. At current source `5de803c2cd`, Linux [job 122593415907](https://github.com/paperclipai/paperclip/actions/runs/40620576702/job/122593415907) passes all 940 native-session-executor cases (61.431 seconds), including the warm external-checkpoint behavior; the shard passes 2,358 cases with one skip. Current [Runner job 122593415475](https://github.com/paperclipai/paperclip/actions/runs/40620576702/job/122593415475) passes the native recovery lane: 115 passes and five container-only skips, including the encrypted restore guard, fresh preparation (63.189 seconds), blocked claim lock (1.226 seconds) and recovery beyond 101 blocked owners (3.897 seconds). The first lane passes 2,215 cases with one skip. The SQL cost test reports zero idle and streaming SQL for both output volumes; this is component evidence, not a production capacity claim. At prior candidate `1ff7bb44f4`, Linux job 122549307215 passes all 938 native-executor cases, including the three-turn external-checkpoint regression. Job 122549306469 passes the native recovery lane with 115 passes and five container-only skips, including the encrypted-restore session guard, fresh preparation (63.664 seconds), claim-lock contention (3.961 seconds), and 101-owner recovery (3.824 seconds). This Mac skips the real-Postgres suite. The earlier full local executor attempt has 921 passes and 16 failures during embedded-Postgres initialization before assertions. The original fixture collisions and their corrections remain recorded. At earlier candidate `8fbe62b41d`, server shard job 122535673841 has 3,876 passes, one failure and one skip in an unchanged expiry-worker retry/drain timing test. The same server/database tree on qualification source `b64cac6951` passes all 20 expiry cases and all 3,877 shard cases. The single diagnostic retry, job 122541891915, passes all 20 expiry cases and all 3,877 shard cases (one skip). No assertion was weakened. The original failure remains recorded and its exact timing race is not proven. All Actions checks and legacy status passed on `8fbe62b41d`, with completed Greptile 5/5 and no open feedback. The next candidate `1ff7bb44f4` also passed all Actions checks and Greptile 5/5. Its Paperclip Review remained queued before the new master conflict required integration. The new candidate needs fresh exact-head CI and reviews; old checks do not qualify it. Migration generation is checkpoint-only, all master SQL and journal entries remain unchanged, and migration checks, server typecheck and all 13 warm-owner variants pass after integration. The first migration check was blocked by TSX sandbox IPC; its permitted repeat passes. The first warm-test command used an incorrect path and ran no tests; the corrected command supplies the 13-case receipt. All 16 database pool-lifetime/options/teardown cases pass after granting loopback permission; the first batch had two EPERM failures and 14 passes. Earlier semantic-tool/HTTP recovery, adoption, encryption-batch, zero-streaming-write and idle-SQL receipts remain source-attributed in the [qualification record](https://github.com/paperclipai/paperclip/blob/2ff66f57da0da35bac1c19a2366ea4b84d87a986/doc/plans/2026-10-22-hot-upgrade-qualification.md). That record retains earlier assertion, setup and cleanup failures and their corrections. No failed gate is waived. The historical 300-session component campaign passes four after phases: steady read p99 is 107–108 ms at 500 reads/s and 100 writes/s; aligned-burst read p99 is 103–105 ms after the bounded pool, versus 584–689 ms before. This comparison isolates the persistence corrections. It does not measure the final stack's hosted active-agent capacity. The hosted zero-agent API pair leaves SQL and CPU essentially unchanged, but both arms fail absolute latency gates and candidate peak cgroup memory fails its gate. The active-agent/viewer matrix, different-artifact upgrade and rollback remain open. Run `PAPERCLIP_TEST_DATABASE_URL=<owned-disposable-postgres> pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`. Previous head `f2db8a6568` had one failed workspace setup gate: [job 122555672441](https://github.com/paperclipai/paperclip/actions/runs/40608984238/job/122555672441) passes all 4,169 UI tests and 285 CLI cases (one skip), but the company import/export suite reaches its 30-second health deadline during doctor startup. Its output does not reach the server-start step. The CLI, server, database, shared, adapter and plugin trees are identical on qualification source `a866619d7b`, whose [job 122555886663](https://github.com/paperclipai/paperclip/actions/runs/40609057937/job/122555886663) passes that fixture and all 286 CLI cases. The one diagnostic retry, [job 122561057225](https://github.com/paperclipai/paperclip/actions/runs/40608984238/job/122561057225), passes the formerly failing fixture and all 286 CLI cases. All Actions checks and legacy status on `f2db8a6568` pass. Its exact-head Greptile completed at 5/5 with no open feedback. The current integrated head requires fresh checks and a completed Paperclip Review before merge. The original failure remains recorded; its exact timing cause is unproven. A repeated failure requires investigation and repair rather than another retry. All current-head CI, legacy status, exact-head completed Greptile 5/5, resolved feedback and ordinary GitHub merge eligibility remain required before merge. The latest integration preserves master’s published Copilot v70 profiles and every published migration. Checkpoint migration `0394_conscious_whizzer.sql` was generated with `pnpm db:generate` and retains the prior idempotent DDL. The Runner vendor rebuild and replay parity checks pass. Server typecheck and all 13 warm-owner variants pass after that rebuild. The imported report-lock patch is byte-identical to master; its intentional patch-context whitespace is retained, and the feature delta passes whitespace checks. No hosted experiment was run. ## Risks The pool adds up to two active connections per instance, shared across agents. Checkpoint delivery can wait behind that bound; it still requires its commit. Steady component-test Postgres CPU rises by about 1.7 percentage points of a two-CPU allocation. These corrections do not remove the stack's control and renewal writes. The migration and key must survive replacement. Fresh-database tests and local load do not establish different-artifact hosted upgrade or rollback continuity. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools and code execution. The runtime does not expose a more specific model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky Leach
|
ea727dc3c2 | Preserve plugin session continuity through warm execution and checkpoint recovery (#17820) | ||
|
Nicky Leach
|
e5c9e60c5a | Avoid session resets when reusing default task directories (#17419) | ||
|
Nicky Leach
|
f7716ec65b | Allow exact warm-state reads during retained task retirement (#17418) | ||
|
Nicky Leach
|
6b4bdc515b | Fix GitHub ownership checks for warm plugin tasks (#17417) | ||
|
Nicky Leach
|
aa0037a137 | Support Codex warm plugin task reuse and recovery (#17390) | ||
|
Nicky LeachandPaperclip
|
a3e8fa568a |
Reduce repeated work during plugin Runner admission (#17377)
## Thinking Path > - Paperclip manages agents and their execution authority. > - A plugin can host Runner tasks on a persistent execution host. > - Admission checks must observe current lifecycle state and credentials. > - The current runtime reader also resolves files, skills, and connections for each check. > - Some providers also repeat the same project authentication check during submit. > - This pull request adds a compact fresh reader and an explicit submit-time validation contract. > - Providers can remove repeated work while Core keeps lease and scope checks. ## Linked Issues or Issue Description **What existing behavior does this improve?** Plugin Runner startup before task submission. **Current behavior** One measured startup made 30 full runtime reads during bootstrap. These reads took 5.759 seconds. Each guard can resolve instructions, skills, and connection assignments even when it only needs current identity and lifecycle state. Providers that already enforce project authentication in submit must also perform a separate Core preflight. **Proposed behavior** Use a lifecycle-owned runtime identity reader for guards. Keep the full runtime reader for synchronization inputs. A trusted provider can declare `projectAuthCheckAtSubmit: true` in its prepared result. Core pins the exact configuration, provider, and project set on the lease. The provider must reject authentication conflicts before credential installation or task start. Providers that omit the flag retain the existing preflight. **Reason and benefit** Estimated savings with a paired consumer: 4–5 seconds from compact runtime checks, plus roughly 0.5–0.9 seconds from removing the duplicate authentication operation. These estimates overlap and are not a measured deployed result. Refs #17367 for the separate run-log HEAD optimization; this PR does not duplicate it. **Breaking changes** No migration. The SDK adds a host method and an optional prepared-result field. Consumers that use them require this Core host version. Older providers retain their existing flow. ## What Changed - Add `agents.getRuntimeIdentity` across SDK, RPC, host service, and test fixtures. - Reuse the existing company, plugin ownership, readiness, and capability checks. Read lifecycle state and protected instance secrets fresh on every call. - Skip instruction, organization, skill, and connection hydration in the compact reader. - Allow explicit submit-time authentication enforcement, with prepared scope pinned and checked again at dispatch. - Emit fixed preparation, project-auth, submit, and connection timing phases. ## Verification - 72 targeted SDK capability, plugin task-owner, and environment-task mediator tests passed. - 33 real embedded-Postgres lifecycle and secret tests passed, including pause, content revision, company isolation, plugin disablement, and secret rotation. - SDK and server TypeScript checks passed. Local Runner/eval TypeScript outputs were built to satisfy server type resolution. - Local Postgres required missing library aliases in ignored dependencies; no product source was changed for that repair. - Full repository suites are delegated to GitHub CI. No live rollout or speedup claim. ## Risks The submit-time flag delegates enforcement to a trusted provider. It is not proof that prepare already checked authentication. The provider must check before VM credential installation or task admission. Core still checks exact prepared scope, active lease, run identity, credential type, and dispatch fencing. Existing providers keep the separate check. Roll back a paired consumer before rolling back this host. ## Model Used OpenAI GPT-6 (Codex), with code execution and repository tools. Exact deployment build and context-window size are not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal ticket id - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandPaperclip
|
a708a13336 |
Separate plugin workspace completion from maintenance inspection (#17329)
## Thinking Path > - Paperclip manages AI agents and their work. > - Storage plugins own remote task workspaces. > - Run completion currently requests the same inspection used for storage cleanup. > - That inspection can fetch remote refs and scan files after the agent has finished. > - Completion needs retained workspace identity and durable writes, not permission to delete files. > - This change adds an optional completion operation and keeps maintenance inspection separate. ## Linked Issues or Issue Description **What happened?** A successful plugin-backed run can fail during completion because the storage provider performs a slow remote-recoverability inspection. Dirty or unpublished output does not need a deletion-safety proof to remain valid task output. **Expected behavior** Completion verifies the assigned workspace and confirms durable writes through the provider. Cleanup still requires the existing full inspection. **Steps to reproduce** Run a task with plugin-owned storage whose inspection fetches remote refs. Let the agent finish, then observe completion waiting for the maintenance inspection. Related context: #16428 added finalization observability. The duplicate search found no existing PR for this provider contract. ## What Changed - Add optional `workspaceProvider.finalizationVersion: 1` and strict `finalize` / `finalized` schemas. - Pass original run and agent context through live completion and recovered completion. - Give the new operation a 30-second deadline and keep ownership checks and lifecycle serialization. - Retain the inspection fallback for providers without the capability. Keep maintenance inspection and removal unchanged. ## Verification - SDK build passed. - Targeted SDK and completion unit tests: 7 passed. - Added integration coverage for capability selection, legacy fallback, maintenance separation, and failure propagation. - Local integration setup timed out starting PostgreSQL; the integration suite passed in GitHub CI. - Full build/typecheck and broader tests passed in GitHub CI (55 successful checks, 2 skipped). Local checks remained targeted as requested. - Diff reviewed for secrets and operational identifiers. ## Risks The capability is opt-in. Providers must not advertise it until their completion implementation is available. An unavailable provider or failed durability barrier remains retryable. No schema or stored-binding migration is required. Test-file size exception: the provider integration suite grows from 973 to 1,002 lines. The 29-line case follows the same prepared binding through legacy completion, opted-in completion, and maintenance inspection using the existing single-connection database fixture. Keeping it there avoids a second embedded-Postgres lifecycle for one transition test. The completion helper unit tests are in a separate focused file. ## Model Used OpenAI GPT-6 Codex, with reasoning and code/test tools. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky Leach
|
412914cfa7 | Read native plugin task output files on the execution host (#17285) | ||
|
Nicky LeachandPaperclip
|
00331e0254 |
Connect native Runner admission to plugin-owned workspaces (#16978)
## Thinking Path > - Paperclip manages work for AI agents. > - A plugin can own an execution environment and its files. > - #17029 supplies the task driver and bounded setup verification. > - Heartbeat still prepares workspace files on Core and expects a shell transport. > - This PR gives the selected plugin ownership of workspace preparation and native task execution. > - Core keeps admission, tool authority, and recovery checks. ## Linked Issues or Issue Description **Subsystem affected** Heartbeat workspace admission and native Runner execution. **Problem or motivation** A task-only plugin cannot run through Core filesystem staging or shell execution. Native admission and recovery must use the same lease and workspace ownership. **Proposed solution** Use the task driver from #17029. Resolve workspace and project identities without Core file operations. Connect native dispatch and restart recovery to that driver. **Alternatives considered** Keep workspace admission and dispatch in one change. A workspace change alone would leave dispatch on the previous transport. An execution change alone would leave Core staging files for the wrong host. ## What Changed - Enable plugin environment selection for native workspace sessions. - Resolve workspace metadata and authorized project references without Core checkout, copy, or provision operations. - Preserve agent-default and primary workspace policies. Use the pinned workspace provider for plugin-backed project storage. Reject unsupported Core service commands and cross-owner reuse. - Connect native execution to the prepared task driver. Keep assigned MCP authority on Core. - Save exact checkpoint identity, transfer selected account refreshes, and require termination proof before replacement or cleanup. - Complete managed plugin workspaces through their storage provider. Use the same check after a restart. Keep branch validation and leave task-owned files on the plugin. - Include default instruction content in runtime synchronization when no instruction directory exists. - Defer new-agent Copilot verification until the saved-agent lifecycle can acquire the agent-owned computer. ## Verification - Rebased onto `origin/master` at `898144f4b3`. Master advanced once during verification. Both manual conflicts joined import blocks: one in `native-session-executor.ts`, and one in `heartbeat/runtime-execution.ts`. The seven patches retain their logic. The first rebased head passed repository typecheck, build, all 862 focused execution/finalizer cases, and all CI gates. The latest head `93e5a72892` passes server typecheck and all 880 focused execution, finalization, and execution-control notification tests. All 54 CI/review checks pass, with two expected skips. Greptile ran automatically and scored this head 5/5 with no unresolved findings. No new review or Apex request was made for these import-only resolutions. - The successful CI workflow initially left three stale job-check reports. Targeted policy and summary reruns cleared the reports. No source change or test fix was needed. - Repository typecheck and build pass: `pnpm -r typecheck` and `pnpm build`. - Focused native execution and restart tests pass: 795 cases. The account and harness suites pass: 159 cases. Workspace admission passes: 48 cases. The SDK task lifecycle, workspace API, runtime-hook, and instruction checks also pass. - The heartbeat integration case uses the merged workspace provider contract and reaches native dispatch with no project files on Core. The new-agent case confirms that creation leaves the agent in `preparing` until lifecycle verification. - Map validation and the Copilot source-identity check pass. No Runner implementation or source-identity bump is in this PR. - The first rebased workspace run found two stale path assertions and a recovery guard that did not recognize task-owned targets. These are fixed. The storage integration fixture also needed its persisted source binding; the production authorization check stays in place. - Two prior review findings required rejection of Core service startup and cross-owner workspace reuse. The existing admission fixtures now check both failures. The installed-account authentication finding was fixed in merged #17029. - Tests reuse the current fixtures. No new provider test matrix or live Runner process is added for these fixes. - The prior head `2db273ec12` passed CI on its first attempt and reached standard Greptile 5/5. Apex scored 4/5 and found local-folder checks during plugin completion and recovery. - The completion fix reproduces both failures. It passes all 110 cases in the runtime execution, workspace finalizer, and workspace provider integration suites. Server typecheck and map checks pass. CI passed on `62d8335b4e` on its first attempt, and standard Greptile scored 5/5. The next Apex review scored 4/5. It found that a failed plugin inspection could abort restart recovery. - The restart fix records a failed workspace receipt and uses the current run retry policy. A regression first reproduced the aborted pass. It now checks retry state, retained results, and completion of another workspace in the same pass. All 68 finalizer cases and server typecheck pass. CI passed on `7d4016323b` on its first attempt. Standard Greptile and Apex both scored 5/5. Apex left one P2 about lost failure codes. - The diagnostic fix keeps the known branch and owner mismatch codes in receipts and run recovery state. It gives repair guidance for those codes and keeps unknown plugin errors generic. All 96 affected finalizer cases and server typecheck pass. Before the rebase, required CI passed on `c2fa61c6c8` on its first attempt. Standard Greptile and Apex both scored 5/5 on that head, with no unresolved findings. The new head retains these reviewed patches. - The earlier Superagent scan on `c2fa61c6c8` was inconclusive after a pinned merge-base checkout failed or timed out. The current scan on `93e5a72892` passes. - The exact rebased head passes all 48 workspace cases and server typecheck. The first local test launch could not bind PostgreSQL inside the sandbox. The authorized local-server run passed. - The prior head reached standard Greptile 5/5. Its first CI run hit an unchanged Runner cleanup timeout and an external build-runner shutdown. Both jobs passed on the single retry. The timeout cause is not established. - A local regression check found that a retained Core workspace could also enter a plugin task without file transfer. Core now rejects this move before provider work. The existing ledger test covers both directions and confirms that the source files stay intact. - The first active standard review scored 4/5. It found that a fresh plugin retry could enter Core Git preparation. This mode now uses the existing unsupported-storage error before workspace resolution or acquisition. Both unsupported-mode cases pass, and server typecheck passes. - No live Cloud or provider validation was run. ## Risks - The prerequisite #17029 is merged. This PR targets master. - Native plugin tasks support per-turn workspace sessions. Core service commands and fresh retries that import retained files remain unsupported. A workspace storage plugin can use the modes and strategies it declares. - Core-managed credentials support Codex and direct Claude accounts. Other managed credential routes fail closed. Other harnesses can use an installed account. - A provider must preserve its prepared configuration and report process termination correctly. Uncertain termination keeps the task pending cleanup. - Cloud deployment and live provider validation remain separate. ## Model Used OpenAI Codex, GPT-6. The exact model ID and context window are not exposed in this session. Tool use includes code editing, local tests, and the GitHub CLI. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal issue details - [x] I have run focused tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |