Commit Graph
312 Commits
Author SHA1 Message Date
Devin FoleyandPaperclip 21bf7ebcee feat(connectors): connect Tailscale networks to SSH environments (#20617)
## Thinking Path

> - Paperclip manages work for AI agents.
> - SSH environments run tasks on computers that users own.
> - A Paperclip server needs network access before it can reach a
private computer.
> - A reusable Tailscale connection belongs in Connectors.
> - Connected machines should become choices in the existing SSH
environment form.
> - This change connects a tailnet and resolves selected machines
through a private network client.
> - Agents retain their choice of harness and AI account.

## Linked Issues or Issue Description

**Subsystem affected**

Connectors, environment settings, SSH execution, company secret storage,
and runtime packaging.

**Problem or motivation**

Entering a Tailscale address in the SSH form does not connect the
Paperclip server to that tailnet. Requiring a separate device key for
every environment also duplicates network setup.

**Proposed solution**

Add a Tailscale Connector with **Connect Tailscale Network**. Use
brokered OAuth consent to enroll one private network client per
connection. Let SSH environments select a connected network and stable
machine identity. Preserve the enrolled node state through the company
vault so a restart does not reuse the one-time token.

**Alternatives considered**

A separate Tailscale environment type would duplicate SSH behavior and
mix network access with harness selection. This change uses the existing
`ssh` driver. Related #15648 adds OAuth client credentials and a health
check; this change uses user consent and supplies outbound task
execution. Related #11743 concerns HTTPS brokering.

**Roadmap alignment**

This extends the existing Connected Apps, environment, and Personal &
Shared AI Accounts capabilities. This PR is stacked on #21248, which
provides SSH account selection independently of Tailscale. The hosted
OAuth broker is a companion control-plane change.

## What Changed

- Preserve Cloud reauthentication in the current browser tab. Explain
canceled or failed consent in the connector and offer a retry of the
same authorized saved connection.
- Record Cloud enrollment starts initiated through the Tailscale
connector in the company activity log.
- Check install-preparation descendant liveness through the kernel
process table in the Runner support test. Keep the direct child’s strict
reaping assertion and reject any non-zombie descendant.
- Align the application image stage contract with the Tailscale client
stage after the upstream Runner-image refactor. Keep the exact stage
list and check every application stage, including `tailscale-client`,
for prohibited Runner compiler commands.
- Group Tailscale consent, discovery, daemon lifecycle, pairing, and
deletion under `server/src/services/connectors/tailscale/`, with their
service tests. Update existing route, runtime, environment, vault, and
company-deletion imports. Keep the existing contracts, authorization,
audit, and transaction owners.
- Handle early stdin closure in the Kubernetes real-shell test helper.
Stop the input stream on `EPIPE`, preserve nonzero child exits, and
reject a zero exit when input was lost. Add two real-process regression
cases. Production transfer code and confinement checks are unchanged.
- Read configured Tailscale executable paths when a client starts, after
supported `.env` loading. Keep explicit command overrides available.
- Discard pending pairing results after switching to Direct SSH or
closing the form. Retain the existing company, connection, machine, and
port checks.
- Store network credentials with an immutable server-owned purpose from
their first vault row. Exclude them from generic catalogs, bindings,
environment projection, and value reads. Use a dedicated audited server
consumer for pinned connector reads. Block unclassified legacy
serialized node state at the same value boundary. Keep SSH private-key
binding separate from connector credentials.
- Gate network queries and setup on verified local-implicit or
instance-admin access. Hide Tailscale catalog rows for other members,
show an admin requirement on direct setup navigation, and avoid device
discovery before management access is known.
- Keep failed-client cleanup in the network manager. A failed request
cannot disconnect a newer retry for the same identity; explicit
connector disconnect and enrollment retirement still stop their
networks.
- Check for readable Ed25519, ECDSA, or RSA host public keys before
pairing edits `authorized_keys`. Print a known-hosts entry for every
available key, or fail before changing authorized keys if none are
available.
- Return self-hosted Cloud enrollment to the Tailscale connector. Save
the unprefixed setup path and let the shared callback add the verified
company prefix. Keep external and unrelated destinations rejected.
- Hide pairing instructions after an SSH port change. Ignore late
key-selection results for the old port. Operators prepare again for
matching known-hosts output.
- Retire connector references before deleting a company or its vault
credential. Stop company daemons after the database commit. Keep failed
secret retirement retryable and prevent late node-state writes from
restoring deleted connections. Discard retired state without blocking
behind the vault deletion lock.
- Reload a mismatched credential before replacing a ready daemon.
Recheck the active entry and manager generation after the read. An older
request cannot stop a newer healthy socket or cross shutdown.
- Pin connector credentials to the exact vault version and SHA-256
digest recorded by trusted enrollment and node-state writes. Ignore
generic secret rotations. Advance both fields with a compare-and-set
guard. Reject unpinned or tampered credentials before network startup.
- Save an explicit Direct SSH selection and clear old Tailscale fields
after config merging. Preserve the operator's entered direct host.
- Route all Tailscale SSH runtime access through the connection service
after SSH-key loading. Pin access to the environment's saved connector
identity and recheck after device discovery. Remove standalone key-based
daemon startup and reject raw enrollment keys or credential-only network
configurations. Align the shared config type and SDK pins.
- Correct two E2E assumptions exposed during connector qualification.
Keep the two-window chat fixture stale until its own mutation. Validate
the original failed run and its permitted write-revocation timestamp in
the legacy recovery browser test.
- Add the Tailscale Connector page, catalog entry, OAuth return flow,
connected-network management, device discovery, and disconnect. Document
all six routes in OpenAPI and the API reference, including their
instance-admin requirement.
- Add company-scoped connection storage in `0480_lazy_lethal_legion`,
trusted credential version/digest fields in `0481_round_pet_avengers`,
and server-owned secret purpose/backfill in
`0482_aberrant_rumiko_fujikawa`. Generate all three migrations through
the maintained command and retain the five newest snapshots.
- Add machine pairing that generates a fresh SSH key per request, stores
its private half in the company vault, and uses the generated public
half in the install command. Do not trust editable saved metadata.
Preserve existing authorized keys, including a file without a final
newline. Keep SSH username, workspace, and trusted host keys explicit.
- Use the existing SSH driver. Resolve the machine's current address
from its stable device ID. Reject disconnected, removed, offline, and
wrong-company selections.
- Hide Network when Direct is the only option. Select a sole connected
tailnet automatically and hide its selector. Remove the requested setup
helper copy. Do not restrict the harness to Claude.
- Run a private userspace Tailscale daemon with a separate socket.
Persist its node state before execution. Restore it after restart, bound
startup, serialize reconnect cleanup, and close it during server
shutdown.
- Guard OAuth completion with a durable attempt marker. Disconnect and
newer authorization invalidate older callbacks. Close replaced daemons
and remove cancelled enrollment secrets without changing newer
connection state.
- Preserve SSH host verification and tailnet-specific session identity.
Honor the configured Tailscale executable path, including spaces and
OpenSSH percent escaping.
- Package checksum-pinned Tailscale clients and their license in the
standard runtime image. Exercise the client build stage in PR CI and
include it in image smoke checks.

## Verification

- Merge status: all CI and image checks pass, and current-head Apex is
5/5 with no unresolved comments. GitHub rejected a direct merge because
the active master ruleset requires code-owner approval for
`.github/workflows/pr-trusted.yml`. The [CODEOWNERS workflow
pattern](https://github.com/paperclipai/paperclip/blob/master/.github/CODEOWNERS#L3)
applies; eligible maintainers already have review requests. Auto-merge
is enabled with the squash method for the verified head. The PR remains
open pending that required approval; no branch policy was bypassed.
- Final revision: `fdb29b2cc366f4945b7f4c3e2b2d25070329a21b`, including
master `999709b0a6dd416af335b5debded13e1de7fcb7a`. [Final CI
verification
passed](https://github.com/paperclipai/paperclip/actions/runs/43518584814/job/132213508320).
All required checks pass, all applicable test/build gates are green, and
the branch is mergeable against master. SSH prerequisite #21248 and
Cloud broker paperclipai/paperclip-cloud#1314 are already merged.
- [Apex reviewed this exact revision at
5/5](https://github.com/paperclipai/paperclip/pull/20617#issuecomment-6776629437).
There are no unresolved review threads. The PR is classified as Critical
impact.
- Final image qualification passes for the [application Runner
path](https://github.com/paperclipai/paperclip/actions/runs/43518584522/job/132206784267),
[amd64 Runner
image](https://github.com/paperclipai/paperclip/actions/runs/43518584544/job/132207296198),
and [arm64 Runner
image](https://github.com/paperclipai/paperclip/actions/runs/43518584544/job/132207296156).
Typechecks, Runner support, native/Rust tests, provider tests,
serialized-server suites, browser shards, and the remaining applicable
CI checks pass. This is source/image CI qualification, not a production
rollout.
- The last master conflict affected only Docker stage declarations and
their boundary test. Both the checksum-pinned Tailscale stage and
upstream `runner-lockfile` guard are retained, including the upstream
`deps` inheritance. The Tailscale client stage is byte-identical. All
180 cases across five Docker/image contract suites pass locally, along
with server and UI typechecks. The maintained 543-input SDK check has no
drift. Runtime and lock pins are unchanged. Secret/PII and whitespace
scans pass.
- Seven OAuth regression cases failed before the fixes. The current-tab
test uses the real handoff helper, checks that the session is stored
before reauthentication navigation, mounts the shared return page,
resumes consent, and verifies handoff cleanup. Callback tests check
trusted saved-connection context, fixed safe outcomes, failure audit
metadata, denied membership and expired-state rejection, JSON API
errors, and no provider text in browser returns. UI retries require a
connection in the current company's catalog. All 80 OAuth
route/service/UI/shared-handoff cases pass; server and UI source/test
typechecks pass.
- [Server shard 4/12 on the prior
refresh](https://github.com/paperclipai/paperclip/actions/runs/43510727031/job/132180301436)
passed 2,784 cases and skipped 21, then failed a reassignment fixture's
`TRUNCATE companies CASCADE` cleanup with PostgreSQL `40P01`. Its
automatic retry found the surviving `REA` prefix. The unchanged test
passes all three cases locally. A controlled reader/cleanup overlap
reproduces the same deadlock, but the exact competing CI query was not
logged. Each case now has a distinct company prefix and identifier; the
owned database is removed at suite end. The three test bodies and
assertions are byte-identical. Eight local reassignment/lock-release
cases pass, and the [final server shard 4/12
passes](https://github.com/paperclipai/paperclip/actions/runs/43518584814/job/132207159392).
Production reassignment logic, deadlines, and retries are unchanged.
- The initial master integration passed 330 cases across 17 suites.
Coverage includes connector state, device discovery, network lifecycle,
pairing, routes, SSH binding and exports, environment persistence, and
connector UI. All three unpublished migrations were regenerated after
master's `0479_short_darkstar`: `0480_lazy_lethal_legion`,
`0481_round_pet_avengers`, and `0482_aberrant_rumiko_fujikawa`. Each SQL
body is byte-identical to its earlier unpublished migration. Generated
snapshots preserve existing master tables and add only the intended
Tailscale table and secret-purpose column. Exactly five snapshots
remain, 0478–0482. Numbering, safety, compatibility, and snapshot-drift
checks pass. The later Docker refresh did not alter this migration
sequence.
- Earlier attempts remain part of the verification record. On `1c020a0`,
Rust cache restoration and server shard 5/12 were interrupted by
shutdown signals, Runner shard 2/3 lost communication with no
retrievable log, and an amd64 provider download failed with npm
`ECONNRESET`. Same-head retries recovered, without establishing the
cause of the communication loss. Later superseded runs also had
cancellations with incomplete logs. These interrupted attempts are not
counted as passes. The final revision has fresh successful CI.
- Local platform limits remain recorded. The Kubernetes suite on macOS
passed 191 cases, failed 12 Linux `/proc/self/fd` transfer cases, and
skipped four Linux-only cases. Unchanged source reproduced the same 12
failures. An initial run also lacked provider dependencies until the
maintained preparation command installed them. Those local attempts are
not green qualification. Final Linux provider CI passes. Local Docker
qualification was unavailable; final image checks passed in CI.
- Before PR separation, a live tailnet/SSH Claude Code task completed on
Ubuntu ARM64 without provider login for the remote user. The selected
Paperclip subscription ran it, wrote the expected file, posted
completion, and marked the task done. File contents and private runtime
cleanup were checked. This is prior live evidence; the final revision
was not live-retested. Other providers have automated coverage only.
Hosted registration for unrelated tailnets and production concurrency
remain qualification limits below.
- The Superagent security and supply-chain reports remain neutral and
inconclusive. Earlier attempts to restart the failed security scan
returned HTTP 404. Supply-chain intelligence lacked coverage for
extracted pinned dependencies and the shared workflow reference. These
reports do not establish successful security qualification. The required
CI and review gates pass; this external coverage gap remains disclosed.

## Risks

- The Superagent security and supply-chain reports are inconclusive. The
security scan re-run request returned HTTP 404, and the supply-chain
scanner lacks coverage for the extracted dependencies and shared
workflow reference. These neutral results are not successful security
qualification.
- The server-owned secret purpose is internal database metadata, not an
HTTP-writable field. The generic secret service owns enforcement so
direct API reads and runtime projection share the same restriction.
Previously copied node state remains valid outside Paperclip until the
enrolled device is revoked in Tailscale; a local disconnect cannot stop
an external copy.
- Tailscale network configurations now require a connected Connector and
selected machine. The unreleased key-only path is removed; any saved
configuration from that path needs Connector setup.
- The connector remains a pilot. The local test used an OAuth app
registered in the consenting user's tailnet. Qualify hosted app
registration for unrelated customer tailnets before general
availability.
- This implementation supports one serving process per instance.
Concurrent replicas need exclusive ownership of the persisted network
identity before production rollout. Preserve the database and company
vault master key across restarts.
- Each machine-pairing preparation creates a new vault secret. Unused
generated keys follow the existing secret deletion lifecycle. A saved
SSH key remains available as an explicit choice.
- Apply additive migrations `0480_lazy_lethal_legion`,
`0481_round_pet_avengers`, and `0482_aberrant_rumiko_fujikawa` before
application rollout. The new credential version and digest columns are
nullable. Existing unpinned connections require fresh OAuth consent; the
migration deliberately does not trust or backfill a generic secret
value. Regenerate or renumber migrations if master advances the sequence
before merge.
- The target needs an SSH server, its chosen harness, and allowed
tailnet access. A reachable network does not grant SSH or provider
authorization. Pairing reads standard `/etc/ssh` host-public-key files;
custom host-key locations require manual known-hosts configuration.
- Roll back the application revision to remove new entry points.
Existing connection secrets and registered Tailscale nodes retain their
separate lifecycle. Do not drop the table or delete node state as part
of an application rollback.
- Production qualification still needs denied-policy, revocation,
cancellation, and deployment-overlap checks. Local Docker and a
production deployment were not performed.

## Model Used

OpenAI Codex, gpt-6.1-sol, xhigh reasoning, with local code execution
and tool use.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used
- [x] I have checked ROADMAP.md and this extends existing connector and
environment features
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have described the issue in-PR using the feature request fields
- [x] I have not referenced internal issues, local URLs, or private
machine identifiers
- [x] My branch name describes the change
- [x] I have run focused tests locally and they pass
- [x] I have added tests for new behavior and failure paths
- [x] I have updated documentation
- [x] I have documented migration and runtime risks
- [x] All Paperclip CI gates are green
- [ ] Required code-owner approval received for the workflow change
- [x] Greptile Apex is 5/5 on the final head with no unresolved comments
- [x] I will address CI and review findings before requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-13 15:15:43 -08:00
Nicky LeachandClaude Opus 5.5 5df1ada9c1 feat(plugins): send configured environment on plugin task resume (#21352)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - #21335 lets a plugin task provider that restarts the Runner on
`resume` opt in (`resumeCredentialVersion: 1`) to receive the claimant
run's harness credential
> - A provider restarting the Runner also needs the agent's configured
environment, which Core sends on `resume` only during a session handoff
> - Without it, a Runner a provider moves to a new computer starts
without the agent's configured environment variables
> - This pull request widens that opt-in to both launch inputs, under a
name that says so, before any provider adopts the old name
> - The benefit is that a provider can restart a parked Runner on resume
with the same credential and environment a `submit` would carry

## Linked Issues or Issue Description

Refs #21289
Follows: #21335

**What existing behavior does this improve?**
The opt-in resume launch inputs added in #21335.

**Subsystem affected**
Plugin SDK environment task schema (`describe` capability), the plugin
Runner task owner's `resume`, and the plugin environment task docs.

**Current behavior**
With `resumeCredentialVersion: 1`, `resume` carries `credential`.
`configuredEnvironment` is sent on `resume` only for a session handoff.

**Proposed behavior**
The capability is renamed `resumeLaunchInputsVersion: 1`. With it,
`resume` carries `credential` as before and also `configuredEnvironment`
(same builder as `submit`) when the provider advertises
`configuredEnvironmentVersion: 1`. Session handoff is unchanged.

**Reason and benefit**
A provider that restarts the Runner on resume gets both launch inputs it
needs.

**Breaking changes**
The capability name changes from `resumeCredentialVersion` to
`resumeLaunchInputsVersion`. No provider advertises the old name yet (it
merged in #21335 and has no adopter), so nothing changes behavior.

## What Changed

- Rename `resumeCredentialVersion` → `resumeLaunchInputsVersion` in the
SDK `describe` result schema, executor plumbing, owner input, docs and
tests.
- With the capability, `resume` also adds `configuredEnvironment` when
the provider advertises `configuredEnvironmentVersion: 1`. A normal
resume does not mark the variables as delivered, since a live Runner
keeps the environment it started with.
- Docs: the "Release and resume" section describes both fields; the
"Configured environment" section points to it.
- The runner-e2e SDK public-input pin is updated.

## Verification

- SDK vitest `tests/environment-task-release-resume.test.ts` and
`tests/environment-tasks.test.ts`: 68 passed.
- Server vitest `plugin-runner-task.test.ts` and
`environment-task-runtime.test.ts`: 153 passed, including a normal
resume with the capability (credential sent, variables added only with
`configuredEnvironmentVersion: 1`, delivered flag unset) and redaction
of both fields from metadata writes and sanitized errors.
- `native-session-executor.test.ts -t "uses a task-only"`: 5 passed.
- runner-e2e `source-sdk-evidence.test.ts` and
`source-sdk-public-type.test.ts`: 46 passed.
- `pnpm typecheck` in `server/` (builds shared and plugin-sdk) passed.
- Not run locally: the full suite and `pnpm verify`; required CI covers
them.

## Risks

- A provider restarting the Runner gets the claimant run's current
variables, which can differ from the ones the parked Runner started
with. That is intended.
- Rollback: revert this pull request. No migration.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, test runs), implemented by a sub-agent and reviewed
before opening.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 13:46:37 -08:00
ef2a49b6df fix(daytona): bound the duplex channel close so a stalled kill cannot retire the worker (#21471)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Cloud agent runs use the Daytona sandbox provider plugin. One shared
plugin worker process serves all Daytona calls on a server.
> - The host closes a duplex channel through the worker. The host waits
10 seconds for the close acknowledgement. When the acknowledgement is
late, the host retires the whole worker with SIGKILL.
> - The worker close waits for the remote PTY kill before it releases
the socket. The Daytona SDK sends that kill as a toolbox request with a
24-hour default timeout. A stalled sandbox can hold it open for longer
than 10 seconds.
> - Then the host kills the shared worker. Every other call in flight on
that worker fails with `Worker process exited (code=null,
signal=SIGKILL)`. This includes other runs' workspace preparation and
restore.
> - This pull request limits the wait for the remote kill to 5 seconds.
Then the worker releases the socket and acknowledges the close.
> - The benefit is that one slow sandbox no longer stops unrelated runs
on the same server.

## Linked Issues or Issue Description

**What happened?**
On a hosted staging server, the host logged `duplex channel close not
acknowledged; retiring worker` six times in about 90 minutes. Each time,
the Daytona worker exited with SIGKILL. Each time, all in-flight calls
on that worker failed. Runs in workspace preparation failed with
`setup_failed` and `Worker process exited (code=null, signal=SIGKILL)`.
Several runs failed together in each burst.

**Expected behavior**
A slow close of one duplex channel does not stop calls that belong to
other channels or other runs.

**Steps to reproduce**
1. Open a duplex channel on a Daytona lease.
2. Make the Daytona PTY kill request stall (the SDK default request
timeout is 24 hours).
3. Close the channel from the host. The worker does not acknowledge
within 10 seconds.
4. The host retires the worker with SIGKILL. All other pending calls
fail.

**Relevant logs or output**
```
info duplex channel close not acknowledged; retiring worker
info retired worker process exited  code=null signal=SIGKILL willRestart=true
JsonRpcCallError: Worker process exited (code=null, signal=SIGKILL)
```

**Additional context**
The login PTY close does not have this problem. It only does the local
socket disconnect. A kill that rejects already gets a close
acknowledgement, because the session close ignores kill errors. This
change gives a slow kill the same result.

No related public PR exists. I searched open and closed PRs for this
failure.

## What Changed

- `duplex-command-stream.ts`: `close()` waits for the remote `kill()`
for at most `DUPLEX_CHANNEL_CLOSE_KILL_TIMEOUT_MS` (5 seconds). Then it
always releases the socket with `disconnect()`. The kill continues in
the background. A kill that answers in time keeps the old order: kill,
then disconnect, then the acknowledgement.
- `duplex-close-deadline.test.ts` (new): regression tests at the session
close and at the worker `onDuplexChannelClose` boundary, and a check
that the deadline stays well inside the host deadline.

## Verification

- Red before the fix: with the old `duplex-command-stream.ts`,
`acknowledges a bound close inside the host deadline when the remote
kill never settles` fails (no acknowledgement after 9 seconds), and the
session close test times out.
- Green after the fix: `pnpm --dir
packages/plugins/sandbox-providers/daytona exec vitest run --config
vitest.config.ts src/duplex-close-deadline.test.ts
src/duplex-command-stream.test.ts src/plugin.test.ts` passes 453 of 453.
This includes the existing tests `joins pending teardown for concurrent
closes while all channel authority stays revoked` and `does not turn a
rejected retired close into an acknowledgement`.
- The safety case `still waits for a kill that answers inside the
deadline before it releases the socket` passes before and after the fix.
- `tsc --noEmit` for the Daytona plugin passes.
- In the full Daytona plugin suite, two GNU tar tests in
`file-sync.test.ts` fail in my sandbox. They also fail on unchanged
`master` in the same sandbox, so they are environment failures. CI
covers them.
- I did not run a live staging workflow. After the deploy, the staging
logs must show no new `duplex channel close not acknowledged; retiring
worker` lines from slow kills.

## Risks

- Low risk. The change is in one Daytona plugin function.
- After a kill timeout, the gateway process can stay alive in the
sandbox until the background kill completes or the lease teardown stops
the sandbox. The socket is closed, so no bytes can move on the channel.
The route stays retired, so the host route cannot open again. This is
the same state as when a kill rejects today.
- The host contract does not change. A malformed or mismatched
acknowledgement still retires the worker.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com>
2026-11-13 12:17:05 -08:00
Nicky LeachandClaude Opus 5.5 41998d551c feat(plugins): send the claimant's credential on plugin task resume (#21335)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Plugin environment drivers run agent tasks on remote computers, and
a warm follow-up run resumes a parked Runner through the `resume` task
operation
> - `resume` carries only the durable identity and a bootstrap ticket,
which is enough while the parked Runner process is still alive
> - A provider sometimes has to start the Runner again to resume the
session: after it moved the parked session to a new sandbox, or after
the computer rebooted
> - A Claude Runner cannot start without its credential, and today only
the portable handoff path (`handoffCredential`) delivers one
> - This pull request lets a provider opt in to receiving the claimant
run's harness credential on `resume`
> - The benefit is that a provider can continue a parked session on a
new computer without Core orchestrating a handoff

## Linked Issues or Issue Description

Refs #21289
Related: #21305

**What existing behavior does this improve?**
Warm resume for plugin task providers that restart the Runner to resume
a session.

**Subsystem affected**
Plugin SDK environment task schema, the server's plugin Runner task
owner, and the plugin environment task docs.

**Current behavior**
`submit` can carry the harness credential (`credential`), but `resume`
cannot. A provider that must start the Runner again on resume has no
credential for a Claude task.

**Proposed behavior**
A provider that returns `resumeCredentialVersion: 1` from `describe`
receives `credential` on `resume` whenever Core would send it on
`submit`, resolved for the claimant run. Other providers see no change.

**Reason and benefit**
Providers can restart a Claude Runner on resume without the Core-run
handoff protocol.

**Breaking changes**
None. The field is opt-in: the SDK worker parses operations strictly, so
Core sends it only to providers that advertise the capability.

## What Changed

- `resume` gains an optional `credential`, with the same schema and
validation as `submit`'s.
- `describe` gains an optional `resumeCredentialVersion: 1` capability.
- The plugin Runner task owner adds the credential on `resume` when the
provider advertised the capability and Core resolved a credential, using
the same resolution as `submit` (`preparePluginRunnerCredentials`).
After a warm rebind, the claimant run's credential is sent.
- `doc/plugins/PLUGIN_ENVIRONMENT_TASKS.md` documents the field and its
handling rules.
- The runner-e2e SDK public-input pin is updated.

`handoffCredential` on the session handoff path is unchanged.

## Verification

- SDK vitest `tests/environment-task-release-resume.test.ts` and
`tests/environment-tasks.test.ts` (68 passed): `resume` accepts and
rejects the same 11 credential cases as `submit`; the capability is
optional and rejects other versions; a resume credential reaches the
plugin through the serialized worker RPC.
- Server vitest `plugin-runner-task.test.ts` and
`environment-task-runtime.test.ts` (151 passed): credential sent with
the capability and a resolved credential; omitted without a credential
or without the capability; the rebound claimant's credential is sent;
the credential appears in no metadata write and not in the sanitized
error, even when the worker error echoes it.
- `native-session-executor.test.ts -t "uses a task-only"` (5 passed):
the executor forwards the capability to the owner.
- runner-e2e `source-sdk-evidence.test.ts` and
`source-sdk-public-type.test.ts` (46 passed).
- `tsc --noEmit` for `packages/plugins/sdk` and `server`.
- Not run locally: the full suite and `pnpm verify`; required CI covers
them.

## Risks

- No server-side provider check on resume, unlike `submit`: the lease
has no reliable harness at resume time. Protection relies on the owner,
whose rebind refuses a changed harness, account or credential identity.
- The credential is the claimant run's current one, which can be newer
than the one that started the parked Runner. That is intended.
- A provider must advertise the capability only after its SDK pin
includes this schema; roll Core out first.
- Rollback: revert this pull request. No migration.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, test runs), implemented by a sub-agent and reviewed
before opening.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 08:40:46 +00:00
Nicky LeachandClaude Opus 5.5 2483cb46c2 feat(plugins): deliver configured environment to plugin Runner tasks (#21280)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - An agent gets environment variables from its execution environment,
its own configuration, its project, and its routine. Secrets are
resolved for each run.
> - A plugin environment can run the agent as a native Runner task on
its own host, through the environment task contract.
> - That contract has no field for the agent's variables. Runs on a
plugin environment get only the variables that the plugin sets itself.
> - This pull request adds a versioned contract field for the values,
sends them on a session handoff, lets Codex commands inherit the
variables in an external sandbox, and lets a plugin protect the
variables that it sets itself.
> - The benefit is that plain and secret variables work on plugin
environments in the same way as on local environments.

## Linked Issues or Issue Description

Refs #21233. That pull request by @lucktastic finds the same gap and
proposes the same capability and field names
(`configuredEnvironmentVersion: 1`, `configuredEnvironment`). This pull
request uses the same names, so providers implement one contract. It
differs in these places:

- Codex commands get the variables through an external-sandbox denylist,
not through key names sent at `prepare`. A retained Runner does not need
a new preparation when a key is added or removed.
- A session handoff `resume` gets the values, because it starts a new
Runner on another computer. Value changes on the same computer need no
new field: Core's process reuse key includes the values, so Core starts
a new task.
- A driver can declare `protectedEnvironmentKeys`. Paperclip rejects or
warns when one is saved.
- The run log names skipped variables only for an older provider.

Thank you to @lucktastic for the diagnosis and the contract shape.

**What happened?**
An agent has environment variables, plain and secret. Its runs on a
plugin task environment do not get them. `PAPERCLIP_CONFIGURED_ENV_KEYS`
in the run lists only the variables that the plugin sets.

**Expected behavior**
Each run gets the agent's configured variables.

**Steps to reproduce**
1. Set a plain and a secret environment variable on an agent.
2. Assign the agent to a plugin environment that provides Runner tasks.
3. Start a run. Read the names in `PAPERCLIP_CONFIGURED_ENV_KEYS` in a
command that the agent runs.
4. The agent's names are not in the list.

## What Changed

- **SDK contract** (`packages/plugins/sdk/src/environment-tasks.ts`):
- A provider advertises `configuredEnvironmentVersion: 1` on `describe`.
- `submit` gets `configuredEnvironment`. `resume` gets it only for a
session handoff.
- Names, count, and sizes have the same limits as the Runner projection.
- **Server**:
- `createRunnerdBackendWithinSessionClaim` gives the run's configured
projection to the plugin task owner. The projection has no provider
credentials and no generated runtime keys.
- `createPluginRunnerTaskOwner` sends the fields only to a provider with
the capability. An older provider gets no new field, and the run log
names the variables that were not set. The log never prints values.
- **Codex** (`codex-security-config.ts`):
- An externally sandboxed launch uses `inherit="all"`,
`ignore_default_excludes=true`, and an `exclude` list of Codex's own
credentials, instead of an `include_only` list fixed at preparation
time.
- A test reads the Runner's Codex environment keys, and fails if a new
key is not classified.
  - Host launches keep the bounded allowlist.
- **runnerd** (`codex_provider.rs`): `runtime_notification_codex_args`
still grants the notification files to read-only profiles when no
allowlist is present. Plan mode in an external sandbox needs these
grants.
- **Protected keys**:
- An environment driver can declare `protectedEnvironmentKeys` in its
manifest.
- Paperclip rejects a protected key on an agent or environment that uses
the driver. The 422 has code `protected_environment_variable`.
- On a project, a routine, or another agent, Paperclip saves the key.
The response has a `configurationWarnings` notice that names the
environments. The project and routine pages show the notice as a toast.
The agent pages already show configuration warnings.
- **Docs**: `doc/plugins/PLUGIN_ENVIRONMENT_TASKS.md` describes the
provider obligations.
- `tests/runner-e2e/source-sdk-public-inputs.json`: refresh the reviewed
SDK source pin.

## Verification

- `packages/plugins/sdk`: `vitest run tests/environment-tasks.test.ts`
(38 pass). Accepts bounded values. Rejects bad names, NUL bytes, too
many variables, and oversized values.
- `server`: `vitest run
src/services/native-runtime/plugin-runner-task.test.ts` (95 pass).
Values go only to a provider with the capability, never in `prepare`.
`resume` sends values only for a handoff.
- `server`: `vitest run src/services/protected-environment-keys.test.ts
src/__tests__/project-routes-env.test.ts` (10 pass). Rejection, notices,
and the project route response.
- `server`: all route suites that import the project, routine,
environment, or agent routes (45 files, 1689 pass).
- `packages/shared`: `vitest run src/validators/plugin.test.ts` (64
pass).
- `packages/paperclip-runner`: `vitest run src/drivers/codex
src/configured-environment.test.ts
src/live/runner-provider-launch.test.ts` (444 pass).
- `packages/paperclip-runner/runner`: `cargo test -p
paperclip-runner-core --lib runtime_notification` (9 pass, 2 ignored as
before).
- `pnpm typecheck` in `packages/shared`, `packages/plugins/sdk`,
`packages/paperclip-runner`, `server`, and `ui`.
- Not run locally: `pnpm verify --full` and browser suites. Required CI
covers the final revision.

## Risks

- **Existing providers:** low risk. Core sends no new field to a
provider without the capability. Their runs behave as before, with one
added log line when variables were skipped.
- **Codex in an external sandbox:** commands now see every variable in
the Runner environment except Codex's own credentials. The execution
environment builds that environment from an allowlist. If the Runner
gives Codex a new credential, the drift test fails until the key is
classified.
- **Saving env:** new rejections happen only when an installed driver
declares `protectedEnvironmentKeys`. No driver declares them today.
- **Providers:** they must keep the values transient and never save or
log them. The doc lists these obligations.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 08:11:23 +00:00
Nicky LeachandClaude Opus 5.5 4f4d823cfc feat(environments): forward the warm idle window to plugin lease acquisition (#21305)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Plugin environment drivers run agent tasks on remote computers, and
warm leases keep a parked Runner between runs for an idle window that
Core records
> - Some providers stop an idle computer on their own schedule (for
example a sandbox auto-stop interval)
> - If that provider timer is longer than the Runner idle window, the
provider keeps paying for a computer whose Runner is already gone; if it
is shorter, a warm resume can find the computer stopped
> - The plugin cannot align the two because it never sees the idle
window at acquire
> - This pull request forwards the warm idle window to
`environmentAcquireLease` as `runnerLifecyclePolicy`
> - The benefit is that a provider can stop its computer just after the
Runner idle window, without Core knowing anything about the provider's
placement

## Linked Issues or Issue Description

Refs #21289
Refs #21066

**What existing behavior does this improve?**
Warm lease setup for plugin environment drivers.

**Subsystem affected**
Server environment runtime (`createPluginEnvironmentDriver`) and the
plugin SDK acquire parameters.

**Current behavior**
Core records the warm lifecycle policy (`runnerTaskLifecyclePolicy`,
including `idleTimeoutMs`) on the lease, but `environmentAcquireLease`
does not receive it, so a provider with its own idle shutdown has to
guess.

**Proposed behavior**
For a warm lease, `environmentAcquireLease` receives
`runnerLifecyclePolicy: { mode: "warm", idleTimeoutMs }`, the same value
Core records. Other lifecycle modes do not receive the field.

**Reason and benefit**
A provider can align its idle shutdown with the Runner idle window.

**Breaking changes**
None. The field is optional and additive.

## What Changed

- `PluginEnvironmentAcquireLeaseParams` has a new optional
`runnerLifecyclePolicy`.
- Core sends it only for warm leases.
- `doc/plugins/PLUGIN_RUNNER_EXECUTION.md` documents the field.
- The runner-e2e SDK public-input pin is updated for the SDK change.

An earlier revision of this PR also made Core try
`environmentResumeLease` before a session handoff. That part is removed:
placement is the plugin's decision, so Core should not decide to retry
the same computer before moving the session. The plugin will own that
inside its own resume instead.

## Verification

- `pnpm typecheck` in `server/` (builds shared and plugin-sdk) passed.
- `vitest run src/__tests__/environment-runtime.test.ts -t "portable
task placement|delegates plugin environment leases with acquisition
budget"` passed (5 cases). The portable test asserts the first warm
acquire receives `runnerLifecyclePolicy` and the lease records
`runnerTaskLifecyclePolicy`; the non-warm delegation test asserts the
field is absent. Making the forwarding unconditional fails both
delegation cases.
- The full suite was not run locally; required CI covers it.

## Risks

- A plugin that does not know the field ignores it.
- Rollback: revert this pull request. No migration.

Note: the branch name still reflects the earlier scope (resume before
handoff).

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 23:45:31 -08:00
Nicky LeachandPaperclip 7f69ef8b44 Prepare and finalize managed workspaces on the active Launcher (#20743)
## Thinking Path

> - Paperclip lets plugins own managed Git workspaces.
> - A portable Runner task can move to a different physical allocation.
> - The checkout must be present on the allocation that launches the
Runner.
> - Workspace preparation and finalization need exact placement when
they retry.
> - This change checks the placement at Runner admission and records it
before finalization.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Workspace preparation and finalization for portable Runner tasks.

**Current behavior**
A task replacement can change the allocation after Git preparation. A
lost finalization reply can leave retry without the original placement.

**Proposed behavior**
Prepare the checkout on the actual allocation before admission. Record
finalization placement so an exact retry can use it.

Portable ownership and controller recovery are in
[#21066](https://github.com/paperclipai/paperclip/pull/21066).

## What Changed

- Reserve the managed workspace before acquiring the portable task
lease.
- Pass the task placement to the workspace provider. Reuse the described
task while placement stays the same and prepare again before admitting a
replacement.
- Record the initial preparation's run, agent, generation, and optional
task placement before dispatch. Inspection and retries on the same
Launcher replay the same request; VM and sandbox preparation use the
same recovery path.
- Keep provider request identity separate from Git credential authority:
reservation reconciliation uses the existing project-consent path in a
host-only scope bound to the plugin and workspace. Foreground
preparation still requires an active run; consent, repository, and
revocation checks are unchanged.
- Acquire the current Launcher before retrying preparation. If placement
changes, use ordinary preparation on the replacement so the provider can
settle the prior writer and recover the retained checkout. No additional
provider discovery call is added.
- Select finalization placement before publishing the operation record,
then dispatch and recover using that saved choice. Retain the Launcher
during bounded finalization retries.
- Preserve interrupted pre-upgrade finalization requests without
placement, including their original operation IDs and provider routing.
No migration or new durable mode flag is required.

## Verification

- Final revision `74ea3e6712`: server typecheck and exact SDK source-pin
verification (538 inputs) passed.
- All 185 tests passed across the full plugin worker manager, project
Git authorization, and plugin workspace integration suites (including 56
workspace tests). Independent verification reran the three lost-reply
cases: inspection, same-Launcher retry, and replacement after deleting
the original Launcher. The tests assert the correct request identity,
durable intent, and clearing that intent after success.
- Before the final retry-accounting fix, all 323 tests passed across
workspace admission, runtime execution, native workspace finalization,
plugin workspace integration, and terminal cleanup retry. Regressions
exercise crashes after operation insertion but before dispatch in both
finalization paths, exact replay after lease release, and retry after
ambiguous placement clears without publishing an incomplete operation.
Pre-upgrade interrupted requests retain their operation IDs despite an
available portable lease.
- Final retry-accounting fix: all 84 native workspace finalizer,
failure, and supersession tests passed. A real missing-plugin regression
first failed on the prior commit, then verified healthy-run progress,
three-attempt exhaustion, terminal cleanup, and absence of a misleading
workspace operation. Independent verification also passed the legacy and
crash-before-dispatch cases.
- Worker-process tests cover both preparation RPC methods: the provider
still receives the original run and agent, maintenance receives
project-scoped credential authority, and foreground or mismatched-scope
requests remain denied. Database tests confirm a stopped run cannot
authorize foreground Git access while valid project consent can
authorize maintenance.
- Existing metadata protection coverage proves callers cannot overwrite
the private preparation intent or read it through workspace responses.
- Two existing Launcher Git tests passed: retry preserves dirty files
and the original base commit without fetching, and retry resumes the
saved checkout intent while rejecting a changed pending branch.
- Implementation was delegated to GPT-6.1 Sol and independently
reviewed. The GitHub stack's automatic rebase preserved the original
workspace-placement patch.

- [Full
CI](https://github.com/paperclipai/paperclip/actions/runs/43293046205/job/131452117706)
passed on `74ea3e6712`, including server and workspace tests, Runner,
end-to-end tests, typecheck, build, and canary dry run. Ordinary
Greptile and
[Apex](https://github.com/paperclipai/paperclip/pull/20743#issuecomment-6788920388)
both scored `74ea3e6712` 5/5, with no unresolved findings.

## Risks

- Live Daytona end-to-end qualification is pending. The checkout record
persists on the mounted volume; replacement preparation relies on the
provider confirming the previous writer is settled before using that
retained checkout. Local tests cover Core request routing and Launcher's
durable checkout recovery separately.

## Model Used

- OpenAI Codex GPT-6 with GPT-6.1 Sol agents for implementation and
independent review.

## Checklist

- [x] Thinking path and model are included.
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] The issue is described using the enhancement template.
- [x] Local checks passed and new behavior has tests.
- [x] All Paperclip CI gates are green.
- [x] Greptile is 5/5 with no open findings.

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-12 12:12:41 -08:00
Nicky LeachandPaperclip a924d2e2c7 Persist portable Runner ownership through controller recovery (#21066)
## Thinking Path

> - Paperclip delegates remote Runner execution to environment plugins.
> - A plugin can move a session to another task allocation.
> - The Core controller can lose an ownership reply or restart before
the Runner starts.
> - The source and target need a durable record of the same request.
> - This change binds restart and attachment to that request and the
original unlaunched task lease.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Recovery of plugin Runner sessions on replacement compute.

**Current behavior**
Core cannot replay a session ownership request from saved lease state
during startup.

**Proposed behavior**
Record the operation on both leases and check the matched ownership
receipt before attaching. Preserve the original pre-admission lease
across controller restart.

Managed environment configuration is in
[#21065](https://github.com/paperclipai/paperclip/pull/21065). Related
warm sandbox cleanup is in
[#20065](https://github.com/paperclipai/paperclip/pull/20065). This
layer preserves the same-sandbox fallback from
[#21001](https://github.com/paperclipai/paperclip/pull/21001). Workspace
placement follows in
[#20743](https://github.com/paperclipai/paperclip/pull/20743).

## What Changed

- Add the optional `sessionHandoffVersion` capability and the ownership
task operation.
- Save matching ownership claims and fence source checkpoint access.
Replay a pending request on the original target.
- Restore a portable session using the durable claim and the current
credentials for the same account. Start a fresh conversation when the
provider reports an unavailable checkpoint, or when Core's persisted
compatibility decision selects a different session after transfer. The
latter requires an unlaunched target; subsequent recovery must match the
fresh task's saved identity. Both paths retain the workspace transfer
and source fencing.
- Reuse the original unlaunched task lease after hard or graceful
controller restart. After a move has attached, validate the surviving
Runner through the saved claim rather than requiring its logical lease
ID to equal the new physical lease.
- Keep older retained leases without a saved account identity on the
existing resume path.
- Apply the existing supported MCP-only compatibility rule to warm
recovery and predecessor accounting. Persist the capability discovered
at admission and reuse it after restart without repeating discovery.
- Keep predecessor-retirement fallback scoped to the same physical
sandbox. A refusal after a portable move preserves the session instead
of requesting a replacement that the plugin driver cannot enforce.

## Verification

- On `1eb527be27`, server typecheck (including Runner build and SDK
dependencies) passed.
- Independent final verification passed 168 tests across portable source
recovery, session resume, controller recovery, and task authority; two
existing tests were skipped. The full portable source integration suite
passed 64 tests.
- The new integration follows confirmed ownership through an
incompatible instruction change, persisted fresh admission, target-only
prepare/auth/submit without the source checkpoint, and reattachment
after controller replacement. It also rejects unpersisted session
selection, ambiguous target leases, and incorrect saved identities.
- Before this final change, independent verification passed 1,667 tests
across executor, session resume, recovery usage, remote warm recovery,
runtime selection, and runtime execution; the environment lifecycle
suite passed 208 tests.
- Existing regressions cover previous-release account metadata,
supported and unsupported MCP changes, provider/instruction
incompatibility, and admission capability reuse without another
discovery call.
- On `51d7eec373`, all four S3 multipart throttle tests passed
independently. CI exposed an existing timing-sensitive test twice; its
20 ms overlap assumption now uses synchronized retries and an event-loop
yield. Production upload code is unchanged. Temporary concurrency caps
of one and eight both fail the retained exact-four assertion.
- Exact SDK source-pin checks passed for both stack heads. The upper
#20743 code patch remains unchanged; its documentation preserves both
layers' behavior.

## Risks

- The handoff protocol is opt-in. Existing plugin leases without
portable ownership metadata keep the previous release's resume path.
Live attachment still requires end-to-end qualification. An unverified
claim blocks startup. Git workspaces prepared on a portable allocation
also need the workspace placement layer.

## Model Used

- OpenAI Codex GPT-6 with GPT-6.1 Sol agents for implementation and
independent review.

## Checklist

- [x] Thinking path and model are included.
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] The issue is described using the enhancement template.
- [x] Local checks passed and new behavior has tests.
- [x] All Paperclip CI gates are green.
- [x] Greptile is 5/5 with no open findings.

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-12 10:43:48 -08:00
DottaandClaude Opus 5.5 90ae3c4337 fix(native): process a superseded export's late stop receipt and stop its sandbox when none arrives (#21143)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A native run can be retired as superseded while its workspace export
fails. Its files stay unsaved, and the run stays the writer of the
task's workspace.
> - #21136 records those files for the automatic workspace recovery.
That record needs the sandbox's stop-and-retain receipt.
> - The native run projection worker checked for that receipt for one
hour after the run ended. After that, it dropped the run. Nothing woke
it when the receipt arrived later.
> - If the receipt came late or never, the run kept the task's workspace
fenced. Every queued run on the task waited and then failed setup.
Nobody owned the problem.
> - This pull request makes the worker process a late receipt, stop and
keep the sandbox itself when no receipt arrives, and never drop the run
silently.
> - The benefit is that the task's queue moves again by itself, and two
writers never use one workspace.

## Linked Issues or Issue Description

Refs #21136 (review thread on
`native-superseded-workspace-export.ts:84`)

**What happened?**
`preserveSupersededWorkspaceExport` returned `recheckAt: null` when the
sandbox stop receipt had not arrived within `finishedAt + 1h`. The
projection worker then deleted its entry. A later stop receipt
(`settleStopOnlyCleanup`) woke only the workspace recovery queue, not
the projection queue. `preserveNativeFailedWorkspace` requires
`hasConfirmedSandboxStopAndRetain`, so no hold was recorded. The run
stayed the dirty writer, and queued runs kept failing setup.

**Expected behavior**
A late receipt is processed at once. When no receipt arrives, Paperclip
stops and keeps the sandbox itself and waits for the provider's
confirmation. The workspace stays held while the sandbox may still run.
When the provider confirms that the sandbox is gone, the task continues
from its last save. When the stop stays unconfirmed for hours, the task
shows one plain notice, and the check continues.

**Steps to reproduce**
1. Retire a native run as superseded after its workspace export fails
(the #21136 shape).
2. Let the run's sandbox stop not settle (for example, the server that
owned the release is replaced).
3. Wait more than one hour after the run ended.
4. See that the projection worker drops the run, and that a later stop
receipt records nothing. The workspace stays held.

## What Changed

- `sandbox-stop-and-retain.ts`: a confirmed stop-only receipt now also
notifies `DELIVERY_QUEUES.nativeRunProjection` (scoped to the company
and run). New `prepareUnconfirmedSourceSandboxStop` puts a
stop-and-retain intent on the source lease from its current status. It
skips a destroyed sandbox, a stop already in progress, and a cleanup in
flight.
- `native-superseded-workspace-export.ts`:
- Within one hour of the run ending, the worker checks every minute
(unchanged).
- After that hour, it requests the stop-and-retain itself. The existing
pending-cleanup sweep dispatches the stop and retries it until the
provider confirms. The worker checks again every 15 minutes, and the
receipt wakes it at once. The workspace stays held.
  - A sandbox that another lease holds is never stopped.
- When the provider confirms that the sandbox was destroyed, the files
cannot be saved. In one transaction (task, run, then workspace locks),
the worker sets aside only that run's workspace epoch, wakes queued-run
dispatch, and records `issue.previous_run_files_lost` on the task.
- When the stop is still unconfirmed six hours after the run ended (or
the sandbox is held by another lease), the task gets one
`issue.previous_run_files_waiting` activity entry. The worker keeps
checking. It never drops the run.
- A stop-only dispatch now settles when the provider confirms that the
exact sandbox is already gone. Before, the runtime accepted only a
`stopped` reply and threw on `destroyed`, so a gone sandbox kept the
stop pending forever.
- `sandbox-stop-and-retain.ts`: new `acceptedStopOnlyReceipt` accepts
`stopped`, or `destroyed` for a stop-and-retain intent (a native
workspace export resume still needs `stopped`). `settleStopOnlyCleanup`
stores a `destroyed` receipt as `released` / `success` /
`workspace_source_unavailable`, removes the stop intent, never writes a
retain receipt, and wakes the projection and workspace recovery queues.
- `environment-runtime.ts`: the plugin `environmentStopLease` and
built-in `stopLease` checks use `acceptedStopOnlyReceipt`.
- `workspace-recovery.ts`: when its own stop settles as `destroyed`, the
source is unavailable (the existing alternative-strategy path) instead
of a binding failure.
- Daytona plugin: stop of a sandbox that Daytona reports as not found
returns `destroyed` (destroy already did this) instead of throwing. boxd
already returns `destroyed`. Other sandbox plugins have no stop-only
method, so nothing changes for them.
- New shared `SANDBOX_ALLOCATION_HOLDER_STATUSES` (`active`, `retained`,
`pending_cleanup`). The stop-only cleanup ownership check and the
worker's "held by another lease" guard both use it, so a retained warm
sandbox is never stopped and the two checks cannot drift.
- `ui/src/lib/activity-format.ts`: plain labels for the two task
activity entries. They do not ask a person to fix anything.
- `doc/execution-semantics.md`: document the rule.
- Not changed: the shared owner prover still refuses to treat a stopped
instance on another host as proof that its child processes stopped.

## Verification

- `server/src/services/superseded-workspace-export-recovery.test.ts` (4
new tests, red/green):
- "checks again every minute while the run's own release may still stop
its sandbox": the recheck branch. Nothing is recorded or stopped, and
the workspace keeps its writer.
- "stops and keeps the sandbox itself when no stop receipt arrived, and
records the files once the late receipt lands": after the hour, the
lease gets a stop-only intent. A second pass changes nothing. The late
receipt wakes the projection queue. The next check records the hold, and
the system recovery frees the workspace. Fails on master (`expected {
…(19) } to match object { status: 'pending_cleanup' }`).
- "names the unconfirmed stop on the task once, and keeps checking, when
the provider never confirms it": the expiry branch. One notice, the
check continues, no hold, and the workspace stays held. Fails on master.
- "continues from the last save when the provider confirms the sandbox
is gone": epoch 1 is set aside, the workspace is free, the task says so,
and the result is not applied. Fails on master.
- Review-round tests (all 5 fail on the first head `a208bfa58f`, pass
now):
- `environment-runtime.test.ts` "settles a plugin stop whose provider
reports the sandbox gone (boxd, Daytona not found) through the cleanup
sweep": provider dispatch through `releaseRunLeases` and
`sweepPendingCleanupLeases`. The lease settles `released` with a
`destroyed` receipt and no retain receipt.
- `environment-runtime.test.ts` "settles a built-in stop whose provider
reports the sandbox gone, without claiming retained files": the built-in
`stopLease` path. No destroy call.
- `superseded-workspace-export-recovery.test.ts` "continues from the
last save when Paperclip's own stop finds the sandbox already gone": the
worker's stop request gets a `destroyed` reply, the projection queue
wakes, and the workspace is freed at epoch 2.
- `superseded-workspace-export-recovery.test.ts` "never stops a sandbox
a retained warm lease still holds, says so once and keeps checking": the
source lease is unchanged, a recheck is returned, and `sandbox_in_use`
is recorded.
- Daytona `plugin.test.ts` "explicit stop confirms a sandbox Daytona no
longer has as destroyed".
- `environment-runtime.test.ts` (207) and Daytona `plugin.test.ts` (421)
pass.
- Related suites pass: `native-run-projection-work`,
`retained-native-session-cleanup-work`, `workspace-recovery-stop-proof`,
`native-failed-workspace-recovery`, `legacy-workspace-restore-recovery`,
`environment-retained-orphan-sandbox`,
`workspace-recovery-retry-copyback`, `workspace-recovery-fallback` (398
tests), and `ui/src/lib/activity-format.test.ts`.
- `tsc --noEmit` for `server`.

## Risks

- Low to medium risk. The new behavior acts only on an ended superseded
run that failed in its workspace export, still holds a dirty workspace,
and has no retained-file hold.
- The worker now stops an ended run's sandbox one hour after the run
ended if no stop receipt arrived. The stop uses the existing stop-only
contract and keeps the sandbox. A sandbox held by another lease is never
stopped.
- A destroyed sandbox's unsaved files are not recoverable. The task
continues from its last save and says so.
- If a provider never confirms a stop, the workspace stays held (by
design: no second writer). The task shows one notice, and Paperclip
keeps retrying.
- Shared cleanup code now treats a confirmed `destroyed` stop-only reply
as settled. Every caller was checked: the cleanup sweep and run release
count it as settled; the workspace recovery treats the source as
unavailable; the fallback treats it as stopped (nothing can run there).
No caller reads it as retained files, because no retain receipt is
written. A native workspace export resume still requires `stopped`.
- Each extra delivery notification per settled stop-only cleanup is
cheap. The projection worker's scoped load finds nothing for runs that
do not match.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code
search, shell, test execution) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 11:54:47 -06:00
DottaandClaude Opus 5.5 482063aeca fix(daytona): reuse or repair a read-only skill bundle instead of failing the sync (#21075)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent turns run in a remote Runner in a Daytona sandbox. Before a
turn starts, Paperclip copies each selected skill into the sandbox as a
read-only bundle.
> - A sandbox is often kept between turns, so the read-only bundle is
already there when the next turn (or a recovery) copies it again.
> - The server writes the same skill files again on its own disk, with
new timestamps. The copy compared the bundles with `tar --diff`, which
also compares timestamps. So the copy saw a difference, extracted over
the read-only directories, and failed with "Cannot open: File exists".
> - The turn then stopped and the task stayed blocked until a person
sent a message.
> - This pull request treats a timestamp-only difference as the same
bundle, and repairs a bundle that really differs instead of failing.
> - The benefit is that a kept sandbox no longer blocks a task when its
skills are copied again.

## Linked Issues or Issue Description

**What happened?**
On staging, two tasks in 30 minutes stopped with "Daytona syncIn extract
command failed (exit 43): tar: .paperclip-repository/...: Cannot open:
File exists". One failure was a normal new turn on a kept sandbox. The
other was a recovery of an interrupted turn. Neither one started again
by itself.

**Expected behavior**
The turn uses the bundle that is already in the sandbox when its content
and permissions match. When the content does not match, the copy repairs
it.

**Steps to reproduce**
1. Copy a read-only (0555/0444) skill directory into a sandbox.
2. Change only the timestamps of the copy (this is what a new server
preparation does).
3. Copy the same skill again. GNU tar fails with "Cannot open: File
exists".

## What Changed

- `file-sync.ts` (read-only directory copy): run the `tar --diff` check
with `LC_ALL=C` and ignore lines that only say "Mod time differs".
Bytes, modes, file types and missing files still count as differences.
- When the bundle really differs, make its read-only subdirectories
writable before the extract. GNU tar then replaces the files and
restores the archived directory modes (`--delay-directory-restore`).
Extra files in the target stay.
- If reopening the directories or the repair extract fails, re-seal only
the directories that were reopened for it (their list is recorded
first). Writable user directories beside the bundle keep their modes.
- `file-sync.test.ts`: the existing GNU tar test now covers a
timestamp-only resync (kept, no failure) and a damaged bundle (repaired
to the source bytes and modes, instead of the old fail-closed result),
and a repair that cannot finish (bundle directories re-sealed to 0555, a
writable user directory left at 0755), and a reopen that hits an
unreadable user directory (bundle re-sealed, no scratch files left).

## Verification

- Red: with the old `file-sync.ts`, the updated test fails with the same
error seen on staging: `tar: references/overview.md: Cannot open: File
exists`.
- Green: `npx vitest run src/file-sync.test.ts` in
`packages/plugins/sandbox-providers/daytona` passes 33/33 with GNU tar.
- I reproduced the root cause by hand with GNU tar 1.35: a re-extract
over a 0555 directory fails with "Cannot open: File exists", and `tar
-df` reports only "Mod time differs" for a copy that differs only in
timestamps.

## Risks

- Low risk. The change affects only read-only (immutable) directory
copies on Daytona.
- A bundle that really differs is now repaired in place instead of
failing the turn. Each file is replaced by tar (unlink and create). The
directory modes come back from the archive.
- If a target has extra read-only subdirectories that are not in the
archive, they become owner-writable after a repair.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, code search, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 06:28:27 -06:00
Nicky LeachandPaperclip 5dfbb04b85 Poll asynchronous workspace provider preparation (#20865)
## Thinking Path

> - Paperclip starts agent runs through a workspace provider when a
project uses remote storage.
> - The provider must prepare the workspace before the agent starts.
> - One long worker call ties the checkout to its transport.
> - Some checkouts take longer than the transport allows.
> - This change lets an opted-in provider accept work and report its
deadline.
> - Core waits for completion through short status calls and stops at
the provider deadline.

## Linked Issues or Issue Description

**What happened?**

A workspace provider can lose its reply when preparation takes longer
than a single transport request. This stops run bootstrap before agent
execution.

**Expected behavior**

Core should wait for a provider-confirmed workspace through short
requests. The provider's deadline must bound the wait.

**Steps to reproduce**

Use a workspace provider that takes longer to prepare a checkout than
its transport can remain open. Start a run that requires a new checkout.

## What Changed

- Add an optional workspace preparation method to the plugin SDK.
- Validate pending receipts and keep the original binding checks on
completed replies.
- Poll only during active preparation. Retry a lost status reply within
the advertised deadline.
- Preserve the original operation for providers that do not offer
asynchronous preparation.
- Reject deadline offers that exceed the existing lifecycle lock budget.
Honor accepted offers without shortening them.
- Checkout verification uses the existing synchronous RPC with a
10-second deadline and a 12-second transport timeout. [Core verification
PR #20876](https://github.com/paperclipai/paperclip/pull/20876)

## Verification

- All 63 workspace SDK and Core integration tests passed with temporary
local PostgreSQL. The tests cover lost status replies, deadline expiry,
oversized offers, and synchronous verification with an async-capable
worker.
- All 58 runner SDK input, evidence, and public type tests passed with
the regenerated source pins. The local server typecheck passed.

## Risks

- This change activates only when the provider advertises the method.
Existing providers keep their current behavior.
- Polling uses one-second intervals during active preparation. It stops
at completion, failure, or the provider deadline.
- No database migration is required.

## Model Used

- OpenAI GPT-6 Codex with local code execution and tests.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-11 10:33:02 -08:00
Nicky LeachandPaperclip b2edea5754 fix(workspaces): verify existing worktrees before fresh retries (#20876)
## Thinking Path

> - Paperclip manages work for AI agents.
> - Retry checks whether the task can keep its existing workspace.
> - A provider can complete checkout before Core receives the result.
> - Recreating a completed worktree costs time and can change its base
commit.
> - Core uses the same ownership and writer checks for local and
provider workspaces.
> - The provider verifies its checkout before Core creates a fresh
workspace.

## Linked Issues or Issue Description

**What happened?**

A retry can allocate a new provider worktree even though the original
checkout completed.

**Steps to reproduce**

Complete workspace checkout through a storage provider. Fail Runner
startup or lose the prepare reply. Retry the task.

**Expected behavior**

Reuse the same worktree when the provider confirms checkout and the
prior run can no longer write to it.

Related: #20865 addresses timeouts while the original prepare is in
progress. This PR checks an earlier checkout when the user retries.

## What Changed

- Check the prior task binding and writer state before asking a storage
provider to verify a checkout.
- Extend the existing prepare operation with an optional verification
mode. Providers opt in through their manifest.
- Return through normal workspace reuse after verification. Keep the
fresh retry fallback when verification cannot establish checkout.

## Verification

- Core workspace preparation, plugin workspace ownership, retry
continuity, and SDK workspace-provider suites passed locally (213
tests).
- Focused cases pass for provider verification, retry through heartbeat
dispatch, and real local Git checkout verification with no plugin
worker. The heartbeat test confirms that ordinary preparation and
adapter dispatch keep the original workspace, branch, and base commit.
- Core server typecheck passed before the final audit and timing edits.
GitHub CI covers the final revision and broader regressions.

## Risks

Provider verification adds one bounded call before a fresh retry. Older
providers keep their current behavior. Verification must preserve the
workspace, branch, and recorded base commit. The normal execution
preparation checks still run before admission. No database migration is
needed.

## Model Used

OpenAI GPT-6 (Codex), with source inspection and local test execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either linked existing issues or described the issue in-PR
following the relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links
- [x] My branch name describes the change and contains no internal issue
details
- [x] I have run targeted tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-11 18:04:18 +00:00
DottaandClaude Opus 5.5 57f1afdf19 fix(daytona): retry a sandbox create that timed out before any agent work (#20828)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - On a Daytona sandbox environment, every new task run first creates a
fresh sandbox
> - When one Daytona create did not start, the plugin waited the full
300 s acquisition budget
> - The run then failed `setup_failed` with `lease_acquire_failed`. No
agent work had started, but nothing retried it
> - On staging this hit two agents in 15 minutes. Each attempt used
about 5 minutes, and the task stopped until a person pressed Retry. The
manual Retry then got a sandbox in 28 s
> - This pull request ends a stuck create early, adopts a create that
started late, and routes the timeout to the bounded bootstrap retry that
already exists for transient provider refusals
> - The benefit is that a stuck create costs about 90 s and recovers by
itself. The task shows Retry only after the bounded attempts are used up

## Linked Issues or Issue Description

Refs #20626 (bounded resume of a reusable sandbox), Refs #20212 (wait
out a Daytona state change), Refs #20609 (force a stuck stop).

**What happened?**
A run on a Daytona sandbox environment failed after 307 s with `Failed
to acquire lease for environment "Paperclip Sandbox" (sandbox): Daytona
lease acquisition timed out; allocation cleanup is pending`. The run
recorded `errorCode: setup_failed`, `failureCause:
lease_acquire_failed`, and `executionRecovery: { kind: "bootstrap",
providerWorkStarted: false }`. The task became blocked with Retry. No
automatic retry was scheduled.

**Steps to reproduce**
1. Use a Daytona sandbox environment with the default `timeoutMs`
(300,000).
2. Start a run while Daytona keeps the new sandbox in `creating`.
3. See that the run waits 300 s, fails `setup_failed`, and is not
retried.

**Expected behavior**
A create that has not started before provider work must not cost 5
minutes. The run must retry by itself within bounds. A late sandbox that
has started must be used. A sandbox that was abandoned must be removed.

**Root cause**
- `packages/plugins/sandbox-providers/daytona/src/plugin.ts`
(`onEnvironmentAcquireLease`): the only bound on the create step was the
whole acquisition budget (`timeoutMs`, default 300 s).
- `server/src/services/environment-provider-unavailable-error.ts`
(`readTransientEnvironmentProviderFailure`): only an HTTP status or
errno on the create call counted as transient. The timeout envelope
carries neither, so the run fell through to generic `setup_failed`,
which `isTransientWorkspacePreparationCode` does not retry.

## What Changed

- Daytona: one fresh acquisition is now capped at 180 s. A create that
has not returned after 90 s is looked up once by its creation name. A
sandbox in `started` is adopted, and setup continues. A sandbox in
`pending_build`, `building_snapshot`, or `pulling_snapshot` keeps
waiting inside the cap. Any other state ends the attempt with the
existing cleanup envelope and an acquisition diagnostic whose elapsed
time reached the expired budget.
- Daytona: a late SDK failure no longer runs the compensating delete for
a sandbox that was adopted.
- Host: `readTransientEnvironmentProviderFailure` reads the validated
acquisition diagnostic. An acquisition whose first failure was its own
deadline is transient. The run fails with
`environment_provider_unavailable`, category `timeout`, and uses the
bounded bootstrap retry (30 s, then 60 s). An earlier setup failure
stays `setup_failed`. Message text is never parsed.
- Host: the task reason for this case is plain: "The sandbox for ... did
not finish starting in time. No agent work had started. Paperclip
retries automatically ...".
- Docs: `doc/execution-semantics.md` and
`doc/plugins/SANDBOX_PROVIDER_CAPABILITIES.md`.

## Verification

Red before the fix, green after:
- `server/src/services/heartbeat/run-completion.test.ts` "routes a fresh
sandbox create that timed out through bounded bootstrap recovery" uses
the staging error shape through the orchestrator. Red: `errorCode` was
`setup_failed`. Green: `environment_provider_unavailable`, category
`timeout`, bounded retry scheduled. The `started` and `early_failure`
cases stay `setup_failed` with no retry.
- `server/src/services/environment-provider-unavailable-error.test.ts`:
deadline diagnostic is transient (red, then green). An early failure is
not. The task message is plain.
- `packages/plugins/sandbox-providers/daytona/src/plugin.test.ts`: "ends
a create still pending at 90 seconds" (red: still pending at 90 s;
green) and "adopts a create that started by 90 seconds" (red; green, and
a late SDK rejection does not delete it). "keeps waiting for a create
that is still preparing its snapshot" is a safety case.
- Existing tests that encoded the 300 s acquisition budget now use the
180 s cap.

Commands run locally:
- `npx vitest run src/plugin.test.ts` (Daytona): 414 passed.
- `npx vitest run src/services/heartbeat/run-completion.test.ts
src/services/environment-provider-unavailable-error.test.ts
src/__tests__/environment-acquisition-diagnostics.test.ts
src/__tests__/task-interruption-query.test.ts`: 229 tests passed. The
run-completion file reports an `afterAll` embedded-Postgres teardown
timeout. The same timeout occurs on unchanged `origin/master`, so it is
a local platform fault.
- Daytona plugin `pnpm typecheck` and server `pnpm typecheck`: pass.
- `pnpm verify`: asked for broader CI coverage, because
`doc/execution-semantics.md` is a global CI input. Required CI supplies
that coverage.

Upgrade safety: nothing new is persisted. A task already blocked by this
failure keeps `canRetry: true`. Its Retry starts a new run with a new
creation record, and that run uses the new path. On staging, the manual
Retry of the affected task got a fresh lease in 28 s. The creation
record of the earlier attempt stays with the creation reconciler.

Not verified: a live Daytona create that stays in `creating`. That
cannot be forced on demand.

## Risks

- A legitimately slow create that stays in `creating` or `starting` for
more than 90 s now ends the attempt and retries. Snapshot preparation
states still wait inside the 180 s cap. Staging creates normally take
about 20 to 30 s.
- An abandoned create is removed by the creation reconciler. It can stay
alive until that reconciler settles the record (the existing settle
window). This PR does not change that window.
- Other sandbox providers get the host classification only when they
send the same acquisition diagnostic.

## Model Used

Claude Opus 5.5 (claude-opus-5-5), Anthropic, used through Claude Code
with tool use (shell, file edits, browser for read-only staging API
reads).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 10:04:17 -06:00
Nicky Leach 4d141cdf64 Carry plugin task refusals and batch project reads (#20762) 2026-11-11 00:49:03 -08:00
DottaandClaude Opus 5.5 42d4499489 fix(daytona): force a stuck stop before starting a retained sandbox (#20609)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - On Daytona, a follow-up task resumes the agent's retained sandbox
before the agent starts
> - The host stops an idle sandbox gracefully. A process that ignores
SIGTERM can leave Daytona in `stopping` for many minutes
> - Resume waited for that state change until its 300 s start deadline.
It never escalated, so each attempt failed after five minutes, and the
bounded retries did the same
> - On staging, a user comment waited about 21 minutes before the agent
started. Two resume attempts each used 299.7 s in the plugin dispatch
> - This pull request forces a stuck stop once, then starts the sandbox
within the same deadline. It also traces the wait with the observed
state
> - The benefit is that a stuck stop costs about 30 s instead of failing
the run, and the next case is diagnosable from the trace

## Linked Issues or Issue Description

**What happened?**
A user commented on a task that used a retained Daytona sandbox. The
first run and its first retry each failed after about 5 min 20 s with
`reusable_sandbox_resume_failed (cause=provider_error)`. The Honeycomb
trace shows `execution.plugin_dispatch` at 299.7 s in both. Inside it
there is only `lease.lookup` (44 ms) and `lease.channel_cleanup`, and no
`lease.start`. So the whole budget went to `waitOutSandboxTransition`,
which polled a sandbox that stayed in a transition state. The later
attempt, after a control-plane restart, found the sandbox `stopped`. Its
`lease.start` took 1.3 s. A transition that ends in `stopped` is a stop.

**Expected behavior**
A sandbox that stays `stopping` for longer than the host's own
graceful-stop budget is forced to stop, then starts. The run does not
fail on the start deadline.

**Steps to reproduce**
1. Retain a Daytona sandbox whose graceful stop does not finish (Daytona
reports `stopping`).
2. Resume or realize the lease.
3. Before this change, the call polls until the 300 s deadline and fails
with "is still stopping at its start deadline".

## What Changed

- `packages/plugins/sandbox-providers/daytona/src/plugin.ts`:
`waitOutSandboxTransition` tracks how long the sandbox stays `stopping`.
After `STUCK_STOPPING_FORCE_AFTER_MS` (30 s) it calls
`sandbox.stop(seconds, true)` once, then keeps polling. A refused force
stop keeps the old wait within the same deadline.
- The wait is a `lease.wait_transition` span. The force stop is a
`lease.force_stop` child span.
- `lease-tracing.ts`: the closed `paperclip.sandbox.startup.state` label
also records `starting`, `stopping`, `restoring`, `archiving` and
`resizing`. Before, these were all `other`.
- `server/src/services/plugin-host-services.ts`: the host admits the two
new span names and the new closed state values.
- `doc/observability.md`: documents the new spans and states.

## Verification

- Red/green: with the plugin change stashed, the three new
stuck-`stopping` tests (realize, refused force stop, resume) fail by
hanging to the deadline. With the change they pass. The "settles on its
own" test confirms that no force stop happens for a normal stop.
- `vitest run` in `packages/plugins/sandbox-providers/daytona`: 14
files, 618 tests passed (8 skipped). `tsc --noEmit` passed.
- `server/src/__tests__/plugin-host-services-span.test.ts`: red before
the allowlist change, 25/25 after.
- `pnpm verify` exited 2 (the doc is a global CI input), so CI supplies
the broad coverage.
- Not verified on a live Daytona sandbox. Hosted acceptance needs the
change deployed to staging.

## Risks

- The force stop sends SIGKILL to processes in a sandbox that is already
stopping and is about to start. The disk persists across a stop, so the
retained workspace is kept. A graceful stop that is still running past
30 s is killed. The host's own graceful stop budget is 30 s, so the host
had already given up on it.
- Daytona can refuse a stop while a state change is in progress. Then
the code waits as before, so the result is no worse than today.
- The root cause is inferred from the trace (a transition that ends in
`stopped`). The new state label confirms or refutes it the next time it
happens.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, browser, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 19:20:30 +00:00
DottaandPaperclip 5bad074baa feat: add individual agent interaction permissions (#20411)
Add eleven sparse incoming and outgoing restrictions, human CLI/API administration, shared mutation enforcement, and pending-delivery checks. Preserve existing defaults, agent settings, and existing authorization boundaries.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-09 20:34:04 -06:00
Nicky LeachandPaperclip 3a5572c9e5 Extend login PTY contracts for existing computers (#20410)
## Thinking Path

> - Paperclip manages agent execution across environments.
> - Environment providers own execution resources.
> - Codex and Claude already share the login PTY transport.
> - A login on an existing computer needs its own allocation and
deadline.
> - Input retries must retain their identity.
> - Codex also needs a fixed credential-file read; Claude already
returns its token through PTY output.
> - This change extends the shared contract and keeps both existing
provider parsers.

## Linked Issues or Issue Description

**Problem**

The existing login contract assumes a temporary sandbox. It cannot
describe a login allocation on an existing computer or export its fixed
credential result without generic execution.

**Proposed solution**

Add shared allocation purpose, command identity, PTY deadline, and input
identity fields for both Codex and Claude. Add one optional
credential-result hook for Codex's auth.json file.

**Alternatives considered**

A separate Claude login implementation would duplicate the current
setup-token parser and owner-bound credential writer. Claude continues
to use that existing path. The new artifact hook is Codex-specific
because Claude's setup-token result comes through the existing shared
PTY output stream.

## What Changed

- Export the login contract types from the main SDK entry point.
- Forward login purpose and command through plugin allocation and retain
them in lease metadata.
- Forward the PTY deadline and give each input write a stable identity.
- End a login route when the worker cannot confirm input delivery.
- Preserve the opening company scope for input, stop, and close
callbacks.
- Advertise and dispatch the optional fixed Codex credential-result
hook.

The allocation, deadline, input, output, and cleanup contracts serve
both providers. This PR does not enable either provider on an existing
computer. The dependent consumer PR connects admission, services,
credential completion, and UI for both providers.

## Verification

- Close-ordering regression: all 91 host worker-manager tests passed,
including concurrent close callers waiting behind a held plugin
acknowledgement after input failure. Server TypeScript check passed. All
current CI gates, ordinary review, and the selected follow-up review
pass on a39fb59bd1.

- Shared and SDK TypeScript builds passed.
- SDK worker dispatch: 31 tests passed.
- Host worker-manager: 90 tests passed, including a nested
worker-to-host scope guard.
- Environment runtime with embedded PostgreSQL: 197 tests passed.
- Added absent-hook, input-failure, and allocation-forwarding checks.
- A running instance is not required to validate this SDK-only
increment. Real provider acceptance follows after the dependent hooks
are connected.

## Risks

The hook and fields are optional. Existing sandbox providers keep their
current path. Failed input delivery now ends its route instead of
waiting for the deadline. There is no database migration.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact
runtime model identifier and context size are not exposed to this agent.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-09 16:50:58 -08:00
DottaandClaude Opus 5.5 088bdf7cfa fix(daytona): wait out a sandbox state change before starting it for workspace realize (#20212)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - On Daytona, every run realizes its workspace in the leased sandbox.
Realizing it first makes sure the sandbox is started
> - Right after a control-plane restart, the sandbox can still be
mid-transition, for example stopping or starting from the previous
lifecycle step
> - The plugin called `sandbox.start` straight away. Daytona rejects
that with "Sandbox state change in progress", so the run failed
`setup_failed` and needed a Retry
> - This pull request waits out a sandbox's state change before starting
it, and retries a start the provider rejected for that reason, within
the existing start deadline
> - The benefit is that a run whose sandbox is briefly transitioning
starts on its own

## Linked Issues or Issue Description

Refs #20211

**What happened?**
Daytona per-turn restart-chaos K13 storm w-20261109T074128-s9503
(revision 9046884811), t07 (claude A1): run `5aabeb55` failed at
07:45:27 with `setup_failed` and "Failed to realize workspace for
environment … (sandbox): Sandbox state change in progress". The sandbox
was mid-transition after the restart, and realize neither waited nor
retried. A Retry succeeded.

**Expected behavior**
Realize waits for a transitional sandbox state to settle, then starts
the sandbox. If the provider still reports the state change, it retries,
all within the existing start deadline. Any other start error fails at
once, and a transition that outlasts the deadline still fails.

**Steps to reproduce**
1. Realize a workspace while the Daytona sandbox is `stopping` or
`starting`, or while `start` returns "Sandbox state change in progress".
2. Before this change, realize fails at once with that error.

**Paperclip version or commit**
9046884811 (wave). Branch rebased on current master.

## What Changed

- `packages/plugins/sandbox-providers/daytona/src/plugin.ts`:
`ensureSandboxStarted` now polls `refreshData` while the sandbox is in a
transitional state.
- The transitional states are `starting`, `stopping`, `restoring`,
`archiving`, `creating`, `pending_build`, `pulling_snapshot`,
`building_snapshot` and `resizing`.
- It polls every second, until the start deadline from `timeoutMs`. Each
read keeps the existing liveness bound.
- If the start fails with "state change in progress", it refreshes and
tries again, within the same deadline.
- The wait, every retry and the start share one deadline. Each read is
bounded by the time left, and the start receives only the remaining
seconds.
- As soon as any state other than `started` is observed, the lease's
stored session is dropped before waiting, so a restart that happened
outside the plugin never reuses a dead session.
  - Starting a settled sandbox (`startSettledSandbox`) is unchanged.
- `plugin.test.ts`:
  - realize waits for a stopping sandbox, then starts it;
  - realize retries a start that was rejected for a state change;
  - realize gives up once the transition outlasts the deadline;
  - another start error fails at once;
  - the start receives only the budget the wait left;
- a restart outside the plugin that the wait sees through opens a fresh
session.
  - The tests stub the API key with `vi.stubEnv`.

## Verification

- Red/green: on master, the wait and the retry cases fail with "Sandbox
state change in progress". With the change they pass, and the deadline
and other-error cases hold.
- `vitest run` in `packages/plugins/sandbox-providers/daytona`: 14
files, 614 tests passed (8 skipped).
- `tsc --noEmit` for the Daytona plugin package: passed.

## Risks

- Low risk. The wait is bounded by the start deadline the plugin already
used. Only transitional states and the provider's state-change rejection
are waited out, and every other error behaves as before.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 03:00:14 -06:00
DottaandClaude Opus 5.5 3bf51913ef fix(daytona): gzip transport compression for sandbox images without zstd (#20104)
## Thinking Path

- Per-turn remote sandboxes are the default, so every remote turn
uploads the workspace into a fresh Daytona sandbox.
- The Daytona plugin compresses large uploads with zstd, but only when
the sandbox has a `zstd` binary. Without one, the upload goes raw: 220
MB for the paperclip seed.
- The default Paperclip runner image has zstd, so it is already covered.
A sandbox image without zstd is not: for example a user-supplied
`config.snapshot` built on plain `debian:bookworm`. Such an image still
has gzip.
- This adds gzip as a fallback codec for those images. It is a
robustness fallback, not a speedup for the default image.

## Linked Issues or Issue Description

Refs #20100

**What happened?**
On a Daytona sandbox image without a `zstd` binary, every remote turn
uploads the workspace seed uncompressed. Measured live with the
paperclip seed: raw takes 18.6–28.8 s, against 6.1–16.0 s compressed.

**Steps to reproduce**
1. Use a Daytona snapshot whose image has gzip but no zstd.
2. Run a remote task on a large repository and inspect the `transfer`
span: there is no `transfer.compression.codec` attribute, and the full
raw size is uploaded.

**Expected behavior**
Large workspace tarballs are compressed in transit whenever the sandbox
can decompress them.

## What Changed

- `packages/plugins/sandbox-providers/daytona/src/file-sync.ts`:
- The existing availability probe (the same round trip as `mkdir`) also
reports `gzip`.
- Codec choice: zstd when the sandbox has it and Node supports it
(unchanged), otherwise gzip, otherwise raw (unchanged).
- gzip runs at level 1. On the 220 MB paperclip seed it produces 69.6 MB
in 1.26 s on the host; zstd -3 produces 57.6 MB in 0.54 s.
- In the sandbox, gzip decompresses with `gzip -dc -- <scratch> > <raw
scratch>`. It reuses the existing explicit `chmod`, atomic `mv -f`,
failure sweep, private host temp directory, and minimum-saving fallback.
- A failed or partial decompress exits 49, the raw scratch is never
promoted, and every scratch file is swept. The workspace sentinel and
`baselineSha256` checks still run before `turn.start`.
- The `transfer.compression.codec` span attribute is now `zstd` or
`gzip` (still a closed set).
- Tests in `file-sync.test.ts` (red on master):
- A real shell promotion on a sandbox that reports no zstd: the upload
is `.gz` and under half the size, `gzip -dc` runs, and the target is
byte-identical with mode 0600 and no scratch left.
- A truncated gzip upload fails the sync, never creates the target, and
leaves no scratch.
- Codec preference: zstd when both are present, gzip when only gzip is,
raw when neither is.

## Verification

- `npx vitest run` in `packages/plugins/sandbox-providers/daytona`: all
pass. The existing zstd suite runs on this host because it has zstd.
- `tsc --noEmit -p packages/plugins/sandbox-providers/daytona`: clean.
- Live Daytona run on the runner image, 2 vCPU, 220 MB seed, through the
real `performSyncIn` with each codec forced through the probe. The
sha256 matched every time.

  | Codec | Times |
  |---|---|
  | raw | 28.8 s, 18.6 s, 20.2 s, 22.8 s |
  | zstd | 6.9 s, 6.1 s, 16.0 s, 8.5 s |
  | gzip | 9.8 s, 16.8 s |

Network variance is large. Both codecs roughly halve the raw upload or
better.

## Risks

- **Host CPU:** about 1.3 s of streamed gzip per large upload, on the
libuv thread pool, and only when the sandbox lacks zstd. Files under 8
MB, or that save less than 10%, still go raw.
- **No compressor at all:** a sandbox without gzip or zstd uploads raw,
as today.
- **Probe caching:** the probe runs on every sync, on the `mkdir` round
trip it already makes, so it adds no latency. It stays uncached,
following the existing design's rule that one principal's observation is
never reused for another.
- **Observability:** one new value, `gzip`, in an existing closed-set
span attribute. The endpoint gate is unchanged.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool
use and code execution, via Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 14:52:56 -06:00
Nicky Leach 17ad23aafe fix(workspaces): back off failed terminal cleanup and retire abandoned reservations through remove (#20020) 2026-11-08 07:12:48 -08:00
DottaandClaude Opus 5.5 7fd98682ca fix(workspaces): give every subtask its own execution workspace (#19860)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents split work into child tasks, and each task runs in an
execution workspace (a checkout, branch, or sandbox folder)
> - Since early 2026, a new child task used its parent's realized
workspace by default: the same checkout, branch and sandbox
> - A parent and its child often run at the same time. Two writers in
one folder lose each other's files, the waking parent is cancelled with
`workspace_busy`, and sandbox lease ownership crosses from the parent
into the child, so retained-file recovery refuses the workspace as
ambiguous
> - The product rule is simple: a subtask gets its own workspace,
independent of the parent's saved files. Paperclip never needs to carry
a parent's unmerged changes into a child
> - This pull request makes that the only behaviour on every creation
path, and moves existing shared children to their own workspace on their
next run, with no user or admin action
> - The benefit is that parents and children never block or overwrite
each other, and recovery ownership stays with one task

## Linked Issues or Issue Description

**What happened?**
A child task inherited its parent's realized execution workspace by
default (`createChild` defaulted to `executionWorkspaceInheritanceMode:
"linkage"`, and `issueService.create` fell back from
`inheritExecutionWorkspaceFromIssueId` to `parentId`). When the parent
woke while the child wrote to the shared workspace, the parent run was
cancelled with `workspace_busy`. In sandbox environments, the lease
ownership chain crossed from the parent into the child, and
retained-file recovery refused the workspace with
`allocation_ownership_ambiguous`.

**Expected behavior**
Every new child task gets its own execution workspace, realized from the
project base exactly as a top-level task in that project. It never uses
the parent's checkout, branch, saved snapshot or unmerged changes. A
parent and its descendant never share a workspace by any route.

**Steps to reproduce**
1. Run a parent task in a project with isolated (git worktree) or
sandbox workspaces, so the parent has a realized execution workspace.
2. From the parent run, create a child task with only `parentId` (or
through the runner `create_task` tool, chat suggestions, decision
effects, plugins, or the New sub-task dialog).
3. See that the child's `executionWorkspaceId` is the parent's, with
`executionWorkspacePreference: "reuse_existing"`.
4. Wake the parent while the child runs in a sandbox environment. The
parent run is cancelled with `workspace_busy`.

**Additional context**
Refs #9233 (the existing `strategy_only` opt-out).
The opt-out `executionWorkspaceInheritanceMode: "strategy_only"` already
existed for accepted-plan decomposition. This change makes that
behaviour (tightened to drop a pinned existing branch) the only
behaviour.

## What Changed

- **One rule for every creation path
(`server/src/services/issues.ts`).** `issueService.create` still uses
the parent (or `inheritExecutionWorkspaceFromIssueId`) as a source for
the project, project workspace and workspace *strategy*, but it never
copies the source's `executionWorkspaceId`, `reuse_existing` preference,
or pinned existing branch. All server paths that create a task go
through `create()`/`createChild()`: issue routes (`POST /issues`, `POST
/issues/:id/children`, the run-bound default parent and run workspace
source), runner `create_task`, chat suggestion acceptance, native status
decisions, accepted-plan decomposition, decision effects, plugins
(`ctx.issues.create`), external agents, watchdog follow-ups, monitor
recovery issues, case handoffs and email channels.
- **Explicit ancestor workspaces are dropped.** If a create names an
`executionWorkspaceId` (or an `existing` workspace selection) that an
ancestor of the new task created or is bound to, `create()` drops it and
keeps only the strategy. Older clients and skills that copied the
parent's id keep working and get an independent workspace.
- **`createChild` no longer has a "linkage" mode.**
`executionWorkspaceInheritanceMode` is still accepted from internal
callers and is ignored.
- **`inheritExecutionWorkspaceFromIssueId` decision.** I kept the field
for API compatibility, but it now copies only the source issue's project
and workspace strategy. I removed its workspace-sharing behaviour for
every pair, not only parent/child: the source task and its follow-up can
run at the same time (the runner, watchdog and status-decision callers
all create the follow-up while the source run is live), so sharing would
again let two tasks write one folder. Explicit sharing between unrelated
tasks stays available through an explicit `existing` workspace
selection.
- **New `server/src/services/task-workspace-lineage.ts`.**
`buildIndependentTaskWorkspaceSettings` (strategy only: mode,
environment, runtime, network egress, shared-workspace concurrency, base
ref, branch template, commands; no `reuse_existing`, no
`operator_branch`, no `existingBranch`), `loadTaskLineageIssueIds` (the
parent chain plus, for a task that depth placement put beside its
requester, the requester's chain via `privacyParentIssueId`) and
`isAncestorExecutionWorkspace` (the workspace's `sourceIssueId` is in
that lineage, or a lineage issue is bound to it). A child in another
project does not take the parent's strategy.
- **Upgrade-safe rebind at admission.**
`executionWorkspaceService.releaseAncestorWorkspaceBindingAtAdmission`
runs at the same admission point as the pending workspace selection
(`heartbeat.ts`). For a child (or a flattened task beside its requester)
that an older release bound to an ancestor's workspace, it clears the
child's binding (keeps the project workspace and strategy, bumps the
binding revision) so the run realizes the child's own workspace. It does
nothing while another run of the child is running, while the child's
earlier run still owns workspace finalization or recovery, for native
restart/lease-owner recovery, for a process-loss retry, or for a Retry
that keeps the current workspace. The parent and its workspace are never
changed.
- **Projects without a project workspace (per-task repository
acquisition).** A parent with no project workspace worked in its own
task files, so its settings (for example `isolated_workspace`) do not
describe the project base. A child of such a parent, and an older child
released from one, takes the project's defaults exactly like a top-level
task, instead of an isolated mode that would resolve to a git worktree
with no git base ("needs a project / project workspace"). When the
child's new workspace is created at admission,
`executionWorkspaceRepositoryService.prepareForAdmission` repeats the
parent's (or requester's) repository requests as the child's own pending
requests (identity, URL, catalog id and requested ref only; no pinned
commit, branch or files). The normal acquisition path then clones a
fresh copy into the child's task files with the child's own access
checks. This runs only for a workspace created by that admission, so
existing tasks never gain repositories.
- **UI.** The New sub-task dialog no longer pre-selects the parent's
workspace. `buildSubIssueDefaults` drops `executionWorkspaceId` and
`reuse_existing`, and derives the parent's independent mode
(`shared_workspace`, `isolated_workspace` or `agent_default`) from its
preference, saved settings or current workspace. The "This sub-task will
no longer use the parent task workspace" warning is removed.
- **Contracts and docs.** Shared validator and plugin SDK comments
describe the new meaning of `inheritExecutionWorkspaceFromIssueId`.
Updated `doc/execution-semantics.md` (new rule in §6 and a wording fix
in the retained-sandbox paragraph), `doc/SPEC-implementation.md` (child
workspace rule, requester keeps *strategy* inheritance, watchdog
follow-ups), `doc/SPEC.md` (task workspace ownership),
`docs/specs/external-task-protocol.md`, `skills/paperclip/SKILL.md`
(step 9 and the workspace bullet),
`skills/paperclip/references/issue-workspaces.md`, the CEO onboarding
`HEARTBEAT.md`, and the plugin SDK README.

### Previous-release story (`rules/upgrade-safe-run-state.mdc`)

- **Migrated on next admission.** A child that the previous release
bound to its parent's (or any ancestor's) execution workspace keeps that
binding until its next normal admission. Then the server clears the
binding and the run realizes the child's own workspace from the project
base. No schema migration is needed. No user or admin action is needed.
- **In-flight writers finish first.** A running child is not
interrupted. Its current run, its native restart recovery (admitted
native input or lease owner), a process-loss retry, and a Retry that
keeps the current workspace all finish in the shared workspace. While a
child's earlier run is still running or its workspace finalization or
recovery is unsettled, the binding stays and a later admission releases
it.
- **No files are dropped.** The release changes only the child's binding
row. The parent keeps its binding, and the shared workspace row, folder,
branch and saved snapshots are not touched. Anything the child left
there stays in the parent's workspace.
- **No new refusal codes.** Saved run identity (native execution input,
leases, checkpoints) is not changed or refused; the rebind only affects
new admissions.

## Verification

- New `server/src/__tests__/subtask-workspace-independence.test.ts`
(embedded Postgres): `create` with `parentId`, `createChild` with no
mode / `"linkage"` / `"strategy_only"`,
`inheritExecutionWorkspaceFromIssueId` pointing at the parent, explicit
parent and grandparent `executionWorkspaceId`, a non-child follow-up,
and unchanged top-level defaults and explicit top-level reuse. 7
creation cases fail on `master` and pass with this change. Admission
cases: an old-release child is rebound while its parent is running and
the parent keeps its workspace; a running child is not rebound (deploy
lands mid-run); an unrelated explicit share is kept; the parent is never
rebound.
- New heartbeat case in
`server/src/__tests__/heartbeat-workspace-busy.test.ts`: an old-release
child bound to its running parent's workspace in a sandbox environment
runs to `succeeded` with no `workspace_busy`, gets a new workspace, and
the parent run and binding stay unchanged. It fails on `master` (the
child waits with `workspace_busy`).
- Updated tests that asserted the old inheritance in
`issues-service.test.ts`, `task-workspace-binding.test.ts`,
`chat-project-tools.test.ts`, `task-workspace-child-trust.test.ts`,
`heartbeat-git-restore-point-resume.test.ts` (a child bound to a closed
parent's expired workspace now starts from the project base and leaves
the parent's restore point alone), and
`ui/src/lib/subIssueDefaults.test.ts`. Remote-workspace reservation,
continuation and writer-wait suites now seed the older-release shared
shape directly (or an unrelated explicit share), so those safety paths
stay covered.
- `npx vitest run` on the changed suites and about 45 related server
suites (heartbeat workspace, recovery admission, issue routes, routines,
plugins), `ui` `subIssueDefaults` and `NewIssueDialog` tests, `pnpm
check:token-gates`, `pnpm -r typecheck`.

- New
`server/src/__tests__/heartbeat-subtask-acquired-repository.test.ts`: a
project with no project workspace, a parent whose task files hold an
acquired repository with an unmerged change, and a child (new, and the
older-release shape bound to the parent's workspace) that runs to
`succeeded` in its own task files with a fresh clone at the project base
and its own ready repository receipt. The parent's checkout keeps its
change. Both cases fail on the previous head with "needs a project /
project workspace".

## Risks

- **Project-level configuration still applies to children exactly as to
top-level tasks.** In a project configured for `shared_workspace`, every
task (top-level or child) uses the project's one folder under the
existing shared-workspace concurrency policy. A branch that the project
itself pins applies to every task in the project. This PR removes only
the parent-to-child carry-over of the realized workspace; it does not
change those project defaults.
- **Standalone follow-ups that an older release bound through
`inheritExecutionWorkspaceFromIssueId`** are migrated only when the
source is in their recorded lineage (parent or privacy parent). Others
keep their explicit share and the existing writer-wait serialization.

- **Behaviour change for agents that relied on a shared checkout.** A
child or follow-up no longer sees the parent's unpushed or uncommitted
changes. Agents must push or merge code they want a subtask to build on.
The skill and onboarding text now say this.
- **Explicit ancestor workspace ids are silently dropped** instead of
rejected, so older clients do not break. The task gets an independent
workspace.
- **First run of a migrated child starts from the project base.** If an
older child had uncommitted work in the parent's workspace, that work
stays in the parent's workspace and is not copied into the child's new
workspace.
- **Admission adds one indexed read** for tasks with a parent and a
bound workspace; the locking transaction and ancestor query run only
when the bound workspace was created by another issue.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use (file edits, shell, test execution) and
extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 18:34:09 -06:00
DottaandClaude Opus 5.5 7475577cdb fix(daytona): keep a runner sandbox alive through the controller reconnect grace (#19754)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A remote agent run executes in a Daytona sandbox and keeps working
while the Paperclip server restarts
> - The server reattaches that runner within a 7-minute reconnect grace
> - Daytona stops an idle sandbox after its `autoStopInterval`, and it
counts only SDK and lifecycle calls as activity
> - With the server down, nothing touches the sandbox, so a 5-minute
auto-stop kills the runner before the server returns
> - This pull request makes the host tell the provider its reconnect
grace, so Daytona never auto-stops a runner sandbox inside that grace
> - The benefit is that a controller outage of up to 7 minutes no longer
turns a healthy remote run into a blocked task

## Linked Issues or Issue Description

Refs #19199 (the same floor for the warm runner idle window). Refs
#19741 (follow-up delivery through restarts; no overlap).

**What happened?**
A restart-chaos wave on Daytona killed the server for 4m51s while a
per-turn remote run was active. The environment used `autoStopInterval:
5`. The last sandbox activity was at 06:09:16. The server came back at
06:14:19 and then crashed again. At 06:15:21 recovery called the
sandbox. The plugin found it not started and started it again. The
runner identity marker was still on disk, but its process was gone:
`native.remote_runner_recovery_failed {stage: process_verification,
reason: process_not_running}`, then
`runner_remote_process_identity_unavailable`. The task went from done to
blocked. An earlier wave showed the same failure after a 4m45s outage.

**Expected behavior**
A sandbox that holds a live remote runner survives any controller outage
up to the reconnect grace. The server reattaches the runner and the run
continues.

**Steps to reproduce**
1. Create a Daytona environment with `autoStopInterval: 5` and a
`paperclip_runner` agent.
2. Start a remote run and kill the server for more than 5 minutes (less
than 7).
3. Restart the server. Recovery reports `process_not_running` and the
task becomes blocked.

## What Changed

- `server`: add `resolveRunnerReconnectGraceMsForRun`. It returns
`REMOTE_RUNNER_RECONNECT_GRACE_MS` for `paperclip_runner` on a sandbox
environment.
- `server`: forward `runnerReconnectGraceMs` to the provider worker on
lease acquisition and on the recovery reattach resume. It uses the same
provider-only path as `warmRunnerIdleTimeoutMs`. It is not lease
identity, lease metadata, or saved environment config.
- Daytona plugin: the effective auto-stop is at least the largest host
floor (warm idle window or reconnect grace) plus 5 minutes. With the
current grace this is 12 minutes. The plugin applies it at creation and
raises an older sandbox's shorter interval on resume (bounded, best
effort).
- Daytona plugin: keep the forwarded grace out of the normalized config.
Update the `autoStopInterval` manifest description.
- `doc/execution-semantics.md`: document the reconnect-grace floor next
to the warm floor.

## Verification

- Red/green Daytona plugin tests: `cd
packages/plugins/sandbox-providers/daytona && npx vitest run --config
vitest.config.ts`. The new cases failed before the fix (for example
`expected 120000 to be >= 720000`, and no `setAutostopInterval` call on
resume). All 605 tests pass after the fix.
- Server tests: `cd server && npx vitest run
src/__tests__/environment-run-orchestrator.test.ts
src/__tests__/environment-runtime.test.ts`. 247 pass. They prove
acquisition forwards the grace only for `paperclip_runner` on a sandbox,
that the reattach resume forwards it, and that it does not reach lease
metadata or saved config.
- `pnpm typecheck` for the plugin and `tsc --noEmit` for the server are
clean.
- `pnpm verify` asks for broad CI coverage because a doc changed.
Required CI supplies it.
- Evidence from the kept wave database: lease metadata shows
`autoStopInterval: 5`, `reuseLease: false`, `runnerLifecycleMode:
per_turn`. The run had no events between 06:09:16 and 06:14:19.

## Risks

- Cost: a runner sandbox that leaks (no cleanup) now auto-stops after 12
minutes instead of a shorter configured value. Cleanup still stops or
destroys released sandboxes. Explicit `0` and longer values are
unchanged. Auto-archive and auto-delete are unchanged.
- I did not inspect the sandbox state in Daytona directly. Cleanup
destroyed it before this investigation. The proof is the timing, the
lease config, and the `process_not_running` result after a restart that
kept the disk.
- No live Daytona replay ran.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell,
file edit, code execution) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 01:36:18 -06:00
github-actions[bot]andlockfile-bot e28158b353 chore(lockfile): refresh pnpm lockfiles (#19617)
## Thinking Path

> - Paperclip uses a dependency lock for reproducible builds.
> - Dependency changes on master need matching lock entries.
> - Paid eval reports also pin a reviewed offline correction.
> - Reports must install before paid model work starts.
> - This automated PR keeps the lock and report proof together.

## Linked Issues or Issue Description

**What happened?**

Dependency manifests on master changed after the last committed lock
refresh.

**Expected behavior**

Frozen installs must match those manifests. Eval reports must retain
matching offline lock proof.

**Steps to reproduce**

Run a frozen install before and after this lock refresh.

## What Changed

- Refresh the dependency lock from the manifests on master.
- Refresh the sandbox provider lockfiles from their manifests on master.
- Include the matching offline report correction and hashes.

## Verification

- Pinned pnpm resolution completed with lifecycle scripts disabled.
- The report proof refresh accepted the reviewed committed baseline.
- The workflow rejected changes outside the lock and report proof files.

## Risks

The lockfile bot merges this PR without CI or review when every added
package version is at least 3 days old on npm. Otherwise the PR waits
for a maintainer, and the daily refresh tries again. The bot merges only
the exact commit this workflow generated from manifests already reviewed
on master. Unknown report lock baselines remain blocked. This update
changes no permissions or paid enable gates.

## Model Used

None — generated by pnpm and the deterministic report-lock updater.

## Checklist

- [x] Thinking path and problem description included
- [x] Generated lock and matching report proof included
- [x] Release-age gate decides between bot merge and maintainer review

Co-authored-by: lockfile-bot <lockfile-bot@users.noreply.github.com>
2026-11-06 21:08:43 +00:00
Nicky LeachandClaude Opus 5.5 7dfdc97b9f fix(deps): let Dependabot update the sandbox providers (#19489)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Sandbox providers (`packages/plugins/sandbox-providers/*`) are
standalone published packages. `pnpm-workspace.yaml` keeps them out of
the root workspace on purpose.
> - `.github/dependabot.yml` watches only `/`, so Dependabot never saw
the providers.
> - Because of this, the Kubernetes provider stayed on zod 3
(GHSA-mrhx-c7xc-m3w5) after the rest of the repo moved to zod `^4.6.6`.
#18839 fixed it by hand.
> - We do not want to keep these versions current by hand in many
places.
> - This pull request lets Dependabot update the providers, and keeps
each provider standalone.
> - The benefit is that a provider dependency that falls behind gets a
pull request that can pass CI.

## Linked Issues or Issue Description

Refs #18839 (adds `scripts/typecheck-sandbox-providers` and fixes
Kubernetes zod by hand).
Refs #18998 (a Dependabot pull request for the Pi distribution that
fails CI by design).

**What existing behavior does this improve?**
Dependabot version updates. They skip every package that the root pnpm
workspace excludes.

**Subsystem affected**
CI and dependency automation: `.github/dependabot.yml`,
`scripts/typecheck-sandbox-providers`.

**Current behavior**
Dependabot does not update the sandbox providers. A provider can stay on
an old or unsafe version, and nobody gets a pull request.

**Proposed behavior**
Dependabot updates the providers each week. Each pull request changes
the provider manifests and the lockfiles that CI installs frozen, so the
pull request can pass CI.

**Reason and benefit**
We stop having to find and fix old provider dependencies by hand.

## What Changed

- `.github/dependabot.yml`: add an npm entry for
`/scripts/typecheck-sandbox-providers`. Dependabot treats this workspace
as one unit. One pull request per dependency changes every provider that
uses the dependency and the shared `pnpm-lock.yaml` together. So the
entry needs no `groups`. One entry per provider would open a separate
pull request for each provider, and each one would fail the frozen
install in CI.
- `.github/dependabot.yml`: add an npm entry for
`/packages/plugins/sandbox-providers/daytona`. Daytona is the only
provider with its own lockfile, and the release build and the Daytona
end-to-end job install it frozen. Each lockfile has exactly one
Dependabot entry, so every pull request changes a manifest and its
lockfile together.
- Take Daytona out of the shared typecheck workspace. This pull request
does not commit `scripts/typecheck-sandbox-providers/pnpm-lock.yaml`:
since #19522, only the lockfile refresh workflow writes it. After merge,
the bot removes Daytona's packages from it (no other version changes;
checked with pnpm 9.15.4). Until then CI resolves it in place.
`scripts/typecheck-test-programs.mjs` now installs Daytona from its own
lockfile (`LOCKED_PROVIDERS`, with `--ignore-workspace`). A stale
lockfile is resolved in place for the run, for both installs.
- `.github/dependabot.yml`: add a comment that tells why the Pi and Kimi
runtime distributions have no entry (see Risks).
- `scripts/typecheck-sandbox-providers/pnpm-workspace.yaml`: list each
provider by its full path. Dependabot does not expand a workspace glob
that starts with `../`, so with the glob it saw no providers.
- `scripts/typecheck-test-programs.test.mjs`: add tests. They check that
the workspace list has no glob and matches `SANDBOX_PROVIDERS` without
`LOCKED_PROVIDERS`, that each locked provider has its lockfile, that a
locked provider installs frozen with `--ignore-workspace`, and that a
stale locked-provider lockfile is resolved in place.
- Add `.npmrc` (`auto-install-peers=true`) in
`scripts/typecheck-sandbox-providers` and in the Daytona provider.
Without it, Dependabot reads the root `.npmrc` (`false`). That rewrites
the lockfile settings, and the frozen install then fails. npm and pnpm
never publish `.npmrc`.
- `scripts/typecheck-sandbox-providers/package.json`: add
`"packageManager": "pnpm@9.15.4"`, the version CI uses. Without it,
Dependabot guesses pnpm 10.
- `doc/DEVELOPING.md`: say that contributors do not refresh or commit
Daytona's lockfile either, and that a new provider goes in both lists.

No provider moves into the root workspace. There is no pnpm catalog. The
root `pnpm-lock.yaml` does not change.

## Verification

I ran the real Dependabot updater on this branch with the Dependabot CLI
(`dependabot update ... --local`, updater image
`dependabot-updater-npm@sha256:c5a9451c…`):

- Before the change, the workspace entry loaded only
`cloudflare/bridge-template` (the one path without a glob). It skipped
all the other providers.
- After the change, `--dep typescript` gave one pull request. It changed
the 9 non-Daytona provider `package.json` files and the shared
`pnpm-lock.yaml`, and did not touch Daytona.
- `--dep wrangler` changed only the wrangler closure in the shared
lockfile. Before the `.npmrc` fix, the same run changed
`autoInstallPeers` to `false` and 757 lockfile lines.
- The Daytona entry (`--dep @daytonaio/sdk`) changed Daytona's
`package.json` and its own lockfile in one pull request, and kept
`autoInstallPeers: true`. It keeps the exact pin exact: `0.203.0`
becomes `0.227.3`.
- Both frozen installs (shared workspace, and Daytona with
`--ignore-workspace`) pass with pnpm 9.15.4.
- Dependabot installed `pnpm@9.15.4` from the new `packageManager`
field.

Other checks:

- `.github/dependabot.yml` passes the SchemaStore `dependabot-2.0`
schema (Ajv). A wrong key fails the same check.
- With pnpm 9.15.4, `install --resolution-only --ignore-scripts
--no-frozen-lockfile` in `scripts/typecheck-sandbox-providers` (the
lockfile bot's command) gives exactly the lockfile that passed the
frozen installs earlier in this pull request.
- The lockfile rule from #19522 allows this pull request's diff: it
changes no lock content.
- `node --test scripts/typecheck-test-programs.test.mjs`: the new test
and the provider tests pass. Two other tests need a root `node_modules`
(`tsc`), which my worktree did not have. CI runs them.

These checks cannot run before merge. GitHub reads `dependabot.yml` only
from the default branch. After merge, open Insights → Dependency graph →
Dependabot and make sure that the two new entries show no errors and run
on Monday.

## Risks

- A dependency in both a non-Daytona provider and Daytona (for example
`typescript`) gets two pull requests: one from each entry. Each one
passes alone, and they can merge in any order.
- The typecheck job now typechecks Daytona's tests against Daytona's own
lockfile instead of the shared one. Its versions are a little older (for
example vitest 4.1.11), so a type difference can show up there.
- Majors of shared dev tools (for example TypeScript 7, Vitest 5) come
as one pull request across all providers. It can need code changes
before it passes. That is the same as the root entry.
- Pi and Kimi get no version updates. Every direct dependency there is a
pinned provider release, and the pins feed the provider's ACPX
`commandDigest`. The Runner checks compare the Pi lock with
`PI_DISTRIBUTION_PINS`, and the Kimi build compares the lock SHA-256
with a pin. So any Dependabot change there fails CI (see #18998), and a
real upgrade starts saved sessions fresh. GitHub security alerts still
cover both lockfiles.
- The Pi zod `3.25.76` pin is still affected by GHSA-mrhx-c7xc-m3w5
(open alert). It is deliberate: `pi-acp@0.0.33` has `zod ^3.25.0` as a
direct dependency, not a peer. A fix needs a `pi-acp` release that
allows zod 4. This pull request does not change it.
- `packages/plugins/examples/plugin-orchestration-smoke-example` is also
outside the root workspace. It uses `workspace:*` dependencies, so it
cannot install alone, and this pull request does not cover it.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, local Docker runs of the
Dependabot CLI).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 11:08:39 -08:00
github-actions[bot]andlockfile-bot 17b4f972bf chore(lockfile): refresh pnpm lockfiles (#19534)
## Thinking Path

> - Paperclip uses a dependency lock for reproducible builds.
> - Dependency changes on master need matching lock entries.
> - Paid eval reports also pin a reviewed offline correction.
> - Reports must install before paid model work starts.
> - This automated PR keeps the lock and report proof together.

## Linked Issues or Issue Description

**What happened?**

Dependency manifests on master changed after the last committed lock
refresh.

**Expected behavior**

Frozen installs must match those manifests. Eval reports must retain
matching offline lock proof.

**Steps to reproduce**

Run a frozen install before and after this lock refresh.

## What Changed

- Refresh the dependency lock from the manifests on master.
- Refresh the sandbox provider lockfiles from their manifests on master.
- Include the matching offline report correction and hashes.

## Verification

- Pinned pnpm resolution completed with lifecycle scripts disabled.
- The report proof refresh accepted the reviewed committed baseline.
- The workflow rejected changes outside the lock and report proof files.

## Risks

The lockfile bot merges this PR without CI or review when every added
package version is at least 3 days old on npm. Otherwise the PR waits
for a maintainer, and the daily refresh tries again. The bot merges only
the exact commit this workflow generated from manifests already reviewed
on master. Unknown report lock baselines remain blocked. This update
changes no permissions or paid enable gates.

## Model Used

None — generated by pnpm and the deterministic report-lock updater.

## Checklist

- [x] Thinking path and problem description included
- [x] Generated lock and matching report proof included
- [x] Release-age gate decides between bot merge and maintainer review

Co-authored-by: lockfile-bot <lockfile-bot@users.noreply.github.com>
2026-11-06 18:09:43 +00:00
DottaandPaperclip c6573962f8 fix(workspaces): recover full sandboxes through read-only export (#19465)
## Thinking Path

> - Paperclip preserves agent work across failed runs and server
restarts.
> - A remote workspace must have a durable save before its source can be
released.
> - Portable capture normally writes a Git mirror and a full archive
inside the sandbox.
> - A full sandbox can prevent capture before the provider's streaming
download starts.
> - Retrying that write cannot create the required free space.
> - This change transfers a checked standalone source and seals its
archive on the controller.
> - The source remains retained unless the existing durable save
completes.

## Linked Issues or Issue Description

Related: #18644 reduced temporary capture space. #18172 added the
provider's disk-free directory transfer. This repair connects portable
capture to that transfer when the archive itself cannot be created.

**What happened?**

Workspace capture returns `remote_workspace_sandbox_disk_full` after
provider work has finished. File recovery repeats the same sandbox
write.

**Expected behavior**

Recover a stable standalone workspace without requiring another full
archive on the source disk. Preserve staged changes, untracked work,
confined links and the original source.

**Steps to reproduce**

1. Create a standalone Git repository with a commit, staged changes and
an untracked result.
2. Use the existing real-command transport fixture. Inject ENOSPC into
the sandbox helper write or archive write.
3. Run portable capture. Before this change it rejects the helper
failure.
4. With the change, restore the saved archive after removing the fixture
source. Check the commit, staged content, untracked result and Git
connectivity.

## What Changed

- Attempt one controller-side capture after an exact sandbox disk-full
failure.
- Read a source inventory without installing a helper. Use the existing
provider bulk transfer with confined links and runtime exclusions.
- Compare the transferred inventory with source fingerprints before and
after sealing.
- Reject copied Git pointers, alternates and administrative symlinks.
Parse config as data without includes; allow only portable keys before
controller Git normalization.
- Preserve socket exclusions. Retain the source and remove incomplete
controller staging after failure.
- Update execution semantics and focused regression coverage. No
database migration or new execution authority.

## Verification

- TDD red on base `2f35eb1080`: the new full-sandbox test failed with
`remote_workspace_sandbox_disk_full` before production edits.
- Green: 202 transport and archive tests passed before the additional
Git configuration guards. At head `8541086cb5`, all 116 transport tests
and the PostgreSQL durability acceptance pass. The final `e24e173ec9`
exclusion guard passes those 116 cases plus its new Git-history
regression across the full run and focused rerun. Environment: macOS
arm64, Node 26.4.0; real Node/Git commands and filesystem transfer.
- Review regressions were also red before their fixes: an internal Git
config symlink bypassed the check, and `core.attributesFile` could
select a controller path. Git administrative symlinks and nonportable
config now fail before normalization. The focused safety selection
passes 11 cases.
- Cases cover helper and archive ENOSPC, commit/staged/untracked
continuity after source removal, confined links, sockets, changed
source, corrupt or extra copied files, unsafe Git metadata, retained
diagnostic text and failed download.
- Real PostgreSQL acceptance passes: normal workspace save recovers one
sandbox ENOSPC, commits the encrypted snapshot, clears its writer, loses
the original allocation, then independently reloads the saved result
twice. The first acceptance attempt used an incorrect expected status
(`available`); the documented state is `committed`. No production change
was needed for that correction.
- Review regressions for untracked Python/Node environments and
excessive Git roots failed before their fixes. Dependency exclusions now
match normal capture. Controller parsing starts only after
whole-tree/repository budgets and layout checks; refspec/settings
budgets and a shared two-minute verification/normalization deadline
bound further work. The excessive-root test verifies zero Git
subprocesses.
- Real provider-boundary red/green: Daytona direct tar and
capacity-fallback PTY transfers first dropped `._result.txt`. The new
optional mapping contract preserves all user names and excludes only
current transfer scratch. Kubernetes implements the same contract.
Explicit caller exclusions and archive validation remain in force.
- Standalone Linux qualification on Node 24.22.2/Debian bookworm: fresh
shared/SDK build; Daytona and Kubernetes typecheck and build; Daytona
file-sync/capacity/diagnostics 68/68 pass; Kubernetes file-sync 31/31
pass, including the actual `/proc` case. Qualification was refreshed
again on the final `be6f7feacc` source tree: current shared/SDK rebuilt;
all 815 staged source/config/manifest hashes match the delivered tree
before and after testing. An initial container run as root bypassed a
read-only file test; the same source passes as uid 1000. Installing zstd
enabled eight initially skipped cases.
- A further review regression proved `.git/pyvenv.cfg` could exclude Git
history as a dependency. The inventory now handles `.git` before
dependency filtering, matching normal capture. The full capture/restore
test verifies the original commit and saved work. The first green
attempt incorrectly compared command-result object identity; correcting
the test to compare stdout passed with no production change.
- Further red/green regressions cover read-only private-copy cleanup, a
regular file replaced by a FIFO before open, case aliases of Git
administrative metadata, and permission preservation under restrictive
umask. Nonblocking no-follow file handles verify identity before and
after reading. Cleanup changes only private controller directories.
Provider extraction preserves ordinary permissions, refuses special mode
bits before writing, and does not adopt source ownership, ACLs,
attributes or flags. Final macOS Daytona tests pass 65/65; the final
Linux qualification above was repeated on these exact provider sources.
- Focused strict TypeScript compilation of the inventory and archive
modules passes.
- The first sandboxed archive run failed three Unix socket fixtures with
EPERM. The permitted run outside that filesystem sandbox passed all
tests.
- Initial full local server typecheck could not resolve the unbuilt
Runner package and produced cascading errors. Exact-head `e24e173ec9` CI
typecheck, build, Runner and browser checks pass. That CI attempt failed
two server shards: the unchanged retained-cleanup backlog test timed out
at 15 seconds and passes locally unchanged; ten cache-reclaim fixture
oracles failed because directory fallback consumed their sealed-archive
counters/faults. Final commit `caed4e4f83` isolates the intended
unavailable-directory boundary while preserving archive counts, recovery
status, integrity and no-second-reclamation assertions. Those ten
failures reproduced locally before the fixture edit. The full affected
file passes 394/395; its sole other HTTP socket-hang-up case passes
unchanged in an isolated rerun. This is combined coverage of all 395
cases, not a single complete green run. The final branch integrates
master `53645a0104` at `98ea07225592930cbb0d18bcf250bc64f6bb13b4`,
preserving upstream workspace phase timings. After integration: 117
transport cases, the PostgreSQL durability acceptance, the existing
object-storage/timing regression, and 12 affected cache cases pass. The
expanded recovery file now has 421 cases. Final-head
`be6f7feacc45c1e32a26a2417e973f1e3752fd8d` CI passes all 55 applicable
checks (three skipped, one neutral review), including the expanded
recovery suite, typecheck, builds and Runner coverage. [CI
result](https://github.com/paperclipai/paperclip/actions/runs/42251345554).
- Provider exclusion regressions were red through four real Linux
transfer paths: Daytona direct archive, direct safe-link repair,
capacity PTY repair, and Kubernetes streaming. A root socket and nested
regular file share a basename; literal glob-character files/subtrees
also survive. Preserve mode now anchors tar exclusions as literal
root-relative paths and escapes `find -path` pruning. Default callers
keep glob semantics. Receipt-count regressions were also red before
correcting preserved-output counting. All 31 Kubernetes and 68 Daytona
tests, builds and typechecks pass on the final source with 815 matching
inputs.
- The final semantics documentation explicitly records special mode
bits, the stricter config allowlist (including innocuous unlisted keys),
and provider qualification limited to Daytona/Kubernetes; these address
the optional current-head review observations without broadening
controller Git authority.
- No Daytona credential is available in this session. Provider transfer
is a local filesystem boundary in these tests. Hosted task recovery and
rollout remain unverified.

## Risks

- Recovery reads the source repeatedly and transfers the standalone
tree. It can take longer than normal capture.
- Linked worktrees, initialized submodules, Git administrative symlinks,
nonportable Git configuration, special mode bits and alternate object
stores remain retained if the normal capture cannot finish. This
fallback does not follow their pointers onto the controller.
- A provider without usable directory export, a changed source, or
insufficient controller space cannot produce a receipt. The original
source stays retained.
- The server, SDK and affected provider plugins must be built from this
revision together. An older provider can fail inventory verification on
names it omits; it cannot publish an incomplete save. No merge,
publication or deployment is included.

## Model Used

OpenAI GPT-6 family through Codex, with reasoning, code editing, shell
tools and automated tests. The exact serving model ID and context window
are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub references)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-06 11:00:32 -06:00
Nicky LeachandClaude Opus 5.5 e1332b997f fix(plugins): align tool run context and interaction input types (#19493)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Plugins add tools and issue-thread interactions through
`@paperclipai/plugin-sdk`
> - The SDK types are the public contract between the host and plugin
authors
> - Two parts of that contract do not match what the host does at
runtime
> - First, `ToolRunContext.projectId` says it is always a project UUID.
The tool gateway sends an empty string when a run has no project. The
direct tool route rejects such runs with 400
> - Second, three interaction helpers ask for the parsed output type, so
plugin authors must pass fields that the host fills in with defaults
> - This pull request makes the types and both tool paths agree with the
runtime
> - The benefit is that agents with no project can run plugin tools, and
plugin authors can omit the fields that have defaults

## Linked Issues or Issue Description

Fixes #3273

Related: #3274 removes the `projectId` requirement from the route. That
change alone is not enough today: the route now also looks up the
project to check its company, and that lookup needs a value. This pull
request also skips that lookup when the run has no project. It gives
plugins one value for "no project" on both tool paths.

The new test typecheck in #18739 and #18839 also listed both problems as
possible product problems.

## What Changed

- `server/src/routes/plugins.ts`: `POST /api/plugins/tools/execute`
accepts a `runContext.projectId` that is missing, `null`, or an empty
string. It passes an empty string to the plugin, the same value that the
tool gateway sends. It checks the project's company only when a project
ID is present. A `projectId` that is not a string returns 400.
- `server/src/routes/openapi.ts`: the request schema marks
`runContext.projectId` as optional and nullable.
- `packages/plugins/sdk/src/types.ts`:
- The `ToolRunContext.projectId` comment says the value is an empty
string when the run has no project.
- `suggestTasks`, `requestConfirmation` and
`requestCheckboxConfirmation` take `CreateIssueThreadInteractionInput`,
as `askUserQuestions` already does. The host parses this input with the
schema, so the defaults apply.
- `packages/plugins/examples/plugin-kitchen-sink-example/src/worker.ts`:
the example passes `undefined` instead of an empty project ID to
`issues.create`.
- `doc/plugins/PLUGIN_SPEC.md` (section 13.10): documents the empty
project ID.
- Tests:
- `server/src/__tests__/plugin-routes-authz.test.ts`: tests a run with
an empty, missing, or `null` project ID, which skips the project lookup.
Also tests that a project ID that is not a string returns 400.
- `packages/plugins/sdk/tests/testing-actions.test.ts`: removes the cast
that the output type required.

## Verification

- `pnpm exec vitest run server/src/__tests__/plugin-routes-authz.test.ts
server/src/__tests__/openapi-routes.test.ts
packages/plugins/sdk/tests/testing-actions.test.ts`: 3 files, 81 tests
pass.
- `cd server && pnpm typecheck`: passes. This includes the source layers
and the three test-tree shards.
- `tsc --noEmit` passes for `packages/plugins/sdk` (`tsconfig.json` and
`tsconfig.test.json`) and for the kitchen-sink example.
- Not run locally: the full `pnpm test:run` and the root `pnpm build`.
Required CI covers them.

## Risks

- Low risk. The direct tool route now accepts calls that it rejected
before. It still checks that the agent and the run belong to the
company. With no project ID, there is no project to check.
- A plugin that reads `runCtx.projectId` can get an empty string on both
tool paths. The tool gateway already sent this value before this change.
Only the documentation is new.
- The SDK parameter types are wider. Existing plugin code still
compiles.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits) and extended thinking.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 16:25:43 +00:00
Nicky LeachandClaude Opus 5.5 83c6832ccb fix(build): typecheck package, Playwright, and e2e test files in CI (#18839)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Runner, plugin, and Playwright tests verify how those agents run,
connect, and appear in the board
> - Most package tsconfigs exclude tests, and Vitest strips types, so a
type error in those tests fails nothing
> - The tests drifted to 803 real type errors, and some of those errors
hid assertions that checked nothing
> - paperclipai/paperclip#18739 and paperclipai/paperclip#18738 check
the server test tree and four test projects, but not these files
> - This pull request adds a required check for these test files, fixes
every error, and fails CI if a test file is in no program
> - The benefit is that a type error in any test file outside
`server/src` now fails CI

## Linked Issues or Issue Description

Refs #18739
Refs #18738

**What happened?**

517 test files were in no program that required CI checks. A type error
in those files did not fail CI.

**Expected behavior**

CI fails on a type error in any test file.

**Steps to reproduce**

Add `export const value: number = "text";` to a test file under
`packages/paperclip-runner/src` on master. CI stays green, because
`tsconfig.json` excludes test files and Vitest strips types.

**Measured gap**

Each row is a program with its tests included, on one thread. File and
error counts are for head `7d4b5e907a`. Memory peaks were measured
before the two latest merges of master.

| Group | Test files | Real errors | Peak |
| --- | ---: | ---: | ---: |
| `paperclip-runner` Node source | 290 | 644 | 1.09 GiB |
| `paperclip-runner` surfaces | 9 | 12 | 0.42 GiB |
| `paperclip-runner` devtools and examples | 14 | 0 | 0.68 GiB |
| `sandbox-providers/kubernetes` | 21 | 65 | 0.44 GiB |
| 8 other sandbox providers and `bridge-template` | 32 | 4 | 0.34 GiB |
| `plugins/sdk` | 22 | 15 | 0.41 GiB |
| `plugin-workspace-diff` | 5 | 0 | 0.37 GiB |
| `plugin-orchestration-smoke-example` | 1 | 1 | 0.15 GiB |
| `kv-demo-mcp-server` | 3 | 0 | 0.13 GiB |
| `packages/db` scripts | 1 | 0 | 1.01 GiB |
| 8 Playwright suites under `tests/` | 52 | 25 | 1.72 GiB |
| `tests/runner-acceptance` | 3 | 0 | 0.58 GiB |
| `tests/e2e`, through layer declarations | 63 | 37 | at most 1.59 GiB |
| `ui` `AttentionInteractionResolver.test.tsx` | 1 | 0 | n/a |
| **Total** | **517** | **803** | |

27 of the 803 errors came from master while this pull request was open.
Each merge of master brought new test code that no check had read.

Errors that were config errors, not real errors:

- One TS2688 for each provider, because the root install does not
install provider dependencies.
- 9 in the Runner Node program (TS7016 and TS6059).
- 2 in `tsconfig.browser.json`. TypeScript 7 removed `baseUrl`, so that
config was invalid.
- 213 in the Playwright suites (`@/*` aliases and Vite types).
- About 60 in `tests/e2e` (import extensions and aliases).

`packages/plugins/create-paperclip-plugin` has no gap. Its tsconfig
excludes its test, but `cli/tsconfig.json` reaches that test through an
import. `server/test-baselines` also has no gap, because
`tests/lifecycle-baseline` reads it.

## What Changed

- Added `scripts/typecheck-test-programs.mjs` and its tests. The
`Typecheck + Release Registry` job runs it. It checks each program in
`PROGRAMS` on one thread.
- The script fails if a tracked `*.test.ts(x)` or `*.spec.ts(x)` file
outside `server/src` is in no program. The compiler file list decides
this, because `include` drops `foo.test.tsx` when `foo.test.ts` exists.
- Added a `tsconfig.test.json` to each uncovered package, and
`tests/tsconfig.playwright.json`.
- Added `tests/e2e` to `PROJECTS` in
`scripts/typecheck-test-projects.mjs`, with a `tsconfig.json` and a
`tsconfig.layers.json`.
- Added `scripts/typecheck-sandbox-providers`. It is a small workspace
that installs provider SDKs from one lockfile. I did not add a lockfile
inside each provider, because
`scripts/build-standalone-public-packages.mjs` uses a frozen install
when one exists, and that changes what a release resolves.
- Corrected `packages/paperclip-runner/tsconfig.browser.json` for
TypeScript 7.
- Set `rootDir` to `..` in
`packages/paperclip-runner/tsconfig.test.json`, because one Runner test
imports from `packages/adapter-utils/src`. The program does not emit
files, so this changes no build output.
- Moved the Kubernetes provider from `vitest ^3.2.4` to `^4.1.8`, the
version the other providers use. Vitest 3 requires `tinypool ^1.1.1`,
which has two critical advisories (GHSA-85c8-ppgw-ccpr and
GHSA-5gmw-xhrv-c9v3). The first patched version is 2.1.2. Its tests have
the same result on both versions: 189 passed, and the same 12
`file-sync` tests failed.
- Moved the Kubernetes provider from `zod ^3.24.2` to `^4.6.6`, and gave
its two `z.record` calls a key schema. This is the only production
change, and it stays in this pull request. See "Production change for
review".
- Renamed `AttentionInteractionResolver.test.ts` to
`AttentionInteractionResolver.replace.test.ts`, so TypeScript reads the
`.test.tsx` beside it.
- Fixed all 803 errors, with changes to test files and test support
only. The fix for `contracts/native-execution.test.ts` is the same as
the fix in paperclipai/paperclip#18739, so those two changes merge
cleanly.
- Updated `doc/DEVELOPING.md`.

This pull request does not edit `server/scripts/typecheck-layers.mjs`.
paperclipai/paperclip#18739 and paperclipai/paperclip#18738 have both
merged, and this branch includes them.

### Production change for review

The new lockfile made dependency review flag `zod@3.25.76` as a
high-severity advisory (GHSA-mrhx-c7xc-m3w5). The first patched version
is 4.6.6, so 3.x has no fix. The published Kubernetes plugin uses zod to
validate operator config. The zod 3 source does not compile against zod
4, so a typecheck-only override cannot fix this.

The existing tests do not prove this change, because they also passed
with the invalid one-argument `z.record`. So I parsed 45 inputs with the
same schema on zod 3.25.76 and zod 4.6.7:

| Result | Inputs | Detail |
| --- | ---: | --- |
| Identical | 23 | Includes defaults, valid config, and a `__proto__`
key |
| Same outcome, new error wording | 21 | For example, `Invalid url` is
now `Invalid URL`. `validateConfig` shows this text to operators. |
| Different outcome | 1 | `jobTtlSecondsAfterFinished: 2 ** 60` was
valid and is now rejected. The new limit is `Number.MAX_SAFE_INTEGER`. |

I did not test this against a real cluster.

The decision is to keep this change in this pull request. Since then,
paperclipai/paperclip#17364 has moved the rest of the repository to zod
4.6.x, so the Kubernetes plugin now matches it.

### Behavior change for review

`reducedMotion` under `use` is not a Playwright option. It had no effect
in 11 configs and 2 `test.use` calls, so those suites ran with
`no-preference`. The options are now under `contextOptions`, so reduced
motion now takes effect. `app-shell-design.spec.ts` asserts a real
transition, so it now asks for `no-preference` itself.

The baseline manifest has `snapshotCount: 0`, so this invalidates no
published baseline. A frozen trial reference PNG can differ.

### Findings

These are product questions. I did not change them, and I did not hide
them with casts:

1. **`PrpEvent` and its schema disagree.** The TypeScript type in
`src/protocol/replay-contract.ts` requires `normalizedSessionId`. The
`required` list in `protocol/schemas/event.schema.json` does not include
it. `validatePrpEvent` accepts events that the type forbids.
2. **`requestCheckboxConfirmation` is stricter than its runtime.** It
uses the output type, so plugin authors must pass `continuationPolicy`
and `allowDeclineReason`, although the schema gives them defaults.
`z.input` would fix this.
3. **`RetainedRunnerdMaintenanceInput.backend.kind` is a free string.**
The quarantine domain comes from `descriptor.kind`, but the release uses
`proof.backend.kind`. If they differ,
`completeRetainedNativeSessionCleanup` returns 0 and the quarantine
stays. The production caller passes `"runner"` today, so this is not a
live bug.
4. **`CodexHarnessSession.close(input?)` is optional, but the contract
requires `{ reason }`.**

These are tests that verified nothing. I corrected them, and they pass:

- `backends/native-instruction-measurement.test.ts` filtered on
`placement === "core"`, which is not a valid placement. It measured an
empty tool list. It now measures 14 tools.
- `drivers/acpx/codex-acpx-driver.test.ts` checked for `usage.updated`,
which is not an event type.
- The `hostFixture()` helper in `drivers/acpx/runtime-host.test.ts`
ignored two overrides, so
`expect(prepareSandbox).not.toHaveBeenCalled()` could not fail.
- `live/live-session.test.ts` set `agent: "pi"` before a spread that
overwrote it.
- `native-distribution-integrity.test.ts` passed an AbortSignal where
`openCommand` expects `{ signal }`.

These are gaps that this pull request does not close:

- **The root lockfile still has some old zod versions.**
paperclipai/paperclip#17364 moved the direct zod dependencies to 4.6.x.
`pnpm-lock.yaml` still resolves `zod@3.25.76`, `4.1.11`, and `4.4.3`,
which are below the patched 4.6.6.
`packages/paperclip-runner/scripts/pi-distribution/package.json` pins
`zod` to `3.25.76`, and some agent SDKs resolve `4.4.3`. Dependency
review does not flag them, because this pull request does not add them.
- **Dead GitHub coverage.** `chat-adapters-ui-providers.spec.ts`
returned early for GitHub, so four later `provider === "github"`
branches never ran. I removed them. They included the private key file
and paste match, and the reconnect flow. Nothing asserts those behaviors
in that spec now.
- **Provider tests do not run in CI.** The root `vitest.config.ts` lists
only Daytona. The other 8 providers, `bridge-template` included, are
typechecked now, but CI does not run their tests.
- **One `packages/db` test never runs.**
`scripts/clean-poisoned-claude-sessions.test.ts` is outside the
`include` of `packages/db/vitest.config.ts`.
- **`plugin-orchestration-smoke-example` does not collect.**
`@paperclipai/shared` is not linked in its `node_modules`. It passes 3
of 3 with an alias.

## Verification

Run these commands:

```sh
node scripts/typecheck-test-programs.mjs --low-memory
pnpm --filter @paperclipai/server typecheck
node scripts/typecheck-test-projects.mjs --low-memory --reuse-declarations
node --test scripts/typecheck-test-programs.test.mjs
```

Results on macOS (arm64), Node 26.12. CI uses Node 24, and its
`Typecheck + Release Registry` job also passes. At head `7d4b5e907a`,
the test programs, `tests/e2e`, and the script tests pass again with 0
errors. The times and peaks below come from the run before the two
latest merges of master:

| Check | Result |
| --- | --- |
| Test programs, `--low-memory` | Exit 0, 0 errors, 42 seconds, peak
1.78 GiB |
| Server layers, low memory | Exit 0, 0 errors, 38 seconds, peak 3.41
GiB |
| Test projects, including `tests/e2e` | Exit 0, 0 errors, 13 seconds,
peak 1.59 GiB |
| Script tests | 12 passed |
| Injected type error in each of 23 places | Each one failed its program
with TS2322 |
| Coverage check | 0 unchecked files, 10 seconds |

I did not run this in a 4 GiB container. The peaks above are `maximum
resident set size` for the largest single process.

**Runtime check of the fixes.** Types do not prove that a test still
works, so I ran the suites before the two latest merges of master:

| Suite | Result |
| --- | --- |
| Runner, full suite | 4,885 passed, 3 failed, 13 skipped |
| Sandbox providers, 10 packages | 939 passed, 12 failed, 36 skipped |
| Plugin SDK | 200 passed |
| `kv-demo-mcp-server` | 12 passed |
| Renamed UI test and its sibling | 4 passed |

All 15 failures exist without these fixes or come from the local
machine:

- **3 Cursor Cloud timeouts.** `artifact-delivery.test.ts` has 1 and
`cursor-cloud-driver.test.ts` has 2. They time out in a full local run,
and all 133 tests in those files with `codex-security-config.test.ts`
pass when run alone. They fail in the same way with those two files at
their unedited versions.
- **12 Kubernetes `file-sync.test.ts` failures.** They fail in the same
way before and after the fixes. The first failure is that
`/proc/self/fd` does not exist on macOS.

After each of the two latest merges of master, I ran only the Runner
test files that I changed:

| Merge | Result |
| --- | --- |
| `885c249cb9`, 8 files | 534 passed |
| `7d4b5e907a`, 7 files | 672 passed, 16 failed |

The 16 failures are in the two `source-reconciliation` test files. They
fail in the same way with master's unedited versions of those files. 15
of them fail with `ACPX sidecar exited at initialize`, so I think my
local build of the Rust and sidecar binaries is out of date. I did not
rebuild them to check. CI builds them from scratch, and all Runner jobs
pass in CI at `7d4b5e907a`.

**Not run:**

- `tests/e2e` and the 8 Playwright suites need a server, so most of them
have only compile checks. `playwright test --list` loads 308 tests from
the main e2e config.
- `packages/plugins/sandbox-providers/kubernetes/test/integration` needs
a cluster, so it has only compile checks.
- I did not run the full `pnpm test:run` or root `pnpm build`. Required
CI runs them.

**Reduced motion.** I ran 6 motion-sensitive Storybook specs in system
Chrome with a temporary config, once with `reduce` and once with
`no-preference`. Both runs had 45 passed and the same 5 failed. So
reduced motion did not change those results. The 5 failures do not
depend on motion. At least one comes from the browser, because system
Chrome prints `oklch(55.6% 0 0)` where the test expects `oklch(0.556 0
0)`. I did not run the other Storybook specs. The `storybook-visual`
label runs that workflow.

## Risks

- **A new lockfile to maintain.** A change to a provider `package.json`
now fails the typecheck job until someone refreshes
`scripts/typecheck-sandbox-providers/pnpm-lock.yaml`. The failure
message gives the command.
- **The job has a new install.** The step downloads provider SDKs, so it
depends on the npm registry. It uses `--frozen-lockfile` and
`--ignore-scripts`.
- **The local install replaces provider `node_modules`.** A contributor
with a standalone provider install must reinstall that provider after
running the script.
- **Reduced motion can change Storybook results.** See "Behavior change
for review". That workflow is not part of required CI.
- **Kubernetes plugin validation changed.** A release of this plugin
ships zod 4. Error wording changes, and very large integers are now
rejected. See "Production change for review".
- **Master moves fast.** Each merge of master brought new test code with
type errors, 27 so far. After this merges, the check reports those
errors in the pull request that adds them.

## Model Used

- Provider: Anthropic Claude, through Claude Code
- Model: Claude Opus 5.5 (`claude-opus-5-5`)
- Capabilities: tool use, code execution, and subagents for the type
fixes in separate files

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [ ] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived
details. The app generated the branch name.
- [ ] I have run tests locally and they pass. See the failures explained
in Verification.
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 07:33:59 -08:00
Nicky LeachandPaperclip 26071353d2 feat(plugins): define settled task release and resume (#19468)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Environment plugins provide Runner tasks through the public SDK.
> - A settled turn can leave resources that a later run can reuse.
> - Core needs a shared contract to end active use and request the same
logical session later.
> - The host must retain control of capacity and report when reuse is
unavailable.
> - This pull request adds an optional release and resume contract with
strict validation and documentation.
> - The benefit is a common boundary for later host and Core integration
across Codex and ACPX.

## Linked Issues or Issue Description

**Subsystem affected**

packages/plugins — plugin SDK and its environment task documentation.

**Problem or motivation**

The SDK has stop and completion operations. It has no explicit contract
to release active use after a settled turn and later request the same
logical session without promising retention or the same process.

**Proposed solution**

Add a versioned optional capability, release for the identified session,
and resume with fresh transient authentication. Define two explicit
resume outcomes: resumed or unavailable. Keep authorization, safe
retries and ordinary attachment requirements explicit.

**Alternatives considered**

A provider-specific command would couple Core to one host. Reusing stop,
completion or archive would change their existing meaning. A new
checkpoint format or Runner mode is unnecessary.

**Roadmap alignment**

This extends the existing plugin and sandbox boundaries in ROADMAP.md.
It adds no Core caller, provider implementation or runtime activation.

**Additional context**

The duplicate search found #19371, which fixes Core failure cleanup for
a session resume owner. Its scope is separate. Related recovery work in
#19157 and admission work in #19196 remain unchanged.

## What Changed

- Add strict release and resume operation/result schemas and optional
releaseResumeVersion: 1.
- Reuse the existing task identity, inferred SDK exports and worker RPC
types.
- Define settled release, host retention policy, atomic reuse
reservation, fresh authentication, retries that cannot release a newer
owner and ordinary recovery.
- Add schema and serialized worker RPC tests, including legacy replies
and failure cases.

## Verification

- SDK typecheck: `node_modules/.bin/tsc -p
packages/plugins/sdk/tsconfig.json --noEmit` passed.
- From `packages/plugins/sdk`, `../../../node_modules/.bin/vitest run
--config vitest.config.ts tests/environment-task-release-resume.test.ts
tests/environment-tasks.test.ts tests/environment-task-error.test.ts
tests/environment-task-error-code.test.ts tests/worker-rpc-host.test.ts`
passed: 94 tests across five suites.
- `git diff --check` passed.
- [Full GitHub
CI](https://github.com/paperclipai/paperclip/actions/runs/42238743059)
passed on `a55376801c91af29b68a13304d085706e4d7ed6f`. Standard Greptile
scored that head 5/5 with no comments or unresolved review threads.
- Full regressions run in GitHub CI, per the requested verification
scope. No VM, deployment or live provider test was run. These tests
prove the SDK boundary, not host retention or live attachment.

## Risks

- This is an optional contract. Runtime behavior stays unchanged until
later host and Core work uses it.
- Older strict Core schemas reject the new description field. Providers
must coordinate capability advertisement with a compatible Core.
- Host implementations must enforce settlement, exact authority,
exclusive claims and safe retries. Unknown outcomes remain errors. An
unavailable result never proves physical termination.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

OpenAI GPT-6.1 Sol (`gpt-6.1-sol`), high reasoning, with repository
inspection, code editing and local test tools. Context window size was
not exposed by the runtime.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-06 07:23:28 -08:00
DottaandClaude Opus 5.5 a4bb059f59 fix(heartbeat): retry transient sandbox provider lease refusals and explain the block (#19412)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A task runs in an execution environment. A managed sandbox
environment gets a lease from a sandbox provider (for example Daytona)
before the agent starts.
> - When the provider answered sandbox create with a temporary error
(HTTP 502), the run failed as the generic `setup_failed`.
> - `setup_failed` is the fail-closed code for unknown setup errors. It
gets no automatic retry, so the task went to `blocked` at once. The task
showed only "Couldn't finish this attempt".
> - A user had to find and press Retry. The Retry then worked, which
shows that the failure was temporary.
> - This pull request classifies temporary provider refusals during
lease acquisition from structured fields. It retries them within the
existing bounded bootstrap budget, and it explains the block when the
retries are used up.
> - The benefit is that a short provider outage no longer stops a task.
When the outage continues, the task shows the cause and a Retry that
works.

## Linked Issues or Issue Description

**What happened?**
A task was assigned to an agent that uses a managed Daytona sandbox
environment. Sandbox creation returned `HTTP 502`, and the cleanup of
the provisional sandbox could not be confirmed. The run failed with
`setup_failed` and the error `Failed to acquire lease for environment
"Paperclip Sandbox" (sandbox): Daytona sandbox creation failed (HTTP
502); cleanup could not be confirmed for paperclip-create-…`. Three
seconds later the task was `blocked`. No automatic retry was scheduled.
The task showed no specific reason. A manual Retry started a new run
that completed normally.

**Expected behavior**
A temporary provider error (HTTP 5xx, 429, timeout) before the agent
starts must retry automatically with bounded backoff. The task must show
why it waits. If the retries are used up, the task must show the cause,
a next action, and a Retry that works on the first click.

**Steps to reproduce**
1. Configure an agent with a sandbox environment.
2. Make the provider return HTTP 502 on sandbox create.
3. Assign a task to the agent.
4. See that the run fails with `setup_failed` and the task goes to
`blocked` with no retry.

**Paperclip version or commit**
`master` at `612da1e8e0`.

## What Changed

- Plugin SDK: an `environmentAcquireLease` error now carries a closed
diagnostic envelope (HTTP status or a closed errno code) across the
worker RPC. It takes the status of the create call only, not of the
compensating cleanup lookup. The existing creation-cleanup ownership
envelope is unchanged. New module: `environment-acquire-error.ts`. The
resume diagnostic helpers are shared.
- Server: new `EnvironmentProviderUnavailableError` and code
`environment_provider_unavailable`. The environment run orchestrator
sets it as the cause of `lease_acquire_failed` only when structured
evidence shows a transient status (408, 425, 429, 500, 502, 503, 504, or
a closed connection errno). It never parses message text. Saved-sandbox
resume and stopped-worker refusals keep their own codes.
- Setup failure (`failRunSetup`): a fresh, pre-dispatch run records
`environment_provider_unavailable` with a `providerFailure` (phase
`startup`, retryability `transient`). The run uses the bounded bootstrap
retry (30 s, then 60 s). An accepted interaction continuation keeps its
own continuation retry, as for `plugin_worker_unavailable`.
- Recovery: the new code is registered with the other bootstrap codes.
It is not sent to the generic continuation retry. It does not create an
execution-reconciliation hold, because no provider work started. On
exhaustion the stranded notice has the title "Sandbox provider was
temporarily unavailable" and a specific next action.
- UI: `ProviderFailureNotice` labels the `sandbox` provider as "Sandbox
provider". The task notice shows the provider failure, "Paperclip will
retry automatically" while a retry is scheduled, and Retry after
exhaustion.
- Docs: `doc/execution-semantics.md` describes the policy and the
sandbox ownership rule.

## Verification

Red before the fix, green after the fix:

- `server/src/__tests__/heartbeat-process-recovery.test.ts` › "transient
sandbox provider refusal during lease acquisition":
- "retries automatically and runs exactly one successful successor
without user action": the real heartbeat. The lease runtime returns a
Daytona 502 once, then delegates. Before the fix: one `failed` run and
no successor. After the fix: one `scheduled_retry` successor with no
hold and a `retry_scheduled` presentation. After promotion, exactly one
successor succeeds.
- "explains the block with a working Retry once automatic retries are
used up": three refusals. The task is `blocked` with a "Sandbox provider
was temporarily unavailable" notice. The presentation is `{ state:
"failed", canRetry: true, providerFailure: { httpStatus: 502 } }`, and
no execution-reconciliation hold exists. One `requestTaskRetry` is
accepted, and its run succeeds.
- `server/src/services/heartbeat/run-completion.test.ts` › "routes a …
sandbox provider lease refusal through bounded bootstrap recovery": the
real orchestrator wrapping. Only the fresh, structured 502 is retried. A
message-only 502, a 400, a run after provider dispatch, and a retained
native run stay `setup_failed`.
- `packages/plugins/sdk/tests/environment-acquire-error.test.ts`: the
real worker RPC. The test checks the create status (not the 404 cleanup
lookup), the errno, no provider text, the unchanged cleanup envelope,
and that forged envelopes are rejected.

Commands:

```sh
pnpm --filter @paperclipai/plugin-sdk exec vitest run tests/environment-acquire-error.test.ts tests/environment-creation-cleanup.test.ts tests/environment-resume-error.test.ts
cd server && npx vitest run src/__tests__/heartbeat-process-recovery.test.ts src/services/recovery/ src/services/heartbeat/run-completion.test.ts src/services/heartbeat/retries.test.ts src/services/legacy-execution-recovery.test.ts src/services/task-interruption-presentation.test.ts src/services/task-retry.test.ts src/__tests__/environment-runtime.test.ts src/__tests__/heartbeat-retry-scheduling.test.ts
cd packages/plugins/sandbox-providers/daytona && npx vitest run
pnpm --filter @paperclipai/plugin-sdk typecheck && pnpm --filter @paperclipai/server typecheck
```

I did not run a live Daytona 502 on staging. Re-check on staging after
deploy (see Risks).

## Risks

- Sandbox leaks or double creation: each retry is a new run with a new
durable creation record and a new provider name
(`paperclip-create-<operationId>`). The failed attempt stays owned by
its own record, and the existing creation reconciler settles it. A retry
never reuses or abandons it. While a provisional sandbox waits for
reconciliation, up to two more sandboxes can be created for the same
task. This is the same as a manual Retry today.
- Plugins built with an older SDK do not send the envelope. Their
refusals stay `setup_failed`, which is the current behavior.
- An in-process error with `status`/`statusCode` 500–504 or a connection
errno during lease acquisition is now retried twice. The budget is
bounded and shared with the other bootstrap codes.
- An RPC timeout on the host side is not classified by this change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code as an agent with tool use (shell, file edits, test runs).
Extended reasoning was on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 00:08:44 -06:00
c9c4949c86 build(deps): bump zod from 4.4.3 to 4.6.6 (#17364)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The shared validators use zod to check API input, and the Runner
contracts check some of the same values again
> - zod below 4.6.6 has a high-severity memory issue
(GHSA-mrhx-c7xc-m3w5), so this bump is necessary
> - zod 4.6 also changes string `min`, `max`, and `length`: they now
count Unicode code points, not UTF-16 units
> - The Runner question parser and the goal readers still use JavaScript
string length, so emoji text can pass zod and then fail later
> - This pull request bumps zod and keeps the string-length bound on
those schemas
> - The benefit is that we get the security fix, and these inputs still
fail at validation

## Linked Issues or Issue Description

Refs GHSA-mrhx-c7xc-m3w5.

**What happened?**
With zod 4.6, three CI tests failed on this bump. Two showed an emoji
body that is over the agent commentary ceiling and that zod now accepts.
The third is the scheduler timeout described under Verification.

**Expected behavior**
A dependency bump does not widen an API limit. Text over the limit gets
a validation error.

**Impact**
Emoji text between the two bounds passed validation and failed later:
- A question set is created, and then each answer gets a 422. The
question stays pending and nobody can answer it. A schema and parser run
confirmed the mismatch. This was not run over HTTP.
- The Muse route returns a 500 because `parsePaperclipQuestionSet`
throws an error that is not an `HttpError`. This comes from reading the
code. It was not run.
- A Runner goal normalizes to null and replaces the stored goal, or it
is marked blocked. This comes from reading the code. It was not run.

## What Changed

- Bump zod to `^4.6.6`. The lockfile resolves 4.6.7. (Dependabot)
- Add `maxStringLength` to `packages/shared/src/validators/text.ts`. It
checks zod `max` and JavaScript string length.
- Use `maxStringLength` for the question-set text fields, the Runner
goal objective, and the agent commentary body.
- Add regression tests for the question-set bound and the goal objective
bound.
- Dependabot recreated the branch on current `master` (`@dependabot
recreate`) to resolve `pnpm-lock.yaml` conflicts. This happened twice.
The lockfile is Dependabot's. Each time, the two fix commits were
applied again without changes.

## Verification

- Compared zod 4.4.3, 4.6.5, 4.6.6, and 4.6.7 directly. The change to
`max` is in 4.6.5 or earlier, and 4.4.3 does not have it.
- Passed locally: `packages/shared` tests `agent-commentary`,
`validators/issue`, and `runner-goal`, plus `tsc --noEmit` in
`packages/shared`.
- Passed locally on the first fix commit: server
`agent-commentary.integration`, `agent-commentary-helper`, and
`openapi-routes`. The later commit changes these schemas again. CI
passed these tests at `b5e9dc7352`.
- Local testing was focused. GitHub CI is the full regression check. CI
passed at `b5e9dc7352`, including `ci / verify` and `ci / e2e`.
- Greptile reviewed `b5e9dc7352`. The score is 5/5, with no inline
comments.
- After the recreate, the fix commits applied cleanly as `bad62c3ba5`
and `3294552e65` on Dependabot's `46789d1487`. Their diff is the same as
the diff Greptile reviewed. The lockfile still resolves zod 4.6.7. The
focused `packages/shared` tests (`validators/issue`, `runner-goal`) and
`tsc --noEmit` in `packages/shared` passed again locally. A second
`pnpm-lock.yaml` conflict came before CI finished at `3294552e65`.
- Second recreate: Dependabot ignored the first `@dependabot recreate`
(no reaction and no update job after 20 minutes). It acted on the second
command and pushed `83c328c15c`. The fix commits applied cleanly as
`dd3e87d93e` and `95b450048f`, with the same diff. The lockfile still
resolves zod 4.6.7. The same focused tests and `tsc --noEmit` passed
again locally. Greptile first scored `95b450048f` 4/5 with one P2: the
description did not explain the source changes. Greptile read
Dependabot's replacement description, which was live during that review.
The description was restored and the thread resolved. The new review
scored `95b450048f` 5/5 with no findings. CI failed at `95b450048f` in
two shards. Neither failure comes from the zod change:
- `server (9/12)`: `server/scripts/reporting-lock-policy.test.mjs` fails
with "Reporting lock changed; review the dependency correction before
install". #19283 added this test today. It requires the committed
`pnpm-lock.yaml` hash to match `tests/runner-e2e/report-lock.json`.
Every pull request that changes the lockfile fails it until the hashes
are renewed, for example #17172 and #17911.
- `workspaces-a (1/2)`:
`cli/src/__tests__/company-import-export-e2e.test.ts` timed out waiting
for its child server. The same test failed the same way on two unrelated
branches in the same hour (runs 42115721271 and 42109846042). A rerun of
this shard at `95b450048f` passed. The rerun of `server (9/12)` failed
again on the reporting-lock test.
- #19356 fixed the reporting-lock gate for Dependabot lockfiles and
merged. A re-run reuses the old merge commit, so the empty commit
`482d76686e` starts a run against current `master`. `pnpm-lock.yaml` and
the reporting-lock proof on `master` did not change since this branch's
base.
- CI at `482d76686e`: `ci / policy` passed, and `server (9/12)` skipped
the reporting-lock test as #19356 intends. `e2e shard (6/8)` failed once
in `cloud-auth.spec.ts` ("apiResponse.json: Response has been disposed",
a Playwright race in a route handler). The same test passed at
`95b450048f` with the same code. A rerun passed. All checks pass,
including `ci / verify` and `ci / e2e`.
- Earlier failure: `issue-monitor-scheduler.test.ts` ("lets the board
check a scheduled monitor on an in_progress task") timed out in CI at
`d60b5f56d8`. It passed 5 of 5 local runs with the first fix. The cause
is unknown. The test waits 3 seconds for heartbeat runs to settle, so CI
load is possible, but this is not proven. The test passed in the CI run
at `b5e9dc7352`. This shows recovery. It does not explain the earlier
timeout.

## Risks

- All other zod string bounds now count code points. Emoji text can be
up to 2N UTF-16 units and 4N UTF-8 bytes (it was N units and at most 3N
bytes). This pull request does not change those bounds.
- An audit of those bounds found no other concrete failure. It checked
index limits, byte limits, `.length` guards, provider limits, and
security `min` checks. It used code search and sampling, so it is not
exhaustive.
- Not verified: Slack `topic` and `purpose` use `.max(250)`, and we did
not confirm how Slack counts that limit.
- `.min(N)` is now stricter for emoji text. A single emoji no longer
passes `.min(2)`.
- `@dependabot rebase` or `@dependabot recreate` on this branch will
remove the manual commits. This happened twice to resolve lockfile
conflicts, and the commits were applied again each time.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code
execution, in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (focused tests only; see
Verification)
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
documentation change needed)
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green (at `482d76686e`)
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
(5/5 at `95b450048f`)
- [x] I will address all Greptile and reviewer comments before
requesting merge

---

Bumps [zod](https://github.com/colinhacks/zod) from 4.4.3 to 4.6.6.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/colinhacks/zod/releases">zod's
releases</a>.</em></p>
<blockquote>
<h2>v4.6.6</h2>
<blockquote>
<p>Six malicious commits pushed to <code>main</code> on September 21
with a compromised git credential were reverted the same day in <a
href="https://redirect.github.com/colinhacks/zod/issues/6620">#6620</a>
and have since been removed from the history of <code>main</code>. None
were ever published to npm. The npm provenance for 4.6.6 names 284243fb,
the pre-rewrite copy of 9c219833 with identical contents.</p>
</blockquote>
<h2>Commits:</h2>
<ul>
<li>0cd3af5e9f50ad11bbcd4407a46a21b8848ba574 docs: load the en locale
directly in Zod Mini</li>
<li>9d5e60f162997b764b40d45aa02481888db50dea docs: import locales from
zod/locales, not z.locales</li>
<li>34a626f307871ecbb0123947559328dc52ed5e3e docs: tighten the z.locales
tree-shaking note</li>
<li>74304ce5b5796b5edb9a88ceb8e170b6ea3e82df release: stage npm versions
instead of publishing them (<a
href="https://redirect.github.com/colinhacks/zod/issues/6622">#6622</a>)</li>
<li>b2ef983e0711ff6517d6020b021071e2e5653efa docs: add Code for Japan to
bronze sponsors</li>
<li>8fb2faf0584828a6a027d8279db36f6f52784462 docs: crop the Code for
Japan logo and list it first in bronze</li>
<li>bacad518c41610d654658172d325874bf0ee6290 docs: square the Code for
Japan logo to match the other bronze tiles</li>
<li>c42596bd1c87e941d7241d10e012c5c196adcc10 release: hold the run on
the release-approval environment until the staged versions are approved
(<a
href="https://redirect.github.com/colinhacks/zod/issues/6638">#6638</a>)</li>
<li>a2ba4a67986a3e6b2b131be30084a2f460d9ea67 fix(types): declare record
output and input as members, not base type arguments (<a
href="https://redirect.github.com/colinhacks/zod/issues/6645">#6645</a>)</li>
<li>a0ebf970f2d752e5f58527cd2942ff48be4f46bd fix(v4): check array, set
and map size bounds before parsing elements (<a
href="https://redirect.github.com/colinhacks/zod/issues/6652">#6652</a>)</li>
<li>529f78dd9431b07f49647fee3aa0819b3f8b66b7 4.6.6</li>
<li>9c219833c3a3764fa1f1ea9c5ecec8254a63c0b1 ci: stage at verbose so a
failed OIDC token exchange says why</li>
</ul>
<h2>v4.6.5</h2>
<h2>Commits:</h2>
<ul>
<li>d2b135cfb7a3582b9eb515756b9166bcb9521f4a docs: add the 4.6.x patch
highlights to the 4.6 post</li>
<li>f1448f7cee00df9fe1e9ad84a000aa1828cc8bc1 docs: fold the 4.6.x patch
highlights into the 4.6 post's own sections</li>
<li>de65a5cb39ed22a507fac935788f718fa88d104f docs: lead the properties
section with the check and add a Zod Mini tab (<a
href="https://redirect.github.com/colinhacks/zod/issues/6598">#6598</a>)</li>
<li>56222cd1532c07bcb91b67df529cab4c0a215330 feat(instanceof): key the
.properties() shape off the instance type (<a
href="https://redirect.github.com/colinhacks/zod/issues/6600">#6600</a>)</li>
<li>ca0229a404818290e6cdcfefcd7eb2d04bcbb543 Revert &quot;feat: add
z.currencyCode() over a vendored ISO 4217 list, refreshed weekly by CI
(<a
href="https://redirect.github.com/colinhacks/zod/issues/6595">#6595</a>)&quot;</li>
<li>cc4cd4ee9c52fcaa10964e48cc144541e41a5ed9 Revert &quot;Revert
&quot;feat: add z.currencyCode() over a vendored ISO 4217 list,
refreshed weekly by CI (<a
href="https://redirect.github.com/colinhacks/zod/issues/6595">#6595</a>)&quot;&quot;</li>
<li>0f3f5ee3ca56c7574bf849e54f79e9a6e02562ee 4.6.5</li>
<li>59bbc03e10c636b9eb3c393dfeb552819774ec21 chore: re-pin the
integration peers to the workspace zod after the 4.6.5 bump</li>
</ul>
<h2>v4.6.4</h2>
<p>A patch on top of <a
href="https://github.com/colinhacks/zod/releases/tag/v4.6.3">4.6.3</a>.</p>
<ul>
<li><a
href="https://github.com/colinhacks/zod/commit/d6bc1e30"><code>d6bc1e30</code></a>
feat: add <code>z.currencyCode()</code> over a vendored ISO 4217 list,
refreshed weekly by CI (<a
href="https://redirect.github.com/colinhacks/zod/pull/6595">#6595</a>)</li>
<li><a
href="https://github.com/colinhacks/zod/commit/ad32d751"><code>ad32d751</code></a>
perf: <code>z.url()</code> rejects an invalid URL with
<code>URL.canParse()</code> instead of a throwing constructor, about 50x
faster; fewer allocations on the validation path (<a
href="https://redirect.github.com/colinhacks/zod/pull/6588">#6588</a>)</li>
<li><a
href="https://github.com/colinhacks/zod/commit/2bb08717"><code>2bb08717</code></a>
chore: re-pin the integration peers to the workspace zod after the 4.6.4
bump</li>
<li><a
href="https://github.com/colinhacks/zod/commit/f6e1701a"><code>f6e1701a</code></a>
chore(deps): bump next to 15.5.25 and vite to 7.3.6 (<a
href="https://redirect.github.com/colinhacks/zod/pull/6153">#6153</a>)</li>
</ul>
<h2>v4.6.3</h2>
<p>A patch on top of <a
href="https://github.com/colinhacks/zod/releases/tag/v4.6.2">4.6.2</a>.</p>
<ul>
<li><a
href="https://github.com/colinhacks/zod/commit/413cce9a"><code>413cce9a</code></a>
fix(v4): make z.properties() a check again (<a
href="https://redirect.github.com/colinhacks/zod/pull/6594">#6594</a>) —
removes the standalone <code>z.properties()</code> schema from 4.6.0;
<code>z.instanceof().properties()</code> and
<code>.check(...z.properties())</code> are unchanged</li>
<li><a
href="https://github.com/colinhacks/zod/commit/75d63ee1"><code>75d63ee1</code></a>
docs: show only the <code>.properties()</code> method form in the 4.6
post</li>
<li><a
href="https://github.com/colinhacks/zod/commit/46da9572"><code>46da9572</code></a>
docs: match the error-message examples to what the parsers emit</li>
</ul>
<h2>v4.6.2</h2>
<p>A patch on top of <a
href="https://github.com/colinhacks/zod/releases/tag/v4.6.1">4.6.1</a>.</p>
<ul>
<li><a
href="https://github.com/colinhacks/zod/commit/9446b5cc"><code>9446b5cc</code></a>
fix: preserve undefined prefault outputs and object keys (<a
href="https://redirect.github.com/colinhacks/zod/pull/6587">#6587</a>) —
closes <a
href="https://redirect.github.com/colinhacks/zod/issues/6585">#6585</a></li>
<li><a
href="https://github.com/colinhacks/zod/commit/0c483c58"><code>0c483c58</code></a>
docs: the <a href="https://zod.dev/blog/zod-4-6">Zod 4.6 announcement
post</a> (<a
href="https://redirect.github.com/colinhacks/zod/pull/6546">#6546</a>)</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/colinhacks/zod/commit/9c219833c3a3764fa1f1ea9c5ecec8254a63c0b1"><code>9c21983</code></a>
ci: stage at verbose so a failed OIDC token exchange says why</li>
<li><a
href="https://github.com/colinhacks/zod/commit/529f78dd9431b07f49647fee3aa0819b3f8b66b7"><code>529f78d</code></a>
4.6.6</li>
<li><a
href="https://github.com/colinhacks/zod/commit/a0ebf970f2d752e5f58527cd2942ff48be4f46bd"><code>a0ebf97</code></a>
fix(v4): check array, set and map size bounds before parsing elements
(<a
href="https://redirect.github.com/colinhacks/zod/issues/6652">#6652</a>)</li>
<li><a
href="https://github.com/colinhacks/zod/commit/a2ba4a67986a3e6b2b131be30084a2f460d9ea67"><code>a2ba4a6</code></a>
fix(types): declare record output and input as members, not base type
argumen...</li>
<li><a
href="https://github.com/colinhacks/zod/commit/c42596bd1c87e941d7241d10e012c5c196adcc10"><code>c42596b</code></a>
release: hold the run on the release-approval environment until the
staged ve...</li>
<li><a
href="https://github.com/colinhacks/zod/commit/bacad518c41610d654658172d325874bf0ee6290"><code>bacad51</code></a>
docs: square the Code for Japan logo to match the other bronze
tiles</li>
<li><a
href="https://github.com/colinhacks/zod/commit/8fb2faf0584828a6a027d8279db36f6f52784462"><code>8fb2faf</code></a>
docs: crop the Code for Japan logo and list it first in bronze</li>
<li><a
href="https://github.com/colinhacks/zod/commit/b2ef983e0711ff6517d6020b021071e2e5653efa"><code>b2ef983</code></a>
docs: add Code for Japan to bronze sponsors</li>
<li><a
href="https://github.com/colinhacks/zod/commit/74304ce5b5796b5edb9a88ceb8e170b6ea3e82df"><code>74304ce</code></a>
release: stage npm versions instead of publishing them (<a
href="https://redirect.github.com/colinhacks/zod/issues/6622">#6622</a>)</li>
<li><a
href="https://github.com/colinhacks/zod/commit/34a626f307871ecbb0123947559328dc52ed5e3e"><code>34a626f</code></a>
docs: tighten the z.locales tree-shaking note</li>
<li>Additional commits viewable in <a
href="https://github.com/colinhacks/zod/compare/v4.4.3...v4.6.6">compare
view</a></li>
</ul>
</details>
<br />

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Nicky Leach <nicky@paperclip.ing>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 17:41:43 -08:00
Nicky LeachandClaude Opus 5.5 e6c74982fe fix(workspaces): report why a workspace provider operation failed (#19362)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A project can store its task workspaces with a workspace storage
provider plugin. Core calls the plugin's `workspaceOperation` method to
prepare, inspect, finalize and remove a workspace.
> - `callWorkspaceProvider` catches every rejection of that call and
throws the same conflict: "Workspace storage operation is incomplete;
retry the same workspace".
> - A provider error, an RPC timeout, a crashed worker and a worker that
never got the call all look the same in the run, the log and the API.
The message tells the user to retry, also when the failure is a fixed
configuration problem that a retry cannot fix.
> - In one diagnosis, three different configuration failures in a row
showed this same message. Finding each cause took trace span counts and
database reads.
> - This pull request lets a provider send a reason from a closed set,
and makes Core tell transport failures apart from provider failures.
> - The benefit is that a failed run says what failed and whether a
retry can help. Provider text and values still never cross into Core.

## Linked Issues or Issue Description

**What happened?**
A workspace provider's `workspaceOperation` failed. The run failed with
"Workspace storage operation is incomplete; retry the same workspace",
`errorCode: setup_failed`, and no reason in the run record or the
`heartbeat execution setup failed` log. Retrying did not help, because
the cause was a provider configuration problem.

**Expected behavior**
Core tells these cases apart: the worker did not receive the call, the
call timed out, the worker stopped during the call, or the provider
reported a failure. For a provider failure, Core shows a bounded reason.
Core tells the user to retry only when a retry can help.

**Steps to reproduce**
1. Install a workspace storage provider plugin whose
`onWorkspaceOperation` throws during `prepare` (for example, because of
missing provider settings).
2. Run an agent task in a project that uses that provider.
3. See the failed run: the error is the generic retry message, and the
run record and logs do not show the cause.

**Paperclip version / commit**
`master` at 5ac05871b5.

**Deployment mode**
Any deployment with a workspace storage provider plugin.

## What Changed

- **SDK reason contract.** New
`packages/plugins/sdk/src/workspace-operation-error.ts`:
`WORKSPACE_OPERATION_FAILURE_REASONS` (11 reasons),
`PluginWorkspaceOperationError`, `workspaceOperationErrorData` and
`readWorkspaceOperationFailureReason`. It uses the same envelope design
as `environment-task-error.ts`. The worker host attaches the envelope
only to `workspaceOperation` errors. It copies only the reason, never
the message.
- **Duck-typed reason.** The worker host also forwards an own
`workspaceFailureReason` property from any error, when the value is in
the closed set. A provider that builds against an older SDK can set the
reason before it can import the class. An inherited or unknown value is
ignored. The value can only select one of Core's fixed messages.
- **Core classification.** New
`server/src/services/workspace-provider-failure.ts`.
`callWorkspaceProvider` maps each rejection to a
`WorkspaceProviderFailureError` (409) with `kind`, `reason` and
`retryable`:
- `worker_unavailable`: no request was sent. The
`PluginWorkerUnavailableError` stays as the cause, so run setup retries
it as `plugin_worker_unavailable` (the existing bounded bootstrap
retry).
- `timeout` and `worker_exited`: the outcome is unknown, so the message
still says to retry the same workspace.
  - `transport`: any other rejection. The message is the old message.
- `provider` with a reason: a fixed message per reason. Only transient
reasons say to retry.
- `provider` without a known reason: "Workspace storage provider
reported a failure; retry the same workspace".
- **Run settlement.** `failRunSetup` records
`resultJson.workspaceProviderFailure` (`kind`, `reason`, `retryable`).
The `heartbeat execution setup failed` log carries the same fields and
the run's `errorCode`. The run keeps `errorCode: setup_failed` (except
for the worker-not-running case above), so the existing setup-failure
retry, recovery and chat paths do not change.
- **Span allowlist.** The host accepts a `plugin.workspace.failure` span
and a `plugin.failure_reason` attribute. The value must be a reason from
the closed set or `unknown`.
- **Docs.** `doc/plugins/PLUGIN_RUNNER_EXECUTION.md` describes the
failure kinds and the reasons. `doc/observability.md` lists the new span
and attribute.

## Verification

- `cd packages/plugins/sdk && npx vitest run`: 21 files, 261 tests pass.
The new `tests/workspace-operation-error.test.ts` sends errors through
the real worker RPC host.
- `cd server && npx vitest run
src/__tests__/workspace-provider-failure.test.ts
src/__tests__/plugin-host-services-span.test.ts
src/services/heartbeat/run-completion.test.ts`: 3 files, 118 tests pass.
These tests cover:
- every transport kind, and that a worker refusal is still found as the
cause;
  - a real worker RPC rejection with a reason, and without a reason;
  - a fixed message and retry decision for every reason;
  - span clamping of `plugin.failure_reason`;
- run setup for a provider reason (no retry, reason in `resultJson` and
in the log) and for a stopped worker (bounded retry).
- `cd server && npx vitest run` for the 4 plugin workspace integration
suites (`plugin-workspaces`, `plugin-fresh-workspaces`,
`project-workspace-provider`, `workspace-delivery-cleanup`): 91 tests
pass. Their lost-reply fixtures still get the old "incomplete" message.
- `npx tsc --noEmit` in `packages/plugins/sdk`: no errors. `npx tsc
--noEmit -p server/tsconfig.json` shows no errors in the changed files.
The remaining errors come from workspace packages that I did not build
locally.
- Not run locally: the full `pnpm -r typecheck`, `pnpm test:run` and
`pnpm build`. CI runs them.

## Risks

- **Behavior change for a stopped worker.** A `workspaceOperation` call
that the worker did not receive now gets `errorCode:
plugin_worker_unavailable` and the bounded bootstrap retry, as other
plugin calls already do. Before, it was a `setup_failed` run with no
retry. No request reached the provider, so the retry is safe.
- **Message changes.** Provider-reported failures and timeouts get new
fixed messages. Code that matches the exact old message only sees it for
`transport` and `remote_incomplete` failures. I did not find a match on
this message outside tests.
- **No new error codes.** Provider failures keep `setup_failed`. Retry,
recovery and chat-retry routing does not change.
- **Duck typing.** Any error with an own `workspaceFailureReason` from
the closed set is forwarded. A provider can only select one of Core's
fixed messages. It cannot send text or values.
- **Observability.** The new span attribute is a closed set and stays
inside the host allowlist. No Telemetry change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, file edits, test runs). Extended thinking was on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Related:
[paperclipai/paperclip#19326](https://github.com/paperclipai/paperclip/pull/19326)
added the same envelope design for Runner revision mismatches. I found
no duplicate PR for this change.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 16:28:49 -08:00
Devin FoleyandPaperclip 8080620fc9 fix(workspaces): size archive deadlines and preserve resume budgets (#19364)
Give large workspace archives a bounded per-transfer deadline without changing
provider settings. Preserve configured post-upload command limits and the saved
provider budget when resuming an exact retained sandbox.

Verification: 341 focused Core/SDK/adapter tests and 588 Daytona tests passed.
Workspace and provider typecheck/build passed. Hosted CI: 54 successful checks,
two intended Storybook skips. Greptile 5/5 with no unresolved review threads.
The bounded full local run passed all six chat shards and stopped at twenty
minutes during general server suites; that full local run remains incomplete.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-05 16:09:32 -08:00
Nicky LeachandClaude Opus 5.5 a6ab6a301c fix(server): retry a run when its remote Runner is out of date (#19326)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - An agent can run on a remote Runner that an environment plugin
provides
> - A release can update the app before the remote Runner, so for a
short time the app sees an older Runner
> - In that window the plugin can refuse the task, but Core cannot tell
this refusal from other errors
> - Today the run fails as `setup_failed` with no retry, and the task
goes to board recovery. Before #19246, it was also reported as a project
authentication conflict
> - Agent verification also counts the refusal as a failed harness test,
and it parks the agent after three tries
> - This pull request gives the plugin a stable error code for a Runner
mismatch, and Core waits and retries when it gets that code
> - The benefit is that a release no longer blocks tasks or parks
agents, and a real authentication conflict keeps its own clear notice

## Linked Issues or Issue Description

Refs #19246 (it stopped reporting every precheck error as a project
authentication conflict)
Refs #19249 (the same bounded bootstrap retry for a stopped plugin
worker)
Refs #19302 (shares `environment-task-error.ts`; its diagnostic envelope
is unchanged)
Refs #19172 (related open PR: it reports an undescribable plugin Runner
as a retryable check in the same precheck)
Refs #18667 (a real `ai_connection_incompatible` conflict, which now
gets its own notice)

**What happened?**
A run started while its remote Runner was older than the app. The
environment plugin refused the task because the installed Runner did not
match the revision it expects. Core recorded a generic setup failure,
scheduled no retry, and blocked the task. Before #19246, Core reported
this as a project authentication conflict. An agent in `verifying`
counted the same refusal as a failed harness test. After three tries it
stopped verification until a person chose Retry.

**Expected behavior**
Core treats a Runner mismatch as temporary. A run retries without
blocking its task. A `verifying` agent checks again until the Runner
matches. Only a real `ai_connection_incompatible` result blocks the
task, with a notice that names the conflict.

**Steps to reproduce**
1. Use an agent on a plugin environment whose installed remote Runner
revision differs from the revision the plugin expects.
2. Wake the agent on a task, or put a new agent through verification.
3. The run fails with `setup_failed` and the task is blocked.
Verification parks after three tries.

**Paperclip version or commit**
master at 377e98f1a2.

**Deployment mode**
Authenticated deployment with a plugin environment that provides a
remote Runner.

## What Changed

- Plugin SDK: add `PluginEnvironmentTaskError` and
`ENVIRONMENT_TASK_ERROR_CODES` with one code,
`runner_revision_mismatch`. The worker RPC host sends it as structured
`environmentTask` error data with the schema
`paperclip/environment-task-error-code/v1`
(`environmentTaskErrorCodeData`, read with
`readEnvironmentTaskErrorCode`). This uses the same envelope method as
the other environment RPC errors. Core reads only the code, never the
message. The source refusal diagnostic from #19302 keeps its own schema
and API unchanged, and it takes precedence on a source `submit`.
- `executeEnvironmentTask` still replaces each worker error with a
generic error. After the #19302 diagnostic check, it now keeps only that
closed code.
- Run setup: a Runner mismatch before provider dispatch gets the error
code `runner_revision_mismatch`. Setup schedules a bounded retry with
its own reason, `runner_revision_wait`, and budget (10 attempts, 60 s
apart). The reason is in the same resource-wait lane as workspace and AI
connection waits, so a wait does not spend the failure retry budget. The
guards are the same as for `plugin_worker_unavailable`. Recovery and
legacy reconciliation treat the code like the other bootstrap-only
codes. `scheduleBoundedRetryForRun` uses the Runner wait reason and
budget for a mismatch run when a caller gives no reason, and the
recovery sweeper reports exhaustion from that budget
(`failedRunRetryBudgetSpent`). An answered interaction that hits a
mismatch also waits for the Runner, and its successor keeps the resolved
interaction continuation identity.
- Move the managed project auth classification from `heartbeat.ts` to
`classifyManagedProjectAuthFailure` in `run-preparation.ts`. Only a
found `ai_connection_incompatible` conflict becomes a configuration
failure. All other errors keep their own code.
- Recovery notices: `ai_connection_incompatible` now has its own notice
instead of the "secret/env bindings" text. An exhausted Runner wait gets
the notice "Remote Runner is out of date".
- Agent verification: the Runner probe and the managed project auth
precheck report a mismatch as a retryable check.
`assertHarnessTestPassed` maps it to the new lifecycle code
`runner_revision_mismatch`. That code never counts toward the
three-failure park. The agent checks again every 60 s and becomes ready
when the Runner matches.
- Document the error code in `doc/plugins/PLUGIN_ENVIRONMENT_TASKS.md`.

## Verification

- `cd packages/plugins/sdk && npx vitest run
tests/environment-task-error-code.test.ts
tests/environment-task-error.test.ts tests/worker-rpc-host.test.ts`: the
code crosses the real `environmentTask` RPC. A plain error with the same
message gets no code. Other methods get no envelope. The #19302
diagnostic tests still pass.
- `cd server && npx vitest run
src/services/heartbeat/run-completion.test.ts`: a fresh or wrapped
mismatch gets `runner_revision_mismatch` and the runner retry budget,
with no configuration failure. A started, untrusted (message only) or
native run keeps `setup_failed` with no retry. A real
`ai_connection_incompatible` stays `configuration_incomplete` with no
retry. Other errors, such as an unreachable execution host, are not
reported as an authentication conflict.
- `npx vitest run src/services/execution-recovery-attempt.test.ts`:
Runner waits do not change `failureRetries`, and earlier failures do not
shorten the wait.
- `npx vitest run src/services/heartbeat/retries.test.ts
src/modules/run-dispatch` (embedded Postgres): a recovery caller without
options gets the Runner wait budget and its exhaustion. An answered
interaction waits for the Runner, keeps its continuation identity and
does not spend failure retries.
- `npx vitest run src/__tests__/environment-task-runtime.test.ts`:
sanitization keeps only the code. No worker text or cause survives.
- `npx vitest run
src/modules/agent-lifecycle/__tests__/verification-retry.test.ts`
(embedded Postgres): a `verifying` agent keeps checking after five
mismatches. It never gets `retryRequired`. It becomes ready when the
next check passes.
- `npx vitest run
src/services/native-runtime/plugin-runner-probe-runner-mismatch.test.ts
src/__tests__/agent-harness-verification-remote.test.ts
src/services/recovery/stranded-notice.test.ts
src/services/recovery/service.pause-durability.test.ts
src/services/legacy-execution-recovery.test.ts`.
- `npx tsc --noEmit` passes for `packages/plugins/sdk` and `server`.
- One test in `src/__tests__/heartbeat-process-recovery.test.ts`
("redacts opaque environment-bound credentials from Sentry diagnostics")
fails on my machine. It fails the same way on the base commit without
this change.
- I did not run the full `pnpm test:run`, `pnpm -r typecheck` or `pnpm
build`. CI covers them.

## Risks

- Low to medium. The change adds one new setup failure code and one
lifecycle code. Other failures keep their current behavior.
- A Runner mismatch must come from a plugin that throws
`PluginEnvironmentTaskError`. Until an environment plugin does that,
behavior is the same as today.
- The run retry budget is 10 minutes. If a Runner update takes longer,
the task goes to board recovery with the Runner notice.
- A remote Runner run without a managed AI connection reaches the Runner
check only after native dispatch. That configuration is not supported,
so this change does not handle it. Such a run keeps the native session
retry (3 attempts, 30 s apart).
- #19172 edits the same precheck call in `agent-environment-test.ts`.
Whichever PR merges second needs a small rebase. Keep the mismatch check
before its generic retryable check.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use (code search, shell and test runs). Reasoning
mode: standard agentic coding.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 14:34:23 -08:00
Nicky LeachandPaperclip 9a79384958 Relay bounded Runner source refusal diagnostics (#19302)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - An environment plugin can submit a Runner continuation from a
retained source.
> - The host can refuse that continuation when its source is unverified
or has an unresolved operation.
> - The worker and Core currently discard that bounded reason.
> - Raw provider errors can contain credentials, text or foreign
identities.
> - This pull request relays only a closed diagnostic and keeps the
existing admission and redaction rules.

## Linked Issues or Issue Description

Refs #19134 and #19157. This independent fix is based on master. A
search found no duplicate relay.

**What happened?**

A source submit can receive HTTP 409 with `SESSION_STATE_UNAVAILABLE`.
The worker drops its structured data. Core then replaces the error with
a generic message. The caller cannot distinguish the two known
diagnostic reasons safely.

**Expected behavior**

Carry only the fixed status, code and reason through SDK worker RPC and
Core. Keep all provider text and identities out of the relayed error. A
diagnostic must not authorize a retry or change state.

**Steps to reproduce**

Submit an authorized retained source through a task provider. Have the
provider throw the recognized state-unavailable error. Read the error
after worker RPC and Core dispatch.

## What Changed

- Export `PluginEnvironmentTaskErrorDiagnostic`,
`environmentTaskErrorData` and `readEnvironmentTaskErrorDiagnostic`.
- Accept only `httpStatus: 409`, `errorCode:
"SESSION_STATE_UNAVAILABLE"` and reason `"unresolved_operation"` or
`"source_state_unverified"`.
- Attach the versioned envelope only to an environment task `submit`
with `resumeFrom`. Replace its provider message with a fixed message.
- Revalidate and reconstruct the closed error in Core. Keep other
failures under the existing generic redaction.
- Document older SDK behavior and the diagnostic-only boundary. No
grant, retry, discard, lifecycle or admission rule changes.

## Verification

- Rebased head `7d02bb3e4c` includes master’s UI fixture fix from
#19342. The sole conflict was in the worker integration test imports. It
now imports both the physical-task result schema from #19300 and the
diagnostic reader and RPC error type from this PR. The 39 SDK/worker
cases and 57 Core/runtime cases passed, including PostgreSQL worker
integration. The SDK build and Runner/server TypeScript checks passed.
[Current-head
CI](https://github.com/paperclipai/paperclip/actions/runs/42111889388)
passed without retries. All 56 check-rollup entries were green,
including Snyk and Socket; [standard Greptile review
2](https://github.com/paperclipai/paperclip/pull/19302#issuecomment-6665823815)
scored this exact head **5/5**, with zero unresolved threads. The PR was
marked ready on November 5 at 21:37 UTC.

- Both new worker relay cases fail against the baseline worker and pass
with the fix.
- 46 SDK/worker cases passed, including both reasons, malformed/foreign
envelopes, extra identities, unsafe getters and non-source operations.
- 51 Core/runtime cases passed. The integration uses real SDK
serialization and dispatch with PostgreSQL through the existing
authorized retained-source submit. It verifies both reasons, malformed
RPC rejection, extra-field removal, preserved source outcome and
existing dispatch bookkeeping. A valid envelope on an ordinary submit or
status stays generic.
- Runner and SDK builds and server TypeScript check passed. An initial
Runner build lacked its eval-kernel build prerequisite; it passed after
that prerequisite was built.
- The first expanded DB fixture used a shared environment name and
failed its unique constraint. It now uses its unique fixture ID; both
complete focused files pass.
- Diff and added-line secret scans passed. [GitHub workflow
42083410387](https://github.com/paperclipai/paperclip/actions/runs/42083410387)
passed on its second attempt. All CI checks completed, and Snyk passed.
Standard Greptile scored exact head `38559ec4df8c` 5/5 with no
unresolved threads. That result applies to the previous head.
- The first GitHub attempt failed in the unchanged concurrent-budget
case in `decision-models.test.ts` (one provider call observed, two
expected). That exact case passed locally without code or assertion
changes. The workspace shard also passed 4,960 tests before the
unchanged CLI company import/export fixture exceeded its 30-second
child-server startup wait. No port-collision cause is claimed; that
fixture deliberately reserves the requested port. Both initial logs are
retained. One failed-job-only retry passed without source changes.

## Risks

Older SDKs keep the generic failure. The envelope is diagnostic evidence
only. Provider operation and call identities are never accepted.
Specific user-visible operation reporting must use Core's retained event
or receipt under its existing authority.

Master does not contain the new claimed predecessor orchestration. This
PR tests the existing retained-source submission boundary. It does not
claim the #19157 source recovery or old-journal end-to-end
qualification. Runtime pins and deployments remain separate.

## Model Used

OpenAI GPT-6.1 Sol (`gpt-6.1-sol`), high reasoning. Used repository
tools, TypeScript, SDK RPC and real PostgreSQL tests.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used with version and capability
details
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have described the issue in-PR using the bug template
- [x] I have not referenced internal or instance-local Paperclip issues
or links
- [x] My branch name describes the change
- [x] I have run the focused tests locally and they pass
- [x] I have added tests for the new behavior and rejection paths
- [x] I have updated relevant documentation
- [x] I have considered and documented risks
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no unresolved findings
- [x] I will address reviewer comments before requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-05 13:38:55 -08:00
Devin FoleyandPaperclip c3550cfb38 fix(daytona): send archive chunks in bounded windows (#19339)
## Thinking Path

> - Paperclip manages agents and saves their work when execution stops.
> - The Daytona plugin saves an archive through a terminal when disk
capacity prevents the normal archive or download copy.
> - That fallback sent one 40 KiB chunk and waited for its host write
acknowledgement.
> - Network delay then applied to every small chunk. A large archive
could exceed the recovery deadline even while bytes were moving safely.
> - This pull request sends a bounded window of chunks and still
acknowledges every completed host write.
> - The benefit is faster recovery over delayed connections, with
bounded memory and the existing size, space and completion checks.

## Linked Issues or Issue Description

Refs #19034, which added the capacity fallback.

**What happened?**

The archive fallback waits for one network round trip after each 40 KiB
chunk. A saved workspace archive can be much larger than that chunk.
Repeating a timed-out transfer does not remove this throughput limit.

**Expected behavior**

Allow a small amount of data to travel before waiting for
acknowledgements. Keep memory bounded, write bytes in order, and stop
after any write or protocol failure.

**Steps to reproduce**

1. Run the real Bash archive sender against a host that delays each
acknowledgement by 400 ms.
2. Transfer 1 MiB plus a partial final chunk with a four-second
deadline.
3. The original implementation times out. The bounded-window
implementation completes and preserves the exact source bytes.

**Paperclip version or commit**

The failing regression uses the original implementation from
`6a66c09f7f7`.

## What Changed

- Send at most sixteen 40 KiB chunks before collecting their
acknowledgements. Drain the final partial window before the completion
receipt.
- Serialize host writes and reject more than sixteen pending frames.
Stop queued writes on the first failure.
- Keep per-frame size checks, total archive limits, the 1 GiB host space
reserve, the transfer deadline and bounded teardown.
- Add a real Bash regression with delayed acknowledgements, plus
overflow and failed-write coverage. Update the plugin documentation.
- Update the agent-creator test fixture for the required directory
membership role introduced by #15183, which otherwise breaks merged CI
typechecking.

## Verification

- Failing first: the final real-Bash regression fails against the
original source with `Daytona archive stream timed out`. It passes with
the change. The empty-stream case passes too.
- Standalone Daytona package `pnpm test`: 578 passed, 8 skipped. The
skips are the existing opt-in live tests.
- Standalone package `pnpm typecheck` and `pnpm build`: passed.
- `pnpm test src/file-sync-capacity.live.test.ts` with a staging Daytona
key: 2 passed on isolated test sandboxes. This covers actual terminal
transport, stop/start, archive and file-copy capacity failures, exact
saved hashes, safe links, cleanup and a second export. The tests removed
their own sandboxes.
- The production dependency assumption is bounded: at most 640 KiB of
decoded payloads can wait for host writes. The parser also has a bounded
encoded buffer. No background work or idle timer is added.
- Initial hosted CI failed with TS2741 in
`ui/src/lib/agent-creator.test.ts` after #15183 made `membershipRole`
required. Rebased onto that change and added the nullable field to the
fixture. UI `pnpm typecheck`, all 4 existing creator-label tests, and
`pnpm check:token-gates` now pass locally. Hosted CI subsequently passed
on the updated head.
- `git diff --check` and a local secret and PII scan: passed.
- Full repository checks were not repeated locally for this standalone
plugin change. Hosted verification at
`bf6b3cb6a622a9523c72b86fdfe616555c8f3ef8`: all 53 check runs passed,
with the 2 opt-in Storybook checks skipped; both required gates (`ci /
verify`, `ci / e2e`) passed. Greptile is 5/5 at this exact head, with no
review threads or merge conflicts. The separate non-required
`security/snyk (cryppadotta)` status remains pending; it is not claimed
as passed.

## Risks

- A burst can queue up to sixteen chunks instead of one. The receiver
enforces this bound and acknowledges a chunk only after its ordered host
write.
- The timeout budget is unchanged. A connection that cannot finish
within that budget can still fail. This change does not retry a provider
turn or change file-recovery authority.
- Source confinement, archive validation, successful producer completion
and publication rules are unchanged. No schema migration.

## Model Used

- OpenAI `gpt-6.1-sol` with xhigh reasoning, in Codex through T3 Code.
Tool use included source edits, real Bash and filesystem tests, and
isolated Daytona tests. The runtime does not expose its context window.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-05 13:12:11 -08:00
Nicky LeachandPaperclip 2506bcdaf4 Admit exact stopped Runner sources under the current run lease (#19134)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native execution keeps provider state and a durable controller
journal.
> - A completed process can stop before the next run attaches.
> - The current run must own recovery of that exact source.
> - An old controller must lose permission to change the consumed source
journal.
> - This pull request adds explicit source admission and protected task
dispatch.
> - The benefit is a narrow recovery boundary that preserves the
original successful result.

## Linked Issues or Issue Description

Native prerequisites #19132 and #19153 are merged into master. This PR
targets master and updates the TypeScript Runner driver contract
alongside Core and SDK admission. This change adds the SDK and Core
admission boundary. Executor selection and fresh source preparation are
a separate change. Related #17916 supplies the workspace export holds
preserved here.

## What Changed

- Plugins advertise `codexSourceRecoveryVersion: 1` for Codex App Server
and `acpxSourceRecoveryVersion: 1` for ACPX. Hosts can qualify either,
both, or neither. Core selects the matching field, because a
stopped-source check for one provider cannot establish the other
provider’s recovery guarantees. ACPX orchestration is tracked separately
in #19196. Use the existing exact source identity and `resumeFrom` under
the authenticated current run. No public recovery mode is added. Require
the claimed source, current account and prepared launch at protected
dispatch.
- Keep the first source binding pinned on the consumer lease, including
after uncertain dispatch. A newly acquired coordinator can refresh only
the same source.
- Claim one stopped successful source under the current run's acquired
native lease. Store the token-free consumer reference in existing lease
metadata. Preserve the source status and result.
- Check the exact six source identity fields, encrypted journal header,
company, task, agent, environment, driver, workspace, account selection,
and current coordinator generation. Keep workspace export and legacy
settlement holds.
- Serialize claims and checkpoint writers in the same coordinator lock
order. Reject stale source writes after consumption. Check the current
consumer before recovered source writes.
- Check source admission before and after protected worker dispatch.
Keep the request and lease owned by the current run while the native
identity remains the source.
- Allow the same owner to prepare B after a fresh source submission,
using the exact consumed-source proof instead of a generic warm-reuse
flag. Recheck that proof after dispatch.
- Preserve ordinary cached adoption and repeated preparation after
launch. The current authenticated owner can differ from the adopted PRP
identity without requesting a new source launch. Keep ordinary suspended
recovery and live task attachment checks. Extract the shared source
resolver to avoid a module cycle, and make ordinary selection honor the
same durable source consumer.

## Verification

- Current head `cac9cf188a` includes merged #19300 and the symmetric
capabilities introduced in `b7e25a64b4`. The task-owner and warm-rebind
fixtures now identify their physical task. All 29 SDK schema cases, 95
Core ownership and source-admission cases, 25 native Codex and ACPX
recovery cases, and the binary metadata check passed on this head. The
SDK and Runner builds and server typecheck passed. Both native providers
reject the other provider’s capability. GitHub CI is green and [standard
Greptile scored this commit
5/5](https://github.com/paperclipai/paperclip/pull/19134#issuecomment-6645574114)
with no open threads.

- Previous head `bda49271a4ae362a15baf3fe6d6d297e25fd2000` is directly
based on master after #19132 and #19153 landed. [GitHub workflow
42072192023](https://github.com/paperclipai/paperclip/actions/runs/42072192023)
passed on its first attempt. All check runs completed successfully or as
intentional workflow skips; Snyk passed. Standard Greptile scored this
exact head 5/5. That review had no unresolved findings.
- Runner and SDK TypeScript builds and server typecheck passed. The
focused SDK, task-owner, task-runtime, PostgreSQL source and authority
suites passed all 161 cases. The production transport-factory getter
regression and 14 native-lane selection checks also passed.
- The Linux native lane executed the new ordinary warm A→B process case.
It kept one Runner/provider process and completed two turns on the same
thread. The eager-getter mutation fails with
`native_runner_source_attachment_pending`; preserving the live getter
passes. The production factory test changes the current owner after
transport creation and checks that same options object.
- PostgreSQL cases cover both concurrent source-writer/claim orders,
exact source submission under the current caller, source-switch denial
after uncertain dispatch, same-source coordinator refresh, stale
same-boot generation, corrupted journal headers, competing consumers,
account changes and export holds.
- The protected dispatch sequence submits A, prepares B and retries B
preparation. It rejects undispatched sources, changed accounts and
foreign target bindings. Source status and accepted result remain
unchanged.
- Ordinary adopted A preparation under current B, repeated ordinary
preparation after launch and ordinary suspended continuation retain
their existing path. Absent or false claims preserve ordinary identity
denial; malformed claims fail closed.
- Earlier development attempts exposed ordinary preparation regressions
and the live-getter omission; the final regressions cover those fixes.
The first embedded PostgreSQL attempt needed missing library aliases
repaired in ignored dependencies. Those attempts are not passing
evidence.
- Historical, superseded CI heads had unchanged UI fake-timer and
Playwright health-response-lifetime failures. Their isolated cases and
scoped retries passed. The response-lifetime cause remains unproven. No
UI/test timeout was changed. The final rebased-head workflow above
passed without a retry.
- Whitespace and added-line secret/PII scans passed. Full workspace
checks ran on GitHub. No live vendor probe or deployment was performed.

## Risks

- Any driver that used the earlier Codex TypeScript option must use
`codexSourceRecoveryVersion`. The new plugin description rejects the
generic spelling. The Launcher’s existing Codex-specific HTTP header can
map to this property when the plugin adopts the new SDK. The change
requires no journal migration or ROM change.

- This is an inactive admission foundation. Executor orchestration is
separate in #19157. Native restore integrity and provider
stopped-source/account validation must be present before executor
activation. The native prerequisite uses ordinary recovery followed by
attachment. The composed executor proofs are in #19157.
- The source claim is durable. A later controller must recover the same
consumer or keep the hold. It cannot select another source after an
ambiguous launch.
- Admission adds bounded database checks around protected startup calls.
Busy issue locks fail closed; other lock waits have a one-second bound.
- No migration, snapshot format, proactive receipt, recurring worker, or
deployment is included. Full workspace checks and platform coverage run
in CI. These tests do not prove vendor history durability or a hosted
rollout.

## Model Used

OpenAI GPT-6.1 Sol (`gpt-6.1-sol`), high reasoning, through Codex with
repository tools and code execution. The runtime does not expose a
context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green on the current head
- [x] Standard Greptile is 5/5 with no unresolved findings on the
current head
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-05 12:55:06 -08:00
Devin FoleyandClaude Opus 5.5 ee72132a44 fix(daytona): bound one-shot exec requests by their command deadline (#19298)
Bound the Daytona one-shot executeCommand request at its command
deadline plus 5 s and return an attributed timed-out result, instead of
waiting for the host RPC deadline. Bound the stdin cleanup the same way.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 11:39:00 -08:00
Devin FoleyandClaude Opus 5.5 7f5d0793bd fix(daytona): record a bounded errno when a sandbox command fails (#19297)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Daytona sandbox provider copies the agent's workspace back to
the host after each turn (syncOut)
> - When the sandbox `tar` fails in the `archive_create` step, the
restore diagnostic records only `exitCode: 2` and `errorCode: "unknown"`
> - The command output is not sent across the plugin RPC boundary, so an
operator cannot tell a full disk from a permission error or a path that
is too long
> - This pull request maps the system error text at the end of a failed
sandbox command's output line to the existing allowlisted `errorCode`
> - The benefit is that the next failure records the real errno, and the
output text stays private

## Linked Issues or Issue Description

Refs #18172
Refs #18212

**What happened?**
A hosted Daytona run failed its workspace restore. The `archive_create`
step exited 2 on all three attempts. The run log recorded
`{"step":"workspace_transfer","transferStep":"archive_create","transferFailureKind":"command_failed","exitCode":2,"errorCode":"unknown"}`.
Two causes are possible. The sandbox disk was full, or a path was longer
than PATH_MAX, for example from deeply nested GitHub helper config
directories before #18212. The diagnostic cannot tell these causes
apart.

**Expected behavior**
A failed sandbox command records a bounded errno such as `ENOSPC`,
`EACCES` or `ENAMETOOLONG` when its output shows one. The output text
itself must not be recorded.

**Steps to reproduce**
1. Run `performSyncOut` against a sandbox whose `tar -c` exits 2 with
`tar: ./x: Cannot open: Permission denied`.
2. Read the environment sync diagnostic from the thrown error.
3. See that `errorCode` is `unknown`.

**Paperclip version or commit**
`master` at 3fca8a1.

## What Changed

- `packages/plugins/sandbox-providers/daytona/src/file-sync.ts`:
`assertSandboxCommandOkWithOutput` maps the strerror suffix of a failed
command's output to an allowlisted errno. It reads only the last 16 KiB
in memory. A line must end with `: <message>`. Fatal causes are checked
first (ENOSPC, EIO, ENAMETOOLONG, EACCES, EPERM, ENOTDIR, EISDIR,
ENOENT), so a warning about a vanished file does not hide the error that
stopped `tar`. The error message, capacity fallback (#18172), restore
and retain-on-failure behavior do not change.
- `packages/plugins/sdk/src/environment-sync-error.ts` and
`packages/adapter-utils/src/workspace-restore-diagnostics.ts`: add
`ENAMETOOLONG` to both allowlists. Both sides still validate
independently.
- Tests: 7 new Daytona cases, plus one allowlist case each in the SDK
and adapter-utils.

## Verification

- `vitest run --config
packages/plugins/sandbox-providers/daytona/vitest.config.ts
packages/plugins/sandbox-providers/daytona/src/file-sync-diagnostics.test.ts`:
18 passed. Without the `file-sync.ts` change, the 7 new cases fail with
`errorCode: "unknown"`.
- `packages/plugins/sdk`: `vitest run
tests/environment-sync-error.test.ts`: 20 passed.
- `packages/adapter-utils`: `vitest run
src/workspace-restore-diagnostics.test.ts`: 50 passed.
- `tsc --noEmit` passes for the SDK and adapter-utils. `git diff
--check` is clean.
- I did not run the Daytona package typecheck or its full suite locally
because the sandbox disk was full. CI covers them.

## Risks

- Low risk. The change adds a diagnostic field only. Control flow, error
messages and the capacity fallback regex do not change.
- Localized `tar` output does not match, so it stays `unknown`. That is
the current behavior.
- `ENAMETOOLONG` is a new allowlisted code on both sides. It is a fixed
string and carries no path or output text.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in an agent harness.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 11:34:41 -08:00
DottaandClaude Opus 5.5 037c58bb22 fix(daytona): keep the provider auto-stop strictly after the warm runner idle window (#19199)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A warm remote agent keeps its sandbox running after a turn. The host
keeps a warm runner for an idle window (`runnerIdleTimeoutMs`, or the
agent's `idleTimeoutMs`)
> - The Daytona provider also stops an idle sandbox after
`autoStopInterval` minutes. The default is 15 minutes
> - On staging, the warm idle window was 900,000 ms (15 minutes). The
two windows were equal
> - A follow-up turn at the end of the warm window resumed a sandbox
that Daytona was stopping. Daytona refused the resume in about 4
seconds, and the run failed at setup
> - #19195 added a bounded retry for that refusal. It did not remove the
cause
> - This pull request makes the two windows non-overlapping by
construction. Daytona derives its auto-stop for a warm reusable sandbox
from the same warm idle value that the host uses
> - The benefit is that a follow-up inside the warm window always finds
a running sandbox, and idle sandboxes still stop

## Linked Issues or Issue Description

Refs #19195 (bounded retry for a refused resume; this change removes the
common cause).

**What happened?**

On a staging soak, three follow-up runs failed at setup with a refused
sandbox resume. Each resume began 15 minutes 8 to 25 seconds after the
previous turn released the sandbox. The warm idle timeout was 900,000
ms. Daytona's default `autoStopInterval` is 15 minutes. Nothing in the
code kept these two values apart.

**Expected behavior**

A reusable sandbox that keeps a warm runner must not auto-stop inside
the warm idle window or at its edge. The provider backstop must fire
after the host's own idle handling, with a margin. Idle sandboxes must
still stop.

**Steps to reproduce**

1. Configure a Daytona sandbox environment with `reuseLease` and a warm
runner with `runnerIdleTimeoutMs: 900000` (or an agent with
`lifecycleMode: warm`, `idleTimeoutMs: 900000`).
2. Run a turn. Do not set `autoStopInterval`.
3. Before this change: the sandbox is created with `autoStopInterval:
15`, equal to the warm window. A follow-up near minute 15 races the
provider stop.

**Paperclip version or commit**

`c656b95b6b` (current `master`).

## What Changed

- Daytona plugin (`plugin.ts`): add `effectiveAutoStopIntervalMinutes`.
For a reusable lease that keeps a warm runner, the effective auto-stop
is at least `ceil(warm idle / 1 min) + 5` minutes. The plugin applies
the floor only when the host forwards `warmRunnerIdleTimeoutMs`, or when
the environment itself selects `runnerLifecycleMode: warm`. An explicit
`0` (disabled), a longer configured value, `inherit` with a per-turn
agent, per-turn, and non-reusable sandboxes keep the configured value.
- Daytona plugin: use the effective value at sandbox creation. On
resume, raise a shorter auto-stop on an existing sandbox. This update is
best effort and bounded to 5 seconds; the #19195 retry still covers a
refusal. The stale-handle refresh threshold keeps the configured
interval, so a cached handle is not trusted for longer before the
provider confirms the new interval. The warm window is not written to
the saved normalized config.
- Host (`runner-environment-lifecycle.ts`): add
`resolveWarmRunnerIdleTimeoutMsForRun`. It returns the run's effective
warm idle window (environment `runnerIdleTimeoutMs`, else the agent
`idleTimeoutMs`) for a warm `paperclip_runner` sandbox run. It shares
the lifecycle-mode resolution with `resolveRunnerEnvironmentForRun`.
- Host (`environment-run-orchestrator.ts`, `environment-runtime.ts`):
pass that value as `warmRunnerIdleTimeoutMs` to the sandbox-provider
driver on acquisition and on remote recovery resume. The driver adds it
to the provider worker config only. Lease identity, reuse matching, and
lease metadata are unchanged.
- Daytona manifest: describe the warm floor on `autoStopInterval`.
- `doc/execution-semantics.md`: document the contract.

## Verification

- Red (before the fix, same tests, with the floor disabled): 5 failures
in `plugin.test.ts` "warm idle window versus Daytona auto-stop". The
staging case failed with `expected 900000 to be greater than or equal to
1200000`. The resume case failed because `setAutostopInterval` was not
called.
- Red (host): the new `environment-runtime.test.ts` case first failed
because the provider worker config had no `runnerIdleTimeoutMs`. The
service wrapper dropped the field. The fix forwards it.
- Green: `npx vitest run --config vitest.config.ts` in
`packages/plugins/sandbox-providers/daytona`: 563 passed. This includes
a setter that never settles (resume completes after the 5-second bound)
and `inherit` with no forwarded window (explicit `autoStopInterval: 2`
stays 2).
- Green: `npx vitest run
src/services/runner-environment-lifecycle.test.ts
src/__tests__/environment-runtime.test.ts
src/__tests__/environment-run-orchestrator.test.ts` in `server/`: 242
passed. This includes recovery resume forwarding the admitted warm
window.
- The regression tests assert the relationship `autoStop >= warm idle +
margin` for several warm windows (125 s, 300 s default, 900 s, 1,800 s)
and for an explicit `autoStopInterval: 15`. They are not constant
checks.
- `node scripts/typecheck-low-memory.mjs` passes. `pnpm typecheck` in
the Daytona package passes.

## Risks

- A warm reusable Daytona sandbox now stays running up to the warm
window plus 5 minutes after its last activity, instead of 15 minutes.
With a 15-minute window this is 20 minutes. Cost is bounded and idle
sandboxes still stop.
- The resume-time `setAutostopInterval` call is best effort. If it
fails, the old interval stays and the #19195 retry still applies.
- Other providers are unchanged. They receive one extra host-owned
config key (`warmRunnerIdleTimeoutMs`) only for warm `paperclip_runner`
runs, and they ignore unknown keys.
- No lease identity change, so existing retained sandboxes are still
reused.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use (shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 01:10:19 -06:00
DottaandClaude Opus 5.5 c656b95b6b fix(environments): retry a refused sandbox resume with the bounded bootstrap budget (#19195)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A warm remote agent keeps its sandbox after a turn, so a follow-up
turn resumes the same sandbox and its files
> - The sandbox provider auto-stops an idle sandbox (Daytona: 15 minutes
by default). The warm native idle window is also 15 minutes
> - A follow-up that reaches the sandbox during that stop gets a
provider refusal. The host keeps the sandbox ("the lease was preserved
and no replacement was created"), which is correct
> - But setup then records a generic `setup_failed`. No retry is
scheduled, no cause is recorded, and the reply that woke the agent is
effectively dropped
> - This pull request gives that refusal its own failure code and routes
it through the existing bounded bootstrap retry. The retry resumes the
same recorded sandbox
> - The benefit is that a follow-up continues a few seconds later on the
same saved sandbox, instead of stopping the task with no next action

## Linked Issues or Issue Description

Refs #18852 (the bounded bootstrap retry that this change reuses).
Related: #19152 and #19187 (copy-back causes), #19178 (Daytona
errored-sandbox recovery before stop-only cleanup), #19182 and #19180
(recovery holds). This change does not overlap with them: they cover
cleanup, copy-back, and recovery after a run. This change covers a
resume refusal before provider work starts.

**What happened?**

On a staging soak, two follow-up turns failed at setup after a
successful first turn:

- A board reply woke the agent. The run failed after 4.9 seconds with
`setup_failed`: `Reusable sandbox lease "…" could not be resumed; the
lease was preserved and no replacement was created.` The issue stayed in
review with no recovery action. The reply was not delivered.
- An earlier run on another task failed with the same message.

Both resume attempts started 15 minutes 8–25 seconds after the previous
turn released the sandbox. The first turns used `keep_running` with a
warm idle timeout of 900,000 ms. The Daytona default `autoStopInterval`
is also 15 minutes. So the resume reached the sandbox while the provider
was stopping it. The provider refused within seconds; this was not a
long resume timeout. The provider cause was not recorded, so this timing
is the strongest evidence. The exact provider state remains an
inference.

**Expected behavior**

A refused resume before provider dispatch is a temporary infrastructure
condition. The host retries it with a small, bounded budget. Each retry
resumes the same recorded sandbox. A refusal does not prove that the
sandbox is lost, so the host never destroys it or creates a replacement.
Only an explicit provider not-found or a workspace identity mismatch
with a verified backup permits replacement (unchanged). The run records
why the provider refused, without provider text.

**Steps to reproduce**

1. Run a warm remote turn that keeps its sandbox.
2. Make the provider's `environmentResumeLease` throw a non-transient
error (for example, a 400/409 while the sandbox stops).
3. Wake the agent with a reply.
4. Before this change: the run fails with `setup_failed`, no retry is
scheduled, and the error has no cause.

**Paperclip version or commit**

Seen on staging at `5b99bdae`. The touched code is unchanged on current
`master` (`f7a4c5678f`).

## What Changed

- Move `ReusableSandboxResumeError` to
`server/src/services/reusable-sandbox-resume-error.ts` (re-exported from
`environment-runtime.ts`). Give it the code
`reusable_sandbox_resume_failed`.
- Add the provider refusal to the error message by closed labels only:
`(cause=<label> status=<http> code=<errno>)`. The labels come from the
existing transport classifier, now exported as
`sandboxProviderFailureLabel`. Provider text, paths, and IDs are never
copied.
- Carry the provider status across the plugin worker RPC. A JSON-RPC
error has only a message and a numeric RPC code. The worker host now
attaches a validated `paperclip/environment-resume-error/v1` envelope
(HTTP status and an allowlisted errno only) to `environmentResumeLease`
failures. The host re-validates it and adds it to the recorded cause.
- In setup-failure settlement, find this error through the orchestrator
wrapper. Record `reusable_sandbox_resume_failed` only when provider
dispatch did not start and the run has no native session or runner
identity (the same guard as `remote_workspace_prepare_timeout`).
- Add the code to `isTransientWorkspacePreparationCode`. The existing
scheduler then gives two retries (30 s, 60 s) with its issue gate, Stop,
pause, and dependency checks.
- Exclude the code from generic continuation recovery and from legacy
reconciliation (no unknown provider actions; bootstrap evidence proves
none started), like the template.
- Add a specific stranded-recovery notice: "Saved sandbox could not be
resumed".
- Document the contract in `doc/execution-semantics.md`.

## Verification

- Red (before the fix, same tests): 6 failures. The `fresh` case
recorded `setup_failed` and did not schedule a retry. The error had no
cause. Legacy reconciliation held the run. Generic recovery granted
another budget.
- Green: `npx vitest run src/services/heartbeat/run-completion.test.ts
src/services/legacy-execution-recovery.test.ts src/services/recovery/
src/__tests__/heartbeat-retry-scheduling.test.ts
src/__tests__/environment-runtime.test.ts
src/services/remote-workspace-transport.test.ts
src/services/heartbeat/retries.test.ts` (in `server/`): 619 tests pass.
Also `src/services/recovery/stranded-notice` and
`src/services/remote-workspace*`: 411 pass.
- Safety regressions:
- `run-completion.test.ts`: only a pre-dispatch legacy failure with no
native identity gets the code and a retry. `started`, `retained`
(native), and an untyped copy of the message stay `setup_failed` with no
retry. The recorded error contains the closed cause and not the provider
path.
- `heartbeat-retry-scheduling.test.ts` (real PostgreSQL): two concurrent
schedulers across a restart create exactly one successor per attempt.
The budget ends after two attempts. Stop, pause, and a new dependency
cancel the scheduled retry. Generic stranded recovery creates no extra
run after exhaustion.
- `packages/plugins/sdk/tests/environment-resume-error.test.ts`: through
the real worker RPC, the `JsonRpcCallError` has no status, and the
envelope gives 409 and the errno. Provider text does not cross. Other
methods get no envelope. Malformed envelopes are rejected.
- `environment-runtime.test.ts`: a throwing resume still preserves the
exact lease. The runtime does not call destroy or acquire. The resume
throws a `JsonRpcCallError` whose message has no status. The thrown
message names `cause=provider_error status=409` from the envelope, and
not the provider text.
- `node scripts/typecheck-low-memory.mjs` passes. `git diff --check`
passes.
- Not verified: a live staging run at this revision. The original tasks
were not retried. This PR is not deployed.

## Risks

- Low. The change affects only runs that fail with this typed error
before provider dispatch. Each retry resumes the same lease and inherits
the source wake context, so a reply is delivered once and no second
executor starts.
- If the provider cannot start the sandbox for longer than about 90
seconds, the two retries end and the task gets the specific recovery
notice. The sandbox and its saved files are kept.
- The error message now has a parenthetical cause before the semicolon.
No code parses this message.
- An issue in review does not get the immediate stranded notice
(unchanged existing behavior). The retry, not the notice, delivers the
reply in that case.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, browser reads of staging diagnostics).
Extended reasoning enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 00:20:21 -06:00
Devin FoleyandPaperclip 895cee28b3 fix(daytona): recover an errored sandbox before stop-only cleanup (#19178)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Remote agent runs use sandbox provider plugins. Daytona is one of
them.
> - After a failed run, Paperclip stops the sandbox and keeps its files
("stop-only cleanup"). A task can continue only when every remote lease
of the failed run has a provider stop receipt.
> - Daytona rejects Stop for a sandbox in its `error` state. A sandbox
that filled its disk went to that state. The plugin called Stop on it,
so every cleanup sweep failed with the same generic error.
> - The lease stayed in pending cleanup without a stop receipt. Each
Retry or comment waited forever with reason `remote_cleanup`. A manual
Daytona Recover was the only fix.
> - This pull request makes the Daytona stop-only hook recover a
recoverable sandbox once, with a bound, and then stop it. When it cannot
stop the sandbox, it returns a typed failure. The server escalates a
terminal failure at once instead of retrying it like a transient error.
> - The benefit is that a full-disk sandbox no longer blocks its task,
and a sandbox that cannot recover is visible to operators on the first
attempt.

## Linked Issues or Issue Description

No public issue exists. Bug description:

**What happened?**

A task sandbox filled its 10 GB disk. Daytona then reported the sandbox
as `state: error`, `desiredState: started` and `recoverable: true`. Its
`errorReason` was a Docker `no space left on device` error. The Daytona
`stopLease` hook calls Stop on any sandbox that is not `stopped` or
`archived`. Daytona rejects Stop in the `error` state. The pending
cleanup sweep ran about every 30 minutes, and it failed on every pass.
The lease stayed `pending_cleanup` with `cleanup_status=failed` and no
stop receipt. Continuation admission
(`server/src/services/explicit-native-continuation.ts`) requires a stop
receipt on every remote lease of the failed run. So each Retry and
comment was deferred with "Waiting for the previous environment to stop.
Your message will start automatically." It never started.

A manual Daytona `POST /api/sandbox/{id}/recover` fixed it. The
sandboxes reached `started` in about one minute. The next sweep stopped
them, recorded the receipt, and the tasks continued.

**Expected behavior**

Stop-only cleanup recovers a recoverable errored sandbox and then stops
it. When the sandbox cannot be recovered, cleanup returns a clear, typed
failure that the server can tell apart from a transient error. Cleanup
never deletes the retained sandbox, and it never loops on Recover.

**Steps to reproduce**

1. Run a task in a Daytona sandbox until the sandbox fills its disk and
Daytona shows `state: error`, `recoverable: true`.
2. Let the run fail, so the server records a `sandboxStopAndRetain`
intent and calls `environmentStopLease`.
3. Before this change, Stop fails with `Sandbox failed to stop with
status: error` on every sweep, and the task cannot continue.
4. The new tests in `plugin.test.ts` reproduce this with a mock sandbox
whose Stop fails from `error`.

**Paperclip version or commit**

Observed on a hosted deployment. Reproduced in unit tests on `master` at
`6452e964a36`.

**Deployment mode**

Managed Daytona sandbox execution with plugin-backed stop-only cleanup.

Related: #18172 (Daytona exports under disk pressure) and #19050
(session reset on every Daytona restart path). This PR does not change
the resume path in `ensureSandboxStarted`, which already recovers a
recoverable sandbox.

## What Changed

- **Daytona plugin
(`packages/plugins/sandbox-providers/daytona/src/plugin.ts`)**: after
`stopLease` confirms the retention policy, a sandbox in `error` goes
through `recoverSandboxForStop`.
- Recoverable: the hook calls `sandbox.recover()` once. The SDK waits
for `started`. The plugin bounds the wait to 20 s, plus the existing
liveness margin for a hung connection. Then the hook stops the sandbox
normally and returns the usual `stopped` receipt.
- Not recoverable: the hook makes no Recover or Stop call. It throws
`PluginEnvironmentStopError("resource_unrecoverable")`.
- Recover fails: the hook reads the provider state once. A sandbox back
in `error` gives `recovery_failed`, or `resource_unrecoverable` if
Daytona now marks it so. A sandbox that is still starting, or a failed
read, gives `recovery_incomplete`.
- The recovered sandbox fails again before Stop finishes (for example,
the disk fills again): a fresh read that shows `error` gives
`recovery_failed`. The hook never calls Recover a second time in the
same call.
- The hook never deletes or archives the sandbox. The non-error stop
path is unchanged.
- **Plugin SDK (`packages/plugins/sdk/src/environment-stop-error.ts`,
new)**: `PluginEnvironmentStopError`, `environmentStopErrorData` and
`readEnvironmentStopFailure`. This follows the existing
`PluginEnvironmentRunnerIngressError` pattern. The envelope holds only
one closed reason. The reader validates the reason and derives
`terminal` from it. `resource_unrecoverable` and `recovery_failed` are
terminal. `recovery_incomplete` is not.
- **Worker RPC host (`packages/plugins/sdk/src/worker-rpc-host.ts`)**:
forwards that envelope as JSON-RPC error `data` for
`environmentStopLease` only.
- **Pending cleanup sweep
(`server/src/services/heartbeat/recovery.ts`)**: the catch reads the
validated envelope.
- On a terminal failure, it escalates on the first attempt. It writes
the one-time "needs operator attention" warning and uses the existing 30
minute retry interval. Before, this happened only at the fifth attempt.
- The failure log uses `errorKind: "provider_stop_terminal"` and
includes the closed `providerStopFailure` reason. The sweep still never
logs exception fields.
- The lease stays `pending_cleanup`. Automatic cleanup continues, and an
explicit Retry still bypasses the cooldown. Untyped and transient
failures keep their current behavior.
- **Docs**: the SDK README ("Stop failures"), the Daytona README,
`doc/plugins/SANDBOX_PROVIDER_CAPABILITIES.md` and
`doc/native-workspace-finalization-recovery.md` describe the new
behavior.

Shared contract note (module boundaries): the SDK envelope is the
smallest addition that lets the host tell a terminal provider stop
failure from a transient one. The worker host drops all other error data
across the RPC boundary, so the existing contracts cannot carry the
signal. The only consumer is the pending cleanup sweep. When a provider
does not throw the new error, or an older host ignores it, behavior is
unchanged. This PR does not touch
`server/src/services/workspace-recovery.ts` or
`sandbox-stop-and-retain.ts`.

## Verification

All commands ran in a clean worktree from `master`. Vitest ran from the
real path.

- Red on `master` (only the new tests added):
- Daytona, with `plugin.ts` from `master`: `npx vitest run --config
vitest.config.ts src/plugin.test.ts -t "provider error state"` exits 1.
All 7 new tests fail. The main case fails with `Sandbox failed to stop
with status: error`, which is the production error.
- SDK, with `worker-rpc-host.ts` from `master`: `npx vitest run
tests/worker-rpc-host.test.ts -t "typed stop failure"` exits 1 (the
typed case fails).
- Server, with `recovery.ts` from `master`: `npx vitest run
src/__tests__/heartbeat-pending-cleanup-sweep.test.ts -t "terminal
provider stop failure"` exits 1. All 3 cases fail (`errorKind:
"destroy_failed"`, no escalation).
- Green on this branch:
- `packages/plugins/sandbox-providers/daytona`: `npx vitest run --config
vitest.config.ts` exits 0 (534 passed, 26 skipped live tests). `pnpm
typecheck` exits 0. `pnpm build` exits 0.
- `packages/plugins/sdk`: `npx vitest run` exits 0 (222 passed). `pnpm
--filter @paperclipai/plugin-sdk typecheck` exits 0.
- `server`: 15 related suites exit 0 (1498 passed): pending cleanup
sweep, `heartbeat/recovery`, environment runtime, environment cleanup
work, native startup cleanup, native remote failure retention, observed
workspace preservation, retained native session cleanup, idle sleep
safety, explicit native continuation, task retry, workspace recovery,
native failed and cancelled workspace preservation, and Daytona close in
the plugin worker manager.
- `pnpm --filter @paperclipai/server typecheck` exits 0. `pnpm
check:module-boundaries` exits 0.
- New tests:
- Daytona: a recoverable errored sandbox is recovered once
(`recover(20)`) after the retention check and before Stop, then stopped.
An unrecoverable one returns `resource_unrecoverable` with no Recover,
Stop, start, delete or archive call. A failed recovery makes exactly one
Recover call for each outcome (`recovery_failed`,
`resource_unrecoverable`, `recovery_incomplete`). A Recover call that
never settles is cut off by the bound. A recovered sandbox that fails
again before Stop gives `recovery_failed` with no second Recover.
- SDK: the envelope holds only the reason. A forged or non-SDK error
gives no data. Unknown reasons, accessor fields and a sender-supplied
`terminal` flag are not trusted. The worker host forwards the typed
failure and drops a plain error's detail.
- Server: terminal reasons escalate on attempt 1 with the long retry and
one operator warning. `recovery_incomplete` keeps the short retry. The
lease stays `pending_cleanup`, and the provider message never reaches
the logs.
- Not run: the full `pnpm -r typecheck`, `pnpm test:run`, `pnpm build`
and `pnpm run typecheck:build-gaps` repo checks. CI runs them. No live
Daytona probe ran. The mocks model the Daytona SDK `recover()` contract:
it waits for `started` and rejects when the sandbox returns to `error`.

## Risks

- Time budget: the host gives `environmentStopLease` at most 60 s. A
recovered stop uses up to 20 s for Recover and up to 30 s for Stop, plus
short reads. A slower recovery returns `recovery_incomplete`. The SDK
call can continue at Daytona after that, so the next sweep finds the
sandbox started and stops it normally. If the host deadline fires while
the worker still stops the sandbox, the next sweep sees `stopped` and
records the receipt.
- Classification: after a failed Recover, the plugin cannot tell a
rejected request from a sandbox that started and failed again. Both give
`recovery_failed` (terminal) when Daytona still shows `error`. A hung
Recover call with a sandbox still in `error` is also terminal. Terminal
failures still retry every 30 minutes, and each retry makes at most one
Recover call. An explicit Retry also makes one attempt. There is no
tight loop. A sandbox that fails Recover every time gets about two
Recover calls an hour until an operator acts.
- Recover restarts the sandbox container before Stop. The plugin runs no
command in it, and the retention policy is confirmed before Recover.
- The continuation message for a terminally failed lease still says the
message "will start automatically". This PR does not change admission. A
follow-up can use the typed reason there.
- Compatibility: a host without this change ignores the new error data
and keeps the old retry behavior. A plugin without this change sends no
typed data.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use (shell, file edits, code execution for tests and typechecks).
The runtime does not expose the context window size to the agent.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (affected suites; full repo
runs are left to CI, as noted above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-04 21:33:48 -08:00
DottaandPaperclip 9e5039f6c8 fix(daytona): retain duplex close ownership after lease cleanup (#19150)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Remote agents use provider plugins to manage sandbox leases and
duplex channels.
> - The host verifies the exact route and session when a bound channel
closes.
> - Daytona lease teardown removed that binding before a later host
close.
> - The incomplete reply made the host retire the shared worker. A later
command then failed during setup.
> - This pull request retains bounded close receipts and joins the
original cleanup.
> - Later work can use the same worker while invalid or unconfirmed
closes still fail closed.

## Linked Issues or Issue Description

**What happened?**

A normal lease teardown can close a duplex channel before the host
closes its bound route. Daytona then replies with only the route ID. The
host requires the original session ID too. It logs `duplex channel close
not acknowledged; retiring worker`, stops the worker, and rejects a
later `environmentExecute` call as stopped.

**Expected behavior**

A later close must identify the original route and session and join its
existing cleanup. It must not revive the channel or restart the worker.
Unknown ownership and an unconfirmed close must retain the existing
failure behavior.

**Steps to reproduce**

1. Acquire two leases through one Daytona provider worker.
2. Open a duplex channel on the first lease.
3. Release that lease, then close the bound channel through the host.
4. Execute a command on the second lease.

**Paperclip version or commit**

The real host/worker regression fails on
`69dea878c0b920b335594debb0286443ba9f6c35`. This branch includes master
`3a72efebe5b050e7594bb309e71a0d5f3072db00`.

**Deployment mode**

Managed Daytona sandbox execution in Cloud. The regression uses a local
real host and SDK worker. Only the external Daytona SDK and PTY are
simulated.

Related work: Refs #19004, Refs #10429, Refs #18065, Refs #19050. These
cover task Retry, managed worker startup, drain release, and stale
provider sessions.

## What Changed

- Retain at most 4096 retired route/session bindings with their original
close promises.
- Remove live channel authority before cleanup starts. Concurrent and
later closes join the same promise and return the stored binding.
- Reject reuse of a retained route ID. Keep unknown and evicted routes
unable to provide a session acknowledgement.
- Add a real child-process host/SDK-worker/Daytona regression with only
the external Daytona SDK and PTY simulated.
- Cover release, cancellation, stop, destroy, resume, a stalled close,
concurrent closes, failed cleanup, forged IDs and bounded eviction.
- Document close receipt scope and its separate provider-termination
checks.

## Verification

- RED before the repair: the actual host and SDK worker reproduced the
retirement log, SIGKILL, stopped state, and later `environmentExecute`
rejection. The direct provider cases also failed because the session ID
was missing.
- Combined-branch provider suite: 364 passed.
- Combined-branch host manager, duplex, managed readiness and composed
suites: 142 passed. The final test-only manifest-loading change passed
all 6 composed cases again.
- Full repository typecheck (`node scripts/typecheck-low-memory.mjs`),
focused server types and the standalone Daytona build passed. The
provider build uses pinned SDK `0.203.0` and TypeScript `5.9.3`.
- The composed cases check the same worker/PID, one later command,
preserved saved content, revoked channel writes/stops, and continued
rejection of commands on closed leases. A stalled close still retires
the worker and dispatches no later command.
- Reproduce with `pnpm --dir server exec vitest run --config
vitest.config.ts
src/__tests__/plugin-worker-manager-daytona-close.test.ts` from a
checkout with the SDK build dependencies prepared. Run the provider
suite with its own Vitest config.
- An initial local typecheck lacked the excluded providers' SDK links
after an install with scripts disabled. The standard root postinstall
linker restored those local links, and full typechecking then passed.
The fixture loads its actual built manifest. Failed setup output is
retained; no type configuration was relaxed.
- The first two composed attempts stopped during fixture setup. They are
not the behavioral RED above. Their failed outputs are retained.
- The first broad local typecheck stopped at an IPC permission error
during preflight. It passed after rerunning with allowed local IPC and
the standard SDK links.
- [Full repository
CI](https://github.com/paperclipai/paperclip/actions/runs/41945022881)
passed all 48 jobs on this head. Selection fell back to the complete
task plan, so the run was unfiltered. The complete check inventory has
54 successes and two expected Storybook skips; the separate Snyk status
passed. Fresh Greptile review on
`daebbeb5005b9d9bd59d32ea30094279048b9213` is [5/5 with no actionable
comments](https://github.com/paperclipai/paperclip/pull/19150#issuecomment-6646621914).
Independent production and fixture review found no actionable findings.
Hosted acceptance of this repair has not run.

## Risks

- A receipt removed at the 4096-entry bound cannot confirm an old bound
close. The existing host guard remains effective in that case.
- Close acknowledgements identify channels. They do not certify remote
process termination or clear unknown external work. Provider termination
receipts and existing recovery gates remain in force.
- Hosted evidence confirms the unacknowledged-close retirement class.
The historical log does not distinguish a missing session ID from a
timeout or another close error. The regression proves and repairs the
missing-binding path; it also preserves timeout retirement.
- The provider is outside the main pnpm workspace. A new Cloud image
must compile its changed source. The unchanged plugin version alone does
not prove activation.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, source editing and tool
execution. The exact model ID and context window are not exposed in this
session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-04 19:15:50 -06:00
DottaandClaude Opus 5.5 367aa16800 fix(daytona): drop the stale session id on every sandbox restart (#19050)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native agent runs can execute inside a Daytona sandbox through the
Daytona sandbox provider plugin
> - The plugin runs every command for a lease in one persistent Daytona
process session and keeps that session id in memory
> - Daytona auto-stops an idle sandbox, and a stopped sandbox loses all
of its process sessions
> - Only the execute and resume hooks dropped the stored session id
before a restart. Workspace realize, sync-in, sync-out, runner ingress,
and interactive setup also restart a stopped sandbox, but they kept the
dead id
> - When host-side workspace preparation stays idle for longer than the
auto-stop interval, sync-in restarts the sandbox, and the next command
sends the dead id. The native run then fails with
`native_session_interrupted: session not found`
> - This pull request moves the session-id clear into the one shared
start helper, so every restart path drops the stale id
> - The benefit is that a run that waits through a long preparation
opens a fresh session instead of failing

## Linked Issues or Issue Description

**What happened?**
Native runs on Daytona failed with `native_session_interrupted` and the
error `session not found`. The failure came about 4 seconds after the
provider session started. In every observed run, the workspace pack
phase took longer than 15 minutes (917 s to 1442 s). This is longer than
the default Daytona `autoStopInterval` of 15 minutes.

**Expected behavior**
After Daytona restarts an auto-stopped sandbox, the next command opens a
new process session and the run continues.

**Steps to reproduce**
1. Run one command on a Daytona lease, so the plugin opens and stores a
session.
2. Let the sandbox stop (Daytona auto-stop after an idle gap).
3. Call workspace realize or sync-in on the lease. The hook restarts the
sandbox.
4. Run the next command. It uses the stored session id, and Daytona
answers `session not found`.

No related open PR exists. The undeployed native fixes on master
(#18992, #19025, #19026, #19004) do not touch the Daytona session store.

## What Changed

- `ensureSandboxStarted` in the Daytona plugin now takes the lease
scope. When the sandbox is not running, it clears the stored session id
before it starts or recovers the sandbox.
- All seven callers (execute, resume, realize workspace, interactive
setup, runner ingress, sync-in, sync-out) pass their lease scope. The
compiler enforces the scope on any new caller.
- The execute hook no longer has its own copy of the clear. The resume
hook keeps its clear because it also closes duplex channels before the
restart.
- Added three regression tests: a fresh session after realize restarts
an auto-stopped sandbox, a fresh session after sync-in restarts it, and
no new session when realize finds the sandbox already running.

## Verification

- Red before the fix: `pnpm exec vitest run src/plugin.test.ts -t
"auto-stopped sandbox|already running"` in
`packages/plugins/sandbox-providers/daytona`. The two restart tests
failed with `expected "vi.fn()" to be called 2 times, but got 1 times`.
The running-sandbox safety test passed.
- Green after the fix: the full Daytona plugin suite (`pnpm exec vitest
run`) passes, 542 passed and 8 skipped.
- `pnpm typecheck` in the Daytona plugin package passes.
- Not done: a live Daytona run that idles past the auto-stop interval.
The fix needs deployment before the staging failure can be checked
again.

## Risks

- Low risk. The clear runs only when the sandbox is not in the `started`
state. A stopped sandbox has no live session, so no live shell can leak.
A running sandbox keeps its stored session, and a test covers this case.
- The recover path for a recoverable `error` state now also drops the
id. A recovered sandbox restarts, so its old session is also gone.
- Duplex channels on a restarted sandbox are still closed only by the
resume hook. This PR does not change that behavior.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, browser reads of staging run
records).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-04 18:24:41 -06:00
Nicky LeachandClaude Opus 5.5 be20d4ad97 fix(projects): keep projects on the storage chosen at creation (#19036)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A project stores its work in Core/local storage or in a storage
plugin
> - Today, Core finds the installed default each time it reads a project
> - So installing a default storage plugin moves existing projects,
without a saved change
> - Agents that cannot reach the new storage then lose access to new
workspaces
> - This pull request makes Core apply the default once, when it creates
a project
> - The benefit is that a project stays on its storage until a person
moves it

## Linked Issues or Issue Description

No issue exists. This describes the problem. I searched open pull
requests for related work and found no duplicate.

**What existing behavior does this improve?**

How Core chooses the storage for a project when a plugin claims
`defaultForProjects`.

**Current behavior**

Installing such a plugin changes the storage of every existing project
with no explicit choice. Disabling the plugin changes it back. Nothing
is saved, so the project has no fixed owner. Execution environments and
storage are separate contracts, so an agent can lose access to new
workspaces on a project that it used before.

**Proposed behavior**

Core saves the storage choice on the project when it creates the
project. A project with no saved choice uses Core/local storage. A
plugin-backed project stays attached to its plugin. A Core/local project
can move to a plugin.

**Reason and benefit**

A project stays on its storage until a person moves it, so installing or
disabling a plugin does not change where work goes.

**Alternatives considered**

- Keep the default at read time and add a process that saves a choice on
existing projects when a plugin arrives. This adds a background step and
a race with new projects.
- Add a new database column. The existing policy field already holds the
choice, so a migration is not necessary.

## What Changed

- `resolveProjectWorkspaceProvider` reads only the saved choice. It no
longer applies an installed default.
- `withDefaultProjectStorage` applies the default when Core creates a
project, including a personal private project. With more than one
default, a project with no explicit choice is refused. This is the
existing conflict rule, now applied at creation.
- `projectService.update` refuses a change of storage on a plugin-backed
project. An update that omits the field keeps the attachment.
- Plugin lifecycle delivery sends project events to a storage provider
only for projects attached to it, and for deleted projects. Core
acknowledges the other events for that provider. Other plugins continue
to receive all project events.
- The project settings control has no "Inherit" option. It is locked on
a plugin-backed project.
- `ProjectWorkspaceProviderConfiguration.source` no longer has the
`"default"` value.
- A project with no repository or folder ignores its saved storage
choice when a task runs. A storage plugin cannot manage this type of
project, so its tasks use a plain task directory. Without this change,
the second run of a task reset the agent session, and remote agents
could not use an empty task directory.
- The plugin SDK README and the plugin authoring guide describe the new
rules.

## Verification

- `npx vitest run
server/src/__tests__/project-workspace-provider.integration.test.ts
server/src/__tests__/plugin-lifecycle-delivery.test.ts
server/src/__tests__/plugin-workspaces.integration.test.ts
packages/plugins/sdk/tests/workspace-provider.test.ts`: 93 tests pass.
- `npx vitest run src/components/ProjectProperties.concurrency.test.tsx`
in `ui/`: 9 tests pass.
- `pnpm --filter @paperclipai/server typecheck` and `pnpm --filter
@paperclipai/ui typecheck` pass.
- `npx vitest run` on `heartbeat-session-preparation`,
`workspace-preparation`, `remote-workspace-source`,
`project-workspace-provider.integration`, and
`plugin-workspaces.integration` in `server/`: 231 tests pass.
- I did not run the full test suite locally. CI runs it.
- I did not test this change on a running instance.

## Risks

- **Behavior change at upgrade:** an existing project that uses a plugin
only through the installed default, with no saved choice, moves to
Core/local storage after upgrade. Its plugin files stay in place, but
new work does not use them until an operator sets
`storageProviderPluginId` on the project. This pull request has no
automatic backfill.
- **Missed lifecycle events:** a provider does not receive events for a
Core/local project. If the project moves to that provider later, the
update event starts its reconcile.
- **One-way move:** a person cannot move a plugin-backed project back
through the API or the UI. This is intended. If a plugin is disabled,
the project fails closed, as it does today for an explicit choice.
- This pull request does not stop a person from assigning an agent to a
project with storage that the agent cannot reach. That is follow-up
work.
- This changes core plugin behavior, and I did not discuss it in `#dev`
first. Maintainers can tell me if they want a discussion before review.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), through Claude Code, with tool
use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-04 13:07:29 -08:00
Devin FoleyandPaperclip cf0ab4f7f4 chore: enable boxd plugin release publishing (#19020)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Sandbox provider plugins let those agents run in remote
environments.
> - The boxd provider exists, but its release entry excludes it from CI
publishing.
> - npm now has the `@paperclipai/plugin-boxd@0.0.0` bootstrap
placeholder.
> - This pull request enables CI publishing and documents npm
installation.
> - A pre-existing OpenCode fixture blocked CI because it qualified a
group-writable host Node binary.
> - This also gives that test its own Linux Node copy with strict
permissions.
> - Operators can install a real release after CI publishes it.

## Linked Issues or Issue Description

Related implementation:
https://github.com/paperclipai/paperclip/pull/17284.
No matching public issue or duplicate release-enrollment PR was found.

**What existing behavior does this improve?**

Publishing and installing the existing boxd sandbox provider.

**Subsystem affected**

`packages/plugins` and the npm release manifest.

**Current behavior**

`publishFromCi` is false. npm has only the `0.0.0` placeholder. The
README says the package is not published and requires a local-path
install.

**Proposed behavior**

Include boxd in CI releases. Document stable installation, explicit
published versions, and the placeholder limit.

**Reason and benefit**

Operators can install the provider through the existing npm plugin
installer after a real release ships.

**Breaking changes**

None. This changes release enrollment and documentation only.

**Additional context**

This follows the existing plugin and sandbox roadmap items. It adds no
new runtime capability.

## What Changed

- Set `publishFromCi: true` for `@paperclipai/plugin-boxd` in the
release manifest.
- Replace the README's unpublished-package notice with npm installation
guidance.
- Explain that the `0.0.0` placeholder has no plugin functionality and
that `latest` can point to it until the first stable release.
- Keep the local development instructions available.
- Fix the existing ingress-retry test to use an owned Linux Node copy
and explicit executable permissions. This follows the existing OpenCode
transport fixture. It does not change production launch checks.

## Verification

- Passed: `node --test scripts/release-package-map.test.mjs
scripts/check-release-package-bootstrap.test.mjs
scripts/bootstrap-npm-package.test.mjs` (31 tests).
- Passed: `node scripts/release-package-map.mjs check`.
- Passed: `PAPERCLIP_RELEASE_BOOTSTRAP_BASE_SHA=origin/master node
scripts/check-release-package-bootstrap.mjs
scripts/release-package-manifest.json
packages/plugins/sandbox-providers/boxd/README.md`.
- Passed: `git diff --check`.
- Confirmed that npm resolves `@paperclipai/plugin-boxd` to version
`0.0.0`.
- Full workspace typecheck, test suite, and build were not run locally.
CI checks remain required before merge.
- Initial CI failure: [native Runner
job](https://github.com/paperclipai/paperclip/actions/runs/41889969053/job/126785636239),
OpenCode ingress retry, `qualified OpenCode proxy command must not be
group- or world-writable`. The failing fixture is byte-identical to the
base commit `3a2249dd576`. It passed `process.execPath` to
qualification, while other OpenCode tests already use a private Linux
Node copy.
- Local focused run: `pnpm exec vitest run
server/src/services/native-runtime/native-ingress-bootstrap-retry.integration.test.ts`
built the real Rust Runner and passed the Codex case. The OpenCode case
stopped at `runner_remote_provider_artifact_incompatible: OpenCode proxy
path belongs to the controller host` on macOS (`/Users/...`). This does
not exercise the new Linux-only copy.
- An unchanged-base OpenCode comparison was attempted with `-t
"reacquires opencode ingress"`, but PostgreSQL setup failed with
`ENOSPC` before the case ran. The local baseline comparison is therefore
inconclusive. Task-owned Rust build outputs were cleaned to recover disk
space.
- At this stage Linux verification was pending. The successful final run
is recorded below.
- No new tests were needed. The existing release-tool tests cover
manifest enrollment, dependency ordering, and bootstrap checks.

- Current-head CI interruption: [workspaces-a shard
1/2](https://github.com/paperclipai/paperclip/actions/runs/41893808561/job/126799012853)
was cancelled after the runner received a shutdown signal. The log shows
no assertion failure. The job rerun request was rejected with HTTP 403
because the workflow was still running. The subsequent fixture
correction starts a new full CI run and supersedes that pending rerun.

- CI at `7fffa315`: [native Runner
job](https://github.com/paperclipai/paperclip/actions/runs/41893808561/job/126799011711)
passed executable permission checks, then failed because Node tried to
resolve `/memfd:paperclip-verified-launch (deleted)`. The private
executable was named `opencode-node`, which bypassed Runner's Node
interpreter recognition. The fixture now names its copy `node`, matching
the existing transport fixture and selecting the CommonJS descriptor
loader.
- Passed a local smoke test of the production CommonJS descriptor loader
using an open, unlinked script file. This is macOS evidence only; the
successful final Linux run below provides the complete ingress retry
evidence.

- Final verification on `bcacac542aa0a1b641b26c099c1e9fc4a1653ec4`: all
54 reported checks passed; three unrelated checks were skipped. [Native
Runner integration
job](https://github.com/paperclipai/paperclip/actions/runs/41895787712/job/126805844773)
passed, including the OpenCode ingress retry regression. The fresh full
run also passed the workspace shard that was interrupted on the previous
attempt. Greptile reports 5/5 for this exact head with no unresolved
review threads. Earlier failures and local verification limits remain
recorded above.

## Risks

- The maintainer confirmed that npm trusted publishing is configured for
this package. The first real CI publish will verify OIDC authentication.
- Merging enables publication in the next successful canary run. An
installation that uses `latest` can still resolve to the placeholder
until the first stable release includes this change.

## Model Used

OpenAI Codex, exact model ID `gpt-6.1-sol`, with xhigh reasoning effort,
repository tool use, and shell execution. The runtime does not expose
the context window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-04 11:36:04 -08:00
DottaandPaperclip 3a2249dd57 fix(runner): retry proven ingress bootstrap failures (#18981)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native execution must retain its admitted model and execution
authority during recovery.
> - A sandbox provider can reject Runner endpoint acquisition before the
Runner gets a credential.
> - The old error path loses the structured HTTP status and leaves that
startup failure ambiguous.
> - A zero counter in an old checkpoint cannot prove that no credential
was issued.
> - This pull request adds a versioned producer proof and carries closed
provider status through the existing RPC.
> - A proved transient startup failure uses the existing retry budget. A
terminal proved-unused run can settle its accounting.

## Linked Issues or Issue Description

Refs #18018 and #18934. Those changes restore existing checkpoints and
ready retained environments. This change covers endpoint acquisition
before Runner dispatch. A search of related public PRs found no
duplicate implementation.

**What happened?**

A provider HTTP 502 during Runner endpoint acquisition becomes an
untyped worker error. Native recovery treats it as ambiguous and stops
without retrying. The run's native usage receipt also stays pending
after startup ends without model work.

**Expected behavior**

Retry a structured transient endpoint failure only when the complete
current controller checkpoint proves that this exact run never issued a
Runner credential. Preserve authentication, configuration, Stop,
ownership and provider-progress gates. Settle zero usage only under the
existing terminal accounting owner.

**Steps to reproduce**

1. Use the controlled endpoint in the Daytona ingress integration test
to return HTTP 502.
2. Register the actual native Runner driver and persist its bootstrap
journal through the encrypted checkpoint writer.
3. Observe the old terminal ambiguous disposition instead of bounded
startup recovery.

**Paperclip version or commit**

Reproduced on the pre-change master source. The original regression used
the production native driver, a real HTTP fault, a real durable control
plane and local Postgres. The new contract additionally tests the pinned
Daytona SDK and real worker RPC.

## What Changed

- The controller owns a v2 persisted state with retained origin and
credential history. Current readers still accept v1. Older writers
reject v2. Restore and checkpoint decoding do not add missing history.
- The proof validates the complete bootstrap journal, all six identity
fields and every delivery, credential and progress control. The server
rechecks its fingerprint under the current controller claim before retry
or accounting changes.
- Daytona and the SDK carry a closed acquisition phase and HTTP status
through the existing ingress RPC. Both public error wrappers omit
response bodies and causes, including Pino cause serialization.
- Only structured HTTP 429 or 5xx ingress failures can use the existing
three-attempt startup budget. HTTP 401/403, configuration and unknown
errors do not gain retry authority.
- A terminal proved-unused run wakes existing accounting and records
zero provider work. It does not create a semantic result, prove physical
termination or release an unknown process/workspace hold.
- The normal automatic second attempt rechecks its exact claim, frozen
execution and matching complete local/encrypted journal before admitting
absent remote Runner state. It keeps the original pending bootstrap
command IDs. A proved unused Codex home can rematerialize sealed skills.
Full raw local state is checked before streaming-event or policy
projection.
- Normal saved-state readers and the existing E2E journal evidence
matcher accept the closed v1/v2 set. The separate legacy no-launch
witness is unchanged. Technical docs describe the contracts.

## Verification

Current head: `cd38f70409cdb8664e14bfbf4efa208f468a0419` (final delta
adds only fixture artifact preparation). Production files are unchanged
from the independently reviewed `93c8a33520` source. Full repository
typecheck and build passed at `96417acb58`; the affected server
typecheck also passes after the final two-line fixture delta. Focused
protocol, raw-proof, real retry, SDK and RPC acceptance pass on this
source. Earlier broad suite counts are listed with their limits below:

- Durable Runner protocol: 187 tests.
- Native executor, driver, startup/accounting and redaction: 1,211
tests, including real Postgres and encrypted checkpoint persistence.
- Normal saved-state restore/projection compatibility: 21 focused cases.
- SDK: 209 tests.
- Standalone Daytona: 544 tests; 8 opt-in live tests skipped.
- Explicit standalone `pnpm -C
packages/plugins/sandbox-providers/daytona exec vitest run --config
vitest.sdk-ingress.config.ts`: 6 cases with real pinned SDK 0.203.0 and
worker RPC ingress acceptance. These include lookup 502/401/403/400,
private-preview 502 after successful lookup/refresh, missing credentials
and private-response exclusion.
- E2E journal evidence: 4 cases across both supported formats, with
wrong ownership, missing stimulus and unknown-schema controls.
- The exact historical controller source rejects a new v2 checkpoint at
decode and restore. A v1 control is accepted. This verifies the
downgrade fence with the actual older parser.
- Startup controls cover issued and pruned credentials, old/truncated
journals, provider progress, exhausted retry accounting, and
checkpoint/Stop/controller-claim races. Race cases retain the current
execution and pending usage.
- New exact admission controls: 19 cases through encrypted persistence
and cold restore. They cover current/stale/expired claims, Stop, pause,
model/trust/account changes, prior progress, issued/pruned credentials,
adopted/old/truncated state, local/canonical fingerprint mismatch,
hidden delivery, unsupported work and company scope. Two additional
red/green cases prove that raw local stream progress and missing event
policy are rejected before lossy checkpoint projection. No case mutates
run, claim, result or accounting state.
- Actual native executor, scheduler and Rust Runner retry: 2 cases.
Codex uses its fake app server. OpenCode uses the real proxy and fake
OpenCode HTTP server with the frozen DeepSeek model and allow permission
configuration. Each path launches no Runner on attempt 1, starts one
Runner and provider on attempt 2, reuses both original command IDs,
commits one accepted result, validates the persisted local workspace,
records nonzero synthetic native usage and settles accounting once. No
zero-work accounting flag survives.

The controlled SDK and protocol tests use local source, isolated
PostgreSQL, a private HTTPS preview and fake provider servers on macOS
arm64 with Node 24.22.2. They create no hosted sandbox and make no model
call. The root workspace intentionally omits standalone provider
dependencies; the explicit SDK target fails if its pinned dependency is
absent and has no conditional skip. They do not qualify hosted rollout
or repair an old unmarked checkpoint. The first full local typecheck
found missing checkout SDK links from an install with scripts disabled;
the repository link helper repaired that setup.

Current-head Linux CI typecheck and build pass. Independent final review
checked 12 source scopes and six raw-proof controls with no remaining
finding. Fresh Greptile review is 5/5 with no unresolved thread.

The full Runner CI suite found two older control-peer test doubles
missing the new proof method. Both now return no proof, with all 27
permission, ordering and cancellation assertions retained and passing.
The new Cargo-dependent native retry suite uses the existing Rust-cached
CI lane. All 39 lane coverage controls pass and prove complete,
duplicate-free suite coverage. The first full test run stopped at an
admitted chat Retry polling failure. CI also found a stale
retained-allocation fixture, now repaired on master by #18984, and a
canceled runner job. A real SDK suite initially relied on a dependency
absent from the root workspace; the explicit standalone target now owns
that acceptance.

The separate Product E2E support run had 12 failures in four untouched
files. All 12 reproduce on the original clean base with independent
dependencies, compiled base source and verified workspace resolution.
These concern workflow expectations, cleanup timeout arguments, a
profile snapshot and operating receipts. They are not passing
acceptance.

The later full local repository test command stopped at a 15-second
timeout in the unchanged GitHub pre-allocation retry `controller` case.
That exact case passed in isolation with unchanged assertions. The broad
Runner run initially omitted `test:typescript:prep` and lacked debug
fake-provider binaries. Required preparation then passed 370 + 44
package contract cases (11 skipped); the prepared full package run
reported 4,932 passed, 9 failed and 13 skipped. All failed families then
passed with unchanged assertions in isolation: 8 ACPX controller tests
and 2 credential controls. The paired original-base ACPX control also
passed. The full-run failures remain recorded; they are not passing
full-suite evidence.

The first Rust-cached native CI run found a missing generated OpenCode
proxy in a fresh checkout. The fixture now runs the existing verified
provider-entrypoint builder. Removing the generated proxy reproduces the
exact ENOENT before the fix; both real retry cases pass after the fix
from the same absent-artifact state.

All current-head CI gates pass, including the aggregate verification
check. Linux native CI passes all 10 suites: 177 tests pass and 14
opt-in cases skip. Both real provider retry cases pass. The full Runner
package passes 297 suites and 4,950 tests, with 1 suite and 10 tests
skipped. The CLI workspace groups pass 4,762 and 286 tests. The first CI
attempt had Runner shutdowns in the Runner package, Canary and three
general server jobs. One unchanged CLI shard also hit child-server
startup and dense ZIP deadlines. Both CLI files passed unchanged in
isolation: 18 tests. One bounded rerun of unsuccessful jobs and their
dependents passed. The original failures remain recorded.

## Risks

- Older controllers reject newly written v2 state on downgrade. Current
controllers retain v1 restore support, but old or unmarked state cannot
gain this new proof.
- An older plugin that omits structured ingress evidence remains held.
The server and provider must use the new producer contract for this
recovery path. The Docker `cloud` target builds the SDK, server/Runner
and standalone Daytona producer from one source tree. The default
`production` target does not build the standalone producer. Hosted
rollout must confirm the new image and active image-owned plugin worker;
a registry version alone does not prove its source. An npm-installed
producer needs an explicit matching artifact upgrade.
- This prevents future proved startup failures. It does not release
historical accounting based on an old checkpoint, change retry budgets,
replace a model or weaken sandbox isolation.
- Zero model usage is separate from process termination, saved files and
resource release. Those owners keep their existing gates.

## Model Used

OpenAI Codex with tool use and code execution. This runtime does not
expose the exact backend model ID or context window to the agent.
Read-only independent agents reviewed checkpoint compatibility and
SDK/RPC error boundaries.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (affected suites; full
local run limits are recorded above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-04 12:16:51 -06:00
DottaandPaperclip 86814c6550 fix(recovery): ready retained sandboxes before preparation (#18934)
## Thinking Path

> - Paperclip manages work performed by AI agents.
> - Restart recovery must preserve each run's workspace and execution
owner.
> - A retained remote lease can refer to an archived sandbox.
> - The reattachment path returned that lease without making it ready.
> - A short project-authentication command then had to start the
sandbox.
> - This PR resumes and verifies the exact retained allocation before
preparation.
> - Recovery keeps the recorded workspace and preserves saved work if
resume fails.

## Linked Issues or Issue Description

**What happened?**

A native run resumed after a controller restart. Its saved sandbox had
been archived. Reattachment returned its lease without a provider
lifecycle resume. The next project-authentication probe tried to start
that sandbox with its 15-second command budget. The start failed before
the probe could inspect project settings. The heartbeat reported a
configuration conflict and stopped work.

**What did you expect to happen?**

Make the exact retained sandbox ready through the existing bounded
lifecycle operation. Verify its allocation before running short
preparation commands. Keep the validated lease and recorded working
directory. Never acquire a replacement in this path.

**Steps to reproduce**

Hold the provider's exact-lease resume pending and call remote
reattachment for an active ephemeral lease. Before this change,
acquisition returns immediately. The added tests hold acquisition until
resume completes and reject missing, changed or expired allocation
evidence. Provider compatibility cases retain the recorded workspace
when resume omits directory metadata or returns the allocation root.

**Deployment mode**

Native Runner in a sandbox with a retained workspace and managed AI
connection. The observed provider was Daytona.

Refs #18910, which restores selection of retries using current remote
workspace storage. This change repairs the next preparation boundary.
Related #17421 and #18559 change project-file inspection; they do not
make retained sandbox leases ready.

## What Changed

- Resume the exact validated sandbox lease before returning it to
preparation.
- Require the receipt to confirm the original provider allocation
without reporting expiration.
- Keep the original database lease and recorded directory. Resume
metadata can describe the allocation root.
- Pass the existing typed resume-directory input through the runtime
facade.
- Preserve retained E2B sandboxes when resume timeout setup or workspace
inspection fails. Keep new-allocation cleanup and explicit
release/destroy behavior.
- Preserve plugin-task reattachment and lease identity checks.
- Update the Stop-accounting fixture to confirm exact-allocation resume
before cleanup, with its budget, ownership and no-paid-work assertions
intact.
- Document the ordering and add readiness, receipt, failure and
ownership regressions.

## Verification

- Red: seven initial retained-lease cases failed before the readiness
change. Two receipt-compatibility cases and four E2B preservation cases
failed before their fixes.
- Green: all four runtime, workspace-preparation, heartbeat-recovery and
orchestrator suites pass: **442 tests**.
- The Stop-accounting CI failure reproduced locally. All **five**
integration cases pass with the exact-allocation resume fixture.
- All **22 E2B tests**, the standalone plugin typecheck and its build
pass.
- **13 Daytona safety cases** pass, including recorded directories,
allocation sentinel rejection, saved-file reads and no replacement
creation. The SDK boundary is simulated; the saved-file case uses real
files and a POSIX shell.
- These focused checks cover **482 passing cases**.
- Full recursive typecheck, final server typecheck and full workspace
build pass on macOS arm64 with Node 24.22.2 and PostgreSQL 18.1.
- Greptile is **5/5** on head
`32f915a475275680bcf2a2aa43ae0c72f4e4b1db`. Both original review threads
are resolved.
- All current-head CI gates pass. The check rollup has 54 successful
checks, three skipped checks and one neutral check.
- The separate Paperclip Review check remains queued. Its public
response says work is paused and the request is saved. The review task
is waiting for saved-file recovery, so no Paperclip assessment of this
head is claimed. The branch-required gate has passed.
- A read-only deployed trace showed an archived sandbox start fail at
the short probe's 15-second deadline. Actual provider reattachment with
this revision has not been exercised. Live recovery remains unverified
until rollout.

## Risks

Recovery now makes one bounded lifecycle call before short preparation
commands. A missing resume capability or a missing, changed or expired
allocation receipt holds recovery. The existing runtime and provider
retain allocation and sentinel verification. The path does not acquire,
release, replace or reseed the lease. Warm providers can return their
existing allocation. E2B retains the existing sandbox if resume setup
fails; explicit lifecycle cleanup still owns its removal. No background
polling, credential-policy change, migration, workflow change or
lockfile change is added.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, source inspection, tool use
and code execution. The exact backend snapshot and context window were
not exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-04 09:04:13 -06:00
DottaandPaperclip 84aa989bf9 Preserve native sessions through legacy workspace enrollment (#18943)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A native session retains its workspace and provider session across
turns.
> - An ephemeral web server must restore those files without its
predecessor's disk.
> - Legacy enrollment changes the workspace authority and can change
both workspace addresses.
> - The retained provider backup checks those addresses as part of its
identity.
> - This pull request preserves the exact addresses in a verified
migration receipt.
> - The benefit is that the next turn can use the original session after
enrollment.

## Linked Issues or Issue Description

Refs #18932, merged as `612d9931a6`. This pull request builds on that
all-Runner durability change. Related: #18918 fixes concurrent workspace
preservation locks. This change covers identity preservation during
offline enrollment.

## What Changed

- The native workspace migration owner records the selected session
bindings in the successful workspace migration operation.
- Runtime selection retains the original logical address only for the
exact agent, task and session.
- Remote workspace preparation restores the original provider address.
It refuses conflicting targets, forged receipts and occupied unowned
roots.
- The environment owner includes the migration operation in the reusable
lease fingerprint. Warm policy stays warm and receives a fresh
allocation through normal lease cleanup.
- Exact migration evidence permits the existing verified provider backup
to restore into that fresh warm allocation.
- Preserve verified provider allocation files through a normalized,
lease-bound digest receipt. Unknown files and altered markers remain
refused; interrupted publication is retryable.
- Retain legacy managed-instruction session fingerprints using the exact
imported content hash and recorded paths, preserving all other
configuration and credential checks.
- Migrate existing local skill archive/manifest objects through upload
claims, readback verification and receipt compare-and-swap; retain
original objects and IDs.
- Add explicit two-turn browser qualification for every released native
Daytona profile, with warm and per-turn variants. The production
maintenance entry runs offline. The old home is deleted only after a
verified export and successor readiness. The oracle checks provider
identity, native session, addresses, reference bytes, question and final
document.

## Verification

- Full repository typecheck passed at `b12ecd823e`. The production
session fix passed 198 session/credential tests. Full repository build
passed at `94bb917947`. The final Rust-only provider fix passed 57 ACPX
provider tests and the exact Linux release build.
- Workspace and transport checks passed: 117 transport/workspace tests,
six final real PostgreSQL migration boundary cases, 178 environment
tests, 539 Daytona tests (eight skips), and the native warm-backup
admission control.
- Local skill migration passed eight PostgreSQL/S3 tests (one existing
opt-in skip). Failed readback retains the source. Successful readback
still works after deletion of the old home.
- Latest Product E2E typecheck and 38 focused harness tests pass.
Earlier catalog/admission/report checks passed. Twelve unrelated
support-test failures reproduce on unchanged base `1a39dea9aa`; these
are not reported as passing.
- Live Codex per-turn at `2d5bb9a6d1` retained the native and provider
sessions, both workspace paths and exact reference bytes after deletion
of the old home. It completed the task and delivered its saved document
and original reference. Its original failure was an output-count grading
defect. The corrected grader passes all 21 checks against retained,
hashed evidence; the original result remains unchanged.
- Live OpenCode per-turn and Codex warm at harness `94bb917947` pass all
21 product checks. Their launcher cleanup failed because a read-only
diagnostic observer kept its disposable database open. The observer now
closes before ordinary database cleanup; a no-provider reproduction
confirms the fix. All owned sandboxes were independently verified
absent, child keys revoked and buckets removed. Original failed campaign
dispositions remain unchanged.
- Claude at `94bb917947` kept its sessions and reference bytes but
omitted the final clickable document link. That user-delivery failure
remains recorded. The fixture now explicitly asks for the link and
records migration evidence before checking navigation. The fresh Claude
attempt at `b12ecd823e` passed both strict migration checks: exact
provider/native session, both paths, original file bytes and
independently downloaded attachment. Its full browser workflow failed
again: it requested unnecessary confirmation, used the answer sentence
as the reference, and omitted the clickable document link. This is
retained as a failed user-workflow result, not a full pass.
- Earlier live failures exposed and reproduced the skill-object,
fresh-allocation marker and managed-instruction fingerprint bugs. The
fixes retain original identity checks. No provider session reset or
accounting bypass was used.
- The real Grok continuation at `b12ecd823e` failed before provider
execution because its immutable-instruction attach policy rejected
relocated registered file copies. A no-provider regression reproduced
the exact rejection. `b05ba3728a` selects the existing strict
authenticated asset-path policy for Grok; content digests, unknown
context fields, session/workspace identity and same-run mutations remain
enforced. The regression and all 57 ACPX provider controls pass. The
Grok browser migration now passes all 21 checks with the b05 Linux
Runner, including provider identity, exact reference bytes and final
user output; the launcher also reports successful cleanup.
- `24fabf1d81` applies the same policy to released ACPX Codex after a
second no-provider reproduction. All 58 ACPX provider controls and the
final Linux release build pass. Its live browser migration passes all 21
checks on the final binary, including exact original session/file
preservation and completed user output.
- The full native-session executor suite passes all 1,121 tests after
updating its existing lease database fixture for the migration-receipt
join. The prior CI failure was this unsupported test-double method; the
production guard still rejects ordinary unproven recovery. CI and
Greptile 5/5 passed at `ca1e0d5fe6`. The first CI attempt lost several
workers and had one rotated-recovery transport failure; its seven
descriptor cases passed locally, and the single unchanged retry passed.
The original failure remains recorded. Staging enrollment has not
started. Initial offline enrollment and subsequent rolling availability
are separate checks.

- The final review follow-up adds real runtime-selection PostgreSQL
cases for exact, different-session and absent receipts, plus
executor-level warm restoration that copies actual backup files and
verifies provider identity and the new lease stamp. Both complete
affected suites pass (1,172 tests), and full repository typecheck
passes. Removing either production migration connection independently
makes its new positive case fail; both source files were restored
byte-for-byte after these negative controls.

- Final commit `0971d373a3` passes all 63 checks (57 successes, five
expected skips, one neutral); Greptile is 5/5 and every review thread is
resolved.

## Risks

- An identity conflict blocks enrollment and keeps the original files.
It does not silently reset the session.
- A new migration generation retires a reusable lease through its
existing owner. The original sandbox must finish normal cleanup before
reuse proceeds.
- Initial enrollment has an offline step. The separate rolling suites
prove uninterrupted deploy behavior after enrollment.
- This adds no schema migration and changes no provider credentials or
accounting gate.

## Model Used

OpenAI Codex, GPT-6 family. The exact serving model build and context
window are not exposed in this session. The agent used reasoning,
terminal tools, code edits, PostgreSQL/filesystem tests and isolated
live browser qualification.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-04 04:43:56 -06:00
DottaandPaperclip 89e3c23062 fix(daytona): stream files when export copies cannot fit (#18712)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A native task must save its files before finalization can release
its writer.
> - Remote workspace capture can produce an archive that fits on the
sandbox disk.
> - Daytona then makes another copy before it downloads that archive.
> - The second copy can run out of space and leave completed work behind
a save hold.
> - This pull request uses the existing bounded stream when that copy
cannot fit.
> - The benefit is a saved archive without a second archive-sized
sandbox allocation.

## Linked Issues or Issue Description

Related: #18644 reduced capture scratch space. #18172 added the
disk-free directory export. #18705 preserves provider checkpoints on a
full disk and reports capture failures. This change handles the later
single-file download copy. It does not depend on #18705 and does not
replace its checkpoint fix.

**What happened?**

A native provider turn returned a successful result and captured its
workspace archive. The Daytona outbound file guard then failed with exit
43 and `No space left on device` while it copied the archive to a
reserved download file. Finalization and file-preservation retries could
not pass this boundary. The workspace kept its writer hold.

**Expected behavior**

Download an existing archive through a confined sandbox process when its
second copy cannot fit. Keep the source and all finalization holds until
the normal save commits.

**Steps to reproduce**

1. Capture a workspace archive in a Daytona sandbox.
2. Leave less free space than the archive needs for another copy.
3. Finalize the task. The outbound single-file guard fails before
download.

**Paperclip version or commit**

Reproduced against `6d72243d8a`. The incident used the native sandbox
runtime. This branch is based on current master.

**Deployment mode**

Cloud with native Paperclip Runner execution and remote workspace
storage.

## What Changed

- Remove partial outbound file copies after a failed snapshot command.
Only copy capacity failures select the stream path. Permission,
confinement, and transport errors still fail.
- Use the existing acknowledged terminal stream to read each file as the
sandbox user. Recheck canonical confinement and regular-file status. Use
a no-follow open. Keep the sandbox disk limit, host free-space reserve,
shared transfer deadline, and bounded teardown.
- Keep host files in private temporary files until the complete batch
has transfer receipts. Never pass a mutable live source to the
privileged download API.
- Add real-file and real-terminal regressions, a disposable Daytona
acceptance case, and workspace verification notes.

## Verification

- Red before repair: a real 5 MiB copy under a controlled file-size
limit failed with exit 43. The same test passes after repair and
compares every source and destination byte.
- The ten capacity cases pass. They cover producer failure after output,
source replacement, stream size limits, permission errors, timeout
teardown, cleanup, and a successful retry. A two-file case completes the
first stream and fails the second. Both old host files stay intact, both
source files survive, and retry saves both files. An isolated
early-publication mutation makes that test fail.
- The initial complete Daytona package suite passes: 537 tests passed, 8
skipped. After the test-only review follow-up, all 60 focused transfer,
stream, and diagnostic tests pass. The package typecheck also passes.
- Real Daytona acceptance passes: an owned Ubuntu 24.04 sandbox stops
and resumes, a copy to `/dev/full` reports ENOSPC, production export
saves the exact archive hash, and a second save after reconnect passes.
The test deletes only its disposable sandbox. This is provider-boundary
acceptance, not a UI journey.
- Assembled finalization acceptance passes with real Postgres and a live
Linux x86_64 Daytona sandbox. The production control-plane port accepts
one fixture result. The production workspace finalizer encounters the
failed copy, saves the files, and commits the task as `done`. A new
database client reads the committed result and exact file hash. The
workspace has no writer or running save, and there is still one run and
one result. No model turn runs. The target resolver is bound to the live
provider fixture. Object storage uses the production encryption and
local-disk service with an S3 admission label; this does not prove an S3
network transfer or a UI journey.
- The live tests used the fixed transfer source with SHA-256
`857b3d8c9ad75170707d95277c17c051ce4582fa79a9c38331d51d2800b36af5`. That
source is identical in this PR head.
- `pnpm -r typecheck` passes locally. The duplicate full local test run
passed all six chat shards before it was stopped during the server lane
in favor of completed current-head CI; a duplicate local build was
skipped. Full current-head CI passes, including typecheck, build,
general and serialized tests, Rust/Runner checks, all eight E2E shards,
and the `ci / e2e` and `ci / verify` aggregate gates.
- Greptile is 5/5 on the current commit with no unresolved review
threads.
- `node scripts/check-maps.mjs` and `git diff --check` pass. Map checks
prove structure only.

## Risks

- The fallback requires the Bash and GNU `dd`/`base64` tools already
required by the existing stream path.
- Terminal streaming is slower than the normal bulk download. It runs
only after a copy capacity failure.
- The workspace capture itself still needs sandbox space. This change
does not fix a full disk before an archive exists.
- The test injects a failed copy sink. It does not exhaust a shared host
disk. No original task was retried, and no deployment or merge is part
of this PR.
- No schema, API permission, writer ownership, or retry-budget change.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository inspection, code
editing, shell tools, browser diagnostics, and automated tests. The
exact serving variant and context-window size are not exposed in this
session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-03 06:11:41 -06:00