Commit Graph
2 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 8a295f944d fix(runner): fail fast when an artifact storage read stalls (#22210)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents use the `download_artifact` tool to copy a saved API artifact
into their workspace
> - The server reads the artifact bytes from storage before it installs
the file
> - This storage read has no deadline. A stalled storage response or
stream keeps the tool call open
> - The provider MCP client stops the call after about five minutes with
the error "The operation timed out."
> - This pull request puts a 60-second deadline on the storage read and
returns a clear, retryable error
> - The benefit is that the agent gets a fast, clear result and can
retry, and does not wait for five minutes

## Linked Issues or Issue Description

**What happened?**
Some `download_artifact` calls stopped for four to five minutes. Then
the agent got only "The operation timed out." In one case, a retry of
the same 1 MB artifact completed in about 20 seconds.

**Expected behavior**
A stalled storage read must fail quickly. The error must tell the agent
that no file was written and that it can retry the same download.

**Steps to reproduce**
1. Give `PaperclipRunnerToolAuthority` a storage service whose
`getObject` never resolves, or whose stream sends one chunk and then
stops.
2. Call `download_artifact` for an artifact in that storage.
3. The call does not settle. Nothing aborts the storage request.

**Root cause**
`#readApiFile` called `storage.getObject(...)` without a `signal` and
read the stream with no deadline. Other storage reads in the runner, for
example chat attachment reads, already have a deadline.

## What Changed

-
`server/src/services/native-runtime/paperclip-runner-tool-authority.ts`:
the artifact branch of `#readApiFile` now uses a 60-second
`AbortSignal.timeout`. It sends the signal to `storage.getObject`. It
destroys the stream when the deadline expires. Every storage stream gets
an error listener before it can be destroyed, so a late reply cannot
raise an uncaught `AbortError`. It returns HTTP 504 with code
`artifact_storage_timeout` and the message "Artifact storage read did
not finish within 60 seconds. No file was written; retry the same
download."
- A new optional binding field, `storageReadTimeoutMs`, sets the
deadline for tests. The default is 60 seconds. The production callers do
not change.
-
`server/src/services/native-runtime/artifact-transfer.integration.test.ts`:
a new regression test covers a storage response that never arrives and a
stream that stops after one chunk.

## Verification

- Failing before the first fix: `cd server && npx vitest run
src/services/native-runtime/artifact-transfer.integration.test.ts -t
"stalls"` failed with `Error: Test timed out in 15000ms.` The
`download_artifact` call never settled.
- Failing before the late-reply fix: the same test now also covers a
storage reply that arrives after the deadline, with a real
`createReadStream` that gets the aborted signal. It failed with an
uncaught `AbortError` (`ABORT_ERR`).
- Passing after both fixes: the same command passes. Each stalled call
fails in less than 5 seconds with the retryable error. All storage
signals are aborted. No uncaught error occurs. The workspace has no
files.
- `npx vitest run
src/services/native-runtime/artifact-transfer.integration.test.ts
src/services/native-runtime/paperclip-runner-tool-authority.test.ts`: 49
passed.
- `npx vitest run
src/services/native-runtime/runner-api-artifacts.integration.test.ts
src/services/native-runtime/runner-api-artifacts.test.ts
src/services/native-runtime/issue-document-batch.integration.test.ts`:
38 passed (first revision).
- `pnpm run typecheck` (server): passed.
- `pnpm verify` selects a very large set of files for this module. I did
not run the full set locally. CI covers it.

## Risks

- Low risk. The deadline applies only to the artifact read from storage,
and artifacts are at most 10 MiB. A healthy read takes seconds.
- A very slow but working storage provider that needs more than 60
seconds for 10 MiB now gets a retryable error.
- The error occurs before any file write, so no partial file stays in
the workspace.
- This change does not fix a slow storage provider. It makes the failure
fast and clear.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution
in a sandboxed workspace.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 15:14:46 -06:00
06bc8db75c fix(runner): install verified API artifacts in execution workspaces (#18494)
## Thinking Path

> - Paperclip manages AI agents and their work.
> - Native agents use authorized API tools to read saved results.
> - Copying a large response as model-generated text can lose bytes.
> - The execution workspace needs an exact copy of the saved artifact.
> - This change copies the authorized bytes without shell API
credentials.
> - The runtime verifies size and SHA-256 before it reports success.
> - A live evaluation checks capture origin, installation and delivered
bytes.

## Linked Issues or Issue Description

**What happened?**

A native agent lost one byte while copying a 24576-byte API text page.
Existing capture and paging controls passed. The runtime had no verified
saved-artifact-to-workspace copy tool. A separate historical screening
refusal does not establish capture corruption.

**Expected behavior**

An authorized native run can install the complete saved artifact. The
stored and installed bytes must match in size and SHA-256. A conflicting
file must remain unchanged. A lost publication receipt must not cause an
unsafe repeated write.

**Steps to reproduce**

Run
`server/src/services/native-runtime/artifact-transfer.integration.test.ts`
on baseline `c46b8f35add9255fc71be294d27f5a7865bcb47f`. Its capability
assertion fails because `download_artifact` is absent. Apply this PR and
run the same regression with the filesystem and API controls. It
verifies 388131 synthetic bytes through the authority and installation
boundary.

Refs #18429. Related #17393, #17086 and #17285 cover paging, transfer
foundations and remote output reads. This PR adds the inbound
installation contract.

## What Changed

- Add `download_artifact` for supported standard-mode native runs, with
a 10 MiB limit.
- Reuse current company, task, run, asset-content and responsible-user
access checks. Check authority again before publication.
- Bind installation to the current execution lease. Stage through
confined Linux descriptors and publish with an atomic create-only link.
Keep API credentials out of execution commands.
- Verify bytes at storage and execution boundaries. Reject unsafe paths,
symlinks, different existing files, truncation and corruption. Reconcile
an unknown publication through verification only; an identical explicit
retry remains safe.
- Add API documentation, access-map entries and content-free audit
records.
- Preserve only the pinned Daytona SDK numeric HTTP status when failed
creation cleanup is unconfirmed. Keep provider exception messages and
headers private.
- Keep only the finished runner binary and portable provider pack in
their builder layers. Remove Rust targets/downloads and the Node install
workspace/store before export. Report tree sizes.
- Add one explicit Codex/Daytona Product E2E cell. Its independent
oracle checks the server capture audit, redacted tool receipts, stored
source, delivered attachment and browser link after reload. Uploaded
substitutes and one-byte loss fail.

## Verification

- Merge update at `6766b46553cab5b9787ade87a002eccaf50e68fc`: merge
current master and preserve both explicit suites in the catalog. Focused
catalog, image and oracle checks: 122 pass. Filesystem and authority
integration: 20 pass. Maps, module boundaries and whitespace checks
pass. New-head CI and Greptile are running. Full local build and
typecheck remain unavailable because cargo is absent.

- Behavioral red/green: the absent-tool regression failed before
implementation and passed afterward. The initial missing-package attempt
is recorded as setup failure.
- Linux filesystem and real PostgreSQL/authority integration: 19
filesystem tests and 1 authority test pass. Existing API authority,
capture and paging controls: 70 pass, 2 skipped. Provider transport in
these integration controls is simulated.
- Qualification calibration: 118 focused tests pass. The real
persisted-event redaction case and a byte-identical uploaded substitute
each failed before their oracle repair. The unchanged controls then
passed. Harness typecheck passes.
- Full recursive typecheck and build pass locally on the integrated
production source at `086f7a2901`. Later commits change only
qualification tests and documentation.
- Structural map/module checks and whitespace checks pass.
- Full local `pnpm test:run` was attempted by the implementation worker
and stopped under its 8 GiB memory limit, with no full-suite verdict.
Current-head CI must supply the full test result.
- The complete local Product E2E support suite has 13 failures. The
unchanged pre-qualification baseline reproduces all 13: host PostgreSQL
semaphore capacity, stale timeout/profile/workflow inventory
expectations, and expired fixed-date allowance fixtures. This is not a
full local suite pass.
- Earlier CI at `086f7a2901` failed a browser test during cleanup
cancellation with a database lock timeout. It remains recorded. All
browser shards passed on both later fully checked heads.
- Two qualification campaigns were canceled during setup before provider
execution while correcting the oracle. They provide no live product
verdict.
- Previous-head CI at `28fa001457f59df1c300c016dc569049e72d0061`: 53
checks pass, 3 intentional selection skips; no failed or pending checks.
- Real image/export regression: campaign 41272351780 compiled the Linux
runner and passed root/unprivileged CLI and transport checks, then
failed BuildKit export with `no space left on device` (exit 102,
filesystem 99.99%). No provider cell ran. Removing the Rust build trees
alone was insufficient: campaign 41273604815 also failed during
provider-pack export with exit 102. Commit `5bbb549c54` also
consolidates dependency installation and pack construction so the
disposable workspace/store can be removed in the same layer. All 122
image/oracle/catalog checks pass. Image and remote-pack publication then
passed in campaign 41275196741 after cleanup changes plus a scoped
60-to-120 GiB expansion of only its delete-on-termination builder disk.
The VM terminated and the expanded volume was deleted. This does not
establish that cleanup alone makes the default 60 GiB builder
sufficient.
- The Daytona admission probe then failed before agent creation, with
unconfirmed provider cleanup. The queued continuity case was canceled
because it shares this prerequisite. No artifact-transfer or continuity
success is claimed.
- Status-diagnostic regression: the original generic wrapper failed the
HTTP-status assertion. After the repair, the full plugin suite passes
501 tests, with 7 skips; its standalone TypeScript check passes.
Controls reject invalid status values, avoid accessors, preserve the
original create status and exclude credential canaries.
- Previous-head Greptile: 5/5; its summary names
`28fa001457f59df1c300c016dc569049e72d0061`. All review threads are
resolved.
- Live artifact transfer and three-turn warm continuity: NOT VERIFIED.
The diagnostic case in
https://github.com/paperclipai/paperclip/actions/runs/41277598289
confirmed Daytona HTTP 401 before agent creation. The configured
`DAYTONA_API_KEY` in the existing `runner-e2e-paid` environment needs
verification or refresh. The queued continuity control was not run after
its shared admission prerequisite failed. No further provider attempt is
planned until the credential changes. Unconfirmed provider cleanup
remains recorded.

## Risks

- Source-provenance changes can repeat the provider dependency install.
The combined build layer trades install-cache granularity for lower
retained disk use.
- Linux descriptor semantics and Node are required on the execution
host. Unsupported providers and restricted runtime profiles fail closed.
- Storage reads buffer at most 10 MiB. A provider outage can leave a
reserved staging file. Partial bytes never become the destination.
- A lost publication receipt can leave a complete file with an unknown
result. Recovery verifies the existing file without installation.
Conflicts are never overwritten.
- Artifact contents remain untrusted. The copy does not bypass screening
or authorize downstream execution.
- Live qualification uses the governed Codex API profile. It does not
establish every model, subscription profile, platform or historical
failure cause.

## Model Used

OpenAI GPT-6 in Codex assisted with diagnosis, code, tests and
qualification. The desktop runtime does not expose a more specific model
deployment ID or context-window size. The model used reasoning, source
inspection, tools and code execution. The native implementation agent is
configured with OpenAI `gpt-6.1-sol`.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com>
Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-02 08:27:09 -06:00