mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-13 07:12:00 +01:00
DottaandClaude Opus 5.5
120017e39d
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - When a cloud task finishes, its workspace is saved as an encrypted
snapshot and its sandbox is released. The board's Files view then reads
that saved copy
> - Saved Files access downloaded, decrypted and extracted the whole
archive for every request, and refused archives over 512 MiB with
`too_large`
> - A finished task in a Paperclip-sized repository saves a snapshot of
about 730 MiB, so its files could not be listed or opened at all
> - The archive format already records each file's size and SHA-256, and
the snapshot records the manifest digest, so single files can be read
and verified without the rest of the archive
> - This pull request reads the saved archive in place, one verified
file at a time
> - The benefit is that users can open any file of a large finished
workspace quickly, with bounded memory and no whole-archive transfer
## Linked Issues or Issue Description
**What happened?**
In restart-chaos waves on Daytona with a real Paperclip clone, the
committed workspace snapshots were 763,944,535 bytes. After the sandbox
was released, every saved Files read returned 422 `too_large` (506 reads
in total), because `createSavedWorkspaceLoader` refuses archives over
`maxArchiveBytes` (512 MiB). The board Files view uses the same loader,
so it fails the same way.
**Expected behavior**
A user can list and open any single file of a saved workspace of
Paperclip-repository size without the server extracting or buffering the
whole archive. Memory and time stay bounded per request, and integrity
is still verified.
**Steps to reproduce**
1. Run a cloud task whose workspace is a large repository, so the final
snapshot exceeds 512 MiB.
2. Let the task finish and its sandbox be released.
3. Open the task's Files view, or call `GET
/api/issues/:id/file-resources/content?workspace=execution&path=…`: it
returns 422 `too_large`.
**Paperclip version or commit**
`master` at f73ff4ace9.
## What Changed
- `openWorkspaceSnapshotRanges`
(`remote-workspace-snapshot-storage.ts`):
- range-reads the snapshot object's header;
- decrypts any plaintext byte range with AES-256-CTR, starting from the
AES-GCM counter block for that range;
- checks that the object size matches the wrapped key's plaintext size.
- New `saved-workspace-archive.ts`:
- range-reads and decrypts the archive header and manifest;
- verifies the manifest's SHA-256 against the snapshot's recorded
`manifestDigest`;
- applies the restore path's manifest checks (safe paths, parent
directories, file sizes and hashes, total size) and computes each file's
offset;
- keeps parsed manifests of recent snapshots within a 128 MiB budget.
- `remote-workspace-files.ts`: with the new `openSavedWorkspaceArchive`
option, saved stat, list, read and download use the verified manifest,
not the extracted-directory helper.
- Path normalization, denied paths, symlink and missing-path errors,
traversal, paging, truncation and text detection mirror the helper
exactly.
- A read or download decrypts only that file into a private temporary
file and verifies its SHA-256 before serving it.
- The changed-files view needs repository status, so it still uses the
bounded extraction. A saved copy beyond its limits now reports
`changed_unavailable`, a specific code, while its other views stay
available.
- `app.ts` and the support inspection route pass the new opener.
- `doc/SPEC-implementation.md` describes in-place saved reads and the
remaining changed-files limits.
## Verification
- Red before the fix, green after: `reads a saved workspace above the
extraction cap in place, one verified file at a time`. With extraction
only (master's Files service), the capped snapshot returns 422
`too_large`. With the archive reader, preview, list and a 256 KiB
download succeed through the ordinary routes. Every storage read is a
byte-range read, and no extraction directory is created.
- Parity: `answers saved stat and list exactly like the extracted
helper` compares archive-backed results with the extracted-helper path
on the same snapshot.
- Stat covers files, directories, nested and binary files, missing
paths, paths through a file, symlinks, denied names, and invalid paths.
- List covers all mode, a subdirectory, paging, queries, a scan limit, a
non-directory, and a denied directory.
- In recent mode, the entries match. Order is by path, because a saved
copy has no modification times; the extracted copy's mtimes were just
its extraction times.
- Integrity: `refuses a saved file whose stored bytes no longer match
its verified manifest`. Tampered stored bytes return 422 `file_changed`,
and no bytes are served.
- `keeps changed files on bounded extraction and names its limit…`: over
the cap, the changed view returns `changed_unavailable` and the other
views still work.
- The existing location-clearing preview test now checks for range reads
and an empty request temp directory, not for an extraction directory.
- Measured locally, with local-disk storage and a 749,006,840-byte
snapshot (88 × 8 MiB pack file, a 10 MiB file, 2,000 small files):
| path | time |
| --- | --- |
| Old: download, decrypt and extract all, once per request | 2,338 ms,
plus a full 749 MB transfer from S3 |
| New, cold: open (header and manifest range reads) | 7 ms; open and
read a small file: 13 ms |
| New, warm (cached manifest) | open 2 ms; open and read: 6 ms |
| New: list 100 entries and download a 10 MiB file | 23 ms |
- `remote-workspace-file-resources`, `remote-workspaces.integration`,
`remote-workspace-save-concurrency`, and
`remote-workspace-snapshot-retention` pass (212 tests).
- `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` in `server` passed.
- Not yet verified: the Daytona R1 wave's file checks after release. The
chaos harness owner will run them on a build with this change.
## Risks
- Range decryption bypasses GCM's whole-object tag. Integrity instead
comes from the manifest's SHA-256 matching the digest recorded in the
database, and from each file's SHA-256 in that manifest. A file is
verified before it is served. Text detection on stat samples the first 4
KiB without verifying the whole file. It only sets a display hint, and
content is always verified before it is shown.
- Saved files carry no modification times, so the recent view of a saved
copy is ordered by path, where it was previously in effectively random
extraction order.
- The changed-files view of a saved copy over 512 MiB still reports
`changed_unavailable`, because it needs a full extraction to run `git
status`.
- No schema or API change.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, local benchmarks).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>