Commit Graph
8 Commits
Author SHA1 Message Date
Dotta 1b7fe33432 fix(workspaces): preserve import verification across process restarts (#20311)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Online import preserves old local files before the web server can
use a disposable filesystem.
> - A parent import must authenticate saved child archives before it can
omit their directories.
> - The operation journal preserves import identity, but completed child
checks existed only in process memory.
> - A container restart therefore repeated downloads, decryption and
restoration that had already succeeded.
> - This pull request saves a versioned verification receipt on each
immutable child snapshot.
> - A replacement process can reuse those checks while it reacquires
locks and verifies current local files.

## Linked Issues or Issue Description

Refs #19854 for verification reuse within a live source fence. Refs
#19873 for cohort import. Refs #20298 for the separate remote-hold
admission fix. Searched related workspace verification and restart PRs;
none provides durable child verification.

**What happened?**

Replacing a server container interrupts a long online import. The
original operation and snapshot survive, but a later invocation
downloads, decrypts and restores completed child archives again. The
current cache ends with the source-lock owner.

**Expected behavior**

Completed immutable checks survive process and scratch loss. Resume
keeps the same operation and snapshot, reuses only correctly bound
verification, and publishes once. Current local files and Git state must
still be checked under fresh source locks.

**Steps to reproduce**

1. Import two nested child workspaces, then start the parent import.
2. Kill the importer after a child verification finishes. Remove its
scratch directory.
3. Resume in a new process. Before this fix, the completed child is
downloaded and restored again.

**Paperclip version or commit**

The regression fails on the previous migration verifier. The final
branch includes master at `189fefb99183953be37f0d1084275fbb0f35d6e2`.

**Deployment mode**

Managed Linux server with PostgreSQL and explicit S3 storage
configuration.

## What Changed

- Add optional opaque storage namespace identity. S3 binds it to the
explicit endpoint, region, bucket, normalized prefix and addressing
style. Implicit endpoints retain full verification.
- Save a versioned child-snapshot receipt only after canonical crypto,
manifest and file restoration succeed and a second HEAD confirms the
same strong ETag and size.
- Bind reuse to the complete snapshot reference, key material, child
lineage, source operation and storage namespace. Require a fresh strong
HEAD and canonical validation that the current secret provider can
unwrap the saved snapshot key.
- Keep current directory/Git capture and source-owner checks on every
invocation. Read-only verifier callers do not write checkpoints.
- Merge checkpoint metadata under a short row lock without changing
parent journal hashes, snapshot retention or update timestamps. Bound
reads and writes. Lock timeouts, statement timeouts, deadlocks and
serialization conflicts are safe cache misses. Other errors remain
fatal.
- Extend real PostgreSQL/Git/archive tests with process death, scratch
removal, resume, changed bindings and single publication.

## Verification

- Baseline RED: two import invocations read each child twice. With
durable receipts, each child is read once across both invocations.
- Real process regression passed on the final rebased head: SIGKILL the
owned importer after one checkpoint, delete all scratch, and resume in a
new Node process. It retains the same operation and snapshot, reads each
child once across both processes, and records one publication.
- 53 focused cases passed in 104.86 seconds on the final production
bytes before rebase. They cover existing nested/cohort behavior,
namespace identity, changed source/reference/key/lineage/ETag, unknown
proof versions, weak validators and bounded row contention.
- After clean rebase and stronger scratch/snapshot assertions, the real
process case passed again in 18.54 seconds. Server TypeScript, module
boundaries and diff checks passed on final head
`9da68ee7ce4a778201ef85e21a11c49eeea7bc16`. Required CI supplies
final-head broad coverage.
- The initial restricted-network dependency setup failed; unchanged
preflight passed with normal network access. Initial TypeScript failures
in test pipe assertions were corrected. No failed attempt is counted as
a pass.
- Review found a missing current-master-key check. A real restart
regression with all child checkpoints complete reproduced unsafe
publication after switching to a different valid master key. Follow-up
`7ea92c2abd9ffdb2e7dd81cecf62c8fc7a6e74a5` delegates to the canonical
snapshot-key owner before both durable and in-memory cache hits.
- The fixed process regressions passed 2/2. Ten key-authority and
existing identity/source guard cases passed. The wrong-key process now
refuses with zero additional child payload GETs, no publication, and
unchanged original journal/snapshot rows. Server typecheck, module
boundaries and diff checks passed on the follow-up.
- Review also found that transient database failures could make the
optional checkpoint cache stop a valid import. Read and save now fall
back only for SQLSTATE `55P03`, `57014`, `40P01`, and `40001`.
Connection loss, identity changes, crypto failures, programming errors
and cancellation remain fatal. Reads use a read-only transaction with
one-second lock and eight-second statement limits.
- The checkpoint database-fault regression failed two cases before the
fix. All four recoverable import cases then passed. The final selected
run passed 21 cases, including fatal-error and cancellation controls,
real row contention, both process-restart cases and key/identity guards.
Server typecheck and boundaries passed. Fault injection is limited to
the checkpoint database boundary; the importer, PostgreSQL, Git,
archives, storage and publication remain real.
- Latest master was merged without rewriting published history. Six real
import/process-restart cases passed again on final head
`8c8e4213a680ebb5c8d6864416bc3a4f271e1e76` in 38.03 seconds. [Required
CI](https://github.com/paperclipai/paperclip/actions/runs/42658248222)
finished with 54 successful checks and four intentional skips; Snyk
passed. [Fresh
review](https://github.com/paperclipai/paperclip/pull/20311#issuecomment-6745354381)
is 5/5 with all four threads addressed and resolved. Hosted verification
after normal release delivery remains pending.

## Risks

- A stale receipt could skip authentication of a different object.
Complete reference/lineage/namespace binding, fresh strong HEAD and
row-locked save validation prevent that reuse. Missing or unsupported
hints trigger full verification.
- Mutable local files can change between attempts. Their source locks
and fresh captures remain required; a checkpoint never certifies current
local bytes.
- Checkpoint persistence can contend with other snapshot owners. It has
one-second lock and eight-second statement limits; the four classified
transient SQL errors retain the normal fully verified path. Other
database errors stop the import so lost source ownership cannot be
hidden.
- No schema, archive format, public API or saved-run identity change.
Older snapshots without receipts remain readable and get full
verification. Existing data and recovery obligations stay intact.

## Model Used

OpenAI Codex, GPT-6.1 Sol (`gpt-6.1-sol`), with reasoning, repository
tools and code execution. Context-window size is not exposed by the
runtime.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-11-09 11:03:48 -06:00
Devin FoleyandPaperclip ee556a266d perf(storage): upload workspace snapshots in parallel verified parts (#19917)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - With remote (sandbox) execution, every agent turn ends by saving the
workspace: capture an archive, encrypt it, upload it to object storage,
and verify it.
> - On a busy S3-backed instance, the upload step is the slowest part of
a save. A 1.5 GB workspace can hold a run open for minutes after the
agent finished.
> - The upload step reads the archive several times, uploads through one
stream with 4 parts in flight, and then downloads the whole object again
to verify it.
> - Parts that use the stream path share one process-wide 32 MiB buffer
window, so saves that run at the same time slow each other down.
> - This pull request uploads parts in parallel straight from the
encrypted file, lets the backend verify every part with Content-MD5, and
replaces the full read-back with a size check for those verified
uploads.
> - The benefit is faster saves, a smaller run-finalization window, and
no interference between concurrent saves.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Remote workspace snapshot upload (`uploadWorkspaceSnapshot`) to
S3-compatible storage.

**Subsystem affected**
Server storage (`server/src/storage/*`) and remote workspace snapshot
storage.

**Current behavior**
For one save, the upload step:
1. hashes the archive;
2. reads the archive again to encrypt it into a temporary file;
3. hashes the encrypted file;
4. uploads it as one stream with 4 parts in flight;
5. downloads the whole object to verify it.

On one hosted instance over 7 days (724 saves, median 731 MB), the
upload step ran at a median of 6.5 MB/s. With no other save running it
reached 11–14 MB/s. With six or more saves at the same time it fell to
4.4 MB/s. From the same region and with the same part settings, a plain
client reached 31 MB/s to the same kind of bucket.

**Proposed behavior**
Upload parts in parallel from byte ranges of the encrypted file, with
Content-MD5 on each part. Confirm only the stored size after a verified
upload. Digest the archive in the same pass that encrypts it.

**Reason and benefit**
Saves finish faster and hold their run open for less time. Concurrent
saves no longer serialize on the shared buffer window. A transient part
failure is retried from disk instead of failing the whole save.

**Breaking changes**
None. `StorageProvider.putObject` may now return a receipt. Providers
that return nothing keep the current behavior, including the full
read-back.

Related: none found.

## What Changed

- `storage/types.ts`, `storage/service.ts`: an upload can name its
`sourceFile`, a private file whose bytes are exactly the body.
`putObject` may return `{ integrity: "server_verified_md5" }`, and
`putFile` passes it through as `PutFileResult.integrity`.
- `storage/s3-multipart.ts`: new `putS3MultipartFile`.
- Each part reads its own byte range of the file. Up to 8 parts upload
at once, with no shared buffer window.
  - Each part carries the Content-MD5 of its range.
  - A failed part is resent from disk, up to 3 attempts.
- The first part that runs out of attempts stops its siblings. The
multipart upload is aborted only after every part settles.
- A source file whose size differs from the declared length is refused
before any request.
- `storage/s3-provider.ts`:
- A `sourceFile` larger than 16 MiB uses `putS3MultipartFile`. A smaller
one uses one PUT with Content-MD5.
- These requests use a second client with `requestChecksumCalculation:
"WHEN_REQUIRED"`, so bodies are plain and the backend verifies
Content-MD5 directly. The default client wraps bodies in aws-chunked
framing with a CRC32 trailer.
  - Other uploads are unchanged.
- `services/remote-workspace-snapshot-storage.ts`:
- Encryption digests the plaintext in the same read. An unchanged file
identity (dev, inode, size, mtime, ctime) across that read replaces the
separate digest pass. A change still fails with `archive_changed` before
any upload.
- The encrypted file is still digested from disk, so the recorded hash
covers the exact bytes that the parts read.
- The upload passes `sourceFile`. When the provider reports
`server_verified_md5`, a HEAD confirms the stored size instead of
downloading the object.
  - The full read-back stays for:
    - other providers (local disk);
    - recovery of earlier attempts;
    - the caller that keeps a verified archive file.

## Verification

- `pnpm --filter @paperclipai/server typecheck`: exit 0.
- `vitest run src/__tests__/storage-s3-multipart.test.ts
src/services/remote-workspace-snapshot-storage.test.ts`: 59 passed. New
tests:
- every part carries the Content-MD5 of its range, and 8 parts are in
flight;
  - a failed part is resent and the upload completes;
- a part that keeps failing aborts the upload only after in-flight
siblings settle;
  - a size mismatch is refused before any request;
- a small file goes as one PUT with Content-MD5, through a real HTTP
server;
- the read-back is skipped only for a server-verified upload, and the
snapshot still restores;
  - the read-back stays when the caller keeps the read-back file;
  - a stored size mismatch is rejected;
  - an archive that changes during encryption is rejected before upload.
- With the source change reverted, the three new snapshot tests fail.
- Real S3-compatible bucket (a Railway bucket, run from a workstation):
- A 3 MiB single PUT stored exactly: HEAD size, ETag and read-back
SHA-256 all matched.
  - A 96 MiB, 13-part upload stored exactly, with the same checks.
  - A part with a wrong Content-MD5 was rejected with `BadDigest`.
- The same backend stored a wrong `x-amz-checksum-sha256` without
checking it. That is why this change uses Content-MD5.
- Measured from a container in the app's region, 1 GiB with the app's
settings (8 MiB parts, 4 in flight), to the same kind of bucket: 31
MB/s. With 16 in flight: 69 MB/s.
- 77 storage-related suites: the only failures are the same 19 that fail
on unmodified `master` in this checkout. They are in 4 files: missing
bundled connector skills and an unbuilt local runner daemon.

## Risks

- Integrity now depends on the backend enforcing Content-MD5 for file
uploads. AWS S3 and the tested S3-compatible backend enforce it. A
backend that ignores Content-MD5 would accept corrupted parts without a
read-back. A later restore still fails closed: AES-GCM authentication
and the plaintext SHA-256 are checked on load.
- Up to 8 parallel part requests per save, instead of 4 shared ones.
Memory per part is a 64 KiB stream buffer.
- Archive-change detection now uses file identity instead of a second
digest. The archive lives in a private temporary directory that the
capture step creates.
- Medium risk overall: the change is limited to file-sourced S3 uploads
and the snapshot verify step.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, test runs, S3 API probes). AI-assisted
investigation, change and tests, with human direction.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
docs describe this path)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-07 14:04:49 -08:00
DottaandPaperclip 5e4cc01073 feat(workspaces): keep remote files off the webserver disk (#18403)
Keep native sandbox workspace files independent of the webserver disk. Use encrypted saved versions and Postgres ownership so a replacement server can recover files and continue work. Enable the path by default for new eligible workspaces when durable storage is configured.

Preserve explicit opt-outs and existing host workspaces. Include safe snapshot repair, Files access, import, migration and retention. Bound saved-Files resources, preserve imported Git recovery, reject unfinished Git operations, publish immutable concurrent captures, and verify permanent object-version deletion.

Validated 471 targeted local controls, build/typechecks, all 40 CI partitions, fresh Greptile 5/5 and security review. The PR records original failed attempts and scoped retries.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-31 14:54:01 -05:00
DottaandPaperclip f1911a4c53 perf(skills): move repository packages out of Postgres (#18082)
<!-- entire-trail-link-start -->
https://entire.com/gh/paperclipai/-/trails/205/changes/paperclip/205
<!-- entire-trail-link-end -->

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents receive selected company skills before each turn.
> - Imported skills can need files from the whole source repository.
> - The current path stores those files in Postgres and reads large JSON
payloads during preparation.
> - This pull request stores verified packages in the configured file or
object store and keeps preparation queries small.
> - It also converts old rows automatically and restores missing files
from the exact saved commit.
> - The result reduces database traffic while preserving skill versions
and repository dependencies.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Company skill import, version storage, and preparation before agent
turns.

**Current behavior**

Repository snapshot bodies live in JSONB. Skill preparation can read
those bodies again even when the files already exist on disk. Historical
packages and native assets also need a bounded cleanup path.

**Proposed behavior**

Store package archives and manifests through the existing storage
service. Keep version identities, receipts, hashes, and recovery
metadata in Postgres. Convert legacy rows after the stored package
passes read-back verification. A missing package can use legacy bytes or
its exact Git commit under current repository authorization. No separate
data-migration command is required.

**Reason and benefit**

The warm path sends no repository bodies through SQL and reads no object
bytes. Cleanup runs through the existing deadline coordinator. Current
versions, active runs, explicit pins, and authored content retain their
protection.

**Breaking changes**

The normal application schema upgrade is required. Single-version and
skill-file responses keep their existing shape. Version lists,
current-version summaries, and test-run version summaries now contain
file names and kinds without file bodies. Diff and restore fetch
selected versions through the existing single-version endpoint. Older
server builds must not write after conversion unless they support the
new package format. The storage override and rollback limits are
documented in `doc/skill-package-storage.md`.

**Additional context**

Related work: #15731 preserves repository dependencies; #17923 changes
version-cache scheduling; #18080 changes delivery to remote Codex homes.
This change owns package persistence, acquisition, and expiry. It does
not replace those delivery or scheduler changes.

## What Changed

- Add company-scoped archive receipts, exact-commit recovery recipes,
migration deadlines, and a durable object-deletion queue.
- Add bounded archive upload, read-back verification, extraction,
concurrent request coalescing, and verified local caches.
- Convert old snapshots and versions automatically. Keep legacy bytes
when verification fails.
- Recover missing files through current source authorization and the
original commit. Validate repository identity, file hashes, modes, and
the aggregate hash.
- Expire unused recoverable Git packages and native assets after 30
days. Preserve active pins and the durable copy of authored or unproven
content.
- Keep warm preparation queries metadata-only. Bound file digest caches
and avoid repeated body hashing when file identities are unchanged.
- Extend the existing storage contract with cancellation. Register
package work with the existing coordinator. Optional conversion retries
retain the original bytes and may wait for ingress without blocking
sleep.
- Add regression tests, operating guidance, a deep trace, benchmark
receipts, and an optional maintenance command.

## Verification

### Final master integration and review repair — October 29, 2026

Current head: `a6065424249e763dfeac37161f1b52a9f25bd9b7`. Commit
`f8bbdd3dbd` resolves the conflicts with master `76a796e964`; the final
two commits repair the Apex deletion findings below.

- Keep both package-maintenance and version-retention notifications in
their write transactions.
- Keep the shared run-pin predicate and master's controller-lease
deadline metadata. Package expiry reads the updated protected-version
result.
- Preserve every master migration entry. Generate migration **0415
(`0415_zippy_zemo.sql`)** after master's 0414. Its skill SQL is
byte-for-byte unchanged from the previously tested idempotent migration.
Retain only the five newest schema snapshots. Migration numbering and
safety checks pass.
- Repair failed file/folder deletion: preserve a complete temporary
subtree until transaction commit, restore on upload or SQL failure, and
restore callback failures before releasing the mutation lock. Both
PostgreSQL regressions reproduced missing files before the fix and now
pass. They cover binary data, executable permissions, empty folders,
unlisted files, unchanged database/version rows, successful retry after
upload-claim expiry, and historical reads.
- Reconcile lost commit replies under the same advisory and row locks.
Check the saved version and source identity before restoring. Keep
committed deletions intact and preserve backups when the database
outcome is unknown or a newer edit exists. Four real-PostgreSQL outcome
tests pass; the prior code failed three of them.
- Full workspace typecheck and production build pass on the final
commit. All **204** focused PostgreSQL package/cache/retention/authoring
and filesystem-boundary tests pass, with none skipped.
- Apex review **18 passes 5/5**, with zero new comments on the exact
final head. All **34** review conversations are resolved.
- [Final-head
CI](https://github.com/paperclipai/paperclip/actions/runs/40914116508)
completes successfully on attempt 2. The head has **55 successful
checks**, two neutral reports, two skipped checks, and a successful Snyk
status. No checks are pending or failing. All eight browser shards pass.
- Runner Vitest 2/2 timed out once in the unchanged ACPX
controller-replacement fixture. The same job passed on the preceding
commit. One unchanged CI retry passes all five ACPX recovery cases,
**2,253** Runner package cases and **126** native server integration
cases; their standard one and five skips remain. The initial timeout's
root cause is not established. Local diagnosis also hit the fixture's
eight-second session.open timeout with an authenticated Runner; no local
Runner pass is claimed.
- Both full local `pnpm test:run` attempts were stopped when Apex
identified deletion defects and source repair became necessary; neither
is claimed as a passing full local run. The final **204** focused tests,
full workspace typecheck, and production build pass. Full-suite
acceptance uses the final-head CI jobs. The superseded merge-head CI
included runners terminated by shutdown signals; each source repair
started a new full run.

- Auto-merge is enabled for squash merge. The final head is
conflict-free and all checks pass. GitHub's active master ruleset still
requires a code-owner approval for the dependency files. The existing
review requests are with `devinfoley`, `nickyleach`, and `forgottendev`;
no policy bypass was used.
- The earlier hosted stress and full native receipts below remain tied
to `a2e597d800`; they are not relabeled as a deployment of this final
integration. The earlier local PostgreSQL startup failure remains
recorded below; PostgreSQL ran successfully for this integration
verification.

### Full native skill execution — October 29, 2026

The full native-agent acceptance gap is closed on candidate
`a2e597d800eb989d565780e7aa947d5368676444`. Two qualifying native turns
used OpenAI API / `gpt-5.4-mini` in real Daytona sandboxes, on the same
disposable hosted TEST tenant as the stress test. A follow-up on the
same task consumed a newly saved skill version.

The task text asked for the assigned skill workflow. It did not contain
the proof nonce or expected receipt. The assigned authored skill
contained a Python verifier and a random package-local nonce. The agent
read the actual installed skill files and executed that verifier against
the imported, automatically converted 1,360-file repository package. It
checked the repository file count and reference/binary bytes, and
directly executed the repository's executable probe. Independent
verification matched the emitted JSON to the nonce and expected byte
hashes. The agent persisted the same receipt as its task response;
native finalization committed, workspace finalization succeeded, and
task state became Done.

| Qualifying turn | End-to-end run time | Recorded skills.prepare |
Repository files | Binary hex | Shell receipt SHA-256 |
|---|---:|---:|---:|---|---|
| Initial task | 47.881 s | 1822 ms | 1360 | `00ff07800a` |
`a1deb30dc095f2306c7aa9cc5ec396da12c68961cf5b809f4d7c20f7fc68493b` |
| Corrected follow-up, new nonce/version | 37.974 s | 377 ms | 1360 |
`00ff07800a` |
`dbcdc33fa1376b4edc136feae1cdf770560a73292467e93619efdda20e4282a7` |

Both successful shell receipts reported `probe: STRESS18082` and
`assetSha256:
e8ea7a2f15f7bb3b332a46adbef848465884c8996595aa234748befd544d1f3a`. Their
proof nonces, selected version IDs and receipt hashes differed, and both
ran on the same task. These are observed timings, not latency SLOs; they
include real provider work. The service stress-test timings remain a
separate measurement boundary.

Earlier failures remain in the evidence:

- The original five-minute Daytona allocation timeout never reached
model execution. After normal owned-stack restore, its durable cleanup
record was `expired` / `success`. The cause of that original allocation
timeout remains unknown; a later success establishes recovery only.
- Initial continuation setup used unsupported generic agent-update
fields and returned HTTP 409 before dispatch. Setup was corrected to use
the supported skill-assignment and versioned instruction commands.
- The intervening completed native turn read repository bytes and ran
the probe, but omitted the verifier JSON under the older test agent's
overly-specific instruction file. Its strict receipt validator failed;
it is not counted as a qualifying turn. Correcting the test agent's
instruction file and sending an explicit task-scoped wake produced the
qualifying follow-up above.
- The pinned application's immutable rate card omits `gpt-5.4-mini`, so
completed usage triggered the normal block-on-unpriced budget policy.
Supported append-only accounting corrections estimated the three
completed runs using the [official OpenAI model rate
card](https://developers.openai.com/api/docs/models/gpt-5.4-mini): $0.75
input, $0.075 cached input, $4.50 output per million tokens, assuming
standard processing. Estimates totaled 12.482655 cents. Original usage
and missing-price records are preserved; the $1 hard stop and
unpriced-usage guard stayed active.
- Model shell attempts included a missing `xxd` command and incorrect
guessed file paths. Those failed calls and subsequent successful actual
verifier calls remain visible in complete event streams. No
zero-tool-error claim is made.

No product source changed during this continuation. The previously
measured contention, cold-start and memory limits remain. The later
merge resolution is covered by the separate integration checks above;
these hosted receipts qualify their recorded candidate.


### Hosted stress test — October 29, 2026

**Hosted acceptance:** full native-agent execution and a changed-skill
follow-up pass on the recorded hosted candidate. The final integration
section above records the subsequent conflict resolution and its
separate checks.

Tested baseline `929e840595c8933ca485ac7a2e99413e1e11ca64` and this
exact candidate sequentially on one disposable TEST tenant, with
external Neon PostgreSQL and actual S3. The fixture contained **1,360
files / 22.4 MB / ten skills sharing one repository**. Measurements
cover deployed skill-preparation and native-context services in an
adjacent diagnostic process, not a complete agent turn. The container
had **24 vCPU / 24 GB**.

| Warm workload | Median before → after | p95 before → after | DB
returned value bytes/preparation before → after |
|---|---:|---:|---:|
| One skill, serial | 1,060 → 125 ms | 1,303 → 155 ms | 22,610,389 →
1,771 |
| One skill, four concurrent | 4,415 → 297 ms | 5,143 → 328 ms |
22,610,389 → 1,771 |
| Ten skills, serial | 5,723 → 535 ms | 6,057 → 600 ms | 224,739,030 →
10,671 |
| Ten skills, four concurrent | 24,410 → 1,986 ms | 24,833 → 2,221 ms |
224,739,030 → 10,671 |

- Candidate warm preparation performed **zero S3 GETs**. Ordinary-pool
queries fell from 16 to 9 for one skill and 61 to 9 for ten, excluding
dedicated lock connections. DB numbers measure returned values before
Drizzle decoding, not wire bytes. Baseline/candidate sample counts were
30/100, 32/100, 5/50 and 8/48 respectively; the small baseline ten-skill
samples are comparative evidence, not SLO estimates.
- **Application-UID soak:** 800 ten-skill preparations over 10.3
minutes, four workers paced by 2.5 seconds; **zero failures**, median
**559 ms**, p95 **775 ms**, p99 **910 ms**, zero warm S3 reads. All 726
accompanying health/company-list requests returned HTTP 200. Their
end-to-end p95 was 569/502 ms, including network latency. The probe
overlaps most of the soak. Soak DB-response capture was limited to its
first 60 seconds.
- **Important remaining costs:** an unpaced 16-way ten-skill burst had
p95 **8.32 seconds** because company-level preparation serializes. Cold
preparation took approximately **1.1–2.5 seconds** for one skill and
**7.4–10.0 seconds** for ten. Ten-skill diagnostic RSS remained about
**1.17 GiB** (baseline 1.18 GiB), and the soak peaked at **1.35 GiB**. A
concurrent whole-container sample was 3.45 GB. This does not establish
suitability for a small-memory host or eliminate per-entrypoint native
asset duplication.
- **Upgrade without a manual data migration passed:** legacy snapshot
JSON shrank from 22,465,691 bytes to `[]`, with a stored package and
zero pending conversions.
- **Recovery passed:** sixteen concurrent cold requests coalesced into
two S3 GETs; full bytes/binary/executable modes matched; same-size
corruption repaired; an injected storage failure preserved warm access
and failed cold access safely. A disabled public-Git source denied
recovery, and after the actual five-minute backoff its exact SHA
reconstructed all 229 files. Cross-company descriptor access was denied.
Private-Git credential revocation was not tested.
- **Retention passed:** reclaimed 27,754,496 allocated cache bytes,
deleted an unreferenced package row and both objects, preserved aged
authored history, and expired/reacquired reconstructible Git content. A
version pinned by the failed native run was correctly retained. Only
owned fixture timestamps were aged.
- **Browser UI passed:** authored skill creation, saved edit surviving
reload, version history, and restoration of earlier content.
- **Initial native attempt (superseded by the qualifying turns above):**
one real run failed at Daytona allocation after five minutes
(`setup_failed`), before any model execution. Provider cleanup also
reported failure. That attempt does not count as a passing native turn.
Its receipt and all evidence remain preserved; its cleanup later
recorded success after normal restoration.
- Initial A/B commands ran as SSH root. Diagnostic-owned lock/cache
permissions caused one setup HTTP 500 and the first UID-1000 soak to
fail; exact fixture paths were repaired. The final soak, retention and
recovery ran as application UID 1000. A fresh UID-1000 one-skill batch
also passed (30 samples, median 116 ms, p95 321 ms, 1,771 DB value
bytes/preparation). Earlier harness-export, SSH, npm provenance, backoff
and UI-wait failures remain recorded alongside their successful retries.
- Private evidence includes raw timings, assertions, HTTP probes,
screenshots and failure/cleanup receipts. Initial failed-campaign
cleanup suspended and retained the owned TEST tenant and released the
shared QA account lease at **15:22:41 UTC**. The tenant was subsequently
resumed under its existing ownership for the native execution
qualification above. Final cleanup completed at **15:53:09 UTC**: the
account lease is released and the tenant is suspended/retained under the
original October 30, 12:24:52 UTC retention deadline. Its separate stack
lease remains held; no active or pending-cleanup environment lease
remained. The original retention window is deliberately preserved even
after successful continuation.

- Earlier hosted-tested head:
`a2e597d800eb989d565780e7aa947d5368676444`, integrated with master
`929e840595`. The earlier merge conflicts were resolved at this head.
[Current-head
CI](https://github.com/paperclipai/paperclip/actions/runs/40884087629)
passed; all checks are complete with 55 successes, two neutral reports,
two skips, and no failures. [Apex review
15](https://github.com/paperclipai/paperclip/pull/18082#issuecomment-6480257258)
passed 5/5 with zero findings on this exact head. All review
conversations are resolved.
- Ten real-filesystem boundary cases and server typecheck pass locally
for the security fix. Coverage includes cache roots, descendants,
individual files, symlink aliases, recursive ancestors, a future
concurrent cache fill, and allowed sibling directories. [Current-head
private-project integration
tests](https://github.com/paperclipai/paperclip/actions/runs/40884087629/job/123483306405)
passed all 29 cases, including authorized and unauthorized imports of a
cached private package. Full typecheck, build, server suites, all eight
browser shards, and Runner checks passed in current-head CI. Local
database tests could not run because PostgreSQL could not initialize;
this limitation is recorded below.
- The earlier generated migration preserved the newest five snapshots.
The final integration renumbers its unchanged SQL to 0415. Migration
safety/numbering checks pass. Its SQL is byte-for-byte identical to the
previously verified idempotent skill migration; repeated application on
this latest schema was blocked by the same local database startup
failure.
- The broader verification before the final migration-only rebase passed
1,025 package, lifecycle, remote-launch, native-session, and
asset-retention tests, 52 focused UI tests, full workspace
typecheck/build, token gates, and map checks. The linked complete CI run
passed before the latest focused fixes.
- A disposable MinIO instance passed multipart upload, verified
read-back, and cold restoration of an 18 MB archive.
- Twenty warm samples with a synthetic 1,349-file, 22.8 MB repository
retained five metadata queries and zero object reads. Median preparation
was 256 ms for one selected skill and 691 ms for ten, with 5 ms emulated
network delay each way. These are local measurements, not production
latency guarantees.
- No production instance was changed. Earlier verification failures and
their disposition are retained below.

<details>
<summary>Earlier failures, review findings, fixes, and rerun
evidence</summary>

- The first CI pass found a stale inventory-span expectation and a
missing repository-read span. Both are fixed; all three observability
tests pass. One browser test copied page text into its YAML parser after
reload; its first unchanged-head retry passed, but the race recurred
after rebasing. The test now waits for the renamed URL and completed
reload before copying the editor, retaining its original assertions. The
earlier full local suite did not complete cleanly: it reported setup
timeouts and assertions, while isolated relevant reruns passed. The
complete current-head CI run closes that broad verification gap. No
production instance was changed.

- The first Apex review scored 2/5 and identified six issues. The fixes
protect authored history even when it shares an in-flight Git upload,
recheck upload expiry during orphan deletion, bound failed-upload
cleanup, retain per-version kind/encoding metadata, add sanitized
failure diagnostics, and correct the migration guide. The final focused
run passes all 140 tests; the preceding eight-file run passed all 149
tests, including local storage and S3 diagnostics. Server typecheck and
migration safety/numbering checks pass after the fixes. The second Apex
review scored 1/5 and found unbounded maintenance storage calls and
mismatched package/Git batch limits. Those are fixed with a 30-second
cancellable cleanup deadline, shared 100 MiB/10,000-file scanner and
storage limits, and bounded Git read batches. The proposed manual
lockfile edit was rejected with evidence of the enforced CI-owned
lockfile policy. All 19 storage/recovery unit tests and server
typechecking pass. The new database-backed cleanup/conversion tests were
locally skipped because the host exhausted its PostgreSQL semaphore
capacity (confirmed directly by initdb). That gap is closed by
current-head CI: [package-store
tests](https://github.com/paperclipai/paperclip/actions/runs/40771320775/job/123109300889)
pass all 18 cases and [package-worker
tests](https://github.com/paperclipai/paperclip/actions/runs/40771320775/job/123109301157)
pass all 7, with no skips in either suite. The corrected YAML browser
shard passes. A separate planning-fixture test failed its cleanup DELETE
with an activity-log foreign key violation after its behavior assertions
passed; its one unchanged-head retry passed. The third Apex review
scored 1/5 and identified a one-slot pool deadlock, projected path
limits, cleanup holding the preparation lock during archive reads, and a
publication/expiry race. These are fixed with transaction-local reads,
prefix-aware bounds, isolated verification, and shared package row locks
with a publication grace. The separate Paperclip review also found stale
historical inspection, eager history downloads, and conversion retries
blocking sleep. Those are fixed. Local PostgreSQL capacity recovered:
the new database regressions ran, including the one-slot fork, paused
storage during a warm turn, and a locked publication during expiry. The
expanded run passed 144/145 tests; its remaining assertion still
expected file bodies from the version list and was updated to fetch the
selected version. The corrected company-skill suite passes all 81 tests,
and the final package/projection run passes all 27. All 52 focused UI
tests pass after updating their query-provider fixtures; the first UI
run exposed a fixture remount, and the follow-up exposed an editing
syntax error, both corrected. Full workspace typecheck and build, token
gates, and map checks pass. The sandbox initially blocked the build
tool’s local IPC socket; the authorized build rerun passed. Master then
claimed migration 0401. The branch is rebased on f8226b0515, the
unchanged SQL is regenerated as 0402, and repeated application passed
twice on the combined schema. Post-rebase workspace typecheck passes.
All 1,025 package, lifecycle, remote-launch, native-session, and
asset-retention tests pass on the combined head. All CI checks passed
for head 9d80a3b52d. The fourth Apex review scored 3/5 and found a
save/edit lock-order cycle, a retry retaining an uncommitted source ID
in its recovery recipe, and a summary read racing conversion. Commit
b7d9237c6a fixes all three. The new concurrency and retry tests pass
within all 61 package/source tests, and server typecheck passes. Master
then claimed migration 0402. The branch is now rebased on 65fa1cd617 and
the unchanged SQL is regenerated as 0403. All 68 package, source, and
worker regressions, server typecheck, migration safety/numbering, and
repeated migration application pass on current head 62553671a0.
Subsequent verification and review outcomes are recorded below.


- The fifth Apex review scored 4/5 and found cached version reads
unnecessarily depending on object storage and conversion reads missing
the coordinator cancellation signal. Commit 1eb90d80be fixes both:
cached reads verify bytes without the preparation lock; manifest and
archive reads carry cancellation through conversion. The first focused
run passed 70/71 tests, including all three new cases, but the existing
diagnostic test observed no warning. Its fixture now sets an explicitly
expired migration deadline; the final worker suite passes all nine
cases. Server typecheck passes.


- The sixth Apex review scored 4/5 with one finding: concurrent cached
inventories were unbounded. Commit 4b7589e14c gives cached inventories
and previews a two-reader/32-waiter limit independent of cold storage.
The first expanded pressure test observed a third reader before the
preview path had the same bound; both paths now share the local limit.
All 73 focused cases pass, followed by all 29 package cases after the
final overload adjustment; server typecheck passes. The separate
Paperclip review found no must-fix or should-fix issues. Its optional
suggestion to persist durability verification was considered: cleanup
intentionally verifies current external bytes before discarding the last
local authored copy, and the missing-object regression requires that
check. Shared-package eviction happens on cold fills with at most two
attempts and a ten-second budget.


- The seventh Apex review reached 5/5 with one remaining P2:
timeout-bearing manifest reads bypassed request coalescing. The security
review also found repeated archive acquisition after a cold
administrative read. Commit 79a80a39d2 shares manifest loads with
per-caller cancellation and cancels the underlying work when the last
caller leaves; coalesces identical cold version/file requests; and
publishes verified restored packages into the shared cache. Publication
uses a short nonblocking lock, while eviction verification runs outside
it. The first expanded run passed 73/74 tests and exposed a missing
originating DB for cache publication inside read-only inspection;
transaction readers now pass that owner explicitly, keeping metadata on
the transaction. The corrected run passed 75/75. A final eviction test
initially expected the shared package to remain after only its separate
runtime projection was reused; it now explicitly accesses the shared
package during verification. All 32 final package tests pass, including
independent caller cancellation, one acquisition across
concurrent/repeated reads, and warm preparation during stalled eviction
verification. Server typecheck passes.


- The eighth Apex review scored 4/5 with one finding: decoded cold
inventories outlived the storage semaphore while waiting for cleanup.
Commit 9d7921d1ec adds a separate two-response/32-waiter lifetime bound
held through cleanup, preserving the independent warm-read queue and
forwarding cancellation through eviction verification. The new
regression pauses cleanup after two responses decode, verifies a third
archive stays queued, and completes warm preparation before releasing
cleanup. All 77 package/source/worker cases and server typecheck pass.
On the preceding head, serialized server shard 6/9 was interrupted with
“The operation was canceled” after 57 seconds of tests; the log contains
no failing assertion. Fresh latest-head CI reruns that shard along with
the other gates.

- The ninth Apex review scored 4/5 with one P1: cold responses could own
both permits while waiting on a one-slot ordinary database pool held by
a queued transaction reader. Commit a94f0e409f reads caller-scoped
metadata before admission and isolates all acquisition/cleanup SQL on a
private connection. The new real-PostgreSQL regression reproduces the
deadlock against the previous head and passes with the fix; all 78
focused cases and server typecheck pass. The previous CI run completed
with 54 successful checks, including both formerly interrupted/stuck
serialized shards.

- The tenth Apex review scored 4/5 with one P1: preloading legacy bodies
before admission could retain many full repositories in queued requests.
Commit 276e353a37 removes that preload and also bounds legacy manifest
derivation/fallback, using private connections inside the I/O slot. All
80 focused tests pass; the final synchronized package suite passes all
36 tests and server typecheck passes.

- Apex review 11 returned 5/5 with zero comments on 276e353a37. While it
ran, master 8bc051245f added migration 0403, creating a metadata
conflict. Merge commit e1783764d9 integrates master and generates
0404_clumsy_leo.sql; its skill SQL is byte-for-byte identical to the
previously verified migration. The combined schema passes migration
safety/numbering, repeated SQL application twice, all 80 focused
PostgreSQL tests, full workspace typecheck, and full build. The
preceding a94f0e409f CI run was superseded and cancelled while its final
runner job was active; no old-head success is claimed for that run.

- Final acceptance: Apex review 12 returned 5/5 with zero comments on
e1783764d9. All 28 review threads are resolved. The final head has 55
successful checks, two neutral reports, two skipped checks, and no
pending or failing checks; the PR workflow completed successfully.
Superseded run 40789121902 was cancelled to free its workflow slot for
the final head; normal cancellation did not finish promptly, so force
cancellation completed it.

- October 29 conflict resolution (867c00673b): master 929e840595 added
installed-selection reuse and bundled relocation, routine cleanup
notifications, a hardened YAML copy helper, and migrations through 0413.
The merge retains those behaviors together with package read scoping and
the public-selection fast path; restricted selections still reauthorize
and the full row read rechecks scope. Generated migration 0414 retains
the existing SQL unchanged. Workspace typecheck/build and migration
checks pass. Local database verification is unavailable: initdb reports
semget / “could not create semaphores: No space left on device”; all 168
cases skipped, and repeated migration application failed at startup.
Docker returned HTTP 500 during container creation, so no Docker test
pass is claimed. Current-head CI must close this gap.

- Apex review 14 scored 3/5 with one P1: a local import could copy
cached private package files into a company-wide skill. Commit
a2e597d800 rejects cache trees and ancestors through a canonical
filesystem boundary, including aliases and ancestors checked before a
concurrent fill. Ten local filesystem tests and server typecheck pass.
The existing private-project integration case now rejects package-cache
imports for both authorized and unauthorized users. The preceding merge
commit’s complete CI run passed, and logs confirm all 168 locally
skipped cases executed successfully: package store 36, sources 35,
package worker 9, runtime selection 4, observability 3, authoring 81.

- Apex review 15 returned 5/5 with zero new findings on a2e597d800. The
separate Paperclip review found no must-fix or should-fix issues. Its
optional suggestions were considered: unexpected filesystem cleanup
failures remain visible, the durable marker cannot replace checking that
bytes still exist before deleting a last local authored copy, and
administrative history batching remains outside the pre-turn
optimization. The cache-path literal centralization suggestion is also
optional; the current producer and guard use the same directory, covered
by the new integration regression. All review conversations are
resolved.

- Earlier October 29 CI/Apex acceptance: a2e597d800 was mergeable at
that check, Apex review 15 passed 5/5 with zero findings, and all review
conversations were resolved. The hosted verification above records the
later acceptance limits and renewed conflicts. Workflow 40884087629
completed successfully, with 55 successful checks, two neutral reports,
two skips, and no pending or failed checks. The project-skill-sources
job log confirms all 29 cases ran and passed, closing the latest
integration gap. The preceding merge workflow also confirmed all 168
focused database cases ran and passed. No local PostgreSQL or Docker
pass is claimed.

</details>

## Risks

- First use after eviction can require storage or Git access. Revoked
grants, removed sources, and unreachable commits can prevent
reconstruction. Recovery never substitutes a newer commit.
- Historical packages without a proven recipe keep durable bytes.
Cleanup favors retention when a run pin cannot be interpreted safely.
- New uploads and conversion use bounded work and persisted retries.
Large initial conversions still add temporary storage and database load.
- Optional cleanup can wait until the next server wake. Supported
notifications assume one owning server process.
- Per-entrypoint native tree duplication and cold preparation cost
remain. The final ten-skill cold sample took 47.6 seconds on the shared
host.
- Generic authored edits can still hold an existing edit transaction
while a package uploads.
- Explicit file/folder deletion uses temporary disk space for rollback.
An uncertain commit with a changed skill or unavailable database retains
that backup and logs its location for recovery.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, repository tools, code
execution, and tests. The exact served model ID and context window are
not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-29 12:19:37 -05:00
DottaandPaperclip 2d0c138122 Expand direct assistant MCP tools for work and configuration (#15380)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - People also use assistants in Codex, Claude, and other MCP clients.
> - The existing assistant connection can read work and create tasks or
comments.
> - It cannot edit tasks, exchange files, or manage normal agent and
project settings.
> - These operations must retain the person's permissions and
Paperclip's execution rules.
> - This pull request adds an explicit operation registry and separately
consented configuration access.
> - Assistants can manage work without receiving credentials or runner
authority.

## Linked Issues or Issue Description

**Subsystem affected**

Cross-cutting: assistant MCP, domain routes, consent UI, storage, and
Product E2E.

**Problem or motivation**

A connected assistant cannot update tasks, maintain documents, attach
files, or configure existing agents, projects, and skills. Users must
leave the assistant for these routine actions.

**Proposed solution**

Add named tools and a restricted API registry to direct connections.
Require separate configuration consent. Reuse domain routes and retry
receipts. File uploads save the attachment when the byte transfer
succeeds.

**Alternatives considered**

Arbitrary REST forwarding would expose administration and credential
operations. Runner impersonation would bypass execution ownership.
Separate upload completion calls add unnecessary client state.

**Roadmap alignment**

Checked ROADMAP.md and related MCP pull requests. This extends the
human-authorized connection from #14933. It does not replace the runner
or introduce agent impersonation.

Companion Cloud routing and directory isolation:
https://github.com/paperclipai/paperclip-cloud/pull/678.

## What Changed

- Add task editing, finish/block, documents/revisions, deliverables,
agent settings/instructions, projects/repositories, and skills/files.
- Add an allowlisted API search/call registry with identical field
restrictions, scopes, and retry identities.
- Add unchecked configuration consent. Existing write grants retain
their current authority.
- Add hashed, expiring file transfer tickets and atomic upload receipts.
No completion call is required.
- Preserve company boundaries, human attribution, active native
execution ownership, and execution review gates.
- Add protocol/domain tests, consent stories, and eight paid Product E2E
workflows.
- Repair two CI fixture races: await cold route setup before assertions,
and wait for asynchronously loaded connection copy. Both fixture suites
pass (24 + 48 tests).

## Verification

- Consent revision: one write-access checkbox controls requested work
and configuration permissions in browser and device flows. All 16
consent tests, UI typecheck/build and token gates pass. Updated
interactive stories cover default approval, opt-out and viewer
restrictions. The paid browser helper uses the new exact label. Real
GPT-5.4 Mini Product E2E passes 2/2 at
`64f96373118eb190f8cba1c2ab17cb979555f3ad` (configuration + permission
denial), campaign `local-2026-10-07T00-51-14-337Z`, no automatic
retries, cleanup passed; $0.04149375 estimated assistant cost plus
unpriced worker usage. Raw results, usage and source fingerprints are
retained in the worktree. UI and Product E2E typechecks pass.

- Prior head `2f246d4b74f1f98c75ebcb37ae6753a748237fac`: all 52 checks
pass; two optional Storybook checks skip. Greptile 5/5 on that head, no
unresolved review threads. Final consent head
`64f96373118eb190f8cba1c2ab17cb979555f3ad` also has all checks passing
and Greptile 5/5 with no unresolved threads. The unchanged Cursor
sandbox test had one 10-second timeout, passed in local isolation, and
passed its single CI rerun; the failed attempt remains in [the CI
run](https://github.com/paperclipai/paperclip/actions/runs/37554106934).
The existing chat retry-denial browser test had one visibility failure;
its single rerun passes, and the failed attempt remains in [the CI
run](https://github.com/paperclipai/paperclip/actions/runs/37542735691).

- Full workspace `pnpm -r typecheck` and `pnpm build` pass at final
runtime source `b2196fae1`. UI token gates pass.
- 139 MCP/OAuth/transfer/privacy tests and 76 grader calibration tests
pass, including one-connection PostgreSQL OAuth and concurrent upload
retries.
- Paid Product E2E: all eight expanded cases qualified across Mini,
Haiku and Sonnet. A merged-source repeat passed 23/24; one Haiku cell
timed out before application startup. Final affected-case qualification
passes 9/9 on all three models with grader v16, including the failed
cell. Automatic retries disabled; failures, costs, source hashes and
independent durable-state/file assertions are retained in [the
verification
record](doc/plans/2026-10-06-expanded-assistant-mcp-verification.md).
- Actual Codex CLI, Claude Code and OpenCode clients completed local
reads/mutations. Codex wrote a report, Claude updated it in a later
conversation, and OpenCode uploaded/downloaded a file with matching
SHA-256 and registered the attachment. Revoking the CLI grant rejects
subsequent bridge initialization.
- Butter staging is verified on final runtime `b2196fae1`
([deployment](https://github.com/paperclipai/paperclip-cloud/actions/runs/37538432138)).
A fresh OpenCode workspace fetched the copied invitation, configured
remote MCP, started OAuth and reached real consent with configuration
unchecked. Invalid transfer tickets return 403 through Cloud. Human
approval for the new persistent staging grant is pending; hosted
task/file success is not yet claimed. The final transaction fix is
deployed.
- Full local `pnpm test:run` passed 15,614 general-server tests but
stopped on two macOS timeouts. The heartbeat test passed in isolation;
the existing 40,000-file Git stress fixture timed out again. Its Linux
CI lane passes. Later local full-suite phases did not run after the
timeout; this is not an all-green local full-suite claim.
- Instructions and security limits are in `doc/public-mcp.md`; the saved
plan is `doc/plans/2026-10-06-expanded-assistant-mcp-tools.md`.

## Risks

- This expands the experimental direct MCP surface. Explicit schemas and
domain permissions must stay synchronized.
- Migration 0311 adds transfer tickets and upload receipts. Expired
orphan cleanup must not remove committed attachments.
- Configuration requires a new consent request containing that scope;
the single write-access choice controls it alongside work mutations.
Refreshing an old grant does not add it.
- The public directory keeps its original ten tools through the
companion Cloud change.
- Hosted consent/work proof remains the final delivery gate. The PR
stays draft while approval of the new staging grant is pending; code
checks and review are green. Merging is a separate action.

## Model Used

OpenAI Codex (GPT-6, tool use and code execution). The exact serving
model ID and context window are not exposed in this session. Paid
evaluation models: gpt-5.4-mini, claude-haiku-4-5-20251001;
claude-sonnet-4-6.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (focused suites; full-suite
macOS limitation disclosed above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-06 20:29:58 -05:00
DottaandPaperclip 3447609d22 fix(runner): stream and page large API responses within capture budgets (#14301)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents use governed API tools to inspect task evidence.
> - Large API results become saved assets with short previews.
> - Reading an asset through the same tool used to create another asset,
so the agent could not reach the rest of the evidence.
> - The 10 MiB response cap also blocked useful large results. Removing
all bounds allowed excessive disk use.
> - This pull request streams responses up to 1 GiB and makes saved text
readable in bounded pages. It adds durable run budgets and capture
admission limits.
> - Agents can inspect complete evidence while tool results, memory use,
and capture work stay bounded.

## Linked Issues or Issue Description

**What happened?**
A large response became an asset. Reading that asset returned another
asset and the same preview. Responses above 10 MiB failed before the
agent could read any page.

**Expected behavior**
The agent can fetch a large response and read its saved text to EOF.
Each page stays bounded. New snapshots have a generous finite limit and
a durable run budget. Existing larger assets remain readable through
byte ranges.

**Steps to reproduce**
1. Call a GET operation that returns more than 10 MiB of text or JSON.
2. Before the fix, the tool returns `api_transport_failure`.
3. With this change, responses up to 1 GiB become streamed snapshots
with artifact references.
4. Read `GET /api/assets/{assetId}/content` with `responseText:
{offsetBytes: 0, limitBytes: 8192}`. Follow `nextOffsetBytes` until
null.

Related work: #14186 added the API fallback tools. #14218 bounded API
discovery.

## What Changed

- Add authenticated UTF-8 text windows to `call_api`, with byte offsets
and total size. Keep each page at or below 24 KiB.
- Stream new responses above 24 KiB through private temporary files into
company-owned assets. Bound each capture to 1 GiB of decoded bytes.
Reject oversized declared lengths before reading and count streamed
bytes before writing.
- Reserve capture budget in the run record before spilling. Allow 4 GiB
per run. Settle successful captures to their actual size. Failed or
interrupted captures retain their full 1 GiB reservation. Run restarts
do not reset the budget.
- Enforce a 20 GiB company snapshot quota with database reservations.
Count legacy snapshots and unfinished storage work across runs and
processes. Asset deletion frees quota.
- Limit large captures to two per company and four per server process.
Hold slots through storage upload and temporary-file cleanup. Use a
10-minute download deadline and 30-second connection/idle-read timeouts.
- Return explicit size, budget, busy, and timeout errors. Preserve
unknown outcomes for mutations whose response cannot be captured.
- Read saved assets through authenticated storage ranges, with at most
two extra bytes for UTF-8 and EOF handling. Unpaged reads return the
existing asset and digest with a bounded preview. Reads create no copies
and do not consume capture budget.
- Keep existing assets above 1 GiB readable in pages. Use safe integer
offsets and PostgreSQL `bigint` asset sizes.
- Stream large S3 uploads through ordered multipart requests. Abort
failed uploads and remove partial local files.
- Revalidate run authority during downloads. Keep company authorization,
GET-only text paging, redirect denial, and mutation replay receipts.
- Document the separate 10 MiB upload limits. This PR does not raise
memory-buffered attachment ingestion limits. Future large video uploads
need streamed ingestion and storage quotas.

## Verification

- Full workspace `pnpm -r typecheck` and `pnpm build` pass after
rebasing on master.
- Focused API and response tests: 1,761 pass. Cover declared and chunked
oversize responses, incorrect Content-Length, exact-limit success,
active-stream deadline, cancellation, cleanup, concurrency admission,
and mutation outcome handling.
- Real HTTP integration: 28 tests pass, including runnerd → PRP →
authority → HTTP, a 12 MiB snapshot, final-page/EOF reads, cross-company
denial, a persisted 3 GiB sparse asset, and large mutation receipt
replay.
- The HTTP suite verifies durable run-budget accounting, simultaneous
runs competing for company quota, legacy snapshot accounting, deletion
refunds, failed-storage reservations, cleaned-failure refunds,
metadata-rollback cleanup refunds, preservation after a lost commit
acknowledgement, and small/saved reads after capture-budget exhaustion.
- A standalone proof streams exactly 1 GiB through the production
capture helper, verifies the final bytes, and removes its temporary
file. It uses repeated 256 KiB chunks and records a peak process RSS of
191 MiB.
- Earlier storage verification covers exact S3 multipart boundaries,
cleanup/abort failures, and a 17 MiB transfer through the real AWS SDK
to a local HTTP S3 endpoint. No cloud S3 qualification was run for this
follow-up.
- The local full test run was interrupted for the company-quota changes.
A later targeted run hit exhausted macOS shared-memory slots before
tests started; two unattached PostgreSQL segments with dead owners were
reclaimed before retrying. All 55 current-head checks pass at
`aebb80ceeeee77d5a56b67bfffd835f2f846878c`, including the full CI test
suite, typecheck, build, browser suites, security scan, and Greptile
(5/5). There are no unresolved review threads. The combined rebased test
catalog also passes (48 tests).
- Earlier paging acceptance passed Daytona and separate staging at
`7739879e9`. Those runs predate the streaming and budget changes.

## Risks

- The 1 GiB response cap and 10-minute active-download deadline are
intentional product limits. Larger live results must use endpoint
pagination or a direct file workflow. Existing larger assets remain
readable through bounded ranges.
- A durable 20 GiB company snapshot quota counts stored runner-api
assets and active/orphan reservations across runs and processes. The
operator can set PAPERCLIP_RUNNER_API_COMPANY_CAPTURE_MAX_BYTES to a
finite value of at least 1 GiB. Deleting snapshots frees capacity;
possible orphan storage must be reconciled before releasing its
reservation.
- A failed capture uses its full reservation. A new large capture needs
a full 1 GiB available, even if it later completes at a smaller size.
Small reads and existing asset pages remain available.
- Concurrency limits apply per server process. The run byte budget is
shared through the database.
- The `integer` to `bigint` migration rewrites asset metadata and takes
an exclusive table lock. File bytes stay in storage.
- A live endpoint is fetched once before returning its snapshot.
Continue reading the saved artifact for stable pages. Mutations may
commit before any size or transport error; inspect state before
retrying.
- Attachment uploads and native file handoffs still default to 10 MiB.
Raising buffered ingestion paths to GiB sizes is separate work.

## Model Used

OpenAI Codex, based on GPT-6, with code execution and repository tools.
The runtime does not expose an exact serving model variant or
context-window size. The earlier paging work also used browser testing
and subagents.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-28 10:36:32 -05:00
Dotta 04a19cbc6e Address artifact PR review feedback 2026-05-30 20:51:46 +00:00
ForgottenandClaude Opus 4.6 fdd2ea6157 feat: add storage system with local disk and S3 providers
Introduces a provider-agnostic storage subsystem for file attachments.
Includes local disk and S3 backends, asset/attachment DB schemas, issue
attachment CRUD routes with multer upload, CLI configure/doctor/env
integration, and enriched issue ancestors with project/goal resolution.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-20 10:31:56 -06:00