PaperClipAI

Jannik/PaperClipAI

0 Fork 0

mirror of https://github.com/paperclipai/paperclip.git synced 2026-11-11 13:24:55 +01:00

Commit Graph

Author	SHA1	Message	Date
Devin Foley	fcb84d472d	feat: already-imported transfer error names the landed company (#12144 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Chunked company-import transfers are deduplicated by content: a byte-identical zip that already finished an apply is rejected > - The rejection said only "this exact package was already imported by a completed transfer" without saying where that import went > - Users who could not find the earlier import read the rejection as data loss and kept retrying, or exported again and created duplicate companies > - This pull request makes the declaration response carry the company the completed apply created, and both clients name it in the error > - The benefit is that the dedupe rejection now points at the existing import instead of implying it vanished ## Linked Issues or Issue Description What existing behavior does this improve? The `alreadyCompleted` rejection when re-declaring a chunked company-import transfer. Subsystem affected Shared transfer contract (`packages/shared/src/company-import-transfer.ts`), transfer declaration route (`server/src/routes/companies.ts`), web import page, CLI import command. Current behavior `POST /api/companies/import/transfers` returns `alreadyCompleted: true` with no pointer to the earlier import. Web and CLI raise "This exact package was already imported by a completed transfer. Re-export the package to import it again." Proposed behavior The response includes an optional `company` field (`{id, name, issuePrefix} \| null`) resolved from the completed run's company link. Web and CLI raise a shared message: `… It created the company "Paperclip" (PAPA) — open it from the company switcher. Re-export the package to import it again.` A company that was deleted since (or a link that was never written) degrades to `null` and the original message. Breaking changes None. The new response field is optional; old clients ignore it. ## What Changed - `CompanyImportTransferCreated` gains optional `company`, plus a shared `buildAlreadyImportedMessage` used by both clients. - The declaration route's `alreadyCompleted` branch resolves the landed company null-safely via `companyService.getById`. - Web (`ui/src/pages/CompanyImport.tsx`) and CLI (`cli/src/commands/client/company.ts`) raise the shared message. ## Verification - `cd packages/shared && npx vitest run src/company-import-transfer.test.ts` — 3 tests (named company, id fallback, no-company original message). - `cd server && npx vitest run src/__tests__/company-import-transfer-routes.test.ts` — 24 tests; the re-declaration test now asserts the company payload and the deleted-company null path. - `cd cli && npx vitest run src/__tests__/company-import-transfer.test.ts` — 17 tests; new test pins the named-company message. - `cd ui && npx vitest run src/pages/CompanyImport.test.tsx` — 23 tests. - `pnpm run typecheck` clean in shared, server, ui, cli. ## Risks - Low risk. The lookup runs only on the `alreadyCompleted` branch and is null-safe; the transfer run is already scoped to the requesting actor (user + instance context in the actor key), so the response never names a company the caller did not import. ## Model Used - Claude Fable 5 (`claude-fable-5`, Anthropic) with extended thinking and tool use, via Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge	2026-08-25 13:51:50 -07:00
Devin Foley	23a1b025c2	feat(server): chunked resumable company import transfers (#11223 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Company import moves large packages into an instance, and since the upload cap rose to 1 GB, the transport is the weak point: one HTTP request, buffered fully in memory, with no resume > - A dropped connection at 90% of an 800 MB upload starts the whole transfer over, and a server restart loses all progress > - This pull request adds the server side of chunked resumable import transfers: a durable run ledger and routes that accept the same import zip as verified ~32 MB parts spooled to disk > - An interrupted transfer resumes from the parts already uploaded — across dropped connections, page refreshes, and server restarts — and peak upload memory drops from the whole package to one part > - The benefit is that large imports become reliable on real-world connections instead of all-or-nothing ## Linked Issues or Issue Description What happened? Large company imports travel as a single HTTP upload. On a slow or flaky connection, any interruption discards all progress and the upload restarts from zero. The server buffers the entire compressed package in memory during upload. A server restart mid-upload loses the transfer entirely. With the upload cap now at 1 GB, these failure modes govern exactly the imports the cap was raised for. Expected behavior A large import upload survives interruptions: already-transferred data is kept and verified, only the missing remainder is re-sent, and the server's memory use during upload is bounded by a part, not the package. Steps to reproduce 1. Import a multi-hundred-MB company package over a connection that drops mid-upload. 2. The upload fails; retrying starts from byte zero. 3. Repeat on an unstable connection and the import may never complete. ## What Changed - New `company_transfer_runs` table (drizzle schema + migration) and `companyTransferRunService`: one row per transfer with a content-derived idempotency key, per-part completion recorded atomically and idempotently, resume scoped to actor and direction, completed runs short-circuiting retries of identical content. - New transfer routes beside the existing import routes, same authorization: declare a sliced zip (`POST /import/transfers` — validates cap, 64 MB part ceiling, contiguity, size sums, sha256 format), upload parts (`PUT .../parts/:n` — raw body, hash-and-size verified before an atomic write to a disk spool under the instance root; re-uploads are no-op successes), poll resume state (`GET .../:id` — missing parts recomputed from disk), and apply (`POST .../:id/apply` — requires all parts, re-verifies the assembled zip against the whole-file hash fail-closed, then feeds the existing import pipeline through factored helpers rather than duplicated logic). - Hourly sweep fails and cleans spools idle for 24 h; a swept transfer honestly reports all parts missing on resume. - Strict UUID gating on run ids before any filesystem path construction. - The existing single-shot upload path is untouched; clients arrive in the follow-up PR. ## Verification - Transfer route suite (embedded Postgres): create/upload/status/apply round-trip with a real imported company, out-of-order parts, wrong-hash part rejected and unrecorded, re-upload no-op, apply-with-missing-parts rejection, resume after failure with prior progress intact, assembled-hash mismatch failing closed with spool deletion, actor scoping 404s, async-job apply, sweep followed by honest resume. - Ledger suite (embedded Postgres): part idempotency, actor/direction scoping, completed-run short-circuit, cancelled runs staying cancelled. - Existing portability route suite unchanged and green; server + db typechecks clean. Exact counts in the PR checks. ## Risks - New routes are additive; the existing import path is untouched. The transfer routes carry the same board authorization as the import routes they sit beside. - Disk spool: bounded by the existing upload cap per transfer, cleaned on success, failure, hash mismatch, and by the 24 h sweep. Spool paths are strict-UUID-gated. - The apply step still materializes the assembled zip in memory once (same profile as today's single-shot import at apply time); upload-time memory drops to one part. - Known limitation, deliberate: transfers are keyed on content alone, so identical package content cannot be imported twice without re-exporting (surfaced explicitly to the caller). Acceptable for v1; noted for review. ## Model Used - Claude Fable 5 (`claude-fable-5`) via Claude Code CLI, with extended thinking and tool use (multi-agent implementation with independent verification). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge	2026-08-11 15:25:07 -07:00

Author

SHA1

Message

Date

Devin Foley

fcb84d472d

feat: already-imported transfer error names the landed company (#12144 )

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Chunked company-import transfers are deduplicated by content: a
byte-identical zip that already finished an apply is rejected
> - The rejection said only "this exact package was already imported by
a completed transfer" without saying where that import went
> - Users who could not find the earlier import read the rejection as
data loss and kept retrying, or exported again and created duplicate
companies
> - This pull request makes the declaration response carry the company
the completed apply created, and both clients name it in the error
> - The benefit is that the dedupe rejection now points at the existing
import instead of implying it vanished

## Linked Issues or Issue Description

**What existing behavior does this improve?**

The `alreadyCompleted` rejection when re-declaring a chunked
company-import transfer.

**Subsystem affected**

Shared transfer contract
(`packages/shared/src/company-import-transfer.ts`), transfer declaration
route (`server/src/routes/companies.ts`), web import page, CLI import
command.

**Current behavior**

`POST /api/companies/import/transfers` returns `alreadyCompleted: true`
with no pointer to the earlier import. Web and CLI raise "This exact
package was already imported by a completed transfer. Re-export the
package to import it again."

**Proposed behavior**

The response includes an optional `company` field (`{id, name,
issuePrefix} | null`) resolved from the completed run's company link.
Web and CLI raise a shared message: `… It created the company
"Paperclip" (PAPA) — open it from the company switcher. Re-export the
package to import it again.` A company that was deleted since (or a link
that was never written) degrades to `null` and the original message.

**Breaking changes**

None. The new response field is optional; old clients ignore it.

## What Changed

- `CompanyImportTransferCreated` gains optional `company`, plus a shared
`buildAlreadyImportedMessage` used by both clients.
- The declaration route's `alreadyCompleted` branch resolves the landed
company null-safely via `companyService.getById`.
- Web (`ui/src/pages/CompanyImport.tsx`) and CLI
(`cli/src/commands/client/company.ts`) raise the shared message.

## Verification

- `cd packages/shared && npx vitest run
src/company-import-transfer.test.ts` — 3 tests (named company, id
fallback, no-company original message).
- `cd server && npx vitest run
src/__tests__/company-import-transfer-routes.test.ts` — 24 tests; the
re-declaration test now asserts the company payload and the
deleted-company null path.
- `cd cli && npx vitest run
src/__tests__/company-import-transfer.test.ts` — 17 tests; new test pins
the named-company message.
- `cd ui && npx vitest run src/pages/CompanyImport.test.tsx` — 23 tests.
- `pnpm run typecheck` clean in shared, server, ui, cli.

## Risks

- Low risk. The lookup runs only on the `alreadyCompleted` branch and is
null-safe; the transfer run is already scoped to the requesting actor
(user + instance context in the actor key), so the response never names
a company the caller did not import.

## Model Used

- Claude Fable 5 (`claude-fable-5`, Anthropic) with extended thinking
and tool use, via Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

2026-08-25 13:51:50 -07:00

Devin Foley

23a1b025c2

feat(server): chunked resumable company import transfers (#11223 )

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Company import moves large packages into an instance, and since the
upload cap rose to 1 GB, the transport is the weak point: one HTTP
request, buffered fully in memory, with no resume
> - A dropped connection at 90% of an 800 MB upload starts the whole
transfer over, and a server restart loses all progress
> - This pull request adds the server side of chunked resumable import
transfers: a durable run ledger and routes that accept the same import
zip as verified ~32 MB parts spooled to disk
> - An interrupted transfer resumes from the parts already uploaded —
across dropped connections, page refreshes, and server restarts — and
peak upload memory drops from the whole package to one part
> - The benefit is that large imports become reliable on real-world
connections instead of all-or-nothing

## Linked Issues or Issue Description

**What happened?**

Large company imports travel as a single HTTP upload. On a slow or flaky
connection, any interruption discards all progress and the upload
restarts from zero. The server buffers the entire compressed package in
memory during upload. A server restart mid-upload loses the transfer
entirely. With the upload cap now at 1 GB, these failure modes govern
exactly the imports the cap was raised for.

**Expected behavior**

A large import upload survives interruptions: already-transferred data
is kept and verified, only the missing remainder is re-sent, and the
server's memory use during upload is bounded by a part, not the package.

**Steps to reproduce**

1. Import a multi-hundred-MB company package over a connection that
drops mid-upload.
2. The upload fails; retrying starts from byte zero.
3. Repeat on an unstable connection and the import may never complete.

## What Changed

- New `company_transfer_runs` table (drizzle schema + migration) and
`companyTransferRunService`: one row per transfer with a content-derived
idempotency key, per-part completion recorded atomically and
idempotently, resume scoped to actor and direction, completed runs
short-circuiting retries of identical content.
- New transfer routes beside the existing import routes, same
authorization: declare a sliced zip (`POST /import/transfers` —
validates cap, 64 MB part ceiling, contiguity, size sums, sha256
format), upload parts (`PUT .../parts/:n` — raw body, hash-and-size
verified before an atomic write to a disk spool under the instance root;
re-uploads are no-op successes), poll resume state (`GET .../:id` —
missing parts recomputed from disk), and apply (`POST .../:id/apply` —
requires all parts, re-verifies the assembled zip against the whole-file
hash fail-closed, then feeds the existing import pipeline through
factored helpers rather than duplicated logic).
- Hourly sweep fails and cleans spools idle for 24 h; a swept transfer
honestly reports all parts missing on resume.
- Strict UUID gating on run ids before any filesystem path construction.
- The existing single-shot upload path is untouched; clients arrive in
the follow-up PR.

## Verification

- Transfer route suite (embedded Postgres): create/upload/status/apply
round-trip with a real imported company, out-of-order parts, wrong-hash
part rejected and unrecorded, re-upload no-op, apply-with-missing-parts
rejection, resume after failure with prior progress intact,
assembled-hash mismatch failing closed with spool deletion, actor
scoping 404s, async-job apply, sweep followed by honest resume.
- Ledger suite (embedded Postgres): part idempotency, actor/direction
scoping, completed-run short-circuit, cancelled runs staying cancelled.
- Existing portability route suite unchanged and green; server + db
typechecks clean. Exact counts in the PR checks.

## Risks

- New routes are additive; the existing import path is untouched. The
transfer routes carry the same board authorization as the import routes
they sit beside.
- Disk spool: bounded by the existing upload cap per transfer, cleaned
on success, failure, hash mismatch, and by the 24 h sweep. Spool paths
are strict-UUID-gated.
- The apply step still materializes the assembled zip in memory once
(same profile as today's single-shot import at apply time); upload-time
memory drops to one part.
- Known limitation, deliberate: transfers are keyed on content alone, so
identical package content cannot be imported twice without re-exporting
(surfaced explicitly to the caller). Acceptable for v1; noted for
review.

## Model Used

- Claude Fable 5 (`claude-fable-5`) via Claude Code CLI, with extended
thinking and tool use (multi-agent implementation with independent
verification).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

2026-08-11 15:25:07 -07:00