PaperClipAI

Jannik/PaperClipAI

0 Fork 0

mirror of https://github.com/paperclipai/paperclip.git synced 2026-10-10 12:07:09 +02:00

Commit Graph

Author	SHA1	Message	Date
DottaandClaude Fable 5	23f34491e2	Fix apiCompression corrupting and dropping Better Auth responses for gzip clients (#9381 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Its server fronts every API route — including Better Auth sign-in — with Express middleware, and #9190 added an `apiCompression` middleware that gzips JSON responses over 1KB > - That middleware buffers `res.write()` chunks with `String(chunk)`, but Better Auth (via better-call) streams `Uint8Array` chunks and commits headers with `writeHead()` before streaming > - `String(Uint8Array)` serializes the body to comma-separated decimal bytes (~3.4x inflation), and once the inflated body crossed the 1KB threshold, `setHeader()` threw `ERR_HTTP_HEADERS_SENT` and the catch handler destroyed the socket > - Every real browser sends `Accept-Encoding: gzip`, so sign-in returned zero bytes (`net::ERR_EMPTY_RESPONSE` / "Failed to fetch"), while curl without `Accept-Encoding` worked — making the bug easy to misdiagnose as a client or network issue > - This pull request makes the middleware byte-safe for `Uint8Array` chunks, passes through responses whose headers are already committed, and falls back to the uncompressed body instead of destroying the connection when compression fails > - The benefit is that browser sign-in (and any other streamed binary-chunk response) works again for gzip-accepting clients, with regression tests locking in all three behaviors ## Linked Issues or Issue Description Refs #9190 (introduced the `apiCompression` middleware). No public GitHub issue exists; bug description: - What happened: Sign-in from any real browser failed with `net::ERR_EMPTY_RESPONSE` / "Failed to fetch". The server logged `ERR_HTTP_HEADERS_SENT` from the compression middleware and destroyed the response socket, so zero bytes reached the client. - Expected: `/api/auth/` responses are delivered intact regardless of the client's `Accept-Encoding`. - Steps to reproduce:* Run the server with API compression active, open the web UI in a browser (which sends `Accept-Encoding: gzip`), and attempt email/password sign-in. The auth response body exceeds ~300 bytes, so after the ~3.4x stringification inflation it crosses the 1024-byte compression threshold and the response is destroyed. `curl` without `Accept-Encoding` succeeds against the same server. - Scope: Any route that streams `Uint8Array` chunks and/or commits headers via `writeHead()` before writing — in practice all Better Auth routes served through better-call. ## What Changed - `server/src/middleware/api-compression.ts`: - Buffer `res.write()` chunks with a `toBodyBuffer()` helper that converts `Uint8Array`/`ArrayBuffer` views via `Buffer.from()` instead of `String()`, so binary chunks are preserved byte-for-byte. - Pass responses through untouched once headers are already sent (`writeHead()`-style streaming), since compression headers can no longer be set at that point. - On any compression failure, write the original uncompressed body instead of calling `res.destroy()`, so clients get a valid (just uncompressed) response rather than a dropped connection. - `server/src/__tests__/api-compression.test.ts`: three new regression tests — small `writeHead`+`Uint8Array` responses are delivered byte-for-byte, large ones no longer drop the connection, and `Uint8Array` JSON bodies gzip without corruption (includes `/api/auth-bridge` and `/api/uint8-json` test routes mirroring better-call's streaming pattern). ## Verification - `cd server && pnpm vitest run src/__tests__/api-compression.test.ts` — 10/10 passing (7 pre-existing + 3 new regression tests). - Manual: with the fix, browser sign-in against a dev instance succeeds for gzip-accepting clients; before the fix the same request returned `net::ERR_EMPTY_RESPONSE`. ## Risks - Low risk. The middleware still compresses large text/JSON responses exactly as before; the changes only affect paths that previously produced corrupted or destroyed responses. - Behavioral shift: responses whose headers were already committed are now delivered uncompressed instead of being (incorrectly) buffered — this is strictly less surprising than the previous corrupted output. - Failure-path shift: a compression error now yields an uncompressed 200 response instead of a dropped connection. ## Model Used - Claude Fable 5 (`claude-fable-5`, Anthropic), extended thinking enabled, running via Claude Code / Paperclip agent harness with tool use (shell, file edit, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>	2026-07-10 16:04:29 -05:00
DottaandPaperclip	176645187c	Fix request storm polling and issue-list coalescing (#9190 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The board UI keeps issue, agent, activity, and run state fresh through polling across several pages and sidebar surfaces > - When multiple components or browser tabs poll the same company data at the same time, the API can receive bursts of duplicate issue-list requests > - Those duplicate requests increase database and server load without returning meaningfully different data > - This pull request adds server-side compression/coalescing plus client-side visibility-aware and cross-tab shared polling > - The benefit is lower request volume during normal board usage while preserving fresh UI data for active users ## Linked Issues or Issue Description No exact public GitHub issue was found for this request. Problem: - The board can issue redundant polling requests for the same issue-list data from multiple UI surfaces and tabs. - In busy operator sessions, those bursts can trigger request-storm behavior and unnecessary issue-list load. - Expected behavior is to reuse identical in-flight work server-side and reduce hidden-tab or duplicate-tab polling client-side while preserving normal refresh behavior. Related public context found during duplicate search: - #8206 covers a different board UI 404-storm scope. - #5165 covers separate issue-list behavior around page-size truncation. ## What Changed - Added API compression middleware foundation and a company/created-at index for heartbeat run access. - Added server-side issue-list request storm detection and identical in-flight request coalescing. - Added UI fetch metadata, visibility-aware polling, and request deduplication for issue/activity/client calls. - Added cross-tab shared polling primitives and wired them into the sidebar, inbox, dashboard, issue, project, routine, and agent surfaces. - Resolved the latest `master` migration collision by keeping upstream `0140_built_in_managed_resources.sql` and renumbering this branch's heartbeat-run index migration to `0141_heartbeat_runs_company_created_at_index.sql`; the SQL uses `CREATE INDEX IF NOT EXISTS` for idempotency. - Stabilized server heartbeat cleanup tests exposed by the PR check matrix. - Fixed the Greptile compression follow-up by weakening strong ETags on encoded JSON responses and bypassing compression for streamed/download responses. ## Verification - `pnpm exec vitest run ui/src/components/IssuesList.test.tsx ui/src/pages/Inbox.test.tsx` — passed after resolving the latest `master` conflict in `IssuesList.tsx` and updating the 200-result cap expectations. - `pnpm check:token-gates` — passed after the UI conflict resolution. - `jq -e '.entries \| length as $n \| (map(.idx) \| unique \| length == $n) and (map(.tag) \| unique \| length == $n)' packages/db/src/migrations/meta/_journal.json` — passed after renumbering the migration to `0141`. - `pnpm exec vitest run server/src/__tests__/api-compression.test.ts` - passed after the compression follow-up. - `pnpm exec vitest run server/src/__tests__/issue-list-assignee-filter-routes.test.ts` - passed after the compression follow-up. - `pnpm --filter @paperclipai/server typecheck` - passed after the compression follow-up. - Greptile Review for head `8dbddac41ec273fda404100b4981ddb912fad57b` - passed after the latest conflict/migration fix; all Greptile review threads are resolved. - `pnpm exec vitest run server/src/__tests__/api-compression.test.ts server/src/__tests__/issue-list-assignee-filter-routes.test.ts ui/src/api/client.test.ts ui/src/api/issues.test.ts ui/src/lib/polling.test.ts ui/src/lib/cross-tab-poll.test.ts ui/src/pages/Inbox.test.tsx ui/src/components/SidebarProjects.test.tsx ui/src/components/SidebarAccountMenu.test.tsx ui/src/lib/issueDetailCache.test.ts` — passed. - `pnpm exec vitest run ui/src/api/client.test.ts` — passed. - `pnpm exec vitest run ui/src/pages/Inbox.test.tsx ui/src/components/SidebarProjects.test.tsx ui/src/components/SidebarAccountMenu.test.tsx ui/src/lib/issueDetailCache.test.ts` — passed. - `pnpm exec vitest run server/src/__tests__/heartbeat-worktree-suppression.test.ts` — passed. - `pnpm exec vitest run server/src/__tests__/low-trust-red-team-routes.test.ts` — passed. - `pnpm exec vitest run server/src/__tests__/heartbeat-workspace-finalize-branch.test.ts` — passed. - `pnpm --filter @paperclipai/ui typecheck` — passed. - `pnpm --filter @paperclipai/server typecheck` — passed. - GitHub PR checks for head `8dbddac41ec273fda404100b4981ddb912fad57b`: all GitHub Actions/status checks passed; Greptile, Superagent, Socket, Snyk, build, typecheck/release registry, general tests, serialized server suites, e2e, canary dry run, policy, and commitperclip review are green; Storybook visual regression and security-review were skipped/neutral by policy. - Confirmed this branch does not include `pnpm-lock.yaml` or `.github/workflows` changes. ## Risks - Medium risk: issue-list coalescing changes request timing and cache semantics for a hot API path. - Medium risk: cross-tab polling uses browser coordination primitives, so older or unusual browser environments need fallback behavior to stay correct. - Low migration risk: the new index migration is ordered after current `master` and uses `CREATE INDEX IF NOT EXISTS`. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used OpenAI Codex coding agent, GPT-5-based model, tool-enabled with shell/git execution. Exact hosted deployment identifier and context-window size were not surfaced in the agent runtime. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-07-09 17:38:34 -05:00

Author

SHA1

Message

Date

DottaandClaude Fable 5

23f34491e2

Fix apiCompression corrupting and dropping Better Auth responses for gzip clients (#9381 )

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Its server fronts every API route — including Better Auth sign-in —
with Express middleware, and #9190 added an `apiCompression` middleware
that gzips JSON responses over 1KB
> - That middleware buffers `res.write()` chunks with `String(chunk)`,
but Better Auth (via better-call) streams `Uint8Array` chunks and
commits headers with `writeHead()` before streaming
> - `String(Uint8Array)` serializes the body to comma-separated decimal
bytes (~3.4x inflation), and once the inflated body crossed the 1KB
threshold, `setHeader()` threw `ERR_HTTP_HEADERS_SENT` and the catch
handler destroyed the socket
> - Every real browser sends `Accept-Encoding: gzip`, so sign-in
returned zero bytes (`net::ERR_EMPTY_RESPONSE` / "Failed to fetch"),
while curl without `Accept-Encoding` worked — making the bug easy to
misdiagnose as a client or network issue
> - This pull request makes the middleware byte-safe for `Uint8Array`
chunks, passes through responses whose headers are already committed,
and falls back to the uncompressed body instead of destroying the
connection when compression fails
> - The benefit is that browser sign-in (and any other streamed
binary-chunk response) works again for gzip-accepting clients, with
regression tests locking in all three behaviors

## Linked Issues or Issue Description

Refs #9190 (introduced the `apiCompression` middleware).

No public GitHub issue exists; bug description:

- **What happened:** Sign-in from any real browser failed with
`net::ERR_EMPTY_RESPONSE` / "Failed to fetch". The server logged
`ERR_HTTP_HEADERS_SENT` from the compression middleware and destroyed
the response socket, so zero bytes reached the client.
- **Expected:** `/api/auth/*` responses are delivered intact regardless
of the client's `Accept-Encoding`.
- **Steps to reproduce:** Run the server with API compression active,
open the web UI in a browser (which sends `Accept-Encoding: gzip`), and
attempt email/password sign-in. The auth response body exceeds ~300
bytes, so after the ~3.4x stringification inflation it crosses the
1024-byte compression threshold and the response is destroyed. `curl`
without `Accept-Encoding` succeeds against the same server.
- **Scope:** Any route that streams `Uint8Array` chunks and/or commits
headers via `writeHead()` before writing — in practice all Better Auth
routes served through better-call.

## What Changed

- `server/src/middleware/api-compression.ts`:
- Buffer `res.write()` chunks with a `toBodyBuffer()` helper that
converts `Uint8Array`/`ArrayBuffer` views via `Buffer.from()` instead of
`String()`, so binary chunks are preserved byte-for-byte.
- Pass responses through untouched once headers are already sent
(`writeHead()`-style streaming), since compression headers can no longer
be set at that point.
- On any compression failure, write the original uncompressed body
instead of calling `res.destroy()`, so clients get a valid (just
uncompressed) response rather than a dropped connection.
- `server/src/__tests__/api-compression.test.ts`: three new regression
tests — small `writeHead`+`Uint8Array` responses are delivered
byte-for-byte, large ones no longer drop the connection, and
`Uint8Array` JSON bodies gzip without corruption (includes
`/api/auth-bridge` and `/api/uint8-json` test routes mirroring
better-call's streaming pattern).

## Verification

- `cd server && pnpm vitest run src/__tests__/api-compression.test.ts` —
10/10 passing (7 pre-existing + 3 new regression tests).
- Manual: with the fix, browser sign-in against a dev instance succeeds
for gzip-accepting clients; before the fix the same request returned
`net::ERR_EMPTY_RESPONSE`.

## Risks

- Low risk. The middleware still compresses large text/JSON responses
exactly as before; the changes only affect paths that previously
produced corrupted or destroyed responses.
- Behavioral shift: responses whose headers were already committed are
now delivered uncompressed instead of being (incorrectly) buffered —
this is strictly less surprising than the previous corrupted output.
- Failure-path shift: a compression error now yields an uncompressed 200
response instead of a dropped connection.

## Model Used

- Claude Fable 5 (`claude-fable-5`, Anthropic), extended thinking
enabled, running via Claude Code / Paperclip agent harness with tool use
(shell, file edit, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

2026-07-10 16:04:29 -05:00

DottaandPaperclip

176645187c

Fix request storm polling and issue-list coalescing (#9190 )

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The board UI keeps issue, agent, activity, and run state fresh
through polling across several pages and sidebar surfaces
> - When multiple components or browser tabs poll the same company data
at the same time, the API can receive bursts of duplicate issue-list
requests
> - Those duplicate requests increase database and server load without
returning meaningfully different data
> - This pull request adds server-side compression/coalescing plus
client-side visibility-aware and cross-tab shared polling
> - The benefit is lower request volume during normal board usage while
preserving fresh UI data for active users

## Linked Issues or Issue Description

No exact public GitHub issue was found for this request.

Problem:
- The board can issue redundant polling requests for the same issue-list
data from multiple UI surfaces and tabs.
- In busy operator sessions, those bursts can trigger request-storm
behavior and unnecessary issue-list load.
- Expected behavior is to reuse identical in-flight work server-side and
reduce hidden-tab or duplicate-tab polling client-side while preserving
normal refresh behavior.

Related public context found during duplicate search:
- #8206 covers a different board UI 404-storm scope.
- #5165 covers separate issue-list behavior around page-size truncation.

## What Changed

- Added API compression middleware foundation and a company/created-at
index for heartbeat run access.
- Added server-side issue-list request storm detection and identical
in-flight request coalescing.
- Added UI fetch metadata, visibility-aware polling, and request
deduplication for issue/activity/client calls.
- Added cross-tab shared polling primitives and wired them into the
sidebar, inbox, dashboard, issue, project, routine, and agent surfaces.
- Resolved the latest `master` migration collision by keeping upstream
`0140_built_in_managed_resources.sql` and renumbering this branch's
heartbeat-run index migration to
`0141_heartbeat_runs_company_created_at_index.sql`; the SQL uses `CREATE
INDEX IF NOT EXISTS` for idempotency.
- Stabilized server heartbeat cleanup tests exposed by the PR check
matrix.
- Fixed the Greptile compression follow-up by weakening strong ETags on
encoded JSON responses and bypassing compression for streamed/download
responses.

## Verification

- `pnpm exec vitest run ui/src/components/IssuesList.test.tsx
ui/src/pages/Inbox.test.tsx` — passed after resolving the latest
`master` conflict in `IssuesList.tsx` and updating the 200-result cap
expectations.
- `pnpm check:token-gates` — passed after the UI conflict resolution.
- `jq -e '.entries | length as $n | (map(.idx) | unique | length == $n)
and (map(.tag) | unique | length == $n)'
packages/db/src/migrations/meta/_journal.json` — passed after
renumbering the migration to `0141`.
- `pnpm exec vitest run server/src/__tests__/api-compression.test.ts` -
passed after the compression follow-up.
- `pnpm exec vitest run
server/src/__tests__/issue-list-assignee-filter-routes.test.ts` - passed
after the compression follow-up.
- `pnpm --filter @paperclipai/server typecheck` - passed after the
compression follow-up.
- Greptile Review for head `8dbddac41ec273fda404100b4981ddb912fad57b` -
passed after the latest conflict/migration fix; all Greptile review
threads are resolved.
- `pnpm exec vitest run server/src/__tests__/api-compression.test.ts
server/src/__tests__/issue-list-assignee-filter-routes.test.ts
ui/src/api/client.test.ts ui/src/api/issues.test.ts
ui/src/lib/polling.test.ts ui/src/lib/cross-tab-poll.test.ts
ui/src/pages/Inbox.test.tsx ui/src/components/SidebarProjects.test.tsx
ui/src/components/SidebarAccountMenu.test.tsx
ui/src/lib/issueDetailCache.test.ts` — passed.
- `pnpm exec vitest run ui/src/api/client.test.ts` — passed.
- `pnpm exec vitest run ui/src/pages/Inbox.test.tsx
ui/src/components/SidebarProjects.test.tsx
ui/src/components/SidebarAccountMenu.test.tsx
ui/src/lib/issueDetailCache.test.ts` — passed.
- `pnpm exec vitest run
server/src/__tests__/heartbeat-worktree-suppression.test.ts` — passed.
- `pnpm exec vitest run
server/src/__tests__/low-trust-red-team-routes.test.ts` — passed.
- `pnpm exec vitest run
server/src/__tests__/heartbeat-workspace-finalize-branch.test.ts` —
passed.
- `pnpm --filter @paperclipai/ui typecheck` — passed.
- `pnpm --filter @paperclipai/server typecheck` — passed.
- GitHub PR checks for head `8dbddac41ec273fda404100b4981ddb912fad57b`:
all GitHub Actions/status checks passed; Greptile, Superagent, Socket,
Snyk, build, typecheck/release registry, general tests, serialized
server suites, e2e, canary dry run, policy, and commitperclip review are
green; Storybook visual regression and security-review were
skipped/neutral by policy.
- Confirmed this branch does not include `pnpm-lock.yaml` or
`.github/workflows` changes.

## Risks

- Medium risk: issue-list coalescing changes request timing and cache
semantics for a hot API path.
- Medium risk: cross-tab polling uses browser coordination primitives,
so older or unusual browser environments need fallback behavior to stay
correct.
- Low migration risk: the new index migration is ordered after current
`master` and uses `CREATE INDEX IF NOT EXISTS`.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

OpenAI Codex coding agent, GPT-5-based model, tool-enabled with
shell/git execution. Exact hosted deployment identifier and
context-window size were not surfaced in the agent runtime.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

2026-07-09 17:38:34 -05:00