3 Commits

Include renames

Author	SHA1	Message	Date
Devin FoleyandPaperclip	3367b75ccc	fix: fence accepted work and cleanup before idle sleep (#15522) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A host can stop an idle instance to reduce unused compute. > - Zero active runs do not prove that requests, saved work, or cleanup are complete. > - A client can disconnect while its request still writes, and failed cleanup can remain only in memory or on disk. > - This pull request holds new admission and checks accepted work, cleanup, and persisted work under an owned drain. > - The host gets an empty report only while those checks remain valid. ## Linked Issues or Issue Description Refs #13413. That companion PR uses the same owned-hold vocabulary for runtime services. This PR covers HTTP requests, scheduler work, accounting, cleanup, and the instance work inventory. It does not include the preview gateway changes. **What existing behavior does this improve?** The instance task-drain API reports process counters. It does not prove that a host can safely stop the instance for idle sleep. **Current behavior** A quiet run set can coexist with an unfinished request, cleanup after a completed run, future work, or a failed accounting write. **Proposed behavior** Provide a bounded owned idle hold. Block new ingress, track accepted handler promises, inspect durable and local work, and return `none` only when the same hold stays quiet through the checks. Keep normal deployment drains compatible. **Reason and benefit** Hosts can identify eligible idle instances without treating a disconnect or failed cleanup write as completed work. ## What Changed - Add `purpose: "idle"`, a bounded TTL, unique owners, and owner-checked release to task drain. Existing holds cannot be replaced by another API request. - Gate HTTP ingress before parsers, auth, webhooks, and MCP. Gate new WebSocket upgrades. Track async handlers in nested Express routers and error middleware until they settle, even after the response or client disconnect. - Count accepted live-event WebSocket authentication through settlement, even after disconnects. Count detached built-in agent, managed-home and runtime-service startup reconciliation after readiness. - Keep health and control mutations tracked. Count control-request authentication separately from the read-only report, including concurrent user/company/membership writes. - Pause new scheduler admissions during idle holds. Count work already in flight, including database backups, and reject scans whose work generation changes. Periodic backups block sleep without a host wake schedule. - Inspect accounting and orphan-cleanup spool directories without skipping temporary or malformed entries. Retain orphan tokens through queue splices, flush failures, and buffer overflow. Keep failed usage capture counted until its database failure fence is written. - Check persisted work across companies in a bounded read-only transaction. Include deferred agent-file cleanup and saved watchdogs whose watched issues are complete. Enabled plugins and unsupported retained work remain blockers. - Require the exact idle owner and completed startup tracking before returning an empty report. Keep reports free of tenant details. - Document the hosting protocol, retry behavior, conservative blockers, and the remaining external provider-stop race. ## Verification - Current head: `8d9a599b732c2047b8c671d4799065d7c90a3567`, rebased on master `941a3fa991aeb97eb1ac390c65b7973b5f6de1ad`. Both heartbeat helper extractions are preserved. GitHub confirms no merge conflicts. - Focused heartbeat renderer/run-log, drain, control-auth, admission, route and PostgreSQL inventory checks: 234 tests passed in 10 suites after the rebase. - The POST task-drain contract includes the expected `409` conflict response. The 84 OpenAPI and instance-settings route tests and server typecheck passed after that final documentation fix. - Final accepted-upgrade/startup regression run: 104 tests passed in five suites, including success and failure after readiness or disconnect. Final server typecheck also passed. - `pnpm -r typecheck`: passed. - `pnpm build`: passed. - `pnpm install --frozen-lockfile` and the PR diff secret scan passed. No dependency or lockfile changes are added by this PR. - The prior verification covered HTTP disconnects and early responses, async error handlers, concurrent authentication writes, signed bootstrap, saved watchdogs, backup promises, orphan cleanup, spools and failed accounting fences. Those tests passed. - The last full local `pnpm test:run` stopped in the general-server phase with 16,658 passing tests and five skill/connector fixture failures caused by an ancestor workspace skill directory. That full local run preceded this rebase and has not been repeated for the import conflict. Full current-head CI passed: 53 successful checks and two conditional skips, with no failures. - All eight review findings are fixed and their threads resolved, including accepted upgrade authentication, detached startup writes and the POST conflict contract. Current-head Apex review is 5/5 with no new findings; all eight review threads remain resolved. - No live provider stop or production change was performed. ## Risks - The hosting controller must use the owned protocol and hold external admission through its final validation and provider stop. A legacy drain cannot authorize idle sleep. During an idle hold, new requests receive 503 with `Retry-After: 1`; the host must handle queueing or retry before enabling this path. - This is a single-process protocol. An unexpected restart after the last validation can race an external provider stop. The host must serialize deploy/wake/sleep operations and bind the validation to the instance it stops. Multiple replicas need shared fencing. - Some retained state conservatively prevents sleep, including every enabled plugin. This PR does not promise that every inactive instance becomes eligible. - The HTTP adapter uses Express 5 router layers. Real Express tests cover nested routes, errors and disconnects. New routes must register before tracking is installed. Detached work must have durable state or explicit work tracking. - Unrecoverable in-memory cleanup debt keeps the instance awake until reconciliation. This change does not make such debt survive an unplanned process crash. No schema migration or provider configuration change is included. ## Model Used OpenAI GPT-6 via Codex, with reasoning, repository inspection, tool use and code execution. The exact deployment variant/model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run focused tests locally and they pass; the full local fixture limitation and clean full CI result are documented above - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-08 08:35:45 -07:00
Devin Foley	2c53437fc9	fix(server): authenticate cloud-proxied browsers on the live-events websocket (#11290) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The UI receives live run/issue events over a websocket at `/api/companies/:id/events/ws`; the server authorizes upgrades with a bearer token or a Better Auth session > - On a cloud-managed deployment, browsers authenticate through trusted `x-paperclip-cloud-*` headers injected by the managing front door — they never hold a local Better Auth session, and the Express middleware lane that understands those headers is not consulted for websocket upgrades > - Every browser websocket upgrade behind the front door therefore resolves no identity and is rejected 403: the live-events socket has never connected on a managed instance, leaving permanent reconnect churn and console failure noise while the UI silently degrades to polling > - This pull request adds a cloud-actor lane to the upgrade authorization, reusing the same trusted-header resolver the HTTP middleware uses > - The benefit is working realtime updates on managed instances, an end to the reconnect churn, and unchanged self-hosted behavior ## Linked Issues or Issue Description No existing issue. Description follows the bug template: **What happened?** On a cloud-managed instance, the browser console shows `WebSocket connection to 'wss://…/api/companies/<id>/events/ws' failed:` repeating indefinitely for every company, on a healthy instance. The server rejects each upgrade with 403 because `authorizeUpgrade` in `server/src/realtime/live-events-ws.ts` only knows bearer tokens and Better Auth sessions, while cloud-proxied browsers authenticate via `x-paperclip-cloud-*` trusted headers (handled only by the Express `actorMiddleware` lane in `server/src/middleware/auth.ts`). **Expected behavior** A browser that authenticates through the trusted cloud headers can open the live-events websocket for any company in its membership scope, exactly as it can call the HTTP API for those companies. **Steps to reproduce** 1. Run Paperclip in `authenticated` mode behind a proxy that injects the `x-paperclip-cloud-*` headers with a valid `PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN`. 2. Load any company page in a browser (no local Better Auth session). 3. HTTP API calls succeed; every `/events/ws` upgrade is rejected 403 and the UI retries forever. ## What Changed - `server/src/middleware/auth.ts`: `resolveCloudTenantActor` now accepts a minimal `CloudActorHeaderSource` (`header(name)`) instead of an Express `Request` — `Request` satisfies it unchanged — plus `cloudActorHeaderSourceFromHeaders` to adapt raw `IncomingMessage.headers`. - `server/src/realtime/live-events-ws.ts`: `authorizeUpgrade` gains an injected `resolveCloudActor` lane, tried before the Better Auth session fallback in `authenticated` mode. A resolved cloud actor is authoritative: the upgrade is authorized only for a company in the actor's membership scope (`companyIds`, the same scope the HTTP lane grants). Absent/unresolvable cloud headers fall through to the session path. - `server/src/index.ts`: wires `resolveCloudActor` through `resolveCloudTenantActor` + the header shim. The resolver self-gates: without `PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN` and a matching trust token it returns null, so self-hosted deployments never take this path. - Tests: upgrade authorized for an in-scope company (session resolver not consulted), rejected for an out-of-scope company, fall-through to session auth when no cloud actor resolves; header-shim resolution from a raw lowercased header map including `string[]` values. ## Verification - `pnpm vitest run server/src/__tests__/live-events-ws.test.ts server/src/middleware/cloud-tenant-actor.test.ts` — 25 tests pass. - `pnpm typecheck` in `server/` — clean. - Not verified live end-to-end: that requires a managed instance running this build; the direct probe evidence (HTTP authenticated fine, every WS upgrade 403) matches the code path exactly. ## Risks Low risk. The new lane only activates when the deployment configures the cloud trust token and the request presents it; both checks already protect the HTTP lane. Authorization scope is the same `companyIds` set the HTTP middleware computes (primary stack company plus the user's real membership rows). The cloud resolver's user/company materialization writes are debounced (existing behavior shared with the HTTP lane), so websocket reconnect storms do not amplify database writes. Self-hosted instances see no behavioral change, covered by the fall-through test. ## Model Used - Claude (Anthropic), Claude Fable 5 (`claude-fable-5`), via Claude Code CLI with extended thinking and tool use (code search, edit, test execution; diagnosis included live websocket handshake probes against a managed instance). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge	2026-08-12 11:22:50 -07:00
Aron PrinsandPaperclip	e93d78b46c	fix(server): harden live events upgrade sockets (#8383) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server exposes live event updates to board and agent clients over WebSocket upgrade requests > - WebSocket upgrade authorization can be asynchronous, leaving the raw HTTP upgrade socket in server-owned code before `ws` takes over > - If the client disconnects during that authorization window, the server can still try to reject or upgrade a closed socket > - A raw socket write after peer disconnect can emit `EPIPE` / `ECONNRESET`, and without a listener that can become process-fatal > - This pull request hardens the pre-`ws` upgrade socket path and adds regression coverage for disconnect/error races > - The benefit is that live event reconnect churn degrades gracefully instead of risking a server crash ## Linked Issues or Issue Description External public context: Refs https://github.com/aronprins/paperclip-desktop/issues/14 No matching open upstream issue or PR was found when searching `paperclipai/paperclip` for `EPIPE`, `rejectUpgrade`, `live-events-ws`, `websocket upgrade`, and related socket-write terms. ### Bug report **What happened?** The live events WebSocket upgrade handler could write a rejection response to the raw upgrade socket after the peer had already disconnected during async authorization. A transport-level `EPIPE` or similar socket error on that raw socket can terminate the server process if it is emitted without an error listener. **Expected behavior** The server should not write to destroyed/non-writable upgrade sockets, should tolerate raw socket errors while authorization is pending, and should not call `handleUpgrade()` after the socket is no longer writable. **Steps to reproduce** 1. Start a Paperclip server with live events enabled. 2. Open a raw WebSocket upgrade request to `/api/companies/:companyId/events/ws`. 3. Disconnect the client before async authorization resolves. 4. Let the server take the reject or upgrade path. 5. Observe that the pre-fix path can still write to or upgrade a closed socket. **Paperclip version or commit** Reproduced by static inspection on `master` before this PR at `950484d20`. **Deployment mode** Any mode using live event WebSocket upgrades. The report was originally observed from local Desktop embedding, but the vulnerable code is in the server package. ## What Changed - Added a writable-state guard for raw upgrade sockets before rejecting or completing an upgrade. - Changed rejection responses from raw `write()` + `destroy()` to guarded `end()` with warning logging if rejection fails synchronously. - Attached a temporary raw socket error listener during the async upgrade authorization window and cleaned it up on socket close or successful `ws.handleUpgrade()`. - Added regression tests for rejecting after an early socket close and for handling raw socket `error` events while authorization is pending. ## Verification - `pnpm exec vitest run server/src/__tests__/live-events-ws.test.ts` - `pnpm --filter @paperclipai/server typecheck` - `git diff --check` ## Risks - Low risk: the change is scoped to the live-events WebSocket upgrade path before `ws` takes ownership of the socket. - Rejected upgrade responses now use graceful `socket.end(...)`; clients should still receive the same HTTP status text when the socket is writable. - The temporary error listener is removed before successful `handleUpgrade()` so normal WebSocket client error handling remains owned by the existing `connection` path. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used OpenAI Codex, GPT-5 coding agent with repository-aware tool use, shell command execution, GitHub connector access, and local code editing. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] If this change affects the UI, I have included before/after screenshots - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-06-20 11:32:49 -07:00