Commit Graph
4 Commits
Author SHA1 Message Date
DottaandPaperclip b94963f91e fix: retry temporary repository catalog failures before dispatch (#19690)
## Thinking Path

> - Paperclip manages AI agents and their work.
> - A task must check repository access before it starts agent work.
> - Repository discovery can fail while the user's existing grant
remains valid.
> - Discovery previously replaced a known temporary provider refusal
with an empty catalog.
> - The task then reported lost access and did not schedule recovery.
> - This change keeps temporary discovery failures distinct and uses the
existing bounded retry owner.
> - Each retry must pass a fresh access check before it can use the
saved repository.

## Linked Issues or Issue Description

**What happened?**

A task with a saved catalog repository stops with an access denial when
GitHub returns an exhausted-quota response or HTTP 503. The task does
not start provider work, but the discovery layer discards the temporary
failure and returns an empty catalog.

**Expected behavior**

Known temporary failures before provider dispatch use the existing
bounded infrastructure retry. The successor checks the same responsible
user's current grants and reads the saved files only after fresh
discovery succeeds. A completed denial remains terminal.

**Steps to reproduce**

1. Bind a task to an authorized catalog repository with saved local
changes.
2. Make its authorized GitHub discovery return HTTP 403 with zero
remaining quota, or HTTP 503.
3. Let discovery recover before the existing retry deadline.
4. Observe one successful successor using the original repository bytes.

**Paperclip version or commit**

Reproduced on master at `b8d94b3ad1`. A search found no existing PR for
this temporary catalog failure.

## What Changed

- Keep a safe typed failure for known HTTP availability and quota
responses. Honor the later provider deadline and bound it to one day.
- Propagate that failure only when fresh discovery cannot find a
required repository. Preserve successful partial catalogs and the
existing connection setup error contract.
- Classify the failure only before provider dispatch. Reuse existing
bounded retry, interaction ownership, and bootstrap settlement rules.
- Keep current grant checks, saved repository selection, normal Stop,
and provider-started failure policy.
- Keep provider deadlines when accepted interactions use their own retry
owner. Recognize secondary HTTP 403 rate limits with valid retry advice
while keeping ordinary permission denials terminal. This matches [GitHub
rate-limit
guidance](https://docs.github.com/en/rest/using-the-rest-api/rate-limits-for-the-rest-api).
- Keep direct repository request responses outside the automatic
bootstrap retry contract. Test successful partial discovery and
missing-required-repository recovery with two independent eligible
connections.
- Document task repository discovery recovery.

## Verification

- Behavioral RED: real heartbeat and PostgreSQL fixtures reached actual
HTTP 403 and503 discovery responses, started no adapter, and found zero
successors instead of one. Both cases failed before the production
change.
- Behavioral GREEN: both cases waited for the real retry deadline, used
the normal queue worker, ran the controlled adapter once after fresh
discovery, and read the original saved bytes. Two cases passed in 74.73
seconds. No external model or provider request was made.
- Deadline RED/GREEN: a combined Retry-After and quota-reset response
initially selected the earlier deadline. The regression now selects the
later deadline.
- Initial repair at `d8d4714a18` (same production source as
`018e8c52d345d79972d7907468080dc9908c299f`): seven affected files
passed, 69 cases in 185.16 seconds. That test-only cleanup then passed
all five real-database acceptance cases in 134.76 seconds.
- Commands: `pnpm exec vitest run` with
`heartbeat-repository-catalog-recovery.test.ts`,
`github-catalog-timing.test.ts`, `github-grant-metadata.test.ts`,
`project-repositories-persistence.test.ts`,
`task-workspace-repositories-persistence.test.ts`,
`github-repository-catalog-failure.test.ts`, and
`services/heartbeat/retries.test.ts` from the server source tree. The
acceptance file also ran fresh at `018e8c52d3`. A fresh five-case run
after the P1 repairs passed at `f490ca6b78` in 122.06 seconds. The final
eight-case file passed fresh at
`e5b2efab11919467850f55fd6345317132b9b672` in 124.62 seconds, using real
owned embedded PostgreSQL. It covers automatic recovery with saved
bytes, genuine denial, provider-started classification, normal Stop, the
direct HTTP route, and both partial-catalog outcomes.
- Review repairs at `f490ca6b787cf77df96d3d6edfb2ab9115eb19a6`: both
reported P1s reproduced RED. The real database interaction scheduler
chose a retry after 24 seconds before the provider deadline of one hour.
A secondary HTTP 403 with valid Retry-After lost its retry
classification. The narrow fixes then passed 44 cases across scheduler,
header, and metadata-contract files in 37.66 seconds.
- A real direct request through the task workspace router and normal
error middleware reproduced a 500 response instead of the existing 403.
Required-ID retry mode now applies only to admission and its rechecks.
The route and both two-connection partial-catalog cases then passed in
29.09 seconds. Both partial cases use actual owned grants and credential
bindings.
- All seven server type layers passed again at final head
`e5b2efab11919467850f55fd6345317132b9b672`. The affected test type layer
passed again after the test-only cleanup. Environment: macOS arm64, Node
26.4.0, pinned pnpm 9.15.4, Vitest 5.0.3, owned disposable PostgreSQL 18
on loopback. The current checkout's shared packages, plugin SDK, and
Runner were built first.
- One local acceptance run under concurrent owned test/typecheck load
hit the unchanged 15-second Stop-case timeout. Its four other cases
passed. The unchanged complete file then passed serially. The timeout
receipt remains recorded; no assertion or timeout was weakened.
- A provider-started failure retains the existing generic adapter retry
policy. It does not receive the catalog no-provider certificate or a
provider retry deadline. Fixture cleanup cancels its scheduled retry
through the normal owner.
- Final head `e5b2efab11919467850f55fd6345317132b9b672` has 59
successful checks, 3 allowed skips, no pending or failed check, and
Greptile 5/5 with no open findings. [Completed CI
workflow](https://github.com/paperclipai/paperclip/actions/runs/42322202977).
- Additional local diagnostic for the late recovery-owner advisory: on
unchanged `e5b2efab`, the dedicated scheduler returned `retry_exhausted`
with attempt 3/max 2. The normal stranded sweeper then escalated once,
created no extra run, and started no adapter, even after catalog access
recovered. One case passed in 17.32 seconds. The existing
failed-predecessor path delegates to that bounded scheduler and returns;
it does not enter a new generic retry budget. No production registry
change was made. This diagnostic was not a CI case. Its local harness is
preserved separately at `74e3854eea3f7c4121fdcd9c8a8ad522cd8c7477` on
`codex/catalog-exhaustion-acceptance`; its production source is
identical to `e5b2efab`.
- Module boundary and whitespace checks passed.

## Risks

- A temporary failure from another eligible connection can delay a truly
missing repository for the existing limit of two attempts. It cannot
grant access.
- Unknown network errors and genuine access denials retain their
existing behavior. This change handles known HTTP failures only.
- Historical provider quota causality is unproven. This repair has a
separate local producer regression. It does not rewrite past failures or
queued user intents.
- Acceptance uses real local database and heartbeat owners with
controlled HTTP and adapter fixtures. It does not claim a live
native-provider recovery.

## Model Used

OpenAI GPT-6.1 Codex (`gpt-6.1-sol`), with high reasoning, repository
tools, code execution, and controlled local PostgreSQL tests. The host
did not expose a context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-06 19:29:40 -06:00
DottaandPaperclip eb12cfc96b perf(github): activate connection work from events and deadlines (#18395)
Reviewed and verified at f77620b9a4.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-31 15:38:43 -05:00
3edba11576 fix(github): stop counting suspended installations as repository access (#17343)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents get GitHub access through a managed GitHub App connection.
**Refresh access** reads the installations and repositories that the
user token can reach.
> - GitHub keeps listing a suspended App installation and all of its
repositories for the user token. But the token then loses write access
and cannot read private repositories.
> - `loadGitHubGrantMetadata` did not read `suspended_at`. It counted a
suspended installation as live access ("all repos") and set the
connection to healthy.
> - So the board showed "connected, all repositories" while agents got a
403 on `git push` and a 404 on private repositories. The suspend webhook
marks the connection unhealthy, but the next refresh or reconnect
reverted that.
> - This pull request skips suspended installations. When all
installations are suspended, Refresh access fails with a clear error and
a link to the installation settings page, where an owner can click
**Unsuspend**.
> - The benefit is that the board shows the real access state, and
operators get a direct path to fix it.

## Linked Issues or Issue Description

**What happened?**
A GitHub App installation on an organization was suspended. The
Paperclip GitHub connection still showed "connected" with access to all
repositories after **Refresh access**. Agents that used the managed
token got `Permission to <org>/<repo>.git denied` (403) on push,
`Resource not accessible by integration` on branch creation, and 404 on
private repositories.

**Expected behavior**
Refresh access must not count a suspended installation as repository
access. When every installation is suspended, Refresh access must fail
with a message that says the installation is suspended, and the page
must link to the installation settings page.

**Steps to reproduce**
1. Connect GitHub to Paperclip through the GitHub App, with an
installation on an organization.
2. On GitHub, suspend that installation (Organization settings → GitHub
Apps → Configure → Suspend).
3. In Paperclip, click **Refresh access** on the GitHub connection.
4. See that the connection stays healthy and reports all repositories,
while agent pushes fail with 403.

**Paperclip version or commit**
`master` at a708a1333.

**Additional context**
`GET /user/installations` returns the suspended installation with
`suspended_at` set and `repository_selection: all`, and `GET
/user/installations/{id}/repositories` still lists its repositories.
Unsuspending the installation restored push access and private reads for
the same token.

## What Changed

- `server/src/services/tool-access/github-access-discovery.ts`:
`loadGitHubGrantMetadata` skips installations that have `suspended_at`
set. It does not add their IDs, owners, repository selection, or
repositories. When there is no usable installation and at least one is
suspended, it throws `github_installation_required` with a "suspended"
message. `managementUrl` points at the suspended installation's settings
page (or the generic installations page when more than one is
suspended).
- `server/src/services/tool-access/catalog/health.ts`:
`checkConnectionHealth` (the Refresh access path) keeps
`installationUrl` and `managementUrl` when it rebuilds a
`github_installation_required` error. Before, the links were dropped.
- `server/src/services/tool-access/catalog/failures.ts`:
`github_installation_required` keeps its own code and returns 422.
Before, it was folded into a generic `paperclip_error` with HTTP 502.
- `ui/src/pages/apps/app-detail/AccessRetryBanner.tsx` (new): the
attention banner with the saved health message and **Retry access**.
After a failed Refresh access with `github_installation_required`, it
shows an **Open installation settings on GitHub** link (https
`github.com` URLs only).
- `ui/src/pages/apps/AppDetail.tsx`: uses `AccessRetryBanner`, refetches
the connection when a refresh fails so the banner shows the saved
message, and keys the page on `connectionId` so a recovery link never
carries over to another connection. The file stays under 1,000 lines.
- Tests:
- `server/src/__tests__/github-grant-metadata.test.ts`: a mix of
suspended and active installations, and an account where all
installations are suspended.
- `server/src/__tests__/github-health-recovery.test.ts`: drives
`checkConnectionHealth` with a suspended-installation GitHub response
and checks the 422, code, and recovery links.
- `ui/src/pages/apps/AppDetail.test.tsx`: a failed Retry access renders
the installation settings link, and navigating to another cached
connection does not show it.

## Verification

- Red/green: with the source changes reverted and the tests kept, the
new tests fail (suspended installation counted as `all`; recovery link
missing from the health-check error; link not rendered; link leaking to
another connection). With the changes, they pass.
- `server`: `github-grant-metadata`, `github-health-recovery`,
`tool-access-service`, `github-operation-credentials`,
`tool-access-oauth-protocol.integration`, `tool-connection-removal`,
`tool-connection-visibility`, and the other `github-*` / `tool-access-*`
suites pass locally (630 tests).
- `ui`: `AppDetail.test.tsx` passes (78 tests). `ui` `tsc --noEmit`
shows no errors in changed files.
- Not run locally: the server `tsc`. The sandbox has a 4 GB memory
limit, and the server typecheck is killed there even on clean `master`.
CI ran the full typecheck and build, and both pass.
- CI: all gates green. One unrelated `workspace-recovery.test.ts` poll
timeout in a server shard passed on re-run.
- Live check: after the suspended installation was unsuspended, a
managed-token `git push --dry-run` succeeded and a private repository
returned 200.

## Risks

- Low risk. The change only affects installations that GitHub marks as
suspended. Active installations behave as before.
- A connection that has only suspended installations now fails Refresh
access with `github_installation_required` (422) and does not report
fake access. This is the intended behavior.
- The recovery link only renders for https `github.com` URLs.
- Keying the app page on `connectionId` remounts it when moving between
connections, which resets per-page state (for example an open rename
field). That reset is intended.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through the
Claude Code agent SDK with tool use (shell, file edit, code execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-26 21:16:40 -07:00
DottaandPaperclip 9db3a5726e refactor(tools): isolate GitHub access discovery (#17144)
## Thinking Path

> - Paperclip manages AI agents and their access to external tools.
> - Tool connections discover repositories through GitHub account
grants.
> - The tool-access service mixes GitHub response parsing with grant
authorization and persistence.
> - These responsibilities have different inputs and failure rules.
> - This PR puts GitHub discovery in one provider module.
> - The service keeps current authority checks and stored grant updates.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

The ownership and review of GitHub access discovery in the tool-access
service.

**Current behavior**

The 21,525-line service contains GitHub pagination, account metadata
validation, and grant lifecycle code.

**Proposed behavior**

A GitHub module returns normalized access metadata. Existing service
exports remain compatible. This is the first PR in a three-part refactor
stack. No behavior change is intended. A search found no existing
tool-access refactor PR or issue.

Stack order: this PR, #17146, then #17147. Each PR targets its
predecessor.

## What Changed

- Move six functions into `tool-access/github-access-discovery.ts`.
Their bodies are unchanged.
- Keep grant selection, credential resolution, refresh retry, membership
locks, revision checks, and webhook updates in the service.
- Add public-entry characterization tests for HTTP error classification
and invalid pagination.
- Record five candidate boundaries and the initial three-PR scope in the
dated plan.
- Reduce `tool-access.ts` from 21,525 lines / 829,519 bytes to 21,237
lines / 820,116 bytes. The new module is 294 lines / 9,722 bytes.

## Verification

- Before and after: `pnpm exec vitest run
server/src/__tests__/github-grant-metadata.test.ts
server/src/__tests__/project-repositories.test.ts` — 13 tests pass.
- Before and after: `pnpm exec vitest run
server/src/__tests__/tool-access-service.test.ts -t 'GitHub|github'` —
11 selected tests pass with disposable PostgreSQL; 455 cases are outside
this selection.
- `pnpm --filter @paperclipai/server typecheck` passes.
- Exact text comparison confirms all six moved bodies match the base.
`git diff --check` passes.
- The initial baseline attempt failed because the plugin SDK had not
been built. The normal workspace prerequisite build resolved it before
behavioral validation.
- Final-stack local verification: `pnpm -r --workspace-concurrency=2
typecheck` and `pnpm build` pass. CI is green at
`07e78dd1b1c26e0bd4234006766444c71e234451`. Greptile is 5/5 with no
actionable findings.

- Full local-suite limitation: `pnpm test:run` was interrupted after 79
minutes during the serial general-server phase, after CI had completed
the full test matrix. Before interruption, it flagged the unchanged
`company-search-service.test.ts` suite (26 database cases skipped). One
isolated rerun passed all 28 cases without code changes. The original
log and a pre-interruption copy are preserved. The original failure
cause was not reproduced. The full local suite and its later groups are
not claimed as passed; the complete CI matrix is green on this stack.

## Risks

- Import wiring could break existing consumers. Existing imports remain
supported and all four production call sites are unchanged.
- Provider requests retain their ordering, timeout, fixed GitHub origin,
and response redaction. The new module has no database dependency or
shared mutable state.
- Tests simulate GitHub responses. They do not qualify live GitHub
access.

## Model Used

OpenAI Codex, GPT-6 family, with code execution and repository tools.
The exact serving model ID and context-window size are not exposed in
this session.

## Checklist


- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-21 22:21:40 -05:00