mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-12 14:19:00 +01:00
DottaandPaperclip
b94963f91e
## Thinking Path > - Paperclip manages AI agents and their work. > - A task must check repository access before it starts agent work. > - Repository discovery can fail while the user's existing grant remains valid. > - Discovery previously replaced a known temporary provider refusal with an empty catalog. > - The task then reported lost access and did not schedule recovery. > - This change keeps temporary discovery failures distinct and uses the existing bounded retry owner. > - Each retry must pass a fresh access check before it can use the saved repository. ## Linked Issues or Issue Description **What happened?** A task with a saved catalog repository stops with an access denial when GitHub returns an exhausted-quota response or HTTP 503. The task does not start provider work, but the discovery layer discards the temporary failure and returns an empty catalog. **Expected behavior** Known temporary failures before provider dispatch use the existing bounded infrastructure retry. The successor checks the same responsible user's current grants and reads the saved files only after fresh discovery succeeds. A completed denial remains terminal. **Steps to reproduce** 1. Bind a task to an authorized catalog repository with saved local changes. 2. Make its authorized GitHub discovery return HTTP 403 with zero remaining quota, or HTTP 503. 3. Let discovery recover before the existing retry deadline. 4. Observe one successful successor using the original repository bytes. **Paperclip version or commit** Reproduced on master at `b8d94b3ad1`. A search found no existing PR for this temporary catalog failure. ## What Changed - Keep a safe typed failure for known HTTP availability and quota responses. Honor the later provider deadline and bound it to one day. - Propagate that failure only when fresh discovery cannot find a required repository. Preserve successful partial catalogs and the existing connection setup error contract. - Classify the failure only before provider dispatch. Reuse existing bounded retry, interaction ownership, and bootstrap settlement rules. - Keep current grant checks, saved repository selection, normal Stop, and provider-started failure policy. - Keep provider deadlines when accepted interactions use their own retry owner. Recognize secondary HTTP 403 rate limits with valid retry advice while keeping ordinary permission denials terminal. This matches [GitHub rate-limit guidance](https://docs.github.com/en/rest/using-the-rest-api/rate-limits-for-the-rest-api). - Keep direct repository request responses outside the automatic bootstrap retry contract. Test successful partial discovery and missing-required-repository recovery with two independent eligible connections. - Document task repository discovery recovery. ## Verification - Behavioral RED: real heartbeat and PostgreSQL fixtures reached actual HTTP 403 and503 discovery responses, started no adapter, and found zero successors instead of one. Both cases failed before the production change. - Behavioral GREEN: both cases waited for the real retry deadline, used the normal queue worker, ran the controlled adapter once after fresh discovery, and read the original saved bytes. Two cases passed in 74.73 seconds. No external model or provider request was made. - Deadline RED/GREEN: a combined Retry-After and quota-reset response initially selected the earlier deadline. The regression now selects the later deadline. - Initial repair at `d8d4714a18` (same production source as `018e8c52d345d79972d7907468080dc9908c299f`): seven affected files passed, 69 cases in 185.16 seconds. That test-only cleanup then passed all five real-database acceptance cases in 134.76 seconds. - Commands: `pnpm exec vitest run` with `heartbeat-repository-catalog-recovery.test.ts`, `github-catalog-timing.test.ts`, `github-grant-metadata.test.ts`, `project-repositories-persistence.test.ts`, `task-workspace-repositories-persistence.test.ts`, `github-repository-catalog-failure.test.ts`, and `services/heartbeat/retries.test.ts` from the server source tree. The acceptance file also ran fresh at `018e8c52d3`. A fresh five-case run after the P1 repairs passed at `f490ca6b78` in 122.06 seconds. The final eight-case file passed fresh at `e5b2efab11919467850f55fd6345317132b9b672` in 124.62 seconds, using real owned embedded PostgreSQL. It covers automatic recovery with saved bytes, genuine denial, provider-started classification, normal Stop, the direct HTTP route, and both partial-catalog outcomes. - Review repairs at `f490ca6b787cf77df96d3d6edfb2ab9115eb19a6`: both reported P1s reproduced RED. The real database interaction scheduler chose a retry after 24 seconds before the provider deadline of one hour. A secondary HTTP 403 with valid Retry-After lost its retry classification. The narrow fixes then passed 44 cases across scheduler, header, and metadata-contract files in 37.66 seconds. - A real direct request through the task workspace router and normal error middleware reproduced a 500 response instead of the existing 403. Required-ID retry mode now applies only to admission and its rechecks. The route and both two-connection partial-catalog cases then passed in 29.09 seconds. Both partial cases use actual owned grants and credential bindings. - All seven server type layers passed again at final head `e5b2efab11919467850f55fd6345317132b9b672`. The affected test type layer passed again after the test-only cleanup. Environment: macOS arm64, Node 26.4.0, pinned pnpm 9.15.4, Vitest 5.0.3, owned disposable PostgreSQL 18 on loopback. The current checkout's shared packages, plugin SDK, and Runner were built first. - One local acceptance run under concurrent owned test/typecheck load hit the unchanged 15-second Stop-case timeout. Its four other cases passed. The unchanged complete file then passed serially. The timeout receipt remains recorded; no assertion or timeout was weakened. - A provider-started failure retains the existing generic adapter retry policy. It does not receive the catalog no-provider certificate or a provider retry deadline. Fixture cleanup cancels its scheduled retry through the normal owner. - Final head `e5b2efab11919467850f55fd6345317132b9b672` has 59 successful checks, 3 allowed skips, no pending or failed check, and Greptile 5/5 with no open findings. [Completed CI workflow](https://github.com/paperclipai/paperclip/actions/runs/42322202977). - Additional local diagnostic for the late recovery-owner advisory: on unchanged `e5b2efab`, the dedicated scheduler returned `retry_exhausted` with attempt 3/max 2. The normal stranded sweeper then escalated once, created no extra run, and started no adapter, even after catalog access recovered. One case passed in 17.32 seconds. The existing failed-predecessor path delegates to that bounded scheduler and returns; it does not enter a new generic retry budget. No production registry change was made. This diagnostic was not a CI case. Its local harness is preserved separately at `74e3854eea3f7c4121fdcd9c8a8ad522cd8c7477` on `codex/catalog-exhaustion-acceptance`; its production source is identical to `e5b2efab`. - Module boundary and whitespace checks passed. ## Risks - A temporary failure from another eligible connection can delay a truly missing repository for the existing limit of two attempts. It cannot grant access. - Unknown network errors and genuine access denials retain their existing behavior. This change handles known HTTP failures only. - Historical provider quota causality is unproven. This repair has a separate local producer regression. It does not rewrite past failures or queued user intents. - Acceptance uses real local database and heartbeat owners with controlled HTTP and adapter fixtures. It does not claim a live native-provider recovery. ## Model Used OpenAI GPT-6.1 Codex (`gpt-6.1-sol`), with high reasoning, repository tools, code execution, and controlled local PostgreSQL tests. The host did not expose a context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
110 lines
7.0 KiB
Markdown
110 lines
7.0 KiB
Markdown
# Project source repositories
|
|
|
|
The Create project dialog accepts a name and optional GitHub repository selections.
|
|
It uses the same `RepositoryEditor` as project Configuration. Description remains
|
|
editable in Configuration. Status, goal links, and target dates remain supported by
|
|
the API but are omitted from creation; Status and Goals are omitted from Configuration.
|
|
Old Overview URLs and saved Overview preferences redirect to Configuration.
|
|
|
|
## API and persistence
|
|
|
|
- `GET /api/companies/:companyId/project-repositories` returns `repositories`,
|
|
`connectionCount`, and `failedConnectionCount`. Repository IDs are GitHub's stable
|
|
numeric IDs represented as strings. Results are deduplicated across accessible
|
|
grants and sorted by full name. Connection labels are display provenance only.
|
|
- `POST /api/companies/:companyId/projects` accepts optional `repositoryIds`.
|
|
The server resolves new selections through the caller's authorized GitHub grants
|
|
before creating the project and all repository workspaces in a transaction.
|
|
The existing `workspace` input remains supported; it cannot be combined with
|
|
`repositoryIds`.
|
|
- `PUT /api/projects/:id/repositories` accepts the selected `repositoryIds` array.
|
|
Accessible retained IDs refresh their canonical name and URL after renames or
|
|
transfers; unavailable retained IDs keep their saved metadata.
|
|
Replacement is transactional. Existing selections may be retained or removed even
|
|
if their GitHub connection becomes unavailable. New identities require current
|
|
access. Legacy URL workspaces are preserved, and matching legacy URLs are adopted
|
|
without creating a duplicate workspace. Local/remote workspace locations survive
|
|
detaching their repository.
|
|
|
|
No schema migration is required. Selected repositories are normal project workspaces
|
|
with `metadata.githubRepositoryId`. Existing manual `repoUrl` workspaces remain
|
|
editable through Configuration and the workspace API. One workspace remains primary.
|
|
Tasks materialize the other distinct repositories as editable checkouts inside their
|
|
workspace, including when no local folders are configured. Local execution and sandbox
|
|
staging use the same layout; sandbox restore preserves each repository's Git history.
|
|
See [Project Repository Checkouts](DEVELOPING.md#project-repository-checkouts) for paths,
|
|
ignore rules, and reuse behavior. Responsible-user credential rules still apply.
|
|
A repository selection never delegates credentials.
|
|
|
|
Fresh tasks using the default repository workspace can clone the same authorized
|
|
remote into a company- and task-scoped source directory when an existing project
|
|
checkout contains another task's restricted workspace. The clone uses the run's
|
|
Git connection and keeps separate Git metadata. Source access checks still apply
|
|
before the task workspace is published. Low-trust review runs cannot use this
|
|
persistent clone fallback. This fallback does not copy unpublished
|
|
local changes or replace reused workspaces, explicit workspace selections, pinned
|
|
branches, or repositories that only exist on the local filesystem.
|
|
Configured branch templates must derive the branch from the task identifier, ID,
|
|
title, or slug. Literal or task-independent templates retain the source denial,
|
|
so a fresh clone cannot replace a selected local branch with a new empty branch.
|
|
New implicit Git workspace defaults record their project-source provenance.
|
|
An untyped checkout binding from an older task is ambiguous and keeps the denial,
|
|
including explicit checkout IDs and inherited parent sources. Admission rechecks
|
|
the current binding rather than inferring a default from the selected checkout ID.
|
|
When the denial is retained, the run stops before the agent starts with a
|
|
`configuration_incomplete` failure (reason `workspace_source_restricted`). The task
|
|
is blocked with a notice that tells the owner to push unpublished work and select a
|
|
new workspace, or to grant access to the restricted task, then retry. Paperclip
|
|
does not report this owner action to Sentry as an unexpected run failure.
|
|
This applies to both isolated-source checks and repository admission for a shared
|
|
project workspace. Neither path replaces the existing workspace or retries the
|
|
same access denial automatically.
|
|
Startup validation of these recorded defaults keeps the task's effective workspace
|
|
strategy. It does not substitute the agent's shared strategy or grant project writes.
|
|
Workspace close and reopen use the recorded isolated source. Both operations
|
|
verify its company, task, repository URL, and separate Git storage before use.
|
|
|
|
## Discovery and setup
|
|
|
|
The server checks company membership, grant ownership/status, and organization-grant
|
|
audiences before loading provider metadata. Connection managers receive no bypass to
|
|
another person's personal repositories. Managed GitHub grants refresh installation
|
|
access; PAT connections use paginated `/user/repos`. Provider failures are reported
|
|
without exposing provider error bodies or credential material. Successful connections
|
|
remain selectable when another connection fails.
|
|
|
|
Task repository preparation distinguishes a temporary provider refusal from a
|
|
completed catalog that denies access. Before provider dispatch, known HTTP
|
|
availability failures and exhausted quotas use the existing bounded infrastructure
|
|
retry and provider deadline. Each successor discovers repositories again under the
|
|
same responsible user's current grants. Empty successful catalogs, revoked grants,
|
|
and permission denials remain terminal. A partial catalog can authorize a requested
|
|
repository only when that repository is present.
|
|
|
|
`ConnectionSetupFlow` owns provider setup in both Apps and project dialogs. Task
|
|
intents retain their existing callback protocol. Standalone dialogs verify the saved
|
|
connection through the API after the sign-in popup returns to the instance. Project
|
|
name and repository drafts stay mounted across setup and cancellation.
|
|
|
|
## UI review and verification
|
|
|
|
`Proposals/Project repos` contains the reviewed states, including loading, failure,
|
|
empty search, disconnected GitHub, multiple repos, legacy URLs, forty selections,
|
|
mobile, and short viewports. The configuration story composes the production page
|
|
properties through an explicit repositories slot. Story setup and saves use fixtures.
|
|
|
|
- Shared visual control: `ui/src/components/RepositoryEditor.tsx`.
|
|
- Data and error handling: `ProjectRepositoryInput.tsx`.
|
|
- Configuration persistence and legacy editing: `ProjectRepositories.tsx` and
|
|
`LegacyProjectRepository.tsx`.
|
|
- Production dialog: `NewProjectDialog.tsx`.
|
|
- Server tests: `project-repositories.test.ts` and
|
|
`project-repositories-persistence.test.ts`.
|
|
- Browser acceptance: `tests/e2e/project-repositories.spec.ts`.
|
|
|
|
The browser suite uses a real temporary server and database. It verifies creation,
|
|
forty persisted repos, mobile scrolling, removal/save/reload, legacy URL editing,
|
|
and rejection without a partial project. Provider discovery is simulated in the
|
|
picker rejection test. GitHub network and popup behavior use deterministic fixtures
|
|
in integration/component tests; the suite does not authorize a real GitHub account.
|