Commit Graph
2464 Commits
Author SHA1 Message Date
DottaandPaperclip 9c442cf48e feat(connections): deploy and configure Supabase and Vercel (#17217)
Add governed Supabase and Vercel deployment/configuration connectors with vaulted credentials, agent access policies, approvals, audit, and bounded provider operations.

Deliver reviewed REST actions to native agents and guard Vercel preview intent on empty projects. Preserve secret-safe request identity and durable retry behavior.

Verified by current-head CI, fresh Greptile 5/5 review, and real-agent deployment/configuration walkthroughs on disposable provider resources. Test resources and temporary provider tokens were removed after verification.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-23 11:37:29 -07:00
Devin FoleyandPaperclip f406b32d08 perf: narrow run authorization reads (#17348)
## Thinking Path

> - Paperclip lets people manage and observe AI agents at work.
> - Live events and run history must enforce current task permissions.
> - Some permission checks load the complete run record for each event
or request.
> - Run context, runner configuration, results, and output can make
those records large.
> - This pull request reads only the persisted fields each access check
needs.
> - The checks still read current permissions and fail closed.

## Linked Issues or Issue Description

Related: #17251, #9155, and #16229. Those changes cover other run
projections, recovery polls, and task history. This change covers
live-event delivery, workspace-operation run checks, and event-list
access.

**What happened?**

Live-event delivery loaded every run column for each subscribed viewer,
although the permission helper used only the company, run scope, and
task binding. Workspace-operation checks did the same. The run-event
endpoint also loaded complete run records before returning separate
event rows. Large contexts and runner configurations therefore increased
database transfer for ordinary access checks.

**Expected behavior**

Read only the stored access binding. Keep company checks, permission
freshness, deleted-task tombstones, and response bodies unchanged.

**Steps to reproduce**

1. Create an issue-bound run with large context, runner configuration,
and output fields.
2. Subscribe an authorized viewer to company live events and publish run
updates.
3. Read the run events and a linked workspace operation.
4. Inspect the run SELECT projections. The previous queries load fields
that the access checks do not use.

**Paperclip version or commit**

The defect is present on the current master base, `62c78e56e53`.

**Deployment mode**

Authenticated server with PostgreSQL. The defect is not
adapter-specific.

## What Changed

- Select only company, scope, and issue bindings for live-event and
workspace-operation run authorization.
- Add a four-field run-access read for event-list requests. Reuse those
columns in the existing log-access projection.
- Preserve per-delivery and per-request permission checks, company
predicates, and event responses.
- Add real PostgreSQL and WebSocket coverage with large run fixtures.
Cover grant revocation, grant restoration, deleted tasks, missing and
foreign-company runs, and failed reads.
- Document the bounded access reads in the existing polling-load guide.

## Verification

- Six focused suites pass: 40 tests. They cover the changed paths,
run-state reads, run logs, disconnect retries, and native event replay.
- The new WebSocket test uses the production delivery path and a real
database. It verifies that the selected SQL omits large run fields while
authorization changes take effect on the same open socket.
- The event-route test preserves the returned event content and denies
the next request after grant revocation.
- Local recursive typecheck, full build, and module-boundary checks
pass. [Exact-head
CI](https://github.com/paperclipai/paperclip/actions/runs/39997370119)
completed successfully, covering the full grouped unit suites,
typecheck, build/runtime boot, and E2E shards. The separate serial local
`pnpm test:run` is still running and is not claimed as completed.
- A later CI shard failed an unchanged email test expecting exactly one
task after permission grants. The unchanged-base test passes at normal
speed; adding only a 2.2-second pause before granting access reproduces
the extra tasks because previously denied deliveries become due for
retry. No email code or assertions are changed in this PR. The original
failure is retained. The single shard rerun passed: 92 suites and 1,489
tests passed, with one suite and two tests skipped by the existing
gates.
- CI initially found a route-test double missing the new access helper.
The follow-up fixes that fixture and adds event reads to malformed-ID
and missing/cross-company checks. All 122 tests across the four
changed-path suites pass; a final rerun of the expanded route suite
passes all 92 tests.
- The first CI run also timed out in the signoff changes-requested
handoff while waiting for a reviewer run after a cancelled execution
blocker. The focused test passes locally on this source. The same
focused test passes three times on unchanged master. Controller-lease
cancellation messages appear on both versions. The original CI timeout
is preserved as an intermittent, unreproduced failure; its exact cause
remains unknown. The signoff shard passes in the final exact-head CI
run.
- Initial fresh-checkout tests could not start because workspace SDK
output and native PostgreSQL library links were missing. Dependency
builds and the package's native-link setup corrected those
prerequisites. The subsequent focused run passed.

Focused command:

```sh
pnpm --filter @paperclipai/server exec vitest run src/__tests__/live-events-run-read.test.ts src/__tests__/heartbeat-run-privacy-routes.test.ts src/services/heartbeat/run-state.test.ts src/__tests__/native-observer-replay.test.ts src/__tests__/heartbeat-run-log.test.ts src/__tests__/heartbeat-run-privacy-disconnect-retry.test.ts
```

## Risks

- A missing access field could cause an incorrect allow or denial. Tests
exercise current grants, revocation, company scope, and deleted-task
bindings through the real consumers.
- No successful permission decision is cached. Every delivery and
request still checks current database state.
- No schema, migration, polling interval, API response, or deployment
configuration changes are included.
- This reduces bytes for the affected reads. It does not establish a
whole-account savings percentage. Hosted transfer measurements remain
separate validation after staging deployment.

## Model Used

OpenAI Codex, a GPT-6 based coding agent, used repository tools, code
execution, and database-backed tests. The session does not expose the
exact backend model identifier or context-window size; those values are
not guessed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-23 11:01:06 -07:00
Devin FoleyandPaperclip 62c78e56e5 fix: describe unexpected git expiration failures in the result (#17341)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Git cache expiration keeps a workspace and reports
`inspection_failed` when an unexpected error interrupts its checks
> - It dropped that error, so the log never said what went wrong
> - This pull request adds a sanitized description of the error to the
expiration result, which the workspace reaper already logs
> - The benefit is that unexpected expiration failures can be diagnosed
without exposing commands or credentials

## Linked Issues or Issue Description

Refs #17248. Bug report described in-PR (bug report template).

**What happened?**
On one instance, a closed workspace was kept with `inspection_failed`.
The log entry carried only that reason, so the cause could not be
determined from the logs.

**Expected behavior**
An unexpected failure is logged with enough detail to diagnose it,
without leaking the command line or credentials.

**Steps to reproduce**
1. Break a closed issue's worktree so a git command fails, for example
by pointing its `.git` file at a missing directory.
2. Run expiration: the log says `inspection_failed` and nothing else.

**Paperclip version or commit:** `master` at 0fb152ee94.
**Deployment mode:** any.
**Agent adapter(s) involved:** not adapter-specific (core).

## What Changed

- `GitExpirationResult.failure` is set when the skip reason is
`inspection_failed`, by all three places that record a skip (the outer
catch and the two base-inspection catches, through one `recordSkip`
helper). It contains:
  - `name`;
  - `code` (for example git's exit code);
  - `at` (the first stack frame);
  - `stderr` (up to 500 characters).
- `describeGitExpirationFailure` builds it. It never includes the error
message, because a failed git command's message contains the full
command line, which can include `http.extraHeader` authentication. In
stderr, credentials in URLs and `authorization`, `token`, `password` or
`secret` values are redacted to the end of the line.
- The workspace reaper already spreads the result into its `Git
workspace cache expiration inspected` log entry, so no logging code
changed.

No code comments were added.

## Verification

- `pnpm exec vitest run src/__tests__/git-workspace-expiration.test.ts
-t "unexpected inspection failure|redacts credentials|while inspecting
the base"` (in `server/`): 4 passed.
- **Broken worktree:** a worktree whose `.git` link points at a missing
directory reports `inspection_failed` with a code, a stack frame and
git's "not a git repository" stderr, and is kept.
- **Failing command:** a real failing `git ls-remote` with credentials
in its arguments produces a description without them or the command.
- **Echoed credentials:** a stderr that echoes a credentialed URL and an
`Authorization: Bearer` header is redacted.
- **Base inspection:** an unreadable directory in the base, which fails
base inspection after the worktree is removed, reports
`inspection_failed` with code `EACCES` and a stack frame. Recording only
the reason in that catch fails this case.
- **Mutation check:** removing the redaction fails the
echoed-credentials case. That case also caught an earlier version that
redacted only the first word of a header value.
- `pnpm exec vitest run git-workspace-expiration execution-workspaces
git-restore-point` (in `server/`): 258 passed.
- `pnpm run typecheck` (in `server/`): pass.
- Not run: server build and boot smoke.

## Risks

- **Log volume:** one bounded object (about 1 KB at most) per failed
inspection.
- **Redaction is pattern-based:** the message and command line, where
git places configuration values, are never logged; stderr is redacted by
pattern.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file edits, tests, read-only inspection of a live
instance).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-23 10:27:35 -07:00
Devin FoleyandPaperclip 0001a4514e fix: keep stale pending restore points from blocking retries (#17344)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Git cache expiration saves restore points and tracks them in
`execution_workspace_restore_points`
> - A unique index allows one row per workspace, generation and checkout
unless the row is deleted
> - A pending row left by an interrupted attempt therefore blocked every
retry for an hour, and superseded rows blocked retries until retention
deleted their objects
> - This pull request narrows the index to live rows and supersedes
stale pending rows before inserting
> - The benefit is that an interrupted expiration no longer stalls a
workspace's cleanup

## Linked Issues or Issue Description

Refs #17257. Bug report described in-PR (bug report template).

**What happened?**
On one instance, a closed workspace with a nested checkout failed
expiration with `inspection_failed` on every sweep. An earlier attempt
had left a `pending` row for the nested checkout at the workspace's next
generation. Each retry stored the main worktree's restore point, then
failed to insert the nested row because the unique index on
`(execution_workspace_id, generation, checkout_name) WHERE status <>
'deleted'` still held the stale row. Running the expiration by hand
reproduced the failing insert.

**Expected behavior**
A retry after an interrupted attempt replaces the stale row and expires
the workspace.

**Steps to reproduce**
1. Leave a `pending` restore point row for a workspace's next generation
and a nested checkout, as an interrupted attempt does.
2. Run expiration with restore points: it fails with `inspection_failed`
until retention supersedes the row after the one-hour pending grace
period.

**Paperclip version or commit:** `master` at 0fb152ee94.
**Deployment mode:** any.
**Agent adapter(s) involved:** not adapter-specific (core).

## What Changed

- **Schema**
(`packages/db/src/schema/execution_workspace_restore_points.ts`,
migration `0375`): the unique index is renamed
`execution_workspace_restore_points_live_status_uniq` and covers only
`pending`, `active` and `expiring`. Superseded, delete-requested and
deleted rows no longer block a new row.
- **Storage** (`storeGitRestorePoint`): in the transaction that inserts
the new `pending` row, any `pending` row for the same workspace,
generation and checkout is first marked `superseded`. The expiration
holds the workspace lifecycle lock, so such a row can only come from an
attempt that already ended. Retention then deletes its object as usual.

No code comments were added.

## Verification

- `pnpm exec vitest run src/__tests__/git-workspace-expiration.test.ts
-t "stale pending restore point"` (in `server/`): 1 passed. A workspace
with a stale `pending` nested row and a `superseded` main row from an
earlier attempt expires with a nested restore point, and the stale row
is superseded.
- **Mutation checks:** removing the supersede step, or keeping the old
index scope in the migration, each reproduces the instance's
`inspection_failed`.
- `pnpm exec vitest run git-restore-point git-workspace-expiration
git-workspace-contents git-workspace-restoration
git-reproducible-managed-checkouts execution-workspaces
heartbeat-git-restore-point-resume` (in `server/`): 274 passed.
- `pnpm run check:migrations` (in `packages/db`): pass.
- `pnpm run typecheck` (in `server/`): pass.
- Not run: server build and boot smoke.

## Risks

- **Migration:** drops and recreates one partial unique index on a small
table. It cannot fail on existing data, because the old index was
stricter. The index is briefly absent while the migration runs.
- **Superseding a pending row:** only rows for the same workspace,
generation and checkout, while the current attempt holds the workspace
lifecycle lock.
- **Not addressed:** why the earlier attempt left the row behind. #17341
logs the underlying error of `inspection_failed` so the next occurrence
can be diagnosed.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file edits, tests, read-only and dry-run
inspection of a live instance).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-23 10:25:59 -07:00
23bd238f05 fix(costs): stop OAuth refreshes from duplicating unverified subscriptions (#16512)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Costs page shows one card for each AI subscription account. Each
card comes from an `ai_subscriptions` row.
> - Paperclip cannot verify an xAI/Grok account with the provider. So
each Grok row used a key made from a hash of the raw credential.
> - The Grok CLI refreshes its OAuth tokens during runs. Paperclip
writes the refreshed `auth.json` back to the vault and keeps the
secret's `aiSessionEpoch`.
> - The next run sees a new credential hash.
`subscriptionService.register()` finds no match and adds a new
subscription row. The old row stays on the Costs page. One Grok
connection becomes many duplicate cards.
> - Claude and OpenAI rows usually avoid this problem, because a
provider profile check merges them. Grok rows never get verified, so
nothing merges them.
> - This pull request keys unverified rows on the credential session
(secret id + `aiSessionEpoch`), not on the token bytes.
> - The benefit is one subscription card per Grok connection through
token refreshes. A real reconnect still gets its own card.

## Linked Issues or Issue Description

No public issue exists. Bug description:

**What happened?**
On a hosted staging instance, the Costs → Subscriptions list showed many
duplicate Grok subscriptions for one Grok connection. Each Grok OAuth
token refresh added one more card.

**Expected behavior**
One Grok connection shows one subscription card. A token refresh does
not change it.

**Steps to reproduce**
1. Connect an xAI/Grok subscription account.
2. Run an agent until the Grok CLI refreshes its OAuth token. Paperclip
writes the refreshed credential back to the vault.
3. Run the agent again.
4. Open Costs → Subscriptions. A second Grok card is there. Each refresh
adds another card.

**Agent adapter(s) involved**
Not adapter-specific (core bug). The trigger is the Grok OAuth refresh
write-back in `server/src/services/ai-connection-runtime.ts`.

**Paperclip version or commit**
`master` at 8aaef59.

## What Changed

- `server/src/services/subscriptions.ts`: For an unverified identity,
`register()` now reads the grant's `ai.credential` secret ref and that
secret's `aiSessionEpoch`. The account key becomes `grant + session +
secretId + epoch`. Before, it was `grant + credential hash`.
- Refresh write-backs use `preserveAiSessionEpoch: true`, so a refreshed
token keeps the same row.
- A reconnect advances the epoch, so it still gets a separate row. A
reconnect can sign in to a different account.
- A grant with no vaulted secret ref falls back to the old
credential-scoped key.
- Unverified rows stay grant-scoped. Only a verified provider
observation can match the billing identity of a different grant.
- Stale sessions are rejected. `SubscriptionConnection` now carries the
`session` (secret id + epoch) captured together with the credential. If
that session does not match the grant's current secret and epoch,
`register()` returns `409` and does not use the live epoch. Before this,
a run could read account A's credential, pause, and register after a
reconnect to account B. It then got B's subscription id, so A's usage
was saved against B.
- `server/src/services/ai-connection-runtime.ts`: The run passes the
secret id and epoch from its existing freshness check (read together
with the credential).
- `server/src/services/subscription-refresh.ts`: Discovery reads the
session before it reads the credential. Then a reconnect at any time
after that read makes `register()` reject the stale session.
- `server/src/__tests__/subscriptions.test.ts`: Added regression tests.
- Two refreshes keep one row and its saved price. A reconnect (epoch
bump) makes a second row, and a refresh after that stays on the second
row.
- A registration with an older epoch, or with no session, is rejected
with `409`. The grant binding stays on the current account.
- `server/src/__tests__/subscription-refresh.test.ts`: The module mock
now exports `subscriptionSession`.
- This pull request adds no code comments. The explanation is in this
description.

## Verification

Run on `master` + this branch (head `11900e4`):

- `cd server && npx vitest run src/__tests__/subscriptions.test.ts
src/__tests__/subscription-refresh.test.ts
src/__tests__/subscription-routes.test.ts
src/__tests__/ai-connections.test.ts
src/__tests__/ai-connection-session.test.ts
src/__tests__/costs-service.test.ts` → 6 files, 191 tests pass.
- The refresh test fails on `master` without the service change. The
refreshed token gets a new subscription id (`expected '6683…' to be
'942b…'`).
- The stale-session test fails when the guard is removed. The stale
registration resolves to the reconnected account's id instead of
rejecting.
- `pnpm -r typecheck` passes for every package except `server` and
`cli`. The runner passes `typecheck:typescript`.
- I could not complete some checks in my sandbox. CI must confirm them:
- The `server` and `cli` typechecks hit the 4 GB sandbox memory limit
(OOM kill). A narrow `tsc --noEmit` of `subscriptions.ts` and its
imports passes with 0 errors. A narrow check of
`ai-connection-runtime.ts` and `subscription-refresh.ts` also hits the
memory limit, because they import most of the server.
  - The full `pnpm test:run` hit the same memory limit.
- `pnpm build` and the runner `typecheck:rust` need `cargo`, and the
sandbox does not have it. This change does not touch Rust or the runner.

Manual check after deploy: Connect Grok and run an agent through a token
refresh. Then make sure that Costs → Subscriptions shows one Grok card.

## Risks

- Low risk. There is no schema change and no migration.
- Existing unverified rows keep their old credential-scoped keys. On the
next refresh after deploy, each existing Grok connection can get one
more row in the new key format. After that, it stays stable.
- This change does not remove duplicate rows that exist now. You can
merge them with the **Link** control on each Costs card. A separate data
repair is planned.
- `register()` does one more indexed read of `company_secrets` (by id)
when a run registers an unverified identity. Discovery does one more
read of that row for each connection. The cached-binding fast path does
not change.
- A run that loses the race with a reconnect now fails its preparation
with `409` ("The AI credential changed. Retry this execution."). This is
the same retry behavior as the existing credential-freshness guard.
Discovery logs the conflict and checks again on the next pass.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, run through
Claude Code as a Paperclip agent with tool use (shell, file edit, test
execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (targeted suites; the full
suite is not complete, see Verification)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes (no
documentation change is necessary)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green (pending CI)
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
(pending review)
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Amy (Opus) <noreply@paperclip.ing>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com>
2026-10-23 10:21:19 -07:00
Nicky LeachandPaperclip a708a13336 Separate plugin workspace completion from maintenance inspection (#17329)
## Thinking Path

> - Paperclip manages AI agents and their work.
> - Storage plugins own remote task workspaces.
> - Run completion currently requests the same inspection used for
storage cleanup.
> - That inspection can fetch remote refs and scan files after the agent
has finished.
> - Completion needs retained workspace identity and durable writes, not
permission to delete files.
> - This change adds an optional completion operation and keeps
maintenance inspection separate.

## Linked Issues or Issue Description

**What happened?**

A successful plugin-backed run can fail during completion because the
storage provider performs a slow remote-recoverability inspection. Dirty
or unpublished output does not need a deletion-safety proof to remain
valid task output.

**Expected behavior**

Completion verifies the assigned workspace and confirms durable writes
through the provider. Cleanup still requires the existing full
inspection.

**Steps to reproduce**

Run a task with plugin-owned storage whose inspection fetches remote
refs. Let the agent finish, then observe completion waiting for the
maintenance inspection.

Related context: #16428 added finalization observability. The duplicate
search found no existing PR for this provider contract.

## What Changed

- Add optional `workspaceProvider.finalizationVersion: 1` and strict
`finalize` / `finalized` schemas.
- Pass original run and agent context through live completion and
recovered completion.
- Give the new operation a 30-second deadline and keep ownership checks
and lifecycle serialization.
- Retain the inspection fallback for providers without the capability.
Keep maintenance inspection and removal unchanged.

## Verification

- SDK build passed.
- Targeted SDK and completion unit tests: 7 passed.
- Added integration coverage for capability selection, legacy fallback,
maintenance separation, and failure propagation.
- Local integration setup timed out starting PostgreSQL; the integration
suite passed in GitHub CI.
- Full build/typecheck and broader tests passed in GitHub CI (55
successful checks, 2 skipped). Local checks remained targeted as
requested.
- Diff reviewed for secrets and operational identifiers.

## Risks

The capability is opt-in. Providers must not advertise it until their
completion implementation is available. An unavailable provider or
failed durability barrier remains retryable. No schema or stored-binding
migration is required.

Test-file size exception: the provider integration suite grows from 973
to 1,002 lines. The 29-line case follows the same prepared binding
through legacy completion, opted-in completion, and maintenance
inspection using the existing single-connection database fixture.
Keeping it there avoids a second embedded-Postgres lifecycle for one
transition test. The completion helper unit tests are in a separate
focused file.

## Model Used

OpenAI GPT-6 Codex, with reasoning and code/test tools.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-23 09:07:35 -07:00
Devin FoleyandPaperclip 0fb152ee94 fix: save stale remote-tracking commits in nested restore points (#17289)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Git cache expiration removes closed issues' worktrees once their
remote, or a restore point, can reproduce them
> - A nested `.paperclip-repositories` checkout's remote-tracking refs
can point at commits the remote has since dropped, such as a branch
deleted after a squash merge
> - Strict expiration rightly keeps such a checkout, because it may hold
the only copy; but restore point mode also kept it forever, because
there was nothing else to save
> - This pull request saves those commits in the checkout's restore
point, so the workspace can be reclaimed without losing them
> - The benefit is that long-lived multi-repository workspaces are
reclaimed in restore point mode

## Linked Issues or Issue Description

Refs #17257. Bug report described in-PR (bug report template).

**What happened?**
With restore points enabled, four closed workspaces on one instance were
kept with `unpushed_managed_checkout`, after a dry run had counted them
as candidates. Each nested `paperclip` checkout was clean, on `main` at
a commit that is on GitHub `main`. But it held about 1,000
remote-tracking refs, and between 28 and 41 of them pointed at branches
the remote has since deleted or force-pushed (147 to 203 commits it no
longer has).

The relaxed check passed, so the checkout needed no restore point beyond
those commits. Capture saved only `HEAD` history, so it found nothing to
save and stored no restore point. The final check before deletion then
ran the strict check, which keeps stale remote-tracking commits, and
rolled back. The dry run never reaches that final check, so it
over-reported. Together these workspaces held about 2.4 GB.

**Expected behavior**
In restore point mode, commits that exist only in remote-tracking refs
are saved, and the workspace expires.

**Steps to reproduce**
1. In a nested checkout, fetch a remote branch, so
`refs/remotes/origin/<branch>` exists.
2. Delete that branch on the remote and garbage-collect it.
3. With restore points enabled, close the issue and run expiration: the
workspace is kept with `unpushed_managed_checkout`.

**Paperclip version or commit:** `master` at 412914cfa7.
**Deployment mode:** any.
**Agent adapter(s) involved:** not adapter-specific (core).

## What Changed

- **Strict mode is unchanged.** An earlier revision of this pull request
ignored remote-tracking refs. Review pointed to `does not trust a stale
remote-tracking ref`, which deliberately protects those commits, so this
pull request keeps that behavior and the test is unchanged.
- **Relaxed nested check** (`assertReproducibleManagedCheckout(..., {
restorePoint: true })`):
- reads all reflogs in one bounded call (`--format=%H %gD`),
deduplicates them by commit, and splits them into local and
remote-tracking entries before applying the tip budget;
  - local entries still have to be covered or upstream, as before;
- remote-tracking tips and remote reflog entries that the remote no
longer has are returned as `preserve` (new `tipsMissingUpstream`,
sharing `commitsMissingUpstream` with the existing check).
- **Capture:** the proof is shared as `provenManagedCheckoutHistory`.
`storeGitRestorePoint({ resolvePreserveCommits })` calls it after taking
the history digest and before capture, then compares the digest again
after capture. A fetch before the digest is included in the preserved
commits, and one after it changes the digest and aborts capture. The
commits flow into `captureRestorePoint`. They are bundled as
history-only commits, within the existing limits (50 preserved tips,
1,000 commits, 256 MB bundle). The checkout therefore gets a restore
point, and the final check uses the relaxed check for it.
- **Final check:** `restorePointHistoryDigest` now also covers
remote-tracking refs and their reflogs, so a fetch, ref change or
reflog-only change after capture rolls expiration back.
- **Restore:** the commits are imported as objects; the stale
remote-tracking refs themselves are not recreated.
- **Docs:** `doc/recoverable-disk-cleanup.md`.

No code comments were added.

## Verification

- `pnpm exec vitest run src/__tests__/git-workspace-expiration.test.ts
src/__tests__/git-reproducible-managed-checkouts.test.ts -t "stale
remote-tracking|remote-tracking refs change"` (in `server/`): 5 passed.
  - The existing strict test passes unchanged.
- A clean nested checkout whose remote-tracking ref points at a deleted,
garbage-collected branch is kept strictly, expires with a restore point,
and reopens on its commit with the stale commit present.
- A remote-tracking ref created after capture rolls expiration back, and
so does a reflog-only change to one.
- 6,000 repeated reflog entries for one commit do not hit the tip
budget.
- **Mutation checks:** each of these fails a case:
- dropping the preserved commits reproduces the instance's symptom
(`unpushed_managed_checkout`);
  - dropping remote-tracking refs or their reflogs from the digest;
  - removing deduplication (`managed_checkout_local_tip_budget`).
- `pnpm exec vitest run git-restore-point git-workspace-expiration
git-workspace-contents git-workspace-restoration
git-reproducible-managed-checkouts execution-workspaces
heartbeat-git-restore-point-resume` (in `server/`): 261 passed.
- `pnpm run typecheck` (in `server/`): pass.
- Not run: server build and boot smoke.

## Risks

- **Only with restore points enabled.** Strict expiration is unchanged.
- **Retention:** saved stale commits are kept until the restore point
expires (90 days after the tree closes by default), then deleted with
it.
- **Limits:** checkouts with more than 50 stale tips, or bundles over
the limits, skip capture and stay kept, as before.
- **Spurious rollbacks:** a fetch during the short window between
capture and deletion rolls back; the next sweep retries.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file edits, tests, read-only inspection of a live
instance).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-23 08:24:53 -07:00
Devin FoleyandPaperclip 6877760904 feat: save ignored files in git restore points (#17290)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Git cache expiration removes closed issues' worktrees after saving a
restore point of anything the remote cannot reproduce
> - Restore points left ignored files out, so any worktree with an
ignored file that was not a known regenerable directory was kept
> - On one instance that was the most common reason a closed worktree
was still kept, mostly build caches and agent evidence files
> - This pull request saves those ignored files in the restore point too
> - The benefit is that more closed worktrees are reclaimed, and nothing
an agent left behind is lost

## Linked Issues or Issue Description

Refs #17248, #17257. Bug report described in-PR (bug report template).

**What happened?**
With restore points enabled, one instance kept 15 of 43 inspected closed
worktrees with `unique_ignored_files`. Among 19 such worktrees on disk:
- 9 held `ui/tsconfig.tsbuildinfo`;
- others held build output outside the regenerable directory names
(`server/ui-dist/`, `test-results/`);
- 2 held materialized skill copies under `.claude/skills/`;
- about 5 held small files agents kept out of git: patches, evidence
archives, recovery receipts.

**Expected behavior**
Ignored files that are not regenerable are saved with the restore point
and restored on reopen, so the worktree can be removed.

**Steps to reproduce**
1. In a closed issue's worktree, leave an ignored file outside
`node_modules`, `dist` and the other regenerable directories (for
example `ui/tsconfig.tsbuildinfo`).
2. With restore points enabled, run expiration: the worktree is kept
with `unique_ignored_files`.

**Paperclip version or commit:** `master` at 412914cfa7.
**Deployment mode:** any.
**Agent adapter(s) involved:** not adapter-specific (core).

## What Changed

**Capture** (`server/src/services/git-restore-point.ts`):
- `restorePointSavesIgnoredFile` defines which ignored files are saved.
It leaves out:
  - `.env` and `.env.*` files;
- `.paperclip-repositories/` (nested checkouts have their own restore
points);
  - `.claude/settings.local.json` (saved with the workspace contents);
- anything inside a directory named in
`REGENERABLE_WORKSPACE_DIR_NAMES`.
- Those files are scanned with the same limits and checks as untracked
files, and snapshotted into a separate `ignored` tree, committed on top
of `HEAD`. Its ref travels in the bundle.
- They are hashed with `hash-object --no-filters` (symlinks by their raw
target bytes), so `.gitattributes` conversions such as `text=auto` never
change their bytes. Every path given to `--stdin-paths` is C-quoted, so
git always reads the literal workspace file, even for names that start
with `"`.
- An ignored nested repository skips capture, which keeps the worktree.
- A worktree whose only extra files are ignored is no longer "nothing to
save".

**Format:** a payload with an ignored tree uses
`paperclip.git-restore-point-payload.v2`, and one without stays `v1`.
Decoding requires `ignored` exactly for `v2`, so older servers reject
the new payloads instead of restoring without the files.

**Restore:** `applyRestorePoint` writes the ignored files directly from
their blobs, read in one `cat-file --batch` call, so no filter is
applied.
- It checks every target before writing, refuses any path that would
leave the worktree, and never overwrites an existing file: a conflict
fails the restore closed.
- If a write fails partway, it removes the files it wrote, so a retry
starts clean. The post-apply verification, the before-deletion snapshot
check and `restorePointSnapshotMatches` all include the ignored tree.
The stored reference (`GitRestorePointRef.ignored`) is validated and
checked against the decoded manifest.

**Expiration** (`git-workspace-restoration.ts`): in restore point mode,
the reproducibility check accepts the ignored files a restore point
saves. Strict mode is unchanged.

**Docs:** `doc/recoverable-disk-cleanup.md`.

No code comments were added.

## Verification

- `pnpm exec vitest run src/__tests__/git-restore-point.test.ts` (in
`server/`): 27 passed. New cases:
- **Round trip:** ignored files round-trip as a `v2` payload with their
modes, while `dist/`, `node_modules/` and `.env` stay out.
- **Conflict:** an existing conflicting file fails the restore closed
and is left as it was.
  - **Nested repository:** an ignored nested repository skips capture.
- **Forged payload:** a `v1` payload claiming an ignored tree is
rejected.
- **Line endings:** an ignored CRLF file in a `* text=auto` repository
restores byte for byte.
- **Partial failure:** a restore that fails partway leaves nothing
behind, and a retry succeeds.
- **Quoted names:** a crafted ignored path that looks like a quoted
absolute path saves and restores the workspace file, never the outside
file it names.
- **Symlinks:** a symlink target with bytes that are not valid UTF-8
restores exactly.
- `pnpm exec vitest run
src/__tests__/git-restore-point-expiration.test.ts` (in `server/`): new
cases.
- A closed worktree whose only extra files are an ignored build cache
and evidence file is kept without restore points, expires with one, and
reopens with those files and without `node_modules`.
  - An ignored file changed after capture rolls back.
- **Mutation checks:** each of these fails at least one case:
  - not relaxing the reproducibility check;
- not capturing ignored files (the end-to-end case then catches the
loss);
  - leaving the ignored tree out of the comparison;
  - not writing the files on apply;
  - hashing with filters;
  - dropping the partial-restore rollback;
  - passing unquoted paths;
  - decoding symlink targets as UTF-8.
- `pnpm exec vitest run git-restore-point git-workspace-expiration
git-workspace-contents git-workspace-restoration execution-workspaces
workspace-compaction heartbeat-git-restore-point-resume
src/services/heartbeat/environment-selection` (in `server/`): 296
passed.
- `pnpm run typecheck` (in `server/`): pass.
- Not run: server build and boot smoke.

## Risks

- **Only with restore points enabled.** Strict expiration is unchanged.
- **Bigger restore points.** Ignored files count toward the existing
limits (20,000 files, 64 MB per file, 256 MB total). Larger caches
outside the regenerable names make capture skip, which keeps the
worktree as before.
- **Rollback:** older servers cannot restore `v2` restore points; they
fail closed. Disable restore points before downgrading.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file edits, tests, read-only inspection of a live
instance).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-23 08:24:31 -07:00
DottaandPaperclip 3f1ad6b25b fix(chat): resume task Retry after retained-source containment (#17317)
## Thinking Path

> - Paperclip manages agent work through task and execution ownership.
> - A task Retry must retain the user's request while the system
verifies an old executor.
> - Connected conversations also retain their external principal and
accepted source batch.
> - Chat Retry admission required containment before it could save a
wait for that recovery.
> - A saved chat receipt could also remain deferred after containment
became available.
> - This pull request lets the exact recovery-bound request wait and
resume through its chat owner.
> - Normal admission still checks containment, current access and policy
before one successor starts.

## Linked Issues or Issue Description

**What happened?**

An ordinary task Retry on a connected conversation failed with
`failed_chat_retry_not_authorized` while its retained remote executor
still needed verification. The new recovery generation existed, but the
chat action failed before it could save a dispatch wait. This occurred
after the repairs in #17267 were confirmed live.

The local regression found a second gap. A processed chat action with a
deferred wake was not reconsidered after containment. Re-entering normal
admission also tried to insert its existing durable receipt again. A
historical hold for the same failed run could also hide the current
Retry's workspace strategy, so the request still waited after the old
executor was contained. Preparation could then read that sibling instead
of the recovery record that owned the fresh successor, losing the
fresh-workspace decision.

**Expected behavior**

Save the exact authorized Retry while its matching recovery generation
verifies the executor. Once containment is recorded, reconsider the same
chat receipt through normal admission. Start one successor with the
original source batch. Keep publication, principal, pause and budget
checks.

**Steps to reproduce**

1. Admit a GitHub or Telegram request and record a failed native run
with a retained remote lease.
2. Leave the executor's termination unconfirmed and exhaust its prior
recovery generation.
3. Use ordinary task Retry to create a new recovery generation.
4. Process the connected Retry. The old admission code fails instead of
retaining a wait.
5. With a deferred receipt, record exact containment and reconcile
again. The old dispatch path leaves it stranded.

**Paperclip version or commit**

Reproduced from `412914cfa78d5abefcc34d3eb8d4bb86e461caac`. The branch
is rebased onto current master.

**Deployment mode**

Hosted tenant for the reported failure. Isolated local PostgreSQL and
the assembled heartbeat/chat services for the regression.

Refs #17267 and #16965. This change does not add a Cloud dependency or
change the absence protocol.

## What Changed

- Permit a saved admission wait only for the exact current task Retry
and active source recovery generation.
- Keep promotion and execution dependent on independent containment
proof.
- Bind a Retry to active automatic recovery without replacing its
generation, attempt count or deadline.
- Share the current exact Retry strategy across historical holds for the
same run. Retain each hold's stop and effect checks.
- Select the canonical recovery record that proves the committed fresh
successor during workspace preparation.
- Let task receipt reconciliation return the same deferred action to its
chat owner.
- Recheck current board and external authority, then update the existing
wake atomically with its successor.
- Preserve the exact chat source batch. Keep generic board-comment
filtering unchanged.
- Add GitHub and Telegram cases for waiting, containment, one successor,
stale generations, source edits, endpoint pause and supersession.
- Update recovery documentation and capability maps.

## Verification

- Red: real PostgreSQL tests reproduced the admission rejection for
GitHub and Telegram. Six active-recovery cases then failed on the
generation-ID assumption. Two same-run historical-hold cases remained
deferred after containment on the old strategy lookup. A preparation
regression then returned no readiness when the historical hold was
stored first.
- Green: 73 selected chat integration tests and 394 continuation, task
Retry, workspace recovery, recovery-state and workspace-preparation
tests passed. The chat tests use the real chat admission and heartbeat
queue. Unrelated capacity stays occupied so no real provider starts.
They assert no executor before containment, one successor afterward,
unchanged source leases while waiting, preserved recovery budgets, and
denial after authority changes. A separate source hold still blocks
work. Both sibling insertion orders pass through the committed-readiness
lookup and real workspace preparation: the new directory has independent
Git storage and files, and unpublished old work remains intact.
- `pnpm -r typecheck`, `node scripts/check-maps.mjs` and `git diff
--check` passed after the review fixes. The source build passed on
`f6660efa48e37a80cd9857217a537606de85b5c8`.
- The local host reached its PostgreSQL semaphore limit. Successful runs
used a test-only loader to reduce disposable cluster connection and
worker limits. No repository database helper or application setting
changed.
- The full local `pnpm test:run` began before the final review edits,
failed in its first server phase and was stopped after those failures.
Later phases did not run. This is not a full local pass. The transient
Retry-notice case passes after fixture isolation; the four Discord
close-owned publication cases also pass in isolation. Other broad-run
failure causes are not established.
- Exact-head CI and security checks pass on
`f6660efa48e37a80cd9857217a537606de85b5c8`; Greptile is 5/5 with zero
unresolved review threads. Optional Paperclip Review completed neutral
with a 5/5 static assessment of all 11 changed files and no findings; it
did not run tests or hosted acceptance. All 57 current-head checks are
terminal: 54 success, two skipped and the optional neutral review, with
no failures or pending checks.
- The real hosted workflow confirmed that the deployed absence repair
recorded containment. A later ordinary UI Retry was accepted and saved a
durable request, but it remains behind the historical source hold. The
useful existing review remains unchanged. This follow-up is not
deployed; a live successful successor and final task disposition remain
unverified. No missing files are claimed recovered.

## Risks

- The existing task reconciliation sweep can restage a deferred chat
action. This reuses its receipt and generation; it does not grant new
execution or publication authority.
- Dispatch must keep the board Retry identity separate from the
connected request's principal. The regression checks these bindings and
preserves current chat authorization at each phase.
- Historical holds reuse a strategy only from the locked same-run
recovery that names the current Retry intent. A different run, malformed
recovery record or independent source hold remains fenced.
- Unknown executors without a matching active recovery still fail
admission. Promotion and execution require containment even when
admission saved a wait.
- No schema or API shape changes. No credential changes, manual
deployment or service restart.

## Model Used

OpenAI GPT-6 through Codex, with repository tools, code execution and
browser verification. The exact served model variant and context window
are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-23 07:51:34 -07:00
DottaandPaperclip 3bd709df8b Operationalize Product E2E accounting, tracing, and scheduling (#17009)
## Thinking Path

> - Paperclip manages agents and the work they perform.
> - Product E2E tests exercise real agents, tools, setup and cleanup.
> - Repeated runs need retained costs and traces, including failed
attempts.
> - A passing retry must not hide earlier spending or feedback.
> - Recurring work also needs admission limits, private reports and
failure detection.
> - This change supplies those controls while keeping paid activation
behind existing gates.

## Linked Issues or Issue Description

Refs #16996 and #17158. This is the consolidated landing of the reviewed
eval operations stack: #17019, #17024, #17025, #17027, #17030, #17033,
#17035, #17038, #17043, #17044, #17048, #17050, #17051, #17092, #17096,
#17100, #17107, #17109, #17112, #17115, #17117, #17121, #17123, #17127,
#17130, #17133, #17134, #17137, #17140, #17143, #17154. The repository
permits squash merges only. A local rehearsal showed that squashing each
slice into master causes repeated conflicts in dependent slices. This PR
preserves their commits and lands their combined implementation in one
reviewed integration. Superseded slices will close after this merge.

## What Changed

- Retain every attempt's cost receipt, including setup, failures,
interruptions and retries. Separate reported amounts, estimates, missing
costs and reserved capacity in reports and dashboards.
- Add bounded request admission, exact model pricecards, gateway error
accounting, cache policy and reconciliation evidence. Refuse unsupported
routes and requests that cannot fit the allowance.
- Integrate Braintrust gateway traces and OTLP correlation, with secret
filtering and experiment imports. Record phase timing, including setup,
feedback capture and cleanup.
- Collect complaint and suggestion evidence after each attempt,
including recovery after failures. Retain explicit review dispositions
independently from the product grade.
- Add gated daily and weekly selections, measured forecasts,
setup/readiness checks, wall-time limits and retained monitor
notifications. Include declared non-model overhead in admission while
reporting the existing subscription separately.
- Keep recurring history private, bound uploads and retention, and use
isolated archive environments. Keep public publication an explicit
operation. Report and publication jobs install only the committed
lockfile, with scripts disabled and isolated stores; stale pins fail
closed.
- Integrate current master, preserve its Product E2E journey billing and
privacy checks, and test that shared setup receipts do not double-count
estimated agent usage. Preserve upstream fixture repairs.

## Verification

- Complete integration baseline
`0649a5ce7544ab20825c2183d2f421157db49f24` includes master
`a3e1d915a7110a4e115b46396a3f79e6dfb4d05e` and every original stack head
as ancestors.
- On that integration baseline, `pnpm test:e2e:runner:unit`: 4,012
Vitest tests passed, two skipped, across 193 files; all 137 Node
source-contract tests passed.
- Focused billing and Product E2E journey contract checks: 24 tests
passed. They cover estimates, unknown receipts, single-owner setup
costs, browser-only execution and private publication guards.
- Monitor and workflow wiring: all 96 Node tests passed. `node
scripts/check-token-gates.mjs`: all gates clean.
- `pnpm test:e2e:runner:typecheck` passes after the standard Runner
TypeScript build refreshed local declarations. The first attempt
rejected the new master `runner_prepared_launch` union member because
this checkout retained an older generated declaration; the source union
already contained that member.
- Current head `492b0708a1c531a023d9ac10f0f3dc6383cc1754` removes
dependency re-resolution from archive-authorized reporting jobs after
the security scan flagged it. All 15 workflow-security tests pass after
updating the old two-install assertion.
- Hosted [CI run
39965460180](https://github.com/paperclipai/paperclip/actions/runs/39965460180)
passes on the current repaired head, including typecheck, build, general
tests, Runner checks, browser shards and canary dry run. Greptile is 5/5
on this exact head with no unresolved threads. The security scan passes
after the fixed lockfile-resolution thread was resolved.
- Paperclip Review returned a terminal neutral check but an incomplete
0-file assessment on the repaired head: its platform blocked both diff
responses as prompt-injection content. This is not a successful code
review. No bypass or fabricated pass was attempted; current-head
Greptile, security and hosted CI are independent gates.
- No paid test, reservation release or recurring-run activation
accompanies this landing. Live route qualification and invoice
reconciliation remain separate evidence.

<details>
<summary>Retained earlier validation failures</summary>

Earlier broad local runs had two 15-second test timeouts (CLI poll
cadence and Retry validation), one embedded-database setup timeout, and
later PostgreSQL semaphore exhaustion. The exact timed-out tests passed
separately without larger timeouts; the broad-run causes were not
established. Those local failures remain distinct from subsequent hosted
CI results. The independent #17158 fix records its Muse fixture
collision, its UUID correction and hosted verification. Local Muse
execution was also blocked by macOS embedded-PostgreSQL startup. No
historical failure is treated as a successful paid qualification.

</details>

## Risks

- Admission is a local control, not a provider invoice cap. Missing
provider receipts or resource charges retain unknown amounts and holds;
estimates do not become billed dollars.
- The bounded path admits only documented local suites and provider
routes. Ordinary legacy or separate Runner protocol campaigns have
different controls. Merging this code does not qualify additional
routes.
- Recurrence still needs the existing enable gates and current
qualification/allowance evidence. Archive permissions and resources must
match the documented private-environment contract.
- Legacy reports may lack complete attempt or phase evidence. Reports
preserve that limitation and do not infer zero usage from missing data.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository tools and code
execution. The exact deployment variant and context-window size are not
exposed by this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (targeted checks;
full-suite host limitation disclosed above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-23 07:18:37 -07:00
DottaandPaperclip 4173059081 fix(inbox): keep advisory GitHub review waits quiet (#17318)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Mine shows a person's participation and requests that need their
response.
> - Automatic GitHub reviews should remain quiet until the assigned work
needs human input.
> - Review bots used completion requests to keep tasks in review while
CI or a PR was still open.
> - Those requests correctly entered Mine, despite the automatic-work
filter from #16356.
> - This pull request gives published reviews an existing provider-event
wait in the in-review state and withdraws superseded completion
requests.
> - Real input requests and separate approval gates keep their current
behavior.

## Linked Issues or Issue Description

Refs #16356
Refs #13314

An advisory score below 5/5, pending or failed CI, or an unmerged PR
does not by itself require a person's approval in Paperclip. The bot can
track those conditions without adding a completion card to Mine.

## What Changed

- Explain the quiet review lifecycle in the GitHub review instructions.
Use a yielded provider-event wait, or an existing saved monitor, after
publication. Report done when the configured completion conditions are
met.
- Keep a confirmed, published review in `in_review` during that wait.
Reuse existing authorization and monitor delivery. Add no schema, timer,
or polling loop.
- Recognize the committed external wait in review attention and
recovery. Shared execution receives a verified publication flag; GitHub
receipt lookup stays with the GitHub module. Other chat waits keep their
current status.
- Withdraw an older system-created completion request only when the same
bot has published both reviews and its new report has no attention
request. Keep explicit cards, separate approval gates, and requests
without the prior review evidence. Finish validation stays read-only;
withdrawal commits atomically with the new status after checking current
governance, the admitted result, and both low-risk contracts.
- Update the product rule, capability maps, and regression coverage.

## Verification

- Final commit `7c502118eec36028d1b4537bb8797ef893940bf0`: 54 GitHub
checks passed, two intentionally skipped, none pending or failed.
Greptile is 5/5, the security scan passed, and all review threads are
resolved.

- Passed: `pnpm -r typecheck` and `pnpm build`. Rechecked server
typecheck and build after the final authorization guard.
- Passed: 195 tests across `external-chat-wait.integration.test.ts`,
`status-arbiter.test.ts`, `review-instructions.test.ts`, and
`issue-liveness.test.ts`. The database suite used an isolated PostgreSQL
18 container through `PAPERCLIP_TEST_DATABASE_URL`.
- Acceptance coverage: 3/5 stays in review outside Mine; 5/5 with
pending completion conditions stays in review; 5/5 with satisfied
conditions finishes quietly; a real repository-access request enters
Mine until resolved. Also covers obsolete-card withdrawal, transaction
rollback, rejected reports, separate approval gates, multiple reviewed
heads, saved monitors, and revoked endpoints, repositories, or
membership.
- Passed: `node scripts/check-maps.mjs`, `node
scripts/check-module-boundaries.mjs`, and `git diff --check`.
- The initial acceptance run exposed a missing authenticated user in the
test fixture. Added the user required by the existing addressee
authorization check. All 154 tests in that database suite then passed,
including the final rerun.
- Review found premature withdrawal during validation, stale repository
access, and a missed receipt when a run reviewed multiple heads. The
revision fixes all three and includes acceptance regressions. The
saved-monitor commit guard now accepts the verified published-review
state.
- `pnpm test:run` was attempted but could not complete on this Mac.
Embedded PostgreSQL failed to allocate semaphores (`semget: No space
left on device`); a direct initialization probe confirmed the host
resource limit. The isolated database run proves the affected suite, not
the full repository suite.
- These tests exercise persisted review receipts, finalization, and Mine
queries. They do not prove a live model followed the new instructions or
submit a live GitHub review.

## Risks

- The bot must follow the new lifecycle instructions. A real explicit
attention request still enters Mine.
- Existing obsolete cards are withdrawn when the same bot publishes a
later review and reports no human dependency. This change does not run a
bulk cleanup of historical inbox entries.
- Review attention now recognizes an external response path. It requires
an applied current status decision, a successful bound run, and current
conversation, repository, and principal authorization.
- No migration or UI change. The full local test suite is unverified
because of the host database limit; CI passes on the final commit.

## Model Used

OpenAI Codex, GPT-6, with reasoning, repository inspection, code
editing, and test execution. The exact deployment model ID and
context-window size are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (195 focused tests;
full-suite limitation above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-23 07:14:40 -07:00
DottaandPaperclip a3e1d915a7 fix(evals): preserve unknown Codex provider cost (#17158)
## Thinking Path

> - Paperclip manages AI agents and their work.
> - Direct Runner evals report usage and costs for each attempt.
> - Codex App Server supplies token counts without a provider USD
receipt.
> - The legacy Runner ledger fills an absent cost with zero.
> - The eval result exposed that ledger value as provider-reported cost.
> - This fix keeps provider cost unknown and preserves the estimate and
budget stop.

## Linked Issues or Issue Description

Refs #17009 and #17154. This is a separate direct Runner accounting fix
based on master.

## What Changed

- Codex direct eval usage now reports null provider USD cost with
unavailable provenance. Token counts and the pinned price estimate
remain available.
- Preserve the original ledger amount as optional
`legacyLedgerCostNanodollars`. Keep the original snapshot receipts
unchanged.
- Apply the existing budget ceiling to the retained ledger amount as
well as the token estimate. Correcting a label does not relax admission
or settlement checks.
- Use the unique operator UUID for Muse test company prefixes,
eliminating collisions from the previous 24-bit random suffix.
- Document the cost distinction in the Runner/Evals integration
contract. Other provider behavior and historical artifacts remain
unchanged.

## Verification

- Reproduced both zero and nonzero legacy amounts being labelled as
provider USD cost before the fix.
- The current eval-session contract and entrypoint tests pass (39
tests). The new regression uses the bounded usage consumer and actual
campaign cost projection. It checks unknown provider cost, the separate
estimate, unchanged raw evidence and the legacy ledger budget stop.
- All 15 Runner report, campaign and metrics Node tests pass. Four
adjacent CLI entrypoint/runtime tests also pass.
- Runner TypeScript validation passes with its standard dependency-build
prerequisite; `git diff --check` passes. Both Runner TypeScript
configurations also passed during the original accounting verification.
- Hosted CI run
[39960367674](https://github.com/paperclipai/paperclip/actions/runs/39960367674)
passes on `7d84f138044bac58cdced6baa2d0ec3d63767bb8`, including
typecheck, build, server tests and browser checks. Greptile and
Paperclip Review both report 5/5 on that commit with no actionable
findings.
- The previous CI failure was a PostgreSQL unique-index collision on
`companies_issue_prefix_idx` during Muse fixture setup. Local Muse
verification could not reach test execution because embedded PostgreSQL
timed out during macOS startup (initially reporting a missing zstd
library). The corrected fixture passed in hosted Linux CI.
- Verification used no model calls. Full repository checks were
performed by hosted CI; local verification was scoped to the changed
contracts and fixture.

## Risks

Consumers that assumed Codex always had a numeric provider cost must
handle null. The contract already permits null for other profiles, and
the report consumer retains missing-cost coverage. The legacy amount
remains available as unverified evidence and still triggers the existing
budget stop. Neither the estimate nor that ledger amount proves an
invoice charge. This change does not rewrite old artifacts or qualify
other provider routes.

## Model Used

OpenAI GPT-6 through Codex, with repository tools and code execution.
The exact deployment variant and context-window size are not exposed by
this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-23 06:36:34 -07:00
Nicky Leach 583ec2443a Read deliverable files before locking mutation rows (#17309) 2026-10-23 05:46:38 -07:00
Nicky Leach 412914cfa7 Read native plugin task output files on the execution host (#17285) 2026-10-22 23:38:09 -07:00
Devin FoleyandPaperclip d69631b01b feat: reopen a fresh workspace with a warning when its restore point expired (#17269)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - #17248 and #17257 let git cache expiration remove closed issues'
worktrees after saving encrypted restore points
> - Restore points are kept for 90 days after an issue tree closes, then
deleted
> - Reopening a workspace whose restore point had expired failed, so the
issue could not get a workspace back
> - This pull request starts a fresh workspace on the same branch
instead, and warns that the saved work is gone
> - The benefit is that old issues can always be resumed, with a clear
record of what was lost

## Linked Issues or Issue Description

Refs #17248, #17257. Bug report described in-PR (bug report template).

**What happened?**
After `PAPERCLIP_GIT_RESTORE_POINT_RETENTION_DAYS` passes, the retention
worker marks the restore point expired and deletes it. Reopening that
workspace then fails with the git cache restoration error, every time.

**Expected behavior**
Reopening succeeds with a fresh workspace on the issue's branch, and the
issue says that unsaved work from before the cleanup is no longer
available.

**Steps to reproduce**
1. Enable restore points and let a closed issue's worktree expire with
unsaved work.
2. Let its restore point expire.
3. Reopen the issue: the workspace rebuild fails.

**Paperclip version or commit:** `master` at 650581b2f0.
**Deployment mode:** any.
**Agent adapter(s) involved:** not adapter-specific (core).

## What Changed

- **Detection:** `gitRestorePointExpired(recipe)` is true when the main
restore point or any nested checkout's restore point has `expiredAt`.
- **Reopen** (`execution-workspaces.ts`):
- `restoreGitCacheBase(..., { freshStart: true })` prepares the base
without the saved commit or the bundle.
- The saved recipe is left out of the metadata handed to the runtime,
and dropped from the stored metadata only when the reopen succeeds, so a
failed attempt can be retried and still warns. The runtime recreates the
worktree on its branch: the branch tip if the repository still has it,
otherwise a new branch from the base ref. When the expired recipe had
nested checkouts, the project's repositories are prepared with
`prepareProjectRepositoryWorkspaces` (without the expired saved state)
right after the worktree is recreated and before the provision command
runs, with the same copy identity and access rechecks as other reopen
copies.
- The result carries `warning`, the `execution_workspace.reopened`
activity records `restorePointExpired` and the warning, and a system
comment on the issue posts the warning in the same transaction.
- **Retention:** `markGitRestorePointExpired` now also marks nested
checkouts' restore points, matched by object key.
- **Docs:** `doc/recoverable-disk-cleanup.md` describes the fresh start.

No code comments were added.

## Verification

- `pnpm exec vitest run
src/__tests__/git-restore-point-expiration.test.ts` (in `server/`): 13
passed. New case: an expired restore point reopens fresh with the
warning and the system comment, on the original branch, even after the
base lost the branch and its unpushed commit; the recipe is gone. A
first attempt that fails keeps the expired recipe, and the retry still
warns and comments exactly once.
- New nested case in `git-workspace-expiration.test.ts`: an expired
nested checkout restore point is marked on the checkout entry and
reopens fresh with the warning, with the nested repository cloned again
at its pushed commit and without its expired uncommitted files.
- **Mutation checks:** each of these fails a case:
  - removing the fresh base path (reopen fails);
  - removing the system comment;
  - handing the runtime the old recipe;
  - dropping the stored recipe before the rebuild succeeds;
  - marking only the main restore point expired;
  - skipping nested repository preparation on a fresh start.
- `pnpm exec vitest run git-workspace-expiration git-workspace-contents
git-restore-point execution-workspaces` plus the environment selection
and heartbeat resume suites (in `server/`): 257 passed.
- `pnpm run typecheck` (in `server/`): pass.
- Not run: server build and boot smoke.

## Risks

- **Only with restore points enabled:** recipes get `expiredAt` only
from the restore point retention worker, so nothing changes otherwise.
- **Lost work is intended:** the fresh workspace has no uncommitted
files and, when the repository no longer has the branch, no unpushed
commits. The warning says so.
- **Recipe dropped:** after a fresh start the workspace no longer
carries the expired recipe, so later expirations start from a clean
slate.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file edits, tests).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-22 22:20:52 -07:00
Nicky Leach 2cd6eb27c7 fix(runner): record the plugin-prepared working directory (#17243) 2026-10-22 20:42:26 -07:00
DottaandPaperclip 4c57da3708 fix(recovery): continue Retry after verified sandbox absence (#17267)
## Thinking Path

> - Paperclip manages AI agents and their work.
> - Task Retry must reconcile a stopped native execution before starting
another one.
> - A retained sandbox can disappear while workspace preservation is
pending.
> - A tenant-key error does not prove absence, especially after
credential rotation.
> - This pull request accepts only a short-lived Cloud observation bound
to the exact source and current preservation owner.
> - Normal Retry admission can then continue in a fresh workspace
without claiming that missing files were recovered.

## Linked Issues or Issue Description

**What happened?**

A native run stops with workspace preservation pending. Its Daytona
allocation no longer exists. Recovery tries to reopen it, then cannot
confirm a stop. The ordinary Retry remains deferred behind cleanup and
repeats a transient failure.

**Expected behavior**

Cloud must independently confirm absence in the recorded provider
organization. Core must verify that proof and settle executor
containment under the current physical owner. Normal Retry admission
must enforce all existing gates and create at most one successor. The
old run must remain failed and missing files must remain unavailable.

**Steps to reproduce**

1. Persist a failed native run with a retained Daytona lease and an
exhausted workspace-recovery generation.
2. Remove the exact provider allocation so reopen and stop cannot
confirm it.
3. Request ordinary task Retry.
4. Observe that recovery reports `provider_unavailable` and leaves the
Retry deferred.

**Paperclip version/commit**

Reproduced on source based on `6d44b80a5edbcc9f69bfc3d520994e70aa9fab6d`
with the prior preservation and Retry repairs present.

**Deployment mode**

Cloud tenant with native Runner and a retained Daytona source. The local
acceptance uses real Core, Cloud, PostgreSQL, signatures, native Runner
and UI, with deterministic model and provider HTTP boundaries.

Refs #16962, #16965, #17053. These repairs cover completion proposals,
durable Retry and retained-source reopening. They do not establish
absence of a missing provider allocation. Related #16809 preserves
orphan sandbox leases; this change requires independent proof before
settling an absent allocation.

## What Changed

- Add an HTTPS client for a purpose-separated, host-bound Cloud
sandbox-absence capability.
- Verify signature, audience, stack, expiry, source identities and
current physical-owner token.
- Settle only executor absence with a metadata and owner
compare-and-set. Preserve source protection and record that workspace
contents are unavailable.
- Leave successor creation, approval, pause, budget and uncertain-effect
checks to normal Retry admission.
- Document the protocol and update the recovery maps.

## Verification

- Red: the new persisted-state regression returned
`provider_unavailable` and scheduled recovery on the unfixed code.
Expected: `source_unavailable` with a contained old execution.
- Green: 384 focused tests pass across workspace recovery, signed
absence, explicit continuation, ordinary Retry, workspace preparation
and provider-owned fresh workspaces.
- Signature tests reject forged, expired and mismatched proofs. HTTP
failures or missing configuration grant no authority.
- Two missing allocations are both reconciled before Retry. A simulated
interruption reuses the first durable absence receipt.
- Concurrent Retry admission produces one queued fresh successor. It
records no successful old result and no recovered-file claim.
- Browser acceptance at this change: one ordinary Retry, two provider
absence reads for two retained sources, one fresh native execution,
useful confirmation and Done after reload. The old run remains failed.
Existing saved review and newer local file bytes remain unchanged. The
deterministic model does not perform external publication; this is local
protocol acceptance, not live provider recovery.
- `pnpm typecheck`, `pnpm build`, and `node scripts/check-maps.mjs`
pass.
- A complete local `pnpm test:run` pass was not obtained. The first run
hit a 15-second timeout in an unchanged status-card test. It was stopped
to apply review fixes before the remaining phases ran. The unchanged
status-card suite then passed 25/25 in isolation with the repository
short temporary-path convention.
- CI exposed a chat nonduplication assertion that compared unordered
database rows. The assertion now sorts both complete row sets by run ID.
Its focused test passes.
- All 58 current-head checks are terminal on
`1e2e881f6b6b1ae1fbfe1f2b2922657ce46e1470`: 55 success, two unaffected
Storybook skips, and one neutral advisory review. Required CI, build,
E2E and security checks pass. Greptile is 5/5 with no unresolved
threads. The optional Paperclip review is neutral advisory 5/5 with
nine-file source coverage and no findings; it did not independently run
tests.
- One unchanged native-runtime test timed out at five seconds on its
first CI attempt. Its suite passed 33/33 in isolation; a single retry of
that lane passed without code changes.

## Risks

- The companion Cloud endpoint must ship before this consumer. Older or
unavailable Cloud versions return no proof and retain the current hold.
- A valid absence receipt permits a fresh workspace attempt. It cannot
restore missing remote files, establish task success or authorize
repeating an uncertain external effect.
- This PR has not been deployed. The original task remains unrecovered
pending rollout and reconciliation of its existing Retry.
- No schema, credential, provider-resource or fleet changes are
included.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, source inspection, code
execution and browser tools. The exact backend variant and
context-window size are not exposed by this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (focused scope above; full
local run limitation recorded)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-22 22:31:52 -05:00
Devin FoleyandPaperclip 5ef783d971 feat: expire idle open issues' worktrees with restore points (#17271)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - #17248 and #17257 let git cache expiration remove a closed issue
tree's worktree after saving an encrypted restore point
> - Issues that stay open but go untouched for weeks still keep their
worktrees forever, so disks keep filling
> - This pull request expires those idle worktrees the same way, and has
the next run restore its workspace automatically
> - The benefit is bounded disk use for stalled open work, with nothing
lost when work resumes

## Linked Issues or Issue Description

Refs #17248, #17257. Bug report described in-PR (bug report template).

**What happened?**
With restore points enabled, expiration only considers worktrees whose
whole issue tree has been `done` or `cancelled` for a while. A worktree
whose issue stays `todo`, `in_progress`, `blocked` or `in_review` with
no activity for weeks is never removed.

**Expected behavior**
An open issue's worktree that has been idle long enough is saved to a
restore point and removed, and the next run on that issue gets it back
exactly.

**Steps to reproduce**
1. Enable git cache expiration and restore points.
2. Leave an issue open with an isolated worktree and no activity for
weeks.
3. The worktree is never removed.

**Paperclip version or commit:** `master` at 650581b2f0.
**Deployment mode:** any.
**Agent adapter(s) involved:** not adapter-specific (core).

## What Changed

**Expiration** (`git-workspace-expiration.ts`):
- `isGitWorkspaceExpirationEligible(..., idleCutoff)`: in idle mode the
closed-tree conditions are replaced by idle ones. No issue in the tree,
and no other issue bound to the workspace, was updated after the cutoff,
and no relevant run was created, started or finished after it. Runs of
bound issues now count as relevant runs in both modes. Every other check
stays, including no live run, recovery action, unsettled finalization,
environment lease or running service.
- `gitWorkspaceLastActivityAt` returns the latest of those timestamps,
sharing the same tree and run scope.
- `expireReconstructableGitWorkspace` takes `idleCutoff` and passes it
to all three eligibility checks.

**Reaper** (`execution-workspaces.ts`):
- For an open tree's isolated git worktree, when restore points and git
cache expiration are on and `idleMs` is set, the reaper computes the
idle deadline (last activity + `PAPERCLIP_GIT_RESTORE_POINT_IDLE_DAYS`).
- Before the deadline, it records the deadline, so the next sweep wakes
then. Open trees cause no recurring checks before that.
- At the deadline, it runs expiration with a restore point. A kept
workspace is retried after an hour, like terminal ones.
- An expiration interrupted after archiving (`completed: false`) is
resumed by later sweeps even while the tree is open, with a retry
scheduled until cleanup finishes.

**Runs** (`heartbeat/environment-selection.ts`):
- When a run's issue is bound to a workspace that git cache expiration
removed (archived, `workspaceRemoved`), the run reopens it as the agent
through `reopenClosedIsolatedExecutionWorkspaceForIssue`, with the
restore point applied.
- It then clears the reopen fence, since the live run now holds the
workspace, and continues with the reopened row.
- A failed restore fails the run instead of silently starting a new
workspace.
- Comments and agent updates on the issue already reopened closed
workspaces through the issue routes.

**Config:** `PAPERCLIP_GIT_RESTORE_POINT_IDLE_DAYS` (default `7`, `0`
turns idle expiry off). Docs in `doc/recoverable-disk-cleanup.md` and
`doc/DEVELOPING.md`.

No code comments were added.

## Verification

- `pnpm exec vitest run
src/__tests__/git-restore-point-expiration.test.ts -t "idle open
issues"` (in `server/`): 7 passed, with real repositories and embedded
Postgres.
- **Round trip:** an open issue's worktree, idle for 8 days with unsaved
work, is skipped without idle mode, expires with a restore point in idle
mode, and reopens exactly.
- **Kept:** issue activity 3 days ago, or a run finished 2 days ago,
keeps the worktree.
- **Interrupted:** an idle expiration interrupted after archiving is
finished by the reaper while the issue stays open, and reopens exactly.
- **Shared workspace:** a recent update on another issue bound to the
workspace keeps it, and moves the last activity time.
- **Reaper:** before the deadline the sweep skips the worktree and
schedules exactly that deadline; at the deadline it expires it with an
active restore point. With idle expiry off, nothing is scheduled or
expired.
- `pnpm exec vitest run
src/services/heartbeat/environment-selection.test.ts` (in `server/`): 14
passed. New cases: a run reopens its expired workspace as the agent,
clears the fence with the reopen's generation and binds the reopened
row; a failed restore fails the run; a workspace archived for other
reasons is left alone.
- `pnpm exec vitest run
src/__tests__/heartbeat-git-restore-point-resume.test.ts` (in
`server/`): 1 passed. Assembled heartbeat journey with no mocked
restore: a real worktree with an unpushed commit and uncommitted notes
is idle-expired, a queued run executes, the agent runs in the original
worktree path with the original workspace id, the commit and notes are
back, the workspace is no longer archived with no reopen fence left, and
the issue stays bound. Disabling the run-side reopen fails it.
- `git-restore-point-expiration-config.test.ts`: the default is 7 and
`0` turns it off.
- **Mutation checks:** each of these fails a case:
  - dropping the idle deadline;
  - dropping the idle cutoff;
  - dropping the run activity condition;
  - not failing the run on a failed restore;
  - not re-reading the reopened workspace;
  - not resuming a pending expiration on an open tree;
  - ignoring bound issues' activity.
- `pnpm exec vitest run git-restore-point git-workspace-expiration
git-workspace-contents execution-workspaces
src/services/heartbeat/environment-selection
src/services/heartbeat/workspace-preparation` plus
`heartbeat-git-restore-point-resume` (in `server/`): 312 passed.
- `pnpm run typecheck` (in `server/`): pass.
- Not run: server build and boot smoke.

## Risks

- **Off unless enabled:** requires
`PAPERCLIP_GIT_CACHE_EXPIRATION_ENABLED=true` and
`PAPERCLIP_GIT_RESTORE_POINTS_ENABLED=true`.
`PAPERCLIP_GIT_RESTORE_POINT_IDLE_DAYS=0` keeps restore points for
closed trees only.
- **First run after idle expiry is slower:** it waits for the restore,
about as long as a reopen.
- **Failed restore blocks runs:** a restore that fails closed (for
example a missing restore point) fails each run on that issue until an
operator fixes or rebinds the workspace. This is deliberate, so work is
never silently continued in an empty workspace.
- **Kept idle workspaces are rechecked hourly**, matching the existing
behavior for closed trees that cannot be expired.
- **Retention:** restore points of open trees are not deleted while the
tree stays open, as before.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file edits, tests).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-22 20:02:21 -07:00
Devin FoleyandPaperclip c5ca29dbef fix(claude): prepare compatible sandbox runtimes during onboarding (#17259)
## Thinking Path

> - Paperclip connects agent accounts and prepares their execution
environments.
> - A managed sandbox image can outlive the model defaults shipped by
the app.
> - Claude setup skipped installation whenever a CLI was already on
PATH.
> - The selected model then failed its minimum-version check after
sign-in succeeded.
> - This pull request prepares a compatible private CLI before testing
or running it.
> - Cloud onboarding also stops asking customers to run inaccessible
shell commands.

## Linked Issues or Issue Description

**What happened?**

A new managed Cloud stack completed Claude subscription sign-in. Its
environment test then failed because the snapshot contained Claude Code
2.1.263, while the selected `claude-opus-5-5` model requires 2.1.280.
The screen displayed Manual debug commands that a hosted customer cannot
run.

**Expected behavior**

Paperclip prepares its managed runtime before testing the selected
account. A stale image CLI does not require the customer to upgrade a
server package. Failed preparation still blocks hiring. Cloud does not
display manual shell debugging.

**Steps to reproduce**

1. Start a managed sandbox with Claude Code 2.1.263 on PATH.
2. Connect a Claude subscription and select `claude-opus-5-5` on the CLI
lane.
3. Connect the agent. Installation is skipped because the command
exists, then the version gate blocks the hello test.

**Paperclip version or commit**

Source baseline: `6204239cfc8c3be79fc185cf3f468ebaa0ab1e3e`.

**Deployment mode**

Authenticated managed Cloud with a Daytona named snapshot. The
correction also applies to the default CLI in other managed sandbox
transports.

Related work: Refs #14993 sets model/runtime requirements; Refs #15993
changes native defaults on qualified targets. Neither prepares an
outdated default sandbox CLI on this path. Searches found no open
duplicate.

## What Changed

- Check the actual default CLI against the selected model's existing
minimum version.
- When a sandbox's default command is outdated, install the exact
required npm package into a private versioned workspace prefix. Reuse a
compatible private installation. Keep the image's global CLI intact.
- Verify the private executable and use that same command for the
account hello test and normal CLI runs, with matching command metadata.
Probe effort support after preparing that executable so the chosen
effort is retained. Keep the model, account, engine and permissions
unchanged.
- Return safe preparation checks to environment tests and normal run
logs, distinguishing installation failure causes without returning raw
installer output or credential-bearing URLs.
- Exclude provider and Paperclip credentials from the installer
environment. Preserve the existing failing version gate if installation
or verification cannot succeed. Local, SSH and explicit custom commands
remain operator-managed.
- Hide Manual debug and legacy operator check hints on Cloud and while
deployment metadata is unknown; preserve check messages and details.
Retain self-hosted help and the detailed failed checks. Cloud copy asks
for a retry instead of asking the customer to repair the environment.
- Add subprocess-backed runtime controls, an account-probe regression
and mounted Cloud/self-hosted wizard controls. Update development
guidance and the affected capability maps.

## Verification

- Passed all Claude source suites and the server Claude execution suite
on head `c941ae44c22d8dafec0df033a129e39ce4ebd966`: **412 passed**, one
existing opt-in test skipped. These include the new command/effort
ordering, safe installer diagnostics, environment forwarding and run-log
regressions.
- On final head `0ecae0e9304f495a57524158b3700b8e9bc21cb8`, passed **208
tests** across all 174 mounted onboarding wizard tests and 34 Claude
execution tests. The failure fixture includes the real command-bearing
hint, and execution asserts both metadata command fields agree.
- Passed `pnpm -r typecheck`, `pnpm build`, `pnpm check:token-gates`,
`node scripts/check-maps.mjs` and `git diff --check`. Full typecheck and
build were repeated successfully on final head
`0ecae0e9304f495a57524158b3700b8e9bc21cb8`. [Latest-head
CI](https://github.com/paperclipai/paperclip/actions/runs/39879208694)
is green: 54 checks passed and two skipped, including browser E2E and
serialized database suites.
- The new runtime controls execute real subprocesses with a controlled
installer. They cover an old CLI, an already compatible CLI, failed or
ineffective installation, local/custom commands, and an unknown version.
The probe regression asserts that the hello call uses the private
executable.
- A separate non-root local subprocess check installed the real npm
`@anthropic-ai/claude-code@2.1.280` package. Its private command
reported 2.1.280, the simulated image CLI remained 2.1.263, and a second
preparation reused the private copy. This ran on macOS ARM64. It did not
use a provider account, send a model prompt, or prove live Daytona
behavior.
- Attempted `pnpm test:run`. Embedded PostgreSQL initialization failed
after five attempts on this Mac with `Postgres init script exited with
code null`. The unchanged native-session database cases then failed.
Stopped only that full-suite process tree after recording the repeated
startup failure. No full local-suite pass is claimed; Linux CI must
cover those tests.
- The first new subprocess fixture had a quoting syntax error. It was
corrected before the passing focused runs. The one-off real-package
script first required an ESM extension correction, then passed. Logs
retain both attempts. A new execution assertion initially inspected the
final sandbox cleanup shell instead of the launched command; it now
checks execution metadata. A broader directory selection also picked up
generated `dist` tests whose Markdown fixtures are not copied by the
build (two ENOENT failures). The explicit source-suite run above passes;
no generated-test pass is claimed.
- Apex review: **5/5**, reviewed final head
`0ecae0e9304f495a57524158b3700b8e9bc21cb8`, all five earlier findings
addressed and resolved, no new findings.
- Merge, deployment and a fresh live Cloud subscription test are still
required. No customer account or production runtime was changed by this
branch.

## Risks

- An outdated sandbox needs npm registry access and up to 120 seconds
for one bounded installation attempt. A failed or unverified install
continues to block execution.
- The versioned package uses sandbox workspace storage. Compatible
cached copies avoid repeat installation within that workspace. Existing
runtime cleanup and sandbox lifecycle still apply.
- The compatible version comes from the adapter's existing model-version
contract. This does not add a model fallback or change the selected
execution engine.
- No schema, credential, permission, workflow or lockfile change.

## Model Used

OpenAI Codex, based on GPT-6, assisted with implementation and
verification. The exact serving model ID and context-window limit are
not exposed in this session. Capabilities used: code editing, shell
execution, API inspection and test execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-22 18:44:53 -07:00
Devin FoleyandPaperclip 650581b2f0 feat: give nested managed checkouts their own restore points (#17257)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - #17248 lets git cache expiration remove a closed issue's worktree
with unsaved work after saving an encrypted restore point
> - Workspaces with nested `.paperclip-repositories` checkouts were
still kept whenever a checkout had unpushed or uncommitted work
> - On one instance that was the second most common reason expiration
skipped a workspace
> - This pull request gives each nested checkout its own restore point
and restores it when the workspace comes back
> - The benefit is that multi-repository workspaces can be cleaned up as
safely as single-repository ones

## Linked Issues or Issue Description

Refs #17238, #17241, #17248. Bug report described in-PR (bug report
template).

**What happened?**
With restore points enabled, a workspace whose nested managed checkout
has unpushed commits or uncommitted files is still skipped with
`unpushed_managed_checkout` or `dirty_or_untracked_files`. On one
instance, `unpushed_managed_checkout` accounted for 32 of the last 111
expiration checks.

**Expected behavior**
A nested checkout's unsaved work is saved like the main worktree's, and
the workspace can expire and later be restored exactly.

**Steps to reproduce**
1. Use a project with an extra repository, so the task worktree contains
`.paperclip-repositories/<name>`.
2. Commit in that checkout without pushing, then close the issue.
3. With restore points enabled, the workspace is never expired.

**Paperclip version or commit:** `master` at 6d44b80a5e.
**Deployment mode:** any.
**Agent adapter(s) involved:** not adapter-specific (core).

## What Changed

**Relaxed checkout check** (`assertReproducibleManagedCheckout(..., {
restorePoint: true })`, used only in restore point mode):
- **Allowed:** the checked-out branch's unpushed commits, plus its
uncommitted, untracked and history-only work.
- **Still blocks:**
- another local branch or tag whose tip the remote lacks and `HEAD` does
not contain (`unpushed_managed_checkout_branches`), because restore
points do not recreate other branches;
- commits in any reflog that are neither on the remote nor in the
bundled history (`HEAD`'s reflog and `ORIG_HEAD`), such as work left in
another branch's reflog;
  - objects reachable from nothing;
  - a `COMMIT_EDITMSG` that matches no commit in the checkout's history.
- **`COMMIT_EDITMSG`:** a nested checkout is a full clone, so its base
git storage check now accepts a `COMMIT_EDITMSG` that matches a commit
in its history, in strict and relaxed mode. It is read only if it is a
regular file of at most 64 KB, opened without following symlinks and
without blocking; anything else keeps the workspace
(`unsupported_commit_message`). The main worktree's existing
`COMMIT_EDITMSG` read uses the same bounded reader. A clean checkout
with only an unpushed commit therefore reports
`unpushed_managed_checkout`, which restore point mode can relax, instead
of `unknown_base_git_files`.

**Capture** (`git-workspace-expiration.ts`):
- **Expiration flow:** strict inspection still runs first.
`unpushed_managed_checkout` is now relaxable. The relaxed prover records
which checkouts need restore points.
- **Storage:** each checkout's restore point is stored under
`.../<generation>/checkouts/<name>/<id>.pcrp`, against a remote-verified
boundary from that checkout. It is recorded on its
`contents.checkouts[]` entry.
- **Activation:** all restore points of the generation are activated
together in the archive transaction. Any nested restore point makes the
recipe `paperclip.git-cache-restoration.v2`, so older servers reject it.
- **Schema:** migration `0374` adds `checkout_name` to
`execution_workspace_restore_points` and includes it in the
live-generation unique index.
- **Final check:** under the expiration locks, each checkout with a
restore point gets the relaxed check and must match its own snapshot.
Checkouts without one get the strict check and must still be clean.
Every restore point, the main worktree's included, records a digest of
its `HEAD` reflog and `ORIG_HEAD` at capture, and that history must be
unchanged, so a commit followed by a reset after capture rolls back.
Rollback drops every restore point reference, returns the recipe to v1,
and supersedes the stored restore points.

**Restore** (`git-workspace-contents.ts`):
- **Re-cloned checkouts:** re-cloning a checkout imports its bundle
before the saved commit is needed. After checking out the saved commit,
it applies and verifies the restore point.
- **Existing checkouts:** the same helper as the main worktree marks a
matching checkout applied, and fails closed instead of overwriting a
diverged one.

**Validation:** `readGitCacheRestoration` and
`savedGitWorkspaceRestoration` validate nested restore point references
against their checkout names. Contents comparison ignores the reference.

**Docs:** `doc/recoverable-disk-cleanup.md` describes nested checkouts.

No code comments were added.

## Verification

- `pnpm exec vitest run src/__tests__/git-workspace-expiration.test.ts
-t "restore points for nested checkouts"` (in `server/`): 10 passed,
with a real nested checkout and embedded Postgres. The cases:
- **Round trip:** an unpushed commit and an untracked file in the nested
checkout survive expiration and reopen. The recipe is v2, the restore
point lives on the checkout entry, and a ledger row carries the checkout
name.
- **Another branch:** unpushed work on another local branch keeps the
workspace.
- **Change after capture:** a nested change after capture rolls back,
keeps the files, returns the recipe to v1, and supersedes the restore
point.
- **Only an unpushed commit:** a clean nested checkout with only an
unpushed commit (and its `COMMIT_EDITMSG`) expires and restores.
- **Reflog-only history:** unpushed history held only in another
branch's reflog keeps the workspace.
- **History after capture:** a commit followed by a reset after capture
rolls back, in a nested checkout and in the main worktree.
- **Unsafe `COMMIT_EDITMSG`:** a named pipe, an oversized file, or a
symlink keeps the workspace without reading it unsafely.
- **Mutation checks:** each of these fails a case:
- removing the other-branch check, the per-checkout snapshot check, the
rollback cleanup, or the checkout apply;
  - rejecting a saved `COMMIT_EDITMSG`;
  - dropping reflog coverage;
  - dropping the history digest check;
  - removing the size cap.

Not covered by a dedicated case: a workspace with two nested checkouts
where only one has a restore point.
- `pnpm exec vitest run git-workspace-expiration git-workspace-contents
git-restore-point execution-workspaces` (in `server/`): 232 passed.
- `pnpm run check:migrations` (in `packages/db`): pass.
- `pnpm run typecheck` (in `server/`): pass.
- The server builds, the runtime bundle builds, and
`scripts/server-boot-smoke.sh node server/dist-runtime/index.mjs`
reaches healthy.

## Risks

- **Off by default:** nothing changes unless
`PAPERCLIP_GIT_RESTORE_POINTS_ENABLED=true`.
- **Other local branches:** a nested checkout's other local branches are
not saved. Any of them with commits the remote lacks keeps the workspace
instead.
- **Migration:** adds a defaulted column and replaces one partial unique
index on a table only written when restore points are enabled.
- **Rollback:** disable the setting. Workspaces already expired with
nested restore points keep v2 recipes that require this code to restore.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file edits, tests, local server builds and boots,
read-only measurements on live instances).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-22 18:42:22 -07:00
Nicky LeachandPaperclip 989ee668e0 Support provider-owned fresh retry workspaces (#17252)
## Thinking Path

> - Paperclip coordinates AI agents and retains their work across
failures.
> - A retry can require a new workspace while preserving the failed
attempt.
> - Core rejects this recovery strategy for provider-owned storage.
> - Providers should implement the required outcome using their own
storage design.
> - This pull request adds a semantic fresh-preparation contract and
delegates recovery preparation.
> - The benefit is that a supported provider can run a fresh retry
without losing prior work.

## Linked Issues or Issue Description

**What happened?**
A retry that selects `fresh_isolated_workspace` fails with
`fresh_workspace_storage_unsupported` for provider-owned storage.

**Expected behavior**
Core asks a capable provider to prepare new storage and attach verified
recovery files. The failed workspace remains intact.

**Steps to reproduce**
Run a task in a task-capable plugin environment. Let the task fail
without a verified workspace preservation receipt. Retry the task so
Core selects a fresh attempt.

**Paperclip version or commit**
6204239cf (rebased on current master).

**Deployment mode**
Self-hosted server with a plugin task environment and managed workspace
provider.

Search found no duplicate implementation. This branch includes the
default-provider changes from #17247 and preserves their source
selection behavior.

## What Changed

- Define an opt-in fresh-preparation contract with a stable attempt
identity and verified recovery file descriptors.
- Describe retained-storage and recovery-attachment guarantees without
requiring a Git storage layout.
- Reserve replacement task bindings with revision checks and keep the
failed workspace intact. Projectless and project-backed task
environments both replay their stable attempt reservation.
- Reconcile an interrupted preparation against its reserved workspace
and stable operation identity.
- Stage verified recovery archives at stable transfer paths. Delegate
attachment materialization to the provider.
- Preserve projectless attempt directory identities through later turns
and pin the intent into each execution lease.
- Protect fresh intent and projectless ownership from caller metadata
replacement or injection.

## Verification

- SDK workspace-provider tests: 4 passed.
- Targeted reservation and task-dispatch tests: 31 passed.
- Recovery transfer staging and lifecycle focus: 7 passed. The separate
cache-cleanup failure reconciliation case also passed.
- After rebasing on current master, 7 focused preparation and ownership
cases passed. These cover inherited provider defaults, fresh
preparation, stable projectless identity, and authorization denial.
- The focused metadata authority regression passed, including forged
intent, replacement, and null metadata patches.
- Three assembled interrupted task-reservation cases passed:
projectless, project with a saved source, and project with a newly
selected default source.
- Targeted tests use worktree SDK sources. No full local build or
typecheck ran.
- Local checks remain targeted as requested. Latest-head GitHub CI
passed: 54 successful checks, including typecheck, build, server suites,
Runner validation, and E2E shards; 2 unrelated Storybook checks were
skipped.
- Greptile reviewed commit 73595b0c61 at
5/5 with zero unresolved threads.
- Live VM E2E has not run.

## Risks

- Existing providers retain their existing behavior. Fresh attempts
require explicit version support.
- The companion provider implementation must materialize recovery
snapshots safely and preserve the old workspace.
- Fresh retries require the companion provider implementation. Providers
without the version marker fail closed.

## Model Used

- OpenAI GPT-6.1 (`gpt-6.1-sol`), reasoning and coding tools. The
runtime does not expose the context-window limit.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-22 17:46:27 -07:00
Devin FoleyandPaperclip 35bd7f6059 Reduce encrypted event retries and GitHub run-query transfer (#17251)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Its Cloud connector delivers encrypted GitHub events to an instance.
> - One failed inner event currently rejects the whole authenticated
batch.
> - Healthy events then remain unacknowledged and are downloaded again.
> - GitHub response and check reconciliation also reads full run records
when it needs only ownership and status.
> - This pull request isolates inner failures and narrows those run
projections.
> - Valid events can finish while failed events remain available for
recovery.

## Linked Issues or Issue Description

**What happened?**

An inner App event encrypted with an unavailable key causes
`leaseEvents` to reject the complete batch. No sibling can finish or
acknowledge. Repeated retries download the same encrypted payloads.
Separately, GitHub response and check reads load prompts, runtime
profiles, results, and log excerpts that their callers do not use.

**Expected behavior**

Authenticate the outer batch and lease before handling individual
events. Process valid siblings independently. Retain failed events
without acknowledging them. Read only the current run fields needed for
response ownership and status checks.

**Steps to reproduce**

1. Seal one event for an old instance key and another for the current
key.
2. Put both events in a correctly authenticated outer lease.
3. Observe that the old client rejects the whole lease.
4. With this change, the valid event processes, the failed ID is
reported locally, and only completed events acknowledge.

No public issue exists. The change does not add a provider integration
or duplicate a roadmap feature.

## What Changed

- Keep outer authentication, environment checks, and lease binding
strict.
- Isolate inner decryption and route-binding failures. Return their
authenticated IDs only to the local poller and count them as failures.
- Leave failed events unacknowledged in the broker. Continue processing
and acknowledging healthy and duplicate siblings.
- Read only run identity, status, and required request context for
GitHub responses. Read only run ID and status for check reconciliation.
- Cover stale keys, swapped routes, invalid outer bindings, all-failed
batches, large unused run payloads, and cross-company/task refusals.

## Verification

- Connector cryptography suite: 47 tests passed.
- Final connector suite: 50 tests passed, including cryptography and
three real PostgreSQL connection-event tests. These cover acknowledgment
isolation, all-failed batches, and the exact safe warning fields.
- GitHub workflow regression suite after the master refresh: 307 tests
passed, including the large-payload/company/task test (1,084 unrelated
tests skipped by the selection).
- Server typecheck passed after the final source changes.
- `pnpm -r typecheck` and `pnpm build` passed after the latest
conflict-free master refresh.
- The broad local `pnpm test:run` was stopped when the branch was
refreshed; it has no complete passing result. All CI gates passed on the
final commit, including general and serialized server suites, runner
verification, build, typecheck, and end-to-end tests. Apex is 5/5 with
all review comments resolved. The last change removes source comments
and expands docs; both edited TypeScript files emit identical JavaScript
before and after that change.
- A read-only sample compared full run records with the new field sets:
response data decreased by about 96%, and check data by over 99%. This
is serialized query output, not a claim of realized billing savings or
CPU reduction.
- An initial database test setup failed because the local dependency
install omitted native-library symlinks. The package's standard setup
script repaired them. The unchanged test timeout then passed.
- Added lines checked for credentials, customer data, private links, and
internal issue references.

## Risks

Failed inner events remain pending and retry under the broker's existing
lease and retention policy. This change does not recover a lost original
private key or introduce a dead-letter deletion path. Outer
authentication failures still reject the complete batch.

Run projections must retain request provenance and task binding fields.
Existing predicates, ordering, identity validation, and publication
authority remain unchanged. The change has no schema migration and adds
no background timer or new provider call.

## Model Used

OpenAI GPT-6 through Codex, with reasoning and shell/GitHub CLI tools.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used with version and capability
details
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs
- [x] I have described the issue in-PR following the bug report template
- [x] I have not referenced internal or instance-local Paperclip issues
or links
- [x] My branch name describes the change and contains no internal
ticket ID
- [x] Focused local regressions and all CI gates pass
- [x] I have added and updated tests
- [x] I have updated relevant documentation
- [x] I have documented risks
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-22 17:40:29 -07:00
Nicky LeachandPaperclip 6204239cfc Support installed default project workspace providers (#17247)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Storage plugins already own project storage and task checkout
preparation.
> - Core only selected these plugins through an explicit project policy.
> - Installation could therefore leave new and existing projects on
incompatible Core preparation.
> - This pull request adds one generic manifest declaration and one
shared effective-policy resolver.
> - Project configuration shows inheritance while durable workspace
bindings retain their owners.

## Linked Issues or Issue Description

**Subsystem affected**

Plugin contracts, project configuration, and execution workspace
preparation.

**Problem or motivation**

An installed storage provider cannot become the company default.
Projects without an explicit provider keep Core preparation, even when
their execution environment cannot use a Core Git checkout.

**Proposed solution**

A provider declares supported `workspaceProvider.defaultForProjects`
mode and strategy. Company-enabled installations supply inherited
defaults. Explicit plugin selections and explicit Core/local
(`storageProviderPluginId: null`) win. Conflicting defaults require an
operator choice. The declaration and implementation contain no
provider-specific identifiers.

**Alternatives considered**

Rewriting every project would lose the distinction between inheritance
and overrides. Selecting by worker health would silently change storage
on an outage.

**Roadmap alignment**

Extends the existing workspace and sandbox provider contracts. The
roadmap has no duplicate default-provider implementation. Related public
work includes #17243, which covers prepared execution directories rather
than provider selection.

## What Changed

- Add a validated generic default-provider declaration and document it
in the SDK.
- Resolve provider selection and Git defaults consistently in project
responses, source selection, heartbeat source resolution, reservation,
and preparation.
- Keep stored policy separate from effective configuration. Add the
normal project configuration selector for inheritance, explicit plugins,
and Core/local.
- Preserve existing local and plugin workspace ownership. Pin selection
through preparation and keep provider failures visible.
- Add database, API, manifest, heartbeat, and UI regressions. Update the
verification maps.

## Verification

- Targeted Vitest runs cover manifest validation, project defaults and
API access, explicit overrides, existing ownership, new/imported task
inheritance, heartbeat routing, source-free tasks, selection pinning
during setup, and UI configuration.
- Combined policy, provider integration, and heartbeat run: 58 tests
pass; 60 unrelated cases excluded. After the final shared-provider
fallback fix, all 47 policy tests pass, including shared and
operator-mode regressions.
- Server and UI TypeScript checks, SDK build, verification-map
validation, credential-pattern scan, added-comment scan, and `git diff
--check` pass.
- A downstream integration uses this Core resolver and preparation
service with a real storage hook and mocked remote infrastructure. A new
Git project reaches the hook without a stored provider ID.
- Standard Greptile and Apex both scored 5/5 on
5d42998466. All review threads are
resolved.
- Final GitHub CI is fully green on
5d42998466. Browser shard 3 initially hit
a PostgreSQL deadlock during cleanup cancellation, which left an
experiment flag enabled for a later test. A failed-job retry passed all
26 browser tests. The workflow completed successfully on attempt 2. Full
final verification was delegated to CI at the user's request.
- No live provider or hosted tenant run was performed.

## Risks

- Installing one default-claiming provider changes effective selection
for unconfigured projects. Existing explicit settings and workspace
ownership remain intact; incompatible explicit settings fail clearly.
- Each configuration/admission resolution reads installed plugin
metadata and the selected company's enablement. Project list reads share
one catalog query. Selection does not call workers or external
providers.
- Deploy the Core contract before any downstream manifest uses the
declaration. No migration or backfill is required. A downstream
immutable image pin must include this change before packaging its
declaration.
- No merge or deployment is authorized by this PR.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository tools, and code
execution. The exact served model ID and context-window size are not
exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have described the issue in-PR following the relevant issue
template
- [x] I have not referenced internal/instance-local Paperclip issues or
links
- [x] My branch name describes the change and contains no internal
ticket id
- [x] I have run targeted tests locally and they pass; full gates are
delegated to CI at the user's request
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green on
5d42998466ad42400564ba34f48124f40c53fbc5; the branch has no merge
conflicts
- [x] Standard Greptile is 5/5 on 5d4299846 after all fixes, with no
unresolved threads
- [x] Apex is 5/5 on 5d42998466, after
standard Greptile 5/5 on that same head
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-22 17:12:29 -07:00
Devin FoleyandPaperclip 6d44b80a5e feat: expire closed git worktrees with unsaved work behind restore points (#17248)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Git cache expiration removes a closed issue's worktree only when the
remote can reproduce it exactly
> - Most closed worktrees hold unpushed commits or uncommitted files, so
they are kept forever and disks fill up
> - #17238 added restore points (an encrypted save of exactly what the
remote cannot reproduce) and #17241 added their lifecycle
> - This pull request lets expiration remove those worktrees after
saving a restore point, and restores them exactly when the work resumes
> - The benefit is bounded disk use for closed work without losing
anything a user could need, behind a setting that is off by default

## Linked Issues or Issue Description

Refs #17238, #17241. Bug report described in-PR (bug report template).

**What happened?**
With `PAPERCLIP_GIT_CACHE_EXPIRATION_ENABLED=true` and dry run off, one
instance's last 111 expiration checks removed nothing:

| Skip reason | Checks |
|---|---|
| `dirty_or_untracked_files` | 39 |
| `unpushed_managed_checkout` | 32 |
| `live_or_unresolved_consumer` | 21 |
| `unique_ignored_files` | 9 |
| commit not on the remote | 10 |

Without expiration, the terminal workspace reaper archives only merged,
clean worktrees. Closed issues' worktrees therefore accumulate without
bound.

**Expected behavior**
A closed worktree whose only obstacle is unsaved work can be removed
after that work is saved, and restored exactly if the issue is reopened
or work resumes.

**Steps to reproduce**
1. Close an issue whose worktree has an unpushed commit or an
uncommitted file.
2. Enable git cache expiration with dry run off.
3. The worktree is never removed.

**Paperclip version or commit:** `master` at 12bb7e1976.
**Deployment mode:** any.
**Agent adapter(s) involved:** not adapter-specific (core).

## What Changed

**Expiration** (`server/src/services/git-workspace-expiration.ts`):
- **When it applies:** with `PAPERCLIP_GIT_RESTORE_POINTS_ENABLED=true`
(default `false`), a worktree that fails the reproducibility check for a
relaxable reason is reconsidered once its issue tree has been closed for
`PAPERCLIP_GIT_RESTORE_POINT_TERMINAL_DELAY_DAYS` (default `3`). The
relaxable reasons are uncommitted or untracked files, a commit not on
the remote, reflog or `ORIG_HEAD` history, and ignored `.env` files. A
leftover `COMMIT_EDITMSG` still blocks expiration unless it matches a
commit in the worktree's reflog, because restore points do not save
draft messages.
- **What still blocks:** a relaxed inspection
(`inspectGitCacheRecipeForRestorePoint`) keeps every other check. Nested
managed checkouts, submodules, special git state, local config and other
unique ignored files still prevent expiration.
- **Capture:** inside the existing source and lifecycle locks, a restore
point is captured with the #17238 library against a remote-verified
boundary (the `HEAD` itself, or its merge base with `origin/HEAD`), then
encrypted, uploaded, read back and hash-checked
(`git-restore-point-storage.ts`).
- **Commit before touching disk:** it is activated in the #17241 table
in the same transaction that archives the workspace with a
`paperclip.git-cache-restoration.v2` recipe referencing it, still before
anything on disk is touched.
- **Final check:** under the file locks, the worktree must still match
the captured snapshot (`HEAD`, index, working tree, modes). Otherwise
the existing rollback runs. The rollback now also drops the restore
point reference and returns the recipe to v1, so a later restore can
never apply a stale restore point over the live worktree; the stored
restore point is marked superseded.
- **Failure paths:** the `pending` row is inserted in a transaction that
first registers a `git-restore-points` notification. If the upload or
read-back fails, the row is marked `superseded` so retention deletes any
uploaded object. Nothing is deleted from disk.
- **Encryption key:** restore points are refused
(`restore_point_key_not_durable`) on `s3` storage unless the secrets
master key comes from `PAPERCLIP_SECRETS_MASTER_KEY`.
- **Dry run** reports `restorePoint: true` candidates without storing
anything.

**Restore** (`git-workspace-restoration.ts`, `workspace-runtime.ts`,
`execution-workspaces.ts`):
- **Recipe versions:** `readGitCacheRestoration` and
`savedGitWorkspaceRestoration` accept v1 without a restore point, or v2
with a valid one. Older servers reject v2 instead of restoring the
commit without the saved work.
- **Before the saved commit is needed:** reopening
(`restoreGitCacheBase`) and the runtime path import the bundle when the
saved commit is missing locally.
- **After the worktree is re-added:** the runtime always imports the
bundle, so history-only commits are present again as objects even when
the base was rebuilt (they are not recreated as branches), then applies
the restore point, verifies the result, and records `appliedAt`. Reopen
persists that through a callback, as it already does for saved contents.
- **Never overwriting:** a worktree that already matches is only marked
applied. A worktree that diverged from the saved commit's clean state is
never overwritten; the restore fails closed and can be retried. Missing,
corrupt or expired restore points also fail closed.

**Retention:** the #17241 worker's expiry hook now marks the recipe's
restore point expired before its object is deleted.

**Config and docs:**
- `server/src/config.ts` adds the two settings;
`doc/recoverable-disk-cleanup.md` and `doc/DEVELOPING.md` describe them.
- No code comments were added.

**Not in this PR (follow-ups):**
- workspaces with nested `.paperclip-repositories` checkouts;
- removing idle worktrees of issues that are still open;
- starting a fresh workspace with a warning when a restore point has
expired.

## Verification

- `pnpm exec vitest run
src/__tests__/git-restore-point-expiration.test.ts` (in `server/`): 12
passed, with real repositories and embedded Postgres. The cases:
- **Round trip:** an unpushed commit, staged and unstaged edits of one
file, a deletion, and an untracked file are restored exactly on reopen,
`.env` is dropped, and the restore point is active. Restoring through
the runtime path works the same way, and a later run never re-applies
over newer work.
- **Kept:** a worktree that is not yet due, one with the feature
disabled (today's skip reason), and one whose encryption key is not
durable.
- **Change after capture:** rolls back, keeps the worktree, returns the
recipe to v1 and supersedes the restore point.
- **Missing restore point:** fails closed, keeps the row archived and
unapplied, and fails closed again on retry.
  - **Dry run:** stores nothing.
  - **Reproducible worktrees:** expire as before, with a v1 recipe.
- **History-only commits:** reopen imports them even after they are
pruned from the base.
  - **Draft messages:** a draft commit message keeps the worktree.
- **Upload failure:** a failed upload marks the pending restore point
for cleanup and keeps the worktree.
- **Mutation checks:** each of these fails at least one case:
  - removing the pre-deletion snapshot check;
  - keeping the restore point reference on rollback;
  - removing the runtime apply;
- removing the re-apply guard (the divergence check then fails closed
instead of overwriting);
- skipping the import for a matching worktree, relaxing the draft
message guard, or not marking a failed upload's row.
- `pnpm exec vitest run
src/__tests__/git-restore-point-expiration-config.test.ts`: 5 passed.
- `pnpm exec vitest run git-workspace-expiration git-workspace-contents
git-restore-point workspace-runtime execution-workspaces` (in
`server/`): 529 passed on the first revision. Two
`workspace-runtime.test.ts` cases about remote base refs fail, and they
also fail on unmodified `master` on this machine. After the review
fixes, the expiration, contents, restore point and execution workspace
suites passed again: 222 tests.
- `pnpm run typecheck` (in `server/`): pass.
- The server builds, the runtime bundle builds, and
`scripts/server-boot-smoke.sh node server/dist-runtime/index.mjs`
reaches healthy.

## Risks

- **Off by default:** nothing changes unless both
`PAPERCLIP_GIT_CACHE_EXPIRATION_ENABLED=true` and
`PAPERCLIP_GIT_RESTORE_POINTS_ENABLED=true` are set. Roll out with
`PAPERCLIP_GIT_CACHE_EXPIRATION_DRY_RUN=true` first to see which
worktrees would receive restore points.
- **Data at rest:** restore points contain source and uncommitted files,
encrypted with keys wrapped by the instance secrets key. Losing that key
makes them unrecoverable, which is why `s3` storage requires an
environment-provided key. Untracked `.env` files are not stored.
- **Interrupted apply:** a crash in the middle of applying a restore
point leaves a worktree that neither matches the restore point nor is
clean. The restore then fails closed rather than overwriting it, and an
operator must reset that worktree to retry.
- **Rollback:** disable the setting. Already-expired worktrees keep
their v2 recipes and restore points, so reverting the code would leave
those recipes unreadable until restored by a server with this change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file edits, tests, local server builds and boots,
read-only measurements on live instances).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-22 16:50:15 -07:00
DottaandPaperclip 9bf7dc0c8f feat(agents): bulk edit connections and default models (#17215)
Add harness-scoped bulk connection and default-model edits to the agent roster, with compatible choices, change previews, per-agent save results, and retries.

Use an icon-based harness filter, keep agent rows readable, remove the status selector and selection hint, and redirect legacy status URLs to the full roster.

Preserve authorization, audit history, unrelated settings, and optimistic concurrency checks through the existing agent update route.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-22 18:47:06 -05:00
Dotta 46760cc588 fix(chat): honor dismissal for all composer requests (#17218)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Task conversations show requests in the feed and above the message
composer.
> - Earlier fixes kept dismissed questions in history, but excluded
completion reviews and other request kinds.
> - Closing those forms left a pending badge, and the server could still
demand a response.
> - This pull request saves human dismissal and removes the request from
composer reminders and ordinary completion gates.
> - People can continue the conversation and reopen the request from
history.

## Linked Issues or Issue Description

Related work: #15229 and #15392.

**What happened?**

Dismissing a completion review closed its form but left it counted as
pending. It could return after reload, block a retry, or cause the agent
to demand another response. Raw runtime requests could also lose their
only usable control when dismissed.

**Expected behavior**

Dismissal leaves an accessible request in the feed and clears the
composer form and pending count. It persists after reload. It does not
submit an answer, approve an action, or wake the agent.

**Steps to reproduce**

1. Open a task with a pending completion review or another composer
request.
2. Dismiss the request, then reload the task.
3. Check that the composer stays clear and the feed entry can reopen the
request.

## What Changed

- Add nullable dismissal fields and a human-only dismissal endpoint with
company, private-task access, resolver, and locked review-policy checks.
- Persist saved-request dismissal on the server. Keep browser
persistence for raw runtime requests and existing question dismissals.
- Keep confirmations, choices, item verdicts, suggested tasks, and
runtime requests accessible in the feed.
- Exclude dismissed ordinary requests from completion and retry gates.
Tell agents not to repeat deferred requests.
- Preserve tool, secret, connection, plan, company-policy, and
configured review authorization gates. Expire deferred non-question
requests when their task completes.
- Preserve the selected form when another dismissal finishes saving.
Keep reopened reviews visible through delayed comment refreshes.
- Add a completion-review Storybook test drive and update design, API,
database, and capability documentation.

## Verification

- `pnpm -r typecheck`: passed. The final UI changes also passed the UI
typecheck.
- `pnpm build`: passed. The final UI changes also passed the UI
production build.
- Focused UI and transcript tests: 550 passed across nine files after
merging the current base branch.
- Focused database, completion, and prompt tests: 105 passed across
three files, including two review-policy regressions that failed before
the fix. The tests reject a review requester excluded by current policy,
preserve the pending gate on denial, and permit an independent reviewer
to defer the request.
- Route regression suite: 81 passed, including denial before reading or
mutating a private-task request.
- `pnpm check:token-gates`: passed.
- Map and Storybook structural checks: passed; 30 map/catalog script
tests passed.
- Browser: used the production task-thread components in the
completion-review fixture. Dismissed, reloaded, reopened from the feed,
and dismissed again. No pending badge or composer takeover remained.
Story callbacks simulate API saves; separate server tests cover
persistence and permissions.
- `pnpm test:run`: stopped after the general-server group reported one
failure in `workspace-runtime.test.ts`: "returns a bounded structured
conflict and exposes only same-company workspace references". The same
test passed in isolation (1 passed, 178 skipped). The cause of the
full-suite failure is not established. The full PR CI suite subsequently
passed on e243cbb001; its two authorization review findings are fixed in
9e1b4478b3. Fresh [full
CI](https://github.com/paperclipai/paperclip/actions/runs/39871944846)
passes on 9e1b4478b3, including the previously failing server and
browser shards. All 54 executed checks passed; two optional Storybook
deployment/visual jobs were skipped. The legacy security status also
passed. Fresh Greptile review is 5/5, the security scan passes, and no
review threads remain open. This full CI run settles the merge gate; the
earlier local workspace-suite failure remains recorded above because its
cause was not established.
- Additional verification: the original failing browser journey passed
locally after the delayed-refresh fix. The expanded page-reload case
also passes on 9e1b4478b3 (52.7s): dismiss, reload, reopen, approve, and
receive the actual tool failure in the continuing task. It exposed a
fixture reading an accepted request before tool execution finished; the
fixture now waits for a stored execution result. The first reload
attempt stopped on a still-loading page at the default five-second
assertion; the reload now uses the suite’s navigation allowance. A
controlled server-restart run timed out waiting for server health before
the UI assertions. Local route-suite runs timed out during concurrent
module loading; sequential imports passed all 81 tests, including the
privacy regression.
- Initial verification failures: a missing SDK build and a story fixture
type error were fixed. PostgreSQL bootstrap then failed because the host
had exhausted its semaphore allowance. After explicit operator approval,
64 verified abandoned PostgreSQL semaphore sets were reclaimed; no live
process was stopped. The database regressions then passed. The
raw-runtime regression exposed a missing feed control; the
implementation now preserves that entry and the regression passes.

## Risks

- Apply the additive database migration before the new server code. It
adds two nullable columns and does not change existing decisions.
- A dismissal is shared task attention state. Only a human with task
read access and current resolver/review authority can persist it. Review
policy is checked under the task lock. The operation is idempotent and
records the dismissing user.
- Raw runtime dismissal remains specific to a browser. Saved interaction
dismissal persists across browsers.
- A deferred review is not an approval. Tests retain protected-action
and configured review gates.
- The browser walkthrough uses a fixture. This branch has not been
deployed to a live instance.

## Model Used

OpenAI Codex, GPT-6, with code execution and browser tools. The exact
backend model ID and context-window size are not exposed in this
session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-10-22 18:37:36 -05:00
Devin FoleyandPaperclip 12bb7e1976 feat: track git workspace restore points and apply their retention (#17241)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Closed issues' git worktrees pile up on disk because they can only
be removed once the remote reproduces them exactly
> - #17238 adds restore points: encrypted saves of the unpushed and
uncommitted parts of a worktree, so the worktree itself can be removed
> - Those saves live in file storage and must be deleted on a schedule,
and promptly when their issue or company is deleted
> - This pull request adds the table that tracks restore points and a
deadline-driven retention worker, with nothing creating restore points
yet
> - The benefit is a reviewable lifecycle foundation before capture is
wired into expiration

## Linked Issues or Issue Description

Refs #17238 (restore point library). Bug report described in-PR (bug
report template).

**What happened?**
Closed issues' worktrees accumulate without bound. For example, one
instance's last 111 git cache expiration checks removed nothing, because
each worktree held unpushed commits or uncommitted files. Restore points
(#17238) will let those worktrees be removed safely. That needs a
durable record of every stored restore point and a retention policy for
deleting them.

**Expected behavior**
- A restore point is kept while its work can still resume.
- It is deleted a fixed time after its issue tree closes.
- It is deleted promptly when its issue or company is deleted.

**Steps to reproduce**
Not applicable on its own: this adds the lifecycle that the capture
integration will use.

**Paperclip version or commit:** `master` at def4726bea.
**Deployment mode:** any.
**Agent adapter(s) involved:** not adapter-specific (core).

## What Changed

**New table `execution_workspace_restore_points`** (migration `0372`,
schema `packages/db/src/schema/execution_workspace_restore_points.ts`):
- **Columns:** storage key, sha256, size, generation, source issue,
`status`, closing time, review/deletion time (`delete_after`), a
`review_generation` counter and last error.
- **Statuses:** `pending`, `active`, `expiring`, `superseded`,
`delete_requested`, `deleted`.
- **No foreign keys:** a row outlives the deletion of its workspace,
issue or company long enough to remove the stored object.
- **Indexes:**
  - the storage key is unique;
  - live generations are unique per workspace;
- an index on status and deadline serves the worker, and a partial index
on active rows' source issue serves the trigger.

**Issue tree trigger** (same migration):
`paperclip_restore_point_tree_changed` runs after an issue's `status` or
`parent_id` changes, and after an issue is deleted. It walks up from the
issue and from its old and new parents (at most 64 levels). For every
active restore point whose source issue is in that set, it makes the row
due now (keeping an earlier due time) and increments
`review_generation`. This happens in the same transaction, so a child's
deletion or move still reaches its former parent's tree.

**New `server/src/services/git-restore-point-ledger.ts`:**
- **`gitRestorePointRetentionService(db, { storage, retentionDays,
beforeExpire })`.** Each `tick` handles only due work, in batches of 50:
- **Review:** an active restore point is checked when its `delete_after`
time arrives. The due check compares against the later of the worker's
clock and the database clock, so a row the trigger just marked is never
skipped. At review:
- **Tree still open:** clear `delete_after`. The trigger marks it due
again on the next relevant issue change, so open trees cause no
recurring checks.
- **Tree `done` or `cancelled`:** expire
`PAPERCLIP_GIT_RESTORE_POINT_RETENTION_DAYS` (default `90`, minimum `1`)
after the tree's latest close. Expiry moves the row to `expiring`, then
runs an optional `beforeExpire` hook, which the capture integration will
use to mark the workspace no longer restorable. Only then does it
request deletion. An `expiring` row's object is never deleted: after a
restart, or if the hook fails, the hook is retried (after 15 minutes
when it failed).
    - **Source issue gone:** delete.
- **Concurrency:** a review result is written only if
`review_generation` is unchanged since the worker read the row. If the
trigger fired in between (for example, the tree closed or reopened), the
write is skipped and the row stays due for the next run. Expiry and
orphan deletion claim the row under the same check before the expiry
hook runs.
- **Abandoned captures:** a `pending` row older than one hour is
deleted.
- **Deletion:** `superseded` and `delete_requested` rows have their
object removed. They become `deleted` only once `headObject` reports the
object gone. Failures, including a local unlink error the provider
swallows, are recorded in `last_error` and retried after 15 minutes.
- **`nextDeadlineAt`** returns the earliest real deadline, or `null`
when nothing is due.
- **`requestRestorePointDeletion(tx, { sourceIssueId } | { companyId
})`** registers a delivery-work notification and marks the matching
`pending`/`active` rows `delete_requested` in the caller's transaction.

**Deletion hooks:**
- `issueService.remove` and `deleteCompany` call
`requestRestorePointDeletion` in their transactions, so a deleted
issue's or company's objects are removed promptly.
- Child issues cannot be deleted with their parent (the parent link
restricts deletion), so each issue removal goes through this hook.

**Scheduling** follows `rules/sleep-aware-background-work.mdc`:
- `registerGitRestorePointWork` registers a `git-restore-points` queue
with the shared delivery-work coordinator.
- The worker keeps no in-memory inputs. Due work is recorded in the
table, by the trigger or by deletion requests. Each run handles it and
schedules only `nextDeadlineAt`, so a failed run loses nothing and is
retried by the coordinator.
- It runs at startup, at real deadlines, and when terminal-workspace
notifications settle after issue changes, but only while restore points
exist. With nothing due, it schedules nothing. A full batch continues
immediately. There is no timer.

**Config and docs:**
- `server/src/config.ts` adds
`PAPERCLIP_GIT_RESTORE_POINT_RETENTION_DAYS`.
- `doc/recoverable-disk-cleanup.md` gains a "Git workspace restore
points" section; `doc/DEVELOPING.md` documents the variable.
- `drizzle-kit generate` pruned `meta/0367_snapshot.json`, following the
repository's snapshot-retention convention.
- No code comments were added.

## Verification

- `pnpm exec vitest run src/__tests__/git-restore-point-ledger.test.ts`
(in `server/`): 26 passed against embedded Postgres, stable across
repeated runs. The cases:
- **Review outcomes:** an open tree clears its deadline and schedules
nothing; a closed tree is scheduled 90 days after its close; a
long-closed tree expires and is deleted; a missing source issue is
deleted.
- **Expiry durability:** a failing hook keeps the row `expiring` with
its object until a retry succeeds; an `expiring` row left by a restart
resumes the hook before deletion; an `expiring` row's object is never
deleted while its hook fails.
- **Trigger:** the last open child closing, a child being deleted, a
child moving to another parent, and the source issue's own status change
all mark the restore point due. Unrelated issues, inactive rows and
already-due rows are untouched. End to end, an open tree's restore point
waits without a deadline, then is scheduled 90 days after its tree
closes.
- **Races:** closing the tree, or reopening it, between the worker's
read and write leaves the row due, and the next run resolves it
correctly. A reopened tree's restore point is not deleted.
- **No rescanning:** 60 retained restore points with future review times
are not evaluated. A backlog of 60 due ones is processed as 50, then 10,
then nothing.
- **Captures and deletion:** abandoned pending rows are deleted while
recent ones are kept; superseded rows are deleted; an object still
present after deletion is not marked deleted; a failed deletion is
retried only after its 15-minute deadline.
- **Deadlines:** the next deadline is correct for each obligation, and
`null` when idle.
- **Deletion hooks:** `issueService.remove` marks only that issue's
restore points, and `deleteCompany` marks all of the company's; the
worker then deletes their objects.
- **Worker:** schedules nothing when idle, and only the next deadline
otherwise; wakes on issue changes only while restore points exist.
- **Mutation checks:** each of these fails at least one case:
  - removing the issue hook or the company hook;
  - making the trigger ignore former parents;
- removing the worker's generation check or the trigger's generation
bump;
- letting storage cleanup delete `expiring` rows, or not resuming the
hook;
  - ignoring open trees;
  - removing the deletion verification;
  - evaluating rows that are not due;
  - dropping the retry deadline.
- `pnpm exec vitest run company-deletion issue-delete issues-service
delivery-work issue-tree` (in `server/`): 190 passed.
- **CI note:** one earlier `Typecheck + Release Registry` run was
cancelled when the GitHub runner received a shutdown signal; it is not a
code failure.
- `pnpm exec vitest run` (in `packages/db`): 211 passed. One run had a
single failure in `issue-comment-derived-attribution-migration.test.ts`,
an older migration's test unrelated to this change. It passed alone and
on the full re-run.
- `pnpm exec vitest run
src/__tests__/git-restore-point-ledger-config.test.ts`: 5 passed.
- `pnpm run check:migrations` (in `packages/db`): numbering and safety
checks pass.
- `pnpm run typecheck` (in `server/`): pass.

## Risks

- **No behavior change today:** nothing creates restore points yet. The
worker runs once at startup, finds no rows and schedules nothing. The
capture integration follows in a separate PR.
- **Trigger cost:** every issue status change, parent change or deletion
now runs a short ancestor walk, plus an `UPDATE` on the new table that
matches only active restore points through a partial index. With no
restore points, the `UPDATE` matches nothing.
- **Missed wakes:** if a wake is ever missed, for example because the
process exits after commit, the due row stays in the table. The worker's
startup run or next deadline processes it.
- **Deletion paths:** issue and company deletion each gain one `UPDATE`
on the new table inside their existing transactions.
- **Migration:** `0372_mature_the_initiative.sql` creates one empty
table, its indexes, and the trigger function with two triggers on
`issues`; nothing is rewritten.
- **Rollback:** revert. The table can stay, since nothing else reads it.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file edits, tests, read-only measurements on live
instances).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-22 15:52:04 -07:00
TonioandClaude Fable 5.1 341b622604 feat(ui): Artifacts V1 behind an experimental setting — Artifacts page, panel tab and task thread (#17228)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents produce artifacts: documents, files, pictures, video and
links. The app shows them on the Artifacts page, in the right panel's
Artifacts tab and in the task thread
> - The artifact directions lab (#17213) settled a presentation for
those surfaces, MVP V1, drawn from the PCLP-Core mocks. It is a
reference, not product code
> - Shipping it means product components on product data, token-only,
behind a switch, one surface at a time, so the current presentation
stays the default until the new one has been used
> - This pull request adds the switch (Settings > Experimental >
Artifacts V1), the `--artifact-*` tokens, the V1 primitives as product
components, and the V1 presentation on all three surfaces behind the
switch: the Artifacts page, the panel's Artifacts tab and the task
thread
> - The benefit is the redesign in the product, reversible by one
setting, with every existing behaviour of those surfaces kept

## Linked Issues or Issue Description

No public issue exists. This section follows the feature request
template; the plan is `doc/plans/2026-10-22-artifacts-mvp-v1.md`.

**Subsystem affected**

The web UI (`ui/`): the Artifacts page, the right panel's Artifacts tab
and the task thread. The experimental settings contract in
`packages/shared` and `server` gains one boolean.

**Problem or motivation**

The three artifact surfaces grew separately and their artifact
presentation is dated and inconsistent. The lab produced a measured
direction; nothing in the product uses it yet.

**Proposed solution**

An experimental setting, `enableArtifactsV1`, default off, with the same
plumbing as the other experimental settings. Product primitives under
`ui/src/components/artifacts/v1` (kind helpers and adapters, the sheet
and print artwork, the stack, the page tiles with the Paper Mono pill,
the row the panel and thread will share), driven by `--artifact-*`
tokens in `index.css` with light and dark values. With the setting on:
the Artifacts page shows one row of views (Tasks, All files, Images,
Videos, Documents, Text, Files) and the V1 tiles, a task with several
artifacts as a stack whose newest artifact is the first page, everything
else as single tiles; the panel's Artifacts tab is one flat list of V1
rows, newest first, with the expandable rows (Markdown review surface,
document annotations, deep links) keeping their behaviour behind the row
as their face; the thread shows a message's files as rows under its
footer and registered deliverables and documents as one row each. The
hover movement respects reduced motion.

**Alternatives considered**

Replacing the current cards outright: no fallback and three surfaces to
land at once. Mounting the lab's components: they are typed on lab
fixtures and keep lab-local tokens, which the token gates reject. The
primitives were ported instead and the lab stays a reference.

**Roadmap alignment**

Supports the "Artifacts & Work Products" theme in `ROADMAP.md`.
Presentation only; no data model change.

**Additional context**

No new API fields: the tiles show title, identifier, count and "Updated
…"; rows take sizes from the work-product and attachment payloads; a
picture's aspect ratio comes from the loaded image, 4:3 until it loads.
The plan records what each surface keeps and what it drops (run sections
in the panel; the "Images · N / Files · N" labels and the four-up grid
in the thread).

## What Changed

- Add the `enableArtifactsV1` experimental setting: shared type,
validator and feature catalog, server defaults and managed keys, the
`useArtifactsV1Enabled` hook, and the Settings > Experimental toggle (in
its alphabetical place).
- Add the `--artifact-*` tokens and the Artifacts V1 component rules to
`ui/src/index.css`: light values on `:root`, dark overrides, and the
hover movement under `prefers-reduced-motion: no-preference`.
- Add `ui/src/components/artifacts/v1/`: `kinds.ts` (kind treatment,
labels, glyphs, the server's media-kind rule, adapters from
`CompanyArtifact`), `geometry.ts` (the measured constants), `when.ts`,
`ArtifactSheet`, `ArtifactPrint`, `ArtifactStack`, `ArtifactFront`,
`ArtifactTile` (stack and single tiles, caption, grid), `ArtifactRow`
(the row and its inline group), `ArtifactsV1Filters`, with the mock's
SVG assets.
- Wire the Artifacts page: with the setting on, the toolbar's filter row
becomes the V1 views and the grid renders V1 tiles; the toolbar takes an
optional `filters` row for this. The old cards and kind filters stay the
default.
- Wire the panel's Artifacts tab (`IssuePropertiesArtifactsTab`): with
the setting on, one flat list of V1 rows, newest first; `DocumentRow`
and `MarkdownWorkProductRow` take a `face` render prop so the V1 row
fronts their unchanged expansion, review and deep-link behaviour;
`PanelFileRow` opens attachments and attachment-backed work products the
way the shipped card does (gallery, text tab, new tab) with open and
download beside the name; other work products are a link row with their
status.
- Wire the task thread: `TaskChatBubble` hangs a message's files under
its footer as V1 rows inline (pictures and video to the gallery, text to
a text tab, files to a new tab) in place of the media grid and file
chips; `TaskChatProtocolCard`'s resource rows render deliverables,
documents and attachments as one V1 row each, opening where the card
did.
- `ArtifactRow` keeps trailing controls beside its own control (no
nested interactive elements) and can be a link, an in-app anchor, a
new-tab anchor or a button; adapters read work products, documents,
attachments and message references.
- Add a catalogued Storybook story, `Components/Artifact V1 primitives`,
under `documents-artifacts`, on fixtures.
- Add tests: the primitives (kinds, dates, print fitting, tiles, rows),
the hook, the Artifacts page, the panel's Artifacts tab, the task-chat
bubble and the protocol card, each with the setting on and off; the
settings page and server tests list the new key.
- Add `doc/plans/2026-10-22-artifacts-mvp-v1.md`.

## Verification

- Run `cd ui && pnpm typecheck`: the Storybook catalog accepts the new
story, and `tsc -b` is clean apart from the five pre-existing
`PluginOrganizationSwitcher` errors.
- Run `pnpm check:token-gates` and `node scripts/check-maps.mjs`: clean.
- Run `cd ui && npx vitest run src/components/artifacts
src/components/issue-properties
src/components/task-chat/TaskChatBubble.test.tsx
src/components/task-chat/TaskChatBubble.artifactsV1.test.tsx
src/components/task-chat/TaskChatProtocolCard.test.tsx
src/components/task-chat/TaskChatProtocolCard.artifactsV1.test.tsx
src/pages/Artifacts.test.tsx src/hooks/useArtifactsV1Enabled.test.ts
src/pages/InstanceExperimentalSettings.test.tsx`: the suites pass (151
tests in the last run).
- Run `cd packages/shared && npx vitest run src/feature-catalog.test.ts`
and `cd server && npx vitest run
src/__tests__/instance-settings-service.test.ts`: pass.
- Manual: open Settings > Experimental, turn on Artifacts V1. On the
Artifacts page the view row reads Tasks, All files, Images, Videos,
Documents, Text, Files; tasks with several artifacts show as stacks,
hovering fans the pages and lifts the artwork (not under reduced
motion); Images shows single tiles with bordered prints. Open a task's
Artifacts tab: one list of rows, newest first; a Markdown work product
still expands into its review surface, a document still opens in its
tab. In the thread, a message's files sit under its footer as rows. Turn
the setting off: everything returns to the current presentation.
- Storybook: `Tasks and conversations / Documents, attachments, and work
products / Components / Artifact V1 primitives`.
- Not run: the full `pnpm test:run` and `pnpm build` (CI runs them), and
a browser pass against a dev instance with real artifacts on the three
surfaces; that pass is the remaining step before this leaves draft.

## Risks

- Off by default; with the setting off, the only changes on a product
path are an optional `filters` prop on the toolbar, an optional `face`
prop on two panel rows, a read of the setting in the bubble and the
protocol card, and the new token declarations.
- With the setting on, the Artifacts page's view tabs set two URL
parameters (`groupBy`, `kind`) at once; the group-by menu and search are
unchanged.
- A picture with unknown dimensions renders at 4:3 until it loads, then
the print refits once; the frame's height is fixed so nothing below it
moves.
- About thirty new `--artifact-*` tokens, namespaced and documented in
place.

## Model Used

- Claude Fable 5.1 by Anthropic, model ID `claude-fable-5-1`, used
through Claude Code in the desktop app. Extended reasoning was on.
Capabilities used: tool use (shell, file edits, a headless Playwright
browser for the lab's measurements, the Figma MCP server for the mocks
during the lab work) and image input. The context window is the Claude
Code session's; its exact size is not surfaced to the session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above (#17213 is the lab this builds on)
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (the suites listed above;
CI runs the full set)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes (the
plan document; no command or product behaviour changes outside the
setting)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-22 15:14:58 -07:00
Devin FoleyandPaperclip 30f402f5fe feat: add a library to capture and restore git workspace restore points (#17238)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Each issue's agent works in a git worktree that Paperclip keeps on
the instance's disk
> - Git cache expiration removes a closed issue's worktree only when the
remote can reproduce it exactly: every commit pushed, nothing
uncommitted
> - Most closed worktrees fail that test, so they are kept forever and
disks fill up
> - A restore point that saves only the part the remote cannot reproduce
would let those worktrees be removed without losing work
> - This pull request adds the library to capture, verify, encrypt and
restore such restore points, with no callers yet
> - The benefit is a small, reviewable foundation for removing unpushed
and uncommitted worktrees safely in follow-up changes

## Linked Issues or Issue Description

Bug report described in-PR (bug report template).

**What happened?**
Closed issues' worktrees accumulate without bound. With
`PAPERCLIP_GIT_CACHE_EXPIRATION_ENABLED=true` and dry run off, one
instance's last 111 expiration checks expired nothing. The skip reasons
were:

| Skip reason | Checks |
|---|---|
| `dirty_or_untracked_files` | 39 |
| `unpushed_managed_checkout` | 32 |
| `live_or_unresolved_consumer` | 21 |
| `unique_ignored_files` | 9 |
| commit not on the remote | 10 |

Without expiration, the terminal workspace reaper only archives merged,
clean worktrees. One local instance archived 37 workspaces across about
6,200 sweeps.

**Expected behavior**
A closed worktree can be removed once anything the remote cannot
reproduce is saved, and restored exactly if the work resumes.

**Steps to reproduce**
1. Close an issue whose worktree has an unpushed commit or an untracked
file.
2. Enable git cache expiration with dry run off.
3. The worktree is never removed; expiration reports
`dirty_or_untracked_files` or `upstream_commit_unverified`.

**Paperclip version or commit:** `master` at def4726bea.
**Deployment mode:** any; observed on a server instance and a local
instance.
**Agent adapter(s) involved:** not adapter-specific (core bug).

## What Changed

New `server/src/services/git-restore-point.ts`, a library that nothing
calls yet:

- **`captureRestorePoint`**
- Snapshots the index and the working tree as deterministic commits in a
temporary object store. The store borrows the repository's objects
through alternates and writes none of its own into the repository.
- Bundles those snapshot commits, unpushed commits, and commits
reachable only from the reflog or `ORIG_HEAD`, against the remote
commits as prerequisites.
- Returns `null` when the remote can already reproduce everything.
Refuses to capture when more than 50 history-only commits exist, rather
than dropping any.
- **Leaves out:** ignored files (such as `node_modules`), and untracked
`.env` and `.env.*` files. Provisioning recreates environment files with
current secrets. Edits to tracked environment files, such as
`.env.example`, are captured.
- **Skips** nested untracked repositories, paths with a filter
attribute, gitlinks and changes over size limits rather than capturing
them partially. Special files are ignored, because git never tracks
them.
- **`currentRestorePointSnapshot`** recomputes the snapshot ids, so a
caller can prove nothing changed between capture and deletion.
- **`importRestorePoint`** verifies the bundle's prerequisites and refs
against the manifest, then imports the objects without leaving refs
behind.
- **`applyRestorePoint`**
  - Requires `HEAD` to match.
- Restores the working tree (`git apply`), the index (`read-tree`) and
non-default file modes.
- Then recomputes the snapshot and requires it to equal the captured
one.
- **`sealRestorePoint` / `openRestorePoint`**
- Encrypt with a fresh AES-256-GCM key per restore point, with the
restore point's identity as additional authenticated data.
- The data key is wrapped by the instance secrets provider
(`localEncryptedProvider`), the same pattern already used for saved
Claude settings.
- Opening checks the ciphertext hash, the key's bound identity, the GCM
tag, and the plaintext hash and size.
- **Isolation:** every git command runs with system and global config
disabled, hooks disabled, and line-ending conversion off.
- New `server/src/__tests__/git-restore-point.test.ts`.
- No code comments were added.

## Verification

- `pnpm exec vitest run src/__tests__/git-restore-point.test.ts` (in
`server/`): 18 passed, using real repositories and a bare remote. The
cases:
- **Full round trip** into a fresh clone that lacks the unpushed
commits, covering:
    - two unpushed commits;
    - staged and unstaged edits of the same file;
    - a deletion, an untracked file, a symlink, `0755` and `0600` modes;
    - `node_modules` and `.env` files left out.
- **No-op:** capture returns `null` for a reproducible workspace, and
creates no objects in the repository (`fsck --unreachable` stays empty).
  - **Snapshot ids** are deterministic and change when contents change.
- **History-only commits** are preserved and leave no refs after import.
- **Skips:** nested repositories and size limits. Special files are
ignored.
  - **Rejections:** missing prerequisites and `HEAD` mismatches.
- **Encryption:** tampering, truncation, a wrong identity and a
different master key are all rejected.
- **Review regressions:** edits to a tracked `.env.example` are
captured; staged files whose contents match the index get limits and
modes; permission-only changes change the snapshot; a configured `diff`
textconv does not alter restored bytes; more than 50 history-only
commits refuses capture.
- **Mutation checks:** removing the `.env` exclusion or the index
restore fails the round trip, and removing each review fix fails its
regression test.
- **Stability:** 12 consecutive runs passed. An earlier version copied
the index for the working-tree snapshot and was racy when a file changed
within the same second. It now starts from `HEAD`'s tree instead.
- `pnpm run typecheck` (in `server/`): pass.

## Risks

- **No behavior change:** nothing calls the library yet.
- **Follow-ups:** wiring it into expiration (storage, metadata
versioning, restore on reopen and on the next run, and retention) comes
in separate PRs behind configuration that defaults to off.
- **Key dependence:** a restore point is only as recoverable as the
instance's secrets master key. The follow-up wiring will refuse to
create restore points when that key is not durable.
- **Rollback:** revert.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file edits, tests, read-only measurements on live
instances).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-22 15:14:51 -07:00
Devin Foley 059ae49cab feat(connections): add Unikraft Cloud connection (#17139)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Apps connections give agents governed access to external services
through vault credentials, grants, action policies, the tool gateway,
and audit.
> - Unikraft Cloud runs unikernel micro-VMs, services, and volumes
across regional metros, and it offers a sandbox plugin for commands and
files.
> - Unikraft is not in the catalog. It has no hosted MCP server, so the
generic "Connect your own MCP server" flow cannot reach it.
> - Daytona (#16483) added a native REST pattern for this case: a
catalog generated from the provider's published OpenAPI specifications
and executed through the gateway.
> - This pull request adds Unikraft on that pattern and moves the shared
native-provider code into one registry, so the shared connection code
does not name each provider.
> - The benefit is governed agent access to all published Unikraft
actions, with the token kept in the vault and sent only to published
Unikraft hosts.

## Linked Issues or Issue Description

Refs #16483

**Problem or motivation**

Teams that run workloads on Unikraft Cloud want their agents to list
instances, read logs, deploy and scale services, and run commands in
sandboxes. Unikraft is not in the Apps catalog. Unikraft has no hosted
MCP server, so operators cannot connect it through the remote MCP flow.

**Proposed solution**

Add a native Unikraft connection. Generate its actions from Unikraft's
official OpenAPI specifications for the platform API and the sandbox
plugin. Pin the specifications to a reviewed commit. The operator
connects an API token. The token stays in the vault and goes only to the
five published metro hosts. Every action takes an optional metro.
Actions without a metro use the connection's default metro, which the
operator can change under Advanced.

**Alternatives considered**

A catalog-only AppDefinition was not possible because no hosted MCP
endpoint exists. Wrapping the `unikraft` CLI was rejected because it
would expose host filesystem behavior and CLI login state. A second copy
of the Daytona bridge was rejected; the shared transport and catalog
contract are now one module with a provider registry.

**Roadmap alignment**

This extends the existing Apps and Connections provider coverage. It
uses the existing credential, grant, policy, gateway, and audit model.
It does not overlap planned core work.

## What Changed

- Added `scripts/generate-unikraft-tools.mjs` and its tests. The
generator reads the platform and sandbox plugin specifications at commit
`5dd107ea1cd8716b83cb86a83ed2a6ee46f99b5b`, resolves shared parameters,
records source SHA-256 hashes, and writes
`server/src/services/unikraft/operations.generated.json`.
- The catalog has 105 actions: 89 platform operations and 16 sandbox
plugin operations. The server-sent audit event stream is recorded in the
inventory and is not advertised as an action.
- Classification: GET operations and sandbox reads and waits are reads.
Delete, stop, suspend, detach, unpin, and sandbox command execution,
signals, and stdin are destructive. All other operations are writes.
- Added `server/src/services/connectors/unikraft/client.ts` and its
generated catalog. It validates arguments against the generated schemas,
accepts only published metros, rejects path traversal, sends JSON bodies
on GET and DELETE as the specification requires, and verifies the token
with `GET /v1/users/quotas`. A sandbox wait that times out (documented
HTTP 408) returns a still-running result instead of a failure.
- Moved the bounded, credential-redacting HTTP sender and the catalog
contract helpers from `daytona.ts` to `native-rest-client.ts`. Added
`native-provider-catalogs.ts`. `tool-access.ts`, the
`tool-access/catalog` and `token-broker` modules,
`connectors/tool-policy.ts`, and `tool-gateway.ts` now use this registry
instead of Daytona-specific checks. Daytona behavior does not change.
- Added the Unikraft definition source, the generated definition, the
brand mark and manifest entry, and the permission review (`liveProof:
not-run`).
- Added `doc/connections/UNIKRAFT.md` with a sequence diagram,
endpoints, setup, actions, and limits. Updated the connections README,
the permission audit, and both capability maps.
- Tests: definition shape, database-backed gateway reads, writes,
deletes, and sandbox commands for personal and shared grants, default
and per-call metros, rejection of unpublished metros and traversal, Off
policies, revocation, stale-contract blocking, credential export denial,
schema validation for all actions, pinned-transport tests for GET and
DELETE bodies, and a browser setup check.

## Verification

- `pnpm exec vitest run packages/shared/src
server/src/__tests__/unikraft-connection.test.ts
server/src/__tests__/unikraft-http-boundary.test.ts
server/src/__tests__/daytona-connection.test.ts
server/src/__tests__/daytona-http-boundary.test.ts
server/src/__tests__/tool-access-service.test.ts
server/src/__tests__/tool-gateway-service.test.ts
server/src/__tests__/tool-gateway.test.ts
ui/src/pages/apps/AppsConnect.test.tsx ui/src/pages/apps/Browse.test.tsx
ui/src/lib/app-brand-assets.test.ts` plus `connector-tool-policy` and
`tool-catalog-publication`: all passed after the rebase onto the split
tool-access modules.
- `node --test scripts/generate-unikraft-tools.test.mjs
scripts/generate-daytona-tools.test.mjs
scripts/app-brand-validation.test.mjs`: 20 passed.
- `node scripts/check-app-brand-assets.mjs`, `node
scripts/check-maps.mjs`, and `git diff --check`: passed.
- `pnpm --filter @paperclipai/shared typecheck`, `pnpm --filter
@paperclipai/server typecheck`, and `pnpm --filter @paperclipai/ui
typecheck`: clean.
- `npx playwright test --config tests/e2e/playwright.config.ts
tests/e2e/unikraft-connector.spec.ts
tests/e2e/daytona-connector.spec.ts`: 2 passed.
- `node scripts/generate-unikraft-tools.mjs` regenerates the committed
catalog with the recorded hashes.
- Manual: on a local instance, open Connectors, search for Unikraft, and
select **Add key**. Confirm the icon, the password field, the key
guidance, and the default metro select under **Change**.
- Not verified: a real Unikraft account. Live provider read and write
proof is pending.

## Risks

- A Unikraft token acts for the whole organization in every metro.
Unikraft offers no narrower token permissions. Assign only trusted
agents and review action permissions.
- Sandbox actions run arbitrary commands inside the selected instance.
They are classified as destructive.
- A timed-out write can still complete at the provider. The bridge does
not retry writes automatically.
- New metros fail closed until the specification is regenerated and
reviewed. Legacy `*.kraft.cloud` hosts are not reachable.
- The Daytona refactor moves code without behavior changes. All Daytona
tests pass without edits.
- The generated JSON snapshot is large because it holds the provider
input contracts. No migration or lockfile change is required.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, 1M context
window, used through Claude Code with tool use (shell, file editing, web
research, browser tests).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-10-22 15:14:26 -07:00
DottaandPaperclip 4dfcb4c9b8 refactor(tools): isolate connection access and removal (#17167)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Connection grants and installations control which identities and
agents can reach tools.
> - Removal must close those access paths and stop runtimes before
cleaning up owned secrets.
> - These responsibilities still shared the setup coordinator after
earlier extractions.
> - This pull request gives grants, installations, credential validation
and removal explicit owners.
> - The benefit is a smaller coordinator and focused access-lifecycle
modules with existing security ordering intact.

## Linked Issues or Issue Description

Depends on #17165. Earlier stack: #17144 → #17146 → #17147 → #17155 →
#17157 → #17159 → #17161 → #17162 → #17165 → this PR.

**What existing behavior does this improve?**

Maintenance of connection audiences, delegation, installation and
removal.

**Current behavior**

The 6,419-line service still owns grant mutation, installation reach and
resumable credential cleanup alongside app setup.

**Proposed behavior**

A grant owner manages audiences and delegation. An installation owner
manages reach and its profile bindings. A removal owner keeps access
revocation, runtime shutdown, secret classification and cleanup
together. The coordinator delegates through typed APIs.

**Reason and benefit**

The entry point falls to 4,821 lines / 198,065 bytes, meeting the
requested under-5,000-line target. Across the full stack it falls from
21,525 lines / 829,519 bytes. App setup remains together in the
coordinator.

## What Changed

- Extract credential validation (75 lines), grants (603), installations
(350) and removal (658).
- Preserve company boundaries, audience advisory locks, membership
locks, assignment locks and audit order.
- Keep the existing runtime supervisor and secret client. Retain live
HTTP options for provider shutdown.
- Keep Slack credential-mutation leasing and resumable owned-secret
cleanup with removal.

## Verification

- Removal, grant-resolution and the stronger real HTTP OAuth fixture
pass before extraction (13 cases); the existing service and publication
baselines also pass.
- The original combined targeted run passed 525 cases across seven
suites. After replaying onto master, 601 cases pass across nine suites,
including Braintrust lifecycle and connector-policy coverage, with one
database worker.
- Full repository typecheck and build passed before merge integration.
After integration, server typecheck, module-boundary and whitespace
gates pass; fresh CI covers the full repository gates.
- All 94 pre-existing function/method bodies match by normalized syntax
comparison. A full-stack TypeScript symbol audit preserves all 18
exports and 89 service methods.
- The earlier local full-suite run was interrupted after 79 minutes and
remains unverified. All current-head CI gates pass on
`d5307fcea2365e51a75d49db82deeed236d19951`, including the full sharded
test matrix, browser suites, typecheck, build and canary dry run.
Greptile is 5/5 with no open review threads after merge integration.

- GitHub canceled duplicate CI run
[39716841632](https://github.com/paperclipai/paperclip/actions/runs/39716841632)
during the atomic stack update. Replacement run
[39716843243](https://github.com/paperclipai/paperclip/actions/runs/39716843243)
passed on the same head; no failing test was retried.

## Risks

- Audience replacement must not erase an inactive member marker and
widen authority. Its advisory and membership locks remain in the grant
owner.
- Uninstall must remove only its own profile bindings. Existing
operator-authored bindings keep their current behavior.
- Removal must remain fail-closed and retryable after provider failures.
Its order, shared-secret retention, supervisor and cleanup references
are unchanged and covered by removal tests.
- No schema, public API, policy or scheduling change is intended. The
first eight PRs have landed; this PR follows the updated catalog
extraction.

## Model Used

OpenAI Codex, GPT-6 family, with repository tools and code execution.
The exact serving model ID and context-window size are not exposed in
this session.

## Checklist


- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-21 22:39:00 -05:00
DottaandPaperclip f7210cd31f refactor(tools): isolate catalog ownership (#17165)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents receive reviewed actions discovered from their connections.
> - Transport, discovery, risk classification, publication and account
inventory lived in the same service.
> - Their ownership differs: transport guards requests, discovery reads
providers, and publication changes the catalog under locks.
> - This pull request gives those phases separate modules with explicit
capabilities.
> - The benefit is a visible discovery-to-publication path that
preserves permission and quarantine behavior.

## Linked Issues or Issue Description

Builds on merged #17162 and preserves the connector policy from #17141.
Earlier stack: #17144 → #17146 → #17147 → #17155 → #17157 → #17159 →
#17161 → #17162 → this PR.

**What existing behavior does this improve?**

Maintenance of tool discovery, catalog publication, health and upstream
account inventory.

**Current behavior**

The 8,988-line service handles HTTP guards, MCP sessions, provider
probes, risk classification, quarantine, cached reads and account sync.

**Proposed behavior**

Discovery returns descriptors through the existing credential and
transport owners. Publication owns catalog writes and default access. A
reader owns refresh deduplication. Health and aggregator inventory have
separate owners.

**Reason and benefit**

The entry point falls to 6,419 lines, a reduction of 2,569. Provider
discovery no longer shares a closure with connection setup and teardown.

## What Changed

- Extract guarded remote HTTP, descriptor policy, discovery,
publication, default access, cached reads, health, attention and
aggregator account sync.
- Keep the public descriptor/risk exports and existing service methods.
- Preserve database locks, transaction-scoped default-access writes,
quarantine decisions, current grant checks and failure classification.
- Preserve per-service catalog flights and aggregator jobs, lease
ownership and live remote-request options.
- New modules range from 39 to 492 lines.

## Verification

- Before and after the move, 470 public service/HTTP integration cases
and 44 publication, HTTP rebinding and aggregator sync cases pass.
- Server typecheck, module-boundary gate and whitespace checks pass.
- All 157 pre-existing function/method bodies match by normalized syntax
comparison.
- Before merge integration, the completed stack passed 525 targeted
cases, repository-wide typecheck and build. Its stronger rotating-token
fixture is included.
- After replaying onto master, the integrated stack passes 601 cases
across nine suites, including Braintrust lifecycle and connector-policy
coverage. Server typecheck and module-boundary checks pass. All 157
moved bodies still match the current master versions.
- The earlier local full-suite run was interrupted after 79 minutes and
remains unverified. All current-head CI gates pass on
`a21595f20809a514b7619d8a8748c56f8e546e2f`, including the full sharded
test matrix, browser suites, typecheck, build and canary dry run.
Greptile is 5/5 with no open review threads after merge integration.

## Risks

- Discovery must keep its current grant and session behavior;
publication must retain its locks and quarantine rules. Existing
authority, schema-drift and publication-race tests exercise these paths.
- Aggregator account observations must not become authorization. Lease,
freshness, failure and credential-fingerprint behavior stay unchanged.
- Live transport options must not be snapshotted. The HTTP owner retains
the same options object.
- No schema, public API, access-policy or scheduling change is intended.

## Model Used

OpenAI Codex, GPT-6 family, with repository tools and code execution.
The exact serving model ID and context-window size are not exposed in
this session.

## Checklist


- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-21 22:38:59 -05:00
DottaandPaperclip ba9e6e4bc5 refactor(tools): isolate authorization lifecycle (#17162)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Browser authorization binds a consenting identity to a company
connection.
> - Direct OAuth, managed-cloud claims and Vercel authorization have
different state and persistence rules.
> - Keeping all of them in the general tool service obscures those rules
and their cleanup owners.
> - This pull request separates state, initiation, provider completion
and explicit identity promotion.
> - The benefit is reviewable authorization flows with their existing
authority checks and ordering preserved.

## Linked Issues or Issue Description

Depends on #17161. Earlier stack: #17144 → #17146 → #17147 → #17155 →
#17157 → #17159 → #17161 → this PR.

**What existing behavior does this improve?**

Maintenance of browser authorization and credential-sharing lifecycle.

**Current behavior**

The 12,019-line service combines provider callbacks, actor-bound states,
authorization URLs and personal-to-company credential promotion.

**Proposed behavior**

Separate owners handle authorization state, start, direct/cloud/Vercel
callbacks, catalog completion and explicit access finalization. Shared
setup policy and identity checks have typed APIs used by the coordinator
and authorization owners.

**Reason and benefit**

The entry point falls to 8,997 lines, a reduction of 3,022.
Provider-specific state consumption and durable-write rules are visible
in their own modules.

## What Changed

- Extract authorization flows and their constraints; the largest new
module is 700 lines.
- Preserve direct/Vercel terminal state consumption and managed-cloud
retryability until claim persistence commits.
- Keep membership and grant locks, secret transaction clients,
interaction resolution and catalog ordering unchanged.
- Keep partial secret acquisition and cleanup together in access
finalization.
- Give shared organization-grant persistence, personal identity checks,
setup policy and response projections named owners.

## Verification

- All 470 public service and real local HTTP/PostgreSQL OAuth cases pass
before and after extraction.
- Coverage includes same-actor state binding, replay/denial, scope
selection, membership revocation, provider failures, refresh concurrency
and persisted usable credentials.
- Server typecheck, module-boundary gate and whitespace checks pass.
- All 182 pre-existing function/method bodies match by normalized syntax
comparison.
- The completed stack passes 525 targeted cases, full repository
typecheck and build. The earlier interrupted local full suite remains
unverified; all current-head CI gates pass; Greptile is 5/5 with no open
review threads.

- Original CI failure retained: [server shard
1/12](https://github.com/paperclipai/paperclip/actions/runs/39710536789/job/119600621627),
`workspace-runtime.test.ts` case “replaces a reused Paperclip dev
runtime whose 2xx health payload is unhealthy”. The fixture failed its
initial health readiness with `ECONNREFUSED` after 97 probes / 10
seconds. The fixture and runtime implementation are unchanged and do not
import this service. The isolated case passes without changing its
timeout or assertions. One retry of only that shard passed at the same
commit. All CI gates are now green; the original failure remains
recorded here.

## Risks

- Provider flows intentionally consume authorization state at different
points. This extraction preserves each flow rather than sharing its
completion algorithm.
- Promotion must remove partially created secrets on failure and retire
source secrets only after persistence. Both phases remain in the same
owner with unchanged ordering.
- Local HTTP fixtures prove assembled application behavior, not live
third-party qualification.
- No schema, public API or authorization policy change is intended.

## Model Used

OpenAI Codex, GPT-6 family, with repository tools and code execution.
The exact serving model ID and context-window size are not exposed in
this session.

## Checklist


- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-21 22:21:43 -05:00
DottaandPaperclip 4fdef9457f refactor(tools): isolate credential ownership (#17161)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Connections need durable credentials that survive authorization and
token rotation.
> - Storage, registration, refresh coordination and credential selection
lived in the general tool service.
> - Each has a different ownership boundary: secrets, process-wide
registration, database leases and per-service refresh work.
> - This pull request gives those responsibilities named modules and
passes their narrow capabilities to consumers.
> - The benefit is visible credential ownership while retaining current
concurrency and failure behavior.

## Linked Issues or Issue Description

Depends on #17159. Earlier stack: #17144 → #17146 → #17147 → #17155 →
#17157 → #17159 → this PR.

**What existing behavior does this improve?**

Maintenance of OAuth credential persistence, refresh and provider
credential access.

**Current behavior**

The 14,877-line service owns registration flights, refresh leases,
secret writes and provider selection alongside app setup.

**Proposed behavior**

Dedicated owners handle credential storage, client registration, refresh
leases, connection/grant refresh, GitHub continuity and Vercel
credential access. Credential selection composes those owners.
Authorization callbacks remain in the coordinator for the next
extraction.

**Reason and benefit**

The entry point falls to 12,019 lines, a reduction of 2,858. Lease
acquisition and cleanup can be reviewed together, independently of
callback completion.

## What Changed

- Extract credential owners and shared single-flight/connection-link
primitives. The largest new module is 650 lines.
- Preserve process-wide client-registration flights, per-service refresh
maps and GitHub sweep state.
- Preserve live cloud-connector lookup, options and environment reads;
use the existing secret client and transaction-scoped clients.
- Extend the real local HTTP/PostgreSQL OAuth fixture with concurrent
refresh across two service instances and authenticated use of rotated
credentials.

## Verification

- The two extended HTTP integration cases pass before extraction for
public and client-secret-basic clients.
- After extraction, all 470 public service and HTTP integration cases
pass, including refresh failure, abandoned leases, secret redaction and
membership revocation cases. The fixture was then strengthened after
review: a second expiry must use the persisted rotated refresh token,
and the provider rejects the old token. Both cases pass before and after
extraction; server typecheck also passes.
- Server typecheck, module-boundary gate and diff whitespace checks
pass.
- All 233 pre-existing function/method bodies match by normalized syntax
comparison.
- Full typecheck/build will run at the stack handoff. The earlier
interrupted local full suite remains unverified; all current-head CI
gates pass, with Greptile 5/5 and no open review findings.

- Prior-head CI failure retained: run 39708550066, general server shard
2/12, `native-session-executor.test.ts` case “preserves a real listening
process and its marker when a health RPC times out” raised
`runner_remote_process_identity_unavailable` /
`process_identity_changed`. Both the fixture and runner implementation
are unchanged. The isolated case passes without changing assertions or
timeouts. The fresh full CI run passes for the test-only review fix; the
prior failure remains recorded here.

## Risks

- Changing state lifetime could permit duplicate refreshes or
registrations. Existing concurrency cases and the real HTTP fixture
exercise these paths.
- A transaction-scoped secret write must stay inside its authority lock.
Function bodies, await ordering and client arguments are preserved.
- The fixture proves assembled local HTTP/database behavior, not live
third-party qualification.
- No schema, public API, policy or scheduling change is intended.

## Model Used

OpenAI Codex, GPT-6 family, with repository tools and code execution.
The exact serving model ID and context-window size are not exposed in
this session.

## Checklist


- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-21 22:21:43 -05:00
DottaandPaperclip ffcd113761 refactor(tools): isolate runtime and activity ownership (#17159)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Operators inspect tool execution, review pending actions and control
local runtimes.
> - These paths shared a large service with token exchange and app
authorization.
> - Runtime state and execution history need explicit owners and stable
public responses.
> - This pull request separates runtime controls, health summaries,
activity reads and the review queue.
> - The benefit is focused modules that preserve the current supervisor
and approval behavior.

## Linked Issues or Issue Description

Depends on #17157. Earlier stack: #17144 → #17146 → #17147 → #17155 →
#17157 → this PR.

**What existing behavior does this improve?**

Maintenance of tool runtime administration and execution activity views.

**Current behavior**

The 16,100-line service contains runtime slot controls, health
calculations, usage history, invocation decisions and action review
queries.

**Proposed behavior**

Runtime modules own slot persistence, operator controls and health
summaries. Activity modules own history, response projections and the
review queue. The public service delegates to these owners.

**Reason and benefit**

The entry point falls to 14,877 lines, a reduction of 1,223 lines. The
coordinator no longer needs execution-history joins or runtime alert
calculations.

## What Changed

- Extract five modules totaling 1,282 lines. The largest is 426 lines.
- Pass the existing supervisor to slot controls. Pass the same live
deployment options, clock and host getter to runtime health.
- Keep activity reads scoped as before. Keep unsigned-request grace and
invalid-signature cancellation in the review-queue owner.
- Preserve public methods, response projections, query order and audit
writes.

## Verification

- The public service and unsigned review-queue tests pass before and
after the move. The combined post-move run passes 469 cases with one
database worker.
- Existing cases exercise actual local stdio stop/restart, route
permissions, stale-slot and audit-failure health, activity attribution,
run decisions and unsigned pending requests.
- Server TypeScript check, module-boundary gate and `git diff --check`
pass.
- Normalized syntax comparison finds all 256 pre-existing
function/method bodies unchanged.
- Full typecheck/build will run at the stack handoff. The earlier
interrupted local full suite remains unverified; all current-head CI
gates pass, with Greptile 5/5 and no open review threads.

## Risks

- Creating another supervisor would split process ownership. The new
module receives the same instance; it does not construct one.
- Copying deployment settings could freeze live values. The health
reader holds the existing options object and getter.
- Review-queue reads can cancel invalid requests. Those writes and the
unsigned-request grace period are preserved and covered by the existing
regression test.
- No schema, public API, authorization policy or background scheduling
change is intended.

## Model Used

OpenAI Codex, GPT-6 family, with repository tools and code execution.
The exact serving model ID and context-window size are not exposed in
this session.

## Checklist


- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-21 22:21:42 -05:00
DottaandPaperclip a42f213244 refactor(tools): isolate connection token broker (#17157)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents can request short-lived tokens for connections that allow
token exchange.
> - Admission, scope checks, exchange protocols and issuance records
lived inside the general tool service.
> - The security flow needs one visible owner while provider payloads
need a separate boundary.
> - This pull request extracts the broker into admission, configuration,
endpoint, exchange and issuance modules.
> - The benefit is a focused issuance flow with its locks, authority
checks and audit order intact.

## Linked Issues or Issue Description

Depends on #17155. Earlier stack: #17144 → #17146 → #17147 → #17155 →
this PR.

**What existing behavior does this improve?**

Maintenance and verification of connection token issuance for agent
runs.

**Current behavior**

A 17,752-line service contains broker configuration, endpoint policy,
credential exchange, admission and issuance.

**Proposed behavior**

A dedicated broker coordinates the same admission and transaction flow.
The exchange owner returns normalized token material. It receives the
existing secret client, including the transaction-scoped client at
issuance time.

**Reason and benefit**

The entry point falls to 16,100 lines, a reduction of 1,652 lines.
Reviewers can see the responsible-user lock and hash-only issuance write
without navigating app setup or OAuth callbacks.

## What Changed

- Extract six broker modules. The largest is 755 lines; together they
contain 1,721 lines.
- Give tool-access audit persistence a named owner shared by the broker
and coordinator.
- Keep current-state checks, policy decisions, rate limits, secret
resolution and transaction order unchanged. Private-host configuration
is still read live.
- Add two public-route integration cases with a real local HTTP server
and disposable PostgreSQL. Cover Pages and RFC 8693 success and upstream
failure.

## Verification

- Baseline: all 466 existing public service tests passed before this
move. The two new real HTTP cases passed on the original implementation
before extraction.
- After extraction: 495 tests pass across the public service and HTTP
rebinding suites, with one database worker. This includes the two new
cases, membership revocation ordering, scope denials, disabled broker
behavior and hash-only persistence.
- Server TypeScript checking, the module-boundary gate and `git diff
--check` pass.
- Normalized syntax comparison finds all 294 pre-existing
function/method bodies unchanged.
- Full typecheck and build passed for the preceding batch. Its
interrupted local full-suite result remains unverified. All current-head
CI gates pass; Greptile is 5/5 with no open review threads.

## Risks

- A new closure could use the wrong secret client or move exchange
outside the membership lock. The existing transaction still passes
`secretService(tx)` to the exchange owner after taking that lock.
- Endpoint validation must preserve live configuration and fail before
credential disclosure. Existing rebinding and private-host tests cover
this path.
- The local fixture proves real HTTP and persisted application behavior.
It does not qualify a live third-party broker.
- No schema, API or access policy change is intended.

## Model Used

OpenAI Codex, GPT-6 family, with repository tools and code execution.
The exact serving model ID and context-window size are not exposed in
this session.

## Checklist


- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-21 22:21:42 -05:00
DottaandPaperclip 233753e577 refactor(tools): isolate profile ownership (#17155)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Tool profiles select the actions each agent can use.
> - Profile reads, review, mutation and assignment resolution shared a
large OAuth and connection service.
> - Those paths have their own company checks, lock order and database
projections.
> - This pull request gives profiles explicit owners while preserving
the existing transactions and public API.
> - The benefit is a smaller coordinator and a reviewable access
boundary.

## Linked Issues or Issue Description

Depends on #17147. Stack: #17144 → #17146 → #17147 → this PR. These PRs
remain open.

**What existing behavior does this improve?**

Maintenance of tool profile management and effective agent assignments.

**Current behavior**

The service entry point contains 19,293 lines. Profile behavior is
spread across its closure and returned object.

**Proposed behavior**

Profile management accepts only a database and clock. Assignment
resolution accepts three named capabilities. Projection and shared lock
functions have explicit homes. Public methods and responses remain
unchanged.

**Reason and benefit**

Reviewers can follow profile transactions, lightweight reads and
precedence without navigating OAuth setup. The entry point falls to
17,752 lines, a reduction of 1,541 lines in this PR.

## What Changed

- Extract profile management, projections and effective assignment
resolution into three modules. Their sizes are 886, 268 and 216 lines.
- Move shared actor/transaction types, identifiers and assignment locks
into named modules. Preserve the public lock export.
- Keep all database reads, transaction boundaries, lock order and method
signatures. Add no cache or background work.
- Record the next ownership cuts toward an entry point below 5,000
lines.

## Verification

- Before and after: 466 cases in `tool-access-service.test.ts` and 33
cases across profile entry validation, effective assignments and catalog
publication. All 499 cases pass with one database worker.
- Server TypeScript check and `node scripts/check-module-boundaries.mjs`
pass. `git diff --check` passes.
- Normalized syntax comparison finds all 329 pre-existing
function/method bodies unchanged. Wiring passes the unchanged public
service tests.
- The previous stack passed full typecheck and build. Its full local
test run was interrupted and remains unverified; its complete CI matrix
passed. This PR will record its own current-head CI result below.

- Current-head CI is green. Current review: Greptile is 5/5 at
`0f5c85905e`, with no inline review threads.
- CI failure preserved: [general server shard
1](https://github.com/paperclipai/paperclip/actions/runs/39704932041/job/119582710174)
failed the unchanged workspace-runtime test that replaces an unhealthy
local process. The replacement refused connections for its original
10-second readiness budget. The fixture and runtime service are
unchanged and do not import tool-access. One isolated run passed without
changes or timeout increases. The failure log is preserved; the cause is
not reproduced. Only that failed job was retried; it passed at the same
head without code changes. The original failure remains recorded above.

## Risks

- Moving profile mutations could change transaction clients or lock
order. Both are unchanged and the existing database tests cover
publication and assignment behavior.
- Effective assignment resolution still uses the same
installed-connection loader and full catalog projection. The lightweight
path still omits large schema and metadata fields.
- No schema, public API or access policy change is intended. Existing
tests provide characterization coverage; no new test mirrors the
extraction.

## Model Used

OpenAI Codex, GPT-6 family, with repository tools and code execution.
The exact serving model ID and context-window size are not exposed in
this session.

## Checklist


- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-21 22:21:42 -05:00
DottaandPaperclip 4413bb262f refactor(tools): isolate OAuth protocol boundaries (#17147)
## Thinking Path

> - Paperclip manages AI agents and their access to external tools.
> - Tool connections use OAuth to obtain credentials from remote
providers.
> - The tool-access service mixed provider protocol parsing with
database and secret writes.
> - This made protocol changes hard to review without reading the full
coordinator.
> - This PR gives discovery, endpoint policy, and client exchange
explicit owners.
> - The coordinator keeps authorization, leases, persistence, and audit
decisions.
> - A real local provider fixture checks the boundary through public
routes.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

The ownership of OAuth discovery, registration, token exchange, and
client metadata.

**Current behavior**

The tool-access service contains over 20,000 lines. Provider responses,
endpoint policy, and stateful operations share the same closure.

**Proposed behavior**

Small protocol modules accept settings and named request capabilities.
They return normalized results. The service owns durable state and
existing public exports. This is part 3 of the stack: #17144, then
#17146, then this PR. It targets #17146's branch. Review only this PR's
diff.

## What Changed

- Extract endpoint policy, metadata parsing/discovery,
registration/token exchange, provider error normalization, and the
public client metadata document.
- Validate remote registration responses in the protocol module. Return
client material to the coordinator for secret and connection writes.
- Remove the unused raw provider response from the internal token
result. Keep every consumed field.
- Preserve process-wide registration single-flight, service-local maps,
refresh leases, membership checks, callback checks, and transaction
ownership.
- Reuse connection-method URL normalization. Keep the endpoint setting
and clock callbacks live.
- Add a real local HTTP OAuth/MCP route fixture for public and
Basic-authenticated clients.
- Reduce `tool-access.ts` from 20,917 lines / 808,777 bytes to 19,293
lines / 749,158 bytes. The seven OAuth modules total 1,697 lines; the
largest has 582 lines. The stack removes 2,232 lines from the original
service.
- Record the next profile-resolution and token-broker boundaries in the
dated plan.

## Verification

- Baseline: 466 service cases and 80 protocol/provider-policy cases
pass. The new two-case HTTP fixture passes before moving production
code.
- After: `pnpm exec vitest run
server/src/__tests__/tool-access-oauth-protocol.integration.test.ts
server/src/__tests__/tool-access-service.test.ts
server/src/__tests__/remote-mcp-connectors.test.ts
server/src/__tests__/remote-http-rebinding.test.ts
server/src/__tests__/oauth-client-metadata.test.ts` — 508 tests pass
with one worker and disposable PostgreSQL.
- The fixture drives connect, metadata discovery, DCR, PKCE code
exchange, callback, catalog review, and finish. It checks persisted
active state, secret references, issuer/auth-method binding, and
wire-level authentication.
- `pnpm -r --workspace-concurrency=2 typecheck` passes. The sandbox-only
attempt could not open a local IPC socket; the same command passed with
local process permissions.
- `node scripts/check-module-boundaries.mjs` and `git diff --check`
pass.
- AST comparison confirms 37 moved bodies are unchanged. Registration
protocol and persistence statements match their original halves. Token
exchange only drops its unused raw result. Preflight uses explicit
defaults and a clock callback.
- Initial fixture authoring used a nonexistent catalog enabled field;
the corrected assertion checks persisted status and reviewer identity.
An initial typecheck found optional defaults at the new boundary; the
caller now passes its already-validated defaults.
- Final-stack local verification: `pnpm -r --workspace-concurrency=2
typecheck` and `pnpm build` pass. Remote CI is green at
`3dc46c132408556ecb31038a9b9621ac20aacefc`. Greptile is 5/5 with no
actionable findings or open inline threads.

- Full local-suite limitation: `pnpm test:run` was interrupted after 79
minutes during the serial general-server phase, after CI had completed
the full test matrix. Before interruption, it flagged the unchanged
`company-search-service.test.ts` suite (26 database cases skipped). One
isolated rerun passed all 28 cases without code changes. The original
log and a pre-interruption copy are preserved. The original failure
cause was not reproduced. The full local suite and its later groups are
not claimed as passed; the complete CI matrix is green on this stack.

## Risks

- OAuth endpoint checks and issuer/resource binding are security
boundaries. Existing denial, redirect, scope, DCR, secret-redaction, and
refresh tests cover the preserved paths.
- A moved closure could capture stale policy or split
registration/refresh ownership. The new modules hold no mutable state.
Policy is read through the existing callback, and state stays in the
service.
- The local fixture proves real HTTP and database behavior. It does not
claim a live third-party provider sign-in.
- No schema, API, authorization policy, or persisted-data migration
changes are intended.

## Model Used

OpenAI Codex, GPT-6 family, with code execution and repository tools.
The exact serving model ID and context-window size are not exposed in
this session.

## Checklist


- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-21 22:21:41 -05:00
DottaandPaperclip 8e95caf9ae refactor(tools): isolate connection method projection (#17146)
## Thinking Path

> - Paperclip manages AI agents and their access to external tools.
> - Connection methods define operator settings and managed tool
arguments.
> - Setup, catalog discovery, and gateway invocation must use the same
projection rules.
> - These rules were embedded in a large database service.
> - This PR gives them one module with a narrow settings contract.
> - Existing provider policy modules still decide their
provider-specific restrictions.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Connection method configuration and tool input projection.

**Current behavior**

The tool-access database service owns settings validation and pure
input/schema transforms. Callers import these through the service.

**Proposed behavior**

A connection-method module owns these transforms and accepts only
connection settings. Existing imports remain compatible. This PR depends
on #17144 and targets its branch. Review only this PR's diff. It is part
2 of the three-PR stack. #17147 follows this PR.

## What Changed

- Move method selection, settings validation, and header/schema/argument
projection into `tool-access/connection-methods.ts`.
- Replace database row parameter types with required config and
transport settings. No new validation is added inside typed calls.
- Keep Honcho, Google Chat, and Gmail policy in their existing modules.
Preserve their call order.
- Retain the service's public exports for routes, gateway consumers, and
tests.
- Reduce `tool-access.ts` from 21,237 lines / 820,116 bytes to 20,917
lines / 808,777 bytes. The new module is 359 lines / 12,665 bytes.

## Verification

- Baseline: all 466 tool-access service tests and 80
protocol/provider-policy cases pass before the move.
- After: `pnpm exec vitest run
server/src/__tests__/tool-access-service.test.ts
server/src/__tests__/memory-connectors.test.ts
server/src/services/google-chat-tool-policy.test.ts
server/src/services/gmail-tool-governance.test.ts` — 506 tests pass with
one worker and disposable PostgreSQL.
- `pnpm --filter @paperclipai/server exec tsc --noEmit` passes after the
existing workspace prerequisites were built.
- AST body comparison confirms the 11 moved functions and private record
parser are unchanged. `git diff --check` passes.
- The first typecheck found optional config in the public DTO. The new
input type now states the existing required-config invariant directly.
- Final-stack local verification: `pnpm -r --workspace-concurrency=2
typecheck` and `pnpm build` pass. Greptile is 5/5 at
`2d0bb86b5830c0306580e3ddc8637c4994b2a75a` with no actionable findings.
- CI passed all non-browser gates. [The first browser
attempt](https://github.com/paperclipai/paperclip/actions/runs/39675461620/job/119484152569)
failed in the unchanged Cases read-only test: the status filter remained
All statuses while the announcement-dismiss locator handler interrupted
the assertion. The same test passed on #17144. The failure log is
preserved. One retry of only the failed jobs passed without code
changes; all CI gates are now green on the same head.

- Full local-suite limitation: `pnpm test:run` was interrupted after 79
minutes during the serial general-server phase, after CI had completed
the full test matrix. Before interruption, it flagged the unchanged
`company-search-service.test.ts` suite (26 database cases skipped). One
isolated rerun passed all 28 cases without code changes. The original
log and a pre-interruption copy are preserved. The original failure
cause was not reproduced. The full local suite and its later groups are
not claimed as passed; the complete CI matrix is green on this stack.

## Risks

- A projection-order change could alter managed arguments or advertised
schemas. Function bodies and call order are unchanged, and tests
exercise setup, discovery, and gateway consumers.
- Legacy Gmail method aliases, GitHub toolset headers, missing unrelated
settings, and disabled method filtering retain their previous behavior.
- The module has no database dependency, new network work, or mutable
state.

## Model Used

OpenAI Codex, GPT-6 family, with code execution and repository tools.
The exact serving model ID and context-window size are not exposed in
this session.

## Checklist


- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-21 22:21:41 -05:00
DottaandPaperclip 9db3a5726e refactor(tools): isolate GitHub access discovery (#17144)
## Thinking Path

> - Paperclip manages AI agents and their access to external tools.
> - Tool connections discover repositories through GitHub account
grants.
> - The tool-access service mixes GitHub response parsing with grant
authorization and persistence.
> - These responsibilities have different inputs and failure rules.
> - This PR puts GitHub discovery in one provider module.
> - The service keeps current authority checks and stored grant updates.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

The ownership and review of GitHub access discovery in the tool-access
service.

**Current behavior**

The 21,525-line service contains GitHub pagination, account metadata
validation, and grant lifecycle code.

**Proposed behavior**

A GitHub module returns normalized access metadata. Existing service
exports remain compatible. This is the first PR in a three-part refactor
stack. No behavior change is intended. A search found no existing
tool-access refactor PR or issue.

Stack order: this PR, #17146, then #17147. Each PR targets its
predecessor.

## What Changed

- Move six functions into `tool-access/github-access-discovery.ts`.
Their bodies are unchanged.
- Keep grant selection, credential resolution, refresh retry, membership
locks, revision checks, and webhook updates in the service.
- Add public-entry characterization tests for HTTP error classification
and invalid pagination.
- Record five candidate boundaries and the initial three-PR scope in the
dated plan.
- Reduce `tool-access.ts` from 21,525 lines / 829,519 bytes to 21,237
lines / 820,116 bytes. The new module is 294 lines / 9,722 bytes.

## Verification

- Before and after: `pnpm exec vitest run
server/src/__tests__/github-grant-metadata.test.ts
server/src/__tests__/project-repositories.test.ts` — 13 tests pass.
- Before and after: `pnpm exec vitest run
server/src/__tests__/tool-access-service.test.ts -t 'GitHub|github'` —
11 selected tests pass with disposable PostgreSQL; 455 cases are outside
this selection.
- `pnpm --filter @paperclipai/server typecheck` passes.
- Exact text comparison confirms all six moved bodies match the base.
`git diff --check` passes.
- The initial baseline attempt failed because the plugin SDK had not
been built. The normal workspace prerequisite build resolved it before
behavioral validation.
- Final-stack local verification: `pnpm -r --workspace-concurrency=2
typecheck` and `pnpm build` pass. CI is green at
`07e78dd1b1c26e0bd4234006766444c71e234451`. Greptile is 5/5 with no
actionable findings.

- Full local-suite limitation: `pnpm test:run` was interrupted after 79
minutes during the serial general-server phase, after CI had completed
the full test matrix. Before interruption, it flagged the unchanged
`company-search-service.test.ts` suite (26 database cases skipped). One
isolated rerun passed all 28 cases without code changes. The original
log and a pre-interruption copy are preserved. The original failure
cause was not reproduced. The full local suite and its later groups are
not claimed as passed; the complete CI matrix is green on this stack.

## Risks

- Import wiring could break existing consumers. Existing imports remain
supported and all four production call sites are unchanged.
- Provider requests retain their ordering, timeout, fixed GitHub origin,
and response redaction. The new module has no database dependency or
shared mutable state.
- Tests simulate GitHub responses. They do not qualify live GitHub
access.

## Model Used

OpenAI Codex, GPT-6 family, with code execution and repository tools.
The exact serving model ID and context-window size are not exposed in
this session.

## Checklist


- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-21 22:21:40 -05:00
DottaandPaperclip 734c265339 Default new Pi agents to Paperclip Runner (#17142)
## Thinking Path

> - Paperclip manages AI agents and their work.
> - Users choose a harness during agent setup.
> - Claude, Codex, and OpenCode already receive their qualified Runner
default.
> - Pi has a qualified native integration but new Pi agents still use
the direct adapter.
> - This change gives new Pi agents the native default on supported
targets.
> - Saved agents keep their runner and compatible settings. Legacy
remains available in Advanced.

## Linked Issues or Issue Description

Refs #17120, #14921, #14922, #14923, #14924, and #14956. Related generic
default work: #15422. This PR is the separate Pi follow-up to the merged
OpenCode change.

**Current behavior**

Choosing Pi for a new agent saves the direct Pi adapter. The native Pi
integration remains behind the generic experimental flag.

**Proposed behavior**

Default new Pi agents to ACPX Pi on macOS ARM64/x64 and GNU/Linux x64
with supported glibc. Keep unsupported targets and explicit Legacy
selections on the direct adapter. Require an explicit provider/model
before test or hire. Permit model-free inactive setup drafts.

**Reason and benefit**

Users choose Pi once and receive its qualified runner. Saved agents
retain their identity and settings. Conversions reject unsupported
legacy thinking and launch settings instead of dropping them.

## What Changed

- Add Pi to the shared platform/default contract and server creation
selection.
- Admit native Pi without the generic experimental flag.
- Preserve provider models, credentials, workspace settings, exact
thinking, and saved native policies.
- Add Pi runner controls, branding, setup resume, onboarding, built-in
configuration, and import handling.
- Add managed Pi OpenRouter routing and matching native provider API-key
accounts. Keep subscription and custom gateway exclusions. Clear
alternate credentials and reject custom Pi provider overrides.
- Keep Pi draft models empty instead of applying Claude's default model.
- Verify setup against the pinned Pi runtime. For native Pi on the
official OpenRouter route, reverify the API key live without requiring a
global Legacy Pi CLI.
- Match Pi’s bounded controller close window to its native snapshot
cleanup and lifetime verification, preserving explicit caller deadlines
and the mandatory drain/suspension proof.
- Bind Pi sidecars to the package that owns the pinned native assets,
including source and vendored server layouts. Other providers retain
their existing package authority.
- Keep protocol-owned finish/block tools out of the ACPX forwarded
catalog. The trusted bridge installs these tools itself; the complete
Rust authorization catalog and terminal receipt ledger remain intact.
This fixes actual Pi session startup and applies to other ACPX
consumers.
- Advance Copilot’s shared-source attestation to v58 because it pins the
shared files touched by Pi. Preserve immutable v57 bytes and historical
decoding; Pi’s own release identity is unchanged.
- Add regression coverage and update the paired maps and developer
documentation.

## Verification

- Passed: full repository typecheck and full production build on final
head 656d304783, including the generated attestation repair.
- Passed: focused shared/server/UI suites covering default selection,
platform fallback, conversion preservation, inactive drafts, managed
account compatibility, credential isolation, OpenRouter routing/model
IDs, exact thinking controls, and saved-account selection. Relevant
batches include 375 server/conversion tests, 267 account/routing tests,
and 110 UI setup/account tests.
- Passed: 179 attestation, historical-profile, qualified-profile, and
native-execution contract tests after CI identified stale Copilot source
hashes. The identity generator check passes; the only delta after the
passing Pi browser run is this Copilot attestation/history repair.
- Passed: 145 managed-account tests, including actual service
preparation and API-key adoption/rejection for native Pi without the
Legacy CLI; 6 pinned-installation probe tests.
- Red then green: an actual task failed session.open because its
54-operation catalog included the bridge-owned terminal tools. The
regression with those tools failed before the filter; all 399 Rust
library tests pass after it. The full product retry exposed an
additional package-authority mismatch. Four authority tests failed
before that repair; 11 authority checks and 14 installation/asset checks
pass afterward. Direct real Pi session initialization with the corrected
package root and catalog succeeds. No model prompt was sent by that
diagnostic.
- Red then green: real Pi posted Hello from Pi but its controller closed
before native cleanup could finish. The cleanup-budget test failed with
the old default and passes with the bounded Pi window. All 46 selected
suspension, drain, and package-authority tests pass after building their
native fixture.
- Passed: token gates, paired map checks, Rust formatting, and git diff
--check.
- Passed real browser acceptance on macOS ARM64, generic native runner
flag disabled, source a433748a12 (Pi code unchanged afterward): the UI
selects Paperclip Runner by default, retains Pi and the selected
model/account, and verifies the pinned runtime plus live OpenRouter key.
A fresh UI-created task, assigned to the UI-created native Pi agent,
posted exactly `Hello from Pi` and reached Done without retry or
restart. The native run and workspace finalization succeeded; the
response and Done persisted after Reload.
- Evidence: 8 preserved browser attempts, including the final passing
18-step journey with all screenshots. Prior failures remain recorded.
The earlier failed checkpoint run recovered on server restart; that
recovery is separate from the fresh final task’s clean success.
- Preserved browser failures: missing account control/inherited Claude
draft model; incompatible managed-account/model-prefix handling; generic
npm probe for Pi; Legacy CLI hello check; duplicate terminal tools and
wrong provider-package authority at actual native session.open; a real
Hello from Pi comment followed by premature checkpoint cleanup timeout.
Each failure has retained screenshots/logs and a corresponding focused
repair. The fresh final task proves these repairs end to end.
- The exact Runner Vitest shard was also run locally after the
attestation fix: the stale-hash failure is gone; 1,833 tests passed, 10
skipped, and two Codex retained-control settlement scenarios failed with
`native_cleanup_maintenance_unproven` during terminal ACK/suspension
replay. Both scenarios passed in an isolated rerun (2/2), and the same
19-test file passed in the preceding Linux CI run. The precise
full-shard timing cause is not established. These Codex scenarios are
outside the changed Pi paths; the original local failures are retained;
no complete local Runner-shard pass is claimed. The corresponding Runner
shard now passes on the final head in hosted Linux CI. All 60 final-head
check runs are now terminal and clean (55 successful, 1 neutral, 4
skipped), and the legacy Snyk status is successful.
- Local full repository tests were attempted but not completed:
provisioning briefly confused a workspace symlink, then a later run was
stopped before browser-driven repairs. No full local-suite pass is
claimed. Final-head hosted CI is green and fresh Greptile review is 5/5
with all review threads resolved; earlier jobs canceled by runner
shutdown are not counted as a pass.

## Risks

- Pi cleanup has a finite 45-second default window, accounting for the
native 30-second snapshot cleanup and 5-second lifetime fence. Other
providers and explicit caller deadlines retain their prior behavior.
- Pi requires explicit operator setup with `paperclipai runtime setup
pi` on the execution host. This PR does not add automatic downloads.
- Linux ARM64, Windows, musl, and unqualified targets retain Legacy.
- Legacy thinking levels minimal, medium, and xhigh cannot be translated
exactly. Native max thinking, warm lifecycle, or restricted permissions
cannot be dropped during a return to Legacy.
- Copilot’s bound profile advances to v58 with exact current launch
admission; v57 remains decodable for recovery. No historical identity is
rewritten.
- Existing agents and immutable run descriptors are not migrated. Cloud
Pi availability is unchanged.

## Model Used

OpenAI GPT-6 through Codex with reasoning, code editing, command
execution, and browser use. The exact serving model ID and
context-window size are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-21 17:10:00 -05:00
DottaandPaperclip 92cfd701c3 feat: add governed Braintrust app connections (#17141)
Add hosted US and EU Braintrust OAuth and API-key connections through the existing governed MCP system.

Keep reviewed tool risks and catalog review policy in connector modules and a narrow registry. Preserve credential ownership, operator decisions, regional setup and changed-tool quarantine. Include official artwork, setup guidance, focused regressions and sanitized qualification evidence.

US OAuth has bounded live read/write, gateway, reconnect and cleanup proof. Remaining account-method and token-lifecycle qualification gaps are documented.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-21 16:19:58 -05:00
DottaandPaperclip efc2ddc7bf fix: stop idle login cleanup polling (#17106)
**Dependency landed:** PR #17129 is merged. This branch is rebased onto
master `0f15f82b19` and preserves its provider-owned completion
contract. The shared save path calls `notifyLoginSessionChanged(tx)`;
the cleanup module invalidates both existing sandbox cleanup workers
without selecting a provider.

## Thinking Path

> - Paperclip manages work for AI agents.
> - Subscription login can allocate a temporary sandbox.
> - Existing cleanup workers scan the database every heartbeat tick,
even with no login work.
> - Empty scans keep idle databases active.
> - This PR uses committed session changes and saved expiry times to
schedule cleanup.
> - Existing rows still drive recovery after failure and restart.

## Linked Issues or Issue Description

**What happened?**
On master `898144f4b3`, the enabled heartbeat scheduler runs both login
reapers every 30 seconds by default. An instance with no pending logins
still queries sessions and leases.

**What did you expect to happen?**
After startup reconciliation, idle login cleanup should make no
recurring SQL or provider calls. New sessions, expiry, failed cleanup
and restart must still recover.

**Steps to reproduce**
Start a claimed instance with heartbeat scheduling enabled and no
pending login sessions. Observe repeated adapter-login and setup-token
cleanup queries at the heartbeat interval.

Related sleep migration: #17094. Searched open issues and PRs. Related
setup-token cleanup work: #15774, #14236 and #16648. This PR does not
change their release-result or null-lease behavior.

## What Changed

- Register separate adapter-login and setup-token workers with the
existing delivery coordinator and scheduler.
- Notify before writes in the owning stores, the lease-tag write, and AI
connection save. Shared save uses a provider-neutral notification; each
cleanup worker checks its own durable predicates. The shared transaction
protocol retains intent until the outer transaction settles.
- Reconstruct session expiry, promotion expiry and concrete cleanup
retries from existing rows. Preserve configured heartbeat spacing to
bound scan bursts.
- Recover tagged orphan leases even with terminal or missing sessions.
Protect live acquisitions that have not attached their lease yet.
- Exclude local-subscription rows from sandbox reapers. Their existing
local-login worker owns cleanup.
- Preserve deferred work across drain and shutdown. Add no tables, Cloud
alarms or host-sleep exemptions.
- Update the inventory and document producer coverage, writer scope and
recovery boundaries.

## Verification

- Updated head: `1d57e13606`. Seven focused suites passed (282 tests):
AI connections, login cleanup, device-login service/reaper,
login-session runtime, and setup-token session/reaper. `pnpm -r
typecheck` and `pnpm build` passed. The duplicate local `pnpm test:run`
was interrupted after CI passed all server shards; it is not counted as
a complete local pass. Full-suite evidence comes from the exact-head CI
run. Greptile reviewed this exact head at 5/5 with no findings or
unresolved threads. [Full CI run
39669285577](https://github.com/paperclipai/paperclip/actions/runs/39669285577)
passed on attempt 2. Both aggregate gates passed; no checks remain
failed or pending.
- Extended the real PostgreSQL AI connection tests for both Anthropic
and OpenAI. The tests check that neither cleanup observer wakes inside
the completion callback, both wake after commit, and an idempotent save
does not emit another wake.
- Existing cleanup tests cover idle/history states, producer wake,
expiry, promotion, cancellation, retry/restart, orphan recovery, nested
commit/rollback, drain, concurrency, and shutdown.
- Current-head CI attempt 1: the unchanged
`sandbox-callback-bridge.test.ts` HTTP/2 preface test failed because its
child process could not bind `127.0.0.1:34033` (`EADDRINUSE`). Its
single unchanged rerun passed. The exact test also passed locally
without changing code or timeouts. This establishes recovery from the
bind collision; it does not establish which process owned the port. No
login code ran in that package test.
- Previous-head failures remain recorded: the login-runtime transaction
fixture was corrected; a workspace-runtime readiness port-bind failure
passed in isolation and fresh CI, but its first failure cause remains
unconfirmed. An earlier combined AI suite timed out in local-login
isolation and passed alone; its timing cause remains unconfirmed. An
earlier local full suite was interrupted and was not counted as passing.

## Risks

- Process-local signals cover one active server with the instrumented
shared database handle. Independent SQL writers or replicas need their
own signal integration.
- Provider resources that never became persisted tagged leases remain
the responsibility of the unchanged general lease/spool recovery.
Setup-token acquisition retains its required provider expiry.
- Cleanup starts are bounded by the configured heartbeat interval.
Writes during a sweep can cause a final follow-up scan before quiet.
- Existing login and resource blockers still prevent unsafe host sleep.
This is worker polling removal, not provider sleep qualification.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository tools and test
execution. The exact serving model ID and context-window size are not
exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes:` / `Closes`
/ `Refs` OR (b) described the issue in-PR following the relevant issue
template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub references)
- [x] My branch name describes the change and contains no
internal/instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-21 15:39:23 -05:00
DottaandPaperclip 0f15f82b19 feat: offer existing Claude and Codex agents a runner upgrade (#17122)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Each agent stores its harness, model, account and runtime
configuration.
> - New Claude and Codex agents now use Paperclip Runner on qualified
targets.
> - Existing agents keep their legacy runner and need a clear upgrade
action.
> - The existing configuration API already translates settings and
records revisions.
> - This pull request adds an explicit upgrade action that uses that
API.
> - Operators can retain supported settings and correct rejected
settings in place.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Existing Claude and Codex agent configuration. Related defaults: #15683
and #15993.

**Current behavior**

Existing agents retain their legacy runner. The operator must find the
runner selector in Advanced to change it.

**Proposed behavior**

Show Upgrade runner when Paperclip Runner is the qualified default for
the saved environment. One click saves pending edits and requests the
existing server conversion. Show any rejected settings without changing
saved configuration.

**Reason and benefit**

Operators can adopt the current runner while retaining their harness,
model, account bindings, instructions and history. Unsupported settings
remain visible and editable.

**Breaking changes**

None. Visiting configuration and saving unrelated changes do not upgrade
an agent. The server API, authorization and conversion rules are
unchanged.

## What Changed

- Add an upgrade notice, pending state, saved result and retry feedback
to the production configuration form.
- Include pending profile, runtime and environment-variable edits in the
upgrade request.
- Bind asynchronous upgrade feedback and save receipts to their original
agent. Navigating away cannot show success on another agent or redirect
back.
- Keep the notice absent for native agents, other harnesses, unavailable
targets and failed discovery.
- Add configuration/API regressions and database session-reset checks
for Claude and Codex.
- Add 13 production-page Storybook states with browser checks in both
themes and on mobile.
- Update the product contract and feature/access maps.

## Verification

- Configuration form and original-agent save receipts: 135 tests passed.
Existing agent route/rename helpers also pass.
- Runner selection, configuration API and database migration
regressions: 220 tests passed.
- Storybook browser checks: 26 passed. Interactive cases wait for
Storybook's visible Pass report. Static cases check the rendered
production form. These use simulated HTTP boundaries.
- Full typecheck, build, map checks and token gates passed on the
current head. Storybook build and public preview verification passed.
- Published fixture preview:
https://storybook.paperclip.page/storybook/branches/codex~2Fagent-runner-upgrade/?path=/story/agents-runner-upgrade--legacy-codex
- All 58 current-head PR checks passed, including the full test shards,
browser suites, build and aggregate gates. The duplicate local pnpm
test:run was stopped after 34 minutes once the same-head full CI suite
passed; it had not finished its general-server phase. No local
full-suite pass is claimed.
- Real disposable local test drive: Codex used gpt-5.4 and Claude used
claude-sonnet-4-6. Both completed one task on Legacy, upgraded through
the browser button, and completed another task on Paperclip Runner with
the same model and API credentials. Independent file checks confirmed
saved instructions and environment bindings before and after conversion.
- Reopened both configurations and compared identity, model, credential
references, instruction hashes, working directories, run policies and
configuration revisions.
- A Claude Chrome setting produced a field-specific rejection. The saved
agent stayed Legacy. Clearing Chrome in the form and clicking Upgrade
runner saved the pending correction and completed conversion.

## Risks

- The existing translator rejects settings that the new runner cannot
honor. The notice shows that error and leaves the form available for
correction.
- Converted agents start fresh sessions. Prior runs remain available.
- Real task execution was checked on local macOS with API credentials.
SSH, sandbox targets and subscription credentials were not exercised in
this change.
- Storybook responses simulate persistence. The real test drive and
database tests provide separate persistence evidence.

## Model Used

- OpenAI GPT-6 through Codex. The exact serving variant and
context-window size were not exposed in this session. Used reasoning,
code editing, shell tools and browser interaction. No subagents.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-21 15:13:42 -05:00
DottaandPaperclip 39e44a9d9b Default new OpenCode agents to Paperclip Runner (#17120)
Default new OpenCode agents to Paperclip Runner on qualified macOS and Linux x64 targets. Preserve saved runner choices, explicit Legacy selection, and unsupported-target fallback. Pi remains a separate change.

Verified the complete browser path with a real OpenRouter connection: create the new agent, assign a task, observe its response and Done status, and reload to confirm persistence. CI passes and Greptile is 5/5 with no unresolved findings.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-21 15:09:15 -05:00
DottaandPaperclip 0a5f1546ee feat(cases): add reusable prompt workflows (#17114)
## Thinking Path

> - Paperclip manages AI agents and their work.
> - Cases keep facts, documents, and one primary task across agent
handoffs.
> - Each case currently needs its own procedure text.
> - Agents need to reuse procedures and move cases between them without
replacing the task.
> - This change adds company workflow definitions with a name and
Markdown prompt.
> - Existing handoff and document transactions preserve authority,
revisions, and recovery.
> - Cases remain behind `experimental.enableCases`.

## Linked Issues or Issue Description

Refs #16959. Related proposal: #3951 uses task graphs and a visual
editor. This change extends existing Cases with prompt definitions and
keeps its current handoff model.

**Problem or motivation**

Agents cannot share a reusable procedure across cases or inspect a
case's movement between procedures.

**Proposed solution**

Add name-and-prompt workflow definitions. Adopt a prompt into the
primary task's revisioned workflow document through a handoff. Keep
phases as free text.

**Alternatives considered**

Defer graphs, workflow versions, run entities, role schemas, authoring
screens, and Spec Kit integration.

**Roadmap alignment**

This extends the existing Cases experiment and same-task review flow. It
adds no work queue or new execution engine. The maintainer approved this
scope in the implementation plan.

## What Changed

- Add company-scoped definition storage, shared request schemas, REST
routes, and four optional Cases tools.
- Accept a workflow ID at kickoff. Agent handoffs can adopt, refresh,
preserve, or detach a workflow.
- Capture prompt text and the document base revision at acceptance.
Apply workflow, phase, and document revision together after outgoing
execution settles.
- Hold conflicting document edits for recovery. Preserve cancellation,
replay, and restart behavior.
- Show current workflow names and captured transition names/phases in
Work, history, and Audit.
- Update instructions, API/tool contracts, capability maps, and the
isolated workflow journey.
- Pin bounded workflow prompt reads to a content fingerprint;
continuation conflicts on edits and can restart safely.
- Remove experimental Cases/workflow/Mermaid guidance from the always-on
native prompt. Extract runtime-context prompt strings into a
CODEOWNERS-protected constants file while preserving the public imports.
- Refresh Copilot’s pinned source identity to v57, including the
extracted prompt file and retaining historical identities.

## Verification

Local checks: repository `pnpm -r typecheck` and `pnpm build` pass on
the prompt-extraction tree (`88222d45d8`). Full repository coverage
previously completed through the supported isolated shard groups (1,082
general server suites, 161 serialized suites, and all three workspace
groups). The earlier default serial run was interrupted and remains
incomplete. The first sharded run retained the failures described below;
targeted corrections and separate limited-concurrency reruns pass. All
CI jobs, security checks, and the 5/5 Greptile review passed on
`fc1bd79c28`. Fresh CI and review are pending for the prompt extraction
in `88222d45d8`. Focused verification passes: real-database handoff
recovery and reuse; 2,002 native-tool/API contract checks; 144 Runner
catalog/discovery checks; 27 Runner context/coverage checks; 37
case/Audit UI checks; and Product E2E oracle calibration. Token gates,
migration safety/numbering, module boundaries, capability maps, and
generated tool/Storybook catalogs pass. The final route/context
corrections pass 49 API/native-tool tests, 90 real-database handoff
tests, and server typecheck; 142 Copilot installation/sidecar tests pass
with the refreshed identity.

Greptile’s first review found that successive prompt pages could mix
definitions after an edit. The fix returns a content fingerprint,
requires it for continuation, and rejects edits with 409. Strict
live/catalog schemas pass (171 Runner checks). The 2,005 API/OpenAPI
tests pass; the initial native regression caught compression’s weak ETag
prefix, and all 23 native integration tests pass after handling it. The
regression covers same-length edits and a four-byte restart window. No
stored version fields or entities were added.

The prompt extraction passes all 462 checks across six focused suites,
including persisted-context recovery and provider transport. The first
local transport attempt lacked the required debug Runner binaries (102
failures, retained); after building the normal prerequisites, all 283
tests in that suite pass without source or assertion changes. Existing
saved snapshots retain their recorded prompt text, and new executions
use core-only prompt v7. The extracted prompt source is now covered by
both CODEOWNERS and Copilot’s source fingerprint.

Live-provider evidence is separate from those simulated checks: the
isolated API-key campaign `reusable-workflows-api-final-2026-10-21`
passed with local code matching `a1d73b4652`. Four actual Codex API runs
read the adopted task documents, moved one case A → B → A on the same
task, completed normally, and passed all nine durable checks. Browser
assertions verified the current workflow/phase, historical names/phases,
five handoff records including terminal completion, and the completed
original task. Cleanup passed. The local report remains under
`tests/runner-e2e/results/reusable-workflows-api-final-2026-10-21/`; no
credentials or screenshots are committed.

CI initially caught an omitted route-inventory entry and a stale pinned
instruction-source hash; both are corrected and separately verified.
Local serialized coverage also recorded an unrelated environment-route
socket disconnect and activity-route timeout; their separate 11-test and
17-test suites passed without code or timeout changes. An unchanged
chunked-webhook check also passed its separate four-test rerun. The
initial adapter/database workspace groups hit time and resource limits
under default worker counts; complete reruns with `VITEST_MAX_WORKERS=2`
pass, including all 55 database files and 211 tests. The UI/CLI groups
passed 837 files and 9,442 tests in the original run. Original failures
remain retained and are not regraded.

Earlier failed live campaigns remain unchanged: subscription HTTP 429
before agent execution; an obsolete exact-four-row oracle that rejected
the valid terminal fifth row; and a chat-marker expectation that did not
account for the existing explicit-comment presentation policy. The final
fixture requires terminal completion and places the marker in its
authoritative comment. Positive/negative oracle tests and the fresh live
campaign pass.

## Risks

- The additive migration adds a table and nullable case/handoff fields.
Existing rows keep null workflow metadata; no assignments are inferred.
- Adoption can wait on an intervening task-document edit. Recovery must
cancel the old intent and accept a new one after readback.
- Shared definition edits affect future adoptions. Existing case copies
keep their revisions.
- The feature remains experimental. This PR does not merge or deploy the
change.
- Live acceptance covers native Codex before the prompt extraction; this
cleanup has contract/transport verification rather than a new
live-provider campaign. The refreshed Copilot identity has static
compatibility evidence, not new live-provider qualification.

## Model Used

OpenAI Codex, configured model ID `gpt-6-astra`, xhigh reasoning. Used
terminal execution, file editing, and tool calls. The active
context-window size is not exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-21 15:00:12 -05:00
DottaandPaperclip 159d006bb4 refactor: let login flows own session completion (#17129)
## Thinking Path

> - Paperclip manages AI agents and their connected accounts.
> - Login flows bind verified credentials to an owned connection.
> - The shared save service currently uses the provider name to choose a
session ID column and completion state.
> - A provider can support more than one login flow. Provider identity
does not define session identity or state.
> - This PR gives every flow the same internal session ID contract and
lets the setup-token owner complete its own claim.
> - Credential writes, session binding, and completion remain in one
transaction.

## Linked Issues or Issue Description

Refs #17106. This is an independent prerequisite for that sleep cleanup
PR. It contains no scheduler or cleanup-worker changes.

**Current behavior**

The connection save service looks up Anthropic sessions by public ID and
all other sessions by internal ID. It also marks every Anthropic session
as stored. Local Anthropic login works because its public and internal
IDs happen to match, and its caller overwrites that intermediate state.

**Proposed behavior**

Every caller passes an internal session ID. A small setup-token
completion service resolves the public ID within its owner scope and
supplies a transaction callback. That callback uses the existing
conditional stored transition. Device and local login flows keep their
own final states.

**Reason and benefit**

Adding or changing a login flow no longer requires provider branches in
the shared session binding path. The service still owns membership
checks, row locking, target validation, credential writes, and
idempotent binding. The flow owns only public-ID resolution and its
completion transition.

**Breaking changes**

The internal TypeScript save signature changes. All production callers
are updated. Public HTTP payloads, database schema, and local session
URLs stay unchanged.

## What Changed

- Replace the ambiguous session ID argument with an internal session
reference and an optional completion callback.
- Run the callback inside the existing save transaction, only for the
first successful binding.
- Move public-ID resolution and the stored transition to
setup-token-ai-connection.ts.
- Update setup-token, device, and local callers.
- Add real PostgreSQL tests for ID selection, owner and target checks,
one-time completion, and rollback after a rejected completion.
- Exercise local Anthropic and OpenAI HTTP completion with distinct
public and internal IDs.
- Document the ownership contract in AI-CONNECTIONS.md.

## Verification

- Passed: pnpm -r typecheck.
- Passed: pnpm exec vitest run
server/src/__tests__/ai-connections.test.ts
server/src/__tests__/agent-device-login-routes.test.ts
server/src/__tests__/agents-claude-oauth-binding.test.ts — 233 tests.
- Passed: pnpm build.
- Passed: full CI, including test shards, typecheck, build, and E2E, on
d98ce6e504:
https://github.com/paperclipai/paperclip/actions/runs/39654210168.
- The additional local pnpm test:run was stopped after the full CI pass
to avoid duplicating the completed run. It is not reported as a
completed local full-suite pass.
- Greptile: 5/5 on d98ce6e504, with no
actionable findings or unresolved threads.
- The first AI suite run had two new fixture failures: a duplicate
environment name across provider cases, and an assertion that ignored
existing shared connections. The fixtures now use distinct names and
query the attempted connection directly. The corrected affected suites
passed.

The first CI run passed all 1,420 assertions in server shard 11/12 but
failed on an unhandled child-process rejection in
`workspace-runtime-exposure.test.ts` (the synthetic EADDRINUSE case).
Both the test and runtime implementation are unchanged from this PR's
base. The two affected parameter cases passed locally unchanged (2
passed, 29 skipped). Inspection shows a possible timing window before
the early-exit promise joins the readiness race; that cause has not been
reproduced conclusively. The single CI retry passed on the same commit.
This demonstrates recovery; the original timing cause was not
conclusively reproduced. Original job:
https://github.com/paperclipai/paperclip/actions/runs/39654210168/job/119410225203.

## Risks

- A completion callback failure must roll back the credential and
session binding. The real-database regression test proves this for a
rejected setup-token claim.
- The callback is internal, awaited inside the existing transaction, and
must use the supplied transaction handle. It adds no network work or
provider SDK dependency.
- Setup-token resolution adds one indexed lookup per completion. The
shared save then locks and validates the row again. No recurring work is
added.
- The setup-token owner now uses the existing conditional completion
guard, including the unconsumed-claim and commit-time expiry checks.
Invalid claims fail closed.
- Live provider sign-in was not exercised; tests use fixture credentials
and real PostgreSQL. No schema migration is needed.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, source inspection, code
editing, and test execution. The exact serving model ID and
context-window size are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-21 14:42:38 -05:00
Devin Foley a1e2173ceb feat(apps): add Ditto connection (#16472)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents reach external services through the Apps catalog. Each
catalog entry is a reviewed `AppDefinition` that connects a provider's
hosted MCP server to Paperclip's shared vault, grants, policies,
gateway, and audit trail.
> - Ditto (`dittowords.com`) is a single source of truth for product
copy. Its hosted MCP server lets agents read style guide rules, search
and reuse approved text, and create, edit, translate and delete text,
components, variants and style guides.
> - Ditto is not in the catalog. Operators must use the generic "Connect
your own MCP server" flow. That flow has no branding, no guidance about
Ditto's `token` API-key scheme, and no warning about permanent deletes.
> - The generic risk classifier also reads several Ditto mutations as
reads: `translate_*` replaces variant text, and `link_*` and `unlink_*`
replace developer IDs.
> - This pull request adds the Ditto definition, its artwork, the
research and permission-review ledger rows, documentation, deterministic
tests, and one reviewed risk-classification rule.
> - The benefit is a branded, governed Ditto connection with browser
sign-in, an API-key option that uses the correct header, and correct
risk levels for every documented tool.

## Linked Issues or Issue Description

**Problem or motivation**

Product and engineering teams use Ditto to keep UI copy consistent. They
want their Paperclip agents to follow the Ditto style guide and reuse
approved strings when they write code. Ditto is not in the Apps catalog.
Operators must paste the MCP URL into the generic remote-MCP flow. That
flow gives no branding and no method guidance. It also does not tell the
operator that Ditto API keys use `Authorization: token <key>`, not
`Bearer`, and it does not warn that delete tools are permanent. The
generic classifier also marks five Ditto mutations as reads, so the
Permissions screen shows them with the wrong risk level.

**Proposed solution**

Add a catalog-only Ditto connection built from the connector playbook.
Browser sign-in uses dynamic client registration on Ditto's Auth0
authorization server and requests the reviewed scopes `openid`,
`profile`, `email` and `offline_access`. An optional method sends a
customer-created workspace API key with the `token` scheme. Both methods
share one `write` capability profile, so the delete and link warning
shows before sign-in. Add a reviewed `ditto` rule to `classifyRisk`:
`list_*`, `get_*` and `search_*` tools, `suggest_edit`, and tools that
Ditto marks read-only are reads; `delete_*` tools are destructive; all
other tools are writes.

**Alternatives considered**

A plugin was not needed because no custom UI, tables, workers, or
webhooks are involved. Requesting no scopes was not chosen, because
Auth0 issues a refresh token only for `offline_access`, so the
connection would expire with its first access token. Ditto's separate
documentation MCP server (`developer.dittowords.com/mcp`) was not added,
because it only searches Ditto's public docs.

**Roadmap alignment**

This extends the existing self-serve remote-MCP connection catalog and
does not overlap planned core work.

## What Changed

- Added the `ditto` provider to `scripts/ingest-app-definitions.mjs`
(category `content`, `token` API-key placement, guidance, warnings,
shared capability profile, description) and regenerated
`packages/shared/src/app-definitions/ditto.json` and the generated
registry.
- Added the Ditto row to the self-serve MCP research ledger with
`dcr_or_api_key` auth and risk tier S4, because the normal tool catalog
includes permanent deletes. Widened `SelfServeMcpResearchEntry.wave` to
include `4`, which existing entries already use.
- Added permission reviews for `ditto/mcp-oauth` (explicit scopes
`openid`, `profile`, `email`, `offline_access`) and `ditto/mcp-api-key`
(provider key), with evidence links.
- Added a reviewed `ditto` rule to `classifyRisk` in
`server/src/services/tool-access.ts`.
- Added Ditto's mark (`ui/public/brands/apps/ditto.png`, the avatar of
Ditto's official GitHub organization) and the brand manifest entry.
- Added gallery copy for the Ditto card.
- Added `doc/connections/DITTO.md` (sign-in sequence diagram, provider
and Paperclip endpoints including `/api/tools/oauth/callback`, scopes,
administrator setup, capabilities and policy, manifest, brand
provenance, validation hook) and linked it from the connections README
and the permission audit.
- Tests: Ditto definition shape, store visibility and artwork, URL
recognition, reviewed scopes, `token` header projection with the key
kept out of connection config, risk classification of every read-looking
Ditto mutation, the connect form's sign-in default, the warning before
sign-in, API-key gating, and the pinned catalog counts.

## Verification

- `pnpm exec vitest run packages/shared/src/app-definitions.test.ts
packages/shared/src/app-definitions-url.test.ts
ui/src/pages/apps/AppsConnect.test.tsx ui/src/pages/apps/Browse.test.tsx
ui/src/lib/app-brand-assets.test.ts
ui/src/pages/apps/AppLogo.brand-assets.test.tsx
server/src/__tests__/tool-access-service.test.ts` — 765 passed.
- `node scripts/check-app-brand-assets.mjs` and `node --test
scripts/app-brand-validation.test.mjs` — passed.
- `pnpm --filter @paperclipai/shared typecheck`, `pnpm --filter
@paperclipai/server typecheck`, `pnpm --filter @paperclipai/ui
typecheck` — clean.
- Manual: on a local instance, open Apps → Browse and confirm the Ditto
card and icon. Open `/apps/connect?source=ditto` and confirm that
sign-in is the default, that the warning shows before sign-in, and that
the API-key method is under Advanced.
- Live metadata probe: an unauthenticated `initialize` on
`https://api.dittowords.com/v2/mcp` returns 401 with
`resource_metadata`. The protected-resource and authorization-server
documents return the recorded endpoints. Dynamic client registration
succeeds, and the authorize endpoint redirects to Ditto's sign-in page
with the reviewed scopes and the `resource` parameter.
- Each of the 41 documented tool names was run through `classifyRisk`
with and without the new rule. Without the rule, `translate_text_items`,
`translate_library_components`, `link_figma_nodes_to_text_item`,
`link_text_items_to_component` and `unlink_text_items_from_component`
were reads.

## Risks

- Low risk to existing providers: the catalog change is additive, and
the new `classifyRisk` branch runs only for the `ditto` source template.
- Ditto delete tools are permanent, and linking a text item to a
component replaces its developer ID. Both methods warn the operator
before connecting, and the API-key helper text tells operators to set
delete and link actions to Ask first.
- Ditto API keys are not scoped, reach the whole workspace, and act as
the user who created them. Paperclip cannot narrow an issued key.
- The permission-review ledger records live proof for both methods as
not run. The lifecycle checklist in `doc/connections/DITTO.md` still
needs a documented pass with a Ditto workspace, including confirmation
that Auth0 issues a refresh token that `/v2/mcp` accepts.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`, 1M context) in Claude Code, with
tool use (shell, file editing, web research).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-10-21 11:29:48 -07:00
6dda8cc2b7 fix: preserve Stop authority and qualify complete recovery journeys (#17105)
## Thinking Path

> - Paperclip lets people manage agents that do useful work.
> - Users need complete tasks, clear questions, retained context, and
dependable recovery.
> - Separate run tests do not prove that the complete user experience
works.
> - Successful retries can hide earlier failures and missing evidence.
> - This PR extends the existing Product E2E harness with three explicit
journeys.
> - It adds a qualification checkpoint over the full retained result
pool.
> - Independent artifacts, state, rendered conversations, and delivery
receipts establish each outcome.

## Linked Issues or Issue Description

Refs #17088, #17089, and #17046. This PR repairs two Stop lifecycle
defects and combines the qualification changes previously reviewed in
#17095, #17090, and #17099. Daily operations remain with #17051. It adds
no second scheduler or recovery engine.

**What happened?**

A completed run can still leave a user without useful output, an
understandable question, or a working continuation. Existing isolated
tests and best-attempt reports do not establish the combined behavior.

**Expected behavior**

A necessary question includes its explanation and resumes once after an
answer. Stop prevents automatic continuation until the user resumes.
Restart during finalization preserves the accepted provider result and
delivers it once. Qualification requires independent evidence for six
journeys on both agents through Slack and Paperclip.

**Steps to reproduce**

List the necessary-question, followup-stop-resume, and
finalization-recovery suites using the documented selectors. Run one
exact cell with the configured model and target. Supply a private
qualification manifest to the existing report command to grade all
retained attempts. Missing proof and incomplete coverage must fail.

## What Changed

- Wake existing accounting in the same committed native Stop closure. A
receipt that arrives before closure must still settle once; incomplete
receipts remain held.
- Require a later user message before native Stop can promote earlier
saved input. Preserve those earlier messages for the authorized
resumption.
- Add explicit necessary-question cells. Verify the actual prompt and
context, answer, one bound continuation, and the delivered document link
and revision.
- Add explicit follow-up, Stop, and later resumption cells. Verify a
live owned provider turn, cancellation acknowledgement, at least five
seconds without an automatic path, retained direction and workspace
context, and an actual browser download.
- Add explicit Daytona finalization-recovery cells. Stop the owned
controller after public result acceptance and during workspace
finalization. Require the original result, restored file hash, resumed
operation, one provider turn, and one receipt-bound delivery.
- Share the receipt-based delivery observer across both recovery
journeys. Keep model prose separate from the authoritative delivered
comment identity.
- Add a 24-cell qualification checkpoint over every retained attempt.
Require three fresh candidate passes per cell and later staging proof
with exact source, runtime, model, settings, instructions, and
independent evidence.
- Reject absent or unsafe manifest evidence. Preserve failed attempts
and incomplete observations in the normal reports.
- Provide four explicit rotating daily selections. Refuse daily
selection until all required live drivers exist. Keep scheduling and
spending with the existing operations owner.
- Document the exact selectors, evidence contract, limitations, and
remaining staging work.

## Verification

- Local Product E2E support: 3,554 passed with two existing skips;
source prerequisites: 128 passed. E2E typecheck passes. The actual
file-chip and Download-control browser regression passes.
- Two real-database regressions fail before the production repair. The
stopped coordinator leaves accounting asleep after a complete receipt;
the saved queue can restart from pre-Stop input. The repaired settlement
and cancellation cohort passes 66 checks. The complete 256-case
continuation suite passes with stronger two-phase Stop expectations, and
66 native executor accounting/Stop safety cases pass. Server typecheck
passes.
- All required CI, repository build/typecheck/tests, and fresh Greptile
5/5 pass at the final head 232d3c52d0,
with no unresolved findings.
- Both actual-model necessary-question development cells passed on the
earlier recorded c4e409d819 source. This is supporting evidence for that
revision, not current qualification.
- Both fresh Stop development cells at 6036cee2f4 stopped, remained
idle, resumed once, reached Done, and settled original and successor
accounting. Their original machine grades remain failed: the observer
counted the generated file-preparation carrier as another final reply.
The corrected observer independently correlates the applied preparation
receipt with the actual attachment/comment binding. Retained-wire
diagnostic replay finds one final delivery for each model without
changing either grade. Those original failures remain retained:
https://github.com/paperclipai/paperclip/actions/runs/39644220458.
- Both fresh actual-model Stop/resume cells pass all five checks at
232d3c52d0:
https://github.com/paperclipai/paperclip/actions/runs/39647010372.
Independent inspection confirms exactly one cancelled original and one
committed successor on the same task, 5.091/5.513-second quiet intervals
before the user resumes, retained workspace/instructions/tool
identities, and one receipt-bound final response. Each visible JSON chip
opens in a tab and its Download control yields verified bytes. The
rendered reference and changed label independently reconstruct the
registered file hash and the original context hash. Both cleanups pass.
These remain local API-key/per-turn development results, not the
required staging/Slack qualification.
- Other retained Stop failures include missed active boundaries, the
real accounting admission failure, and a preparation observer that
required nativeIssueId before runtime selection. They remain recorded at
https://github.com/paperclipai/paperclip/actions/runs/39636796645 and
https://github.com/paperclipai/paperclip/actions/runs/39642060165. The
preparation observer now validates all present task IDs and still
requires selected native execution at the active checkpoint.
- The remote finalization attempt failed during image export with no
disk space before a provider started. Remote crash acceptance remains
unverified.
- Negative calibration rejects wrong or absent ownership/evidence, stale
settings, unsafe files, repeated admissions, unacknowledged Stop,
premature resumption, incomplete receipts, lost context, genuine
duplicate final replies, and missing delivery. A prepared-looking body
cannot suppress a comment; absent, rejected, foreign, or mismatched file
receipts and attachment bindings also cannot suppress it.
- Core staging/Slack drivers, 72 fresh candidate passes, deployed
staging proof, and recovery of the original user-visible task remain
incomplete. No measured reliability percentage or completed
qualification is claimed.

## Risks

These are explicit development cells. Their API-key and per-turn runs do
not qualify deployed subscription authentication, warm reuse, or Slack.
A fast provider can miss the Stop boundary; that attempt fails. The
remote crash boundary depends on independently observed public state and
an owned controller. The checkpoint is opt-in through a private manifest
and cannot replace the missing live observers. The two production
changes affect native Stop settlement and saved-input promotion. They
use the existing transaction signals and admission path. A complete
receipt can settle without another provider turn. Partial accounting and
all ownership, cleanup, pause, and budget gates remain authoritative.

## Model Used

OpenAI Codex, GPT-6 family, with repository tools and code execution.
The runtime does not expose its exact deployment ID or context-window
size. Those values are not inferred.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Paperclip <paperclip@paperclip.ing>
2026-10-21 13:02:27 -05:00
Nicky LeachandPaperclip 42715d459f Support authenticated WebSocket routes and agent views in plugins (#17111)
## Thinking Path

> - Paperclip manages agents and their work.
> - Plugins provide optional views and connect to external runtimes.
> - A live runtime view needs a binary browser connection.
> - The current plugin API supports JSON routes but cannot proxy that
connection.
> - Agent detail tabs are declared in the plugin contract but are not
mounted on agent pages.
> - This PR completes those two host capabilities and keeps upstream
credentials on the server.

## Linked Issues or Issue Description

**Subsystem affected**

Plugin scoped API routes and agent detail views.

**Problem or motivation**

Plugins cannot expose a binary or text stream through a browser session
without exposing an upstream credential or adding a separate browser
gateway. Agent-scoped detail contributions also have no mount on the
agent page.

**Proposed solution**

Allow declared board GET routes to select a server-only WSS upstream.
Run upgrades through the existing HTTP authentication pipeline and
trusted Origin check. Add agent company resolution with the existing
`agent:read` decision. Mount contributed detail views in agent
navigation.

**Alternatives considered**

A separately hosted viewer needs its own browser credentials and
cross-origin policy. JSON or SSE transport adds encoding and buffering
to a binary protocol.

**Roadmap alignment**

This extends the existing plugin API and detail-tab contract. It adds no
desktop provider to Core. I checked ROADMAP.md and searched related open
PRs; I found no equivalent transport and agent view change.

## What Changed

- Add declared WebSocket routes requiring board authentication, GET, and
`http.outbound`. Reject body-based company resolution because browser
upgrades cannot send a body.
- Keep worker-selected upstream URLs and headers out of browser JSON
responses. Strip browser cookies and Origin. Preserve binary frames and
socket backpressure.
- Close both sockets on disconnect or worker shutdown. Reauthorize on
reconnect after a five-minute connection lifetime. Preserve HTTP
idle-work accounting for the stream lifetime.
- Resolve agent scope through `agent:read` and mount generic agent
plugin detail views, with current agent/view breadcrumbs and browser
titles.

## Verification

- Final head `9d8d269040acf9dc7bb65524876406af522dacb5`: CI passes
(including Runner Rust, build, and browser E2E); Greptile 5/5; no
unresolved review threads; no merge conflicts. Verified 2026-10-21.
- Core server TypeScript check: `node node_modules/typescript/bin/tsc -p
server/tsconfig.json --noEmit`.
- Focused server suites: 17 passing tests
(`server/src/__tests__/plugin-scoped-api-routes.test.ts` and
`server/src/http/server.test.ts`), including arbitrary build-output and
metrics channels with real local text and binary WebSocket round trips,
disconnect cleanup, denied actors and origins, credential exclusion,
agent-read denial, and cross-company 404 equivalence. The test replaces
TLS with local HTTP while retaining the real upgrade and binary stream.
- UI suites: 9 passing tests for independent Insights and Notes
contributions, removed contributions, built-in route preservation,
sidebar links, direct plugin entry and heading updates after tab
navigation. These contributions do not require the WebSocket transport.
- Shared validator suite: 42 passing tests, including rejecting
body-based WebSocket resolution while retaining ordinary HTTP body
resolution and accepting query/entity scopes.
- UI token gates pass.
- UI TypeScript check: `node ../scripts/storybook-catalog.mjs` then `tsc
-b`.
- Shared and plugin SDK builds pass.
- Full Runner build reached its unrelated Rust step and stopped because
cargo is unavailable. Runner TypeScript build was run separately. No
live remote VM was used.

## Risks

- This adds a browser ingress path for trusted installed plugins. The
plugin must select its upstream from a trusted binding. HTTP requests
cannot retrieve the upstream descriptor.
- Membership or binding changes are rechecked on reconnect. Connections
last at most five minutes, and worker shutdown is checked every five
seconds.
- Initial CI caught a missing cross-company 404 guard and an old test
that assumed exactly one upgrade listener. Both were corrected.
Greptile's body-resolution and stale-heading findings are fixed with
regression coverage; latest-head CI passes and Greptile scored 5/5 with
no unresolved findings.
- No schema migration. Rollback restores the previous Core image;
plugins using this transport must be disabled or rolled back first.

## Model Used

OpenAI Codex, based on GPT-6, with reasoning and code/tool execution.
The exact runtime model ID and context-window size are not exposed in
this session.

## Checklist

- [x] Thinking path and issue description included
- [x] Model described
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] No internal issues, private links, or runtime credentials included
- [x] Descriptive branch name
- [x] Focused tests and relevant typechecks run
- [x] Tests and documentation updated
- [x] Risks documented
- [x] All CI gates green
- [x] Greptile 5/5 with no unresolved findings
- [x] I will address review findings before requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-21 10:10:00 -07:00