mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-31 14:16:39 +01:00
d6dfbb495db44b60faa37416675e8ffbb9cc10d0
282
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Devin FoleyandPaperclip
|
1a79522a4e |
perf: narrow run-to-issue authorization reads (#17405)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Run pages show links to issues associated with a run. > - The issue-link route checks the run's company and issue visibility before it reads those links. > - That check needs four binding fields, but the route reads the full run. > - Full runs can include large prompt, output, and result fields. > - This pull request uses the existing narrow authorization read. > - The route transfers less database data and keeps its permission checks and response contract. ## Linked Issues or Issue Description **Problem or motivation** `GET /api/heartbeat-runs/:runId/issues` calls `heartbeat.getRun` for a permission check. Its consumers only need the run ID, company ID, scope kind, and issue ID. The full read also fetches context, output excerpts, result data, and runtime metadata. Repeated link reads pay for that unused data. **Proposed solution** Use `getRunReadAccess`, the existing four-field read from #17348. Keep authentication, company access, company-scope authorization, run issue visibility, and linked-issue filtering in their current owners. Keep the existing empty response for missing and inaccessible runs. **Alternatives considered** A new query contract is unnecessary because the existing read provides all required fields. A permission cache could become stale. Changing the full run reader could affect consumers that need run details. **Roadmap alignment** This is a focused improvement to an existing route. It adds no feature, route, provider, cache, or background work. Searches for run-issue links, narrow run authorization, and `getRunReadAccess` found no open duplicate. Related work: #17348. ## What Changed - Change the issue-link route to use the existing narrow run read. - Reject full run reads in the route tests. Cover readable company and issue scopes, denied issue access, missing bindings, anonymous requests, and the missing/cross-company empty response. - Add a developer note about the read contract. ## Verification - `pnpm --filter @paperclipai/server exec vitest run src/__tests__/activity-routes.test.ts src/services/heartbeat/run-state.test.ts src/__tests__/agent-live-run-routes.test.ts`: 126 tests passed. - `pnpm build`: passed. - `pnpm -r typecheck`: passed. - The route tests use the real run visibility helper with only the four binding fields. The full reader throws if a route calls it. The existing run-state suite checks the exact four-column selection and covers database wiring with disposable Postgres. - `git diff --check`: passed. Reviewed the committed diff for secrets and private identifiers before push. - Apex review: 5/5 on exact head `888a4ce9ea88bb2b284eb5e28fafc01b6f48b386`, completed successfully, with zero review threads. - Full current-head CI passed: 54 successful checks and two expected skips. [Workflow attempt 2](https://github.com/paperclipai/paperclip/actions/runs/40051399577/attempts/2). [Initial serialized shard failure](https://github.com/paperclipai/paperclip/actions/runs/40051399577/job/120744710081): the first company-history case in unchanged `agent-live-run-routes.test.ts` hit its 15-second timeout; the shard had 391 passing tests and one failure. That file, its agent route, the heartbeat state reader, and the service export are unchanged from base. The five company-history cases pass on unchanged base `9fb28d06ffebbb615f413765078aeeb7c4291311` (9.12s suite) and this head (8.29s suite). They also passed in the 126-test focused run. The timeout was not reproduced locally; its cause is not established. Only the failed shard was retried on the same head. It passed in attempt 2. This establishes recovery; the initial timeout remains in this record and its cause is unproven. - The prior local full-suite failures and baseline triage remain documented in #17384. No new local full-suite pass is claimed here. This change uses focused local tests and the full current-head CI gate. - No hosted user journey, staging deployment, production deployment, or billed savings is claimed for this follow-up. ## Risks - Run visibility depends on the canonical `scopeKind` and `issueId`. The existing helper returns those same fields. It does not use prompt or result data for access decisions. - Linked issues still pass the existing SQL visibility filter. Missing and cross-company runs still return `200 []`. Anonymous requests still fail before the run read. - The database query count stays the same. The change reduces data selected for this authorization read. - No schema or API response changes. ## Model Used OpenAI Codex, GPT-6 family. The exact deployment model ID and context window are not exposed in this session. Used reasoning, repository tools, and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
70de7d49e9 |
Migrate standalone nested clones onto the shared project clone (#17407)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Each task runs in a git worktree. Extra project repositories sit in
the task under `.paperclip-repositories/`.
> - #17382 made new nested checkouts git worktrees of one shared clone
per project repository. Existing standalone nested clones stayed as full
clones.
> - Long-lived tasks keep those clones for a long time. Many tasks hold
the same large repository, so most of the disk is duplicate `.git` data.
> - #17369 added a locked and verified move of isolated workspaces onto
the shared clone, with a reaper sweep behind a setting.
> - This pull request adds the same kind of locked and verified
conversion for standalone nested clones in open tasks, and runs it from
the same reaper path behind a new setting.
> - The benefit is that the duplicate clones go away without losing any
state or local history, and the disk stops filling.
## Linked Issues or Issue Description
Refs #17382 #17369. Bug report described in-PR (bug report template).
**What happened?**
On one instance, 44 open task worktrees in one project each hold a
standalone nested clone of the same large repository. Together they use
about 25 GB. About 16 GB of that is duplicate `.git` data. Another
project has 33 smaller clones. The clones are removed only when their
tasks expire, and many tasks stay open for a long time. The disk keeps
filling.
**Expected behavior**
Existing standalone nested clones in open tasks become worktrees of the
project's shared clone, with the same state and local history, and the
standalone clones are deleted.
**Steps to reproduce**
1. Prepare a task with a nested project repository before #17382, or
without a shared clone, so that `.paperclip-repositories/<name>` is a
standalone clone.
2. Create the project's shared clone, as #17382 does for new tasks.
3. Observe that the old checkout stays a full clone until the task
expires.
**Paperclip version or commit:** `master` at
|
||
|
Devin FoleyandPaperclip
|
0c3b5ce273 |
perf: filter sandbox provider lookups in Postgres (#17395)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Sandbox execution selects a provider from installed plugin manifests. > - Driver-key lookup reads every plugin row before it finds a matching declaration. > - Repeated lookups transfer manifests for unrelated plugins. > - The registry owns this database read and can filter candidates before returning rows. > - This pull request adds that filter and keeps the existing provider selection checks. > - The returned data now grows with matching providers instead of the full catalog. ## Linked Issues or Issue Description **Problem or motivation** `resolvePluginSandboxProviderDriverByKey` calls the unfiltered registry catalog on each lookup. The caller only needs plugins with a sandbox-provider declaration for one driver key. Unrelated plugin manifests still cross the database connection and are decoded in the server. **Proposed solution** Add an optional sandbox-provider driver-key filter to `pluginRegistryService().list`. Match the key and kind on the same JSON declaration in Postgres. Pass the key from the existing resolver. Keep full matching rows, install order, status checks and worker checks. **Alternatives considered** A catalog cache would need an invalidation contract for plugin updates and independent writers. This change keeps fresh database reads. Returning only one SQL row could skip a later eligible provider when an earlier worker is unavailable. The filter therefore keeps all matching candidates. **Roadmap alignment** This is a small performance fix to existing plugin execution. It adds no product feature or provider. Searches for sandbox/driver lookup and plugin catalog/filter work found no duplicate PR or issue. ## What Changed - Add an optional JSONB containment filter to the existing registry list read. - Use it in sandbox-provider lookup by driver key. The unfiltered catalog call keeps its existing behavior. - Add real-Postgres regression coverage and a short developer note. ## Verification - Four focused provider-selection/recovery/seam suites passed: 27 tests. - `pnpm build`: passed. - `pnpm -r typecheck`: passed. - The new Postgres tests seed large unrelated manifests and verify the actual resolver SQL filters rows in the database. They cover key/kind matching within one declaration, unfiltered reads, install order, disabled and unavailable workers, live status/declaration changes, unknown keys and bound unusual input. - The filter keeps the existing SQL call count. It does not add a network call, cache, timer, or background worker. - `git diff --check`: passed. Reviewed the staged diff for secrets and private account data before push. - Full PR CI passed on `6f97c984a931416b6347e9885d840557c54d8aab`: 54 successful checks and two expected skips. [Workflow evidence](https://github.com/paperclipai/paperclip/actions/runs/40041294925). - Apex review: 5/5 on that exact head, completed cleanly, with no comments or unresolved threads. Merge state is clean. - The prior local full run on parent work is documented in #17384. It did not pass because of test setup, host Git-default assumptions and transient failures. Those results were preserved and triaged against unchanged code. This change uses focused local checks and full current-head CI; no new local full-suite pass is claimed. - No hosted user journey, live provider throughput, production deployment or billed savings is claimed. ## Risks - JSONB containment checks the stored manifest before returning rows. It still scans candidate metadata in the database; no index or SQL-call reduction is claimed. - Both key and kind must match the same declaration. The regression fixture includes a plugin with those values on different declarations. - Selection still uses the existing install-order and live readiness/worker checks. Matching disabled records remain visible to callers that do not require a running worker. - No schema, API response, provider protocol, file-copy behavior or scheduling changes. ## Model Used OpenAI Codex, GPT-6 family. The exact deployment model ID and context window are not exposed in this session. Used reasoning, repository tools and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
8549ecaf9a |
Re-home isolated repository workspaces onto the shared project clone (#17369)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Each isolated task workspace is a Git worktree on the server disk,
and the terminal workspace reaper and git cache expiration keep that
disk usage bounded
> - For some days, workspace preparation gave each isolated
`git_worktree` workspace its own full clone under
`isolated-repository-sources/`, with the worktree inside it. #16963
removed that creation path, but the existing clones stay, and restoring
an expired workspace still recreates its private clone
> - On one instance, 30 such clones use about 23 GB: 12.1 GB of `.git`
data, 5.2 GB of unused default-branch checkouts and 5.75 GB of real
worktrees. 28 of them back open workspaces, and the disk is nearly full
> - This pull request moves each such open workspace onto the project's
shared managed clone with its exact state, deletes the private clone,
and makes restore use the shared clone
> - The benefit is that disk use drops to one shared clone per project
repository, with no lost work and no change for workspaces that cannot
move safely
## Linked Issues or Issue Description
Refs #16963. Bug report described in-PR (bug report template).
**What happened?**
On one instance, 30 per-issue clones under
`isolated-repository-sources/<companyId>/<issueId>/checkout` use about
23 GB. 28 of them back open workspaces. No process moves these
workspaces to the shared project clone that new workspaces use, and git
cache expiration does not remove a clone while its workspace is open.
Reopening an expired workspace whose recipe has `sourceKind:
"isolated_remote"` clones the full repository again. Separately, a
restore point with a non-default file mode cannot be restored: the
reference read back from jsonb has reordered `modes` keys, so
`loadGitRestorePoint` fails with `restore_point_manifest_mismatch`.
**Expected behavior**
An open workspace with a private clone moves to the shared project clone
with the same branch, HEAD, index, worktree, untracked and ignored
files, and file modes. The private clone is then deleted. Reopening an
expired workspace uses the shared clone when one exists. A restore point
with non-default file modes restores.
**Steps to reproduce**
1. Create an isolated `git_worktree` workspace with
`materializeIsolatedRemoteWorkspaceSource` and record
`isolatedRemoteSource` metadata, as preparation did before #16963.
2. Keep its issue open. The clone stays, because no sweep moves or
removes it.
3. Expire the workspace with a restore point that has a file with mode
`0600`, then reopen it. The private clone is created again, and the
restore fails with `restore_point_manifest_mismatch`.
**Paperclip version or commit:** `master` at
|
||
|
Nicky Leach
|
100477e887 | Avoid live workspace inspection in task context reads (#17396) | ||
|
Devin FoleyandPaperclip
|
ec2e0d6407 |
Share one clone across nested project repository checkouts (#17382)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A project can attach extra repositories, and every task worktree
gets a managed checkout of each one under
`.paperclip-repositories/<name>`
> - Each of these nested checkouts is a full clone. The clone uses
`--dissociate`, so no objects are shared between tasks
> - On one instance, 44 task worktrees held nested checkouts of the same
repository: 24 GB of 27 GB of worktrees. Each new task added about 0.5
GB, and the disk filled twice in one day
> - Main task worktrees already avoid this, because they are git
worktrees of one managed project clone
> - This pull request makes nested checkouts git worktrees of one shared
clone per project and repository, and adapts expiration, restore points,
restore and cleanup to them without weaker safety checks
> - The benefit is that repository objects are stored once, and a new
task adds only its working files
## Linked Issues or Issue Description
No public issue exists. This is the in-PR bug report.
**What happened?**
Every task in a project with extra repositories got a full, independent
clone of each extra repository under `.paperclip-repositories/`. On one
instance this used 24 GB of 27 GB of task worktrees (16 GB of it `.git`
data) and filled the disk twice in one day.
**Expected behavior**
Nested checkouts of the same repository share one object store, like
main task worktrees share the managed project clone. Disk use per new
task is only its working files.
**Steps to reproduce**
1. Create a project with a primary repository and a second repository
that has a repository URL and no local folder.
2. Run several tasks in isolated git worktrees in that project.
3. Look at `<task worktree>/.paperclip-repositories/<name>/.git`: each
one is a full directory with its own copy of every object.
**Paperclip version or commit:** `master` at
|
||
|
Devin FoleyandPaperclip
|
4660b22f6b |
perf: narrow run bindings during event ingestion (#17378)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agent runs append native events and retry-exhaustion receipts. > - These writes lock the run row and verify its company and agent bindings. > - The binding check fetched the full run, including prompts, configuration, and output. > - This pull request reads only the two fields that the check uses. > - The benefit is less database transfer while the same row lock protects sequence allocation and replay checks. ## Linked Issues or Issue Description Related: #17355 and #17348 reduce other run-read payloads. GitHub searches for event-read and narrow-run PRs found no duplicate of this event-ingestion change. **What existing behavior does this improve?** The run binding read during native event and retry-exhaustion ingestion. **Current behavior** The transaction uses a full-row `SELECT ... FOR UPDATE` to check only `companyId` and `agentId`. Large run context, runner configuration, and output cross the database connection for each such check. **Proposed behavior** Select only the company and agent IDs. Keep the same run ID predicate, row lock, transaction, sequence allocator, replay cursor, and binding checks. **Reason and benefit** The check does not use the other fields. Narrowing the query removes that repeated transfer without a cache or an authorization change. **Breaking changes** None. No API response, schema, event, or scheduling changes. ## What Changed - Narrow the existing run-row read inside `appendHeartbeatRunEvent`. The event service continues to own its transaction and binding validation. - Exercise native and retry-exhaustion writes and replays with large stored payloads. Record the executed SQL to prove the lock reads only the two binding fields. - Extend missing-run and company/agent mismatch coverage to both transaction paths. Keep the existing concurrent-writer and rollback tests. - Document the read boundary. ## Verification - Focused PostgreSQL event append and sequencing suites: 9 tests passed. They include concurrent writers, replay conflicts, rollback, cursor checks, large payloads, and company/agent/missing-run denials. - The first new retry-exhaustion fixture failed its duplicate assertion because it omitted the exhaustion fields from the event payload. The existing producer contract stores those fields in the payload. The fixture now does the same; the rerun passed. No production retry or replay logic changed. - `pnpm build`: passed. - `pnpm -r typecheck`: passed. - Greptile Apex: 5/5 on `76a6d3ee813ae47919b1ec8e8b88e01e58d5655d`, clean check and no review threads. The [full PR workflow](https://github.com/paperclipai/paperclip/actions/runs/40028758665) passed on that exact head: 54 checks passed and two conditional checks were skipped. The branch is conflict-free. - The full local all-in-one suite is not repeated for this follow-up. Full exact-head CI will supply the broader suite coverage. - No hosted deployment or bill reduction is claimed for this follow-up. ## Risks - Low scope: only the selected columns change. The `FOR UPDATE` lock, transaction boundaries, sequence reservation, duplicate checks, and failure handling remain in place. - Native replay and legacy event writers must still serialize on the same row. Existing real PostgreSQL concurrency tests cover that boundary. - No migration, provider change, new timer, or operator configuration is required. ## Model Used OpenAI Codex (GPT-6 family), with code execution and repository tools. The runtime does not expose the exact model ID or context window. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (focused suites above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
dfe7df0015 |
fix(native): expire settled native workspace sync folders (#17358)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs sync their workspace with a remote sandbox through a
per-run folder under `<instance>/native-workspace-sync/<run>/`
> - The folder holds a descriptor, a manifest, and workspace and git
seed archives, and a seed can be more than a gigabyte
> - Only a few finalization commit paths delete the folder, so many runs
leave it on disk forever
> - Instance disks fill up with folders that no code will read again
> - This pull request adds a bounded, deadline-driven retention worker
that removes a folder only when no reader can still need it
> - The benefit is that instance disks stop growing with stale sync
folders, and recovery and export paths keep their files
## Linked Issues or Issue Description
**What happened?**
`cleanupNativeWorkspaceSync(runId)` is the only code that deletes
`native-workspace-sync/<run>/`. It runs only when finalization commits
through specific paths. Runs that end in `terminal_failure`, retained
recovery, reconstruction, superseded runs, commits through other paths,
and server crashes leave the folder forever. On one instance, 22 folders
older than 7 days held 6.8 GB. Single seed archives were up to 1.4 GB.
**Expected behavior**
A run's sync folder is removed after a retention window, once the run
and its finalization are settled and no recovery or export path can
still read it. The shared `generations` seed cache stays.
**Steps to reproduce**
1. Run a native agent with a remote sandbox and let the run end in
`terminal_failure` (or crash the server during finalization).
2. Wait more than 7 days.
3. Look in `<instance>/native-workspace-sync/`. The run's folder and its
seed archives are still there.
**Paperclip version or commit**
`master` at
|
||
|
Devin FoleyandPaperclip
|
ef9e17304c |
perf: narrow runtime progress database reads (#17355)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Active runs publish workspace and provider progress. > - Each update must check the current run status before it publishes. > - This check fetched the full run, including prompts and runner configuration. > - This pull request reads only the four fields that progress publication uses. > - The benefit is less database transfer while each update still checks durable state. ## Linked Issues or Issue Description Related: #17348 narrows authorization reads. This PR applies a separate projection to runtime progress. GitHub searches for runtime progress and run-status read PRs found no duplicate. **What existing behavior does this improve?** The database read before a runtime progress update. **Current behavior** Each update calls `getRun`. This reads large run context and runner configuration even though progress publication only uses the run ID, company ID, agent ID, and status. **Proposed behavior** Use a fresh four-column read for each update. Keep the existing active-status check, event fields, error propagation, and late-progress suppression. **Reason and benefit** Transfer progress can emit many updates for one run. A small status check avoids repeatedly transferring payloads that it does not use. **Breaking changes** None. No API response, schema, scheduling, or cancellation policy changes. ## What Changed - Add `getRunStatus` to the run-state owner. The full reader remains available to consumers that need execution data. - Wire the narrow reader into the lifecycle progress check. Remove the unused context field from that check's input contract. - Cover the exact selected fields, missing runs, database errors, large stored payloads, terminal transitions, and the public heartbeat service wiring. Document the read boundary. ## Verification - Run-state and lifecycle suites: 44 tests passed before the additional public-service regression. - Lifecycle, public-service wiring, and native cancellation suites: 36 tests passed. - `pnpm -r typecheck`: passed. - `pnpm build`: passed. - The [full PR workflow](https://github.com/paperclipai/paperclip/actions/runs/40013092548) passed on `95bf496f689049812646b5ccddcdf2cca21efad0`. All 54 reported checks passed; two conditional checks were skipped. This includes test shards, browser checks, Runner verification, production bundle boot, and clean npm installation. - `GIT_CONFIG_COUNT=1 GIT_CONFIG_KEY_0=init.defaultBranch GIT_CONFIG_VALUE_0=master pnpm test:run`: an additional local all-in-one run is still in progress; no full-local-pass claim. The process-scoped Git default matches existing master-only fixtures and does not change user configuration. - The initial description check failed because the dedup-search checkbox was missing. Searches are complete and the template checkbox is now included. The rerun passed on the same source commit. - Greptile Apex: 5/5 on `95bf496f689049812646b5ccddcdf2cca21efad0`, no findings or review threads. - No hosted deployment or bill reduction is claimed. CI and Apex are complete; this PR is merge-ready. ## Risks - Low scope: the query selects fewer columns and preserves the existing status read on every progress callback. It adds no worker, cache, timer, or idle query. - A run can still finish after the status read and before publication, as it could before this change. This PR does not change that concurrency boundary. - No migration or operator configuration is required. ## Model Used OpenAI Codex (GPT-6 family). The runtime does not expose the exact model ID or context window. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (focused suites above; full suite is still running) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
f406b32d08 |
perf: narrow run authorization reads (#17348)
## Thinking Path > - Paperclip lets people manage and observe AI agents at work. > - Live events and run history must enforce current task permissions. > - Some permission checks load the complete run record for each event or request. > - Run context, runner configuration, results, and output can make those records large. > - This pull request reads only the persisted fields each access check needs. > - The checks still read current permissions and fail closed. ## Linked Issues or Issue Description Related: #17251, #9155, and #16229. Those changes cover other run projections, recovery polls, and task history. This change covers live-event delivery, workspace-operation run checks, and event-list access. **What happened?** Live-event delivery loaded every run column for each subscribed viewer, although the permission helper used only the company, run scope, and task binding. Workspace-operation checks did the same. The run-event endpoint also loaded complete run records before returning separate event rows. Large contexts and runner configurations therefore increased database transfer for ordinary access checks. **Expected behavior** Read only the stored access binding. Keep company checks, permission freshness, deleted-task tombstones, and response bodies unchanged. **Steps to reproduce** 1. Create an issue-bound run with large context, runner configuration, and output fields. 2. Subscribe an authorized viewer to company live events and publish run updates. 3. Read the run events and a linked workspace operation. 4. Inspect the run SELECT projections. The previous queries load fields that the access checks do not use. **Paperclip version or commit** The defect is present on the current master base, `62c78e56e53`. **Deployment mode** Authenticated server with PostgreSQL. The defect is not adapter-specific. ## What Changed - Select only company, scope, and issue bindings for live-event and workspace-operation run authorization. - Add a four-field run-access read for event-list requests. Reuse those columns in the existing log-access projection. - Preserve per-delivery and per-request permission checks, company predicates, and event responses. - Add real PostgreSQL and WebSocket coverage with large run fixtures. Cover grant revocation, grant restoration, deleted tasks, missing and foreign-company runs, and failed reads. - Document the bounded access reads in the existing polling-load guide. ## Verification - Six focused suites pass: 40 tests. They cover the changed paths, run-state reads, run logs, disconnect retries, and native event replay. - The new WebSocket test uses the production delivery path and a real database. It verifies that the selected SQL omits large run fields while authorization changes take effect on the same open socket. - The event-route test preserves the returned event content and denies the next request after grant revocation. - Local recursive typecheck, full build, and module-boundary checks pass. [Exact-head CI](https://github.com/paperclipai/paperclip/actions/runs/39997370119) completed successfully, covering the full grouped unit suites, typecheck, build/runtime boot, and E2E shards. The separate serial local `pnpm test:run` is still running and is not claimed as completed. - A later CI shard failed an unchanged email test expecting exactly one task after permission grants. The unchanged-base test passes at normal speed; adding only a 2.2-second pause before granting access reproduces the extra tasks because previously denied deliveries become due for retry. No email code or assertions are changed in this PR. The original failure is retained. The single shard rerun passed: 92 suites and 1,489 tests passed, with one suite and two tests skipped by the existing gates. - CI initially found a route-test double missing the new access helper. The follow-up fixes that fixture and adds event reads to malformed-ID and missing/cross-company checks. All 122 tests across the four changed-path suites pass; a final rerun of the expanded route suite passes all 92 tests. - The first CI run also timed out in the signoff changes-requested handoff while waiting for a reviewer run after a cancelled execution blocker. The focused test passes locally on this source. The same focused test passes three times on unchanged master. Controller-lease cancellation messages appear on both versions. The original CI timeout is preserved as an intermittent, unreproduced failure; its exact cause remains unknown. The signoff shard passes in the final exact-head CI run. - Initial fresh-checkout tests could not start because workspace SDK output and native PostgreSQL library links were missing. Dependency builds and the package's native-link setup corrected those prerequisites. The subsequent focused run passed. Focused command: ```sh pnpm --filter @paperclipai/server exec vitest run src/__tests__/live-events-run-read.test.ts src/__tests__/heartbeat-run-privacy-routes.test.ts src/services/heartbeat/run-state.test.ts src/__tests__/native-observer-replay.test.ts src/__tests__/heartbeat-run-log.test.ts src/__tests__/heartbeat-run-privacy-disconnect-retry.test.ts ``` ## Risks - A missing access field could cause an incorrect allow or denial. Tests exercise current grants, revocation, company scope, and deleted-task bindings through the real consumers. - No successful permission decision is cached. Every delivery and request still checks current database state. - No schema, migration, polling interval, API response, or deployment configuration changes are included. - This reduces bytes for the affected reads. It does not establish a whole-account savings percentage. Hosted transfer measurements remain separate validation after staging deployment. ## Model Used OpenAI Codex, a GPT-6 based coding agent, used repository tools, code execution, and database-backed tests. The session does not expose the exact backend model identifier or context-window size; those values are not guessed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
5ef783d971 |
feat: expire idle open issues' worktrees with restore points (#17271)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - #17248 and #17257 let git cache expiration remove a closed issue
tree's worktree after saving an encrypted restore point
> - Issues that stay open but go untouched for weeks still keep their
worktrees forever, so disks keep filling
> - This pull request expires those idle worktrees the same way, and has
the next run restore its workspace automatically
> - The benefit is bounded disk use for stalled open work, with nothing
lost when work resumes
## Linked Issues or Issue Description
Refs #17248, #17257. Bug report described in-PR (bug report template).
**What happened?**
With restore points enabled, expiration only considers worktrees whose
whole issue tree has been `done` or `cancelled` for a while. A worktree
whose issue stays `todo`, `in_progress`, `blocked` or `in_review` with
no activity for weeks is never removed.
**Expected behavior**
An open issue's worktree that has been idle long enough is saved to a
restore point and removed, and the next run on that issue gets it back
exactly.
**Steps to reproduce**
1. Enable git cache expiration and restore points.
2. Leave an issue open with an isolated worktree and no activity for
weeks.
3. The worktree is never removed.
**Paperclip version or commit:** `master` at
|
||
|
Devin FoleyandPaperclip
|
c5ca29dbef |
fix(claude): prepare compatible sandbox runtimes during onboarding (#17259)
## Thinking Path > - Paperclip connects agent accounts and prepares their execution environments. > - A managed sandbox image can outlive the model defaults shipped by the app. > - Claude setup skipped installation whenever a CLI was already on PATH. > - The selected model then failed its minimum-version check after sign-in succeeded. > - This pull request prepares a compatible private CLI before testing or running it. > - Cloud onboarding also stops asking customers to run inaccessible shell commands. ## Linked Issues or Issue Description **What happened?** A new managed Cloud stack completed Claude subscription sign-in. Its environment test then failed because the snapshot contained Claude Code 2.1.263, while the selected `claude-opus-5-5` model requires 2.1.280. The screen displayed Manual debug commands that a hosted customer cannot run. **Expected behavior** Paperclip prepares its managed runtime before testing the selected account. A stale image CLI does not require the customer to upgrade a server package. Failed preparation still blocks hiring. Cloud does not display manual shell debugging. **Steps to reproduce** 1. Start a managed sandbox with Claude Code 2.1.263 on PATH. 2. Connect a Claude subscription and select `claude-opus-5-5` on the CLI lane. 3. Connect the agent. Installation is skipped because the command exists, then the version gate blocks the hello test. **Paperclip version or commit** Source baseline: `6204239cfc8c3be79fc185cf3f468ebaa0ab1e3e`. **Deployment mode** Authenticated managed Cloud with a Daytona named snapshot. The correction also applies to the default CLI in other managed sandbox transports. Related work: Refs #14993 sets model/runtime requirements; Refs #15993 changes native defaults on qualified targets. Neither prepares an outdated default sandbox CLI on this path. Searches found no open duplicate. ## What Changed - Check the actual default CLI against the selected model's existing minimum version. - When a sandbox's default command is outdated, install the exact required npm package into a private versioned workspace prefix. Reuse a compatible private installation. Keep the image's global CLI intact. - Verify the private executable and use that same command for the account hello test and normal CLI runs, with matching command metadata. Probe effort support after preparing that executable so the chosen effort is retained. Keep the model, account, engine and permissions unchanged. - Return safe preparation checks to environment tests and normal run logs, distinguishing installation failure causes without returning raw installer output or credential-bearing URLs. - Exclude provider and Paperclip credentials from the installer environment. Preserve the existing failing version gate if installation or verification cannot succeed. Local, SSH and explicit custom commands remain operator-managed. - Hide Manual debug and legacy operator check hints on Cloud and while deployment metadata is unknown; preserve check messages and details. Retain self-hosted help and the detailed failed checks. Cloud copy asks for a retry instead of asking the customer to repair the environment. - Add subprocess-backed runtime controls, an account-probe regression and mounted Cloud/self-hosted wizard controls. Update development guidance and the affected capability maps. ## Verification - Passed all Claude source suites and the server Claude execution suite on head `c941ae44c22d8dafec0df033a129e39ce4ebd966`: **412 passed**, one existing opt-in test skipped. These include the new command/effort ordering, safe installer diagnostics, environment forwarding and run-log regressions. - On final head `0ecae0e9304f495a57524158b3700b8e9bc21cb8`, passed **208 tests** across all 174 mounted onboarding wizard tests and 34 Claude execution tests. The failure fixture includes the real command-bearing hint, and execution asserts both metadata command fields agree. - Passed `pnpm -r typecheck`, `pnpm build`, `pnpm check:token-gates`, `node scripts/check-maps.mjs` and `git diff --check`. Full typecheck and build were repeated successfully on final head `0ecae0e9304f495a57524158b3700b8e9bc21cb8`. [Latest-head CI](https://github.com/paperclipai/paperclip/actions/runs/39879208694) is green: 54 checks passed and two skipped, including browser E2E and serialized database suites. - The new runtime controls execute real subprocesses with a controlled installer. They cover an old CLI, an already compatible CLI, failed or ineffective installation, local/custom commands, and an unknown version. The probe regression asserts that the hello call uses the private executable. - A separate non-root local subprocess check installed the real npm `@anthropic-ai/claude-code@2.1.280` package. Its private command reported 2.1.280, the simulated image CLI remained 2.1.263, and a second preparation reused the private copy. This ran on macOS ARM64. It did not use a provider account, send a model prompt, or prove live Daytona behavior. - Attempted `pnpm test:run`. Embedded PostgreSQL initialization failed after five attempts on this Mac with `Postgres init script exited with code null`. The unchanged native-session database cases then failed. Stopped only that full-suite process tree after recording the repeated startup failure. No full local-suite pass is claimed; Linux CI must cover those tests. - The first new subprocess fixture had a quoting syntax error. It was corrected before the passing focused runs. The one-off real-package script first required an ESM extension correction, then passed. Logs retain both attempts. A new execution assertion initially inspected the final sandbox cleanup shell instead of the launched command; it now checks execution metadata. A broader directory selection also picked up generated `dist` tests whose Markdown fixtures are not copied by the build (two ENOENT failures). The explicit source-suite run above passes; no generated-test pass is claimed. - Apex review: **5/5**, reviewed final head `0ecae0e9304f495a57524158b3700b8e9bc21cb8`, all five earlier findings addressed and resolved, no new findings. - Merge, deployment and a fresh live Cloud subscription test are still required. No customer account or production runtime was changed by this branch. ## Risks - An outdated sandbox needs npm registry access and up to 120 seconds for one bounded installation attempt. A failed or unverified install continues to block execution. - The versioned package uses sandbox workspace storage. Compatible cached copies avoid repeat installation within that workspace. Existing runtime cleanup and sandbox lifecycle still apply. - The compatible version comes from the adapter's existing model-version contract. This does not add a model fallback or change the selected execution engine. - No schema, credential, permission, workflow or lockfile change. ## Model Used OpenAI Codex, based on GPT-6, assisted with implementation and verification. The exact serving model ID and context-window limit are not exposed in this session. Capabilities used: code editing, shell execution, API inspection and test execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
6d44b80a5e |
feat: expire closed git worktrees with unsaved work behind restore points (#17248)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Git cache expiration removes a closed issue's worktree only when the
remote can reproduce it exactly
> - Most closed worktrees hold unpushed commits or uncommitted files, so
they are kept forever and disks fill up
> - #17238 added restore points (an encrypted save of exactly what the
remote cannot reproduce) and #17241 added their lifecycle
> - This pull request lets expiration remove those worktrees after
saving a restore point, and restores them exactly when the work resumes
> - The benefit is bounded disk use for closed work without losing
anything a user could need, behind a setting that is off by default
## Linked Issues or Issue Description
Refs #17238, #17241. Bug report described in-PR (bug report template).
**What happened?**
With `PAPERCLIP_GIT_CACHE_EXPIRATION_ENABLED=true` and dry run off, one
instance's last 111 expiration checks removed nothing:
| Skip reason | Checks |
|---|---|
| `dirty_or_untracked_files` | 39 |
| `unpushed_managed_checkout` | 32 |
| `live_or_unresolved_consumer` | 21 |
| `unique_ignored_files` | 9 |
| commit not on the remote | 10 |
Without expiration, the terminal workspace reaper archives only merged,
clean worktrees. Closed issues' worktrees therefore accumulate without
bound.
**Expected behavior**
A closed worktree whose only obstacle is unsaved work can be removed
after that work is saved, and restored exactly if the issue is reopened
or work resumes.
**Steps to reproduce**
1. Close an issue whose worktree has an unpushed commit or an
uncommitted file.
2. Enable git cache expiration with dry run off.
3. The worktree is never removed.
**Paperclip version or commit:** `master` at
|
||
|
Dotta
|
46760cc588 |
fix(chat): honor dismissal for all composer requests (#17218)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Task conversations show requests in the feed and above the message composer. > - Earlier fixes kept dismissed questions in history, but excluded completion reviews and other request kinds. > - Closing those forms left a pending badge, and the server could still demand a response. > - This pull request saves human dismissal and removes the request from composer reminders and ordinary completion gates. > - People can continue the conversation and reopen the request from history. ## Linked Issues or Issue Description Related work: #15229 and #15392. **What happened?** Dismissing a completion review closed its form but left it counted as pending. It could return after reload, block a retry, or cause the agent to demand another response. Raw runtime requests could also lose their only usable control when dismissed. **Expected behavior** Dismissal leaves an accessible request in the feed and clears the composer form and pending count. It persists after reload. It does not submit an answer, approve an action, or wake the agent. **Steps to reproduce** 1. Open a task with a pending completion review or another composer request. 2. Dismiss the request, then reload the task. 3. Check that the composer stays clear and the feed entry can reopen the request. ## What Changed - Add nullable dismissal fields and a human-only dismissal endpoint with company, private-task access, resolver, and locked review-policy checks. - Persist saved-request dismissal on the server. Keep browser persistence for raw runtime requests and existing question dismissals. - Keep confirmations, choices, item verdicts, suggested tasks, and runtime requests accessible in the feed. - Exclude dismissed ordinary requests from completion and retry gates. Tell agents not to repeat deferred requests. - Preserve tool, secret, connection, plan, company-policy, and configured review authorization gates. Expire deferred non-question requests when their task completes. - Preserve the selected form when another dismissal finishes saving. Keep reopened reviews visible through delayed comment refreshes. - Add a completion-review Storybook test drive and update design, API, database, and capability documentation. ## Verification - `pnpm -r typecheck`: passed. The final UI changes also passed the UI typecheck. - `pnpm build`: passed. The final UI changes also passed the UI production build. - Focused UI and transcript tests: 550 passed across nine files after merging the current base branch. - Focused database, completion, and prompt tests: 105 passed across three files, including two review-policy regressions that failed before the fix. The tests reject a review requester excluded by current policy, preserve the pending gate on denial, and permit an independent reviewer to defer the request. - Route regression suite: 81 passed, including denial before reading or mutating a private-task request. - `pnpm check:token-gates`: passed. - Map and Storybook structural checks: passed; 30 map/catalog script tests passed. - Browser: used the production task-thread components in the completion-review fixture. Dismissed, reloaded, reopened from the feed, and dismissed again. No pending badge or composer takeover remained. Story callbacks simulate API saves; separate server tests cover persistence and permissions. - `pnpm test:run`: stopped after the general-server group reported one failure in `workspace-runtime.test.ts`: "returns a bounded structured conflict and exposes only same-company workspace references". The same test passed in isolation (1 passed, 178 skipped). The cause of the full-suite failure is not established. The full PR CI suite subsequently passed on e243cbb001; its two authorization review findings are fixed in |
||
|
Devin FoleyandPaperclip
|
12bb7e1976 |
feat: track git workspace restore points and apply their retention (#17241)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Closed issues' git worktrees pile up on disk because they can only
be removed once the remote reproduces them exactly
> - #17238 adds restore points: encrypted saves of the unpushed and
uncommitted parts of a worktree, so the worktree itself can be removed
> - Those saves live in file storage and must be deleted on a schedule,
and promptly when their issue or company is deleted
> - This pull request adds the table that tracks restore points and a
deadline-driven retention worker, with nothing creating restore points
yet
> - The benefit is a reviewable lifecycle foundation before capture is
wired into expiration
## Linked Issues or Issue Description
Refs #17238 (restore point library). Bug report described in-PR (bug
report template).
**What happened?**
Closed issues' worktrees accumulate without bound. For example, one
instance's last 111 git cache expiration checks removed nothing, because
each worktree held unpushed commits or uncommitted files. Restore points
(#17238) will let those worktrees be removed safely. That needs a
durable record of every stored restore point and a retention policy for
deleting them.
**Expected behavior**
- A restore point is kept while its work can still resume.
- It is deleted a fixed time after its issue tree closes.
- It is deleted promptly when its issue or company is deleted.
**Steps to reproduce**
Not applicable on its own: this adds the lifecycle that the capture
integration will use.
**Paperclip version or commit:** `master` at
|
||
|
DottaandPaperclip
|
734c265339 |
Default new Pi agents to Paperclip Runner (#17142)
## Thinking Path > - Paperclip manages AI agents and their work. > - Users choose a harness during agent setup. > - Claude, Codex, and OpenCode already receive their qualified Runner default. > - Pi has a qualified native integration but new Pi agents still use the direct adapter. > - This change gives new Pi agents the native default on supported targets. > - Saved agents keep their runner and compatible settings. Legacy remains available in Advanced. ## Linked Issues or Issue Description Refs #17120, #14921, #14922, #14923, #14924, and #14956. Related generic default work: #15422. This PR is the separate Pi follow-up to the merged OpenCode change. **Current behavior** Choosing Pi for a new agent saves the direct Pi adapter. The native Pi integration remains behind the generic experimental flag. **Proposed behavior** Default new Pi agents to ACPX Pi on macOS ARM64/x64 and GNU/Linux x64 with supported glibc. Keep unsupported targets and explicit Legacy selections on the direct adapter. Require an explicit provider/model before test or hire. Permit model-free inactive setup drafts. **Reason and benefit** Users choose Pi once and receive its qualified runner. Saved agents retain their identity and settings. Conversions reject unsupported legacy thinking and launch settings instead of dropping them. ## What Changed - Add Pi to the shared platform/default contract and server creation selection. - Admit native Pi without the generic experimental flag. - Preserve provider models, credentials, workspace settings, exact thinking, and saved native policies. - Add Pi runner controls, branding, setup resume, onboarding, built-in configuration, and import handling. - Add managed Pi OpenRouter routing and matching native provider API-key accounts. Keep subscription and custom gateway exclusions. Clear alternate credentials and reject custom Pi provider overrides. - Keep Pi draft models empty instead of applying Claude's default model. - Verify setup against the pinned Pi runtime. For native Pi on the official OpenRouter route, reverify the API key live without requiring a global Legacy Pi CLI. - Match Pi’s bounded controller close window to its native snapshot cleanup and lifetime verification, preserving explicit caller deadlines and the mandatory drain/suspension proof. - Bind Pi sidecars to the package that owns the pinned native assets, including source and vendored server layouts. Other providers retain their existing package authority. - Keep protocol-owned finish/block tools out of the ACPX forwarded catalog. The trusted bridge installs these tools itself; the complete Rust authorization catalog and terminal receipt ledger remain intact. This fixes actual Pi session startup and applies to other ACPX consumers. - Advance Copilot’s shared-source attestation to v58 because it pins the shared files touched by Pi. Preserve immutable v57 bytes and historical decoding; Pi’s own release identity is unchanged. - Add regression coverage and update the paired maps and developer documentation. ## Verification - Passed: full repository typecheck and full production build on final head |
||
|
DottaandPaperclip
|
39e44a9d9b |
Default new OpenCode agents to Paperclip Runner (#17120)
Default new OpenCode agents to Paperclip Runner on qualified macOS and Linux x64 targets. Preserve saved runner choices, explicit Legacy selection, and unsupported-target fallback. Pi remains a separate change. Verified the complete browser path with a real OpenRouter connection: create the new agent, assign a task, observe its response and Done status, and reload to confirm persistence. CI passes and Greptile is 5/5 with no unresolved findings. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
c6ee96a3df |
fix: stop idle MCP event polling (#17036)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - MCP task subscriptions and Dot mailboxes notify external clients when work changes. > - Their delivery workers queried SQL every two seconds, including when no work existed. > - Those queries prevent idle databases from becoming quiet across many hosted instances. > - Existing activity, mailbox, subscription, and delivery records already preserve the work. > - This pull request uses committed-write signals and saved retry deadlines to drive delivery. > - Idle subscriptions become quiet while startup, retry, drain, and shutdown recovery remain supported. ## Linked Issues or Issue Description Related sleep migrations: #15625, #17000, #17008. Searched open PRs and issues for MCP event sleep/polling work; no duplicate found. **What happened?** Each public MCP and Dot event service started a two-second interval. It checked settings, scanned subscriptions, created delivery receipts, cleaned expired records, and attempted delivery even without changes. **Expected behavior** An empty or subscribed-but-idle service produces no recurring SQL or network traffic. Committed events trigger prompt delivery. Only unfinished work retains a retry deadline. **Steps to reproduce** Configure a public MCP origin and start the server. Observe recurring settings queries with MCP disabled, or subscription/queue queries with it enabled. No incoming task changes are required. **Paperclip version or commit** Initially inspected master at `8b4ab65b1c3d863d282752e83307943720a5c85c`; rebased onto `c2daf906d9` after master advanced. The only conflict was overlapping imports in the heartbeat queue; both were preserved. ## What Changed - Register separate MCP and Dot workers with the existing delivery coordinator. Remove both unconditional intervals. - Notify from committed activity, comment, mailbox, subscription, and experimental-settings writes. Centralize comment insertion in one small helper so normal callers do not manage recovery state. - Preserve outer transaction ownership, uncertainty reconciliation, and requests deferred during drain or standby. - Restore delivery retry/abandoned-claim deadlines on startup. Scan subscriptions in pages of 20; drain full source and delivery batches without another external event. Scope subscription refreshes to their subscription or shared Dot binding, issue activity to its task, comment discovery to its company, and mailbox/assignment changes to their binding. Retry deadlines do not rescan unchanged sources. - Run expired-record cleanup opportunistically. Preserve on-use expiry checks and the existing one-time automatic Dot challenge behavior. - Add real PostgreSQL lifecycle coverage and guarded external test database options for existing MCP/Dot integrations. Document producers and the supported writer model. ## Verification - Final head `e54fb34485921016ef988f7df9cca173a9ebdfc8`: all CI checks passed (55 successful checks/statuses, one neutral review, two intentionally skipped Storybook checks). Mergeable with no unresolved review threads. Greptile: 5/5 on this exact commit; security scan passed. - All 284 affected tests passed in one local run after rebasing: 16 real-PostgreSQL lifecycle tests, 24 coordinator/scheduler/standby tests, 84 public MCP tests, 54 Dot runner tests, and 106 interaction-service tests. - Server TypeScript passed after refreshing generated Runner/plugin SDK dependencies from master. Full workspace typecheck and build passed before the rebase; final CI repeated typecheck and build successfully on the rebased head. - Coverage includes SQL/network quiet with configuration and history, committed writes, outer commit/rollback, drain/standby, retry restoration, reenable/unsubscribe, concurrent writes, shutdown, source/delivery/subscription batch boundaries, Dot challenges and reconnects, native broker fences, comment discovery, and company-isolated scans. - CI exposed an existing lock-order test whose insert proxy missed nested transactions and waited indefinitely. The fixture now observes nested inserts, preserves the lock assertion, and fails explicitly if interception is missed. The complete 106-test interaction suite and the formerly hanging CI shard pass. The warm-standby/settings test doubles were also updated for the new coordinator/transaction boundaries. - Earlier fixture failures (required timestamps, cleanup, external database selection, async waits) were corrected. One existing OAuth first-poll timing assertion and one E2E company-cleanup race passed on single reruns; the final full CI run passed without a rerun. No OAuth or company-deletion behavior was changed. - The local full `pnpm test:run` attempt could not initialize embedded PostgreSQL after five attempts. It was stopped; affected integrations were verified with separate disposable PostgreSQL 17 databases, and final CI supplied the full-suite result. Embedded-only local skips are not counted as passes. ## Risks - Notifications use one live server/database scope. Independent SQL writers and separate live replicas do not trigger these process-local workers. Startup recovers durable state; continuous multi-writer discovery needs a notification/ingress contract. - Idle physical cleanup waits for startup or another enabled sweep. Subscription and authorization expiry remain enforced on use. - Existing delivery claims, retry bounds, signatures, and receipt deduplication remain in place. No new tables or migrations. - Host sleep-safety checks remain conservative. MCP/Dot retry deadlines are not yet covered by Paperclip Cloud's scheduled-wake contract. This proves worker quiet, not deployed host suspension or fleet savings. ## Model Used OpenAI Codex (GPT-6), with reasoning, local code execution, and tool use. The runtime does not expose a more specific model version or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
30b753f614 |
perf(budgets): scope accounting and deliver enforcement asynchronously (#17046)
## Thinking Path > - Paperclip manages AI agents and their paid work. > - Budgets must prevent new work and stop reported overruns. > - Cost settlement also ran serial lifecycle reconciliation for every eligible company agent. > - That made request latency grow with unrelated idle agents and delayed results on stop delivery. > - This change separates atomic admission and accounting from durable asynchronous enforcement. > - Scoped live usage detects overruns without writing provisional ledger charges. > - The result is constant query scope for ordinary settlement and recoverable stop delivery. ## Linked Issues or Issue Description **What happened?** Settlement and budget checks entered company-wide lifecycle reconciliation. A company with 1,000 unrelated idle agents performed 1,001 accounting transactions for one ordinary cost evaluation. User requests could also wait for provider termination. Running usage checkpoints did not enforce reported monetary exposure. **Expected behavior** Keep the final reservation gate authoritative. Commit charges, scope blocks, and enforcement intent before returning. Deliver stop requests after commit. Read only applicable policies and execution scopes for ordinary charges. Stop long-running work when supported reported or priced usage establishes an overrun. **Steps to reproduce** 1. Create a disposable company with one company budget and idle agents. 2. Evaluate a charge below the budget with 0, 17, and 1,000 unrelated idle agents. 3. Compare query counts, company accounting transactions, and duration. 4. Use the controlled partial-usage and delayed-stop fixtures to check enforcement and request responsiveness. **Paperclip version or commit** The measured baseline is `6c9b87d0922f1ebfc43e8b0c36889ac5ba7b9837` from `master`. Related public work: Refs #16976, #2006, #3539, #13763, #14296, #16853, #13075. These cover the affected service charge, cancellation, and accounting paths. This change removes their shared reconciliation overhead. ## What Changed - Batch applicable policy exposure under the existing company accounting lock. Preserve exact fractional money and zero-reservation dispatch fences. - Commit a durable final budget dispatch claim before provider I/O. Retain a separate final issue/run ownership transaction through synchronous handoff, without holding the company accounting lock or awaiting provider work. Preserve native recovery lease checks and duplicate handoff fencing. Browser handoff rechecks current policy; it reuses a consumed rate slot only for the identical policy generation. - Separate ledger spend, live observed spend, and effective held capacity. Use the maximum of reservation and live observation for each unfinished operation. - Deliver budget cancellation through the existing coordinator. Revalidate generations and persist queued/running stop fences before external termination. Bound stop concurrency to four. - Apply agent holds only to the affected agent. Project budgets leave other projects running. Project real company transitions in bounded lifecycle batches. - Replace periodic broad reconciliation with durable pending work, startup repair, receipt replay, and concrete retry/window deadlines. Retain drain and idle-safety checks. - Enforce accepted cumulative usage checkpoints without provisional charges. Final accounting atomically replaces live exposure and settles reservations. - Add operator-gated budget spans, regression coverage, map updates, and a real-PostgreSQL comparison harness. No schema or first-party telemetry change. ## Verification - Full workspace `pnpm -r typecheck` passes on the final ownership fix, rebased onto `bbcf5dfe98`. Production `pnpm build` passes on final head `4c5f4de8b1`. Map, module-boundary, token, and Storybook catalog checks pass. - Focused real-PostgreSQL coverage includes the 292-case rebased accounting/dispatch run, 63 browser cases, 52 run-control/cancellation cases, 42 policy-generation cases, 77 lifecycle/commentary cases, and the exact Muse confirmation regression. Native executor unit coverage passes all 768 cases. Final ownership coverage passes: 74 adapter/use-case cases, nine public regression cases, and the additional deleted-project race. These exercise competing ownership, budget changes and charges between phases, zero/nonzero duplicate claims, rollback, recovery contention, read-only capacity rechecks, and controlled HTTP startup after commit. Another 74 pure/startup/tracing cases pass. The complete CI matrix and both 5/5 reviews passed on `1ac9b58`; its external security scan identified a stale-owner handoff race. The final fix restores synchronous issue/run ownership fencing after the accounting transaction commits. Final head `8e84f55ce8` has fresh Greptile 5/5 with zero new comments, Paperclip review 5/5, and a passing security scan. The complete latest-head [CI matrix](https://github.com/paperclipai/paperclip/actions/runs/39535189808) passes, including budget, browser, Runner, build, and canary jobs. The unchanged Runner close fixture passed on its sole same-head job retry. - Broad local `pnpm test:run` invocations were stopped before required source updates (rebase and review fixes). They are not claimed as full passes. Final full-suite coverage comes from the complete latest-head CI matrix; local focused coverage and the incomplete broad invocations are reported separately. - Reproduce performance with `pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-budget-enforcement.ts --baseline-ref=6c9b87d0922f1ebfc43e8b0c36889ac5ba7b9837`; repeat with `--reverse`. The harness uses disposable real PostgreSQL, 20 warm samples per variant/count, and records raw timings, SQL/lock counts, agent reads, unchanged agent state, and query plans. Final production-source measurement revision: `eff713a4f72c7f4b7a4002a701cb481d4bfccd95`; the extended harness is committed in `4c5f4de8b1`. | Unrelated idle agents | Baseline median / p95 | New median / p95 | Baseline SQL / locks | New SQL / locks | New agent reads | | --- | ---: | ---: | ---: | ---: | ---: | | 0 | 1.80 / 2.33 ms | 2.99 / 3.88 ms | 5 / 1 | 7 / 1 | 0 | | 17 | 15.53 / 19.55 ms | 2.24 / 2.60 ms | 90 / 18 | 7 / 1 | 0 | | 1,000 | 978.35 / 1,790.43 ms | 2.16 / 2.52 ms | 5,005 / 1,001 | 7 / 1 | 0 | - These measurements use local PostgreSQL 18.1 and Node 26.4.0 arm64. They isolate the cost-evaluation core and complete admission path, without hosted-region or provider latency. Prior-revision reversed-order controls also preserved the constant query counts; the final measurements above use the final production source. - Complete reserve → final ownership gate → immediate offline provider result is measured separately, with one responsible agent and one active operation at each unrelated-agent count. Admission p95 baseline/new: 0 agents **11.54 / 29.77 ms**; 17 agents **9.31 / 27.62 ms**; 1,000 agents **24.51 / 74.05 ms**. Baseline admission was already scoped; the stronger final fence adds bounded overhead. New counts stay constant at 53 Drizzle statements, three transactions, two company-accounting locks, and five agent-reading statements bound to the responsible agent. Zero unrelated-agent reads, exactly one provider callback per sample, and no leftover runs or holds. All admission scenarios stay below 250 ms. No external provider calls are included. - The partial-usage/delayed-termination fixture passed 20/20: receipt persistence p95 13 ms, receipt return to observed durable Stop fence p95 28 ms, complete receipt-start-to-fence p95 50 ms, unrelated database request p95 1 ms while termination was blocked. After releasing the controlled termination fixture, acknowledgement p95 was 14 ms. These are separately measured phases; fixture acknowledgement is not live-provider termination latency. - Models and provider deadlines are unchanged. No deployment, live inference, or staging acceptance is claimed. A quiet 30-minute follow-up waits for the merged revision on staging, then repeats classifier and controlled budget-stop acceptance in staging, recording hosted bookkeeping, queue delay, and actual provider acknowledgement separately. - Development failures retained: finalizer notification regressions (fixed; 27/27 affected cases passed), invalid browser admission fixture assumptions (fixed; browser coverage now passes 63/63), and first-head lifecycle/dispatch CI regressions (reproduced and repaired). A prior-head Cases E2E test hit its unchanged announcement-dismiss handler race while opening a filter; the same journey passed on `1ac9b58`. The external security review then found a stale-owner race between accounting commit and provider entry; final verification covers a competing run in that gap while retaining provider I/O outside the accounting lock. Early PostgreSQL startup attempts hit local semaphore pressure; serial reruns succeeded. The final-head Runner lane had one failure among 1,823 tests: the unchanged `timeout-attachment` fixture failed at its first process close after 6.28 seconds. The sole same-head job retry passed all 1,819 active tests (four skipped), including that fixture in 1.85 seconds. Three isolated local probes could not run correctly because the Rust fake-provider fixture binary was absent; cleanup masked the original startup failure. CI builds that fixture through its normal lane. These probes are inconclusive and are not counted as passes. ## Risks - This changes accounting, lifecycle, and cancellation ordering. Durable generations, source/sequence checks, ownership leases, and company locking remain required for correctness. - Provider stop acknowledgement can be slow or fail. Stop intent remains pending until existing termination proof succeeds; cancelled row status alone does not acknowledge delivery. - Mid-run limits depend on provider usage reports or supported price calculations. Providers that report only at completion cannot provide exact mid-call detection. - Signals are process-local wake hints for the supported single-active-writer model. Durable state and startup recovery remain authoritative. - There are no migrations, new queue tables, or cached balances. Repeat the classifier and controlled budget-stop acceptance after normal rollout. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, tool use, code execution, and parallel coding agents. The host does not expose the exact backend model revision or context-window size, so those values are not asserted. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
15176963b1 |
fix: stop idle skill-source polling (#17008)
## Thinking Path > - Paperclip manages agent work and the skills that agents use. > - Project repositories create durable import jobs, and installed sources can refresh automatically. > - The server queried these records every five seconds, even when no work existed. > - Idle tenant databases need to stop receiving those discovery queries. > - This change uses the existing transaction notifications and delivery coordinator. > - Saved refresh, retry, and lease deadlines determine when the worker runs next. > - Cloud does not yet wake these deadlines, so outstanding skill work retains the instance. ## Linked Issues or Issue Description **What existing behavior does this improve?** Skill-source background synchronization. This continues the idle-work migrations in #15625 and #17000. A search found no overlapping open polling-removal PR. **Current behavior** On master `b11010b0d4`, startup installs a five-second skill-source interval and also calls it from the external-object scheduler. Each sweep queries active companies, import capacity, pending imports, and automatic refreshes. **Proposed behavior** Run at startup, after a relevant transaction settles, or at a saved deadline. Stay SQL-quiet when no eligible work remains. Preserve import admission, authorization, atomic claims, retry backoff, and the existing five-minute upstream check. **Reason and benefit** Remove recurring database activity from inactive tenant instances without requiring new tables or a new scheduler. ## What Changed - Replace both recurring skill-sync calls with one delivery-coordinator registration. - Notify from attachment/detachment, source creation/settings/manual refresh/disconnection, project/company deletion, and company status changes, including budget resume. - Restore the earliest eligible `nextSyncAt` or lease-expiry deadline. Due backlogs and conflicts retry after five seconds; no idle recovery cadence remains. - Admit deferred imports again after a batch frees capacity. Preserve requests through drain, standby, restart, and commits during a sweep. - Add sleep-safety predicates for imports and eligible automatic sources until Cloud supports these deadlines. - Document the lifecycle and process-local writer contract. No schema or API changes. - Guard external test database URLs before migrations or cleanup: require a `_test` database name and reject database/options overrides. ## Verification - 147 targeted tests pass: 39 server-startup tests, 14 new real-PostgreSQL lifecycle tests, 20 shared coordinator/scheduler tests, 63 existing import/refresh/retry regression tests, and 11 test-database guard cases. - CI exposed a stale startup mock without the new `start` method. The fixture is updated and all 39 startup tests pass locally. - Tests exercise real enqueue/settings/company-reactivation producers, nested commit, rollback, saved retry/lease restart, manual refresh failure, capacity admission, concurrent commit, shutdown cancellation, and SQL quiet after startup or pause. - Sleep-safety tests reject sleep for future imports and automatic refreshes, and accept inactive retained state. - `pnpm -r typecheck` and `pnpm build` pass. - `pnpm test:run` was attempted and stopped after repeated embedded-PostgreSQL startup failures in unrelated suites. The two existing import/refresh suites initially skipped for the same host limitation; after adding explicit disposable-database options, all 63 regression tests passed against PostgreSQL 17. Full CI passed on `ddb3ee51369baa8c7153bae9fd88efb8b903f2bc`: 55 successful checks, one neutral review check, and two intentional skips. ## Risks - Notifications are process-local. Independent SQL writers and separate live replicas do not wake this worker; startup reconstructs persisted work. - Enabled automatic sources still contact GitHub every five minutes. Their upstream changes have no event ingress here. - Cloud scheduled wakes currently cover routines and issue monitors, not skill sync. This change retains instances with outstanding skill obligations. It does not claim measured host/database suspension or fleet savings. - No change to source authorization, installed content on failure, or existing claim semantics. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository tools, and code execution. The host does not expose an exact model revision or context-window size to this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
b11010b0d4 |
Simplify failure banners and show only the latest error (#16982)
Remove dashed failure dividers and show only the latest adjacent run or recovery error. Preserve permitted retries, pending state and unrelated notice boundaries. Add production-component Storybook examples and browser regression coverage. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
505e1a5582 |
feat(cases): add persistent workflows with same-task agent handoffs (#16959)
## Thinking Path > - Paperclip manages AI agents and their work. > - An agent workflow needs persistent facts and a safe way to transfer work. > - Creating a task for each phase splits the objective, discussion, workspace, and review policy. > - A case can retain the facts while one primary task moves between agents. > - Transfers must survive process failure and preserve private access, budgets, and completion gates. > - This pull request adds durable case handoffs, native case tools, and read-only case pages. > - Prompts describe the workflow. Existing task controls decide when work can run and finish. ## Linked Issues or Issue Description **Problem or motivation** Prompt-driven agent teams need a persistent record that can move through triage, implementation, verification, and correction. Ordinary reassignment does not provide a durable instruction, safe release barrier, or recoverable delivery receipt for an agent handing off its own running task. **Proposed solution** Extend standalone cases with one primary task, a descriptive phase, retained access sources, and a handoff ledger. Apply assignments after source execution and workspace settlement. Deliver each transfer through the existing durable wake queue. Provide `create_case`, `read_case`, `update_case`, and `handoff_case` in the native runner. Creation requires kickoff and explicit workflow text. An explicit handoff to `done` closes the case when normal task completion is accepted. **Alternatives considered** Do not add a workflow engine, filesystem queue, configured state graph, or one task per phase. The workflow document combines transition prose with an optional Mermaid `stateDiagram-v2` map. The server does not execute the diagram. The Pipelines experiment removed on master stays removed. **Roadmap alignment** This extends Agent Reviews and Approvals, Enforced Outcomes, Bounded Run Recovery, and Activity Log attribution. Existing approval, review, budget, privacy, workspace, and responsible-user rules remain authoritative. Related work: [ordinary handoff wake recovery](https://github.com/paperclipai/paperclip/pull/15019). This feature adds a separate persisted case-handoff receipt and release barrier. ## What Changed - Add case routing fields, retained access sources, durable handoff history, delivery leases, and replay-safe migrations. - Keep the same primary task across agent phases. Preserve each returning agent's own session and the task's workspace. - Fence outgoing mutations, wait for provider and workspace release, and recover delivery without duplicate task comments or runs. Recheck current assignment permissions as the original caller before applying a delayed transfer. Scope a run Stop to that run’s own handoff intents, so repeated old Stop requests preserve newer work. Cancelling terminal intent without a source run also requires fenced current task ownership, including native retries and preparation with an old checkout. For Cloud callers, revalidate current stack membership with an uncached Cloud RPC and derive owner elevation from the current managed setting. - Keep case reads, documents, assets, Audit, and live events inside retained task/project permissions. - Add native case creation/read/update/handoff tools. Store required workflow text and initial documents on the first task. Page oversized case facts and referenced document bodies through the dedicated tool with bounded UTF-8 JSON, current routing versions, snapshot conflicts, and current source authorization on every page. - Add read-only case lists and detail pages, task-style list controls, compact dotted field tables, linked handoff tasks, and case filters in Audit. - Add native/legacy integration tests, migration/restart tests, browser coverage, and bounded live receipt fixtures. Extend the trusted sandbox condition to cover the new subscription fixture profile. - Hide obsolete paused-agent warnings on completed or cancelled task threads. - Synchronize the company-switch settings test with React effects after a CI failure. Preserve its security assertions and production remount behavior. ## Verification The branch directly includes master `9ef434fe79`. The required `ci / verify` and `ci / e2e` gates pass on head `765fccd67c`. [Current-head CI](https://github.com/paperclipai/paperclip/actions/runs/39464348614) passes typecheck, build, tests, and all eight browser shards. Security checks pass. [Greptile gives this exact head 5/5](https://github.com/paperclipai/paperclip/pull/16959#issuecomment-6291156549), with no unresolved review threads. The optional Paperclip Review check remains queued because its assigned agent is paused. It is not a required branch check. No check result was overridden. Local workspace typecheck and build pass after the master merge, along with database migration checks, token gates, map checks, Storybook catalog checks, and the Copilot profile identity check. Standard builds refresh Runner vendor output. Focused checks pass 2,008 API/paging/context tests and 7 strict case protocol tests. The final commit changes one test fixture only; its ten tests, UI typecheck, and token gates pass. No production source changed in that commit. On current head `765fccd67c`, all **194 critical workflow tests** executed and passed with no skipped tests in those files: - 38 Stop/ownership tests, including board completion intent, native retries, stale Stop, and preparation with an old checkout. - 2 actual legacy process/HTTP handoff tests, including scratch failure, a closed-stdio child that continues writing, and recovery through a fresh worker after the process group stops. - 5 migration tests, including SQL column metadata, preserved null authority context, and all six prior-numbering receipts. - 3 process-settlement and 5 scratch tests. - 88 handoff-service, 18 heartbeat handoff, and 21 native case-tool integration tests. - 7 bounded case/document-read projection tests and 7 strict protocol-contract tests. The current CI checkout was `fed65d49`, merging `765fccd67c` into master `9ef434fe79`. Its new document regressions reconstruct exact Unicode/escaped bodies and revisions through real native authority and authenticated HTTP with generic API tools disabled. They cover case-owned, primary-task, and linked-task documents; routing on every page; stale document snapshots; access loss and unlinking; unrelated/foreign/private sources; system documents; and quarantined content. Active-task document tools retain their existing behavior. These are current-head test results from the first CI attempt; the browser-only retry did not rerun these suites. The full `64fdd3403f` CI failed one separate settings test: after a company switch, it observed an unsettled API-key draft. The production component and test were unchanged from prior green source. Production switches remount both keyed settings layers; the test directly reused an unkeyed view and used a timer tick instead of waiting for React effects. The final commit replaces that fixture synchronization with React `act`. The exact secret-clearing assertions and all timeouts remain unchanged. The original assertion failure remains recorded; no unchanged retry was used. The narrow unchanged baseline and fixed fixture each pass 10 tests. The upstream result-paging merge preserves case capture namespaces and current asset privacy. Its API integration suite executed 39 of 41 tests successfully, with two skipped; the optional assigned-result PRP integration test was skipped. Neither skipped suite is counted in the 194 critical tests or as live provider proof. All 361 upstream migration entries and SQL bodies remain unchanged. Six case migrations follow master as 0363–0368 with identical SQL hashes. Official Drizzle generators produced the combined metadata and five retained snapshots. Generation reports no schema changes, and the snapshot drift test passes. The earlier `ce361ae5d0` CI passed on attempt 1, including its then-current 179 critical tests. Fresh review later identified unusable document references with generic API tools disabled; the bounded document reads and ten additional authenticated HTTP tests address that finding. The full local `pnpm test:run` attempt was cancelled after this Mac exhausted PostgreSQL semaphore capacity; it is **not** counted as a pass. Later database attempts also stopped before assertions or skipped, including one sandbox socket denial. Two original stale-Stop regressions did reproduce the bug against the old source. A REST serialization assertion passed after two preserved database-startup failures. No test timeout was increased. All 194 assertions listed above executed and passed in current-head CI. The complete 385-file, 3,963-test UI shard also passed after the fixture repair. A configured-provider receipt workflow passed 13 backend checks across triage → implementation → verification → correction → done, using one task and four work runs. The saved receipt is revision 2 with total 6 USD; case and task both closed. The original canonical attempt remains failed because its browser assertion confused the terminal destination with completed-task tooltips. The corrected full browser verifier passed on the same retained database and original runs, with no additional provider calls. The last clean-build browser readback is on `6a86c54a53`; later changes affect recovery, authorization, bounded reads, and settlement. Accounted work cost is a $1.010574 rate-card estimate; provider billing and setup-probe usage are unavailable. Copilot profile 49 passed the unchanged seven-fixture private protocol pack (46/46 checks, no retries) and the product file-edit canary after master's completion-replay change. All 60 bound implementation hashes remain unchanged on the current source. Historical profile-47/48 identities and results remain intact. See the [profile-49 compatibility record](https://github.com/paperclipai/paperclip/blob/codex/persistent-case-handoffs/doc/plans/2026-10-20-copilot-completion-recovery-compatibility-v49.json). This is bounded compatibility evidence, not a fresh full matrix, installed-package product test, or Daytona rollout. Preserved CI failures and retries: - On `20662930ac`, Slack managed setup failed a five-second success assertion after reload. Its test/application path was unchanged from a prior green head. One unchanged-source retry passed all 25 tests in that shard; no source or timeout changed. - On `9402b4f17a`, two CI workers received shutdown signals at 15:24:19 UTC. Neither log had an assertion failure before interruption. One unchanged-source retry of each passed all Runner static checks and all 32 tests in browser shard 6; both aggregates passed. The next head, `7b4374edaf`, passed all CI without retries. - On `295f289b5a`, serialized shard 9 timed out during its first device-login test, then the next test found an active login lease. Direct login/test paths were unchanged from a prior green head. Cold import plus a timed-out async body using the next test's mutable actor/store is the supported fixture-race explanation; no request trace proves that sequence. The original failure remains recorded. One unchanged-source retry passed all 294 tests in 19 files, including all 41 device-login tests. No login source or timeout changed. The subsequent `ce361ae5d0` CI passed without retries. - On `64fdd3403f`, one company-switch settings assertion observed an unsettled draft. The final test-only commit uses React `act` without changing assertions, timeouts, or production code. On `765fccd67c`, all ten settings tests and the complete 385-file, 3,963-test UI shard passed. - On `765fccd67c`, browser shard 3 completed the split-handoff assertions, then cleanup received HTTP 500 from run cancellation after a one-second PostgreSQL row-lock timeout. The extra chat run was reporting completed work while teardown began. Cleanup stopped before restoring a global experiment flag, causing a later assertion to fail. Relevant production and test paths are unchanged from an earlier green commit. Stop's lock order is consistent, and no nested self-lock was found; the exact PostgreSQL blocker was not recorded. One unchanged-shard retry passed. The original failure, trace, and downstream flag failure remain recorded. No Stop implementation, cleanup assertion, or timeout changed. The retry supports intermittent cleanup contention; it does not identify the original blocking transaction. ## Risks - Local process transfers require host-owned PID/group absence. Missing or uncertain termination evidence keeps delivery pending, independently of scratch setup. - Handoff admission and task cancellation share execution ownership. New regression tests cover temporary holds, late source writes, and uncertain provider termination. - Retained private sources use an intersection of existing permissions. This intentionally fails closed when a source is missing or an existing task cannot preserve that boundary. - Migrations add tables and indexes. They preserve master history and can replay over prior feature installs. Migration 0368 adds nullable internal authorization provenance with no default or authority backfill. Historical transfers still awaiting release without this context fail closed; an operator must cancel them and request a new handoff. - Delayed Cloud handoffs depend on the current Cloud membership RPC. An unavailable, ambiguous, expired, or mismatched response holds the handoff for retry; it does not reuse cached authority or grant access. - Bounded reads reload and reauthorize the selected projection per page. Underlying work grows with the data size and number of pages. Changed data or routing requires restarting from offset zero without the old snapshot. - Phase names and workflow diagrams remain advisory. They do not satisfy review or approval gates. - This is a large, cross-cutting PR. The maintainer requested one PR for the whole feature; the commits separate data, runtime, UI, and qualification work. ## Model Used OpenAI Codex, GPT-6 family, with tool use, code execution, and delegated code review. The exact deployment model ID and context-window size were not exposed in the session. Live fixture agents used the explicitly configured provider model recorded in their test evidence. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
1f3695ac67 |
fix: expire native session failover backups after their issues close (#16866)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native agent sessions that run in a remote sandbox keep a host-side
failover copy of the sandbox's harness state, so a replacement sandbox
can resume the conversation
> - Each session keeps that copy forever, and one copy can reach several
gigabytes
> - On a busy instance these copies fill the volume, and no existing
cleanup touches them
> - Once a session's work is closed, its copy matters only if someone
reopens that work later
> - This pull request expires the copy after the session's issues have
been closed for a configurable window, and makes a reopened issue start
a fresh session
> - The benefit is bounded disk use without breaking active,
recoverable, or leased sessions
## Linked Issues or Issue Description
Bug report described in-PR (bug report template).
**What happened?**
After every remote turn, `checkpointRemoteRunner` writes the sandbox's
runner state and provider session home (for example `codex-home`) to
`durable-sessions/<scope>/failover-backups/current`. Only the same
session's next checkpoint replaces it. No code path deletes a session's
copy when the session ends, its issue closes, or its agent goes away, so
the instance volume fills. On one instance:
- 246 sessions held 17 GB of copies, 84 of them older than a week;
- 24 new sessions added 5 GB in a single day, one copy alone 3.2 GB.
**Expected behavior**
Disk used by failover copies stays bounded. A copy is kept while its
session can still resume (open issue, live run, pending recovery, or
live lease), and expires once the session's issues have been closed for
a while. A reopened issue then starts a fresh native session, instead of
failing with `runner_harness_backup_unavailable` on a replacement
sandbox.
**Steps to reproduce**
1. Run agents with a native runtime in a remote sandbox environment.
2. Let them complete many issues.
3. Watch
`<instance>/runtime/paperclip-runner/durable-sessions/*/failover-backups/current`
grow without bound after the issues close.
**Paperclip version or commit:** `master` at
|
||
|
DottaandPaperclip
|
813e808af4 |
Remove filesystem history authorization from workspace preparation (#16963)
Remove physical-root source/reader authorization and sticky directory classifications from workspace preparation. Preserve current task/run authority, app-record privacy, credential authorization, containment, and lifecycle/publication integrity.
Validate replacement selections against persisted new inputs and refresh verified checkout ownership so bounded reuse survives earlier workspace relocation. Add real PostgreSQL/Git regressions, a reproducible history-scaling benchmark, and documentation.
Full CI passed all 48 jobs and Greptile scored 5/5 at
|
||
|
DottaandPaperclip
|
4970d5ed3d |
fix(skills): unblock agent attachment and preserve Latest selections (#16899)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Operators attach company skills to agents from the skill library. > - Attachment first reads the agent's skill snapshot to preserve other assignments. > - That read materialized and validated repository files for the entire library, even in read-only mode. > - The response also treated resolved runtime versions as stored pins. Saving could then fail with beta pins disabled. > - This pull request makes version-cache inspection lightweight and returns the stored assignment choices. > - The benefit is a responsive attachment flow that preserves Latest and explicit pins. ## Linked Issues or Issue Description **What happened?** Adding a repository skill to an agent could stall before the save request. The prerequisite snapshot read projected or checked every file in every installed repository skill. Once it returned, an existing Latest assignment could become a version pin in the save payload. The server then rejected that save when beta skill pins were disabled. **Expected behavior** Reading assignments must not prepare all repository snapshots for execution. Adding a skill must preserve the other assignments and their Latest or pinned choices. **Steps to reproduce** 1. Install repository-backed skills with stored immutable versions and cold disk caches. 2. Assign one repository skill to a Claude agent with Latest selected. Leave beta skill pins disabled. 3. Open another skill, select Add to agent, choose the same agent, and save. 4. Observe the expensive prerequisite read and the beta-pin rejection after that read completes. **Paperclip version or commit** Reproduced from master at `ac9459bae3329395f10bc81c638a374eef152baa`. **Deployment mode** Observed in hosted staging. Reproduced with a disposable local instance, embedded PostgreSQL, and the Claude local adapter on macOS arm64 with Node 26.4.0. Related: https://github.com/paperclipai/paperclip/pull/9995 adds a separate read-only assignment endpoint for peer-agent access. This fix keeps the existing configuration-gated snapshot and its permissions. ## What Changed - Respect read-only listing for current and pinned version caches. Check directory and entrypoint safety without projecting repository contents, reading all support files, or updating cache access records. - Keep complete snapshot restoration and byte validation at run preparation. Missing caches are restored from stored data. Corrupted files are repaired there. - Return stored assignment version choices in both list and sync responses. Keep runtime version details in the snapshot entries. - Add database-backed regressions for cold and warm reads, pinned versions, execution repair, and unsafe entrypoints. Add HTTP regressions for save round trips and explicit pins. - Update developer documentation and verification recipes. ## Verification - Red: both latest and pinned read-only cache regressions failed on the original code. The original code created caches during listing. - Red: the HTTP save-roundtrip regression returned 400 with the beta-pin error. The explicit-pin regression also failed. - Green: the updated API and cache suites passed all 145 tests. The three related company-skill service suites also passed. - Real browser: attached a repository skill to a paused test agent with another repository skill already set to Latest and beta pins disabled. The success toast appeared. The skill's Agents tab retained the assignment after reload. API reads confirmed both assignments remained unpinned and no repository version caches were created. - Local synthetic fixture: six skills share a 606-file repository snapshot. A cold HTTP skill-list request completed in 241 ms. This is local evidence, not a staging latency measurement. - Real service comparison on that fixture: the baseline cold list wrote 3,648 version/cache files. A warm baseline list read 3,642 version files and wrote six access records (457 ms). The fixed warm list read and wrote zero version/cache files (61 ms). Both services returned the same 16 library entries. - `pnpm -r typecheck` passed. The final API edit also passed the server typecheck. - `pnpm build` passed. - `node scripts/check-maps.mjs` and `git diff --check` passed. - The broad `pnpm test:run` invocation reported six failures in the untouched `tool-gateway-service.test.ts` suite. An isolated rerun passed all 59 tests. The broad run was stopped after recording those failures, so it is not a green full-local result. The PR CI matrix separately passed those suites. Greptile completed on the current commit with 5/5, zero comments, and no open threads. - One CI shard was interrupted by runner shutdown, with no failed assertion in its log. A single failed-job retry passed on the same commit. The aggregate CI gate passed. All current-head PR checks are complete with no failing or pending checks. - This PR does not claim that the fix has been deployed to staging. ## Risks - Settings reads check cache locations and entrypoints. They defer full support-file validation to execution. Regression tests verify that execution still repairs corrupt bytes and restores missing versions. - The response now distinguishes stored pins from resolved runtime versions. Explicit historical pins remain intact. Company, configuration, connector, and beta-pin authorization gates keep their existing behavior. - No schema migration or dependency change. ## Model Used - OpenAI GPT-6 through Codex, with reasoning, code execution, and browser tools. The exact runtime build ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
7c58ea07b2 |
fix(issues): guard prepared task promotion by status version (#16893)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Integrations can prepare a backlog task before its assigned agent starts. > - A person can change its status or assignment during preparation. > - An unconditional promotion can replace that newer decision. > - Safe promotion needs a saved revision that includes every assignment path. > - This pull request checks that revision under the task row lock and derives policy cleanup from the locked row. > - The benefit is safe initialization recovery with the existing company, task, and run authority. ## Linked Issues or Issue Description Prerequisite #16896 has merged. This PR now targets master and contains only the conditional promotion API. The prerequisite fixed the confirmed baseline first-upload browser regression. **What happened?** A client that saves a task's initial documents and then sends an unconditional status update can override a concurrent status or assignment decision. The update validator previously dropped an expected version, so clients could not make this promotion conditional. Some direct assignment and release paths also did not advance the revision. **Expected behavior** A prepared task moves from backlog to todo only if its original status and assignment revision still matches. A stale request returns 409 before a task mutation, audit publication, or wake. Policy cleanup preserves unrelated saved settings. **Steps to reproduce** 1. Create a backlog task and save its statusVersion before preparing it. 2. Release its assignment, or change an assignee and then restore that assignee. 3. Send `{ "status": "todo", "expectedStatusVersion": <original revision> }` to its PATCH endpoint. 4. The stale promotion must return 409. A matching revision on an unchanged backlog task must succeed. Duplicate and related PR searches found no existing prepared-task promotion fix. ## What Changed - Accept an optional nonnegative expectedStatusVersion for exactly a status-only backlog-to-todo promotion. Reject extra public fields before parsing can discard them. - Check the current revision and backlog state under the issue row lock. Reject stale requests before checkout adoption or execution-control effects. - Add SQL migration 0358 to advance the revision for both assignee fields at the database boundary. Release, administrative release, direct writes, and assignment round trips now invalidate an old version. - Derive guarded policy cleanup from the locked row. Preserve review limits and protected authorization when removing a monitor. Reject derived ownership changes or review decisions. - Add schema, real HTTP/PostgreSQL, concurrency, migration replay, and policy regression coverage. Update the response type, OpenAPI, developer docs, and access map. ## Verification - Red before the initial fix: stale requests returned 200 instead of 409, validation removed the expected version, and a stale active task could adopt a checkout. - Review found two further defects. Four new release/direct-assignment tests showed unchanged revisions. The real HTTP/service test returned 422 for a valid two-field promotion after monitor cleanup. Helper tests also proved that cleanup lost valid review limits and protected authorization. All now pass after focused repairs. - Green: 124 route and policy tests, including 7 real Express/PostgreSQL cases, 37 existing route tests, and 80 policy/helper tests. The real row-lock race preserves a policy edit committed while promotion waits. - Green: all 23 database client tests, plus 9 focused assignment and replay checks. Historical migration 0227 receipt recovery cannot remove the new assignment trigger. Reapplying migration 0358 leaves existing rows unchanged and installs one trigger. Combined status/assignment writes increment once; unchanged assignees do not increment. - Earlier verification passed all 218 schema/route/service cases and the generated OpenAPI capability test. Rebased focused regressions passed all 27 selected cases. Final server TypeScript passed after all repairs; shared TypeScript passed. Migration safety/numbering, five-snapshot retention, map validation, and diff checks pass. - The baseline browser attachment case failed twice on the earlier head. Separate #16896 proved the lost-composer cause, passed 335 component tests and the real first-upload journey, and has green CI with a 5/5 review. No assertion or timeout was weakened. - Some early local checks were blocked by missing or stale sibling dependency artifacts. This worktree now has its own frozen offline install and standard SDK prerequisites. The failures and successful reruns remain in the task evidence. - Before rebasing, head `3304bb00de6ad6a248b5e046c008072ad20ac8f5` passed all CI (55 successful checks and one neutral), Greptile 5/5, and a secondary source review at 5/5 with no unresolved findings. The secondary reviewer did not execute project tests, inspect CI, or exercise a live agent journey. Earlier incomplete and must-fix assessments remain visible; both established findings are repaired above. - After #16896 merged, all five CAS commits were rebased onto master `87d9386ae637c652d1fed09d2046be1bcaeb413b`. Migration 0358 was regenerated because master added another 0357. Range comparison shows no other change to the CAS patches; inherited UI commits are absent. The original head is preserved locally. - Rebased head `1c0264e51a280b1dbf597725a7e42e72d4be1b1b` passes all 124 route/policy tests, 9 selected assignment/promotion tests, and all 24 database client/snapshot-drift tests. Database build, migration checks, generated snapshot linkage, five-snapshot retention (0354–0358), maps, and diff checks pass. Fresh-head CI and review are pending. No deployment or live agent journey was performed for this API. ## Risks - The guard supports prepared backlog promotion only. Other statuses or mixed public mutation fields return 422. A stale revision or non-backlog row returns 409. - Preserve the original version from a server that already advertises this guard before initialization. Older versions did not track every assignment path. Reconcile older or uncertain initialization instead of fetching a new revision and blindly retrying. - Migration 0358 is SQL-only and idempotent. It adds no table, column, or task backfill. Its separate trigger survives historical status-migration replay. - Existing authorization and unguarded updates remain in place. Guarded promotion does not provide a general lifecycle or review-stage control API. ## Model Used OpenAI Codex, GPT-6 family. The exact serving model ID and context window are not exposed by this session. Used tool calls, code execution, and test-first regression verification. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
8193abb936 |
feat(claude): default qualified macOS and Linux targets to Paperclip Runner (#15997)
## Thinking Path > - Paperclip must run the selected agent on its selected host. > - Claude needs verified packages for each supported target. > - A Mac controller must select the correct Linux provider pack. > - New defaults must follow package and execution evidence. > - This PR adds Claude platform packaging and target-specific pack selection. > - Existing Codex Linux ARM64 behavior remains intact. ## Linked Issues or Issue Description Refs #15683. This PR depends on #15993 and #15994. Related scope: #15422 and #15224. ## What Changed - Reuse the merged four-target daemon producer/consumer matrix. Add target-owned Linux provider packs and carry their immutable identities through release assembly and vendoring. - Reuse master’s landed pinned Claude SDK ARM64 executable admission and retain package-integrity, architecture and no-follow admission. - Select target-specific remote packs so Mac controllers can use Linux hosts without a development pack override. - Preserve read-only runtime context on SSH startup and resume without accepting unverified old content. - Check ordinary installed Claude readiness without injected package authority. Retain Codex and Linux x64 consumer checks. - Extend Claude automatic target selection for the remaining qualified package targets. Retain unsupported-platform and external-override behavior. - Qualify managed Daytona snapshot names through provider IDs during Claude Connect. Pin native acquisition to the measured ID. Preserve a revocation marker after failed probes or target edits, and block native acquisition until requalification instead of falling back to a mutable name. Preserve immutable-image proof, company and account boundaries, and Legacy acquisition. ## Verification - Reconciled with master `9179d8dcc3e908c217fde4f63c9d08ccf8132eeb` through the prerequisite stack, retaining the merged startup fix and latest Codex Linux ARM64 behavior. - [Four-target exact-source package qualification](https://github.com/paperclipai/paperclip/actions/runs/39267321383) passed on source `04e0202ca81387b17cbe7fd094a8338648356fe9`: all four native builds, public-package assembly, Linux x64 verification and clean installed consumers on Apple Silicon, Intel Mac and Linux ARM64. The later master reconciliation adds shared OpenRouter attribution and changes runtime source. This historical result keeps its original source identity; [later qualification](https://github.com/paperclipai/paperclip/actions/runs/39281100929) passed at `dca62c7aeb4b45f0dd91a8e18d386a16a2658ef8`. It predates the subsequent TypeScript startup fix. - Fresh local product acceptance on integrated source `b2c9fe43f9cd3bc5dd9d22552da0148845ba71a4` completed a useful file and follow-up through native Claude/Opus 5.5, High effort and the API-key connection. Both actual browser downloads were independently checked; the revision preserves the original body and session continuity. Total task estimates: $0.518432800. Ordinary web and API creation saved defaults with the native experiment disabled. - Local focused verification passed 993 startup/session tests, 243 UI tests, 12 workflow security tests, direct server/UI TypeScript checks, and token/palette gates before the last review changes. Initial DB-backed snapshot pin/revocation coverage passed 18 cases; the final serial route suites pass 122 credential/permission and 232 snapshot/environment cases after correcting a test hook timeout. Current-head cloud suites pass. - Preserved the failed pre-provider local fixture and its full pending reservation. The successful fixture has no unpriced or pending usage. Historical failed Cloud journeys, accounting holds, source identities and subscription limitations remain in the dated plan. - All Docker/Rust release production was performed in cloud CI. No local Docker image was built. An inadvertently invoked local Rust prerequisite was stopped. - Current-head cloud CI is green. All three stack PRs have fresh Greptile 5/5 reviews and zero unresolved review threads. Required typecheck, test and build gates ran in cloud CI; Docker work stayed remote. Earlier failed checks remain in their original runs. - The final master merge preserves task-loading, permission, skill-retention and OpenRouter changes. Pi uses a fresh combined profile 27, preserving both historical profile-24 declarations; Copilot remains profile 22. Focused protocol checks passed 155 cases (one skipped), the updated Pi attestation check passed 18 (one skipped), and 91 evaluation/catalog checks passed. - Latest-application manual continuation passed in 37.942 seconds on `695e5c005a85bddc8711d3fcddcaf6b7e1fb64d0`, with High effort, native Opus 5.5, Anthropic API attribution, the same resumed Claude session, an independently checked 635-byte file and persisted Done status. It uses the earlier verified Mac daemon; the preceding-source package qualification passed at `dca62c7aeb4b45f0dd91a8e18d386a16a2658ef8`. That package run predates the small TypeScript startup fix and is not relabeled as qualification for it. - Upgrade limitation: changed compiled sidecar identity rejected old durable launch state and recovery hit session-open bounds before eventually resuming. That attempt was operator-cancelled and retained; the successful manual continuation does not qualify seamless upgrades. - Local latency profile, Opus 5.5/High unchanged: three alternating real-ACP loopback samples per variant reduce median provider-ready time from 5.081s to 4.034s. The matched real API task startup improves from 5.533s to 4.348s, but total reply time changes from 17.933s to 20.221s because agent work varies. This establishes a startup saving of about one second, not a consistent total-reply speedup. A real candidate follow-up passed in 20.146s and resumed the same saved session. The loopback fixture makes no external model requests; failed initial probes remain retained. - The startup change passes 107 focused runtime/effort/lease tests, 53 profile checks (one skipped), TypeScript checking and the TypeScript-only runtime build. Native hire fixture checks pass 70 tests; their first local run retained a socket-hangup failure and stale OpenCode expectation, both accounted for before the passing rerun. Broader required checks run in cloud CI. ## Risks - **Experience needs improvement:** successful connection setup still shows “Connection needs attention” because runtime checks are deferred. Cold provider-pack upload took 122–126 seconds per run while the task mainly showed “Thinking.” The tested Core-only preview excludes private Cloud UI plugins; this is not full Cloud distribution or subscription qualification. Desktop dark theme was tested; mobile and comprehensive accessibility were not. The portable report validates, but gallery interaction could not be inspected because host preview was queued. - The optional supply-chain scan returned neutral/inconclusive because GitHub Actions package coverage and infrastructure parsing are unavailable. It is not counted as a successful supply-chain qualification. Exact package, daemon and image integrity checks passed. - A stale, missing or wrong-architecture pack must fail admission. It must not change runner or target silently. - Provider-pack source changes invalidate affected earlier execution results. Existing unknown-cost and cleanup holds remain intact. - Live subscription execution remains unverified under the API-key qualification direction. Built-in authentication flows are retained. ARM64 cloud-image production is deferred. Merge and production deployment remain separate. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository tools, code execution and browser testing. The exact serving model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] Focused local tests and final cloud verification pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
45b67f1d93 |
feat: default new Apple Silicon Claude agents to Paperclip Runner (#15993)
## Thinking Path > - Paperclip lets users choose an agent harness and account. > - The server resolves the runner for new agents. > - Claude still needs the same ordinary native-default journey as Codex. > - Existing agents must retain their runner, model and account. > - This PR defaults new Apple Silicon Claude agents to Paperclip Runner with Opus 5.5. > - Users can connect and work without a runner override. ## Linked Issues or Issue Description Refs #15683. This PR depends on #15994 and precedes #15997. Related scope: #15422, #13978 and #15224. ## What Changed - Resolve new Apple Silicon Claude agents to `paperclip_runner`, `provider: acpx`, `acpxAgent: claude`. Keep explicit Legacy, models, provider routing and saved execution. - Use `claude-opus-5-5` for ordinary Claude defaults and retain explicit overrides. - Reuse existing API-key/subscription connections, credential storage and checks. Generalize the shared Advanced runner control with Claude branding. - Carry configured effort into the shared v9 contract. Preserve compatible permissions, instructions, skills, timeouts and session behavior. Require Legacy for unsupported settings. - Qualify a new immutable managed Claude target at explicit Connect through the existing permission-gated probe. Use fresh scoped discovery for test and hire. Discard stale results and require an explicit Connect after settings change. - Retain merged Codex onboarding recovery, auth preparation, import/effort corrections and Grok Bot gates. - Check the Claude first-task wizard's saved default directly. Update configuration/runtime documentation and exact Opus 5.5 accounting. - Align hire fixtures with the merged host-environment restriction. Native agent-authenticated hires cannot inject host env settings; both creation endpoints verify that no extra agent is saved. ## Verification - Updated onto master `9179d8dcc3e908c217fde4f63c9d08ccf8132eeb` through #15994. Native input v9 carries effort while preserving v7 home modes and v8 Muse. Codex defaults remain unchanged. - Focused local verification passed 285 server tests, 323 UI tests, 124 setup/registry tests, and two v9 permission-pin transport cases. Direct server/UI TypeScript checks passed. An initial 30-second hook timeout was resolved by serial verification with a longer hook budget: all 122 credential/permission cases and 232 snapshot/environment cases pass. Direct server TypeScript checking also passes. - Ordinary web creation with the experiment disabled saved native Claude, Opus 5.5 and High effort without runner/model overrides. The actual product task and follow-up both succeeded on integrated source `b2c9fe43f9cd3bc5dd9d22552da0148845ba71a4` with the cloud-built daemon from `04e0202ca81387b17cbe7fd094a8338648356fe9`. The final master reconciliation also changes shared runtime source. Historical proof retains its source identity; a final-source cloud qualification is running before refreshing the preview daemon. - Both actual browser downloads were checked: v1 has three sections/six checkboxes (451 bytes); v2 preserves the original body and adds two Rollback checks (612 bytes). The original attachment stayed byte-identical. The follow-up records session resume and the same provider session. Total estimated API usage: $0.518432800, with no pending/unpriced usage in the successful fixture. - Fixed the live-discovered setup incompatibility: native API-key validation no longer depends on the global Legacy Claude CLI. The existing provider endpoint verifies the key without a paid prompt; subscription/custom routing probes remain intact. - Fixed security review's native-hire credential redirection: agent-supplied URLs, proxy/launch settings and other host environment controls are rejected before inheritance. Native credential references remain supported and partial edits use the saved harness. Added create/hire/update regression coverage. - Current-head cloud CI is green. All three stack PRs have fresh Greptile 5/5 reviews and zero unresolved review threads. Required typecheck, test and build gates ran in cloud CI; Docker work stayed remote. Earlier failed checks remain in their original runs. - Docker and native release builds ran in cloud infrastructure. An inadvertent local Rust prerequisite was stopped; no local Docker build was run. - The final master merge preserves task-loading, permission, skill-retention and OpenRouter changes. Pi uses a fresh combined profile 27, preserving both historical profile-24 declarations; Copilot remains profile 22. Focused protocol checks passed 155 cases (one skipped), the updated Pi attestation check passed 18 (one skipped), and 91 evaluation/catalog checks passed. - Latest-application manual continuation passed in 37.942 seconds on `695e5c005a85bddc8711d3fcddcaf6b7e1fb64d0`, with High effort, native Opus 5.5, Anthropic API attribution, the same resumed Claude session, an independently checked 635-byte file and persisted Done status. It uses the earlier verified Mac daemon; the preceding-source package qualification passed at `dca62c7aeb4b45f0dd91a8e18d386a16a2658ef8`. That package run predates the small TypeScript startup fix and is not relabeled as qualification for it. - Upgrade limitation: changed compiled sidecar identity rejected old durable launch state and recovery hit session-open bounds before eventually resuming. That attempt was operator-cancelled and retained; the successful manual continuation does not qualify seamless upgrades. - Local latency profile, Opus 5.5/High unchanged: three alternating real-ACP loopback samples per variant reduce median provider-ready time from 5.081s to 4.034s. The matched real API task startup improves from 5.533s to 4.348s, but total reply time changes from 17.933s to 20.221s because agent work varies. This establishes a startup saving of about one second, not a consistent total-reply speedup. A real candidate follow-up passed in 20.146s and resumed the same saved session. The loopback fixture makes no external model requests; failed initial probes remain retained. - The startup change passes 107 focused runtime/effort/lease tests, 53 profile checks (one skipped), TypeScript checking and the TypeScript-only runtime build. Native hire fixture checks pass 70 tests; their first local run retained a socket-hangup failure and stale OpenCode expectation, both accounted for before the passing rerun. Broader required checks run in cloud CI. ## Risks - Missing packages, authentication rejection, unsupported settings and unavailable models must fail explicitly. Existing agents retain saved execution; explicit runner changes use existing revision/session invalidation. - Subscription follow-up remains unverified; remaining live acceptance uses API keys per human steering. Historical accounting holds and the separate $50 ceiling remain intact. - The subscription acceptance limit remains explicit for review. No Claude database migration is introduced. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository tools, code execution and browser testing. The exact serving model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] Focused local tests and final cloud verification pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
3e5a9f5afe |
perf(ui): load recent tasks with bounded summaries (#16851)
Replace passive recent-task detail reads with bounded, authorized summaries. Keep account isolation, stored history, task actions, and live recovery behavior. Overlap access checks with shell settings and avoid duplicate session and health reads on the first healthy socket opening. Retain domain reconciliation. Add focused coverage and local performance evidence. Isolate the existing cache capacity test from entry expiry on slow CI runners. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
06c5bc471d |
fix(skills): unblock retention for legacy run selections (#16852)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agent runs can select saved skill versions. > - The server keeps those files in a cache that must protect active and unresolved runs. > - Older runs can lack the newer pin map even when their saved execution input records the full selection. > - Finished interrupted runs can also keep a company blocked after all execution and cleanup have settled. > - This pull request uses existing selection evidence and recognizes settled interruptions so safe cache expiration can resume. ## Linked Issues or Issue Description Related: #16433. **What happened?** Skill cache retention reports `unknown_run_pins` for a whole company when it finds a finished interrupted run without pins, or an older native run whose complete selection is stored only in its execution input. Old cache directories cannot expire. **Expected behavior** Release pins from fully settled interruptions. Preserve exact versions from valid saved native selections. Continue to block cleanup when selection evidence is missing, invalid, or belongs to another run. **Steps to reproduce** 1. Materialize an old, restorable skill version and let its idle grace period expire. 2. Add a finished interrupted run with settled ownership and cleanup, or an unresolved native run with a valid saved execution input but no modern pin map. 3. Run the retention service. Before this change, it skips the company with `unknown_run_pins`. ## What Changed - Treat `interrupted` as terminal only when the existing finish, controller, finalization, and environment cleanup checks allow it. - For native runs with no modern pin-map field, parse the saved execution input with the existing recovery parser. Validate its digests and company, run, agent, and issue binding. - Keep captured version IDs protected. Unversioned skills use separate captured asset bundles. Do not replace a malformed modern map or write any recovery records. - Add database regressions and document the fallback. Remove an unused status list that no longer represented the actual eligibility query. ## Verification - Passed 41 focused tests: `skill-version-cache.test.ts`, `skill-version-retention.test.ts`, and `skill-version-retention-config.test.ts`. - The tests perform real PostgreSQL reads, directory eviction, and restoration from saved bytes with upstream unavailable. They cover leases, unfinished cleanup, invalid digests, invalid versions, and mismatched identities. - Passed `node scripts/check-module-boundaries.mjs` and `git diff --check`. - A read-only evaluation of the candidate selector against existing legacy data cleared the unknown-pin blocker while retaining exact version pins. - Passed `pnpm -r --workspace-concurrency=2 typecheck` and `pnpm build`. The full remote CI suite passed, including all server, workspace, Runner, and end-to-end shards. The full local test command remains in progress; its focused cache suites passed as recorded above. - Greptile reviewed this head at 5/5 with no actionable comments or unresolved threads. - Initial local database attempts did not run the cases: sandbox access prevented a listener, then an install with scripts disabled lacked PostgreSQL library symlinks. After running the package's symlink setup, the binary started and all focused tests passed. ## Risks The main risk is releasing a cache still in use. The existing ownership and cleanup checks remain in force. The native fallback applies only when the modern field is absent and the saved input passes the same parser used for recovery. Invalid or missing evidence still blocks the company. Current versions and explicit agent pins stay protected. Saved version rows and runtime asset bundles are not deleted. There is no migration. The selector reads execution inputs only for legacy native runs that need the fallback. Normal runs still load only their pin maps and binding columns. A representative read-only selector check completed in 98 ms; this is a sample, not a latency guarantee. ## Model Used OpenAI Codex, GPT-6 family, with tool-assisted coding and verification. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
56e09cd19f |
docs: add native Runner integration playbook (#16846)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native Runner connects those agents to harnesses through provider drivers. > - Adding a harness crosses protocol, setup, lifecycle, packaging, and product boundaries. > - Existing guidance does not provide one complete procedure for this work. > - This pull request adds a native Runner skill, detailed playbook, and working templates. > - Contributors can use them to define a release, implement the integration, and verify the delivered result. ## Linked Issues or Issue Description **Issue type** Missing documentation. **Where is the issue?** Native Runner contribution guidance in `doc/DEVELOPING.md` and `packages/paperclip-runner/README.md`. **What's wrong?** Contributors must reconstruct the requirements, protocol mapping, normal setup, lifecycle rules, packaging, and acceptance process when they add a harness. The legacy direct-adapter recipe covers a different execution path. **Suggested fix** Add a self-contained native Runner procedure. Include capability research, a finite release contract, implementation boundaries, reproducible tests, normal installation, and release evidence. State each lesson directly so contributors need no private chats, tasks, or evidence archives. The duplicate search found no existing playbook PR. ## What Changed - Add `create-runner-adapter` for new native Runner harness integrations. - Add a detailed playbook based on the Pi, Cursor, and Copilot integration lessons. - Cover model and instruction admission, connection setup, rich events, durable interactions, recovery, cancellation, packaging, and Product E2E acceptance. - Add templates for scope, capabilities, attempts, failures, jobs, and release handoff. - Link the procedure from the development guide and Runner README. - Keep legacy adapters as a compatibility boundary. ## Verification - `git diff origin/master --check` passed. - `node scripts/validate-doc-links.mjs` passed from `packages/paperclip-runner` and checked 53 documentation files. - All 71 Markdown links in the new reusable files resolve to repository paths or public provider documentation. - Skill YAML, naming, description, and scaffold checks passed. - A private-reference audit passed for the reusable skill, playbook, and templates. - The legacy direct-adapter skill matches the base version. - Full local runtime checks were attempted with pnpm 9.15.4. `pnpm -r typecheck` stopped because Node type definitions are missing. `pnpm test:run` and `pnpm build` stopped because `cli/node_modules/tsx` is missing. This checkout has no installed dependencies. These commands did not establish runtime test or build results. - Full remote CI passed on commit `88b4fc4832d95807da1f5e48029a183b3dc62b6c`: build, typecheck, general and serialized server tests, Runner checks, browser tests, and the canary dry run. [CI run](https://github.com/paperclipai/paperclip/actions/runs/39262297681). - All 55 applicable checks passed. The two Storybook checks were skipped for this change. - Greptile gave the current commit 5/5. There are no review comments or unresolved threads. ## Risks Low risk. The procedure can drift as the Runner contracts and test entry points change. It records its source baseline and tells contributors to inspect current declarations. Historical lessons describe failure patterns; they do not certify current binaries. ## Model Used OpenAI Codex with `gpt-6-astra` and `gpt-6.1-sol`, both using xhigh reasoning. Context-window sizes were not exposed. The models used repository search, terminal tools, and file edits. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used, including available version and capability details - [x] I have checked ROADMAP.md and confirmed this documentation does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs - [x] I have described the issue in this PR using the documentation issue template fields - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal ticket or instance details - [x] I have run the relevant local documentation checks; they pass. Blocked runtime checks are recorded above - [x] I have considered tests for this documentation-only change and validated its links and skill metadata - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented the risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
98ca6ff7c5 |
perf: reuse verified provider files during sandbox startup (#16493)
Build provider-pack transport layers once, reuse verified image files in private directories, and upload only missing layers. Preserve archive integrity checks and directory fallback on unsupported hosts. Verified with 44 focused tests, all 48 main CI jobs, cloud Docker Runner checks, and Greptile 5/5 on the merged head. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
2f6ad2226a |
fix(heartbeat): reconcile stranded running agent status (#16791)
Serialize agent completion status updates with new execution starts. Repair stranded running flags through existing recovery after durable work and cleanup settle. Share exact failed-lease cleanup proof with idle sleep, bound recovery work, and keep repair failures from blocking unrelated recovery. Verified with 54 successful CI checks, 292 affected regressions after review fixes, workspace typecheck/build, and Greptile Apex 5/5 with no unresolved comments on the merged head. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
b85a69b334 |
test(server): record observer load in native event replay (#16800)
Record observers and polling in synthetic native replay measurements, with zero-viewer controls and correctness checks. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
790b729086 |
fix: restore nested project repositories and settings after expiration (#16691)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents use isolated Git workspaces with managed project repository copies. > - Finished workspaces can expire when Git can reproduce their contents. > - The saved recipe records the main commit but omits nested commits and local settings. > - Reopening can therefore use newer nested code and lose merged user settings. > - This pull request saves and restores those contents before workspace provisioning. > - It also permits verified FETCH_HEAD metadata so ordinary fetches do not block safe expiration. ## Linked Issues or Issue Description Refs #16457, #16586, #16621. Related performance work: #16579. **What happened?** After a finished workspace expires, its nested project checkout can reopen at the current upstream commit instead of its previous commit. Cleanup also treats local Claude settings as disposable, although that file can contain user settings. Ordinary linked-worktree FETCH_HEAD metadata prevents otherwise safe cleanup. **Expected behavior** Restore the main and nested repositories at their saved commits. Preserve local settings bytes and permissions. Skip files and Git state that cannot be reproduced. Permit standard fetched metadata only when its commits remain recoverable. **Steps to reproduce** 1. Prepare a managed nested project repository at commit A and add ignored local settings. 2. Advance upstream to B, finish the task tree, and expire its workspace and base. 3. Reopen the workspace through the normal service path. Before this fix, nested preparation selects B. ## What Changed - Extend the existing restoration recipe with exact nested commits, branches, source identity, and encrypted local settings. Keep the schema compatible with older recipes. - Restore nested repositories and settings before provisioning. Recheck company/project source access and preserve conflicting files on retry. - Recheck recorded contents under cleanup locks before deletion. Persist lock ownership before atomic publication so process exits recover safely and foreign locks stay intact. Preserve changed or unsupported contents, including tracked settings. - Keep saved project repository selections through normal preparation. Checkpoint restored contents before provisioning so failed setup can retry without overwriting later edits. - Accept bounded linked-worktree FETCH_HEAD records only after local ancestry or fresh upstream proof. Preserve local-only fetched commits. - Add real Git/Postgres lifecycle cases and focused content, access, publication-retry, and fetched-state controls. Document the behavior. ## Verification - `pnpm -r typecheck` passed. - `pnpm build` passed. - The original real Git/Postgres experiment failed with nested B instead of A. The same path passes with this change. - The integrated lifecycle suite passed 42 cases in 91.66s. Related preparation, credential, content, and Git metadata controls passed 46 cases in 24.41s. The final heartbeat checkpoint persistence case passed separately in 8.17s. - Coverage includes exact main/nested commits, encrypted settings and modes, upstream failure and retry, provisioning order, concurrent reopening, changed unique files, tracked settings, malformed metadata, and company boundaries. - Exact-head CI passed the implementation, typecheck, build, security, and test matrices. One browser shard passed on a single rerun after an unrelated fixture teardown raced an activity-log write; its feature assertions passed in the initial attempt. - The broad local test run was stopped after about 12 minutes of unrelated serial tests. Full parallel CI is required before merge; no local full-suite pass is claimed. - Single-workspace deployment, real-backlog reclamation, and ordinary product reopening remain required acceptance work after merge. ## Risks - Expired Git repositories still depend on upstream access. A failed restoration returns the existing retryable error and never substitutes the default branch. - Saved settings use the existing local encryption provider. Recovery requires that encryption key to remain available. - Unsupported files, tracked settings, unknown sources, and unverified Git state remain on disk. This can reduce reclaimed space. - No database migration, recovery-retention change, public API, or fleet-default change is included. ## Model Used OpenAI Codex, GPT-6 family, with tool use, code execution, and an independent agent review. The runtime does not expose a more specific model identifier. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
5555817c2b |
fix(server): let Git expiration verify public commits without a run identity (#16621)
Git expiration created its auth provider with only the issue ID; with a managed GitHub connection that provider throws outside a run ("No managed GitHub identity is available for this run"), so every upstream commit proof failed and was logged as inspection_failed. Expiration now falls back to anonymous access when the provider throws: public repositories verify, private ones stay kept. Upstream verification failures are reported as upstream_commit_unverified, and a managed checkout whose upstream cannot be reached or authenticated as managed_checkout_upstream_unavailable; local proof-store failures keep their own error.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
|
||
|
DottaandPaperclip
|
669d329f25 |
perf(slack): avoid unused catalog revision scans (#16630)
Slack catalog discovery now checks live task, user, endpoint and credentials without calculating an unused compatibility revision. Full authority callers retain the unchanged revision and search-token path. Validation: 44 affected cases, full repository typecheck/build, current-head full CI/E2E and security checks, Greptile Base 5/5 with no unresolved findings. Identical production catalog output with revision statements reduced from two to zero in real-PostgreSQL comparisons. Hosted latency acceptance remains pending. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
0528cc7ee0 |
perf: overlap native GitHub and assigned-tool setup (#16620)
Prepare managed GitHub access and fresh assigned-MCP discovery concurrently for remote native Codex. Wait for both before provider construction and drain both on failure through the existing run cleanup. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
dc499ae02b |
perf(server): reject shared Git cleanup paths before history scans (#16614)
Check the existing global execution/project shared-path veto before scanning retained run contexts. Preserve the full unshared-path query, lifecycle locks and all deletion-phase rechecks. Verify with 35 focused real PostgreSQL/Git cases, full repository typecheck/build, exact-head full CI/E2E and Greptile Base 5/5. A read-only same-snapshot query pair retained the false verdict while reducing round trip from1089ms to9.4ms; this is not a full-agent latency result. The first security scan was inconclusive after an infrastructure checkout failure; one actual rescan passed. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
9a3290fe10 |
perf(runner): overlap default remote artifact preparation (#16608)
Prepare the native runner and default image Codex harness together on providers that declare concurrent sync support. Preserve all executable checks, wait for both operations on failure, and retain serial explicit harness installs. Verified with 881 affected cases, full typecheck/build, exact-head full CI and Greptile Base 5/5. The real Daytona component median fell from 839ms to 432ms; full deployed reply latency remains unproven. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
73659c6b27 |
fix(server): let Git expiration discard Paperclip-managed checkouts (#16586)
Reconstructable Git expiration rejected nearly every terminal worktree with unique_ignored_files, because Paperclip writes two ignored paths into every worktree itself: .claude/settings.local.json and the managed project checkouts under .paperclip-repositories/. Expiration now accepts the settings file, and accepts a managed checkout only through a prover the expiration worker supplies. The prover maps the checkout back to its project workspace row and proves it reproducible against that row's repository URL: worktree and stock-storage checks, no linked worktrees, no unreachable objects, and every ref, reflog and HEAD commit reachable from upstream refs fetched fresh into a proof store. Every inspection Git command now runs with core.fsmonitor disabled. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
f5534717b0 |
fix(runtime): preserve unexported native work during lease cleanup (#16583)
Preserve the exact native workspace source until its export is finalized. Apply the same guard to cleanup sweeps and direct environment teardown. Keep stop-only recovery and healthy warm retention available. Verified with 317 targeted tests, server typecheck, module boundary check, all current-head CI gates, and Greptile 5/5. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
ce0eff7d1f |
perf: reduce fresh workspace clone and sandbox staging waits (#16487)
## Thinking Path > - Paperclip manages AI agents and their work. > - Each run needs a private workspace and safe persistence. > - Fresh isolated repository sources download upstream Git history again. > - Sandbox staging also verifies an optional seed cache before it uploads the workspace. > - That verification holds a company lock during a full extraction and hash scan. > - This pull request reuses pristine Git objects through an authenticated remote clone. > - It also overlaps seed verification with upload and moves verification outside the publication lock. > - The benefit is less repeated transfer and less serial preparation work, with the same task isolation and recovery contract. ## Linked Issues or Issue Description **What existing behavior does this improve?** Fresh task-private repository preparation and inbound sandbox workspace staging. **Current behavior** Each isolated remote source clones the full upstream history. Optional seed verification then extracts and hashes another full tree before upload. One workspace's verification can hold the company cache lock while other workspaces wait. **Proposed behavior** Use a company and credential scoped cache of pristine upstream objects for GitHub HTTPS sources. Each task still clones from the current remote. It disconnects from the cache before publication. Run seed verification alongside upload. Hold the company lock only for final capacity checks and publication. **Reason and benefit** These operations lie on the workspace preparation path. Removing repeated Git pack transfer and serial verification can reduce startup time. Local measurements below cover these boundaries. They do not establish hosted agent latency. Related: #16425, #16454, #16398, #16461, #16468, #16473. The cache inspection and durable folder proposal in #13376 covers separate product work. ## What Changed - Add a pristine upstream Git object cache for fresh isolated GitHub HTTPS sources. Key it by company, repository and current credential fingerprint. Do not store credentials. - Keep online clone authorization and current remote refs. Use `--reference-if-able --dissociate` so task objects remain independent. A reference failure retries once online without the cache and resolves credentials again. - Cap cache storage at 64 entries and 4 GiB per company, with 2 GiB per entry. Skip contended publication. Bound priming by a cooperative five-second budget and a 4,096-entry scan allowance. Keep the existing free-space reserve. - Overlap optional seed verification and required workspace upload. Settle both before returning or deleting temporary files. Recheck quotas and company directory identity under the publication lock. - Keep the independent fresh upstream reproducibility proof, task commits, dirty source copies, ignored files, nested repositories and durable recovery behavior. - Suppress optional Git server-info generation during dissociation so generated storage passes the existing safe-expiration checks. This uses [the supported Git setting](https://git-scm.com/docs/git-config/2.43.0#Documentation/git-config.txt-repackupdateServerInfo). - Add bounded timing phases, regression coverage and two repeatable local benchmarks. No first-party Telemetry or dependency changes. ## Verification - `pnpm -r typecheck`: passed. - `pnpm build`: passed. - Affected adapter suites: 240 tests passed. Native workspace persistence/history and tracing: 33 tests passed. Server checkout/isolation/tracing batch: 53 tests passed. Final object-cache authority, path and capacity suite: 12 tests passed. These batches overlap. - Cache and expiration/restoration lifecycle batch: 34 tests passed. A direct generated-storage check covers cached clones. - Real HTTP Git fixture: a matching cache does not bypass revoked remote access. Other fixtures cover current refs/tags, hidden cached commits, independent objects after cache removal, corruption fallback, credential/company scope, quota limits and publication contention. - Initial restricted runs hit SQLite write and Unix socket permission failures. The adapter batch passed with disposable writable instance state and socket access. The first typecheck rejected the test use of `Promise.withResolvers`; typed deferred gates now use the supported library contract. - Repository benchmark on final head `9047368bf3b7fa2061e00d698b8b51cf4ae1da32`: five alternating samples, real local `file://` Git transport, synthetic 32 MiB history, no injected latency. Median 1143.8 ms to 240.5 ms; ranges 1085.7–1486.6 ms and 188.0–254.8 ms. Received Git pack bytes fell from 33,569,788 to zero. Initial cache priming took 1208.4 ms and is reported separately. - Sandbox benchmark, five alternating samples per mode on 10,000 files with real packing, extraction and verification: with a controlled 3-second transfer delay, median 9.450 s to 7.620 s (ranges 9.280–12.115 s and 7.296–7.983 s). With no injected delay, median 6.808 s to 6.450 s (ranges 6.255–7.029 s and 5.709–6.902 s). The no-delay ranges overlap; local copying competes with verification for disk. Both modes verify final tree contents, ignored-file exclusion and Git HEAD. These are synthetic local fixtures, not hosted agent runs. - Reproduce with `pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-repository-object-cache.ts` and `pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-workspace-staging.ts --baseline-ref=307eb16c0c --transfer-delay-ms=3000`. - Prior head `af766bc820` received Greptile 5/5 with no findings. Its E2E shard 7 failed in `plan-task-fixture` cleanup: company deletion raced an `activity_log` foreign key. That fixture has no provider run or Git workspace. The failure is retained here; final-head CI must pass. The corrected corruption fixture now forces a pack and verifies a real damaged archive. Current head `9047368bf3b7fa2061e00d698b8b51cf4ae1da32` has Greptile 5/5 with no review threads or findings. Final head has 55 passing checks and two optional Storybook skips. [The first workspace-shard attempt](https://github.com/paperclipai/paperclip/actions/runs/38802441857/job/116655485350) passed all 1,778 cases, then reported a late `EPIPE` from unchanged `server-utils.ts` while the ACP reuse test was active. That case mocks workspace preparation and passed locally in isolation. [Its single retry on the same commit](https://github.com/paperclipai/paperclip/actions/runs/38802441857/job/116659728034) passed, as did the dependent verification gate. The earlier E2E cleanup fixture passed on this final head. Both initial failures remain recorded here. A full local suite was not duplicated; the affected boundaries were run locally and the complete suite runs in CI. ## Risks - First cache population adds local copying. Cold clones, credential rotation, capacity exhaustion and a corrupt cache retain the online clone path. No eviction is added. Cached data is bounded, but many companies can each consume their allowance. - Individual filesystem calls and cleanup can exceed the cooperative priming budget. Failed reference clones make one additional online attempt. - Parallel verification and upload compete for disk and CPU. The local no-delay benchmark shows a smaller gain than a slow-transfer case. Verification still checks contents before cache publication, and required upload failures remain failures. - Local fixtures do not prove hosted throughput or the 15–30-second end-to-end target. The fresh checkout still writes its working tree, and model startup, copyback and other preparation costs remain. ## Model Used OpenAI Codex, GPT-6 family. The exact serving model ID and context window are not exposed in this session. Used reasoning, source editing, shell tools and test execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal task identifiers - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
06689c1b34 |
perf(skills): coalesce inventory before run cache locking (#16576)
Share each active company skill-inventory refresh before taking the version-cache lock. Read actor visibility under the lock and keep snapshot publication protected through durable run pinning. Preserve empty-selection behavior and the list API. Validation: 176 affected cases, full repository typecheck/build, fresh full CI/E2E, security and exact-head Apex 5/5 pass. The local six-reader latency experiment improves from 50.4s to 18.3s; live response latency is a separate acceptance check. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
8556539364 |
perf(adapter-utils): batch remote GitHub launcher staging (#16552)
Stage the nine managed GitHub launcher files through one bounded control command. Preserve per-file locks, content verification, atomic replacement and control-channel dispatch. Real Daytona staging decreased from about one second to 0.24–0.34 seconds. Full CI, E2E, Superagent and Apex 5/5 passed on the reviewed head. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
78b833c495 |
Dispatch routine and monitor work at saved deadlines (#16514)
## Thinking Path > - Paperclip manages work that agents run for a company. > - Routines and issue monitors store their next run time in the database. > - The app currently waits for the heartbeat interval to find that work. > - This can delay work after a workspace wakes before its saved deadline. > - This pull request uses the existing work coordinator to schedule each known deadline. > - It also parses each cron expression once, so sparse schedules do less work on the event loop. > - Agents can start due work sooner while the existing execution checks stay in place. ## Linked Issues or Issue Description **What happened?** Routine and issue-monitor dispatch depends on the heartbeat scan interval. A workspace can wake early and still wait for the next scan. A sparse cron search also parses the same expression for every minute it checks. In a local annual-schedule test, the deadline callback ran on time but schedule calculation delayed the routine record by 13.6 seconds. **Expected behavior** Use a saved future deadline to request the existing dispatch scan. Restore deadlines at startup and update them after a producer transaction settles. Do not let future timers keep an idle workspace active. Parse a cron expression once for each search. **Steps to reproduce** 1. Set an active routine and an assigned issue monitor to the same future deadline. 2. Choose a deadline between heartbeat scans. Use an annual routine expression to exercise a sparse search. 3. Compare the saved deadline with the routine firing and agent start times. **Paperclip version or commit** Source checkout based on |
||
|
DottaandPaperclip
|
67a9d5cc37 |
perf(runner): bypass skill cache waits for empty selections (#16498)
Native runs with no skill files no longer queue behind company skill-cache work. Preserve saved empty selections and the existing exclusion for selected and legacy operational skills. Verified with real PostgreSQL held-lock regressions, 20 distinct affected cases, full repository typecheck/build, fresh full CI/E2E/security and exact-head Apex 5/5. Hosted latency remains to be verified. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
Dotta
|
f1d925b726 |
fix(issues): keep workspace restoration failures retryable (#16489)
## Thinking Path > - Paperclip manages agent work and its saved files. > - Finished Git workspaces can expire and restore from a saved commit. > - A task comment must restore a closed workspace before the comment is saved. > - Failed restoration returned HTTP 503 and hid the restoration message. > - The composer treats 5xx writes as uncertain, so it blocked the draft instead of offering a clear retry. > - This change returns a definite rejection before the write and preserves the restoration message. > - Users can restore repository access and retry the same draft. ## Linked Issues or Issue Description Refs #16457 and #16476. Live acceptance found this gap with a disposable expired workspace and an unavailable upstream. The saved commit and unique-file control stayed intact. No fallback workspace or comment was created. The UI still showed an uncertain comment result because the route returned 503. ## What Changed - Return HTTP 409 for a rejected workspace rebuild before the consuming write. - Preserve the service's safe restoration message in the response. - Cover rejected comment, comment-update, and checkout paths. - Document the response and draft retry behavior. ## Verification - `pnpm exec vitest run server/src/__tests__/issue-closed-workspace-routes.test.ts ui/src/api/issues.test.ts`: 38 passed in 26.80s. - Route tests assert that failed restoration does not save a comment or update the task. - Existing client controls preserve 409 rejections and still treat network failures and 5xx writes as uncertain. - `pnpm -r typecheck` and `pnpm build` passed. Full PR CI passed on attempt 2: [run 38804449401](https://github.com/paperclipai/paperclip/actions/runs/38804449401). The first attempt timed out in the unrelated fresh personal-AI connection browser case while waiting for its first reply. Logs and browser evidence were inspected and retained; only failed jobs were rerun once, and that shard passed. - Retained live fixture will verify the error, retry, exact saved commit, and subsequent persisted work after deployment. ## Risks Clients now receive 409 instead of 503 for this known pre-write rejection. They must retry after repository access is restored. The route uses the service's fixed messages, not raw Git errors. This change does not alter restoration, authorization, cleanup, or genuine uncertain-write handling. ## Model Used OpenAI Codex, GPT-6 family, with code execution and browser tools. The exact deployment model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge |
||
|
DottaandPaperclip
|
59a70dc9e2 |
fix(workspaces): expire reproducible Git files and restore exact commits (#16457)
## Thinking Path > - Paperclip manages AI agents and their task workspaces. > - Isolated Git worktrees and private repository copies remain on the host after tasks finish. > - These files can fill the disk even when every commit is available upstream. > - Cleanup must retain unique files, local Git state, live work, and recovery data. > - This pull request records and verifies an exact-commit restoration recipe before expiration. > - Normal reopening restores that commit even after upstream advances. > - The benefit is lower host disk use while finished tasks remain available for later work. ## Linked Issues or Issue Description Refs #10285 and #2022. Follows #16433 for skill-cache expiration. Related #11936 provides separate host-local cleanup tooling; this change extends the existing company-scoped lifecycle worker and normal workspace reopening. ROADMAP.md does not specify a competing implementation. ## What Changed - Add opt-in expiration for verified Paperclip-owned Git worktrees after their whole task tree has been done or cancelled for 24 hours. - Preserve active and scheduled runs, previews, pending reopening, unresolved recovery, dirty files, unknown ignored files, and unsupported or unique Git state. - Verify an authenticated fetch in a fresh object store. Save repository identity, exact commit, branch, and ownership before any deletion. - Rename eligible files into recorded managed trash. Resume interrupted cleanup under source and lifecycle locks. - Expire private base clones only after consumer, file, reference, reflog, and object checks pass. Preserve shared project base clones. - Rebuild missing bases and worktrees through normal reopening at the recorded commit. Return an explicit retryable error when repository access fails. - Add independent workspace/base switches, dry-run reports, reclaimed-byte counts, and operational documentation. No table, public API, UI, or recovery-retention change. ## Verification - Real local bare remote plus Postgres: create at A, advance upstream to B, expire the worktree and private base, reopen at A, then make and retain a subsequent commit. - Final lifecycle suite: all 20 tests passed in 46.41 seconds at `d72947a2e45be5d0c912debb03c46b15863b4798`. Review fixes cover second-cycle expiration, newer-branch recovery, normal preparation after restoration, branch adoption, and Git administration path controls. Eleven existing branch reconciliation controls also passed. Covers unique files and Git state, live descendants, scheduled runs, recovery, previews, unauthorized access, upstream failure, concurrent reopening, interrupted deletion, shared bases, and nested private worktrees. - Final guard regressions: 3 affected tests passed. Existing workspace/runtime/preparation/source controls: 328 tests passed in 238.69 seconds. - Full repository typecheck and build passed locally. The full local test attempt did not pass: initial native-session cases hit macOS Postgres shared-memory exhaustion, source changes invalidated two discovery cases, and the checkout disappeared near the end. The final-code retry of the remaining skill and unrelated controls passed all 103 tests. All checks passed on this exact head, including typecheck, general and serialized tests, Runner checks, build, canary dry-run, browser E2E, and security scans. Greptile reviewed the head at 5/5 with all findings resolved. - Fixed a blocking CI policy false positive for `docker-compose-secrets.test.mjs`. The test still rejects GitHub secret expressions, including dot, bracket, and multiline forms. Its full 15-test suite passes. - Map validation and `git diff --check` passed. - Staging dry-run inspection, measured reclamation, and ordinary product acceptance remain pending. No live acceptance is claimed. ## Risks - Git restoration depends on continued upstream availability and credentials. Cleanup skips unverified commits; restoration never substitutes the default branch. - File deletion requires multiple conservative proofs. Unknown or unsupported state is retained and can limit reclamation. - Source and lifecycle locks serialize preparation, reopening, and expiration. Remote verification has a deadline but can delay work on the same source. - Start with both expiration switches disabled. When enabled, dry-run remains the default. Rollback disables further deletion while retaining restoration support. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository inspection, tool use, and code execution. The exact backend model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (focused tests; full local limitation documented above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
68a09dcb5f |
perf(ci): shorten the test execution critical path (#16434)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Contributors wait for the full PR workflow before they can deliver fixes. > - Server shards still execute independent files in series, and a single Runner file serializes two long real-clock waits. > - Faster work outside the critical path does not prove a faster workflow. > - Overlap independent files while preserving isolation, test cases and real production deadlines. > - Compare complete baseline and candidate workflows, separating queue delay from execution cost. ## Linked Issues or Issue Description Refs #16429. Related duration refresh: #12577. **What existing behavior does this improve?** The elapsed time from PR CI creation to the last required check. **Current behavior** General server shards use one worker. One Runner transport file takes about 333 seconds on Linux, including over 100 seconds in two real-clock regressions. These are the longest verification lanes after #16429. **Proposed behavior** Run independent general-server files in at most two isolated forks. Move the two intact Runner deadline regressions into a separate file to overlap their waits in the existing pool. Run two chat partitions per existing job, each with its own temporary home and embedded PostgreSQL. Preserve serial execution within files and every assertion and timing boundary. ## What Changed - Qualify both serial and parallel forks with real worker processes, isolated homes/configuration, timers, globals and environment. - Require both parallel workers to arrive before either exits. - Use two isolated forks for sharded general-server tests; retain one worker with an explicit shared test database. Other test groups keep their existing concurrency. - Extract the lease-expiry and Pi cold-admission regressions verbatim into a separate Runner test file; retain real waits and process/lease assertions. - Split each sharded chat job into two independent invocations; preserve serial shared-database execution. Await both children before failing the parent. Prove disjoint coverage and failure propagation. - Update the paired Linux benchmark to compare a complete general-server shard with identical case inventories. - Document concurrency and deadline-test placement. No production behavior, workflow topology or runner count changes. - Allow 15 seconds for snapshot integration cases that cold-start multiple real TypeScript workers. On the repeat, one-worker cases took 2.5–3 seconds, so queued or sequential starts could exceed the default five seconds. Preserve all cleanup, queue-release and filesystem assertions; these cases do not enforce a production latency deadline. ## Verification All 48 jobs passed on scheduling head `5d718e18` in [attempt 1](https://github.com/paperclipai/paperclip/actions/runs/38763885288/attempts/1). The [bounded repeat](https://github.com/paperclipai/paperclip/actions/runs/38763885288/attempts/2) failed two existing snapshot-worker tests at their default five-second timeout. Its shorter duration is **not** counted as a successful performance sample. The latest head `3c894462` adds a targeted test-budget repair and [passes all 48 CI jobs](https://github.com/paperclipai/paperclip/actions/runs/38768433532) in **8m06s** (workflow creation to final required workflow job). This is **70 seconds / 12.6% shorter** than the 9m16s intermediate run, but the earlier 9m23s result remains part of the evidence: a stable average improvement is not established. Both branch-protection checks (`ci / verify` and `ci / e2e`) pass. The non-required Superagent Security Scan is still running; external PR scans are separate from this workflow duration. | Measurement | Before | After | What it establishes | | --- | ---: | ---: | --- | | General-server shard, paired on one Linux host | 430.94s | 292.99s | 32.0% faster; identical 1,047 case inventory | | Chat job 1 | 460s | 211s | Same full chat inventory spread across isolated partitions | | Chat job 2 | 408s | 274s | Serial cases within each child, separate PostgreSQL instances | | Chat job 3 | 408s | 216s | Total chat runner time fell from 1,276s to 701s | | Full PR workflow, first chat candidate | 9m16s | 9m23s | Not an end-to-end win in this pair: unchanged Build rose from 365s to 465s | | Full PR workflow, final repaired head | 9m16s | 8m06s | All 48 jobs pass; 70 seconds faster observed, not a controlled causal estimate | - Latest [paired benchmark](https://github.com/paperclipai/paperclip/actions/runs/38763885223) passes. Earlier paired server measurements were 33.3% and 37.4% faster. Unchanged fixture timing noise is not attributed to this PR. - All **1,337 chat case identities and outcomes match exactly** across [before-chat](https://github.com/paperclipai/paperclip/actions/runs/38760937988) and [after-chat](https://github.com/paperclipai/paperclip/actions/runs/38763885288/attempts/1), without duplicates or removals. Whole-suite records rose from 44,563 to 44,574: three new launcher checks and eight upstream warm-owner tests. Remaining name differences were generated fixture IDs/timestamps/PIDs, not removed tests. - The final run used `m7i-flex.xlarge` for Build (408s), with all chat jobs completing in 238s / 318s / 280s. It used the same actual merge base `afe5b341` as the first chat candidate. Optional profiling was disabled after the bounded measurement experiment; the full 48-job correctness workflow still ran. - The fleet used `m7i.xlarge` for the earlier Build and `m6a.xlarge` for the first chat candidate Build. Several unchanged compile/install/probe phases slowed together. This does not prove hardware alone caused the difference, and full-run attribution must account for runner/cache variance and the upstream base change. - Original full baseline: [all 48 jobs passed](https://github.com/paperclipai/paperclip/actions/runs/38753547876/attempts/2), 12m07s, including a server job that queued over five minutes. It is not a clean basis for attributing the whole elapsed-time reduction to this PR. An interrupted baseline and two misrouted candidate attempts were excluded and preserved; routing/trust rules were not weakened. - Local: 32 sharding/launcher checks pass, including disjoint coverage, shared-database fallback and a failed child that waits for its sibling. Real paired chat partitions passed 416 cases in 217.63s on independent embedded databases. - Both extracted Runner deadline regressions pass unchanged with real waits. An initial local attempt rejected a Node executable under the controller home; the identical binary in an owned non-home path passed without weakening that guard. - Real worker-isolation proof, actionlint and diff checks pass. Latest-head Greptile is **5/5**, with no review threads or actionable findings. - The eight snapshot-worker cases pass locally with a private `PAPERCLIP_HOME`. The first direct local invocation failed to open the default-home lock database under the sandbox; no production guard was changed. ## Risks Parallel files and paired chat databases may expose shared-state assumptions or increase peak memory. Cases stay serial within each invocation, and explicit shared databases keep the serial path. Complete server coverage and the full 48-job CI workflow pass. Check external required PR scans before merge; the full-workflow measurements are observational and do not establish a steady 12.6% speedup. Queueing, cache state and runner variance must be distinguished from execution improvements. ## Model Used OpenAI GPT-6 through Codex, with repository inspection, code execution and GitHub tools. The exact serving model suffix and context capacity are not exposed to this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green (48 jobs; both required branch-protection checks pass; non-required external security scan pending) - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |