mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-19 20:44:16 +02:00
8f1cb11a034b82e93fa414f8e77beeb702cc14e8
1091
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
8f1cb11a03 |
fix(workspaces): name the owner action when a shared source is restricted (#16545)
A reused task workspace in a shared project checkout keeps its source denial when another task's restricted worktree shares that checkout (#16095 deliberately does not clone over existing workspaces). The run then failed as an opaque `setup_failed` 403 and the task was blocked with a generic "no live execution path" notice. Convert that exact denial into a `configuration_incomplete` failure with reason `workspace_source_restricted`, give the task notice and the recovery action the owner step, and keep this pre-dispatch owner action out of the unexpected Sentry run-failure stream. The reason and predicate live in a dependency-free module. Access checks are unchanged. Co-Authored-By: Paperclip <Paperclip-Paperclip@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
f1d4bc1617 |
perf(slack): resolve only signing secret during webhook checks (#16548)
Resolve only the current Slack signing secret during workspace checks and authenticated setup evidence. Preserve full runtime credential resolution, company bounds, rotation, and auditing. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
78b833c495 |
Dispatch routine and monitor work at saved deadlines (#16514)
## Thinking Path > - Paperclip manages work that agents run for a company. > - Routines and issue monitors store their next run time in the database. > - The app currently waits for the heartbeat interval to find that work. > - This can delay work after a workspace wakes before its saved deadline. > - This pull request uses the existing work coordinator to schedule each known deadline. > - It also parses each cron expression once, so sparse schedules do less work on the event loop. > - Agents can start due work sooner while the existing execution checks stay in place. ## Linked Issues or Issue Description **What happened?** Routine and issue-monitor dispatch depends on the heartbeat scan interval. A workspace can wake early and still wait for the next scan. A sparse cron search also parses the same expression for every minute it checks. In a local annual-schedule test, the deadline callback ran on time but schedule calculation delayed the routine record by 13.6 seconds. **Expected behavior** Use a saved future deadline to request the existing dispatch scan. Restore deadlines at startup and update them after a producer transaction settles. Do not let future timers keep an idle workspace active. Parse a cron expression once for each search. **Steps to reproduce** 1. Set an active routine and an assigned issue monitor to the same future deadline. 2. Choose a deadline between heartbeat scans. Use an annual routine expression to exercise a sparse search. 3. Compare the saved deadline with the routine firing and agent start times. **Paperclip version or commit** Source checkout based on |
||
|
DottaandPaperclip
|
0e58f29eb7 |
fix(observability): trace pre-dispatch controller waits (#16515)
Measure usage startup, MCP handoff, dispatch admission, and budget reservation with the existing endpoint-gated tracer. Preserve outer attempt context across asynchronous provider dispatch.
Validated with 135 focused tests, full typecheck/build, fresh full CI/E2E, Superagent, and Apex 5/5 on
|
||
|
Devin FoleyandPaperclip
|
c7ea5c0216 |
perf(server): skip settled committed finalizations in unscoped passes (#16485)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - On restart, `/api/health` reports `starting` until startup recovery finishes, and deploys wait for it > - Native recovery and the orphan reap each run an unscoped native finalization reconciliation pass, and readiness waits for both > - The pass selects every committed finalization with a result, including runs that ended long ago, and spends dozens of sequential queries on each one > - On a busy instance this is about one minute per pass, so about two minutes of every restart, and the periodic recovery repeats it on every tick > - This pull request skips settled committed finalizations in unscoped passes, and keeps a small rotating upkeep page so their upkeep still runs > - The benefit is faster restarts and less steady database load, with no change for unsettled work ## Linked Issues or Issue Description Related: #16330 and #16329 (startup and reconciliation timing). **What existing behavior does this improve?** Startup recovery time and steady-state recovery load. **Current behavior** - The unscoped candidate scan in `reconcileNativeFinalizations` returns every native finalization with a result and phase other than `terminal_failure`. Nothing excludes rows that are already settled. - Each committed row then takes the full per-row path: pending-effect, assessment, evidence and decision reads, presentation repair (twice), the workspace barrier read, a `finalizeNativeRun` re-read of the committed coordinator, and workspace settlement (lease release check and agent status update). That's about 40–60 sequential round trips per row. - Startup runs the unscoped pass twice (native recovery, then the orphan reap), and readiness waits for both. The periodic orphan reap runs it again on every tick. - On one instance, all 268 candidates were committed finalizations whose runs had ended. Native recovery took 50–80 s and the orphan reap 47–80 s on every restart, with database round trips of about 3.4 ms. **Proposed behavior** An unscoped pass skips a committed finalization when: - its run is `succeeded`, `failed`, `cancelled` or `timed_out`, and - the run finished more than an hour ago, and - its decision has no `pending` or `failed` status-decision effect. Each unscoped pass also includes up to 10 such settled rows in a rotating order, so the presentation and evidence upkeep still reaches every row over time. Scoped passes (explicit run IDs) are unchanged. **Reason and benefit** These settled rows have nothing left to finish. Recent rows, unfinished runs and rows with undelivered effects stay in every pass, so crash-window repairs and effect delivery behave as before. Startup and periodic passes no longer scale with the full history of committed runs. **Breaking changes** None. A late repair on a settled row can be delayed until its turn in the upkeep rotation instead of the next pass. ## What Changed - `server/src/services/native-runtime/native-finalization-reconciler.ts`: - `settledCommittedFinalization(now)`: the settled predicate above, including a `NOT EXISTS` on undelivered `status_decision_effects` for the same company, issue and decision. - The unscoped candidate scan adds `NOT settled OR run in the upkeep page`. - `nextSettledCommittedUpkeep`: up to `NATIVE_SETTLED_COMMITTED_UPKEEP_PER_PASS` (10) settled run IDs after an in-process cursor, ordered by run ID, wrapping at the end. - New constants `NATIVE_SETTLED_COMMITTED_GRACE_MS` (1 hour) and `NATIVE_SETTLED_COMMITTED_UPKEEP_PER_PASS`, plus `resetSettledCommittedUpkeepForTests`. - New `server/src/__tests__/native-finalization-settled-candidates.test.ts`. - `doc/execution-semantics.md`: describes the settled rule and the upkeep rotation. ## Verification - `pnpm exec vitest run src/__tests__/native-finalization-settled-candidates.test.ts` (in `server/`): 4 passed. The test mocks the timing hook to capture the candidate count and stop the pass after the scan: - 25 settled rows give 10, 10, 5, then 10 again over four passes (rotation and wrap). - A row finished 10 minutes ago, a running row, and a terminal row without `finished_at` are all candidates. - A settled row whose decision has a `failed` effect is a candidate. - A scoped pass with an explicit run ID still selects a settled row. - `pnpm exec vitest run` for `native-finalization-recovery`, `legacy-finalization-regression`, `native-run-finalizer-telemetry`, `native-workspace-finalizer`, `native-finalization-reconciler-timing`, `external-chat-wait.integration` and `native-status-arbiter-corpus`: 316 passed. - `heartbeat/recovery.test.ts`: the 8 tests that do not need a database passed. The 20 database tests were skipped locally because embedded PostgreSQL could not start (the host's shared-memory segments were exhausted by other processes). CI runs them. - `pnpm typecheck` in `server/`: passed. ## Risks - A settled row that needs a late repair (for example a missing chat presentation) waits for its turn in the upkeep rotation, up to `ceil(settled rows / 10)` passes, instead of the next pass. - The upkeep cursor is in memory. After a restart the rotation starts from the beginning; the work is idempotent. - A run left `running` with a committed finalization is never considered settled, so it is still projected before the reaper's active-run loop. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code with tool use (shell, file edits, test runs, database and log analysis). A separate Claude agent mapped the recovery call tree. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Huanyi Xie
|
c4a16cbe98 |
fix(plugins): follow local package path changes in dev watcher (#15247)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Local plugin development uses file changes to restart the plugin worker. > - A plugin can move to a different local package directory. > - The watcher rejected every later registration for the same plugin ID. > - This pull request replaces the watcher when the resolved package path changes. > - The benefit is that current plugin files trigger reloads and previous files stop doing so. ## Linked Issues or Issue Description Refs: #14955. This change addresses its second observation: the watcher keeps the previous local directory after its package path changes. The issue also tracks stored configuration replay as a separate problem. Related PRs: #11336 shares the lifecycle manager across routes and the dispatcher. #14381 controls when production file watching is enabled. #10100 orders configuration delivery. Those changes address other parts of plugin reload behavior. ## What Changed - Store the resolved package path with each watcher. Keep repeated registrations of the same path as a no-op. Replace the watcher for a different path. - Cancel pending changes from the previous directory. Ignore late change, error, and timer callbacks from a replaced watcher. - Apply only the latest pending directory lookup. Manual registration, unwatching, and shutdown invalidate earlier lookups. - Stop a failed watcher without cancelling its pending replacement lookup. Explicit unwatching, disable, unload, and shutdown still cancel pending registrations. - Stop the old watcher when the new directory is unavailable or the plugin no longer has a local package path. - Document the path-change behavior and add twelve regression cases. ## Verification - `pnpm exec vitest run server/src/__tests__/plugin-dev-watcher.test.ts server/src/__tests__/plugin-lifecycle-restart.test.ts`: 18 tests passed. Regressions cover path changes, stale callbacks, lookup order, explicit cancellation, and watcher errors during a pending replacement. The current-watcher error case fails before the stop-versus-cancel fix; the stale-watcher error control passes. - A local smoke check used real Chokidar watchers and two real temporary plugin directories. It confirmed that a pending old change is cancelled, later old-directory changes are ignored, and a new-directory change triggers one restart. - `pnpm -r typecheck` passed on the path and lookup guards. The final watcher-error fix passed `pnpm --filter @paperclipai/server exec tsc --noEmit`. - `pnpm build` passed on the initial path-change fix. The final source passed `pnpm --filter @paperclipai/server build`. - `git diff --check`: passed. - Greptile reviewed `bef3182` at 5/5 with no new findings. The watcher-error review thread is resolved. CI is still running. - The full `pnpm test:run` suite was not repeated for this watcher-only change. The affected suites ran locally. The broad test shards will run in CI. ## Risks - Low risk. The watcher follows the current registered path. An unavailable replacement stops watching until another valid registration arrives. - Closing a filesystem watcher is asynchronous. Callback identity checks prevent an old watcher from acting on its replacement. - No database migration, API shape, or dependency change is required. ## Model Used - OpenAI Codex based on GPT-6, with reasoning, tool use, and code execution. The runtime does not expose the exact serving model ID or context window. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge |
||
|
DottaandPaperclip
|
becdc3b08e |
perf(slack): remove delays before mention acknowledgements (#16491)
Slack mentions now start eyes reactions after the authorized task/comment commits, independently of attachments, subscription and scheduler acceptance. Omit unused optional URL-preview polling while retaining message text, links and files. Keep receipt failures and missing-file warnings independent, including terminal cleanup of older multi-line diagnostics. Verified with full current-head CI and E2E, Superagent and legacy security checks, Apex 5/5 with no unresolved findings, 19 selected real-PostgreSQL receipt/authorization/cleanup cases, and full frozen-source typecheck/build. The first CI attempt's unchanged Cursor probe timeout is retained in the PR; the one failed-only retry passed under unchanged deadlines. Live Slack latency still requires deployed acceptance. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
|
81cea89d6b |
fix(cli): require confirmation for instructions file deletes (#11504)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The CLI exposes board/operator controls for managing agent configuration and instructions. > - Agent instructions files influence future agent behavior, so deleting one is a destructive configuration mutation. > - Nearby destructive CLI commands already require explicit `--yes` confirmation before sending DELETE requests. > - `agent instructions-file:delete` skipped that local confirmation guard and could delete a bundle file from a scripted or mistyped command. > - This pull request aligns that command with the existing destructive-command safety convention. > - The benefit is a small, predictable operator-safety guard with focused regression coverage. ## Linked Issues or Issue Description Bug report: ### Pre-submission checklist - [x] I have searched existing open and closed issues and this is not a duplicate. - [x] I am on the latest released version of Paperclip (or can reproduce on `master`). - [x] I have confirmed the error originates in Paperclip itself — not in my agent adapter, API provider, or local configuration. ### What happened? `paperclipai agent instructions-file:delete` deleted an agent instructions bundle file without requiring explicit CLI confirmation. ### Expected behavior Destructive CLI delete operations should require `--yes` before sending a DELETE request. ### Steps to reproduce 1. Run `npx paperclipai agent instructions-file:delete <agent-id> --path AGENTS.md`. 2. Observe that the CLI immediately issues `DELETE /api/agents/:agentId/instructions-bundle/file?path=AGENTS.md`. 3. No local confirmation guard blocks the destructive request. ### Paperclip version or commit `fd472d02ba950519ca5eff75ed9fd55cf976f4c2` ### Deployment mode Local dev (`pnpm dev`) ### Installation method Built from source (`pnpm dev` / `pnpm build`) ### Agent adapter(s) involved - [x] Not adapter-specific (core bug) ### Database mode Not database-related ### Access context Board (human operator) ### Node.js version `v20.x` ### Operating system macOS ### Relevant logs or output ```text Current behavior is local CLI behavior: the command proceeds to the DELETE call without requiring `--yes`. ``` ### Relevant config (if applicable) None. ### Additional context This command already sits next to other destructive CLI actions such as `agent delete`, which require explicit confirmation. Aligning the guard makes the CLI safety model consistent. ### Privacy checklist - [x] I have reviewed all pasted output for PII (usernames, file paths, API keys, tokens, company names) and redacted where necessary. ## What Changed - Added a `--yes` confirmation option to `agent instructions-file:delete`. - Refuse the command locally before resolving API context or sending a DELETE request when confirmation is missing. - Updated CLI parity coverage for confirmed and refused instructions-file deletion. - Updated the CLI documentation example to include `--yes`. ## Verification - `/Users/xinran/Downloads/dev/paperclip-pr-work/node_modules/.bin/vitest run cli/src/__tests__/agent-lifecycle.test.ts --config cli/vitest.config.ts` equivalent executed from the PR worktree using temporary dependency symlinks: 3 tests passed. - `git diff --check` ## Risks - Low risk: this only changes one destructive CLI command and matches existing `agent delete`, `issue delete`, and other delete-command confirmation behavior. - Scripts that intentionally delete instructions files must add `--yes`, which is the intended explicit confirmation path. - No database, API contract, or lockfile changes. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - OpenAI Codex, GPT-5-based coding agent in Codex desktop with shell/tool use and local test execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: 馨冉 <xinxincui239@gmail.com> Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
67a9d5cc37 |
perf(runner): bypass skill cache waits for empty selections (#16498)
Native runs with no skill files no longer queue behind company skill-cache work. Preserve saved empty selections and the existing exclusion for selected and legacy operational skills. Verified with real PostgreSQL held-lock regressions, 20 distinct affected cases, full repository typecheck/build, fresh full CI/E2E/security and exact-head Apex 5/5. Hosted latency remains to be verified. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandPaperclip
|
ec3eb1968b |
feat(runner): expose credential updates through PRP (#16463)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The Runner uses PRP to send harness events to the host. > - A harness can replace its stored OAuth tokens during a run. > - The host has no PRP signal that tells it to read the updated credential. > - This pull request adds a durable credential update event and observers for supported stores. > - The host can react before it acknowledges the event. Token values stay outside PRP. ## Linked Issues or Issue Description **Subsystem affected** `packages/paperclip-runner`: native provider execution, PRP schemas, and host event delivery. **Problem or motivation** A Paperclip host that consumes PRP needs to react when a harness changes its refresh-token credential. Existing credential copy-back paths do not provide a common event during a warm session. **Proposed solution** Add `auth.credentials.updated` as a P0 runner event. Include only the provider, binding ID, revision, and detection method. Observe native Codex and Codex/Grok ACPX credential files. Retain each pending event across runner restart until the durable outbox accepts it. Raw PRP consumers opt in with `observeCredentials: true` before session open. The bundled transport opts in for fresh sessions. **Alternatives considered** `harness.diagnostic` would mix a runner observation with harness diagnostics. Sending token values through PRP would expose credentials to event logs. Both approaches are unnecessary. **Roadmap alignment** I checked `ROADMAP.md`. This change extends the existing Runner protocol and does not duplicate a planned core feature. **Additional context** Refs: #11094, #14056, #12696. These describe credential copy-back and staging work. This PR adds the notification contract. It does not replace those private credential channels or claim to fix their full scope. ## What Changed - Add strict credential update payload validation to PRP v1, v2, and v3. Regenerate validators, the manifest, and the event catalog. - Watch supported credential directories with `notify`. Add a periodic read as a fallback and check at terminal and close boundaries. - Persist a private baseline and pending event together. Notify on account identity changes and retain the new baseline for later refreshes. Ignore incomplete files, unsafe files, and metadata changes. - Forward the event through the host transport. Test commit-before-ACK delivery and replay. - Commit pending runner observations before suspend and shutdown commands complete. Preserve the provider event backlog. - Create the private ACPX runtime directory before configuring observation. Cover fresh session startup with observation enabled. - Fence asynchronous PRP commits to the current authenticated connection and run authority. Ignore replaced callbacks while preserving the maintenance drain of admitted work. - Deliver observations found during final provider cleanup before clearing terminal recovery state. Retain the outbox and recovery state when the host ACK is lost. - Advance the pending Copilot candidate profile to v19 for the changed shared PRP source hashes. Preserve its v18 history and pending qualification status. Include the current fixture in the Docker build context. - Commit credential updates through an awaited host callback before PRP ACK, including during idle and close. Persist them in a separate run-log source namespace with company, agent, session, and runner checks. - Preserve public event metadata through server redaction. Register the event in task run details. - Document opt-in behavior, supported stores, event semantics, and the local run-log path. ## Verification - All 45 durable runner control tests and 24 native executor integration tests pass. - Ten Rust credential observer tests pass. They cover atomic replacement, polling, recovery, account-only changes and later refreshes for Codex and Grok, unsafe files, fresh runtime creation, and the Grok close handoff. Account identity stays private. - The full Runner TypeScript suite passes: 3,565 tests in 221 files. An additional 12 tests are skipped by the suite. - All 214 focused tests pass for the contract, Codex event routing, catalog, and durable host delivery. - All 13 Node protocol contract tests pass. - The exact Docker context integrity command passes: `docker buildx build --file .github/docker-context-checks.Dockerfile .`. - Rust formatting and generated protocol checks pass. - `pnpm -r typecheck` and `pnpm build` pass. The full local `pnpm test:run` was attempted with CI-compatible `init.defaultBranch=master`, then stopped after a reproducible failure outside this change: `git-workspace-expiration.test.ts` expects one re-expired workspace but gets zero. The same case fails in isolation on this macOS checkout. Full repository validation is provided by CI; the focused Runner and host tests pass locally. - All 253 native transport and restart tests pass against the final release binary. - All 3 durable recovery tests and 139 candidate profile and native admission tests pass. - All 41 host redaction and task protocol surface tests pass. `pnpm check:token-gates` passes. - All 467 CI policy tests pass, including the 15 CI profiling tests. The upstream secret-reference test fix is retained from master. No workflow changes are included. - The Apex host-delivery fix passes 381 targeted Runner tests, six host database tests, repository typecheck, and the production build. The Runner tests cover commit-before-ACK, retry after a rejected host commit, warm attachment, and close after the turn queue ends. - Prior head `2f125cfcb` passed CI, standard Greptile 5/5, and Apex 5/5. One CI shard was interrupted by GitHub runner shutdown; its single rerun passed. Both Apex findings are addressed: idle/cleanup host delivery and late callback reconnect fencing. Current head `5db78f082` passes CI (56 successful statuses and four optional skips) and standard Greptile 5/5. Apex review of the account-change update is 5/5 on this exact head, with no actionable issues. - All 121 durable control-plane and credential callback tests pass, including account-only change delivery, subsequent refresh on the new account during close, host commit retry, late success/failure races, forward cursor delivery on a replacement connection, and the existing maintenance retirement contract. All 27 native provider and durable recovery integration tests pass after the account-change update. ## Risks - File changes signal a stored credential update. They do not prove that an OAuth refresh succeeded. Changes between reads can be combined. - Unix Codex and Codex/Grok ACPX stores are supported. Other harnesses and credentials held only in memory need their own integration. - Hosts must update their event schemas before opting in. Older hosts omit the flag and receive no new event. Older runners ignore the flag. - The host must use its existing private credential channel and account checks to read or copy the token. Account switches trigger notification but do not authorize accepting the new account. The event grants no credential access. - Tests use synthetic credentials. No live OAuth refresh has been forced. ## Model Used OpenAI Codex, exact model ID `gpt-6.1-sol`, with high reasoning effort, repository inspection, tool use, and code execution. The session does not expose its context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
8843887c24 |
fix(runner): preserve managed GitHub access in OpenCode (#16488)
## Thinking Path > - Paperclip manages AI agents and their authorized connections. > - Native OpenCode agents use managed Git and GitHub launchers for shell work. > - Each operation must obtain credentials from the active run's broker. > - Two OpenCode environment filters dropped that broker binding before the shell started. > - The shell therefore ran anonymously even when connector tools had authorized access. > - This pull request preserves the explicit broker environment through both filters. > - The benefit is working GitHub shell access with current identity and revocation checks. ## Linked Issues or Issue Description **What happened?** Native OpenCode shell commands reported `GitHub capability_missing` despite an authorized GitHub connection. Both runner startup and provider startup dropped the controller's broker settings. Connector MCP calls used a separate path and could still succeed. **Expected behavior** A managed shell operation obtains credentials for the current run. A later operation uses the new identity after rotation and loses access after revocation. Ambient host credentials do not supply authority. **Steps to reproduce** 1. Start the native GitHub broker with an active run binding. 2. Pass its environment through native OpenCode startup. 3. Run `gh api user` from the provider shell. 4. Observe anonymous access before this fix. **Paperclip version or commit** Reproduced on master `6943904bad` before this change. **Deployment mode** Native Paperclip Runner with OpenCode, including remote execution. Related: #16401 fixes the separate oversized OpenCode SSE failure and is already merged. The same live-driver regression was verified locally against its exact pre-fix source and current source. This PR does not duplicate that fix. Searches found no open PR for this native broker propagation defect; #15309 concerns the legacy OpenCode adapter. ## What Changed - Use the existing GitHub credential environment projection at both OpenCode launch boundaries. - Preserve explicit controller authority while excluding an ambient host broker. - Exercise the real launcher and broker from an OpenCode child process, directly and through runner projection. - Check operation-time identity rotation, revocation, and exclusion of controller API credentials and persistent GitHub tokens. - Document the managed OpenCode shell contract. ## Verification - Red: both new `native-github-access.test.ts` cases returned `anonymous` instead of `fixture-a` before production changes. - Green: both cases pass after the fix, including `fixture-a` to `fixture-b` rotation and anonymous operation after revocation. - Full native GitHub broker suite: 18 tests pass. - OpenCode, provider-event, and configured-environment suites: 141 tests pass across eight files. - Separate SSE red/green verification: the existing live-driver regression times out on `fd057f6ac4^` and passes with merged #16401. The full OpenCode group also passes after restoration. - Full repository build and typecheck pass locally. All CI jobs pass for `23d734293f6a5c6d6128cae0c8773661816daf9a`. Final GitHub rollup: 55 successful checks/statuses, two skipped checks, and one neutral check; none pending or failing. Greptile gave that commit 5/5 with no findings or unresolved threads. - Paperclip Review spent over 23 minutes preparing its runtime without a provider assessment. Its run was cancelled for recovery and ended neutral (review execution did not complete). This neutral result is not a code assessment. No check policy or branch protection was changed. - The full local Vitest attempt first hit embedded PostgreSQL startup failures. Running the unchanged `initdb --version` and `postgres --version` independently showed missing native library symlinks. The dependency had been installed with scripts disabled. Running its bundled `hydrate-symlinks.js` restored startup; all 15 affected native-session cases and all 112 AI connection tests then passed without application code changes. The original full-suite attempt was stopped after the setup failure and its log is retained. A complete local suite pass is not claimed. - The broker and executable are local fixtures. No live GitHub account, credential change, or test PR is required by these regressions. ## Risks The change affects credential delivery into OpenCode processes. It reuses the existing bounded GitHub projection and accepts broker authority only from explicit input. Each shell operation still uses the normal broker, scope, identity, and revocation rules. Unrelated controller secrets remain excluded. There are no database or permission-policy changes. ## Model Used OpenAI `gpt-6.1-sol`, through Codex, with xhigh reasoning and repository tools/code execution. The session does not report its context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run the focused regression and affected suites locally and they pass; the full local attempt and setup failure are disclosed above - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green (Paperclip Review is neutral/incomplete, as disclosed above) - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
0b62c736a0 |
feat(connections): add generic Daytona API and Toolbox access (#16483)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Apps connections give agents controlled access to external services. > - Daytona provides sandbox, file, command, Git, and account APIs. > - Operators need to assign this access without exposing a server credential. > - A small task-specific tool list cannot serve general Daytona work. > - This pull request adds the published HTTP actions through the existing connection gateway. > - Operators keep control of identity, agent access, and each action. ## Linked Issues or Issue Description **Problem or motivation** Paperclip has a Daytona sandbox provider for its own execution infrastructure. It lacks a normal Apps connection for agents to manage an operator's Daytona account. Reusing the server's provider credential would bypass normal connection ownership and action permissions. **Proposed solution** Add a Daytona connection with personal or shared vault credentials. Generate its action catalog from the official platform and Toolbox API specifications. Use the existing run-bound gateway, grants, action policies, approvals, revocation, and audit paths. Sandbox IDs come from Daytona; tools need no Paperclip task or lease ID. **Alternatives considered** A four-action diagnostic subset cannot support general provider work. Launching the provider CLI would expose host filesystem behavior. Native OAuth is not included because a supported third-party registration path was not verified. The native method uses an organization API key. Existing Composio setup is a separate possible OAuth route and is not qualified by this change. **Roadmap alignment** This extends the existing Apps and Connections provider coverage in `ROADMAP.md`. It uses that system's credential and permission model. GitHub searches found no existing Daytona Apps connector PR or issue. ## What Changed - Add the Daytona catalog entry, provider source override, official theme logos, permission review, and setup documentation. - Generate 274 HTTP actions: 169 platform and 105 Toolbox operations. Record two WebSocket upgrades separately; do not advertise them as HTTP tools. Record source URLs and hashes. - Add provider-focused execution for sandboxes, snapshots, volumes, files, Git, commands, computer use, and account resources. Provider authority still limits which endpoints accept an organization key. - Validate action arguments. Resolve Toolbox targets through the fixed platform API. Check sandbox identity and organization before using an approved Daytona proxy. Keep normal guarded network requests and vault resolution. - Transfer file bytes without reading or writing host files. Share one caller deadline across prerequisite reads, dispatch, and response reading. Bound requests and responses. Do not automatically retry writes with unknown outcomes. Redact the selected credential from results and errors. - Classify credential access, access changes, interactive input, command execution, file permissions, and deletion (including TTL and auto-delete) as destructive actions. Keep new and changed actions under catalog review. - Upgrade saved catalogs and retire the earlier diagnostic names. Hide retired actions from the UI and agent discovery even when provider refresh fails. Preserve existing action choices and historical records. Reject stale same-name contracts during agent discovery and dispatch before a board refresh. - Add database-backed gateway and catalog-upgrade regressions, schema generator tests, a browser setup check, and both capability map updates. ## Verification - Passed on the review-fix head: 57 focused Vitest cases (18 Daytona database/client cases, five HTTP boundary cases, and 34 shared catalog cases), four generator tests, `pnpm -r typecheck`, and `pnpm build`. The five HTTP cases use a real local server through the pinned production transport. The normal browser setup check and 17 generator/artwork cases passed on the initial PR head; those UI and artwork paths are unchanged. - Passed on this branch: brand asset validation, token gates, map structure checks, and `git diff --check`. All latest-head GitHub checks pass on `ea979ccb3a64dc9508807823540f935c4316ea35`. Apex is 5/5, with all five findings addressed and zero unresolved threads. The branch has no merge conflicts. - The initial catalog-generation command could not find the external research corpus. The supported `--definitions-only` mode regenerated the definitions successfully. Unrelated generated drift was excluded. - An earlier full local Vitest attempt on the preview branch failed when bundled PostgreSQL could not start on macOS. After replay onto current master, all three unchanged event-sequencing cases pass with bundled PostgreSQL; the earlier startup cause remains uncertain. - The first full-suite command on this branch used my external test database override. Chat fixtures require a current migrated schema and failed because `agents.lifecycle_state` was absent in that disposable database. A focused comparison reproduces failure with the override and passes with normal setup. That attempt was stopped, and the full suite restarted without the override. Focused Daytona cases use their own migrated PostgreSQL 16 databases. - The clean local full-suite attempt recorded embedded PostgreSQL startup failures in two unchanged native-runtime cases and two failures in unchanged Git fixtures. It was stopped after these failures were isolated; no full local suite pass is claimed. The 15 matching native-runtime cases pass separately. The Git cases fail with the local `init.defaultBranch=main` setting and pass with `GIT_CONFIG_COUNT=1 GIT_CONFIG_KEY_0=init.defaultBranch GIT_CONFIG_VALUE_0=master` limited to the test command. No global Git setting was changed. Full Linux CI is green. - Fixtures use real database persistence and run-bound gateway sessions with provider responses replaced by fixtures. They cover personal/shared vault ownership, reads and writes, file bytes, Off policies, revocation, organization mismatch, untrusted proxy rejection, output bounds, saved catalog upgrades, failed refresh, and permission preservation. - The first Apex review found five valid issues: multipart transport compatibility, destructive classification, caller deadlines, audit filters, and stale saved contracts. All five were fixed with regressions before the second review. A typecheck then found the nullable saved schema hash; admission now accepts that stored type and rejects null as stale. - The browser check covers setup controls without submitting a provider key. No real Daytona account, provider write, native OAuth flow, or agent model run is qualified. Live provider read/write proof remains pending a test account. ## Risks - Daytona keys have broad Toolbox authority, including keys with no selected resource scopes. Only assign this connection to trusted agents and review its action permissions. - Remote writes or commands can continue after a timeout. Calls do not automatically retry an uncertain outcome. - The bridge targets Daytona Cloud. Proxy hosts are restricted; an unrecognized future regional host fails closed. - Files and responses have an 8 MiB bound. Each action has one caller deadline, capped at 30 seconds across all provider requests and response reading. Interactive WebSocket sessions are not exposed as HTTP actions. - The generated JSON snapshot exceeds 1,000 lines because it holds the provider input contracts. Generator logic and execution code remain in separate small modules. Updates require explicit regeneration and catalog review. - No database migration or lockfile change is required. ## Model Used OpenAI Codex, GPT-6. The runtime identifies the model family but does not expose the exact backend model ID or context window. Used reasoning, repository inspection, code editing, shell tools, database verification, and browser tests. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
61831ba7b8 |
refactor: defer adapter launcher imports during workspace snapshots (#16492)
Defer adapter launcher imports until a directory merge resolves its lock root. Snapshot-only workers retain the same hashes, modes, exclusions, and ownership checks. Verification: 63 focused cases; full typecheck/build; exact-head full CI/E2E/security; Apex 5/5. Local native Daytona replies: 66.939s cold and 26.485s warm. Hosted staging and the cold latency target remain unverified. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
Dotta
|
f1d925b726 |
fix(issues): keep workspace restoration failures retryable (#16489)
## Thinking Path > - Paperclip manages agent work and its saved files. > - Finished Git workspaces can expire and restore from a saved commit. > - A task comment must restore a closed workspace before the comment is saved. > - Failed restoration returned HTTP 503 and hid the restoration message. > - The composer treats 5xx writes as uncertain, so it blocked the draft instead of offering a clear retry. > - This change returns a definite rejection before the write and preserves the restoration message. > - Users can restore repository access and retry the same draft. ## Linked Issues or Issue Description Refs #16457 and #16476. Live acceptance found this gap with a disposable expired workspace and an unavailable upstream. The saved commit and unique-file control stayed intact. No fallback workspace or comment was created. The UI still showed an uncertain comment result because the route returned 503. ## What Changed - Return HTTP 409 for a rejected workspace rebuild before the consuming write. - Preserve the service's safe restoration message in the response. - Cover rejected comment, comment-update, and checkout paths. - Document the response and draft retry behavior. ## Verification - `pnpm exec vitest run server/src/__tests__/issue-closed-workspace-routes.test.ts ui/src/api/issues.test.ts`: 38 passed in 26.80s. - Route tests assert that failed restoration does not save a comment or update the task. - Existing client controls preserve 409 rejections and still treat network failures and 5xx writes as uncertain. - `pnpm -r typecheck` and `pnpm build` passed. Full PR CI passed on attempt 2: [run 38804449401](https://github.com/paperclipai/paperclip/actions/runs/38804449401). The first attempt timed out in the unrelated fresh personal-AI connection browser case while waiting for its first reply. Logs and browser evidence were inspected and retained; only failed jobs were rerun once, and that shard passed. - Retained live fixture will verify the error, retry, exact saved commit, and subsequent persisted work after deployment. ## Risks Clients now receive 409 instead of 503 for this known pre-write rejection. They must retry after repository access is restored. The route uses the service's fixed messages, not raw Git errors. This change does not alter restoration, authorization, cleanup, or genuine uncertain-write handling. ## Model Used OpenAI Codex, GPT-6 family, with code execution and browser tools. The exact deployment model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge |
||
|
DottaandPaperclip
|
6943904bad |
perf: stage independent remote GitHub launchers concurrently (#16484)
Stage at most three independent remote GitHub launcher files at once only when the existing effective concurrent-sync capability permits it. Preserve serial fallback, per-file locks and integrity checks, and settle every started upload before failure returns. Verification: 119 affected tests pass (one existing Linux-only oracle skipped on macOS); full repository typecheck/build pass; same-sandbox real-Daytona component comparison preserves all bytes/modes; current-head full CI/E2E/Superagent/Snyk and Apex 5/5 pass. The full native timing target remains separate and is not claimed by this change. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
676d3ec8d4 |
perf(ci): reuse package downloads and overlap runtime checks (#16474)
## Thinking Path > - Paperclip contributors wait for CI before delivering changes. > - Build still finishes last after #16460. > - CI already downloads the public npm graph while the build runs. > - Bundled-package preparation ignores that cache and downloads the same archives again. > - Reuse the existing cache and overlap independent runtime checks while preserving fresh installs, integrity, offline lifecycle and admission verification. > - Measure staging and the full workflow separately. ## Linked Issues or Issue Description Refs #16460 and #16434. This is a focused follow-up to merged #16460, now based directly on master. **Current behavior:** The Build workflow warms `PAPERCLIP_PUBLIC_INSTALL_NPM_CACHE`, but `startBundledInstalls` inherits the host's default npm cache. Server staging took 31.17 seconds in the latest passing run, before packing and the clean consumer install. **Proposed behavior:** Pass the verifier environment, including the explicitly configured warm cache, to the three existing staging children. Without an explicit shared cache, keep the usual producer npm configuration. **Alternatives considered:** Copying an installed dependency tree would bypass normal fresh installation. Adding jobs would add setup cost. This change reuses only npm's integrity-checked download cache. It does not remove coverage or change product behavior and does not overlap a roadmap feature. Open PR title search found no equivalent cache wiring change. ## What Changed - Initialize the existing consumer cache before package staging and expose it to producer npm only when explicitly configured. - Pass that environment to the existing bundled-install children. - Extend the child orchestration check to verify cache propagation and document the cache boundary. - Start Pi setup after the offline lifecycle, in its own download sandbox, while Codex and Grok probes run in separate offline sandboxes. Join successful Pi setup before lock verification and Pi admission. Name and clean up only the verifier-owned setup container. ## Verification - 33 focused public-install, closure and child-orchestration tests pass on Node 24.21.0. - A real server dependency staging experiment used `prepare-bundled-package.mjs`, its actual manifests, provider overrides and release patches. Built application payloads were omitted from this dependency-only measurement. - Cold cache: **16.139s**. After the actual CI warm-up script: **6.980s**. A fully warm producer cache took **6.127s**. These are local sequential measurements, not a CI wall-clock claim; filesystem warmth and registry timing vary. - All **28,329 files/links**, including both lockfiles, were byte-identical between the cold output and the actual-workflow-warm output. - The warm-up itself took 49.631s; CI already runs it during Rust restore/build. This change adds no warm-up operation. - First follow-up run [passed all 48 jobs](https://github.com/paperclipai/paperclip/actions/runs/38791277425), and Greptile gave the cache-only head 5/5. Server staging fell **31.172s → 17.565s** and total staging **70.169s → 54.869s**. But complete CI was **8m20s**, slower than the parent's 8m04s; longer Codex/Pi probe times outweighed that saving. This is not a full-run speed improvement claim. - The next revision overlaps the independent probes instead of serializing Pi setup behind them. All 46 focused sandbox, staging, closure and process-supervision tests pass, including failed background children, parent exit and cancellation. Pre-retarget head `590873a44` passes all 48 jobs and Greptile 5/5 with no review findings. After #16460 merged, head `fe4f8d38e` incorporates master and retargets this PR directly to master. Its diff remains only four optimization/documentation/test files; 46 focused checks pass again. Fresh full CI and review are pending. The unchanged parent conflict-resolution run passed all 48 jobs in 8m04s: https://github.com/paperclipai/paperclip/actions/runs/38789785045. Final paired runs against the same updated master, both on `m7i-flex.xlarge`: | Measurement | Parent #16460 | This PR | | --- | ---: | ---: | | Bundled server staging | 22.220s | 14.202s | | All package staging | 61.277s | 48.304s | | Clean consumer install | 67.617s | 73.490s | | Pi setup remaining wait | 25.428s (serial setup) | 20.719s | | Build job | 385s | 368s | | Build start after workflow creation | 23s | 136s | | Entire workflow, including admission and aggregates | 452s (7m32s) | 552s (9m12s) | [Parent run](https://github.com/paperclipai/paperclip/actions/runs/38792983313), [final run](https://github.com/paperclipai/paperclip/actions/runs/38792986392). Build execution improved by **17s (4.4%)** in this pair. The total workflow did **not** improve: it waited 113 seconds longer to start Build, and other lanes/aggregates also contribute. The overlapping setup ran for 25.667s while Codex/Grok took 4.946s, leaving 20.719s to wait. Actual installed Codex, Grok and Pi checks all passed with zero provider calls. These are individual samples with filesystem/network/CPU variance, not a claimed steady-state average. The earlier slower cache-only run is retained above. No tests were removed and no jobs were added. ## Risks Shared-cache contention can reduce the benefit if warm-up is still running. npm handles concurrent content-addressed cache writers and checks integrity; misses still download normally. Two runtime containers can now overlap and contend for host resources; the full CI run measures that tradeoff. They retain separate network policies and per-container bounds. Pi writes only its provider assets and a private receipt; the other probes inspect Codex/Grok. No installed tree, lock, test result or build output is reused. The consumer still starts empty and performs the full lifecycle and runtime checks. Normal unconfigured producer cache behavior is preserved. ## Model Used OpenAI GPT-6 through Codex, with repository inspection, execution and GitHub tools. The serving model suffix and context capacity are not exposed to this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either linked existing issues or described the issue in this PR - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch name describes the change and contains no private ticket identifier - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation - [x] I have considered and documented the risks - [ ] All Paperclip CI gates are green (prior head passed; rerunning after master merge) - [ ] Greptile is 5/5 with no open P2s, recommendations or follow-ups (prior head 5/5; current head pending) - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
3898f6a45f |
fix(native): honor duplex transport for GitHub callback bridges (#16455)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native remote runners use a session-owned broker for managed GitHub operations. > - That broker uses the existing authenticated sandbox callback bridge. > - The native caller omitted the server-selected duplex flag. An enabled and capable target still used file polling. > - Warm sessions must observe a changed flag and channel loss before the next admission. > - This pull request forwards the existing flag and retires unavailable or changed bridges before retained instruction preparation. > - The benefit is less idle remote traffic without changing credential authority or the default transport. ## Linked Issues or Issue Description **What happened?** A native GitHub callback bridge selected `queue_v1` even when its sandbox target had duplex enabled and supported it. The file bridge continues remote queue checks while the warm session remains available. **Expected behavior** An enabled, capable target should use the existing HTTP/2 bridge. Disabled or unsupported targets and failed channel startup must retain the authenticated file fallback. A changed server flag must take effect at the next normal warm-session boundary. **Steps to reproduce** 1. Create native GitHub access on a sandbox target with the server-selected duplex flag enabled, verified duplex capability and a working channel. 2. Observe the selected callback transport. 3. Before this fix, it is `queue_v1`. The added real-wire regression expects `http2_v1` and fails on the old source. ## What Changed - Forward the existing server-selected duplex flag through the native GitHub broker. - Record that transport selection on the broker. Retire the warm provider and broker through the existing cleanup path when the flag changes. - Read live channel failure when checking broker readiness. Mark normal session cleanup as an orderly completion. - Include the selected flag in instruction preparation identity. Retire a failed broker before adopting retained instructions, while preserving existing ownership and cleanup fences. - Add real local HTTP/2 credential-rotation and file-fallback controls. Add native warm-session flag-change controls. - Document selection, fallback and warm-session behavior. - Scan CI secret expressions in one forward pass, including quoted braces and escaped apostrophes. Preserve direct, bracket, multiline and whole-context references while rejecting filename and literal-string false positives. ## Verification - Current head `41440b07ea45e9e882e36ed495a66e501c8dd4df` resolves the actual CI-policy conflict with master `307eb16c0c59abd9e0626fde9201939b262e7f5f`. It retains upstream's stricter whole-expression credential guard, including quoted secret text, and uses the forward scanner instead of a backtracking regex. All **16 CI policy tests** pass, including the bounded 28- and 10,000-apostrophe controls in both policy modes. The reported old regex still has its recorded two-second timeout (14 passed, one failed). On this final merged source, **809 affected native cases** pass with zero selected skips (65.20s). Full repository typecheck (31.44s) and build (22.13s) pass. Frozen source hashes stayed unchanged. Fresh exact-head full CI, configured security and completed Apex remain pending. - Before master integration, the frozen native repair passed repository `pnpm -r typecheck` passed (33.0s) and `pnpm build` passed (32.1s). - Before the latest master integration, all **800 affected cases passed with zero selected skips**: complete native executor (725), resume (50), GitHub broker (16), environment duplex-gate (4), and heartbeat instruction-copy (5) suites. The final run took 95.3s. Source hashes stayed unchanged through these checks. - The initial head passed 788 affected cases and full CI, but Apex scored it **3/5**. It found a dead HTTP/2 broker could remain reusable and a flag change could fail after retained instruction preparation. Both findings required the new repair. Three added before-fix regressions failed on channel availability, orderly close, and preparation identity. - The first repair run failed three new warm-instruction fixtures because fake provider close left its recorded state ready. The corrected fixture writes a suspended checkpoint without weakening the production ownership guard. A separate real-wire run then failed two availability assertions because its helper spread had frozen the readiness getter. The helper now preserves the getter. All failed attempts remain in the local evidence. - One disposable real Daytona sandbox exercised both transports on the initial transport patch through the production native broker and callback bridge. The provider shell wrapper used the then-current production declarations. During equal two-second idle windows, file mode issued **9 remote commands** and HTTP/2 issued **0**. Both variants rotated synthetic credentials between two run bindings and denied idle requests. Cleanup deleted the sandbox. - Daytona component setup took 3.64s for file mode and 4.74s for HTTP/2. This is an idle-traffic reduction, not a startup-speed claim. That component run predates the availability and preparation repair; it does not qualify the repaired paths on Daytona. The new failure controls use a real local HTTP/2 channel. The credential resolver was synthetic and no model was called. Real GitHub-account grants, staging replies, and the end-to-end latency target remain unverified by this component test. - Repaired-head CI found a policy-test false positive: a broad `secrets.` regex matched an upstream filename. The unchanged test reproduced the failure, and the corrected version passed all 15 controls. Master later supplied a stronger direct/bracket check, which we retained when resolving the merge conflict. Review then found that formatted expressions could stop that check at a quoted brace. The new reported-case baseline failed (one failed, 14 passed). The final check treats quoted strings as complete tokens, respects expression boundaries, and passes all 15 cases. The later master conflict also added `toJSON(secrets)` controls. The old branch pattern failed that whole-context case. Review subsequently found exponential backtracking with escaped apostrophes; the final forward scanner retains the quoted-boundary and whole-context controls and passes all 15 tests, including the bounded long-expression regression. No workflow or credential handling changed. All failed attempts remain recorded. - The full local repository test suite was not rerun. [Prior-head full CI](https://github.com/paperclipai/paperclip/actions/runs/38775801821), including `ci / verify` and `ci / e2e`, passed on previous head `e5f8bdb425081cf08110c0f7423f96609e405920`. All 52 check runs succeeded, two optional Storybook jobs skipped, and legacy Snyk passed. Completed Apex review is **5/5** on that exact head, with no unresolved review threads. ## Risks - Only an explicit server-selected opt-in can use HTTP/2. The existing provider-capability, channel-startup and authenticated file-fallback checks remain in the shared bridge. - Toggling the flag or losing the HTTP/2 channel replaces an idle retained provider at the next run boundary. Instruction preparation must make the same retirement decision. Normal ownership, settled history and cleanup checks remain in effect. - Lower idle traffic may reduce provider/controller pressure, but no staging latency improvement is claimed. This feature remains inactive when the existing flag is off. Other preparation and finalization costs remain. - No schema, API, UI, credential-selection, tracing endpoint or default-setting changes. ## Model Used - OpenAI Codex, based on GPT-6, with tool-assisted reasoning and code execution. The exact hosted snapshot ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
307eb16c0c |
perf(ci): avoid duplicate extraction in public install checks (#16460)
## Thinking Path > - Paperclip manages AI agents for work. > - Contributors wait for CI before they can deliver changes. > - After #16434, Build is the longest CI job. > - Its clean public npm install spends about 90 seconds installing packed artifacts. > - npm extracts local archives to read metadata before it installs them. > - Serve metadata from those same packed artifacts, then let normal npm verify and install their bytes once. > - Preserve the offline lifecycle and runtime checks, and measure the complete workflow. ## Linked Issues or Issue Description Refs #16434 and #16429. **Current behavior:** The Linux Build check installs every staged package by local tarball path. npm's file fetcher extracts an archive to obtain its manifest. Installation then extracts it again. In the last green run, Build took 408 seconds; consumer installation took 89.7 seconds. The full workflow took 8m06s. **Proposed behavior:** Read manifests directly from the packed archives. Serve those manifests and their SHA-512-bound tarballs from a temporary registry on container loopback. Use normal npm installation into the empty consumer. Other packages still resolve from the public npm registry. **Alternatives:** More parallel jobs would add setup and transfer work. Skipping the install or runtime probes would remove useful coverage. This change removes duplicate extraction instead. It does not add a product feature or duplicate a roadmap item. ## What Changed - Build a registry catalog from the actual tarball manifests and archive hashes. - Run the short-lived registry and npm in the existing unprivileged download container. No host port is exposed. - Preserve public dependency tarball URLs, archive integrity checks, disabled initial lifecycle scripts, the offline rebuild, lock checks and all Codex, Grok and Pi probes. - Close the registry before offline verification. Keep native ARM and Mac install qualification unchanged. - Add real npm parity and failure tests. Record metadata preparation time in the existing timing receipt. - Fix an existing policy false positive: the filename `docker-compose-secrets.test.mjs` is not a workflow credential. Inspect GitHub expressions and retain rejection of Honeycomb keys, dot/bracket secret reads and `toJSON(secrets)`. ## Verification - 31 focused install-contract checks pass on Node 24.21.0. All 16 targeted CI profile-policy tests also pass. - Real npm file and registry installs produce identical installed files, bundled dependencies, optional dependencies and executable links. Both defer lifecycle execution; offline rebuild works after the registry closes and preserves the lock. - npm rejects an archive whose gzip header changes while its extracted payload remains valid. The CLI propagates the failure. - The permanent real-npm parity test now includes non-catalog `is-number@7.0.0`. Both paths install it from its official npmjs tarball, and the test asserts the lock URL. Fetches have a 10-second bound and no retries. This addresses the first Greptile finding. A separate live probe also confirmed public provenance. - The first local run hit the host sandbox's loopback `EPERM`; the same checks passed with local listener access. An initial corrupted-gzip negative fixture hit decompression failure before integrity verification; the final fixture preserves valid gzip and specifically proves integrity rejection. An old command-shape assertion was updated for the new entry point without removing its ordering checks. - The [first CI run](https://github.com/paperclipai/paperclip/actions/runs/38778371043) passed all test lanes and Build, but the existing credential-guard false positive failed policy and both aggregates. Its 7m25s full duration is **not** counted as a passing performance sample. The guard repair and public-download regression are on the new head. - The real Linux install passed all 18 packed packages, the lifecycle sentinel, offline rebuild, unchanged lock, official Codex payload/lease checks, Grok missing/present checks and Pi admission. The receipt confirms `artifactTransport: loopback-registry`. No provider calls ran. - Compared with the previous green run, consumer install fell **89.72s → 66.29s** (26.1%), including 1.87s to prepare metadata. Build fell **408s → 374s**. Both Build hosts were `m7i-flex.xlarge`; these are separate runs, so registry/cache and upstream changes remain confounders. - The base advanced since the previous 8m06s run, including runtime and test changes. Treat that duration as historical context, not an identical-source control. - Pre-conflict head `d8f9de1c` [passes all 48 workflow jobs](https://github.com/paperclipai/paperclip/actions/runs/38779549190), including both required branch-protection checks. Greptile is **5/5** on this head; its review thread is resolved. The non-required Superagent Security Scan is still running. | Measurement | Previous green run | Final green run | Observed change | | --- | ---: | ---: | ---: | | Full CI, workflow creation through final aggregate | 8m06s | **7m35s** | **31s shorter (6.4%)** | | Build job execution | 408s | **389s** | **19s shorter** | | Clean consumer install, including catalog preparation | 89.72s | **69.69s** | **20.03s less (22.3%)** | The final catalog preparation took 1.87s and is included in the install figure. Both Build hosts were `m7i-flex.xlarge`. Build began 42 seconds after workflow creation instead of 53 seconds in the historical run, so faster queue/setup admission explains part of the 31-second total. Upstream code also advanced; these measurements are not a controlled causal estimate of the complete gain. The two candidate install measurements (66.29s and 69.69s) support the narrower reduction in install work. No tests or runtime probes were removed. - Conflict-resolution head `ea8287b81` [passes all 48 workflow jobs](https://github.com/paperclipai/paperclip/actions/runs/38789785045), including both required gates. Greptile re-reviewed this exact head at 5/5, with the sole earlier thread resolved. The conflict was the independent filename-policy fix; the stronger expression guard is retained. This run took **8m04s** overall, Build 423s, consumer installation 80.56s and bundled-server staging 31.17s. This confirms meaningful run variance: the earlier 7m35s is a sample, not a guaranteed new wall-clock duration. - A second independent master update touched the same credential test. Final head `1c740b73e` resolves that conflict while retaining the same stronger guard. [All 48 jobs pass](https://github.com/paperclipai/paperclip/actions/runs/38792983313); Greptile re-reviewed this exact head at 5/5. This run took **7m32s** overall, Build **385s**, consumer install **67.62s**. Auto-merge is enabled but GitHub requires code-owner approval. ## Risks The initial install now uses registry package specs instead of file specs. The parity test checks the resulting installed bytes. Full CI must still exercise real bundled provider graphs and all offline probes. The temporary consumer lock contains loopback artifact URLs; it is not exported or reused for another installation. Native release consumers retain their existing artifact install path. Queue, host and cache variance limit attribution from separate workflow runs. ## Model Used OpenAI GPT-6 through Codex, with repository inspection, execution and GitHub tools. The serving model suffix and context capacity are not exposed to this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either linked existing issues or described the issue in this PR - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch name describes the change and contains no private ticket identifier - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation - [x] I have considered and documented the risks - [x] All Paperclip CI gates are green (48 jobs and both required gates on conflict-resolution head) - [x] Greptile is 5/5 with no open P2s, recommendations or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
2aca41464b |
fix(workspaces): preserve compaction and expire standard Git bases (#16476)
## Thinking Path > - Paperclip manages AI agents and their task workspaces. > - Finished tasks can release reproducible Git files and restore them later. > - Unique work must stay on disk, while regenerable dependencies can be removed. > - Linux Git templates include an empty legacy directory that blocked base cleanup. > - The new expiration branch also skipped the existing guarded dependency cleanup. > - This change accepts the standard empty directory and retains that cleanup order. > - Private bases can expire, and protected workspaces still shed regenerable files. ## Linked Issues or Issue Description Refs: #16457 Live acceptance of #16457 removed a worktree but retained its private base because Linux Git created an empty `.git/branches` directory. A pending-base dry run then tried to resolve the already removed worktree and hid the preservation reason. A local integration comparison also showed that enabling Git expiration skipped existing dependency compaction. It deferred pending startup compaction by an hour. Both failures were reproduced before the fix. ## What Changed - Accept `.git/branches` only when it is an empty real directory. Preserve populated entries, files, and symlinks. - Inspect pending base cleanup without resolving an absent worktree. - Run existing guarded compaction before Git expiration. Exclude already closed rows from compaction and retain its pending deadline. - Keep failed expiration checks from reaching legacy whole-workspace deletion. Existing compaction remains independent of Git expiration dry-run mode. - Extend real Git/PostgreSQL tests and retain the acceptance failures in the delivery state note. ## Verification - Linux-template baseline: private-base expiration failed before the fix. The first sandboxed attempt could not bind PostgreSQL; the authorized retry reproduced the behavior. - Compaction baseline: two regression cases failed in 7.67 seconds. They reproduced skipped dependency cleanup and the lost startup deadline. - After the fixes, `pnpm exec vitest run server/src/__tests__/execution-workspace-compaction.test.ts server/src/__tests__/git-workspace-expiration.test.ts server/src/__tests__/terminal-workspace-work.test.ts` passed all 49 cases in 43.79 seconds. - Controls cover unique work, active runs, unfinished descendants, previews, concurrent reopening, interrupted removal, exact-commit restoration, and retryable upstream failures. - Full `pnpm -r typecheck` and `pnpm build` passed for the final commit. All exact-head CI and security checks passed. Greptile is 5/5 with all findings resolved. - Final staging acceptance will let the restarted worker resume the retained base cleanup, then reopen normally at the recorded commit and persist subsequent work. The base has not been manually removed. The superseded preview build was cancelled before dispatching a replacement. ## Risks - Legacy branch definitions can hold unique data. Only an empty real directory qualifies for deletion. - Compaction preserves its existing lifecycle guard and regenerable-directory allowlist. It cannot fall through to legacy deletion after an expiration failure. - Already closed workspaces must only resume their recorded expiration; they do not enter compaction. ## Model Used OpenAI Codex, GPT-6.1-sol, with reasoning, code execution, Git, database, and browser tools. The exact context-window size is not exposed by this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
c53bdd9eb3 |
perf: capture sparse checkpoint files in bounded batches (#16473)
Capture sparse checkpoint files in bounded batches while retaining content and mutation checks. Prevent inserts after SQLite automatic rollback and drain active readers before database close. Verified: 209 affected cases, final six capture controls, repository typecheck/build, real Daytona helper validation, and exact-head full CI/E2E, security and Apex 5/5. The matched component median saves 1.05s; the whole-agent latency target remains open. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
5ec93dae01 |
fix(connectors): avoid redundant run preparation discovery (#16461)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Run preparation selects connector skills and tools under current permissions. > - An assignment check can build an unrelated company tool catalog and wait for a shared discovery slot. > - Native startup repeats assignment discovery to obtain static tool definitions. > - Slack also locks and reconciles an unchanged catalog, then repeats task authorization. > - This pull request removes that redundant startup work and preserves live checks at tool execution. > - The benefit is less preparation work, less contention, and unchanged permission and approval enforcement. ## Linked Issues or Issue Description Related: #16454 (detailed preparation tracing), #16425 (workspace preparation performance). **What happened?** Connector configuration added avoidable work to run preparation. Browser Use assignment authorized the whole company catalog, including other connector families. Four connector resolvers ran in series. Native controller startup repeated all assignment resolution when any connector skill was pinned. Slack reconciled an unchanged catalog under an advisory lock and checked task authority twice. **Expected behavior** Prepare only the connector data needed by the run. Reuse static tool definitions from pinned skills. Preserve current company, assignment, identity, profile, policy, grant, and approval checks when an operation executes. **Steps to reproduce** 1. Configure Browser Use and an unrelated MCP catalog with 500 tools for one company. 2. Resolve Browser Use resources for an authorized task. The old path reads and authorizes the unrelated catalog. 3. Occupy both full discovery slots. The old assignment check waits for those slots. 4. Resolve a linked Slack task repeatedly with an existing current catalog. The old path still acquires a catalog lock and repeats authority checks. **Paperclip version or commit** Base: `68a09dcb5f50e3fb6ceaaeac2706d96db97fc9b3`. **Deployment mode** Observed in hosted native runs. The local reproduction uses a source checkout and real embedded PostgreSQL. ## What Changed - Restrict Browser Use catalog reads and policy decisions to active Browser Use connections. Preserve guest restrictions, reviewed catalog filters, live grants, and deterministic resource order. Keep assignment checks out of the full discovery queue. - Resolve the four independent connector families concurrently. Wait for outstanding work to settle before propagating a failure, and retain registration order. - Register native direct connector tool definitions from pinned connector skills. Keep live assignment resolution and governed execution on every tool call. - Compare Slack catalog hashes before reconciliation. Current catalogs skip the catalog transaction and lock. Check task authority once after current profile checks. Initial setup and upgrades retain locked reconciliation and checks before and after it. - Precompute immutable bundled Slack descriptor hashes. Preserve disabled entries, profile state, removed bindings, and review metadata. - Document the startup paths and add focused contention, revocation, catalog upgrade, and failure-settlement regressions. - Repair a pre-existing CI guard false positive: the filename `docker-compose-secrets.test.mjs` matched `secrets.`. Check GitHub secret expressions instead, with dot, bracket, and whole-context controls. No workflow changes. ## Verification - Conflict resolution: merge `b1566ca494f4af6e8d4418fa37393c5caee02582` from master into head `55140b1609ae7bd21a038eb75a48c2683299e4b4`. Combine both secret-expression guards and regression cases. Preserve current workspace, native recovery, and Daytona changes. All 467 policy tests, 787 focused native/connector/Browser Use tests, and server typecheck passed locally after integration. Fresh full CI passed all 55 checks on this head, with two optional Storybook jobs skipped. Greptile reviewed this exact head at 5/5 with no unresolved threads. - `pnpm -r typecheck` — passed. - `pnpm build` — passed. - Focused native startup, connector concurrency, and AgentMail suites — 763 tests passed. - Complete Browser Use suite — 33 tests passed. Focused Slack catalog, linked task, retained provenance/revocation, and pinned native tool tests passed. The complete runner tool authority suite passed (26 tests). The 1,337-case chat suite passed 1,335 tests and had two failures: the original new transaction-count assertion and a 15-second timeout in the publication-batch case. Both cases passed an isolated rerun. The publication timeout occurred during concurrent local checks; the passing rerun does not erase it. CI must still pass the chat shards. - The first focused run found two incorrect new assertions: policy evaluation performs a second catalog lookup, and an explicit allow policy overrides a profile default. The tests now distinguish the scoped catalog query and revoke the governing policy. The focused Browser Use rerun passed all 33 cases after these assertion fixes. - Full-suite test verification used the CI shards of `pnpm test:run`. Local runs focused on the changed boundaries; the complete repository test command was not duplicated locally. - `node --test '.github/scripts/tests/*.test.mjs'` — all 467 policy tests passed. The CI failure reproduced on the unchanged base guard before this correction. Actual GitHub secret expressions still match the guard. - CI on the final head passed all chat shards, typecheck, build, Runner checks, browser shards, and other test lanes. One existing workspace-runtime case hit `EADDRINUSE` for a randomly selected test listener in server shard 1/12 (1,162 tests passed in that shard). The shard passed its single retry on the same SHA with no source changes. The first failure remains recorded here. Previous reviewed head `7191ac40691ddb1ef38b86665c95cc14c57ea390`: all 55 checks passed, two optional Storybook jobs skipped, and no pending or failed checks. Greptile is 5/5 on this head with no unresolved review threads. The external security scan passed. Local before/after measurements use the old and new implementations against the same fixture. Both paths are warmed. Seven samples alternate execution order. The database is real embedded PostgreSQL. No latency is injected. These measurements do not establish hosted speed. | Local path | Before median (range) | After median (range) | Work reduction | | --- | --- | --- | --- | | Browser Use assignment with 500 unrelated MCP tools | 44.020 ms (33.994–107.328) | 26.502 ms (20.385–99.037) | 26 → 22 statements; unrelated catalog and discovery queue removed | | Slack assignment with current catalog and linked responsible user | 49.723 ms (43.624–80.653) | 22.141 ms (19.781–30.818) | 5 → 2 database transactions; no catalog lock on the current path | The contention regression fails against the old implementation after a bounded five-second wait, then passes against the optimized implementation. It occupies the real global discovery scheduler and verifies that Browser Use assignment completes without releasing those slots. It records catalog SQL to verify connection scope. Revocation cases remove policy, catalog availability, and grants between calls. Existing Slack integration cases exercise retained provenance, removed profile bindings, identity revocation, approved writes, and current responsible-user restrictions. ## Risks - Pinned native tool definitions can remain visible after access is revoked. They confer no authority. Invocation rechecks current assignments and fails closed before an operation executes. Newly available connector families still require a newly admitted skill selection. - Concurrent resolution can increase instantaneous database reads to four connector families per configuration call. It is a fixed bound. Outstanding work settles before a failed preparation returns. - Slack skips reconciliation only when all bundled version hashes and an existing profile match. Missing or changed definitions still use the existing locked path. Operator revocations remain effective. - This PR addresses connector preparation. Workspace transfer, provider startup, copyback, and queue delay can still keep total latency above the user's goal. Hosted improvement needs a deployed before/after trace. - No schema, lockfile, workflow, telemetry event, or authorization cache changes. ## Model Used OpenAI GPT-6 (Codex), with reasoning, repository inspection, code execution, and GitHub tools. The runtime does not expose the exact serving model ID or context window. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
db11d899cf |
fix(native): recover remote sessions while preserving warm runs (#16451)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native agents keep remote processes and conversation history across warm turns. > - A completed turn can leave handoff authority only in controller memory. > - Losing that controller can block a follow-up even while its sandbox survives. > - This pull request verifies durable ownership and reconnects the exact prior process before handing it to the next run. > - Healthy follow-ups stay warm. Recovery preserves saved work without replaying a completed turn. ## Linked Issues or Issue Description **What happened?** A remote warm turn can finish without suspended backup authority. A replacement controller rejects that prior owner even when its exact process and completed checkpoint remain. A single unavailable identity probe can also stop an otherwise valid reattachment. **Expected behavior** Keep healthy follow-ups warm. Recover a completed session after proving that its controller stopped and that the successor holds current authority. Reuse compatible processes; otherwise save and resume the conversation. Retry unavailable identity reads within one shared deadline. **Steps to reproduce** Use `doc/testing-native-remote-continuity.md`. Its Linux cases execute real turns through PostgreSQL, TLS ingress, and the Rust runner. They kill a separate controller process, recover old checkpoint metadata, and change process configuration. The provider is deterministic. Related: #16432 and #16441 fix live warm-owner handoffs. #16341 adds recovery diagnostics. #14695 covers provider process retention. ## What Changed - Preserve warm process reuse between healthy remote turns. - Verify a committed prior result, stopped controller, matching checkpoint and sandbox, and current task/coordinator ownership before recovering an orphaned warm session. - Reconnect through the prior run route, then use the existing authenticated run attachment to change authority. - Persist a process-configuration digest. Close and save recovered sessions when old or changed configuration cannot support reuse. - Retry unavailable state, marker, and process reads within one 30-second deadline. Keep signals and definitive identity failures single-attempt. - Record closed recovery diagnostics, including unreadable or malformed saved state. - Retire the recovery-only ownership check after verified attachment, so later warm turns use their current authority. - Add real-process Linux regressions and refusal controls. - Expose the existing source-only file-recovery form for retained native holds, with the same board authority, evidence, and exact stop-receipt requirements as legacy holds. Recording recovered files does not change task status or replay a run. ## Verification - `pnpm -r typecheck` passed; the server check was repeated for the third-turn fix. - `pnpm build` passed after the final UI change. The UI typecheck, token gates, and map checks also pass. - Four Linux continuity cases passed: warm reuse, controller loss, legacy checkpoint metadata, and changed configuration. Each creates one conversation and starts exactly three turns, including a healthy follow-up after recovery. The fixture uses expired prior leases; the third turn keeps the runner and provider processes. - Greptile caught the retained recovery gate; the third-turn regression now covers it. An initial test omitted the heartbeat’s saved input and left a test process after failure; the fixture was corrected and its disposable container reset before the passing run. - The current focused executor/restart group passes all 749 tests, including the original failure classifications. - Native file-recovery and UI tests pass (52 tests), including blocked tasks and both native and legacy holds. - After merging current master and retaining its equivalent CI expression check, the CI policy tests pass (15 tests) and all four three-turn Linux continuity cases pass again on `dec418ae55ff21b4fb067132613658cce445f181`. - `pnpm test:run` completed its general server group with 20,572 passes, 103 skips, and three failures: the skill-audit child process and two HTTP test timeouts. The same three suites then passed in isolation (89 tests) without code changes. That local command stopped before later groups; the complete final-head CI, including general, serialized, Runner, build, and browser gates, is green. - Greptile reviewed the combined head `dec418ae55ff21b4fb067132613658cce445f181` at 5/5 with no actionable findings; both earlier findings are resolved. All final-head CI gates and the security scan passed. Staging deployment and retained-incident disposition remain deployment acceptance work. ## Risks - Recovery requires exact durable evidence. Live conflicting controllers, replacement sandboxes, Stop/pause, and uncertain effects must remain blocked. - After controller loss, tool-contract or process-configuration changes can restart the provider. Normal healthy follow-ups retain their processes. - The new idle-recovery path has Linux fake-provider evidence. Live Codex/OpenCode and retained quarantined-state recovery still need staging qualification. - No database schema, telemetry, or runner protocol changes. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository inspection, code execution, and browser tools. The session does not expose a more specific model version or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (targeted checks and isolated reruns; full-run limitation recorded above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green on `dec418ae55ff21b4fb067132613658cce445f181` - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
59a70dc9e2 |
fix(workspaces): expire reproducible Git files and restore exact commits (#16457)
## Thinking Path > - Paperclip manages AI agents and their task workspaces. > - Isolated Git worktrees and private repository copies remain on the host after tasks finish. > - These files can fill the disk even when every commit is available upstream. > - Cleanup must retain unique files, local Git state, live work, and recovery data. > - This pull request records and verifies an exact-commit restoration recipe before expiration. > - Normal reopening restores that commit even after upstream advances. > - The benefit is lower host disk use while finished tasks remain available for later work. ## Linked Issues or Issue Description Refs #10285 and #2022. Follows #16433 for skill-cache expiration. Related #11936 provides separate host-local cleanup tooling; this change extends the existing company-scoped lifecycle worker and normal workspace reopening. ROADMAP.md does not specify a competing implementation. ## What Changed - Add opt-in expiration for verified Paperclip-owned Git worktrees after their whole task tree has been done or cancelled for 24 hours. - Preserve active and scheduled runs, previews, pending reopening, unresolved recovery, dirty files, unknown ignored files, and unsupported or unique Git state. - Verify an authenticated fetch in a fresh object store. Save repository identity, exact commit, branch, and ownership before any deletion. - Rename eligible files into recorded managed trash. Resume interrupted cleanup under source and lifecycle locks. - Expire private base clones only after consumer, file, reference, reflog, and object checks pass. Preserve shared project base clones. - Rebuild missing bases and worktrees through normal reopening at the recorded commit. Return an explicit retryable error when repository access fails. - Add independent workspace/base switches, dry-run reports, reclaimed-byte counts, and operational documentation. No table, public API, UI, or recovery-retention change. ## Verification - Real local bare remote plus Postgres: create at A, advance upstream to B, expire the worktree and private base, reopen at A, then make and retain a subsequent commit. - Final lifecycle suite: all 20 tests passed in 46.41 seconds at `d72947a2e45be5d0c912debb03c46b15863b4798`. Review fixes cover second-cycle expiration, newer-branch recovery, normal preparation after restoration, branch adoption, and Git administration path controls. Eleven existing branch reconciliation controls also passed. Covers unique files and Git state, live descendants, scheduled runs, recovery, previews, unauthorized access, upstream failure, concurrent reopening, interrupted deletion, shared bases, and nested private worktrees. - Final guard regressions: 3 affected tests passed. Existing workspace/runtime/preparation/source controls: 328 tests passed in 238.69 seconds. - Full repository typecheck and build passed locally. The full local test attempt did not pass: initial native-session cases hit macOS Postgres shared-memory exhaustion, source changes invalidated two discovery cases, and the checkout disappeared near the end. The final-code retry of the remaining skill and unrelated controls passed all 103 tests. All checks passed on this exact head, including typecheck, general and serialized tests, Runner checks, build, canary dry-run, browser E2E, and security scans. Greptile reviewed the head at 5/5 with all findings resolved. - Fixed a blocking CI policy false positive for `docker-compose-secrets.test.mjs`. The test still rejects GitHub secret expressions, including dot, bracket, and multiline forms. Its full 15-test suite passes. - Map validation and `git diff --check` passed. - Staging dry-run inspection, measured reclamation, and ordinary product acceptance remain pending. No live acceptance is claimed. ## Risks - Git restoration depends on continued upstream availability and credentials. Cleanup skips unverified commits; restoration never substitutes the default branch. - File deletion requires multiple conservative proofs. Unknown or unsupported state is retained and can limit reclamation. - Source and lifecycle locks serialize preparation, reopening, and expiration. Remote verification has a deadline but can delay work on the same source. - Start with both expiration switches disabled. When enabled, dry-run remains the default. Rollback disables further deletion while retaining restoration support. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository inspection, tool use, and code execution. The exact backend model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (focused tests; full local limitation documented above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
68a09dcb5f |
perf(ci): shorten the test execution critical path (#16434)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Contributors wait for the full PR workflow before they can deliver fixes. > - Server shards still execute independent files in series, and a single Runner file serializes two long real-clock waits. > - Faster work outside the critical path does not prove a faster workflow. > - Overlap independent files while preserving isolation, test cases and real production deadlines. > - Compare complete baseline and candidate workflows, separating queue delay from execution cost. ## Linked Issues or Issue Description Refs #16429. Related duration refresh: #12577. **What existing behavior does this improve?** The elapsed time from PR CI creation to the last required check. **Current behavior** General server shards use one worker. One Runner transport file takes about 333 seconds on Linux, including over 100 seconds in two real-clock regressions. These are the longest verification lanes after #16429. **Proposed behavior** Run independent general-server files in at most two isolated forks. Move the two intact Runner deadline regressions into a separate file to overlap their waits in the existing pool. Run two chat partitions per existing job, each with its own temporary home and embedded PostgreSQL. Preserve serial execution within files and every assertion and timing boundary. ## What Changed - Qualify both serial and parallel forks with real worker processes, isolated homes/configuration, timers, globals and environment. - Require both parallel workers to arrive before either exits. - Use two isolated forks for sharded general-server tests; retain one worker with an explicit shared test database. Other test groups keep their existing concurrency. - Extract the lease-expiry and Pi cold-admission regressions verbatim into a separate Runner test file; retain real waits and process/lease assertions. - Split each sharded chat job into two independent invocations; preserve serial shared-database execution. Await both children before failing the parent. Prove disjoint coverage and failure propagation. - Update the paired Linux benchmark to compare a complete general-server shard with identical case inventories. - Document concurrency and deadline-test placement. No production behavior, workflow topology or runner count changes. - Allow 15 seconds for snapshot integration cases that cold-start multiple real TypeScript workers. On the repeat, one-worker cases took 2.5–3 seconds, so queued or sequential starts could exceed the default five seconds. Preserve all cleanup, queue-release and filesystem assertions; these cases do not enforce a production latency deadline. ## Verification All 48 jobs passed on scheduling head `5d718e18` in [attempt 1](https://github.com/paperclipai/paperclip/actions/runs/38763885288/attempts/1). The [bounded repeat](https://github.com/paperclipai/paperclip/actions/runs/38763885288/attempts/2) failed two existing snapshot-worker tests at their default five-second timeout. Its shorter duration is **not** counted as a successful performance sample. The latest head `3c894462` adds a targeted test-budget repair and [passes all 48 CI jobs](https://github.com/paperclipai/paperclip/actions/runs/38768433532) in **8m06s** (workflow creation to final required workflow job). This is **70 seconds / 12.6% shorter** than the 9m16s intermediate run, but the earlier 9m23s result remains part of the evidence: a stable average improvement is not established. Both branch-protection checks (`ci / verify` and `ci / e2e`) pass. The non-required Superagent Security Scan is still running; external PR scans are separate from this workflow duration. | Measurement | Before | After | What it establishes | | --- | ---: | ---: | --- | | General-server shard, paired on one Linux host | 430.94s | 292.99s | 32.0% faster; identical 1,047 case inventory | | Chat job 1 | 460s | 211s | Same full chat inventory spread across isolated partitions | | Chat job 2 | 408s | 274s | Serial cases within each child, separate PostgreSQL instances | | Chat job 3 | 408s | 216s | Total chat runner time fell from 1,276s to 701s | | Full PR workflow, first chat candidate | 9m16s | 9m23s | Not an end-to-end win in this pair: unchanged Build rose from 365s to 465s | | Full PR workflow, final repaired head | 9m16s | 8m06s | All 48 jobs pass; 70 seconds faster observed, not a controlled causal estimate | - Latest [paired benchmark](https://github.com/paperclipai/paperclip/actions/runs/38763885223) passes. Earlier paired server measurements were 33.3% and 37.4% faster. Unchanged fixture timing noise is not attributed to this PR. - All **1,337 chat case identities and outcomes match exactly** across [before-chat](https://github.com/paperclipai/paperclip/actions/runs/38760937988) and [after-chat](https://github.com/paperclipai/paperclip/actions/runs/38763885288/attempts/1), without duplicates or removals. Whole-suite records rose from 44,563 to 44,574: three new launcher checks and eight upstream warm-owner tests. Remaining name differences were generated fixture IDs/timestamps/PIDs, not removed tests. - The final run used `m7i-flex.xlarge` for Build (408s), with all chat jobs completing in 238s / 318s / 280s. It used the same actual merge base `afe5b341` as the first chat candidate. Optional profiling was disabled after the bounded measurement experiment; the full 48-job correctness workflow still ran. - The fleet used `m7i.xlarge` for the earlier Build and `m6a.xlarge` for the first chat candidate Build. Several unchanged compile/install/probe phases slowed together. This does not prove hardware alone caused the difference, and full-run attribution must account for runner/cache variance and the upstream base change. - Original full baseline: [all 48 jobs passed](https://github.com/paperclipai/paperclip/actions/runs/38753547876/attempts/2), 12m07s, including a server job that queued over five minutes. It is not a clean basis for attributing the whole elapsed-time reduction to this PR. An interrupted baseline and two misrouted candidate attempts were excluded and preserved; routing/trust rules were not weakened. - Local: 32 sharding/launcher checks pass, including disjoint coverage, shared-database fallback and a failed child that waits for its sibling. Real paired chat partitions passed 416 cases in 217.63s on independent embedded databases. - Both extracted Runner deadline regressions pass unchanged with real waits. An initial local attempt rejected a Node executable under the controller home; the identical binary in an owned non-home path passed without weakening that guard. - Real worker-isolation proof, actionlint and diff checks pass. Latest-head Greptile is **5/5**, with no review threads or actionable findings. - The eight snapshot-worker cases pass locally with a private `PAPERCLIP_HOME`. The first direct local invocation failed to open the default-home lock database under the sandbox; no production guard was changed. ## Risks Parallel files and paired chat databases may expose shared-state assumptions or increase peak memory. Cases stay serial within each invocation, and explicit shared databases keep the serial path. Complete server coverage and the full 48-job CI workflow pass. Check external required PR scans before merge; the full-workflow measurements are observational and do not establish a steady 12.6% speedup. Queueing, cache state and runner variance must be distinguished from execution improvements. ## Model Used OpenAI GPT-6 through Codex, with repository inspection, code execution and GitHub tools. The exact serving model suffix and context capacity are not exposed to this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green (48 jobs; both required branch-protection checks pass; non-required external security scan pending) - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
25b16e65fc |
fix(observability): detail workspace and connector preparation (#16454)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The server prepares a repository, sandbox workspace, and connector access before an agent can reply. > - Existing traces measure large preparation phases but leave important work inside those phases unclear. > - Repository source materialization and Browser Use discovery can do more work than their callers suggest. > - New detail must preserve access checks, run ownership, and the existing work order. > - This pull request adds fixed OpenTelemetry spans inside those operations. > - Operators can use one deployed trace to select the next performance fix. ## Linked Issues or Issue Description Refs: #16419, #16425, #16428. **What existing behavior does this improve?** Startup traces for server workspace preparation, native sandbox synchronization, and connector assignment resolution. **Current behavior** Workspace and connector aggregate spans can take many seconds. The trace does not separate private source cloning, seed reuse, archive work, progress persistence, or each connector resolver. Browser Use resource selection builds the full authorized tool catalog. The configuration and native backend can each resolve connector assignments. **Proposed behavior** Show the awaited work as fixed child spans. Separate discovery queue wait from catalog construction. Preserve the initiating run context when another run releases a scheduler slot. Add no new lifecycle event writes for workspace detail callbacks. **Reason and benefit** Operators need evidence about the hosted critical path before another performance change. These spans distinguish filesystem work, provider transfer, authorization, transaction entry, locks, and reporting overhead. **Breaking changes** None. Authorization, query order, transaction ownership, synchronization, existing progress events, and task results retain their contracts. The change adds opt-in Observability spans under the existing OTLP endpoint gate. ## What Changed - Trace private source directory setup, Git credentials, clone, requested ref, source identity, and checkout publication. - Trace sandbox source capture, seed cache/replay, shallow clone and archive, overlay copy/archive, verification, upload, warm stamps, and descriptor persistence. - Separate existing awaited progress reporting from useful workspace work. Keep new detail callbacks trace-only. - Trace each connector resolver and Browser Use catalog/grant work. Split discovery queueing, descriptor sources, and policy evaluation. - Trace Slack authority, credential reads, catalog synchronization, transaction entry/settlement, and advisory locks. - Update the real filesystem staging trace expectations and the Honeycomb investigation guide. - Add a real scheduler regression for a queued job released by another run. Assert run ownership, parent spans, and exactly one completed queue span. ## Verification - Passed: server and adapter-utils TypeScript checks (`pnpm --filter @paperclipai/server exec tsc --noEmit` and `pnpm --filter @paperclipai/adapter-utils exec tsc --noEmit`). - Passed: 252 checks across 9 focused files. These cover real filesystem staging, native workspace synchronization, trace ownership, endpoint gating, and workspace preparation. - Passed after review: 26 trace-context, tracing-failure, and discovery-scheduler checks across 3 files (these overlap the first run). The new queued-discovery regression fails when context binding is removed and passes after restoration. An initial submillisecond timestamp assertion was replaced with explicit span-completion order to avoid SDK clock precision assumptions. - Passed on implementation commit `0e68968a4bd61001893d833f281f12bb110edbe1`: `pnpm -r typecheck` and `pnpm build`. The later commit updates only an engine test expectation. - The first CI workspace lane found an old engine trace-tree expectation. It expected the earlier span set. The updated assertion now checks the new detail spans, their parents, and single completion. Its targeted executor check passes. - The full test suite, build, release registry, and end-to-end gates pass in CI on final head `40e37d89ef679109d60a96bb5731a0616efeaa43`: 55 passing checks and 2 optional Storybook skips. The full suite was not duplicated locally after the focused run. - Greptile reviewed the final head at 5/5. No unresolved review threads remain. - After deployment, filter Honeycomb by the hashed run ID and verify `service.version`. Inspect `heartbeat.prepare.workspace`, `runtime_selection.workspace_sync`, and the connector assignment parents. The guide in `doc/observability.md` explains each new family and the nested intervals. - No hosted run has used this new instrumentation yet. Local checks cannot establish hosted latency improvement. ## Risks - Tracing adds spans when an operator configures an OTLP endpoint. Detail remains capped at 16 spans per phase per execution attempt. - Parent and child durations overlap. The documentation explains how to avoid counting them twice. - A discovery job can start in another run's scheduler continuation. The bound callback preserves the initiating async context. - New spans export fixed names and the existing hashed run ID. They add no paths, repository URLs, connector resource IDs, credentials, tool schemas, or exception text. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, tool use, and code execution. The exact served model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
86c965ce7a |
fix(skills): expire unused version caches and restore saved contents (#16433)
Expire idle skill-version caches with durable runtime pins, bounded discovery, conservative retention, and database-backed restoration. Co-Authored-By: Dotta <cryppadotta@users.noreply.github.com> Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
Huanyi Xie
|
95692b7785 |
fix(docker): forward tool action signing secret in Compose (#15202)
## Thinking Path > - Paperclip can run through Docker Compose. > - Approval-gated tools need an independent signing secret. > - The guide sets that secret before starting Compose. > - Neither Compose file passes it into the container. > - This pull request forwards the operator's configured secret. > - Documented setup can issue signed tool-action approvals. ## Linked Issues or Issue Description Fixes: #15064 ## What Changed - Forward PAPERCLIP_TOOL_ACTION_SIGNING_SECRET in both shipped Compose environments. - Document persistent deployment secrets and reuse on restart. - Explain that it is independent from the session secret. - Keep an empty default so deployments without approval-gated tools retain their existing startup behavior. ## Verification - Docker Compose config before the fix drops a supplied synthetic secret in both files. - After the fix, both preserve the supplied value. - Both also resolve successfully when the optional signing secret is absent. - Used --env-file /dev/null and synthetic values. No containers were started. - `node --test scripts/docker-compose-secrets.test.mjs`: 2 passed. - The Docker context CI lane runs this integration test. - git diff --check passes. ## Risks The value is visible to the container as intended. Operators must protect it as a secret. This does not change admission rules or require a new startup value for existing deployments. ## Model Used OpenAI Codex, based on GPT-6, assisted with source review, implementation and local test execution. The exact deployment model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (targeted local checks and full remote CI pass) - [x] I have added or updated tests where applicable (Hermes regression / real Compose interpolation checks) - [x] I have updated relevant documentation to reflect my changes (Hermes has no configuration changes; Compose documentation is updated) - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge |
||
|
DottaandPaperclip
|
60e03c002d |
fix(grok): hold new runs while connection needs recovery (#16445)
## Thinking Path > - Paperclip manages agents and their work. > - Grok Bot assignments use a managed Runner and a verified connection. > - Stop can leave the connection held for remote-work inspection. > - The queue started new runs before it checked that connection. > - Those runs allocated Runners, failed, and caused more preparation attempts. > - This pull request checks readiness at queue admission. > - Pending input waits for recovery without using managed compute. ## Linked Issues or Issue Description Refs #16427 and #16422. The open Grok changes cover the coding adapter and do not duplicate this fix. **What happened?** A real Stop test held the Grok connection for inspection. A second task was pending. Four new preparation attempts acquired managed Runners, then failed because the connection was held. No new external assignment or webhook was sent. Pause stopped the loop, and all leases completed cleanup. **Expected behavior** Keep fresh input in the durable queue until its connection is ready. Do not allocate a Runner to discover a known connection hold. Keep existing session reconciliation under its recorded authority. **Steps to reproduce** 1. Connect Grok Bot and start an assignment. 2. Queue another task for that agent. 3. Stop the first assignment and leave the inspection hold in place. 4. Run queue recovery. The pending task repeatedly starts preparation and fails. **Paperclip version or commit** Observed on packaged Core |
||
|
Leia MarshallandLeia Marshall
|
e4ce989099 |
Add opt-in Gmail archive capability (#16331)
## Thinking Path > - Paperclip separates Gmail capabilities by reviewed OAuth scope and tool policy. > - Existing Gmail connections support reading and draft creation but not bounded archiving. > - Gmail archive requires `gmail.modify`, so it must not widen existing read or draft grants. > - This change adds an opt-in archive profile and limits it to removing only the `INBOX` label. > - A final pre-dispatch check preserves that boundary for approved and replayed calls. ## Linked Issues or Issue Description **What existing behavior does this improve?** This improves the existing Gmail connection and tool-governance behavior. **Subsystem affected** Cross-cutting: `packages/shared` and `server`. **Current behavior** Paperclip can read Gmail and create drafts. It does not provide an opt-in, bounded action for archiving a thread. **Proposed behavior** Paperclip offers a separate Read & archive connection. It exposes `unlabel_thread` only for that connection and accepts only `threadId` with `labelIds: ["INBOX"]`. **Reason and benefit** Users can archive processed Gmail threads while sending, trashing, and other label changes remain disabled. **Breaking changes** None. Existing read and draft methods keep their current scopes and defaults. Related work: #15545 adds a unified Google Workspace connector and keeps Gmail draft-only. #15412 adds a separate Gmail REST server without label mutations. Neither PR adds this bounded archive capability. ## What Changed - Added the `gmail.archive` connector profile. - Added managed and customer-owned archive connection methods. - Added an Inbox-only schema and final pre-dispatch argument guard. - Classified archive as destructive and added Ask-first defaults for the archive methods. - Added permission-review records and focused tests. ## Verification - `pnpm --filter @paperclipai/server typecheck` - `pnpm --filter @paperclipai/server build` - `pnpm --filter @paperclipai/shared typecheck` - Focused Vitest suites: 42 tests passed. - App-definition ingestion: 103 definitions and 0 review flags. - `git diff --check` ## Risks The main risk is accidental scope or operation expansion. The new profile is opt-in. Existing profiles retain their scopes. The gateway validates the exact Inbox-only payload immediately before dispatch. Production rollout requires separate OAuth configuration and fresh consent. ## Model Used Unverified — reliable runtime model metadata was not available. Confirm the model attribution before publishing. ## Checklist - [x] I have included a concise design rationale - [x] I have specified that model attribution is unverified - [x] I have checked ROADMAP.md and found no Gmail archive entry - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in this PR - [x] I have not referenced internal or instance-local issues - [x] My branch name describes the change - [x] Local tests and builds pass - [x] I have added or updated tests - [x] I have updated relevant documentation and permission records - [x] I have documented risks - [ ] All hosted CI gates are green - [ ] Greptile and human review are complete --------- Co-authored-by: Leia Marshall <leia@paperclip.ing> |
||
|
DottaandPaperclip
|
afe5b3418d |
fix(native): verify warm ownership before configuration rotation
Track the exact prior warm-owner run ID separately from mutable configuration. Keep configuration retirement, terminal-state checks and failed-close fences. Add current-v5 handoff regressions and document the recovery boundary. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
7ec6fd0972 |
perf(ci): remove repeated test setup and fixture work (#16429)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - CI must check changes without making contributors wait for repeated setup. > - A complete CI profile showed 158 serialized Vitest launches and repeated dependency installs. > - A large fixture also transferred 40,000 files to prove two different size boundaries. > - This PR removes repeated controller, install, build, transfer, and wait work. > - It keeps isolated test processes and all useful regression boundaries. > - Paired command benchmarks check speed and test inventories before we claim a benefit. ## Linked Issues or Issue Description Refs #16423 for the complete CI profiling data path. No duplicate optimization PR was found. **Current behavior** Each serialized file starts a new Vitest controller. Each CI lane restores the pnpm store and installs dependencies. Large Git scanning and staging share one 40,000-file fixture. A readiness regression waits over a minute for the default database idle deadline. **Proposed behavior** Use one controller per serialized shard, with a fresh fork for each file. Restore exact installed dependencies and certified shared/SDK build outputs when inputs match. Keep the normal install and build paths on a miss. Separate the 32 MiB Git scan proof from staging above command argument limits. Cross a real database idle deadline with a one-second fixture setting. **Reason and benefit** The paired Linux CI benchmark reduced the complete 566-case serialized shard from 303.43s to 246.96s (18.6%). The Git/database fixture command fell from 156.54s to 33.78s (78.4%). Exact dependency/build restoration reduced setup from 56s to 19s (66.1%). These are measured command/setup times. They do not establish a controlled reduction in the full workflow's queue-to-completion time. **Breaking changes** None to application behavior. Tests still run in full. Test results are not cached. Cache misses preserve the existing lockfile resolution fallback. ## What Changed - Batch serialized files into one isolated, single-worker Vitest invocation per shard. Give each worker its own home, config and instance ID. - Add a real two-order isolation check for processes, environment, globals, timers and filesystem state. - Add exact dependency keys for all manifests, lockfile, workspace settings, patches, installer, runtime, platform, libc, pnpm, store and checkout path. Cache only installed package directories, not source fixtures. - Add a separate content key for the shared/SDK dependency graph. Retain the existing source/output fingerprint verifier and rebuild path. Only an exact canonical master push can populate normal caches. Stale lockfile fallback runs cannot publish entries. - Keep four real Git filename lanes above 32 MiB. Use a separate 4,000-file staging/deletion case above 2 MiB. Check every staged file, unusual names, cleanup and bounded commands. - Configure a real one-second PostgreSQL idle timeout in the readiness fixture. Assert that the backend connection changes and that another connection can write the parent row. - Add an opt-in, unprivileged Linux benchmark under `profile-ci`. It compares the tested merge tree with its actual first parent on one runner. It also removes installed outputs and qualifies a real cache restore using temporary PR-scoped entries. ## Verification - Local Node 24.19.0: unchanged serialized shard 255.69s; candidate 231.44s; all 566 cases pass in both. - Local fixture command: unchanged 189.13s; candidate 43.39s. Candidate scan 18.58s, staging 13.41s, database case 2.80s. All three candidate cases pass. - Negative control: moving readiness back inside the real parent-lock transaction fails the shortened regression at the parent-write assertion. The production source was restored and is absent from the diff. - Worker isolation check passes in both file orders. - 565 CI policy, partition and workflow regression checks pass with local HTTP/IPC access enabled. The first sandboxed attempt was denied listen access; it was not a code failure. - `actionlint` passes for the three changed workflows. - [All 48 main CI jobs passed](https://github.com/paperclipai/paperclip/actions/runs/38745643045) on head `5bec2edb2b194e74352ca697f3fd1d0ac03ce967`, including full typecheck, tests, build, browser E2E and release checks. All 158 serialized files and 3,250 cases passed through 9 controller invocations. Full typecheck/test/build were verified in CI; they were not all repeated locally. - [Paired Linux benchmark passed](https://github.com/paperclipai/paperclip/actions/runs/38745643127) on Node 24.21.0. Its baseline is `57fa6b358de62922cfa9a67766b2a227c434fcdf`; its candidate merge tree is `f68f1b32966090f11647502e31852130c20237cf`. The artifact contains results, exact case inventories, setup times and raw logs. Serialized case identities match exactly. All three fixture contracts execute in both variants. | Measured work | Baseline | Candidate | Reduction | |---|---:|---:|---:| | Complete serialized shard, 566 cases | 303.43s | 246.96s | 18.6% | | Git/database fixture command | 156.54s | 33.78s | 78.4% | | Dependency/build setup, cold vs exact hit | 56s | 19s | 66.1% | - [Honeycomb profile export succeeded on its second attempt](https://github.com/paperclipai/paperclip/actions/runs/38747237615/attempts/2). The first attempt failed on GitHub HTTP 503 while downloading an artifact, before export. Only the export was retried; tests were not repeated. The temporary PR-scoped qualification caches were removed after validation, reclaiming 3.34 GB. Timing artifacts remain available. - Greptile is 5/5 on the current head, with no unresolved review threads. Its P1 checkout-prefix finding was fixed in `5bec2edb2`; the corrected reporter paths and paired comparison passed. Earlier runs were superseded while this was fixed. A copy-batching experiment did not speed up the Git fixture and was replaced by the two-boundary fixture split. ## Risks - A persistent controller retains transform state longer. The local shard's maximum RSS rose from about 1.52 GB to 1.99 GB. Full CI must confirm all shards fit the runner budget. - Materialized caches are larger than metadata caches. The Linux benchmark measured real archive transfer and validation after removing installed outputs. Its hit took 19s versus 56s cold. Normal PR lanes cannot publish cache entries. Exact misses run installation. Canonical cache warming requires a frozen-lockfile install; the existing stale-lockfile fallback remains uncached. - The PR uses the trusted reusable workflow from master. The paired benchmark directly exercises the candidate dependency action; normal workflow adoption still requires this PR to merge. This PR does not change the trust references. The completed main workflow took 11m02 including queue time. No controlled full-workflow latency improvement is claimed before this workflow adopts the cache change. - Only deterministic dependency/build outputs are reused. Broad dependency-based test omission and caching successful test results are deliberately excluded from this change. - The real Runner lease minimum and cold-admission waits remain. They protect protocol boundaries and were not shortened by weakening production limits. ## Model Used OpenAI GPT-6 through Codex, with reasoning, shell execution and code tools. The runtime does not expose a more specific deployment ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
69eb3b6eff |
perf: reduce repository admission filesystem and write overhead (#16425)
Reduce repository preparation latency by resolving each distinct workspace path once per scan, loading overlapping root provenance, and batching source retention writes. Preserve company authorization, fresh alias checks, and preparation timing. Add a reproducible benchmark and regressions for source identity retention, symlink retargeting, and bounded filesystem and database work. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
5a9e6b6315 |
fix(grok): avoid false file recovery after Stop (#16427)
## Thinking Path > - Paperclip lets people manage AI agents and their work. > - Grok Bot uses a managed Runner and a fixed external work session. > - Its admitted contract gives it no access to Paperclip workspace files. > - The Runner still has a sync descriptor to support durable recovery. > - Cancellation treated that descriptor as proof that files needed repair. > - This pull request uses the admitted access contract while preserving ownership settlement. > - Users get the correct recovery action after Stop, with native file safeguards intact. ## Linked Issues or Issue Description Refs #16219 and #16422. The duplicate search found no open Grok workspace correction. **What happened?** A real Grok Stop test cancelled its run, fenced its assignment, denied a stale progress update and cleaned up the sandbox. Paperclip also created a file-repair hold. The recovery panel told the user to recover workspace files, although the admitted external contract has workspace access set to none. **Expected behavior** Keep the stopped session fenced and require inspection before new work. Keep failed-run ownership settlement. Do not invent a file export obligation for a session that cannot access the workspace. Native sessions with writable or unknown workspace access must keep their file safeguards. **Steps to reproduce** 1. Connect a Grok Bot and assign a task. 2. Let it accept the task and save a Paperclip checkpoint document. 3. Use Stop while it is working. 4. Inspect the task recovery panel. It asks for file repair after sandbox cleanup succeeds. **Paperclip version or commit** Observed on Core |
||
|
DottaandPaperclip
|
57fa6b358d |
fix(observability): separate workspace finalization and copyback spans (#16428)
Give native workspace finalization its own span name. Keep copyback and commit as separate child measurements, with an exported-span regression that preserves the workspace operation result. Validation: 40 runtime/tracing tests and server typecheck passed locally. Final-head CI, E2E and security checks passed. Apex review: 5/5, zero findings and no unresolved threads. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
cccde7122b |
Trace execution preparation and durable response persistence (#16419)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The server prepares an execution environment before it starts an agent. > - Operators need to distinguish preparation, provider wait, and response persistence. > - Existing preparation phases start separate traces, and the native session trace ends before the final response is saved. > - This pull request adds an execution-attempt trace and detailed spans at the existing work boundaries. > - The benefit is a measurable startup and settlement path without changing execution policy. ## Linked Issues or Issue Description **What existing behavior does this improve?** Operator-configured OpenTelemetry tracing for agent preparation, startup, and durable response storage. **Subsystem affected** Server orchestration, native Runner startup, and the Honeycomb run link. **Current behavior** Preparation phases start independent roots. Workspace lock waits, authorization groups, runtime asset work, provider-pack fallback, and the gap after provider completion are hard to attribute. **Proposed behavior** Use a server-owned async context for each execution attempt. Parent nested work to that attempt. Link the separately sampled native session because its ingress and queue history can begin earlier. Record transaction entry, lock waits, grouped authorization, asset preparation, artifact fallback, copyback, and response commit. **Reason and benefit** Operators can identify expensive work from recorded boundaries instead of inferring the cause from elapsed time. Nested and concurrent durations remain visible and must not be added together. **Breaking changes** No API or execution-policy change. New Observability span names and closed count attributes are documented. The existing local run-performance schema also records the additional provider-pack and harness measurements. Refs: #16178, #16205, #16330. ## What Changed - Add `task.execution_attempt` with same-run async ownership, independent root sampling, and native-session links. - Measure workspace transaction entry and settlement, advisory and row waits, source scans, reader groups, notifications, and writes. - Measure runtime selection and each concurrent instructions, skills, and MCP branch. Separate manifest, cache, materialization, and verification work. - Measure provider-pack discovery, verified reuse or upload, final verification, and remote harness state checks. - Measure workspace settlement, accounting, liveness, response authorization, comment commit, and publication enqueue. - Preserve the silent endpoint-off path, including when another library has registered a tracer. Bound repeated detail spans to 16 per name per attempt. - Extend the Honeycomb run link and document the actual coverage and remaining gaps. ## Verification - After resolving the conflict with retained-reader batching on `b98cdaea531c8447637cd4dcc19c61b3f5f41844`, all 61 targeted query, authorization, workspace-preparation and trace-boundary tests passed. The server typecheck passed. Reader tests still bound query count, avoid loading retained run payloads and enforce fresh project grants on every publication attempt. - Before that base integration, the tracing/exporter group passed 855 tests with 2 optional-SDK skips across 10 files and no inherited OTLP endpoint. It covers real PostgreSQL waits/rollback, durable response commit, assembled heartbeat preparation, concurrent parentage, sampling and tracer failure isolation. Native workspace/history tests passed 23 cases. These results remain supporting evidence for the unchanged tracing paths. - Full workspace typecheck and build passed during implementation. Full sharded CI, release validation, E2E and Apex 5/5 passed on prior head `8cebad80f3856dbcff6f3a6d5d34880ff77a20f2`. Fresh CI, security and Apex review are required on the current head before merge. - `pnpm check:token-gates` passed. No styles changed. - Earlier CI runs exposed exact-source-expression assertions changed by wrappers, missing run hashes on transaction-boundary spans, and sampling fixtures without an enabled endpoint. The fixes preserve the calls, add the declared hash and span cap, and enable tracing explicitly in the sampling fixtures. Focused repairs passed; the full prior-head CI passed after those repairs. Earlier failed run: https://github.com/paperclipai/paperclip/actions/runs/38729131366. - The full local `pnpm test:run` was stopped after the base advanced during that process. Its mixed-version results are not final-head evidence. A full local-suite pass is not claimed. - The old-head Superagent scan stalled at 35%. The current head has a fresh scan; no security pass is inferred from the old pending result. - No deployed Honeycomb acceptance is claimed by this code change. ## Risks - Enabled tracing creates more spans. Repeated detail spans are bounded, and inventories use group spans. - The attempt and native session remain linked traces. Historical queue time can precede the attempt, and nested intervals overlap. - A committed comment proves durable server storage. Publication enqueue and realtime dispatch do not prove external delivery or browser paint. - New custom attributes use only the existing run hash and validated file/byte counts. No SQL, paths, permissions, content, or exception text is added. ## Model Used OpenAI GPT-6 through Codex. The serving model ID and context window size are not exposed in this session. The agent used reasoning, tools, code editing, and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
7420af26fc |
fix(ci): finish large Honeycomb profile exports (#16423)
## Thinking Path > - Paperclip uses pull request CI to check changes before merge. > - The merged CI profiler records a selected full PR suite in Honeycomb. > - A real 48-job profile exposed an export failure after Honeycomb accepted only part of the trace. > - The same run exposed three fixture subprocesses that looked like incomplete test launches, one direct reporter without a launcher marker, and one overlong test name. > - The exporter needs to handle temporary ingest rejection without sending accepted spans again. > - This pull request fixes those cases and documents the trace display limit. > - The benefit is a complete, checkable CI waterfall with test-file timings. ## Linked Issues or Issue Description **What happened?** The first live `profile-ci` run on #16423 completed all 48 jobs, but the post-run exporter stopped when Honeycomb rejected part of a batch. The initial trace had no root span. Its coverage flag also reported four unfinished invocations and one invalid timing record. **Expected behavior** The exporter should pace large uploads, retry only temporarily rejected events, and publish the root after every child is accepted. Completed reporters should count as complete. Long test names should not erase test timing. **Steps to reproduce** 1. Add `profile-ci` to a CI CODEOWNER PR with the merged profiler. 2. Run the full PR suite, as in [run 38732093469](https://github.com/paperclipai/paperclip/actions/runs/38732093469). 3. Inspect the separate [failed exporter run](https://github.com/paperclipai/paperclip/actions/runs/38733393370) and the trace coverage fields. **Paperclip version or commit** `12612b06eb4332844ac4c8770a1fb6c6b7eccbc9` (PR run head, based on merged #16378). ## What Changed - Pace Honeycomb uploads and retry only transiently rejected events from each batch. - Preserve overlong test timings with bounded names and count complete reporter pairs without a launcher record. - Keep nested Vitest fixture runs from inheriting CI profiling and creating false coverage gaps. - Clarify ordinary CI access, large-trace display limits, and export behavior in the operator guide. ## Verification - [Full PR suite](https://github.com/paperclipai/paperclip/actions/runs/38732093469/attempts/1): 48 of 48 jobs succeeded; 10m 29s wall time. The selected jobs used the AWS public PR fleet. The protected exporter used a separate GitHub-hosted runner. - [Clean Honeycomb file waterfall](https://ui.honeycomb.io/paperclip/environments/test/datasets/paperclip-ci/trace?trace_id=01ed0c541b0442e613a030ffb6dc8d9b&trace_start_ts=1792065955&trace_end_ts=1792066584): 48 jobs, 822 steps, 2,622 file spans, and `ci.profile_complete=true`. It was reconstructed from that run's 45 timing artifacts after identifying three nested fixture records. The full trace also recorded 44,497 tests; Honeycomb's chart omits some deep test spans because it displays at most 32,000 spans per trace. - The original exporter stopped after a partially accepted batch. A local replay of that batch succeeded. I used the authorized local test-environment key to complete the original upload and create the file-level view. No key was added to this branch or PR. - `node --test .github/scripts/tests/ci-profile.test.mjs scripts/ci-profile/reporters.test.mjs`: 18 passed. - `pnpm exec vitest run server/src/__tests__/vitest-chat-shards.test.ts`: 1 passed. - `git diff --check`: passed. - The local `pnpm -r typecheck` and profiled fixture rerun could not start: the package-manager shim could not verify the `pnpm@9.15.4` registry signature because its fetches failed. `pnpm test:run` and `pnpm build` were not run locally; hosted CI is pending. - The updated exporter has not yet run from protected `master`; this PR remains draft until CI and review finish. ## Risks - A 48,000-span export now takes about three extra minutes due to pacing. Retry limits still fail a persistently rejected batch and keep the root unpublished. - Honeycomb may retain duplicates from an already partial export. The clean file-level evidence trace uses a separate trace ID. - The API key stays only in the protected `ci-profiling` environment. PR jobs write timing artifacts but never receive it. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — see `CONTRIBUTING.md`. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, code execution, GitHub API, and browser inspection. The runtime does not expose the exact serving model ID or context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
ea4aa185b0 |
perf: batch retained-reader authorization reads (#16424)
Batch retained-reader project checks through the existing multi-actor read API. Share raw database reads within one call, retain actor-specific privacy and supplied membership data, and avoid loading large saved-run payloads for identity checks. Keep current-reader writes, distinct saved-run policies, company root fences and later-call revocation checks. Add query-count and access-control regressions. All 255 affected cases and full repository typecheck/build pass. Hosted response-time acceptance remains separate. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
7f9d3d1929 |
fix(grok): avoid routine wakes for proactive admission (#16422)
## Thinking Path > - Paperclip lets people manage AI agents and their work. > - Grok Bot can request governed work from its own conversation. > - Managed Runner admission can take longer than the initial MCP response. > - The dispatcher treated a delayed offer as a reason to start another routine. > - The original conversation was already waiting for that same work. > - This pull request keeps admission in its original request and cancels old prepared echoes. > - Users get one task interaction while normal assigned-task wakes remain available. ## Linked Issues or Issue Description Refs #16219 and #16394. The duplicate search found no other proactive Grok correction. **What happened?** A real idle Grok conversation created one Paperclip task and reached accepted completion. During admission, the dispatcher also sent a routine webhook after the offer was ten seconds old. Two acceptance requests and two title writes were recorded. The extra routine can compete with the original conversation. **Expected behavior** Immediate and delayed Bot-initiated admission stay with the original request. The Bot checks the same request ID and arguments until admission succeeds or fails. A new routine must not echo that work back to the Bot. **Steps to reproduce** 1. Ask an idle Grok Bot to create and complete a Paperclip task. 2. Let managed admission return pending. 3. Leave its offered assignment unaccepted for more than ten seconds. 4. Run the dispatcher. It sends another routine webhook for that proactive offer. **Paperclip version or commit** Observed through the actual Grok client on Core |
||
|
DottaandPaperclip
|
6acec4266e |
feat(ci): profile full CODEOWNER PR suites in Honeycomb (#16378)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Its PR workflow runs dozens of parallel build and test jobs. > - Individual job durations do not show the elapsed cost of the whole suite. > - Maintainers need one shared timeline to find slow work and repeated setup. > - This pull request adds full-suite profiles for selected CI CODEOWNER PRs. > - Honeycomb shows each job, step, test file, and supported test on that timeline. > - The benefit is evidence for changes that reduce total CI time. ## Linked Issues or Issue Description **What existing behavior does this improve?** The main PR workflow has about 48 jobs. Maintainers need to compare their overlapping work and find which jobs delay completion. **Current behavior** GitHub shows job and step durations separately. Test files and tests are not joined to a full workflow trace. **Proposed behavior** A CI CODEOWNER adds `profile-ci` to a PR. The tested code stays on that PR branch. All main PR lanes run, including lanes normally skipped for stack position or release notes. A trusted post-run exporter sends one wall-clock trace to Honeycomb. **Reason and benefit** The trace supports decisions about which CI work to speed up or remove. Ordinary PRs do not export traces. The roadmap and related-PR search found no duplicate CI profiling work. ## What Changed - Add label selection and a full-suite override. Verify the PR author, original actor, and rerun actors against the trusted `.github/**` CODEOWNERS rule. - Record Vitest, Playwright, and Node test file/test timings in local artifacts. Keep ordinary reporter behavior and failing exit codes. - Add a trusted `workflow_run` exporter. Use actual GitHub timestamps for all jobs, setup, steps, waiting, and teardown. Mark missing test data and partial reruns. - Keep the ingest key out of PR jobs. Store it only in the `ci-profiling` environment, restricted to `master`. The exporter reads bounded JSON and never executes artifact code. - Record expected test invocations outside reporters so a missing invocation cannot appear complete. Preserve `node --test` on older PR checkouts. Flush large shard inventories before exit. - Add security, timing, routing, and real-reporter checks. Document operation and activation. Protect the exporter with CODEOWNERS. ## Verification - 509 focused workflow/reporter/shard checks passed. They include outside authors, outside rerun actors, bots, stale revisions, malicious ZIP paths, incomplete reports, and partial ingestion failures. - Final revision `3bf5d9017d` adds multi-file qualification and fixes short-file step parenting and duplicate Node file-wrapper events. Greptile reviewed this exact revision at 5/5 with no unresolved threads. [Main PR CI run](https://github.com/paperclipai/paperclip/actions/runs/38728040302): the initial attempt had 54 successful checks, 3 intentional skips, and one serialized-shard failure from an unchanged workspace-authorization test exceeding its 15-second timeout. The single failed-shard retry passed: the final result is 55 successful checks and 3 intentional skips. The same timeout reproduces locally with profiling disabled; both assertions pass with a diagnostic 60-second timeout override (about 23 seconds total). No source or CI timeout was changed. - A dry run against real GitHub attempt 2 of run `38677445871` produced exactly 3 executed jobs, 40 spans, and 510 seconds, excluding 45 carried-over jobs. The partial trace remains explicitly incomplete; this validation did not upload data. - `actionlint` passed for all four changed workflows. `git diff --check` passed. - Full Node 24 `pnpm -r typecheck` and `pnpm build` passed. The local full `pnpm test:run` was stopped after it reported a setup-token timeout and a distribution-plugin-catalog fixture failure. The 30 setup-token tests passed on focused rerun. The plugin fixture fails at `rmSync` of a directory symlink on this macOS/Node 24.11.0 host; the same failure reproduces with the unchanged base Vitest configuration and unchanged test/service files (10 other tests pass). The local full test suite is therefore not claimed green. - A real root-config Vitest probe passed 13 tests with profiling both off and on. Multi-file Vitest, Playwright, and Node probes each verify two distinct filename rows and four correctly parented tests, including repeated test names across files. Separate passing probes exercise the full exporter hierarchy `workflow → job → step → filename → tests`. Intentional failing probes preserve failing exit status. - Verified all 48 measured jobs in latest-head run `38728040302` used AWS: one `paperclip-ci-routing-x64` gate and 47 `paperclip-public-pr-x64` jobs. Profiling preserves AWS authorization and its GitHub-hosted fallback. The separate privileged exporter runs on `ubuntu-latest` outside the measured suite. - [Live Honeycomb baseline](https://ui.honeycomb.io/paperclip/environments/test/datasets/paperclip-ci/trace/oaPMzw8zGwG): 48 jobs, 731 spans, 479 seconds. It matches [GitHub run 38671274294](https://github.com/paperclipai/paperclip/actions/runs/38671274294). This historical trace has job/step data only; it is not proof of the future label-triggered test-reporting journey. - Verified the dedicated Honeycomb key is ingest-only with dataset creation disabled. Verified the GitHub environment allows only the `master` branch. No repository-level secret of that name exists. Both credential values were scanned absent from source and branch history. After merge, add `profile-ci` to a CI CODEOWNER PR, wait for the entire PR run and exporter, then follow the run-summary link. Collapse job children to compare parallel lanes; expand a job to inspect its steps and tests. ## Risks - Activation requires merging the trusted workflow definitions to `master`. This PR continues using the current trusted reusable workflows until then. - Selected runs produce many spans and artifacts. Oversized artifacts fail export instead of truncating data. Missing or cancelled reporters are marked incomplete. - GitHub timings have second precision. Node test times use reporter receipt time. Rust retains job/step timing but lacks per-test wall timestamps. - CI CODEOWNERS are individual users in the explicit `.github/**` rule. Unsupported team rules and identity lookup failures disable profiling. - An interrupted upload can leave partial data. Retrying uses the same span IDs, but the Events API does not promise deduplication. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, code execution, repository tools, and browser verification. The runtime does not expose the exact serving model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
7fc2dc8b32 |
fix: isolate workspace snapshots from controller work (#16414)
Move native sandbox snapshot captures off the controller callback queue while preserving the existing capture and restore contract. Keep legacy adapters inline, bound captures to two workers, and verify cleanup and reordered worker completion. Validation: 461 affected cases, full typecheck/build, exact-head CI/E2E/Superagent and Apex 5/5. The full local canonical suite remains running; no full local pass or under-30-second staging response is claimed. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
Devin Foley
|
8365872743 |
fix(codex-local): retry ACP model capacity failures as transient upstream (#16407)
Classify typed Codex ACP `service` failures about model capacity or server overload as `codex_transient_upstream` / `transient_upstream`, so the existing bounded transient retry and continuation recovery apply. In `unexpected` Sentry mode, skip only runs carrying that full classified evidence; `all` mode still reports them. |
||
|
Devin FoleyandClaude Opus 5.5
|
11d69d04c7 |
fix(native-runtime): report remote recovery labels thrown before coordinator save (#16406)
Restart reattachment can fail in the executor before the native coordinator saves its result. The run failure report now reads the closed remote recovery stage and reason from the caught error when the saved result lacks them. The saved result keeps priority and the same closed-vocabulary validation applies. Add a regression for thrown, wrapped-transport, saved-priority, and out-of-vocabulary cases. Full current-head CI and Greptile 5/5 passed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Devin FoleyandPaperclip
|
307c27e3d8 |
fix(native): reclaim task execution before same-run recovery
Restore the original task execution binding atomically during native same-run recovery. Preserve current owners, stops, pauses, assignments, and dependency waits. Isolate candidate failures, bound deferral writes, prioritize undated retries, and allow recovery ahead of queued waiters without premature queue promotion. Add PostgreSQL attachment, ownership, contention, fairness, and queue regressions. Verification: 99 focused tests, workspace typecheck, build, full current-head CI, and Apex 5/5. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
af4191f45e |
fix: keep workspace notices from suppressing final replies (#16400)
Keep stored system notices visible without letting them occupy the final reply slot. Read presentation metadata in live and recovery selection, and cover explicit replies, progress receipts, company boundaries, and native replay. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
|
8d9d2f9ddf |
fix(recovery): preserve deliberate stops and dependency waits
Preserve deliberate assignee blocks and operator stops during dependency and run-release recovery. Attribute native waits to their bound owner. Ignore unrelated edits and shared-workspace exports from another issue. Continue after genuine dependency changes while preserving independently authorized user input. Keep the original contributor's implementation and review history. Add native ownership, unrelated-edit, later readiness, and shared-workspace regressions. The shared-workspace loop was reproduced before the fix. Co-Authored-By: Devin Foley <devin@paperclip.ing> Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
c68a483cd9 |
fix: replay prepared warm seeds when generation caching is unavailable (#16399)
Replay this preparation’s verified private workspace seed after failed warm validation, including when the generation cache refuses publication. Preserve host-current semantics and rebuild if the private seed is missing or changed. Verified 89 focused cases, full typecheck/build, exact-head CI/E2E/security and Apex 5/5. Live PAP latency acceptance remains separate. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
5bc4e878e7 |
fix: reuse verified seeds after warm workspace fallback (#16398)
Preserve the verified company/workspace seed-cache scope when a warm sandbox requires host-current staging. Keep remote identity checks, archive digests and rebuilding for missing or corrupt cache generations. 85 focused cases and full repository typecheck/build pass. Exact-head full CI/E2E, Superagent, legacy Snyk and Apex 5/5 pass with no unresolved findings. No staging response-time claim is made. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
79c34058cd |
fix(native): retain unexported workspaces after execution failure (#16397)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native agents can edit files in a remote sandbox before they submit a result. > - A process identity or startup failure can end the run before file export. > - Cleanup can then delete the sandbox and its unpublished files. > - This pull request retains the exact source and records a file repair action before cleanup. > - Operators can recover the files without replaying the failed turn or changing the task owner. ## Linked Issues or Issue Description **What happened?** A native remote process had completed local edits when process identity recovery failed. The run still had a prepared workspace reference and no accepted result. Cleanup deleted the source sandbox. No file recovery card appeared. A restarted run can hit the same path during setup, before the native executor starts. **Expected behavior** Retain the exact unexported source. Show the existing file recovery card. Require source stop and file recovery evidence. Keep task ownership, status, and external action reconciliation separate from file repair. **Steps to reproduce** 1. Run a native agent in a remote ephemeral workspace and make unpublished edits. 2. Fail process identity recovery before result acceptance or file export. 3. Allow failure handling and lease cleanup to run. 4. Before this change, cleanup can destroy the source without a file repair action. Refs: #16341 for related process identity diagnostics. Refs: #16349 for startup ownership fencing. This change addresses file retention after failure. ## What Changed - Record source retention and a file recovery action in the terminal native failure transaction. - Cover restarted preparation failures before the terminal setup writer and cleanup run. - Require the exact company, run, lease, provider allocation, and workspace binding. Require the exact live recovery claim to settle an owned preparation failure. Refuse stale or competing claims. - Reuse the existing stop-only cleanup and file-only reconciliation path. Leave accepted results and scheduled same-run recovery on their existing paths. - Document the retention rule and add database and runtime boundary regressions. ## Verification - All 1,242 tests passed across the full heartbeat recovery suite, native executor, native source retention, runtime teardown, run completion, and legacy file recovery. - The integrated missing-descriptor regression fails before the inner-handler fix and passes after it. The runtime retention and owned restart regressions also reproduce their prior failures. - The tests cover exact source identity, current and stale claims, atomic rollback, cancellation, idempotence, and file-only repair on a completed task. - `pnpm -r typecheck` passed on the final code. - `pnpm build` passed on the final code. - Full repository CI passed on `3945eb2ba5c228c5d4992da02b765dcf2e634cb8`: 54 successful checks and two skipped Storybook checks. - Apex scored 5/5 on that exact commit. Both earlier findings are fixed and all review threads are resolved. - The unsharded local `pnpm test:run` is still in progress. It is not counted as a local full-suite pass; complete suite coverage is verified in CI. ## Risks - Failed runs can retain a sandbox that needs operator cleanup after file recovery. This uses the existing retention policy and recovery controls. - This does not recover files from allocations that were already deleted. - No schema migration, UI change, or telemetry change is required. ## Model Used - OpenAI GPT-6 through Codex. Tool use, code execution, and reasoning assisted this change. The environment identifies the model as GPT-6; it does not expose a more specific serving snapshot or context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
074c67fea8 |
fix(grok): avoid routine wakes for accepted work (#16394)
## Thinking Path
> - Paperclip lets people manage AI agents and their work.
> - Grok Bot receives assignments through routine webhooks and uses
agent MCP tools.
> - Each accepted assignment belongs to the current external
interaction.
> - A pending tool result or comment could start another routine while
that work was still active.
> - Quiet MCP traffic cannot prove that Grok has stopped working.
> - This pull request keeps receipts and feedback durable for the
current interaction to read.
> - New task routines start only for offers that the Bot has not
accepted.
## Linked Issues or Issue Description
Refs #16219 and #16305. The duplicate search also found #12991, which
covers instruction files for local Grok runs.
**What happened?**
The dispatcher could start another routine for an unread result after a
period without a routine reply. It could also wake accepted work for a
task comment. A queued delivery could still start after the Bot had
claimed the offer.
**Expected behavior**
Accepted work stays in its current interaction. The Bot polls the
original operation receipt and reads queued feedback at a safe boundary.
Acknowledgements of a sent wake remain valid after acceptance.
**Steps to reproduce**
1. Accept a Grok assignment and leave it active.
2. Add an unread comment, or complete a tool operation whose call
returned pending.
3. Run the dispatcher after its quiet period.
4. Observe another webhook call. Also queue an offer wake, accept the
offer before sending, and observe that delivery still starts.
**Paperclip version or commit**
Observed during real-client qualification on Core
|
||
|
DottaandPaperclip
|
854ab294cc |
fix: restore repository admission and Claude startup on Linux ARM64 (#16372)
Batch retained workspace admission reads while preserving individual privacy and trust decisions. Preserve bounded saved-policy projections when integrating current master, and qualify the pinned Claude executable on Linux ARM64.
Verified exact head
|