mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-21 16:27:18 +02:00
cfa9dce295970a85b3d8ee14cab0eccf50bc72e2
5343
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Dotta
|
cfa9dce295 |
refactor(runner): remove unreferenced Paperclip Runner test fixtures (#16543)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - The Paperclip Runner package keeps fake agent servers and versioned
profile identity declarations under `test/fixtures/` for its test
suites.
> - Five of those fixture files have no reference anywhere in the
repository: no test, script, generator, workflow step, or document loads
or spawns them.
> - Dead fixtures add maintenance cost and make the referenced sibling
fixtures harder to trust.
> - This pull request deletes only those five unreferenced fixture files
and adds an audit receipt with the reference evidence.
> - The benefit is a smaller test-fixture surface with zero loss of
executed coverage.
## Linked Issues or Issue Description
**What existing behavior does this improve?**
The test-suite hygiene of the Paperclip Runner package. The repository
keeps unused test-support files that no suite loads. This slows audits
and makes the referenced fixtures harder to trust.
**Subsystem affected**
`packages/paperclip-runner` test fixtures.
**Current behavior**
Five files under `packages/paperclip-runner/test/fixtures/` have zero
references anywhere in the repository:
- `fake-native-resume-codex-app-server.mjs` (277 LOC): an old fake codex
app-server for resume testing. Resume coverage now runs through
`fake-final-burst-codex-app-server.mjs` (used by
`server/src/services/native-runtime/native-session-resume.test.ts:287`)
and `fake-durable-codex-app-server.mjs` (used by
`packages/paperclip-runner/src/live/live-session.test.ts:2192`).
- `copilot-profile-v{3,4,5,6}-identity.json` (64 LOC total): orphaned
profile identity declarations. Copilot profile tests,
`native-backend-factory.test.ts`, and
`scripts/generate-acpx-profiles.mjs` reference only v7 through v19 by
literal path. No dynamic `copilot-profile-v${...}` path exists in the
repository (the only template-literal fixture path is
`test-fixtures/pi-acp/profile-v${version}-identity.json`).
**Proposed behavior**
Delete only the five unreferenced fixture files. Keep every referenced
fixture. Keep all registered test cases unchanged.
**Reason and benefit**
Smaller test-fixture surface: 341 test/support lines removed with zero
loss of executed coverage. The audit receipt
`doc/plans/2026-10-16-runner-dead-fixture-test-audit.md` records the
per-file reference evidence.
**Breaking changes**
None. No production code, no registered test case, and no test selection
path changes.
**Additional context**
A repository-wide `git grep` for each removed filename stem returns zero
external references across all tracked file types. The
`hashFiles('packages/paperclip-runner/test/fixtures/**')` occurrences in
the workflows are Rust cache-key inputs, not execution selectors;
deleting files only changes that cache hash.
## What Changed
- Deleted
`packages/paperclip-runner/test/fixtures/fake-native-resume-codex-app-server.mjs`.
- Deleted
`packages/paperclip-runner/test/fixtures/copilot-profile-v{3,4,5,6}-identity.json`.
- Added the audit receipt
`doc/plans/2026-10-16-runner-dead-fixture-test-audit.md` with per-file
reference evidence.
- 341 test/support lines removed. Zero production, documentation, or
registered test-case changes.
## Verification
- Repo-wide `git grep -i` for each removed filename stem
(`fake-native-resume`, `native-resume-codex`, `copilot-profile-v3` …
`copilot-profile-v6`) across all tracked file types returns zero
external references.
- Selection audit: `packages/paperclip-runner/vitest.config.ts` includes
only `src/**`, `devtools/**`, `examples/**`, and two named
`scripts/*.test.mjs`; the server project selects `server/src/**` and
`server/scripts/**`; the `test:typescript:prep` `node --test` list names
no fixture. No workflow executes anything under `test/fixtures/` — the
`pr-trusted.yml`/`release-verify.yml`
`hashFiles('packages/paperclip-runner/test/fixtures/**')` occurrences
are Rust cache-key inputs, not execution selectors.
- Rust tests use their own `fake-codex-app-server` binary in
`runner/crates/runner-core`, not these `.mjs` fixtures.
- Searched open and merged PRs for each filename: zero overlap (`gh
search prs`).
- Repo-wide `pnpm -r typecheck`, `pnpm test:run`, and `pnpm build` were
not run this session: the removed files are unreferenced, absent from
every vitest/node-test include pattern, and excluded from all tsconfigs,
so no compiled or executed path changes. No runtime CI savings are
claimed.
- All Paperclip CI gates green on head `ce24ab3` (verified 2026-10-16
~09:00Z): every `ci /` shard (General tests, e2e shards, serialized
server suites, Verify Paperclip Runner rust/vitest/static, Build,
Typecheck + Release Registry, policy, Docker context integrity, Canary
Dry Run, gate), `review`, `Superagent Security Scan`, `Socket Security`
PR alerts + project report, and `Contributor trust` all pass; the only
skipped jobs are the Storybook deploy/visual-regression jobs, which are
not required for this change. Greptile Review: 5/5 confidence, "Low
impact", no actionable issues, last reviewed commit `ce24ab3`.
## Risks
Low risk. No executed coverage or selection path changes. The only side
effect is a one-time change to the `hashFiles` Rust cache key that lists
`test/fixtures/**`, which simply invalidates that cache on the next CI
run.
## Model Used
Zhipu `z-ai/glm-5.3` (GLM), served through OpenRouter, running as the
Paperclip Test Gardener agent. Context window size unavailable to the
agent. Capabilities used: shell tooling, git, GitHub CLI search, and
file inspection; no image or code execution capabilities were used.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` /
`Closes: #` / `Refs: #` OR (b) described the issue in-PR following the
relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
canary/v2026.1018.0-canary.2
|
||
|
Devin FoleyandPaperclip
|
e30550ef39 |
refactor: remove code comments added by recent PRs (#16681)
CONTRIBUTING.md says PRs add no code comments, but #16485, #16586, #16621 and #16672 added some. This removes the 215 comment-only lines git blame on master attributes to those merge commits, across 12 code files, plus one blank line the removal left doubled. Rewritten (not added) comments in scripts/assert-cloud-image-sentry.mjs and comments in .github workflows are left alone. No code changes. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
bebeb389f1 |
perf(server): do not watch bundled plugins for file changes (#16677)
Bundled sandbox-provider plugins are installed from local paths inside the image, so production started a chokidar dev watcher for each of them over files that never change. createPluginDevWatcher now takes unwatchedRoots (compared in resolved and real-path form), and the app passes the bundled plugin catalog root; plugins installed elsewhere are watched as before, and a plugin moving into an unwatched root closes its old watcher and cancels any pending restart. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
d30648dee8 |
Preserve accepted messages for paused recipients (#16679)
Keep accepted deferred requests pending while their recipient is paused and deliver through normal gates after resume. Preserve company, task hold, and cleanup checks. Verification: 247 focused checks, recursive typecheck, the full CI test and build matrix, and Apex 5/5. The local Git fixture initial-branch limitation is documented in the PR. Co-Authored-By: Paperclip <noreply@paperclip.ing>canary/v2026.1018.0-canary.1 |
||
|
Devin FoleyandPaperclip
|
391dd15ff2 |
fix: recover deferred messages and completed unblock waits (#16673)
Recover accepted deferred messages through the latest finished issue run and recognize exact completed agent unblock notifications as passive idle waits. Preserve assignment handoffs, pause and Stop, failed outcomes, and cleanup ownership. Honor local and SSH release contracts and atomically settle cleanup claims. Add PostgreSQL recovery, concurrency, and interruption regressions. Validation: recursive typecheck and build pass; 678 focused tests pass; full CI passes; Apex review is 5/5 with no unresolved findings. Co-Authored-By: Paperclip <noreply@paperclip.ing>canary/v2026.1018.0-canary.0 |
||
|
Devin FoleyandPaperclip
|
27ab902d3d |
perf(docker): start the server from a precompiled runtime bundle (#16672)
The production image started the server through the tsx loader, which resolved about 3,100 modules and transpiled about 1,200 workspace TypeScript files on every boot (7.6-20 s before listen on a live instance). A build:runtime step now bundles server/dist/index.js with esbuild, compiling in the workspace packages, and the image starts node server/dist-runtime/index.mjs. Server dependencies stay bare so runtime resolution (including the cloud image's optional peers) is unchanged; each module keeps its original import.meta location; dynamic imports stay split; the entry presents the original dist/index.js as argv[1]. The avatar worker loads through tsx's API when the main thread has no loader, tini runs as a subreaper (-s), and PR CI boots the bundle with scripts/server-boot-smoke.sh. Co-Authored-By: Paperclip <noreply@paperclip.ing>canary/v2026.1017.0-canary.8 |
||
|
|
938a441944 |
fix(acpx): end Claude turns whose pending tool call stops receiving updates (#16671)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Hosted `claude_local` runs use the ACPX engine. Paperclip sends
runtime MCP servers to Claude, and Claude can start subagents (the Task
tool) that call those MCP tools.
> - #16360 set `MCP_TOOL_TIMEOUT` for Claude. That bound works: in a
local test with the same Claude binary, a direct MCP call and a subagent
MCP call to a server that never answers both ended at the bound.
> - A hosted run still hit the 4h sandbox timeout on a release that has
#16360. Its run log shows a subagent that started four MCP calls. ACP
reported the four calls as `pending`, and they never got another update.
About 40 other MCP calls in the same minutes finished in 3–25 seconds.
So the four calls most likely never reached the MCP server, and
`MCP_TOOL_TIMEOUT` did not apply to them.
> - The subagent's Task call got an `in_progress` heartbeat every 30
seconds for the full 4 hours. The turn looked active, so no existing
bound stopped it.
> - This pull request adds a stalled-tool watchdog to the ACPX engine.
It ends the turn when a pending tool call gets no update for 30 minutes
and no other tool call finishes in that time.
> - The benefit is that a stalled run fails in about 30 minutes with a
clear error code, not after 4 hours with a general timeout.
## Linked Issues or Issue Description
- Refs #16360 (MCP client deadline; this PR covers the stall that the
deadline cannot reach)
- Refs #15962 (stop-time diagnostics that showed the pending tools and
their age)
**What happened?**
A hosted `claude_local` run gave lookups to two subagents. One subagent
started four MCP tool calls. ACP reported each call as `pending`. No
call got a result, an update, or an error. The other subagent finished
in 3 minutes. The stalled subagent's Task call got a heartbeat every 30
seconds, so the last-event age at stop was only 28 seconds. The oldest
pending tool was 3h58m old. The sandbox execution timeout ended the run
after 4 hours.
**Expected behavior**
A tool call that never gets a result ends the turn after a bounded time.
The run fails with an error that names the stall.
**Steps to reproduce**
1. Start an ACP `claude_local` run that uses a subagent to call runtime
MCP tools.
2. Make one of the subagent's tool calls stay `pending` with no further
ACP update (in production this happened without a known trigger).
3. See that the run continues until the adapter execution timeout,
because the Task heartbeats keep the turn active.
**Paperclip version or commit**
`master` at
canary/v2026.1017.0-canary.7
|
||
|
Devin FoleyandPaperclip
|
4415665d67 |
fix(server): allow proved idle waits and verify historical cleanup (#16633)
## Thinking Path > - Paperclip manages agents, task continuations, integrations, and their execution resources. > - Automatic idle sleep needs durable evidence that accepted work and provider cleanup are settled. > - Response-driven questions, passive board escalations, and completed connection receipts currently look like unfinished work. > - Legacy ephemeral leases and terminal image sessions need actual deletion proof before their history can become idle. > - This PR adds narrow evidence-based exceptions and read-only provider verification while preserving queues, outcomes, and ownership fences. > - Idle instances can sleep without losing responses, automatic reconciliation, or cleanup obligations. ## Linked Issues or Issue Description **What happened?** Instances with no executable work can remain awake because pending human responses, passive recovery records, outbound integration configuration, or completed delivery history are counted as background work. Older failed ephemeral leases and finished setup sessions also lack the evidence needed to distinguish retained resources from deleted allocations. **Expected behavior** Permit idle sleep only for proved response-driven waits and settled history. Continue blocking real queued work, inbound integrations without a wake contract, unfinished accounting/retries, retained resources, and uncertain provider cleanup. **Steps to reproduce** Finish an issue's producing run, leave a pending question or confirmation, and request an owned scheduled idle report. Separately retain a processed connection receipt or a failed ephemeral lease whose provider allocation has already been deleted. The old checks keep those instances awake even after executable work has finished. Related public groundwork: #15522, #15599, #15761, #15897, #15907, and #16325. Searched existing public idle-sleep PRs; this addresses additional blockers in the current predicates. ## What Changed - Recognize supported response-driven waits and passive board recoveries without changing task state, response audiences, or continuation policies. Bind passive issue generation into routine-only alarms too. Keep automatically polled merge-confirmation cards awake. - Treat processed events, delivered intents, disabled drafts, and outbound MCP configuration as history/configuration where proved safe. Preserve pending deliveries, downstream wake requests, and inbound GitHub bindings. - Permit only pristine Slack setup drafts with a matching disabled channel connection, no provisioning registration, and no prior provider identity/activity. A saved slash-command name remains inert configuration. - Repair missed routine execution-issue status hooks in bounded awake-only maintenance. Lock and re-read exact owners, apply existing completion/transient-failure rules, and preserve normal continuation after an issue resumes. - Allow human-only connection setup to wait only when no reusable active connection of its purpose exists; available access reblocks sleep for reconciliation. The delivery worker shares this proof, releases its local idle hold, and retains maintenance only while already awake. Passive-only ticks pause before admission during an owned drain, allowing slow backups to finish; new producer notifications still reconcile and invalidate the backup. - Add optional `environmentInspectLease` SDK discovery/dispatch and read-only Daytona/fake-provider implementations. Old workers remain conservative. Daytona status inspection uses a fresh provider GET without reconnecting or poisoning unused-worker eligibility. - Bind historical credential reads to an unrotated initial secret version and its immutable resolved-value hash; reject ambiguous histories and ambient Daytona credential/API-endpoint fallbacks before querying provider state. - Verify bounded rotating batches of historical resources on already-awake Cloud instances, outside readiness. Store host-owned deletion evidence bound to exact row identity/version and reject concurrent ownership changes. Preserve failure outcomes, cleanup/export intents, and run continuation authority. - Preserve cancellation provenance, defer contended routine rows individually, and cap actual inspection RPCs across caller timeouts and worker exits. Add database, SDK, provider, configuration-scope, and race regression coverage and update idle protocol documentation. ## Verification - `pnpm -r typecheck` and `pnpm build` passed on `24fd1c0f5ec`; server typecheck/build passed again after the final coordinator-only fix in `60f23e6e4b6`. - Final server regressions: 258 sleep-safety/connection cases and all 17 coordinator cases passed. Earlier focused checks also covered 331 combined idle/cleanup cases, 34 custom-image cases, 39 credential/cleanup cases, 39 connection-history cases, and 80 existing routine cases. Counts overlap and are not additive. - SDK: 153 passed. Daytona: 406 passed with 14 existing opt-in skips. The provider implementation also typechecks against the locked SDK used by the Cloud wrapper. - Dependency boundaries were exercised with real child workers holding provider inspection replies beyond caller timeouts, a real PostgreSQL row lock delaying one routine owner, and real connection creation → queue notification → worker hold release → resumption after access is enabled without another notification. A controlled six-second backup goes through the real coordinator, task drain, work signals, and idle-safety generation checks; maintenance permits completion, while a new committed transaction invalidates it. - Full local `pnpm test:run` completed its general-server phase with 21,122 passing, 29 failing, and 109 skipped tests; later phases did not run. That long run spanned source edits and is not a final-revision pass. The affected Slack draft assertion passes on current source. Restoring `/usr/sbin` for `lsof` produced 233 passing tests and two Git-fixture failures across the other five files (3 existing skips). Both remaining assertions pass with command-scoped `init.defaultBranch=master`; this host defaults to main and leaves both branches in their fixture. No source/global Git setting changed for that comparison. Original failure logs are retained. - A parallel typecheck/build attempt raced generated Runner outputs and failed an import. Running the build sequentially passed with no source change. - Local database checks use the maintained test-constructor seam with native PostgreSQL 16.14 because bundled PostgreSQL 18 is unreliable on this host. No database assertions were skipped in the focused runs. An interrupted combined run was superseded by the separate passing coordinator and safety/connection runs; no owned test database processes remain running. - Apex reviewed final head `60f23e6e4b615a08591e19d86443a6c35c091ce1`: **5/5**, zero unresolved threads. All current-head CI gates are green, with expected policy skips. The branch is conflict-free and GitHub reports a clean merge state. - Current-head CI initially failed `plan-task-fixture.spec.ts:66` during fixture deletion: the final company DELETE hit `activity_log_company_id_companies_id_fk` after the feature assertions completed. The deletion path is unchanged here; the final diff only changes drain-gated passive maintenance. The preceding revision passed this shard. Failure evidence is retained in [the original CI job](https://github.com/paperclipai/paperclip/actions/runs/39006936642/job/117321244396); the targeted retry passed on the unchanged commit ([retry job](https://github.com/paperclipai/paperclip/actions/runs/39006936642/job/117322535013)). This shows recovery, not that the underlying company-deletion race is resolved. - These checks do not claim staging/production deployment or live provider cleanup proof. Provider tests simulate responses. Full Cloud staging verification requires a compatible Core/private-plugin composition after the Core revision is on master. ## Risks Sleep eligibility changes require the existing owned drain, final generation validation, local-work checks, and plugin drain. Unsupported states fail closed. Queued agent input and routine execution remain blocked. Outbound MCP exemptions exclude inbound GitHub grants; other unsupported inbound integrations remain awake. The inspection RPC is optional and requires a matching live worker method. Its actual outstanding calls are capped at two across workers even after the caller times out. Historical lease inspection requires its recorded plugin/config; setup inspection requires an unchanged recorded config fingerprint. Not-found under replacement account credentials is not historical deletion proof: ambiguous/rotated secret histories defer verification, and initial-version reads must match their recorded value hashes. Missing provenance (including legacy image sessions without a recorded config fingerprint), live/stopped resources, provider authorization failures, and concurrent changes defer verification. Routine repair replays existing completion/transient-failure behavior and timestamps at reconciliation time; it does not execute or discard deferred comments. No migration or destructive provider operation is added. The new marker does not grant run-resume authority. Full Cloud staging validation requires a compatible Core/private-plugin composition; no production deployment is included. ## Model Used OpenAI GPT-6 Codex, with repository/tool access and code execution. Exact model identifier and context-window size are not exposed by this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run relevant tests locally; passing focused checks and full-run limitations are documented above - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>canary/v2026.1017.0-canary.6 |
||
|
Nicky Leach
|
2723168083 | test(runner): control the durable resume watchdog (#16623) canary/v2026.1017.0-canary.5 | ||
|
Devin FoleyandPaperclip
|
5555817c2b |
fix(server): let Git expiration verify public commits without a run identity (#16621)
Git expiration created its auth provider with only the issue ID; with a managed GitHub connection that provider throws outside a run ("No managed GitHub identity is available for this run"), so every upstream commit proof failed and was logged as inspection_failed. Expiration now falls back to anonymous access when the provider throws: public repositories verify, private ones stay kept. Upstream verification failures are reported as upstream_commit_unverified, and a managed checkout whose upstream cannot be reached or authenticated as managed_checkout_upstream_unavailable; local proof-store failures keep their own error.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
canary/v2026.1017.0-canary.4
nightly/v2026.1017.0-nightly.0
|
||
|
DottaandPaperclip
|
669d329f25 |
perf(slack): avoid unused catalog revision scans (#16630)
Slack catalog discovery now checks live task, user, endpoint and credentials without calculating an unused compatibility revision. Full authority callers retain the unchanged revision and search-token path. Validation: 44 affected cases, full repository typecheck/build, current-head full CI/E2E and security checks, Greptile Base 5/5 with no unresolved findings. Identical production catalog output with revision statements reduced from two to zero in real-PostgreSQL comparisons. Hosted latency acceptance remains pending. Co-Authored-By: Paperclip <noreply@paperclip.ing>canary/v2026.1017.0-canary.3 |
||
|
Devin FoleyandClaude Opus 5.5
|
32ee2e9a6a |
fix(heartbeat): let a live legacy controller finish before the finalization deadline fails its run (#16622)
The execution-control reconciler now checks hasLiveLegacyController inside its locked transaction before failing a run. While a live legacy controller lease owns the run, the reconciler moves executionControlDeadlineAt to the lease expiry instead of failing, reporting, or surfacing it. Once the lease lapses, the next sweep fails and reports the run once, as before. Native runs are unchanged. Adds an embedded-Postgres regression test that fails on master and passes with the fix. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1017.0-canary.2 |
||
|
DottaandPaperclip
|
0528cc7ee0 |
perf: overlap native GitHub and assigned-tool setup (#16620)
Prepare managed GitHub access and fresh assigned-MCP discovery concurrently for remote native Codex. Wait for both before provider construction and drain both on failure through the existing run cleanup. Co-Authored-By: Paperclip <noreply@paperclip.ing>canary/v2026.1017.0-canary.1 |
||
|
Nicky LeachandPaperclip
|
5f5602bba7 |
feat(plugins): synchronize standing connection assignments (#16603)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Lifecycle plugins prepare agent runtimes outside the control plane. > - They need committed connection assignment changes without polling. > - A reader and its revision hooks must cover the same fields. > - This pull request adds a bounded standing-assignment snapshot and its matching hooks. > - Plugins can react to assignment changes while execution keeps its existing access checks. ## Linked Issues or Issue Description **What happened?** The agent runtime reader does not expose standing connection assignments. Changes to connection installs, active profiles, and connector feature switches do not request synchronization. Adding company-wide invalidation also exposes lock inversions in assignment writers and AI adoption. **Expected behavior** The owning lifecycle plugin can read a bounded assignment snapshot. Every committed change to its stored fields advances the runtime revision. Rollback removes both the change and its event. Assignment writers finish without a lock-order cycle. **Steps to reproduce** 1. Install a connection or bind an active profile to an agent. 2. Read its runtime inputs through the owning lifecycle plugin. 3. Rename or disable the connection, revoke the assignment, or switch connector features. 4. Expect a committed runtime event and an updated snapshot without periodic polling. **Additional context** Extracted from #16503. Catalog publication #16602 and the separate test maintenance PRs #16601 and #16610 are merged. This PR now targets master and is the active review. Tool-definition hooks #16604 and access/credential hooks #16605 remain drafts without review requests. ## What Changed - Add SDK types and an owning-plugin runtime snapshot for installed connections, active profile IDs, and effective connector feature switches. - Reject more than 512 connections or 128 active profiles instead of returning partial data. - Add migration 0354 for the exact stored fields used by the snapshot, with transaction-coalesced invalidation. Agent-targeted bindings and installs notify only their old/new target agents; project/routine/issue/gateway bindings are outside the snapshot and produce no notification. - Use ordered agent locks that are compatible with foreign-key references and exclude runtime readers. Prelock complete old/new target sets before replacing assignments or restoring managed profiles. - Record one company baseline at delivery startup for process-managed feature overlays. - Lock assignment inputs before install saves, task completion, and email assignment. Setup finalization reads the locked connection state and prelocks company agents before activating, re-enabling, or reviving a profile. - Make profile binding and reassignment writes transactional and lock parent profiles first. Keep target validation and profile/detail reads on the reserved transaction connection, including with a one-connection pool. - Lock company agents in order before validated AI adoption. Keep adoption and shared configuration writes at `FOR NO KEY UPDATE`, avoiding an upgrade that can deadlock foreign-key inserts during gateway creation. Configuration fields cannot change agent identity; lifecycle transitions and deletion retain their existing locks. - Document supported coverage and the inputs reserved for later PRs. ## Verification - Thirteen affected suites passed: 757 tests covering runtime hooks/assignments/skills, concurrent skill publication, tool access/catalog, task connection intents, lifecycle delivery, email, AI accounts, migration histories, snapshot drift, and the agent lifecycle module. This includes the merged catalog PR's OAuth and overlapping-refresh regressions. - The assignment suite passes all 24 cases, including bounded snapshots, company/agent isolation, rollback, health-field no-ops, process-start feature overlays, and real lock contention. - Four one-connection PostgreSQL regressions reproduce the earlier helper-read hangs before the correction; bind, unbind, delete, and reassign complete with it. The earlier adoption/gateway probe exposed both lock upgrades: changing only validation still deadlocks in shared configuration. Current scoped coverage verifies adoption with agent-targeted binding writes; gateway-only bindings no longer invalidate this snapshot. - The original ordered-adoption regression reproduces PostgreSQL `40P01` without ordered company-agent locking. Scoped-recipient tests reproduce unrelated-peer journaling before the trigger correction and verify repeated bind/unbind operations leave peers untouched, ignored binding types produce no events, both reassignment recipients are reached, and coalescing remains intact. - Six replacement contention cases cover installs, existing setup, activation, re-enabling, profile revival, and profile reassignment. Negative controls reproduce `40P01` without the corresponding target/company prelocks; all cases pass with the guards. Activation decisions use the connection row reread under lock. - An initial contention fixture failed because its settings connection lacked an MCP URL. Supplying the fixture URL and controlled lookup corrected that test setup error. - Server typecheck, database migration numbering/safety, five-snapshot retention, feature-map structural validation, and `git diff --check` passed. Migration metadata preserves historical journal entries and adds no schema change. The complete diff was reviewed for secrets and personal data before push. - Earlier hosted PR CI runs 38983440861 and 38985530069 passed; the superseded promotion run 38983440470 was cancelled. Superagent acknowledged the recipient fix and accepted the documented company-wide cost at `359b5b2f30a8a0248177367e458663d5379810d1`. Its two standard Greptile summaries were conditional on a later setup correction and were not counted as independent acceptance; the setup correction is now included here. - At `7da246aab46274db1a120a0546abbb404c5427c7`, hosted PR run 38986958988 passed: all 55 checks succeeded, with two optional Storybook checks skipped. Superagent passed, all review threads are resolved, and standard Greptile independently returned 5/5 with no blocking findings (review completed 2026-10-17 00:40 UTC). Apex independently returned 5/5 with no actionable findings (completed 2026-10-17 00:50 UTC). CI was rechecked after Apex and remains green; the branch is mergeable against master. Full local build and repository-wide tests were not run. ## Risks Migration 0354 adds database functions and triggers without changing table schema. Apply it through the normal migration flow. Agent-targeted bindings and installs only notify their affected agents. Explicit company-wide assignments, connection metadata, profile activation/removal, and connector feature switches retain synchronous, transaction-atomic company fanout. Each such transaction takes ordered row locks and inserts up to one connector runtime event per eligible company agent: O(number of eligible agents) work and journal storage, with no company-size cap or cross-transaction mutation quota. This remains an intentional scope tradeoff for this PR; deferred fanout is not implemented. Many of these conservative changes can request preparation even if an individual snapshot stays unchanged. Snapshot limits fail closed. This PR provides desired assignments, not credentials or task authorization. Later PRs must add mutation coverage before consumers can rely on immediate synchronization of tool definitions or credentials. ## Model Used OpenAI Codex, GPT-6, with reasoning, repository inspection, code execution, and testing. The exact deployment model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>canary/v2026.1017.0-canary.0 |
||
|
Nicky LeachandPaperclip
|
190080642e |
perf(tools): publish catalog refreshes in bounded chunks (#16602)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - MCP discovery publishes tools that setup and access profiles use. > - Large catalogs currently write one row at a time. > - Repeated page entries can produce duplicate results, and profile writes can partly commit. > - This pull request publishes tools in bounded chunks and makes profile synchronization atomic. > - The result supports large catalogs before connector runtime hooks are enabled. ## Linked Issues or Issue Description **What happened?** Catalog refresh performs one database write per tool. Repeated tool names across discovery pages appear more than once in the result. Default-profile synchronization uses separate statements, so failure can leave a profile without its binding. OAuth callbacks also acquire connection/application locks in the reverse order from publication and setup, allowing a concurrent refresh to deadlock. **Expected behavior** Publish at most 500 tools per catalog statement. Keep one result per tool name, using its last descriptor. Preserve existing IDs, first-seen timestamps, disabled actions, and unchanged review decisions. Commit or roll back default-profile changes together. OAuth persistence must acquire application then connection locks before secret or connection writes. **Steps to reproduce** 1. Refresh a synthetic MCP catalog with 2,500 tools. 2. Return the same tool on two discovery pages with different descriptions. 3. Fail the profile-binding insert during default-profile creation. 4. Overlap personal, organization, or managed OAuth persistence with setup finalization or discovery of a new tool. **Additional context** This is the first feature extracted from #16503. The separate test-only maintenance PRs #16601 and #16610 are merged. This PR now targets master and is the active review. Standing assignments #16603, tool definitions #16604, and access/credential hooks #16605 remain stacked drafts without review requests. ## What Changed - Deduplicate discovery results by tool name and keep the last descriptor. - Upsert catalog rows in transactions of at most 500 tools. - Preserve row identity and operator-disabled status on conflict. Determine default-access candidates from the rows read under each publication lock, so an overlapping refresh cannot restore an action turned Off after another refresh finishes. - Lock the application before the connection during publication, setup finalization, removal, and all three OAuth persistence paths. The OAuth ordering fix is included here so this PR is independently safe; it does not depend on #16605. - Synchronize the managed default profile, its entries, bindings, and managed policies in one transaction. - Add PostgreSQL publication tests and document partial-failure recovery. - Target the existing callback failure injection at profile creation instead of a transaction ordinal. ## Verification - `pnpm exec vitest run server/src/__tests__/tool-catalog-publication.test.ts server/src/__tests__/tool-access-service.test.ts`: 436 tests passed with the OAuth and overlapping-refresh corrections on top of the merged #16610 fix. - The new tests exercise real PostgreSQL with 250 and 2,500 tools, repeated discovery pages, preserved choices, lock contention, and failures between chunks and during profile creation. - Four selected regression cases fail with the production source restored to the parent commit. The implementation was then restored and rechecked. - Six additional real-PostgreSQL regressions cover personal, organization, and managed OAuth persistence against both setup finalization and new-tool discovery. All six reproduce PostgreSQL deadlock `40P01` with the OAuth ordering fix removed; all six pass with it restored. Discovery uses synthetic credential headers to isolate publication locking from credential rotation. - Apex initially returned 3/5 on `0d6013a9649388041975406fe4b0247cd0e695aa`, identifying an overlapping-refresh permission race. Two new PostgreSQL regressions cover both absent and quarantined tools; both prove an agent goes from deny back to allow without the correction, and remains denied with it. The test pauses one refresh after its initial catalog snapshot, completes another refresh, turns the action Off through the service, and resumes the stale refresh. - `pnpm --filter @paperclipai/server typecheck`: passed. - `node scripts/check-maps.mjs`: passed (structural validation only). - `git diff --check`: passed. - Reviewed the complete diff for secrets and personal data before push. - Earlier hosted run 38975428064 failed the maximum-size newline skill-audit watchdog; other interrupted jobs reported runner shutdown/cancellation. The original watchdog failure did not identify whether startup or auditing timed out, so its exact cause remains unknown. The separate merged #16610 isolates the audit deadline from module startup and preserves the audit hang guard. This branch is rebased onto that fix. - Full local build and repository-wide tests were not run. At the previous head `0d6013a9649388041975406fe4b0247cd0e695aa`, all 55 hosted checks succeeded (two optional Storybook checks skipped), and standard Greptile returned 5/5 before Apex identified the permission race above. At corrected head `dc120d2bc879c54c14347db30e74c753bcff61e6`, all 55 hosted checks succeeded (two optional Storybook checks skipped), and a fresh standard review returned 5/5 with no open findings. The second Apex review also returned 5/5 on that exact head (successful Apex check completed 2026-10-16T23:36:44Z). Both earlier findings are addressed, there are no open review threads, and the branch is mergeable against master. ## Risks Each successful catalog chunk remains committed if a later operation fails. A retry completes publication. Provider discovery occurs outside database locks. Default-profile changes commit separately from the catalog chunks and roll back together on failure. Tests use a controlled MCP response; they do not qualify a live provider. There is no schema migration or new plugin API in this PR. ## Model Used OpenAI Codex, GPT-6, with reasoning, repository inspection, code execution, and testing. The exact deployment model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
dc499ae02b |
perf(server): reject shared Git cleanup paths before history scans (#16614)
Check the existing global execution/project shared-path veto before scanning retained run contexts. Preserve the full unshared-path query, lifecycle locks and all deletion-phase rechecks. Verify with 35 focused real PostgreSQL/Git cases, full repository typecheck/build, exact-head full CI/E2E and Greptile Base 5/5. A read-only same-snapshot query pair retained the false verdict while reducing round trip from1089ms to9.4ms; this is not a full-agent latency result. The first security scan was inconclusive after an infrastructure checkout failure; one actual rescan passed. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
Nicky Leach
|
564d4e1bce | test(skills): separate audit deadline from process startup (#16610) canary/v2026.1016.0-canary.20 | ||
|
DottaandPaperclip
|
9a3290fe10 |
perf(runner): overlap default remote artifact preparation (#16608)
Prepare the native runner and default image Codex harness together on providers that declare concurrent sync support. Preserve all executable checks, wait for both operations on failure, and retain serial explicit harness installs. Verified with 881 affected cases, full typecheck/build, exact-head full CI and Greptile Base 5/5. The real Daytona component median fell from 839ms to 432ms; full deployed reply latency remains unproven. Co-Authored-By: Paperclip <noreply@paperclip.ing>canary/v2026.1016.0-canary.19 |
||
|
Nicky Leach
|
c0ac2772b0 | test: stabilize migration and recovery fixtures (#16601) | ||
|
Devin FoleyandPaperclip
|
73659c6b27 |
fix(server): let Git expiration discard Paperclip-managed checkouts (#16586)
Reconstructable Git expiration rejected nearly every terminal worktree with unique_ignored_files, because Paperclip writes two ignored paths into every worktree itself: .claude/settings.local.json and the managed project checkouts under .paperclip-repositories/. Expiration now accepts the settings file, and accepts a managed checkout only through a prover the expiration worker supplies. The prover maps the checkout back to its project workspace row and proves it reproducible against that row's repository URL: worktree and stock-storage checks, no linked worktrees, no unreachable objects, and every ref, reflog and HEAD commit reachable from upstream refs fetched fresh into a proof store. Every inspection Git command now runs with core.fsmonitor disabled. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
3db33c131e |
perf(slack): overlap endpoint preparation and narrow credential reads (#16593)
Prepare Slack endpoints in pairs with a per-call concurrency limit of two, and resolve only the current bot token for live tool authority. Preserve candidate-order errors, active-sibling settlement, credential rotation and authorization revocation. Verified with real PostgreSQL regression controls, full repository CI/E2E, security checks and current-head Greptile Plus5/5 with no unresolved findings. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandPaperclip
|
d1463fb79e |
Preserve saved delivery deadlines across clock ticks (#16589)
## Thinking Path > - Paperclip manages AI agent work and message delivery. > - Delivery workers restore saved retry and maintenance deadlines. > - The coordinator converts each future deadline to a delay and back. > - A clock tick between those reads can move the deadline. > - This pull request keeps the saved future timestamp. > - A deterministic test covers the clock tick. ## Linked Issues or Issue Description Refs #16503. This separate fix addresses an unrelated CI failure found during its review. **What happened?** The unchanged delayed GitHub publication restart test intermittently reports a wake deadline one millisecond after the saved timestamp. The coordinator subtracts one clock reading from the deadline, then adds the delay to a later reading. **Expected behavior** Keep a future saved deadline exactly. Preserve the current retry behavior for expired deadlines during idle reconciliation. **Steps to reproduce** Register a task with a future `nextRunAt` timestamp. Advance `Date.now()` by one millisecond on each read. The old coordinator reports `nextWakeAt()` one millisecond after the saved timestamp. The new unit test forces this sequence. ## What Changed - Let the internal scheduler accept the existing absolute future deadline. - Keep relative retry and immediate wake scheduling unchanged. - Add a deterministic advancing-clock regression. Keep the chat integration assertion unchanged. ## Verification - Based on `origin/master` at `c91f7a6e5f`. - Coordinator suite and the unchanged delayed GitHub publication restart case: 16 passed, 1346 unrelated cases skipped. - The new regression fails against the original coordinator with the exact one-millisecond offset. - Server typecheck passed. - On head `d2db3e15d6dfafd390bab59529474abebb19d09c`, all 55 hosted checks passed, with 2 expected skips. This includes all server/chat shards, Runner checks, build, canary dry run, security checks, and end-to-end tests. - Standard Greptile and Apex are both 5/5 on that exact head, with no review threads. Apex completed after green CI; final CI verification remains fully green and the branch is mergeable. - `git diff --check` and the pre-push secret/PII scan passed. ## Risks Low risk. The change keeps the saved future timestamp. Existing tests cover retry, idle drain, preemption, and shutdown behavior. No schema or API change is required. ## Model Used OpenAI Codex, GPT-6, with reasoning, repository inspection, code execution, and testing. The exact deployment model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub references) - [x] My branch name describes the change and contains no internal Paperclip ticket id - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes (no public contract changed) - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>canary/v2026.1016.0-canary.16 |
||
|
DottaandPaperclip
|
93ab7bad11 |
Add immutable Muse clients and OAuth grant contracts (#16447)
Add immutable Muse client assets and the grant-linked private contract. Keep the grant migration safe for replay and partial schema recovery, preserving exact preview receipts and existing data. Keep public wake signals separate from private provider authority. Refs #16447. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
2cc7992369 |
Share the external Runner lifecycle and add strict Muse contracts (#16444)
Add the external Runner foundation with distinct Dot/Grok V6, canonical home-mode V7 and Muse V8 contracts. Keep generated schemas and profiles synchronized. Bound default question-answer command IDs in a disjoint namespace while preserving existing IDs and durable replay. Preserve canonical migration history and add normally generated, replay-safe Muse schema milestones with literal preview-history fixtures. Refs #16444. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
4de6335c3b |
perf(server): trim saved run JSON from agent authentication (#16577)
## Thinking Path > - Paperclip coordinates AI agents that work for companies. > - Agent JWT authentication checks each request against its saved run. > - Saved runs can contain large prompts and result payloads. > - The first run lookup needs only identity fields, cancellation state, and the conversation flag. > - Reading all saved JSON transfers data that this lookup does not use. > - This PR selects the required JSON fields in PostgreSQL and preserves the existing authorization decisions. ## Linked Issues or Issue Description **What existing behavior does this improve?** The initial saved-run lookup in agent JWT authentication. **Subsystem affected** Server authentication and database queries. **Current behavior** The lookup transfers the full context and result JSON on each request. It uses only the cancellation state and conversation flag from those values. **Proposed behavior** Select the required JSON fields in PostgreSQL. Keep the run, company, and agent predicates. Read current state on every request. **Reason and benefit** A synthetic lookup with a 96 KiB prompt and 4 KiB result returns 102,520 JSON bytes before this change and 134 bytes after it. This reduces data transferred by this query. **Breaking changes** None intended. Identity fields, write revocation, read access, and cancellation response codes retain their current behavior. Related: #9280 addresses authentication round-trip counts. This PR only reduces the data returned by the existing initial run lookup. ## What Changed - Extract the initial run lookup and select only the JSON fields that authentication consumes. - Add real PostgreSQL tests for legacy JSON shapes, fresh cancellation state, payload size, and run/company/agent scope. - Add a reproducible benchmark that loads the original query from an explicit baseline commit. - Document the lookup and benchmark. ## Verification - Baseline: `06689c1b3410000bcd65ff40cb469e600209c4db`. - `pnpm -r typecheck`: passed. - `pnpm build`: passed. - Final server typecheck: passed. - Final-head GitHub checks: all green at `2be1c38e6bf51616b69993cf84cee1eef9657517`. Apex: 5/5, zero unresolved threads, conflict-free. - `pnpm check:module-boundaries`: passed. The standalone synthetic benchmark now uses the existing root benchmark directory and resolves Drizzle through the server workspace. - Focused HTTP middleware and database tests: 29 passed on real Homebrew PostgreSQL 16.14. The SQL test compares 270 combinations of status, cancellation JSON, and conversation JSON. It also checks fresh state and company/agent/run scope. - Broad native verification: 873 passed across the HTTP middleware, SQL projection, run diagnostics, and native executor suites. - The default local full command completed its general server lane with 12,090 passed and 8,885 skipped tests. It could not finish the root workflow: the host could not start the bundled PostgreSQL; a diagnostics assertion matched a directory name; and Git fixtures retained the host default branch. These failures remain in the verification record. All 68 failed files now pass on replay with disposable Homebrew PostgreSQL, an isolated checkout, and command-scoped Git defaults. The 68-file replay passed 2,588 tests and left two email skill-resolution failures. Moving the checkout outside an unrelated ancestor skills directory resolved those last failures: all 44 tests in that file passed. Product code and test assertions were unchanged for these host adjustments. Benchmark command: ```sh pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-agent-authentication.ts --baseline-ref=06689c1b3410000bcd65ff40cb469e600209c4db ``` The final benchmark ran at treatment commit `2be1c38e6bf51616b69993cf84cee1eef9657517` with a clean worktree. The local run used PostgreSQL 16.14 and the maintained test constructor seam to start disposable Homebrew clusters. The bundled PostgreSQL binary could not start on this host. Both versions use the same schema and predicates. Each case has two warmup rounds and ten measured rounds with alternating order. Random content prevents unrealistic compression. Complete authentication decisions match. | Context / result | Before JSON bytes | After JSON bytes | Before median | After median | | --- | ---: | ---: | ---: | ---: | | 1 KiB / 4 KiB | 5,240 | 134 | 0.157 ms | 0.157 ms | | 96 KiB / 4 KiB | 102,520 | 134 | 0.331 ms | 0.140 ms | These are local query round trips, including result transfer and decoding. The small control has no material latency gain. The large case improves by about 2.4x. Both plans read the same single row. The primary result is reduced response size. This does not measure HTTP latency, server CPU savings, or fleet-wide cost savings. Full samples and plans are retained with the verification evidence. ## Risks - JSON types must retain their meaning. The real database tests compare the consumed authorization contract with the original projection, including SQL null, JSON null, scalars, arrays, objects, string booleans, and cancellation states. - The change covers the first JWT run lookup. Later run identity capture keeps its existing full-run reads and transaction locks. Those reads remain a separate cost. - The change adds no schema migration or authorization cache. ## Model Used OpenAI GPT-6 through Codex, with reasoning, code editing, and shell tools. This session did not expose a more specific serving model ID or its context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
c91f7a6e5f |
perf(runner): overlap verified sandbox launch assets (#16585)
Native sandbox launch preparation stages fresh harness files and immutable runtime context in sequence. Overlap them only under the verified concurrent-sync capability, settle every started transfer before launch or cleanup, and preserve serial SSH behavior and error priority. A matched real-Daytona comparison reduces staging from 1,552 ms to 1,017 ms. The larger active-run difference includes model variation and is not attributed to this change. Add an explicit single-attempt diagnostic and synchronize stale Pi evidence metadata with its shipped profile. Validation: 764 native runner/tracing cases, 2,764 support cases, repository typecheck/build, fresh exact-head full CI/E2E/security and Apex 5/5 pass. Observability remains a silent no-op when no endpoint is configured. Co-Authored-By: Paperclip <noreply@paperclip.ing>canary/v2026.1016.0-canary.15 |
||
|
DottaandPaperclip
|
f5534717b0 |
fix(runtime): preserve unexported native work during lease cleanup (#16583)
Preserve the exact native workspace source until its export is finalized. Apply the same guard to cleanup sweeps and direct environment teardown. Keep stop-only recovery and healthy warm retention available. Verified with 317 targeted tests, server typecheck, module boundary check, all current-head CI gates, and Greptile 5/5. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandPaperclip
|
6f8ad3d77b |
Add verified session-local stdio MCP transport to Runner (#16526)
## Thinking Path > - Paperclip manages AI agent runs and their tool authority. > - Runner supplies provider sessions with assigned MCP connections. > - The current native connection binding supports an HTTP gateway only. > - Execution hosts also have private local tools with stdio MCP interfaces. > - Those tools need session-local registration and safe concurrency. > - This pull request adds verified local stdio assignments beside the existing gateway. > - Providers receive the existing tool schemas and image content. ## Linked Issues or Issue Description **Problem or motivation** Runner can supply a native provider session with an HTTP MCP gateway, but cannot also assign a verified local stdio MCP server without persistent provider configuration. Execution hosts need a session-local transport for local MCP tools, coordination for calls sharing a resource, and native schema/image delivery. The pinned native Codex client drops MCP images when `structuredContent` is also present. **Proposed solution** Add general stdio MCP transport support using host-assigned executable digests, arguments, environment and a private Unix socket identifying the shared resource. Tool implementations supply their own schemas and responses; Runner forwards their native MCP schemas and content. Register the transport for the current provider session. Serialize calls across processes, keep unknown outcomes fenced, and authorize assigned native Codex tools without a new prompt while routing explicit tool confirmations through existing PRP permission requests. Retain the HTTP gateway and applicable permission policy. **Alternatives considered** Reimplementing tools or translating their actions into provider-specific APIs would tie the transport to individual tool implementations. Global provider config would leak session assignments. Shell commands alone do not deliver native MCP images or preserve MCP approval requests. **Roadmap alignment** This completes the existing Runner transport boundary. Related: #10690 (operator config for the legacy ACPX engine), #11605 (gateway delivery of installed stdio connections), #16486 (preconfigured provider homes). This change assigns local tools on the execution host and leaves those administration paths intact. ## What Changed - Add reserved, verified stdio bindings for native Codex and through the shared ACPX runtime. ACPX clients must support stdio MCP assignments. Read-only sessions omit the binding. No home/project config is rewritten. - Run the backend with only its assigned environment through the verified executable launch path. - Serialize all calls sharing a private Unix socket across sessions. Metadata remains responsive. Queued cancellation prevents execution; active calls remain fenced until acknowledgement. Unknown outcomes require a resource restart. - Authorize host-assigned native Codex tools without a new per-call prompt, matching existing execution-host shell utilities. Preserve explicit tool deny/prompt rules. Route requested MCP tool confirmations through one-time PRP approvals, bound to the active thread and turn. Keep shell escalation disabled and existing tool deny rules effective. ACPX adapters retain their applicable ACPX permission policy. - Preserve schemas and image blocks. For native Codex 0.160/0.162, retain structured metadata as MCP text on image-bearing results to avoid the client's image-loss bug. ACPX adapters receive the original response. - Document transport authority, cancellation and recovery limits. ## Verification - `pnpm -r typecheck` and `pnpm build` passed with pnpm 9.15.4 and Rust 1.97.1 on macOS arm64. - Before Apex, full `paperclip-runner-core` Rust tests passed. After the fixes, all 379 library tests and eight subprocess tests using a generic stdio MCP fixture pass. Those subprocess tests cover schema/image passthrough, assigned-only environment, executable drift, call serialization, cancellation and unknown outcomes. The native Codex suite passed 91 tests with two intentionally ignored; `receipt_limit_rejects_the_call_and_keeps_polling_when_interrupt_fails` timed out waiting for `turn.interrupted`, including an isolated retry. The unchanged pre-fix commit `80917b301a2309f7966a466cf0c198a2112be2d5` reproduces the same timeout in isolation; this local native-suite run is not claimed as fully passing. - All 252 native controller tests pass against the final narrowed approval branch. Focused TypeScript transport/permission suites passed (298 tests); final binding, read-only host, policy and adapter checks passed (105 tests). - **Provider-client smoke coverage:** pinned Codex 0.160.0 through native Runner and Claude SDK 0.3.286 used local stdio MCP servers and loopback model API fixtures without live model credentials. These clients define the smoke-test coverage; this is not an exhaustive adapter compatibility test. Image responses reached the next model request as typed image blocks. Native Codex checks also verified unattended execution without approval requests, explicit per-tool prompt denial without backend execution, disabled-tool rules, and stale-turn approval rejection. - Two Greptile concurrency findings were reproduced with failing subprocess tests and fixed: pending cancellation under output backpressure, and a ctime-based fence reset after chmod. Initial checks exposed Codex CLI option-scope and key-quoting errors, which were corrected. The existing ordinary elicitation regression also exposed an overly broad request branch; the final branch handles explicit tool approvals only. - `pnpm test:run` was attempted. The broad local server suite hit persistent embedded PostgreSQL startup failures and unrelated local Git fixture failures; it was stopped with the failure log retained. Those local results are not claimed as passing. Hosted Linux CI gates passed on the reviewed runtime commit, as recorded below. - First Apex review found three additional defects, now fixed with regressions: keep the resource-lock socket snapshot instead of sampling the socket twice; project native permission requests with durable turn/item IDs while retaining provider IDs in details; validate provider response conversion before marking a request as settling. The retry regression failed before the fix. Focused Codex request/adapter checks pass (23 tests), server permission projection passes (16 tests), and all 379 Runner library tests pass. The broader database-backed bridge suite was attempted but its local PostgreSQL setup timed out; 29 tests did not execute. This local failure is retained, not claimed as passing. - Runtime acceptance before the documentation clarification, on `bd4fccae001157beb5bc76ff17c1920a5db3105a`: standard Greptile 5/5, then Apex 5/5 with no unresolved threads; all 60 checks complete successfully or intentionally skip. Ten CI jobs were interrupted by the same runner shutdown, including one stale job that required cancellation before retry; the same-commit retry passed. Hosted Rust tests also pass, including the native suite with the local baseline timeout noted above. - `git diff --check`, map validation, and pre-push secret/PII scan passed. ## Risks - No live model account or deployment isolation qualification was run. The loopback fixture proves the actual vendor-client request format, not model interpretation. - Stdio MCP support uses the current host-assignment contract: each binding requires a verified executable and a private Unix socket identifying the serialized resource. Other platforms retain the HTTP path. No provider version was upgraded. - A lost acknowledgement blocks subsequent calls until the host restarts the assigned resource. Direct shell calls are outside this MCP fence and retain the existing shell policy. - Preconfigured Claude remains unattended and denies interactive approval requests. Host assignment authorizes native Codex local tools; it must be supplied only for sessions already authorized to use the resource. ## Model Used OpenAI Codex, GPT-6, with reasoning and repository/tool execution. The session does not expose a more specific model identifier or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
ce0eff7d1f |
perf: reduce fresh workspace clone and sandbox staging waits (#16487)
## Thinking Path > - Paperclip manages AI agents and their work. > - Each run needs a private workspace and safe persistence. > - Fresh isolated repository sources download upstream Git history again. > - Sandbox staging also verifies an optional seed cache before it uploads the workspace. > - That verification holds a company lock during a full extraction and hash scan. > - This pull request reuses pristine Git objects through an authenticated remote clone. > - It also overlaps seed verification with upload and moves verification outside the publication lock. > - The benefit is less repeated transfer and less serial preparation work, with the same task isolation and recovery contract. ## Linked Issues or Issue Description **What existing behavior does this improve?** Fresh task-private repository preparation and inbound sandbox workspace staging. **Current behavior** Each isolated remote source clones the full upstream history. Optional seed verification then extracts and hashes another full tree before upload. One workspace's verification can hold the company cache lock while other workspaces wait. **Proposed behavior** Use a company and credential scoped cache of pristine upstream objects for GitHub HTTPS sources. Each task still clones from the current remote. It disconnects from the cache before publication. Run seed verification alongside upload. Hold the company lock only for final capacity checks and publication. **Reason and benefit** These operations lie on the workspace preparation path. Removing repeated Git pack transfer and serial verification can reduce startup time. Local measurements below cover these boundaries. They do not establish hosted agent latency. Related: #16425, #16454, #16398, #16461, #16468, #16473. The cache inspection and durable folder proposal in #13376 covers separate product work. ## What Changed - Add a pristine upstream Git object cache for fresh isolated GitHub HTTPS sources. Key it by company, repository and current credential fingerprint. Do not store credentials. - Keep online clone authorization and current remote refs. Use `--reference-if-able --dissociate` so task objects remain independent. A reference failure retries once online without the cache and resolves credentials again. - Cap cache storage at 64 entries and 4 GiB per company, with 2 GiB per entry. Skip contended publication. Bound priming by a cooperative five-second budget and a 4,096-entry scan allowance. Keep the existing free-space reserve. - Overlap optional seed verification and required workspace upload. Settle both before returning or deleting temporary files. Recheck quotas and company directory identity under the publication lock. - Keep the independent fresh upstream reproducibility proof, task commits, dirty source copies, ignored files, nested repositories and durable recovery behavior. - Suppress optional Git server-info generation during dissociation so generated storage passes the existing safe-expiration checks. This uses [the supported Git setting](https://git-scm.com/docs/git-config/2.43.0#Documentation/git-config.txt-repackupdateServerInfo). - Add bounded timing phases, regression coverage and two repeatable local benchmarks. No first-party Telemetry or dependency changes. ## Verification - `pnpm -r typecheck`: passed. - `pnpm build`: passed. - Affected adapter suites: 240 tests passed. Native workspace persistence/history and tracing: 33 tests passed. Server checkout/isolation/tracing batch: 53 tests passed. Final object-cache authority, path and capacity suite: 12 tests passed. These batches overlap. - Cache and expiration/restoration lifecycle batch: 34 tests passed. A direct generated-storage check covers cached clones. - Real HTTP Git fixture: a matching cache does not bypass revoked remote access. Other fixtures cover current refs/tags, hidden cached commits, independent objects after cache removal, corruption fallback, credential/company scope, quota limits and publication contention. - Initial restricted runs hit SQLite write and Unix socket permission failures. The adapter batch passed with disposable writable instance state and socket access. The first typecheck rejected the test use of `Promise.withResolvers`; typed deferred gates now use the supported library contract. - Repository benchmark on final head `9047368bf3b7fa2061e00d698b8b51cf4ae1da32`: five alternating samples, real local `file://` Git transport, synthetic 32 MiB history, no injected latency. Median 1143.8 ms to 240.5 ms; ranges 1085.7–1486.6 ms and 188.0–254.8 ms. Received Git pack bytes fell from 33,569,788 to zero. Initial cache priming took 1208.4 ms and is reported separately. - Sandbox benchmark, five alternating samples per mode on 10,000 files with real packing, extraction and verification: with a controlled 3-second transfer delay, median 9.450 s to 7.620 s (ranges 9.280–12.115 s and 7.296–7.983 s). With no injected delay, median 6.808 s to 6.450 s (ranges 6.255–7.029 s and 5.709–6.902 s). The no-delay ranges overlap; local copying competes with verification for disk. Both modes verify final tree contents, ignored-file exclusion and Git HEAD. These are synthetic local fixtures, not hosted agent runs. - Reproduce with `pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-repository-object-cache.ts` and `pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-workspace-staging.ts --baseline-ref=307eb16c0c --transfer-delay-ms=3000`. - Prior head `af766bc820` received Greptile 5/5 with no findings. Its E2E shard 7 failed in `plan-task-fixture` cleanup: company deletion raced an `activity_log` foreign key. That fixture has no provider run or Git workspace. The failure is retained here; final-head CI must pass. The corrected corruption fixture now forces a pack and verifies a real damaged archive. Current head `9047368bf3b7fa2061e00d698b8b51cf4ae1da32` has Greptile 5/5 with no review threads or findings. Final head has 55 passing checks and two optional Storybook skips. [The first workspace-shard attempt](https://github.com/paperclipai/paperclip/actions/runs/38802441857/job/116655485350) passed all 1,778 cases, then reported a late `EPIPE` from unchanged `server-utils.ts` while the ACP reuse test was active. That case mocks workspace preparation and passed locally in isolation. [Its single retry on the same commit](https://github.com/paperclipai/paperclip/actions/runs/38802441857/job/116659728034) passed, as did the dependent verification gate. The earlier E2E cleanup fixture passed on this final head. Both initial failures remain recorded here. A full local suite was not duplicated; the affected boundaries were run locally and the complete suite runs in CI. ## Risks - First cache population adds local copying. Cold clones, credential rotation, capacity exhaustion and a corrupt cache retain the online clone path. No eviction is added. Cached data is bounded, but many companies can each consume their allowance. - Individual filesystem calls and cleanup can exceed the cooperative priming budget. Failed reference clones make one additional online attempt. - Parallel verification and upload compete for disk and CPU. The local no-delay benchmark shows a smaller gain than a slow-transfer case. Verification still checks contents before cache publication, and required upload failures remain failures. - Local fixtures do not prove hosted throughput or the 15–30-second end-to-end target. The fresh checkout still writes its working tree, and model startup, copyback and other preparation costs remain. ## Model Used OpenAI Codex, GPT-6 family. The exact serving model ID and context window are not exposed in this session. Used reasoning, source editing, shell tools and test execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal task identifiers - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>canary/v2026.1016.0-canary.14 |
||
|
Devin FoleyandPaperclip
|
647116dd4b |
fix(adapter-utils): allow sandbox agents to read run diagnostics (#16575)
## Thinking Path > - Paperclip manages agent work and its execution history. > - Sandbox agents use a callback bridge to reach the control plane. > - The API already provides authorized run diagnostics. > - The bridge denied those read routes before the API could check access. > - Health scans could see task activity but could not inspect failed runs. > - This pull request admits exact GET routes on both bridge transports. > - Linked task results also apply the caller's task-read filter. > - Agents can inspect failures through their existing run identity. ## Linked Issues or Issue Description **What happened?** A sandbox health scan received `403 Route not allowed` for run history, run details, events, and task recovery records. The same agent could read task activity and connected provider state. The bridge stopped these diagnostic requests before they reached the API. **Expected behavior** The bridge should admit the existing diagnostic GET routes. The API must still enforce company and task access and redact run content. Execution and recovery writes must stay denied by these rules. **Steps to reproduce** 1. Start an agent in a sandbox with the normal callback bridge. 2. Use its injected API credential to request company run history. 3. Request a run's details or events. 4. Observe the bridge's route denial before the API handles the request. **Paperclip version or commit** Reproduced with the callback route list on master at `9f4a956b906`. **Deployment mode** Authenticated server with a managed sandbox. Both queue and HTTP/2 transports use this route list. Related work: Refs #15854 adds separate cost-read routes to this list. It does not add run diagnostics. Searches found no duplicate run-diagnostic change. ## What Changed - Admit nine existing diagnostic GET paths: company history and live runs; run details, events, logs, workspace operations and task links; task runs and recovery records. - Reject other methods, extra path segments, execution control, recovery resolution, and provider-trace administration. - Filter linked tasks by the caller's task-read predicate in both activity and original-task queries. Keep those results in the run's company. A readable run must not expose private task titles. - Test both route lists and the HTTP/2 request path. Preserve query parameters and controller denial responses. Add a database regression with public and private tasks linked to a readable company run. - Document bounded diagnostic reads, private-run stubs, access checks, and credential handling in the agent API reference. Update both affected run access maps with the sandbox entry points and verification limits. ## Verification - Passed both bridge suites: 139 tests. - Passed the server run-route, activity-route, and secret-redaction suites: 117 tests, including the new task-link filter test. The cold route-import test needed a 30-second timeout on this host. - Passed `pnpm -r typecheck`, `pnpm build`, the adapter-utils typecheck, both generated capability checks, and `node scripts/check-maps.mjs`. - Attempted `pnpm test:run`. The bundled PostgreSQL binary cannot start on this host. The native session suite reported 15 database startup failures; the privacy suite skipped its 15 cases. Stopped that full local attempt after those failures. No full local-suite pass is claimed. The final 16-case privacy suite also skipped locally for the same host limitation; Linux CI ran all 16 cases successfully. - One unchanged queue-response test hit a file-read race during the combined local run. A separate rerun of both complete bridge suites passed all 139 cases. The transient race was not independently isolated; the original failure is retained in this record. - All CI gates pass on head `97c65cec459d3fc2787083d05ae6ec51e3fd7b70`. Linux CI passed all 16 database-backed run privacy cases, including the new public/private task-link regression. These also cover private-run denial, authorized owner/reviewer reads, redacted list stubs, credential diagnostics, and deleted-task handling: [serialized shard 1](https://github.com/paperclipai/paperclip/actions/runs/38925932547/job/117057907621). - Apex reviewed the final head at 5/5. Both findings are addressed and their review threads are resolved. - After deployment, run a new sandbox audit. Confirm it can list and inspect authorized failed runs without a board credential. This has not been deployed or live-tested by this PR. ## Risks - Sandbox agents can reach more diagnostic GET routes. The existing server access checks and redaction still apply. - Linked task results now omit tasks that the caller cannot read. Authorized task links remain visible. - This change does not grant execution control or recover a failed workspace. - Existing bridge sessions may need a new run to load the updated route list. - No schema, credential, UI, workflow, or lockfile change. ## Model Used OpenAI Codex, based on GPT-6, assisted with implementation and verification. The exact serving model ID and context-window limit are not exposed in this session. Capabilities used: code editing, shell execution, and test execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
06689c1b34 |
perf(skills): coalesce inventory before run cache locking (#16576)
Share each active company skill-inventory refresh before taking the version-cache lock. Read actor visibility under the lock and keep snapshot publication protected through durable run pinning. Preserve empty-selection behavior and the list API. Validation: 176 affected cases, full repository typecheck/build, fresh full CI/E2E, security and exact-head Apex 5/5 pass. The local six-reader latency experiment improves from 50.4s to 18.3s; live response latency is a separate acceptance check. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandPaperclip
|
5fbece9dcc |
Synchronize assigned skills through agent runtime hooks (#16501)
## Thinking Path > - Paperclip manages shared skills and agent assignments. > - Runtime plugins need installed skills before a task starts. > - Skill content can change without changing the agent row. > - Source access and version pins also affect delivered files. > - This pull request feeds those changes into the durable runtime stream. > - The runtime reader resolves the same immutable skill sources used for execution. ## Linked Issues or Issue Description Builds on merged PR #16499. This pull request now targets `master` and contains only the five skill synchronization commits. **Problem** Plugins receive no durable notification when an assigned skill changes. A saved assignment can also become inaccessible after private-source access changes. **Proposed solution** Invalidate referencing agents at installed skill mutations. Invalidate inherited access when source bindings, project privacy, or policy change. Return authorized skill sources and version IDs through the owning runtime reader. **Alternatives considered** Activity-only subscriptions miss service writers. Per-task synchronization adds preparation work. Copying current files would ignore version pins. **Roadmap alignment** This extends existing skills, source synchronization, and plugin runtime support. ## What Changed - Defer multi-skill source runtime hooks until commit, after publication holds all skill rows. Keep file-save journaling immediate to preserve filesystem rollback. - Add transactional invalidations for skill content, companion files, versions, rename, deletion, policy, source access, and pin-setting changes. - Recognize canonical keys and supported legacy aliases. - Reuse the execution skill resolver, current agent access, and instance version-pin rules. - Resolve and materialize sources outside the agent row lock. Recheck ownership and runtime revision afterward. - Keep task-only mentions and running-task snapshots in the execution path. - Document conservative invalidations for pinned assignments and inherited access. ## Verification - Rebased onto origin/master `9f4a956b90`, which contains merged PR #16499. All five commits are unchanged according to `git range-diff`; no conflict resolutions or source changes were needed. - On rebased head `58b065941`, all 237 focused tests passed: runtime skill hooks/source publication (41), plus content service, assignment routes, policy routes, and project source access (196). - Server typecheck passed after rebuilding Runner with the installed Rust toolchain. Migration numbering/safety and map checks passed; snapshot pruning made no changes. Migration CLI checks required local IPC permission. - Real PostgreSQL coverage includes concurrent source publication/version saves, policy-only tightening/restoration, referencing-agent fanout, company isolation, no-op counters, supporting files, rename/deletion, rollback, private-source access, pinned/latest bytes, and unassignment. - Earlier stack validation passed workspace typecheck/build. Fresh hosted CI and review must verify the rebased head before handoff. - The broad local suite was previously limited by an unchanged diagnostic test that rejects "private" in an error stack under this /private/tmp worktree; that unchanged suite passed all 108 tests in the original checkout. Hosted full-suite checks provide exact-head verification. - Diff whitespace and full PR secret/PII scan passed. - All hosted CI/security checks are green on `58b065941`; fresh standard Greptile and Apex both returned 5/5 with no findings or unresolved threads. The PR targets master and is mergeable without conflicts. ## Risks - Apply migration 0348 after the runtime-stream migration in #16499. - Content invalidations include pinned assignments. The resolved version stays pinned; consumers can avoid writes when its version/hash is unchanged. - Project access and source changes invalidate only agents assigned skills from the affected sources. Company policy still invalidates the company. Source polling bookkeeping and popularity counters do not create work. - The reader rebuilds installed caches without upstream requests. Concurrent changes cause a bounded retry and then a conflict. - Private transport integration and live remote computer qualification remain separate. ## Model Used OpenAI Codex, GPT-6, with reasoning, repository inspection, code execution, and testing. The exact deployment model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub references) - [x] My branch name describes the change and contains no internal Paperclip ticket id - [x] I have run tests locally and they pass (focused suites; broad local limitation documented above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>canary/v2026.1016.0-canary.12 |
||
|
Nicky LeachandPaperclip
|
9f4a956b90 |
Add durable agent runtime synchronization hooks (#16499)
## Thinking Path > - Paperclip manages agents and their runtime configuration. > - Plugins can maintain external agent computers. > - Configuration and file saves currently miss durable plugin delivery. > - A transient activity notification cannot recover after interruption. > - This pull request adds an opt-in agent runtime resource stream. > - The existing dispatcher delivers and acknowledges its committed work. > - Plugins can prepare computers when configuration changes. ## Linked Issues or Issue Description Related: #15631 and #16203 establish agent lifecycle and native durable delivery. **Problem** Plugins cannot reliably react to agent configuration and instruction changes without polling or waiting for execution. **Proposed solution** Add an opt-in `agent_runtime` stream to the existing resource journal. Record changes at the database mutation and managed file audit boundaries. Wake the dispatcher with content-free commit notifications. Provide scoped runtime inputs to the selected lifecycle plugin. **Alternatives considered** Route-only activity events miss service and background writers. Scheduled scans delay changes and repeatedly query unchanged state. **Roadmap alignment** This extends the existing plugin and agent configuration work. ## What Changed - Add durable update/delete events for agent runtime state, including an existing-agent baseline. - Capture configuration changes, instruction edits, deletion, restore, and managed run writeback. - Add commit notifications and a reconnect scan. Register one listener for configured, enabled runtime subscribers. Release it when none remain. - Keep HTTP startup independent of queued plugin callbacks. - Add `ctx.agents.getRuntime`. Require lifecycle capability, invocation company, enabled plugin, and owning lifecycle selection. - Keep legacy agent/project subscriptions isolated. Manual inbox reads retain both legacy streams. - Resolve recovered instruction paths without reading or hashing bundle contents. Use the company reporting graph for runtime health. - Repair missed file events on identical save/deletion retries after a journal failure; preserve content tokens and conditional-write checks. - Correlate file audits with runtime revisions to deduplicate recorded retries without a new table. Bound the lookup with the existing company/timestamp index. - Document the contract, migration, access boundaries, and session-connection requirement. ## Verification - Rebased the full stack onto origin/master after #16507. - Final replayed stack: `pnpm -r typecheck` and `pnpm build` passed. - All 132 affected runtime, skill, connector, publication, instruction, managed-agent, and delivery tests passed after the latest-master rebase. Workspace typecheck and build passed. - Journal failure injection verifies recovery on identical file writes, deletions, and instruction retries from an acknowledged inbox. - Deduplication follow-up: 80 runtime/instruction/delivery tests and server typecheck passed. Repeated retries create no audit/events; old hashes returning after an intervening commit still repair delivery. Built-in/native instruction writer regressions passed all 56 tests. - PR 1 follow-up tests cover valid/cyclic reporting chains, listener registration failure, multiple subscribers, global/company disable, re-enable, unload, and startup with blocked callbacks. - Database coverage includes rollback, journal failure, no-op suppression, file saves/deletion, ordering/replay, owner/company denial, independent-client commits, and real listener loss/reconnect. - SDK/shared suites passed 1,059 tests during implementation. CLI/delivery regressions also passed. - Migration safety, map checks, diff whitespace, and local secrets/PII scan passed. - Hosted CI and security checks are fully green at `b296405e2`. Standard Greptile and Apex both returned 5/5 with no unresolved findings. - The broad local suite encountered an unchanged diagnostic test that rejects the word "private" from an error stack. This worktree lives under /private/tmp. The same unchanged diagnostic suite passed all 108 tests in the original checkout. Hosted full-suite checks provide exact-head verification. ## Risks - Apply migration 0347 before enabling a runtime subscriber. Runtime notifications require session-level PostgreSQL LISTEN. Transaction-mode poolers are unsupported for this opt-in feature. - Database triggers share the mutation transaction. Journal failure prevents configuration commits. Trigger work adds a journal insert to relevant changes. - Filesystem publication and SQL commit are separate boundaries. Direct filesystem edits do not emit hooks. External clients must fence stale revisions and retain conditional-write checks. - The reader returns only that plugin's own instance secrets. Keep them out of UI data, logs, plugin state, and agent files. - No private transport integration or live remote computer qualification is included. ## Model Used OpenAI Codex, GPT-6, with reasoning, repository inspection, code execution, and testing. The exact deployment model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub references) - [x] My branch name describes the change and contains no internal Paperclip ticket id - [x] I have run tests locally and they pass (focused suites; broad local limitation documented above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
b294abb9eb |
perf(server): append ordinary run events in one statement (#16569)
Ordinary run events now validate their binding, allocate a sequence and insert the decoded receipt in one atomic PostgreSQL statement. Native replay and retry-exhaustion transactions keep their existing behavior. Verification: exact-head full CI/E2E, Superagent, legacy security and Apex 5/5 passed. PostgreSQL concurrency, rollback and payload controls pass. Controlled network-delay comparison reduced median event persistence from about192ms to48ms; no staging response-time claim is made before deployment. Co-Authored-By: Paperclip <noreply@paperclip.ing>canary/v2026.1016.0-canary.11 |
||
|
DottaandPaperclip
|
d75b3fc33f |
perf(daytona): observe command exit before log-follow closure (#16561)
Return completed Daytona session commands after recorded exit and a final log snapshot. Keep bounded early status reads, command deadlines, late callback fences and stream recovery. Co-Authored-By: Paperclip <noreply@paperclip.ing>canary/v2026.1016.0-canary.10 |
||
|
DottaandPaperclip
|
43ffe8ce95 |
perf: skip conditions for unselected tool profile entries (#16556)
Check exact tool profile selectors before evaluating conditions. Matched entries retain fresh policy condition checks and existing approval, deny, binding, and decision-phase behavior. Validation: 70 affected cases pass with zero skips. Full repository typecheck and build pass. A balanced same-database comparison over 47,931 entries preserves all decision JSON and reduces 200 decisions from about 1.05 seconds to 0.40 seconds. Fresh exact-head full CI, E2E, Superagent, legacy security, and Apex 5/5 passed with no unresolved findings. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
8556539364 |
perf(adapter-utils): batch remote GitHub launcher staging (#16552)
Stage the nine managed GitHub launcher files through one bounded control command. Preserve per-file locks, content verification, atomic replacement and control-channel dispatch. Real Daytona staging decreased from about one second to 0.24–0.34 seconds. Full CI, E2E, Superagent and Apex 5/5 passed on the reviewed head. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
|
8f1cb11a03 |
fix(workspaces): name the owner action when a shared source is restricted (#16545)
A reused task workspace in a shared project checkout keeps its source denial when another task's restricted worktree shares that checkout (#16095 deliberately does not clone over existing workspaces). The run then failed as an opaque `setup_failed` 403 and the task was blocked with a generic "no live execution path" notice. Convert that exact denial into a `configuration_incomplete` failure with reason `workspace_source_restricted`, give the task notice and the recovery action the owner step, and keep this pre-dispatch owner action out of the unexpected Sentry run-failure stream. The reason and predicate live in a dependency-free module. Access checks are unchanged. Co-Authored-By: Paperclip <Paperclip-Paperclip@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1016.0-canary.8 |
||
|
DottaandPaperclip
|
f1d4bc1617 |
perf(slack): resolve only signing secret during webhook checks (#16548)
Resolve only the current Slack signing secret during workspace checks and authenticated setup evidence. Preserve full runtime credential resolution, company bounds, rotation, and auditing. Co-Authored-By: Paperclip <noreply@paperclip.ing>nightly/v2026.1016.0-nightly.0 canary/v2026.1016.0-canary.7 |
||
|
Devin FoleyandPaperclip
|
78b833c495 |
Dispatch routine and monitor work at saved deadlines (#16514)
## Thinking Path > - Paperclip manages work that agents run for a company. > - Routines and issue monitors store their next run time in the database. > - The app currently waits for the heartbeat interval to find that work. > - This can delay work after a workspace wakes before its saved deadline. > - This pull request uses the existing work coordinator to schedule each known deadline. > - It also parses each cron expression once, so sparse schedules do less work on the event loop. > - Agents can start due work sooner while the existing execution checks stay in place. ## Linked Issues or Issue Description **What happened?** Routine and issue-monitor dispatch depends on the heartbeat scan interval. A workspace can wake early and still wait for the next scan. A sparse cron search also parses the same expression for every minute it checks. In a local annual-schedule test, the deadline callback ran on time but schedule calculation delayed the routine record by 13.6 seconds. **Expected behavior** Use a saved future deadline to request the existing dispatch scan. Restore deadlines at startup and update them after a producer transaction settles. Do not let future timers keep an idle workspace active. Parse a cron expression once for each search. **Steps to reproduce** 1. Set an active routine and an assigned issue monitor to the same future deadline. 2. Choose a deadline between heartbeat scans. Use an annual routine expression to exercise a sparse search. 3. Compare the saved deadline with the routine firing and agent start times. **Paperclip version or commit** Source checkout based oncanary/v2026.1016.0-canary.6 |
||
|
DottaandPaperclip
|
82a12f1a8b |
perf(skills): batch project source authorization reads (#16536)
Batch project-source read authorization through the existing decideReads path. Keep company scope, source union rules and fresh permission decisions. The local80-project fixture reduces select builders from322 to4 and complete legacy preparation from689ms to128ms. All88 affected cases and both fresh-database cases pass. Full current-head CI/E2E, security and Apex5/5 pass with no unresolved findings. This is component proof; hosted latency still needs verification. Co-Authored-By: Paperclip <noreply@paperclip.ing>canary/v2026.1016.0-canary.5 |
||
|
DottaandPaperclip
|
185483e090 |
perf(slack): keep authorization revision rows inside PostgreSQL (#16527)
Compute Slack authorization revisions in one PostgreSQL statement while retaining all existing scope and invalidation inputs. Return one ordered digest instead of materializing retained profile rows in Node. Validated with 19 affected cases, full typecheck/build, final-head CI/E2E/Superagent, and Apex 5/5. Co-Authored-By: Paperclip <noreply@paperclip.ing>canary/v2026.1016.0-canary.4 |
||
|
github-actions[bot]andlockfile-bot
|
560d895306 |
chore(lockfile): refresh pnpm-lock.yaml (#16516)
Auto-generated lockfile refresh after dependencies changed on master. This PR only updates pnpm-lock.yaml. Co-authored-by: lockfile-bot <lockfile-bot@users.noreply.github.com> |
||
|
DottaandPaperclip
|
90924d61b3 |
fix(tools): batch profile selector validation (#16521)
Batch company-scoped profile selector validation while preserving both existing write boundaries and exact selectors. Verified with 81 affected local cases, full repository typecheck/build, current-head CI/E2E, Superagent and Apex 5/5; no unresolved findings. Co-Authored-By: Paperclip <noreply@paperclip.ing>canary/v2026.1016.0-canary.3 |
||
|
DottaandPaperclip
|
0e58f29eb7 |
fix(observability): trace pre-dispatch controller waits (#16515)
Measure usage startup, MCP handoff, dispatch admission, and budget reservation with the existing endpoint-gated tracer. Preserve outer attempt context across asynchronous provider dispatch.
Validated with 135 focused tests, full typecheck/build, fresh full CI/E2E, Superagent, and Apex 5/5 on
canary/v2026.1016.0-canary.2
|
||
|
Nicky LeachandPaperclip
|
fa9f3f8a08 |
feat(runner): support preconfigured user homes for Codex and ACPX (#16486)
## Thinking Path > - Paperclip manages AI agents and their work. > - Runner launches the selected provider and keeps durable session state. > - Runner currently prepares private provider homes for each session. > - Operators can already have working login, skills, and MCP configuration on the execution machine. > - This PR adds an explicit mode that uses that existing setup. > - Runner keeps its task bridge and session ownership while leaving the user's configuration in place. ## Linked Issues or Issue Description **Problem or motivation:** Rebuilding a provider home is unnecessary when the execution account is already configured. **Proposed solution:** Add `homeMode: "preconfigured"` to Runner adapter config and runnerd transport options. Keep `"isolated"` as the default. Initially support native Codex and ACPX Claude. **Alternatives considered:** Continue copying user configuration into a private home, or add an approval workflow. Both add work outside this feature's scope. **Roadmap alignment:** This extends the existing cross-provider Runner. It does not add another runtime. Related work: #14056 covers managed Codex credentials. #14474 covers command environment delivery. This mode uses user-owned credentials and retains the existing launch boundaries. ## What Changed - Add a closed v7 launch contract for the selected home mode. Existing contracts retain their behavior. - Keep homeMode under operator control. Reject task overrides and agent-authenticated changes, including removal through replacements, adapter changes, or rollback. Ignore stored task values and exclude it from automatic hire inheritance. - Skip managed AI login, assigned-skill materialization, and external MCP setup in preconfigured mode. Preserve the Runner task/completion bridge. - Keep protocol state and provider lifetime fences scoped to the Runner session. Different sessions can share a home. - Fail preconfigured ACPX planning/read-only launches before provider admission until a read-only user-home launch is qualified. Use isolated mode or native Codex for planning. - Reject ACPX approval requests. Keep native Codex on `never`. Local permission rules and hooks remain effective. Initially require the default Runner ACPX `approve-all` setting; other Runner policies are unqualified in this mode and fail before launch. - Keep the internal ACPX bridge in a stable session namespace, preserving an existing user MCP server named `paperclip`. - Reject preconfigured ACPX Codex before launch until a session permission profile can enforce Runner-state denial. Remove the unprotected full-access mode switch; use native Codex or isolated mode for that model family. - Bump Pi to profile 23 because its attestation covers shared Runner source changed here. Preserve profile 22 for historical recovery, and update Docker/package verification inputs. - Fence recovery on mode and home bindings. Fail missing provider-history recovery without starting a replacement session. Exclude the external Codex home from backups and credential cleanup. - Discover remote HOME on the execution machine. Do not project controller HOME or XDG paths onto it. - Separate task instructions from Codex ACP isolation and update the generated provider digest. - Keep remote configured-environment manifests consistent when dropping controller paths. Preserve exact Claude model IDs through per-launch environment settings, without editing the user home or patching the provider. ## Verification - Targeted local checks cover native contracts and launch allowlists, user-config preservation, provider admission, permissions, session recovery, remote environment projection, heartbeat preparation, and operator-only configuration. Runner, shared, and server TypeScript checks and the Rust binary build pass; generated ACPX profiles are current. - Following the latest master integration, all 757 server configuration/executor tests and 115 Runner admission/recovery tests passed. Additional review-fix checks passed: 103 planning contract/host tests, 78 provider-backend tests, 95 route/preparation/inheritance tests, 43 task-validator tests, and 43 workspace-policy tests. Counts describe separate runs and can overlap. - The real pinned Claude ACP/SDK regression reproduces exact-model aliasing without the fix (`claude-sonnet-5-5` becomes `sonnet`) and passes with the fix. All 89 adapter tests pass. User settings remain byte-for-byte unchanged. - Localhost-only Claude SDK probes with synthetic credentials verify the private bridge runs, local `ask` reaches the existing reject callback, local `deny` blocks without a callback, and a user MCP server named `paperclip` remains available. No user model allowlist is required. - Review regressions cover fail-closed provider admission, saved-mode recovery before routing, controller-home path removal without configured-environment manifest drift, pre-launch rejection of unqualified read-only ACPX launches, and denial of task/agent changes to the operator-owned home mode. - Before rebase, `d4d4ee2aebe4cb88c4f4d9ef8e22fed4b36b87be` received standard Greptile 5/5 with no unresolved threads. Prior Apex findings are fixed and covered by regressions. No manual Greptile retrigger is requested for this rebase, per reviewer direction. - The prior PR head and master hit the same GitHub runner encoding crash in workspace-b on three PR attempts, including diagnostic logging ([prior PR CI](https://github.com/paperclipai/paperclip/actions/runs/38818474563/job/116714538405), [prior master CI](https://github.com/paperclipai/paperclip/actions/runs/38819528748/job/116712423193)). Rebased onto `origin/master` at `c3ffa3f0dd` to pull in its bridge-test index correction. The rebased head `edd153db410bea063b645e0000aa68649f287b00` passes all CI checks ([CI run](https://github.com/paperclipai/paperclip/actions/runs/38824676914)), including workspace-b. Greptile's automatic standard review is 5/5 on this exact head, with no unresolved threads; no manual review was requested. - Local workspace-b investigation passed 911 shared tests and 3,868 tests across the remaining 15 workspace projects. The catalog run separately hit an existing ignored `.agents/skills/design-guide/SKILL.md` description-budget violation in this checkout; that file is untracked, outside the PR, and left untouched. Repeating shared tests with CI reporting enabled also passed. - Rebase verification: the final Git tree exactly matches the previously reviewed branch combined with current master. All 192 Runner admission/contract tests, 61 server preparation/provider tests, and 68 bridge tests pass, as do Runner/server TypeScript and generated-profile checks. The real Claude adapter fixture hit an `ENOTEMPTY` cleanup race on its initial run; all 89 adapter tests passed on retry. - Apex on rebased head `edd153db` returned 2/5: ACPX Codex selected full access without enforcing Runner-state denial, and agent configuration replacement could remove the operator mode. Both are fixed in `26e14dbbf`. Preconfigured ACPX Codex is temporarily unqualified; re-enabling it requires session permission enforcement on thread/turn operations and a real filesystem-denial probe. The config guard covers effective replacements, adapter changes, and rollback while allowing ordinary agent merges and operator removal. All 190 focused Runner tests, 103 server tests, both typechecks, and generated-profile checks pass. Standard review and CI passed before the next Apex request. - CI on `26e14dbbf` passed 1,967 Runner-shard tests but hit the previously observed `ENOTEMPTY` fixture cleanup race after successful Claude model admission. `eb83284fb` adds bounded filesystem retries to that fixture's teardown; all 87 adapter tests pass. Assertions and production cleanup behavior are unchanged. Head `eb83284fb42ee19a15d9a93b106c1bbb07e1695e` passes all 59 CI checks ([CI run](https://github.com/paperclipai/paperclip/actions/runs/38829500857)) and standard Greptile at 5/5 with no unresolved threads. Apex also completed at **5/5** on that exact head ([Apex result](https://github.com/paperclipai/paperclip/pull/16486#issuecomment-6201559527)). All review threads are resolved, the branch has no merge conflicts, and the PR remains unmerged. - No live paid-provider or remote-machine smoke was performed. The lockfile remains unchanged under the repository's refresh-bot policy; local dependency installation resolved the changed patch hash. ## Risks This explicit opt-in grants Codex tools access to the execution home (read-only in planning), including credentials. Native Codex separately denies Runner-owned protocol state. ACPX Claude retains its existing provider trust boundary, and the isolated default is unchanged. Providers may refresh credentials and update their own metadata. User configuration can block tools or the completion reporter. Existing blocked results and failure diagnostics remain the recovery path. Edit the local settings and retry; no new approval workflow is added. External provider history remains on the execution machine and is not backed up by Runner. Remote HOME discovery requires `getent` and fails explicitly when unavailable. Other harnesses need separate qualification. The Codex ACP bundle digest changes. Existing sessions tied to the old bundle may need a fresh session. No database migration is required. ## Model Used OpenAI Codex, GPT-6, with reasoning and code execution. The exact runtime model ID was not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR following the feature issue template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal ticket id - [x] I have run targeted tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented risks above --------- Co-authored-by: Paperclip <noreply@paperclip.ing>canary/v2026.1016.0-canary.1 |
||
|
Devin FoleyandPaperclip
|
c7ea5c0216 |
perf(server): skip settled committed finalizations in unscoped passes (#16485)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - On restart, `/api/health` reports `starting` until startup recovery finishes, and deploys wait for it > - Native recovery and the orphan reap each run an unscoped native finalization reconciliation pass, and readiness waits for both > - The pass selects every committed finalization with a result, including runs that ended long ago, and spends dozens of sequential queries on each one > - On a busy instance this is about one minute per pass, so about two minutes of every restart, and the periodic recovery repeats it on every tick > - This pull request skips settled committed finalizations in unscoped passes, and keeps a small rotating upkeep page so their upkeep still runs > - The benefit is faster restarts and less steady database load, with no change for unsettled work ## Linked Issues or Issue Description Related: #16330 and #16329 (startup and reconciliation timing). **What existing behavior does this improve?** Startup recovery time and steady-state recovery load. **Current behavior** - The unscoped candidate scan in `reconcileNativeFinalizations` returns every native finalization with a result and phase other than `terminal_failure`. Nothing excludes rows that are already settled. - Each committed row then takes the full per-row path: pending-effect, assessment, evidence and decision reads, presentation repair (twice), the workspace barrier read, a `finalizeNativeRun` re-read of the committed coordinator, and workspace settlement (lease release check and agent status update). That's about 40–60 sequential round trips per row. - Startup runs the unscoped pass twice (native recovery, then the orphan reap), and readiness waits for both. The periodic orphan reap runs it again on every tick. - On one instance, all 268 candidates were committed finalizations whose runs had ended. Native recovery took 50–80 s and the orphan reap 47–80 s on every restart, with database round trips of about 3.4 ms. **Proposed behavior** An unscoped pass skips a committed finalization when: - its run is `succeeded`, `failed`, `cancelled` or `timed_out`, and - the run finished more than an hour ago, and - its decision has no `pending` or `failed` status-decision effect. Each unscoped pass also includes up to 10 such settled rows in a rotating order, so the presentation and evidence upkeep still reaches every row over time. Scoped passes (explicit run IDs) are unchanged. **Reason and benefit** These settled rows have nothing left to finish. Recent rows, unfinished runs and rows with undelivered effects stay in every pass, so crash-window repairs and effect delivery behave as before. Startup and periodic passes no longer scale with the full history of committed runs. **Breaking changes** None. A late repair on a settled row can be delayed until its turn in the upkeep rotation instead of the next pass. ## What Changed - `server/src/services/native-runtime/native-finalization-reconciler.ts`: - `settledCommittedFinalization(now)`: the settled predicate above, including a `NOT EXISTS` on undelivered `status_decision_effects` for the same company, issue and decision. - The unscoped candidate scan adds `NOT settled OR run in the upkeep page`. - `nextSettledCommittedUpkeep`: up to `NATIVE_SETTLED_COMMITTED_UPKEEP_PER_PASS` (10) settled run IDs after an in-process cursor, ordered by run ID, wrapping at the end. - New constants `NATIVE_SETTLED_COMMITTED_GRACE_MS` (1 hour) and `NATIVE_SETTLED_COMMITTED_UPKEEP_PER_PASS`, plus `resetSettledCommittedUpkeepForTests`. - New `server/src/__tests__/native-finalization-settled-candidates.test.ts`. - `doc/execution-semantics.md`: describes the settled rule and the upkeep rotation. ## Verification - `pnpm exec vitest run src/__tests__/native-finalization-settled-candidates.test.ts` (in `server/`): 4 passed. The test mocks the timing hook to capture the candidate count and stop the pass after the scan: - 25 settled rows give 10, 10, 5, then 10 again over four passes (rotation and wrap). - A row finished 10 minutes ago, a running row, and a terminal row without `finished_at` are all candidates. - A settled row whose decision has a `failed` effect is a candidate. - A scoped pass with an explicit run ID still selects a settled row. - `pnpm exec vitest run` for `native-finalization-recovery`, `legacy-finalization-regression`, `native-run-finalizer-telemetry`, `native-workspace-finalizer`, `native-finalization-reconciler-timing`, `external-chat-wait.integration` and `native-status-arbiter-corpus`: 316 passed. - `heartbeat/recovery.test.ts`: the 8 tests that do not need a database passed. The 20 database tests were skipped locally because embedded PostgreSQL could not start (the host's shared-memory segments were exhausted by other processes). CI runs them. - `pnpm typecheck` in `server/`: passed. ## Risks - A settled row that needs a late repair (for example a missing chat presentation) waits for its turn in the upkeep rotation, up to `ceil(settled rows / 10)` passes, instead of the next pass. - The upkeep cursor is in memory. After a restart the rotation starts from the beginning; the work is idempotent. - A run left `running` with a committed finalization is never considered settled, so it is still projected before the reaper's active-run loop. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code with tool use (shell, file edits, test runs, database and log analysis). A separate Claude agent mapped the recovery call tree. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>canary/v2026.1016.0-canary.0 |
||
|
Nicky LeachandPaperclip
|
c3ffa3f0dd |
test(adapter-utils): fix feedback request index in bridge round trip (#16507)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Sandbox agents use the callback bridge to call the Paperclip API. > - The bridge round-trip test checks request delivery and credential removal. > - #12785 added an accepted API-key request before the feedback request. > - The feedback assertions still read request 2, which is now the GitHub credentials request. > - This PR checks request 3 and verifies that all four accepted requests reached the host. > - This restores the shared test gate while retaining the Unicode payload and token checks. ## Linked Issues or Issue Description Related PR: #12785 introduced the stale feedback index. The reported failure appears on #16486 and master. Searches for `workspaces-b`, `sandbox-callback-bridge`, `surrogate`, and `EncoderFallbackException` found no duplicate fix. **What happened?** The `General tests (workspaces-b)` job fails on PRs and master. The hosted PR job and AWS master job both report `System.Text.EncoderFallbackException` for an unpaired `\uD83D`. The runner exits during the test step and does not upload that step's log. The unchanged master test reproduces a deterministic assertion failure at the feedback request. It compares `/runtime-tools/github/credentials` with `/api/companies/company-1/agent-commentary` and prints a diff containing 5,000 crying emoji. The assertion output is the likely trigger for the runner's Unicode logging crash. - [Reported PR job](https://github.com/paperclipai/paperclip/actions/runs/38818474563/job/116719880033) - [Master job with the same runner exception](https://github.com/paperclipai/paperclip/actions/runs/38821128038/job/116717690951) **Expected behavior** The test checks the actual feedback request, verifies its full payload, and confirms that its authorization header is removed. Both shared CI lanes complete normally. **Steps to reproduce** 1. Check out master at `c4a16cbe981f96e2756c7014103c88da67e5514f`. 2. Install dependencies with pnpm 9.15.4. 3. Run `node_modules/.bin/vitest run packages/adapter-utils/src/sandbox-callback-bridge.test.ts -t 'round-trips localhost bridge requests'`. 4. Observe the feedback assertion reading the preceding GitHub credentials request. **Paperclip version or commit** Master at `c4a16cbe981f96e2756c7014103c88da67e5514f`. **Deployment mode** Built from source. GitHub-hosted PR CI and AWS post-merge CI. ## What Changed - Assert that four accepted requests reached the host. - Check feedback delivery and authorization stripping on request 3. - Retain the large Unicode payload and all existing authentication checks. ## Verification - Unchanged master: the focused round-trip test fails at the stale feedback index. - Fixed branch: all 68 tests in `sandbox-callback-bridge.test.ts` pass with real localhost gateways. - `pnpm --filter @paperclipai/adapter-utils typecheck` passes. - `git diff --check` passes. - Exact CI group: `node scripts/run-vitest-stable.mjs --mode general --group general-workspaces-b` passes (4,800 tests passed, 28 skipped across 17 projects). - `pnpm --filter @paperclipai/adapter-utils build` passes. - Greptile reviewed current head `3de15d3374354809ea84ca4df5f93c2473715f02`: 5/5, success, no comments or actionable findings. - All GitHub checks pass on current head `3de15d3374354809ea84ca4df5f93c2473715f02`: 55 successful checks and 2 intentional skips, including `workspaces-b`, build, typecheck, Runner verification, and e2e. - Full local `pnpm test:run` was started and stopped after the CI test lanes passed; it did not complete locally. No worktree-owned test processes remained afterward. - Full local typecheck and build were attempted. Both stop at the Runner Rust commands because `cargo` is absent from the shell PATH. The changed package typecheck passes. The complete repository typecheck and build pass in CI. - The first sandboxed local reproduction could not bind localhost (`EPERM`). It was repeated with listener permission and reached the deterministic assertion failure described above. ## Risks Low risk. This changes three test lines. Runtime behavior, credentials, workflow settings, and payload coverage stay the same. The GitHub runner Unicode logging defect remains upstream; this fix removes the incorrect assertion that triggers it in this test. ## Model Used OpenAI GPT-6 in Codex, with reasoning, repository inspection, and command execution. The exact runtime model identifier and context window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>canary/v2026.1015.0-canary.24 |
||
|
|
8aaef59339 |
fix(pi-local): add stdout compaction to stop quadratic run-log growth (#10249)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The `pi_local` adapter runs the `pi` CLI in `--mode json` and
records its NDJSON stream as a run-log per turn
> - In JSON mode `pi` emits one `message_update` event per streaming
token, each carrying the **full cumulative assistant message**, so a
single turn's run-log grows quadratically with output length (measured
~77 MB for one thinking-model turn)
> - The root cause is in `pi`, but the Pi project declined to add a
compact mode (earendil-works/pi#3905, closed the same day), so Paperclip
has to filter on the consumer side
> - This PR adds a dedicated, config-gated compactor at the `pi_local`
stdout boundary (`stdoutLogMode`: `raw` | `compact`, default `compact`)
that strips the cumulative snapshots while preserving streaming deltas,
terminal events, and parser/UI output
> - The benefit is linear run-log growth (~150× smaller in practice)
with **no change** to `parsePiJsonl` results, token usage, or live
typing — keeping storage and replay bounded
## Linked Issues or Issue Description
- **Refs #5699** — same root cause ("full accumulated partial state" →
46 MB log, per that PR). This PR takes a **different, single-concern
approach**: a dedicated `stdout-compaction.ts` module with an explicit
`stdoutLogMode` config (default `compact`, fail-safe fallback to `raw`),
scoped to `pi_local` only. #5699 instead filters inline in
`buffered-on-log.ts`, bundles an unrelated `claude-local` change, and is
currently **conflicting** + stale (last updated 2026-05-14). Happy to
coordinate with @ukint-vs on a merged direction.
- **Refs earendil-works/pi#3905** — proposed adding a compact JSON mode
upstream in `pi`; closed by the maintainer, which is why this filters on
the Paperclip side rather than the producer.
- **Related #1846** (run-log/retention TTL) — complementary: this PR
reduces what's stored per run; #1846 addresses how long rows live.
- No public GitHub issue exists that is an exact duplicate of the
`pi_local` quadratic-growth symptom; the problem is tracked via #5699
above. (No internal/instance-local references used.)
## What Changed
- **`packages/adapters/pi-local/src/server/stdout-compaction.ts`** (new,
123 lines) — `createPiStdoutLogCompactor(mode)`. `raw` passes lines
through unchanged; `compact` strips the cumulative
`message`/`assistantMessageEvent.partial` from `message_update` events
while preserving streaming deltas, terminal events, and passthrough of
unknown event types. Default mode `compact`; an invalid explicit mode
fails safely to `raw`.
- **`packages/adapters/pi-local/src/server/execute.ts`** — read the mode
via `resolvePiStdoutLogMode(config)` and apply the compactor at the
stdout → `onLog` (run-log) boundary.
- **`packages/adapters/pi-local/src/index.ts`** — expose the
`stdoutLogMode` config option (documented inline: `string`, optional,
default `"compact"`).
- **`packages/adapters/pi-local/src/ui/parse-stdout.ts`** — confirm the
stdout parser consumes compacted output unchanged.
- **Tests** — new `stdout-compaction.test.ts` (359 lines) covering
modes, edge cases, and `parsePiJsonl` replay equivalence; updated
`execute.remote.test.ts`.
## Verification
- **Unit:** `pnpm --filter @paperclipai/adapter-pi-local exec vitest
run` → **121/121 passing (12 files)**.
- **Typecheck:** `pnpm --filter @paperclipai/adapter-pi-local exec tsc
--noEmit` → **clean, 0 errors**.
- **Real-data replay:** a real `pi --mode json` stream (`zai/glm-5.2`,
thinking=high) compacted line-by-line through this compactor:
- Raw **76.96 MB** → Compact **525.5 KB** = **99.33% reduction
(≈150×)**.
- 4503 lines in, 4503 out (**0 dropped**); **0** `message_update` with a
cumulative `.message` remaining; **4489** streaming deltas preserved.
- **Parser equivalence:** `parsePiJsonl(raw) ≡ parsePiJsonl(compact)` —
**0 errors** on both; identical `finalMessage` (13,369 chars); identical
`usage` (`inputTokens 2896, outputTokens 4491, cachedInputTokens 6848`).
Result, token accounting, and live-typing are unchanged.
## Risks
- **Behavioral shift:** default `compact` changes what is stored in
run-logs vs today. Operators who need the verbatim stream can set
`stdoutLogMode: "raw"`. Flagging for reviewer preference — happy to flip
the default to `raw` for upstream if that's preferred.
- **Ordering with secret redaction (#8027 / #9856):** compaction is
structural and runs at the stdout boundary **before** redaction; it only
removes cumulative duplicates and never alters token content, so it does
not change what redaction sees.
- **Unknown event types** pass through unchanged (forward-compatible
with future `pi` events).
- Low risk otherwise — the change is contained to the `pi_local`
adapter.
## Model Used
OpenAI `gpt-5.6-luna` — via pi's `openai-codex` provider (272K context
window). Used for implementation and for the real-data measurement in
Verification.
Reasoning/thinking: extended thinking enabled at level `high` (pi
`--thinking high`; maps to OpenAI reasoning effort). `gpt-5.6-luna` is a
reasoning-capable model.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: commitperclip[bot] <284812584+commitperclip[bot]@users.noreply.github.com>
|
||
|
Huanyi Xie
|
c4a16cbe98 |
fix(plugins): follow local package path changes in dev watcher (#15247)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Local plugin development uses file changes to restart the plugin worker. > - A plugin can move to a different local package directory. > - The watcher rejected every later registration for the same plugin ID. > - This pull request replaces the watcher when the resolved package path changes. > - The benefit is that current plugin files trigger reloads and previous files stop doing so. ## Linked Issues or Issue Description Refs: #14955. This change addresses its second observation: the watcher keeps the previous local directory after its package path changes. The issue also tracks stored configuration replay as a separate problem. Related PRs: #11336 shares the lifecycle manager across routes and the dispatcher. #14381 controls when production file watching is enabled. #10100 orders configuration delivery. Those changes address other parts of plugin reload behavior. ## What Changed - Store the resolved package path with each watcher. Keep repeated registrations of the same path as a no-op. Replace the watcher for a different path. - Cancel pending changes from the previous directory. Ignore late change, error, and timer callbacks from a replaced watcher. - Apply only the latest pending directory lookup. Manual registration, unwatching, and shutdown invalidate earlier lookups. - Stop a failed watcher without cancelling its pending replacement lookup. Explicit unwatching, disable, unload, and shutdown still cancel pending registrations. - Stop the old watcher when the new directory is unavailable or the plugin no longer has a local package path. - Document the path-change behavior and add twelve regression cases. ## Verification - `pnpm exec vitest run server/src/__tests__/plugin-dev-watcher.test.ts server/src/__tests__/plugin-lifecycle-restart.test.ts`: 18 tests passed. Regressions cover path changes, stale callbacks, lookup order, explicit cancellation, and watcher errors during a pending replacement. The current-watcher error case fails before the stop-versus-cancel fix; the stale-watcher error control passes. - A local smoke check used real Chokidar watchers and two real temporary plugin directories. It confirmed that a pending old change is cancelled, later old-directory changes are ignored, and a new-directory change triggers one restart. - `pnpm -r typecheck` passed on the path and lookup guards. The final watcher-error fix passed `pnpm --filter @paperclipai/server exec tsc --noEmit`. - `pnpm build` passed on the initial path-change fix. The final source passed `pnpm --filter @paperclipai/server build`. - `git diff --check`: passed. - Greptile reviewed `bef3182` at 5/5 with no new findings. The watcher-error review thread is resolved. CI is still running. - The full `pnpm test:run` suite was not repeated for this watcher-only change. The affected suites ran locally. The broad test shards will run in CI. ## Risks - Low risk. The watcher follows the current registered path. An unavailable replacement stops watching until another valid registration arrives. - Closing a filesystem watcher is asynchronous. Callback identity checks prevent an old watcher from acting on its replacement. - No database migration, API shape, or dependency change is required. ## Model Used - OpenAI Codex based on GPT-6, with reasoning, tool use, and code execution. The runtime does not expose the exact serving model ID or context window. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge |