Jannik/PaperClipAI
1
0 Fork 0
mirror of https://github.com/paperclipai/paperclip.git synced 2026-10-21 08:22:20 +02:00
Code Issues Packages Projects Releases Wiki Activity
5,346 Commits 1,227 Branches 1,999 Tags
e6dcc6b0959bdfc3bc93239c3d95e14c9fc8d457
Commit Graph
1118 Commits
Author SHA1 Message Date
Devin FoleyandPaperclip 63fbb714e6 fix(server): release idle tracking after aborted static UI files (#16685) 
## Thinking Path

> - Paperclip runs work for AI agents.
> - Hosts must prove that accepted work has finished before idle sleep.
> - HTTP request tracking keeps unfinished writes active after a client
leaves.
> - Static file middleware has no returned completion promise.
> - An aborted UI download can leave its request active forever.
> - This PR completes only a matched file response and keeps other work
tracked.

## Linked Issues or Issue Description

Refs #15522. Found while qualifying #16679. A search found no duplicate
static-download idle fix.

**What happened?**

Closing a client during a static UI download leaves one tracked request
active. The response has closed, but the file middleware never calls the
wrapped response end method. The instance cannot report idle after that
request.

**Expected behavior**

A closed, matched static file response must not leave a permanent work
hold. Unfinished application writes must remain tracked. A missing file
must keep the normal fallback and error paths.

**Steps to reproduce**

1. Serve a large local file through the static UI middleware with idle
admission enabled.
2. Start the download over HTTP.
3. Disconnect the client after its first data chunk.
4. Observe that the tracked request remains active with the original
middleware.

**Paperclip version or commit**

Reproduced on the behavior in
`d30648dee89720358271c276b9fc28cf83f18be7`. This branch starts at
`cfa9dce295970a85b3d8ee14cab0eccf50bc72e2`.

## What Changed

- Mark the completion of a known read-only response without dropping
pending middleware promises.
- Wrap static UI file serving. A successful file match is required
before a closed response can complete its tracking. A late match
rechecks an already closed response. Missing files keep the normal
fallback.
- Use the wrapper for hashed assets, other static UI files, and public
development files. Preserve caller headers and cache options.
- Test both HTTP abort timings, nested routers, conservative
mounted-application holds, parent writes, mutating GET fallbacks, and
callback-only work.
- Update the idle contract and its verification map. Correct the
paused-message recipe from hosted qualification: an explicit Pause
cancels older queued work, so test new accepted input after the pause
and await the normal resume lifecycle.

## Verification

- The original two real stream-abort regressions fail with the original
Express static behavior.
- Three review regressions fail on
`e260f4694afab8447b0cb203f439cb3ccad254e8`: a client closing before file
lookup, an unfinished parent in a mounted application, and an unfinished
parent in a root without installed handler tracking. Fixed sources were
restored byte for byte after the comparison.
- Two focused suites: 16 passed. They cover nested routers, both abort
timings, hidden parent writes, late missing-file fallbacks, contents,
HEAD, conditional responses, and caller headers.
- Module and map checks pass. Map checks are structural only.
- `pnpm -r typecheck` and `pnpm build`: passed on
`f242ee739344484995d3addf81635512370778c8`.
- Full CI matrix: 54 successful checks and 2 intentional Storybook
skips.
- Apex: 5/5 on the exact current head, with zero unresolved threads.
- Full local maintained test groups are still running before final
handoff.
- The initial local broad run was stopped before changing source for the
P1 review findings. Two file-mode fixture failures in that run came from
the scratch runner's inherited `0077` mask; the same suite passes with
the normal child test mask `0022`, without source or host configuration
changes. The replacement run uses `0022` for test children and
command-scoped `init.defaultBranch=master` for existing Git fixture
assumptions.
- This PR has not been deployed to a hosted environment.

## Risks

Only a matched file served by the admitted application with installed
handler tracking can complete this way. Routers share that scope.
Mounted Express applications and roots without handler tracking keep the
existing conservative response hold; this change does not claim their
hidden writes are tracked. An unfinished tracked parent promise still
blocks sleep after a file client disconnects. Redirects, missing files,
application callbacks, and other GET requests retain their guards. No
authentication, company scope, provider cleanup, or scheduling gate is
removed. No schema or API response changes.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, source editing, and shell
tools. The session does not expose a more specific serving model ID or
its context window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-17 20:28:57 -07:00
DottaandPaperclip 1f27427ffd perf(server): batch retained workspace reader identities and policies (#16683) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Task startup checks who can read each physical workspace.
> - A shared root retains the identities of earlier runs.
> - The service reloads the same agents, tasks, projects and run
policies for each reader.
> - Database round trips grow with the retained run history and hold the
company provenance lock.
> - This pull request batches these reads within each guarded operation.
Each run keeps its own identity and policy.
> - The benefit is less task preparation time while current access
checks remain in place.

## Linked Issues or Issue Description

Refs #16424.

**What existing behavior does this improve?**

Workspace admission, repository publication and batched read
authorization.

**Current behavior**

A synthetic shared workspace with 64 retained readers makes 680 SQL
calls during a publication check. The same principals are loaded
repeatedly.

**Proposed behavior**

Load retained identities and saved policies in batches of at most 500
IDs. Load project settings once per guarded operation. Reload records
and permission grants on the next operation.

**Reason and benefit**

The local service benchmark with an injected 4 ms loopback round trip
changed from about 7.0 seconds to 0.5 seconds and from 680 SQL calls to
47. This measures synthetic PostgreSQL data, not hosted response
latency.

## What Changed

- Extract retained-reader identity loading and acquisition trust checks
into a focused service module. Select only the identity and trust fields
used by these checks.
- Batch company-scoped run, agent and task lookups. Keep each run's task
binding, responsible user and policy checks.
- Batch saved run-policy reads in read authorization. Keep company and
agent identity checks and ordinary lookup behavior for noncanonical IDs.
- Reuse project settings within one root authorization operation.
- Add query budgets, identity failures, batch-boundary coverage, policy
comparisons and grant-revocation tests.
- Document snapshot lifetime and query scope.

## Verification

- PASS: 197 focused workspace and authorization tests.
- PASS: `pnpm -r typecheck`.
- PASS: `pnpm build`.
- PASS: all current-head hosted CI test groups, Runner checks, E2E
shards, security checks and builds (55 successful checks; two unrelated
Storybook jobs skipped). The local monolithic `pnpm test:run` was
stopped after all equivalent hosted CI groups passed. It has no complete
local result; the hosted matrix supplies the full-suite proof.
- Synthetic real-PostgreSQL benchmark: 383 workspaces and 64 retained
readers; 4 ms simulated loopback round trip. Publication check: 680 ->
47 SQL calls; 7.0-7.1 s -> 0.48-0.51 s. Three concurrent checks:
20.0-23.1 s -> 1.52-2.12 s. Setup and provider execution are excluded.
Live reply timing remains unverified.
- Regression coverage compares complete batched and ordinary
authorization decisions. It checks distinct saved policies for the same
principals, missing identities, company mismatch, task mismatch,
responsible-user mismatch, revoked project grants and changes after the
first operation.
- No new schema, API or UI contract.

## Risks

- A batch must not combine different runs' identities or policies. Each
lookup and policy decision retains its run, company and agent checks.
- Snapshots last for one guarded operation. Later admissions and
publications reload data. No authorization cache survives the operation.
- Restricted project trust checks keep their existing per-run validation
path. Hosted latency remains an acceptance gate.

## Model Used

OpenAI `gpt-6.1-sol` through Codex, with xhigh reasoning, code execution
and tool use. The runtime does not expose the configured context-window
size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-17 21:52:08 -05:00
Dotta cfa9dce295 refactor(runner): remove unreferenced Paperclip Runner test fixtures (#16543) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The Paperclip Runner package keeps fake agent servers and versioned
profile identity declarations under `test/fixtures/` for its test
suites.
> - Five of those fixture files have no reference anywhere in the
repository: no test, script, generator, workflow step, or document loads
or spawns them.
> - Dead fixtures add maintenance cost and make the referenced sibling
fixtures harder to trust.
> - This pull request deletes only those five unreferenced fixture files
and adds an audit receipt with the reference evidence.
> - The benefit is a smaller test-fixture surface with zero loss of
executed coverage.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

The test-suite hygiene of the Paperclip Runner package. The repository
keeps unused test-support files that no suite loads. This slows audits
and makes the referenced fixtures harder to trust.

**Subsystem affected**

`packages/paperclip-runner` test fixtures.

**Current behavior**

Five files under `packages/paperclip-runner/test/fixtures/` have zero
references anywhere in the repository:

- `fake-native-resume-codex-app-server.mjs` (277 LOC): an old fake codex
app-server for resume testing. Resume coverage now runs through
`fake-final-burst-codex-app-server.mjs` (used by
`server/src/services/native-runtime/native-session-resume.test.ts:287`)
and `fake-durable-codex-app-server.mjs` (used by
`packages/paperclip-runner/src/live/live-session.test.ts:2192`).
- `copilot-profile-v{3,4,5,6}-identity.json` (64 LOC total): orphaned
profile identity declarations. Copilot profile tests,
`native-backend-factory.test.ts`, and
`scripts/generate-acpx-profiles.mjs` reference only v7 through v19 by
literal path. No dynamic `copilot-profile-v${...}` path exists in the
repository (the only template-literal fixture path is
`test-fixtures/pi-acp/profile-v${version}-identity.json`).

**Proposed behavior**

Delete only the five unreferenced fixture files. Keep every referenced
fixture. Keep all registered test cases unchanged.

**Reason and benefit**

Smaller test-fixture surface: 341 test/support lines removed with zero
loss of executed coverage. The audit receipt
`doc/plans/2026-10-16-runner-dead-fixture-test-audit.md` records the
per-file reference evidence.

**Breaking changes**

None. No production code, no registered test case, and no test selection
path changes.

**Additional context**

A repository-wide `git grep` for each removed filename stem returns zero
external references across all tracked file types. The
`hashFiles('packages/paperclip-runner/test/fixtures/**')` occurrences in
the workflows are Rust cache-key inputs, not execution selectors;
deleting files only changes that cache hash.

## What Changed

- Deleted
`packages/paperclip-runner/test/fixtures/fake-native-resume-codex-app-server.mjs`.
- Deleted
`packages/paperclip-runner/test/fixtures/copilot-profile-v{3,4,5,6}-identity.json`.
- Added the audit receipt
`doc/plans/2026-10-16-runner-dead-fixture-test-audit.md` with per-file
reference evidence.
- 341 test/support lines removed. Zero production, documentation, or
registered test-case changes.

## Verification

- Repo-wide `git grep -i` for each removed filename stem
(`fake-native-resume`, `native-resume-codex`, `copilot-profile-v3` …
`copilot-profile-v6`) across all tracked file types returns zero
external references.
- Selection audit: `packages/paperclip-runner/vitest.config.ts` includes
only `src/**`, `devtools/**`, `examples/**`, and two named
`scripts/*.test.mjs`; the server project selects `server/src/**` and
`server/scripts/**`; the `test:typescript:prep` `node --test` list names
no fixture. No workflow executes anything under `test/fixtures/` — the
`pr-trusted.yml`/`release-verify.yml`
`hashFiles('packages/paperclip-runner/test/fixtures/**')` occurrences
are Rust cache-key inputs, not execution selectors.
- Rust tests use their own `fake-codex-app-server` binary in
`runner/crates/runner-core`, not these `.mjs` fixtures.
- Searched open and merged PRs for each filename: zero overlap (`gh
search prs`).
- Repo-wide `pnpm -r typecheck`, `pnpm test:run`, and `pnpm build` were
not run this session: the removed files are unreferenced, absent from
every vitest/node-test include pattern, and excluded from all tsconfigs,
so no compiled or executed path changes. No runtime CI savings are
claimed.
- All Paperclip CI gates green on head `ce24ab3` (verified 2026-10-16
~09:00Z): every `ci /` shard (General tests, e2e shards, serialized
server suites, Verify Paperclip Runner rust/vitest/static, Build,
Typecheck + Release Registry, policy, Docker context integrity, Canary
Dry Run, gate), `review`, `Superagent Security Scan`, `Socket Security`
PR alerts + project report, and `Contributor trust` all pass; the only
skipped jobs are the Storybook deploy/visual-regression jobs, which are
not required for this change. Greptile Review: 5/5 confidence, "Low
impact", no actionable issues, last reviewed commit `ce24ab3`.

## Risks

Low risk. No executed coverage or selection path changes. The only side
effect is a one-time change to the `hashFiles` Rust cache key that lists
`test/fixtures/**`, which simply invalidates that cache on the next CI
run.

## Model Used

Zhipu `z-ai/glm-5.3` (GLM), served through OpenRouter, running as the
Paperclip Test Gardener agent. Context window size unavailable to the
agent. Capabilities used: shell tooling, git, GitHub CLI search, and
file inspection; no image or code execution capabilities were used.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` /
`Closes: #` / `Refs: #` OR (b) described the issue in-PR following the
relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
 2026-10-17 21:18:11 -05:00
Devin FoleyandPaperclip d30648dee8 Preserve accepted messages for paused recipients (#16679) 
Keep accepted deferred requests pending while their recipient is paused and deliver through normal gates after resume. Preserve company, task hold, and cleanup checks.

Verification: 247 focused checks, recursive typecheck, the full CI test and build matrix, and Apex 5/5. The local Git fixture initial-branch limitation is documented in the PR.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-17 18:38:17 -07:00
Devin FoleyandPaperclip 391dd15ff2 fix: recover deferred messages and completed unblock waits (#16673) 
Recover accepted deferred messages through the latest finished issue run and
recognize exact completed agent unblock notifications as passive idle waits.

Preserve assignment handoffs, pause and Stop, failed outcomes, and cleanup
ownership. Honor local and SSH release contracts and atomically settle cleanup
claims. Add PostgreSQL recovery, concurrency, and interruption regressions.

Validation: recursive typecheck and build pass; 678 focused tests pass; full CI
passes; Apex review is 5/5 with no unresolved findings.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-17 16:53:29 -07:00
Devin FoleyandPaperclip 27ab902d3d perf(docker): start the server from a precompiled runtime bundle (#16672) 
The production image started the server through the tsx loader, which resolved about 3,100 modules and transpiled about 1,200 workspace TypeScript files on every boot (7.6-20 s before listen on a live instance). A build:runtime step now bundles server/dist/index.js with esbuild, compiling in the workspace packages, and the image starts node server/dist-runtime/index.mjs. Server dependencies stay bare so runtime resolution (including the cloud image's optional peers) is unchanged; each module keeps its original import.meta location; dynamic imports stay split; the entry presents the original dist/index.js as argv[1]. The avatar worker loads through tsx's API when the main thread has no loader, tini runs as a subreaper (-s), and PR CI boots the bundle with scripts/server-boot-smoke.sh.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-17 16:22:35 -07:00
Devin FoleydevinfoleyClaude Opus 5.5
 		938a441944 fix(acpx): end Claude turns whose pending tool call stops receiving updates (#16671) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Hosted `claude_local` runs use the ACPX engine. Paperclip sends
runtime MCP servers to Claude, and Claude can start subagents (the Task
tool) that call those MCP tools.
> - #16360 set `MCP_TOOL_TIMEOUT` for Claude. That bound works: in a
local test with the same Claude binary, a direct MCP call and a subagent
MCP call to a server that never answers both ended at the bound.
> - A hosted run still hit the 4h sandbox timeout on a release that has
#16360. Its run log shows a subagent that started four MCP calls. ACP
reported the four calls as `pending`, and they never got another update.
About 40 other MCP calls in the same minutes finished in 3–25 seconds.
So the four calls most likely never reached the MCP server, and
`MCP_TOOL_TIMEOUT` did not apply to them.
> - The subagent's Task call got an `in_progress` heartbeat every 30
seconds for the full 4 hours. The turn looked active, so no existing
bound stopped it.
> - This pull request adds a stalled-tool watchdog to the ACPX engine.
It ends the turn when a pending tool call gets no update for 30 minutes
and no other tool call finishes in that time.
> - The benefit is that a stalled run fails in about 30 minutes with a
clear error code, not after 4 hours with a general timeout.

## Linked Issues or Issue Description

- Refs #16360 (MCP client deadline; this PR covers the stall that the
deadline cannot reach)
- Refs #15962 (stop-time diagnostics that showed the pending tools and
their age)

**What happened?**

A hosted `claude_local` run gave lookups to two subagents. One subagent
started four MCP tool calls. ACP reported each call as `pending`. No
call got a result, an update, or an error. The other subagent finished
in 3 minutes. The stalled subagent's Task call got a heartbeat every 30
seconds, so the last-event age at stop was only 28 seconds. The oldest
pending tool was 3h58m old. The sandbox execution timeout ended the run
after 4 hours.

**Expected behavior**

A tool call that never gets a result ends the turn after a bounded time.
The run fails with an error that names the stall.

**Steps to reproduce**

1. Start an ACP `claude_local` run that uses a subagent to call runtime
MCP tools.
2. Make one of the subagent's tool calls stay `pending` with no further
ACP update (in production this happened without a known trigger).
3. See that the run continues until the adapter execution timeout,
because the Task heartbeats keep the turn active.

**Paperclip version or commit**

`master` at 4415665.

## What Changed

- Add `ACPX_STALLED_TOOL_TIMEOUT_MS` (30 minutes),
`ACPX_STALLED_TOOL_MARGIN_MS` (15 minutes), the check interval, and the
`acpx_tool_stalled` error code in `acpx-engine/constants.ts`.
- Add `resolveStalledToolTimeoutMs()`. Claude turns use 30 minutes, or
`MCP_TOOL_TIMEOUT` / `BASH_MAX_TIMEOUT_MS` plus 15 minutes if that is
longer. Other agents are not watched. The adapter config field
`stalledToolTimeoutSec` overrides the bound for any agent, and 0
disables it. The bound is not part of the session fingerprint.
- The engine records the time of the last update for each tool call, and
the time a tool call last reached a final status.
- During a turn, the engine checks the pending tool calls every 60
seconds. If a pending call has had no update for the bound, and no tool
call reached a final status in that time, the engine:
- records the stop-time activity (the same fields as a timeout or an
operator stop),
- writes a `[paperclip]` stderr line with the closed tool kind and the
silent time,
- cancels the turn, and starts the existing fail-fast cancel deadline
that the duplex-loss path uses. The deadline helper is now shared by
both paths.
- Finalization reports the stall as a failure: `errorCode:
"acpx_tool_stalled"`, `resultJson.status: "failed"`,
`acpStalledToolKind` (one closed ACP `ToolKind` label or `unknown`), and
`acpStalledToolSilentMs`. A timeout or a duplex loss still takes
priority. A turn that completed anyway keeps its success.
- The server Sentry diagnostics allowlist the two new fields. It drops
the kind if it is not one closed label.
- Document the watchdog in `doc/observability.md` and the
`stalledToolTimeoutSec` field in the `claude_local` config notes.

## Verification

- `npx vitest run src/acpx-engine` in `packages/adapter-utils`: 21
files, 422 tests pass.
- New engine tests:
- A silent pending call next to a Task call that keeps getting
heartbeats ends the turn with `acpx_tool_stalled`, the closed kind, and
the silent time. No tool title or ID appears in the diagnostics.
- An agent that ignores the cancel: the fail-fast deadline closes the
stream and the run fails with `acpx_tool_stalled`.
  - A long tool that keeps getting updates is not cancelled.
- A queued call that waits behind slow calls that keep finishing is not
cancelled.
- `stalledToolTimeoutSec: 0` disables the watchdog, and only the
wall-clock timeout ends the turn.
- `resolveStalledToolTimeoutMs()` defaults, env bounds, other agents,
and config override.
- Mutation checks: with the watchdog disabled, the two stall tests time
out (the same hang as production). Without the "no tool finished" guard,
the queued-call test fails.
- `npx vitest run
src/services/__tests__/run-failure-diagnostics.test.ts` in `server`: 115
tests pass.
- `npx tsc --noEmit -p packages/adapter-utils`: passes.

## Risks

- A legitimate tool call that gets no ACP update for 30 minutes while no
other tool call finishes now fails the run. Claude's own tool bounds
(MCP 15 minutes, Bash 10 minutes by default) end healthy calls before
that. If an operator raises `MCP_TOOL_TIMEOUT` or `BASH_MAX_TIMEOUT_MS`
in the env, the bound moves up with it. Operators can also set
`stalledToolTimeoutSec` or disable the watchdog with 0.
- A long sync subagent stays safe while its Task call gets heartbeats.
Only a call that is itself silent can trip the watchdog.
- The run fails instead of continuing. The failure uses the normal
failed-run recovery, and it reports to Sentry under its own error code,
not under the generic timeout.
- This PR does not find why the four calls never got a result. Claude
runs with `--permission-mode default`, so each MCP call makes an ACP
permission round trip, and the ACP stream goes through the remote
process-session relay. A lost request or response on that path is the
most likely cause. The new diagnostics show how often the stall happens
and for which tool kind.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with tool use
and code execution. It read the run log and the ACP agent and client
code, edited the code, and ran the tests.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-17 14:51:49 -07:00
Devin FoleyandPaperclip 4415665d67 fix(server): allow proved idle waits and verify historical cleanup (#16633) 
## Thinking Path

> - Paperclip manages agents, task continuations, integrations, and
their execution resources.
> - Automatic idle sleep needs durable evidence that accepted work and
provider cleanup are settled.
> - Response-driven questions, passive board escalations, and completed
connection receipts currently look like unfinished work.
> - Legacy ephemeral leases and terminal image sessions need actual
deletion proof before their history can become idle.
> - This PR adds narrow evidence-based exceptions and read-only provider
verification while preserving queues, outcomes, and ownership fences.
> - Idle instances can sleep without losing responses, automatic
reconciliation, or cleanup obligations.

## Linked Issues or Issue Description

**What happened?**

Instances with no executable work can remain awake because pending human
responses, passive recovery records, outbound integration configuration,
or completed delivery history are counted as background work. Older
failed ephemeral leases and finished setup sessions also lack the
evidence needed to distinguish retained resources from deleted
allocations.

**Expected behavior**

Permit idle sleep only for proved response-driven waits and settled
history. Continue blocking real queued work, inbound integrations
without a wake contract, unfinished accounting/retries, retained
resources, and uncertain provider cleanup.

**Steps to reproduce**

Finish an issue's producing run, leave a pending question or
confirmation, and request an owned scheduled idle report. Separately
retain a processed connection receipt or a failed ephemeral lease whose
provider allocation has already been deleted. The old checks keep those
instances awake even after executable work has finished.

Related public groundwork: #15522, #15599, #15761, #15897, #15907, and
#16325. Searched existing public idle-sleep PRs; this addresses
additional blockers in the current predicates.

## What Changed

- Recognize supported response-driven waits and passive board recoveries
without changing task state, response audiences, or continuation
policies. Bind passive issue generation into routine-only alarms too.
Keep automatically polled merge-confirmation cards awake.
- Treat processed events, delivered intents, disabled drafts, and
outbound MCP configuration as history/configuration where proved safe.
Preserve pending deliveries, downstream wake requests, and inbound
GitHub bindings.
- Permit only pristine Slack setup drafts with a matching disabled
channel connection, no provisioning registration, and no prior provider
identity/activity. A saved slash-command name remains inert
configuration.
- Repair missed routine execution-issue status hooks in bounded
awake-only maintenance. Lock and re-read exact owners, apply existing
completion/transient-failure rules, and preserve normal continuation
after an issue resumes.
- Allow human-only connection setup to wait only when no reusable active
connection of its purpose exists; available access reblocks sleep for
reconciliation. The delivery worker shares this proof, releases its
local idle hold, and retains maintenance only while already awake.
Passive-only ticks pause before admission during an owned drain,
allowing slow backups to finish; new producer notifications still
reconcile and invalidate the backup.
- Add optional `environmentInspectLease` SDK discovery/dispatch and
read-only Daytona/fake-provider implementations. Old workers remain
conservative. Daytona status inspection uses a fresh provider GET
without reconnecting or poisoning unused-worker eligibility.
- Bind historical credential reads to an unrotated initial secret
version and its immutable resolved-value hash; reject ambiguous
histories and ambient Daytona credential/API-endpoint fallbacks before
querying provider state.
- Verify bounded rotating batches of historical resources on
already-awake Cloud instances, outside readiness. Store host-owned
deletion evidence bound to exact row identity/version and reject
concurrent ownership changes. Preserve failure outcomes, cleanup/export
intents, and run continuation authority.
- Preserve cancellation provenance, defer contended routine rows
individually, and cap actual inspection RPCs across caller timeouts and
worker exits. Add database, SDK, provider, configuration-scope, and race
regression coverage and update idle protocol documentation.

## Verification

- `pnpm -r typecheck` and `pnpm build` passed on `24fd1c0f5ec`; server
typecheck/build passed again after the final coordinator-only fix in
`60f23e6e4b6`.
- Final server regressions: 258 sleep-safety/connection cases and all 17
coordinator cases passed. Earlier focused checks also covered 331
combined idle/cleanup cases, 34 custom-image cases, 39
credential/cleanup cases, 39 connection-history cases, and 80 existing
routine cases. Counts overlap and are not additive.
- SDK: 153 passed. Daytona: 406 passed with 14 existing opt-in skips.
The provider implementation also typechecks against the locked SDK used
by the Cloud wrapper.
- Dependency boundaries were exercised with real child workers holding
provider inspection replies beyond caller timeouts, a real PostgreSQL
row lock delaying one routine owner, and real connection creation →
queue notification → worker hold release → resumption after access is
enabled without another notification. A controlled six-second backup
goes through the real coordinator, task drain, work signals, and
idle-safety generation checks; maintenance permits completion, while a
new committed transaction invalidates it.
- Full local `pnpm test:run` completed its general-server phase with
21,122 passing, 29 failing, and 109 skipped tests; later phases did not
run. That long run spanned source edits and is not a final-revision
pass. The affected Slack draft assertion passes on current source.
Restoring `/usr/sbin` for `lsof` produced 233 passing tests and two
Git-fixture failures across the other five files (3 existing skips).
Both remaining assertions pass with command-scoped
`init.defaultBranch=master`; this host defaults to main and leaves both
branches in their fixture. No source/global Git setting changed for that
comparison. Original failure logs are retained.
- A parallel typecheck/build attempt raced generated Runner outputs and
failed an import. Running the build sequentially passed with no source
change.
- Local database checks use the maintained test-constructor seam with
native PostgreSQL 16.14 because bundled PostgreSQL 18 is unreliable on
this host. No database assertions were skipped in the focused runs. An
interrupted combined run was superseded by the separate passing
coordinator and safety/connection runs; no owned test database processes
remain running.
- Apex reviewed final head `60f23e6e4b615a08591e19d86443a6c35c091ce1`:
**5/5**, zero unresolved threads. All current-head CI gates are green,
with expected policy skips. The branch is conflict-free and GitHub
reports a clean merge state.
- Current-head CI initially failed `plan-task-fixture.spec.ts:66` during
fixture deletion: the final company DELETE hit
`activity_log_company_id_companies_id_fk` after the feature assertions
completed. The deletion path is unchanged here; the final diff only
changes drain-gated passive maintenance. The preceding revision passed
this shard. Failure evidence is retained in [the original CI
job](https://github.com/paperclipai/paperclip/actions/runs/39006936642/job/117321244396);
the targeted retry passed on the unchanged commit ([retry
job](https://github.com/paperclipai/paperclip/actions/runs/39006936642/job/117322535013)).
This shows recovery, not that the underlying company-deletion race is
resolved.
- These checks do not claim staging/production deployment or live
provider cleanup proof. Provider tests simulate responses. Full Cloud
staging verification requires a compatible Core/private-plugin
composition after the Core revision is on master.

## Risks

Sleep eligibility changes require the existing owned drain, final
generation validation, local-work checks, and plugin drain. Unsupported
states fail closed. Queued agent input and routine execution remain
blocked. Outbound MCP exemptions exclude inbound GitHub grants; other
unsupported inbound integrations remain awake.

The inspection RPC is optional and requires a matching live worker
method. Its actual outstanding calls are capped at two across workers
even after the caller times out. Historical lease inspection requires
its recorded plugin/config; setup inspection requires an unchanged
recorded config fingerprint. Not-found under replacement account
credentials is not historical deletion proof: ambiguous/rotated secret
histories defer verification, and initial-version reads must match their
recorded value hashes. Missing provenance (including legacy image
sessions without a recorded config fingerprint), live/stopped resources,
provider authorization failures, and concurrent changes defer
verification. Routine repair replays existing
completion/transient-failure behavior and timestamps at reconciliation
time; it does not execute or discard deferred comments. No migration or
destructive provider operation is added. The new marker does not grant
run-resume authority. Full Cloud staging validation requires a
compatible Core/private-plugin composition; no production deployment is
included.

## Model Used

OpenAI GPT-6 Codex, with repository/tool access and code execution.
Exact model identifier and context-window size are not exposed by this
session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run relevant tests locally; passing focused checks and
full-run limitations are documented above
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-17 07:01:32 -07:00
Devin FoleyandPaperclip 5555817c2b fix(server): let Git expiration verify public commits without a run identity (#16621) 
Git expiration created its auth provider with only the issue ID; with a managed GitHub connection that provider throws outside a run ("No managed GitHub identity is available for this run"), so every upstream commit proof failed and was logged as inspection_failed. Expiration now falls back to anonymous access when the provider throws: public repositories verify, private ones stay kept. Upstream verification failures are reported as upstream_commit_unverified, and a managed checkout whose upstream cannot be reached or authenticated as managed_checkout_upstream_unavailable; local proof-store failures keep their own error.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-16 20:54:30 -07:00
DottaandPaperclip 669d329f25 perf(slack): avoid unused catalog revision scans (#16630) 
Slack catalog discovery now checks live task, user, endpoint and credentials without calculating an unused compatibility revision. Full authority callers retain the unchanged revision and search-token path.

Validation: 44 affected cases, full repository typecheck/build, current-head full CI/E2E and security checks, Greptile Base 5/5 with no unresolved findings. Identical production catalog output with revision statements reduced from two to zero in real-PostgreSQL comparisons. Hosted latency acceptance remains pending.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-16 22:21:03 -05:00
DottaandPaperclip 0528cc7ee0 perf: overlap native GitHub and assigned-tool setup (#16620) 
Prepare managed GitHub access and fresh assigned-MCP discovery concurrently for remote native Codex. Wait for both before provider construction and drain both on failure through the existing run cleanup.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-16 20:19:28 -05:00
Nicky LeachandPaperclip 5f5602bba7 feat(plugins): synchronize standing connection assignments (#16603) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Lifecycle plugins prepare agent runtimes outside the control plane.
> - They need committed connection assignment changes without polling.
> - A reader and its revision hooks must cover the same fields.
> - This pull request adds a bounded standing-assignment snapshot and
its matching hooks.
> - Plugins can react to assignment changes while execution keeps its
existing access checks.

## Linked Issues or Issue Description

**What happened?**

The agent runtime reader does not expose standing connection
assignments. Changes to connection installs, active profiles, and
connector feature switches do not request synchronization. Adding
company-wide invalidation also exposes lock inversions in assignment
writers and AI adoption.

**Expected behavior**

The owning lifecycle plugin can read a bounded assignment snapshot.
Every committed change to its stored fields advances the runtime
revision. Rollback removes both the change and its event. Assignment
writers finish without a lock-order cycle.

**Steps to reproduce**

1. Install a connection or bind an active profile to an agent.
2. Read its runtime inputs through the owning lifecycle plugin.
3. Rename or disable the connection, revoke the assignment, or switch
connector features.
4. Expect a committed runtime event and an updated snapshot without
periodic polling.

**Additional context**

Extracted from #16503. Catalog publication #16602 and the separate test
maintenance PRs #16601 and #16610 are merged. This PR now targets master
and is the active review. Tool-definition hooks #16604 and
access/credential hooks #16605 remain drafts without review requests.

## What Changed

- Add SDK types and an owning-plugin runtime snapshot for installed
connections, active profile IDs, and effective connector feature
switches.
- Reject more than 512 connections or 128 active profiles instead of
returning partial data.
- Add migration 0354 for the exact stored fields used by the snapshot,
with transaction-coalesced invalidation. Agent-targeted bindings and
installs notify only their old/new target agents;
project/routine/issue/gateway bindings are outside the snapshot and
produce no notification.
- Use ordered agent locks that are compatible with foreign-key
references and exclude runtime readers. Prelock complete old/new target
sets before replacing assignments or restoring managed profiles.
- Record one company baseline at delivery startup for process-managed
feature overlays.
- Lock assignment inputs before install saves, task completion, and
email assignment. Setup finalization reads the locked connection state
and prelocks company agents before activating, re-enabling, or reviving
a profile.
- Make profile binding and reassignment writes transactional and lock
parent profiles first. Keep target validation and profile/detail reads
on the reserved transaction connection, including with a one-connection
pool.
- Lock company agents in order before validated AI adoption. Keep
adoption and shared configuration writes at `FOR NO KEY UPDATE`,
avoiding an upgrade that can deadlock foreign-key inserts during gateway
creation. Configuration fields cannot change agent identity; lifecycle
transitions and deletion retain their existing locks.
- Document supported coverage and the inputs reserved for later PRs.

## Verification

- Thirteen affected suites passed: 757 tests covering runtime
hooks/assignments/skills, concurrent skill publication, tool
access/catalog, task connection intents, lifecycle delivery, email, AI
accounts, migration histories, snapshot drift, and the agent lifecycle
module. This includes the merged catalog PR's OAuth and
overlapping-refresh regressions.
- The assignment suite passes all 24 cases, including bounded snapshots,
company/agent isolation, rollback, health-field no-ops, process-start
feature overlays, and real lock contention.
- Four one-connection PostgreSQL regressions reproduce the earlier
helper-read hangs before the correction; bind, unbind, delete, and
reassign complete with it. The earlier adoption/gateway probe exposed
both lock upgrades: changing only validation still deadlocks in shared
configuration. Current scoped coverage verifies adoption with
agent-targeted binding writes; gateway-only bindings no longer
invalidate this snapshot.
- The original ordered-adoption regression reproduces PostgreSQL `40P01`
without ordered company-agent locking. Scoped-recipient tests reproduce
unrelated-peer journaling before the trigger correction and verify
repeated bind/unbind operations leave peers untouched, ignored binding
types produce no events, both reassignment recipients are reached, and
coalescing remains intact.
- Six replacement contention cases cover installs, existing setup,
activation, re-enabling, profile revival, and profile reassignment.
Negative controls reproduce `40P01` without the corresponding
target/company prelocks; all cases pass with the guards. Activation
decisions use the connection row reread under lock.
- An initial contention fixture failed because its settings connection
lacked an MCP URL. Supplying the fixture URL and controlled lookup
corrected that test setup error.
- Server typecheck, database migration numbering/safety, five-snapshot
retention, feature-map structural validation, and `git diff --check`
passed. Migration metadata preserves historical journal entries and adds
no schema change. The complete diff was reviewed for secrets and
personal data before push.
- Earlier hosted PR CI runs 38983440861 and 38985530069 passed; the
superseded promotion run 38983440470 was cancelled. Superagent
acknowledged the recipient fix and accepted the documented company-wide
cost at `359b5b2f30a8a0248177367e458663d5379810d1`. Its two standard
Greptile summaries were conditional on a later setup correction and were
not counted as independent acceptance; the setup correction is now
included here.
- At `7da246aab46274db1a120a0546abbb404c5427c7`, hosted PR run
38986958988 passed: all 55 checks succeeded, with two optional Storybook
checks skipped. Superagent passed, all review threads are resolved, and
standard Greptile independently returned 5/5 with no blocking findings
(review completed 2026-10-17 00:40 UTC). Apex independently returned 5/5
with no actionable findings (completed 2026-10-17 00:50 UTC). CI was
rechecked after Apex and remains green; the branch is mergeable against
master. Full local build and repository-wide tests were not run.

## Risks

Migration 0354 adds database functions and triggers without changing
table schema. Apply it through the normal migration flow. Agent-targeted
bindings and installs only notify their affected agents. Explicit
company-wide assignments, connection metadata, profile
activation/removal, and connector feature switches retain synchronous,
transaction-atomic company fanout. Each such transaction takes ordered
row locks and inserts up to one connector runtime event per eligible
company agent: O(number of eligible agents) work and journal storage,
with no company-size cap or cross-transaction mutation quota. This
remains an intentional scope tradeoff for this PR; deferred fanout is
not implemented. Many of these conservative changes can request
preparation even if an individual snapshot stays unchanged. Snapshot
limits fail closed. This PR provides desired assignments, not
credentials or task authorization. Later PRs must add mutation coverage
before consumers can rely on immediate synchronization of tool
definitions or credentials.

## Model Used

OpenAI Codex, GPT-6, with reasoning, repository inspection, code
execution, and testing. The exact deployment model ID and context window
are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-16 17:55:47 -07:00
Nicky LeachandPaperclip 190080642e perf(tools): publish catalog refreshes in bounded chunks (#16602) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - MCP discovery publishes tools that setup and access profiles use.
> - Large catalogs currently write one row at a time.
> - Repeated page entries can produce duplicate results, and profile
writes can partly commit.
> - This pull request publishes tools in bounded chunks and makes
profile synchronization atomic.
> - The result supports large catalogs before connector runtime hooks
are enabled.

## Linked Issues or Issue Description

**What happened?**

Catalog refresh performs one database write per tool. Repeated tool
names across discovery pages appear more than once in the result.
Default-profile synchronization uses separate statements, so failure can
leave a profile without its binding. OAuth callbacks also acquire
connection/application locks in the reverse order from publication and
setup, allowing a concurrent refresh to deadlock.

**Expected behavior**

Publish at most 500 tools per catalog statement. Keep one result per
tool name, using its last descriptor. Preserve existing IDs, first-seen
timestamps, disabled actions, and unchanged review decisions. Commit or
roll back default-profile changes together. OAuth persistence must
acquire application then connection locks before secret or connection
writes.

**Steps to reproduce**

1. Refresh a synthetic MCP catalog with 2,500 tools.
2. Return the same tool on two discovery pages with different
descriptions.
3. Fail the profile-binding insert during default-profile creation.
4. Overlap personal, organization, or managed OAuth persistence with
setup finalization or discovery of a new tool.

**Additional context**

This is the first feature extracted from #16503. The separate test-only
maintenance PRs #16601 and #16610 are merged. This PR now targets master
and is the active review. Standing assignments #16603, tool definitions
#16604, and access/credential hooks #16605 remain stacked drafts without
review requests.

## What Changed

- Deduplicate discovery results by tool name and keep the last
descriptor.
- Upsert catalog rows in transactions of at most 500 tools.
- Preserve row identity and operator-disabled status on conflict.
Determine default-access candidates from the rows read under each
publication lock, so an overlapping refresh cannot restore an action
turned Off after another refresh finishes.
- Lock the application before the connection during publication, setup
finalization, removal, and all three OAuth persistence paths. The OAuth
ordering fix is included here so this PR is independently safe; it does
not depend on #16605.
- Synchronize the managed default profile, its entries, bindings, and
managed policies in one transaction.
- Add PostgreSQL publication tests and document partial-failure
recovery.
- Target the existing callback failure injection at profile creation
instead of a transaction ordinal.

## Verification

- `pnpm exec vitest run
server/src/__tests__/tool-catalog-publication.test.ts
server/src/__tests__/tool-access-service.test.ts`: 436 tests passed with
the OAuth and overlapping-refresh corrections on top of the merged
#16610 fix.
- The new tests exercise real PostgreSQL with 250 and 2,500 tools,
repeated discovery pages, preserved choices, lock contention, and
failures between chunks and during profile creation.
- Four selected regression cases fail with the production source
restored to the parent commit. The implementation was then restored and
rechecked.
- Six additional real-PostgreSQL regressions cover personal,
organization, and managed OAuth persistence against both setup
finalization and new-tool discovery. All six reproduce PostgreSQL
deadlock `40P01` with the OAuth ordering fix removed; all six pass with
it restored. Discovery uses synthetic credential headers to isolate
publication locking from credential rotation.
- Apex initially returned 3/5 on
`0d6013a9649388041975406fe4b0247cd0e695aa`, identifying an
overlapping-refresh permission race. Two new PostgreSQL regressions
cover both absent and quarantined tools; both prove an agent goes from
deny back to allow without the correction, and remains denied with it.
The test pauses one refresh after its initial catalog snapshot,
completes another refresh, turns the action Off through the service, and
resumes the stale refresh.
- `pnpm --filter @paperclipai/server typecheck`: passed.
- `node scripts/check-maps.mjs`: passed (structural validation only).
- `git diff --check`: passed.
- Reviewed the complete diff for secrets and personal data before push.
- Earlier hosted run 38975428064 failed the maximum-size newline
skill-audit watchdog; other interrupted jobs reported runner
shutdown/cancellation. The original watchdog failure did not identify
whether startup or auditing timed out, so its exact cause remains
unknown. The separate merged #16610 isolates the audit deadline from
module startup and preserves the audit hang guard. This branch is
rebased onto that fix.
- Full local build and repository-wide tests were not run. At the
previous head `0d6013a9649388041975406fe4b0247cd0e695aa`, all 55 hosted
checks succeeded (two optional Storybook checks skipped), and standard
Greptile returned 5/5 before Apex identified the permission race above.
At corrected head `dc120d2bc879c54c14347db30e74c753bcff61e6`, all 55
hosted checks succeeded (two optional Storybook checks skipped), and a
fresh standard review returned 5/5 with no open findings. The second
Apex review also returned 5/5 on that exact head (successful Apex check
completed 2026-10-16T23:36:44Z). Both earlier findings are addressed,
there are no open review threads, and the branch is mergeable against
master.

## Risks

Each successful catalog chunk remains committed if a later operation
fails. A retry completes publication. Provider discovery occurs outside
database locks. Default-profile changes commit separately from the
catalog chunks and roll back together on failure. Tests use a controlled
MCP response; they do not qualify a live provider. There is no schema
migration or new plugin API in this PR.

## Model Used

OpenAI Codex, GPT-6, with reasoning, repository inspection, code
execution, and testing. The exact deployment model ID and context window
are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-16 16:40:12 -07:00
DottaandPaperclip dc499ae02b perf(server): reject shared Git cleanup paths before history scans (#16614) 
Check the existing global execution/project shared-path veto before scanning retained run contexts. Preserve the full unshared-path query, lifecycle locks and all deletion-phase rechecks.

Verify with 35 focused real PostgreSQL/Git cases, full repository typecheck/build, exact-head full CI/E2E and Greptile Base 5/5. A read-only same-snapshot query pair retained the false verdict while reducing round trip from1089ms to9.4ms; this is not a full-agent latency result. The first security scan was inconclusive after an infrastructure checkout failure; one actual rescan passed.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-16 18:17:15 -05:00
DottaandPaperclip 9a3290fe10 perf(runner): overlap default remote artifact preparation (#16608) 
Prepare the native runner and default image Codex harness together on providers that declare concurrent sync support. Preserve all executable checks, wait for both operations on failure, and retain serial explicit harness installs.

Verified with 881 affected cases, full typecheck/build, exact-head full CI and Greptile Base 5/5. The real Daytona component median fell from 839ms to 432ms; full deployed reply latency remains unproven.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-16 17:29:27 -05:00
Devin FoleyandPaperclip 73659c6b27 fix(server): let Git expiration discard Paperclip-managed checkouts (#16586) 
Reconstructable Git expiration rejected nearly every terminal worktree with unique_ignored_files, because Paperclip writes two ignored paths into every worktree itself: .claude/settings.local.json and the managed project checkouts under .paperclip-repositories/. Expiration now accepts the settings file, and accepts a managed checkout only through a prover the expiration worker supplies. The prover maps the checkout back to its project workspace row and proves it reproducible against that row's repository URL: worktree and stock-storage checks, no linked worktrees, no unreachable objects, and every ref, reflog and HEAD commit reachable from upstream refs fetched fresh into a proof store. Every inspection Git command now runs with core.fsmonitor disabled.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-16 14:31:06 -07:00
DottaandPaperclip 3db33c131e perf(slack): overlap endpoint preparation and narrow credential reads (#16593) 
Prepare Slack endpoints in pairs with a per-call concurrency limit of two, and resolve only the current bot token for live tool authority. Preserve candidate-order errors, active-sibling settlement, credential rotation and authorization revocation.

Verified with real PostgreSQL regression controls, full repository CI/E2E, security checks and current-head Greptile Plus5/5 with no unresolved findings.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-16 16:05:46 -05:00
DottaandPaperclip 2cc7992369 Share the external Runner lifecycle and add strict Muse contracts (#16444) 
Add the external Runner foundation with distinct Dot/Grok V6, canonical home-mode V7 and Muse V8 contracts.

Keep generated schemas and profiles synchronized. Bound default question-answer command IDs in a disjoint namespace while preserving existing IDs and durable replay. Preserve canonical migration history and add normally generated, replay-safe Muse schema milestones with literal preview-history fixtures.

Refs #16444.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-16 13:46:50 -05:00
Devin FoleyandPaperclip 4de6335c3b perf(server): trim saved run JSON from agent authentication (#16577) 
## Thinking Path

> - Paperclip coordinates AI agents that work for companies.
> - Agent JWT authentication checks each request against its saved run.
> - Saved runs can contain large prompts and result payloads.
> - The first run lookup needs only identity fields, cancellation state,
and the conversation flag.
> - Reading all saved JSON transfers data that this lookup does not use.
> - This PR selects the required JSON fields in PostgreSQL and preserves
the existing authorization decisions.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

The initial saved-run lookup in agent JWT authentication.

**Subsystem affected**

Server authentication and database queries.

**Current behavior**

The lookup transfers the full context and result JSON on each request.
It uses only the cancellation state and conversation flag from those
values.

**Proposed behavior**

Select the required JSON fields in PostgreSQL. Keep the run, company,
and agent predicates. Read current state on every request.

**Reason and benefit**

A synthetic lookup with a 96 KiB prompt and 4 KiB result returns 102,520
JSON bytes before this change and 134 bytes after it. This reduces data
transferred by this query.

**Breaking changes**

None intended. Identity fields, write revocation, read access, and
cancellation response codes retain their current behavior.

Related: #9280 addresses authentication round-trip counts. This PR only
reduces the data returned by the existing initial run lookup.

## What Changed

- Extract the initial run lookup and select only the JSON fields that
authentication consumes.
- Add real PostgreSQL tests for legacy JSON shapes, fresh cancellation
state, payload size, and run/company/agent scope.
- Add a reproducible benchmark that loads the original query from an
explicit baseline commit.
- Document the lookup and benchmark.

## Verification

- Baseline: `06689c1b3410000bcd65ff40cb469e600209c4db`.
- `pnpm -r typecheck`: passed.
- `pnpm build`: passed.
- Final server typecheck: passed.
- Final-head GitHub checks: all green at
`2be1c38e6bf51616b69993cf84cee1eef9657517`. Apex: 5/5, zero unresolved
threads, conflict-free.
- `pnpm check:module-boundaries`: passed. The standalone synthetic
benchmark now uses the existing root benchmark directory and resolves
Drizzle through the server workspace.
- Focused HTTP middleware and database tests: 29 passed on real Homebrew
PostgreSQL 16.14. The SQL test compares 270 combinations of status,
cancellation JSON, and conversation JSON. It also checks fresh state and
company/agent/run scope.
- Broad native verification: 873 passed across the HTTP middleware, SQL
projection, run diagnostics, and native executor suites.
- The default local full command completed its general server lane with
12,090 passed and 8,885 skipped tests. It could not finish the root
workflow: the host could not start the bundled PostgreSQL; a diagnostics
assertion matched a directory name; and Git fixtures retained the host
default branch. These failures remain in the verification record. All 68
failed files now pass on replay with disposable Homebrew PostgreSQL, an
isolated checkout, and command-scoped Git defaults. The 68-file replay
passed 2,588 tests and left two email skill-resolution failures. Moving
the checkout outside an unrelated ancestor skills directory resolved
those last failures: all 44 tests in that file passed. Product code and
test assertions were unchanged for these host adjustments.

Benchmark command:

```sh
pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-agent-authentication.ts --baseline-ref=06689c1b3410000bcd65ff40cb469e600209c4db
```

The final benchmark ran at treatment commit
`2be1c38e6bf51616b69993cf84cee1eef9657517` with a clean worktree. The
local run used PostgreSQL 16.14 and the maintained test constructor seam
to start disposable Homebrew clusters. The bundled PostgreSQL binary
could not start on this host. Both versions use the same schema and
predicates. Each case has two warmup rounds and ten measured rounds with
alternating order. Random content prevents unrealistic compression.
Complete authentication decisions match.

| Context / result | Before JSON bytes | After JSON bytes | Before
median | After median |
| --- | ---: | ---: | ---: | ---: |
| 1 KiB / 4 KiB | 5,240 | 134 | 0.157 ms | 0.157 ms |
| 96 KiB / 4 KiB | 102,520 | 134 | 0.331 ms | 0.140 ms |

These are local query round trips, including result transfer and
decoding. The small control has no material latency gain. The large case
improves by about 2.4x. Both plans read the same single row. The primary
result is reduced response size. This does not measure HTTP latency,
server CPU savings, or fleet-wide cost savings. Full samples and plans
are retained with the verification evidence.

## Risks

- JSON types must retain their meaning. The real database tests compare
the consumed authorization contract with the original projection,
including SQL null, JSON null, scalars, arrays, objects, string
booleans, and cancellation states.
- The change covers the first JWT run lookup. Later run identity capture
keeps its existing full-run reads and transaction locks. Those reads
remain a separate cost.
- The change adds no schema migration or authorization cache.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, code editing, and shell
tools. This session did not expose a more specific serving model ID or
its context window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-16 11:44:14 -07:00
DottaandPaperclip c91f7a6e5f perf(runner): overlap verified sandbox launch assets (#16585) 
Native sandbox launch preparation stages fresh harness files and immutable runtime context in sequence. Overlap them only under the verified concurrent-sync capability, settle every started transfer before launch or cleanup, and preserve serial SSH behavior and error priority.

A matched real-Daytona comparison reduces staging from 1,552 ms to 1,017 ms. The larger active-run difference includes model variation and is not attributed to this change. Add an explicit single-attempt diagnostic and synchronize stale Pi evidence metadata with its shipped profile.

Validation: 764 native runner/tracing cases, 2,764 support cases, repository typecheck/build, fresh exact-head full CI/E2E/security and Apex 5/5 pass. Observability remains a silent no-op when no endpoint is configured.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-16 13:35:26 -05:00
DottaandPaperclip f5534717b0 fix(runtime): preserve unexported native work during lease cleanup (#16583) 
Preserve the exact native workspace source until its export is finalized. Apply the same guard to cleanup sweeps and direct environment teardown. Keep stop-only recovery and healthy warm retention available.

Verified with 317 targeted tests, server typecheck, module boundary check, all current-head CI gates, and Greptile 5/5.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-16 12:56:09 -05:00
DottaandPaperclip ce0eff7d1f perf: reduce fresh workspace clone and sandbox staging waits (#16487) 
## Thinking Path

> - Paperclip manages AI agents and their work.
> - Each run needs a private workspace and safe persistence.
> - Fresh isolated repository sources download upstream Git history
again.
> - Sandbox staging also verifies an optional seed cache before it
uploads the workspace.
> - That verification holds a company lock during a full extraction and
hash scan.
> - This pull request reuses pristine Git objects through an
authenticated remote clone.
> - It also overlaps seed verification with upload and moves
verification outside the publication lock.
> - The benefit is less repeated transfer and less serial preparation
work, with the same task isolation and recovery contract.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Fresh task-private repository preparation and inbound sandbox workspace
staging.

**Current behavior**

Each isolated remote source clones the full upstream history. Optional
seed verification then extracts and hashes another full tree before
upload. One workspace's verification can hold the company cache lock
while other workspaces wait.

**Proposed behavior**

Use a company and credential scoped cache of pristine upstream objects
for GitHub HTTPS sources. Each task still clones from the current
remote. It disconnects from the cache before publication. Run seed
verification alongside upload. Hold the company lock only for final
capacity checks and publication.

**Reason and benefit**

These operations lie on the workspace preparation path. Removing
repeated Git pack transfer and serial verification can reduce startup
time. Local measurements below cover these boundaries. They do not
establish hosted agent latency.

Related: #16425, #16454, #16398, #16461, #16468, #16473. The cache
inspection and durable folder proposal in #13376 covers separate product
work.

## What Changed

- Add a pristine upstream Git object cache for fresh isolated GitHub
HTTPS sources. Key it by company, repository and current credential
fingerprint. Do not store credentials.
- Keep online clone authorization and current remote refs. Use
`--reference-if-able --dissociate` so task objects remain independent. A
reference failure retries once online without the cache and resolves
credentials again.
- Cap cache storage at 64 entries and 4 GiB per company, with 2 GiB per
entry. Skip contended publication. Bound priming by a cooperative
five-second budget and a 4,096-entry scan allowance. Keep the existing
free-space reserve.
- Overlap optional seed verification and required workspace upload.
Settle both before returning or deleting temporary files. Recheck quotas
and company directory identity under the publication lock.
- Keep the independent fresh upstream reproducibility proof, task
commits, dirty source copies, ignored files, nested repositories and
durable recovery behavior.
- Suppress optional Git server-info generation during dissociation so
generated storage passes the existing safe-expiration checks. This uses
[the supported Git
setting](https://git-scm.com/docs/git-config/2.43.0#Documentation/git-config.txt-repackupdateServerInfo).
- Add bounded timing phases, regression coverage and two repeatable
local benchmarks. No first-party Telemetry or dependency changes.

## Verification

- `pnpm -r typecheck`: passed.
- `pnpm build`: passed.
- Affected adapter suites: 240 tests passed. Native workspace
persistence/history and tracing: 33 tests passed. Server
checkout/isolation/tracing batch: 53 tests passed. Final object-cache
authority, path and capacity suite: 12 tests passed. These batches
overlap.
- Cache and expiration/restoration lifecycle batch: 34 tests passed. A
direct generated-storage check covers cached clones.
- Real HTTP Git fixture: a matching cache does not bypass revoked remote
access. Other fixtures cover current refs/tags, hidden cached commits,
independent objects after cache removal, corruption fallback,
credential/company scope, quota limits and publication contention.
- Initial restricted runs hit SQLite write and Unix socket permission
failures. The adapter batch passed with disposable writable instance
state and socket access. The first typecheck rejected the test use of
`Promise.withResolvers`; typed deferred gates now use the supported
library contract.
- Repository benchmark on final head
`9047368bf3b7fa2061e00d698b8b51cf4ae1da32`: five alternating samples,
real local `file://` Git transport, synthetic 32 MiB history, no
injected latency. Median 1143.8 ms to 240.5 ms; ranges 1085.7–1486.6 ms
and 188.0–254.8 ms. Received Git pack bytes fell from 33,569,788 to
zero. Initial cache priming took 1208.4 ms and is reported separately.
- Sandbox benchmark, five alternating samples per mode on 10,000 files
with real packing, extraction and verification: with a controlled
3-second transfer delay, median 9.450 s to 7.620 s (ranges 9.280–12.115
s and 7.296–7.983 s). With no injected delay, median 6.808 s to 6.450 s
(ranges 6.255–7.029 s and 5.709–6.902 s). The no-delay ranges overlap;
local copying competes with verification for disk. Both modes verify
final tree contents, ignored-file exclusion and Git HEAD. These are
synthetic local fixtures, not hosted agent runs.
- Reproduce with `pnpm --filter @paperclipai/server exec tsx
../scripts/benchmark-repository-object-cache.ts` and `pnpm --filter
@paperclipai/server exec tsx ../scripts/benchmark-workspace-staging.ts
--baseline-ref=307eb16c0c --transfer-delay-ms=3000`.
- Prior head `af766bc820` received Greptile 5/5 with no findings. Its
E2E shard 7 failed in `plan-task-fixture` cleanup: company deletion
raced an `activity_log` foreign key. That fixture has no provider run or
Git workspace. The failure is retained here; final-head CI must pass.
The corrected corruption fixture now forces a pack and verifies a real
damaged archive. Current head `9047368bf3b7fa2061e00d698b8b51cf4ae1da32`
has Greptile 5/5 with no review threads or findings. Final head has 55
passing checks and two optional Storybook skips. [The first
workspace-shard
attempt](https://github.com/paperclipai/paperclip/actions/runs/38802441857/job/116655485350)
passed all 1,778 cases, then reported a late `EPIPE` from unchanged
`server-utils.ts` while the ACP reuse test was active. That case mocks
workspace preparation and passed locally in isolation. [Its single retry
on the same
commit](https://github.com/paperclipai/paperclip/actions/runs/38802441857/job/116659728034)
passed, as did the dependent verification gate. The earlier E2E cleanup
fixture passed on this final head. Both initial failures remain recorded
here. A full local suite was not duplicated; the affected boundaries
were run locally and the complete suite runs in CI.

## Risks

- First cache population adds local copying. Cold clones, credential
rotation, capacity exhaustion and a corrupt cache retain the online
clone path. No eviction is added. Cached data is bounded, but many
companies can each consume their allowance.
- Individual filesystem calls and cleanup can exceed the cooperative
priming budget. Failed reference clones make one additional online
attempt.
- Parallel verification and upload compete for disk and CPU. The local
no-delay benchmark shows a smaller gain than a slow-transfer case.
Verification still checks contents before cache publication, and
required upload failures remain failures.
- Local fixtures do not prove hosted throughput or the 15–30-second
end-to-end target. The fresh checkout still writes its working tree, and
model startup, copyback and other preparation costs remain.

## Model Used

OpenAI Codex, GPT-6 family. The exact serving model ID and context
window are not exposed in this session. Used reasoning, source editing,
shell tools and test execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` /
`Closes: #` / `Refs: #` OR (b) described the issue in-PR following the
relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal task
identifiers
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-16 12:21:47 -05:00
DottaandPaperclip 06689c1b34 perf(skills): coalesce inventory before run cache locking (#16576) 
Share each active company skill-inventory refresh before taking the version-cache lock. Read actor visibility under the lock and keep snapshot publication protected through durable run pinning. Preserve empty-selection behavior and the list API.

Validation: 176 affected cases, full repository typecheck/build, fresh full CI/E2E, security and exact-head Apex 5/5 pass. The local six-reader latency experiment improves from 50.4s to 18.3s; live response latency is a separate acceptance check.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-16 10:39:02 -05:00
Nicky LeachandPaperclip 5fbece9dcc Synchronize assigned skills through agent runtime hooks (#16501) 
## Thinking Path

> - Paperclip manages shared skills and agent assignments.
> - Runtime plugins need installed skills before a task starts.
> - Skill content can change without changing the agent row.
> - Source access and version pins also affect delivered files.
> - This pull request feeds those changes into the durable runtime
stream.
> - The runtime reader resolves the same immutable skill sources used
for execution.

## Linked Issues or Issue Description

Builds on merged PR #16499. This pull request now targets `master` and
contains only the five skill synchronization commits.

**Problem**

Plugins receive no durable notification when an assigned skill changes.
A saved assignment can also become inaccessible after private-source
access changes.

**Proposed solution**

Invalidate referencing agents at installed skill mutations. Invalidate
inherited access when source bindings, project privacy, or policy
change. Return authorized skill sources and version IDs through the
owning runtime reader.

**Alternatives considered**

Activity-only subscriptions miss service writers. Per-task
synchronization adds preparation work. Copying current files would
ignore version pins.

**Roadmap alignment**

This extends existing skills, source synchronization, and plugin runtime
support.

## What Changed

- Defer multi-skill source runtime hooks until commit, after publication
holds all skill rows. Keep file-save journaling immediate to preserve
filesystem rollback.

- Add transactional invalidations for skill content, companion files,
versions, rename, deletion, policy, source access, and pin-setting
changes.
- Recognize canonical keys and supported legacy aliases.
- Reuse the execution skill resolver, current agent access, and instance
version-pin rules.
- Resolve and materialize sources outside the agent row lock. Recheck
ownership and runtime revision afterward.
- Keep task-only mentions and running-task snapshots in the execution
path.
- Document conservative invalidations for pinned assignments and
inherited access.

## Verification

- Rebased onto origin/master `9f4a956b90`, which contains merged PR
#16499. All five commits are unchanged according to `git range-diff`; no
conflict resolutions or source changes were needed.
- On rebased head `58b065941`, all 237 focused tests passed: runtime
skill hooks/source publication (41), plus content service, assignment
routes, policy routes, and project source access (196).
- Server typecheck passed after rebuilding Runner with the installed
Rust toolchain. Migration numbering/safety and map checks passed;
snapshot pruning made no changes. Migration CLI checks required local
IPC permission.
- Real PostgreSQL coverage includes concurrent source
publication/version saves, policy-only tightening/restoration,
referencing-agent fanout, company isolation, no-op counters, supporting
files, rename/deletion, rollback, private-source access, pinned/latest
bytes, and unassignment.
- Earlier stack validation passed workspace typecheck/build. Fresh
hosted CI and review must verify the rebased head before handoff.
- The broad local suite was previously limited by an unchanged
diagnostic test that rejects "private" in an error stack under this
/private/tmp worktree; that unchanged suite passed all 108 tests in the
original checkout. Hosted full-suite checks provide exact-head
verification.
- Diff whitespace and full PR secret/PII scan passed.
- All hosted CI/security checks are green on `58b065941`; fresh standard
Greptile and Apex both returned 5/5 with no findings or unresolved
threads. The PR targets master and is mergeable without conflicts.

## Risks

- Apply migration 0348 after the runtime-stream migration in #16499.
- Content invalidations include pinned assignments. The resolved version
stays pinned; consumers can avoid writes when its version/hash is
unchanged.
- Project access and source changes invalidate only agents assigned
skills from the affected sources. Company policy still invalidates the
company. Source polling bookkeeping and popularity counters do not
create work.
- The reader rebuilds installed caches without upstream requests.
Concurrent changes cause a bounded retry and then a conflict.
- Private transport integration and live remote computer qualification
remain separate.

## Model Used

OpenAI Codex, GPT-6, with reasoning, repository inspection, code
execution, and testing. The exact deployment model ID and context window
are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` /
`Closes: #` / `Refs: #` OR (b) described the issue in-PR following the
relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub references)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id
- [x] I have run tests locally and they pass (focused suites; broad
local limitation documented above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-16 08:28:47 -07:00
Nicky LeachandPaperclip 9f4a956b90 Add durable agent runtime synchronization hooks (#16499) 
## Thinking Path

> - Paperclip manages agents and their runtime configuration.
> - Plugins can maintain external agent computers.
> - Configuration and file saves currently miss durable plugin delivery.
> - A transient activity notification cannot recover after interruption.
> - This pull request adds an opt-in agent runtime resource stream.
> - The existing dispatcher delivers and acknowledges its committed
work.
> - Plugins can prepare computers when configuration changes.

## Linked Issues or Issue Description

Related: #15631 and #16203 establish agent lifecycle and native durable
delivery.

**Problem**

Plugins cannot reliably react to agent configuration and instruction
changes without polling or waiting for execution.

**Proposed solution**

Add an opt-in `agent_runtime` stream to the existing resource journal.
Record changes at the database mutation and managed file audit
boundaries. Wake the dispatcher with content-free commit notifications.
Provide scoped runtime inputs to the selected lifecycle plugin.

**Alternatives considered**

Route-only activity events miss service and background writers.
Scheduled scans delay changes and repeatedly query unchanged state.

**Roadmap alignment**

This extends the existing plugin and agent configuration work.

## What Changed

- Add durable update/delete events for agent runtime state, including an
existing-agent baseline.
- Capture configuration changes, instruction edits, deletion, restore,
and managed run writeback.
- Add commit notifications and a reconnect scan. Register one listener
for configured, enabled runtime subscribers. Release it when none
remain.
- Keep HTTP startup independent of queued plugin callbacks.
- Add `ctx.agents.getRuntime`. Require lifecycle capability, invocation
company, enabled plugin, and owning lifecycle selection.
- Keep legacy agent/project subscriptions isolated. Manual inbox reads
retain both legacy streams.
- Resolve recovered instruction paths without reading or hashing bundle
contents. Use the company reporting graph for runtime health.
- Repair missed file events on identical save/deletion retries after a
journal failure; preserve content tokens and conditional-write checks.
- Correlate file audits with runtime revisions to deduplicate recorded
retries without a new table. Bound the lookup with the existing
company/timestamp index.
- Document the contract, migration, access boundaries, and
session-connection requirement.

## Verification

- Rebased the full stack onto origin/master after #16507.
- Final replayed stack: `pnpm -r typecheck` and `pnpm build` passed.
- All 132 affected runtime, skill, connector, publication, instruction,
managed-agent, and delivery tests passed after the latest-master rebase.
Workspace typecheck and build passed.
- Journal failure injection verifies recovery on identical file writes,
deletions, and instruction retries from an acknowledged inbox.
- Deduplication follow-up: 80 runtime/instruction/delivery tests and
server typecheck passed. Repeated retries create no audit/events; old
hashes returning after an intervening commit still repair delivery.
Built-in/native instruction writer regressions passed all 56 tests.
- PR 1 follow-up tests cover valid/cyclic reporting chains, listener
registration failure, multiple subscribers, global/company disable,
re-enable, unload, and startup with blocked callbacks.
- Database coverage includes rollback, journal failure, no-op
suppression, file saves/deletion, ordering/replay, owner/company denial,
independent-client commits, and real listener loss/reconnect.
- SDK/shared suites passed 1,059 tests during implementation.
CLI/delivery regressions also passed.
- Migration safety, map checks, diff whitespace, and local secrets/PII
scan passed.
- Hosted CI and security checks are fully green at `b296405e2`. Standard
Greptile and Apex both returned 5/5 with no unresolved findings.
- The broad local suite encountered an unchanged diagnostic test that
rejects the word "private" from an error stack. This worktree lives
under /private/tmp. The same unchanged diagnostic suite passed all 108
tests in the original checkout. Hosted full-suite checks provide
exact-head verification.

## Risks

- Apply migration 0347 before enabling a runtime subscriber. Runtime
notifications require session-level PostgreSQL LISTEN. Transaction-mode
poolers are unsupported for this opt-in feature.
- Database triggers share the mutation transaction. Journal failure
prevents configuration commits. Trigger work adds a journal insert to
relevant changes.
- Filesystem publication and SQL commit are separate boundaries. Direct
filesystem edits do not emit hooks. External clients must fence stale
revisions and retain conditional-write checks.
- The reader returns only that plugin's own instance secrets. Keep them
out of UI data, logs, plugin state, and agent files.
- No private transport integration or live remote computer qualification
is included.

## Model Used

OpenAI Codex, GPT-6, with reasoning, repository inspection, code
execution, and testing. The exact deployment model ID and context window
are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` /
`Closes: #` / `Refs: #` OR (b) described the issue in-PR following the
relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub references)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id
- [x] I have run tests locally and they pass (focused suites; broad
local limitation documented above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-16 07:53:04 -07:00
DottaandPaperclip b294abb9eb perf(server): append ordinary run events in one statement (#16569) 
Ordinary run events now validate their binding, allocate a sequence and insert the decoded receipt in one atomic PostgreSQL statement. Native replay and retry-exhaustion transactions keep their existing behavior.

Verification: exact-head full CI/E2E, Superagent, legacy security and Apex 5/5 passed. PostgreSQL concurrency, rollback and payload controls pass. Controlled network-delay comparison reduced median event persistence from about192ms to48ms; no staging response-time claim is made before deployment.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-16 08:51:01 -05:00
DottaandPaperclip 8556539364 perf(adapter-utils): batch remote GitHub launcher staging (#16552) 
Stage the nine managed GitHub launcher files through one bounded control command. Preserve per-file locks, content verification, atomic replacement and control-channel dispatch.

Real Daytona staging decreased from about one second to 0.24–0.34 seconds. Full CI, E2E, Superagent and Apex 5/5 passed on the reviewed head.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-16 04:59:21 -05:00
Devin FoleyPaperclipClaude Opus 5.5
 		8f1cb11a03 fix(workspaces): name the owner action when a shared source is restricted (#16545) 
A reused task workspace in a shared project checkout keeps its source
denial when another task's restricted worktree shares that checkout
(#16095 deliberately does not clone over existing workspaces). The run
then failed as an opaque `setup_failed` 403 and the task was blocked
with a generic "no live execution path" notice.

Convert that exact denial into a `configuration_incomplete` failure
with reason `workspace_source_restricted`, give the task notice and the
recovery action the owner step, and keep this pre-dispatch owner action
out of the unexpected Sentry run-failure stream. The reason and
predicate live in a dependency-free module. Access checks are unchanged.

Co-Authored-By: Paperclip <Paperclip-Paperclip@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-16 01:33:53 -07:00
DottaandPaperclip f1d4bc1617 perf(slack): resolve only signing secret during webhook checks (#16548) 
Resolve only the current Slack signing secret during workspace checks and authenticated setup evidence. Preserve full runtime credential resolution, company bounds, rotation, and auditing.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-16 03:05:05 -05:00
Devin FoleyandPaperclip 78b833c495 Dispatch routine and monitor work at saved deadlines (#16514) 
## Thinking Path

> - Paperclip manages work that agents run for a company.
> - Routines and issue monitors store their next run time in the
database.
> - The app currently waits for the heartbeat interval to find that
work.
> - This can delay work after a workspace wakes before its saved
deadline.
> - This pull request uses the existing work coordinator to schedule
each known deadline.
> - It also parses each cron expression once, so sparse schedules do
less work on the event loop.
> - Agents can start due work sooner while the existing execution checks
stay in place.

## Linked Issues or Issue Description

**What happened?**

Routine and issue-monitor dispatch depends on the heartbeat scan
interval. A workspace can wake early and still wait for the next scan. A
sparse cron search also parses the same expression for every minute it
checks. In a local annual-schedule test, the deadline callback ran on
time but schedule calculation delayed the routine record by 13.6
seconds.

**Expected behavior**

Use a saved future deadline to request the existing dispatch scan.
Restore deadlines at startup and update them after a producer
transaction settles. Do not let future timers keep an idle workspace
active. Parse a cron expression once for each search.

**Steps to reproduce**

1. Set an active routine and an assigned issue monitor to the same
future deadline.
2. Choose a deadline between heartbeat scans. Use an annual routine
expression to exercise a sparse search.
3. Compare the saved deadline with the routine firing and agent start
times.

**Paperclip version or commit**

Source checkout based on c7ea5c0216.

Related public changes: #16215 reports routine deadlines for external
wake. #8034 caches time-zone formatters in the cron search. This change
reuses those contracts.

## What Changed

- Register separate routine and issue-monitor workers with the existing
delivery coordinator after startup housekeeping.
- Restore the earliest eligible future deadline from the database after
each scan.
- Notify the matching worker when routine or monitor writes settle,
including the native `set_task_monitor` tool. Keep the heartbeat
interval as a recovery scan for blocked or externally written work.
- Retain the scan-start cutoff so a deadline crossed during a slow scan
receives an immediate follow-up scan.
- Retain the coordinator gates for standby, idle drain, and shutdown.
Future timers do not count as active work. Bound expired deadline
retries during idle reconciliation so a paused worker cannot spin.
- Parse a cron expression once during a next-run search. Skip impossible
UTC dates before time-zone formatting. Preserve the existing calendar
and time-zone matching.
- Add clock, transaction, failure, SQL eligibility, and producer
notification tests. Document the scheduler behavior.

## Verification

- `pnpm -r typecheck`: passed.
- `pnpm build`: passed.
- Full CI, browser tests, and security scan: passed on final head
9fc857dcec. Greptile Apex is 5/5 with no
unresolved comments.
- Final focused startup, deadline, routine, SQL eligibility, native
monitor, cron, and coordinator tests: 159 passed across seven suites.
The crossed-deadline and native-notification regressions failed before
their fixes.
- CI caught the existing startup suppression assertion. It failed
locally before restoring the outer timer-dispatch guard; all 38 startup
tests now pass.
- A canceled-drain regression also failed before the fix. Future timers
keep their saved time; only expired deadlines use the bounded idle
retry. Server typecheck and build passed again.
- The idle-drain regression reproduced 22 checks in 20 ms before its
fix. After the fix, 43 coordinator and related worker tests passed.
Server typecheck and build passed again after the review fixes.
- The broad local `pnpm test:run` attempt reported existing
host-sensitive failures. Two Git fixture tests fail identically on
unchanged c7ea5c0 because the host default `main` leaves an extra branch
in an intended master-only remote. Both pass with a command-scoped
`init.defaultBranch=master` setting. No Git settings were changed on
disk.
- A diagnostics test also fails on both unchanged master and this branch
because its broad `private` substring assertion matches the canonical
macOS temporary source path in an error stack. A local Anthropic hire
assertion failed only in the broad run; all 70 tests in that file pass
on both source revisions when run separately. Its broad-run cause is not
determined. These failures remain recorded. The full exact-head CI
matrix passed; the broad local development attempt was then stopped.
- An earlier full run was interrupted after PostgreSQL reported host
shared-memory exhaustion. One retry was stopped when timing required the
UTC optimization. Acceptance then used an owned database on the existing
local PostgreSQL service. No shared services were stopped or reset.
- Local dispatch: routine record at +0.102 s, routine agent start at
+0.171 s, monitor agent start at +0.159 s. Both scripted agents
succeeded using real run-scoped API authentication. No model provider
was called. No duplicate runs occurred during the 70-second follow-up.
Platform: macOS, Node 25.6.1, isolated database on local PostgreSQL 16,
source checkout. The final run used a fresh database and a clean
checkout of 9fc857dcec.
- The first timing run exposed the repeated cron parsing described
above. A diagnostic run showed the timer fired 1 ms after the deadline
while the annual search blocked the event loop. Parsing once reduced
local dispatch to 5.9 seconds, which still missed the five-second test
target. The next change skips impossible UTC dates before formatting. A
second regression failed before that fix. The final timing result and
duplicate check passed.
- A separate acceptance setup failed because the direct local server had
no agent signing key. That owned fixture was removed. The retry uses an
ephemeral key in its isolated process environment.
- Owned fixtures were deleted, the local runtime exited normally, and
the disposable databases were dropped. Shared services remain running.
- No production deployment or npm publication is part of this PR.

## Risks

- Known deadlines now trigger a scan between heartbeat ticks. Existing
atomic claims, suppression checks, budgets, and execution rules still
decide whether work runs.
- Direct database writers rely on the heartbeat recovery scan. Overdue
blocked work also retains that recovery interval to avoid a tight retry
loop.
- This is not a hard real-time guarantee. Provider startup, queue load,
and other synchronous work can still delay agent execution.
- No database migration or API schema change. Revert this commit to
restore interval-only dispatch.

## Model Used

OpenAI GPT-6 (Codex), with reasoning, repository tools, terminal
execution, and automated tests. The session does not expose a more
specific model version or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass for the changed areas;
full-suite CI and local exceptions are documented above
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-15 22:44:17 -07:00
DottaandPaperclip 0e58f29eb7 fix(observability): trace pre-dispatch controller waits (#16515) 
Measure usage startup, MCP handoff, dispatch admission, and budget reservation with the existing endpoint-gated tracer. Preserve outer attempt context across asynchronous provider dispatch.

Validated with 135 focused tests, full typecheck/build, fresh full CI/E2E, Superagent, and Apex 5/5 on 9b28865bb3.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-15 20:30:05 -05:00
Devin FoleyandPaperclip c7ea5c0216 perf(server): skip settled committed finalizations in unscoped passes (#16485) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - On restart, `/api/health` reports `starting` until startup recovery
finishes, and deploys wait for it
> - Native recovery and the orphan reap each run an unscoped native
finalization reconciliation pass, and readiness waits for both
> - The pass selects every committed finalization with a result,
including runs that ended long ago, and spends dozens of sequential
queries on each one
> - On a busy instance this is about one minute per pass, so about two
minutes of every restart, and the periodic recovery repeats it on every
tick
> - This pull request skips settled committed finalizations in unscoped
passes, and keeps a small rotating upkeep page so their upkeep still
runs
> - The benefit is faster restarts and less steady database load, with
no change for unsettled work

## Linked Issues or Issue Description

Related: #16330 and #16329 (startup and reconciliation timing).

**What existing behavior does this improve?**
Startup recovery time and steady-state recovery load.

**Current behavior**
- The unscoped candidate scan in `reconcileNativeFinalizations` returns
every native finalization with a result and phase other than
`terminal_failure`. Nothing excludes rows that are already settled.
- Each committed row then takes the full per-row path: pending-effect,
assessment, evidence and decision reads, presentation repair (twice),
the workspace barrier read, a `finalizeNativeRun` re-read of the
committed coordinator, and workspace settlement (lease release check and
agent status update). That's about 40–60 sequential round trips per row.
- Startup runs the unscoped pass twice (native recovery, then the orphan
reap), and readiness waits for both. The periodic orphan reap runs it
again on every tick.
- On one instance, all 268 candidates were committed finalizations whose
runs had ended. Native recovery took 50–80 s and the orphan reap 47–80 s
on every restart, with database round trips of about 3.4 ms.

**Proposed behavior**
An unscoped pass skips a committed finalization when:
- its run is `succeeded`, `failed`, `cancelled` or `timed_out`, and
- the run finished more than an hour ago, and
- its decision has no `pending` or `failed` status-decision effect.

Each unscoped pass also includes up to 10 such settled rows in a
rotating order, so the presentation and evidence upkeep still reaches
every row over time. Scoped passes (explicit run IDs) are unchanged.

**Reason and benefit**
These settled rows have nothing left to finish. Recent rows, unfinished
runs and rows with undelivered effects stay in every pass, so
crash-window repairs and effect delivery behave as before. Startup and
periodic passes no longer scale with the full history of committed runs.

**Breaking changes**
None. A late repair on a settled row can be delayed until its turn in
the upkeep rotation instead of the next pass.

## What Changed

-
`server/src/services/native-runtime/native-finalization-reconciler.ts`:
- `settledCommittedFinalization(now)`: the settled predicate above,
including a `NOT EXISTS` on undelivered `status_decision_effects` for
the same company, issue and decision.
- The unscoped candidate scan adds `NOT settled OR run in the upkeep
page`.
- `nextSettledCommittedUpkeep`: up to
`NATIVE_SETTLED_COMMITTED_UPKEEP_PER_PASS` (10) settled run IDs after an
in-process cursor, ordered by run ID, wrapping at the end.
- New constants `NATIVE_SETTLED_COMMITTED_GRACE_MS` (1 hour) and
`NATIVE_SETTLED_COMMITTED_UPKEEP_PER_PASS`, plus
`resetSettledCommittedUpkeepForTests`.
- New
`server/src/__tests__/native-finalization-settled-candidates.test.ts`.
- `doc/execution-semantics.md`: describes the settled rule and the
upkeep rotation.

## Verification

- `pnpm exec vitest run
src/__tests__/native-finalization-settled-candidates.test.ts` (in
`server/`): 4 passed. The test mocks the timing hook to capture the
candidate count and stop the pass after the scan:
- 25 settled rows give 10, 10, 5, then 10 again over four passes
(rotation and wrap).
- A row finished 10 minutes ago, a running row, and a terminal row
without `finished_at` are all candidates.
  - A settled row whose decision has a `failed` effect is a candidate.
  - A scoped pass with an explicit run ID still selects a settled row.
- `pnpm exec vitest run` for `native-finalization-recovery`,
`legacy-finalization-regression`, `native-run-finalizer-telemetry`,
`native-workspace-finalizer`, `native-finalization-reconciler-timing`,
`external-chat-wait.integration` and `native-status-arbiter-corpus`: 316
passed.
- `heartbeat/recovery.test.ts`: the 8 tests that do not need a database
passed. The 20 database tests were skipped locally because embedded
PostgreSQL could not start (the host's shared-memory segments were
exhausted by other processes). CI runs them.
- `pnpm typecheck` in `server/`: passed.

## Risks

- A settled row that needs a late repair (for example a missing chat
presentation) waits for its turn in the upkeep rotation, up to
`ceil(settled rows / 10)` passes, instead of the next pass.
- The upkeep cursor is in memory. After a restart the rotation starts
from the beginning; the work is idempotent.
- A run left `running` with a committed finalization is never considered
settled, so it is still projected before the reaper's active-run loop.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file edits, test runs, database and log analysis).
A separate Claude agent mapped the recovery call tree.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-15 16:42:15 -07:00
Huanyi Xie c4a16cbe98 fix(plugins): follow local package path changes in dev watcher (#15247) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Local plugin development uses file changes to restart the plugin
worker.
> - A plugin can move to a different local package directory.
> - The watcher rejected every later registration for the same plugin
ID.
> - This pull request replaces the watcher when the resolved package
path changes.
> - The benefit is that current plugin files trigger reloads and
previous files stop doing so.

## Linked Issues or Issue Description

Refs: #14955. This change addresses its second observation: the watcher
keeps the previous local directory after its package path changes. The
issue also tracks stored configuration replay as a separate problem.

Related PRs: #11336 shares the lifecycle manager across routes and the
dispatcher. #14381 controls when production file watching is enabled.
#10100 orders configuration delivery. Those changes address other parts
of plugin reload behavior.

## What Changed

- Store the resolved package path with each watcher. Keep repeated
registrations of the same path as a no-op. Replace the watcher for a
different path.
- Cancel pending changes from the previous directory. Ignore late
change, error, and timer callbacks from a replaced watcher.
- Apply only the latest pending directory lookup. Manual registration,
unwatching, and shutdown invalidate earlier lookups.
- Stop a failed watcher without cancelling its pending replacement
lookup. Explicit unwatching, disable, unload, and shutdown still cancel
pending registrations.
- Stop the old watcher when the new directory is unavailable or the
plugin no longer has a local package path.
- Document the path-change behavior and add twelve regression cases.

## Verification

- `pnpm exec vitest run server/src/__tests__/plugin-dev-watcher.test.ts
server/src/__tests__/plugin-lifecycle-restart.test.ts`: 18 tests passed.
Regressions cover path changes, stale callbacks, lookup order, explicit
cancellation, and watcher errors during a pending replacement. The
current-watcher error case fails before the stop-versus-cancel fix; the
stale-watcher error control passes.
- A local smoke check used real Chokidar watchers and two real temporary
plugin directories. It confirmed that a pending old change is cancelled,
later old-directory changes are ignored, and a new-directory change
triggers one restart.
- `pnpm -r typecheck` passed on the path and lookup guards. The final
watcher-error fix passed `pnpm --filter @paperclipai/server exec tsc
--noEmit`.
- `pnpm build` passed on the initial path-change fix. The final source
passed `pnpm --filter @paperclipai/server build`.
- `git diff --check`: passed.
- Greptile reviewed `bef3182` at 5/5 with no new findings. The
watcher-error review thread is resolved. CI is still running.
- The full `pnpm test:run` suite was not repeated for this watcher-only
change. The affected suites ran locally. The broad test shards will run
in CI.

## Risks

- Low risk. The watcher follows the current registered path. An
unavailable replacement stops watching until another valid registration
arrives.
- Closing a filesystem watcher is asynchronous. Callback identity checks
prevent an old watcher from acting on its replacement.
- No database migration, API shape, or dependency change is required.

## Model Used

- OpenAI Codex based on GPT-6, with reasoning, tool use, and code
execution. The runtime does not expose the exact serving model ID or
context window.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
 2026-10-15 16:13:35 -07:00
DottaandPaperclip becdc3b08e perf(slack): remove delays before mention acknowledgements (#16491) 
Slack mentions now start eyes reactions after the authorized task/comment commits, independently of attachments, subscription and scheduler acceptance. Omit unused optional URL-preview polling while retaining message text, links and files. Keep receipt failures and missing-file warnings independent, including terminal cleanup of older multi-line diagnostics.

Verified with full current-head CI and E2E, Superagent and legacy security checks, Apex 5/5 with no unresolved findings, 19 selected real-PostgreSQL receipt/authorization/cleanup cases, and full frozen-source typecheck/build. The first CI attempt's unchanged Cursor probe timeout is retained in the PR; the one failed-only retry passed under unchanged deadlines. Live Slack latency still requires deployed acceptance.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-15 17:36:50 -05:00
馨冉馨冉Paperclip
 		81cea89d6b fix(cli): require confirmation for instructions file deletes (#11504) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The CLI exposes board/operator controls for managing agent
configuration and instructions.
> - Agent instructions files influence future agent behavior, so
deleting one is a destructive configuration mutation.
> - Nearby destructive CLI commands already require explicit `--yes`
confirmation before sending DELETE requests.
> - `agent instructions-file:delete` skipped that local confirmation
guard and could delete a bundle file from a scripted or mistyped
command.
> - This pull request aligns that command with the existing
destructive-command safety convention.
> - The benefit is a small, predictable operator-safety guard with
focused regression coverage.

## Linked Issues or Issue Description

Bug report:

### Pre-submission checklist

- [x] I have searched existing open and closed issues and this is not a
duplicate.
- [x] I am on the latest released version of Paperclip (or can reproduce
on `master`).
- [x] I have confirmed the error originates in Paperclip itself — not in
my agent adapter, API provider, or local configuration.

### What happened?

`paperclipai agent instructions-file:delete` deleted an agent
instructions bundle file without requiring explicit CLI confirmation.

### Expected behavior

Destructive CLI delete operations should require `--yes` before sending
a DELETE request.

### Steps to reproduce

1. Run `npx paperclipai agent instructions-file:delete <agent-id> --path
AGENTS.md`.
2. Observe that the CLI immediately issues `DELETE
/api/agents/:agentId/instructions-bundle/file?path=AGENTS.md`.
3. No local confirmation guard blocks the destructive request.

### Paperclip version or commit

`fd472d02ba950519ca5eff75ed9fd55cf976f4c2`

### Deployment mode

Local dev (`pnpm dev`)

### Installation method

Built from source (`pnpm dev` / `pnpm build`)

### Agent adapter(s) involved

- [x] Not adapter-specific (core bug)

### Database mode

Not database-related

### Access context

Board (human operator)

### Node.js version

`v20.x`

### Operating system

macOS

### Relevant logs or output

```text
Current behavior is local CLI behavior: the command proceeds to the DELETE call without requiring `--yes`.
```

### Relevant config (if applicable)

None.

### Additional context

This command already sits next to other destructive CLI actions such as
`agent delete`, which require explicit confirmation. Aligning the guard
makes the CLI safety model consistent.

### Privacy checklist

- [x] I have reviewed all pasted output for PII (usernames, file paths,
API keys, tokens, company names) and redacted where necessary.

## What Changed

- Added a `--yes` confirmation option to `agent
instructions-file:delete`.
- Refuse the command locally before resolving API context or sending a
DELETE request when confirmation is missing.
- Updated CLI parity coverage for confirmed and refused
instructions-file deletion.
- Updated the CLI documentation example to include `--yes`.

## Verification

-
`/Users/xinran/Downloads/dev/paperclip-pr-work/node_modules/.bin/vitest
run cli/src/__tests__/agent-lifecycle.test.ts --config
cli/vitest.config.ts` equivalent executed from the PR worktree using
temporary dependency symlinks: 3 tests passed.
- `git diff --check`

## Risks

- Low risk: this only changes one destructive CLI command and matches
existing `agent delete`, `issue delete`, and other delete-command
confirmation behavior.
- Scripts that intentionally delete instructions files must add `--yes`,
which is the intended explicit confirmation path.
- No database, API contract, or lockfile changes.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- OpenAI Codex, GPT-5-based coding agent in Codex desktop with
shell/tool use and local test execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: 馨冉 <xinxincui239@gmail.com>
Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-15 15:36:23 -07:00
DottaandPaperclip 67a9d5cc37 perf(runner): bypass skill cache waits for empty selections (#16498) 
Native runs with no skill files no longer queue behind company skill-cache work. Preserve saved empty selections and the existing exclusion for selected and legacy operational skills.

Verified with real PostgreSQL held-lock regressions, 20 distinct affected cases, full repository typecheck/build, fresh full CI/E2E/security and exact-head Apex 5/5. Hosted latency remains to be verified.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-15 17:01:57 -05:00
Nicky LeachandPaperclip ec3eb1968b feat(runner): expose credential updates through PRP (#16463) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The Runner uses PRP to send harness events to the host.
> - A harness can replace its stored OAuth tokens during a run.
> - The host has no PRP signal that tells it to read the updated
credential.
> - This pull request adds a durable credential update event and
observers for supported stores.
> - The host can react before it acknowledges the event. Token values
stay outside PRP.

## Linked Issues or Issue Description

**Subsystem affected**

`packages/paperclip-runner`: native provider execution, PRP schemas, and
host event delivery.

**Problem or motivation**

A Paperclip host that consumes PRP needs to react when a harness changes
its refresh-token credential. Existing credential copy-back paths do not
provide a common event during a warm session.

**Proposed solution**

Add `auth.credentials.updated` as a P0 runner event. Include only the
provider, binding ID, revision, and detection method. Observe native
Codex and Codex/Grok ACPX credential files. Retain each pending event
across runner restart until the durable outbox accepts it. Raw PRP
consumers opt in with `observeCredentials: true` before session open.
The bundled transport opts in for fresh sessions.

**Alternatives considered**

`harness.diagnostic` would mix a runner observation with harness
diagnostics. Sending token values through PRP would expose credentials
to event logs. Both approaches are unnecessary.

**Roadmap alignment**

I checked `ROADMAP.md`. This change extends the existing Runner protocol
and does not duplicate a planned core feature.

**Additional context**

Refs: #11094, #14056, #12696. These describe credential copy-back and
staging work. This PR adds the notification contract. It does not
replace those private credential channels or claim to fix their full
scope.

## What Changed

- Add strict credential update payload validation to PRP v1, v2, and v3.
Regenerate validators, the manifest, and the event catalog.
- Watch supported credential directories with `notify`. Add a periodic
read as a fallback and check at terminal and close boundaries.
- Persist a private baseline and pending event together. Notify on
account identity changes and retain the new baseline for later
refreshes. Ignore incomplete files, unsafe files, and metadata changes.
- Forward the event through the host transport. Test commit-before-ACK
delivery and replay.
- Commit pending runner observations before suspend and shutdown
commands complete. Preserve the provider event backlog.
- Create the private ACPX runtime directory before configuring
observation. Cover fresh session startup with observation enabled.
- Fence asynchronous PRP commits to the current authenticated connection
and run authority. Ignore replaced callbacks while preserving the
maintenance drain of admitted work.
- Deliver observations found during final provider cleanup before
clearing terminal recovery state. Retain the outbox and recovery state
when the host ACK is lost.
- Advance the pending Copilot candidate profile to v19 for the changed
shared PRP source hashes. Preserve its v18 history and pending
qualification status. Include the current fixture in the Docker build
context.
- Commit credential updates through an awaited host callback before PRP
ACK, including during idle and close. Persist them in a separate run-log
source namespace with company, agent, session, and runner checks.
- Preserve public event metadata through server redaction. Register the
event in task run details.
- Document opt-in behavior, supported stores, event semantics, and the
local run-log path.

## Verification

- All 45 durable runner control tests and 24 native executor integration
tests pass.
- Ten Rust credential observer tests pass. They cover atomic
replacement, polling, recovery, account-only changes and later refreshes
for Codex and Grok, unsafe files, fresh runtime creation, and the Grok
close handoff. Account identity stays private.
- The full Runner TypeScript suite passes: 3,565 tests in 221 files. An
additional 12 tests are skipped by the suite.
- All 214 focused tests pass for the contract, Codex event routing,
catalog, and durable host delivery.
- All 13 Node protocol contract tests pass.
- The exact Docker context integrity command passes: `docker buildx
build --file .github/docker-context-checks.Dockerfile .`.
- Rust formatting and generated protocol checks pass.
- `pnpm -r typecheck` and `pnpm build` pass. The full local `pnpm
test:run` was attempted with CI-compatible `init.defaultBranch=master`,
then stopped after a reproducible failure outside this change:
`git-workspace-expiration.test.ts` expects one re-expired workspace but
gets zero. The same case fails in isolation on this macOS checkout. Full
repository validation is provided by CI; the focused Runner and host
tests pass locally.
- All 253 native transport and restart tests pass against the final
release binary.
- All 3 durable recovery tests and 139 candidate profile and native
admission tests pass.
- All 41 host redaction and task protocol surface tests pass. `pnpm
check:token-gates` passes.
- All 467 CI policy tests pass, including the 15 CI profiling tests. The
upstream secret-reference test fix is retained from master. No workflow
changes are included.

- The Apex host-delivery fix passes 381 targeted Runner tests, six host
database tests, repository typecheck, and the production build. The
Runner tests cover commit-before-ACK, retry after a rejected host
commit, warm attachment, and close after the turn queue ends.
- Prior head `2f125cfcb` passed CI, standard Greptile 5/5, and Apex 5/5.
One CI shard was interrupted by GitHub runner shutdown; its single rerun
passed. Both Apex findings are addressed: idle/cleanup host delivery and
late callback reconnect fencing. Current head `5db78f082` passes CI (56
successful statuses and four optional skips) and standard Greptile 5/5.
Apex review of the account-change update is 5/5 on this exact head, with
no actionable issues.
- All 121 durable control-plane and credential callback tests pass,
including account-only change delivery, subsequent refresh on the new
account during close, host commit retry, late success/failure races,
forward cursor delivery on a replacement connection, and the existing
maintenance retirement contract. All 27 native provider and durable
recovery integration tests pass after the account-change update.

## Risks

- File changes signal a stored credential update. They do not prove that
an OAuth refresh succeeded. Changes between reads can be combined.
- Unix Codex and Codex/Grok ACPX stores are supported. Other harnesses
and credentials held only in memory need their own integration.
- Hosts must update their event schemas before opting in. Older hosts
omit the flag and receive no new event. Older runners ignore the flag.
- The host must use its existing private credential channel and account
checks to read or copy the token. Account switches trigger notification
but do not authorize accepting the new account. The event grants no
credential access.
- Tests use synthetic credentials. No live OAuth refresh has been
forced.

## Model Used

OpenAI Codex, exact model ID `gpt-6.1-sol`, with high reasoning effort,
repository inspection, tool use, and code execution. The session does
not expose its context window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-15 14:49:00 -07:00
DottaandPaperclip 8843887c24 fix(runner): preserve managed GitHub access in OpenCode (#16488) 
## Thinking Path

> - Paperclip manages AI agents and their authorized connections.
> - Native OpenCode agents use managed Git and GitHub launchers for
shell work.
> - Each operation must obtain credentials from the active run's broker.
> - Two OpenCode environment filters dropped that broker binding before
the shell started.
> - The shell therefore ran anonymously even when connector tools had
authorized access.
> - This pull request preserves the explicit broker environment through
both filters.
> - The benefit is working GitHub shell access with current identity and
revocation checks.

## Linked Issues or Issue Description

**What happened?**

Native OpenCode shell commands reported `GitHub capability_missing`
despite an authorized GitHub connection. Both runner startup and
provider startup dropped the controller's broker settings. Connector MCP
calls used a separate path and could still succeed.

**Expected behavior**

A managed shell operation obtains credentials for the current run. A
later operation uses the new identity after rotation and loses access
after revocation. Ambient host credentials do not supply authority.

**Steps to reproduce**

1. Start the native GitHub broker with an active run binding.
2. Pass its environment through native OpenCode startup.
3. Run `gh api user` from the provider shell.
4. Observe anonymous access before this fix.

**Paperclip version or commit**

Reproduced on master `6943904bad` before this change.

**Deployment mode**

Native Paperclip Runner with OpenCode, including remote execution.

Related: #16401 fixes the separate oversized OpenCode SSE failure and is
already merged. The same live-driver regression was verified locally
against its exact pre-fix source and current source. This PR does not
duplicate that fix. Searches found no open PR for this native broker
propagation defect; #15309 concerns the legacy OpenCode adapter.

## What Changed

- Use the existing GitHub credential environment projection at both
OpenCode launch boundaries.
- Preserve explicit controller authority while excluding an ambient host
broker.
- Exercise the real launcher and broker from an OpenCode child process,
directly and through runner projection.
- Check operation-time identity rotation, revocation, and exclusion of
controller API credentials and persistent GitHub tokens.
- Document the managed OpenCode shell contract.

## Verification

- Red: both new `native-github-access.test.ts` cases returned
`anonymous` instead of `fixture-a` before production changes.
- Green: both cases pass after the fix, including `fixture-a` to
`fixture-b` rotation and anonymous operation after revocation.
- Full native GitHub broker suite: 18 tests pass.
- OpenCode, provider-event, and configured-environment suites: 141 tests
pass across eight files.
- Separate SSE red/green verification: the existing live-driver
regression times out on `fd057f6ac4^` and passes with merged #16401. The
full OpenCode group also passes after restoration.
- Full repository build and typecheck pass locally. All CI jobs pass for
`23d734293f6a5c6d6128cae0c8773661816daf9a`. Final GitHub rollup: 55
successful checks/statuses, two skipped checks, and one neutral check;
none pending or failing. Greptile gave that commit 5/5 with no findings
or unresolved threads.
- Paperclip Review spent over 23 minutes preparing its runtime without a
provider assessment. Its run was cancelled for recovery and ended
neutral (review execution did not complete). This neutral result is not
a code assessment. No check policy or branch protection was changed.
- The full local Vitest attempt first hit embedded PostgreSQL startup
failures. Running the unchanged `initdb --version` and `postgres
--version` independently showed missing native library symlinks. The
dependency had been installed with scripts disabled. Running its bundled
`hydrate-symlinks.js` restored startup; all 15 affected native-session
cases and all 112 AI connection tests then passed without application
code changes. The original full-suite attempt was stopped after the
setup failure and its log is retained. A complete local suite pass is
not claimed.
- The broker and executable are local fixtures. No live GitHub account,
credential change, or test PR is required by these regressions.

## Risks

The change affects credential delivery into OpenCode processes. It
reuses the existing bounded GitHub projection and accepts broker
authority only from explicit input. Each shell operation still uses the
normal broker, scope, identity, and revocation rules. Unrelated
controller secrets remain excluded. There are no database or
permission-policy changes.

## Model Used

OpenAI `gpt-6.1-sol`, through Codex, with xhigh reasoning and repository
tools/code execution. The session does not report its context-window
size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run the focused regression and affected suites locally and
they pass; the full local attempt and setup failure are disclosed above
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green (Paperclip Review is
neutral/incomplete, as disclosed above)
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-15 16:15:38 -05:00
Devin FoleyandPaperclip 0b62c736a0 feat(connections): add generic Daytona API and Toolbox access (#16483) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Apps connections give agents controlled access to external services.
> - Daytona provides sandbox, file, command, Git, and account APIs.
> - Operators need to assign this access without exposing a server
credential.
> - A small task-specific tool list cannot serve general Daytona work.
> - This pull request adds the published HTTP actions through the
existing connection gateway.
> - Operators keep control of identity, agent access, and each action.

## Linked Issues or Issue Description

**Problem or motivation**

Paperclip has a Daytona sandbox provider for its own execution
infrastructure. It lacks a normal Apps connection for agents to manage
an operator's Daytona account. Reusing the server's provider credential
would bypass normal connection ownership and action permissions.

**Proposed solution**

Add a Daytona connection with personal or shared vault credentials.
Generate its action catalog from the official platform and Toolbox API
specifications. Use the existing run-bound gateway, grants, action
policies, approvals, revocation, and audit paths. Sandbox IDs come from
Daytona; tools need no Paperclip task or lease ID.

**Alternatives considered**

A four-action diagnostic subset cannot support general provider work.
Launching the provider CLI would expose host filesystem behavior. Native
OAuth is not included because a supported third-party registration path
was not verified. The native method uses an organization API key.
Existing Composio setup is a separate possible OAuth route and is not
qualified by this change.

**Roadmap alignment**

This extends the existing Apps and Connections provider coverage in
`ROADMAP.md`. It uses that system's credential and permission model.
GitHub searches found no existing Daytona Apps connector PR or issue.

## What Changed

- Add the Daytona catalog entry, provider source override, official
theme logos, permission review, and setup documentation.
- Generate 274 HTTP actions: 169 platform and 105 Toolbox operations.
Record two WebSocket upgrades separately; do not advertise them as HTTP
tools. Record source URLs and hashes.
- Add provider-focused execution for sandboxes, snapshots, volumes,
files, Git, commands, computer use, and account resources. Provider
authority still limits which endpoints accept an organization key.
- Validate action arguments. Resolve Toolbox targets through the fixed
platform API. Check sandbox identity and organization before using an
approved Daytona proxy. Keep normal guarded network requests and vault
resolution.
- Transfer file bytes without reading or writing host files. Share one
caller deadline across prerequisite reads, dispatch, and response
reading. Bound requests and responses. Do not automatically retry writes
with unknown outcomes. Redact the selected credential from results and
errors.
- Classify credential access, access changes, interactive input, command
execution, file permissions, and deletion (including TTL and
auto-delete) as destructive actions. Keep new and changed actions under
catalog review.
- Upgrade saved catalogs and retire the earlier diagnostic names. Hide
retired actions from the UI and agent discovery even when provider
refresh fails. Preserve existing action choices and historical records.
Reject stale same-name contracts during agent discovery and dispatch
before a board refresh.
- Add database-backed gateway and catalog-upgrade regressions, schema
generator tests, a browser setup check, and both capability map updates.

## Verification

- Passed on the review-fix head: 57 focused Vitest cases (18 Daytona
database/client cases, five HTTP boundary cases, and 34 shared catalog
cases), four generator tests, `pnpm -r typecheck`, and `pnpm build`. The
five HTTP cases use a real local server through the pinned production
transport. The normal browser setup check and 17 generator/artwork cases
passed on the initial PR head; those UI and artwork paths are unchanged.
- Passed on this branch: brand asset validation, token gates, map
structure checks, and `git diff --check`. All latest-head GitHub checks
pass on `ea979ccb3a64dc9508807823540f935c4316ea35`. Apex is 5/5, with
all five findings addressed and zero unresolved threads. The branch has
no merge conflicts.
- The initial catalog-generation command could not find the external
research corpus. The supported `--definitions-only` mode regenerated the
definitions successfully. Unrelated generated drift was excluded.
- An earlier full local Vitest attempt on the preview branch failed when
bundled PostgreSQL could not start on macOS. After replay onto current
master, all three unchanged event-sequencing cases pass with bundled
PostgreSQL; the earlier startup cause remains uncertain.
- The first full-suite command on this branch used my external test
database override. Chat fixtures require a current migrated schema and
failed because `agents.lifecycle_state` was absent in that disposable
database. A focused comparison reproduces failure with the override and
passes with normal setup. That attempt was stopped, and the full suite
restarted without the override. Focused Daytona cases use their own
migrated PostgreSQL 16 databases.
- The clean local full-suite attempt recorded embedded PostgreSQL
startup failures in two unchanged native-runtime cases and two failures
in unchanged Git fixtures. It was stopped after these failures were
isolated; no full local suite pass is claimed. The 15 matching
native-runtime cases pass separately. The Git cases fail with the local
`init.defaultBranch=main` setting and pass with `GIT_CONFIG_COUNT=1
GIT_CONFIG_KEY_0=init.defaultBranch GIT_CONFIG_VALUE_0=master` limited
to the test command. No global Git setting was changed. Full Linux CI is
green.
- Fixtures use real database persistence and run-bound gateway sessions
with provider responses replaced by fixtures. They cover personal/shared
vault ownership, reads and writes, file bytes, Off policies, revocation,
organization mismatch, untrusted proxy rejection, output bounds, saved
catalog upgrades, failed refresh, and permission preservation.
- The first Apex review found five valid issues: multipart transport
compatibility, destructive classification, caller deadlines, audit
filters, and stale saved contracts. All five were fixed with regressions
before the second review. A typecheck then found the nullable saved
schema hash; admission now accepts that stored type and rejects null as
stale.
- The browser check covers setup controls without submitting a provider
key. No real Daytona account, provider write, native OAuth flow, or
agent model run is qualified. Live provider read/write proof remains
pending a test account.

## Risks

- Daytona keys have broad Toolbox authority, including keys with no
selected resource scopes. Only assign this connection to trusted agents
and review its action permissions.
- Remote writes or commands can continue after a timeout. Calls do not
automatically retry an uncertain outcome.
- The bridge targets Daytona Cloud. Proxy hosts are restricted; an
unrecognized future regional host fails closed.
- Files and responses have an 8 MiB bound. Each action has one caller
deadline, capped at 30 seconds across all provider requests and response
reading. Interactive WebSocket sessions are not exposed as HTTP actions.
- The generated JSON snapshot exceeds 1,000 lines because it holds the
provider input contracts. Generator logic and execution code remain in
separate small modules. Updates require explicit regeneration and
catalog review.
- No database migration or lockfile change is required.

## Model Used

OpenAI Codex, GPT-6. The runtime identifies the model family but does
not expose the exact backend model ID or context window. Used reasoning,
repository inspection, code editing, shell tools, database verification,
and browser tests.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-15 14:05:26 -07:00
DottaandPaperclip 61831ba7b8 refactor: defer adapter launcher imports during workspace snapshots (#16492) 
Defer adapter launcher imports until a directory merge resolves its lock root. Snapshot-only workers retain the same hashes, modes, exclusions, and ownership checks.

Verification: 63 focused cases; full typecheck/build; exact-head full CI/E2E/security; Apex 5/5. Local native Daytona replies: 66.939s cold and 26.485s warm. Hosted staging and the cold latency target remain unverified.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-15 16:03:47 -05:00
Dotta f1d925b726 fix(issues): keep workspace restoration failures retryable (#16489) 
## Thinking Path

> - Paperclip manages agent work and its saved files.
> - Finished Git workspaces can expire and restore from a saved commit.
> - A task comment must restore a closed workspace before the comment is
saved.
> - Failed restoration returned HTTP 503 and hid the restoration
message.
> - The composer treats 5xx writes as uncertain, so it blocked the draft
instead of offering a clear retry.
> - This change returns a definite rejection before the write and
preserves the restoration message.
> - Users can restore repository access and retry the same draft.

## Linked Issues or Issue Description

Refs #16457 and #16476.

Live acceptance found this gap with a disposable expired workspace and
an unavailable upstream. The saved commit and unique-file control stayed
intact. No fallback workspace or comment was created. The UI still
showed an uncertain comment result because the route returned 503.

## What Changed

- Return HTTP 409 for a rejected workspace rebuild before the consuming
write.
- Preserve the service's safe restoration message in the response.
- Cover rejected comment, comment-update, and checkout paths.
- Document the response and draft retry behavior.

## Verification

- `pnpm exec vitest run
server/src/__tests__/issue-closed-workspace-routes.test.ts
ui/src/api/issues.test.ts`: 38 passed in 26.80s.
- Route tests assert that failed restoration does not save a comment or
update the task.
- Existing client controls preserve 409 rejections and still treat
network failures and 5xx writes as uncertain.
- `pnpm -r typecheck` and `pnpm build` passed. Full PR CI passed on
attempt 2: [run
38804449401](https://github.com/paperclipai/paperclip/actions/runs/38804449401).
The first attempt timed out in the unrelated fresh personal-AI
connection browser case while waiting for its first reply. Logs and
browser evidence were inspected and retained; only failed jobs were
rerun once, and that shard passed.
- Retained live fixture will verify the error, retry, exact saved
commit, and subsequent persisted work after deployment.

## Risks

Clients now receive 409 instead of 503 for this known pre-write
rejection. They must retry after repository access is restored. The
route uses the service's fixed messages, not raw Git errors. This change
does not alter restoration, authorization, cleanup, or genuine
uncertain-write handling.

## Model Used

OpenAI Codex, GPT-6 family, with code execution and browser tools. The
exact deployment model ID and context-window size are not exposed in
this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
 2026-10-15 15:59:31 -05:00
DottaandPaperclip 6943904bad perf: stage independent remote GitHub launchers concurrently (#16484) 
Stage at most three independent remote GitHub launcher files at once only when the existing effective concurrent-sync capability permits it. Preserve serial fallback, per-file locks and integrity checks, and settle every started upload before failure returns.

Verification: 119 affected tests pass (one existing Linux-only oracle skipped on macOS); full repository typecheck/build pass; same-sandbox real-Daytona component comparison preserves all bytes/modes; current-head full CI/E2E/Superagent/Snyk and Apex 5/5 pass. The full native timing target remains separate and is not claimed by this change.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-15 15:36:43 -05:00
DottaandPaperclip 676d3ec8d4 perf(ci): reuse package downloads and overlap runtime checks (#16474) 
## Thinking Path

> - Paperclip contributors wait for CI before delivering changes.
> - Build still finishes last after #16460.
> - CI already downloads the public npm graph while the build runs.
> - Bundled-package preparation ignores that cache and downloads the
same archives again.
> - Reuse the existing cache and overlap independent runtime checks
while preserving fresh installs, integrity, offline lifecycle and
admission verification.
> - Measure staging and the full workflow separately.

## Linked Issues or Issue Description

Refs #16460 and #16434. This is a focused follow-up to merged #16460,
now based directly on master.

**Current behavior:**
The Build workflow warms `PAPERCLIP_PUBLIC_INSTALL_NPM_CACHE`, but
`startBundledInstalls` inherits the host's default npm cache. Server
staging took 31.17 seconds in the latest passing run, before packing and
the clean consumer install.

**Proposed behavior:**
Pass the verifier environment, including the explicitly configured warm
cache, to the three existing staging children. Without an explicit
shared cache, keep the usual producer npm configuration.

**Alternatives considered:**
Copying an installed dependency tree would bypass normal fresh
installation. Adding jobs would add setup cost. This change reuses only
npm's integrity-checked download cache. It does not remove coverage or
change product behavior and does not overlap a roadmap feature. Open PR
title search found no equivalent cache wiring change.

## What Changed

- Initialize the existing consumer cache before package staging and
expose it to producer npm only when explicitly configured.
- Pass that environment to the existing bundled-install children.
- Extend the child orchestration check to verify cache propagation and
document the cache boundary.
- Start Pi setup after the offline lifecycle, in its own download
sandbox, while Codex and Grok probes run in separate offline sandboxes.
Join successful Pi setup before lock verification and Pi admission. Name
and clean up only the verifier-owned setup container.

## Verification

- 33 focused public-install, closure and child-orchestration tests pass
on Node 24.21.0.
- A real server dependency staging experiment used
`prepare-bundled-package.mjs`, its actual manifests, provider overrides
and release patches. Built application payloads were omitted from this
dependency-only measurement.
- Cold cache: **16.139s**. After the actual CI warm-up script:
**6.980s**. A fully warm producer cache took **6.127s**. These are local
sequential measurements, not a CI wall-clock claim; filesystem warmth
and registry timing vary.
- All **28,329 files/links**, including both lockfiles, were
byte-identical between the cold output and the actual-workflow-warm
output.
- The warm-up itself took 49.631s; CI already runs it during Rust
restore/build. This change adds no warm-up operation.
- First follow-up run [passed all 48
jobs](https://github.com/paperclipai/paperclip/actions/runs/38791277425),
and Greptile gave the cache-only head 5/5. Server staging fell **31.172s
→ 17.565s** and total staging **70.169s → 54.869s**. But complete CI was
**8m20s**, slower than the parent's 8m04s; longer Codex/Pi probe times
outweighed that saving. This is not a full-run speed improvement claim.
- The next revision overlaps the independent probes instead of
serializing Pi setup behind them. All 46 focused sandbox, staging,
closure and process-supervision tests pass, including failed background
children, parent exit and cancellation. Pre-retarget head `590873a44`
passes all 48 jobs and Greptile 5/5 with no review findings. After
#16460 merged, head `fe4f8d38e` incorporates master and retargets this
PR directly to master. Its diff remains only four
optimization/documentation/test files; 46 focused checks pass again.
Fresh full CI and review are pending. The unchanged parent
conflict-resolution run passed all 48 jobs in 8m04s:
https://github.com/paperclipai/paperclip/actions/runs/38789785045.

Final paired runs against the same updated master, both on
`m7i-flex.xlarge`:

| Measurement | Parent #16460 | This PR |
| --- | ---: | ---: |
| Bundled server staging | 22.220s | 14.202s |
| All package staging | 61.277s | 48.304s |
| Clean consumer install | 67.617s | 73.490s |
| Pi setup remaining wait | 25.428s (serial setup) | 20.719s |
| Build job | 385s | 368s |
| Build start after workflow creation | 23s | 136s |
| Entire workflow, including admission and aggregates | 452s (7m32s) |
552s (9m12s) |

[Parent
run](https://github.com/paperclipai/paperclip/actions/runs/38792983313),
[final
run](https://github.com/paperclipai/paperclip/actions/runs/38792986392).

Build execution improved by **17s (4.4%)** in this pair. The total
workflow did **not** improve: it waited 113 seconds longer to start
Build, and other lanes/aggregates also contribute. The overlapping setup
ran for 25.667s while Codex/Grok took 4.946s, leaving 20.719s to wait.
Actual installed Codex, Grok and Pi checks all passed with zero provider
calls. These are individual samples with filesystem/network/CPU
variance, not a claimed steady-state average. The earlier slower
cache-only run is retained above. No tests were removed and no jobs were
added.

## Risks

Shared-cache contention can reduce the benefit if warm-up is still
running. npm handles concurrent content-addressed cache writers and
checks integrity; misses still download normally. Two runtime containers
can now overlap and contend for host resources; the full CI run measures
that tradeoff. They retain separate network policies and per-container
bounds. Pi writes only its provider assets and a private receipt; the
other probes inspect Codex/Grok. No installed tree, lock, test result or
build output is reused. The consumer still starts empty and performs the
full lifecycle and runtime checks. Normal unconfigured producer cache
behavior is preserved.

## Model Used

OpenAI GPT-6 through Codex, with repository inspection, execution and
GitHub tools. The serving model suffix and context capacity are not
exposed to this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either linked existing issues or described the issue in
this PR
- [x] I have not referenced internal or instance-local Paperclip issues
or links
- [x] My branch name describes the change and contains no private ticket
identifier
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation
- [x] I have considered and documented the risks
- [ ] All Paperclip CI gates are green (prior head passed; rerunning
after master merge)
- [ ] Greptile is 5/5 with no open P2s, recommendations or follow-ups
(prior head 5/5; current head pending)
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-15 15:16:52 -05:00
DottaandPaperclip 3898f6a45f fix(native): honor duplex transport for GitHub callback bridges (#16455) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native remote runners use a session-owned broker for managed GitHub
operations.
> - That broker uses the existing authenticated sandbox callback bridge.
> - The native caller omitted the server-selected duplex flag. An
enabled and capable target still used file polling.
> - Warm sessions must observe a changed flag and channel loss before
the next admission.
> - This pull request forwards the existing flag and retires unavailable
or changed bridges before retained instruction preparation.
> - The benefit is less idle remote traffic without changing credential
authority or the default transport.

## Linked Issues or Issue Description

**What happened?**

A native GitHub callback bridge selected `queue_v1` even when its
sandbox target had duplex enabled and supported it. The file bridge
continues remote queue checks while the warm session remains available.

**Expected behavior**

An enabled, capable target should use the existing HTTP/2 bridge.
Disabled or unsupported targets and failed channel startup must retain
the authenticated file fallback. A changed server flag must take effect
at the next normal warm-session boundary.

**Steps to reproduce**

1. Create native GitHub access on a sandbox target with the
server-selected duplex flag enabled, verified duplex capability and a
working channel.
2. Observe the selected callback transport.
3. Before this fix, it is `queue_v1`. The added real-wire regression
expects `http2_v1` and fails on the old source.

## What Changed

- Forward the existing server-selected duplex flag through the native
GitHub broker.
- Record that transport selection on the broker. Retire the warm
provider and broker through the existing cleanup path when the flag
changes.
- Read live channel failure when checking broker readiness. Mark normal
session cleanup as an orderly completion.
- Include the selected flag in instruction preparation identity. Retire
a failed broker before adopting retained instructions, while preserving
existing ownership and cleanup fences.
- Add real local HTTP/2 credential-rotation and file-fallback controls.
Add native warm-session flag-change controls.
- Document selection, fallback and warm-session behavior.
- Scan CI secret expressions in one forward pass, including quoted
braces and escaped apostrophes. Preserve direct, bracket, multiline and
whole-context references while rejecting filename and literal-string
false positives.

## Verification

- Current head `41440b07ea45e9e882e36ed495a66e501c8dd4df` resolves the
actual CI-policy conflict with master
`307eb16c0c59abd9e0626fde9201939b262e7f5f`. It retains upstream's
stricter whole-expression credential guard, including quoted secret
text, and uses the forward scanner instead of a backtracking regex. All
**16 CI policy tests** pass, including the bounded 28- and
10,000-apostrophe controls in both policy modes. The reported old regex
still has its recorded two-second timeout (14 passed, one failed). On
this final merged source, **809 affected native cases** pass with zero
selected skips (65.20s). Full repository typecheck (31.44s) and build
(22.13s) pass. Frozen source hashes stayed unchanged. Fresh exact-head
full CI, configured security and completed Apex remain pending.

- Before master integration, the frozen native repair passed repository
`pnpm -r typecheck` passed (33.0s) and `pnpm build` passed (32.1s).
- Before the latest master integration, all **800 affected cases passed
with zero selected skips**: complete native executor (725), resume (50),
GitHub broker (16), environment duplex-gate (4), and heartbeat
instruction-copy (5) suites. The final run took 95.3s. Source hashes
stayed unchanged through these checks.
- The initial head passed 788 affected cases and full CI, but Apex
scored it **3/5**. It found a dead HTTP/2 broker could remain reusable
and a flag change could fail after retained instruction preparation.
Both findings required the new repair. Three added before-fix
regressions failed on channel availability, orderly close, and
preparation identity.
- The first repair run failed three new warm-instruction fixtures
because fake provider close left its recorded state ready. The corrected
fixture writes a suspended checkpoint without weakening the production
ownership guard. A separate real-wire run then failed two availability
assertions because its helper spread had frozen the readiness getter.
The helper now preserves the getter. All failed attempts remain in the
local evidence.
- One disposable real Daytona sandbox exercised both transports on the
initial transport patch through the production native broker and
callback bridge. The provider shell wrapper used the then-current
production declarations. During equal two-second idle windows, file mode
issued **9 remote commands** and HTTP/2 issued **0**. Both variants
rotated synthetic credentials between two run bindings and denied idle
requests. Cleanup deleted the sandbox.
- Daytona component setup took 3.64s for file mode and 4.74s for HTTP/2.
This is an idle-traffic reduction, not a startup-speed claim. That
component run predates the availability and preparation repair; it does
not qualify the repaired paths on Daytona. The new failure controls use
a real local HTTP/2 channel. The credential resolver was synthetic and
no model was called. Real GitHub-account grants, staging replies, and
the end-to-end latency target remain unverified by this component test.
- Repaired-head CI found a policy-test false positive: a broad
`secrets.` regex matched an upstream filename. The unchanged test
reproduced the failure, and the corrected version passed all 15
controls. Master later supplied a stronger direct/bracket check, which
we retained when resolving the merge conflict. Review then found that
formatted expressions could stop that check at a quoted brace. The new
reported-case baseline failed (one failed, 14 passed). The final check
treats quoted strings as complete tokens, respects expression
boundaries, and passes all 15 cases. The later master conflict also
added `toJSON(secrets)` controls. The old branch pattern failed that
whole-context case. Review subsequently found exponential backtracking
with escaped apostrophes; the final forward scanner retains the
quoted-boundary and whole-context controls and passes all 15 tests,
including the bounded long-expression regression. No workflow or
credential handling changed. All failed attempts remain recorded.

- The full local repository test suite was not rerun. [Prior-head full
CI](https://github.com/paperclipai/paperclip/actions/runs/38775801821),
including `ci / verify` and `ci / e2e`, passed on previous head
`e5f8bdb425081cf08110c0f7423f96609e405920`. All 52 check runs succeeded,
two optional Storybook jobs skipped, and legacy Snyk passed. Completed
Apex review is **5/5** on that exact head, with no unresolved review
threads.

## Risks

- Only an explicit server-selected opt-in can use HTTP/2. The existing
provider-capability, channel-startup and authenticated file-fallback
checks remain in the shared bridge.
- Toggling the flag or losing the HTTP/2 channel replaces an idle
retained provider at the next run boundary. Instruction preparation must
make the same retirement decision. Normal ownership, settled history and
cleanup checks remain in effect.
- Lower idle traffic may reduce provider/controller pressure, but no
staging latency improvement is claimed. This feature remains inactive
when the existing flag is off. Other preparation and finalization costs
remain.
- No schema, API, UI, credential-selection, tracing endpoint or
default-setting changes.

## Model Used

- OpenAI Codex, based on GPT-6, with tool-assisted reasoning and code
execution. The exact hosted snapshot ID and context-window size are not
exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-15 15:15:55 -05:00
DottaandPaperclip 307eb16c0c perf(ci): avoid duplicate extraction in public install checks (#16460) 
## Thinking Path

> - Paperclip manages AI agents for work.
> - Contributors wait for CI before they can deliver changes.
> - After #16434, Build is the longest CI job.
> - Its clean public npm install spends about 90 seconds installing
packed artifacts.
> - npm extracts local archives to read metadata before it installs
them.
> - Serve metadata from those same packed artifacts, then let normal npm
verify and install their bytes once.
> - Preserve the offline lifecycle and runtime checks, and measure the
complete workflow.

## Linked Issues or Issue Description

Refs #16434 and #16429.

**Current behavior:** The Linux Build check installs every staged
package by local tarball path. npm's file fetcher extracts an archive to
obtain its manifest. Installation then extracts it again. In the last
green run, Build took 408 seconds; consumer installation took 89.7
seconds. The full workflow took 8m06s.

**Proposed behavior:** Read manifests directly from the packed archives.
Serve those manifests and their SHA-512-bound tarballs from a temporary
registry on container loopback. Use normal npm installation into the
empty consumer. Other packages still resolve from the public npm
registry.

**Alternatives:** More parallel jobs would add setup and transfer work.
Skipping the install or runtime probes would remove useful coverage.
This change removes duplicate extraction instead. It does not add a
product feature or duplicate a roadmap item.

## What Changed

- Build a registry catalog from the actual tarball manifests and archive
hashes.
- Run the short-lived registry and npm in the existing unprivileged
download container. No host port is exposed.
- Preserve public dependency tarball URLs, archive integrity checks,
disabled initial lifecycle scripts, the offline rebuild, lock checks and
all Codex, Grok and Pi probes.
- Close the registry before offline verification. Keep native ARM and
Mac install qualification unchanged.
- Add real npm parity and failure tests. Record metadata preparation
time in the existing timing receipt.
- Fix an existing policy false positive: the filename
`docker-compose-secrets.test.mjs` is not a workflow credential. Inspect
GitHub expressions and retain rejection of Honeycomb keys, dot/bracket
secret reads and `toJSON(secrets)`.

## Verification

- 31 focused install-contract checks pass on Node 24.21.0. All 16
targeted CI profile-policy tests also pass.
- Real npm file and registry installs produce identical installed files,
bundled dependencies, optional dependencies and executable links. Both
defer lifecycle execution; offline rebuild works after the registry
closes and preserves the lock.
- npm rejects an archive whose gzip header changes while its extracted
payload remains valid. The CLI propagates the failure.
- The permanent real-npm parity test now includes non-catalog
`is-number@7.0.0`. Both paths install it from its official npmjs
tarball, and the test asserts the lock URL. Fetches have a 10-second
bound and no retries. This addresses the first Greptile finding. A
separate live probe also confirmed public provenance.
- The first local run hit the host sandbox's loopback `EPERM`; the same
checks passed with local listener access. An initial corrupted-gzip
negative fixture hit decompression failure before integrity
verification; the final fixture preserves valid gzip and specifically
proves integrity rejection. An old command-shape assertion was updated
for the new entry point without removing its ordering checks.
- The [first CI
run](https://github.com/paperclipai/paperclip/actions/runs/38778371043)
passed all test lanes and Build, but the existing credential-guard false
positive failed policy and both aggregates. Its 7m25s full duration is
**not** counted as a passing performance sample. The guard repair and
public-download regression are on the new head.
- The real Linux install passed all 18 packed packages, the lifecycle
sentinel, offline rebuild, unchanged lock, official Codex payload/lease
checks, Grok missing/present checks and Pi admission. The receipt
confirms `artifactTransport: loopback-registry`. No provider calls ran.
- Compared with the previous green run, consumer install fell **89.72s →
66.29s** (26.1%), including 1.87s to prepare metadata. Build fell **408s
→ 374s**. Both Build hosts were `m7i-flex.xlarge`; these are separate
runs, so registry/cache and upstream changes remain confounders.
- The base advanced since the previous 8m06s run, including runtime and
test changes. Treat that duration as historical context, not an
identical-source control.
- Pre-conflict head `d8f9de1c` [passes all 48 workflow
jobs](https://github.com/paperclipai/paperclip/actions/runs/38779549190),
including both required branch-protection checks. Greptile is **5/5** on
this head; its review thread is resolved. The non-required Superagent
Security Scan is still running.

| Measurement | Previous green run | Final green run | Observed change |
| --- | ---: | ---: | ---: |
| Full CI, workflow creation through final aggregate | 8m06s | **7m35s**
| **31s shorter (6.4%)** |
| Build job execution | 408s | **389s** | **19s shorter** |
| Clean consumer install, including catalog preparation | 89.72s |
**69.69s** | **20.03s less (22.3%)** |

The final catalog preparation took 1.87s and is included in the install
figure. Both Build hosts were `m7i-flex.xlarge`. Build began 42 seconds
after workflow creation instead of 53 seconds in the historical run, so
faster queue/setup admission explains part of the 31-second total.
Upstream code also advanced; these measurements are not a controlled
causal estimate of the complete gain. The two candidate install
measurements (66.29s and 69.69s) support the narrower reduction in
install work. No tests or runtime probes were removed.

- Conflict-resolution head `ea8287b81` [passes all 48 workflow
jobs](https://github.com/paperclipai/paperclip/actions/runs/38789785045),
including both required gates. Greptile re-reviewed this exact head at
5/5, with the sole earlier thread resolved. The conflict was the
independent filename-policy fix; the stronger expression guard is
retained. This run took **8m04s** overall, Build 423s, consumer
installation 80.56s and bundled-server staging 31.17s. This confirms
meaningful run variance: the earlier 7m35s is a sample, not a guaranteed
new wall-clock duration.

- A second independent master update touched the same credential test.
Final head `1c740b73e` resolves that conflict while retaining the same
stronger guard. [All 48 jobs
pass](https://github.com/paperclipai/paperclip/actions/runs/38792983313);
Greptile re-reviewed this exact head at 5/5. This run took **7m32s**
overall, Build **385s**, consumer install **67.62s**. Auto-merge is
enabled but GitHub requires code-owner approval.

## Risks

The initial install now uses registry package specs instead of file
specs. The parity test checks the resulting installed bytes. Full CI
must still exercise real bundled provider graphs and all offline probes.
The temporary consumer lock contains loopback artifact URLs; it is not
exported or reused for another installation. Native release consumers
retain their existing artifact install path. Queue, host and cache
variance limit attribution from separate workflow runs.

## Model Used

OpenAI GPT-6 through Codex, with repository inspection, execution and
GitHub tools. The serving model suffix and context capacity are not
exposed to this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either linked existing issues or described the issue in
this PR
- [x] I have not referenced internal or instance-local Paperclip issues
or links
- [x] My branch name describes the change and contains no private ticket
identifier
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation
- [x] I have considered and documented the risks
- [x] All Paperclip CI gates are green (48 jobs and both required gates
on conflict-resolution head)
- [x] Greptile is 5/5 with no open P2s, recommendations or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-15 14:52:15 -05:00
DottaandPaperclip 2aca41464b fix(workspaces): preserve compaction and expire standard Git bases (#16476) 
## Thinking Path

> - Paperclip manages AI agents and their task workspaces.
> - Finished tasks can release reproducible Git files and restore them
later.
> - Unique work must stay on disk, while regenerable dependencies can be
removed.
> - Linux Git templates include an empty legacy directory that blocked
base cleanup.
> - The new expiration branch also skipped the existing guarded
dependency cleanup.
> - This change accepts the standard empty directory and retains that
cleanup order.
> - Private bases can expire, and protected workspaces still shed
regenerable files.

## Linked Issues or Issue Description

Refs: #16457

Live acceptance of #16457 removed a worktree but retained its private
base because Linux Git created an empty `.git/branches` directory. A
pending-base dry run then tried to resolve the already removed worktree
and hid the preservation reason.

A local integration comparison also showed that enabling Git expiration
skipped existing dependency compaction. It deferred pending startup
compaction by an hour. Both failures were reproduced before the fix.

## What Changed

- Accept `.git/branches` only when it is an empty real directory.
Preserve populated entries, files, and symlinks.
- Inspect pending base cleanup without resolving an absent worktree.
- Run existing guarded compaction before Git expiration. Exclude already
closed rows from compaction and retain its pending deadline.
- Keep failed expiration checks from reaching legacy whole-workspace
deletion. Existing compaction remains independent of Git expiration
dry-run mode.
- Extend real Git/PostgreSQL tests and retain the acceptance failures in
the delivery state note.

## Verification

- Linux-template baseline: private-base expiration failed before the
fix. The first sandboxed attempt could not bind PostgreSQL; the
authorized retry reproduced the behavior.
- Compaction baseline: two regression cases failed in 7.67 seconds. They
reproduced skipped dependency cleanup and the lost startup deadline.
- After the fixes, `pnpm exec vitest run
server/src/__tests__/execution-workspace-compaction.test.ts
server/src/__tests__/git-workspace-expiration.test.ts
server/src/__tests__/terminal-workspace-work.test.ts` passed all 49
cases in 43.79 seconds.
- Controls cover unique work, active runs, unfinished descendants,
previews, concurrent reopening, interrupted removal, exact-commit
restoration, and retryable upstream failures.
- Full `pnpm -r typecheck` and `pnpm build` passed for the final commit.
All exact-head CI and security checks passed. Greptile is 5/5 with all
findings resolved.
- Final staging acceptance will let the restarted worker resume the
retained base cleanup, then reopen normally at the recorded commit and
persist subsequent work. The base has not been manually removed. The
superseded preview build was cancelled before dispatching a replacement.

## Risks

- Legacy branch definitions can hold unique data. Only an empty real
directory qualifies for deletion.
- Compaction preserves its existing lifecycle guard and
regenerable-directory allowlist. It cannot fall through to legacy
deletion after an expiration failure.
- Already closed workspaces must only resume their recorded expiration;
they do not enter compaction.

## Model Used

OpenAI Codex, GPT-6.1-sol, with reasoning, code execution, Git,
database, and browser tools. The exact context-window size is not
exposed by this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-15 14:36:00 -05:00
DottaandPaperclip c53bdd9eb3 perf: capture sparse checkpoint files in bounded batches (#16473) 
Capture sparse checkpoint files in bounded batches while retaining content and mutation checks. Prevent inserts after SQLite automatic rollback and drain active readers before database close.

Verified: 209 affected cases, final six capture controls, repository typecheck/build, real Daytona helper validation, and exact-head full CI/E2E, security and Apex 5/5. The matched component median saves 1.05s; the whole-agent latency target remains open.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-15 14:32:44 -05:00
DottaandPaperclip 5ec93dae01 fix(connectors): avoid redundant run preparation discovery (#16461) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Run preparation selects connector skills and tools under current
permissions.
> - An assignment check can build an unrelated company tool catalog and
wait for a shared discovery slot.
> - Native startup repeats assignment discovery to obtain static tool
definitions.
> - Slack also locks and reconciles an unchanged catalog, then repeats
task authorization.
> - This pull request removes that redundant startup work and preserves
live checks at tool execution.
> - The benefit is less preparation work, less contention, and unchanged
permission and approval enforcement.

## Linked Issues or Issue Description

Related: #16454 (detailed preparation tracing), #16425 (workspace
preparation performance).

**What happened?**

Connector configuration added avoidable work to run preparation. Browser
Use assignment authorized the whole company catalog, including other
connector families. Four connector resolvers ran in series. Native
controller startup repeated all assignment resolution when any connector
skill was pinned. Slack reconciled an unchanged catalog under an
advisory lock and checked task authority twice.

**Expected behavior**

Prepare only the connector data needed by the run. Reuse static tool
definitions from pinned skills. Preserve current company, assignment,
identity, profile, policy, grant, and approval checks when an operation
executes.

**Steps to reproduce**

1. Configure Browser Use and an unrelated MCP catalog with 500 tools for
one company.
2. Resolve Browser Use resources for an authorized task. The old path
reads and authorizes the unrelated catalog.
3. Occupy both full discovery slots. The old assignment check waits for
those slots.
4. Resolve a linked Slack task repeatedly with an existing current
catalog. The old path still acquires a catalog lock and repeats
authority checks.

**Paperclip version or commit**

Base: `68a09dcb5f50e3fb6ceaaeac2706d96db97fc9b3`.

**Deployment mode**

Observed in hosted native runs. The local reproduction uses a source
checkout and real embedded PostgreSQL.

## What Changed

- Restrict Browser Use catalog reads and policy decisions to active
Browser Use connections. Preserve guest restrictions, reviewed catalog
filters, live grants, and deterministic resource order. Keep assignment
checks out of the full discovery queue.
- Resolve the four independent connector families concurrently. Wait for
outstanding work to settle before propagating a failure, and retain
registration order.
- Register native direct connector tool definitions from pinned
connector skills. Keep live assignment resolution and governed execution
on every tool call.
- Compare Slack catalog hashes before reconciliation. Current catalogs
skip the catalog transaction and lock. Check task authority once after
current profile checks. Initial setup and upgrades retain locked
reconciliation and checks before and after it.
- Precompute immutable bundled Slack descriptor hashes. Preserve
disabled entries, profile state, removed bindings, and review metadata.
- Document the startup paths and add focused contention, revocation,
catalog upgrade, and failure-settlement regressions.
- Repair a pre-existing CI guard false positive: the filename
`docker-compose-secrets.test.mjs` matched `secrets.`. Check GitHub
secret expressions instead, with dot, bracket, and whole-context
controls. No workflow changes.

## Verification

- Conflict resolution: merge `b1566ca494f4af6e8d4418fa37393c5caee02582`
from master into head `55140b1609ae7bd21a038eb75a48c2683299e4b4`.
Combine both secret-expression guards and regression cases. Preserve
current workspace, native recovery, and Daytona changes. All 467 policy
tests, 787 focused native/connector/Browser Use tests, and server
typecheck passed locally after integration. Fresh full CI passed all 55
checks on this head, with two optional Storybook jobs skipped. Greptile
reviewed this exact head at 5/5 with no unresolved threads.

- `pnpm -r typecheck` — passed.
- `pnpm build` — passed.
- Focused native startup, connector concurrency, and AgentMail suites —
763 tests passed.
- Complete Browser Use suite — 33 tests passed. Focused Slack catalog,
linked task, retained provenance/revocation, and pinned native tool
tests passed. The complete runner tool authority suite passed (26
tests). The 1,337-case chat suite passed 1,335 tests and had two
failures: the original new transaction-count assertion and a 15-second
timeout in the publication-batch case. Both cases passed an isolated
rerun. The publication timeout occurred during concurrent local checks;
the passing rerun does not erase it. CI must still pass the chat shards.
- The first focused run found two incorrect new assertions: policy
evaluation performs a second catalog lookup, and an explicit allow
policy overrides a profile default. The tests now distinguish the scoped
catalog query and revoke the governing policy. The focused Browser Use
rerun passed all 33 cases after these assertion fixes.
- Full-suite test verification used the CI shards of `pnpm test:run`.
Local runs focused on the changed boundaries; the complete repository
test command was not duplicated locally.
- `node --test '.github/scripts/tests/*.test.mjs'` — all 467 policy
tests passed. The CI failure reproduced on the unchanged base guard
before this correction. Actual GitHub secret expressions still match the
guard.
- CI on the final head passed all chat shards, typecheck, build, Runner
checks, browser shards, and other test lanes. One existing
workspace-runtime case hit `EADDRINUSE` for a randomly selected test
listener in server shard 1/12 (1,162 tests passed in that shard). The
shard passed its single retry on the same SHA with no source changes.
The first failure remains recorded here. Previous reviewed head
`7191ac40691ddb1ef38b86665c95cc14c57ea390`: all 55 checks passed, two
optional Storybook jobs skipped, and no pending or failed checks.
Greptile is 5/5 on this head with no unresolved review threads. The
external security scan passed.

Local before/after measurements use the old and new implementations
against the same fixture. Both paths are warmed. Seven samples alternate
execution order. The database is real embedded PostgreSQL. No latency is
injected. These measurements do not establish hosted speed.

| Local path | Before median (range) | After median (range) | Work
reduction |
| --- | --- | --- | --- |
| Browser Use assignment with 500 unrelated MCP tools | 44.020 ms
(33.994–107.328) | 26.502 ms (20.385–99.037) | 26 → 22 statements;
unrelated catalog and discovery queue removed |
| Slack assignment with current catalog and linked responsible user |
49.723 ms (43.624–80.653) | 22.141 ms (19.781–30.818) | 5 → 2 database
transactions; no catalog lock on the current path |

The contention regression fails against the old implementation after a
bounded five-second wait, then passes against the optimized
implementation. It occupies the real global discovery scheduler and
verifies that Browser Use assignment completes without releasing those
slots. It records catalog SQL to verify connection scope. Revocation
cases remove policy, catalog availability, and grants between calls.
Existing Slack integration cases exercise retained provenance, removed
profile bindings, identity revocation, approved writes, and current
responsible-user restrictions.

## Risks

- Pinned native tool definitions can remain visible after access is
revoked. They confer no authority. Invocation rechecks current
assignments and fails closed before an operation executes. Newly
available connector families still require a newly admitted skill
selection.
- Concurrent resolution can increase instantaneous database reads to
four connector families per configuration call. It is a fixed bound.
Outstanding work settles before a failed preparation returns.
- Slack skips reconciliation only when all bundled version hashes and an
existing profile match. Missing or changed definitions still use the
existing locked path. Operator revocations remain effective.
- This PR addresses connector preparation. Workspace transfer, provider
startup, copyback, and queue delay can still keep total latency above
the user's goal. Hosted improvement needs a deployed before/after trace.
- No schema, lockfile, workflow, telemetry event, or authorization cache
changes.

## Model Used

OpenAI GPT-6 (Codex), with reasoning, repository inspection, code
execution, and GitHub tools. The runtime does not expose the exact
serving model ID or context window.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-15 14:08:01 -05:00
DottaandPaperclip db11d899cf fix(native): recover remote sessions while preserving warm runs (#16451) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native agents keep remote processes and conversation history across
warm turns.
> - A completed turn can leave handoff authority only in controller
memory.
> - Losing that controller can block a follow-up even while its sandbox
survives.
> - This pull request verifies durable ownership and reconnects the
exact prior process before handing it to the next run.
> - Healthy follow-ups stay warm. Recovery preserves saved work without
replaying a completed turn.

## Linked Issues or Issue Description

**What happened?**

A remote warm turn can finish without suspended backup authority. A
replacement controller rejects that prior owner even when its exact
process and completed checkpoint remain. A single unavailable identity
probe can also stop an otherwise valid reattachment.

**Expected behavior**

Keep healthy follow-ups warm. Recover a completed session after proving
that its controller stopped and that the successor holds current
authority. Reuse compatible processes; otherwise save and resume the
conversation. Retry unavailable identity reads within one shared
deadline.

**Steps to reproduce**

Use `doc/testing-native-remote-continuity.md`. Its Linux cases execute
real turns through PostgreSQL, TLS ingress, and the Rust runner. They
kill a separate controller process, recover old checkpoint metadata, and
change process configuration. The provider is deterministic.

Related: #16432 and #16441 fix live warm-owner handoffs. #16341 adds
recovery diagnostics. #14695 covers provider process retention.

## What Changed

- Preserve warm process reuse between healthy remote turns.
- Verify a committed prior result, stopped controller, matching
checkpoint and sandbox, and current task/coordinator ownership before
recovering an orphaned warm session.
- Reconnect through the prior run route, then use the existing
authenticated run attachment to change authority.
- Persist a process-configuration digest. Close and save recovered
sessions when old or changed configuration cannot support reuse.
- Retry unavailable state, marker, and process reads within one
30-second deadline. Keep signals and definitive identity failures
single-attempt.
- Record closed recovery diagnostics, including unreadable or malformed
saved state.
- Retire the recovery-only ownership check after verified attachment, so
later warm turns use their current authority.
- Add real-process Linux regressions and refusal controls.
- Expose the existing source-only file-recovery form for retained native
holds, with the same board authority, evidence, and exact stop-receipt
requirements as legacy holds. Recording recovered files does not change
task status or replay a run.

## Verification

- `pnpm -r typecheck` passed; the server check was repeated for the
third-turn fix.
- `pnpm build` passed after the final UI change. The UI typecheck, token
gates, and map checks also pass.
- Four Linux continuity cases passed: warm reuse, controller loss,
legacy checkpoint metadata, and changed configuration. Each creates one
conversation and starts exactly three turns, including a healthy
follow-up after recovery. The fixture uses expired prior leases; the
third turn keeps the runner and provider processes.
- Greptile caught the retained recovery gate; the third-turn regression
now covers it. An initial test omitted the heartbeat’s saved input and
left a test process after failure; the fixture was corrected and its
disposable container reset before the passing run.
- The current focused executor/restart group passes all 749 tests,
including the original failure classifications.
- Native file-recovery and UI tests pass (52 tests), including blocked
tasks and both native and legacy holds.
- After merging current master and retaining its equivalent CI
expression check, the CI policy tests pass (15 tests) and all four
three-turn Linux continuity cases pass again on
`dec418ae55ff21b4fb067132613658cce445f181`.
- `pnpm test:run` completed its general server group with 20,572 passes,
103 skips, and three failures: the skill-audit child process and two
HTTP test timeouts. The same three suites then passed in isolation (89
tests) without code changes. That local command stopped before later
groups; the complete final-head CI, including general, serialized,
Runner, build, and browser gates, is green.
- Greptile reviewed the combined head
`dec418ae55ff21b4fb067132613658cce445f181` at 5/5 with no actionable
findings; both earlier findings are resolved. All final-head CI gates
and the security scan passed. Staging deployment and retained-incident
disposition remain deployment acceptance work.

## Risks

- Recovery requires exact durable evidence. Live conflicting
controllers, replacement sandboxes, Stop/pause, and uncertain effects
must remain blocked.
- After controller loss, tool-contract or process-configuration changes
can restart the provider. Normal healthy follow-ups retain their
processes.
- The new idle-recovery path has Linux fake-provider evidence. Live
Codex/OpenCode and retained quarantined-state recovery still need
staging qualification.
- No database schema, telemetry, or runner protocol changes.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository inspection, code
execution, and browser tools. The session does not expose a more
specific model version or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (targeted checks and
isolated reruns; full-run limitation recorded above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green on
`dec418ae55ff21b4fb067132613658cce445f181`
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-15 13:26:09 -05:00
DottaandPaperclip 59a70dc9e2 fix(workspaces): expire reproducible Git files and restore exact commits (#16457) 
## Thinking Path

> - Paperclip manages AI agents and their task workspaces.
> - Isolated Git worktrees and private repository copies remain on the
host after tasks finish.
> - These files can fill the disk even when every commit is available
upstream.
> - Cleanup must retain unique files, local Git state, live work, and
recovery data.
> - This pull request records and verifies an exact-commit restoration
recipe before expiration.
> - Normal reopening restores that commit even after upstream advances.
> - The benefit is lower host disk use while finished tasks remain
available for later work.

## Linked Issues or Issue Description

Refs #10285 and #2022. Follows #16433 for skill-cache expiration.
Related #11936 provides separate host-local cleanup tooling; this change
extends the existing company-scoped lifecycle worker and normal
workspace reopening. ROADMAP.md does not specify a competing
implementation.

## What Changed

- Add opt-in expiration for verified Paperclip-owned Git worktrees after
their whole task tree has been done or cancelled for 24 hours.
- Preserve active and scheduled runs, previews, pending reopening,
unresolved recovery, dirty files, unknown ignored files, and unsupported
or unique Git state.
- Verify an authenticated fetch in a fresh object store. Save repository
identity, exact commit, branch, and ownership before any deletion.
- Rename eligible files into recorded managed trash. Resume interrupted
cleanup under source and lifecycle locks.
- Expire private base clones only after consumer, file, reference,
reflog, and object checks pass. Preserve shared project base clones.
- Rebuild missing bases and worktrees through normal reopening at the
recorded commit. Return an explicit retryable error when repository
access fails.
- Add independent workspace/base switches, dry-run reports,
reclaimed-byte counts, and operational documentation. No table, public
API, UI, or recovery-retention change.

## Verification

- Real local bare remote plus Postgres: create at A, advance upstream to
B, expire the worktree and private base, reopen at A, then make and
retain a subsequent commit.
- Final lifecycle suite: all 20 tests passed in 46.41 seconds at
`d72947a2e45be5d0c912debb03c46b15863b4798`. Review fixes cover
second-cycle expiration, newer-branch recovery, normal preparation after
restoration, branch adoption, and Git administration path controls.
Eleven existing branch reconciliation controls also passed. Covers
unique files and Git state, live descendants, scheduled runs, recovery,
previews, unauthorized access, upstream failure, concurrent reopening,
interrupted deletion, shared bases, and nested private worktrees.
- Final guard regressions: 3 affected tests passed. Existing
workspace/runtime/preparation/source controls: 328 tests passed in
238.69 seconds.
- Full repository typecheck and build passed locally. The full local
test attempt did not pass: initial native-session cases hit macOS
Postgres shared-memory exhaustion, source changes invalidated two
discovery cases, and the checkout disappeared near the end. The
final-code retry of the remaining skill and unrelated controls passed
all 103 tests. All checks passed on this exact head, including
typecheck, general and serialized tests, Runner checks, build, canary
dry-run, browser E2E, and security scans. Greptile reviewed the head at
5/5 with all findings resolved.
- Fixed a blocking CI policy false positive for
`docker-compose-secrets.test.mjs`. The test still rejects GitHub secret
expressions, including dot, bracket, and multiline forms. Its full
15-test suite passes.
- Map validation and `git diff --check` passed.
- Staging dry-run inspection, measured reclamation, and ordinary product
acceptance remain pending. No live acceptance is claimed.

## Risks

- Git restoration depends on continued upstream availability and
credentials. Cleanup skips unverified commits; restoration never
substitutes the default branch.
- File deletion requires multiple conservative proofs. Unknown or
unsupported state is retained and can limit reclamation.
- Source and lifecycle locks serialize preparation, reopening, and
expiration. Remote verification has a deadline but can delay work on the
same source.
- Start with both expiration switches disabled. When enabled, dry-run
remains the default. Rollback disables further deletion while retaining
restoration support.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository inspection, tool
use, and code execution. The exact backend model ID and context-window
size are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (focused tests; full local
limitation documented above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-15 13:09:59 -05:00