mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
ddfbbfff202dfba53b39b342ec4e8a5afbf584ae
2405
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
d4498ae7ac |
fix(runner): settle every recorded tool result and free sessions blocked by a closed turn's pending call (#21490)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents call Paperclip tools through the Runner. The Runner records
each call and returns the controller's result to the provider (here
OpenCode)
> - In chat-eval cohort 13, a `list_workflows` result was recorded but
never reached OpenCode. OpenCode waited about 290 s, and the call stayed
pending
> - The next run's `run.attach` then failed with
`durable_pending_tool_calls`, so the person's follow-up question never
got an answer
> - This pull request fixes the result shape that caused the rejection,
makes the Runner settle every recorded result, and lets the next run
settle a call that a closed turn left pending
> - The benefit is that a tool result always reaches the agent, and a
stuck call can no longer block every later turn
## Linked Issues or Issue Description
**What happened?**
Chat-eval cohort 13 (OpenCode,
`openrouter/deepseek/deepseek-v4.1-flash-1017`, master `df1b12ad8e`). In
`cases.case-history` and `cases.create-case`, the CEO called
`list_documents` (or `list_projects`) and `list_workflows` (call id 5)
together. The server answered `GET /companies/{id}/workflows` with 200
in 584 ms. The OpenCode MCP bridge diagnostic for call 5 says `outcome:
"unanswered"`, `written: false`, `handlerMs: 289038` (290723 in
create-case). The sibling call was answered in about 1 s. The run ended
with `runner_settlement_incomplete`: `semanticTools.pending` held call 5
and its `semantic_tool.result` delivery command had status `failed`. The
next question started a run whose `run.attach` was refused: `run.attach
requires a settled Codex provider session: durable_pending_tool_calls`.
The run failed in 17 s, and the question was not answered.
Cause: `executeNativeWorkflowTool` returned the Runner API result
unchanged: `{ ok: true, status: 200, operationId: "GET
/api/companies/{companyId}/workflows", ... }`. Every other API-backed
tool renames that field to `apiOperationId`. The Rust tool bridge
(`semantic_response_value`) reads an object with a boolean `ok` and an
`operationId` as a semantic envelope that must name this tool and call.
The REST route did not match, so `apply_result` failed with `semantic
result envelope does not match its provider call`. Nothing was sent to
the provider, and the call stayed pending forever. The red test
reproduces this exact error. The parallel call is not the trigger: the
sibling call settled normally, and the OpenCode MCP bridge answered
other parallel pairs in the same runs. `list_workflows` first appears in
cohort 13, so cohorts 11 and 12 did not hit it. All four workflow tools
had this shape.
**Expected behavior**
Every recorded tool result reaches the provider. When a recorded result
cannot be delivered, the provider gets an error at once and the call
settles. A call that a closed turn left pending does not block the next
run.
**Steps to reproduce**
1. Run an OpenCode agent with Cases enabled on the durable Runner.
2. Ask a question that makes the agent call `list_workflows`.
3. See the call wait about 290 s. Then ask a follow-up question and see
`run.attach` fail with `durable_pending_tool_calls`.
**Paperclip version or commit**
`master` at `df1b12ad8e`.
## What Changed
- `server/src/services/native-runtime/native-workflow-tools.ts`:
workflow tool results name the route `apiOperationId`, like case tools
and `call_api`.
- `provider_bridge.rs`: add `settle_undeliverable_result`. When the
bridge rejects a recorded result for a call it still holds, the call
settles with a bounded, non-retryable runner error. Conflicting or
unknown results still fail as before.
- `provider_backend.rs` (`deliver_semantic_result`): use that
settlement, deliver the error to the provider, and record a
`semantic_tool_result_undeliverable` diagnostic with the call,
operation, reason and the rejected result's digest.
- `provider_bridge.rs` and `provider_backend.rs` (`attach_run`): add
`expire_closed_turn_calls`. When nothing is live (no active or ambiguous
turn, no provider request waiting), `run.attach` settles calls that a
closed turn left pending as an expired failure with an unknown outcome,
and records `semantic_tool_result_expired`. The provider stopped waiting
when its turn ended, and the new run's controller never delivers the old
run's results. A call that a live turn still waits on still blocks
attachment. This follows the existing `tool_call_interrupted_by_restart`
rule: never execute the operation again, and record an unknown outcome
when no result can ever arrive.
- Docs: `doc/run-log-events.md` and
`packages/paperclip-runner/docs/durable-recovery.md` describe both
diagnostics.
## Verification
- Red/green, Rust (`cd packages/paperclip-runner/runner && cargo test -p
paperclip-runner-core --test codex_provider --
a_rejected_recorded_result run_attach_expires`):
- Before:
`a_rejected_recorded_result_still_answers_the_provider_and_frees_the_session`
fails with `semantic result envelope does not match its provider call`
(the production error).
`run_attach_expires_a_call_left_pending_by_a_closed_turn` fails with
`run.attach requires a settled Codex provider session:
durable_pending_tool_calls` (the production refusal).
- After: both pass. The fake provider logs `tool-response:failure`, so
it got an answer at once. The next `run.attach` succeeds.
- Parallel calls (`cargo test -p paperclip-runner-core --lib --
provider_bridge`):
`a_rejected_result_in_a_parallel_pair_still_settles_its_own_call` sends
the cohort 13 pair (`list_documents` + `list_workflows`). The rejected
call settles, its sibling keeps its own result, and the next turn
starts. `only_a_closed_turn_expires_its_pending_calls` checks that an
open turn expires nothing.
- Upgrade and mid-run cutover:
`run_attach_expires_a_call_left_pending_by_a_closed_turn` saves state
with a pending call after the turn ends, stops that Runner, and attaches
with a new Runner process from the saved state.
`durable_backend_settles_pending_tools_when_recovery_finds_the_turn_ended`
(unchanged) still passes: a late result in the same run still settles as
`settled_after_turn`.
- Red/green, server (`cd server && npx vitest run
src/services/native-runtime/native-case-tools.integration.test.ts -t
workflows`): before, `expected { ok: true, status: 201, … } to not have
property "operationId"`; after, 2 passed.
- `cargo test -p paperclip-runner-core`: 643 lib tests pass. One lib
test (`fenced_receipt_output_keeps_prefix_when_stdout_never_closes`)
failed once with `EPERM` while signaling a process group under parallel
load. It passes alone 3 of 3 and does not touch this change. The
integration test binaries pass.
- Paid narrow run for `cases.case-history` and `cases.create-case`: see
the comment below.
## Risks
- Previous-release story: no persisted shape changes. State saved by the
previous Runner with a call pending after its turn closed carries
forward. The new Runner settles that call at the next `run.attach`, and
the session continues (tested from saved state with a new process). A
previous-release Runner that receives the corrected `apiOperationId`
result delivers it normally, because the result is no longer read as an
envelope.
- Behavior change: a call left by a closed turn is now settled at the
next run's attachment, not kept forever. Its outcome is recorded as
unknown, and the operation is never run again. A provider turn that is
still live keeps the old refusal (`durable_pending_tool_calls`,
`provider_side_effect_pending`), and the controller still never replaces
the session.
- A recorded result that the bridge rejects now becomes an error for the
agent, not an endless wait. The agent is told the action may have
completed and must check the current state before repeating it.
- Latency: nothing is added before turn start. The attach-time check
walks the in-memory pending calls (usually none) and adds no I/O beyond
the existing state save. The rejected-delivery path now answers in the
same command instead of after the provider's ~290 s client timeout, and
the follow-up run no longer fails after 17 s.
## Model Used
- Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude
Code with tool use and code execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Refs #21398 (first release where `list_workflows` was called in this
eval), #19733 (the attach refusal), #20886 (earlier OpenCode delivery
fix).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
9b581d689b |
fix(chat): report an outcome once and create no run when an earlier reply gave it (#21489)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - In agent chat, a finished delegated task or a declined tool action wakes the chat agent so it tells the person > - #21401 stopped a second completion report when an earlier reply already gave it > - But the suppressed completion still created a zero-second heartbeat run, and declined tool actions still got a second report > - Run accounting must count only real turns, and the person must get each outcome once > - This pull request settles an already-reported outcome before it becomes a run, and applies the same rule to declined tool actions > - The benefit is one report per outcome, no phantom run, and no extra model call ## Linked Issues or Issue Description Refs #21401 **What happened?** Chat eval cohort 13 (OpenCode, `openrouter/deepseek/deepseek-v4.1-flash-1017`, master `df1b12ad8e`) showed three problems after #21401. 1. `questions-and-approvals.duplicate-answer`: the completion wake was suppressed, but it still created a 0-second run (`runtimeMode: legacy`, no usage). The bounded-native-runs check rejected that run. 2. `skills.studio-test`: the CEO's single reply reported the Studio result after it read the test task done (0 passed, 1 failed, links COM-2). The completion was then suppressed, but the eval waited for a separate completion run and timed out after 240 s. 3. `questions-and-approvals.decline-tool`: the person declined the tool review one second after the turn for their message started. That turn read the declined action and said so. The decline continuation, which waited behind that turn, then posted the same declined outcome again. **Expected behavior** Each finished task or declined tool action is reported once in the conversation. An outcome that an earlier reply already reported creates no heartbeat run. **Steps to reproduce** 1. In chat, delegate a short task, or start a governed tool read that needs approval. 2. While it waits, send a message that wakes the chat agent. 3. Let the task finish, or decline the tool review, while that turn runs. The turn reads the outcome and reports it. 4. See the completion wake become a 0-second run, or see a second decline report from the decline continuation. **Paperclip version or commit** master `df1b12ad8e` ## What Changed - `server/src/services/chat-completion-delivery.ts`: `settleReportedChatCompletions` marks each undispatched completion that an earlier reply reported as delivered, with that reply as its response. It returns true only when nothing else is owed. The outbox `deliver` also checks this before it requests a wake, so no wake and no run exist for an already-reported completion. - `server/src/modules/wake-queue`: a new transaction port, `skipAlreadyReportedChatWake`, runs in the release drain before a deferred wake is promoted. When the earlier reply already reported every outcome of the wake, the wake ends `skipped` ("Already reported by an earlier reply in this conversation") and no run is created. Wakes that carry queued comments or a failed-chat retry are never skipped. - `server/src/services/chat-interaction-outcome-report.ts` (new): the same structural rule for declined tool actions. A conversation agent run that lists its interactions records `chat_interaction.declined_read` for each declined tool action. The decline continuation is settled when the same agent replied in the conversation after the decline, from a run with that receipt, posted after the read. The rule covers only declined tool-action confirmations. Answers, plan reviews and accepted actions keep their own turn. - `server/src/routes/issues.ts`: `GET /issues/:id/interactions` calls the recorder for agent runs. - The run-start check from #21401 stays as a fallback for a completion turn that was admitted before the earlier reply landed. - `tests/runner-e2e/chat-catalog-domains/integrator-skill-studio.ts`: the Studio oracle (v5) accepts either the completion turn's reply or an earlier CEO reply as the delivery. The earlier reply counts only from saved state: no completion run exists, the replying run has a `chat_handoff.completed_read` receipt for the test task, the reply was posted after that read, it is the only such reply, and it links the saved result. No wording check is added. - `doc/run-log-events.md`: documents both read receipts and the skipped wake outcome. ## Verification - `cd server && npx vitest run src/__tests__/chat-completion-delivery.test.ts src/modules/wake-queue/adapters/chat-declined-outcome-reported.test.ts src/modules/wake-queue/application/use-cases.test.ts`: all pass. - Red: with the drain check and the outbox check disabled, 5 tests fail: "starts no completion wake or run ...", "settles a deferred completion wake without a run ... (true)", "settles the decline continuation with no run ...", and both "settles an already-reported chat wake without a run" cases. - Negative controls stay green: a linking reply without a done read, a decline the earlier turn never read, a non-tool-action card, and an accepted card all still run their continuation. - `npx vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/chat-catalog-domains/integrator-skill-studio.test.ts`: 34 pass. Red on the old oracle: "accepts the earlier reply the server took as the completion report" fails. New negative cases reject no done read, a read of another task, a reply before the read, a reply from another run, two replies, an extra completion run, and a missing task link. - A narrow paid eval run of `duplicate-answer`, `studio-test` and `decline-tool` on this branch follows in a comment. ## Risks - A completion turn admitted before the earlier reply landed (a narrow race) still ends as a short succeeded run with no model call, as in #21401. - If an earlier reply read the outcome but did not mention it, the person does not get a second message. The read receipt, the reply-after-read order and, for completions, the task link make this unlikely. - The decline rule covers only declined tool-action confirmations. Other interaction outcomes are unchanged. ## Model Used - Claude, claude-opus-5-5, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
890dcbdd66 |
fix(native): wait for a live warm session owner during a deploy overlap (#21627)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A deploy swaps servers with an overlap: the outgoing server still serves while the incoming server starts new runs > - A warm agent session stays open in the server process that ran the last turn, after that run finished > - The incoming server took over a warm session while the outgoing server was alive and still held it, because the finished run's finalization lease ends at commit > - The takeover then failed, the task went blocked with a Retry, and in one case a retry loop ran 27 times on the same sandbox > - This pull request makes the incoming server wait while the server that holds the warm session is alive, makes the outgoing server close idle warm sessions as soon as it leaves new runs to the replacement, and stops the retry loop when the promised fresh sandbox does not come > - The benefit is that a follow-up during a deploy finishes without a person, and no Runner ever gets two controllers ## Linked Issues or Issue Description **What happened?** A zero-downtime release-swap test (Gate B, wave B1, master |
||
|
|
d25667f370 |
perf(db): keep postgres.js connection queues proportional to live connections (#21601)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The server reads PostgreSQL through postgres.js, with a patched
driver in `patches/postgres@3.4.9.patch`
> - The driver keeps its pool connections in small FIFO queues (`open`,
`busy`, `full` and others) in `src/queue.js`
> - The `open` queue keeps an empty slot for every connection it hands
out until it is completely empty, and `remove` scans every slot
> - A busy pool always has some idle connections, so `open` never
empties. Each query then scans one slot for every query the process has
already run
> - The main thread of a long-running server therefore spends more time
on each query as the process ages
> - This pull request compacts the queue once empty slots outnumber
queued items, and makes `remove` search only the queued items
> - The benefit is that query dispatch cost stays flat for the life of
the process instead of growing linearly
## Linked Issues or Issue Description
**What happened?**
A hosted Paperclip server spends more of its main thread in the
postgres.js dispatcher the longer it runs. A CPU profile 14 minutes
after start showed the driver's `handler` (`src/index.js:350`) at 9.6%
self time, plus 3.8% more in the same inlined queue work. The server had
about 117 open sockets, a pool maximum of 500, and about 1,000
statements per second.
The cause is in `src/queue.js` (and `cjs/src/queue.js`). `shift` resets
its backing array only when the queue is completely empty. Otherwise it
leaves an `undefined` slot. `handler` takes a connection with
`open.shift()`. Then `move` calls `open.remove(c)` for a connection that
`shift` already took out, so `indexOf` scans the whole backing array and
finds nothing. When the query finishes, `move(c, open)` pushes the
connection back. The backing array grows by about one slot per query.
**Steps to reproduce**
1. Open a pool with more connections than the number of concurrent
queries (for example 24 connections and 8 concurrent queries).
2. Run 200,000 `select 1` queries.
3. Measure main-thread CPU per query for each block of 20,000 queries.
It grows from 30.6 µs to 48.7 µs.
4. Or run the same queue operations without a database: with 117
connections and 8 in flight, the cost per query is 16 µs at 200,000
queries and 151 µs at 1,000,000 queries.
**Expected behavior**
The cost to dispatch a query depends on the number of connections in the
queues, not on how many queries the process has run.
**Environment**
postgres.js 3.4.9 with the Paperclip patch, Node.js 24 and 26,
PostgreSQL 17/18, hosted and local servers.
## What Changed
- `patches/postgres@3.4.9.patch`: adds one hunk each to `src/queue.js`
(ESM, which the server loads) and `cjs/src/queue.js` (CommonJS). All
existing hunks are unchanged. The added hunk is:
```diff
remove: (x) => {
- const index = xs.indexOf(x)
- return index === -1
+ const at = xs.indexOf(x, index)
+ return at === -1
? null
- : (xs.splice(index, 1), x)
+ : (xs.splice(at, 1), x)
},
...
if (index === xs.length) {
index = 0
xs = []
+ } else if (index > 1024 && index * 2 > xs.length) {
+ xs = xs.slice(index)
+ index = 0
} else {
xs[index - 1] = undefined
}
```
The compaction copies only the queued items, and only after more than
1,024 shifts and after empty slots outnumber queued items. Its cost is
therefore amortized O(1) per `shift`. FIFO order, `length`, `push`,
`shift` (including `shift` on an empty queue) and `remove` (including
`null` for an absent item) keep the same results.
- `packages/db/src/postgres-queue.test.ts`: for both the ESM and
CommonJS builds, checks FIFO order, `length` and `remove` results. It
compares the queue with a plain array over 300,000 random operations
that cross the compaction threshold. It also runs 1,000,000 queries in
the pattern where `open` never empties, and checks that the slots
scanned by `remove` and the backing array size stay bounded by the live
connections.
- `packages/db/src/__fixtures__/postgres-queue-pool.mjs` and
`packages/db/src/postgres-queue-pool.test.ts`: run a real pool (24
connections, 8 concurrent queries, 20,000 queries) against embedded
PostgreSQL for both builds. They check that a dispatch scans no more
slots than there are connections.
- `packages/db/scripts/benchmark-postgres-queue.ts`: reports per-query
cost for the queue simulation (1,000,000 queries) and for the real pool
(200,000 queries, 10 blocks).
- `doc/DATABASE.md`: describes the fix, the tests and the benchmark
command.
- `pnpm-lock.yaml` is not changed. The lockfile refresh workflow updates
the patch hash, as it did after #21432.
## Verification
- Red before the fix: `pnpm --filter @paperclipai/db exec vitest run
src/postgres-queue.test.ts src/postgres-queue-pool.test.ts` failed 4
tests. The unit test scanned 13,340 slots (limit 125). The real pool
scanned 5,017 slots in its first 5,000 queries (limit 24). The semantic
tests passed before and after the fix.
- Green after the fix: the same command, plus
`src/postgres-receive-buffer.test.ts` and
`src/postgres-connection-recovery.test.ts`, passes 32 of 32 tests.
- `pnpm --filter @paperclipai/db exec tsc --noEmit` passes.
- `pnpm verify` exits 2 because the patch is a global CI input. Required
CI supplies the broad coverage.
- Benchmark: `pnpm --filter @paperclipai/db exec tsx
scripts/benchmark-postgres-queue.ts` on Apple Silicon, Node.js 26.4,
embedded PostgreSQL 18:
| Measurement | Before | After |
| --- | --- | --- |
| Simulation, µs per query at 200k / 400k / 600k / 800k / 1M queries |
16.0 / 50.4 / 81.3 / 128.4 / 150.7 | 0.070 / 0.070 / 0.071 / 0.073 /
0.072 |
| Real pool, main-thread CPU µs per query, first vs last 20k of 200k |
30.6 vs 48.7 (+59%) | 22.8 vs 14.6 (first block includes warm-up) |
| Real pool, CPU µs per query, blocks 2 to 10 | 29.6 rising to 48.7 |
13.3 to 15.9, flat |
## Risks
- Low risk. The change is limited to the driver's internal queue. The
results of every queue operation are the same, and the semantic test
compares them with a plain array over 300,000 random operations in both
builds.
- `remove` now searches from the first queued item. Slots before it are
always empty, so it cannot miss a queued item. Before this change,
`remove(undefined)` could remove an empty slot. The driver never calls
it that way.
- Compaction allocates a new array with only the queued items. It runs
at most once per 1,024 shifts, so memory use falls.
- The lockfile refresh workflow updates the patch hash in
`pnpm-lock.yaml` after merge, as it did after #21432.
- A possible follow-up is to offer the same fix to the upstream
postgres.js project. This pull request does not do that.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file editing, test execution) and extended
thinking.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
f452b449fd |
fix(recovery): stop re-checking settled native runs that cannot be released (#21590)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The server releases the process identity (`process_pid`,
`process_group_id`, `process_started_at`) of a finished native run when
it can prove that the runner process stopped (#17374)
> - A background queue (`settledNativeProcesses`) finds these runs and
checks them. A check that cannot finish ("skipped" or "pending") writes
nothing, so the run stays in the queue and every recovery hint checks it
again
> - A finished run that a replaced container recorded can never pass the
check, because only the host that recorded the process can prove that it
stopped. Each deploy adds more of these runs
> - On one hosted instance this loop ran about 10 checks per second and
about 55 SQL statements per second, and the server main thread was about
95% busy with only a few live runs
> - This pull request moves each "cannot make progress" reason into the
queue's SQL predicate, adds a growing backoff for "pending" runs, and
makes one check cost one locking statement
> - The benefit is that runs which cannot progress cost nothing until
something that can change the answer happens, and a run that becomes
releasable is still released at once
## Linked Issues or Issue Description
No public issue exists. Bug report:
**What happened?**
The settled native process recovery queue checks the same finished
native runs again on every recovery hint and every 30 seconds, forever.
With 300 such runs and one company-wide hint per second it ran 9 checks
per second and 84 SQL statements per second (9.4 statements per check,
2.2 ms of server CPU per check) on embedded PostgreSQL. A hosted
instance showed the same rates. The runs that never finish are: runs
recorded by a host that is gone (each redeploy starts a new container
with a new hostname), runs whose coordinator phase is not settled, runs
with no checkpoint and no provider evidence, and runs whose local
process is still alive (they defer 30 seconds with no limit). Startup
discovery (`discoverSettledNativeRunProcesses`) also opens one
transaction for every such run on every boot.
**Expected behavior**
A run that cannot make progress is not read again until something that
can change the answer happens. A run that becomes releasable (its phase
settles, provider evidence arrives, its own host checks it, its sandbox
receipt arrives, its process exits) is released promptly.
**Steps to reproduce**
1. Create a failed native run with a recorded local process identity
from another host (or an unsettled coordinator phase, or no checkpoint
and no provider evidence, or a live process).
2. Register the residual recovery work and send company-wide
`notifyLiveRunRecovery` hints.
3. Count the run's `for update` checks: on `master` every hint checks
the run again.
`server/src/__tests__/settled-native-process-rechecks.test.ts` automates
these steps.
**Paperclip version or commit**
`master` at
|
||
|
|
e8c79235ca |
perf(attention): scope run lifecycle retention hints and read only used columns (#21579)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The `attention-retention` queue archives old attention items (failed runs, reviews, questions, recovery actions, blocked tasks). A company hint makes it reload every candidate of the company, 100 per page, with eligibility reads for each page. > - Several run lifecycle producers sent that company hint: every agent status change after a run, the execution-control writes and legacy run terminalization when they name only a run, scheduled retries, and the Runner's task monitor tool. > - Each page also read whole rows it did not use: full `heartbeat_runs` rows (recent ones carry tens of KB of `context_snapshot` and `runner_profile_json`), full `issues` rows (descriptions), and the whole agent roster with its configuration. > - This pull request names the task graph or the source kind in those hints, and reads only the columns each eligibility decision uses. > - The benefit is fewer company reloads per run event and far fewer result bytes per reconstruction, with the same decisions. ## Linked Issues or Issue Description No issue exists. This describes the problem with the fields of `.github/ISSUE_TEMPLATE/enhancement.yml`. Stacked on #21552 (it builds on that pull request's change to `execution-control-notifications.ts`); review that one first. **What existing behavior does this improve?** The attention retention queue (`server/src/services/attention-retention.ts`) and the producers that wake it on run lifecycle paths. **Subsystem affected** Server background retention (`attention-retention.ts`, `retention-work-notifications.ts`, `execution-control-notifications.ts`, `heartbeat/agent-status.ts`, `heartbeat/retries.ts`, `native-runtime/paperclip-runner-tool-authority.ts`). **Current behavior** - `heartbeat/agent-status.ts` sends `{ companyId }` on every agent status change after a run (two places). `heartbeat/retries.ts` sends it for each scheduled retry. `writeExecutionControl` and `notifyExecutionRunWork` send it when the scope names only a run. The Runner's `set_task_monitor` tool sends it for one task. - On a local fixture with 1,500 open tasks and run churn, the queue loaded 47 company candidate rows per second from these hints. - `pageEvidence` read `select *` from `heartbeat_runs`, `issue_recovery_actions` and (through `getTableColumns`) `issues`; `reviewEligibility` read `select *` from `issues`; `interactionEligibility` read the full agent roster; the per-entry `check` repeated full-row reads. **Proposed behavior** - Agent status changes send `{ companyId, sourceKind: "agent_error_alert" }`. The status moves between `running`, `idle` and `error`. All three are invokable for blocker attention (`BLOCKER_ATTENTION_INVOKABLE_AGENT_STATUSES`) and for questions (`agent-eligibility.ts`). Failed-run attention only excludes terminated agents, and this path never sets that. So only agent error alerts can change. - Run-scoped execution-control writes, legacy run terminalization and scheduled retries use `notifyIssueAttentionRetentionWork` with the run id (the run's task, its parents and the tasks it blocks; company fallback past 100). This is the same helper as #21521. The task monitor tool scopes to its task. - `notifyIssueAttentionRetentionWork` matched the run's saved task id with `issue.id::text = …`. It now validates and casts the id, so the primary key is used, as #21552 does for recovery. - Narrow reads, same decisions: - runs: `id`, `agent_id`, `created_at` and the saved task id computed in SQL with the same rules as `readRunIssueId` (`issueId` unless it is absent or JSON null, then `taskId`; only non-empty strings). - recovery actions: `id`, `source_issue_id`. - issues for evidence and `visibleIssue`: the blocker-attention input columns plus `unblock_descriptor` and `blocked_transition_at`. - review issues: `id`, `company_id`, `status`, `execution_state`, `assignee_user_id`. - agent roster for question addressees: `id`, `company_id`, `name`, `reports_to`, `status` (the `AgentOrgRow` that `evaluateAgentInvokability` takes). **Reason and benefit** A run event no longer reloads every attention candidate of the company. A reconstruction no longer transfers whole run, task and agent rows. ## What Changed - `server/src/services/attention-retention.ts`: column lists for runs, recovery actions, issues and the roster; the run probe type replaces whole rows. - `server/src/services/heartbeat/agent-status.ts`, `heartbeat/retries.ts`, `execution-control-notifications.ts`, `native-runtime/paperclip-runner-tool-authority.ts`: scoped attention hints. - `server/src/services/retention-work-notifications.ts`: indexed lookup of a run's saved task. - `server/src/__tests__/attention-retention-work.test.ts`: two tests. ## Verification - `reads only the fields its decisions use from large runs, reviews and agents` (I saw it fail first): 50 failed runs, 50 tasks in review with large descriptions, 10 questions to a paused agent, about 40 KB of payload in each run's snapshot, profile and stdout and in the agents' configuration. It loads the three kinds with the full-size response capture (`full-database-response-capture.ts`). Before: `heartbeat_runs` 6,317,700 bytes, `agents` 84,336 bytes, `issues` 4,235,810 bytes. After: under the test bounds of 20,000, 5,000 and 60,000 bytes. The decisions are the same before and after: 50 failed runs, 40 reviews and 10 questions. - `scopes run lifecycle attention hints to the run's task graph and agent status to agent alerts` (I saw it fail first): `writeExecutionControl` and `notifyExecutionRunWork` with a run id send the hint for the task and its parent. `finalizeAgentStatus` sends the agent-alert hint. Before, all three sent `{ companyId }`. - `attention-retention-work` (57 tests), `attention-retention-candidates`, `attention-service`, `execution-control-work` and `heartbeat-agent-status-recovery` pass, each file run alone (in `server/`, against a dedicated local `_test` database). - `pnpm check:module-boundaries` passes. Not done in this pull request, and why: - I did not reproduce `syncItems` re-syncing unchanged rows from the retention `check`. On my fixture (300 stalled blocked tasks, 4,500 old recovery-action and blocker retention rows, run churn), `syncItems` ran 0 times in a 180-second window and `decision_retention` reads were 0 per second. `check` calls `syncItems` only for entries that are already due (older than the 90-day archive delay), and a due entry is archived and leaves the candidates. So I did not batch `syncItems` or skip checks of unchanged rows. - `readIssues` still selects the `issues` columns that blocker attention needs. I did not narrow `issueService.listBlockerAttentionBatch`. ## Risks - A producer that changes more than the run's task graph would now miss the wider effect. Company-level changes (agent configuration, company status, budgets) keep their company hints. The 90-day archive delay makes a missed reload cheap: the item is reloaded on the next hint that covers it. - The saved task id now comes from SQL. The SQL mirrors `readRunIssueId` for strings, absent keys and JSON null. A non-string `issueId` value now yields no task, as before. - No schema change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
3c3e6c2d75 |
fix(retention): stop rescanning unchanged pinned workspace snapshots (#21571)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Remote workspaces keep saved versions in the
`execution_workspace_snapshots` ledger, and a retention sweep retires
versions that no reference holds
> - Every execution notification wakes this sweep, and the sweep
examined every pinned version again on each pass
> - Some pins never change. A migration cohort supersession leaves
failed operations with `lastFailure` and no `retiredSave`, and these
hold their snapshots until a separate review releases them
> - On a busy server the sweep ran four statements per pinned row, many
times each second, on a main thread that was already saturated
> - This pull request records the reference that pins a version and
leaves that version out of the candidate query until that reference
releases
> - The benefit is that an execution notification costs one candidate
query, not one transaction per pinned row
## Linked Issues or Issue Description
No public issue exists. Bug report:
**What happened?**
A hosted instance ran four statements at 13.4 calls per second each: the
workspace read, the importing-parent read and the operation read in
`pinned()`, and the locked ledger row read. The sweep in
`server/src/services/remote-workspace-snapshot-retention.ts` caused
them. A pinned superseded row keeps `status = 'superseded'` with
`delete_after = null`, so the candidate query selects it on every pass.
Each pass examines up to 50 rows. A truncated page runs again after one
second. Each run-work notification (`notifyExecutionRunWork`) sets
`rescanRequested`, which starts the scan again. The sequence stopped at
the operation read because a failed operation with `lastFailure` and no
`retiredSave` pinned these rows.
`supersedeExpiredRemoteWorkspaceMigrationCohort` writes such operations,
and they hold their snapshots permanently by design (see
`doc/DATABASE.md`, "Workspace cohort admission proofs").
**Expected behavior**
A pinned version is examined again only when the reference that pins it
can have changed. An idle or busy server does not repeat reads for
unchanged pinned history. A released version is retired on its normal
deadline.
**Steps to reproduce**
1. Create 300 superseded snapshots. Pin each with a failed
`remote_workspace_migration` operation that has `lastFailure` and no
`retiredSave`.
2. Register the retention worker on a delivery coordinator and let the
first pass finish.
3. Send `notifyExecutionRunWork` about 13 times each second for 60
seconds.
4. Count statements with `pg_stat_statements`.
**Paperclip version or commit**
`master` at
|
||
|
|
3ae7aba643 |
perf(db): reuse dedicated connections instead of opening one per call (#21567)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server uses `withDedicatedDbConnection` (packages/db) when an action must hold one database session outside the main pool: advisory transaction locks, lock keepalive, and `pg_settings` reads for workspace lifecycle, finalization, source locks, skill caches and status cards > - Each call built a new postgres.js client, opened a new physical connection (TCP, TLS and startup through the pooler), built a new drizzle client with the full schema, and then ended all of it > - On staging the dedicated application name shows about 21 connection setups per second, and the main thread is pinned (event-loop utilization 0.965) > - This pull request keeps a small set of idle single-connection clients per owner and reuses them > - The benefit is that a dedicated action costs about 0.2 ms of main-thread time instead of 4 to 8 ms, with the same exclusive-session guarantee ## Linked Issues or Issue Description **What existing behavior does this improve?** `withDedicatedDbConnection` opens and closes one new database client and one new physical connection for every call. **Subsystem affected** Database client (`packages/db`) and the server services that take dedicated advisory-lock sessions. **Current behavior** Each call builds a new postgres.js client, opens a new connection (TCP, TLS and startup through the pooler), and builds a new drizzle client with the full schema. On staging (Neon statement statistics, 326 s window) `SET application_name='paperclip-workspace-finalization-lock'` ran 21.4 times per second. Only this factory uses that application name. In a 30 s staging CPU profile, drizzle `relations.ts` was 2.2% inclusive, `n:net` 5.8% and postgres.js construction 0.3%. Part of that is this per-call setup. **Proposed behavior** A caller still gets exclusive use of one session for the whole action. The session and its drizzle client are reused by later actions, so the per-call cost is the action's own queries, not a full connection setup. **Reason and benefit** The server main thread is saturated on staging (event-loop utilization 0.965). Reuse removes about 5 ms of main-thread work per call. **Breaking changes** None. The exported function keeps its signature. `closeDedicatedDbConnections` and `DEDICATED_CONNECTION_IDLE_MAX` are new exports. ## What Changed - `packages/db/src/client.ts`: each `createDb` owner now has one set of dedicated single-connection clients, shared with every private client derived from that owner (so nested dedicated actions use the same set). - An action takes an idle client (last in, first out) or opens a new one. It never waits. Callers hold advisory locks and nest dedicated actions, so a waiting queue could deadlock. - On success the client returns to the idle set if fewer than `DEDICATED_CONNECTION_IDLE_MAX` (4) are idle. Otherwise it is closed. - When the action throws, the client is closed and never reused. This covers a failed or cancelled transaction, a broken socket, and an interrupted lock wait. - When an action ends the client itself (the source-lock cancel path calls `$client.end`), the client is never reused. - Idle clients keep the owner's idle timeout and max lifetime, so postgres.js still closes idle physical connections after `DATABASE_IDLE_TIMEOUT_SECONDS`. - New `closeDedicatedDbConnections(db, timeoutSeconds)` closes the idle set and stops retention. Ending the owner `$client` does the same. A later action still works with a connection that closes on release. - `server/src/index.ts`: shutdown closes the dedicated set next to the other private pools. - `doc/DATABASE.md`: documents the idle cap, the "not an admission limit" rule, and the transaction-scoped session state contract. - Session state audit: every current caller uses `pg_advisory_xact_lock` / `pg_try_advisory_xact_lock` and `set_config(..., true)`. No caller uses session advisory locks, `SET` without `LOCAL`, `LISTEN`, or temporary tables. A reset round trip per call is therefore not added. ## Verification - New `packages/db/src/dedicated-connection-pool.test.ts` (embedded PostgreSQL, 10 tests). - Red on `origin/master`: 200 sequential calls used 200 distinct sessions (`expected 200 to be 1`). All 10 tests failed. - Green with this change, 3 runs in a row: 200 sequential calls use 1 session; 7 concurrent callers each hold a different session for the whole transaction, and afterwards at most 4 stay open and later calls reuse them; a failed transaction's connection is closed; a terminated socket during the action is not reused; an idle connection terminated by the server reconnects; a client the action ended is not reused; transaction-local settings do not leak; shutdown and owner `$client.end` close every retained connection; nested actions do not block. - Focused suites pass: packages/db `work-signals`, `prepared-queries-postgres`, `response-capture-postgres`, `checkpoint-pool`, `client`, `client-options`, `client-teardown-registry`, `postgres-connection-recovery`, `client-disconnect-replay` (132 tests). Server suites that use dedicated connections: `skill-package-store`, `status-card-evaluation-work` (updated: it now proves advisory locks are released, since sessions are reused), `agent-instruction-working-copies`, `workspace-recovery-retry-copyback`, `native-execution-lease-batch`, `company-skills-empty-runtime`, `decisions-service`, `legacy-controller-lease`, `server-startup-feedback-export` (467 tests). - `pnpm verify --explain` selects 2779 files for a `packages/db/src/client.ts` change. Per AGENTS.md I ran the focused checks above and let CI give the broad coverage. - Benchmark: 200 calls of `withDedicatedDbConnection(db, d => d.transaction(tx => tx.execute(select pg_try_advisory_xact_lock(42))))` against local embedded PostgreSQL 18, 10 warm-up calls, 5 alternating before/after process pairs per mode. Main-thread time is `eventLoopUtilization().active`. | Mode | Variant | Wall ms/call (median, range) | Main-thread ms/call (median, range) | | --- | --- | --- | --- | | TLS | before | 35.55 (32.93–37.52) | 5.18 (4.68–7.95) | | TLS | after | 0.19 (0.17–0.21) | 0.18 (0.17–0.20) | | Plain | before | 18.39 (17.91–20.53) | 3.65 (3.58–4.30) | | Plain | after | 0.17 (0.13–0.18) | 0.13 (0.12–0.14) | At the staging rate of about 21 calls per second, this removes roughly 100 ms of main-thread work per second (TLS figures; staging CPUs may be slower). ## Risks - A retained connection that the server or network drops while idle: postgres.js reconnects on the next query when it has seen the close (tested). If the drop and the reuse race, the first query of that action fails like a main-pool query would. Before this change, a dedicated action always started on a fresh connection. - Through a transaction-mode pooler, `SET application_name=...` is a server-side parameter switch, so its count may not drop as much as the client-side connection count. Measure the deployed effect with pooler client connections and main-thread CPU, not only that statement. - Concurrency above 4 opens extra connections exactly as before; they close on release. Total retained connections per owner grow by at most 4 idle clients, which still close after the idle timeout. - An action that sets session-level state would now leak it to the next action. The contract is documented on `withDedicatedDbConnection` and in `doc/DATABASE.md`; no current caller does this. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
bbed32450c |
perf(load-model): model recorded native run rates, the recorded event mix and coalesced delta batches (#21549)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - `scripts/benchmark-agent-scaling-load.ts` (#21418) estimates how many concurrent agents one server process can carry > - It assumed 20 text deltas/s and 1 other event/s per agent, and every other event was an `item.completed` > - Recorded hosted native runs show about 0.2 deltas/s and 0.5 other events/s, with a mix in which 39% of the other events are server trace spans that take a cheaper path > - This pull request makes the recorded rates and mix the model's default, keeps the old rates as an explicit stress profile, adds a burst profile, and mirrors the delta batching of #21520 > - The benefit is a model that measures what production does, so the next optimizations target the real per-agent cost ## Linked Issues or Issue Description Refs #21418 (load model and plan). Refs #21520 (delta batching that the model now mirrors). **What existing behavior does this improve?** The load model's default rates, its event mix and its delta batching rule, and the plan document's measurements. **Subsystem affected** `scripts/benchmark-agent-scaling-load.ts`, its test `server/src/__tests__/agent-scaling-load-model.test.ts`, and `doc/plans/2026-11-13-thousand-concurrent-agents.md`. **Current behavior** The model defaults to 20 deltas/s and 1 other event/s per agent. Every other event is a native `item.completed` with a chat-progress wake and a checkpoint save. It cuts a delta batch on the timer even while the previous batch commits. **Proposed behavior** - `PROFILE=recorded` (default): 0.2 deltas/s and 0.5 other events/s per agent. `PROFILE=burst`: 2.5 and 3, a run's busiest minute. `PROFILE=stress`: the earlier 20 and 1. API requests stay at 0.3/s. Each rate can still be set on its own. - `EVENT_MIX=recorded` (default) sends other events in the recorded proportions. A `run.performance.span` is an ordinary append with no wake and no checkpoint, as the server's run trace writes it. `tool.execution.*` and `item.completed` are native appends with a chat-progress wake, then a checkpoint save. `EVENT_MIX=<type>` sends one type only. - Delta batching follows #21520: deltas that arrive while a batch commits all join one next batch. - The report adds `profile` and `eventsIssuedByType`. **Reason and benefit** At the recorded rates the per-agent cost is the non-delta runner events and their checkpoint saves, not deltas. The plan and the next optimizations need that picture. **Breaking changes** None for the product. Anyone who runs the model with no options now gets the recorded profile; `PROFILE=stress` gives the earlier load. ## What Changed - Profiles, the recorded event mix (a 100-slot cycle per agent, offset by agent so the fleet sends the mix at every moment), the span append path, and the coalesced delta batches. - The model test now checks the profile in the report and that both server trace spans and runner events with a progress wake are in the mix, for the recorded profile and for the stress profile with one transaction per delta. - The plan has a new section "Load model at recorded rates": the recorded source (40 recent native runs on one hosted workspace, as aggregates), the profiles, the mix and its server paths, fresh runs at 500, 1,000 and 2,000 agents plus burst and stress, and what they show. It records the 3-connection checkpoint pool as a computed per-process limit on a hosted database, and that SQL building in Drizzle costs about as much main-thread time as result parsing on the hosted instance. A new section "Considered and not done" records why `run.performance.span` rows stay (about 3% of the statements per agent, with readers) and why checkpoint saves per critical event only are not done now. ## Verification - `server/src/__tests__/agent-scaling-load-model.test.ts` passes (2 tests). The new assertions fail against the previous script: it reports no `profile` and no `eventsIssuedByType`. - `server/src/__tests__/agent-scaling-load-model.test.ts` also checks each profile's rates with no overrides, and that a rate set on its own wins. - Load model runs, recorded in the plan (60 s, main pool of 10, embedded PostgreSQL on the same machine, each run started below a 1-minute load average of 15): recorded rates at 1,000 agents (event-loop utilization 0.61) and 2,000 (0.85) finish every operation with no failure; 500 agents all in their busiest minute run at 0.95 with p99 lateness under 0.4 s. About 4.4 statements per agent per second at recorded rates, 35% of them on the checkpoint pool. - Earlier runs at load averages of 20–55 had shown one process falling behind at 500 burst agents and at 2,000 recorded; the quieter runs after the model fixes (event order, spans off the runner stream) do not, and the plan says so. The checkpoint pool as a per-process limit is now stated as a computed limit for a hosted round trip (about 400 agents in a busy minute at 1 ms), not as measured. - Greptile findings fixed: a waiting delta batch no longer takes deltas sent after the next runner event, takes every delta queued when it starts, and spans no longer join the runner event stream. ## Risks - Low: a benchmark script, its test and a plan document. No product code changes. - The machine that measured the plan's table was shared; the plan marks the rows that the machine distorted. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
67652580ee |
perf(tool-access): index enabled policies so each decision checks only candidates (#21563)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Tool access policies decide whether an agent can call each tool, and
an MCP tools/list response decides access once for each listed tool
> - Each decision matched the raw selector of every enabled policy two
times, so one listing cost tools x policies selector checks on the
server's main thread
> - On staging, a company with 363 enabled policies kept the main thread
at 95-98% of a core: policy matching was 22% of CPU time, with single 80
ms runs inside the selector match
> - This pull request compiles the enabled policies once for each
decision cache and indexes them by the selector key that rejects most
(catalog entry, then tool name)
> - The benefit is that each decision checks only the policies that can
match its tool, with the same decisions, so a listing no longer blocks
the event loop
## Linked Issues or Issue Description
**What existing behavior does this improve?**
Tool access decisions for MCP tool listings
(`toolAccessPolicyService.decide`, called by `decideToolsForListing` in
the tool gateway).
**Subsystem affected**
Server: tool access policy evaluation
(`server/src/services/tool-access-policy.ts`).
**Current behavior**
Each decision loops over all enabled policies two times. The first loop
matches the selector and schema-checks the conditions of every
conditioned policy. The second loop evaluates conditions and matches the
selector of every policy. The selector match reads and allocates every
selector key in a fixed order, so a catalog entry mismatch is found only
at the end. A listing of 800 tools with 365 policies does 305,600
selector evaluations. On staging this was 22% of main-thread time
(event-loop utilization 0.965, delay p99 144 ms, max 710 ms).
**Proposed behavior**
The decision cache holds the enabled policies compiled once: parsed
selectors, an index by catalog entry and tool name, and a memoized
conditions schema check. Each decision evaluates only the candidate
policies, in the original priority and creation order.
**Reason and benefit**
Listing cost becomes proportional to the policies that can match each
tool. In the benchmark, one listing goes from about 164 ms to 12 ms of
main-thread time, and the longest event-loop block goes from 84 ms to 4
ms.
**Breaking changes**
None. The policy data model, API, and listing contract do not change.
## What Changed
- Add `server/src/services/tool-access/policy-selector-index.ts`. It
compiles a policy selector into a matcher with the same semantics as the
raw `selectorMatches`, and it indexes policies by `catalogEntryId`, then
by `toolName` (matched against the tool name and the upstream tool name)
for policies without a catalog entry selector. Policies with neither key
are candidates for every call. Candidates keep their original order.
- `decideInternal` caches the compiled index under the same
`enabled-policies:${companyId}` key. The new `matchEnabledToolPolicies`
keeps the two-step semantics: the first selected policy with unsupported
conditions denies the call before any match applies, then the selected
policies with matching conditions go to the unchanged decision loop
(block, rate limit, trust rule, require approval, allow, explanations).
- Decisions without a cache, and decisions that consume a rate limit
(which read fresh rows), compile the rows they read in the same way.
- The conditions schema check runs once per policy and cache, not once
per decision.
- Conditions are now evaluated only for policies whose selector matches.
Before, a malformed legacy condition on an unrelated policy could throw
in every decision. A selected policy with such conditions still gets the
unsupported-conditions deny first.
- Add `server/scripts/benchmark-tool-policy-matching.ts` and a section
in `doc/mcp-discovery-performance.md`.
## Verification
- Red/green regression:
`server/src/__tests__/tool-access-policy-index.test.ts` "reads policy
selectors once per decision batch, however many tools it decides". It
decides 40 tools with one cache against a staging-shaped set of 365
policies and counts reads of the raw policy selectors. On
`origin/master` it fails: 308,110 reads for 40 tools against 7,711 for
the first tool. With this change the count stays at the first-tool
count. The test also checks that cached and uncached decisions are
equal.
- Equivalence: the same file compares the compiled selector with the raw
`selectorMatches` for 404 selectors x 300 contexts (including empty,
whitespace, non-string, string-plural, duplicate and contradictory
values). It compares the indexed policy selection with the old unindexed
two-step selection (copied into the test) for 5 random sets of 400
policies x 400 contexts, and for the staging-shaped set over 800 tools.
The sets include catalog-entry-only, catalog-entry+connection,
multi-key, tool-name, unindexed, conditioned, unsupported-condition,
block, rate_limit, trust_rule, allow and legacy policy types with mixed
priorities.
- Index edge cases: an earlier tool-name policy (matched through the
upstream name), an unindexed policy and an empty catalog-entry selector
apply before a catalog-entry block; an unsupported-condition policy in
another bucket still denies. A DB-backed test checks the same through
`decide`, with and without a cache.
- Focused suites pass locally: `tool-access-policy-index`,
`tool-access-policy-service`, `tool-gateway-listing-memory`,
`connector-tool-policy`, `tool-gateway`, `tool-gateway-service` (6
files, 406 tests). `node server/scripts/typecheck-layers.mjs` passes.
- `pnpm verify` exits 2 (the new benchmark script is a broad input), so
CI supplies the broad coverage.
- Benchmark, local embedded PostgreSQL, macOS arm64, Node 24. One
listing of 800 tools, 365 enabled policies (staging shape plus the 2
fixture policies), 16 concurrent decisions with one cache as in the
gateway. Three runs of 9 alternating rounds after 2 warm-up rounds. Run
3 uses the event-loop delay histogram that also covers the end of the
listing. Both versions return identical decisions.
| Per listing | `origin/master` median (min-max) | This PR median
(min-max) |
| --- | --- | --- |
| Elapsed, run 1 | 157.5 ms (146.1-192.8) | 11.4 ms (10.3-15.3) |
| Elapsed, run 2 | 165.3 ms (161.7-185.4) | 11.9 ms (10.1-14.0) |
| Process CPU, run 1 | 160.5 ms (149.7-188.9) | 13.7 ms (11.1-21.0) |
| Process CPU, run 2 | 162.8 ms (157.2-192.7) | 12.3 ms (9.6-20.5) |
| Elapsed, run 3 | 163.9 ms (151.6-185.3) | 12.4 ms (10.4-18.5) |
| Process CPU, run 3 | 165.4 ms (155.7-214.9) | 14.5 ms (10.6-20.2) |
| Longest event-loop block, run 3 | 84.1 ms (76.9-95.1) | 3.7 ms
(3.0-5.0) |
| Selector evaluations | 305,600 | 1,164 |
| Raw selector property reads | 6,155,830 | 9,490 (compile only) |
```sh
cd server && pnpm exec tsx scripts/benchmark-tool-policy-matching.ts
--baseline-ref origin/master --tools 800 --rounds 9
```
- Not verified: the hosted staging profile after deploy.
## Risks
- Low risk. The decision loop is unchanged; only the set of policies
that reach it is narrowed, and every left-out policy has a selector that
rejects the call. The equivalence tests compare against the old
selection.
- The index lives only in the per-listing decision cache (or for one
uncached decision). There is no cross-request cache, so policy edits
apply on the next listing as before.
- The cached index memoizes the conditions schema check on shared
entries. The check is pure, so concurrent decisions in one listing get
the same result.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, local test and benchmark runs).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
b7922570e1 |
fix(runtime): verify GitHub bridge health before agent startup (#21537)
Verify the inactive GitHub credential endpoint from the execution target
before publishing launcher settings. If HTTP/2 fails during startup,
stop it and verify one file-bridge attempt. Stop an unverified final
bridge before returning isolated anonymous launchers.
Regression tests cover early transport loss, delayed loss reporting,
credential isolation, and failed file-bridge cleanup. The master update
keeps the current MCP approval fixture.
Verification: 96 focused tests and server typecheck pass after conflict
resolution. Full CI passes on
|
||
|
|
6826800425 |
feat(ci): report the required checks on merge-queue commits (#21545)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Each merge to `master` publishes the standard image, and the time from merge to published image limits how fast a fix reaches users > - #21096 lets `docker.yml` push an image that the merge queue already built, but the merge queue is not on > - The `master` ruleset requires `ci / verify` and `ci / e2e`. Only `pr.yml` reports them, and only for pull requests > - A merge queue waits for the same two names on its merge-group commit. No workflow reports them there, so every queued merge would stall > - This pull request adds a workflow that reports both names on merge-group commits > - The benefit is that the owner can turn the merge queue on with one ruleset change ## Linked Issues or Issue Description Refs #21096 **What existing behavior does this improve?** The merge-queue image build from #21096. It cannot run until the merge queue is on, and the merge queue cannot be turned on safely. **Current behavior** `ci / verify` and `ci / e2e` come from `pr.yml`, which runs on `pull_request` only. With a merge queue, GitHub waits for both checks on the merge-group commit. They never arrive. Each queued pull request would time out and leave the queue. **Proposed behavior** `merge-queue-checks.yml` runs on `merge_group` and reports `ci / verify` and `ci / e2e`. It runs no tests. It does not check out the queued commit. It has no token permissions and no secrets. **Reason and benefit** A pull request enters the queue only after both checks pass on the pull request. A queued merge therefore has the same protection as a direct merge today. The exact-source gate still tests each `master` commit before publication. Real tests on the combined tree can replace these jobs later under the same names. ## What Changed - Added `.github/workflows/merge-queue-checks.yml`. It reports `ci / verify` and `ci / e2e` on merge-group commits. - Added `scripts/__tests__/merge-queue-checks-workflow.test.mjs`. It fails if the names differ from the names that `pr.yml` reports, or if the workflow gains a checkout, an action, a secret, a token permission or a self-hosted runner. - `pr-trusted.yml` now runs that test and `scripts/merge-group-image.test.mjs` in a step that skips test files an older checkout does not have. The second test came with #21096 but no CI step ran it. - `scripts/merge-group-image.mjs find` now waits up to 15 minutes for a merge-queue image build that is still queued or running. The queue can land a commit before its image build ends. Without the wait, the `master` push always builds from source. - `doc/RELEASING.md` explains the workflow and lists the ruleset settings that turn the queue on and off. ## Verification - `node --test scripts/__tests__/merge-queue-checks-workflow.test.mjs`: 2 pass. - `node --test scripts/merge-group-image.test.mjs`: 10 pass, with new cases for unfinished builds. - `node --test scripts/__tests__/release-verify-workflow.test.mjs`: pass. - `actionlint` on both changed workflows: no findings. - Not verified: no `merge_group` event can fire before the queue is on. The first queued merge is the live test. `pr.yml` calls `pr-trusted.yml@master`, so the new CI step runs only after this merges. Owner steps after merge: 1. In the `master` ruleset, add **Require merge queue**: Squash, build concurrency 5, group size 1 to 5, status check timeout 60 minutes. 2. After the first queued merge, confirm that the `master` commit SHA equals the SHA of its merge-group runs. ## Risks - Low risk while the queue is off. The workflow has no trigger other than `merge_group`. - With the queue on, the queue does not test the combined tree. Two pull requests that pass alone can still break `master` together. This is the same as today. - A merge-queue image build that hangs delays the source build by up to 15 minutes. A build that fails does not. - If GitHub does not accept a job named `ci / verify` for the required context, queued merges time out. Rollback: remove the merge queue rule. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
f6c5986b61 |
feat(connections): machine sign-in (MCP OAuth client credentials) for organization connections (#21473)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents use apps through connections to remote MCP servers
> - Today an organization connection must sign in as a person (browser
sign-in) or use a static key
> - Service accounts and CI-style integrations have no person who can
sign in
> - The MCP OAuth Client Credentials extension (2026-07-28) defines
machine-to-machine sign-in for this case
> - This pull request adds "Machine sign-in" for organization
connections, with a Paperclip-held signing key (recommended) or a client
secret
> - The benefit is that organizations can connect automation-owned MCP
servers safely, with credentials only in the vault and bound to one
sign-in service
## Linked Issues or Issue Description
**Subsystem affected**
Connections: remote MCP OAuth (server `tool-access/oauth`, tool gateway
token refresh) and the custom MCP server setup flow in the UI.
**Problem or motivation**
Organization-owned automation (service accounts, CI integrations) cannot
connect an MCP server that requires OAuth, because no person can
complete a browser sign-in. The MCP OAuth Client Credentials extension
(`io.modelcontextprotocol/oauth-client-credentials`) solves this with
`grant_type=client_credentials` and RFC 7523 signed client assertions.
**Proposed solution**
Add machine sign-in for organization connections only. Paperclip creates
a keypair per connection, keeps the private key in the company vault,
and shows the public key (PEM and JWK) for the admin to register. A
client secret is also supported. The sign-in service (issuer and token
endpoint) is pinned at setup. Tokens use the existing organization grant
refresh path, so caching, concurrent dedupe and 401 re-request reuse the
existing lease and single flight.
**Alternatives considered**
Reuse the deployment-wide `PAPERCLIP_TOOL_OAUTH_*` client env path that
already sent `client_credentials`. Rejected: it is not per connection,
it keeps secrets in env, and it has no issuer pin. Route machine sign-in
through `connectGalleryApp`. Rejected: its rollback deletes the draft on
a failed test, which would lose a key the admin already registered.
**Roadmap alignment**
Phase 1 of the enterprise MCP plan (no enterprise single sign-on
needed). OIDC login and enterprise-managed authorization (ID-JAG) are
later phases.
## What Changed
- `oauth/client-credentials.ts`: keypair generation (RS256, or ES256
when the service requires it), RFC 7638 key IDs, and RFC 7523 assertions
(`iss` = `sub` = client ID, `aud` = token endpoint, 60 s `exp`, unique
`jti`) with Node `crypto` only.
- `oauth/client.ts`: the token exchange accepts a signed assertion and
then sends only `client_assertion_type` and `client_assertion` (no
`client_id` or secret), as the extension specifies.
- `oauth/client-credentials-service.ts`: discovery of the sign-in
service (protected-resource metadata, then RFC 8414/OIDC metadata with
issuer check, or a token URL entered directly), issuer pinning,
vault-only key and secret handling, draft creation, test-and-activate,
and no-downtime rotation (test the new secret first; stage a new key and
switch only after the provider accepts it).
- `oauth/grant-refresh.ts` and `oauth/connection-refresh.ts`: machine
sign-in requests a new access token with the connection's own
credentials instead of needing a refresh token. It never falls back to
deployment-wide OAuth clients.
- `catalog/discovery.ts`: a rejected token on a machine sign-in
connection never re-runs discovery or rewrites the pinned endpoints.
- `mcp-http.ts` and the tool gateway: declare
`io.modelcontextprotocol/oauth-client-credentials` in client
capabilities only for connections that use it.
- Routes (with OpenAPI and activity logging): `POST
/companies/{companyId}/tools/machine-sign-in`, `GET|POST
/tool-connections/{id}/machine-sign-in[/test|/rotate|/rotate/complete]`.
Removal activity now records `machineSignIn: true`. The runner API
treats these routes as sensitive.
- Shared validators and types for machine sign-in.
- UI: "Machine sign-in" under Advanced authentication in the custom MCP
server flow (organization connections only), `MachineSignInSetup`
(choose Paperclip key or client secret, register the public key, test),
and Storybook stories for each setup state.
- Docs: `doc/connections/MCP-CLIENT-CREDENTIALS.md`, linked from
`doc/connections/README.md` and `GENERIC-REMOTE-MCP.md`.
## Verification
- `server/src/__tests__/tool-access-client-credentials.test.ts`: 16/16
pass. It uses a fake sign-in service that verifies real signatures
against registered keys, and a fake MCP server. It covers RS256 and
ES256 assertion claims and signatures, assertion-only client
authentication, the secret flow (`client_secret_basic`), discovery and
direct token URL, issuer pinning (refuses a different sign-in service
and sends it nothing), the pinned endpoint surviving a hostile
challenge, the token cache (0 token requests while live), concurrent
dedupe (5 callers on 2 service instances produce 1 token request), 401
re-request, key and secret rotation, a refused personal connection,
secret redaction in responses, rows and activity, and vault cleanup on
removal.
- Red/green: with the grant-refresh machine sign-in branch and the
discovery pin guard disabled, 5 runtime tests fail (pin, cache/dedupe,
401 re-request, both rotations). Restored: 16/16 pass. The other tests
cover new routes and services that do not exist on `master`.
- `ui/src/features/connections/MachineSignInSetup.test.tsx`: 3/3.
`ConnectionSetupFlow.architecture` and `.oauth` tests: 19/19.
- `openapi-routes.test.ts`: 51/51. `runner-api.test.ts`: 2153/2153.
`mcp-http.test.ts`: 9/9. Existing `tool-access-service.test.ts` refresh
and legacy client-credentials tests: 54/54 (filtered).
- Server package `tsc --noEmit`: clean. UI `tsc`: clean except the
pre-existing generated Storybook catalog imports.
- Not run locally (machine under heavy load): full suites, repo-wide
typecheck, browser check of the live app. CI covers breadth.
## Risks
- No migration. New config lives in `config.oauth` (`grantType:
"client_credentials"` plus `clientAuth`). Older env-configured
client-credentials connections have no `clientAuth` and keep their
original path.
- The capability declaration adds one extension key only for machine
sign-in connections. Other requests are unchanged.
- Rotation has no settings-page control yet; it is available through the
API. This is a follow-up.
- Low risk to agent boot and time to first token: a live token is read
exactly like a browser sign-in token.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
dc018f9bc0 |
Keep agent-owned connections healthy during catalog refresh (#21539)
Use dedicated agent grants for catalog refresh, GitHub health checks, and reconnect status. Preserve shared connection health when a different agent lacks authorization. Add PostgreSQL and HTTP regressions for credential selection, stale health recovery, caller isolation, and actual provider rejection. Correct the MCP demo approval fixture to submit a valid saved-card option through normal answer validation. Validation: 629 focused tests, server typecheck, all CI checks, and Greptile 5/5 with no unresolved review comments. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
|
a726dce4f0 |
feat(mcp): show live progress for long connected-app tool calls in the task chat (#21449)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents call tools from connected apps through the Paperclip MCP gateway > - Some tool calls take a long time: exports, builds, large searches > - The MCP progress utility lets an app report how far a call is, but Paperclip ignored it > - So the task chat showed only a spinner until the call finished, with no sign of life > - This pull request reads the app's progress and shows it on the tool's row while the call runs > - The benefit is that people can see that a long call is working, how far it is, and how long it has run ## Linked Issues or Issue Description **Subsystem affected** Connected apps (MCP gateway) and the task chat. **Problem or motivation** A long tool call from a connected MCP app shows only a spinner in the task chat. Paperclip does not send a `progressToken`, and it drops `notifications/progress` from the app. **Proposed solution** Send a `progressToken` on each `tools/call` that an agent run makes. Read the app's progress notifications from the call's response stream. Show a progress bar (or an indeterminate sweep), the latest message, and elapsed time on the running tool row. Keep the latest state so that a reload mid-call shows it. **Alternatives considered** A new transport for progress. The existing company live-events socket already carries per-run events with run read checks, so this change uses it. **Additional context** Spec: https://modelcontextprotocol.io/specification/2026-07-28/basic/utilities/progress ## What Changed - `mcp-http.ts`: `mcpHttpRequestInit` can add `_meta.progressToken`. `readMcpHttpResponse` gives each notification on the response stream to an observer before the result. The observer cannot fail or delay the call. New `mcpProgressNotificationOf` matches a notification to its token. - `tool-gateway.ts`: remote HTTP and local stdio `tools/call` from an agent run send a per-call token and forward progress to a reporter. Calls without a run send no token. - New `server/src/services/tool-call-progress.ts`: checks each update (finite, increasing, plain single-line text of at most 200 characters). It publishes a `heartbeat.run.tool_progress` live event at most every 250 ms (latest wins) and saves the latest update on the invocation at most every 1 s. Nothing is awaited by the call. A final `done` event goes out when the call ends. - DB: nullable `tool_invocations.progress` jsonb column (migration `0479`, `ADD COLUMN` only). A save never writes onto a finished call. - Live delivery: the new event follows its run like transcript output. A client gets it only for runs it follows, after the run read check, in the run's company only. - New route `GET /api/issues/:issueId/tool-call-progress` (task read check) returns the latest progress of the task's running calls, for a reload mid-call. Documented in `docs/api/issues.md`. - UI: `ToolCallProgressProvider` (one per chat) keeps the latest update per call. A running tool row subscribes and shows a bar or sweep, the message, the percent and elapsed time (design tokens, reduced motion honored). Finished rows drop it and never fetch or re-render for progress. `LiveUpdatesProvider` ignores the event for query invalidation. - `packages/kv-demo-mcp-server`: new `slow_with_progress` demo tool (`steps`, `stepMs`, `withTotal`) in its own file. - Storybook: `Tasks/Tool progress` stories (running with total, running without total, done). - Docs: new `doc/connections/MCP-PROGRESS.md`; `MCP-PROTOCOL-VERSIONS.md` no longer lists progress as unsupported. ## Verification - Red/green: on the original source, 5 of the new server tests fail (token in `_meta`, SSE with interleaved progress and result, notification matching, run-interest delivery, gateway end-to-end). With this change they pass. - `server/src/__tests__/mcp-http.test.ts` (13), `tool-call-progress.test.ts` (9: throttling, latest wins, bounded text, out-of-order drop, finish, failures never throw, company/run/tool scoping), `tool-call-progress-store.test.ts` (2, embedded Postgres: company and task scoping, no write onto a finished call), `realtime/live-event-delivery.test.ts` (run interest), `tool-gateway.test.ts` end-to-end against the real SDK kv demo server over Streamable HTTP: 6 provider updates over about 1.2 s give 3 to 7 throttled events, all for the right company, run, task and tool, the last one `done`; no event reaches another company; the invocation keeps the progress; the result is unchanged. - UI: `tool-call-progress.test.tsx` (10: name matching, row states, plain-text rendering, drop on complete, restore after reload, no fetch for finished rows). All 67 task-chat test files pass. - `packages/kv-demo-mcp-server`: 13 tests, including progress over HTTP with and without a total. - Storybook: checked the three stories in a browser. The bar fills to 40%, the sweep animates, the elapsed time ticks, and the done row shows no progress. - `pnpm verify`: see the CI checks for the final revision. ## Risks - Low. The migration adds one nullable column. - An SSE response is now bounded per message, not per stream, so handled notifications (such as progress) do not use up the result's size limit. Each message is still bounded. - Progress code runs only while a run's tool call is in flight. It adds a few bytes to the request and throttled, fire-and-forget work. Agent boot and time to first token do not change. - A row finds its call by run and tool name. If one run calls the same tool twice at the same time, both rows show the newer call's progress. - Progress from a call that became an MCP Task is not shown. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
81d06cfc97 |
feat(mcp-tasks): add a demo MCP server that uses Tasks, test against it, and keep audit records out of agent errors (#21464)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents call tools in connected apps through the tool gateway, which speaks MCP to each app > - MCP Tasks (#21263) lets an app answer a long tool call with a durable task. It shipped behind the experimental setting `enableMcpTasks`, off by default > - No public app returns tasks without credentials, so nobody had seen the feature work against a real server, and it was hard to understand > - This pull request adds a small demo app that uses Tasks, tests the gateway against it over HTTP, and records a live end-to-end run > - The live run found one bug: every task stop sent the call's audit record, with a server stack trace, to the agent. This pull request fixes it > - The benefit is a repeatable way to try and test Tasks, plain-language docs, and evidence for the decision to turn the setting on ## Linked Issues or Issue Description Refs #21389 Refs #21263 ## What Changed - Add an MCP Tasks demo server to `packages/kv-demo-mcp-server` (new bin `paperclip-tasks-demo-mcp-server`; run `node packages/kv-demo-mcp-server/dist/tasks/main.js` after a build). It is stateless MCP 2026-07-28 with `server/discover`, and it declares `io.modelcontextprotocol/tasks`. It has six tools: `slow_report`, `needs_approval` (form question), `will_fail`, `cancellable`, `expires`, and `quick_echo`. It saves task state to a file, so it can restart mid-task. It needs no credentials. It has a status page at `/` and JSON at `/api/state`. - I put it in the existing demo package, so there is no new workspace package and no `pnpm-lock.yaml` change. - Add `server/src/__tests__/mcp-tasks-demo-server.test.ts`. It runs the real gateway against the real demo server over HTTP, with one test per tool plus a restart. - Fix: `server/src/routes/tool-gateway.ts` no longer sends `details.execution` to callers. That record has request IDs and the server-side error stack. The audit still keeps it. Add `server/src/__tests__/tool-gateway-agent-errors.test.ts` (red before the fix, green after). - Docs: `doc/connections/MCP-TASKS.md` now has "In plain words", "What a person sees", "Try it yourself", the live-run results, and a recommendation. The package README documents the demo server. - `enableMcpTasks` stays off by default. ## Verification - `packages/kv-demo-mcp-server`: `npx vitest run` gives 25/25 pass (13 new demo-server tests). `tsc -p tsconfig.test.json` is clean. - `server`: `mcp-tasks-demo-server.test.ts` gives 8/8 pass. `tool-gateway-agent-errors.test.ts` gives 2/2 pass, and it failed 2/2 before the fix. - Existing suites that cover the changed route and Tasks: `mcp-tasks.test.ts` and `tool-gateway-discovery-http.test.ts` (results in a comment below). - `pnpm verify` exits 2, because the doc is a global CI input. Required CI gives the broad coverage. - Live end-to-end run on a local `test-drive` instance with `enableMcpTasks` on. The agent was Codex `gpt-6-luna` at low effort. The demo was connected through "Connect your own MCP server": | Behaviour | Result | `tools/call` count at the app | | --- | --- | --- | | Polling (`slow_report`, 45 s) | Delivered after 22 polls | 1 | | Question (`needs_approval`) | The card was answered in the UI. `tasks/update` was sent, and the task finished | 1 | | Failure (`will_fail`) | `provider_task_failed` with the app's message | 1 | | Cancel (`cancellable`) | Stop on the run sent `tasks/cancel` | 1 | | Expiry (`expires`, 15 s) | `provider_task_expired`, no hang | 1 | | Restart (`slow_report`, 120 s) | The server was down for 2 min 23 s. The result was delivered once, and the agent was woken once | 1 | After the run, all 6 `tool_mcp_tasks` rows were settled and none had a next poll. ## Risks - Low risk. The demo server is new code in a demo package, and the bin is additive. - The route change removes `execution` from tool-gateway error responses. No UI code or test reads that field from these responses. The audit and tool-call events still record it. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), in Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
1f388ed385 |
Start follow-up runs normally when a warm sandbox's window ends during its idle close (#21504)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A warm sandbox keeps an agent's Runner alive between runs, so the next run on the task starts fast > - When a run ends, the server closes the idle warm Runner in-process. A separate SQL worker stops the sandbox when its warm window ends (#21410, merged today) > - Both steps start at the same moment. The sandbox stop wins, so the idle close fails. The failed owner stays registered, and the next run throws `native_instruction_materialization_handoff_unavailable` before any provider work > - Recovery must just work: a follow-up run must never fail because a predecessor's warm copy is gone > - This pull request lets the idle close finish before the sandbox stops, prepares instructions fresh when the warm copy cannot be handed off, and releases an owner whose sandbox the host already gave up > - The benefit is that automatic follow-up runs start normally after a warm window ends ## Linked Issues or Issue Description Refs #21472 (base of this stacked PR), #21410 (introduced the warm-window stop). **What happened?** In a Daytona two-server chaos wave (`w-20261113T200824-s7106`), three automatic follow-up runs failed before any provider work with `native_instruction_materialization_handoff_unavailable`: run `5f68e194`, run `d6bd5a97` and run `d1198e40`. None of them was at a release swap. In each case, the predecessor's idle close had failed first with `provider_state_unreadable` / "Daytona sandbox lease … is no longer active". The cause: the warm-window stop from #21410 stopped the sandbox at the same moment that the idle close started. For the first run's predecessor, the close was requested at 20:30:34 and the sandbox got SIGTERM at 20:30:37.8. **What did you expect?** The follow-up run prepares its instructions fresh and continues. The warm-window stop does not interrupt the host's own idle close. **Steps to reproduce** 1. Run a warm Daytona task until it ends, so its warm window starts. 2. Let the warm window end while the idle timer closes the Runner. 3. Start a follow-up run on the same task. It fails with `native_instruction_materialization_handoff_unavailable`. Two of the three tasks then ended stranded. That downstream part (the claim-gate cancellation with `execution_reconciliation_required` and the stranded notice) is owned and fixed by another session. It is **not** in this PR. ## What Changed - `server/src/services/native-runtime/native-session-executor.ts` `claimWarmNativeInstructionCopy`: after it retires a stale warm owner whose copy cannot be adopted, it returns `null`. The caller then prepares instructions fresh. Before, it threw `native_instruction_materialization_handoff_unavailable`. - `server/src/services/warm-sandbox-expiry.ts`: the warm-window stop now waits `WARM_SANDBOX_IDLE_CLOSE_MARGIN_MS` (2 minutes) past the window. The in-process idle close (close grace plus the 30 s proof read) finishes first. - `native-session-executor.ts` idle timer: when an idle close fails, it reads the lease rows for that sandbox. If every row is no longer `active` or `retained` (the host gave the sandbox up), it releases the in-process owner. The next run then prepares fresh. - `doc/execution-semantics.md`: one paragraph about the stop margin and fresh preparation. ## Verification Red → green (each test fails on the base `35ba76d0ef` and passes on this branch): - `native-session-executor.test.ts` › "hands off collection before probing or retiring a warm owner: unavailable" — red: throws `handoff_unavailable`; green: returns `null` (fresh preparation). - `native-session-executor.test.ts` › "forgets a warm owner whose idle close failed only once the host gave up its sandbox: pending_cleanup / released / retained" — red on all three; green: the owner is released for `pending_cleanup` and `released`, and kept (retired by the successor) for `retained`. - `warm-sandbox-expiry.test.ts` › "leaves the host's idle close of the warm runner time to finish before stopping its sandbox" — red: the sandbox stops 3 s after the window; green: it stops only after the margin. Two existing tests ("keeps a sandbox still inside its warm window, and wakes when the window ends", "reconciles a warm sandbox left running across a restart") now expect the deadline with the margin. Local runs on this branch: - `server`: `tsc --noEmit` clean. - `vitest run src/__tests__/warm-sandbox-expiry.test.ts src/services/heartbeat/instruction-working-copy.test.ts`: 24/24 passed. - `vitest run src/services/native-runtime/native-session-executor.test.ts -t "hands off collection before probing or retiring a warm owner|forgets a warm owner whose idle close failed"`: 10/10 passed. Not run: a Daytona wave with this change. ## Risks - Warm sandboxes now stop up to 2 minutes later than the configured window. This is a small cost increase per warm sandbox. I chose a fixed margin over an in-process "is a close running?" check because the stop is pure SQL. The margin works when two servers run at the same time (the server that runs the stop may not be the server that owns the Runner), and it needs no busy-loop or cross-server signal. - The margin does not cover a re-armed idle timer or a close that takes longer than 2 minutes. Fresh preparation (fix 1) and the owner release (fix 3) cover those cases. - Not run on Daytona. The evidence is from the wave above and the unit tests. - In the wave, a `restore_failed` copy-back onto the just-restarted sandbox was also seen. I did not find its root cause. It is not changed here. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edits, test runs) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
a4c570aede |
feat(mcp): ask an app's form questions in the task chat and send the answer back (MCP elicitation) (#21466)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents use connected MCP apps through the tool gateway. > - An MCP app can ask the person a question in the middle of a tool call (MCP elicitation). There are two kinds: a form, or a link to open. > - Today a form question is a dead end: > - A 2026-07-28 server's form `input_required` stops with "needs information Paperclip can't collect yet". > - An older server's `elicitation/create` makes a card, but the answer is never sent back, and the server waits on an open stream. > - Paperclip does not declare form support, so compliant servers never ask. > - This pull request shows the question as a card in the task chat. The card names the app, and only a person can answer it. When the person answers, the agent wakes. The agent's repeated call sends the answer to the app exactly once, for both protocol eras. > - The benefit: apps that need a choice or a detail from the person now finish their work. Restarts lose nothing, and no call is left waiting. ## Linked Issues or Issue Description Refs #21140 (stateless MCP client; this PR closes its "form elicitation" follow-up). Refs #21263 (MCP Tasks; this PR shares its card code). Refs #21389 (MCP Tasks follow-ups: the card answer validation item). **Is your feature request related to a problem?** When a connected MCP app asks the person a form question during a tool call, the call stops and cannot continue. A 2026-07-28 server gets a "needs input" error. An older server gets a card whose answer never reaches it. **Describe the solution you'd like** Show the question in the task chat. Let a person answer, skip, or leave it. Wake the agent when the question is resolved. Send the answer on the agent's repeated call, once, with the server's exact state. **Describe alternatives you've considered** Holding the tool call open until the person answers. We rejected this: it holds a stream and a run slot, and it does not survive deploys or restarts. The plan (`doc/plans/2026-11-13-mcp-elicitation.md`) records nine design decisions with their alternatives. ## What Changed - **Plan:** `doc/plans/2026-11-13-mcp-elicitation.md` covers the spec, the inventory, the decisions, the journey, tests, and risks. - **New module `server/src/services/mcp-elicitation.ts`.** It is the single mapping for direct calls and MCP Tasks: - It turns each spec form field into a real card field: text with defaults, number, yes/no, single choice (`enum` and `oneOf`), and multiple choice (`items.enum` and `items.anyOf`). - It turns card answers back into typed `content`. - Answered sends `accept`. Skip sends `decline`. Withdrawn, closed, or expired sends `cancel`. - It refuses forms that ask for passwords, keys, tokens, card numbers, or one-time codes. - It keeps app text plain, so links in it are not clickable. - It seals `requestState` with AES-256-GCM, bound to the company and connection. - **New table `tool_mcp_elicitations`** (migration 0479). It is company-scoped, with one row per question round. It holds a keyed call identity, the sealed state, the questions and their cards, a claim time, and a 24-hour expiry. - **Gateway, stateless servers:** - Paperclip declares `elicitation: {form, url}` on task tool calls. - A form-only `input_required` saves the round, makes the cards, and stops with a wait message. - A repeated call before the person answers stops again without contacting the app. - After the answer, the repeated call claims the round and sends `inputResponses` and the exact `requestState`, once. - A failed send releases the claim. A crash mid-send lets the claim lapse. - **Gateway, older servers:** - Paperclip declares form support at `initialize`. - The first question gets `cancel` at once, so the server never waits. - When the same question (same words and schema) arrives on the repeated call, Paperclip answers it from the card. - **Gateway, mixed rounds and retries:** - A form beside a URL step shows the link first. - A repeated call that stopped for a question runs again in the same run. It is no longer replayed by the per-run idempotency key. - **Cards:** the title is "*App* needs your input". The resolver policy is `human_only`. "Send to *App*" sends the answer. A user comment does not supersede the card. - **Activity log:** new events `tool_gateway.elicitation_sent`, `tool_gateway.elicitation_refused`, and `tool_gateway.elicitation_expired`. They record the action only, never the values. - **MCP Tasks:** task cards use the same mapping and gain the app name, real fields, and `human_only`. A secret or an unsupported form stays unsupported. - **Storybook:** `Tasks and conversations/Questions and approvals/MCP App Questions` has five states: Asking, Answered, Skipped, Expired, and TwoQuestions. The fixtures are generated from the server mapper's output. - **Docs:** `doc/connections/MCP-ELICITATION.md` (linked from the README) is new. `MCP-PROTOCOL-VERSIONS.md` and `MCP-TASKS.md` are updated. ## Verification - `server/src/__tests__/mcp-elicitation.test.ts` has 16 tests. They pass on the branch. - **Red/green:** 9 of the 9 gateway tests fail with master's `tool-gateway.ts` and `mcp-http.ts`. The 7 pure unit tests cover the new module. - The tests use a fake 2026-07-28 server and a fake 2025-06-18 server that asks over the open stream and waits for the reply. - Covered: answer, skip (`decline`), withdraw (`cancel`), two questions at once, a restart while waiting, a crash mid-send, another company with the same app and arguments, expiry, a refused password field, an agent that tries to answer (rejected), and capabilities. - Existing suites: `mcp-tasks`, `mcp-protocol-versions`, `tool-gateway-service`, `tool-gateway`, and `remote-mcp-connectors`. See the results comment below. - Storybook: the five stories render in headless Chromium, and the play tests pass. The only console errors are the 404s from the backend-less API calls that every story makes. - `tsc --noEmit` is clean for `server` and `ui`. `node scripts/storybook-catalog.mjs` passes. ## Risks - **Older servers may start asking questions.** Declaring form support at `initialize` can make older servers ask where they did not before. Mitigation: the first question always gets `cancel` at once, so no call hangs, and the agent gets a clear next step. - **Per-call cost.** Each 2026-07-28 HTTP tool call in a task makes one indexed lookup for a waiting round. Nothing is added before the first token or to agent boot. - **Migration.** One additive table with no backfill, so it is safe to apply online. - **Changed Tasks behavior.** A task card that is withdrawn or closed now sends `cancel` instead of `decline`, as the spec defines. A skipped card still sends `decline`. - **Stale cards.** Expiry is checked only when the agent calls again. A card can sit past 24 hours until the next call. Then it is withdrawn and the app asks again. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code with tool use (shell, file edits, headless browser), with extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
e6df84a992 |
fix(ci): count a step cancelled as "A task was canceled." as runner loss when the log shows the shutdown (#21514)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - PR CI runs on AWS spot runners, and `ci-runner-loss-rerun.yml` re-runs a PR's first attempt once when every failed job lost its runner (#21291) > - On 2026-11-13 a watcher was added to the runner image that stops the GitHub runner about 10 seconds after AWS announces a spot reclaim, so the lost job ends in about 28 seconds instead of up to 10 minutes > - When the runner is stopped this way, the cancelled step's check annotation is usually "The operation was canceled.", but a step that was still starting up gets "A task was canceled." instead > - The classifier only recognised the first text, so it counted such a job as a real failure and refused the re-run, even though the job log has the shutdown-signal line > - This pull request recognises both cancellation texts, still only together with shutdown evidence in the log > - The benefit is that lost jobs stopped early in a step are re-run automatically too ## Linked Issues or Issue Description Refs #21291 **What happened?** In the first hour after the spot-notice watcher went live, a `ci / Build` job was stopped 13 s after it started. Its job log has `##[error]The runner has received a shutdown signal...`, but its only failure annotation was `A task was canceled.`. `classifyJob` treated that annotation as a real failure, so the whole run was not re-run. **Expected behavior** A job whose log shows the shutdown signal is counted as runner loss, whichever cancellation text the step was given. **Actual behavior** It is counted as a real failure, and the run stays red until a person re-runs it. ## What Changed - `.github/scripts/runner-loss-rerun.mjs`: the cancellation annotation check accepts "The operation was canceled." and "A task was canceled.". The log check still decides. A cancellation with no shutdown or lost-communication line in the log is still a real failure, and its evidence now quotes the actual annotation text. - `.github/scripts/tests/runner-loss-rerun.test.mjs`: a test for "A task was canceled." with and without shutdown evidence. - `doc/DEVELOPING.md`: the runner-loss paragraph names both texts. ## Verification - Red: the new test fails against the unchanged script (25 pass, 1 fail). - Green: `node --test .github/scripts/tests/runner-loss-rerun.test.mjs`: 26 pass. - `node --test '.github/scripts/tests/*.test.mjs'` (the `policy` job's command): 808 pass. - The workflow runs from `master`, so this PR's own CI cannot exercise the new behaviour; the tests are the evidence. ## Risks - A job cancelled as "A task was canceled." for another reason is still treated as a real failure unless its log has the shutdown or lost-communication line, so a person cancelling a run still blocks the automatic re-run. - No change to triggers, permissions, the once-only rule or the stale-head checks. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
0a96f972e1 |
Connect Tailscale networks through device sign-in (#21526)
## Thinking Path > - Paperclip manages AI agents and the environments where they run. > - SSH environments can reach machines through a company Tailscale connection. > - The initial connector requires a managed OAuth app that Tailscale limits to its own tailnet. > - Customers need to connect their own tailnets without registering an app or supplying a key. > - This pull request uses Tailscale's normal device sign-in for new connections. > - Customers approve the Paperclip device and then select their machines in SSH environments. ## Linked Issues or Issue Description **Problem or motivation** The current Tailscale enrollment path cannot serve independent customer tailnets through the documented self-service OAuth app model. It also requires Cloud broker configuration and pilot admission before a customer can start. **Proposed solution** Start a private Tailscale device in the tenant and open its normal sign-in link. Save its encrypted device state before reporting a successful connection. Retain the existing SSH machine selection and agent account behavior. **Alternatives considered** Auth-key entry would require customers to copy a credential. A public OAuth integration still needs a confirmed registration path. Device sign-in uses a documented Tailscale flow without either requirement. See [device authentication](https://tailscale.com/docs/features/tsnet#device-creation-and-authentication) and [OAuth app limitations](https://tailscale.com/docs/features/oauth-apps). **Related PR search** Searched GitHub PRs for `tailscale` and `device sign-in`, and issues for `tailscale`. This follows the merged network connector in [#20617](https://github.com/paperclipai/paperclip/pull/20617) and preserves the SSH agent-account support in [#21248](https://github.com/paperclipai/paperclip/pull/21248). The open [#15648](https://github.com/paperclipai/paperclip/pull/15648) proposes storing an OAuth client and checking its health; this change instead enrolls a device through interactive sign-in so customers do not need an OAuth app. ## What Changed - New connections use device sign-in. Existing connected devices and pending legacy OAuth callbacks remain supported. - Pending sign-ins are bound to their company and initiating user. They have a fixed ten-minute deadline. Device state uses server-owned vault secrets with trusted version and digest checks. - The active Tailscale CLI waits for consent and device approval. Cancellation, expiry, secret deletion, and stale completion cannot restore a cancelled connection. - The connector stays open during sign-in. It provides a link when popups are blocked and shows device approval separately. Progress checks stop on completion, failure, expiry, or unmount. - Shared response types, OpenAPI contracts, SDK source pins, and the API/setup documentation describe the new flow. No database migration or Cloud provider configuration is required. ## Verification - 109 focused tests cover network processes, the real database and encrypted vault, routes, UI, and legacy connection behavior. - Server and UI typechecks pass. - Follow-up `29cf296a597`: reproduced the missing OpenAPI route failure locally. All 70 OpenAPI and Tailscale route tests and server typecheck pass after synchronizing the API contract. - Follow-up `79de05f38b9`: the transient-recovery regression failed before the fix; all 20 connector UI tests and UI typecheck pass after it. The query-error policy check reports no findings. - The public SDK input registry check passes. - [CI attempt 2](https://github.com/paperclipai/paperclip/actions/runs/43535594257/attempts/2) passed on the final head, `29cf296a597e470d432b84767050bc539caa280e`: 57 successful checks, no failures or pending checks. The interrupted canary, E2E shard, and serialized-suite jobs passed on rerun. Greptile is 5/5 on this head with no unresolved review threads. - `pnpm verify --explain` exits 2 because the documentation change selects broad CI coverage. Focused local checks passed; broad CI passed on `29cf296a597e470d432b84767050bc539caa280e`. - Live local qualification on `29cf296a597e470d432b84767050bc539caa280e`: native device sign-in completed with interactive user approval. An isolated Ubuntu 24.04 ARM64 EC2 machine appeared in discovery and passed SSH pairing and workspace probes. The assigned Claude Code agent used a Paperclip-managed subscription, created `hello_world.txt` containing exactly `Hello, world!` plus a newline, posted a completion comment, and marked the task done. An independent read through the private connector verified the bytes, remote user, and hostname. The remote machine had no stored Claude login and no leftover temporary credential directories. - Restarted the local Paperclip server and verified the same Tailscale device identity, online peer discovery, successful SSH probe, and unchanged file without new consent. Temporary public SSH ingress was removed. - Local qualification covers one user tailnet and the final Core revision. Browser UI acceptance, two unrelated tailnets, a hosted restart, and an agent task on the final staging revision are not yet verified. ## Risks - The enrollment method changes. It must pass the live qualification above before broad rollout claims. - Existing encrypted device state and the vault master key must survive a restart or redeploy. Device expiry or removal still requires reconnecting. - The existing one-serving-process-per-instance constraint remains. Concurrent replicas need durable network ownership. - Pending setup uses bounded progress requests and an event-driven CLI with a deadline. It adds no idle database scan. ## Model Used OpenAI GPT-6 Astra (`gpt-6-astra`) through Codex in T3 Code, with xhigh reasoning. Context window size was not exposed by the runtime. No subagents were used. ## Checklist - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] My branch name describes the change and contains no internal ticket or instance details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation - [x] I have considered and documented the risks - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
2178c3df88 |
fix(native): keep a slow but live Runner on retry instead of replacing it (#21462)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a recovery read to a remote Runner goes unanswered, the next attempt of the same run carries `replaceUnansweredRunner` > - That attempt verified the Runner was alive and then refused to adopt it anyway. On a resumed sandbox lease the reuse check then failed with `resumed_not_suspended` and left the Runner running. Otherwise a second Runner was launched, and its launch fence killed the busy Runner, which lost the provider turn. > - After a late reconnect a Runner is often slow, not dead: it is still replaying the events it saved with no controller attached > - This pull request makes the attempt check progress first. It adopts a live Runner that is making progress and replaces only a Runner that made no progress. > - The benefit is that a slow but working Runner keeps its turn, and the retry no longer causes the failure it was meant to fix. ## Linked Issues or Issue Description **What happened?** Hosted staging, release swaps on 2026-11-13. The retry after a first-attempt timeout failed with these codes (counts from the traced sample of 136 failures): - `runner_ingress_unavailable: runnerd process failed: runner_harness_state_mismatch: resumed_not_suspended` (part of 40 `ingress` failures) - `runner_remote_process_identity_unavailable` (25): the Runner was gone by the time the retry checked, because the launch fence or a previous retry had killed it - "restored the conversation after process loss, but…" / `provider_turn_lost_on_restore` (10): the replacement Runner restored from the old state and the provider turn was gone Runs that crossed a swap with a reconnect gap under 150 s: 0 of 9 failed. Over 420 s: 14 of 15 failed. The process ID of the Runner changed inside the same sandbox in the failed runs, which shows the replacement. Code path: `nativeRecoveryCommandUnanswered` grants an extra attempt with `replaceUnansweredRunner: true`. `verifyRemoteRunnerOrSettledExit` verifies the Runner is alive and then returns `undefined` because of that flag, so `adoptedProcess` is `undefined`. **Expected behavior** A live Runner that is making progress is adopted. A Runner that made no progress is stopped with proof and then restored from its own journal. **Steps to reproduce** 1. Reattach a remote run whose previous attempt left a recovery read unanswered (`replaceUnansweredRunner: true`), with the Runner alive and its saved event sequence past the controller's last acknowledgement. 2. On master the attempt launches a second Runner, or fails with `resumed_not_suspended` on a resumed sandbox lease. 3. The new tests in `native-session-executor.test.ts` and `native-runner-restart-recovery.integration.test.ts` do this. **Paperclip version or commit** `master` at |
||
|
|
1140984b82 |
feat(plugins): tell an opted-in workspace provider the run's environment (#21512)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A plugin can be a project workspace provider and also an environment
driver for the agent runs that use those workspaces
> - Core asks the workspace provider to prepare the project workspace
before the run acquires its environment lease
> - The provider knows the run, agent, project, and execution workspace,
but not the environment that the run selected
> - A provider whose compute exists per run (for example, a sandbox that
it creates from the environment's driver config) cannot start or find
the run's compute for the workspace work
> - This pull request lets a provider opt in to receive the run's
selected environment, when that environment is one of the provider's own
drivers
> - The benefit is that such a provider can prepare the workspace on the
run's own compute, with no change to the order of operations in Core
## Linked Issues or Issue Description
**What existing behavior does this improve?**
Plugin workspace preparation for runs whose environment is a driver of
the same plugin.
**Subsystem affected**
Plugin SDK workspace provider params, the plugin manifest workspace
provider declaration, and `callWorkspaceProvider` /
`preparePluginWorkspace` in the server.
**Current behavior**
`prepare` params carry `runId`, `agentId`, `projectId`, and
`executionWorkspaceId`. They carry `taskLease` only when Core acquired
the lease first. They never name the run's environment.
**Proposed behavior**
A provider that declares `workspaceProvider.workspaceEnvironmentVersion:
1` receives `environment: { id, driverKey, config }` when the run's
selected environment is a plugin environment of that same provider.
`config` is the environment's `driverConfig`, the value that the
driver's acquire receives.
## What Changed
- SDK: optional strict `environment` field on
`pluginWorkspaceParamsSchema`.
- Manifest: optional `workspaceProvider.workspaceEnvironmentVersion: 1`,
next to the sibling version fields. Core sends the new field only to
providers that declare it, because providers parse the params with the
strict schema.
- Server: `callWorkspaceProvider` adds `environment` only when the
provider opted in, the environment driver is `plugin`, its `pluginKey`
is this provider, and it names a `driverKey`. The field is not part of
the operation ID.
- Server: `preparePluginWorkspace` passes the run's environment to its
prepare call. This covers normal, verify-existing, fresh, and
replacement prepares.
- Calls that do not have the run's environment do not send it: finalize,
inspect, remove, reservation reconciliation, and the retry-candidate
verification in `fresh-workspace-continuation.ts`.
- Lease acquisition, the `sessionHandoffVersion` gate, and the order of
operations do not change.
- Docs: `doc/plugins/PLUGIN_RUNNER_EXECUTION.md` describes the
capability and its exceptions.
## Verification
- `packages/plugins/sdk/tests/workspace-provider.test.ts`: 11 passed.
The schema accepts the field and rejects malformed values.
- `packages/shared/src/validators/plugin.test.ts`: 65 passed. The
manifest accepts only version 1.
- `server`: `plugin-workspaces.integration`,
`plugin-fresh-workspaces.integration`, `project-environments`, and
`heartbeat/workspace-preparation`: 226 passed, against embedded
Postgres. The new integration test checks the exact shape, an unchanged
operation ID, verification prepares, and omission for an undeclared
provider, another plugin's environment, non-plugin drivers, a missing
`driverKey`, and no environment. It fails when the server stops passing
the environment.
- `pnpm typecheck` passes in `packages/shared`, `packages/plugins/sdk`,
and `server`.
- The full suite was not run locally. CI runs it.
## Risks
- Low risk. Providers that do not declare the capability get the same
params as before.
- `config` is the stored `driverConfig`, in the same form that the
driver's acquire gets. Secret references in it reach the provider in
that same form.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, code search, test execution).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [ ] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
21bf7ebcee |
feat(connectors): connect Tailscale networks to SSH environments (#20617)
## Thinking Path > - Paperclip manages work for AI agents. > - SSH environments run tasks on computers that users own. > - A Paperclip server needs network access before it can reach a private computer. > - A reusable Tailscale connection belongs in Connectors. > - Connected machines should become choices in the existing SSH environment form. > - This change connects a tailnet and resolves selected machines through a private network client. > - Agents retain their choice of harness and AI account. ## Linked Issues or Issue Description **Subsystem affected** Connectors, environment settings, SSH execution, company secret storage, and runtime packaging. **Problem or motivation** Entering a Tailscale address in the SSH form does not connect the Paperclip server to that tailnet. Requiring a separate device key for every environment also duplicates network setup. **Proposed solution** Add a Tailscale Connector with **Connect Tailscale Network**. Use brokered OAuth consent to enroll one private network client per connection. Let SSH environments select a connected network and stable machine identity. Preserve the enrolled node state through the company vault so a restart does not reuse the one-time token. **Alternatives considered** A separate Tailscale environment type would duplicate SSH behavior and mix network access with harness selection. This change uses the existing `ssh` driver. Related #15648 adds OAuth client credentials and a health check; this change uses user consent and supplies outbound task execution. Related #11743 concerns HTTPS brokering. **Roadmap alignment** This extends the existing Connected Apps, environment, and Personal & Shared AI Accounts capabilities. This PR is stacked on #21248, which provides SSH account selection independently of Tailscale. The hosted OAuth broker is a companion control-plane change. ## What Changed - Preserve Cloud reauthentication in the current browser tab. Explain canceled or failed consent in the connector and offer a retry of the same authorized saved connection. - Record Cloud enrollment starts initiated through the Tailscale connector in the company activity log. - Check install-preparation descendant liveness through the kernel process table in the Runner support test. Keep the direct child’s strict reaping assertion and reject any non-zombie descendant. - Align the application image stage contract with the Tailscale client stage after the upstream Runner-image refactor. Keep the exact stage list and check every application stage, including `tailscale-client`, for prohibited Runner compiler commands. - Group Tailscale consent, discovery, daemon lifecycle, pairing, and deletion under `server/src/services/connectors/tailscale/`, with their service tests. Update existing route, runtime, environment, vault, and company-deletion imports. Keep the existing contracts, authorization, audit, and transaction owners. - Handle early stdin closure in the Kubernetes real-shell test helper. Stop the input stream on `EPIPE`, preserve nonzero child exits, and reject a zero exit when input was lost. Add two real-process regression cases. Production transfer code and confinement checks are unchanged. - Read configured Tailscale executable paths when a client starts, after supported `.env` loading. Keep explicit command overrides available. - Discard pending pairing results after switching to Direct SSH or closing the form. Retain the existing company, connection, machine, and port checks. - Store network credentials with an immutable server-owned purpose from their first vault row. Exclude them from generic catalogs, bindings, environment projection, and value reads. Use a dedicated audited server consumer for pinned connector reads. Block unclassified legacy serialized node state at the same value boundary. Keep SSH private-key binding separate from connector credentials. - Gate network queries and setup on verified local-implicit or instance-admin access. Hide Tailscale catalog rows for other members, show an admin requirement on direct setup navigation, and avoid device discovery before management access is known. - Keep failed-client cleanup in the network manager. A failed request cannot disconnect a newer retry for the same identity; explicit connector disconnect and enrollment retirement still stop their networks. - Check for readable Ed25519, ECDSA, or RSA host public keys before pairing edits `authorized_keys`. Print a known-hosts entry for every available key, or fail before changing authorized keys if none are available. - Return self-hosted Cloud enrollment to the Tailscale connector. Save the unprefixed setup path and let the shared callback add the verified company prefix. Keep external and unrelated destinations rejected. - Hide pairing instructions after an SSH port change. Ignore late key-selection results for the old port. Operators prepare again for matching known-hosts output. - Retire connector references before deleting a company or its vault credential. Stop company daemons after the database commit. Keep failed secret retirement retryable and prevent late node-state writes from restoring deleted connections. Discard retired state without blocking behind the vault deletion lock. - Reload a mismatched credential before replacing a ready daemon. Recheck the active entry and manager generation after the read. An older request cannot stop a newer healthy socket or cross shutdown. - Pin connector credentials to the exact vault version and SHA-256 digest recorded by trusted enrollment and node-state writes. Ignore generic secret rotations. Advance both fields with a compare-and-set guard. Reject unpinned or tampered credentials before network startup. - Save an explicit Direct SSH selection and clear old Tailscale fields after config merging. Preserve the operator's entered direct host. - Route all Tailscale SSH runtime access through the connection service after SSH-key loading. Pin access to the environment's saved connector identity and recheck after device discovery. Remove standalone key-based daemon startup and reject raw enrollment keys or credential-only network configurations. Align the shared config type and SDK pins. - Correct two E2E assumptions exposed during connector qualification. Keep the two-window chat fixture stale until its own mutation. Validate the original failed run and its permitted write-revocation timestamp in the legacy recovery browser test. - Add the Tailscale Connector page, catalog entry, OAuth return flow, connected-network management, device discovery, and disconnect. Document all six routes in OpenAPI and the API reference, including their instance-admin requirement. - Add company-scoped connection storage in `0480_lazy_lethal_legion`, trusted credential version/digest fields in `0481_round_pet_avengers`, and server-owned secret purpose/backfill in `0482_aberrant_rumiko_fujikawa`. Generate all three migrations through the maintained command and retain the five newest snapshots. - Add machine pairing that generates a fresh SSH key per request, stores its private half in the company vault, and uses the generated public half in the install command. Do not trust editable saved metadata. Preserve existing authorized keys, including a file without a final newline. Keep SSH username, workspace, and trusted host keys explicit. - Use the existing SSH driver. Resolve the machine's current address from its stable device ID. Reject disconnected, removed, offline, and wrong-company selections. - Hide Network when Direct is the only option. Select a sole connected tailnet automatically and hide its selector. Remove the requested setup helper copy. Do not restrict the harness to Claude. - Run a private userspace Tailscale daemon with a separate socket. Persist its node state before execution. Restore it after restart, bound startup, serialize reconnect cleanup, and close it during server shutdown. - Guard OAuth completion with a durable attempt marker. Disconnect and newer authorization invalidate older callbacks. Close replaced daemons and remove cancelled enrollment secrets without changing newer connection state. - Preserve SSH host verification and tailnet-specific session identity. Honor the configured Tailscale executable path, including spaces and OpenSSH percent escaping. - Package checksum-pinned Tailscale clients and their license in the standard runtime image. Exercise the client build stage in PR CI and include it in image smoke checks. ## Verification - Merge status: all CI and image checks pass, and current-head Apex is 5/5 with no unresolved comments. GitHub rejected a direct merge because the active master ruleset requires code-owner approval for `.github/workflows/pr-trusted.yml`. The [CODEOWNERS workflow pattern](https://github.com/paperclipai/paperclip/blob/master/.github/CODEOWNERS#L3) applies; eligible maintainers already have review requests. Auto-merge is enabled with the squash method for the verified head. The PR remains open pending that required approval; no branch policy was bypassed. - Final revision: `fdb29b2cc366f4945b7f4c3e2b2d25070329a21b`, including master `999709b0a6dd416af335b5debded13e1de7fcb7a`. [Final CI verification passed](https://github.com/paperclipai/paperclip/actions/runs/43518584814/job/132213508320). All required checks pass, all applicable test/build gates are green, and the branch is mergeable against master. SSH prerequisite #21248 and Cloud broker paperclipai/paperclip-cloud#1314 are already merged. - [Apex reviewed this exact revision at 5/5](https://github.com/paperclipai/paperclip/pull/20617#issuecomment-6776629437). There are no unresolved review threads. The PR is classified as Critical impact. - Final image qualification passes for the [application Runner path](https://github.com/paperclipai/paperclip/actions/runs/43518584522/job/132206784267), [amd64 Runner image](https://github.com/paperclipai/paperclip/actions/runs/43518584544/job/132207296198), and [arm64 Runner image](https://github.com/paperclipai/paperclip/actions/runs/43518584544/job/132207296156). Typechecks, Runner support, native/Rust tests, provider tests, serialized-server suites, browser shards, and the remaining applicable CI checks pass. This is source/image CI qualification, not a production rollout. - The last master conflict affected only Docker stage declarations and their boundary test. Both the checksum-pinned Tailscale stage and upstream `runner-lockfile` guard are retained, including the upstream `deps` inheritance. The Tailscale client stage is byte-identical. All 180 cases across five Docker/image contract suites pass locally, along with server and UI typechecks. The maintained 543-input SDK check has no drift. Runtime and lock pins are unchanged. Secret/PII and whitespace scans pass. - Seven OAuth regression cases failed before the fixes. The current-tab test uses the real handoff helper, checks that the session is stored before reauthentication navigation, mounts the shared return page, resumes consent, and verifies handoff cleanup. Callback tests check trusted saved-connection context, fixed safe outcomes, failure audit metadata, denied membership and expired-state rejection, JSON API errors, and no provider text in browser returns. UI retries require a connection in the current company's catalog. All 80 OAuth route/service/UI/shared-handoff cases pass; server and UI source/test typechecks pass. - [Server shard 4/12 on the prior refresh](https://github.com/paperclipai/paperclip/actions/runs/43510727031/job/132180301436) passed 2,784 cases and skipped 21, then failed a reassignment fixture's `TRUNCATE companies CASCADE` cleanup with PostgreSQL `40P01`. Its automatic retry found the surviving `REA` prefix. The unchanged test passes all three cases locally. A controlled reader/cleanup overlap reproduces the same deadlock, but the exact competing CI query was not logged. Each case now has a distinct company prefix and identifier; the owned database is removed at suite end. The three test bodies and assertions are byte-identical. Eight local reassignment/lock-release cases pass, and the [final server shard 4/12 passes](https://github.com/paperclipai/paperclip/actions/runs/43518584814/job/132207159392). Production reassignment logic, deadlines, and retries are unchanged. - The initial master integration passed 330 cases across 17 suites. Coverage includes connector state, device discovery, network lifecycle, pairing, routes, SSH binding and exports, environment persistence, and connector UI. All three unpublished migrations were regenerated after master's `0479_short_darkstar`: `0480_lazy_lethal_legion`, `0481_round_pet_avengers`, and `0482_aberrant_rumiko_fujikawa`. Each SQL body is byte-identical to its earlier unpublished migration. Generated snapshots preserve existing master tables and add only the intended Tailscale table and secret-purpose column. Exactly five snapshots remain, 0478–0482. Numbering, safety, compatibility, and snapshot-drift checks pass. The later Docker refresh did not alter this migration sequence. - Earlier attempts remain part of the verification record. On `1c020a0`, Rust cache restoration and server shard 5/12 were interrupted by shutdown signals, Runner shard 2/3 lost communication with no retrievable log, and an amd64 provider download failed with npm `ECONNRESET`. Same-head retries recovered, without establishing the cause of the communication loss. Later superseded runs also had cancellations with incomplete logs. These interrupted attempts are not counted as passes. The final revision has fresh successful CI. - Local platform limits remain recorded. The Kubernetes suite on macOS passed 191 cases, failed 12 Linux `/proc/self/fd` transfer cases, and skipped four Linux-only cases. Unchanged source reproduced the same 12 failures. An initial run also lacked provider dependencies until the maintained preparation command installed them. Those local attempts are not green qualification. Final Linux provider CI passes. Local Docker qualification was unavailable; final image checks passed in CI. - Before PR separation, a live tailnet/SSH Claude Code task completed on Ubuntu ARM64 without provider login for the remote user. The selected Paperclip subscription ran it, wrote the expected file, posted completion, and marked the task done. File contents and private runtime cleanup were checked. This is prior live evidence; the final revision was not live-retested. Other providers have automated coverage only. Hosted registration for unrelated tailnets and production concurrency remain qualification limits below. - The Superagent security and supply-chain reports remain neutral and inconclusive. Earlier attempts to restart the failed security scan returned HTTP 404. Supply-chain intelligence lacked coverage for extracted pinned dependencies and the shared workflow reference. These reports do not establish successful security qualification. The required CI and review gates pass; this external coverage gap remains disclosed. ## Risks - The Superagent security and supply-chain reports are inconclusive. The security scan re-run request returned HTTP 404, and the supply-chain scanner lacks coverage for the extracted dependencies and shared workflow reference. These neutral results are not successful security qualification. - The server-owned secret purpose is internal database metadata, not an HTTP-writable field. The generic secret service owns enforcement so direct API reads and runtime projection share the same restriction. Previously copied node state remains valid outside Paperclip until the enrolled device is revoked in Tailscale; a local disconnect cannot stop an external copy. - Tailscale network configurations now require a connected Connector and selected machine. The unreleased key-only path is removed; any saved configuration from that path needs Connector setup. - The connector remains a pilot. The local test used an OAuth app registered in the consenting user's tailnet. Qualify hosted app registration for unrelated customer tailnets before general availability. - This implementation supports one serving process per instance. Concurrent replicas need exclusive ownership of the persisted network identity before production rollout. Preserve the database and company vault master key across restarts. - Each machine-pairing preparation creates a new vault secret. Unused generated keys follow the existing secret deletion lifecycle. A saved SSH key remains available as an explicit choice. - Apply additive migrations `0480_lazy_lethal_legion`, `0481_round_pet_avengers`, and `0482_aberrant_rumiko_fujikawa` before application rollout. The new credential version and digest columns are nullable. Existing unpinned connections require fresh OAuth consent; the migration deliberately does not trust or backfill a generic secret value. Regenerate or renumber migrations if master advances the sequence before merge. - The target needs an SSH server, its chosen harness, and allowed tailnet access. A reachable network does not grant SSH or provider authorization. Pairing reads standard `/etc/ssh` host-public-key files; custom host-key locations require manual known-hosts configuration. - Roll back the application revision to remove new entry points. Existing connection secrets and registered Tailscale nodes retain their separate lifecycle. Do not drop the table or delete node state as part of an application rollback. - Production qualification still needs denied-policy, revocation, cancellation, and deployment-overlap checks. Local Docker and a production deployment were not performed. ## Model Used OpenAI Codex, gpt-6.1-sol, xhigh reasoning, with local code execution and tool use. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used - [x] I have checked ROADMAP.md and this extends existing connector and environment features - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR using the feature request fields - [x] I have not referenced internal issues, local URLs, or private machine identifiers - [x] My branch name describes the change - [x] I have run focused tests locally and they pass - [x] I have added tests for new behavior and failure paths - [x] I have updated documentation - [x] I have documented migration and runtime risks - [x] All Paperclip CI gates are green - [ ] Required code-owner approval received for the workflow change - [x] Greptile Apex is 5/5 on the final head with no unresolved comments - [x] I will address CI and review findings before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
647641a99b |
feat(ci): let docker.yml copy the published Runner image, behind a repository variable (#21340)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner is the part that executes agent work. `master` publishes it alone as an attested image, once for each set of its inputs > - Since #21332 the application image takes its Runner only from the `runner-artifacts` stage, and a build can replace that stage with the published image > - No workflow does that yet, so every application image still builds its own Runner, and nothing on an application image says which Runner it has > - A build that copies the published Runner must also build the lockfile that Runner was built from. The workflow refreshes the lockfile before every build, and the server build then refuses the published Runner > - This pull request lets `docker.yml` copy the published Runner image and label the application image with it, behind a repository variable that is off, and makes that build use the committed lockfile > - The benefit is one Runner for all images of the same inputs, named on the image, with a switch that goes back to today's builds without a revert ## Linked Issues or Issue Description Refs #21204, #21229, #21230 (the Runner image), #21306 (the resolver), #21307 (the report-only job), #21308 (the server reads a pack from its Runner image), and #21331, #21332, #21333 (the application image takes its Runner from `runner-artifacts`). No issue exists for this change. **Problem or motivation** `docker.yml` builds the application image on `master` and on tags. Each build makes its own Runner. The report-only job that runs the resolver on `master` feeds no build. Downstream image composers cannot tell from an application image which Runner it has. There is a second problem, which only shows when a build copies the published Runner. The Runner's build stamp covers the whole `pnpm-lock.yaml`. The step that vendors the Runner into the server fails when the stamp does not match the tree. `docker.yml` refreshes `pnpm-lock.yaml` on the runner before every build. The refresh rewrites the file whenever the manifest of any package is ahead of it, also a package the Runner does not read. On the tree of this pull request it rewrites the file even for a pristine checkout: the registry changed the deprecation text of one package. The published Runner was built from the committed file, so the image build would fail. **Proposed solution** One job resolves the Runner once for the run. Both architecture builds need it and add its answer to their build: a build context that replaces the `runner-artifacts` stage, two labels, and the build argument with which the build checks the Runner. A build that copies the published Runner does not refresh the lockfile: it builds the committed tree, the tree the Runner was published for. The job runs only when the repository variable `RUNNER_IMAGE_CONSUME_ENABLED` is exactly `true`. This pull request merges with the variable unset, so no image changes when it merges. **Alternatives considered** - Resolve in each architecture build. The two builds could then disagree, and one image index would hold two Runners. - Write the labels in the `Dockerfile`. A `LABEL` cannot be left out for an inline build. - For the lockfile, copy the Runner image's `/pnpm-lock.yaml` over `/app/pnpm-lock.yaml` in the `build` stage. Nothing after that step reads the file: the two vendor checks are its only readers in `build`, `production` copies it and takes `node_modules` from `deps`, both `cloud-*` stages install with `--ignore-workspace --no-lockfile`, and the server reads only a plugin's own lockfile and the pack's own lockfile at run time. But the image would then ship a lockfile that its `node_modules` did not come from, the copy changes that file's layer for every inline build, and each `Dockerfile` edit makes a new Runner image. - For the lockfile, bind-mount the Runner image's lockfile over the file for the two steps that run the vendor check. This also edits the `Dockerfile`, and it makes the lockfile part of the check always true for a published Runner. - For the lockfile, let the resolver refresh and compare. The answer would depend on the registry at two different moments. - Skipping the refresh needs no `Dockerfile` edit, keeps the stamp check as strict as it is, and builds exactly the tree the inputs digest was computed from. **Roadmap alignment** This is build infrastructure. It adds no product feature and does not overlap an item in `ROADMAP.md`. This pull request is built on `runner/app-image-consume`. Merge in this order: 1. #21331, #21332, #21333: the application image takes its Runner from the `runner-artifacts` stage, checks the inputs digest a build names, and stops early on a stale lockfile. 2. `runner/app-image-consume`: the wait and the `unsupported` answer of the resolver. 3. This pull request. The same change for the preview cloud image build in `release.yml` is deferred. It has no pull request yet. ## What Changed - `.github/workflows/docker.yml`: the job `runner_image` replaces `runner_image_report`. - It runs only when `vars.RUNNER_IMAGE_CONSUME_ENABLED == 'true'`, in this repository. A fork never runs it. - Steps: `inputs` with no credential; `resolve`; then `gh attestation verify` of the full index against the signer `runner-image.yml` on `refs/heads/master`. - For a `master` push, `resolve` runs with `--wait` and the job token. For a tag or a manual run it does not wait and receives no token. - Only the verification step writes the job's outputs. An image that does not verify fails that step with a warning and writes no output. The job is `continue-on-error`: the job is red, the run is not, and the builds are inline. - Permissions: `contents: read`, `actions: read`, `attestations: read`. No registry login, no write. - Limits: 5 minutes for `inputs`, 55 for the lookup, 5 for the verification, 70 for the job. The wait of 40 minutes ends inside them. - `build-and-push` needs `runner_image` and adds its outputs to the build: `build-contexts`, one more line of `build-args` (`PAPERCLIP_RUNNER_EXPECT_INPUTS_DIGEST`), two more lines of `labels`. With empty outputs these are the values the build had before. - The step "Refresh lockfile for Docker build context" runs only when the answer is not `published`. - The step that looks for a merge-queue image of the commit runs only when the answer is not `published`. The two steps after it already depend on its result. - Labels on an image that copied the published Runner: `io.paperclip.runner.image=ghcr.io/paperclipai/paperclip-runner@sha256:<full index digest>` and `io.paperclip.runner.inputs-digest=<digest>`. An image without them made its own Runner. - The `Dockerfile` does not change. The Runner image digest does not move. - Tests that pin lines this changes follow: `scripts/merge-group-image.test.mjs` (the condition of the merge-queue lookup), `scripts/reuse-canary-image.test.mjs` (the `needs` of the build job), and `docker-build-stamp.test.ts` (no workflow names a build context; `docker.yml` now names exactly one). - `server/src/__tests__/docker-runner-workflows.test.ts`: the report-only tests give way to tests of the new job. - `server/scripts/prepare-runner-vendor.test.mjs`: one fixture test of the lockfile case. - `doc/runner-image-contract.md` and `doc/DOCKER.md`: how an application image gets its Runner, the labels and what their absence means, the variable, the wait, the verification, the lockfile rule, the merge-queue rule and the rollback. ## Verification ### Real builds `docker buildx build --target build --platform linux/arm64 --output type=cacheonly`, with a `docker-container` builder made for this test. The Runner image is the publication form (`PAPERCLIP_RUNNER_IMAGE_PUBLICATION=1`, `--target runner-artifacts`) built from a clean export of this branch, exported as an OCI layout: inputs `c4f9d089…34f0`, image `sha256:bef7e9cb…ffef`. The application builds receive `--build-context runner-artifacts=oci-layout://…@sha256:bef7e9cb…` and `PAPERCLIP_RUNNER_EXPECT_INPUTS_DIGEST=c4f9d089…`. The refresh command is the one in the workflow, run with pnpm 9.15.4 in the export: `pnpm install --resolution-only --ignore-scripts --no-frozen-lockfile`. | Build context | Result | | --- | --- | | The committed tree | Passes in 78 s. Both vendor checks: `reusing the current Runner build (inputs unchanged …; 1333 files and 1860 outputs checked)` | | `ui/package.json`: `clsx` `^2.1.1` → `^2.1.0`, then the refresh command (what the workflow did before this change) | Fails at `RUN node server/scripts/prepare-runner-vendor.mjs`: `PAPERCLIP_RUNNER_VENDOR_REQUIRE_CURRENT=1 forbids the Runner rebuild this step needs (inputs changed since the recorded build; 1333 files checked)` | | The committed tree, then only the refresh command | Fails the same way. The refresh changed only the `deprecated` text of one package | | The same `ui/package.json` change with no refresh (what the workflow does now for a `published` answer) | Passes in 66 s. `deps` resolves the manifest inside the image; both vendor checks reuse the Runner | | One line added to `packages/paperclip-runner/src/index.ts` | Fails at the vendor step, as before: a Runner of other inputs is still refused | Lockfile behind the Runner's own manifests (`packages/paperclip-runner/package.json`: `smol-toml` `^1.9.1` → `^1.9.0`), no build context: | Step | Result | | --- | --- | | `runner-image-resolve.mjs inputs`, then `resolve` | `lock: behind`, no digest; `inline (lock-behind)` | | Build without the refresh | Fails in 4 s at the `runner-lockfile` stage: `ERROR: pnpm-lock.yaml is behind the manifests the Runner is installed from …` | | Build after the refresh command (what the workflow does for an `inline` answer) | Passes in 193 s: the `runner-lockfile` stage passes, the build makes its own Runner with its pack, and both vendor checks reuse it | ### Tests macOS (Node 26.12.0) and Linux (`node:24.22.3-bookworm`, linux/arm64, with `jq` and `zstd` added), on this branch: - `pnpm exec vitest run` on `docker-runner-workflows`, `runner-image-resolve`, `runner-image-workflow`, `docker-build-stamp`, `container-init-reaping`, `cloud-image-bundled-plugins`, `docker-image-layer-reuse`, `cloud-image-sentry`, `runner-lockfile-guards` and `runner-image-identity` in `server/src/__tests__`, and `server/scripts/prepare-runner-vendor.test.mjs`: 225 tests in 11 files pass on Linux; on macOS 224 pass and 1 is skipped (it needs bash 4.4). - `node --test .github/scripts/tests/*.test.mjs scripts/merge-group-image.test.mjs scripts/reuse-canary-image.test.mjs scripts/cloud-source-verification.test.mjs scripts/standard-image-contract.test.mjs scripts/preview-artifacts.test.mjs scripts/__tests__/release-verify-workflow.test.mjs scripts/__tests__/release-dry-run-notes.test.mjs`: 991 of 991 pass on both. - The changed test files typecheck with the options of `server/src/__tests__/tsconfig.json`. What the workflow tests hold: - Variable off: the job's condition is the variable and the repository, with no alternative. With empty outputs, the build step has exactly the 3 build arguments, 0 build contexts and 1 label entry it had before, and the refresh step runs. - Published: both architecture builds are one job that reads only the outputs of `runner_image`, with no architecture in those lines. - The verification step runs as real shell against a stub `gh`. When `gh` succeeds it writes the context, the labels and the build argument. When `gh` fails the step exits 1, writes no output and prints the warning. The step has no `continue-on-error`; the job has. The condition of the build job does not name the resolver job and has no `success()`. - The lookup step runs as real shell against a stub `node`. It passes `--wait` only when the event is a `master` push. - Credentials: only the lookup and the verification steps have a token. The steps that build have no `env`. - The refresh step has the condition `needs.runner_image.outputs.mode != 'published'`, and it is the only `pnpm install` of the build job. - The fixture test gives the vendor step a Runner that another tree built for the committed lockfile. The step reuses it. After a change of one specifier of another package in the lockfile the step fails and names the lockfile. With the committed lockfile back it reuses the Runner again. - The merge-queue workflow and the pull-request check do not name the resolver, the variable, a build context or a Runner label. Read-only checks against the real services: - `gh attestation verify oci://ghcr.io/paperclipai/paperclip-runner@sha256:0cd8250b…1bf7 --repo paperclipai/paperclip --signer-workflow paperclipai/paperclip/.github/workflows/runner-image.yml --source-ref refs/heads/master`: exit 0 for the full index, and exit 0 for the slim index `sha256:69ab2001…69b3`. - The same command with `--signer-workflow …/docker.yml`: exit 1 for both. With `--source-ref refs/heads/not-master`: exit 1. With `--repo cli/cli`: exit 1. For a digest that does not exist: exit 1. With no token: exit 4. - The parsed YAML of `docker.yml` differs from its base only in the resolver job, the `needs` of the build job, the conditions of the refresh step and of the merge-queue lookup, and the three build inputs. Never run: - `docker.yml`. Only GitHub can run it, and it has no pull-request trigger. - A build with the variable on. The first one is the first run after a maintainer sets the variable. - A build that pulls the published Runner from the registry as `docker-image://`. The real builds used the same image form from a local OCI layout. - An `amd64` build. - The `production` and `cloud` targets. The real builds stopped at `build`, the last stage that checks the Runner. ## Risks - With the variable on, an application build on `master` starts after the resolver job. That is 2 to 3 minutes when the image exists, and up to 40 minutes from the push when the commit's own Runner publication is in progress. `docker.yml` holds one concurrency slot for each ref, so a long wait can make more `master` commits lose their image build to a newer push. - A build that copies the published Runner ships the committed `pnpm-lock.yaml` in `/app`, while `deps` resolves newer manifests of other packages inside the image. A local `docker build` has always done this; a CI image did not. Nothing in the image reads that file. - Any other build that gives the `Dockerfile` a published Runner and a refreshed lockfile fails at the vendor step. That is the intended answer to a tree the Runner was not built for, and the message names the lockfile. - The pack of an image that copied the published Runner names the commit that published the Runner (`runnerSourceRevision`), which can be older than the commit of the image. A consumer that compares the two must accept that before the variable is set. - A failed verification makes the resolver job red on a green run. That is on purpose: it means a published tag whose attestation does not verify. - `scripts/merge-group-image.mjs` refuses to promote a merge-queue image for a push that changes `docker.yml`. The push that lands this change is such a push. The merge queue is not enabled. - With the variable off there is no report of the resolver on `master` any more: the job that replaced the report-only job does not run. - Rollback: set the variable to another value or delete it. The next run builds inline, with the refresh. Images that exist do not change. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude Code with tool use (shell, file edits, local Docker builds, read-only calls to the GitHub API and the container registry). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
e824c5cc16 |
feat(ci): let the Runner image resolver wait for the publication of a master push (#21339)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner is the part that executes agent work. `master` publishes it alone as the image `ghcr.io/paperclipai/paperclip-runner`, once for each set of its inputs > - The application image will copy that Runner instead of making its own. `scripts/runner-image-resolve.mjs` tells a build whether the Runner of its inputs is published > - On a `master` push, the commit's own run of `runner-image.yml` is often still publishing when the application build asks. The resolver answers once, so that build is told `absent` for an image that exists some minutes later > - The resolver also cannot say that a tree is not safe to give a published Runner to. A `Dockerfile` that does not check the Runner it receives builds with any Runner > - This pull request adds an opt-in wait for the commit's own publication run, and an `unsupported` answer for a `Dockerfile` that cannot check its Runner > - The benefit is that the next pull request can let `docker.yml` copy the published Runner without a blind timeout and without a label that is not true ## Linked Issues or Issue Description Refs #21204, #21229, #21230 (the Runner image), #21306 (the resolver), #21307 (the report-only job), and #21331, #21332, #21333 (the application image takes its Runner from `runner-artifacts`). No issue exists for this change. **Problem or motivation** A `master` push starts `runner-image.yml` and `docker.yml` at the same time. Publication takes about 30 minutes from the push, and several of those minutes are runner queue time. An application build that asks the resolver at once gets `absent` for every commit that changes the Runner's inputs, and for commits that follow one closely. A fixed poll of the tag has the opposite fault: when no image will ever come (the lockfile is behind, or the run failed), every build waits the full timeout. **Proposed solution** `resolve --wait` waits for a run, not for a time. It finds the runs of `runner-image.yml` for the push of the commit to `master`, and it looks up the image again when they end. No caller uses the flag in this pull request, so no workflow changes behavior. **Alternatives considered** - Poll the tag until a timeout. Every build with no image to come waits the full timeout. - Call `runner-image.yml` from `docker.yml` as a reusable workflow. Publication would then share the one concurrency slot of `docker.yml`. - Wait until the run completes and only then look. The run goes on after publication (it writes the build cache), so the wait would last longer than needed. The command looks up the image while a run is in progress. - Verify the attestation in the resolver. The workflow that uses the answer does it, in a step of its own, in the next pull request. **Roadmap alignment** This is build infrastructure. It adds no product feature and does not overlap an item in `ROADMAP.md`. This pull request is built on #21333. Merge in this order: 1. #21331, #21332, #21333: the application image takes its Runner from the `runner-artifacts` stage, checks the inputs digest a build names, and stops early on a stale lockfile. 2. This pull request: the wait and the `unsupported` answer of the resolver. 3. `runner/app-image-consume-docker`: `docker.yml` copies the published Runner, behind a repository variable. The same change for the preview cloud image build in `release.yml` is deferred. It has no pull request yet. ## What Changed - `scripts/runner-image-resolve.mjs` - `resolve --dir <directory> --wait`. When the first answer is `absent` or `incomplete`, the command lists the runs of `runner-image.yml` for the commit (Actions API, `GITHUB_SHA`, `GITHUB_TOKEN`) and looks again when they end. - Only a run of the `push` event on `master` in this repository counts. A pull-request, scheduled or manual run of the same commit is never waited for. The request filters by event and branch, and the script checks each run again. - A run that fails, is cancelled or publishes nothing ends the wait when it ends. With several runs for one commit, the command waits for all of them. - While a run is in progress, the command looks up the image every 30 seconds, so the wait ends at publication. - A run that is not listed yet is looked for during 2 minutes. - The limit is 40 minutes from the push: `RUNNER_IMAGE_WAIT_MINUTES`, the one constant. The push time is the creation time of the oldest run of the commit, so a build that starts late waits less. - Three failed API requests in a row end the wait. No token or no full commit SHA means no wait. - After every ending the command looks up the image one last time. No ending is an error. Without an image the answer is `inline`, and the command prints a notice with the reason. - `inputs` reads the `Dockerfile` first. Without a line `ARG PAPERCLIP_RUNNER_EXPECT_INPUTS_DIGEST` it runs no build, and `resolve` answers `inline` with the reason `unsupported`. - `resolve` reports `build_args`: the value of that argument for a `published` answer. - The wait and the list of runs take the clock, the sleep and `fetch` as arguments, so the tests use no network and do not sleep. - `server/src/__tests__/runner-image-resolve.test.ts`: every outcome of the wait and the answer that follows it, the run filter, the limit measured from the push, the API failures, and the command line. - `doc/runner-image-contract.md`: a "Waiting for a publication" section, `build_args`, and the `unsupported` answer. ## Verification macOS (Node 26.12.0) and Linux (`node:24.22.3-bookworm`, linux/arm64, with `jq` and `zstd` added), on this branch on top of #21333: - `pnpm exec vitest run` on `runner-image-resolve`, `docker-runner-workflows`, `runner-image-workflow`, `docker-build-stamp`, `container-init-reaping`, `cloud-image-bundled-plugins`, `docker-image-layer-reuse`, `cloud-image-sentry`, `runner-lockfile-guards` and `runner-image-identity` in `server/src/__tests__`, and `server/scripts/prepare-runner-vendor.test.mjs`: 218 tests in 11 files pass on Linux; on macOS 217 pass and 1 is skipped (it needs bash 4.4). - `node --test .github/scripts/tests/*.test.mjs scripts/merge-group-image.test.mjs scripts/reuse-canary-image.test.mjs scripts/cloud-source-verification.test.mjs scripts/standard-image-contract.test.mjs scripts/preview-artifacts.test.mjs scripts/__tests__/release-verify-workflow.test.mjs scripts/__tests__/release-dry-run-notes.test.mjs`: 991 of 991 pass on both. - The changed test file typechecks with the options of `server/src/__tests__/tsconfig.json`. Read-only checks against the real services: - The lookup, with no credential, answered `published` for the inputs `d4df566f…af00` and `0e6662ad…d1ca`, with the index digests that `master` published, and `absent` for a digest that does not exist. - The real answer of the Actions API for the `runner-image.yml` runs of four `master` commits went through the wait, with a clock that does not sleep. Three completed runs ended it at once. A queued run ended it at the limit of 40 minutes from its creation time. A commit with no run ended it after the 2 minutes for a run that is not listed. - Of the 60 newest runs of `runner-image.yml`, the filter accepted the 19 `push` runs on `master` and none of the 41 pull-request runs. - `runner-image-resolve.mjs inputs` ran on a clean export of the tree of the next pull request, with a `docker-container` builder. The answer was `lock: current` and the inputs digest `c4f9d089…34f0`. The `Dockerfile` of #21332 declares the argument, so the answer was not `unsupported`. The same command on a tree with one changed range in `packages/paperclip-runner/package.json` answered `lock: behind`, and `resolve` then answered `inline (lock-behind)`. Never run: - `resolve --wait` in a workflow. No workflow calls it in this pull request. - The full test suite and the full typecheck. CI must supply them. ## Risks - `resolve` now prints a notice for every `inline` answer. The report-only job in `docker.yml` therefore shows one notice on each `master` push whose Runner image is not published yet, until the next pull request replaces the job. Nothing reads that job. - `inputs` answers `unsupported` for a tree whose `Dockerfile` does not declare `ARG PAPERCLIP_RUNNER_EXPECT_INPUTS_DIGEST` at the start of a line. #21332 adds that line; a later edit that moves or renames it turns every answer to `inline`. - The wait is bounded by time from the push, not by its own duration. A build that starts more than 40 minutes after the push looks once and does not wait. - A registry that does not answer adds at most about 3 minutes to one lookup, because the registry client retries. A caller must give the step more than 40 minutes. The next pull request gives it 55. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude Code with tool use (shell, file edits, read-only calls to the GitHub API and the container registry). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
999709b0a6 |
fix(docker): stop a stale-lockfile build early and keep beta and stable releases buildable (#21333)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The application image now takes its Runner from the `runner-artifacts` stages, which install with a frozen lockfile > - A contributor cannot commit `pnpm-lock.yaml`. After a change to a manifest of the Runner, the lockfile on `master` is behind until the refresh merges > - In that time a plain `docker build` fails late, inside a long step, with a message that is easy to miss. And a beta or a stable release can be cut from such a commit, because the release workflows refresh the lockfile for their own build > - This pull request adds an early check with a clear instruction to the image build, and the same check to the beta and the stable release > - The benefit is a failure in seconds that says what to run, and beta and stable tags that always build from source ## Linked Issues or Issue Description This is the third part of a chain. The first part is #21331. The second part takes the Runner of the application image from the `runner-artifacts` stage. Refs #21204 and #21306 (the Runner image and its lockfile rule). No issue exists for this change. **What happened?** A tree with a lockfile behind the manifests of the Runner fails `docker build` after the daemon compile (169 s in the test). The error is `ERR_PNPM_OUTDATED_LOCKFILE` in the middle of the largest `RUN` of the `Dockerfile`. **Expected behavior** The build fails within seconds of the check running. The message names the files and the command to run. The application's dependency install starts from the check, so it cannot start before the check passes. Stages that do not depend on it (the Runner's toolchain download and daemon compile, and the tool layer of `production`) can already be running, and the failed check cancels them. A beta or a stable release from such a commit does not start. How often this happens: of 72 nightly tags since 2026-08-24, 9 are on a commit whose lockfile is behind the manifests of the Runner (2026-09-01, 09-18 through 09-23, 09-28 and 10-26). Of 12 beta tags, none is. A beta is the promotion of a nightly, so a beta can be cut from such a commit. **Steps to reproduce** 1. Change a dependency range in `packages/paperclip-runner/package.json`. Do not refresh `pnpm-lock.yaml`. 2. Run `docker build .`. **Paperclip version or commit** `master` with the previous pull request of this chain. **Deployment mode** Not applicable. This is the image build and the release tooling. ## What Changed - `Dockerfile`: a `runner-lockfile` stage copies only the manifests the Runner build installs from, the lockfile and the patches. It runs the frozen, filtered, offline lockfile check, with the arguments of `runnerLockState`. `deps` starts from it. For a stale lockfile it prints the files and `pnpm install --resolution-only --ignore-scripts --no-frozen-lockfile`. For any other failure it prints the output of pnpm and says so. - `scripts/check-release-runner-lockfile.mjs`: exports the Runner inputs of a commit with `git archive` and gives them to `runnerLockState` (`scripts/runner-image-publish.mjs`). Exit 0 for current, 1 for behind, 2 for an error. - `scripts/release.sh` and `scripts/release-lib.sh`: `require_runner_lockfile_current` runs for the `beta` and the `stable` channel, after `--print-version` exits and before any build. It runs in a dry run too. It does not run for `canary` or `nightly`. - Both beta paths end in the `publish_beta` job: the promotion of a nightly, and a `candidate/beta-*` branch. That job checks out the commit that `select_beta` chose and runs `release.sh beta` there. So the check reads the committed files of the commit the beta is built from. A beta is refused before its soak starts. The stable that promotes it is checked again. - Tests: `server/src/__tests__/runner-lockfile-guards.test.ts` (18 tests). Three of them pin the checkout of `publish_beta`, `preview_stable` and `publish_stable` to the selected commit. Two tests plus one stub in `scripts/__tests__/release-dry-run-notes.test.mjs` cover the beta, the candidate beta and the stable path. - Docs: `doc/DOCKER.md`, `doc/RELEASING.md`, `doc/runner-image-contract.md`. ## Verification Builds: `linux/arm64`, local. They ran before the rebase onto `master`, except the last one. `master` changed neither the `Dockerfile` nor a manifest in between. - A tree with `ajv` changed from `^8.20.0` to `^8.19.0` in the manifest of the Runner: `docker buildx build --target build` fails after 6 s at `runner-lockfile`, with the message. - The same tree after `pnpm install --resolution-only --ignore-scripts --no-frozen-lockfile` on the host, which is what the image workflows run: `--target build` builds. The check passes, the Runner stages install, and the vendor step logs `reusing the current Runner build` in both of its runs. - After the rebase: `build` and `runner-provider-pack` built from this branch without a build commit. The check passed. - The release check with the real pnpm: `behind` for the first tree, `current` for the second, exit 0 for this checkout. - `vitest run server/src/__tests__/runner-lockfile-guards.test.ts` with the other changed server tests, on macOS and in `node:24.22.3-bookworm` (249 pass in 11 files on Linux). The test file typechecks. - `node --test scripts/__tests__/release-dry-run-notes.test.mjs`: 14 pass. `scripts/release-lib.test.mjs`: 15 pass on Linux; on macOS 4 of its tests fail on `master` too. `release-verify-workflow.test.mjs` and `acpx-patch-packaging.test.mjs`, which read the release scripts: pass. - `docker buildx build --check`: no warning. - The Docker builds above ran before the second part of the chain got its expectation step and before the beta guard. The expectation step was built on the second part alone. This branch as it is now was not built again; its tests ran. - Not run: the release workflow, any `linux/amd64` build, the full test suite. Only CI and a release dry run can prove the release path. ## Risks - **A beta can be refused.** When the newest nightly is on a commit whose lockfile is behind, `release.sh beta` stops, in the dry run and in the real run. The operator must promote a nightly whose commit includes the lockfile refresh, or cut a candidate branch that includes it. `node scripts/check-release-runner-lockfile.mjs --commit <ref>` gives the answer before the dispatch. - **A stable can be refused** only when its beta was cut before this change, or with release tooling that does not have the check. Promotions run the release tooling of the source commit. - The check runs in `release.sh`. In the beta workflow it runs in `publish_beta`, after the approval of the `npm-beta` environment and after the candidate verification. In the stable workflow it runs after verification, not in the `preflight_stable` job. The selection jobs have no pnpm. - `release.sh stable` now needs `git`, `tar` and `pnpm` on `PATH` before its first build step. The workflow has them. - `scripts/__tests__/release-dry-run-notes.test.mjs` is not in a CI command today. The Vitest file carries the enforced tests. - The image build has one more small stage. It reads only manifests, so a source change does not run it again. - The digest of the Runner image changes again, because its identity covers the whole `Dockerfile`. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), from Anthropic, through Claude Code. The model used tools: it edited files, ran the tests and ran the Docker builds that this description reports. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
05fef9f27b |
perf(server): add a load model of per-agent server cost and the plan for 1,000 concurrent agents (#21418)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - One server process controls every agent run of a workspace > - A hosted workspace slows down with about 50 agents running at the same time > - The target is 1,000 agents at the same time, served by several server processes > - This pull request adds the plan and a local load model that measures what each running agent costs the server > - The benefit is a measured ranking of the causes, and a repeatable before/after number for each fix ## Linked Issues or Issue Description **What existing behavior does this improve?** The server controls agent runs. With about 50 runs active, requests wait and run events fall behind. **Current behavior** Each text delta from an agent is one database transaction. In the load model, 50 agents at 20 deltas per second issue about 3,900 statements per second, and the process completes about half of the offered deltas. Agent authentication slows from 7 ms to 464 ms because it shares the connection pool. **Proposed behavior** The plan gives three phases: measure, decrease the cost per agent, then let several server processes own runs. Each fix is one pull request with a test and a before/after number from the load model. **Reason and benefit** The ranking was an estimate from code. The load model replaces the estimate with a measurement and gives each fix a number to improve. ## What Changed - Added `doc/plans/2026-11-13-thousand-concurrent-agents.md`: finish line, evidence, load model result, phases, and owner decisions. It now also has a "Load model after Phase 1" section. That section has a table at 50, 100, 200 and 400 agents and a row at 200 agents with deltas and leases as before Phase 1. The 400-agent delta count was measured before the count fix and is marked as not reliable. - Added `scripts/benchmark-agent-scaling-load.ts`: N simulated agents call the real event append, controller checkpoint save, lease renewal, and agent authentication code against an embedded PostgreSQL. It reports statements for each database pool, event-loop utilization, and lateness for each source. - Deltas: by default the model commits deltas as the server port does after #21460. The first text after another event commits at once. Later deltas commit in one transaction (`appendHeartbeatRunEventBatch`) per 250 ms or 100 deltas, and the stream does not wait for the receipts. `DELTA_BATCH_MS=0` keeps one transaction per delta, as before batching. - Leases: by default the model renews every lease through the shared schedule from #21444 (`startNativeSessionExecutionLeaseRenewal`, `renewNativeExecutionLeasesNow`). `LEASE_MODE=single` keeps one transaction per run, as before. - The commit counter is read before the checkpoint drain, so commits that come after the window do not count in it. - Per-source counts freeze when the window closes. An operation that settles later, while queued work drains, is not done. - Batch lease mode needs `LEASE_S` of at most 20. The shared schedule renews a remote lease on its own 20 s after its last renewal, and the model does not count that work, so the script rejects a longer interval. - Added `server/src/__tests__/agent-scaling-load-model.test.ts`. It runs the model end to end in both modes: the default (batched deltas and shared lease renewal) and `DELTA_BATCH_MS=0 LEASE_MODE=single`. It checks that every source finishes without failures and that each pool is counted. ## Verification - `pnpm --filter @paperclipai/server exec vitest run src/__tests__/agent-scaling-load-model.test.ts` on a tree that contains #21460 and #21444: 2 of 2 pass (both modes). The test also checks that no source reports more done than offered. - `pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-agent-scaling-load.ts` with `AGENTS=50 SECONDS=20` and higher agent counts. The results are in the plan. - `pnpm check:module-boundaries` passes. - No product code changed. ## Risks Low risk. The pull request adds a document and a script that no product code imports. The load model runs on loopback PostgreSQL, so its absolute numbers are not a prediction for a hosted database. The plan states this limit. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), Claude Code, with tool use and code execution. Subagents on the same model read the code paths. - The two-mode load model test and the merge with master: Claude Opus 5.5 (`claude-opus-5-5`), Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [ ] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
0ead74b4f3 |
feat(slack): show useful progress without extra messages (#20827)
## Thinking Path > - Paperclip helps people manage AI agents and their work. > - Chat connectors show agent work in the source conversation. > - Slack progress uses a small set of general status messages. > - Native runtimes already mark some completed assistant notes as public progress. > - The server can select those notes and control their size and rate. > - This pull request keeps one progress message and sends the final answer as a new reply. > - Users can follow useful changes without reading each tool event. ## Linked Issues or Issue Description **What existing behavior does this improve?** Automatic progress updates in Slack task threads. **Current behavior** Users see general messages such as “Maya is making progress”. The first answer can replace that message. A source update that arrives during the progress delay can be lost if no later event arrives. **Proposed behavior** Show the latest short public note in one message. Wait 10 seconds before the first update and at least 30 seconds between updates. Recover pending work after a restart. Send the final answer as a fresh reply. **Reason and benefit** Users can see what the agent checks and what it found. The thread stays short. The model needs no new tool call or summary model. Related change: [resolved-card chat bindings](https://github.com/paperclipai/paperclip/pull/20825). This branch includes that master revision. The progress change uses the existing source authority checks. ## What Changed - Select completed `agentMessage` items from the normalized PRP `progress` channel. Keep unknown channels, reasoning, tool output, and raw provider fields out of progress text. - Sanitize each note. Keep at most two sentences and 400 characters. Add a task link when a public board URL is available. - Prefer public notes over general tool status. Remove repeated text and combine bursts into the latest pending note. - Use run events and the publication outbox as saved state. Use the shared deadline worker for the grace period and update delay. A delivery receipt wakes the worker when an earlier update is still in flight. Settled history does not poll. - Apply the run's registered-secret filter before text selection or storage. Reuse the latest usable note within a sweep. Invalidate the cache when the source event or secret registry changes. - Reuse the exact source-bound Slack fast-response receipt when it replaced the working placeholder. Check its current outbound message link before editing it. - Send Slack answers as new replies. After delivery, queue a separate cleanup edit for a generic working message. Keep its current outbound link as the edit authority. Leave useful public notes in place. Cancel pending public progress when an answer, question, or terminal run takes over. Keep the existing delivery policy for other connectors. - Add one short Slack prompt instruction. Update product documentation and regression tests. The Slack module owns public-note selection, cadence, lane eligibility, and cleanup. The shared progress service holds its existing issue and run locks and calls a narrow provider policy. The secret registry supplies redaction and a version token for the per-sweep note cache. The existing outbox owns delivery, retries, and authority checks. The Slack module owns receipt reuse. Shared delivery changes supply the receipt wake and select the Slack reply policy. No schema or configuration change is required. ## Verification - The initial content-selection regression failed before the implementation and passed after it. - The follow-up reproduced two review defects before the fixes. A registered synthetic secret appeared in saved progress. A generic working message stayed active after the answer. Both regressions now pass. - The final content, database-worker, query, protocol-boundary, and redaction selection passes 78 tests across four files. - The earlier affected transport selection passes all 65 cases. It covers public progress, native cadence, separate answers, Slack and Telegram retry isolation, FIFO restart, and cleanup ownership. Its first run passed 64 cases and found one old assertion that rejected the intended cleanup edit. The corrected assertion checks the exact successor message ID. - Full server typecheck passes: `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm --filter @paperclipai/server typecheck`. An earlier attempt found two transaction type errors. The helper contracts now accept their actual database operations. The final run passes all source and test layers. - The broad Slack selection stopped after 189 passing cases when the sandbox killed its worker with SIGKILL. That attempt is not a pass. The focused transport run above supplies affected local coverage. Required CI must supply broad coverage on the final head. - `pnpm verify --explain` returns exit 2 because the product documentation change requires broad CI coverage. No full local suite was started. - Preflight installed dependencies and built the SDK. Cargo is absent, so local Rust checks could not run. No Rust source changed. - [CI on `02552bb584`](https://github.com/paperclipai/paperclip/actions/runs/43493597242) passed build, typecheck, policy, all browser shards, Runner/Rust checks, and the other test lanes. One chat shard failed in three Slack cases. Each expected one edit after an answer was already sent. The intended cleanup adds a “Response sent.” edit before the later failure notice. All three failures reproduced locally. The test assertions now check both exact edit texts and the original run's message ID. They also check that another sweep adds no edits. - The latest local run passes all 85 selected transport cases. It includes all 18 Slack/Telegram answer and failure orders, plus progress, deferred FIFO, retry isolation, and cleanup coverage. Command: `pnpm exec vitest run server/src/__tests__/chat-channels.integration.test.ts -t 'preserves selected answer and failure lanes|public progress|native progress|keeps intermediate.*agent comments|keeps a.*retry.*queue|FIFO'`. - [Final CI](https://github.com/paperclipai/paperclip/actions/runs/43499797545) passed on `55ec4c053fe5263d9033d807777dbf2e679407e3`. All 52 executed jobs passed. This includes build, typecheck, general tests, serialized tests, all eight browser shards, Runner/Rust checks, and the required aggregates. The affected chat shard now passes. The test-file selection job was skipped because this run used full coverage. - [Greptile](https://github.com/paperclipai/paperclip/runs/132142301124) completed on the same head with 5/5 and no new findings. All four inline review threads are resolved. The review job and Socket checks passed. Of 61 check runs, 57 succeeded, three were skipped, and one was neutral. No checks remain pending or failed. The PR is mergeable with a clean state. - [Superagent Security Scan](https://github.com/paperclipai/paperclip/runs/132142345827) returned a neutral result with “PR scan workflow failed”. It supplied no security verdict. Do not count it as a successful scan. - Live Slack notification and unread behavior has not been tested. Service tests use real PostgreSQL and a fake provider runtime. ## Risks - Only runtimes that map commentary to the explicit public `progress` channel supply contextual prose. Other runtimes retain general status messages. - Slack final answers now create a fresh reply. This is an intended delivery change. - Public commentary must suit the current conversation audience. Text sanitization adds protection; it does not authorize sharing unrelated private material. - Source events and outbox rows supply recovery state. No new timer table or recurring idle scan is added. - The existing large integration suite retains its provider fixtures and lifecycle assertions. No source file crosses the 1,000-line threshold in this change. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code tools. The exact deployment ID was not supplied. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
5df1ada9c1 |
feat(plugins): send configured environment on plugin task resume (#21352)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - #21335 lets a plugin task provider that restarts the Runner on `resume` opt in (`resumeCredentialVersion: 1`) to receive the claimant run's harness credential > - A provider restarting the Runner also needs the agent's configured environment, which Core sends on `resume` only during a session handoff > - Without it, a Runner a provider moves to a new computer starts without the agent's configured environment variables > - This pull request widens that opt-in to both launch inputs, under a name that says so, before any provider adopts the old name > - The benefit is that a provider can restart a parked Runner on resume with the same credential and environment a `submit` would carry ## Linked Issues or Issue Description Refs #21289 Follows: #21335 **What existing behavior does this improve?** The opt-in resume launch inputs added in #21335. **Subsystem affected** Plugin SDK environment task schema (`describe` capability), the plugin Runner task owner's `resume`, and the plugin environment task docs. **Current behavior** With `resumeCredentialVersion: 1`, `resume` carries `credential`. `configuredEnvironment` is sent on `resume` only for a session handoff. **Proposed behavior** The capability is renamed `resumeLaunchInputsVersion: 1`. With it, `resume` carries `credential` as before and also `configuredEnvironment` (same builder as `submit`) when the provider advertises `configuredEnvironmentVersion: 1`. Session handoff is unchanged. **Reason and benefit** A provider that restarts the Runner on resume gets both launch inputs it needs. **Breaking changes** The capability name changes from `resumeCredentialVersion` to `resumeLaunchInputsVersion`. No provider advertises the old name yet (it merged in #21335 and has no adopter), so nothing changes behavior. ## What Changed - Rename `resumeCredentialVersion` → `resumeLaunchInputsVersion` in the SDK `describe` result schema, executor plumbing, owner input, docs and tests. - With the capability, `resume` also adds `configuredEnvironment` when the provider advertises `configuredEnvironmentVersion: 1`. A normal resume does not mark the variables as delivered, since a live Runner keeps the environment it started with. - Docs: the "Release and resume" section describes both fields; the "Configured environment" section points to it. - The runner-e2e SDK public-input pin is updated. ## Verification - SDK vitest `tests/environment-task-release-resume.test.ts` and `tests/environment-tasks.test.ts`: 68 passed. - Server vitest `plugin-runner-task.test.ts` and `environment-task-runtime.test.ts`: 153 passed, including a normal resume with the capability (credential sent, variables added only with `configuredEnvironmentVersion: 1`, delivered flag unset) and redaction of both fields from metadata writes and sanitized errors. - `native-session-executor.test.ts -t "uses a task-only"`: 5 passed. - runner-e2e `source-sdk-evidence.test.ts` and `source-sdk-public-type.test.ts`: 46 passed. - `pnpm typecheck` in `server/` (builds shared and plugin-sdk) passed. - Not run locally: the full suite and `pnpm verify`; required CI covers them. ## Risks - A provider restarting the Runner gets the claimant run's current variables, which can differ from the ones the parked Runner started with. That is intended. - Rollback: revert this pull request. No migration. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, test runs), implemented by a sub-agent and reviewed before opening. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
12aa08721c |
feat(ssh): use the agent AI account in new environments (#21248)
Use the agent-selected AI account for new SSH environments. Preserve
remote-machine authentication for existing environments and expose it as
an advanced option. Keep managed credentials private while preserving
supported conversation state and credential refreshes across turns.
Validated with focused route, adapter, lifecycle, and real-process tests;
final CI passed on
|
||
|
|
8f105e743a |
fix(accounting): recover closed legacy runs without inventing usage (#21493)
Preserve exact physical process closure independently from incomplete cost reporting. Recover a missing receipt through the existing authorized accounting Retry without replaying provider work. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
|
0d832a02d3 |
perf(ci): shorten the general server and chat test lanes (#21393)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Every pull request runs the `PR` CI workflow before it can merge > - The `General tests (server (N/12))` and `General tests (chat (N/3))` jobs finish last, so they set the wall-clock time of CI > - Before this change, server shards took 9.5 to 11.7 minutes and chat shards took 6.4 to 9.6 minutes; the target is 7 minutes or less > - Measurement shows that the chat lane was unbalanced and that the server lane is limited by CPU on the 4-vCPU runners > - This pull request balances the chat lane by recorded test time, runs three chat partitions per job, and takes general server weights from PR runs > - The benefit is a chat lane under 7 minutes and measured evidence for the shard count that the server lane needs ## Linked Issues or Issue Description **What existing behavior does this improve?** The PR CI general server and chat test jobs. They are the slowest jobs of every pull request. **Subsystem affected** CI test scheduling: `scripts/run-vitest-stable.mjs`, `scripts/test-line-shard.mjs` and `scripts/ci/refresh-test-durations.mjs`. **Current behavior** Baseline: green `PR` runs [43437558012](https://github.com/paperclipai/paperclip/actions/runs/43437558012) and [43441184316](https://github.com/paperclipai/paperclip/actions/runs/43441184316), all on m7i-flex.xlarge (4 vCPU, 16 GB). - Server shards: 9.5 to 11.7 job minutes, 502 to 654 s test step, effective file concurrency 2.5 to 2.9 of 3 workers. Vitest reports `tests 61-68%, import 25-31%, transform 6-7%, setup 1%`. Each shard sums 1,410 to 1,765 s of file time. The longest file is 303 s (`deferred-execution-receipt-work.test.ts`), so the floor is total work divided by workers, not one file. - Chat shards: 9.3 to 9.6, 7.6 to 7.8 and 6.4 to 6.5 job minutes. Each job ran two partitions of `chat-channels.integration.test.ts`. The partitions were balanced by collected-case count, but one `it.each` line can run many cases. One of six partitions recorded 532 s of test time against a 377 s mean. **Proposed behavior** - Chat partitions keep each source line whole and balance by recorded time per collected entry. - Each chat job runs three partitions in place of two. - General server weights come from the server jobs of merged PR runs, which run files with the same concurrency as the lane they place. **Reason and benefit** The slowest job sets the CI wall-clock time for every pull request. The chat lane goes from a 9.3-9.6 minute maximum to 6.5-6.8 minutes. ## What Changed - `scripts/test-line-shard.mjs`: `partitionTestLines` takes optional recorded durations. With durations, it balances source lines by recorded time; entries without a record take the median. Without durations, the behavior is unchanged. - `scripts/chat-shard-durations.json` (new): recorded time per collected chat entry, from the CI profile records of PR run [43444854171](https://github.com/paperclipai/paperclip/actions/runs/43444854171). 926 of 928 collected entries are measured; the other two are the bogus collection entries that the launcher already drops. - `scripts/ci/refresh-chat-shard-durations.mjs` (new): builds that manifest from downloaded `ci-profile-*` artifacts. Each executed case maps to the collected `it`/`it.each` entry that registers it. A case without an entry fails the refresh. - `scripts/run-vitest-stable.mjs`: a sharded chat job runs three partitions in place of two, and the chat launcher passes the recorded durations. A chat job collects the suite once with `vitest list` in place of once per partition, and the per-partition filter checks run at the same time. Before, each partition ran two blocking collections of the 86,000-line file one after another. The server worker count stays at three. - `scripts/ci/refresh-test-durations.mjs` and `scripts/ci/pr-run-durations.mjs`: the weekly refresh reads general server weights from the two newest merged-PR runs. Each run must have the full server shard count from `.github/ci-task-matrix.json` and cover at least 90% of the lane. Serialized weights still come from master Cloud readiness runs. - `scripts/general-server-shard-durations.json`: refreshed with the new source (PR runs 43439145752 and 43439279537). - Tests: `scripts/__tests__/run-vitest-stable-shard.test.mjs`, `scripts/ci/test-selection-pr-timings.test.mjs` and `server/src/__tests__/vitest-chat-shards.test.ts` cover the weighted partition, the entry mapping, the manifest format, the PR-run collection rules and the three-partition chat launcher. - `doc/DEVELOPING.md`: describes the three chat partitions, the chat weights and how to refresh them, and the new source of the server weights. ## Verification Experiments ran in this PR's own CI with a temporary resource sampler (removed before review). | Setting (run) | Server step vs. same shard's baseline | CPU busy | Max memory used | Result | |---|---|---|---|---| | 3 workers (43444854171, shards 1-4) | +8% (profiling overhead) | 62-73% | 6.8 GB | 1 flaky retry | | 4 workers (43444854171, shards 5-8) | -4% | 82-86% | 8.2 GB | clean | | 5 workers (43444854171, shards 9-12) | -10% | 70-93% | 9.5 GB | 1 hard failure: `decision-models.test.ts` timed out at 15 s twice | | 4 workers (43450469416) | mixed m5a/m5n/m6i runners | 78-89% | 7.7 GB | 1 hard failure: `workspace-recovery-warm-handoff.test.ts` timed out at 30 s on m5a | The server lane is CPU bound: at four workers the CPU is 82-89% busy and wall time improves by only about 10%. Five workers fail tests. Memory is not the limit (no swap, no memory pressure, at least 6 GB free). Node's compile cache and Postgres `fsync=off` showed no attributable gain (import share stayed 29-34%; `fsync=off` cut iowait from 2-4% to 0.2-0.7% of CPU time), so neither is in this PR. Final configuration (server: 3 workers with refreshed weights; chat: 3 weighted partitions per job), job minutes / test-step seconds / effective concurrency: | Shard | Baseline 43437558012 | Baseline 43441184316 | A: 43465156503 attempt 1 | B: attempt 2 | C: attempt 3 | |---|---|---|---|---|---| | server 1/12 | 11.0 / 609 / 2.81 | 10.2 / 562 / 2.82 | 10.1 / 535 / 2.88 | 11.6 / 615 / 2.86 | 11.3 / 600 / 2.86 | | server 2/12 | 10.2 / 567 / 2.71 | 10.9 / 607 / 2.79 | 9.7 / 510 / 2.85 | 8.9 / 466 / 2.86 | 11.9 / 635 / 2.83 | | server 3/12 | 10.9 / 609 / 2.88 | 10.9 / 601 / 2.86 | 9.4 / 498 / 2.81 | 11.7 / 623 / 2.80 | 20.3 / 1109 / 0.00 | | server 4/12 | 11.0 / 613 / 2.77 | 11.2 / 622 / 2.84 | 11.3 / 602 / 2.84 | 9.3 / 487 / 2.83 | 12.5 / 676 / 2.84 | | server 5/12 | 10.1 / 555 / 2.83 | 9.9 / 544 / 2.84 | 10.2 / 542 / 2.85 | 11.6 / 618 / 2.82 | 11.9 / 640 / 2.84 | | server 6/12 | 9.9 / 551 / 2.85 | 10.4 / 579 / 2.85 | 9.8 / 516 / 2.69 | 9.1 / 480 / 2.68 | 9.2 / 480 / 2.67 | | server 7/12 | 10.8 / 596 / 2.84 | 10.1 / 558 / 2.86 | 10.8 / 575 / 2.80 | 11.8 / 631 / 2.80 | 20.8 / 1129 / 2.69 | | server 8/12 | 11.7 / 654 / 2.62 | 11.2 / 624 / 2.56 | 9.8 / 513 / 2.83 | 11.2 / 589 / 2.82 | 19.4 / 1059 / 2.40 | | server 9/12 | 9.7 / 533 / 2.85 | 9.7 / 533 / 2.84 | 11.5 / 615 / 2.68 | 9.8 / 515 / 2.66 | 13.0 / 703 / 2.68 | | server 10/12 | 9.8 / 540 / 2.79 | 9.5 / 502 / 2.81 | 10.6 / 561 / 2.84 | 11.8 / 625 / 2.83 | 9.3 / 488 / 2.83 | | server 11/12 | 10.9 / 605 / 2.83 | 11.3 / 627 / 2.75 | 10.3 / 545 / 2.75 | 9.5 / 498 / 2.72 | 20.3 / 1107 / 0.00 | | server 12/12 | 10.6 / 592 / 2.50 | 11.7 / 645 / 2.53 | 9.6 / 503 / 2.81 | 8.9 / 468 / 2.82 | 19.7 / 1065 / 2.70 | | chat 1/3 | 9.6 / 528 / 1.56 | 9.3 / 511 / 1.58 | 6.6 / 310 / 2.69 | 5.7 / 272 / 2.65 | 6.8 / 331 / 2.69 | | chat 2/3 | 7.8 / 418 / 1.74 | 7.6 / 408 / 1.75 | 6.3 / 305 / 2.57 | 6.5 / 311 / 2.58 | 5.5 / 260 / 2.58 | | chat 3/3 | 6.5 / 337 / 1.74 | 6.4 / 333 / 1.73 | 6.0 / 285 / 2.65 | 5.4 / 258 / 2.68 | 5.8 / 262 / 2.69 | Summary: chat maximum 9.3-9.6 min before, 6.6 / 6.5 / 6.8 min in A / B / C. Server maximum 11.7 min before, 11.5 / 11.8 min in A / B; the server worker setting is unchanged, so its run-to-run spread is mostly runner type (m8i.xlarge shards took 8.9-9.8 min, m6a.xlarge 11.9-13.0 min). Runners: A all m7i.xlarge; B m8i/m6i; C m6a/m6in/m8i/m5a. In C, the five server shards on m5a.xlarge summed 2,545-3,037 s of file time against about 1,700 s elsewhere; three reached the 20-minute job limit and two failed tests on 15 s timeouts. The server lane settings in this PR are the same as master's, so this is a fleet finding: m5a.xlarge cannot run a server shard inside the job limit (follow-up for the fleet configuration, outside this PR). Flakiness of the final setting: the nine chat jobs in A, B and C had no failure and no retried test. Server shards in A and B had no failure and no retried test. Before the last commit (single chat collection), three runs of 43456289623 had chat at 6.7-7.5 min with no chat retries; server retries there (2 in one run, 1 in another) were on the unchanged three-worker server setting. Local checks (correctness only): - `node --test scripts/__tests__/run-vitest-stable-shard.test.mjs scripts/ci/test-selection-pr-timings.test.mjs scripts/ci/test-selection-timings.test.mjs`: 68 passed. - `pnpm exec vitest run --project @paperclipai/server server/src/__tests__/vitest-chat-shards.test.ts`: passed. - `pnpm verify` exits 2 (global CI input changed), so the required CI checks supply the broad coverage. ## Risks - Three chat partitions per job use more memory: the peak was 11.4 GB of 16 GB, with at least 4.2 GB free and no memory pressure. CPU was 55-63% busy. - Stale chat weights only change balance, never which tests run. Shard membership is still checked with `vitest list` before each run. - The weekly refresh now downloads general server logs from merged-PR runs. Runs that selected a subset of files are skipped before download because their shard count is smaller. - The server lane is still above 7 minutes. See the remaining floor below. **Remaining floor and shard-count recommendation (for a follow-up under `.github/`)** A server shard sums about 1,600 s of file time, and 3 workers on 4 vCPUs give at most about 2.8 effective concurrency, so the test step is about 560-640 s. Job overhead is about 50 s. A job of 7 minutes needs a test step of about 360 s, so about 1,000 s of file time per shard. The current 19,200 s of general server file time therefore needs about 19-20 shards of 4 vCPUs at 3 workers. More workers on the same runners do not help, because the CPU is already the limit. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell, GitHub CLI) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
0199c963e2 |
feat(apps): ship the redesigned sign-in step for a pasted MCP server (#21469)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Apps lets an operator connect a remote MCP server by pasting its
address ("Connect your own MCP server")
> - Step 2 of that flow put every sign-in choice in an "Advanced
authentication" box with wrapped pill buttons and a hint paragraph
> - The box hid the choice behind a vague label, and its shape did not
match the access line ("Connects for everyone… Change") right below it
> - The owner approved a redesign: a stated sign-in default with a
Change link, and five options with their fields directly below each
option
> - This pull request wires that redesign into the real screen, removes
the old step, and updates copy that named the removed box
> - The benefit is a screen that reads as two stated answers and one
action, with the same default path (paste the address, press Check link)
## Linked Issues or Issue Description
No public issue exists. Enhancement:
**What existing behavior does this improve?**
The sign-in section of the "Connect your own MCP server" step in Apps.
**Subsystem affected**
UI: Apps connection setup (`ui/src/features/connections`). Small copy
changes in the server OAuth client registration message and docs.
**Current behavior**
Step 2 shows an "Advanced authentication" collapsible with five wrapped
pill buttons and a hint paragraph below them. The fields for the chosen
mode appear further down. Guidance banners tell the operator to look
"under Advanced authentication". The page also shows the "Unverified
server" badge twice (header and body). The stepper reads "Pick app · Add
your key", even when the server needs no key.
**Proposed behavior**
Step 2 states the sign-in choice in one line ("Paperclip asks the server
how to sign in.") with a Change link. Change shows five radio cards: Let
Paperclip check, Key or token, Browser sign-in, Custom headers, No
sign-in needed. The fields for the selected option are directly below
it. Done collapses the list to a one-line summary (for example "Signs in
with 1 custom header."). The list stays open while the chosen option
still needs a value. Client ID and secret are behind "Use your own
client ID" unless they have values or the server asked for them. The
stepper reads "Pick app · Connect".
**Reason and benefit**
The choice is visible as a stated default in the same shape as the
access line. Each option keeps its fields next to it. No copy points at
a box that does not exist.
**Breaking changes**
None. The five auth modes and their request payloads do not change.
**Additional context**
Related open PRs change server behavior for the same flow. They are not
duplicates. Refs #15192, Refs #15450.
## What Changed
- `ui/src/features/connections/GenericMcpConnectStep.tsx`: new step
component (same props as the old `LinkConnectStep`). It uses stable
field ids `generic-mcp-key`, `generic-mcp-client-id`,
`generic-mcp-client-secret` and a `data-testid` on the sign-in
Change/Done link.
- `ui/src/features/connections/ConnectionSetupFlow.tsx`: render
`GenericMcpConnectStep`; delete `LinkConnectStep`,
`GENERIC_AUTH_MODE_OPTIONS`, `SegmentedOption` and unused imports;
change the pasted-link stepper label from "Add your key" to "Connect"
(`STEP_LABELS` is used only by the pasted-link flow; curated apps keep
their own `["Add your key"]`).
- `ui/src/pages/apps/generic-mcp-connect.ts`: `oauth_challenge` and
`oauth_manual_client_required` guidance now say "below" instead of
"under Advanced authentication".
- `server/src/services/tool-access/oauth/client-registration.ts`: the
manual-client message no longer names the removed box.
- `doc/connections/GENERIC-REMOTE-MCP.md`: describe the sign-in line,
Change, and the five options.
- Comments in `packages/shared` and `server/src/services/tool-access.ts`
reworded. `tests/runner-e2e/source-sdk-public-inputs.json` regenerated
for the two shared files (blob hashes only).
- `ui/storybook/stories/connection-setup/byo-mcp-sign-in.stories.tsx`:
14 stories for the shipped component; annotation is now `current` /
`production`.
- Tests: `AppsConnect.test.tsx` generic MCP block uses the sign-in
toggle and radio options; new tests for the collapsed summary, the
open-while-empty-key rule, and a single "Unverified server" label. New
`GenericMcpConnectStep.test.ts` unit-tests `genericMcpAuthSummary`.
Before: "Advanced authentication" box with pill buttons and a hint;
fields below the pills; two "Unverified server" badges; stepper "Pick
app · Add your key".
After: one sign-in line with Change; five radio cards with fields under
the selected card; one "Unverified server" badge in the header; stepper
"Pick app · Connect". Storybook: `byo-mcp-sign-in--choosing`,
`byo-mcp-sign-in--server-asked-for-key`.
## Verification
- `cd ui && pnpm exec vitest run src/pages/apps/AppsConnect.test.tsx
src/pages/apps/generic-mcp-connect.test.ts
src/features/connections/GenericMcpConnectStep.test.ts
src/features/connections/ConnectionSetupFlow.architecture.test.ts
src/features/connections/ConnectionSetupFlow.oauth.test.ts`: 5 files,
300 tests pass.
- `cd ui && node ../scripts/run-tsc.mjs -b`: pass.
- `pnpm -s typecheck` in `packages/shared` and `server`: pass.
- `pnpm check:token-gates`: all gates clean.
- `node scripts/storybook-catalog.mjs`: pass (generated indexes not
committed).
- `pnpm verify`: exit 2 (broader verification required, because
`doc/connections/GENERIC-REMOTE-MCP.md` is a global CI input). Required
CI supplies that coverage. The focused files above were rerun on the
final revision after the rebase.
- Headless Playwright against the Storybook dev server: the Default,
Choosing and Server-asked-for-key stories render the new step, the new
guidance copy, and the "Pick app · Connect" stepper.
## Risks
- Low risk. UI change on one screen; request payloads
(`genericConnectPayload`) do not change.
- Tests that click the first collapsed "Change" button now skip the
sign-in toggle, because the access line also has a "Change" button.
- The server message for `oauth_manual_client_required` changes text. No
code matches on the text; the client uses the error code.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, headless Playwright).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
|
||
|
|
7773b52f05 |
fix(codex): preserve sandbox certificate trust in Codex shell commands (#20364)
<!-- Written in ASD-STE100 style: short sentences, active voice. --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Paperclip Runner and the Codex adapters run agent shell commands inside a sandboxed Codex session with a strict environment allowlist > - The controller builds `shell_environment_policy.include_only` before the sandbox injects certificate-trust variables such as `SSL_CERT_FILE` and `GIT_SSL_CAINFO` > - The allowlist drops the injected variables, so managed `git` and `gh` commands cannot verify HTTPS against a runtime-provided CA > - This pull request allows the certificate-trust variable names through every environment policy stage, from the Rust process supervisor to the shell allowlist > - The benefit is that managed `git` and `gh` work again in sandboxes that require a runtime CA, while provider-credential isolation and the bounded allowlist stay in place ## Linked Issues or Issue Description - Fixes: #20019 - Refs #20038 — community pull request for the same defect. This change reviews the approach independently. The commit credits the community author as co-author. ## What Changed - Add `CERTIFICATE_ENVIRONMENT_KEYS` (`SSL_CERT_FILE`, `SSL_CERT_DIR`, `GIT_SSL_CAINFO`, `NODE_EXTRA_CA_CERTS`, `DENO_CERT`, `REQUESTS_CA_BUNDLE`, `CURL_CA_BUNDLE`) to `packages/paperclip-runner/src/configured-environment.ts` - Native Codex shell policy (`codex-security-config.ts`): always put the certificate names in `include_only`; set inheritance to `"all"` with `ignore_default_excludes=true`, bounded by the explicit allowlist - Pass the certificate names through the app-server environment filter, the ACPX spawn environment filter, the ACPX runtime sandbox `include_only`, the runner platform environment keys, and the Rust process supervisor base key list - Legacy and ACPX-engine Codex shell allowlists (`packages/adapter-utils/src/codex-shell-environment.ts`): always include the certificate names - Add a synthetic private-CA HTTPS journey test: a real `git ls-remote` fails TLS verification without the injected CA and succeeds with it, through the policy-filtered child environment - Document the contract in `doc/sandbox-shell-environment.md` and link it from `doc/DEVELOPING.md` ## Verification - `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/drivers/codex/codex-security-config.test.ts src/configured-environment.test.ts src/drivers/codex/codex-shell-certificate-trust.test.ts src/drivers/codex/codex-app-server-driver.context-control.test.ts src/drivers/acpx/runtime-sandbox.test.ts` — 104 tests pass - `pnpm --filter @paperclipai/adapter-utils exec vitest run src/codex-shell-environment.test.ts src/acpx-engine/execute.test.ts` — 270 tests pass - `pnpm --filter @paperclipai/adapter-codex-local exec vitest run src/server/execute.test.ts` — 10 tests pass - Codex driver lifecycle, conformance, recovery, and boundaries suites pass (103 tests); environment-builder suites pass (307 tests) - Red proof: with only the source fix reverted, the new regressions fail. The journey test shows `GIT_SSL_CAINFO` is undefined in the shell environment, and `git` fails TLS verification against the private-CA fixture - `cargo check` passes for `paperclip-runner-core`; `rustfmt --check` passes for the edited Rust file - The journey test needs `git` and `openssl`; it skips when either tool is absent ## Risks - The native shell policy now always sets `shell_environment_policy.inherit="all"` with `ignore_default_excludes=true`. Before, it set `"none"` when no credentials or task keys were projected. The explicit `include_only` allowlist still bounds every inherited name, so no new secret names can cross - The allowlist grows by seven non-secret certificate-trust names. Their values point at CA files. The sandbox filesystem policy still controls access to those files; this change grants no filesystem access - No persisted run, session, or launch identity changes shape. Sessions saved by the previous release resume unchanged; the new policy applies when the next launch builds its arguments - Provider credentials keep their existing exclusions at every stage ## Model Used - Claude Fable 5 (Anthropic, model ID `claude-fable-5`), extended thinking, agentic tool use, run through Claude Code. The implementation was written and reviewed independently against the community design in #20038; the community author is credited as co-author in the commit. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green (pending: checks run after this PR opens) - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups (pending review) - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: everlastingcrown <everlastingcrown@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
d018a9d1cd |
fix(queue): retain chat bindings on resolved card continuations (#20825)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - A task can receive a card response while its agent runs.
> - The queue saves this response for the next turn.
> - Chat authorization requires the saved inbound message binding.
> - Queue promotion removed this binding when a prior turn received the
message.
> - This pull request retains live bindings for resolved card responses.
> - The next turn can check current chat access and continue the
approved work.
## Linked Issues or Issue Description
**What happened?**
An accepted plan card queued behind a chat run could start with a chat
source and no message IDs. Reviewed chat authorization then failed with
context_mismatch before provider work started.
**Expected behavior**
A resolved card must retain its live chat binding through queue
promotion. The new run must check current access. A consumed
message-only wake must not start another turn.
**Steps to reproduce**
1. Start a chat run with an inbound message.
2. Queue a card response with that message binding.
3. Complete the first run and release its queue.
4. Inspect the promoted context and check chat authorization.
Related work: #20689 cancels consumed message-only wakes. #19156
protects a successor checkout. This change retains the binding for an
independent card response.
## What Changed
- Retain delivered message bindings for resolved chat card responses
during queue promotion.
- Keep deleted, foreign-task, and self-authored message filters and
current chat authorization.
- Add a persisted queue regression and Slack/Discord
promotion-to-authorization checks, including access revocation.
- Document the queue behavior.
## Verification
- Before the fix, the persisted queue test failed because the promoted
context had no wakeCommentIds. Three existing tests passed.
- The first focused run passed 108 tests. The added Slack and Discord
checks passed after promotion and refused revoked access.
- Final head
|
||
|
|
d738ed5ee4 |
fix(ui): show each related task once in the Tasks tab (#20315)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The Tasks tab shows related tasks in separate sections. > - A task can be both a subtask and a task created from the current task. > - The tab showed that task in Subtasks and again in its project group. > - This pull request removes tasks from project groups when another section already shows them. > - Users can scan each task once and keep the subtask progress view. ## Linked Issues or Issue Description **What happened?** The Tasks tab shows the same task in Next action or Other subtasks and again in a project group. **Expected behavior** Show each task once. Hide a project group when all its tasks appear in other sections. **Steps to reproduce** 1. Open a task that created subtasks in the same project. 2. Open the Tasks tab. 3. Compare Next action and Other subtasks with the project section. **Paperclip version or commit** Reproduced in component tests on source commit `189fefb99183`. **Deployment mode** Built from source. The fault is in the shared UI component. No related open pull request was found in searches for duplicate tasks and TaskDetailTasksPanel. ## What Changed - Exclude task IDs already shown as subtasks, ancestors, the creation source, or a root blocker from project groups. - Show a creation source that is also an ancestor only in Created from. - Hide empty project groups, including No project. - Use an index for project lookup. - Update the product contract and regression tests. ## Verification - Before the fix, four regression cases failed because task rows or links appeared twice. - After the fix, both relation-panel test files pass: 19 tests. Run from `ui`: `node ../node_modules/vitest/vitest.mjs run --config vitest.config.ts src/components/task-detail/TaskDetailTasksPanel.test.tsx src/components/task-detail/TaskDetailRelationsPanel.test.tsx`. - Tests cover active and completed subtasks, progress, independent collapse, duplicate input rows, empty project groups, ancestors, the creation source, and a root blocker. - A test with 800 tasks and 120 projects checks unique rows and bounds task and project lookups. - `pnpm check:token-gates` and `git diff --check` pass. - The first CI typecheck found an incomplete blocker-attention test fixture. The fixture now supplies all required fields. The 19 focused tests pass again. - `pnpm --filter @paperclipai/ui typecheck` passes after `pnpm --filter @paperclipai/plugin-sdk ensure-build-deps`. The first local attempt lacked the built SDK; the retry built that dependency. - `pnpm verify` returns exit 2 because the product document requires broad CI coverage. CI typecheck, build, general tests, browser shards, and serialized server suites pass on the current head. - Full local preflight could not finish: the OpenCode CLI install hook failed, and Cargo is absent. The UI tests ran after installing dependencies with install hooks disabled. - No full-stack browser acceptance was run. This change has component DOM evidence. - Greptile gives the current head `f3bfd8fb47154810b85578ee6731246238caf67a` 5/5 with no review threads. - All 58 current-head checks pass. [CI attempt 2](https://github.com/paperclipai/paperclip/actions/runs/42657701260/attempts/2) succeeded on `f3bfd8fb47154810b85578ee6731246238caf67a`. - The first Rust CI job failed in `reaping_stops_a_dead_runners_provider_and_its_escaped_tool_job`. Its report listed one unattributed orphan after cleanup. The same Rust code passed in the previous run. [One job retry](https://github.com/paperclipai/paperclip/actions/runs/42657701260/job/129311740923) passed without code changes. The original failure is retained here; its exact cause was not established. This PR does not change Rust code. ## Risks - Low risk. This changes UI grouping only. Stored task relationships and API data stay the same. - A project section can disappear when every task in it already appears above. This is the expected result. ## Model Used - OpenAI GPT-6, through Codex, with reasoning, repository tools, and code execution. The runtime does not expose a more specific model build or context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
c6301c92ff |
fix(ci): keep the typecheck job working on stacked PRs and stop its CLI guidance test timing out (#21451)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Every pull request runs the required `Typecheck + Release Registry` job from the trusted workflow on `master` > - #21404 replaced that job's six sequential steps with one script, `scripts/ci-typecheck-lane.mjs`, that runs independent checks in parallel > - PR runs take the workflow from `master` but check out the PR's merge commit. A stacked PR's merge commit is built on its base branch, so it can lack a script that only `master` has > - The lane also ran the CLI guidance test beside the typechecks at a lower CPU priority, and one of its cases is slow enough to cross its 15 second timeout there > - This pull request makes the job fall back to the earlier commands when the script is absent, and runs the CLI guidance test before the parallel checks > - The benefit is that stacked PRs typecheck again and the job stops failing on a timeout that has nothing to do with the PR ## Linked Issues or Issue Description Refs #21404 **What happened?** In the first eight PR runs after #21404 merged (2026-11-13 18:06Z), the typecheck job failed three times for reasons unrelated to the PRs: - Twice on #20617, a PR stacked on `feat/ssh-agent-ai-accounts`: `Error: Cannot find module '/home/runner/_work/paperclip/paperclip/scripts/ci-typecheck-lane.mjs'` (jobs 132076887259 and 132079570039). The job failed 0 seconds into the step. - Once on run 43480469363 (job 132075678943): `CLI guidance safety failed with exit code 1`, from `Error: Test timed out in 15000ms.` in `allows only exact-allowlist pnpm paperclipai commands on every guidance surface`, which took 17059 ms. The same case passes in about 5 seconds when the test runs alone. **Expected behavior** The typecheck job passes or fails on the PR's own code, including for stacked PRs. **Actual behavior** Stacked PRs always fail the job. Other PRs fail it intermittently (1 of 6 runs that reached the lane). **Steps to reproduce** Open a PR whose base branch does not contain `scripts/ci-typecheck-lane.mjs`, or run the lane on a loaded 4 vCPU runner. ## What Changed - `.github/workflows/pr-trusted.yml`: the lane step runs only when `scripts/ci-typecheck-lane.mjs` exists in the checkout. A new step runs the six earlier commands, in their earlier order, when it does not. Only the `typecheck_release_registry` job changed. - `scripts/ci-typecheck-lane.mjs`: the CLI guidance test moved from the parallel checks to the end of the sequential preparation steps, at normal priority. - Tests: `scripts/__tests__/ci-typecheck-lane.test.mjs` and `.github/scripts/tests/ci-plan.test.mjs` assert the two conditions, that the earlier commands appear only in the fallback step, and that the CLI guidance test is a preparation step. - `doc/DEVELOPING.md`: the "Typecheck lane in CI" section describes both. ## Verification - `node --test scripts/__tests__/ci-typecheck-lane.test.mjs`: 8 pass. - `node --test '.github/scripts/tests/*.test.mjs'` (the `policy` job's command): 807 pass. - `node --test scripts/__tests__/runner-release-crate-cache.test.mjs`: 11 pass. - Not exercised by this PR's own CI: PR runs use the workflow from `master`, so the new conditions first run after merge. The script change does run here, because the script is read from the PR's checkout. - Not measured: the lane's wall time with the CLI guidance test moved. The test takes 2 to 5 seconds alone, so the expected cost is that much. ## Risks - The fallback step repeats six commands in the workflow. It is needed only until open stacked PRs are based on a branch that has the script, and can be removed later. - `hashFiles` returns an empty string when no file matches, which is what the two conditions rely on. - If a future PR deletes or renames the script, the job silently uses the slower commands instead of failing. The wiring tests would fail in that PR. ## Model Used Claude Fable 5.2 (`claude-fable-5-2`), via Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
37393f5dc5 |
fix(browser-tracing): send API spans only for measured page loads, without timing events (#21441)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Hosted instances can send browser page-load traces to Honeycomb (#21156, #21392) > - On staging, one hour in one workspace sent about 64,000 Honeycomb events, and only about 15 were `page.load` > - Two sources make the volume: nine network timing span events on each API request span, and request spans for background polling in idle tabs > - The page-load metrics use neither of them > - This pull request drops the timing events and records API request spans only while a page load is measured > - The benefit is about 30x fewer events in a browser test, with the same page-load data and the same server trace joins ## Linked Issues or Issue Description Refs #21156, Refs #21392 **What existing behavior does this improve?** Browser tracing exports a span for every same-origin `/api/` fetch and nine network timing events on each of those spans. **Subsystem affected** UI browser tracing (`ui/src/lib/browser-tracing/`), on hosted instances with browser tracing turned on. **Current behavior** Honeycomb stores each span event as its own event. Each request span becomes about 10 events. Idle tabs poll (for example live runs every 3 s on a task page), so most spans are requests that no page load made. On staging: about 6,600 `GET` spans plus about 6,400 of each timing event per hour, against about 15 `page.load` spans. **Proposed behavior** Request spans carry no timing events. A request is recorded only when it starts while a `page.load` measurement is open. Other requests send no span and no `traceparent`. **Reason and benefit** Event volume drops by about 30x for an idle tab. `page.load`, Web Vitals, and the Resource Timing backfill do not change. No exported request span lacks its `page.load` parent, and no server span points at a browser span that was never sent. ## What Changed - `sdk.ts`: the fetch instrumentation runs with `ignoreNetworkEvents: true` (an option of the installed `@opentelemetry/instrumentation-fetch` 0.221.0). - `sdk.ts`: new `measuredRequestsOnly` wraps the tracer provider that only the fetch instrumentation uses. When no `page.load` is open, it returns a non-recording span with an invalid context. Nothing is exported. The W3C propagator sends no `traceparent` for an invalid context, so the server traces that request alone, as for an unsampled tab. The `page.load`, Web Vitals, and backfill spans use the normal tracer and do not change. - Failed idle requests are not kept. The server's own request span records server errors. To keep them, the browser must record every idle request and send its `traceparent` before it knows the status, and that brings back the dangling joins. - `span-sanitizer.ts`: comment update only. - `doc/browser-performance-tracing.md`: new "Live API spans" section with the rule, the note that timing events are not sent, and updated volume and trace-header text. ## Verification - Red then green (`ui/src/lib/browser-tracing/sdk.test.ts`): - `records no span and sends no traceparent for an API request made while no page load is measured`: failed on the old code (an idle request sent `traceparent`), passes now. - `sends request spans without per-request network timing events`: failed on the old code (9 timing events on the request span), passes now. - All 56 tests in `ui/src/lib/browser-tracing/` pass. `tsc --noEmit` for `ui` passes. - `pnpm verify` exits 2 because a doc is a global CI input. CI supplies the broad coverage. - Real browser: local dev server on an isolated empty instance with 8 tasks, a stand-in OTLP collector, headless Chromium, and the tracing config injected into the session response. Scenario: hard load of the all-tasks page, click a task, then 2 minutes idle. One Honeycomb event = one span or one span event. | Name | master | this branch | | --- | ---: | ---: | | `page.load` | 2 | 2 | | `web_vital.lcp` | 1 | 1 | | `HTTP` (backfill) | 33 | 32 | | `GET` | 165 | 18 | | `POST` | 1 | 1 | | each of `fetchStart`, `domainLookupStart`, `domainLookupEnd`, `connectStart`, `connectEnd`, `requestStart`, `responseStart`, `responseEnd` | 165 | 0 | | **Total events** | **1,522** | **54** | - `secureConnectionStart` does not appear locally because the dev server is plain HTTP. On staging it is the ninth event. - Branch: the hard `page.load` has 32 `HTTP` children and the client `page.load` of the task has 19 live children (`GET`, `POST`). 0 request spans without a `page.load` parent (master: 146). 19 requests sent `traceparent`, and all 19 point at an exported span (master: 172 sent, 6 point at spans never exported). - Idle traffic in this test (210 `/api/` requests in total): `GET /api/issues/:id/live-runs` 88 times in 2 minutes, and `GET /api/health` 44 times (a dev-server-only 2 s poll). Most of the rest are one-off refetches. - Bundle (`vite build`): entry 3,826,982 B raw both, gzip 1,156,148 → 1,156,173 B (+25 B, chunk hash). Lazy SDK chunk 85,423 → 85,825 B raw, 27,723 → 27,859 B gzip. ## Risks - Low risk. The change is limited to the operator-gated browser tracing path on hosted instances. - Behavior shift: Honeycomb no longer has browser spans for requests made outside a page load, such as polling and mutations in a loaded page. Their server spans are still recorded as their own traces. - A request that starts during a page load and finishes after it is still recorded, as a child of that load. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, headless Chromium). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
5859e7cfa6 |
perf(apps): avoid repeated writes during permission saves (#21033)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Apps give agents access to tools through saved profiles and approval rules. > - A save replaces the full set of action entries and access bindings. > - It also updates each Ask first rule and each retired rule in a separate query. > - Adding one agent can therefore cause hundreds of database round trips. > - This pull request keeps unchanged rows and batches the required changes. > - The benefit is faster saves with the same action permissions. ## Linked Issues or Issue Description **What happened?** An app permission save repeats database work for actions and agents that did not change. With 100 actions and 500 retired rules, one agent addition makes 636 queries. Of these, 599 update approval policies. Related boundary coverage: Refs #19973. This change keeps that issue's Off and agent-selection protections. **Expected behavior** Adding an agent applies the saved Allowed, Ask first, and Off choices. The save changes only the rows that need a change. Independent approval rules remain in force. **Steps to reproduce** 1. Create an app with 100 active actions and 99 Ask first rules. 2. Add 500 disabled app approval rules. 3. Save access for three agents. Then add a fourth agent. 4. Count database queries during the save. **Paperclip version or commit** Baseline: `51d77648bc62`. **Deployment mode** Built from source. The reproduction uses disposable local PostgreSQL. ## What Changed - Keep action entries and access bindings when their saved values match. - Insert new bindings in one query. Remove only bindings and entries that no longer apply. - Read managed approval rules for the selected app. Skip unchanged rules and rules that are already disabled. - Batch new or changed Ask first rules and disable stale enabled rules in one query. - Keep existing policy conditions, priority, author, and creation time when a rule changes. - Add real PostgreSQL regression tests and a repeatable benchmark. Document independent approval rules. ## Verification - Red: the original regression made 2,995 policy writes across five saves and failed the zero-write check. - Green: 636 focused service and runtime tests passed after the rebase. CI supplies the broad checks required by `pnpm verify` (exit 2). - Benchmark: `node --import ./server/node_modules/tsx/dist/loader.mjs scripts/benchmark-app-permission-save.ts --baseline-ref=51d77648bc62 --output=/tmp/app-permission-save.json`. - Local benchmark: two warmups and six samples in alternating order. Restore three agents outside each timed sample, then add the fourth agent. Median save time fell from 458.51 ms to 38.05 ms (12.0 times faster). Query count fell from 636 to 32. Unchanged policy writes fell from 599 to zero. - Compare all saved profile, entry, binding, and policy values after mapping fixture IDs and excluding generated row IDs and write times. Check Allowed and Ask first decisions for the added agent. Tests also cover Off, an unassigned agent, additive access, removal, independent policies, company boundaries, and re-enabled rules. - CI passes on `069e2c76252a19ec3c11a061de0506eccb0463d1`: [PR workflow](https://github.com/paperclipai/paperclip/actions/runs/43162600023). This includes typecheck, build, server tests, and all eight browser shards. - [Greptile](https://github.com/paperclipai/paperclip/pull/21033#issuecomment-6815608829) gives the final head 5/5. Its one finding is resolved. The separate Paperclip Review check is still queued. Its review task stopped after a Runner failure. - Hosted request timing and the specific reported approval prompt are not verified. No live permissions changed. ## Risks - The save now retains unchanged row IDs and timestamps. Consumers must use permission values to determine access. - Existing transaction locks and permission checks remain in place. The runtime concurrency tests exercise these paths. - The benchmark measures a local service with synthetic data. It does not establish hosted endpoint latency. - No migration or startup change is required. ## Model Used OpenAI Codex, GPT-6. Exact provider model ID is not exposed in this session. Used reasoning, tool calls, and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
6ad4529c5a |
fix(native): preserve pack proof across interrupted reattachment (#21246)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - The server can connect again to a live Runner after a restart.
> - Each new controller claim changes provider-pack admission to
pending.
> - The claim kept the previous verified proof only in local memory.
> - If recovery failed before its callback, the next claim lost that
proof.
> - This pull request saves a small reference to the verified proof with
the claim.
> - The next claim can check the same receipt without weakening
admission checks.
## Linked Issues or Issue Description
**What happened?**
A restart recovery attempt timed out before its admission callback. The
next attempt refused the saved pack with `source_state_pending`. It then
failed with `native_provider_artifacts_admission_unverified`.
**Expected behavior**
The next attempt can reuse the previous verified proof for the same
receipt, session, Runner, and lease. Current admission must remain
pending until its normal callback succeeds.
**Steps to reproduce**
1. Save verified provider-pack admission for a run.
2. Claim that run again. Commit the pending admission state.
3. Interrupt recovery before the session admission callback.
4. Claim the same run again and reconnect to the same Runner.
5. Observe that the old code refuses the saved receipt.
Related work: #20682 and #20850 address lock waits. #21085 adds refusal
logs. #21161 addresses command waits after restart. This change
addresses lost admission proof across interrupted claims. The original
transport timeout remains a separate investigation.
## What Changed
- The admission module prepares the claim patch and its recovery source.
- The claim saves a versioned reference with the source run, receipt
digest, and previous verified state. It adds no copy of provider files
or credentials.
- A later pending claim can use that reference only if the receipt is
unchanged. Normal company, agent, session, Runner, and lease checks
still apply.
- Missing proof, unknown state, a changed receipt, or a different Runner
stays refused. The callback clears the reference when it records its
decision.
- The executor uses this helper under its existing claim lock. Existing
artifact-retention and ownership checks stay in place.
- The execution guide states the previous-release behavior. Old verified
records seed the reference. Old pending records without a reference
remain refused. Older readers ignore the field and keep their existing
refusal behavior. No migration is needed.
## Verification
- The new interrupted-claim regression failed before the fix: admission
returned false instead of true.
- Focused admission, PostgreSQL fencing, and executor lease tests passed
after the rebase: 61 tests on commit
|
||
|
|
08be8be414 |
fix(recovery): report exact retained sandbox resumes as resumed (#21259)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A controller restart must keep the same remote execution owner. > - A retained sandbox lease stores the result of its first acquisition. > - That result can say replacement after the same allocation resumes. > - The Runner then rejects a valid resume as a changed sandbox. > - This pull request reports the verified operation as resumed. > - The saved acquisition history and identity checks stay intact. ## Linked Issues or Issue Description Refs #18934. That change makes the retained sandbox ready before preparation. This change corrects the acquisition receipt after that resume. **What happened?** A remote native run resumed the exact retained sandbox after a controller restart. The lease still carried a historical replacement receipt. Runner verification rejected it with `native_remote_recovery_lease_mismatch`, stage `runner_verification`, field `replacement_sandbox`. **What did you expect to happen?** Report the verified exact-allocation operation as resumed. Preserve the original saved history. Reject a changed, absent or expired allocation. **Steps to reproduce** Save an active lease with historical replacement metadata. Resume that exact provider allocation through `acquireForRun`. Resolve its execution target. Before the fix, the target still reports replacement and Runner verification refuses it. **Deployment mode** Native Runner on a remote sandbox. The reported provider was Daytona. ## What Changed - Return a separate operation receipt after the exact retained allocation passes the existing resume checks. - Pass the receipt through heartbeat preparation and apply it only to the execution target. - Keep the saved lease metadata and workspace unchanged. - Test historical created, replacement and resumed receipts. Test ordinary and remote-authority realization through the real execution target resolver. Check every saved metadata write. - Document the difference between saved acquisition history and the current resume operation. ## Verification - Head: `74d03e8ddadd1274891bea99a10455795c500557`. - Red: three historical receipt cases fail without the resume receipt. - Review regression red: realization writes resumed over the historical replacement receipt. The saved metadata loses the previous allocation and cause. - Green: all 69 orchestrator tests pass. Ordinary and remote-authority realization preserve history in every metadata write and return resumed in the target. - Green: 135 focused orchestrator and remote controller adoption cases pass. Identity and changed-allocation refusals remain effective. - Server typecheck passes, including its TypeScript build dependencies and test layers. - `git diff --check` passes. - `pnpm verify` exits 2 because the execution-semantics document needs broad CI. Required CI supplies that coverage. - Linux x64, Node 25.9.0. Provider boundaries are simulated. - The first post-merge attempt could not collect tests because the compiled plugin SDK lacked the new taskLease schema. Shared and plugin SDK builds passed, then the unchanged selection passed. - Full preflight stopped because Cargo is absent. Dependency installation completed. The later server TypeScript build and typecheck passed. - Live retained Runner reattachment is not verified. This workspace has no provider credentials. No rollout or original-task recovery by this patch is claimed. - Required current-head CI and automated review remain pending. ## Risks A wrong receipt could hide an actual replacement. The projection occurs only after the existing company, task, run, agent, workspace, provider allocation, expiry and directory checks pass. The provider must resume the exact allocation. Runner process and identity verification still runs. No replacement is acquired and no saved acquisition history is changed. The heartbeat change only carries the typed receipt between the owning orchestrator stages. Callers without a receipt keep their existing behavior. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository inspection, code execution and tool use. The exact backend snapshot and context window are not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
2b1d0fc825 |
fix(connectors): keep result classification alive during run progress (#20763)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - App connectors screen returned content before they give it to an agent. > - The model screen checks run identity and reserves its budget in one transaction. > - That transaction took a repeatable-read snapshot before it locked the run. > - A run event could change the row first and cause all four attempts to fail. > - This pull request uses READ COMMITTED for this admission under the existing locks. > - The benefit is that connector results can pass screening while run progress continues. ## Linked Issues or Issue Description Refs #20385 and #20484. These changes fixed other run accounting paths. They did not change classifier admission. **What happened?** A connector read can return the accounting contention error before the model screen starts. A real gateway test reproduces SQLSTATE `40001` at `prompt-injection-prevention.ts` when it reads `heartbeat_runs FOR SHARE`. Each attempt commits a run event after the transaction starts its snapshot. **Expected behavior** Return screened connector content and save its classifier charge while run events continue. Refuse paid classification if the run stops, its identity changes, or access or budget checks fail. **Steps to reproduce** Run this test with the production source from `9574065a28`: `cd server && pnpm exec vitest run src/__tests__/prompt-injection-gateway.test.ts -t 'returns a screened connector result'` The test fails after four `40001` errors. It passes with this change. ## What Changed - Use READ COMMITTED only for prompt-injection admission. Keep company, project, configuration, agent, run and secret share locks. - Keep classifier settlement at repeatable read. Keep provider calls outside transaction retries. - Test screened results, stored charges and settled holds under run progress. Test four concurrent reads and refusal after a run stops or changes identity. - Document the transaction boundary. ### Hypothesis and thesis for a fix **Hypothesis:** the classifier admission snapshot becomes stale before its run share lock. The gateway regression confirms this cause in current source. The historical connector error alone does not identify its SQL statement. **Thesis for a fix:** read current run state under the existing locks at READ COMMITTED. This removes stale snapshots caused by run events. The change belongs to classifier admission because every screened connector result uses this service. Investigation source: [Braintrust pattern](https://www.braintrust.dev/app/Paperclip/p/Paperclip%20Staging/patterns/d5369c32-4c0f-43a1-af94-a01fef0a697b). Access is required to read the traces. **Proof:** baseline `9574065a28` failed with `40001` after four attempts. The same single-read regression passed after the change. The full focused suites then passed, including four concurrent reads, stored charges and refusal controls. ## Verification - Real PostgreSQL gateway regression: baseline red in 28.07 seconds; candidate green in 23.16 seconds. - `cd server && pnpm exec vitest run src/__tests__/prompt-injection-gateway.test.ts src/__tests__/prompt-injection-prevention.test.ts`: 54 tests passed. After sync with master `1ac8b6a711`, all 54 passed again in 34.46 seconds. - The tests use the real gateway, content guards, classifier service and database. The MCP server and paid model are simulated. This is not a live deployment test. - Existing controls cover company boundaries, budgets, credentials, configuration changes, interruption and receipt recovery. - Staging health reported `16024ffa2`. Its classifier and accounting source matches the reproduced baseline and includes #20385 and #20484. The revision at the earlier failure times remains unknown. - `pnpm verify` selected 2,512 of 3,303 test files because its dependency policy retained 2,510 files conservatively. The broad local run was stopped after the selection became clear. It is not a pass. The focused 54-test result above is the local proof; required CI supplies broad coverage. An earlier selection was also invalidated by a concurrent commit. - Full preflight could not finish because this host has no Cargo. Pinned dependencies and shared/plugin SDK builds completed. The focused tests passed without Rust. - [Greptile reviewed this head at 5/5](https://github.com/paperclipai/paperclip/pull/20763#issuecomment-6790238737). No inline thread is open. [Paperclip completed its assessment at 5/5 with no findings](https://github.com/paperclipai/paperclip/pull/20763#issuecomment-6790312753). [Required CI passed for head `643bbe51faa70dcb17f372b9e85e3cbbcb724b9d`](https://github.com/paperclipai/paperclip/actions/runs/42981473514), including typecheck, build, 12 server test shards, nine serialized suites, eight E2E shards and Runner checks. Snyk and Socket passed. The full inventory contains 58 completed check runs; the neutral Paperclip check is a completed 5/5 assessment, not an incomplete review. ## Risks - READ COMMITTED uses a new snapshot for each statement. Existing admission locks still protect the run status, identity, configuration and secret checks. Budget checks and reservation writes still commit together. - Concurrent independent admissions retain existing budget behavior. This change adds no retry of paid provider work. - No schema, API or saved receipt format changes. - Historical failing SQL is unavailable. The source regression proves this failure path; it does not prove that every historical example has the same cause. - The change is unmerged and not deployed. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools and code execution. The exact deployment model ID and context-window size are not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal Paperclip ticket id - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
cfa959add7 |
refactor(slack): correct stale status comment and add production acceptance checklist (#20840)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Slack chat connector lets people talk to their agents in Slack > - The connector is close to production, but its live evidence is out of date > - The September live results use old revisions, and the October setup run did not test follow-up, reauthorization, or removal > - Cloud managed setup has never run against real Slack, and its docs say not to enable it until it has > - This pull request adds one checklist for a fresh live pass on current master, and corrects one stale code comment > - The benefit is a clear finish line for Slack production readiness on self-hosted and Cloud ## Linked Issues or Issue Description **Issue type** Missing documentation. **Where is the issue?** `doc/connections/SLACK-AUTOMATIC-SETUP.md`, `doc/plans/chat-adapters/2026-09-05-slack-live-qualification-result.md`, `doc/testing/slack-cloud-qualification.md`, and `server/src/services/chat-channels.ts`. **What's wrong?** The Slack live qualification is spread over three documents. Their evidence uses older revisions. No document lists what a production pass must cover for Cloud managed setup. A code comment says the manifest does not request `assistant:write`, but the manifest does request it. **Suggested fix** Add one dated acceptance checklist with pass conditions per row. Correct the comment. ## What Changed - Added `doc/plans/2026-11-11-slack-production-acceptance.md`. It has sections for self-hosted automatic setup, manual and existing apps, conversation behavior, Cloud managed setup on staging, and distribution questions (Marketplace-only workspaces, non-Marketplace rate limits, scope justification, zero-copy). - Corrected the comment in `server/src/services/chat-channels.ts`. It now says that Slack shows the working state through `agents.sessions.setStatus` and the lifecycle reply. No code behavior changed. ## Verification - Documentation and a comment only. No runtime behavior changed. - I checked the doc links and the referenced files against current master. ## Risks - Low risk. The change is documentation and one comment. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and web search. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [ ] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
64ec273845 |
perf(ci): run the typecheck lane's checks in parallel and drop its duplicate server type check (#21404)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Every pull request runs the trusted PR workflow, and the `Typecheck + Release Registry` job is one of its required checks > - That job takes about 9.2 minutes on the 4 vCPU / 16 GB fleet runner (run 43344818387), against a target of 7 minutes or less for every job > - The job runs six steps one after another, and the first step type-checks the whole server program a second time, at about 13.5 GiB, which makes the runner swap > - Typecheck results are the check that authors most want back quickly, so this job is on the critical path of most pull requests > - This pull request removes the duplicate server type check and runs the independent checks in parallel through one small script, inside the same memory budget > - The benefit is about 39% less wall time for the job's checks on the same machine, with peak memory down from about 14 GiB (plus swap) to about 8.5 GiB ## Linked Issues or Issue Description **What existing behavior does this improve?** The `Typecheck + Release Registry` job in `.github/workflows/pr-trusted.yml`. It takes about 9.2 minutes. The target for every job in the workflow is 7 minutes or less. **Subsystem affected** CI (`.github/workflows/pr-trusted.yml`) and the typecheck scripts under `scripts/` and `server/scripts/`. **Current behavior** After about 75 s of setup, the job runs six steps in sequence on a 4 vCPU runner, about 470 s in all (run 43344818387): `typecheck:build-gaps` 121 s, test programs 85 s, server layers 128 s, test projects 38 s, CLI guidance test 5 s, release registry tests 95 s. Most of these steps use one thread. The `typecheck:build-gaps` step builds the server with a full type check of the whole server program. The server layer step then checks the same files again, and the `Build` job checks them a third time. That full program peaks at about 13.5 GiB, so the job peaks at about 14 GiB of 16 GiB and uses swap. **Proposed behavior** The job runs `node scripts/ci-typecheck-lane.mjs`. The script builds the server without type checks (`PAPERCLIP_TSC_EMIT_ONLY=1`, as the Docker image does) and still checks the runtime assets. It runs the steps that write shared files first, one at a time. Then it runs the remaining checks in parallel, one per CPU, with the server declaration layers first, because the test layers and test projects read them. The checks that are not on that path run at a lower CPU priority. **Reason and benefit** On the same runner, the job's checks take 61% of the time and about 60% of the memory (see Verification). The job should take about 6 minutes on the fleet instead of 9.2. **Breaking changes** None. The same files are type-checked with the same options. Local commands do not change. `pnpm run typecheck:build-gaps` still works as before. ## What Changed - `scripts/ci-typecheck-lane.mjs` (new): runs the job's checks. It writes shared outputs first, in sequence: links, plugin SDK, server build without type checks, the runtime asset check, the sandbox provider installs, and the Runner typecheck (it rebuilds the eval kernel, which other checks read). Then it runs these checks in parallel: the server declaration layers, each server test layer, each test project, the test programs, the other build-gap workspaces, the CLI guidance test and the release registry tests. A passing check prints its output in a collapsed log group. A failing check prints its full output between `=== FAILED` and `=== end of` lines, with an error annotation. The script ends with a table of each check's start, duration and result, and exits non-zero if any check fails. - `server/scripts/typecheck-layers.mjs`: new `--layer <name>` option that checks one layer against the configs and declarations of an earlier `--declarations-only` run, and an exported `listLayers`. - `scripts/typecheck-test-programs.mjs`: new `--prepare-only` and `--skip-prepare` options, so the installs happen before the parallel checks start. - `scripts/run-typecheck-build-gaps.mjs`: new `--list-packages` option, so the script still finds the build-gap workspaces and still fails when one has no `typecheck` script. - `.github/workflows/pr-trusted.yml`: the `typecheck_release_registry` job runs the script instead of the six steps. No other job changes. - Tests: new `scripts/__tests__/ci-typecheck-lane.test.mjs` (added to `test:release-registry`), new cases in `server/scripts/typecheck-layers.test.mjs` and `scripts/typecheck-test-programs.test.mjs`, and the workflow wiring assertions in `.github/scripts/tests/ci-plan.test.mjs` and `scripts/__tests__/runner-release-crate-cache.test.mjs`. - `tests/runner-e2e/source-sdk-public-inputs.json`: repinned, because `package.json` changed. - `doc/DEVELOPING.md`: new "Typecheck lane in CI" section, and the new options. ## Verification This PR's own CI ran the old job definition, because PR runs use the workflow from master. So I measured the commands on a stand-in machine: a temporary workflow (`.github/workflows/typecheck-lane-bench-temp.yml`, now deleted) on GitHub-hosted `ubuntu-latest` (4 vCPU Intel Xeon 6975P-C, 15.7 GiB, no secrets, `contents: read`). It ran in runs [43444356017](https://github.com/paperclipai/paperclip/actions/runs/43444356017), [43445393958](https://github.com/paperclipai/paperclip/actions/runs/43445393958), [43446786105](https://github.com/paperclipai/paperclip/actions/runs/43446786105), [43448707837](https://github.com/paperclipai/paperclip/actions/runs/43448707837) and [43450786493](https://github.com/paperclipai/paperclip/actions/runs/43450786493). Each step ran under `/usr/bin/time -v`, and a sampler read `/proc/meminfo` every 0.25 s. **Old steps, one at a time (3 runs, run 43445393958):** | Step | Wall s | CPU s | Peak RSS of one process | System memory peak | | --- | --- | --- | --- | --- | | `typecheck:build-gaps` | 128–155 | 376–400 | 13.2–13.6 GiB | 14.2–14.7 GiB | | ...of which the full server `tsc` | 40–51 | 146–164 | 12.8–13.8 GiB | 13.7–14.7 GiB, swap up to 738 MiB | | ...replaced by the emit-only server build | 6–10 | 20–22 | 2.9 GiB | 3.9 GiB | | Test programs | 67–87 | 147–170 | 1.9 GiB | 2.9 GiB | | Server layers | 102–127 | 150–172 | 4.0–4.2 GiB | 5.4–5.6 GiB | | Test projects | 31–37 | 49–59 | 2.1–2.2 GiB | 3.1–3.2 GiB | | CLI guidance test | 2–4 | 2–3 | 0.2 GiB | 1.6 GiB | | Release registry tests | 82–95 | 159–183 | 1.7 GiB | 3.0 GiB | CPU seconds are about twice the wall seconds for the `tsc` steps even on one checker thread. So each step uses about two of the four vCPUs. **Old sequence and new lane on the same runner (4 runs, run 43450786493).** Each job copied one installed checkout, then ran both, in alternating order: | Run | Old steps, wall s | New lane, wall s | Ratio | Old memory peak | New memory peak | | --- | --- | --- | --- | --- | --- | | 1 | 503 | 314 | 0.62 | 13.98 GiB | 8.33 GiB | | 2 | 647 | 388 | 0.60 | 14.04 GiB, swap 637 MiB | 8.13 GiB | | 3 | 627 | 385 | 0.61 | 13.98 GiB | 8.63 GiB | | 4 | 382 | 237 | 0.62 | 14.10 GiB, swap 1.1 GiB | 7.99 GiB | Hosted runners differ in speed by up to 1.7 times, so only the same-runner ratio is meaningful. Both arms include a cold compile of the Runner release crate (45–80 s) that the fleet restores from cache. The fleet ran the old steps in 470 s; at the measured ratio, the new lane should take about 290 s, and the job about 6 minutes with setup. The new lane peaks at about 55% of memory, with no swap. Local checks (on a loaded Mac, for correctness only): - `node --test scripts/__tests__/ci-typecheck-lane.test.mjs` — 8 pass. - `node --test .github/scripts/tests/*.test.mjs` — 807 pass. - `node --test scripts/__tests__/e2e-shard.test.mjs scripts/__tests__/release-verify-workflow.test.mjs scripts/__tests__/runner-release-crate-cache.test.mjs scripts/__tests__/run-typecheck-build-gaps.test.mjs` — pass. - `pnpm exec vitest run scripts/typecheck-layers.test.mjs` in `server/` — the new cases pass. ## Risks - This changes a required job that every PR depends on, and its own CI cannot prove it. Please watch the first `Typecheck + Release Registry` runs on master after merge: the wall time (expect about 6 minutes), the summary table at the end of the step, and any `terminated by` message. - A type error now shows in that check's `=== FAILED` block, not in a step of its own. Passing checks are folded, so the failure is the only expanded block. - When the server declaration layers fail, the test layers and test projects are reported as skipped. Before, the test layers still ran after a source error. Fix the source errors first; the next run checks the tests. - The Honeycomb CI time board and any step-name queries see one step, `Typecheck, CLI guidance safety and release registry tests`, instead of six. The script prints the duration of each check. - The release registry tests run beside the type checks at normal priority. Their slowest test took 46–78 s here, against a 180 s timeout, and 114 s on the slowest stand-in runner. - Low risk for memory: the new lane peaked at 8.0–8.6 GiB of 15.7 GiB, far below the old 14 GiB. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), through Claude Code, with tool use (shell, GitHub CLI, file editing). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
bb253c6a99 |
fix(ui): save new tasks in the background (#20831)
## Thinking Path > - Paperclip lets people manage AI agents and their tasks. > - The New task composer is the entry point for new work. > - The composer currently waits for the server to save a task before it closes. > - A slow request prevents the person from starting the next task. > - This change closes the composer as soon as it keeps the submitted request. > - Separate request records keep retries and files safe while another draft is open. > - The person can submit the next task without waiting for the first save. ## Linked Issues or Issue Description **What happened?** The New task composer stays open until task creation and file uploads finish. A slow save blocks the next task. **Expected behavior** Enter closes the composer at once. The save continues in the background. A failed save restores the request. A later draft stays intact. **Steps to reproduce** 1. Open New task and enter a request. 2. Delay the create response. 3. Submit the request. The current composer stays open until the response arrives. **Paperclip version or commit** Reproduced on source commit `a4e89288ec`. **Deployment mode** Board UI built from source. The save behavior does not depend on the server deployment mode. Related transport work: #13428. The open PR and issue searches did not find a duplicate for background New task saves. ## What Changed - Add an account-scoped queue around the existing durable submit hook. Keep each request and creation key before closing the composer. Send queued requests in order. - Keep submitted draft settings and files separate from the next draft. Resume keyed requests after a reload. Keep legacy recovery records readable. - Reopen a failed draft when the composer is free. If another draft is open, show a retry action beside it. An earlier receipt cannot clear the later draft or cause late navigation. - Add regression tests for slow saves, consecutive tasks, rejections, reloads, file ownership, and account lookup. Update the product and development docs. ## Verification - Red/green: the slow-response regression failed before the fix because the composer did not close. It passes after the fix. A second red/green case proves that missing files from an earlier task cannot change the next draft's parent or goal. - Final focused run: `pnpm exec vitest run ui/src/hooks/useBackgroundSubmitQueue.test.tsx ui/src/components/NewIssueDialog.test.tsx ui/src/hooks/useDurableSubmit.test.tsx ui/src/hooks/useStagedIssueFiles.test.tsx ui/src/lib/pending-send.test.ts ui/src/components/task-chat/TaskChatComposer.test.tsx` — 6 files and 316 tests passed on Linux with Node 25.9.0. - `pnpm --filter @paperclipai/ui typecheck` passed. `git diff --check` passed. - An earlier `pnpm verify` run selected 544 test files and overlapped the last edits. It reported the new account-lookup file-scoping assertion before the worker guard was complete. The stale run was stopped. The final focused run passes that case. Do not use the earlier run as proof for the final revision. - Store each queued request in its own browser entry. A submission or receipt in one tab cannot erase another tab's queued tasks. A red/green two-tab regression proves recovery after reload. Legacy queue migration also passes. - Follow-up recovery checks: a saved rejection cannot replace a newer saved draft or a fresh launch. Edits survive close, reopen, and reload. A retry keeps its original hidden workspace context and request key. The new regressions failed before the fixes and pass after them. - Updated the branch from `master`. This includes the SDK pin repair in #20834 and the span test repair in #21149, which address the two original CI failures. The SDK input drift check now reports no stale pins. - CI must provide broad checks for the final commit. Browser and release suites were not run; this change does not modify those suites. The dependency preflight installed packages and built the shared and plugin SDK outputs. Its later Runner step could not run because Cargo is absent in this UI workspace. ## Risks - Requests now close the composer before the server confirms a save. A failed request must stay visible and recoverable. The tests cover rejection, retry, reload, and a newer draft. - The queue and file snapshots use browser storage. If storage is unavailable, requests remain in memory and a reload can require the files again. - Server creation, authorization, and duplicate-request handling retain their existing APIs. No database migration is required. - The queue sends requests in order. An uncertain first save can delay later server saves while the person continues to compose. ## Model Used - OpenAI GPT-6 through Codex, with reasoning, shell tools, and code execution. The exact service model ID and context window are not exposed in this run. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
84a4d345b3 |
refactor(docker): take the application image's Runner from the runner-artifacts stage (#21332)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner is the part that executes agent work. `master` publishes it alone as the image `ghcr.io/paperclipai/paperclip-runner`, built by the `runner-artifacts` stages of the `Dockerfile` > - The application image does not use those stages. It has five stages of its own that compile the daemon, it builds the Runner again inside the server build, and it builds a second provider pack > - So one `Dockerfile` holds two recipes for the Runner, and the two results differ: another base image and another daemon binary > - This pull request removes the second recipe. The application image copies the Runner out of the `runner-artifacts` stage, which the same build runs once > - The benefit is one Runner recipe, and an application image whose Runner is the Runner the contract describes. A later pull request can replace the stage with the published image ## Linked Issues or Issue Description This is the second part of a chain. The first part is #21331, which removes the cache check that cannot run on this `Dockerfile`. Refs #21204, #21229, #21230 (the Runner image), #21306 (the no-rebuild mode of the vendor step), #21307 (the pull-request check of the application image) and #21312. No issue exists for this change. **Problem or motivation** The stages `rust-toolchain`, `rust-chef`, `runner-plan`, `runner-deps` and `runner-build` compile the daemon on Debian trixie. The server build then runs the full build of the Runner package. The `runner-provider-pack` stage builds the provider pack. The `runner-artifacts` stages do all of that again for the Runner image, on Debian bookworm, with a stripped daemon. **Proposed solution** Read only the `runner-artifacts` stage from the application stages. Build it inline in every build. Do not pass a published image yet. **Alternatives considered** - Pass the published Runner image to the build in this pull request. That needs the resolver, labels and a rule for commits without an image. It is the next step, and this one must be safe alone. - Keep the old stages as a fallback. Two recipes is the problem. ## What Changed - `Dockerfile`: remove the five stages. `build` starts from `base`. It copies `/runner-package/dist` and the build stamp from `runner-artifacts`, sets `PAPERCLIP_RUNNER_VENDOR_REQUIRE_CURRENT=1` and runs the vendor check before the first application build. - `Dockerfile`: `runner-provider-pack` starts from `base`. One `RUN` with a bind mount of `runner-artifacts` copies the pack, the receipt and the archives, and moves the large files to their own tree. `build` and `runner-provider-pack` now come after the stages they read. - `Dockerfile`: the three `COPY` instructions of `production` that take the pack are byte-identical. A test pins all three. - `Dockerfile`: `build` declares the build argument `PAPERCLIP_RUNNER_EXPECT_INPUTS_DIGEST`, empty by default, and one step runs the new `scripts/check-runner-image-expectation.mjs` on `/runner-image.json` of `runner-artifacts`. **This pull request declares the argument. No build sets it. The next pull request, which gives a published Runner image to the build, is its first user.** With the argument set, the build fails unless `runner-artifacts` is the published image of exactly those inputs. `docker-app-check.yml` also runs when the script changes. - Comments: the `Dockerfile` has 220 comment lines, down from 247. The comments of the two moved stages moved with them, unchanged, so the diff shows them as added lines. Eight comments that named a removed stage or a step that moved get the smallest correction. Two comments in `prepare-runner-vendor.mjs` and `build-inputs.mjs` said that the `Dockerfile` relies on the standalone rebuild; that clause is removed. - Tests: replace the two cargo-chef tests of `docker-build-stamp.test.ts` with three tests of the new shape. Update the pack test, `docker-image-layer-reuse.test.ts`, `vite-build-commit.test.ts`, the stage rules of `runner-image-identity.test.ts`, and the include-layout test of `runner-release-crate-cache.test.mjs`, which now reads `runner-artifacts-runnerd`. - Docs: `doc/DOCKER.md`, `doc/DEVELOPING.md`, and a new section "The application image" in `doc/runner-image-contract.md`. ## Verification All builds: `linux/arm64`, local, `--build-arg PAPERCLIP_BUILD_COMMIT=<sha>` with one commit for every build. The builds ran before the rebase onto `master`, on the head of #21310 as it was before it merged. `master` changed neither the `Dockerfile` nor an input of the Runner in between. - `docker buildx build --target production`, cold, before and after: 506 s and 410 s. Compressed size: 2,971 MB and 2,900 MB. - The image config and the text of all 33 history rows are identical. The row `COPY /provider-pack-transport/ /opt/paperclip-runner/ # buildkit` matches the pattern of `retained-provider-pack-oci.ts` exactly once. - A comparison of every file of the two images (path, type, size, mode, link target, and SHA-256 under `/app` and `/opt`). The differences are in "Risks". All other files are byte-identical, the root `node_modules` included. - From the image after: the four `/app/packages/paperclip-runner/...` paths and the contents of `/opt/paperclip-runner/` exist. `paperclip-runnerd --build-metadata` runs. `provider-pack.json` has `runnerSourceRevision` equal to the build commit and no `runnerInputsDigest`. The entry points `index.js`, `live/index.js` and `testing.js` of the Runner import. - The vendor step logged `reusing the current Runner build` in both of its runs. Every `Compiling` line of the log is in `runner-artifacts-runnerd`. - `--target build`, `--target runner-provider-pack` and `--target cloud` built. The `build` stage has no `gcc`, `cc`, `cargo`, `rustc` or `pkg-config`. - `--target runner-artifacts` and `runner-artifacts-slim`, before and after: `runner-image-layout.mjs check` and `runner-image-oci.mjs check-layers` pass on both, in `node:24.22.3-bookworm`. The daemon, `provider-pack.json` and the two stable layers have the same digests before and after. - Without a build commit: `build` and `runner-provider-pack` built, and the pack was skipped with the same message. This run was on the next branch of this chain, after the rebase. The last steps of `production` were not run for this case. - The expectation argument, on this `Dockerfile`, `--target build --output type=cacheonly`, after the rebase: - No argument, inline Runner: the build passes (233 s). The step passes. - Argument, inline Runner: the build fails at the step. The message says that the `runner-artifacts` of the build are not a published Runner image. - Argument `d4df566f…` and the published image of those inputs as `--build-context runner-artifacts=docker-image://ghcr.io/paperclipai/paperclip-runner@sha256:0cd8250b…`: the step passes, and the build runs no Runner stage. The vendor step then fails with `inputs changed since the recorded build`, because this tree differs from the tree of that image by one comment line in `build-inputs.mjs`. With that line put back, the same build completes, and the vendor step reuses the published build. - Another digest with the same context: the build fails at the step. The message names the inputs the image has. - A tree whose lockfile is behind the manifests of the Runner fails at the frozen install of the Runner stages, after 169 s, with the message that stage already had. - `docker buildx build --check`: no warning, before and after. `scripts/check-docker-deps-stage.mjs` and `scripts/check-node-version-policy.mjs` pass. - Tests, on macOS and in `node:24.22.3-bookworm`, after the rebase: the changed server test files with the other tests that read the `Dockerfile` or the image workflows (231 pass in 10 files on Linux), `vite-build-commit.test.ts` (10 pass), `runner-release-crate-cache.test.mjs` (11 pass), `docker-deps-postinstall.test.mjs`. The changed server test files typecheck. - Not run: the full test suite, any `linux/amd64` build, `podman`, the workflows and the release pipeline. Only CI can prove amd64, `docker.yml`, `docker-merge-group.yml`, `release.yml`, `docker-app-check.yml` and `runner-image.yml`. ## Risks - **The digest of the Runner image changes.** Its identity covers the whole `Dockerfile`. The next `master` push publishes a new `inputs-<digest>`. The contents do not change: see the comparison above. - **A build that is given a Runner of other inputs fails; it does not ship that Runner.** With the argument, it fails at the expectation step. Without the argument, it fails at the vendor step when the build inputs of the Runner differ. One comment line in `build-inputs.mjs` changes those build inputs in this pull request, so the image published before this pull request cannot be vendored into a tree after it. - **What a consumer of the application image sees** (measured on `linux/arm64`): - The daemon is the stripped binary compiled on Debian bookworm: 23.2 MB, was 36.4 MB, at `/app/packages/paperclip-runner/dist/bin/paperclip-runnerd` and at the vendored copy in `server/dist`. Its hash changes. It has no symbols for a crash backtrace. - `/app/packages/paperclip-eval-kernel/dist` is absent. Only `dist/eval/workflow-harness.js` of the Runner imports that package. Only `eval/index.js` exports the harness again, and no module imports that file. The server runtime bundle does not name the package. - The provider pack: `provider-pack.json` is byte-identical. Two provider executables and one licence file are hard links to their other copy in the pack, where they were separate files. One empty directory is absent. The pack is 195 MB smaller, and the layer of its large files gets a new digest once. - The transport archives have other bytes, because the Runner build gives large files a fixed timestamp. The receipt names the same files with the same digests. - Paths, modes, owners, the image config and the layer order do not change. - **What a contributor and a maintainer of the image build see:** - **The image build does not run the drift checks of the Runner.** Before, the server build ran the full `build` script of the Runner, so a failed drift check failed the image build on `master`. Now the Runner stages run only the output steps of that script. Repository CI runs the checks on every pull request. `.github/docker-context-checks.Dockerfile` and three exceptions in `.dockerignore` exist for those checks in the image build; a separate follow-up removes them, and this chain does not edit them. - **The dependency-caching layer of the daemon is gone.** Before, cargo-chef compiled the Rust dependencies in a layer of their own, and a change to the Rust source compiled only the Runner's own crates. Now such a change compiles the dependencies again (146 s here for the whole daemon). A change to the server, the UI or the Runner's TypeScript still reuses the compiled daemon. - **A lockfile behind the manifests of the Runner fails the build.** Before, `docker build` resolved it inside the image. The image workflows refresh the lockfile first. A local build prints the command to run. The next pull request makes this failure early and clear. - **The build needs more disk**: about 25 GB of builder storage here, was about 20 GB, with the image export. - Rollback: revert this pull request. Images that exist do not change. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), from Anthropic, through Claude Code. The model used tools: it edited files, ran the tests and ran the Docker builds that this description reports. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
2de4787f41 |
fix(db): release a postgres.js connection's receive buffer once it is fully read (#21432)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The server reads PostgreSQL through postgres.js, with a patched
driver in `patches/postgres@3.4.9.patch`
> - A hosted server keeps about 0.5 GB of `external` memory (Buffers)
for its whole life, with 340 open database connections
> - postgres.js keeps a zero-length view of the buffer of each
connection's last read. The view keeps the whole buffer alive until that
connection reads again
> - An idle connection therefore pins the socket chunk or the whole
buffer of its largest recent message, which can be one MB or more
> - This pull request replaces a fully read buffer with an empty one in
the driver patch
> - The benefit is that idle connections hold no receive buffers. At 340
connections that each read a 1 MB row, external memory drops from about
320 MB to 0
## Linked Issues or Issue Description
**What happened?**
A hosted Paperclip server holds 455 to 520 MB of `external` memory for
its whole life. It has about 340 open database connections. Each idle
postgres.js connection keeps the buffer of its last read.
**Steps to reproduce**
1. Open 80 or 340 pool connections.
2. Read a 1 MB row on each connection.
3. Drop the results, let the connections go idle, and force garbage
collection.
4. Measure `process.memoryUsage().external`. Each idle connection still
holds about 0.95 MB.
**Expected behavior**
An idle connection holds no buffer from a result that it has completely
read.
**Environment**
postgres.js 3.4.9 with the Paperclip patch, Node.js 24 and 26,
PostgreSQL 17/18.
## What Changed
- `patches/postgres@3.4.9.patch`: in `data()` of `src/connection.js` and
`cjs/src/connection.js`, the driver now replaces the receive buffer with
an empty buffer when it has handled every message in it. A partly read
buffer is not changed. The other hunks of the patch are unchanged. I
regenerated the patch with `pnpm patch postgres@3.4.9`, edited the two
files, and ran `pnpm patch-commit`. I did not commit the lockfile change
from that command.
- `packages/db/src/postgres-receive-buffer.test.ts` and
`packages/db/src/__fixtures__/postgres-receive-buffer.mjs`: a regression
test for ESM and CommonJS on real PostgreSQL.
- It checks that results are reassembled correctly for a 3 MB row, for
161 row sizes around one 64 KB socket chunk (to land message boundaries
at the end of a chunk), and for 20,000 small rows that arrive many to a
chunk.
- It records a weak reference to each large receive buffer and checks
that none of them stays alive after the connection becomes idle.
- `packages/db/scripts/benchmark-postgres-receive-buffer.ts`: one
command that reports heap, external and rss per connection, before and
after large rows.
- `doc/DATABASE.md`: documents the new patch behavior, the test and the
benchmark.
## Verification
- Red/green: with the new line removed from the installed driver, the
fixture fails for ESM and CommonJS ("an idle connection must not keep
the receive buffer of its last result"). With the line in place it
passes, repeated twice for each format.
- `pnpm --filter @paperclipai/db exec vitest run
src/postgres-receive-buffer.test.ts
src/postgres-connection-recovery.test.ts`: 20 passed.
- `pnpm --filter @paperclipai/db exec vitest run src/backup-lib.test.ts
src/backup-lib-binary.test.ts src/client-disconnect-replay.test.ts
src/client-teardown-registry.test.ts src/work-signals.test.ts`: 36
passed. In 1 of 4 runs, the JavaScript backup test (COPY and cursor
reads) timed out on a machine with load average 20 to 34. It took 39 s
and 25 s with the change and 21 s without it in passing runs. Read
throughput is the same with and without the change: a 200,000-row select
plus a COPY TO STDOUT has a median of 319 and 348 ms with it, and 329 ms
without it.
- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/prepared-statements-driver.test.ts
src/__tests__/prepared-statements.test.ts`: 5 passed.
- `pnpm --filter @paperclipai/db typecheck`: passed.
- Benchmark: `node --expose-gc --import tsx
packages/db/scripts/benchmark-postgres-receive-buffer.ts` with a 1 MB
row on each connection, numbers after garbage collection:
| Connections | External per idle connection, before | After | External
held in total, before | After |
| --- | --- | --- | --- | --- |
| 80 | 917 KB | 1 KB | 72 MB | 0 MB |
| 340 | 960 KB | 0 KB | 319 MB | 0 MB |
The rss per connection also went down (80 connections: 1,555 to 1,375
KB; 340 connections: 1,308 to 714 KB). The allocator keeps some freed
memory, so rss changes less than external memory.
## Risks
- Low risk. The change runs only when the buffer has no unread bytes. A
message split across chunks keeps the same reassembly path. The test
covers split messages, exact chunk boundaries and many messages in one
chunk.
- This saves external memory (Buffers outside the JavaScript heap). It
does not reduce `heapUsed`. It does not fix the out-of-memory crashes of
the hosted server by itself.
- The patch must be removed or ported when Paperclip upgrades
postgres.js. The test fails if a driver version keeps the buffer again.
## Model Used
- Claude Opus 5.5 (Anthropic, model ID `claude-opus-5-5`), used in
Claude Code with tool use (shell, file editing, local PostgreSQL), heap
snapshots and allocation measurement.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
bef2aac905 |
chore(ci): remove the Docker dependency cache check of the Runner stages (#21331)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner is the part that executes agent work. The application image contains it > - Today the application image compiles the Runner in its own stages, with a cargo-chef dependency layer > - A pull-request job proves that a fresh builder reuses that layer. The job runs a script that appends to the `Dockerfile` and names those stages > - The next pull request removes those stages. The application image then takes its Runner from the `runner-artifacts` stages. The job cannot run against that `Dockerfile` > - This pull request removes the job and its script > - The benefit is that the next pull request changes only the image build, and its checks stay green ## Linked Issues or Issue Description Refs #21204, #21229, #21230, #21306 and #21307 (the Runner image and the preparation for this change). No issue exists for this change. **Problem or motivation** `.github/workflows/docker-runner-check.yml` has a pull-request job, `Compile isolated native Runner`. It runs `scripts/check-docker-runner-cache.sh`. The script adds stages to a copy of the `Dockerfile` that start from `runner-build` and copy from `runner-plan`. It reads the build log for the `runner-deps` step. The job runs on every pull request that changes the `Dockerfile`. The pull request that removes those stages would fail this job. **Proposed solution** Remove the job, its trigger and its script first, in a pull request of its own. **Alternatives considered** - Remove the job in the same pull request as the stages. That pull request is already large. - Rewrite the script for the `runner-artifacts` stages. Those stages have no dependency layer, so the property does not exist. `runner-image.yml` already builds them on a pull request. This is the first of three pull requests that move the application image onto the `runner-artifacts` stages. It has to merge first: the removed job runs on every `Dockerfile` change and builds stages that the second pull request deletes. The other two follow on top of it. ## What Changed - Delete `scripts/check-docker-runner-cache.sh`. - Remove the `runner` job and the `pull_request` trigger from `docker-runner-check.yml`. Only the manual jobs remain. The concurrency group no longer reads a pull request number. - Remove `Docker Runner check` from the workflow list of `pr-merge-review.yml`. That notifier acts only on pull-request runs, and the workflow has none now. - Remove the paragraph of `doc/DOCKER.md` that describes the check. ## Verification - `node --test .github/scripts/tests/notify-pr-merge-review.test.mjs`: 22 pass. - `vitest run server/src/__tests__/runner-image-workflow.test.ts`: pass (1 test is skipped on macOS, where bash is 3.2). - `vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/workflow-security.test.ts tests/runner-e2e/workflow-target.test.ts`: 28 pass on macOS. These files read `docker-runner-check.yml`. - The same commands in a `node:24.22.3-bookworm` container: all pass, except one test of `workflow-target.test.ts`. It fails with `jq: command not found`. It reads another workflow, and the container has no `jq`. - Not run: the full test suite, and the workflows themselves. Only CI can run the workflows. ## Risks - Until the next pull request merges, a pull request that changes only the native inputs of the Runner (`packages/paperclip-runner/runner/**`, `protocol/**`, `rust-toolchain.toml`) does not build the `runner-build` stage of the application image. `runner-image.yml` builds the daemon from the same files in the `runner-artifacts` stages on such a pull request. `docker-app-check.yml` builds `runner-build` when the `Dockerfile` or `.dockerignore` changes. - No check proves the cargo-chef layer reuse after this change. The next pull request removes that layer. - The job was not a required check. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), from Anthropic, through Claude Code. The model used tools: it edited files, ran the tests and ran the Docker builds that this description reports. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
2659ff1d22 |
fix(release): retry Cloud readiness jobs that lost their Spot runner (#21399)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Every push to `master` publishes a canary, and the canary's image reaches staging > - Each canary waits for `Cloud source verified v1` from the Cloud readiness run of its exact SHA > - Cloud readiness runs on AWS Spot workers. When a worker is reclaimed, the gate failed and the image watcher cancelled the readiness run > - Nothing retried that attempt. On a busy master the newest commit's release failed, and merged work did not reach staging until a person re-ran the run > - This pull request re-runs the failed jobs of a readiness attempt whose every failure is a lost runner. A real check failure still fails closed > - The benefit is that a Spot reclaim no longer holds a release ## Linked Issues or Issue Description **What happened?** On 2026-11-13, #21266 merged as `8af889f612`. No canary that contained it published for more than an hour. The release workflow keeps one running and one pending canary. A newer push replaces the pending one, so the newest commit always runs next. That part works as designed. The failures came from the readiness gate: | Release (SHA) | Readiness run / attempt | Gate that blocked it | Cause | | --- | --- | --- | --- | | `11840f9904` | 43435593630 / 2 | `Reuse exact-source verification` | Spot workers reclaimed at 13:49:56Z. The run was then cancelled by `github-actions[bot]` (the image watcher). Dotta re-ran attempt 3 by hand | | `8af889f612` | 43434587048 / 2 | Release superseded while pending; readiness failed in the same reclaim | Spot reclaim | | `9fbfd93c9b` | 43423674511 / 2 | `Reuse exact-source verification` | Spot reclaim | | `6917d0568d` | 43439942063 / 1 | `Reuse exact-source verification` | Flaky test `decision-models.test.ts` (concurrent budget admission) | | `18fb6ee571` | 43422042261 / 2 | `Reuse exact-source verification` | Flaky test `fresh-continuation-fallback.test.ts` (truncate) | | `5ede64f247` | 43421480302 / 1 | `Reuse exact-source verification` | Both flaky tests | Over the last 200 master pushes, release runs ended 14 success, 33 failure, and 150 cancelled (superseded while pending, which is expected). **Expected behavior** A lost runner is not a source failure. The gate retries the lost jobs and publishes the newest commit that passes. **Steps to reproduce** See `scripts/cloud-readiness-runner-loss.test.mjs`, "busy-master simulation". It replays the 13:49:56Z reclaim through both waiters. ## What Changed - `scripts/cloud-readiness-runner-loss.mjs` (new): classifies a failed readiness attempt with the existing PR classifier (`.github/scripts/runner-loss-rerun.mjs`). It is a runner loss only when every causing job (`failure`, `timed_out`, `startup_failure`) lost its runner. Cancelled siblings are fallout. A run that is still going is cancelled first, then `rerun-failed-jobs` runs on the next poll. The ceiling is readiness attempt 4. Writes go only to `cancel` or `rerun-failed-jobs` of a canonical run, after an identity check. - `scripts/cloud-source-verification.mjs` (canary gate in `release.yml`): on `SourceVerificationFailed`, ask the retry first. The source wait goes from 60 to 90 minutes. - `scripts/watch-release-source.mjs` (master image watcher in `docker.yml`): on a runner loss, keep the image attempt alive and wait for the retried proof. Do not cancel it. - `.github/workflows/release.yml`: `verify_canary` timeout goes from 70 to 100 minutes. - `.github/workflows/pr-trusted.yml`: run the new test file. - `doc/RELEASING.md`: new "Busy master and lost runners" section. ## Verification - `node --test scripts/cloud-readiness-runner-loss.test.mjs scripts/cloud-source-verification.test.mjs .github/scripts/tests/*.test.mjs scripts/__tests__/release-verify-workflow.test.mjs`: all pass. `release-verify-workflow.test.mjs` passes 51 of 51 but does not exit locally. This also happens on `origin/master` without this change. - Red/green: with the two waiter files reverted, 3 of the 6 simulation tests fail (the canary gate, two reclaims in a row, and the image watcher). With the change, all 20 pass. - Dry run against real runs (read-only, writes captured): 43435593630/2, 43434587048/2, and 43423674511/2 give `rerun-failed-jobs`. 43439942063/1, 43421480302/1, and 43422042261/2 (real test failures) give `failed` with no write. - `pnpm verify` exits 2 because a workflow file changed. Required CI supplies broad coverage. ## Risks - A real failure classified as a runner loss would be retried. The classifier needs the runner's own shutdown or lost-communication `##[error]` line in the job log. It is the same classifier PR CI uses, and unreadable evidence fails closed. - On a reclaim, the waiter cancels the readiness run to retry sooner. Jobs that were still healthy run again. This costs runner time, not correctness. - Flaky tests still fail a canary. This change does not retry them. They need their own fixes. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, GitHub CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
51c81c537b |
fix(browser-tracing): end hidden loads, cap timeouts, backfill hard-load API spans, record list view (#21392)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Hosted instances send real-user page-load traces to Honeycomb (#21156), so we can set page-load SLOs > - Staging data showed four measurement gaps: 140 of 147 hard loads were hidden tabs that became 30 s timeouts, a few timeouts lasted about 2 hours after the computer slept, hard loads had no API child spans, and list views on the tasks page could not be told apart > - These gaps make every average and percentile wrong, so the data cannot back an SLO yet > - This pull request ends hidden loads as `hidden`, caps timeouts at exactly 30 s and marks suspends, backfills hard-load API spans from Resource Timing, and records the list view from an allowlist > - The benefit is page-load numbers that describe what users see, with no cost on the critical path ## Linked Issues or Issue Description Refs #21156 **What is the issue?** Browser page-load tracing records misleading data. A load in a background tab never paints, so it always ends as a 30 s `timeout`. A timeout that fires late (after sleep) reports hours. Hard loads have no API spans because the SDK loads after the page settles. `?view=` is stripped, so all task list views share one route. **Steps to reproduce** Open the all-tasks page in a background tab on a traced instance. The `page.load` span is `outcome=timeout`, `was_hidden=true`, about 30.4 s (reproduced locally on master: 30,359 ms and 30,988 ms). **Expected behavior** A hidden load is counted separately and ends quickly. A timeout lasts exactly 30 s. A hard load shows its API requests. The list view is a dimension. ## What Changed - `page-load-recorder.ts`: new outcome `hidden`. A load that starts hidden, or hides before it is ready, ends at the moment the tab hid (same rule `web-vitals` uses for LCP/FCP). For a hard load, the controller reads Chromium `visibility-state` entries, so a hide before tracing started is caught. A redirect while still hidden renames the hidden span instead of starting a second one. - `page-load-recorder.ts`: any end observed after start + 30 s is a `timeout` that ends at exactly start + 30 s. `paperclip.page.suspended=true` marks an end more than 5 s past the deadline or a gap of more than 5 s between frames. - `controller.ts`: a `PerformanceObserver` (`type: resource`, `buffered: true`) keeps only this origin's `/api/` fetch/XHR entries (max 500) until the SDK loads. It does not depend on the 250-entry resource buffer. - `sdk.ts`: when the SDK loads, it creates `HTTP` child spans under each `page.load` that started before the SDK, for requests that started in the load window and before live fetch tracing began (no duplicates). The name is `HTTP` because Resource Timing has no method (OpenTelemetry's name for an unknown method). Status comes only from `responseStatus` when the browser exposes it. The URL goes through the same sanitizer as live fetch spans and becomes `paperclip.api.area`. Spans carry `paperclip.api.timing_source=resource_timing`. - `page-view.ts`: `paperclip.page.view` for `/:companyPrefix/issues` (values from `TASK_VIEW_KEYS`, `default` when missing) and `/:companyPrefix/decisions` (values from `ATTENTION_PERSONAL_VIEWS`, `mine` when missing). Any other value is `other`. - `span-sanitizer.ts`: allowlist adds `paperclip.page.suspended`, `paperclip.page.view`, `paperclip.api.timing_source`. - `doc/browser-performance-tracing.md`: outcome list, the meaning of "loaded" for hidden tabs, suspend handling, page views, backfilled API spans, attribute table, and example queries that filter `outcome = ready`. ## Verification - Red then green: 17 new tests failed on the old code and `page-view.test.ts` failed to import; all 50 tests in `ui/src/lib/browser-tracing` pass after the change. `PageLoadTracker.test.tsx` passes. `tsc --noEmit` for `ui` passes. - `pnpm verify` exits 2 (a doc is a global CI input), so CI supplies broad coverage. - Real browser (local dev server on an empty isolated instance, stand-in OTLP collector, headless Chromium, tracing config injected into the session response): - Background tab (raw CDP, `Target.createTarget background:true`): `outcome=hidden`, duration 0 ms. Master: `outcome=timeout`, 30,359 ms. - Tab covered 400 ms into the load: `outcome=hidden`, 434 ms. Master: `timeout`, 30,988 ms. - Hard load of the all-tasks page: `page.load` with 13 to 38 `HTTP` child spans (areas `companies`, `plugins`, `cli-auth`, and others, status 200 where exposed), same trace, parent = `page.load`. - `?view=mine`: `paperclip.page.view=mine`. - Leak scan of every exported body: no URL, host, company id, company prefix, user id, or query text. - Entry bundle: 3,822,045 → 3,823,991 B raw (+1.9 KB), 1,154,280 → 1,155,097 B gzip (+0.8 KB, 0.07%). Lazy SDK chunk: 84,322 → 85,229 B raw, 27,304 → 27,630 B gzip. ## Risks - Low risk. The change is limited to the operator-gated Observability path on hosted instances; self-hosted installs never start the controller. - Behavior shift for dashboards: hidden tabs are now `outcome=hidden` instead of `timeout`, so timeout rates drop. Queries should filter `outcome = ready`. - The observer adds one callback per batch of finished requests while tracing is on, until the SDK loads. It keeps at most 500 entries. - Backfilled spans do not join server traces because no `traceparent` was sent. The docs say so. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, headless Chromium). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |