mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 20:12:20 +01:00
3fc2198759e80eec0142e40f06f34196e6cf7f59
7826
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
3fc2198759 |
fix(runner): treat tool-call markup in OpenCode text as an unexecuted call (#21361)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Paperclip Runner drives OpenCode and turns its event stream into the run's progress, reply, and completion result > - Some model hosts return a tool call as raw tool-call syntax inside the assistant text, not as a structured tool call > - The OpenCode driver streamed that syntax as progress and accepted it as the final answer, so the run succeeded with markup as its reply and the intended tool call never ran > - This pull request treats leaked tool-call markup as an unexecuted, malformed tool call > - The benefit is that the person never sees raw call syntax as a reply, and the agent gets one bounded correction to make the call properly ## Linked Issues or Issue Description **What happened?** An agent run that uses an OpenRouter model can end with raw tool-call markup as its reply. The model meant to call the completion tool. The host put the call into the message text as DeepSeek DSML markup (for example `<|DSML| invoke name="...">`). The runner accepted that text as the final answer and settled the run as succeeded. The completion call did not run, so follow-up automation repeated the work. **Steps to reproduce** 1. Run an OpenCode agent with a DeepSeek model through an OpenRouter host that leaks tool-call syntax into text. 2. Let the model end its turn with a completion call that the host returns as text. 3. Look at the run reply. **Expected behavior** The runner does not show the markup and does not accept it as the reply. It tells the model that the tool did not run and asks it once to call the tool through the tool interface. **Paperclip version or commit** `master` at the time of this pull request. **Additional context** Today the markup streams as progress and becomes the run's final reply. The run succeeds without its completion call. ## What Changed - Added `tool-call-markup.ts` in the OpenCode driver. It detects leaked tool-call markup in one linear scan: DSML and DeepSeek call-token sentinels anywhere, and `<function_calls>` / `<tool_call>` wrappers only at a line start with a call body. It ignores fenced code blocks. - The driver holds markup (and a trailing fragment that can still become markup) out of streamed progress text. - A completed text part that contains markup is not a final-answer candidate. The driver emits the `opencode_tool_call_markup_discarded` diagnostic and records the position, so earlier text cannot become the reply either. - The turn then goes through the existing one-shot empty-response correction. The correction text says the tool did not run and asks the model to call tools only through the tool interface. - Regression tests: detection cases (recorded DSML shape, other families, prose and code-fence negatives, streaming monotonicity) and a driver case that replays the recorded markup. ## Verification - `cd packages/paperclip-runner && npx vitest run src/drivers/opencode/` passes (20 files, 263 tests). - Red/green: the new `tool-call-markup` case in `empty-response.test.ts` fails on the unchanged driver (the markup is selected as the final reply) and passes with this change. - `npx tsc --noEmit -p packages/paperclip-runner` passes. - Cost: the scan adds about 22 µs for each streamed update of a 20 KB answer with `<` on every line (43 ms over 1,960 updates), and 0.05 ms for each completed part. It adds no waits and does not touch boot or the start of a turn. ## Risks - Low risk. A real answer that puts DSML sentinels or a `<function_calls>`/`<tool_call>` wrapper with a call body outside a code fence is now treated as an unexecuted call. The correction then asks once for a proper answer. - If the model leaks markup again after the correction, the existing empty-response failure path applies. - No schema, API, or saved run-state change. ## Model Used - Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
a090a6a932 |
fix(runner): stop a model from resending the same rejected tool call (#21360)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents act through Paperclip tools such as `call_api`, and the tool
layer refuses an invalid call before it sends anything
> - Some model hosts drop optional tool arguments. A `PATCH` call then
arrives without `body` every time, even when the model says it added it
> - The refusal already names the missing argument, but nothing stops
the model from resending the same call. In one recorded run the agent
sent the same rejected call 12 times in 80 seconds and never finished
the task
> - This pull request makes the tool layer recognise a repeated
identical validation failure, correct it once with the exact arguments
that arrived, and then stop running that exact call for the rest of the
run
> - The benefit is that a dropped argument ends in a clear correction or
a reported blocker, not a long loop that burns time and tokens
## Linked Issues or Issue Description
**What happened?**
An agent created a task and then tried to update it with `call_api`
`PATCH /api/issues/{id}`. Every call arrived with `operationId`,
`pathParams`, `query` and `responseText`, but no `body`. The tool layer
refused each call with "requires body" and showed the shape to resend.
The model's reasoning said that the body was missing, but each new call
arrived without `body` again. The agent sent the same call 12 times and
was still running when the test ended.
**Expected behavior**
After the same call fails twice in the same way, the agent should get
one clear correction that says what arrived and what is missing.
Paperclip should then refuse to run that exact call again, so the agent
changes the call or reports the blocker.
**Paperclip version or commit**
`master` at
|
||
|
|
5ede64f247 |
fix(recovery): settle an older run's saved-work hold, fold a waiting Retry into the isolated continuation, and plain recovery wording (#21348)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run is interrupted, a saved-work hold protects the files it
left. The fresh-continuation fallback then continues the task in a new
run, or asks a person to press Retry.
> - Staging found that an older run's saved-work hold made the dispatch
gate cancel every new run of the task. The hold was never settled, so
the task could not run again, and no person could act on the reason.
> - Recovery notices also showed internal codes and words such as
"board", "inspect", "repair" and "no live execution path". People cannot
act on these.
> - A person's Retry pressed while an isolated continuation was waiting
for delivery was saved, and then ran a second time after the
continuation.
> - This pull request settles an older run's hold through the fallback's
existing machinery, uses plain wording in every remaining recovery
notice (server and UI together), and folds a waiting Retry into the
isolated continuation that answers it.
> - The benefit is that a task held by an older run continues by itself,
or on one Retry, in a fresh isolated workspace. People read only what
happened and the one thing they can do, and a Retry never starts a
redundant second run.
## Linked Issues or Issue Description
**What happened?**
1. An earlier run's saved-work hold (for example "Automatic recovery of
saved work failed … Retry to try again") made the dispatch gate cancel
every new run of the task with `execution_reconciliation_required`. The
fresh-continuation fallback only took holds of the task's latest run.
Each refused run looked like a newer run, so the hold was never settled.
2. Recovery notices and recovery-task comments showed internal codes,
raw IDs, "board", "inspect", "repair" and "no live execution path". The
hold for remote files that could not be restored asked a person to
"verify its exact stop receipt" and to "record workspaceRepairEvidence".
3. On a projectless task whose old legacy writer might still be alive, a
person's Retry pressed after the isolated continuation was recorded, but
before it was delivered, stayed deferred. When the continuation run
ended, task release promoted that Retry into a second, sequential run.
Concurrent writers were already impossible on this path, because the
Retry waited behind the continuation. The second run was redundant: the
continuation was the answer to the person's Retry.
**Expected behavior**
An older run's hold does not block new runs indefinitely. The task
continues in an isolated workspace, by itself while the interruption is
recent, or on a person's Retry. Notices say what happened and the one
thing to do. A person's Retry answered by the continuation never starts
another run.
**Steps to reproduce**
1. Leave a hold from an older run whose saved-work recovery gave up,
then start a new run of the task. Before this change, the run is
cancelled at dispatch every time and the hold stays.
2. Open a task whose run was cut off and its retries exhausted. Before
this change, the notice reads "No live execution path … visible for
intervention".
3. On a projectless task with an isolated continuation recorded, press
Retry before delivery, then let the continuation end. Before this
change, a second run follows it.
## What Changed
- Older-run holds (`fresh-continuation-fallback.ts`,
`modules/run-dispatch/adapters/postgres.ts`):
- A run that was cancelled at dispatch (status `cancelled`, never
started, `execution_reconciliation_required`) never ran and follows no
run. The held-run listing and the fallback's successor check now ignore
it.
- The older run's hold goes through the fallback's existing machinery. A
saved-work recovery that is still in progress keeps its bounded wait.
Holds that a person owns stay. Past the bound, the hold is settled and
the task continues in a new isolated workspace. Held files are kept,
recorded as `kept_not_applied`, and never applied.
- One recency rule, `interruptionEligibility` (the #21222 rule), judges
recency by the newest run that the hold refused at dispatch. It no
longer uses the end time of the hold's own run. If the last attempt is
older than 24 hours, no automatic continuation starts. A person's Retry
always continues the task.
- After the dispatch gate refuses a run for another run's hold, it hands
that hold to the fallback in the same transaction. The normal dispatch
path is not changed.
- A person's Retry and the isolated continuation (`heartbeat/queue.ts`,
`execution-recovery-resolution.ts`, new
`fresh-continuation-retry-fold.ts`):
- The wake-admission transaction that inserts an isolated continuation
run (context `freshContinuationIsolation`) also coalesces the person's
waiting `task_retry` wake for that task, under the task row lock. This
applies only to a Retry with no message attached. The wake gets `status:
coalesced` and `runId` set to the continuation. A server stop after the
commit cannot leave the Retry unfolded.
- The delivery pass is idempotent about this. An action whose own
continuation is already queued is no longer hidden from the pass. When
the pass finds that continuation, it marks the action delivered and
folds any Retry that is still waiting.
- Before this change, the fold covered only a Retry that was already
waiting when the continuation was recorded. A Retry that carries a
person's message is not folded. Only a Retry whose source run is the
continuation's own source run is folded, and only while the continuation
is queued or running. A Retry for the continuation's own failure gets
its own run.
- Plain wording (server and UI together):
- "No live execution path" is now "Task could not continue". One seed
(`buildNoRunningExecutionNoticeSeed`) builds it for every producer.
- Exhausted saved-work recovery: "Paperclip could not recover the
agent's saved files … Press Retry to try again".
- The non-retryable continuation notice no longer shows the error code.
Recovery-task comments no longer show `stranded_issue_recovery`, raw IDs
or retry reason codes.
- `legacy-execution-recovery.ts` uses plain text for remote files that
could not be restored.
- `IssueRecoveryActionCard`: "Board decision required" is now "Waiting
for you".
- The UI still recognizes notices saved with the old text
(`adjacent-errors.ts`, `system-notice-humanizer.ts`).
- The precise codes stay in the evidence and in the notice details.
- Merge with current master: `recovery/service.ts` keeps master's
three-way choice (retry exhausted / continuation refused /
non-retryable). The non-retryable branch now uses the plain notice
without the code.
- Code comments that the original commits added are removed
(CONTRIBUTING).
## Verification
Red-to-green tests:
- `server/src/services/fresh-continuation-fallback.test.ts`:
- "continues a task held by an older run's saved-work hold in a new
isolated workspace…"
- "keeps the wait of a saved-work recovery that is still advancing…"
- "keeps a hold a person owns blocking…"
- "leaves an old hold for a person's Retry, and that Retry runs
isolated" (refused run and waiting Retry)
- "continues a recent restart-stopped run whose prepared remote
workspace could not be restored…" (red on the wording)
- "leaves an older restart-stopped remote restore failure for a person's
Retry, which then runs isolated"
- Same file, "consumes a person's Retry exactly once when the server
stops right after admitting the isolated continuation": the test runs
the admission and nothing after it. The Retry is already coalesced. The
test then resets it to unfolded, as an older release would leave it, and
runs the next delivery pass. The result is exactly one run and the Retry
consumed. Red without the in-transaction fold, and red without the
delivery-pass fold.
- Same file, "never folds a Retry for the admitted continuation's own
failure into that ended continuation": the continuation ends (real
`cancelRun`), then the person presses Retry on it. The Retry gets its
own run. Red without the source-run and liveness guard.
- Same file, "runs a person's Retry pressed before delivery as the
recorded isolated continuation, never in the shared directory": a
projectless task, an old legacy writer that may still be alive, and an
isolated continuation pending. Delivery starts exactly one run, which
carries `freshContinuationIsolation` in a `fresh_isolated_workspace`.
Extended here: after the continuation run ends (real `cancelRun` release
path, then `resumeQueuedRuns` and delivery), no run comes from the
person's Retry. The Retry wake stays `coalesced` into the continuation.
Red without the delivery fold (the Retry became a second run), green
with it.
Local runs on this branch (based on master
|
||
|
|
0ec155ac31 |
fix(recovery): enqueue a fresh interruption's continuation at once and record why a folded wake did not start (#21354)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a run stops before it finishes, for example when a server swap reaches a sandbox late, the fresh-continuation fallback records one continuation of the task from its saved work > - The execution-continuation worker turns that record into a queued run. It took one record per agent per pass, and it took the oldest held record first > - After a large swap, the backlog of old records is hours deep. A record for a run that ended a few minutes ago waited behind the whole backlog. The record also stayed outside the queue while the agent's queue was deeper than its concurrency limit > - Meanwhile the task showed "todo" with nothing running, and the assigned-work sweep's wakes for it were folded into the pending record without any trace > - This pull request queues a fresh interruption's continuation on the next pass, keeps a share of every pass for the old backlog, and records a skipped wake row when a system wake is folded into a pending continuation > - The benefit is that an interrupted task gets its next run promptly, it waits in the agent's queue where the dispatcher ranks it, and nothing is lost or invisible ## Linked Issues or Issue Description Refs #21324, #21257, #21234 (earlier pacing and dispatch ranking that this builds on). **What happened?** On a hosted staging instance, five tasks had runs that ended after a server swap ("The runner restored the conversation after process loss, but the previous turn is no longer active."). The fallback recorded a continuation for each task 3 to 6 minutes after the run ended. Each record stayed `continuationDelivery: pending` for 30 to 110 minutes with no delivery attempt. During that time the worker delivered continuations whose source runs ended 4 to 7 hours earlier, one per agent per pass. The tasks read as failed and needing Retry. The sweep tried 3 to 4 times per task, and each wake returned `null` without a wake row. **Expected behavior** A recorded continuation for a recent interruption becomes a queued run within about one delivery pass. Old backlog continues to drain and is not starved. A wake that is not started records why. **Steps to reproduce** 1. Give an agent a queue that is deeper than its concurrency limit. 2. Record pending continuations for older interruptions that are held past the hold bound. 3. Record a continuation for a run that ended a few minutes ago. 4. Run `deliverReconciledExecutions`. Before this change the fresh record is not delivered. The oldest held record is delivered first, one per agent per pass. **Agent adapter(s) involved** Not adapter-specific (core bug). ## What Changed - `deliverReconciledExecutions` now sorts pending continuations into two classes by when the source run ended. "Fresh" means that the source run ended within `RECOVERY_BACKLOG_AGE_MS` (30 minutes), which is the same value the dispatcher uses to tell fresh recovery from backlog. - A fresh continuation is enqueued without the queue-depth room check. It is limited only by the per-agent limit (one fresh continuation per agent per pass) and the per-company cap per pass. Among fresh records, the newest interruption goes first. A record whose wake was refused moves behind records that were not attempted yet. - Backlog continuations keep their earlier rules. These are the queue-depth room check, the bypass after the 5-minute hold bound, oldest held first, and one per agent per pass. - Each pass runs three phases: fresh up to the company cap minus `RECOVERY_CONTINUATION_BACKLOG_SHARE_PER_PASS` (1), then backlog up to the cap, then fresh again up to the cap. When both classes wait, the backlog gets at least 1 of the 5 slots per company per pass. The per-company cap (5) does not change, so a swap that interrupts many runs does not enqueue them all at once. All admissions still go through the one `wake(...)` call. - In `enqueueWakeup`, a system wake that is folded into the task's pending fresh continuation now writes a `skipped` wake row with reason `issue_continuation_pending`. Before this change it returned `null` with no trace. - Tests in `recovery-wake-herd.test.ts`: - New: a fresh continuation is queued on the first pass while the agent's queue is deeper than its limit. - New: a fresh continuation is delivered ahead of an hours-old backlog for the same agent, and the backlog keeps draining. - New: many fresh continuations for one agent are enqueued one per pass, newest first, and none is left behind. - New: the backlog keeps a share of each pass while fresh continuations fill the company cap. - New: a refused fresh wake stays pending and is enqueued on the next pass. - New: a folded system wake leaves a skipped row. - Updated: existing tests that used a recent source run to mean "held because the queue is full" now use backlog-age sources, and assertions that depended on exactly one admission per agent per pass now check the record under test. ## Verification - Red before the fix: the 6 new tests failed. The 18 earlier tests still passed. - Green after the fix: `cd server && npx vitest run src/__tests__/recovery-wake-herd.test.ts` gives 24 passed. - `npx vitest run src/__tests__/execution-control-work.test.ts` gives 27 passed. `src/__tests__/issue-recovery-actions.test.ts` gives 112 passed. `src/services/fresh-continuation-fallback.test.ts` gives 62 passed. `src/services/workspace-recovery-fallback.test.ts` gives 19 passed. `src/services/native-runtime/native-safe-replacement.test.ts` gives 62 passed. - `cd server && npx tsc --noEmit -p tsconfig.json` passes. ## Risks - Behavior shift: an agent with a full queue now gets fresh recovery continuations queued at once. They wait in its queue, where the dispatcher ranking (person-initiated first, recovery share) decides when they start. They are no longer held outside the queue. The per-company cap per pass still limits the burst. - Old backlog drains at a similar pace as before. It always gets at least one slot per company per pass when it waits. - The skipped wake row adds one small row each time the sweep retries a task that has a pending continuation. The sweep spaces its retries, so the volume is low. - No migration. No API or UI contract change. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits, read-only staging API calls). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
e23549cd9c |
fix(presentation): a task with a continuation waiting to start shows as continuing, not as needing Retry (#21353)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run dies (for example during a server swap), the
fresh-continuation fallback records that Paperclip will continue the
task in a fresh run from saved work
> - The record says `continuationDelivery: "pending"` until a delivery
pass starts the new run. Delivery is paced, so a record can wait a long
time behind an older backlog
> - The task view showed such a task as "working" only for 15 minutes
after the record was made. After that it showed a failed attempt with a
Retry button, although the record still said Paperclip would continue
the task
> - Product rule: a task that the system has recorded it will continue
by itself never shows as failed with a Retry button
> - This pull request ties the "continuing" state to the record's own
delivery state instead of a timer
> - The benefit is that the task view stays truthful however long
delivery takes, and Retry comes back as soon as the system gives up
## Linked Issues or Issue Description
Refs #21264, #21324, #21297, #21320, #21200
**What happened?**
On hosted staging (build
|
||
|
|
386dbe76a1 |
fix(evals): grade the company-secret setup reply by its rendered link (#21350)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat catalog evals check that a chat request reaches the correct saved outcome > - The configure-company-secret case checks that chat sends the person to the organization secret entry and never asks for the key in chat > - The case found the settings route by splitting the reply's markdown into tokens, and it judged the reply's wording with regular expressions > - The same tokenizer failed the personal-secret case: a link followed by a colon lost its route (fixed in #21344) > - This pull request grades the company setup reply the same way: it finds the rendered link, clicks it, and requires the New secret entry to appear > - The benefit is that the case measures what a person can actually do, and the reply's wording is left to human review ## Linked Issues or Issue Description Refs #21344 **What happened?** `companySecretSetupURL` read routes from the reply's markdown with a regular expression. It removed only `.,;!?])}` from the end of each token, so a reply such as `Open [Secrets](/COM/company/settings/secrets): choose New secret` yielded the route `/COM/company/settings/secrets):` and failed, although the rendered link was correct. The same bug failed the personal-secret case. The reply was also gated on wording patterns. **Expected behavior** A reply passes when its rendered message has a visible link to this company's secrets settings, and a person who clicks it reaches the organization secret entry. **Steps to reproduce** Grade a company setup reply whose markdown link to the secrets settings is followed by `:`. The old `companySecretSetupURL` returns `undefined`, although `deploymentSecretGuidance` accepts the same reply. ## What Changed - Add `external-secret-setup-link.ts`: the shared rendered-link check. It finds a visible link in the reply's rendered message to this company's `/company/settings/secrets`. It clicks the link, requires the case's entry control to be visible, and then returns to the conversation. The personal-secret check (`external-personal-secret-link.ts`) now uses it, with no change in behavior. - Add `external-company-secret-link.ts`. The company entry is **New secret** on the company **Secrets** tab, which is selected if needed. The company link takes no query. - `external-company-secret.ts`: - The comment handoff no longer checks the reply against wording patterns. - The setup link is followed only when no question or confirmation card carries the handoff. - The proof is recorded as `company-secret-reply-link.json`, and the `honest-secure-human-handoff` grade uses it. The proof must belong to a reply of the initial run. - Card paths are unchanged. - No scenario YAML changed, so the TestOps manifest needs no entry. `chat-catalog-descriptors.json` lists the new source files. ## Verification - `external-company-secret-link.test.ts`: - An ungated unit test covers the link-target rules. - A Chromium test (gated by `PAPERCLIP_CHAT_EVAL_SECRET_ROUTE_CANARY=1`, like the other secret-route browser tests, and run locally) shows that these cases pass: a link followed by a colon; a company tab selected after the click; tabs that render after navigation. - These cases fail: no link (wording only); a plain-text path; wrong-company and foreign links; a link with a query; a hidden link; a page with no secret entry. - Red check in the same test: the old `companySecretSetupURL` returns `undefined` for the colon-terminated link, while its wording guard accepts the reply. - The company grade tests now require the link proof. They reject a missing proof, a proof bound to another comment, and a proof that did not reach the entry. - Personal and company secret tests, catalog consistency and contract tests: 350 passed. `pnpm test:e2e:runner:typecheck` passes. ## Risks - Low risk. The change is in eval code only. - A company setup reply without a clickable settings link (only menu directions) no longer passes the comment path. Question and confirmation card handoffs are unchanged. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
f0679ee0c7 |
fix(recovery): bound automatic attempts per task, one deadline per interruption, and batch the invariant report's reads (#21347)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - When a server swap or a failure interrupts an agent's task, recovery
continues the task automatically: the assigned-work sweep, the
scheduler's bounded retries and native session restart recovery.
> - Release-swap testing on staging found four gaps in that recovery.
The assigned-work report took 60–81 s. One lasting outage was retried by
two owners under two separate caps. A server with a fast clock could
take a session lease that another server still held. Each server
generation started its own 15-minute recovery wait, so the waits chained
for hours.
> - Each gap either slows recovery down or lets a task loop or stall
without any limit.
> - This pull request batches the report's reads, gives each task one
shared hourly cap for automatic attempts, uses the database clock to
judge a lease, and gives each interruption one deadline across server
generations.
> - The benefit is that recovery stays fast and bounded. A lasting
outage gets at most six automatic attempts an hour. A run whose provider
makes no progress ends after 15 minutes, and the fallback then continues
it in isolation.
## Linked Issues or Issue Description
**What happened?**
1. On staging, the assigned-work invariant report took 60–81 s with more
than 1,000 open agent tasks. For each task with nothing live or queued,
the report read the tree pause gate in sequence: one round trip, plus
one per ancestor while a pause hold existed. It also read the company's
agent list once per assignee.
2. A lasting repository catalog outage was retried by two owners. Each
sweep continuation started a fresh run. Each failure of that run got two
more bounded transient retries from the scheduler. One failure could
start up to 18 runs an hour.
3. `claimNativeSessionResumptions` compared the lease expiry with the
claiming server's own clock. A server whose clock ran fast could take a
lease that another server still held.
4. Each server generation's restart-recovery claim started its own
15-minute wait for the provider to recover. Swaps came every 10–14
minutes, so the waits chained for up to 12 generations and none of them
expired.
**Expected behavior**
The report reads pause holds in a fixed number of statements. One task
gets at most one hourly cap of automatic attempts across all owners. The
database clock judges a lease. An interruption with no provider progress
ends after one 15-minute deadline, whatever the number of server
generations.
**Steps to reproduce**
1. Seed 1,500 open tasks with 300 held roots and 400 queued wakes, then
run the invariant report. Before this change it used 2,279 statements.
2. Make the repository catalog unavailable for longer than an hour on an
assigned task. Count the runs.
3. Let a server whose clock is ten minutes fast claim a native session
resumption whose lease is still live.
4. Replace the server every 7 minutes while a native run's provider
makes no progress.
## What Changed
- `issue-tree-control.ts`: a new batched pause-hold read. For a set of
tasks it uses at most two statements: one to find any active hold, then
one recursive ancestor read. It has the same depth bound and cycle guard
as the per-issue gate. `assigned-work-sweep.ts` uses it, and reads the
org chart once per batch to judge each assignee.
- New `recovery/task-automatic-attempts.ts`: one per-task ledger of
automatic attempts in the last hour (sweep continuations plus the
scheduler's `transient_failure` retries). The sweep counts both for its
cap and its spacing (`assigned-work-sweep.ts`). The scheduler's
transient retry lane (`heartbeat/retries.ts`) reads the same ledger.
Once the sweep has continued a task in the hour, a transient retry that
would exceed the cap of six is reported as exhausted. If the sweep never
continued a task, the scheduler keeps its own chain bound.
- `native-finalization-reconciler.ts` (`claimNativeSessionResumptions`):
the compare-and-set also requires the other owner's lease to have lapsed
by `clock_timestamp()`.
- `native-restart-recovery.ts`: each restart-recovery claim records
`interruptionStartedAt` in its recovery-history entry. A later claim
keeps that value while the interrupted attempt shows no provider
progress, and resets it when there is progress. A claim made 15 minutes
or more after the interruption began, with no progress, ends the run as
no longer resumable. This uses the existing terminal path (run failed,
slots and task lock freed), so the fresh-continuation fallback continues
the task in isolation. The bound is
`NATIVE_INTERRUPTION_NO_PROGRESS_DEADLINE_MS` in
`fresh-continuation-fallback.ts`, set equal to the fallback's
exact-recovery cap. Coordinators saved by older releases have no start
time, so the first claim by this code starts the clock. The normal run
path is not changed.
- Review fixes:
- A graceful drain no longer counts as "no progress". The claim always
reads provider progress. The drain only decides whether the resume
spends an attempt, so a run that progressed before a deploy is never
ended by the deadline.
- The transient lane checks the hourly cap again inside the task-locked
transaction, after it reuses an existing successor and before it inserts
a new one. The sweep already reads the ledger under the same lock.
- A same-run resume claim that the database clock rejects (another
owner's lease is still live) no longer pushes that owner's
`nextAttemptAt` out.
- Code comments that the original commits added are removed
(CONTRIBUTING). Merged with current master (includes #21336):
`assigned-work-sweep.ts` keeps master's `BLOCK_ATTRIBUTION_WINDOW_MS`
import and adds the ledger import.
## Verification
Red-to-green tests, each red on the parent commit and green with the
fix:
- `server/src/__tests__/assigned-work-sweep-round-trips.test.ts` "reads
pause holds and assignees once per batch on 1,500 open tasks…": 1,500
tasks, 300 held roots plus 300 children, 400 queued wakes, 20 agents.
Red: 1,100 per-task gate reads and 2,279 statements. Green: 0 and 239.
Report verdict counts are asserted to be the same before and after.
- Same file, "the batched pause read gives the per-issue gate's answer:
deep chains, cycles, released holds, other companies": checks that the
batched read gives the same result as the per-issue gate.
- `server/src/__tests__/assigned-work-invariant.test.ts` "counts the
scheduler's transient retries of the same task against the one
continuation cap": red (repaired), green (deferred_cap).
- `server/src/services/heartbeat/retries.test.ts` "a lasting outage gets
at most the task's hourly cap of automatic attempts across the sweep and
the transient lane".
- `server/src/__tests__/native-session-resumption.test.ts` "judges
another owner's lease by the database clock, not a server clock that
runs fast".
-
`server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`:
"resumes a run that progressed for 20 minutes before a graceful drain
suspended it" (red before: the deadline ended it) and "still ends a run
with no provider progress at the deadline when a graceful drain
suspended it".
- `server/src/services/heartbeat/retries.test.ts` "re-checks the task's
hourly cap under the task lock when the sweep records an attempt while
the retry is being prepared" (red before: attempt seven was scheduled).
- `server/src/__tests__/native-session-resumption.test.ts`: the
clock-skew test also asserts that `nextAttemptAt` is not pushed out (red
before).
-
`server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`:
"ends a run whose generations show no progress at the deadline from the
first interruption, not per generation", "restarts the clock when a
generation's resume made provider progress", "starts the clock at the
first claim for a coordinator saved before the deadline existed".
Local runs on this branch (based on master
|
||
|
|
ff971149c2 |
fix(dispatch): bound how long a queued run waits and keep a person's interrupted Retry person-first (#21349)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The heartbeat queue decides which queued run an agent starts when a slot frees up > - Recent changes start person-initiated work first and give recovery runs one start in four > - A system continuation of a person's interrupted Retry ranked as old-backlog recovery, and the recovery share always went to fresher recovery runs > - With a steady stream of new work, that continuation never started, so the person's task sat queued for hours > - This pull request adds aging and a longest-waiting share, keeps a person's interrupted request person-first, and passes the turn on when a long-queued run cannot be claimed > - The benefit is that every ready queued run starts within a bounded number of starts, and a person's Retry is not lost behind system work ## Linked Issues or Issue Description **What happened?** On a hosted staging instance, one agent had 28 runs running and kept starting newer runs. The company had 67 queued runs, and 54 of them were older than one hour. A person pressed Retry on a task. The Retry run was interrupted by a server restart two minutes later. The system queued an automatic continuation (`issue_continuation_needed`) with `previousRunId` set to the interrupted Retry run. That continuation stayed queued for more than 2.5 hours. The task showed as "working". Other queued runs of the same person (some queued for more than 10 hours) also did not start. Root cause: - The dispatcher ranked the continuation as `recovery_backlog`, the last tier, because it was a system wake and its source run ended more than 30 minutes ago. The person-first check only looked at wake requests attached to the queued run itself. It did not look at the source run, which was the person's `task_retry`. - The 1-in-4 recovery share picked the first recovery run in sort order. Fresh recovery runs sort before backlog runs, and fresh recovery runs kept arriving, so the backlog run never got the share. - Nothing aged a queued run. A run in a lower tier or with a lower priority could wait forever. **Expected behavior** A system continuation of a person's Retry ranks as person-initiated. Every ready queued run starts within a bounded number of starts, even when fresh person and system work keeps arriving. A run that keeps failing to claim is reported and does not block the next-oldest run. **Steps to reproduce** 1. Give an agent one free slot and several queued runs. 2. Queue a person's `task_retry` run, start it, and interrupt it. 3. Queue a system continuation with `previousRunId` set to the interrupted run, and let its source age past 30 minutes. 4. Keep adding fresh system and person work. 5. See that the continuation never starts. **Paperclip version or commit** Staging Core |
||
|
|
2b69cdfcd6 |
fix(evals): grade the personal-secret setup reply by its rendered link (#21344)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat catalog evals check that a chat request reaches the correct saved outcome > - The personal-secret case checks that chat sends the person to their own secure value entry, and never asks for the value in chat > - The case judged the reply's wording with regular expressions, and it found the settings route by splitting the reply's markdown into tokens > - A valid reply failed because its link was followed by a colon, so the tokenizer read the route as `/COM/company/settings/secrets):` > - This pull request grades the rendered link instead: it finds the visible link, clicks it, and requires the person's own-value entry to appear > - The benefit is that the case measures what a person can actually do, and the reply's wording is left to human review ## Linked Issues or Issue Description **What happened?** `chat-features.runner-opencode.local.secrets.personal-secret` failed with "Actual reply must direct the supported own-value settings path for this definition". The retained screenshot and DOM snapshot show a correct reply. The turn rendered as two messages, and both linked **My secrets** to `/COM/company/settings/secrets` and said **Set my value**. The wording check accepted the reply. The link check returned nothing. It extracted routes from the reply's markdown with a regular expression and removed only `.,;!?])}` from the end of each token. In the final message the link was followed by a colon (`[My secrets](/COM/company/settings/secrets): choose`), so the token was `/COM/company/settings/secrets):`, which does not match the settings path. An earlier passing run had `,` after the link, which the check removed. **Expected behavior** A reply passes when its rendered message has a visible link that a person can click to reach their own value entry for the required definition. Wording is not graded by keywords. **Steps to reproduce** Grade a setup reply whose markdown link to the secrets settings is followed by `:`. The old `personalSecretSetupURL` returns `undefined` for the retained reply, although its wording check returns `true`. ## What Changed - Add `tests/runner-e2e/chat-catalog-domains/external-personal-secret-link.ts`. It finds a visible link in the reply's rendered message (`#comment-<id>`) whose target is this company's `/company/settings/secrets`. An optional `definition` query must name the required definition. Other origins, companies, paths, credentials, fragments and query keys are refused. It clicks the link and requires the person's own-value entry to be visible. A link that names the definition opens that definition's detail sheet, so the check requires **Set my value** in the sheet, which must show `PERSONAL_PUBLISHING_KEY`. Otherwise it selects **My secrets** if that tab is not active and requires a visible **Set value** button on the row bound to the required definition (`my-secret-<definitionId>`). It then returns to the conversation and records the proof as `personal-secret-reply-link.json`. - In `external-personal-secret.ts`, the setup reply is no longer checked against wording patterns. It must still be the agent's own comment from the bound run. The grade uses the recorded link proof, which must belong to a reply of the initial run (a turn can post several messages). The reply link is now followed before the human-entry phase. - The confirmation and question paths are unchanged. When a question or confirmation card already carries the handoff, the reply link is not checked, so an optional link cannot fail those paths. - No scenario YAML changed, so the TestOps manifest needs no entry. `chat-catalog-descriptors.json` lists the new source file for this case. ## Verification - `external-personal-secret-link.test.ts`: - An ungated unit test covers the link-target rules. - A Chromium test (gated like the other secret-route browser tests by `PAPERCLIP_CHAT_EVAL_SECRET_ROUTE_CANARY=1`, run locally) uses the retained two-message reply. These cases pass: the retained reply; a link carrying `?definition=` that opens the definition's detail sheet; a page where the **My secrets** tab is selected after the click. - These cases fail: no link (wording only); a plain-text path; a wrong-company link; a foreign link; a wrong-page link; a hidden link; a link that reaches another definition's row; a definition link whose sheet shows another key; a link that reaches no own-value entry. - Red check: the old `personalSecretSetupURL` returns `undefined` for the retained reply, while `personalSecretGuidance` returns `true`. - The personal-secret test files, catalog consistency and contract tests pass (210 tests). `pnpm test:e2e:runner:typecheck` passes. - A narrow paid run of the case on this branch is linked in a comment. ## Risks - Low risk. The change is in eval code only. - A reply that points to the settings page without a clickable link (for example, text menu directions only) no longer passes this path. That matches the stated contract: the person reaches their entry through a link in the reply. Question and confirmation cards are unchanged. - The company-secret case uses the same markdown tokenizer and could hit the same trailing-colon problem. It is not changed here. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
70be453c38 |
test(e2e): accept either order when a person's message and the fallback settle a legacy hold (#21275)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a run is cut off, either a person's message or the fresh-continuation fallback's disposition worker can settle the task's recovery hold, and the task continues. > - The e2e spec `legacy-failure-continuation` "message" case asserted an internal field that depends on which of the two settles the hold first. > - When the fallback's worker wins, it stamps the cut-off run as superseded. The spec then fails, even though the person sees the same outcome. > - This pull request makes the spec assert what a person sees, not which path won. > - The benefit is that the spec stops flaking for every PR and in the release source check. ## Linked Issues or Issue Description Refs #21135, #21250 **What happened?** `tests/e2e/legacy-failure-continuation.spec.ts` › "legacy startup hold: message reaches a new agent response" failed on #21250 at |
||
|
|
566843820c |
fix(recovery): a queued continuation is not shown blocked, recent temporary failures continue, and notices say what a person can do (#21336)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run is interrupted, recovery continues the task, shows the
person a notice, and sweeps for tasks that nothing will continue.
> - On staging, a task stayed "Blocked / needs attention" for 24 minutes
while its Retry's run was queued.
> - Two tasks escalated to the board after a sandbox resume timeout, a
temporary failure, and nobody continued them.
> - Recovery notices asked a "Board operator" to inspect evidence and
repair sandboxes. Recovery must work without that.
> - A children-completed system wake could start on a parent that was
already closed.
> - This pull request fixes those four gaps.
> - The benefit is that interrupted tasks continue on their own, and
notices tell a person only what happened and the one thing they can do.
## Linked Issues or Issue Description
Refs #21250, #21257 (merged), #21258, #21234
**What happened?**
1. A task whose continuation was queued was shown as blocked (one
staging task). Execution recovery blocked the task at 23:51:28, and
restart recovery's Retry queued the successor at 23:51:50 through
`admitExplicitNativeContinuation`. That path resolves the interruption's
holds, but it never lifted the block those holds owned. The run started
at 00:15, so the task was queued for 24 minutes and shown blocked the
whole time. This was not a race: the issue's `updatedAt` equals
`blockedTransitionAt`, so nothing wrote to the issue after the Retry.
2. Board-owned stranded holds after a temporary failure had no continuer
(two staging tasks). A person's Retry failed while resuming the saved
sandbox (`reusable_sandbox_resume_failed`, cause `transport_timeout`,
`providerWorkStarted=false`), and stranded recovery escalated the task
to the board. The sweep leaves a blocked task that has any active hold
to the hold's owner. The fresh-continuation fallback takes only
execution-reconciliation holds. So nobody continued these tasks.
3. Recovery notices addressed a "Board operator" and asked people to
inspect evidence and repair runtimes, sandboxes or workspaces.
4. An `issue_children_completed` system wake could start on a parent
that was already done or cancelled.
**Expected behavior**
A task with a queued continuation is not shown blocked. A recent
temporary failure is continued automatically. A notice says what
happened and the one thing a person can do. A system wake does not start
on a closed task.
**Steps to reproduce**
1. Fail a run so that execution recovery blocks the task, then press
Retry while the agent's queue is full. The task stays Blocked until the
run starts.
2. Make a saved-sandbox resume time out on a person's Retry. The task
escalates to the board and stays blocked.
## What Changed
- **Queued continuation is not shown blocked**
(`explicit-native-continuation.ts`,
`recovery/assigned-work-invariant.ts`,
`recovery/assigned-work-sweep.ts`):
- `admitExplicitNativeContinuation` (a person's Retry, restart
recovery's Retry, or a saved message) moves the task from Blocked back
to To do when the replaced interruption set the block. That means the
hold's own block receipt, or a block written at or after the source run
ended with no later decision. A declared unblock action or an open
blocker keeps the block.
- New invariant violation `blocked_with_live_run`: the task is blocked,
has a queued or running run, and is not waiting on anyone else. Under
the row lock, the sweep lifts that block without starting a run or
spending the cap. Its suspect pre-filter now lets blocked tasks with
live runs through.
- **Recent temporary failures are continued** (new
`recovery/stranded-hold-continuation.ts`,
`recovery/assigned-work-sweep.ts`, `fresh-continuation-fallback.ts`,
`recovery/stranded-notice.ts`):
- The assigned-work sweep is the single owner of board-owned
`stranded_assigned_issue` holds whose failed run ended within the last
24 hours with a restart-class code, or with a temporary setup failure
before provider work (for example a sandbox resume refused for a
timeout).
- Under the row lock it resolves the hold (outcome `restored`, owner
system), restores the task's status from before the block, and queues
one continuation under the usual per-task cap and spacing.
- Older escalations and lasting causes (access denied, lost sandbox,
unknown) stay with their owner. The fallback explicitly never takes a
`stranded_assigned_issue` hold.
- **Plain recovery notices** (`recovery/service.ts`,
`recovery/stranded-notice.ts`, `recovery/successful-run-handoff.ts`):
see the wording table below.
- **Closed parent** (`heartbeat/queue.ts`): `issue_children_completed`
is added to the claim-time cancel for system continuations on a done or
cancelled task. A person's saved message still runs, and so does
onboarding's completion report (`onboardingCompletion: true`), which is
meant to run on a done parent.
- **Review fixes** (Greptile, first round):
- A person's block is never lifted by a Retry or saved message. The
block-time inference applies only when no hold carries a block receipt,
and a block whose activity entry was written by a person is kept. This
is the same signal the sweep uses (`attributeBlock`, so `blockedBy:
"user"` is a wait).
- The sweep's suspect query lets a blocked task with a live run through
the queued/claimed-wake filter, so a continuation queued by
`enqueueWakeup` (a queued wake bound to its run) reaches the
`blocked_with_live_run` repair.
- The saved-sandbox and workspace-setup timeout notices promise an
automatic retry only when the sweep will continue that hold. Both use
one decision (`strandedRunCauseIsTemporary`, inside
`strandedHoldContinuesAutomatically`). Otherwise the notice says to
press Retry.
### Old → new wording
| Where | Old | New |
|---|---|---|
| service.ts hold nextAction, successful run with missing state | Board
operator: inspect the run evidence, then explicitly choose a valid issue
disposition... | The agent finished without saying whether the task is
done. Mark the task done if it is, or press Retry to let the agent
continue from the saved work. |
| service.ts, process_lost / default | Board operator: inspect the retry
history / inspect the evidence, repair the runtime if appropriate... |
This attempt could not be finished. Press Retry to start it again from
the saved work. (`RETRY_FROM_SAVED_WORK_NEXT_ACTION`) |
| service.ts, provider_quota | Wait for provider quota recovery, then
retry the original assignee; do not wake a takeover owner. | The AI
provider's usage limit was reached. The same agent continues the task
once the limit resets. |
| service.ts, environment provider | No agent work started. Retry the
task once the environment provider is available. | The sandbox service
was unavailable, so no agent work started. Press Retry once it is
available again. |
| service.ts, inactivity | Board operator: inspect the inactivity
evidence... | The agent stopped responding, so this attempt was ended.
Press Retry to start it again from the saved work. |
| service.ts, worktree branch incoherence | Board operator: repair the
source task git worktree branch incoherence... | The task's workspace no
longer matches its branch, so the agent did not start. Press Retry to
start it again. |
| service.ts, base materialization | Board operator: repair the project
workspace repository URL or clone access... | Paperclip could not copy
the project's repository. Restore access to the repository in the
project settings, then press Retry. |
| service.ts, workspace link | Board operator: repair the source task
workspace link, project workspace cwd, or git checkout... | The task's
workspace could not be used, so the agent did not start. Press Retry to
start it again. |
| service.ts, base ref | Check the starting branch and repository
access, then repair the task's branch... | The starting branch for this
task's workspace was not found in the repository. Set a starting branch
that exists, then press Retry. |
| service.ts, sandbox plugin | Board operator: the sandbox provider
plugin named in the run failure is not ready; ... explicitly retry... |
The plugin that starts this agent's sandbox is not running. To fix it,
<remedy>, then press Retry. |
| service.ts, missing secret | Board operator: bind the missing
secret(s) named in the run failure... | A secret this agent needs is not
set. Add it in the agent's settings, then press Retry. |
| service.ts, review participant | Board operator: repair the failed
review participant path... | The reviewer did not finish the review.
Choose a reviewer, or press Retry to ask the same reviewer again. |
| service.ts failure summary | Latest retry failure details were
withheld from the issue thread; inspect the linked run for evidence. |
The failure details are kept with the run. |
| service.ts nested recovery comment | the assigned recovery owner or
board operator should fix the runtime/adapter problem... | press Retry
on the original task to start it again from the saved work, then close
this task. |
| service.ts recovery-issue stop comment | the current recovery owner
should inspect the failed run evidence, restore a live execution path...
| press Retry to start this task again from the saved work. |
| service.ts deliberate-wait exhaustion hold and comment | Inspect the
evidence and choose whether to repair, retry... / Next action: repair
the liveness disposition or request an explicit source-owner decision. |
The agent stopped without saying what happens next with this task. Mark
the task done if it is, or press Retry... / Next action: mark the task
done if it is, or press Retry... |
| stranded-notice REMOTE_WORKSPACE_RESTORE_NEXT_ACTION | Open the failed
run... inspect and move it aside, then retry | The saved work could not
be put back into the sandbox, so no agent work started. The saved
version was kept. Press Retry to start again from it. |
| stranded-notice scan notices (timeout, plugin worker, saturated,
output limit, failed, cancelled) | Check repository access and server
load / Check the worker status / Inspect the failed run... | "<what
happened>, so no agent work started. Press Retry ..." |
| stranded-notice workspace validation seed | ...Moving it to `blocked`
so the workspace link, cwd, or git checkout can be repaired | The task's
workspace could not be used, so the agent did not start. Press Retry to
start it again. |
| stranded-notice base ref seed | ...blocked until its workspace
configuration or repository access is repaired / Check that the
configured base ref exists... | starting branch not found / Set a
starting branch that exists in the repository, then press Retry. |
| stranded-notice stale shared holder | Check whether the other run is
still working and stop it if it is stuck... | Stop the other run on this
task, or choose a separate workspace for this task, then press Retry. |
| stranded-notice secret default | required secret/env bindings are
missing. Moving it to `blocked` so an operator can bind... | The agent
did not start because a secret it needs is not set. Add the secret in
the agent's settings, then press Retry. |
| stranded-notice review seeds (2) | ...so the board can inspect the
evidence, repair the reviewer runtime... | The reviewer ... did not
finish / cannot take work. Choose a reviewer, or press Retry ... |
| stranded-notice default body | could not restore a live execution
path... visible for intervention | This attempt could not be finished
automatically. Press Retry to start it again from the saved work. |
| stranded-notice metadata | Board decision required / "Inspect the
evidence, then retry ... repair the execution path ..." / "The recovery
owner should either restore a live execution path..." | Waiting for you
/ `RETRY_FROM_SAVED_WORK_NEXT_ACTION` / The assigned agent continues the
task from the saved work |
| successful-run-handoff.ts | Board decision required / inspect the
evidence, then retry the original owner... | Waiting for you / Mark the
task done if it is, or press Retry to let the agent continue from the
saved work |
| stranded-notice, saved sandbox resume timeout and workspace setup
timeout | asked to inspect the run or the provider | No agent work
started; Paperclip tries again automatically. |
Kept on purpose:
- `buildImmediateExecutionPathRecoveryNoticeSeed` ("No live execution
path" title, "retried continuation ... visible for intervention" body).
The UI recognizes this exact text
(`ui/src/components/task-chat/adjacent-errors.ts` regex and title,
`ui/src/lib/system-notice-humanizer.ts`) and humanizes it, so changing
it needs a UI change in the same PR.
- The UI's own "Board decision required"
(`IssueRecoveryActionCard.tsx`).
- Recovery-issue comments that still contain internal codes
(`stranded_issue_recovery`).
- The agent-facing disposition instruction in
`successful-run-handoff.ts`.
### Deliberately changed test expectations
Each of these asserted the old wording or the old behavior:
- `server/src/services/recovery/stranded-notice.test.ts`: stale shared
holder; base ref body and next action; scan timeout metadata; plugin
worker metadata; secret default body; owner row is now "Waiting for
you".
- `server/src/__tests__/heartbeat-process-recovery.test.ts`: owner row
is now "Waiting for you" (3 places); failure-summary line;
board-escalation nextAction expectations (review participant,
process_lost, missing disposition, deliberate wait, default); sandbox
plugin; workspace link and validation comments; missing secret action
and comment; review participant comment.
- `server/src/__tests__/assigned-work-invariant.test.ts`: the old "left
to the hold owner" test now uses a 30-hour-old run. A recent temporary
failure is now continued, which is the new behavior.
- `server/src/services/heartbeat/queue.test.ts`:
`issue_children_completed` is now in the claim-time cancel list.
- `server/src/services/heartbeat/run-completion.test.ts`: the
restore-failure next action now says "The saved version was kept".
- `server/src/__tests__/issue-recovery-actions.test.ts` and
`server/src/__tests__/heartbeat-workspace-branch-containment.test.ts`:
the branch-incoherence next action now says the "workspace no longer
matches its branch".
- `server/src/services/recovery/stranded-notice.test.ts`: the timeout
notice is checked for both an automatic and a manual retry.
## Verification
Red-to-green tests (each failed before its fix and passes after it):
- `task-retry.test.ts` "lifts the failure's block once the Retry's run
is queued..."; invariant unit test and sweep integration test for
`blocked_with_live_run`.
- sweep integration test "continues a recent board escalation of a
temporary setup failure...". A recent one is continued; a 30-hour-old
one and `provider_access_denied` are left with their owner. Plus a
`stranded-notice.test.ts` case.
- the wording expectations above.
- `heartbeat/queue.test.ts`.
Review-fix tests, each red before its fix:
- `task-retry.test.ts` "lifts the failure's block...": new `newer` case
(the hold's receipt names an older status version) and `person` case (a
person's activity entry set the block). Both stay blocked.
- `__tests__/assigned-work-invariant.test.ts` "a recovery block whose
task already has a queued run...": now seeds a queued wake bound to the
queued run, as `enqueueWakeup` does, and asserts `listSuspects` returns
the task.
- `heartbeat-dependency-scheduling.test.ts` "dispatches only a verified
completed onboarding report" (it failed in CI on the first push) passes
again.
Evidence:
- Item 1: the staging task's activity log. The issue's `updatedAt`
equals `blockedTransitionAt`; nothing wrote after the Retry.
- Item 2: the two staging runs ended at 12:28Z and 14:23Z on Nov 12,
inside the 24-hour window.
- Ruled out, so there is no change for it: a third staging task is
waiting on a real person's gate. A pending `request_confirmation` was
created before the fallback run failed.
Local runs, on master after #21257 (
|
||
|
|
cb94108484 |
fix(native): finish the run from a provider turn a timed-out start left running (#21338)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A native Codex run talks to a remote Runner through a controller. A
deploy or a same-run retry can reattach a new controller to the live
Runner.
> - A slow turn start can make the first controller give up with
"runnerd did not report the provider turn identity". Codex still runs
that turn.
> - The retry reattaches the Runner and adopts that turn, because the
Runner accepted it for this run's latest turn.start.
> - If the adopted turn finishes while recovery still runs, the saved
checkpoint has no active turn. The runtime then tries to start a fresh
turn, and the run fails with "session cannot start another turn". The
Runner is lost and the task needs a Retry.
> - This pull request makes the runtime treat a terminal that the
recovered session newly settled as the adopted provider terminal when
the saved checkpoint bound no turn.
> - The benefit is that a turn which a timed-out start left running
finishes the same run on the reattached Runner, with its session
context, and with no second turn/start.
## Linked Issues or Issue Description
**What happened?**
On staging, Codex runs on a remote Runner lost the Runner after a
same-run reattachment. Each chain started with a turn.start that timed
out ("runnerd did not report the provider turn identity") while Codex
ran the turn anyway. After the reattachment, the run failed with a
protocol integrity fault (thread_binding_mismatch,
tool_binding_mismatch, or native_event_replay_conflict "Provider
authoritative notification item/started did not name its execution
owner"). The task then fell back to a fresh session and lost its
context. A local reproduction with the real runnerd and the fake Codex
app server shows the same trigger also fails when the abandoned turn
finishes during the reattachment: the runtime tries a new turn and fails
with "session cannot start another turn".
**Expected behavior**
A turn that a timed-out turn.start left running belongs to the same run.
The reattached run adopts it, binds its late notifications, and finishes
from its outcome. Recovery does not kill the Runner and does not need a
manual Retry. A real cross-run or cross-task binding violation still
fails closed.
**Steps to reproduce**
1. Start a native Codex run on a remote Runner whose provider is slow to
emit turn/started after it accepts turn/start.
2. Let the controller give up with "runnerd did not report the provider
turn identity" while the Runner stays alive.
3. Let the same-run retry reattach the Runner.
4. Let the provider turn start and finish after the reattachment.
5. See the run fail instead of finishing from the adopted turn.
**Paperclip version or commit**
master at
|
||
|
|
d16becac04 |
test(evals): fix cohort-10 secrets entry, steering graders, private-task settle and queue brief (#21296)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat eval catalog checks real chat journeys against saved
product state, not reply wording
> - Full cohort 10 failed six cases where the product worked but the
eval was out of date or raced
> - #21225 redesigned Settings → Secrets, and #21243 changed when a
queued comment counts as delivered
> - The private secret steps, two steering graders, one final run check
and one fixture did not follow
> - This pull request updates those eval layers and keeps each one able
to fail a broken product
> - The benefit is that these six cases measure the product again
instead of the stale harness
## Linked Issues or Issue Description
Refs #21225
Refs #21243
**What happened?**
In full cohort 10 (OpenCode, local), six cases failed for eval reasons:
- `secrets.configure-company-secret` and `secrets.personal-secret`: the
private browser step waited for the old "Create secret" dialog and its
Provider field, and for the old personal form text. #21225 renamed the
dialog to "New secret", removed Provider, moved Key under "More
options", and moved the definition name into the "Set your value for …"
heading.
- `steering.steer-active` and `steering.discard-queued-message`:
Morgan's running turn read the queued comments through task history and
cited them in its accepted result. Under #21243 that is delivery, so no
continuation ran. The graders still required one more run and timed out.
- `tasks.private-task`: the final run check read the ledger while the
CEO's `chat_task_completed` follow-up was still running.
- `steering.queue-follow-up`: the brief was only a reference string, so
the continuation asked a question instead of building.
**Expected behavior**
Each case grades the product's current contract on saved state, and
still fails a broken product.
**Steps to reproduce**
Run the six cases at `origin/master` with the OpenCode local profile.
See the cohort-10 evidence for each case.
## What Changed
- `external-private-browser-worker.ts`: the company step opens "New
secret", finds the dialog by its "Who provides the value" radiogroup,
selects Organization, opens "More options" for a requested key, and
picks Paperclip-encrypted storage only when the "Stored in" picker is
shown. The saved secret must now have provider `local_encrypted`. The
personal step binds the form by the heading "Set your value for Personal
publishing credential". Credential handling is unchanged.
`manual-private-*.ts` is unchanged.
- `external-company-secret-create-dialog.test.ts`: the canary page uses
the new dialog. It adds a several-vaults variant (valid), a
wrong-storage result and the retired "Create secret" form (both
rejected). Source anchors follow `Secrets.tsx`.
- `external-personal-secret-entry-dialog.test.ts`: checks that the
worker heading matches what `SetMyUserSecretDialog.tsx` renders, and
rejects another definition and the "Replace your value" form.
- New `runtime-cited-delivery.ts`: `owedAfterTurn` (queued comments the
turn did not read at the current revision and cite) and
`continuationCarriesOwed` (one continuation carries each owed comment
once, with its text and the earlier-read warning, and none that the turn
handled).
- `runtime-active-steering.ts` (oracle `steer-active.v9`): records the
queued comments (person and any CEO relay) and the worker's receipts.
Delivery passes if every queued comment was handled in turn with no
continuation, or one continuation carries exactly the owed ones. Run
count is 2 plus one when anything is owed. The website the worker
registered last must still render blue.
- `runtime-discard.ts` (oracle `runtime-discard.v7`, `minimumRunCount:
4`): the same rule for the font note and the later contact request. The
run that handled the contact request must save its labelled section. The
discarded request must never be cited. The gate no longer says "Leave
saved follow-ups for their ordinary continuation", because that
contradicts the #21243 contract.
- `internal-private-delegation.ts` (oracle `private-task.v3`):
`verifyResult` waits for the delegated work and its completion follow-up
to settle, in the pattern of `identity-plan-review.ts`. A failed or
cancelled run still stops the wait. A worker still running at the
deadline stays bounded active work.
- `runtime-queue.ts` (oracle `runtime-queue.v4`): the gate gives a
complete website brief (product, page, hero copy, features, pricing,
company details). It does not contain the word "contact", so the brief
does not complete the queued requirement.
- Oracle bumps for the secrets cases:
`company-secret-controller-baseline.v22`, `personal-secret.v21`.
`chat-catalog-descriptors.json` follows all bumps and source files. No
scenario YAML changed, so the TestOps manifest is unchanged.
## Why each new check still catches a broken product
- Secrets: the old check matched removed UI text, so it failed a working
product. The new check uses roles and labels, and still fails a missing
form, a failed POST, wrong saved metadata and storage outside Paperclip.
- Steering: the old check required a continuation, which #21243 removes
when the turn cites the comment. The new check still fails an uncited
read with no continuation (the lost-comment bug), a citation without a
current-revision read, a cited comment that was not applied (no blue
page or no saved label), a replay of a handled comment, and any extra
run.
- Private task: the old check read the ledger at a random point after
acceptance. The case still fails if the follow-up fails or is cancelled.
- Queue brief: the grader is unchanged. The brief test fails if the
brief leaks the contact requirement.
## Verification
- `npx vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/chat-catalog tests/runner-e2e/chat-catalog-domains`:
285 files, 3948 tests pass.
- Calibration uses the retained cohort-10 receipts. The steer-active
worker read and cited both queued comments. The discard worker read and
cited the font note and the later contact request. Both now pass.
Reworded-valid variants (target handled in turn, relay or font note owed
to one continuation) pass. The broken variants listed above fail.
- `PAPERCLIP_CHAT_EVAL_SECRET_ROUTE_CANARY=1` secrets canaries: 3 of 3
pass. RED: the old worker against the new dialog page fails
(`locator.fill` timeout).
- `pnpm test:e2e:runner:typecheck`: pass. `node --test
tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs`: 10 of 10.
`node tests/runner-e2e/source-sdk-public-inputs.mjs
|
||
|
|
43aabb7080 |
fix(runner): keep configured secret values out of OpenCode output, storage and model requests (#21330)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A person can configure a secret that Paperclip delivers to an agent's commands as an environment variable > - The value must stay out of what the agent runtime stores and what it sends to the model > - With the OpenCode runtime, a command that printed the variable put the value in OpenCode's tool output. OpenCode stored that output in its session database and sent it to the model. Paperclip's own run log already redacted it > - Paperclip did not tell the runner which configured values are secrets, so the runner could not protect them without also hiding ordinary settings > - This pull request sends secret names (never values) to the runner, filters exact secret values before OpenCode captures any tool output, redacts them from model requests, and tells the agent how to check a secret without printing it > - The benefit is that a configured secret stays out of model context and out of runtime storage, even when an agent prints it ## Linked Issues or Issue Description **What happened?** The `chat-features.runner-opencode.local.secrets.personal-secret` eval failed because the persisted-state scan found a configured secret value in OpenCode's session database. The person entered the value through the normal secret entry form. Nobody pasted it in chat. The agent ran a shell command that printed the environment variable it was given. Paperclip's run log redacted the value. OpenCode stored the raw output in its database and included it in the next model request. **Expected behavior** A configured secret must never reach model context or runtime persistence. Ordinary configured values must not be redacted. **Steps to reproduce** Configure a secret environment variable for an OpenCode agent. Have the agent run a command that prints the variable. Then search the OpenCode session data and the model requests for the value. ## What Changed - **Secret names cross to the runner.** `configuredEnvironmentProjection` takes the controller's secret key names and adds `PAPERCLIP_SECRET_ENV_KEYS`. This marker holds only the names of configured entries that came from secrets. It travels with the configured projection across the TypeScript and Rust runner boundaries (`configured-environment.ts`, `configured_environment.rs`). The server passes `secretKeys` in `runtime-execution.ts`. - **Command output is filtered before OpenCode reads it.** For a launch that has configured secrets, the runner sets OpenCode's `shell` to a small POSIX wrapper (`drivers/opencode/output-custody.ts`). The wrapper runs the shell OpenCode would use anyway. It sends stdout and stderr through an `awk` filter that replaces exact secret values. OpenCode never receives the raw bytes, so its streamed previews, its database and its model requests stay clean. The wrapper keeps the exit status. It does not wait for background processes. The values come from the environment that OpenCode already holds and are never written to disk. - **Other tool results are rewritten before they are saved.** A local OpenCode plugin rewrites completed non-shell results (file reads, searches, MCP results) in `tool.execute.after`. OpenCode runs that hook before it stores or returns the result. The wrapper refuses to run any command until the plugin has started for the current launch (per-launch nonce). So no command can write a secret to a file that a later read returns without filtering. - **Model requests are redacted as a second layer.** The OpenCode provider proxy removes exact secret values from the string values of each parsed model request. Field names and other literals are never changed. A quick byte scan for each value, raw and JSON-escaped, sends the original bytes unchanged when no value is present. - **The agent gets guidance.** Native task prompts list the configured secret names. They say to check a secret with `[ -n "$VAR" ]` and never to print, echo, log or write a secret value. - **No added startup delay.** The wrapper is content-addressed under the runtime directory and reused across launches, so it is not rewritten on every launch. OpenCode's plugin-dependency install is marked as already done, so loading the plugin does not add wait time to the first tool call. - **Only exact values are redacted.** Command output is filtered line by line, so each line of a multi-line value that has at least 4 characters is redacted too. If a configured secret has no line of 4 or more characters, the launch is refused with `opencode_output_custody_unavailable` instead of running unprotected. ## Verification - `packages/paperclip-runner/src/drivers/opencode/output-custody.test.ts` (10 tests): the wrapper filters stdout, stderr and a partial last line; keeps exit codes (7, and 143 for SIGTERM); filters each line of a multi-line value; refuses to run before plugin readiness or with an old nonce; returns at once when a background process keeps running; leaves ordinary values unchanged. The plugin rewrites nested results. Model-request redaction covers raw and JSON-escaped values and never changes field names or literals (a secret equal to `true` leaves `"stream": true` intact). Secrets the filter cannot match are refused. No value is written to disk. - `opencode-server-driver.test.ts`: one saved session launches three times. First with no secret marker (older controller), then with the marker, then with no marker again (rollback). The first launch is unchanged, the second applies custody, and the third launch still starts. This test fails without the driver change. - `configured-environment.test.ts`: the marker names only secret entries, crosses the runner spawn and OpenCode boundaries, reads as "no secrets" when absent, and is refused when malformed. Rust: `secret_name_marker_crosses_only_with_its_projection`. - `server/src/__tests__/configured-secret-guidance.test.ts`: guidance content, and no guidance when there are no secrets. - Real OpenCode 1.18.34 qualification (`PAPERCLIP_OPENCODE_QUALIFY=1 node --test scripts/qualify-opencode-runtime.test.mjs`). A fake model asks for a command that prints the secret and writes it to a file, then a read of that file. Unprotected baseline: the value is in the session database and in model requests (red). With custody: it is in neither, and the exit status is kept (green). - **Latency** (local, macOS, real OpenCode, 8 runs each): boot 629 ms vs 697 ms baseline; first model request after the prompt 678 ms vs 698 ms (no regression); first command 139 ms vs 191 ms; each later command +9 ms (36 vs 27 ms). Proxy cost per request when no secret is present (the normal case): 33 µs at 50 KiB, 164 µs at 300 KiB, 0.55 ms at 1 MiB, 2.3 ms at 4 MiB. When a secret is present: 91 µs, 0.48 ms, 1.6 ms and 6.8 ms. The scan runs only for launches that have configured secrets, and it adds nothing to OpenCode's own startup. - A narrow paid run of `chat-features.runner-opencode.local.secrets.personal-secret` on this branch is linked in a comment. ## Risks - **Previous-release story.** The marker is new and optional. A runner that receives no marker (older controller) launches exactly as before. An older runner drops the marker at its environment boundary. The change saves no new run or session state. OpenCode configuration is regenerated on every launch, so sessions saved before this change resume under custody (covered by the cutover test). Warm-process reuse keys include the marker, so a warm process from before the deploy is not reused for a run that has secrets. - **Commands go through the wrapper.** Output arrives line by line. Output written after the shell exits is dropped, which matches how OpenCode stops reading at exit. If `awk`, `mkfifo` or `rm` is missing, the launch is refused with `opencode_output_custody_unavailable` and does not run unprotected. - **Scope.** Only exact values are redacted. An agent that encodes a value (for example base64) is out of scope. The model-request layer applies when Paperclip routes the model through its proxy (OpenRouter and Paperclip connections). ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1113.0-canary.5 |
||
|
|
0986c3eb3a |
fix(native): let a same-run reattachment take over its own retained remote Runner cleanup (#21328)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs on a remote sandbox (Daytona) use a detached `paperclip-runnerd`. After a recoverable failure, the server resumes the same run and reattaches to that Runner > - #20923 keeps an unreachable remote Runner alive when the failed attempt closes, so that the same-run reattachment can adopt it > - The same close also leaves the session in the in-process cleanup quarantine. The quarantine refuses every new session of the task, and that includes the reattachment that the Runner was kept for > - Close retries cannot settle it. After the first close fails, the Runner transport returns the same rejected close for every retry > - On staging, this was the largest failure class: 14 Codex runs in one hour failed with `native_session_cleanup_quarantined`, while only 5 runs succeeded. Each task then needed Retry > - This pull request lets the same run's reattachment to that Runner, in the same remote sandbox, take over its own retained cleanup > - The benefit is that the task continues on its own, with no spent attempt, no failed run and no Retry ## Linked Issues or Issue Description Refs #20923, #21031 **What happened?** A remote native run (Codex on Daytona) failed a turn with a recoverable error, for example `runnerd did not report the provider turn identity` or `native_finalization_missing: session returned no semantic result`. The close then logged `runner did not prove durable suspension before checkpoint` and `remote runner left alive unsettled for same-run reattachment`. Each later same-run reattachment was refused at admission: - When the close error was retryable, admission waited out the grace and failed with `native_session_cleanup_quarantined: prior session cleanup remains incomplete`. That spent a same-run attempt. After the attempt budget, the run failed. All close retries replayed the same failed close within milliseconds (`runner transport close requested: native session quarantined cleanup recovery`), so they could never settle the quarantine. - When the close error was `native_session_close_unrecoverable` (operator-required), the next attempt failed at once with `native_session_cleanup_quarantined: the previous session is still shutting down; Paperclip finishes the cleanup automatically and the saved work is kept.` That code is a permanent failure. "Paperclip finishes the cleanup automatically" did not happen in the process. Only a server restart (a deploy) removed the in-memory quarantine, and by then the run had failed and the task waited for Retry. Staging restarts every 20 to 45 minutes, so attempts often end mid-cleanup and meet this gate. **Expected behavior** The same run's reattachment adopts the Runner that its earlier attempt kept for it. It does not wait for a close that cannot settle, and it does not spend an attempt. The task continues without Retry. Other runs of the task and other sandboxes stay gated. **Steps to reproduce** 1. Start a remote native run, and let its turn fail with a recoverable transport error while the Runner cannot prove suspension. 2. The close keeps the Runner alive and quarantines the session. 3. The server schedules the same-run reattachment 30 s later. 4. The reattachment fails with `native_session_cleanup_quarantined`, and the task needs Retry. **Paperclip version or commit** Deployed Core `c730872974` on staging. Reproduced on `master` at `49f0030f06`. **Deployment mode** Hosted cloud (staging), Daytona remote sandboxes. **Agent adapter(s) involved** `paperclip_runner` native runtime with the Codex app server backend. ## What Changed - `packages/paperclip-runner/src/native-session-runtime.ts`: `executeNativeSession` takes a new option `retainedRunnerReattachment`. With a remote cleanup scope, admission retires quarantine entries that belong to the same run in the same remote scope. Removal stops later close retries. A close attempt that is still in flight stays an admission owner until it settles, so the old and new controllers do not overlap. Other runs, other sandboxes and local backends keep the gate. - `server/src/services/native-runtime/native-session-executor.ts`: sets `retainedRunnerReattachment` only for a `reattach_remote_runner` recovery of the same run. - `doc/execution-semantics.md`: records the takeover. - Tests: - `packages/paperclip-runner/src/native-session-runtime.test.ts`: four new cases. An operator-required close with a retained Runner. A retryable close whose retries replay one failure. A close that a deploy interrupts while it is in flight (the reattachment waits for it, then takes over). Controls: another run, a run without the takeover option and a local backend stay refused. - `server/src/services/native-runtime/native-session-executor.test.ts`: asserts that the option is set for a same-run remote reattachment and is not set for a fresh sandbox run. ## Verification - Red on `master` (runtime source restored): 3 of the 4 new runtime cases fail with exactly the two staging errors (`the previous session is still shutting down…` and `prior session cleanup remains incomplete`). The control case passes. - Red on `master` (executor source restored): the 3 executor assertions fail. - Green with this change: - `cd packages/paperclip-runner && npx vitest run src/native-session-runtime.test.ts` (294 passed) - `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts -t "exact Stop cleanup|fresh sandbox|retirement"` (46 passed) - `cd server && npx vitest run src/__tests__/heartbeat-native-cleanup-admission-hold.test.ts src/services/quarantined-task-retry.test.ts src/__tests__/completed-turn-cleanup-hold-settle.test.ts` (61 passed) - `npx tsc --noEmit` in `packages/paperclip-runner` and `server` passes. - Staging evidence: runs `7b7ded99` and `27c487ef` show the sequence above in their events and logs. ## Risks - The takeover applies only when the server recovers the same run with `reattach_remote_runner` in the same remote cleanup scope. This is the case that #20923 keeps the Runner alive for. After a deploy the quarantine is already gone and the same reattachment runs today, so the takeover gives the in-process retry the same behavior as a restart. - A close attempt that is in flight is never overlapped. The reattachment waits for it within the existing admission grace. - The reattachment still verifies Runner ownership and its controller generation before it adopts the Runner. If the Runner is gone, the existing relaunch or reconciliation path handles it. - No persisted shape changes. Saved state from the previous release is not affected because the quarantine lives only in process memory. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
d1d71a7755 |
test(restart-chaos): staging twin cells, follow-up/queue/subtask grading, and full Daytona cleanup (#21325)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart-chaos harness (`tests/restart-chaos/`) proves that a release swap (two servers that overlap, then the old one stops) does not break running agent work > - The staging twin runs that swap with two local servers and real Daytona sandboxes, and it is the gate before any hosted proof > - A loaded twin wave (48 tasks, 20 runs plus 20 queued, 2 swaps) reported 31 per-task failures and 2 extra turns, but the product did the correct thing in almost all of them > - Three harness defects made the noise: the graders called a required follow-up run an extra turn, called a healthy queued run an orphan, and called a parent that waited for its subtasks a stall. Cleanup also left archived Daytona sandboxes in the shared org on every wave > - This pull request lands the twin harness work and fixes the three grader defects and the cleanup gap, each with a red/green harness test > - The benefit is a stop-rule grade that shows only real failures (here: 1 Retry banner) and makes the follow-up wait visible, and a wave that leaves no sandbox behind ## Linked Issues or Issue Description Refs #21175 and #21227 (the server's comment and child-result delivery rules; this harness takes their outcome from the server's wake record and does not re-check them). **What happened?** Wave `w-20261113T052703-s7301` (loaded twin L1, on Daytona) graded 31 tasks FAIL and listed 2 extra turns. The release-swap grade was 82 / 82 tasks completed without human action. - t007 and t010 (OpenCode): a person posted a follow-up comment during the first run (05:44:56 and 05:49:21). OpenCode takes no input mid-turn. The first run finished `done` at 05:53 and said "No board comments arrived". The deferred wake was promoted, and the second run answered the comment after 16 to 17 minutes in the 20-deep queue. The graders counted the second run as an extra turn and as a `completion_once` duplicate. - t021 to t048: each waited about 483 s queued for a slot in the deliberate 20-deep queue. U1 allows 5 minutes queued plus 3 minutes without an owner, so all 28 failed `no_owner_no_action`. - t001: a parent ended its turn `yielded` to wait for its two subtasks. The observer saw no run for 3 minutes, called it a stall, and rested it `in_progress`. The parent completed later through its `issue_children_completed` wake. R1, R3, R5 and U1 failed. - Cleanup: every Daytona wave left 1 to 8 archived `paperclip-create-*` setup sandboxes and reported `remainingSandboxes: []`. Daytona does not delete archived sandboxes. **Expected behavior** - A run that starts with a person's comment that no earlier run received is required. Only a run that starts with no undelivered input is an extra turn. - A run queued while every slot of its agent is taken is healthy for the declared queue time. A run queued while its agent has a free slot still fails. - A parent that waits for open subtasks is in a named wait, not a stall. - Cleanup deletes the sandboxes that this wave created and fails when one remains. **Steps to reproduce** 1. Run the L1 loaded twin (command in the README section "Staging twin cells"). 2. Read `staging-mix-summary.md` and `SUMMARY.md`. ## What Changed - Cherry-picked the 8 twin harness commits from `fastloop/staging-twin-loaded`: staging-mix under the release-swap driver on Daytona, `--server-from` on Daytona, probe agents verified while the first tasks run, image matched to the `--server-from` checkout, and the knobs `RESTART_CHAOS_K15_KILL_OLD_AFTER_SECONDS` / `_KILL_OLD_SWAPS`, `RESTART_CHAOS_K15_FREEZE_NEW_SECONDS` / `_FREEZE_SWAPS`, `RESTART_CHAOS_MIX_BANNER_REST_SECONDS` and `RESTART_CHAOS_MIX_FIRST_SLOT_SECONDS`. A misplaced doc block in `supervisor.ts` is moved back to its function. No code comments are added. - Cleanup (`daytona-sweep.ts`): after the environment delete, the harness deletes every sandbox that carries both of this wave's UUID labels (company and environment), archived ones included. `assertHarnessMayMutate` still refuses any sandbox without both labels and any `fleet-sbx-*` snapshot. The `paperclip-create-*` name refusal is removed, because every plugin sandbox (the fleet's included) has that name, so the labels are the real ownership proof. A sandbox left after deletion fails the wave (`undeletedSandboxes`, warm-shutdown verdict, `ttft-baseline`). - Follow-ups (`follow-up-delivery.ts`, `wake-evidence.ts`, `staging-mix-retry.ts`, `staging-mix.ts`): delivery comes from the server's own record. A comment that the server judged delivered never becomes a run. A run that the server started from a wake carrying a person's comment posted after the first run started (a retry included) is required, not an extra turn. `completion_once` keeps the first completion as the baseline and does not count a later completion by such a run (truth-table test). The grader does not re-check the server's delivery rule; the server's tests cover it. The summary adds `extra turns: N` and `follow-up answered in a later run: n, queued wait p50/max, posted to answered p50/max`. - Declared queue (`universal.ts`, `agent-slots.ts`, `wave-grade-inputs.ts`): with `--mix-queue` above 0, U1 allows ceil(queue / runs) × (longest turn + 5 min) queued (27 minutes for L1). Only queued time while the run's agent has a free slot counts against the 5-minute bound. Slot use comes from a new `wake-evidence/agent-slots.json` (run start/finish and each agent's `maxConcurrentRuns`). - Subtask wait (`child-wait.ts`, `observer.ts`, `grader.ts`): an idle parent after a `yielded` or `blocked` turn, until its `issue_children_completed` wake, is a named wait (`waitingOnChildren`) for R5 and U1. The observer no longer gives up on a parent while a subtask is open. An older record that rested during that wait takes the completion from the wake evidence. - Review follow-ups: the first staging-mix slot moves earlier without adding a swap; each sandbox delete pass has a deadline; a frozen incoming server is always resumed (`server-freeze.ts`); the first completion is always the baseline for `completion_once` (truth-table test); a follow-up run that failed before its turn keeps the answer owed for its retry; `agent-slots.json` saves the server's effective run cap (`maxConcurrentRunsSql`); the wave-end cleanup flow moved to `daytona-cleanup.ts` (`main.ts` stays under 1,000 lines). - README: one-command entries for the twin cell (G, G2, G3) and L1, the knobs, the stop-rule grading, the new summary lines, the U1 queue rule, the subtask wait and the cleanup rule. ## Verification - Red/green tests (red = new tests against the previous source): - Cleanup, `daytona-sweep.test.ts`: red 4 failed (for example "expected [] to deeply equal [ 'archived', 'archiving', 'started' ]": nothing was deleted). Green: pass. - Follow-ups, declared queue, subtask wait (`staging-mix-retry.test.ts`, `universal.test.ts`, `child-wait.test.ts`, `wake-evidence.test.ts`): red 6 failed (for example the t010 run listed as an extra turn, U1 fail for the t030 queue shape, t001 "rested_not_done"). Green: pass. The tests use the real timings of t001, t010 and t030. - Full harness suite: `npx vitest run --config tests/restart-chaos/vitest.config.ts`: 71 files, 811 tests pass. `npx tsc -p tests/restart-chaos/tsconfig.json --noEmit`: clean. - `pnpm verify` exits 2 (the harness README is a global CI input). Required CI supplies the broad coverage. - One-off offline analysis (a scratch script, not part of this PR): the saved wave `w-20261113T052703-s7301` regraded with these graders, from a copy of its results: | | Before | After | | --- | --- | --- | | Per-task FAIL | 31 (t001 R1, t007 and t010 `completion_once`, t021 to t048 U1) | 0 | | Extra turns | 2 (t007, t010) | 0 | | Retry banner shown | 1 (t020, 3 s, completed without a click) | 1 (t020) | | Follow-up answered in a later run | not shown | 4: t001, t007, t010, t012. Queued wait p50 983 s, max 1739 s. Posted to answered p50 1420 s, max 2255 s | t001 and t012 started their follow-up run before completion, so they were never extra turns. They are listed because the wait is real. This wave predates `agent-slots.json`, so the regrade used only the declared queue bound for U1. - No Daytona wave ran for this PR. There is no live proof here that the cleanup deletes archived sandboxes in the real org or that the new evidence file is written on a live wave. The delete path uses the same `DELETE /sandbox/<id>` call as the Daytona SDK's `deleteSandbox`. ## Risks - The harness now deletes Daytona sandboxes in an org that the Cloud fleet shares. The guard is the wave's two freshly minted UUID labels, filtered on the server and checked again on every row, plus the `fleet-sbx-*` refusal. A sandbox without both labels is never deleted. - When the server wrongly re-delivers a comment that a finished run already received, the harness counts that run as a required follow-up, not as an extra turn. The server's own delivery tests catch that class, not this grader. `completion_once` still fails when that run also reports completion and is not the first. - The declared queue bound is generous (27 minutes for L1). A stuck queue with full slots for longer than that still fails. A queue with a free slot fails after 5 minutes when slot evidence exists. Without slot evidence only the declared bound applies. - Test harness only. No product code changes. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
22c9300dac |
fix(recovery): a run that fails before the agent is asked anything continues by itself on a fresh computer (#21297)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run on a remote sandbox opens a provider session and then
waits at a launch gate before the first prompt goes to the provider
> - On remote OpenCode sandboxes the gate saves the provider launch
files; when that save fails, no prompt was ever sent
> - Today that failure ends as `native_session_interrupted` with
ambiguous recovery evidence, so the task goes blocked and the user must
press Retry
> - Nothing was sent to the provider, so nothing can be duplicated: the
system can safely continue by itself
> - This pull request types that failure, proves it from the Runner turn
ledger at the failure point, and continues the task on a fresh sandbox
in the existing interrupted-preparation lane
> - The benefit is that a setup failure before the first prompt no
longer needs a human, and the user sees plain words instead of internal
codes
## Linked Issues or Issue Description
Refs #21287
Refs #21304
**What happened?**
On a remote OpenCode sandbox, the provider launch-files gate failed
before the first prompt (for example
`native_provider_artifacts_transfer_timeout`). The run ended blocked
with a Retry banner. The run error showed internal codes and raw byte
counts to the user.
**Expected behavior**
A failure that happens before any prompt reaches the provider continues
automatically on a fresh sandbox. The task shows as working while the
continuation is pending. The visible text is plain and has no internal
codes or raw numbers.
**Steps to reproduce**
1. Use an OpenCode agent on a Daytona sandbox environment with
controller persistence.
2. Arm the crash point `native.provider_artifacts.before_capture` in
fail mode (or let the launch-file upload time out).
3. Start a task. The first run fails at the gate before any prompt.
4. On master the task goes blocked with Retry. With this change a
successor starts on a new sandbox and the task completes.
**Agent adapter(s) involved**
OpenCode (native runtime, remote sandbox).
## What Changed
- New typed cause `native_failed_before_provider_turn`
(`server/src/services/native-runtime/failed-before-provider-turn.ts`).
The executor raises it only when the launch gate fails, the run was not
cancelled, and the Runner turn ledger proves that no turn command was
issued. The proof is taken at the failure point, before the session
closes. In all other cases the original error is rethrown and today's
path stays.
- The executor quarantines the failed run's session root, so the
successor starts in a new root.
- `heartbeat/predecessor-retirement-fallback.ts` accepts the new cause:
it records the proof on `runnerProfileJson.failedBeforeProviderTurn`,
marks the run's reusable sandbox leases for replacement, and schedules
the successor in the existing `interrupted_preparation` lane (same cap
of 12 and the same backoff).
- `remote-workspace-unchanged-proof.ts` accepts the runner process,
runner command and pre-turn events only when that proof record is
present. It still refuses any turn evidence, an unsettled save by
another run on the same workspace, and setup writes. When the proof
refuses, the run's epoch is abandoned unapplied: the files are kept and
recorded, and the next run restores the last committed save into a fresh
root.
- When the proof refuses and the epoch is abandoned, the replacement
mark stays. The next acquisition replaces the sandbox. #21304's
retention hold makes `retireReplacedReusableSandboxLease` park the old
lease as `retained` instead of `pending_cleanup`, so the task continues
on a fresh sandbox and the sandbox that holds the abandoned files is
kept. When the proof holds (only the Runner's own session folder was
written), the old sandbox is replaced and cleaned up as usual.
- `task-interruption-presentation.ts` shows a pending
`interrupted_preparation` retry as `working` with no Retry until it is
due (plus the dispatch grace).
- The visible run error is now plain: "Setting up the computer took too
long, so the task is continuing on a fresh one." (for a setup timeout)
or "The computer could not be set up before the agent started, so the
task is continuing on a fresh one." The technical cause code stays in
`runnerProfileJson.failedBeforeProviderTurn.technicalCause`, the server
log and the run log.
- New crash point `native.provider_artifacts.before_capture` (fail mode
throws the typed transfer timeout at the gate).
`tests/runner-e2e/source-sdk-public-inputs.json` is refreshed for the
changed registry file.
- `tests/restart-chaos/main.ts`: `RESTART_CHAOS_EXTERNAL_CRASH_CONTROL`
lets an outside driver arm fail-mode crash points without a restart
fault.
## Verification
Local (targeted):
- `server/src/__tests__/heartbeat-native-predispatch-retry.test.ts`: 8/8
pass. New cases: fresh remote sandbox and fresh plugin-task sandbox
continuation (now also asserts the plain visible text with no
identifier-looking tokens, and the technical cause in the proof record),
the ambiguous hold when the ledger cannot prove no turn, and backoff
plus the lane cap.
- `server/src/services/remote-workspace-unchanged-release.test.ts`:
27/27 pass. New cases: hand-on when the ledger proves no turn; refusal
on a recorded turn, an unsettled predecessor save, and setup writes. Two
more cases follow the marked lease through the next acquisition's
replacement decision (`readReplaceBeforeReuseMarker` +
`reusableSandboxReplacementPlan` + `retireReplacedReusableSandboxLease`)
and then through `sweepPendingCleanupLeases`. "replaces the marked
sandbox when the failed run provably left its files unchanged": the old
lease goes to `pending_cleanup` and the sweep destroys it. "continues on
a fresh sandbox and parks the one that holds abandoned files, which
cleanup leaves alone": the old lease is `retained` with the
abandoned-files reason and is held by the receipt, the sweep does not
destroy it, and the file is still there.
- Red without the product change (product files reset to the base, tests
kept): 8 failed, 25 passed of 33. All 8 failures are the new cases. The
visible-text assertion alone was red on the previous commit: the error
read "The computer could not be prepared before the agent started
(native_provider_artifacts_transfer_timeout: the upload of 536870912
bytes did not finish within 120 s). The task continues on a fresh
computer."
- `task-interruption-presentation.test.ts`,
`native-provider-artifact-gate.test.ts`,
`crash-point-call-sites.test.ts`: 79/79 pass.
`packages/shared/src/crash-points.test.ts`: 16/16.
`tests/restart-chaos/crash-points.test.ts`: 12/12.
- `tsc --noEmit` clean for `server`, `packages/shared` and
`tests/restart-chaos`.
Daytona (local servers, Daytona sandboxes, image `sha256:787c3e9c…`,
these commits rebased onto
|
||
|
|
41998d551c |
feat(plugins): send the claimant's credential on plugin task resume (#21335)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Plugin environment drivers run agent tasks on remote computers, and a warm follow-up run resumes a parked Runner through the `resume` task operation > - `resume` carries only the durable identity and a bootstrap ticket, which is enough while the parked Runner process is still alive > - A provider sometimes has to start the Runner again to resume the session: after it moved the parked session to a new sandbox, or after the computer rebooted > - A Claude Runner cannot start without its credential, and today only the portable handoff path (`handoffCredential`) delivers one > - This pull request lets a provider opt in to receiving the claimant run's harness credential on `resume` > - The benefit is that a provider can continue a parked session on a new computer without Core orchestrating a handoff ## Linked Issues or Issue Description Refs #21289 Related: #21305 **What existing behavior does this improve?** Warm resume for plugin task providers that restart the Runner to resume a session. **Subsystem affected** Plugin SDK environment task schema, the server's plugin Runner task owner, and the plugin environment task docs. **Current behavior** `submit` can carry the harness credential (`credential`), but `resume` cannot. A provider that must start the Runner again on resume has no credential for a Claude task. **Proposed behavior** A provider that returns `resumeCredentialVersion: 1` from `describe` receives `credential` on `resume` whenever Core would send it on `submit`, resolved for the claimant run. Other providers see no change. **Reason and benefit** Providers can restart a Claude Runner on resume without the Core-run handoff protocol. **Breaking changes** None. The field is opt-in: the SDK worker parses operations strictly, so Core sends it only to providers that advertise the capability. ## What Changed - `resume` gains an optional `credential`, with the same schema and validation as `submit`'s. - `describe` gains an optional `resumeCredentialVersion: 1` capability. - The plugin Runner task owner adds the credential on `resume` when the provider advertised the capability and Core resolved a credential, using the same resolution as `submit` (`preparePluginRunnerCredentials`). After a warm rebind, the claimant run's credential is sent. - `doc/plugins/PLUGIN_ENVIRONMENT_TASKS.md` documents the field and its handling rules. - The runner-e2e SDK public-input pin is updated. `handoffCredential` on the session handoff path is unchanged. ## Verification - SDK vitest `tests/environment-task-release-resume.test.ts` and `tests/environment-tasks.test.ts` (68 passed): `resume` accepts and rejects the same 11 credential cases as `submit`; the capability is optional and rejects other versions; a resume credential reaches the plugin through the serialized worker RPC. - Server vitest `plugin-runner-task.test.ts` and `environment-task-runtime.test.ts` (151 passed): credential sent with the capability and a resolved credential; omitted without a credential or without the capability; the rebound claimant's credential is sent; the credential appears in no metadata write and not in the sanitized error, even when the worker error echoes it. - `native-session-executor.test.ts -t "uses a task-only"` (5 passed): the executor forwards the capability to the owner. - runner-e2e `source-sdk-evidence.test.ts` and `source-sdk-public-type.test.ts` (46 passed). - `tsc --noEmit` for `packages/plugins/sdk` and `server`. - Not run locally: the full suite and `pnpm verify`; required CI covers them. ## Risks - No server-side provider check on resume, unlike `submit`: the lease has no reliable harness at resume time. Protection relies on the owner, whose rebind refuses a changed harness, account or credential identity. - The credential is the claimant run's current one, which can be newer than the one that started the parked Runner. That is intended. - A provider must advertise the capability only after its SDK pin includes this schema; roll Core out first. - Rollback: revert this pull request. No migration. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, test runs), implemented by a sub-agent and reviewed before opening. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
e063587edd |
fix(recovery): repair stale pending recovery states and tell the fresh run what the cut-off run already did (#21257)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The assigned-work invariant and its periodic sweep find open tasks that nothing will continue, and continue them once. > - The fresh-continuation fallback continues a cut-off run in a new run, with an opening note about what the cut-off run did. > - In release-swap waves, the invariant believed some "pending" states forever, so stalled tasks were reported OK and never repaired. > - The opening note did not say what the cut-off run had already created or read, so fresh runs created subtasks again and posted DONE twice. > - This pull request bounds the stale "pending" states and adds the cut-off run's work to the opening note. > - The benefit is that a stalled task gets repaired within minutes, and a fresh run does not repeat completed work. ## Linked Issues or Issue Description Refs #21135, #21203, #21234 **What happened?** 1. The invariant counted a task as pending without limit when its projection said "system recovery working", or when a wake was deferred behind a run that had already ended, or when a wake was claimed but no run was ever created. In release-swap waves, an undelivered continuation and follow-ups that were never promoted were reported OK for 17 minutes or more and not repaired. The suspect listing also excluded every task with a deferred wake, so the periodic sweep did not find them. 2. A fresh continuation's opening note listed only actions with unknown outcomes. Subtasks, comments and documents that the cut-off run had completed were not mentioned, and follow-ups it had already read were not mentioned. Continuations created the same subtasks again and posted the DONE comment twice. **Expected behavior** A stale pending state stops counting after a bounded time, and the sweep continues the task once. The fresh run knows what the cut-off run already created and read. **Steps to reproduce** 1. Run a release-swap wave where a run is cut off after it created subtasks and posted a comment, and where a follow-up waits behind it. 2. See the continuation create the subtasks again, or see the task stay "working" with no run while the invariant reports it OK. ## What Changed - Invariant bounds (`server/src/services/recovery/assigned-work-invariant.ts`, `server/src/services/recovery/assigned-work-sweep.ts`): - "System recovery working" counts as pending only for `SYSTEM_RECOVERY_WORKING_BOUND_MS` (17 min) after the task's last run ended: the fallback's progress bound (15 min) plus one disposition re-check. - A deferred wake with no live run counts as pending only for 5 minutes after it was created. A claimed wake with no run counts only for 5 minutes after it was claimed. - The suspect listing uses the same bounds, so the periodic sweep finds these tasks. - A held continuation (paced delivery from #21234) counts as the task's pending wake only for `PENDING_CONTINUATION_BOUND_MS` (7 min) after it was recorded: the pacing admits a held continuation within 5 minutes even when the queue is full (`RECOVERY_CONTINUATION_MAX_HOLD_MS`), plus 2 minutes of 15-second delivery passes. Inside the bound the task still shows "working" with no Retry, as #21234 intends. - The sweep's wake still goes through admission, which hands a held task to the fallback. It never starts a second writer. - A continuation whose delivery is stuck is repaired, not just no longer counted (`server/src/services/recovery-continuation-bounds.ts`, `execution-recovery-resolution.ts`, `fresh-continuation-fallback.ts`): - One module owns `RECOVERY_CONTINUATION_MAX_HOLD_MS` (5 min) and `PENDING_CONTINUATION_BOUND_MS` (that plus 2 min). The invariant, wake admission and delivery all read them from there. - Delivery re-drives first. A continuation held past 5 minutes was already admitted when the agent's queue was full, but the delivery page skipped every record of an agent that had another recovery run queued, at any age. That exclusion now applies only within the hold bound. - Admission never replaces the recorded continuation. The recorded continuation alone carries the isolation and fresh-session decision. A wake that `hasPendingFreshContinuation` coalesces into it now also notifies the delivery queue. Delivery then re-drives the same record, with the same isolation key, `forceFreshSession`, and one-successor idempotency key. ### Where the isolation decision is read, and every path that starts the successor Run preparation reads the decision in one place: `readFreshWorkspaceReadiness` (`server/src/services/fresh-workspace-continuation.ts`), called from `heartbeat/workspace-preparation.ts`. It reads `contextSnapshot.freshContinuationIsolation`, which names the record, and loads it through `readFreshContinuationReadiness`. Only a run that carries that key gets a new isolated workspace. | Path that can start the successor | How it keeps the isolation | Test | |---|---|---| | Delivery worker (`deliverReconciledExecutions`) | Sends the record's own wake with `freshContinuationIsolation` and `forceFreshSession` | `fresh-continuation-fallback.test.ts` "gives the continuation a new isolated workspace while the old local writer may be alive" | | Re-drive after the bound | The same delivery wake, admitted past the hold bound even when the queue is full or another recovery run is queued | "re-drives an isolated continuation pending past the bound itself; a repair wake never replaces it"; `recovery-wake-herd.test.ts` "re-drives a continuation held past the hold bound..." | | Sweep / monitor / timer wake | Never becomes the successor: coalesced into the record, which it re-drives | "records a pending continuation for an ordinary continuation wake and delivers it once"; the re-drive test above | | Saved input (a person's message) | Waits behind the isolated continuation and is promoted when that run is released (#21250) | `postgres.test.ts` "... (released, continuation pending)" (#21250) | | A person's Retry started before delivery | Delivery counts the Retry's run as the successor. The Retry run does not carry `freshContinuationIsolation`. On a task with a workspace binding, the fallback cleared the binding, so a new workspace is prepared. On a projectless task this path is **not yet covered** | "never starts a second run when a person's Retry already continued the task" (one successor only). Isolation on this path is a known gap, reported for follow-up | - Opening note (`server/src/services/fresh-continuation-fallback.ts`): - Names the subtasks, comments and document saves that the cut-off run created, and asks the agent not to create them again. The record keeps the list as `createdObjects`. - Lists the person's messages that the cut-off run had read (through `taskHistoryReadsForRun`, from #21203) before what that run then posted, so the agent answers a re-delivered follow-up once. Only the revision that was read counts as read. A message a person edited after the read is listed as changed ("treat them as new"). - The note is written when the continuation is recorded. Nothing is added before `turn.start`. ## Verification Red-to-green tests (each failed before its fix and passes after it): - `server/src/services/recovery/assigned-work-invariant.test.ts` (unit) and `server/src/__tests__/assigned-work-invariant.test.ts` (sweep integration): stale "working", a stale deferred wake, and a claimed wake with no run become violations past their bound and stay pending inside it. - `server/src/services/fresh-continuation-fallback.test.ts`: "names the subtasks, comments and documents the cut-off run already created" and "names the follow-up the cut-off run had read and its own final comment". - `fresh-continuation-fallback.test.ts` "re-drives an isolated continuation pending past the bound itself; a repair wake never replaces it" (projectless task, writer possibly alive). It was red on |
||
|
|
7e7452e88f |
fix(native): keep a reattached Runner alive when its provider recovery deadline ends (#21321)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native Codex runs on remote sandbox Runners. A same-run reattachment
adopts a live Runner after a failed attempt or a server deploy.
> - On staging, a remote Codex Runner is often gone when the next
attempt reattaches (`process_verification` / `process_not_running`). The
task then continues in a fresh session and loses its context.
> - The deploy does not stop the Runner. The logs show `controller
authority detached for restart; durable runner left alive`, and the new
controller adopts the same Runner after the deploy.
> - Our own cleanup stops it. The adopted Runner answers its recovery
reads slowly while its in-flight turn runs. The 900 s provider recovery
deadline then aborts the recovery, and the abort closes the transport
with `turn.stop` and `runner.suspend` and signals the process.
> - This pull request makes the recovery deadline detach from the
Runner, as a recovery read timeout already does. The server marks the
failure as an unanswered adopted Runner, so the next attempt replaces
the Runner from its saved journal and resumes the same provider thread.
> - The benefit is that a deadline no longer kills a live Runner and no
longer forces a fresh session. The task keeps its session context.
## Linked Issues or Issue Description
Refs #21314, #21272, #21161, #21023
**What happened?**
Codex runs on Daytona remote Runners failed often on staging. In three
sampled chains (attempt 2 of each), the same-run reattachment adopted
the live Runner, authenticated in under 1 s, and then waited the full
deadline: `native session provider recovery timed out after 900000ms`
(909 s, 930 s and 915 s). The deadline cleanup sent `turn.stop` and
`runner.suspend` (`provider turn stop timed out before runner
suspension`). The next attempt found the Runner gone 25–180 s later
(`process_not_running`, `runnerLifecycle: ready`, empty stderr) and
continued in a fresh session. During the 900 s wait, the in-flight turn
was still running Paperclip tools (`tool.execute` spans at 04:43 in a
recovery that started at 04:29).
**Expected behavior**
A remote Runner survives a recovery deadline and a control-plane deploy.
The next attempt continues the same provider session. A Stop still stops
the Runner.
**Steps to reproduce**
1. Start a Codex run on a remote Runner and let a turn start.
2. Fail the attempt so that its close keeps the Runner (for example, a
turn start that outlives its bound).
3. Let the same-run reattachment adopt the Runner, and make the Runner
stop answering recovery commands.
4. Let the provider recovery deadline expire.
5. See that the Runner process is gone and the next attempt reports
`process_not_running`.
**Paperclip version or commit**
master at
|
||
|
|
4fe34df748 |
Do not show Retry on a parent with open subtasks while its queued message is settled (#21320)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The task view shows a "Retry" notice when a task needs a person to start it again. > - The product rule is that a task that continues by itself never shows this notice, not even for a few seconds. > - A parent task that waits for its subtasks showed the notice for 3.4 seconds after its run succeeded. > - The cause is the queued-message grace in the task interruption presentation: it does not apply to `blocked` tasks. > - #21200 kept `blocked` out of that grace on purpose, so this pull request adds only the exact observed shape: a parent with open subtasks and a queued task message. > - The benefit is that a parent that waits for its subtasks no longer shows a false failure, and every other blocked task keeps its Retry. ## Linked Issues or Issue Description Refs #21264, #21200 (added the queued-message grace and kept `blocked` out of it) **What happened?** A Codex parent task ran across a two-server release swap. Its run waited for two subtasks and then succeeded with the disposition `blocked`. The task status became `blocked`. A user comment (a deferred wake, status `deferred_issue_execution`) had been queued 13 minutes earlier, while the run was active. For 3.4 seconds after the run finished, the task presentation was `state: "failed"`, `canRetry: true`. Then the release drain cancelled the wake as "Queued messages were already delivered to a completed run", and the presentation changed to `resolved`. The subtasks finished later. Their `issue_children_completed` wake continued the parent, and it completed without help. The cause is in `projectTaskInterruptionState` in `server/src/services/task-interruption-presentation.ts`. The `queuedDuringRun` check shows a succeeded run with user input queued before it finished as `working` for up to 2 minutes. It applies only to `todo`, `in_progress` and `in_review`. For a `blocked` task the presentation fell through to the last line: `failed` with `canRetry: true`. The workspace finalization deferral (`native_workspace_finalization_busy`) in the same window does not change the presentation. The presentation does not read it. **How this differs from what #21200 narrowed** #21200 first applied the grace to every status. Its commit `a3fae9bb67` removed `blocked` because input queued on a blocked task is not promoted by itself, and because a recovery that gives up hands its hold to the board and must end with Retry. This change does not add `blocked` back. It adds one narrow case, and all of these must be true: - The latest run **succeeded**. A failed, interrupted or given-up run is not affected. - The task is `blocked` **and has open subtasks of its own** (not `done` or `cancelled`). Their completion wakes the parent again (`issue_children_completed`). - The pending item is an ordinary **task comment** wake (`commentId` or `wakeCommentIds` in its payload), not a Retry request (`taskRetry.sourceRunId`). - The comment was queued before the run finished, the run finished less than 2 minutes ago, and saved-work recovery is not exhausted. These are the existing bounds. A blocked task that waits for a person or a board decision has no open subtasks, so it keeps Retry as before. **Expected behavior** After the run succeeds, the parent shows `working` with no Retry while its queued message is settled. Then it shows `resolved` (message already delivered) or `working` (message promoted to a new run). If nothing settles the message in 2 minutes, Retry shows. **Steps to reproduce** 1. Assign a parent task that creates subtasks and waits for them. 2. Post a comment on the parent while its run is active. 3. Let the run succeed with the disposition `blocked`. 4. Read the task presentation in the first second after the run finishes. It shows `failed` with Retry. **Paperclip version or commit** master `bbafa78fe2`. The wave that found it ran on `62d673cc3f`. ## What Changed - `server/src/services/task-interruption-presentation.ts`: - The batched wait read also reads whether the wake carries a task comment (`queuedComment`). - One company-scoped read finds which `blocked` tasks with a succeeded latest run and a queued comment have open subtasks (`waitingOnOpenSubtasks`). It runs only for those tasks. - The `queuedDuringRun` grace also applies to such a task. The `todo`, `in_progress` and `in_review` behavior does not change. - `server/src/__tests__/task-interruption-automatic-path.test.ts`, new embedded-Postgres tests: - The exact observed state: a `blocked` parent with two open subtasks, a run that succeeded 1 second ago, a user comment wake queued 13 minutes earlier. Shows `working`, no Retry. - The same parent after the wake is cancelled as already delivered: `resolved`. - The same parent after the wake is promoted to a new run: `working` on the new run. - Guard: the same parent after the 2-minute grace: `failed` with Retry. - Guard: a `blocked` task with a queued comment and no subtasks, or only `done` or `cancelled` subtasks (waiting for a person or a board decision): `failed` with Retry, unchanged. - Guard: a parent with open subtasks whose queued wake carries no comment: `failed` with Retry, unchanged. - Guard: a parent with open subtasks and a queued comment whose run failed and whose recovery hold went to the board: `failed` with Retry. - Guard: a parent with open subtasks and a queued comment whose saved-work recovery is exhausted: `failed` with Retry. - A parent whose run succeeded with no queued message stays `resolved` when its subtasks are open, blocked or cancelled. - The existing #21200 assertion (a `blocked` task with queued input and no subtasks shows Retry) is kept unchanged. ## Verification - `cd server && npx vitest run src/__tests__/task-interruption-automatic-path.test.ts` - Red on master `bbafa78fe2`: "shows a parent whose run just succeeded waiting on its subtasks as working while its queued message is settled" returns `state: "failed"`, `canRetry: true`. This is the value that the wave recorded. The other 15 tests pass on master. - Green with this change: 16 of 16. - A plain `blocked` inclusion fails 3 of the guards: the #21200 assertion, "still offers Retry on a blocked task with a queued message and no open subtasks" and "still offers Retry on a parent with open subtasks when its queued wake carries no message". This shows that the narrow condition is necessary. - `task-interruption-presentation.test.ts`, `task-interruption-query.test.ts`: green (116 tests with the file above). - `cd server && npx tsc --noEmit -p .`: 0 errors. - Not verified: no new chaos wave. The flash was seen once in a loaded release-swap wave. The server test reproduces its saved state. ## Risks - Low risk. The change is presentation only. It does not change wake promotion, dispatch or Retry. - A parent with open subtasks and a stranded queued comment now shows `working` for up to 2 minutes after its run succeeded, and then Retry. Before, it showed Retry at once. - One extra indexed read (`issues_company_parent_idx`) runs only for blocked tasks with a succeeded latest run and a queued comment. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, in Claude Code with tool use (shell, file edits, test execution). Extended reasoning was available. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
d9a4bb23fe |
fix(runner): wait for a slow Codex turn start while it reports progress (#21329)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Codex agents run through the Paperclip runner, and the server-side transport starts each provider turn > - After Codex accepts a turn, the transport waits a fixed 30 s for that turn's `turn/started` before it reports the turn identity > - Codex starts its MCP servers before it starts the turn, so a healthy cold start in a loaded sandbox often takes longer than 30 s > - The transport then failed a live turn with `runnerd did not report the provider turn identity`, and the abandoned turn's late notifications broke the reattachment (`did not name its execution owner`, `runner_harness_state_mismatch: resumed_not_suspended`) > - This pull request sizes the start window to the measured cold start and extends it while the provider keeps reporting startup progress > - The benefit is that slow but healthy Codex starts succeed on the first attempt, and a dead start still fails at a bound ## Linked Issues or Issue Description **What happened?** On staging, Codex runs on Daytona failed at turn start with `native_session_interrupted: runnerd did not report the provider turn identity`. The run log shows the same sequence each time. The runner accepts the turn. Codex then reports startup progress (a bubblewrap config warning, `account/updated`, and `mcpServer/startupStatus/updated` failures for an MCP server that is not logged in) for about 30 s. The transport gives up 30 s after acceptance. A few seconds later, Codex sends the turn's `turn/started`, which now arrives at a closed turn (`protocol_failure:turn_binding_mismatch`). The same-run reattachment then hits that orphaned turn. It fails with `Provider authoritative notification item/started did not name its execution owner.` or `runner_harness_state_mismatch: resumed_not_suspended`. Of 198 recent runs in the staging company, 52 contain this turn identity failure. In 68 successful starts, submit to `turn/started` took p50 25 s, p75 37 s, p90 50 s, p95 54 s and max 76 s. The fixed 30 s wait sits near the median. **Expected behavior** A slow but healthy Codex start succeeds. A start that makes no progress, whose Runner exits, or that the runner fails still fails at a bound and recovers through the normal same-run resume. **Steps to reproduce** 1. Start a Codex turn whose provider accepts the turn but sends `turn/started` after the 30 s default window (on staging: several MCP servers and a loaded sandbox). 2. Let the provider report startup notices during the wait, and let the runner not prove the accepted turn live in time (on staging no turn was adopted). 3. See `runnerd did not report the provider turn identity`, then the reattachment conflict. The new tests reproduce this with the fake Codex app server (`--turn-started-delay-ms` with `--turn-started-progress-ms`) and a runner snapshot that cannot prove liveness. **Paperclip version or commit** Staging Core `c730872974` (`2026.1106.0+2253`). Base of this PR: `origin/master`. **Deployment mode** Authenticated, public, managed cloud (staging). Remote runner on Daytona. **Agent adapter(s) involved** `paperclip_runner` with the Codex provider (`gpt-6.1-sol`). ## What Changed - `runnerd-codex-transport.ts`: the default wait for an accepted turn's `turn/started` is now 90 s (measured max 76 s), not 30 s. An explicit `turnStartTimeoutMs` keeps its old bound. - The wait restarts its window each time the runner commits a provider notice, or a runner diagnostic that names the accepted turn. The total wait stops at 180 s (option `turnStartProgressCeilingMs`, so tests can lower it). A provider relaunch after the accepted `turn.start` does not extend it. - The adoption check (runner snapshot proves the accepted turn live) now runs before each window end, not only before the first one. A Runner exit and a runner failure of the pending turn still fail at once. - `fake-codex-app-server`: new `--turn-started-progress-ms` flag sends `mcpServer/startupStatus/updated` failures while it delays `turn/started`. - `doc/execution-semantics.md`: documents the start window, the progress rule and the staging measurements. ## Verification - Red on `origin/master` transport (with the new fake flag): `keeps waiting for an accepted Codex turn while its startup keeps reporting progress` failed after 4.6 s, and `gives a silent Codex cold start the measured default start window` failed after 30.9 s. Both failed with `runnerd did not report the provider turn identity`. - Green: `npx vitest run src/live/runnerd-codex-transport-deadlines.test.ts` passes 9/9. This includes a ceiling test: provider notices every 1 s and no `turn/started` fail at an 8 s test ceiling, and the test goes red with the ceiling cap removed. It also includes the existing adoption, no-adoption-within-bound and runner-failure tests, and a new test that a silent start still fails within its bound. - `npx vitest run` on the adoption, cut-turn-start, stall, turn-start-timeout-retention, remote-retention, restart-continuation and controller-outage transport suites passes 48/48. - `npx vitest run src/live/runnerd-codex-transport.test.ts src/live/live-session.test.ts src/live/turn-stream.test.ts`: 391/392 pass. The one failure (`stops a lost runnerd's provider and escaped tool job from its lineage journal (journal=true)`, 1 s, process-lineage timing under full-suite load) passes when run alone, both on this branch and on `origin/master`. - `tsc --noEmit` for `packages/paperclip-runner` and `cargo fmt --check` pass. ## Risks - A silent dead start (no Runner exit, no provider output, no runner failure) now fails after 90 s, not 30 s. The Runner exit and runner-failure paths still fail at once, so this applies only to a provider that stops without a trace. - A provider that keeps sending notices but never starts the turn waits at most 180 s. - No schema, API or UI change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, browser reads of staging run data). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge Refs #21272, #21120 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
314b893678 |
fix(native-runtime): deliver assigned gateway tools to every native harness through the PRP relay (#21274)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - An agent can have app connections. Paperclip gives the agent the
tools of those apps through an MCP gateway, with its approval and audit
rules.
> - Native harnesses reach the gateway in two ways today. In the relay,
the server keeps the gateway token and offers the tools over the
authenticated Runner channel. In the direct way, the server gives the
gateway URL and a bearer token to the harness.
> - Which way a harness uses comes from a list of special cases that
grew one harness at a time. Local Codex and all ACPX agents use the
direct way. The others use the relay.
> - The direct way puts a run-scoped token in a provider process. A
process that the agent starts can read it. A provider process that
serves more than one run can hold a revoked token.
> - This pull request replaces the list with one rule. Each native
harness gets its assigned gateway tools through the relay.
> - The benefit is one behavior for all native harnesses, and no gateway
token in a provider process.
## Linked Issues or Issue Description
No public issue exists. The issue is described here.
Refs #13942 (the relay), #17149 (ACPX accepts relayed tools) and #19912
(the design for turn-scoped ACPX MCP authority).
**What existing behavior does this improve?**
How a native harness reaches the tools of its assigned MCP gateway.
**Subsystem affected**
Native runtime in the server (`native-session-executor.ts`,
`native-session-resume.ts`). One Runner test file. One Rust test.
**Current behavior**
- `relayAssignedMcp` is true for plugin tasks, Cursor Cloud, OpenCode,
Codex on a remote target, and Dot.
- For local Codex and for each ACPX agent (Claude, Codex, Cursor, Pi,
Kimi, Grok, Copilot, Antigravity), Core puts
`PAPERCLIP_NATIVE_MCP_NAME`, `PAPERCLIP_NATIVE_MCP_URL` and
`PAPERCLIP_NATIVE_MCP_TOKEN` in the Runner environment. The ACPX host
gives them to the provider as an HTTP MCP server. The local Codex home
gets them in `config.toml`.
- The token is run-scoped. Core revokes it when the run ends.
**Proposed behavior**
- A native run that has an assigned gateway uses the relay. The
condition names no provider.
- Core builds the gateway tools on the control plane, offers them with
the other Runner tools, dispatches each call over PRP, and authorizes
each call again through the gateway.
- Core removes the three `PAPERCLIP_NATIVE_MCP_*` variables from each
Runner environment.
- The tool contract fingerprint has the relay marker for each target,
not only for remote targets.
**Reason and benefit**
- All native harnesses have the same behavior.
- No provider process holds a gateway token.
- A sandbox does not need to reach the gateway URL of the host.
## What Changed
- `server/src/services/native-runtime/native-session-executor.ts`
- The list of special cases is gone. Core validates the binding and
builds the relayed tools when a binding is present.
- Core does not list the gateway for a review run, Grok Bot or Muse. The
tool authority never offered app tools to those. A gateway error can
therefore not fail those runs.
- OpenCode keeps its `on_demand` catalog. That option depends on the
provider, not on the relay.
- Core removes the three variables from each Runner environment.
- `server/src/services/native-runtime/native-session-resume.ts`: the
relay marker in the tool contract fingerprint is unconditional.
- `doc/execution-semantics.md`: the text about the tool contract
fingerprint now states the rule, the tool names and the cutover.
- App tool schemas. New module
`packages/paperclip-runner/src/drivers/tool-input-schema.ts` compiles
each schema by itself, in its declared dialect (draft-07, 2019-09 or
2020-12), with no shared `$id` registry. Core uses it when it builds the
relayed tools (`assigned-mcp-tools.ts`). Core does not offer a tool
whose schema has an unsupported dialect or does not compile, and writes
a log entry with the tool name and the reason. The run does not fail.
The ACPX tool bridge and the OpenCode MCP bridge use the same function.
The tools of Paperclip itself stay strict.
- Large arguments. The ACPX sidecar sends the arguments of a tool call
to runnerd with no display limit. The limit is the tool-call admission
limit of runnerd, 256 KiB. A larger call fails with a tool error that
gives the limit. The sidecar does not dispatch it.
- Tests in `native-session-executor.test.ts`,
`native-session-resume.test.ts`, `runtime-context-materializer.test.ts`
and `acpx_provider_backend.rs`.
Not changed in this pull request: the direct code path. It is still in
the Runner, but it receives no binding. #21311 deletes it.
## Behavior that each harness sees
| Harness | Before | After |
|---|---|---|
| Codex on a remote target, OpenCode, Cursor Cloud, Dot, plugin tasks |
Relay | No change |
| Local Codex | Direct: a `paperclip-assigned` server in `config.toml` |
Relay. App tools are dynamic tools named `app_<name>_<digest>` |
| ACPX agents | Direct: a second HTTP MCP server | Relay. App tools are
on the `paperclip` bridge as `mcp__paperclip__app_<name>_<digest>` |
| ACPX with a preconfigured home | No gateway | No change |
| Grok Bot, Muse | Got the variables and ignored them | Do not get the
variables |
- A relayed tool name has a maximum of 64 characters. With the
`mcp__paperclip__` prefix it has a maximum of 80.
- Permission behavior keeps the same intent. For Claude in the
`approve-reads` and `approve-paperclip` modes, app tools ask for
approval, as before. For the other ACPX agents the permission decision
does not use the server name.
- In a planning run or an ask run, the relay offers only tools with read
risk. The direct gateway offered all tools. This is stricter.
- When the catalog is too large, each harness gets the two compact tools
`paperclip_search_assigned_tools` and `paperclip_call_assigned_tool`.
- The four helper tools for resources and prompts
(`paperclip_list_resources`, `paperclip_read_resource`,
`paperclip_list_prompts`, `paperclip_get_prompt`) are not in the relay.
No harness loses them: a run token permits only `tools/list` and
`tools/call` (`run-preparation.ts`), so these four tools already fail
with `gateway_token_action_denied` on the direct path.
## State saved by the previous release
- **Local sessions of each provider start again one time.** The tool
contract fingerprint changes for local targets. The next run sees a
different fingerprint, starts a new provider session with the full task
context in the same workspace, and keeps the old saved state. This is
the existing rotation for a tool contract change. Codex needs it,
because Codex cannot add tools to a thread that it resumes. For the
local providers that already used the relay, the restart is not
necessary. It is accepted so that the rule names no provider.
- **Remote sessions continue.** Their fingerprint does not change. For
ACPX, the attach of the next run replaces the saved tool set.
- **A kept Codex home.** Paperclip writes `config.toml` again at each
start and recovery. The old gateway server and its token are removed.
- **A deploy lands during a run, and the Runner survives.** The
controller attaches to the same Runner for the same run. The old
provider keeps its direct gateway connection. The controller detach
keeps the gateway token of that run, and Paperclip revokes it when the
run ends.
- **A deploy lands during a run, and the Runner is dead.** The Runner
starts again for the same run with its saved tool set and no direct
gateway. The assigned app tools are not available for the remainder of
that one run. The next run gets them through the relay. This applies to
each harness that moves to the relay. It has no test.
- Paperclip refuses no saved state and discards none.
## Verification
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/native-session-resume.test.ts`: `1508 passed
| 7 skipped`, 1 failed. The failed test is "archives a damaged prior
epoch before completing guarded same-task replacement with real runnerd"
(`ENOENT … authority-epochs`). It fails in the same way on `master` at
|
||
|
|
fde0328ff5 |
fix(recovery): deliver every pending continuation within a bounded time (#21324)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Execution recovery continues a task in a fresh run after a restart or a lost run left it unfinished > - #21234 paced these continuations: at most 1 per agent and 5 per company per pass, and a continuation waits while its agent already has a queued recovery run or a full queue > - A continuation that waited 5 minutes must enter the queue anyway, but that bound applied only to the full-queue check. The "agent already has a queued recovery run" check was in the page query, and it had no bound. > - On staging, an agent had queued recovery runs that did not start for hours. Its pending continuation was never read again, and its task showed "working" with nothing running or queued > - This pull request applies the 5-minute bound to both checks, stops one undeliverable continuation from using its agent's or company's slot, and stops the task view from showing "working" for a continuation that is overdue > - The benefit is that every pending continuation enters the queue within a bounded time, also after a restart, and the pacing still protects people from recovery herds ## Linked Issues or Issue Description Refs #21234, #21135 **What happened?** On staging, a task had a fresh-continuation hold with `continuationDelivery: "pending"` since about 03:30Z. More than 3.5 hours later, no run had started for the task. The task view showed the interruption state `working` with `canRetry: false`, so the user saw nothing wrong. The task's agent had 20 concurrency slots and 11 running runs. It also had queued `issue_continuation_needed` runs from 22:54Z that had not started. The delivery pass reads pending continuations with a query that leaves out every agent that has a queued recovery run (`issue_recovery_action_restored` or `issue_continuation_needed`). That condition had no time bound, so the pass never read this continuation again. The 5-minute admission bound was never reached. Across the company, 203 open tasks showed `working` while the API showed no queued or running run for them. 162 queued runs were more than 1 hour old. **Expected behavior** Every pending continuation enters its agent's queue within a bounded time, also when the agent's queue holds recovery runs that do not start, and also after a server restart. Pacing still limits how many continuations enter a queue at the same time. A task does not show "working" when its continuation is overdue and nothing runs or is queued. **Steps to reproduce** 1. Give an agent a concurrency limit of 1. 2. Put one queued `issue_recovery_action_restored` run on that agent that does not start. 3. Record a pending fresh continuation for another task of the same agent. 4. Wait more than 5 minutes, and let the 15-second continuation pass run. 5. See that the continuation stays `pending`, the task shows `working`, and no run starts. **Paperclip version or commit** master at |
||
|
|
2483cb46c2 |
feat(plugins): deliver configured environment to plugin Runner tasks (#21280)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - An agent gets environment variables from its execution environment, its own configuration, its project, and its routine. Secrets are resolved for each run. > - A plugin environment can run the agent as a native Runner task on its own host, through the environment task contract. > - That contract has no field for the agent's variables. Runs on a plugin environment get only the variables that the plugin sets itself. > - This pull request adds a versioned contract field for the values, sends them on a session handoff, lets Codex commands inherit the variables in an external sandbox, and lets a plugin protect the variables that it sets itself. > - The benefit is that plain and secret variables work on plugin environments in the same way as on local environments. ## Linked Issues or Issue Description Refs #21233. That pull request by @lucktastic finds the same gap and proposes the same capability and field names (`configuredEnvironmentVersion: 1`, `configuredEnvironment`). This pull request uses the same names, so providers implement one contract. It differs in these places: - Codex commands get the variables through an external-sandbox denylist, not through key names sent at `prepare`. A retained Runner does not need a new preparation when a key is added or removed. - A session handoff `resume` gets the values, because it starts a new Runner on another computer. Value changes on the same computer need no new field: Core's process reuse key includes the values, so Core starts a new task. - A driver can declare `protectedEnvironmentKeys`. Paperclip rejects or warns when one is saved. - The run log names skipped variables only for an older provider. Thank you to @lucktastic for the diagnosis and the contract shape. **What happened?** An agent has environment variables, plain and secret. Its runs on a plugin task environment do not get them. `PAPERCLIP_CONFIGURED_ENV_KEYS` in the run lists only the variables that the plugin sets. **Expected behavior** Each run gets the agent's configured variables. **Steps to reproduce** 1. Set a plain and a secret environment variable on an agent. 2. Assign the agent to a plugin environment that provides Runner tasks. 3. Start a run. Read the names in `PAPERCLIP_CONFIGURED_ENV_KEYS` in a command that the agent runs. 4. The agent's names are not in the list. ## What Changed - **SDK contract** (`packages/plugins/sdk/src/environment-tasks.ts`): - A provider advertises `configuredEnvironmentVersion: 1` on `describe`. - `submit` gets `configuredEnvironment`. `resume` gets it only for a session handoff. - Names, count, and sizes have the same limits as the Runner projection. - **Server**: - `createRunnerdBackendWithinSessionClaim` gives the run's configured projection to the plugin task owner. The projection has no provider credentials and no generated runtime keys. - `createPluginRunnerTaskOwner` sends the fields only to a provider with the capability. An older provider gets no new field, and the run log names the variables that were not set. The log never prints values. - **Codex** (`codex-security-config.ts`): - An externally sandboxed launch uses `inherit="all"`, `ignore_default_excludes=true`, and an `exclude` list of Codex's own credentials, instead of an `include_only` list fixed at preparation time. - A test reads the Runner's Codex environment keys, and fails if a new key is not classified. - Host launches keep the bounded allowlist. - **runnerd** (`codex_provider.rs`): `runtime_notification_codex_args` still grants the notification files to read-only profiles when no allowlist is present. Plan mode in an external sandbox needs these grants. - **Protected keys**: - An environment driver can declare `protectedEnvironmentKeys` in its manifest. - Paperclip rejects a protected key on an agent or environment that uses the driver. The 422 has code `protected_environment_variable`. - On a project, a routine, or another agent, Paperclip saves the key. The response has a `configurationWarnings` notice that names the environments. The project and routine pages show the notice as a toast. The agent pages already show configuration warnings. - **Docs**: `doc/plugins/PLUGIN_ENVIRONMENT_TASKS.md` describes the provider obligations. - `tests/runner-e2e/source-sdk-public-inputs.json`: refresh the reviewed SDK source pin. ## Verification - `packages/plugins/sdk`: `vitest run tests/environment-tasks.test.ts` (38 pass). Accepts bounded values. Rejects bad names, NUL bytes, too many variables, and oversized values. - `server`: `vitest run src/services/native-runtime/plugin-runner-task.test.ts` (95 pass). Values go only to a provider with the capability, never in `prepare`. `resume` sends values only for a handoff. - `server`: `vitest run src/services/protected-environment-keys.test.ts src/__tests__/project-routes-env.test.ts` (10 pass). Rejection, notices, and the project route response. - `server`: all route suites that import the project, routine, environment, or agent routes (45 files, 1689 pass). - `packages/shared`: `vitest run src/validators/plugin.test.ts` (64 pass). - `packages/paperclip-runner`: `vitest run src/drivers/codex src/configured-environment.test.ts src/live/runner-provider-launch.test.ts` (444 pass). - `packages/paperclip-runner/runner`: `cargo test -p paperclip-runner-core --lib runtime_notification` (9 pass, 2 ignored as before). - `pnpm typecheck` in `packages/shared`, `packages/plugins/sdk`, `packages/paperclip-runner`, `server`, and `ui`. - Not run locally: `pnpm verify --full` and browser suites. Required CI covers the final revision. ## Risks - **Existing providers:** low risk. Core sends no new field to a provider without the capability. Their runs behave as before, with one added log line when variables were skipped. - **Codex in an external sandbox:** commands now see every variable in the Runner environment except Codex's own credentials. The execution environment builds that environment from an allowlist. If the Runner gives Codex a new credential, the drift test fails until the key is classified. - **Saving env:** new rejections happen only when an installed driver declares `protectedEnvironmentKeys`. No driver declares them today. - **Providers:** they must keep the values transient and never save or log them. The doc lists these obligations. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
e73fc71331 |
fix(environments): continue on a new sandbox when an idle runner sandbox never starts (#21323)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A task that runs on a reusable cloud sandbox resumes the same sandbox on the next wake > - On staging, Daytona left two idle sandboxes in the `archiving` state for more than three hours, so no resume call could succeed > - The native runner leases had no verified harness backup, so the run never replaced the sandbox, even though the task files were saved > - Each retry spent six minutes on the same stuck sandbox, the bounded retries ran out, and the task waited for a person to press Retry > - This pull request continues on a new sandbox when the provider has proven it cannot start the old one and the workspace is saved, and it keeps the old sandbox > - The benefit is that the task recovers by itself and a sandbox that may hold unsaved runner state is never destroyed ## Linked Issues or Issue Description Refs #20626, #21113 **What happened?** Eight Codie runs on staging failed in one hour with `reusable_sandbox_resume_failed`: `Reusable sandbox lease "<id>" could not be resumed (cause=transport_timeout); the lease was preserved and no replacement was created.` They came in two clusters (05:59 to 06:00Z and 06:32 to 06:38Z). Each run took about six minutes, so the 330-second idle resume budget from #21113 was applied. All runs were already `transient_failure_retry` wakes. Investigation of one instance (lease `c23d6120…`, run `d16c272e…`): - The lease was `released` at 02:37Z with a confirmed stop-and-retain. No live run held it, so the lease was idle and the extended budget applied. - A read-only Daytona API read at 06:5xZ showed the sandbox in state `archiving` (desired state `archived`), last updated 03:38Z, one hour after the stop (the auto-archive interval is 60 minutes). The second sandbox (`e0e6cfe5…`) showed the same state since 03:35Z. Daytona does not start a sandbox while it archives, so every resume call timed out. More time or a new HTTP connection could not help. - The task workspace was clean and had a committed saved snapshot. - The agent uses `paperclip_runner`. The last run on the sandbox failed while it saved the runner session, so the lease had no `nativeHarnessBackup` stamp. `reusableSandboxReplacementIsSafe` requires a verified harness backup for a runner lease, so the run never replaced the sandbox. It failed strictly, the bounded retries (2) ran out, and the task waited for a manual Retry. **Expected behavior** When an idle sandbox never starts through the whole provider start deadline and the task files are saved, the run continues on a new sandbox automatically. A sandbox that may hold unsaved runner state is not destroyed. **Steps to reproduce** 1. Run a `paperclip_runner` task on a reusable Daytona sandbox. Let the last run end without a harness backup (for example, the runner exits while it saves the session). The workspace is clean with a committed snapshot. 2. Let Daytona start to archive the stopped sandbox and get stuck in `archiving`. 3. Wake the task. Every `environmentResumeLease` call times out, the run fails after about 330 seconds with `reusable_sandbox_resume_failed`, and the retries fail the same way. **Paperclip version or commit** Staging Core `0c04ff76eb` (current `master` has the same code path). **Deployment mode** Hosted staging (cloud), Daytona sandbox provider plugin, reusable leases on. **Agent adapter(s) involved** `paperclip_runner` (native runner) on a Daytona sandbox environment. **Root cause** `server/src/services/reusable-sandbox-resume-replacement.ts`: a runner lease without a verified harness backup was never replaceable, for any failure. The only other path was to retry the same sandbox, which cannot work while the provider is stuck. Replacement also always retired the old lease to `pending_cleanup`, so allowing it without a backup would have destroyed the only copy of the runner session state. ## What Changed - `reusable-sandbox-resume-replacement.ts`: `reusableSandboxReplacementIsSafe` becomes `reusableSandboxReplacementPlan`. It returns `retire` (same rules as before), `keep`, or `null`. It returns `keep` for a runner lease without a verified harness backup only when the caller passes `providerCannotStart`. All other conditions still apply: idle lease, same workspace, clean workspace, committed snapshot, this run is the writer, and no access refusal. - Add `keepReplacedReusableSandboxLease`. After the replacement lease is recorded, one guarded write marks the old lease with `reusableSandboxKeptAfterReplacement`. The lease keeps its status and its sandbox. Nothing destroys it. - `environment-runtime.ts`: set `providerCannotStart` only when the idle start deadline was extended and the run then used all of it on timeouts. Lease selection skips a lease with the keep marker, so later runs do not spend time on it and obsolete-lease cleanup does not destroy it. - The replacement records `replacement` / `resume_failed` with no harness backup. The native executor already starts a fresh session in a new sandbox in this case (`freshReplacement`), and workspace preparation restores the committed snapshot. - The keep plan needs every resume call to time out. A rate limit, a reset, or any other failure in between keeps the strict behavior. - `reusable-sandbox-kept-marker.ts`: the marker key and a SQL guard. Every automatic teardown path skips a marked lease: retained-orphan listing and settle, `destroyReusableSandboxLease` (obsolete cleanup, expiry, run destroy), `destroyClaimedSandboxLease` (issue and workspace teardown), recovered-workspace cleanup, and abandoned-file release. Only an operator-consented environment delete still destroys it. - `doc/execution-semantics.md`: document the rule. ## Verification - Red before the fix: the new staging-shape test in `server/src/__tests__/environment-runtime-reusable-resume-replacement.test.ts` (released runner lease, no harness backup, clean committed snapshot, every resume times out) failed with the staging error `could not be resumed (cause=transport_timeout); the lease was preserved and no replacement was created.` - Review round: "keeps an idle sandbox with no harness backup when some resume calls failed for another reason", "never destroys a sandbox kept after replacement through the run or workspace teardown paths", and the retained-orphan case "keeps a warm sandbox while a replacement kept it" each failed before the fix and pass after it. `environment-retained-orphan-sandbox.test.ts`, `abandoned-workspace-files-hold.test.ts`, `workspace-recovery-preparation-gates.test.ts`, `workspace-recovery-admission.test.ts`: all pass. - Green after the fix: 22/22 pass in the replacement suite. The run gets a new sandbox within 330 seconds, makes no destroy call, the old lease stays `released` with the keep marker, and the next run resumes the new sandbox only. With unsaved work or no saved snapshot, the run still fails strictly and does not mark the old lease. Earlier tests (a refusal without a backup, a live owner, a dirty workspace with a timeout) are unchanged. - `pnpm exec vitest run src/__tests__/environment-runtime.test.ts src/services/environment-runtime-reusable-lease-probe.test.ts src/__tests__/reusable-leases-default.test.ts`: 215 pass. - `tsc -p server/tsconfig.json --noEmit` passes. - Not run: a live replacement on staging. It is verified after deploy on the next wake of the affected tasks. ## Risks - A runner task that hits this path starts a fresh runner session. The files come from the saved snapshot, and the task, comments, and run history are kept. Before this change, the task stopped and waited for a person. - The kept sandbox stays with the provider (archived storage) until someone removes it. This is the cost of never destroying unsaved runner state. - The rule needs the full extended start deadline (about 330 seconds) of timeouts, so a slow but healthy start still resumes the same sandbox. - No schema change. Old leases follow the same rule. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1113.0-canary.4 |
||
|
|
503a736195 |
feat(ci): read a registry build cache in the Runner image build (#21310)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner is the part that executes agent work: a daemon (`paperclip-runnerd`) and a provider pack of agent tools > - `.github/workflows/runner-image.yml` builds the Runner alone and publishes it from `master` as `ghcr.io/paperclipai/paperclip-runner`, one time for each set of inputs > - The build used no cache, so every run compiled the daemon again. The first publication took 25m46s from push to tag > - The toolchain and the daemon change only with the Rust sources. Most commits that change the Runner's inputs do not touch them > - This pull request lets the build legs read a registry build cache for those stages. Only a new job can write the cache, and only for a `master` push > - A weekly run builds with no cache, so the cache cannot hide a broken build for long > - The benefit is about 3 minutes less on each build leg when the Rust sources did not change, with no change to the published image ## Linked Issues or Issue Description Refs #21230 (publication of the Runner image), #21229 and #21204 (the build this change makes faster). No issue exists for this change. **What existing behavior does this improve?** The Runner image build in `.github/workflows/runner-image.yml`. It builds the same image in less time. **Current behavior** Each build leg starts from an empty builder. It installs the Rust toolchain and compiles the daemon in every run. In the first `master` publication (run 43365670428) the amd64 leg spent 7m22s in the build step. The toolchain and the cargo build took 3m26s of that. **Proposed behavior** The build legs read `ghcr.io/paperclipai/paperclip-runner-buildcache:<arch>` for the stages up to the compiled daemon. A new `cache` job writes that cache after a publication. A scheduled run builds with no cache one time each week. **Reason and benefit** The stages before the last one read the Rust sources, the protocol files and the toolchain pin. In the last 30 days, 438 `master` commits changed an input of the image, and at most 181 of them changed a file those stages read. For the other commits, and for each pull request that does not touch those files, the build step is about 3 minutes shorter on each leg. **Breaking changes** None. The `Dockerfile` and each file in the image identity are unchanged. The published digest for a set of inputs does not change. This is part 4 of 4 of a stack ([stack #21309](https://github.com/paperclipai/paperclip/stack/21309)). Merge the parts in order: 1. #21306: the resolver and the no-rebuild mode 2. #21307: the report-only job in `docker.yml` and the pull-request check of the application image's Runner path 3. #21308: reading a retained provider pack from its Runner image 4. #21310: a registry build cache for the Runner image build (this pull request) This pull request uses no code from parts 1 to 3. It is stacked on them because all four parts edit `doc/runner-image-contract.md`. ## What Changed - The build legs pass `--cache-from type=registry,ref=ghcr.io/paperclipai/paperclip-runner-buildcache:<arch>` to the first build command. They do not log in. Their permissions stay `contents: read` and `packages: read`. - A new `cache` job writes the cache with `--cache-to type=registry,...,mode=max` and `--output type=cacheonly`. It runs only for a `master` push in the canonical repository, for inputs that had no image. Its permissions are `contents: read` and `packages: write`. It has no `id-token` and no `attestations` permission. - The `cache` job needs the `publish` job, and no job needs the `cache` job. A cache write cannot delay or fail a publication. - The `cache` job builds the target `runner-artifacts-runnerd` and no later stage. The last stage takes the commit as a build argument, so no other commit can reuse it. The install and its dependency lifecycle scripts are in that stage, so the job with the write token does not run them. - The `cache` job has one concurrency group for each architecture, a job time limit and a step time limit. - A `schedule` trigger starts a weekly build. It passes `--no-cache` to the first command and reads no cache. It runs all checks and publishes nothing. A fork's schedule runs nothing. - `server/src/__tests__/runner-image-workflow.test.ts` pins the new security properties: which jobs hold `packages: write`, that the build legs have no credential and cannot write the cache, the conditions of the `cache` job, that the `publish` job does not depend on it, and that a scheduled run cannot publish or write the cache. - `doc/runner-image-contract.md` has a "Build cache" section. ## Verification Tests, on macOS (Node 26.12.0) and in a `node:24.22.3-bookworm` container (Linux arm64, Node 24.22.3): - `pnpm exec vitest run src/__tests__/runner-image-workflow.test.ts src/__tests__/runner-image-identity.test.ts` in `server/`: 55 of 55 tests pass on both. - `node --test .github/scripts/tests/*.test.mjs`: 807 of 807 tests pass on both. The container needs `jq`. - `pnpm verify` exits 2 for this change, because a workflow file is a global CI input. Required CI supplies the broad coverage. - Each new assertion fails when the workflow breaks its rule. I checked seven such edits, for example: the `publish` job needs the `cache` job, the `cache` job gets `id-token: write`, a build leg gets `--cache-to`. Cache mechanics, on linux/arm64 with a local registry, BuildKit v0.34.2 (the version the hosted runners use), and an empty builder for each build: | Build of both variants | Time | Result | | --- | --- | --- | | No cache, cache ref does not exist | 260 s | An importer error in the log, exit 0 | | One changed TypeScript file, with the cache | 149 s | Toolchain and cargo steps cached | | Unchanged inputs and a new commit, with the cache | 129 s | Toolchain and cargo steps cached | | Each stage in the cache (a whole-build cache, which this change does not write) | 16 s | Both OCI archives complete | | `--no-cache` on the first command | 272 s | No cached step in the first command. The second command reuses all of it | - The images built from the cache pass `runner-image-layout.mjs check` and `runner-image-oci.mjs check-layers`. The daemon has the same SHA-256 as the daemon of a cold build. - A cache ref that does not exist, a host that refuses the connection, a name that does not resolve, and the real `ghcr.io/paperclipai/paperclip-runner-buildcache:arm64` (which does not exist yet) each give an importer error in the log and a complete build with exit 0. - The `cache` job command on an empty builder with the cache present takes 6 s. It downloads no layer. Not tested before merge: - The `cache` job. It runs only on `master`. - The scheduled run. It starts only from the default branch. - The cache package. It does not exist. I did not test a private package, which answers an anonymous read with an authorization error. I expect the same importer error and a cold build. - A fully cached build on the hosted runners. The local test used the same BuildKit version and a different buildx version (0.37.0, and the runners have 0.38.1). ## Risks - **A second job can write packages.** Before this change, only the `publish` job held `packages: write`, and it runs no build. Now the `cache` job holds it too, and it runs BuildKit on the repository's `Dockerfile`. GitHub cannot limit that token to one package, so it can also write to `ghcr.io/paperclipai/paperclip-runner`. The limits are: the job runs only for a `master` push in the canonical repository, the token goes only to the registry login, the build steps run inside BuildKit without it, the job does not run the install, and the job cannot sign. An image that it wrote has no attestation. - **Part of a published image can come from an earlier run.** The daemon in a published image can come from a step that an earlier `master` run compiled and cached. The attestation names the run that published, not the run that compiled. BuildKit reuses a step only for the same instruction and the same file content, but a party that can write the cache package can replace what a later build reuses. Write access to the cache package must stay as narrow as write access to the image package. - **A cached download is not repeated.** The toolchain installer, the toolchain and the crates are not downloaded again until the inputs of their step change. The weekly cold build finds a download that is gone. It does not compare its output with the cache. - **A registry failure after the cache import fails the leg.** A cache that cannot be read at the start is not a failure. A layer download that fails later is a failure. Run the job again. - **The cache package starts private.** GitHub creates it on the first write. A maintainer must make it public one time. Until then each build runs cold and nothing fails. - **The first cache write waits for an input change.** The merge of this pull request changes no input of the image, so its `master` run builds nothing and writes no cache. - **A cancelled cache job marks its run as cancelled.** When three cache writes for one architecture wait at the same time, GitHub cancels the middle one. The image of that run is published. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, through Claude Code. Tool use and shell execution. It wrote the change, ran the local builds and tests, and wrote this description. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
4f4d823cfc |
feat(environments): forward the warm idle window to plugin lease acquisition (#21305)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Plugin environment drivers run agent tasks on remote computers, and warm leases keep a parked Runner between runs for an idle window that Core records > - Some providers stop an idle computer on their own schedule (for example a sandbox auto-stop interval) > - If that provider timer is longer than the Runner idle window, the provider keeps paying for a computer whose Runner is already gone; if it is shorter, a warm resume can find the computer stopped > - The plugin cannot align the two because it never sees the idle window at acquire > - This pull request forwards the warm idle window to `environmentAcquireLease` as `runnerLifecyclePolicy` > - The benefit is that a provider can stop its computer just after the Runner idle window, without Core knowing anything about the provider's placement ## Linked Issues or Issue Description Refs #21289 Refs #21066 **What existing behavior does this improve?** Warm lease setup for plugin environment drivers. **Subsystem affected** Server environment runtime (`createPluginEnvironmentDriver`) and the plugin SDK acquire parameters. **Current behavior** Core records the warm lifecycle policy (`runnerTaskLifecyclePolicy`, including `idleTimeoutMs`) on the lease, but `environmentAcquireLease` does not receive it, so a provider with its own idle shutdown has to guess. **Proposed behavior** For a warm lease, `environmentAcquireLease` receives `runnerLifecyclePolicy: { mode: "warm", idleTimeoutMs }`, the same value Core records. Other lifecycle modes do not receive the field. **Reason and benefit** A provider can align its idle shutdown with the Runner idle window. **Breaking changes** None. The field is optional and additive. ## What Changed - `PluginEnvironmentAcquireLeaseParams` has a new optional `runnerLifecyclePolicy`. - Core sends it only for warm leases. - `doc/plugins/PLUGIN_RUNNER_EXECUTION.md` documents the field. - The runner-e2e SDK public-input pin is updated for the SDK change. An earlier revision of this PR also made Core try `environmentResumeLease` before a session handoff. That part is removed: placement is the plugin's decision, so Core should not decide to retry the same computer before moving the session. The plugin will own that inside its own resume instead. ## Verification - `pnpm typecheck` in `server/` (builds shared and plugin-sdk) passed. - `vitest run src/__tests__/environment-runtime.test.ts -t "portable task placement|delegates plugin environment leases with acquisition budget"` passed (5 cases). The portable test asserts the first warm acquire receives `runnerLifecyclePolicy` and the lease records `runnerTaskLifecyclePolicy`; the non-warm delegation test asserts the field is absent. Making the forwarding unconditional fails both delegation cases. - The full suite was not run locally; required CI covers it. ## Risks - A plugin that does not know the field ignores it. - Rollback: revert this pull request. No migration. Note: the branch name still reflects the earlier scope (resume before handoff). ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
86f9555695 |
fix(native): give a Runner whose first unanswered read came on the last attempt one replacement attempt (#21322)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native Codex runs execute in a remote Runner, and a same-run retry reattaches that Runner after an interruption > - #21272 made the next attempt replace a Runner that left its recovery read unanswered, instead of adopting it again > - On staging, three runs after #21272 went live still failed: their first unanswered read came on the third and last attempt, so no next attempt existed > - This pull request gives such a run one more attempt, used only for that replacement > - The benefit is that the replacement from #21272 can run when a restart used up the middle attempt ## Linked Issues or Issue Description Refs #21272, Refs #21161 **What happened?** After #21272 ( |
||
|
|
49f0030f06 |
fix(heartbeat): rank a person's joined comment first and refill free run slots (#21277)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The heartbeat queue starts each agent's queued runs, up to the agent's `maxConcurrentRuns` > - #21234 starts person-initiated runs before system recovery runs, and it reads "person-initiated" from the run's own wake request > - When a person comments on a task that already has a queued system run, the comment joins that run. The run keeps its system wake, so dispatch ranks the person's message as system work > - Dispatch is event driven. The queued-run re-drive backstop covers only agents with no live run, so an agent with several slots keeps a free slot empty until one of its runs finishes, and runs can take hours > - This pull request makes dispatch read the person wakes that joined each queued run, and makes the re-drive backstop cover an agent that still has a free slot > - The benefit is that a person's comment starts first, and free slots fill within about a minute instead of waiting for an unrelated run to finish ## Linked Issues or Issue Description Refs #21234, #20986 **What happened?** On staging, one agent had `maxConcurrentRuns` 20, 18 running runs and 76 queued runs at 03:12Z. No run started on that agent from 03:01:33Z to 03:16:57Z (15.4 minutes). From 03:11:10Z, after a run finished, at least 2 slots were free. Other agents in the same company started runs during that time. Runs started again only after the next deploy restarted the server. Five queued runs came from a person: comments at 02:43Z, 02:44Z and 02:45Z, and Retries at 02:46Z, 02:49Z and 03:01Z. The comment runs had the wake reasons `issue_reopened_via_comment` and `issue_commented`, the invocation source `automation`, and no person on the run's own wake request. **Expected behavior** A run that carries a person's comment ranks as person-initiated, also when the comment joined an existing queued run. When an agent has a free slot and a queued run is ready, the run starts promptly, without waiting for another run on the agent to finish. **Steps to reproduce** 1. Give an agent `maxConcurrentRuns` 3. Start 2 runs on it. 2. Queue a third run while no dispatch signal reaches the queue (for example, a claim declines it while a gate is still closed, and the gate then clears). 3. See that `listUndispatchedQueuedRuns` returns nothing, because the agent has a live run. The run stays queued until one of the 2 running runs finishes. 4. Separately, queue a system run on a task, then post a person's comment on the task so that the comment wake coalesces into that queued run. Queue older system runs on other tasks of the same single-slot agent. 5. See that dispatch starts the older system runs first. **Root cause** 1. `coalesceIntoActiveExecutionRun` (`server/src/modules/wake-queue/adapters/postgres.ts`) merges the person's comment context into the queued run and records the person's wake as a separate `coalesced` row with `runId` set to that run. The run's own `wakeupRequestId` still points to the system wake. `queuedRunDispatcherForAgent` (`server/src/services/heartbeat/queue.ts`) only read the wake behind `wakeupRequestId`. The task routes do record the comment author as the requester (`requestedByActorType: "user"`). The person was lost only at dispatch time. 2. `listUndispatchedQueuedRunPage` (`server/src/services/queued-run-dispatch-liveness.ts`) skipped every agent with any `running` or `scheduled_retry` run. That rule assumes one slot per agent. For an agent with several slots, a lost or declined dispatch is not re-driven until a run on that agent finishes. This change is not in `claimQueuedRun`, the fresh-isolated preflight, the wake-queue promoter or `fresh-continuation-fallback.ts`. **Paperclip version or commit** Staging Core `ddc0aacad8` (contains #21234 `ade7e0643f`). This branch is based on master `6cb0dd0c6b`. **Deployment mode** Authenticated, public, Paperclip Cloud managed staging. **Agent adapter(s) involved** `paperclip_runner` (Codex native runtime). The fix is adapter-independent. ## What Changed - `queuedRunDispatcherForAgent` also reads person wakes whose `runId` is a queued run of the agent (by `run_id IN (queued run ids)`, so an earlier saved comment that a run adopted is included). A run with such a wake ranks as person-initiated and is never counted as system recovery. - `listUndispatchedQueuedRuns` now returns a queued run when its agent has fewer `running` plus `scheduled_retry` runs than its `maxConcurrentRuns`, and its task has no `running` or `scheduled_retry` run. The re-drive backstop and the ownership I7 check use this query. - The query counts each agent's live runs once, in one grouped subquery that is joined to the queued runs. - Migration 0476 adds `agent_wakeup_requests_company_run_idx` (`company_id`, `run_id`, partial on a non-null run) with `CREATE INDEX CONCURRENTLY IF NOT EXISTS`. It is listed in `MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS` and classifies as `online_compatible`. `doc/DATABASE.md` describes it. - Added `maxConcurrentRunsSql()` to `timer-policy.ts`. It reads the same limit as `parseHeartbeatPolicy`, including the single-run runner providers, which now share one constant. ## Verification - Red/green: `server/src/__tests__/recovery-wake-herd.test.ts` "starts a queued run that a person's comment joined ahead of older system work". It fails on master (`expected null not to be null`: the older system run started first) and passes with this change. - Red/green: `server/src/__tests__/ownership-reconciliation-report.test.ts` "re-drives a queued run while its agent still has a free slot, and not once every slot is running". It fails on master (`expected [] to deeply equal [ Array(1) ]`) and passes with this change. - Red/green: "starts a queued run that adopted a person's comment saved before every queued run". It fails with a `requestedAt` bound and passes with the `run_id` lookup. - Red/green: "counts an agent's live runs in one grouped aggregate, not once per queued run" runs EXPLAIN on the page query. It fails with a per-row correlated count and passes with the grouped subquery. - New: "uses the agent's single-run limit for a runner provider that runs one turn at a time". - Two existing tests now set `maxConcurrentRuns: 1` so that they keep testing "the agent's only slot is busy". - `npx vitest run src/__tests__/recovery-wake-herd.test.ts src/services/heartbeat/queued-run-dispatch-order.test.ts src/__tests__/ownership-reconciliation-report.test.ts src/services/queued-run-dispatch-liveness.test.ts src/services/heartbeat/dispatch-work.test.ts`: 71 passed. - `npx tsc --noEmit -p server/tsconfig.json`: no errors. ## Risks - The re-drive backstop now also covers multi-slot agents with live runs. Each queued run is re-driven at most on its existing backoff (1 minute, then doubling to 15 minutes). A queued run whose task already has a live run is not re-driven. - The ownership I7 report can now list a queued run on a multi-slot agent that has a free slot. This is the stall that this change targets. - One more indexed read per dispatch pass (`agent_wakeup_requests` by company and run id). The new index is built concurrently, so the previous release keeps writing during the build. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, read-only staging API reads in a browser tab). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
74096c19f4 |
fix(evals): admit the in-flight chat completion follow-up on a settled source (#21313)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat catalog evals check that a chat request reaches the correct saved outcome > - The `source-settled-with-active-work` run policy checks that the CEO chat is settled while delegated work can continue > - After a delegated task finishes, the product starts its normal `chat_task_completed` follow-up on the source chat > - The policy required every source run to be `succeeded` and the source to be `waiting`, so a re-grade during that follow-up failed > - This pull request admits that one product follow-up while it is in flight, and keeps every other source run strict > - The benefit is that the eval grades the saved private-delegation outcome, not the timing of the reload ## Linked Issues or Issue Description **What happened?** The OpenCode cell of `chat-features.runner-opencode.local.tasks.private-task` failed with "Independent oracle survives browser reload: expected true, received false". The saved state was correct. The CEO answered, Morgan saved the private draft in the private project, and the unrelated member could not read it. The grade passed for its full 5-second stability window. Morgan's run then finished, and 73 ms later the product started the CEO `chat_task_completed` follow-up on the source chat. The harness reloaded the page and re-graded while that follow-up was running. The only failed check was `catalog-settled-source-separate-work`, because it required all source runs to be `succeeded`. **Expected behavior** The case already counts this follow-up in its run bound (`expectedRunCount: 3`, `minimumRunCount: 2`). A queued or running product follow-up on an answered source must not fail the case. **Steps to reproduce** Run the private-task case. Let the worker finish within about 5 seconds of the first passing grade. The post-reload re-grade sees the follow-up as `running` and fails. ## What Changed - Add `chatCatalogSourceCompletionFollowUp` to `tests/runner-e2e/chat-catalog-run-policy.ts`. It identifies the lead's `chat_task_completed` run on the source whose every completion update names separate work in the run ledger. - In the `source-settled-with-active-work` check, admit at most one such follow-up while it is `queued`, `running` or `succeeded`. Accept a source `conversationState` of `active` only while that follow-up is in flight. - Keep the other rules strict. The original answer must exist and must have succeeded. Any other active, failed or extra source run still fails. The declared run-count bound still applies. - Also admit the follow-up before its turn is prepared. The server claims the run first and saves the completion updates after that, so a queued run, or a running run in preparation, has only its delivery IDs. It qualifies only while queued or running, when automation woke it, and after a separate worker run in the ledger succeeded. - Add a red/green test that reproduces the observed race and covers the negative cases. ## Verification - `npx vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/chat-catalog-run-policy.test.ts`: the new test fails before the change (1 failed, 13 passed) and passes after it (14 passed). The queued-snapshot cases were added after review. - The related runner-e2e tests (`chat-catalog-run-policy`, `chat-catalog-readiness`, `chat-catalog-failure-retirement`, `runtime-dependency`, `internal`) pass: 200 tests. - `pnpm test:e2e:runner:typecheck` passes. - Paid run [43374419951](https://github.com/paperclipai/paperclip/actions/runs/43374419951) on the first commit of this branch: `chat-features.runner-opencode.local.tasks.private-task` passed. ## Risks - Low risk. The change is in eval grading only. It does not change product code. - The check accepts a follow-up that is still running at grade time. A later failure of that follow-up is not observed by this case. Completion delivery is covered by its own completion cases. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
23c2343015 |
feat(server): read a retained provider pack from its Runner image (#21308)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner executes agent work in a sandbox. It needs a provider pack of agent tools, and the server uploads that pack when the sandbox does not have it > - A retained session keeps the pack it started with. When that pack is not the server's own, the server downloads the pack's transport archives from a registry (#19779) > - Today the server looks for them in the application image tagged `sha-<commit that built the pack>` > - `master` now publishes the Runner one time for each set of inputs, as `ghcr.io/paperclipai/paperclip-runner`. A pack from that image names the commit that published the image, and few commits get an application image, so that tag often does not exist > - This pull request reads such a pack from its Runner image, and keeps the old source for every other pack and for a Runner image that cannot be read > - The benefit is that a retained session can still get its pack when the pack came out of the published Runner image ## Linked Issues or Issue Description Refs #19779 (the retained pack source), #21228 (the pack field `runnerInputsDigest`), and #21204, #21229, #21230 (the Runner image). No issue exists for this change. **Problem or motivation** `retained-provider-pack-oci.ts` reads `ghcr.io/paperclipai/paperclip:sha-<runnerSourceRevision>`. It needs that image to exist, to carry the pack, and to have one layer with a known build-history line. For a pack from a published Runner image, `runnerSourceRevision` is the commit that published the image. `docker.yml` writes `sha-<commit>` only when its `master` run for that commit completes, and a newer push can replace a waiting run. The restart-chaos harness has the same assumption in another place: it requires the pack of a sandbox image to be built by the commit in the image's `org.opencontainers.image.revision` label. **Proposed solution** A pack that names its Runner image in `payload.runnerInputsDigest` is read from `ghcr.io/paperclipai/paperclip-runner:inputs-<digest>`. A sandbox image can name the commit that built its Runner in a label of its own. No application image carries a pack with that field yet, and no image carries that label, so behaviour today does not change. **Alternatives considered** - Keep the application image as the only source and require an application image for every Runner publication. Publication would then depend on `docker.yml`, which serves one ref at a time. - Take a fixed number of layers ("the image has four, take the last two"). A server in the field cannot be changed when the image gets another layer. The server now uses the rule that holds for any number of layers: the full image is the slim image plus transport layers. - Fall back to the application image for every failure of the Runner image. A Runner image that exists and does not match the pack is a fault, and a second source would hide it. **Roadmap alignment** This is build and runtime infrastructure. It adds no product feature and does not overlap an item in `ROADMAP.md`. This is part 3 of 4 of a stack ([stack #21309](https://github.com/paperclipai/paperclip/stack/21309)). Merge the parts in order: 1. #21306: the resolver and the no-rebuild mode 2. #21307: the report-only job in `docker.yml` and the pull-request check of the application image's Runner path 3. #21308: reading a retained provider pack from its Runner image (this pull request) 4. #21310: a registry build cache for the Runner image build This pull request uses no code from parts 1 and 2. It is stacked on them because all four parts edit `doc/runner-image-contract.md`. ## What Changed - `server/src/services/native-runtime/retained-provider-pack-oci.ts`, for a pack whose `payload.runnerInputsDigest` is 64 hexadecimal characters: - The server reads the full index `inputs-<digest>` of the Runner image. The index must carry `io.paperclip.runner.inputs-digest` equal to the pack's value, `org.opencontainers.image.revision` equal to the pack's `runnerSourceRevision`, and a valid digest in `io.paperclip.runner.slim-digest`. - It reads the slim index by that digest, never by the slim tag, and checks the bytes it gets against the digest. - For the pack's platform, the layer digests of the slim image must be the first layer digests of the full image, in the same order. The layers after them are the transport layers. There must be at least one and at most eight, and their total size has the existing limit. - The server downloads only those layers. It checks each against its descriptor and refuses every entry that is not a transport archive, the receipt or the `provider-pack.archives` directory. A file may not be in two layers. - When the Runner image cannot be read, the server reads the application image as before. That covers a registry error status for the token, for either index or for either manifest: for example 404 for a tag or a slim index that does not exist. Nothing is written before that point. - A Runner image that can be read and does not match fails with `runner_remote_provider_source_invalid`. The server does not try the application image then. - The same file, for every other pack: the old path, unchanged in what it checks. Its code moved into functions that both paths use (`request`, `pullToken`, `platformImage`, `receiveTransportLayer`). - The downloaded layer file is removed as soon as it is unpacked. Before, the importer removed it after its checks, so `retained-provider-pack-source.ts` loses that one line. - The cache record gets an optional `runnerImage` field with the inputs digest and the digests of the transport layers. `transportLayerDigest` is the last of them and `transportLayerBytes` their total. An older server ignores the new field. - `tests/restart-chaos/daytona-image.ts`: when the image has a non-empty label `io.paperclip.runner.source-revision`, the pack must be built by that commit. Otherwise the pack must be built by the commit in `org.opencontainers.image.revision`, as before. The receipt field `providerPackRunnerSourceRevision` records the pack's commit. - `doc/runner-image-contract.md` has a new section, "Reading the transport archives alone". `tests/restart-chaos/README.md` describes the label. One choice needs a reviewer's eye. When the full index can be read and the slim index cannot (a registry error status), the server falls back to the application image; it does not fail. Without the slim manifest the server cannot tell which layers are transport layers, so it has not accepted any byte of the Runner image, and the old source has its own complete checks. A slim index that is read and has other bytes than its digest still fails. ## Verification Linux (`node:24.22.3-bookworm`, GNU tar 1.34, zstd 1.5.4, the worktree mounted read-only): - In `server/`: `retained-provider-pack-source.test.ts` (34 tests, 15 of them new), `provider-pack-archive.test.ts`, `remote-provider-pack-verification.test.ts`, `remote-provider-pack.test.ts`: 127 tests pass in 4 files. - `node node_modules/vitest/vitest.mjs run --config tests/restart-chaos/vitest.config.ts tests/restart-chaos/daytona-image.test.ts`: 25 tests pass, 5 of them new. macOS (Node 26.12.0): - `daytona-image.test.ts`: 25 tests pass. - `retained-provider-pack-source.test.ts`: the 15 new tests pass. Six older tests fail, with and without this change. They run the real uploader, which needs GNU tar, and this host has bsdtar. - `tsc` on the two changed server files and the test, and on the two restart-chaos files, with the settings of their test projects: no error. The new cases: - Read from the Runner image with 6 registry requests (two indexes, two manifests, two layers), none to the application image and none for the slim tag. - Take every layer after the slim image's: one transport layer, and three. - Fall back when the tag does not exist, when the token is refused, and when the slim index cannot be reached. - Refuse, with no request to the application image: an index for other inputs, an index for another revision, no slim annotation, a slim annotation that is a tag, a slim index with other bytes than its digest, a slim image that is not a prefix of the full image, a slim image equal to the full image, nine transport layers, and a transport layer that holds a pack file. - Four deliberate faults in the reader each make a test fail: the slim index bytes are not checked, no prefix check, the slim annotation is not validated, and no bounds on the number of transport layers. A real run, in the same Linux container (`linux/arm64`), with no credential: - The script took `provider-pack/provider-pack.json` of `linux/arm64` from the published image `ghcr.io/paperclipai/paperclip-runner:inputs-0e6662ad9c8c6451556c76bf326723645d4afaab2decc73cd5e0025f7dd8d1ca`. Pack digest `sha256:604c29cb32a181843cc4d425871f88c0e65c8185b0a158c801fda4eb1bfce52f`, `runnerSourceRevision` `25e9a9c2e9277f3be8649be8c809cc81bde6f0c5`. - `resolveProviderPackUploadSource`, with another pack as the server's own, returned `{ archiveOnly: true }` after 23 seconds. That function calls `receiveRetainedProviderPack` and then the importer's checks: the manifest inside the overlay archive is the selected manifest, every archive matches the receipt, and the content of every file archive matches the digest of its file. - The server code made 9 requests: one anonymous token request, the full index, the `arm64` manifest, the slim index `sha256:f63ea436…4f3c` by digest, its `arm64` manifest, and two layers, each with one redirect. No request went to the application image. - The cache record: ```json { "schema": "paperclip.retained-provider-pack-source.v1", "sourceRevision": "25e9a9c2e9277f3be8649be8c809cc81bde6f0c5", "providerPackDigest": "sha256:604c29cb32a181843cc4d425871f88c0e65c8185b0a158c801fda4eb1bfce52f", "imageIndexDigest": "sha256:84f2c051aa4ed16c5cc138c08e3533fa417e92177f7d15af5f2aa3e0ba467162", "imageManifestDigest": "sha256:73dbe864fbab6208616f0aa6f1610a7ece4ef22f68a73d30901cbaea60139589", "imageConfigDigest": "sha256:740e48463d5caf0bfead4b923cdbd8b220acb93d058419c3db9d00c7f1831d65", "transportLayerDigest": "sha256:35433b9f788641666b470f41c37fef9fc9b872262b7edd5bd79f0e288f75bcdf", "transportLayerBytes": 426602696, "runnerImage": { "inputsDigest": "0e6662ad9c8c6451556c76bf326723645d4afaab2decc73cd5e0025f7dd8d1ca", "transportLayerDigests": [ "sha256:6a340088ca2cfa85f10f0385d9c8f5cdccb22c3a2bbb2062620b1d90616ba5b5", "sha256:35433b9f788641666b470f41c37fef9fc9b872262b7edd5bd79f0e288f75bcdf" ] }, "receiptSha256": "2afdd26645845369d9e851b1e6e2884eb6c3884ea7c4866a1eeb2efb9fb230d7", "verifiedTarPaths": true } ``` - `uploadVerifiedProviderPack` then installed the pack from that source through a local stand-in for the sandbox, with no further registry request, in 4 seconds. The installed tree is 1,249,060,841 bytes. Its `provider-pack.json` is the selected manifest, all 7 artifacts of the manifest have the digest the manifest names, and the pack's own Node prints `v24.22.3`. Never run: - The same real run for `linux/amd64`, whose transport layers are about 1 GB. - A session in a real sandbox. The real run used a local stand-in for the sandbox. - The fallback against the real registry. Fixtures cover it. - The full test suite and the full typecheck. CI must supply them. ## Risks - The old path was moved into shared functions. Its checks are the same, and its 19 existing tests pass on Linux, but a review must read that code as changed. - The layer file of the old path is now removed before the importer's checks, not after them. The importer does not read it. - The reader depends on two rules of the image contract: the full index names its slim index, and the full image starts with the slim image's layers. A Runner image that breaks either rule is refused, with no fallback. - The fallback covers a registry error status only. A network error ends the import, as it does today. - The label `io.paperclip.runner.source-revision` is new and no image carries it yet. The change to the sandbox image that sets it must use this name. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude Code with tool use (shell, file edits, local Docker builds). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
0477f622d6 |
fix(recovery): keep the sandbox that holds abandoned unsaved files out of replacement and cleanup (#21304)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents run in remote sandboxes, and the workspace files live in those sandboxes until a save > - When a run cannot resume exactly, the fresh continuation fallback (#21135) abandons the old workspace epoch and keeps its unsaved files "recorded and unapplied" on the old sandbox > - The abandonment clears the dirty flag, the writer and the location, so the workspace looks clean. No code reads the abandonment receipt > - A later failed resume can then replace the sandbox, and cleanup can destroy it. That sandbox holds the only copy of the unsaved files > - This pull request adds one shared hold. The task continues on a new sandbox, and the old sandbox is parked until its files are saved, discarded, or kept for 7 days > - The benefit is that the fallback keeps its promise. Abandoned files are not lost ## Linked Issues or Issue Description Refs #21135 Refs #21297 **What happened?** After the fresh continuation fallback abandoned a workspace epoch, the receipt in `remoteWorkspaceAbandonedEpochs` was write-only. `reusableSandboxReplacementIsSafe` saw a clean workspace with a committed save and allowed replacement. `retireReplacedReusableSandboxLease` then moved the old lease to `pending_cleanup`, and the cleanup sweep destroyed the sandbox with the unsaved files. `nativeWorkspaceExportPendingForLease` (the check that the sweep, the retained orphan settlement and the scoped destroy paths use) also answered "nothing pending". **What did you expect?** A sandbox that holds abandoned, unsaved files stays until the files are saved, a person discards them, or a stated retention period ends. **Steps to reproduce** 1. Abandon a remote workspace epoch whose location names sandbox L. 2. Let the next run fail to resume L. 3. The replacement decision says safe, L goes to `pending_cleanup`, and the sweep destroys it. ## What Changed - New `server/src/services/remote-workspace-abandoned-files-hold.ts` holds the single predicate for "this lease holds abandoned files". It is one SQL fragment (`abandonedWorkspaceFilesHeldSql`) plus a TypeScript wrapper (`abandonedWorkspaceFilesHeldOnLease`). A lease is held while a workspace in its company has an abandoned-epoch receipt whose `location.providerLeaseId` equals the lease's provider lease id, and that receipt has no `released` object. A receipt written by the current release (no new fields) counts as unreleased. - Replace and park. A failed resume of a held sandbox still lets the run replace it, so the task continues at once on a new sandbox restored from the last committed save. `retireReplacedReusableSandboxLease` parks the old lease as `retained` with the `abandonedWorkspaceFilesParked` marker (failure reason `abandoned_workspace_files_held`) instead of `pending_cleanup`. `reusableLeaseCanBeResumed` and `reusableLeaseCanBeCleanedUp` skip a parked lease, so later runs never select it again. The check is a metadata read with no query, so the run path does no extra work. - Every path that can destroy a sandbox now honours the hold: - `nativeWorkspaceExportPendingForLease` answers pending for a held lease. This covers the pending cleanup sweep (defer), the orphaned active lease sweep, the retained orphan settlement `heldBy`, `destroyClaimedSandboxLease` (terminal workspace reaper `destroyRemoteWorkspaceLease`, `destroyReusableSandboxLeases` and the environment delete path, and run lease release with `destroy`), the `destroyRunLease` dispatcher, and the plugin sandbox teardown retry. - The run lease settlement (`environment_run_lease_active`) is the one caller that ignores the hold (`{ destructive: false }`). It never destroys a sandbox. It stops the sandbox and keeps it, or it moves a lease without a sandbox to `pending_cleanup`, where the sweep parks a held lease. If it waited for the hold, a dead run's lease would stay `active` and block the retry for the whole retention period. - `destroyReusableSandboxLease` parks the lease and returns without a provider call. This covers obsolete reusable lease cleanup, the destroy after an expired or mismatched resume, release with `expired`, and the sandbox driver's `destroyRunLease`. - The `retryPendingSandboxTeardown` dispatcher refuses a destructive teardown of a held lease. Stop-only cleanup still runs, because a stop keeps the filesystem. - The retained orphan listing (`alwaysHeldSql`) uses the same SQL fragment, so held orphans are not polled. - Release. `releaseAbandonedWorkspaceFiles` marks a receipt `released` with a reason (`recovered`, `discarded`, `retention_expired`) and an actor. It moves the parked leases for that sandbox to `pending_cleanup` when no other unreleased receipt holds them, and it writes a `workspace.abandoned_files_released` activity entry on the task in plain language. No export path for abandoned epochs exists today. The `recovered` reason is the hook for one; this pull request adds no new export mechanism. - Retention expiry. New receipts carry `retainUntil` (7 days). An old receipt uses `abandonedAt` plus 7 days. `nextEnvironmentCleanupAt` includes the earliest retention end of any unreleased receipt. The shared cleanup worker therefore wakes at that real deadline, and a restart rebuilds it from the database. At the deadline the sweep runs `expireAbandonedWorkspaceFileHolds`. This releases each due receipt as an attributed system decision (actor `workspace-retention`) and logs the held receipts and parked leases for each company. Each expiry always writes a task entry: "Unsaved files from an interrupted run were kept for N days, and their retention period has ended." The entry adds "They are scheduled for removal." only when a parked lease was queued for cleanup. A separate entry, "... kept for N days and have now been removed.", is written only after the provider confirms the delete. Each receipt is released in its own transaction. A failed receipt is logged and skipped, and it is retried after a backoff (5 minutes, doubling, at most 6 hours). The other receipts are still processed. The entries reference a run or a task only if it still exists. If the task was deleted, the decision goes to the server log only. - Upgrade. A `pending_cleanup` lease (not in flight, not stop-only) that an unreleased receipt holds goes back to `retained` and is parked by the sweep. A sandbox that a release before this change already destroyed cannot be recovered. Its receipt ends at its retention time like any other receipt. - Receipt trim (review P1). `abandonUnverifiedRemoteWorkspaceEpoch` trims only released receipts and keeps the last 20 of them. It never drops an unreleased receipt. A new abandonment of the same sandbox and root replaces the unreleased receipt for that root. So a workspace holds at most one unreleased receipt for each distinct sandbox and root abandoned in the last 7 days. ## Verification - `server/src/__tests__/abandoned-workspace-files-hold.test.ts` (19 tests). It follows the lease through the real acquisition path with a failed resume, the real `abandonUnverifiedRemoteWorkspaceEpoch`, the real `sweepPendingCleanupLeases`, and `settleRetainedOrphanSandbox`: - `gives the next run a new sandbox at once and parks the held one, which cleanup and later runs leave alone` - `pulls a held lease already queued for cleanup back to retained, and leaves it on the provider` - `treats a receipt written before release fields existed as unreleased` - `records a retention expiry on the receipt` - `hands the parked lease to normal cleanup once the files are recovered` / `... discarded` - `removes the files only after the retention period, and says removed only after the provider confirms it` - `does not report removal when the provider delete fails` - `records the end of retention without scheduling removal when another receipt still holds the sandbox` - `records the end of retention when it expires before the lease was ever parked` - `wakes the cleanup worker at the receipt's retention end, also after a restart, and then goes quiet` - `releases a receipt whose run was deleted and records it without the run reference` - `releases a receipt whose task was deleted without writing a task entry` - `keeps expiring later receipts when one fails, and retries the failed one after a backoff` - `keeps a held sandbox when the retained orphan settlement reviews it` - `lists the sandbox for the retained orphan settlement again once the receipt is released` - `keeps holding the first sandbox after many later abandonments on another sandbox` - `replaces and cleans the sandbox as before when the receipt names a different sandbox` - `replaces and cleans the sandbox as before when no epoch was abandoned` - Red on the base revision ( |
||
|
|
e363b0e52d |
fix(docker): keep local build output out of the Runner image's build context (#21312)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner is the part that executes agent work. `master` publishes it alone as the image `ghcr.io/paperclipai/paperclip-runner`, one time for each set of Runner inputs (#21204, #21229, #21230) > - The name of a published image is a digest of every file the Runner build receives from the build context > - `.dockerignore` keeps the Runner's own build output (`packages/paperclip-runner/dist`) out of the context, but not the eval kernel's (`packages/paperclip-eval-kernel/dist`), which the Runner's build also writes > - A tree where the Runner was built therefore has another inputs digest than a clean checkout of the same commit > - The Runner package's Vite builds write five more directories with the same problem > - This pull request adds the missing ignore rules, and a test that finds the next such directory > - The benefit is that a developer's tree and a clean checkout of the same commit name the same Runner image ## Linked Issues or Issue Description Refs #21204, which added the image identity. No issue exists for this change. **Problem or motivation** The inputs digest covers every file the build receives under `packages/paperclip-eval-kernel`. The Runner's build compiles the eval kernel and writes `packages/paperclip-eval-kernel/dist`. After a local Runner build, that directory is in the build context, so the digest of the tree differs from the digest of a clean checkout. A lookup of the published Runner image for such a tree finds nothing. CI checkouts are clean, so no published image is affected. **Proposed solution** Add `packages/paperclip-eval-kernel/dist` and `packages/paperclip-runner/dist-*` to `.dockerignore`, next to the rule for the Runner's own `dist`. Add a test that derives every build-output directory inside the image's inputs and requires a rule for each. **Alternatives considered** - Exclude the directory in the identity script only. Then the digest would not describe what the build receives, which is the property the identity is built on. **Roadmap alignment** This is build infrastructure. It adds no product feature and does not overlap an item in `ROADMAP.md`. This pull request stands alone. It is based on `master` and needs no other open pull request. ## What Changed - `.dockerignore`: two rules. - `packages/paperclip-eval-kernel/dist`: the Runner's own build writes it. - `packages/paperclip-runner/dist-*`: the output of the Runner package's Vite builds (`dist-browser`, `dist-issue-thread`, `dist-scenarios`, `dist-sdk`, `dist-standalone`). The Runner's `build` script does not write them, but a tree where one of those builds ran had the same digest drift. No tracked path matches the rule, and no later `!` rule re-includes those directories. - `server/src/__tests__/runner-image-identity.test.ts`: one new test. For each package among the identity's inputs it collects the `outDir` of every tsconfig a script compiles with `tsc -p`, the `target` directory beside every Cargo manifest a script names, and the `outDir` of every Vite configuration. It requires that the last `.dockerignore` rule matching each directory is an exclusion. Today that is eight directories. A new output directory inside the inputs fails the test until it has a rule. - No image build reads that directory from the host. The Runner stages, the `build` stage of the application image and `docker/daytona-runner` all compile the kernel themselves (`tsc -p`, not incremental), and the eval kernel is a development dependency of the Runner only. ## Verification The inputs digest, as the `runner-artifacts-identity` build target reports it, on a `linux/arm64` Docker host: | Tree | Without the rule | With the rule | | --- | --- | --- | | clean export at `62d673cc3f` | `0e6662ad…` | `0e6662ad…` | | the same, with an eval-kernel `dist` | `e4cf3095…` | `0e6662ad…` | | clean export at `bc71c23e0a` | `88fe2929…` | `88fe2929…` | | the same, with an eval-kernel `dist` | `ac331275…` | `88fe2929…` | | clean export at `731b52453c` | `88fe2929…` | `88fe2929…` | | the same, with two files in `dist-browser` | `44fe867d…` | `88fe2929…` | | the same, with one file in each of the five Vite directories | not measured | `88fe2929…` | - `0e6662ad…` is the digest of the image that `master` published at `25e9a9c2e9`, so the rule does not change the digest of a clean checkout. - A worktree after a real `pnpm --filter @paperclipai/paperclip-eval-kernel build` also gave the clean digest with the rule. - `runner-image-identity.test.ts` (42 tests) and `tests/runner-e2e/daytona-image.test.ts` (19 tests), which reads `.dockerignore`, pass on macOS and in `node:24.22.3-bookworm`. - The new test fails on the `.dockerignore` of `master` and names `packages/paperclip-eval-kernel/dist`. It fails without the `dist-*` rule and names `packages/paperclip-runner/dist-browser`. It fails without either of the two older Runner rules, and with a trailing `!` rule that re-includes one of the directories. Never run: - Any image build with the new rules beyond the identity target, including `.github/docker-context-checks.Dockerfile`. On a CI checkout the directory does not exist, so the rules remove nothing there. - `linux/amd64`. - The full test suite and the full typecheck. CI must supply them. ## Risks - **The sandbox image's content id changes one time.** `.dockerignore` is an input of that id (`tests/runner-e2e/daytona-image-content.ts`). At `62d673cc3f` the id goes from `be208823…` to `1e13a151…`. The next end-to-end run builds a new sandbox image, and the restart-chaos harness needs an image built from this tree. The image's contents do not change. - Low otherwise: the rules remove directories that no image build reads. The `Dockerfile`, `docker/daytona-runner/Dockerfile` and `.github/docker-context-checks.Dockerfile` name none of them and run no Vite build. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude Code with tool use (shell, file edits, local Docker builds). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
bbafa78fe2 |
fix(retry): offer Retry only when admission can take it, and start held saved-work Retries (#21315)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run fails and its saved work is held for recovery, the task
shows a Retry button. Admission then decides if the Retry can start a
new turn.
> - The task view, the task's execution blocker, and admission each
computed their own answer. The task view said `canRetry: true`. The
blocker said `canRetry: false`, and its own `workspaceRecovery` part
said `canRetry: true`.
> - Admission refused these Retries with no reason. The receipt kept the
generic `execution_recovery` wait ("Retry is waiting for execution
checks"), and each later click joined the same held receipt. Nothing
ran.
> - The refusals had real causes. A preparation proof rejected
host-written workspace notices in the run log. A repair proof did not
accept a snapshot that the failed preparation only restored and never
changed.
> - This pull request fixes both proofs, names every admission refusal
of a held Retry, and makes the task view and the blocker read one shared
Retry answer.
> - The benefit is that Retry starts work in these cases, and Retry is
offered only when admission can take it.
## Linked Issues or Issue Description
**What happened?**
On hosted staging, several blocked tasks showed a Retry button. Pressing
Retry was accepted but nothing ran. The task said "Retry is waiting for
execution checks. Your message is saved." The task view had
`interruption.canRetry: true` and `retryWait.reason:
execution_recovery`. The task's `executionBlocker` had `canRetry:
false`.
- Saved-work recovery failed with `source_unavailable`. The legacy
preparation had failed before the provider started
(`executionRecovery.providerWorkStarted: false`), its sandbox epoch was
abandoned, and its sandbox was destroyed. The blocker contradicted
itself: `workspaceRecovery.canRetry: true`, top-level `canRetry: false`.
- Saved work showed "recovered", but `workspaceRepairRequired` stayed
`true`. The failed preparation had restored a snapshot that an earlier
run produced. The repair proof accepted only a snapshot that the run
itself saved.
**Expected behavior**
The task view, the execution blocker, and admission give the same Retry
answer. A Retry that admission can take starts work. When saved work
cannot be recovered, Retry continues in a fresh session from the
recorded work. Every admission refusal of a held Retry has a named
reason. No Retry is offered for a refusal that another click cannot
change.
**Steps to reproduce**
1. Let a legacy `paperclip_runner` preparation fail before the provider
starts, after the host writes its workspace notices ("Execution
workspace reuse freshness action ...", "Skipping saved session resume
...") to the run log.
2. Let saved-work recovery end with `source_unavailable` and
`use_alternative_strategy`.
3. Press Retry.
4. Observe that the Retry receipt is held with `executionWait.reason =
execution_recovery` and no run starts.
Or:
1. Let a legacy preparation restore the task's saved snapshot and fail
with `setup_failed` before the provider starts.
2. Let saved-work recovery complete.
3. Observe that the blocker keeps `workspaceRepairRequired: true`, and
that Retry is held with `execution_recovery`.
**Paperclip version or commit**
Staging on `origin/master` `62d673cc3f` (after #21267 and #21295).
**Deployment mode**
Hosted cloud (staging), remote Daytona sandbox runner.
**Agent adapter(s) involved**
`paperclip_runner` (legacy runtime runs, native agent).
Refs #21267, #21295.
## What Changed
- `hasVerifiedPreparationLog` (`verified-preparation-log.ts`) now
accepts a log made only of host notices that the host writes before it
hands the turn to an adapter: the mentioned-skills line (first record
only, checked against the saved skill selection) and the workspace
freshness and session-reset notices. Every record must have the same
attempt, sequential `seq`, `stdout`, and the excerpt and watermark must
match. Any other line still keeps the run held.
- `verifiedUnchangedRestoredSnapshot`
(`remote-workspace-snapshot-reuse-receipt.ts`) proves that a failed
preparation's saved state is the snapshot it restored. It needs
`executionStage: preparing`, `providerWorkStarted: false`, exactly one
succeeded restore of that snapshot at that epoch and lease, and no
snapshot save by the run. `verifiedWorkspaceRepairReceipt` uses it next
to the existing reuse proof.
- `savedWorkOffersRetry` (`workspace-recovery-receipt.ts`) is the one
rule for Retry on a saved-work hold. Retry is offered when the saved
work is recovered and verified, or when recovery is exhausted in a state
that only a Retry restarts (a new recovery generation, or the
fresh-session fallback after `source_unavailable`). Retry is not offered
while recovery re-arms by itself (a stop recheck or a newer capture
policy), or while a copyback owner's stop is unconfirmed. The blocker
uses this rule for the top-level `canRetry` and for
`workspaceRecovery.canRetry`, so the two parts cannot contradict.
- `task-retry-eligibility.ts` lists the held-Retry refusals that another
click cannot change. The task view and the blocker both use it.
- Admission (`explicit-native-continuation.ts`) names each refusal of a
held request that was silent before: `previous_execution_unverified`,
`execution_settling`, `cleanup_pending`, and `remote_cleanup`. It names
a refusal only when an execution blocker holds the task, so a request
without a hold still takes the normal path.
- The task view now reads the blocker's answer for a task with a
saved-work hold. It shows no Retry when the blocker offers none, and it
names the recovery wait.
## Verification
- Red/green: `cd server && npx vitest run
src/services/saved-work-retry-eligibility.test.ts`. The tests fail on
`origin/master` sources and pass with this change. They are built from
the staging shapes:
- offers the same Retry in the task view and the blocker when saved work
is no longer available
- offers no Retry in either view while saved-work recovery waits for an
unconfirmed file copy to stop
- continues in a fresh session from the recorded work when saved work is
no longer available
- verifies the repair of a restored snapshot that the failed preparation
never changed
- keeps the repair unverified when the failed preparation may have
changed the restored snapshot
- names a refusal that another click cannot change and offers no Retry
for it in either view
- names the admission refusal of a held Retry and offers Retry only when
admission would take it
- `npx vitest run src/services/verified-preparation-log.test.ts`: new
cases accept the host notices and keep a provider line, a different
attempt, or a late skills line held.
- Related suites: deferred execution wait, execution blocker read,
explicit native continuation, governed wait runner loss, legacy
preparation retry hold, remote workspace snapshot reuse, task
interruption presentation, task retry, and all `workspace-recovery-*`
suites.
- `npx tsc --noEmit -p server/tsconfig.json` passes.
## Risks
- Medium-low. The preparation-log proof accepts more host lines. Each
accepted line has a fixed host format, and the run must still pass every
other preparation proof (events, leases, no provider evidence).
- The restored-snapshot proof applies only when the run recorded that
the provider never started and it never saved its own snapshot.
- Admission now records a named wait instead of the generic one for some
held requests. It changes no admission decision.
- The task view calls `getExecutionBlocker` for tasks that offer Retry
and have a saved-work hold. These tasks are few, so the extra reads are
small.
- Not covered here: a copyback owner on a replaced deployment still
needs its stop confirmed before saved work can be copied out of a full
sandbox. The task view now shows that wait instead of a Retry that
admission refuses.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edit, browser for read-only staging API
reads).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
b555087f0f |
fix(native): continue again when a deploy interrupts an automatic continuation (#21314)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native Codex runs execute in a remote Runner (Daytona) that the server controls; a deploy restarts the server while runs are in flight > - When the restart's reattachment proves the Runner process gone mid-turn, #21039 makes the server request one automatic Retry that continues in a fresh session from the saved work, without replaying the turn > - Staging deploys every 20 to 45 minutes, and Codex runs often take longer, so that automatic continuation is often interrupted by the next deploy too > - The hook treated that second loss as "automatic Retry already used" and left the task blocked for a person (`native_continuation_requires_reconciliation`) > - This pull request gives an automatic continuation one more fresh-session continuation each time a server restart or deploy took its controller, and a deploy spends no automatic budget > - The benefit is that a task interrupted by repeated deploys continues by itself, and no provider turn that may have run is sent again ## Linked Issues or Issue Description Refs #21039, Refs #21052, Refs #21272, Refs #21023, Refs #21120, Refs #21161 **What happened?** On staging, the largest failure class for a Codex agent on Daytona was `runner_remote_process_identity_unavailable` with error code `native_continuation_requires_reconciliation`: 18 runs in 3 hours. The error name is misleading. It is the shared code for every refused remote reattachment (`REMOTE_RUNNER_RECOVERY_ERROR_CODE`). In every sampled run the refusal was `process_verification` / `process_not_running`: the Runner process was gone, its stderr log was empty, and the last turn event was `turn.submitted` or `turn.accepted` (unsettled). It was not a missing identity file. Sampled 12 such runs. In 9 the #21039 hook worked: it recorded `native.interrupted_turn_outcome_unknown` and requested the fresh-session Retry. The tasks that stayed blocked had one shape: 1. A run lost its Runner after a deploy. The hook requested the automatic Retry (key `task-retry:restart-interrupted-turn:<run>`). 2. The Retry run started a fresh session. The next deploy came during its turn (a graceful `SIGTERM` shutdown transition, then "Recovering the existing native runner process after server restart"). 3. The same-run reattachment found that Runner gone too (`process_not_running`). 4. The hook returned `automatic_retry_used`, because the run's Retry intent came from the hook. The task stayed blocked. Examples: a `task_retry` run created 40 s after its predecessor failed (the hook's Retry), interrupted by the 05:17 deploy, failed at 05:18, task blocked. A second chain (assignment, then the hook's Retry) failed the same way at 05:12. **Expected behavior** A deploy is not the task's failure. When a server restart or deploy took the controller of an automatic continuation and its Runner is proven gone, the server continues the task again in a fresh session from the saved work. It never replays the turn. A loss without restart evidence keeps the existing one-time rule. **Steps to reproduce** 1. Start a long Codex run in a remote Daytona Runner. 2. Deploy the server during its provider turn so that the reattachment finds the Runner gone. The server requests the automatic Retry. 3. Deploy again during the Retry run's turn so that its Runner is also lost. 4. See the Retry run fail with `runner_remote_process_identity_unavailable` and the task stay blocked with no new run. **Paperclip version or commit** Staging Core `25e9a9c2e9`. Fixed on `master` after `731b52453c`. **Deployment mode** Hosted (staging), remote Daytona sandbox Runner. **Agent adapter(s) involved** Codex (native Runner, `paperclip-runnerd`, `codex_app_server`). ## What Changed - `restart-interrupted-turn.ts`: `lostControllerToRestart` finds restart evidence on the run: a `native.recovery.transition` with `restartKind: "graceful"` (shutdown), or a restart claim (`reattach_remote_runner`, `reattach_existing_runner`, `resume_dead_runner`, `bootstrap_incomplete`) without the `same_run_retry` trigger. - Both `automatic_retry_used` refusals (a person's Retry that the hook requested, and a system retry of owed work) now continue when the run has restart evidence. A deploy interruption spends no budget, so a task that only deploys interrupt never runs out. Each continuation needs a new deploy during its run, so deploys bound it. A loss without restart evidence keeps the one-time rule. All other gates are unchanged: proven Runner exit, unknown-turn record, the issue still assigned and open, idempotency per run, and the Retry's admission gates. - `doc/execution-semantics.md` describes the rule. ## Verification - Red before the fix, green after: `server/src/services/runner-lost-mid-turn-continuation.test.ts` › "an automatic continuation that the next deploy interrupted". The test seeds the staging sequence: a lost run, the hook's Retry run with a `turn.submitted` turn, the shutdown and restart transitions with staging payloads, then a same-run reattachment that proves the Runner gone. On the old code the two continuation tests return `automatic_retry_used`. With the fix they request one fresh-session Retry, record the unknown turn outcome once, recover the saved work and queue the successor with `forceFreshSession: true`. - A chain of 12 deploy continuations, each with a system retry that inherits its Retry receipt, still continues (red on the first revision of this PR, which had a budget of 8). Losses without deploy evidence stay refused (with and without earlier deploy continuations). - `npx vitest run src/services/runner-lost-mid-turn-continuation.test.ts` 23/23 pass. `governed-wait-runner-loss.test.ts` and `workspace-recovery-retry-copyback.test.ts` 103/103 pass. - `pnpm --filter @paperclipai/server typecheck` passes. - `pnpm verify` exits 2 (a global doc input changed), so required CI supplies the broad coverage. ## Risks - Low to medium. A task whose Runner is lost on every deploy now continues after each deploy instead of once. Each continuation is a full run, so it costs tokens. The deploy count bounds it, because each continuation needs a new restart during its run. - Not fixed here: why the reattached Runner is gone after a deploy. In the sampled runs the reattachment after the restart hung in `native session provider recovery timed out after 900000ms`, and the next attempt found the process gone with empty stderr. That needs its own investigation. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use (shell, browser read-only API GETs on staging). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
03f9b83e52 |
feat(ci): report the Runner image on master and build the app image's Runner path on pull requests (#21307)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner is the part that executes agent work, and `master` now publishes it alone as the image `ghcr.io/paperclipai/paperclip-runner` > - The application image will soon copy that Runner instead of compiling its own. Part 1 of this stack adds the resolver that picks the Runner for a build > - Two gaps make that next change risky. The resolver has never run in CI on `master`, so its lookup time and how often a commit has a published Runner are unknown. And `docker.yml` has no pull-request trigger, so a change to how the application image gets its Runner first runs after it merged > - This pull request adds a report-only job that runs the resolver on every `master` push, and a pull-request check that builds the Runner path of the application image > - The benefit is real data from `master` before any build depends on the resolver, and a failed check on the pull request instead of a failed image build on `master` ## Linked Issues or Issue Description Refs #21204, #21229 and #21230 (the Runner image). No issue exists for this change. **Problem or motivation** The resolver decides whether a build copies the published Runner. On `master`, a commit that changes the Runner's inputs is ahead of its publication for some minutes, and nobody knows yet how often an application build will meet that. The application image itself is built only on `master` and on tags. The change that removes the Runner's compile stages from that image will edit the `Dockerfile`, and today nothing builds those stages on a pull request. **Proposed solution** Add one job to `docker.yml` that only reports, and one new workflow that only checks. Neither changes what an image contains, and no existing job reads either of them. **Alternatives considered** - Run the resolver inside the build legs. A failure there could fail the image build. A job of its own, with `continue-on-error`, cannot. - Give the lookup the job token. The Runner package is public, so an anonymous lookup works and the job needs no credential. - Build `production` in the pull-request check. It adds three `COPY` instructions and a readability check to the two stages that this check builds, at the cost of the tool layer and two more copies of several gigabytes. - Trigger the check on every Runner source change. `runner-image.yml` already builds the Runner on those pull requests, and CI builds the server and the UI. **Roadmap alignment** This is build infrastructure. It adds no product feature and does not overlap an item in `ROADMAP.md`. This is part 2 of 4 of a stack ([stack #21309](https://github.com/paperclipai/paperclip/stack/21309)). Merge the parts in order: 1. #21306: the resolver and the no-rebuild mode 2. #21307: the report-only job in `docker.yml` and the pull-request check of the application image's Runner path (this pull request) 3. #21308: reading a retained provider pack from its Runner image 4. #21310: a registry build cache for the Runner image build ## What Changed - `.github/workflows/docker.yml`: new job `runner_image_report`. - It runs only for a `master` push in this repository. - It runs `runner-image-resolve.mjs inputs`, then `resolve`, and the result goes to the job summary with the line "Report only: no build read this result." - It has `continue-on-error: true`, declares no `outputs`, and no job needs it. Its failure does not fail the run. - It holds no credential. Its only permission is `contents: read`, the checkout does not keep its token, and the lookup is anonymous. It does not log in to a registry and writes to none. - Time limits: 8 minutes for the job, 5 for the step that builds, 2 for the lookup. - It is the first job in the file. Existing tests cut the other jobs out of the file by name, up to the next job or to the end of the file, so no other place leaves those tests unchanged. - `.github/workflows/docker-app-check.yml` (new): on a pull request, builds the `build` and `runner-provider-pack` targets with `PAPERCLIP_BUILD_COMMIT` set, for `linux/amd64` only. - `permissions: {}` at the top, `contents: read` for the job, no credential, no `pull_request_target`. - It exports nothing (`--output type=cacheonly`) and pushes nothing. It reads the public build cache that `docker.yml` writes and never writes it. - It refreshes the lockfile in the build context first, as `docker.yml` does. - Time limits: 60 minutes for the job, 45 for the two builds. - Trigger paths, and why: | Path | Reason | | --- | --- | | `.github/workflows/docker-app-check.yml` | the check itself | | `Dockerfile`, `.dockerignore` | the recipe, and the files it receives | | `server/scripts/prepare-runner-vendor.mjs` | the step of `build` that reuses or rebuilds the Runner | | `packages/paperclip-runner/scripts/build-inputs.mjs` | the build stamp that step compares with the files the image build received | - `server/src/__tests__/docker-runner-workflows.test.ts` (new, 90 lines): only security properties. The report job has no credential and one read permission, nothing can read it, and its two steps have time limits. The check has no credential, cannot publish, and exports nothing. In both, every action is pinned by commit and no shell script holds an expression. - `doc/runner-image-contract.md` and `doc/DOCKER.md` name the job and the check. ## Verification macOS (Node 26.12.0): - In `server/`: `docker-runner-workflows`, `runner-image-workflow`, `docker-build-stamp`, `container-init-reaping`, `cloud-image-bundled-plugins`, `docker-image-layer-reuse` and `cloud-image-sentry` tests: 52 tests pass in 7 files. - `tests/runner-e2e` workflow tests (`workflow-security`, `private-trust-workflow-security`, `workflow-image-only`, `paid-runner-routing`): 27 tests pass in 4 files. - `node --test .github/scripts/tests/*.test.mjs scripts/merge-group-image.test.mjs scripts/reuse-canary-image.test.mjs scripts/cloud-source-verification.test.mjs scripts/standard-image-contract.test.mjs scripts/__tests__/release-verify-workflow.test.mjs`: 962 of 962 pass. - Both workflow files parse as YAML. - `docker buildx build --target runner-artifacts-context --cache-from type=registry,ref=ghcr.io/paperclipai/paperclip:buildcache-amd64 --output type=cacheonly .` succeeds, so both flags are accepted and the cache is readable. Linux (`node:24.22.3-bookworm`, with `zstd` and `jq` added): the same three sets pass: 52, 27 and 962 of 962. The two commands the report job runs were run by hand, with no credential, on clean exports of the tree. For the inputs `0e6662ad…d1ca` the answer was `published`, with the two index digests that `master` published. For a commit whose image was not yet published the answer was `inline`, `absent`. Part 1 has the output. With the job at the end of `docker.yml`, two existing tests failed. They are the reason for its place: `docker-canary-promotion.test.mjs` reads the last job to the end of the file, and `release-verify-workflow.test.mjs` allows no BuildKit configuration after `merge-and-push`. Never run: - Either workflow. Only GitHub can run them. This pull request starts the check by its own path; the report job first runs on `master`. - The two builds of the check (`--target build`, `--target runner-provider-pack`) on a local host. Their time and disk use on a hosted runner are not known. - The full test suite and the full typecheck. CI must supply them. ## Risks - The check builds the whole `build` stage, with the UI and server builds. Its duration is not measured. It runs only when one of five paths changes, and its first real run is on this pull request. - `docker.yml` holds one concurrency slot for each ref until all its jobs end. The report job normally ends long before the build legs. A stalled job can hold the slot for at most 8 minutes when the legs end sooner. - `scripts/watch-release-source.mjs` warns about every job of the image run that waited more than 60 seconds for a runner. The report job is one more job that it can name. - `scripts/merge-group-image.mjs` refuses to promote a merge-queue image for a push that changes `docker.yml`. The push that lands this change is such a push. The merge queue is not enabled. - The lookup has 2 minutes. The registry client retries a failing request for longer than that, so a registry outage ends the step by its time limit, and the job then reports nothing. The run stays green. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude Code with tool use (shell, file edits, local Docker builds). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
ff91bfe2dc |
fix(runner): withhold Claude's self-scheduling tools from Paperclip ACPX sessions (#21276)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Paperclip owns scheduling. An agent uses Paperclip tools such as task monitors and routines to continue work later. > - Paperclip Runner runs Claude Code through the ACPX sidecar. > - Claude Code offers its own scheduling tools to the model: `CronCreate`, `CronDelete`, `CronList` and `ScheduleWakeup`. With these tools the provider can wake itself. Paperclip does not see or govern that work. > - Today the Runner stops the Claude process when a turn ends, so these tools have little effect. A later change keeps the Claude process alive between turns. > - This pull request removes the four tools from Claude sessions that Paperclip starts through ACPX. > - The benefit is that a Claude agent schedules work only through Paperclip. ## Linked Issues or Issue Description No public issue exists. The issue is described here. Refs #19912 (the design for turn-scoped ACPX MCP authority). **What existing behavior does this improve?** The set of tools that Claude Code offers the model in a Paperclip ACPX session. **Subsystem affected** Paperclip Runner, ACPX provider (Claude). **Current behavior** A Paperclip ACPX Claude session offers the model 26 built-in tools. Four of them are `CronCreate`, `CronDelete`, `CronList` and `ScheduleWakeup`. **Proposed behavior** - The Claude settings that Paperclip writes into the session home deny the four tools. Claude Code then does not offer them. - Each Claude launch sets `CLAUDE_CODE_DISABLE_CRON=1`. This stops the cron scheduler of Claude Code and removes the three cron tools. - All other tools stay. `SendMessage`, `ListAgents`, `Workflow`, `Agent` and `TaskStop` do not change. **Reason and benefit** The provider cannot schedule its own wake-ups. Paperclip tools stay the only way to schedule work. ## What Changed - `packages/paperclip-runner/src/drivers/acpx/permission-policy.ts`: new constant `CLAUDE_SELF_SCHEDULING_TOOLS` with the four tool names. - `packages/paperclip-runner/src/drivers/acpx/runtime-sandbox.ts`: the Claude `settings.json` now always has `permissions.deny` with the four tools, in each permission mode. The `allow` rules do not change. - `packages/paperclip-runner/src/drivers/acpx/environment.ts`: `createSanitizedAcpxSpawnInput` sets `CLAUDE_CODE_DISABLE_CRON=1` for each Claude launch. The host cannot override it. - New test `claude-scheduling-tools.claude-acp.test.ts`. It prepares the session home with the production `prepareAcpxRuntimeSandbox`. It then runs the pinned `claude-agent-acp` and the pinned Claude Code client against a local fake Anthropic API. It reads the tools that the client offers the model. - Unit tests in `environment.test.ts` and `runtime-sandbox.test.ts`. One assertion in `runtime-host.test.ts` now expects the `deny` list. - No change to the `claude-agent-acp` patch, the lockfile or a pinned digest. No change to Codex. ## State saved by the previous release - A session that the previous release saved continues with no change. The new restriction is not part of a saved identity or digest. - Each time Paperclip starts Claude for a session, it writes its own `settings.json` again with the deny list and sets `CLAUDE_CODE_DISABLE_CRON`. This includes a resumed session and a start after an upgrade. - A resumed session keeps its identity, its home and its transcript. It loses the four tools from its next start. - A Claude process that the previous release started keeps its tools until it stops. Paperclip does not refuse it and does not classify it differently. Its next start gets the restriction. - No refusal code is necessary, because no saved state becomes incompatible. ## Verification - `cd packages/paperclip-runner && npx vitest run src/drivers/acpx/claude-scheduling-tools.claude-acp.test.ts` - Without the product change: `2 failed` (`expected [ Array(26) ] to not include 'CronCreate'`). - With the product change: `2 passed`. - `npx vitest run src/drivers/acpx/claude-scheduling-tools.claude-acp.test.ts src/drivers/acpx/environment.test.ts src/drivers/acpx/runtime-sandbox.test.ts src/drivers/acpx/runtime-host.test.ts`: `4 passed (4)` files, `147 passed` tests. - `npx tsc --noEmit -p tsconfig.json` in `packages/paperclip-runner`: no errors. - Tools offered after the change, isolated home: the same 26 tools without `CronCreate`, `CronDelete`, `CronList` and `ScheduleWakeup`. - `npx vitest run src/drivers/acpx/runtime-host.test.ts -t "previous release"`: the new test "resumes a Claude session saved by the previous release and withholds self-scheduling tools on relaunch" fails without the product change and passes with it. The test saves a session, puts its home into the shape of the previous release, and resumes it through `AcpxRuntimeHost.open`. - `npx tsc --noEmit -p tsconfig.test.json` in `packages/paperclip-runner`: no errors. - The fixture in `claude-scheduling-tools.claude-acp.test.ts` now stops the process group of the provider and waits for it before it removes the fixture directory. - Not run: a live run with a real Claude account, and a remote Runner. Not run: `pnpm verify` and the full suite. CI supplies that coverage. ## Risks - **A preconfigured home still offers `ScheduleWakeup`.** In that mode Paperclip does not write the Claude settings of the user, so the deny list does not apply. The environment variable removes only the three cron tools. To remove `ScheduleWakeup` there, the Runner must pass `disallowedTools` to the session. ACPX 0.22 does not forward that option. This is a follow-up. - **The new test runs the real pinned Claude Code client.** It needs the platform binary, as `runner-tool-bridge.claude-schema.test.ts` does today. It uses no credentials and no network. - **A prompt or skill that tells Claude to use these tools no longer works.** No Paperclip instruction refers to them. - The prepared harness home and remote Runners use the same code path. They are not tested separately. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the change, reviewed the diff and ran the verification again. - Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with tool use (shell, file edits, local test runs), investigated the options and wrote the change and the tests. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
bcadbde449 |
feat(runner): record the OpenRouter host that served each OpenCode model call (#21298)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - OpenCode agents can use models through OpenRouter, and the Runner proxies those model calls > - OpenRouter routes one model across many hosts, and these hosts use different quantizations (fp4, mxfp4, fp8) > - Chat evaluations with identical prompts behave very differently between runs, but the run log does not record which host served a call > - The run usage shows `provider: "deepseek"` and `biller: "openrouter"`, and `providerRequestId` is always `null` > - This pull request records the serving host and the OpenRouter generation ID for each model call in the run log > - The benefit is that we can compare evaluation cohorts by host, and the cost event names a real OpenRouter generation ## Linked Issues or Issue Description **Subsystem affected** Runner OpenCode provider proxy, runnerd usage normalization, and native run usage accounting. **Problem or motivation** OpenRouter's default routing sends a request to one of many hosts. Each host can use a different quantization. Chat evaluation results change a lot between runs with identical prompts. Paperclip does not record which host served a call, so we cannot compare results by host. `usageJson.providerRequestId` is always `null` for OpenCode runs. **Proposed solution** Every OpenRouter completion response has a top-level `provider` field and an `id` (the generation ID). Each streamed chunk has them too. The Runner proxy reads them from the first chunk after it forwards the bytes. The Runner attaches them to the usage of that call. The values go to the run log, and the run's first generation ID goes to `usageJson.providerRequestId`. **Alternatives considered** A new PRP event type such as `provider.upstream.served` would need a new event enum value. An older control plane rejects an unknown event type. A `harness.diagnostic` event shows in the task chat as a diagnostic. The usage payload already flows to every consumer, and older consumers ignore extra fields. Thus this change uses the usage payload. **Additional context** Run-log change only (AGENTS.md section 7). The data stays in `heartbeat_run_events` and `heartbeat_runs.usageJson`. It does not touch `packages/shared/src/telemetry/` or OpenTelemetry. ## What Changed - New `createUpstreamServedTap` (`packages/paperclip-runner/src/drivers/opencode/upstream-served.ts`). It is a pass-through stream. It pushes each chunk downstream first, then it reads `provider` and `id` from the first SSE `data:` event (or from a JSON body at its end). It scans at most 64 KiB of a stream and 1 MiB of a JSON body. It skips encoded bodies and unknown content types. Any parse problem or recorder error stops the scan and leaves the response unchanged. - `startOpenCodeProviderProxy` uses the tap only for 2xx responses. The OpenCode session keeps the most recent served call. Each `item.completed` usage payload and `session.usage()` now carry `upstreamProvider` and `providerRequestId`. The usage dedupe fingerprint does not include these fields, so the number of usage events does not change. - The OpenCode app-server proxy adds the two fields to `thread/tokenUsage/updated`. Rust `usage.reported` now fills its existing `providerRequestId` field and adds an optional `upstreamProvider`. The TypeScript runnerd transport keeps both fields when it rehydrates the usage notification. `usage.schema.json` documents `upstreamProvider` (the protocol manifest and schema bundle are regenerated). - Server native turn accounting keeps the first generation ID the run reports. The usage checkpoint and the native adapter result carry it as `providerRequestId`, so `usageJson.providerRequestId` and the final cost event name it. - `doc/run-log-events.md` documents the fields in a new "OpenRouter serving host" section. ## Verification - Red/green: `opencode-server-driver.test.ts` "passes OpenRouter's stream through unchanged and records the host that served it on the turn's usage". The test sends a streamed fixture with a comment line, a host chunk, a malformed chunk, and `[DONE]` through the proxy. It checks that the bytes are identical and that the turn's usage names `Novita` and the generation ID. The test fails without the proxy wiring. - `upstream-served.test.ts`: streamed fixture, chunk split inside a multibyte character, non-streamed JSON, malformed chunks, a truncated JSON body, an encoded body, the 64 KiB limit, and a recorder that throws. - `opencode-proxy-usage.test.ts`, `runnerd-codex-transport.test.ts` (rehydration), Rust `provider_events::tests::carries_the_serving_host_and_generation_on_proxy_usage`, and server `native-usage-accounting.test.ts` (first generation ID stays stable across turns). - Local: OpenCode driver, OpenCode proxy, and runnerd transport suites (31 files, 596 tests) pass. `cargo test -p paperclip-runner-core` passes (one timing test failed once under load and passed alone). Runner and server `tsc --noEmit` pass. Protocol manifest, type, and coverage checks pass. - Latency: a local benchmark proxied a 400-chunk SSE stream 200 times with the tap and 200 times without it. The median time to first byte was 292 µs with the tap and 259 µs without it (+33 µs, which is the extra stream stage). The total stream time changed from 491.4 ms to 492.5 ms. The tap spends about 5.5 µs of CPU to inspect the first chunk, after it has forwarded that chunk. OpenRouter time to first token is hundreds of milliseconds. - Paid evidence: one single-case run of `chat-features.runner-opencode.local.skills.fork-read-only` on this branch. The result is in a comment on this pull request. ## Risks - Low risk. All new fields are optional and additive. An older control plane ignores the extra usage fields. `usage.schema.json` is not bound by the event schema, and no validator enforces it on `usage.reported`. - Saved state from the previous release: this change adds no persisted run, session, checkpoint, or launch state. A saved session resumes unchanged. A run that is in flight during a deploy reports usage without the new fields until its Runner is new. `usageJson.providerRequestId` then stays `null`, as before. - `providerRequestId` on the run and its cost event is the run's first generation ID, not each call's ID. The value is stable, so a late usage receipt still matches the posted charge. Per-call IDs and hosts are in the run log. - If OpenCode starts a new model call before it reports the usage of the previous call, a usage event can name the newer call. OpenCode reports a message's usage when its step ends, so this is not expected. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude Code as an agent with tool use (shell, file edits, test runs). Extended reasoning was on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
c730872974 |
fix(recovery): a chat turn waiting on the person is not stranded work (#21302)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - In a chat, the agent's turn ends and the conversation waits for the person's next message. The stranded-work sweep must leave that wait alone > - The sweep recognizes a finished chat turn by `finalizationReasonCode: "conversation_turn_finished"` on the run record > - Run completion re-reads the run through the 64 KiB read projection. When the stored result is larger, that projection drops the finalization fields, and completion wrote the reduced copy back > - The chat turn then lost its finalization reason. Its reply was not presented, and the sweep queued `issue.productive_terminal_continuation_recovery` 30 s later. That is an extra run while the agent waits on a person > - This pull request keeps the full stored result through run completion, and makes the sweep read the committed decision from its durable rows > - The benefit is that a turn waiting on a person is never treated as stranded work ## Linked Issues or Issue Description **What happened?** In the paid chat eval `app-permissions.new-quarantined-action`, the CEO's chat turn yielded with a `response_wake` continuation and waited for a person to allow an app action. The stored run result was 113,937 bytes. Its saved record had the read projection's `truncated`, `truncationReason: oversized_result_json` and `originalSizeBytes` fields and no `finalizationReasonCode` or `decisionId`. The run's presentation decision was `interaction_owns_conversation_response`, so no reply comment was written. 30 s later the sweep queued `issue.productive_terminal_continuation_recovery`, and the case failed its run limit. **Steps to reproduce** 1. Finish a native chat turn that yields `response_wake`, so finalization records `conversation_turn_finished`. 2. Let the stored run result be larger than 64 KiB when run completion reads it. 3. Run completion writes back the reduced copy without `finalizationReasonCode`. 4. Run the stranded-work sweep. It queues a productive continuation run. **Expected behavior** The run record keeps its finalization fields, and the sweep leaves the waiting chat turn alone. No new run starts until the person replies. ## What Changed - `server/src/services/heartbeat/run-completion.ts`: run completion reads the run with `unsafeFullResultJson` before it merges the native result. It no longer writes the read projection's reduced copy back. - `server/src/services/native-runtime/native-response-wake-wait.ts`: `hasCommittedNativeConversationTurnWait` checks the run's committed finalization decision (`conversation_turn_finished`) through `native_run_finalizations` and `status_decisions` for the conversation's own agent. - `server/src/services/recovery/service.ts`: the sweep's finished-chat-turn skip also uses that durable check, so runs already saved with the reduced copy are recognized. - Tests: `run-completion.test.ts` keeps the finalization record when the stored result is larger than the read projection. `heartbeat-process-recovery.test.ts` finalizes a real chat turn with the eval's yielded result, saves the reduced run record, and expects two sweeps to start no run. ## Verification - `cd server && npx vitest run src/services/heartbeat/run-completion.test.ts -t "larger than the read projection"` — fails on master (the saved result is the truncated copy) and passes here. - `cd server && npx vitest run src/__tests__/heartbeat-process-recovery.test.ts -t "chat turn that yielded"` — fails on master (`continuationRequeued: 1`) and passes here. - `cd server && npx vitest run src/services/heartbeat/run-completion.test.ts src/__tests__/heartbeat-process-recovery.test.ts` (614 passed). - `cd server && npx tsc --noEmit` passes. - Paid E2E: see the combined-branch run in the comments. ## Risks - Run completion now reads the full stored result for one run. This is one row; the read projection still serves every API read. - The sweep check adds one indexed query for chat issues whose latest run succeeded and whose run record lacks the reason. - Previous-release saved state: no persisted shape changes. Runs already saved with the reduced copy are covered by the durable decision check. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>nightly/v2026.1113.0-nightly.0 canary/v2026.1113.0-canary.3 |
||
|
|
f79e2c5d84 |
feat(ci): add the Runner image resolver and a no-rebuild mode for the Runner vendor step (#21306)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner is the part that executes agent work: a daemon (`paperclip-runnerd`) and a provider pack of agent tools > - `master` now publishes the Runner alone as the image `ghcr.io/paperclipai/paperclip-runner`, one time for each set of Runner inputs (#21204, #21229, #21230) > - The application image still compiles its own Runner on every build. The next steps make it copy the published Runner instead > - Before a build can do that, it must answer one question the same way as every other build: "does a published Runner exist for exactly this tree?" It must also fail clearly when the Runner it was given does not match the tree > - This pull request adds the script that answers the question, and a mode of the Runner vendor step that fails instead of rebuilding > - The benefit is that the later change to the image build is small, and that a wrong or unreachable answer can only mean "build the Runner yourself" ## Linked Issues or Issue Description Refs #21204, #21229 and #21230 (the Runner image). No issue exists for this change. **Problem or motivation** Three builders make images that contain the Runner: `docker.yml`, the merge-queue build and the release preview build. Each one would have to repeat three steps to find the published Runner of its tree: check the lockfile, compute the inputs digest in the build, and look up `inputs-<digest>`. Three copies can decide differently. A build that copies a Runner built elsewhere also has no Rust toolchain. If the copied Runner does not match the tree, `prepare-runner-vendor.mjs` starts a rebuild, and the build fails late at `cargo` with a message that hides the cause. **Proposed solution** Add `scripts/runner-image-resolve.mjs`, the one place that decides. Add `PAPERCLIP_RUNNER_VENDOR_REQUIRE_CURRENT=1` to `prepare-runner-vendor.mjs`. Nothing calls the resolver and nothing sets the variable in this pull request, so no workflow job changes. **Alternatives considered** - One command that builds and looks up. Two commands keep a registry credential, when a caller has one, out of the step that builds. - A wait for the publication of the commit. The decision whether the application build waits is open, so the resolver has no wait mode. It reports `absent` and stops. - A build argument that names the expected inputs. No `Dockerfile` declares one yet. The change that makes the application image copy the Runner will add both ends together. **Roadmap alignment** This is build infrastructure. It adds no product feature and does not overlap an item in `ROADMAP.md`. This is part 1 of 4 of a stack ([stack #21309](https://github.com/paperclipai/paperclip/stack/21309)). Merge the parts in order: 1. #21306: the resolver and the no-rebuild mode (this pull request) 2. #21307: the report-only job in `docker.yml` and the pull-request check of the application image's Runner path 3. #21308: reading a retained provider pack from its Runner image 4. #21310: a registry build cache for the Runner image build ## What Changed - `scripts/runner-image-resolve.mjs` (new, 185 lines), with two commands that share a directory: - `inputs --dir <dir>` exports the `runner-artifacts-context` target, runs the lockfile check on that export, and builds the `runner-artifacts-identity` target when the lockfile is current. It reads no credential. A lockfile that is behind stops there: no image is published for such a tree. - `resolve --dir <dir>` looks up `inputs-<digest>` and prints `mode`, `reason`, `inputs_digest`, `image`, `slim_image`, `build_contexts` and `labels`. It also writes them to `$GITHUB_OUTPUT` and a short report to `$GITHUB_STEP_SUMMARY`. - `mode` is `published` only for a complete publication whose full index names these inputs and both platforms. Both images are pinned by index digest. The slim digest is the one the full index names in `io.paperclip.runner.slim-digest`; the slim tag is never read on its own. - Every other answer is `inline`, with the last four values empty: | `reason` | Meaning | | --- | --- | | `lock-behind` | `pnpm-lock.yaml` is behind the Runner's manifests | | `absent` | no tag exists | | `incomplete`, `broken` | a publication stopped, or its two tags do not belong together | | `mismatch` | the full index names other inputs, or lacks a platform | | `unknown` | the registry refused, failed or could not be reached, or the lookup threw | - The resolver reuses `runnerLockState`, `readRunnerImageManifest`, `runnerPublicationState` and `runnerImageTags` from `scripts/runner-image-publish.mjs`. That file now exports `environmentCredentials`; it has no other change. - `server/scripts/prepare-runner-vendor.mjs`: with `PAPERCLIP_RUNNER_VENDOR_REQUIRE_CURRENT=1`, the step reuses a current build as before. When the build is not current, it prints the reason, starts no rebuild, leaves the stamp in place and exits with status 1. It also fails when `PAPERCLIP_RUNNER_VENDOR_REBUILD=1` is set at the same time. Without the variable the step is the same as before. - `doc/runner-image-contract.md` has a new section, "Choosing the Runner for an image build". `doc/DEVELOPING.md` describes the new variable. - The resolver is not an input of the Runner image, so the inputs digest does not change. ## Verification macOS (Node 26.12.0), in `server/`: - `node node_modules/vitest/vitest.mjs run scripts/prepare-runner-vendor.test.mjs src/__tests__/runner-image-resolve.test.ts src/__tests__/runner-image-publish.test.ts src/__tests__/runner-image-identity.test.ts src/__tests__/runner-image-workflow.test.ts src/__tests__/server-package-build-script.test.ts`: 179 tests pass in 6 files. 27 of them are new resolver tests and 3 are new tests of the vendor step, whose end-to-end test also has a new case. - `tsc` on the new test file, with the settings of `server/src/__tests__/tsconfig.json`: no error. Linux (`node:24.22.3-bookworm`, GNU tar 1.34, the worktree mounted read-only): the same files, 179 tests pass. Real runs, on clean exports of the tree (`git archive`), with no credential, on a `linux/arm64` Docker host: - On `master` at `62d673cc3f`: `inputs` gave `lock=current` and `inputs_digest=0e6662ad9c8c6451556c76bf326723645d4afaab2decc73cd5e0025f7dd8d1ca`. `resolve` gave `mode=published`, `image=ghcr.io/paperclipai/paperclip-runner@sha256:84f2c051aa4ed16c5cc138c08e3533fa417e92177f7d15af5f2aa3e0ba467162` and `slim_image=ghcr.io/paperclipai/paperclip-runner@sha256:f63ea43666ed708f1b0e5d2ade8bdc002885bf04252b9ed6cee21f7920e54f3c`. These are the two indexes that `master` published for those inputs. - The same tree with one line added to a Runner source file: another digest, `mode=inline`, `reason=absent`. - On `master` at `bc71c23e0a`, which changed the Runner's inputs, before its image was published: digest `88fe2929321b4706c77f11a098572ce990fec4449d505544c62ef3a84c28308a`, `mode=inline`, `reason=absent`. - Four deliberate faults in the resolver each make a test fail: no index check, only `absent` is inline, the identity is built for a lockfile that is behind, and the slim image is taken by tag. Never run: - A lookup with a registry credential. The package is public, so no caller needs one now. - Any workflow. No workflow calls the resolver in this pull request. - `linux/amd64`. - The full test suite, the full typecheck and `pnpm build`. `pnpm verify --explain` reports exit 2, "broader verification required", so CI must supply them. ## Risks - Low for existing behaviour: nothing calls the resolver, and the vendor step is the same unless the new variable is set. - The resolver does not verify the attestation of the image. The workflow that consumes the image must do that from the digest the resolver reports. - The resolver reports both build contexts, `runner-artifacts` and `runner-artifacts-slim`. A build that reads only one stage gets one context it does not use. - A `broken` publication is `inline`, not a failure. The Runner image workflow fails for that state. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude Code with tool use (shell, file edits, local Docker builds). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
731b52453c |
fix(environments): verify workspaces on run leases and release settled tasks (#21289)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Core owns run leases and delegates remote execution to environment plugins. > - A retry must verify its old checkout on the placement that will run the task. > - A completed task must still release its provider resources. > - This change uses the existing lease for workspace verification and always calls the release hook after settlement. > - Saved settlement proof lets cleanup retry without repeating execution work. ## Linked Issues or Issue Description Refs #21237. That PR records stop proof before lease release. This PR keeps provider cleanup after settlement and changes workspace verification sequencing. Both touch task completion; preserve both changes if the merge order requires conflict resolution. **What happened?** Workspace verification ran before a portable task had a lease, so a provider could not verify the checkout on its selected placement. Core also skipped the provider release hook after successful task completion. This could leave resources allocated after the run ended. **Expected behavior** Verify the old checkout through the selected run lease. A confirmed refusal can use the authorized fresh-workspace fallback. A transport failure must remain retryable. Call the provider release hook after settlement and retry failed cleanup without changing the run result. **Steps to reproduce** 1. Retry a task whose prior checkout uses plugin-managed storage and portable task placement. 2. Verify that the plugin receives the same task lease for verification and preparation. 3. Complete a task, fail the provider release once, and reconstruct the cleanup controller. 4. Confirm that cleanup calls only the release hook and retains the completed run outcome. ## What Changed - Acquire the logical run lease before provider workspace verification and reuse it for preparation and execution. - Preserve the existing session and workspace choices. Recompute workspace-dependent inputs only after a definite refusal; acquire credentials and managed runtime inputs once. - Save the exact refusal and lease before cleanup. Use a workspace-scoped creation key only for the resulting fresh attempt. - Replay interrupted verification on the same unlaunched lease. Keep transport errors retryable and reject replay after execution starts. - Always call the provider release hook after task settlement. Pass refreshed settlement proof and route failed releases through the existing cleanup reaper. - Add database regressions for call order, restart, retained writers, refusal, transport errors, and settlement-only cleanup replay. `plugin-workspaces.ts` grows from 997 to 1,001 lines. The four added lines extend the existing workspace transaction and lease callbacks. Keeping the checks there preserves the ownership assertion, provider result validation, and binding update in one operation. Extracting only the verification branch would split that transaction without removing a separate responsibility. ## Verification - Server TypeScript check passed: `tsc -p server/tsconfig.json --noEmit`. The complete `server/scripts/typecheck-layers.mjs` check also passed, including all test-tree layers. - The first CI typecheck caught a readonly array in a test fixture that the narrower source check did not include. Commit |
||
|
|
61ab829d5d |
fix(queue): another agent's comment no longer hides a pending approval (#21303)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a person approves a new plan while the agent is busy, the approval waits in the task's message queue. Its Interrupt control stops the current attempt and starts a fresh one with the approved plan > - The task view shows one queue at a time. A saved card response (an approval) gives way to any later queued message, so it never hides the person's own later messages > - Another agent's comment (here, the CEO's direction with `resume: true`) is also a queued message. It took over the view, and the approval and its Interrupt control disappeared > - The person could then not interrupt for the plan they approved > - This pull request lets only the person's own later messages hide a pending approval. A queue that holds only agent comments no longer hides it > - The benefit is that the Interrupt control for an approved plan stays available until the person uses it or writes a newer message ## Linked Issues or Issue Description **What happened?** In the paid chat eval `steering.interrupt-with-direction`, the person approved a saved simpler plan while Morgan was busy. The approval queued with `requiresFreshSession`, and its Interrupt control showed. The CEO then posted its own direction comment on the task at 04:03:21. That comment queued as a separate deferred wake. `GET /queued-comments` then returned only the CEO comment's queue. The approval's Interrupt control (`task-chat-queued-interrupt-<approval id>`) never rendered, the click timed out, and no interrupt request was sent. The approval was not lost: its deferred wake was still saved. Only the view and its control were hidden. **Steps to reproduce** 1. Approve a plan on a busy task, so the approval waits for a fresh session. 2. Let another agent comment on the task, so its comment queues after the approval. 3. Read `GET /api/issues/:id/queued-comments`. The approval is gone from the queue. **Expected behavior** The approval stays the visible queue with its Interrupt control. A newer message from the person still takes the view, as before. ## What Changed - `server/src/routes/issues.ts`: `findQueuedCommentWake` returns a later message queue only when it holds a message from a person. A queue with only agent comments yields to a pending saved response. With no saved response it is still returned, as before. - `server/src/__tests__/issue-queued-comments-routes.test.ts`: the eval ordering (a fresh-session approval, then another agent's comment) keeps the approval as the queue, and a later person message in that queue still takes the view. ## Verification - `cd server && npx vitest run src/__tests__/issue-queued-comments-routes.test.ts -t "another agent's comment queues after it"` — fails on master (the CEO comment's queue is returned) and passes here. - `cd server && npx vitest run src/__tests__/issue-queued-comments-routes.test.ts` (all tests passed). - `cd server && npx tsc --noEmit` passes. - Paid E2E: see the combined-branch run in the comments. ## Risks - While a saved response waits, queued agent-only comments are not shown in the composer queue. They stay saved and are delivered by the normal deferred-wake path, as they were before the person approved anything. The person's own messages are never hidden. - One indexed comment lookup per queued message wake during the queue read. - No persisted shape or API contract change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
99d9e608f4 |
fix(heartbeat): hold the new owner's run while the reassigned owner's run settles (#21301)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a person reassigns a task, the old owner's native run is cancelled and the new owner's run is queued > - The old run's executor keeps closing its provider session for several seconds after the run is marked cancelled > - #21031 holds a queued run while the task's previous session close is pending. That hold reads the runner's cleanup gate, which exists only after the close begins > - The stale-lock sweep cleared the task lock as soon as the old run was terminal. The new owner's run was then claimed before the close began, met the close as it escalated, and failed with `native_session_cleanup_quarantined` > - This pull request keeps the new owner's run queued while the old owner's stopped run still settles in this process, and wakes it when that executor finishes > - The benefit is that a reassignment never fails the new owner's run and needs no Retry ## Linked Issues or Issue Description Refs #21031 **What happened?** In the paid chat eval `tasks.edit-properties`, the old owner's run `6cf2ee38` was cancelled for reassignment at 04:03:47.611. Its process stopped only at 04:03:57. The stale-lock sweep cleared the task lock at 04:03:47. The new owner's run was claimed at 04:03:48.626. The runner admission waited for the old close, which escalated to operator-required at 04:03:56, and the new run failed with `native_session_cleanup_quarantined`. Why #21031 did not apply: its hold reads the runner's in-memory cleanup gate. At claim time the old executor had not started its session close, so no gate existed. The existing `predecessor_settling_in_process` hold covers only the same agent, and a reassignment changes the agent. #21260 does not touch this path: its new checkpoint runs only before a turn in a recovered session, and both sessions here were new. **Steps to reproduce** 1. Run a native task for agent A. 2. Reassign it to agent B. A's run is cancelled, but its executor is still settling. 3. The stale-lock sweep clears the task lock. 4. B's queued run is claimed before A's session close begins. **Expected behavior** B's run stays queued until A's executor has finished. It starts by itself after that, then #21031's gate covers any close that is still settling. ## What Changed - `server/src/services/heartbeat/queue.ts`: the `predecessor_settling_in_process` hold in `claimQueuedRun` also holds for another agent's native run on the same task when that run is terminal but its executor is still active in this process. A live run of another agent (for example a reviewer) does not hold it. - `server/src/services/heartbeat/queue.ts`: `notifyOtherAgentsQueuedOnSettledIssue` sends a queued-run dispatch signal when a native executor settles and another agent has a queued run on the same task. It does nothing when no run waits. - `server/src/services/heartbeat.ts`: the executor calls it after its cleanup, next to the existing same-agent `startNextQueuedRunForAgent`. - `server/src/services/heartbeat/queue.test.ts`: the exact ordering (cancelled old-owner run still in the executor set, lock cleared, new owner's run queued), a control for a live run of another agent, and the dispatch wake. ## Verification - `cd server && npx vitest run src/services/heartbeat/queue.test.ts -t "reassigned owner|another agent's live run|next owner once"` — the hold and the wake tests fail on master and pass here. - `cd server && npx vitest run src/services/heartbeat/queue.test.ts src/__tests__/heartbeat-native-cleanup-admission-hold.test.ts src/services/queued-run-dispatch-liveness.test.ts src/__tests__/heartbeat-workspace-busy.test.ts` (all passed). - `cd server && npx tsc --noEmit` passes. - Paid E2E: see the combined-branch run in the comments. ## Risks - A queued run on a task waits while another agent's terminal native run on that task is still settling in this process. That wait ends when the executor finishes, and the new wake re-drives dispatch. The queued-run liveness redrive still covers a lost signal. - The wake is event-driven: one indexed query when a native executor settles, and one notification only when another agent waits on that task. It adds no polling. - Previous-release saved state: no persisted shape changes. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
f82ec65800 |
fix(runs): a person's Stop wins over lease-release teardown (#21300)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A person can stop an agent run with Stop (cancel). The run must then end as cancelled > - A run that a person stops just as it starts tears down its environment lease at the same moment > - The lease-release teardown wrote `interrupted` (`lease_released_before_terminal`). It recognized only an acknowledged Stop, and it decided from a run snapshot that it read before the Stop request landed > - An interrupted run looks like an infrastructure failure, and it is a recovery candidate. That is against the person's decision > - This pull request makes a recorded person Stop win over the teardown in both orderings > - The benefit is that a person's Stop always ends the run as cancelled ## Linked Issues or Issue Description **What happened?** In the paid chat eval `steering.stop-versus-pause`, the fixture cancelled a resumed run as it reached native running. The Stop request was recorded at 04:02:11.049. The teardown wrote `interrupted` with `lease_released_before_terminal` at 04:02:11.083. The native cancellation intent was still `pending`, so the teardown did not see it as a Stop. The eval needs `cancelled`. **Steps to reproduce** 1. Start a native run. 2. Record a person Stop request on the run (the cancel route does this first). 3. Let the lease-release teardown run before the cancel finalizer writes `cancelled`. 4. The run ends `interrupted`. **Expected behavior** The run ends `cancelled`. This is true when the Stop is recorded before the teardown reads the run, and when it is recorded between that read and the write. ## What Changed - `server/src/services/acknowledged-native-stop.ts`: `hasRecordedUserStopRequest` and `recordedUserStopRequestSql` recognize a person Stop that the cancel route recorded (`cancelledByActorType: "user"`, a user id, and a requested or acknowledged execution cancellation or a startup cancellation fence). - `server/src/services/heartbeat/run-lifecycle.ts`: `terminalizeRunOnLeaseRelease` re-reads the run before it decides. A recorded person Stop now gives `cancelled`. The `interrupted` write is fenced on "no person Stop recorded", so a Stop that lands between the read and the write makes the write fail. The function then reads again and writes `cancelled`. - `server/src/__tests__/heartbeat-run-lease-release-terminalization.test.ts`: two orderings (Stop recorded before the teardown read; Stop recorded after the read, with the stale snapshot) and a control (an agent-only request still reports `interrupted`). ## Verification - `cd server && npx vitest run src/__tests__/heartbeat-run-lease-release-terminalization.test.ts` — both new ordering tests fail on master (`interrupted`, `lease_released_before_terminal`) and pass here. - `cd server && npx vitest run src/__tests__/heartbeat-run-lease-release-terminalization.test.ts src/__tests__/heartbeat-run-terminalize-before-release.test.ts src/services/heartbeat/run-lifecycle.test.ts` (67 passed). - `cd server && npx tsc --noEmit` passes. - Paid E2E: see the combined-branch run in the comments. ## Risks - Low. Only the lease-release teardown changes, and only when a person Stop is recorded on the run. The cancel path already wrote `cancelled` for the same run when it won the race. - The teardown now makes one more read of the run row. This is one indexed primary-key read per teardown. - Previous-release saved state: no persisted shape changes. Runs saved by the previous release carry the same Stop fields. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
bc71c23e0a |
fix(test): remove the Pi fault calibration file race and halve its runtime (#21286)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner support suite checks the Pi provider fault helper. The
helper kills the exact Pi child process on Linux with a pidfd.
> - `tests/runner-e2e/pi-provider-fault.test.ts` failed in 3 unrelated
PRs on 2026-11-12. One failure passed on a rerun of the same commit.
> - The failures had two causes. A fixture file race gave a JSON parse
error. A CPU-heavy calibration ran past the 25 s outer timeout on the
slower shared fleet.
> - This pull request removes the file race and removes about half of
the calibration work. It also states the outer timeout margin.
> - The benefit is a deterministic test that runs in about 3.4 s instead
of about 5.9 s on Linux.
## Linked Issues or Issue Description
**What happened?**
The test "calibrates the exact Pi fault helper with metadata negatives
and native Linux pidfd" failed intermittently in the "Runner support
suite" lane. There were two errors:
- Job 131195539083: `test_real_descriptor_launch_then_exact_child_pidfd
(descriptor=7)` failed at `json.loads(ready.read_text())` with
`JSONDecodeError: Expecting value: line 1 column 1 (char 0)`.
- Jobs 131420813114 and 131420614853: `spawnSync /usr/bin/python3
ETIMEDOUT` after 25 s. Both jobs ran on the `paperclip-public-pr-x64`
fleet (m5n.xlarge, 4 cores, load average about 4). In those jobs, every
file in the lane ran about 1.8x slower than on the hosted `ubuntu-24.04`
image.
**Expected behavior**
The test passes on every run of the same commit.
**Steps to reproduce**
1. Error A: add a delay between file creation and write in the fixture
child (for example `openSync`, then `Atomics.wait` for 500 ms, then
`writeSync`). Run `/usr/bin/python3 -B
tests/runner-e2e/pi-provider-fault.test.py` on Linux. All four real
calibrations fail with the same `JSONDecodeError`.
2. Error B: run the suite on a loaded 4-core runner. The Python suite
needs more than 25 s.
**Paperclip version or commit**
`master` at
|
||
|
|
e65a3095a7 |
fix(runner): quiesce the Core-dialed route once the Runner commits its suspend (#21245)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Core runs agent sessions through the Paperclip Runner. With provider
ingress, Core dials the Runner over a PRP WebSocket.
> - When Core closes a session, it asks the Runner to suspend. The
Runner suspends, reports the result, and exits.
> - Core's ingress connection did not know that this exit was expected.
It treated the drop as a transport loss and redialed the Runner. Each
redial got a 404 until the close released the route.
> - The release (park) path already quiesces the connection before the
Runner goes away. The suspend path did not.
> - This pull request makes the suspend close quiesce the route in the
same way, once the Runner commits its suspend.
> - The benefit is that a clean suspend no longer looks like a lost
Runner. A drop before the suspend is committed still redials and
recovers as before.
## Linked Issues or Issue Description
**What happened?**
After a provider-ingress session closed cleanly, Core redialed the
Runner's ingress endpoint several times and got 404 each time. The
Runner state was `suspended`, `runner.drain` and `runner.suspend` had
both completed, and the Runner had exited with code 0. The redials
followed the outbound reconnect backoff (about 250, 500, 1000 and 2000
ms) and stopped when the registration release closed the outbound.
**Expected behavior**
After Core commits to `runner.suspend` and the Runner reports it
completed, the Runner's exit is expected. Core must not redial or
classify the drop as a transport loss. The release path already works
this way: it calls `outbound.quiesce()` before it releases the task.
**Steps to reproduce**
1. Open a session through a Core-dialed route (plugin provider ingress
or the `listen_ws` driver).
2. Close the session. Core drains, queues `runner.suspend`, and the
Runner suspends and exits.
3. See that the outbound reconnect loop redials the gone Runner until
the registration release closes it.
**Paperclip version or commit**
`master` at
|
||
|
|
62d673cc3f |
fix(retry): start a held Retry once a stopped preparation is proven stopped (#21295)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - When an agent run fails, a person can press Retry. Admission starts
a new turn from the recorded work only after it proves that the previous
execution stopped.
> - For a legacy runner run that failed while it prepared its sandbox,
the proof reads the run's events and its environment lease.
> - After the automatic retry budget is spent, execution recovery
settles a no-replay hold and appends its own lifecycle event to the run.
The proof accepted only the error and measured events, so this platform
event made the proof fail.
> - The proof also asked for a workspace repair receipt for every run
with a lease, even when the run never recorded a remote workspace and
its sandbox was destroyed. No repair could ever exist, so the proof
could never pass.
> - Admission then refused the Retry without a named reason. The receipt
kept the generic `execution_recovery` wait ("Retry is waiting for
execution checks"), and every later Retry joined the same held receipt.
> - This pull request accepts the recovery hold's own settlement event
when it is bound to a settled hold for the same run, and asks for a
workspace repair only when the run holds prepared files.
> - The benefit is that Retry starts work at once when the previous
preparation is proven stopped, and it continues from the recorded work.
## Linked Issues or Issue Description
**What happened?**
A board user pressed Retry on a blocked task. The Retry was accepted,
but no run started for more than 13 hours. The task said "Retry is
waiting for execution checks. Your message is saved." A second Retry the
next day attached to the same held request and also did nothing. The
execution blocker said `legacy_execution_requires_reconciliation` with
`canRetry: true` and "A person can use Retry on this task ... to
continue once the previous execution has stopped." The previous run was
a legacy `paperclip_runner` run that failed with `setup_failed` while
preparing (`executionRecovery.kind = bootstrap`, `providerWorkStarted =
false`). It had no process identity, no controller lease, and its
Daytona sandbox lease had a `destroyed` termination receipt. Execution
recovery had settled a no-replay hold on it and appended the "Automatic
recovery stopped" lifecycle event.
**Expected behavior**
When the previous execution is proven stopped, a person's Retry starts a
new turn promptly and continues from the recorded work. The task view,
the execution blocker and admission agree that Retry works.
**Steps to reproduce**
1. Let a legacy `paperclip_runner` run fail while it prepares a remote
sandbox (`setup_failed`, bootstrap evidence with `providerWorkStarted =
false`) on its third automatic attempt.
2. Let the sandbox lease be released with a `destroyed` termination
receipt.
3. Let execution recovery settle the
`legacy_execution_requires_reconciliation` hold. It appends the
"Automatic recovery stopped" lifecycle event to the run.
4. Press Retry on the task.
5. Observe that the Retry receipt keeps `executionWait.reason =
execution_recovery` and no run starts. Press Retry again and observe
that it attaches to the same held receipt.
**Paperclip version or commit**
Staging Core `53608bd1d6`. Fixed on top of `origin/master` `c4f639ed1b`.
**Deployment mode**
Hosted cloud (staging), Daytona remote sandbox.
**Agent adapter(s) involved**
`paperclip_runner` (legacy runtime).
Related: #21267 (a Retry held on a resolved decision,
`decision_pending`). This pull request does not change its files.
## What Changed
- `readFailedPreparationSettlement` (`failed-preparation-settlement.ts`)
accepts one trailing host lifecycle event: the no-replay settlement
event that execution recovery appends. The event must have the exact
payload shape and must name a resolved
`legacy_execution_requires_reconciliation` hold whose evidence binds the
same run. Any other event still fails the proof.
- `failedNativePreparation` (`cancelled-native-startup.ts`) asks for a
verified workspace repair only when the run holds prepared files: a
recorded remote workspace, a workspace restore failure, or required
workspace recovery. A run with no recorded remote workspace and a
stopped sandbox has nothing to repair. The lease termination receipt,
the bootstrap evidence and the settled-event proof are still required.
- New regression test `legacy-preparation-retry-hold.test.ts` uses the
exact shape of the staging run.
## Verification
- Red/green: `cd server && npx vitest run
src/services/legacy-preparation-retry-hold.test.ts`.
- "starts the person's Retry once the stopped preparation's sandbox is
destroyed" fails on `origin/master` (the receipt keeps
`executionWait.reason = execution_recovery`, no run) and passes with
this change (one queued successor with `previousRunId` set to the failed
run and the Retry request id).
- The Retry stays held when the proof does not hold: a settlement event
bound to another run's hold, a provider execution event, a lease without
a termination receipt, and a recorded remote workspace without a repair
receipt.
- Related suites pass (299 tests): `task-retry`,
`saved-bootstrap-continuation`, `interrupted-preparation-hold-settle`,
`workspace-recovery-preparation`,
`workspace-recovery-preparation-gates`, `heartbeat-session-preparation`.
- `npx tsc --noEmit -p server/tsconfig.json` passes.
## Risks
- Low. The change only widens two proofs for a run that never started
provider work: the platform's own settlement event is accepted only when
it binds a settled hold for the same run, and the repair receipt is
skipped only when the run recorded no files to repair.
- Retry still starts a new turn. It never replays the stopped run.
- Not in scope: two other tasks show the same generic wait with a
different cause on the saved-work side. Their blocker says `canRetry:
false` while the task view offers Retry. One has saved-work recovery
that failed with `source_unavailable`. The other shows recovered work
without a verified repair receipt. They need a separate fix.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edit, browser for read-only staging API
reads).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
db3d7223e0 |
fix(verify): report stale SDK public input pins locally, before CI (#21283)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner support suite proves the plugin SDK public types against
pinned source. The pins are in
`tests/runner-e2e/source-sdk-public-inputs.json`.
> - The pins cover every file in `packages/shared/src` and
`packages/plugins/sdk/src`, and the SDK build inputs (for example
`pnpm-lock.yaml`).
> - A PR that edits a pinned file and does not refresh the registry
fails three files in the "Runner support suite" lane.
> - Between 2026-11-12 10:27Z and 2026-11-13 01:14Z, this failure
occurred in 14 CI runs on 12 PRs.
> - The local check, `pnpm verify`, returned exit 2 ("let CI cover it")
for these changes. It did not mention the pins. Authors pushed, and CI
failed.
> - This pull request makes `pnpm verify` find stale pins before the
push. It also makes the opaque fixture error name the file and the fix.
> - The benefit is that authors find and fix the problem in seconds,
locally, instead of after a 6-minute CI lane.
## Linked Issues or Issue Description
**What happened?**
Three files in the "Runner support suite" lane failed together on many
unrelated PRs: `source-sdk-evidence.test.ts`,
`source-sdk-public-inputs.test.mjs` and
`source-sdk-public-type.test.ts`. Two of the three failed with the
opaque error `Reviewed SDK reference source changed`.
**Root cause**
The pins on master are not stale. The tests pass on master (58/58). Each
failing PR edited a pinned input and did not refresh the registry. I
compared the PR heads with their merge base:
- All 14 failing heads (12 PRs) changed a pinned input. None of them
changed `source-sdk-public-inputs.json`.
- All 19 heads in the same window that changed a pinned input and passed
this lane also refreshed the registry. This includes later heads of
three PRs that failed first (#21119, #21121, #21156).
- Two of the 12 PRs are Dependabot PRs that change only `pnpm-lock.yaml`
(#19593, #19594). The lockfile refresh bot already resyncs the pins for
lock changes on master.
**Steps to reproduce**
1. Start from `origin/master`.
2. Add one line to `packages/shared/src/types/heartbeat.ts` and commit
it.
3. Run `node node_modules/vitest/vitest.mjs run --config
tests/runner-e2e/vitest.config.ts source-sdk-evidence.test.ts
source-sdk-public-inputs.test.mjs source-sdk-public-type.test.ts`.
4. The result is the same as CI: 3 files fail, with the same 4 failures.
5. Run `pnpm verify --explain`. It returns exit 2 and does not mention
the pins.
**Expected behavior**
`pnpm verify` reports the stale pins and the refresh command before the
author pushes.
## What Changed
- `tests/runner-e2e/source-sdk-public-inputs.mjs`: add
`staleSdkPublicInputsInWorkingTree(root)`. It applies the existing drift
guard to the working tree (tracked and untracked files that git does not
ignore). Add `isSdkPublicInput(file)` and the shared failure message
`staleSdkPublicInputsMessage(stale)`. The committed-tree guard keeps its
behavior and now uses a shared comparison.
- `scripts/local-verify.mjs`: when the diff touches a pinned input, the
registry or `report-lock.json`, `pnpm verify` checks the pins first. If
they are stale, it exits 1 and prints the files and the refresh command.
The check reads files only. It does not write the registry.
- `tests/runner-e2e/source-sdk-reviewed-fixture.ts`: the error `Reviewed
SDK reference source changed` now names the file and gives the refresh
command.
- `tests/runner-e2e/source-sdk-public-inputs.test.mjs`: new test for the
working-tree guard (uncommitted edit, untracked file, ignored file). The
CI drift test uses the shared message.
- `doc/DEVELOPING.md`: document the new check under "Incremental local
verification".
No assertion is weakened. No pin is changed. No file under `.github/` is
changed.
## Verification
- Master, without this change: the three files pass (58/58).
- Red: on master plus a one-line edit to
`packages/shared/src/types/heartbeat.ts`, the three files fail with the
CI signature. `pnpm verify --explain` returns exit 2 and does not
mention the pins.
- With this change on the same red commit: `pnpm verify` exits 1 in
about 15 s and prints `tests/runner-e2e/source-sdk-public-inputs.json is
stale for 1 pinned input(s): packages/shared/src/types/heartbeat.ts
(changed). Commit your change, run ... --write, and commit the updated
registry.`
- Green: I followed the printed command (`node
tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --write`, then
commit). The three files then pass (59/59, with the new test).
- `node --test scripts/__tests__/local-verify.test.mjs`: 14/14 pass.
- `pnpm verify` on this branch returns exit 2 because
`scripts/local-verify.mjs` is a global CI input. Required CI supplies
the broad coverage.
## Risks
- Low risk. The new check only reads files. It runs only when the diff
touches a pinned input.
- `pnpm verify` now fails (exit 1) for a change that also fails in CI.
Before, it returned exit 2.
- The check needs a working `git ls-files` in the repository. `pnpm
verify` already needs git.
- Dependabot lock PRs still fail this lane until the lockfile bot
refreshes the pins. This PR does not change that path.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude (Anthropic), Claude Code agent with tool use (shell, git,
GitHub CLI). Commits are attributed to Claude Fable 5.2.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
|
||
|
|
0c63f0fdb3 |
test(skills): budget list read bytes for fixture rows, not the bundled catalog (#21285)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The skill library list reconciles the bundled skill catalog, folders, and local sources on each read > - #18023 added a test that limits the `company_skills` bytes one warm `list()` call reads, so the folder pass and the list query keep their narrow projections > - The limit counts every `company_skills` read, so it also counts the bundled catalog rows that every company gets > - Each new bundled skill adds about 9.6 KB to that total, so pull requests that only add skills under `skills/` fail a test about projections > - This pull request measures a catalog-only company next to the fixture company and applies the same limit to the difference > - The benefit is that the test fails only when the fixture rows are read too widely, not when the catalog grows ## Linked Issues or Issue Description Refs #18023 (added the test). Refs #20760, #20764, #20767, #20768, #20769 (failed on this test). **What happened?** `companySkillService.list > keeps folder classification and stored content while narrowing inventory refresh reads` failed on five open pull requests that only add bundled skills under `skills/`. The error was `expected 1163219 to be less than 950000`. Other runs had 983607, 1035956, 982321, and 985561. The test passes on master. **Expected behavior** The test must fail when `list()` reads the unused metadata or the Markdown of company skill rows. It must not fail when the bundled catalog gets larger. **Steps to reproduce** 1. Check out master. 2. Run `git checkout <head of #20760> -- skills` to add its 34 bundled skills. 3. Run `pnpm --filter @paperclipai/server exec vitest run src/__tests__/company-skills-service.test.ts -t "narrowing inventory refresh"`. 4. The test fails with `expected 1163504 to be less than 950000`. **Paperclip version or commit** `9b613562b70b7bea9e7562ddc1087d4c1b1788b1` (master). ## What Changed - The test creates a second company that has only the bundled catalog. - The test lists both companies warm and captures the `company_skills` response-value bytes of each call. - The test applies the existing 950,000-byte limit to the difference. The difference is about 663 KB and comes only from the ten fixture rows. - The test also checks that the fixture company lists the catalog rows plus the ten fixture rows. - No service code changes. ## Verification Where the bytes come from (master, one warm `list()` call): | Read | Master catalog | Catalog of #20760 | | --- | --- | --- | | Main list query (`company-skills.ts:3324`) | 671,003 B, 20 rows | 691,694 B, 52 rows | | Bundled reconciliation `getByKey` reads (one per bundled skill) | 165,423 B, 13 calls | 471,771 B, 45 calls | | Total (old assertion) | 836,465 B, pass | 1,163,504 B, fail | | Fixture company minus catalog-only company (new assertion) | 663,153 B, pass | 663,166 B, pass | Red and green: - Old test, master `skills/` plus the `skills/` tree of #20760: fails, `expected 1163504 to be less than 950000`. - New test, same enlarged catalog: passes, difference 663,166. - New test, master: passes, difference 663,153. The new test still catches the regression it exists for. I widened each projection on purpose with the enlarged catalog, ran the test, and reverted: - Folder pass reads the full `metadata` column instead of `sourceKind` only: fails, `expected 1323122 to be less than 950000`. - Main list query also reads `markdown`: fails, `expected 1183162 to be less than 950000`. Commands: - `pnpm --filter @paperclipai/server exec vitest run src/__tests__/company-skills-service.test.ts`: 81 passed. - `pnpm verify`: not completed. It selected 2,615 of 3,435 test files (conservative selection), and I stopped it to keep load low on a shared machine. CI ran the full required checks on the final commit: 54 passed, 4 skipped. ## Risks - Low risk. The change is in one test only. - The new test runs one more warm `list()` call, about 0.5 s more. - The bound is unchanged. A widened projection on the fixture rows still adds at least 540 KB and fails. - This test no longer limits the bytes the bundled catalog adds to each list call. Each bundled skill adds about 9.6 KB per warm call, because `ensureBundledSkills` calls `getByKey` with the full row for each bundled skill. That cost is real but separate. A fix should narrow that read, not let this test limit the catalog size. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes (not applicable: test-only change) - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green (Superagent Security Scan is neutral: "Scan inconclusive") - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |