mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-13 15:55:46 +01:00
b7446969d73ea9c99eaf8d89d7a7249b5a986fcf
7573
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandClaude Opus 5.5
|
b7446969d7 |
fix(evals): give choose-capable-target a saveable computer and grade it structurally (#20881)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat catalog evals check that agents answer real user questions through the product. > - The case `execution-environments.choose-capable-target` asks "Which computer can run Morgan with Claude Code?" while environment reads are board-only. > - Its setup saved a built-in fake sandbox as the "unsupported" computer. The product refuses to save fake sandboxes (since #4449), so setup failed with 422 before any agent run, and the test API hid the reason. > - No computer that the product can save is advertised as unable to run Claude Code, so the original contrast cannot be built through supported APIs. > - This pull request changes the contrast to advertised support versus runtime health, grades the reply by structure instead of keywords, and makes setup failures show the product's validation reason. > - The benefit is that the case can measure its user promise, and a future setup 422 says why. ## Linked Issues or Issue Description **What happened** In the full chat catalog run, setup failed with `POST /api/companies/<id>/environments returned 422; body withheld`. A focused server test shows the reason: "Built-in fake sandbox environments are reserved for internal probes and cannot be saved." The fixture payload was `driver: "sandbox", config: { provider: "fake" }`. **Expected behavior** Setup builds its computers only through supported APIs, the agent run starts, and the grader checks the user promise without reply keywords. A setup validation failure states the product's reason. **Steps to reproduce** 1. Run `chat-features.runner-opencode.local.execution-environments.choose-capable-target` on master. 2. Setup posts a fake sandbox environment and gets 422. 3. `cd server && pnpm exec vitest run src/__tests__/environment-routes.test.ts -t "rejects persisted fake sandbox environments"` shows the same rule. ## What Changed - `runtime-capability-choice.ts`: setup saves an SSH computer on a reserved `.invalid` host (no key, never probed) beside the local fixture computer. Production capabilities advertise Claude Code on both. The `enableEnvironments` experimental setting is still set only through `PATCH /api/instance/settings/experimental` and restored in cleanup. - Grader: removed the reply regex and keyword checks. New checks: agent gets 403 on capabilities and probe; the reply renders a link to `/<prefix>/company/settings/instance/environments` (or a target's edit page); the reply has no board-only SSH settings (host, user, path); targets (including status and metadata) and the worker are unchanged; exactly one chat turn and no tasks. `verifyResult` clicks the rendered link and checks that the page shows both computers and loads the same capabilities. - `execution-environments.yaml`: context changed from "one target supports the harness and another does not" to support versus health, with the reason. Recorded in `chat-full-catalog-testops-manifest.json` `changedFields`, pinned in the manifest test, and the inventory digest is updated. - `oracleVersion` is now `paperclip.chat-catalog.runtime-capability-choice.v2` in code and in `chat-catalog-descriptors.json`; `sourceFiles` adds `internal-support.ts`, `server/src/routes/authz.ts` and `server/src/services/environment-config.ts`. - `api.ts`: cookie-session 400/409/422 failures now include the product's `error` message, at most 240 characters, with request string values (8 or more characters), the session cookie and long token-like strings redacted. `details` and all other statuses stay withheld. - Includes a cherry-pick of #20880 (agent environment denials link the environment settings page). Without it, the agent has no route to give. ## Verification - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts` on `runtime-capability-choice.test.ts`, `runtime-computers.test.ts`, `chat-catalog-contract.test.ts`, `chat-catalog-consistency.test.ts`, `api-private-transport.test.ts`, `api.test.ts`, `api-response-reading.test.ts`: 73 passed. - Grader calibration: valid and reworded replies with the settings link pass. Missing, unprefixed, wrong-prefix, cross-origin and wrong-page links fail. Replies with the SSH host, user or path fail. Unsupported capability, reachable host, 200 boundary, changed target or worker, an extra run and a created task fail. - `node --test tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs`: 10 passed. - `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`: unchanged. - `npx tsc --noEmit -p tests/runner-e2e`: no errors in changed files. - Paid run still needed: `chat-features.runner-opencode.local.execution-environments.choose-capable-target`. ## Risks - Premise change: the case no longer has a saved computer that is advertised as unable to run Claude Code, because the product cannot save one. The "avoid an incompatible target" clause stays in the expected outcome, but no saved target exercises it. - The reply's claims about health are not graded by words. A reviewer must read the paid run's reply. - The 400/409/422 reason can show product text that the test API hid before. It is bounded and redacted. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
c45f54ee74 |
fix(native): stop or release unclosable idle warm Runners on environment delete, and say why a close failed (#20957)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs keep a warm Runner in a Daytona sandbox between turns of
a task, so the next turn starts fast
> - Each idle warm Runner has an in-memory owner in the controller.
Deleting an environment closes these owners before it destroys the
sandboxes
> - In every warm chaos wave today, some owners never closed. The server
logged only `{"failed":2}`. Environment delete returned 409 forever, and
all sandboxes of the wave stayed `started` until their 60-minute TTL
> - On staging, that is leaked sandboxes (cost) and owners that block
the work after them
> - This pull request logs and reports the exact close error, stops an
unsettled idle Runner when its environment is deleted, and releases an
owner whose sandbox is already destroyed
> - The benefit is that environment delete finishes or names the exact
run and error, and every warm chaos wave now grades a clean shutdown
## Linked Issues or Issue Description
**What happened?**
After warm Daytona chaos waves (W2, W2Q, W2C cells, `--faults K0`) on
Codex, Claude and OpenCode, the server logged `idle native session
cleanup failed before controller shutdown {"idleSessions":{"failed":2}}`
with no reason. `DELETE
/api/environments/:id?destroyReusableSandboxLeases=true` returned 409
"warm native runner did not shut down cleanly" on every retry. Eight
sandboxes per wave stayed `started`.
**Root cause.**
`closeIdleWarmNativeSessions`
(`server/src/services/native-runtime/native-session-executor.ts`)
dropped the close error (`catch {}`) and counted it. The owners that
failed are the W2Q owners. A W2Q task parks a question and is answered
by a later run on a fresh sandbox. With the reason logged, a wave shows
two causes for the same owner:
1. The owner's close cannot settle (`native_session_close_unrecoverable:
runner did not durably suspend before checkpoint`, the governed-wait
shape from #20844). Only a successor reset could force a stop
(`discardUnsettledSession`). An idle sweep or environment delete had no
forced stop.
2. The owner's sandbox lease was already destroyed by the
pending-cleanup sweep after the successor moved to a fresh sandbox
(#20863). Every close then fails with `Daytona sandbox lease <id> is no
longer active.`, because the stop RPC has no sandbox to reach. The owner
stays registered for ever.
The Runner close memoizes its rejection, so every later attempt fails at
once with the same error. This is not the "controller sequence must be
contiguous" exit (#20946): no close failure in these waves names a
sequence mismatch.
**Expected behavior**
An idle warm owner with no turn in flight never blocks environment
delete for ever. Every failed close names its run and its error.
**Steps to reproduce**
1. Run `tests/restart-chaos/main.ts --env daytona --daytona-warm
--adapters claude --workloads W2,W2Q,W2C --faults K0 --concurrency 8
--tasks 8`.
2. Read `<wave>/sandbox-sweep.json`: `environmentDeleteBlocked` is set
and the sandboxes are `started`.
## What Changed
- `closeIdleWarmNativeSessions` now returns `failures: [{ runId,
issueId, environmentId, providerLeaseId, reason }]`. The reason is the
close error with its code, its settlement record and its cause. It also
logs `idle warm native sessions did not close` with that list. The
idle-timeout close logs its error too.
- `closeWarmNativeSessionsForEnvironment` takes `forceStopUnsettled` and
`db`. The environment delete route sets both. When the graceful close
fails, Core retires the owner as #20844 does: one confirmed Launcher
`stop`, the credential snapshot, `complete`, then instruction
collection. A failed forced stop is added to the reason.
- When retirement still fails and the owner's sandbox lease has a
`destroyed` termination receipt, Core releases the owner. It does not
contact the sandbox. It logs `warm native session's sandbox was already
destroyed; released its owner` with the close error.
- The environment delete 409 now names each run and its reasons, and
`details.nativeRunnerSessions.failures` lists them.
- Idle timeout, controller restart and tenant sleep do not force a stop.
Their conversation can still be resumed, so they only log the reason.
- Chaos harness: new check `warm-shutdown.ts`, "no leaked sandboxes /
clean warm shutdown". Every warm wave (`--daytona-warm` or `--lifecycle
warm`) runs it. It fails the wave on a blocked delete, on a sandbox left
running after cleanup, or on a warm session that did not close at
environment delete or controller shutdown. It writes
`<wave>/warm-shutdown.json` and a section in `SUMMARY.md`.
- Docs: `doc/plugins/PLUGIN_RUNNER_EXECUTION.md` and the chaos README.
## Verification
- RED on master: `native-session-executor.test.ts -t "stops an idle warm
owner whose close cannot settle"` fails: `expected { closed: 0, busy: 0,
failed: 1 } to match object { closed: 0, failed: 1, failures: [...] }`
(a bare count, no reason). `environment-routes.test.ts -t "names the
warm runner session"` fails: the 409 does not name the run. On the first
fix commit, `-t "sandbox lease is gone"` fails: the lease state is never
read, and the owner stays.
- GREEN: those tests pass. The executor suite, environment routes, run
control, recovery, heartbeat process recovery, instance settings, the
idle-sleep, eviction, release-resume, restart-recovery and
warm-checkpoint integration suites pass (11 files, 2471 tests).
`tests/restart-chaos` unit test `warm-shutdown` passes. Server and
harness `tsc --noEmit` pass.
- Daytona warm waves (Claude, W2/W2Q/W2C, K0, 8 tasks). The runner image
`sha256:e04d9ce4…` matches
|
||
|
DottaandClaude Opus 5.5
|
068d814735 |
test(chaos): run waves in Cloud's volume-free server mode; make W2 phase 1 yield (#20930)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness (`tests/restart-chaos/`) runs real agent tasks against an isolated control plane and injects faults > - Cloud servers run in the volume-free mode (`PAPERCLIP_EPHEMERAL_SERVER=true`, durable agent files, run logs in object storage) > - The harness never set that mode, so every wave ran a different server mode from staging. Paths that exist only in that mode were never tested. One of them is the per-agent directory operation lock, so `--fail-recovery-step directory-lock` could never reach its step with any workload > - The re-wake workloads W2 and W2C also lost coverage: an agent that reported phase 1 with disposition `done` closed the task, and the harness never sent the re-wake (four of eight Claude cells on 2026-11-11) > - This pull request adds `--ephemeral-server` and makes the W2/W2C brief ask for a `yielded` phase-1 report > - The benefit is that chaos waves can run in staging's server mode and the re-wake cells test the re-wake on every provider ## Linked Issues or Issue Description **What is the problem?** The restart chaos harness cannot run the control plane in the volume-free mode that Cloud and staging use. The `directory-lock` recovery step (`agent_directory.operation_lock.before_acquire`) is reached only for disposable remote copies, which need `PAPERCLIP_EPHEMERAL_SERVER=true`. W2 and W2C cells also stop before the re-wake when the agent reports phase 1 as `done`. **What is the expected behavior?** A wave can opt in to the volume-free mode on Daytona, and every restart of the wave keeps that mode. The W2 and W2C cells reach the re-wake. **Steps to reproduce** Run `--fail-recovery-step agent_directory.operation_lock.before_acquire:1` on Daytona with any workload: the check `recovery_step_failed_once` reports "never reached". Run W2 on Claude: the first run ends with the task `done` and `rewake_sent` is "never sent". ## What Changed - `assertIsolatedServerEnvironment` (`tests/runner-e2e/harness-env.ts`) takes `expected.generatedSecretsMasterKey`. It admits `PAPERCLIP_SECRETS_MASTER_KEY` only when that parameter is set and equals the value exactly. `PAPERCLIP_SECRETS_MASTER_KEY_FILE` stays refused. The check runs over the final server environment. - `--k15-fresh-home` implies `--ephemeral-server` and defaults `--storage` to `s3`. A fresh instance home without the volume-free mode loses the agents' instruction bundles, which staging keeps in Postgres and object storage. - `--ephemeral-server` (`config.ts`, `types.ts`, `main.ts`, `supervisor.ts`, new `ephemeral-server.ts` with its unit test `ephemeral-server.test.ts`): it needs `--storage s3` and `--env daytona` and refuses K14. Every server of the wave gets `PAPERCLIP_EPHEMERAL_SERVER`, `PAPERCLIP_DURABLE_AGENT_FILES`, `PAPERCLIP_SHARED_UI_ASSETS`, the wave's Postgres as `PAPERCLIP_DATABASE_LISTEN_URL`, one stable `PAPERCLIP_SECRETS_MASTER_KEY` (also written to the config key file) and `RUN_LOG_S3_*` in the wave's object store. - `rewake.ts`: the W2/W2C phase-1 brief asks for `paperclip_finish` with disposition `yielded` and a `response_wake` continuation. - W2/W2C tasks are created in `backlog` with their assignee, and a board comment starts phase 1 (`startComment` in `workloads.ts`, used by `observer.ts`). A passive board wait needs a comment-started run; without it the yield queues an immediate continuation and phase 1 wakes itself. The start comment is posted once (saved-comment check plus a stable `clientRequestId`), and the brief keeps phase 2 out of blocking `remainingWork`. - Volume-free mode drops the repository workloads (B1, B5, F1, R1) from the default list and refuses them when listed: their fixture is a `local_path` project source, which a volume-free server refuses. - `README.md`: a section for the new flag and a note on the brief. - `shape-cells.test.ts`: option parsing and refusals for the new flag. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts tests/restart-chaos/shape-cells.test.ts` passes (16 tests). - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - Live proof wave (Daytona, Codex, W2, K1 and K15 with a fresh home, `--storage s3 --ephemeral-server`): wave `w-20261111T205259-s6301` (2 tasks). The first server and both replacements (K15 with a fresh home, then K1) started and reached ready in the volume-free mode. The server process had `PAPERCLIP_EPHEMERAL_SERVER=true`, `PAPERCLIP_DURABLE_AGENT_FILES=true` and `RUN_LOG_S3_BUCKET` set. Both W2 tasks left phase 1 `in_progress` (yielded), received the re-wake and ended `done`. The wave also found product faults (reported separately, not fixed here): a re-wake after K1 failed with `runner_harness_state_mismatch: backup_without_reusable_lease` and needed Retry. - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/support.test.ts` passes, including the new master-key admission test (a matching wave key passes; an inherited key, a mismatched key and `PAPERCLIP_SECRETS_MASTER_KEY_FILE` are refused). ## Risks - Low risk. The flag is opt-in and the harness is test-only. The brief change alters the W2/W2C prompt, so results from older W2/W2C seeds are not directly comparable. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
ef755cecc6 |
fix(evals): grade case facts by the saved fact, bind case kickoffs to the CEO run, stop block-on-design on a missing blocker (#20948)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat eval catalog runs real chat journeys and grades saved
product state
> - Three cases failed in runs 43088288923 and 43101125481. In two of
them the product did the right thing
> - `cases.update-case-facts` used a word allow-list on a saved field.
`cases.create-case` read a field the product never fills for a new task
> - `task-coordination.block-on-design` timed out in the browser on a
missing row, which hid the real cause
> - This pull request grades what the user asked for, binds records by
structure, and stops with the real cause
> - The benefit is that each case fails only when the product fails, and
says why
## Linked Issues or Issue Description
**What happened**
- `cases.update-case-facts`: the CEO saved `browser: "Mobile Safari only
(bug reproduces only on mobile Safari)"` and kept the other fields. The
grader allowed only the words "safari" and mobile qualifiers, so "bug
reproduces" failed the first phase. The earlier rewording of the
scripted follow-up was not the cause: it is sent only after this phase.
- `cases.create-case`: the CEO created the case, the `workflow` and
`customer_brief` documents and Morgan's kickoff in one run, before
Morgan started. The grader required the kickoff's `sourceRunId` to be a
CEO run. The product sets `sourceRunId` to the run that holds the task
being handed off, so it is always null when the kickoff creates the
task. The grade never passed, so the harness kept waiting. Morgan then
found no app to test, handed the case back to the CEO, and the extra CEO
run broke the run bound.
- `task-coordination.block-on-design`: the CEO saved no Design blocker.
The scripted human then waited 30 s for the "Blocked by" row and failed
with a locator timeout.
**Expected behavior**
Each case accepts every valid product behavior and rejects the failures
that matter. A failure names its cause.
**Steps to reproduce**
1. Regrade the retained observation of run 43101125481 for
`cases.create-case` with the old grader: `real-first-morgan-kickoff` and
`required-documents-persist-before-investigation` fail although the
records are correct.
2. Call `corrected({ browser: "Mobile Safari only (bug reproduces only
on mobile Safari)" })` with the old grader: it returns false.
3. Run `task-coordination.block-on-design` when the CEO saves no
blocker: it fails on a UI locator instead of on the missing blocker.
## What Changed
- `cases.update-case-facts` (oracle v8): grade the saved correction
structurally, with no word lists. The browser fact differs from the
case's own setup value, and no field keeps the setup's stale literal
"not yet confirmed", in the saved case and in the reviewer's handoff
context. A person reads whether the wording says mobile Safari.
- `cases.create-case` (oracle case-create.v2): bind the kickoff and its
documents to the CEO chat run that was active when the kickoff was
saved, not to `sourceRunId`. Admit one CEO run on the primary task woken
by Morgan's own case handoff back to the CEO, and add a check that binds
it to that handoff. Run bound is now 2 to 3.
- Shared run policy (`chat-catalog-run-policy.ts`,
`chat-catalog-contract.ts`): new optional `maximumLeadCaseHandbackRuns`
for `source-settled-with-active-work`. The default is 0, so other cases
grade exactly as before. No other oracle version changes.
- `task-coordination.block-on-design` (oracle v9): after the first turn,
stop with "The CEO did not save Design as a blocker on Website" when the
edge is missing.
- Descriptors updated for the three cases.
## Verification
- `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts` on
`integrator-case-create`, `integrator-case-facts`, `runtime-dependency`,
`chat-catalog-run-policy`, `chat-catalog-readiness`,
`chat-catalog-failure-retirement`, `chat-catalog-consistency` and
`chat-ordinary-phase-callers` tests: all pass.
- Calibration in both directions:
- case facts: the retained values of runs 43088288923 and 43107042937
and four rewordings pass. The unchanged setup value, a kept stale
literal (in any field), empty facts and a first turn that left the value
unchanged fail.
- create-case: the retained run 43101125481 regrades to pass. Documents
from another run, a kickoff outside the CEO run, a kickoff by Morgan, a
handback with no handoff, a handback woken another way and two handbacks
fail.
- run policy: an undeclared handback, a lead run woken another way and a
second handback fail.
- `npx tsc --noEmit -p tests/runner-e2e`: no errors in changed files.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`:
unchanged.
- End-to-end: chat eval run linked in a comment below.
## Risks
- Low risk. Only eval code changes. The shared run-policy option is
opt-in, and its default keeps the old result.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
9c56e5465c |
fix(native): keep the agent's own unanswered question open after a turn ends without a report (#20934)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - An agent can ask the person a question and wait for the answer. Native task status comes from the server's status arbiter, not from model prose > - When a native turn ends without `paperclip_finish`, the Runner sends a default report: `done`, objective satisfied, with the agent's final message as the only evidence > - A later comment from the person moves an open question into history (#15392), so the question no longer gates completion > - Together these close a task as `done` while the agent is still waiting for its own answer and has produced nothing. No Retry appears and nothing flags it > - This pull request keeps such a task `in_review` on the agent's own unanswered question when the only completion evidence is the agent's final message > - The benefit is that a "waiting for your answer" turn can no longer become a silent false completion ## Linked Issues or Issue Description Refs #15392 **What happened?** Restart-chaos wave on master |
||
|
DottaandClaude Opus 5.5
|
65ebd96bcf |
fix(chat): keep the person's brief as a document on a new case's primary task (#20954)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - With Cases enabled, a person can ask chat to open a case from a
brief and hand it to an agent
> - The case scenario expects the brief and a workflow document on the
primary task before work starts
> - In one run the CEO copied the brief's facts into case fields but
saved only the workflow document
> - The Cases chat guidance says nothing about where the person's brief
belongs
> - This pull request adds one guidance sentence: save the brief with
`initialDocuments` in the same case-handoff request
> - The benefit is that the assignee always has the person's source,
saved before its run begins
## Linked Issues or Issue Description
**What happened**
In the chat eval `cases.create-case` (run 43107042937), the person
attached a customer bug brief and asked the CEO to open a case and have
Morgan investigate. The CEO created the case, its fields, the workflow
document and the kickoff. It did not save the brief on the primary task.
In an earlier run (43101125481) the CEO did save it, so the behavior
depends on the model's choice.
**Expected behavior**
The new primary task has the person's brief as a document, saved in the
request that creates it, before the assignee starts.
**Steps to reproduce**
1. Enable Cases. Attach a brief in chat with the CEO.
2. Ask: "Open a case for this customer bug, record the facts from my
brief, and have Morgan investigate first."
3. Read the primary task's documents: in some runs only `workflow`
exists.
## What Changed
- `CASES_CHAT_DIRECTIVE` (rendered only while Cases is enabled): one
sentence. When the person opens a case from a brief, save the brief as a
document on the primary task with `initialDocuments` on `POST
/api/companies/{companyId}/case-handoffs`. Case fields do not replace
the brief.
- `case-history-api-discovery.test.ts`: a regression test for the new
guidance.
## Verification
- Red: the new test failed before the change. Green: `npx vitest run
src/__tests__/case-history-api-discovery.test.ts
src/__tests__/agent-conversations.test.ts` (67 passed).
- The route already accepts `initialDocuments` for a newly created
primary task (`packages/shared/src/case-handoffs.ts`).
- End-to-end: chat eval run linked in a comment below.
## Risks
- Low risk. The sentence renders only when Cases is enabled. No API or
permission changes.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
6cfb1f3d87 |
fix(environments): resume a warm sandbox that its finished predecessor is still releasing (#20953)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A warm Paperclip Runner session keeps its Daytona sandbox between runs. The next run of the same task resumes that sandbox through a `reuse_by_environment` lease. > - Acquisition only resumes a lease that is `released`, `retained`, or held by the same run. A finished run releases its lease only after it saves its instruction and workspace files, which takes several seconds. > - A board wake that arrives in that window does not see the warm lease. It creates a fresh sandbox. The session's durable backup cannot be restored there, because its source lease is still held. The run fails with `native_session_recovery_failed: runner_harness_state_mismatch: backup_without_reusable_lease` and the task needs Retry, which starts a new provider session. > - This pull request makes acquisition wait, for a short bounded time, for a lease that a recently finished predecessor is still releasing, and then resume it. > - The benefit is that the follow-up continues the same warm session in the same sandbox, with no Retry, no restore and no lost provider context. ## Linked Issues or Issue Description **What happened?** Restart-chaos wave `w-20261111T205259-s6301` (Codex, Daytona warm, volume-free server, `--ephemeral-server` from #20930), task t01. The board re-wake run `e7129ec7` started at 20:57:44. Its predecessor `5cf8c306` reported `run.terminal` at 20:57:38 but saved its instruction copy until 20:57:51, so its warm lease was still `active`. At 20:57:53 the new run recorded `startup reuse: environment (fresh)`. It then failed with `runner_harness_state_mismatch: backup_without_reusable_lease` (error code `native_session_interrupted`). Recording the `workspace_finalize` operation then failed with a Postgres deadlock, and the run settled `native_finalization_missing`. Retry run `79aa18ec` worked with a fresh provider session. The K1 restart in that wave was at 20:55:32 and is not the cause: six warm runs after it reused the sandbox normally. **Root cause** `reusableLeaseCanBeResumed` in `server/src/services/environment-runtime.ts` excludes an `active` lease held by another run. That rule is correct for a running predecessor. For a predecessor that has already ended, the `active` state is a short release tail. The new run creates a sandbox, and `retiredSourceAuthorizesNativeBackupRestore` (`native-retired-backup-source.ts`) correctly refuses to restore the backup from a source lease that is not yet `released` or `retained` (`idleWarmSourceRetiresBackup`). The executor's last branch (`native-session-executor.ts`, host preparation, created allocation) then throws `backup_without_reusable_lease`. **How often** Seen in 2 of 361 retained wave result directories on this machine (`s6301` today, `s5201` earlier). It needs a board wake within a few seconds of the predecessor's terminal event, so it is timing dependent. ## What Changed - `server/src/services/environment-runtime.ts`: add `waitForSettlingReusableLease`. Before the plugin and built-in sandbox paths select a reusable lease, acquisition polls (250 ms) for up to 20 s while a matching `reuse_by_environment` lease is `active` under a different run that is `succeeded`, `failed`, `cancelled` or `timed_out` and finished in the last two minutes. It then continues with the normal selection, which resumes the released lease. - A running predecessor causes no wait (unchanged: the new run gets a new sandbox). A predecessor that ended more than two minutes ago and never released causes no wait. - `server/src/__tests__/environment-runtime.test.ts`: two red/green cases (succeeded and failed predecessor release the lease 400 ms after the successor starts acquiring) and one case for an old unreleased lease (no wait, new sandbox, old lease untouched). - `doc/execution-semantics.md`: describe the wait next to the existing warm-reuse rules. ## Verification - Red on master: `pnpm --filter @paperclipai/server exec vitest run src/__tests__/environment-runtime.test.ts -t predecessor` fails both cases with `expected 'warm-2' to be 'warm-1'`: the successor created a second sandbox instead of resuming the first. - Green: the same command passes (3 tests). `environment-runtime.test.ts`, `plugin-warm-tasks.integration.test.ts` and `environment-run-orchestrator.test.ts` pass (288 tests). Server `tsc --noEmit` passes. - Not verified in a chaos wave. The window is a few seconds wide and appeared in 2 of 361 waves, so one wave cannot show the change. ## Risks - A successor can start up to 20 s later when its predecessor's release is slow. Creating a sandbox and restoring a backup takes longer, and failing needs a Retry. The two-minute bound keeps a stuck, never-released lease from delaying every later run. - If a caller held a lock that the predecessor's release needs, the wait ends at 20 s and acquisition behaves as before. - Compared with #20863 (fresh-sandbox fallback) and #20811 (unsaved provider state marker): this change does not alter either. It reduces how often a warm continuation reaches the fresh-sandbox branch. The backup restore proof is unchanged, so an unproven backup still fails closed with a working Retry. - The `workspace_finalize` deadlock in the same run is a separate defect and is not fixed here. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edit, test runs). Default context window, extended reasoning on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
e01ba16bfd |
fix(native): continue the interrupted turn after a relaunch that failed before its runner existed (#20944)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native agent runs survive a server restart: recovery verifies the old runner stopped, then resumes the same provider session and its interrupted turn > - Recovery is a chain of steps. When one step fails once, the next attempt must still be allowed to finish the work > - Here the first recovery step was correct, its runner spawn failed once (EAGAIN), and the next attempt refused to continue the interrupted turn. The run died and needed a manual Retry > - This pull request lets the scheduled same-run resume prove that the failed launch left nothing behind, and then continue the turn > - The benefit is that a single failed spawn during recovery no longer turns a recoverable lost turn into a terminal failure ## Linked Issues or Issue Description **What happened?** Restart-chaos wave `w-20261111T190113-s5303` (local, codex, A1, fault K1 with `--fail-recovery-step runnerd-start:1`) on master `683fd1e9a5`. Run timeline: 1. K1 killed the server mid-turn. Recovery recorded `native.local_process_stopped` for the dead runner and chose `resume_dead_runner`. 2. That attempt wrote `native.process_start_requested`, then the spawn failed with `spawn paperclip-runnerd EAGAIN`. The executor persisted a scheduled same-run resume (`recoveryMode: exact_checkpoint_resume`, `providerSessionEstablished: true`, `checkpointExists: true`, `recoverable: true`). 3. The scheduled attempt resumed the session. The runner reported `provider_turn_lost_on_restore` (recoverable). The run ended `native_provider_terminal_failed` with "Automatic recovery stopped", and the task needed Retry. The same seed with K1 and no injected step passes. Root cause: in `executePaperclipNativeSession` (`native-session-executor.ts`), `resumeInterruptedTurn` is `restartRecovery.kind === "resume_dead_runner" || scheduledResumeContinuation`. The second attempt is not a restart claim, so it depends on `scheduledResumeContinuationAllowed` (`scheduled-resume-continuation.ts`). That decision refused with `local_stop_unverified`: `readNativeLocalProcessStop` reads the newest launch event, and the newest one was the failed `process_start_requested` with no process identity and no stop receipt. The checkpoint evidence was correct; the stop evidence was the gap. The refusal was not logged. **Expected behavior** When a launch fails before its runner exists, a scheduled same-run resume proves on this host that no runner for the run exists and that the launch wrote no runner state, records that stop, and continues the interrupted turn. A live runner for the run still blocks it. **Steps to reproduce** `node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env local --adapters codex --workloads A1 --faults K1 --fail-recovery-step runnerd-start:1 --fault-after-seconds 10 --concurrency 1 --tasks 1 --seed 5303` ## What Changed - `scheduled-resume-continuation.ts`: when the decision refuses only for `local_stop_unverified` on a local target, `scheduledResumeContinuationAllowed` calls the existing `recordUnidentifiedLocalRunnerStop` (the same proof explicit continuations use). It requires that the newest event is this host's launch request, that no process carries `--run-id <runId>`, and that no runner state for the run changed after the launch request. On success it records `native.local_process_stopped` (`basis: unidentified_runner_absent`) in the claim transaction and decides again. All other checks (exact checkpoint, recoverable, session established, checkpoint exists, no replacement, admitted attempt) still apply first. - `native-session-executor.ts`: logs the refusal reason when a scheduled `exact_checkpoint_resume` will not continue the interrupted turn. - `native-scheduled-resume-continuation.integration.test.ts`: two cases. A failed relaunch after a verified stop now continues the turn (`resumeInterruptedTurn: true`, attempt 2). A failed relaunch while a process with `--run-id <runId>` is alive still refuses. The fixture now advances `nextEventSeq` after inserting events directly. - `tests/restart-chaos/README.md`: records what the existing `runnerd-start:1` cell found and what now passes. ## Verification - RED on master: "continues the lost turn after a relaunch that failed before its runner existed" fails with `expected [ false ] to deeply equal [ true ]` (the runtime gets `resumeInterruptedTurn: false`, which is the path that produced `provider_turn_lost_on_restore` → `native_provider_terminal_failed`). - GREEN: `scheduled-resume-continuation`, `native-scheduled-resume-continuation.integration`, `native-exact-resume-failure`, `native-restart-recovery`, `native-runner-restart-recovery.integration`, `native-session-executor`, `native-session-resumption`, `heartbeat-native-restart-suspension` pass (1797 tests). `pnpm --filter @paperclipai/server typecheck` passes. - Chaos rerun of seed 5303 with `runnerd-start:1`: result is added in a comment on this PR. ## Risks - The proof is host-local: it only applies on the host that wrote the launch request, the same as explicit continuations. Remote targets and restart claims are unchanged. - Old saved state: a run already stuck in a scheduled resume after a failed spawn now continues on its next attempt instead of failing. That is the intended recovery; the checkpoint and attempt limits still bound it. - Sibling recovery steps (`teardown`, `directory-lock`) are Daytona-only. The saved waves that armed them (`s5311`, `s5312`) never reached the step, so this PR does not change or prove them. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
d9d5958d09 |
fix(runner): reject an out-of-order controller command instead of exiting (#20946)
## Thinking Path - Paperclip's controller and the sandbox Runner exchange numbered commands; the Runner refuses a command whose number is not exactly one past the last it processed. - Staging's most common "Runner gone" failure was a follow-up controller whose journal was behind the Runner. #20928 removed the cause on the server (a warm follow-up now launches from the journal its retired predecessor saved) and added a defence that reconciles a journal behind only by the predecessor's settled close. - The Runner still treated an out-of-order command as fatal: `durable/state.rs` returned an error and the process exited. A controller with a stale journal from any other cause (state saved by an older release, a mid-run deploy) therefore lost the Runner process and the run could only fail as "Runner gone" (`runner_remote_process_identity_unavailable`). - This pull request makes the Runner refuse such a command with a precise result instead of exiting, and makes the server rest that run with a precise code and a Retry whose next run reconciles the journal (the #20928 defence) before it opens the session. ## Linked Issues or Issue Description - Refs #20928 (the server-side root-cause fix and the journal reconciliation this relies on) - Refs #20907 (makes runnerd exit text visible in the run log) - Duplicate search: searched PRs for "controller sequence contiguous", "controller_sequence_mismatch" and "out-of-order command"; none changes the Runner's handling. ## What Changed - `runner-core/src/durable/state.rs`: a new command whose `controllerSeq` is not `lastControllerCommandSeq + 1` (behind or a gap) now yields `CommandDisposition::Reject` with `status: "rejected"` and `{ code: "controller_sequence_mismatch", message, expectedControllerSeq, receivedControllerSeq }`. The command is not executed or journaled and the last sequence is unchanged; the process keeps running. Replay of an already-processed command and the compacted-history rejection are unchanged. - `native-session-executor.ts`: a failure carrying `controller_sequence_mismatch` is classified as `native_controller_journal_behind_runner`, a permanent (terminal) failure with a Retry action that says no provider work began and that Retry reconciles the journal from the saved checkpoint or stops with a precise error. - Tests: - Rust `tests/durable_recovery.rs`: out-of-order (behind and gap) commands are rejected with the precise code, the journal is unchanged, and the next contiguous command executes. The existing `command_gaps_and_identifier_reuse_fail_closed` unit test now asserts the rejection instead of an error. - Real runnerd `src/live/runnerd-stale-controller-journal.test.ts`: a follow-up controller with a journal missing the predecessor's close now gets `"code":"controller_sequence_mismatch"`, the Runner process stays alive and its durable sequence is unchanged; with the current journal it resumes. - Server: the classification and terminal disposition of the new code. ## Verification - Red (master Runner): the Rust test panics on `expect("an out-of-order command is not fatal")`; the real-runnerd test receives `runnerd exited unexpectedly with code 1: paperclip-runnerd: controller sequence must be contiguous: expected 6, received 4`. - Green: `cargo test -p paperclip-runner-core --test durable_recovery` (4 passed); `cargo test -p paperclip-runner-core` lib suite (the only other failure, `task_memory::tests::monitor_bounds_notices`, is a timing test that passes alone); `npx vitest run src/live/runnerd-stale-controller-journal.test.ts` with a runnerd built from this branch (2 passed); server `native-session-executor.test.ts` focused tests (71 passed). - `pnpm --filter @paperclipai/paperclip-runner check:acpx-profiles`: profiles match the release manifest. No ACPX profile, attestation, pin or pack contract schema changed (`rules/acpx-profile-identity.mdc`): the change is Runner source behaviour with an existing command-result shape, so no profile version or `commandDigest` changes. ## Risks - A controller that is *ahead* of the Runner (a gap) is now rejected instead of fatal. Neither case executes the command; the Runner journal is unchanged, so this is strictly safer than exiting. - The rested run needs a Retry. Before this change the same run failed after its retries as "Runner gone" with no precise cause. - Runner source changes the Daytona image content ID; a new image is needed for chaos waves on this revision. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), 1M-token context, tool use (shell, file edits, tests) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
2719145ae2 |
fix(chat): save a requested start order as a first-class blocker (#20947)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - People use chat to tell the CEO agent how work must be ordered
> - A person said "Do not start the website until the design is
approved"
> - The CEO only promised to hold the work. It did not save a blocker,
so nothing stopped the Website task
> - Chat guidance names blockers only when chat creates new tasks. It
says nothing about an order between existing tasks
> - This pull request adds one guidance line: save the order as a
blocker on the waiting task
> - The benefit is that Paperclip enforces the order and wakes the
assignee when the prerequisite is done
## Linked Issues or Issue Description
**What happened**
In the chat eval `task-coordination.block-on-design` (run 43088288923),
the person asked the CEO to keep Website from starting until Design is
approved. The CEO read both tasks and replied "I'm holding both until
the design is actually approved". It saved no blocker. Website stayed
startable.
**Expected behavior**
The CEO adds Design to Website's `blockedByIssueIds`. Website cannot
start while Design is open. Its assignee wakes when Design is done.
**Steps to reproduce**
1. Create Design and Website tasks in backlog. Assign Website to an
agent.
2. In chat with the CEO, say "Do not start the website until the design
is approved."
3. Read Website: `blockedBy` is empty in some runs, because the guidance
does not tell the agent to save the order.
## What Changed
- `AGENT_CHAT_DIRECTIVE`: one paragraph. When the person says an
existing task must not start until another is done or approved, add the
prerequisite to the waiting task's `blockedByIssueIds` with `PATCH
/api/issues/{id}`, and keep existing blockers. A comment or a promise
does not stop the work. Link both tasks.
- `agent-conversations.test.ts`: a regression test for the new guidance,
in the directive and in the rendered chat prompt.
## Verification
- Red: the new test failed before the guidance change.
- Green: `npx vitest run src/__tests__/agent-conversations.test.ts
src/__tests__/case-history-api-discovery.test.ts` (66 passed).
- Local scripted-human check (dev server, Playwright): with the blocker
saved, the Website task shows the Design row under "Blocked by", a Todo
click returns 200 without a Morgan run, Design done wakes Morgan with
`issue_blockers_resolved`, and the follow-up chat message sends.
- End-to-end: chat eval run linked in a comment below.
## Risks
- Low risk. The change adds about 470 characters to the chat prompt. It
changes no API or permission. The route and field already exist and
agents already use them.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
Devin FoleyandPaperclip
|
6a43890326 |
Fix image attachments in remote native workspaces (#20939)
## Thinking Path > - Paperclip manages AI agents and their work. > - Native runs must receive the files that users attach to their messages. > - Remote wake preparation returned file metadata without readable bytes. > - The historical chat reader also rejected every remote target. > - Command-backed sandboxes and SSH targets already provide a bound command transport. > - This pull request stages verified files through that transport and clears them after the turn. > - Agents can then open attached images with their normal image reader. ## Linked Issues or Issue Description Refs #19740. This change covers SSH and command-backed sandbox execution, including Daytona. Plugin Runner tasks without a command transport still need a separate versioned file-transfer contract. This PR does not close that broader issue. The open attachment-recovery PR #20778 concerns provider-session attachment journals, not input files. **What happened?** A user uploaded an image and sent it to an agent. The remote native run received its filename and size, but no readable file. Wake preparation reported `remote_workspace_staging_unsupported`. The historical read fallback also refused remote execution. **Expected behavior** Copy the exact admitted attachment into the bound remote workspace. Let the provider read the image. Preserve company, task, source-comment, size, and hash checks. Clear temporary file contents when the turn ends. **Steps to reproduce** 1. Select a native agent on a command-backed sandbox or Linux SSH environment. 2. Attach a PNG to its current task or in-app chat message. 3. Ask the agent to inspect it. 4. Check that wake preparation supplies a readable workspace-relative image path. ## What Changed - Add a remote staging owner with bounded commands, stdin transfer, hash verification, opaque paths, and safe image extensions. - Confine Linux filesystem operations through directory descriptors. Refuse symlinks and replacement files during cleanup. - Bind current wake files and authorized historical external-chat reads to the selected lease transport. - Join active writes during cancellation. Retain failed cleanup work until scope close. - Tell providers to open image files with an image-capable reader. - Handle stdin stream errors in the shared SSH transport. Failed uploads return failure instead of raising an unhandled EPIPE or reporting success. - Add database, transport, cancellation, integrity, cleanup, executor-wiring, and Linux filesystem regressions. Document support and limits. ## Verification - Prepared an isolated source checkout with `node scripts/agent-preflight.mjs --skip-rust-debug`. - Focused suites: `pnpm exec vitest run server/src/services/native-runtime/remote-attachment-staging.test.ts server/src/services/native-runtime/native-runner-file-handoff.test.ts server/src/services/native-runtime/chat-attachment-read.test.ts` — 74 passed, 4 Linux-only tests skipped. - Executor checks: `pnpm exec vitest run server/src/services/native-runtime/native-session-executor.test.ts -t 'attachment staging to the remote|stages from the authenticated|verifies frozen staged admission'` — 7 passed. - Server typecheck: `pnpm --filter @paperclipai/server typecheck`, followed by the server layered check after the final changes. - `git diff --check` passed. The local secret, PII, and private-reference diff scan found no matches. - `pnpm verify` returned exit 2 because the documentation change requires broad CI. No full local suite is claimed. - The four Linux filesystem tests were skipped locally because this macOS host has no available Docker daemon. Linux CI executed all 17 remote-staging tests successfully, including those four confinement tests: [server shard 10](https://github.com/paperclipai/paperclip/actions/runs/43102035096/job/130796646266). - SSH regression: the real pipe test reproduced two unhandled EPIPE errors before the fix, plus false success when the peer exited with code zero. The fixed SSH tests and existing argv tests pass (12 tests). The remote-staging transport suite also passes (13 tests; 4 Linux tests require CI). - `pnpm --filter @paperclipai/adapter-utils typecheck` passed after the SSH change. - Final revision `1c5a9288b8d5f267ddeac84638ff6075dbbbf8ec`: all applicable CI jobs passed, including typecheck, build, general and serialized tests, Runner checks, canary dry run, and all eight browser shards. [CI run](https://github.com/paperclipai/paperclip/actions/runs/43102035096). Linux CI also passed all 6 native SSH tests: [server shard 5](https://github.com/paperclipai/paperclip/actions/runs/43102035096/job/130796646349). - Greptile reviewed that final revision at 5/5 with no remaining findings. The review thread is resolved. GitHub reports the PR mergeable with a clean merge state. - No live model turn or deployed release is claimed. ## Risks - Remote staging requires Linux, Node, and a command transport bound to the selected lease. Plugin Runner tasks without that transport retain their unavailable result. - Each file uses bounded independent commands. Slow transports can delay preparation or cleanup. File bytes stay out of command arguments and provider-facing error messages. - An abrupt controller or remote-host failure can bypass cleanup. Ephemeral sandbox disposal remains the backstop. Retained files are never blindly overwritten or deleted. - Previous-release provider sessions and checkpoints retain their format and identity. New turns receive newly staged input files. Existing restart-admission checks still apply. No migration, runner-image change, or account-credential transfer is required. ## Model Used OpenAI `gpt-6.1-sol` through the Codex harness in T3 Code, with xhigh reasoning, repository inspection, code execution, and tests. No subagents were used. The context-window size is not exposed by this runtime. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
757a20d2f1 |
fix(runner): keep remote recovery and launch through a short plugin RPC stall (#20945)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A remote run starts `paperclip-runnerd` in a sandbox. The controller
reads and controls it through the environment plugin
(`environmentExecute`).
> - #20940 made a stalled supervision probe "unknown" while the Runner
stays connected. Other callers still turn the same plugin stall into a
terminal failure.
> - During restart reattachment, the saved-state read, the
identity-marker read and the process check share a 30-second budget. One
plugin RPC that stalls for 40 seconds uses all of it, so a short stall
fails a reattachment to a healthy Runner.
> - At launch, a stalled launch call or identity read fails the run with
`runner_process_termination_unconfirmed` or
`runner_remote_process_identity_unavailable`, although the runner may be
healthy.
> - This pull request treats a plugin RPC timeout as "unknown" and
retries the read-only call within a bounded stall budget. A stalled
launch call is not repeated: the controller reads the launch nonce's
identity marker instead.
> - The benefit is that a short plugin stall during recovery or launch
no longer fails a healthy run. Every positive answer from the sandbox
keeps its meaning, and the bound still fails closed.
## Linked Issues or Issue Description
Refs #20940, #20909
**What happened?**
A plugin `environmentExecute` RPC stall of more than 40 seconds makes a
remote recovery read fail. Plugin RPCs time out at the requested timeout
plus a 30-second overhead, so a 10-second probe can take 40 seconds. The
reattachment read budget is 30 seconds. One stalled read therefore
failed reattachment with `probe_transport_failed` or `probe_timeout`. At
launch, a stalled launch call rejected the run with exit unconfirmed. A
stalled identity read (20-second wait, 2-second reads that can take 32
seconds) failed the run with
`runner_remote_process_identity_unavailable`.
**Expected behavior**
A plugin RPC timeout is a stall of the plugin, not an answer from the
sandbox. Read-only recovery reads retry within a bound. A launch call
that stalls is not repeated. The launch nonce's marker decides whether
the one runner was launched. At the bound, the run fails closed and
Retry works.
**Steps to reproduce**
1. Start a remote (Daytona) run.
2. Restart the controller so that the run reattaches, or start a new
run.
3. While the controller reads the sandbox, stop the plugin worker for 45
seconds.
4. The reattachment or launch fails.
**Paperclip version or commit**
master at
|
||
|
|
5e2243f081 |
fix(ui): align sidebar avatars and navigation icons (#20926)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The left navigation mixes agent avatars, task status icons, and page icons. > - Agent avatars used 20px boxes. Task and page icons used 16px boxes. > - The different widths moved agent labels 4px to the right of other labels. > - This pull request gives these icons the same 16px square slot. > - Shared label positions make pinned and recent items easier to scan. ## Linked Issues or Issue Description Refs #19685, which added mixed agent and page pins. No duplicate alignment PR was found. **What happened?** Agent avatars in pinned and recent navigation were wider than task and page icons. Their labels started 4px farther to the right. Inline icon layout also produced different box heights. **Expected behavior** Use the existing 16px task and page icon size for all leading navigation icons. Center each icon in a square slot. Start the labels at the same horizontal position. **Steps to reproduce** 1. Pin an agent conversation and a page such as Routines. 2. Open agent conversations and tasks so both appear in Recents. 3. Compare the icon boxes and label positions across the rows. **Paperclip version or commit** Base commit: `c5e6ab5ca813`. **Deployment mode** Local source checkout. This is a client layout issue. ## What Changed - Use 16px agent avatars for pinned and recent conversation rows. - Give `SidebarNavItem` a fixed 16px square icon slot with centered contents and left-aligned text. The existing token scale supplies the size. - Add light and dark Storybook examples with mixed icons and long labels. The browser assertions compare slot sizes, label positions, and glyph bounds. - Add a production `CoordinatorSidebar` regression test that compares pinned and recent avatar dimensions with a task status icon. ## Verification - `pnpm exec vitest run ui/src/components/SidebarNavItem.test.tsx ui/src/components/CoordinatorSidebar.test.tsx ui/src/components/SidebarAgents.test.tsx` passes: 59 tests across three files. - Regression proof: the new production-avatar test fails with the original 20px avatars (`expected 20 to be 16`) and passes with 16px avatars. This addresses the initial review's test coverage gap. - `pnpm check:token-gates` passes. - `pnpm --filter @paperclipai/ui typecheck` passes. - `pnpm --filter @paperclipai/ui build` passes on the final source tree. - `pnpm verify --base origin/master` returns exit 2 because the new Storybook file needs broad CI coverage. The focused local tests run separately. CI must supply the remaining coverage. - Browser inspection on macOS checked all 13 mixed rows in light and dark themes. Every icon slot measured 16 by 16 CSS pixels. Every label started at the same position. Before and after captures use production components with fixture data; they do not prove server persistence or permissions. - Review `sidebar-icon-alignment--dark` and `sidebar-icon-alignment--light` in Storybook to inspect the mixed rows and run their layout assertions. - [Greptile review](https://github.com/paperclipai/paperclip/pull/20926#issuecomment-6806138954): 5/5 on `e779f4707e7826d577b23a2991f3f36755e954be`. The initial 4/5 review requested production Recents coverage. The added regression test addresses that finding; the thread is resolved. - CI queue recovery: the superseded first run was canceled. The next run stalled before runner assignment; GitHub accepted force-cancel but left it queued and refused the rerun request. The final empty commit requests fresh CI without changing the source tree. Required CI is tracked in [run 43095809539](https://github.com/paperclipai/paperclip/actions/runs/43095809539). Its jobs are still queued; CI completion is not yet claimed. - External blocker: after more than 15 minutes, the fresh run still has zero jobs. Other repository PR runs are also queued. A repository Actions or runner administrator must restore dispatch before the required typecheck, test, build, and E2E gates can report results. The root cause of the dispatch delay is not established. ## Risks - Low risk. The shared icon slot also affects other navigation rows. Existing badge, rail, and expanded-state tests cover the shared component. - Agent avatars become smaller in pinned and recent navigation, as requested. No data or API contracts change. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected. This change is a scoped UI bug fix. ## Model Used - OpenAI GPT-6 via Codex. The host does not expose the exact served model snapshot or context window size. Used reasoning, code editing, shell execution, and browser inspection. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Scott Tong <scott@scottsmbpm5max.lan> Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
612cc7e3e6 |
fix(heartbeat): renew every legacy controller lease in one shared statement (#20924)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A legacy run holds a durable controller lease while it prepares. The server renews the lease every 10 seconds, and it stops the run if no renewal succeeds within the 60 second lease. > - Each run renewed in its own transaction on a private pool of two connections. Each renewal used six round trips (begin, statement timeout, transaction id, notify, update, commit) and woke live-run recovery for that run. > - When many runs prepare at the same time on a loaded database, the renewals queue behind each other. The queue grows with the number of runs, and every run loses its lease at about the same time. > - A healthy server must not fail its own runs because it has many of them. > - This pull request renews all leases of one server process through one shared queue. Requests that arrive while a renewal is in flight share the next statement. > - The benefit is that the renewal cost no longer grows with the number of live runs, so a burst cannot starve the lease. ## Linked Issues or Issue Description **What happened?** On a staging server that did not restart, 22 runs failed with `controller_lease_lost` in about 13 minutes. Every run was a legacy run in the `preparing` stage with `providerWorkStarted: false`. Every loss had cause `renewal_deadline`. The renewal had been pending 28 to 48 seconds (`renewalPendingMs`). The deadline callback ran on time (`deadlineLateMs` 3 to 861 ms), so the event loop was not blocked. The successful renewals before the loss had each taken 20 to 30 seconds, because the deadline fired only 25 to 48 seconds after the last renewal (`sinceRenewalMs`). About 30 runs were live in the process. The first wave started when a user retried about 14 blocked tasks at once. The automatic retries of the first wave then failed the same way, which made the second wave. 12 of 16 affected tasks became blocked. **Steps to reproduce** 1. Start about 30 legacy runs in one server process. 2. Make each renewal transaction slow (for example, a loaded database). 3. Wait one lease period. **Expected behavior** The leases stay owned. The renewal cost does not grow with the number of runs. **Actual behavior** All runs fail with `controller_lease_lost` (`renewal_deadline`). ## What Changed - `server/src/services/legacy-controller-lease.ts`: All controllers of one process renew through one queue per database. One flush is in flight at a time. Requests that arrive during a flush share the next flush, so a renewal waits behind at most one other statement. A flush is one transaction on the private lease pool: `statement_timeout` and one statement that renews all requested rows. - The statement locks the requested rows with `for update skip locked`. A row that another transaction holds is reported busy, and only that run retries after 250 ms. It does not stall the other runs in the batch. A row that is not this boot's live legacy lease is reported rejected, as before. - Renewal no longer emits the `active-execution-recovery` notification. A renewal only moves the deadline later. Recovery already has a deadline at the earlier expiry and rereads the moved deadline then. This removes one recovery pass, `pg_current_xact_id` and `pg_notify` per renewal. - The lease contract does not change: 60 second lease, 10 second renewal, boot-id fence, expired leases cannot renew, the local deadline is measured from the attempt start, and the loss causes and timings are the same. - `doc/run-log-events.md`: Documents the shared renewal queue. Upgrade story: no persisted shape changes. The row columns, lease length and boot-id fence are the same. A lease held by the previous release has a different boot id, so the new batch rejects it and leaves it untouched. A test covers a previous-release lease in the same batch. ## Verification - Red on `origin/master`, green with this change: - `keeps thirty preparing controllers owned when every renewal takes four seconds on a two-connection pool`: before, all 30 controllers lost the lease with `renewal_deadline` (the staging shape). After, none lose it. - `renews fifteen concurrent controllers in one statement` (real Postgres): before, 15 update statements. After, 1. - `renews through one private statement without waking live-run recovery`: before, the renewal woke `active-execution-recovery`. After, it does not. - `live-recovery-work.test.ts` `renews without waking recovery and re-reads the moved controller deadline at the old one`: before, 2 recovery passes per renewal. After, none, and recovery still moves to the new deadline at the old one. - Safety cases: a row lock on one run does not delay another run; a previous-release lease in the same batch stays untouched; a terminated lease connection is retried; existing deadline, retry, rejection and revocation tests pass. - `npx vitest run src/services/legacy-controller-lease.test.ts src/__tests__/live-recovery-work.test.ts`: 88 passed. - `pnpm verify`: see the CI checks on this head. ## Risks - Medium-low. All renewals of a process now share one statement. A failed flush fails every request in it; each watcher retries independently, as it did before. - A busy row is retried every 250 ms until the lease deadline. Before, the renewal waited on the row lock for up to the statement timeout. The loss behavior is the same. - Recovery wakes at the old lease expiry instead of after each renewal. It finds the live lease and moves to the new expiry. - Not in scope: the downstream workspace recovery that blocked tasks after a lost lease (`legacy_execution_requires_reconciliation`, `source_unavailable`). > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (claude-opus-5-5), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
228b0169be |
fix(dispatch): let the answer to the agent's own question continue the task it blocked on (#20931)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - An agent can ask the person a question (`request_human_input`) and wait for the answer > - A later comment from the person moves an open question into history (#15392). The agent still needs the answer, so it reports a blocker and the task becomes Blocked > - The person then answers. Dispatch cancels the answer's continuation with `issue_not_in_progress`, because only a dependency-blocked task may take an answer while Blocked > - The task rests Blocked with a failed run and a Retry. The answer is lost until a person clicks Retry > - This pull request lets the answer continue the task when the same agent blocked it to wait for that answer, proven from durable records > - The benefit is that the person's answer always reaches the agent that asked, without a Retry ## Linked Issues or Issue Description Refs #15392, Refs #20872 **What happened?** Restart-chaos waves with no fault injected fail every W2Q task on every provider (Claude, OpenCode, Codex) with `R3:retry_needed`. For Claude task t03 on master |
||
|
DottaandClaude Fable 5.2
|
b01959cb15 |
fix(recovery): release a handed-off run's saved-files hold for the person it handed the task to (#20941)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - An agent can hand its own task to a person for review. With #20884, a native run that does this keeps running and finishes its turn. > - A restart can land between that hand-off and the end of the same turn. Restart recovery then settles the run as failed and keeps its unsaved sandbox files, so no work is lost. > - Automatic workspace recovery saves those files. But the workspace-repair hold then stayed on the person's task, because the hold only released for a newer agent owner. > - This pull request releases that hold for the person who received the task through the stopped run's own recorded hand-off, once the files are saved. > - The benefit is that the person's task rests in review with no stale "Saved work recovered" hold, and the agent's files are still saved first. ## Linked Issues or Issue Description Follow-up to #20884 (merged). It was reviewed there as a known limit: "Restart blocks the person's task". It uses the `resultJson.humanSelfHandoff` record that #20884 added. Related: #19184, which released the same hold for a newer agent owner. **What happened?** A native run handed its own task to a person, and a restart landed before the run finished its turn. Restart recovery settled the run and retained its files. After automatic recovery saved them, the task still had a workspace-repair hold ("Saved work recovered. Send a message to continue."). The person had nothing to do with that run's work, and nothing released the hold. **Expected behavior** The hold protects the workspace until the files are saved. After that, the task rests with the person and has no execution blocker. **Steps to reproduce** 1. A native run records a hand-off to a person, and the task is now `in_review` with that person. 2. A restart settles the run as failed, with its prepared remote workspace retained. 3. Automatic workspace recovery completes. 4. Before this fix, `getExecutionBlocker` still returns the run's workspace-repair hold. ## What Changed - `server/src/services/execution-blocker.ts`: `restoreHoldReleasedForCurrentOwnerPredicate` now also releases a recovered file-preservation hold in one more case. The task must be assigned to a person, and the stopped run's `resultJson.humanSelfHandoff` must name that task and that person. The existing proof is still required: the completed-recovery receipt bound to the action, and a copyback rather than a snapshot-only save. Until the files are saved, the hold stays. - `server/src/services/workspace-recovery-admission.test.ts`: two embedded-Postgres tests. - The hold blocks the person's task before recovery and releases after it. - A hand-off recorded for a different person keeps the hold. - `doc/execution-semantics.md`: one sentence on the restart case. ## Verification - Red: with the `execution-blocker.ts` change removed, `releases a handed-off run's recovered files hold for the person it handed the task to` fails. `getExecutionBlocker` still returns the hold after `recoverRetainedWorkspace`. - Green: `cd server && npx vitest run src/services/workspace-recovery-admission.test.ts src/__tests__/interrupted-preparation-hold-settle.test.ts src/services/explicit-native-continuation.test.ts src/services/native-runtime/native-human-self-handoff.test.ts`. All 617 tests pass. The server typecheck passes. - Chaos: the H2 workload from #20884 covers the hand-off with a K1 restart after it. Locally the runs had no unsaved files, so no hold appeared. This PR's case needs a remote sandbox with unsaved files. **No Daytona wave was run.** ## Risks - The release applies only when the stopped run's own record names the current person and task, and only after the completed-recovery receipt. Other holds, snapshot-only saves and unsafe archives are unchanged. - No schema change. A run without the record behaves as before. - This branch is rebased onto master after #20884 merged. The only conflict was an adjacent import line. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic), model ID `claude-opus-5-5`, through Claude Code with tool use and code execution. The session does not expose the context window or reasoning mode. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
DottaandClaude
|
0bf3236bc8 |
test(restart-chaos): staging class R cells CR1-CR3 and sandbox capture on every Daytona wave (#20909)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart-chaos harness (`tests/restart-chaos/`) reproduces staging failures before users meet them > - Staging fails with `runner_remote_process_identity_unavailable` when the remote Runner or its provider is gone after a deploy, a stall or a re-wake, and nobody has seen the sandbox side > - The harness had no cell for a provider-only death or a plugin RPC stall, and its sandbox capture ran only on K15 waves > - This pull request adds the staging class R cells CR1–CR3, and captures the sandbox side on every Daytona wave > - The benefit is that a wave that reproduces the refusal also keeps the evidence of why the Runner or provider died ## Linked Issues or Issue Description **What is the problem?** The harness cannot answer why a remote runnerd or its provider dies. Its capture ran only around K15 restarts. It recorded no process start times, sockets or provider log files. It had no fault that kills only the provider or stalls a plugin RPC. **What should change?** Add cells for each staging shape, and capture the sandbox before, during and after each one, on every Daytona wave. **Context** Refs #20907 (keeps runnerd's stderr in the sandbox, and reports it on failed recovery). Refs #20893 (W2 re-wake cells). ## What Changed - `class-r.ts` (new) adds three faults, selected with `--faults`: - **CR1:** a K15 redeploy with a held outage. `--cr1-hold-seconds`, 0–600, default 360. - **CR2:** SIGKILL of only the provider process under the run's `paperclip-runnerd`, inside its sandbox. Runnerd stays alive. By default the controller stays up; `--cr2-hold-seconds` adds a K15 outage. - **CR3:** SIGSTOP of the server's Daytona plugin worker. `--cr3-stall-seconds`, 41–600, default 45. Each CR task must pass the `class_r_fault_applied` check. - `daytona-runner-logs.ts` adds `CLASS_R_PROBE`. It reads process start times and `wchan`, sockets, the runtime file listing, and the tails of the Runner's and the provider's state and log files. It also adds `executeInWaveSandboxes`, the fixed-script runner that CR2 uses. - `main.ts` and `supervisor.ts`: - every Daytona wave captures with the class R probe when a task rests and at wave end; - CR cells also capture before the fault, after the stop, before the restart, and every 15 s for 180 s after ready; - `--sandbox-watch-seconds` captures on a clock for the whole wave. - The README has a new "Staging class R cells" section. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 49 files, 600 tests pass, including the new `class-r.test.ts`. It covers option parsing, planning, the kill command's quoting and its refusal of anything but a run id, plugin worker selection, the watch schedule and the applied check. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - Daytona waves run from this branch plus #20907. The results are in the follow-up comment. ## Risks - This change is limited to the harness. CR2 is its only in-sandbox mutation: a SIGKILL of the provider, run by a fixed script only in sandboxes labelled with the wave's company and environment. The script never signals runnerd. - More captures add Daytona toolbox calls on every Daytona wave. Each capture has a 45 s deadline, and a failed capture is recorded and never fails the wave. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, test runs, Daytona waves). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
48d23d1530 |
fix(runner): keep a connected remote Runner through a stalled plugin supervision probe (#20940)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A remote run starts `paperclip-runnerd` in a sandbox. The controller
monitors the process with a health probe that goes through the
environment plugin (`environmentExecute`).
> - A stall of the Daytona plugin worker makes that probe RPC time out
after 40 seconds. The monitor treated the timeout as "process
supervision failed".
> - The transport then failed closed with
`runner_process_termination_unconfirmed`. The run ended
`notification_transport_failed`, `recoverable: false`, and the close
path suspended a healthy Runner. The task rested blocked with Retry.
> - Nothing was wrong with the sandbox, the Runner or the provider. The
Runner held its own authenticated connection to the controller during
the full stall.
> - This pull request makes a failed or timed-out probe "unknown". While
the Runner stays connected on its own channel, the monitor keeps the
same runner and turn and probes again, for a bounded time.
> - The benefit is that a short plugin stall no longer kills a healthy
run. The fail-closed cases stay the same.
## Linked Issues or Issue Description
Refs #20909 (chaos cell CR3, which reproduces this failure)
**What happened?**
A 45-second stall of the Daytona plugin worker killed a healthy run.
Chaos cell CR3 reproduced it 2 of 2 times (OpenCode and Claude,
Daytona). A staging run had the same failure with no deploy: a plugin
`environmentExecute` RPC stalled for more than 40 seconds. The run
failed with "Notification transport failed closed:
runner_process_termination_unconfirmed: runnerd process supervision
failed: RPC call \"environmentExecute\" timed out after 40000ms". The
close path then sent `runner.suspend`, and the healthy Runner suspended
and exited. The task rested blocked with Retry.
**Expected behavior**
A probe that times out or fails in transport is not proof that the
process stopped. While the Runner stays connected on its own channel,
the run continues. If supervision cannot come back within a bound, the
run rests with a working Retry. A probe that positively reports the
process gone still starts the existing recovery.
**Steps to reproduce**
1. Start a remote (Daytona) run with a T1 workload.
2. Send SIGSTOP to the server's Daytona plugin worker for 45 seconds,
then SIGCONT.
3. The run fails `notification_transport_failed` about 40 seconds into
the stall.
**Paperclip version or commit**
master at
|
||
|
DottaandClaude Opus 5.5
|
884a9e19ea |
fix(native): launch a warm follow-up from its predecessor's saved journal, and restart a Runner that exited before session.open (#20928)
## Thinking Path - Paperclip runs agent work through a controller (server) and a runner (`paperclip-runnerd`) that exchange numbered commands; the runner refuses any command whose number is not exactly one past the last it processed. - Staging's most common failure is "Runner gone" (`runner_remote_process_identity_unavailable`, `native.remote_runner_recovery_failed` with `process_not_running`, `runnerLifecycle: ready`, `runnerSettled: false`). Chaos wave `w-20261111T192658-s5201` (Daytona, warm, Codex, W2/W2Q/W2C re-wake cells) captured it live: the new runner wrote `controller sequence must be contiguous: expected 13, received 11` (and `expected 5, received 3`) and exited 0.15 s after authenticating. - In all three captured failures the follow-up run started 1–2 s before the predecessor's warm session was closed (`transport_close_requested` for the predecessor arrives after the follow-up's `run started`). The close is issued by the follow-up's own admission. - Admission first materializes the controller projection for the session root. With an idle retained warm owner the root only holds the external-persistence marker, so `restoreWarmRunnerdController` copies that owner's saved checkpoint into the root. Later in the same admission, `executePaperclipNativeSessionWithinScope` finds the owner cannot be reused (configuration, process or GitHub access changed) and retires it. Retirement sends `runner.drain` and `runner.suspend`; both are saved to the owner's checkpoint before they are sent, but the root still holds the projection taken before the close. - The follow-up then opens its controller from that projection. Its journal is two commands behind the runner's durable `lastControllerCommandSeq`, so its first command repeats sequence 11; the runner treats this as fatal and exits, and every later attempt only reattaches to a process that is gone. - So the saved controller journal was persisted correctly; it was not selected: the projection is never refreshed after the retirement that extends it. ## Linked Issues or Issue Description - Refs #20874 (the runner-exit proof this extends from a rejected `session.open` to a Runner that exited before answering it) - Refs #20907 (makes the runnerd exit text, including `controller sequence must be contiguous`, visible in the run log; independent of this fix) - Refs #20853 (related: resumes a remote Runner that died at a settled `ready` cut; this PR covers the unsettled `ready` state left by an exit before `session.open`) - Duplicate search: searched open and closed PRs for "controller sequence contiguous", "runner_remote_process_identity_unavailable" and "Runner gone"; none fixes the stale controller projection. ## What Changed - `server/src/services/native-runtime/warm-controller-journal.ts` (new): - `refreshRetiredWarmOwnerProjection` reloads the retired owner's canonical checkpoint into the session root after retirement, only when the root still projects that owner and the saved journal extends the projected one. A diverged journal holds admission with `native_runner_checkpoint_projection_held:retired_owner_journal_diverged`; a projection changed underneath holds with `...:journal_changed`. - `reconcileControllerJournalWithRunner` is the defence for state already saved in the bad shape (old-saved/new-resumed, mid-run deploys). When the remote runner state shows its journal ahead of the controller projection and the runner's last command is a completed `runner.suspend`, it reconciles only if every extra sequence is a completed `command_close_drain_*`/`command_close_suspend_*` in the runner's processed-command journal and the predecessor's canonical checkpoint holds exactly those commands. Anything else refuses before launch with `native_controller_recovery_missing_evidence:controller_journal_behind_runner` instead of guessing. A runner whose last command is not a completed suspend is left to the existing live-adoption checks. - Defence in depth for the server side of the same failure (`native_session_recovery_failed: runnerd exited while waiting for session.open`, then `runner_remote_process_identity_unavailable` on every same-run retry): a Runner proven to have exited while this attempt's opening command (`session.open`, or the recovery probe) was still unanswered is a failed fresh start, not a live Runner to reattach. - `runnerd-codex-transport.ts`: `runnerExitConfirmed()` (#20874's proof) now also holds for a resumed session whose provider identity was preloaded, when the attempt's opening command is still pending at the proven exit. - The Codex driver's recovery result, `HarnessDriverBackend` and `native-session-runtime` carry that proof on the `native_session_recovery_failed` error. - The executor records `runnerExitedBeforeSessionOpen` on the settled failure; restart recovery puts `exitedBeforeSessionOpen` on the `reattach_remote_runner` claim; `verifyRemoteRunnerOrSettledExit` and `verifyStagedRemoteRunner` then start a new Runner when the process is gone and the saved Runner state has no pending terminal delivery, provider cleanup or warm transition (any lifecycle other than an active turn). Attempt budget and identity checks are unchanged. - `native-session-executor.ts`: calls the refresh right after the configuration-changed retirement of a warm owner, and the reconciliation in `restoreRemoteSuspendedRunnerCheckpoint` with the runner state it already reads (no added remote read, no wait before `turn.start`). - Tests: - `native-session-executor.test.ts`: new `changed-external-checkpoint` scenario of "rotates an owned warm codex remote runner before checking changed configuration" with a real `DurablePrpControlPlane`: the retired owner's close saves drain/suspend, and the journal the follow-up launches from must end with them. - `warm-controller-journal.test.ts`: refresh, divergence hold, untouched marker/other run, reconcile of a settled close, refusal when the canonical journal cannot prove the close, refusal for any other extra command, and no action for a live runner. - `packages/paperclip-runner/src/live/runnerd-stale-controller-journal.test.ts`: real `paperclip-runnerd` + fake Codex. A follow-up controller on the predecessor's suspended runner with a journal that lacks the close commands reproduces `controller sequence must be contiguous` and loses the runner; with the current journal it resumes. ## Verification - Red for the added part: `verifyRemoteRunnerOrSettledExit` with the staging state (process gone, lifecycle `ready`, unsettled outbox) rejects with `runner_remote_process_identity_unavailable` without the change; with `exitedBeforeSessionOpen` it resolves to a fresh start, and stays fail-closed for a pending cleanup, terminal delivery, active turn or exhausted budget. Without the propagation, the backend recovery result lacks `runnerExitConfirmed`; the real-runnerd stale-journal test asserts the transport's exit proof after the contiguity exit (`expected false to be true` on the old transport). - Red on master (executor unchanged, new scenario): `AssertionError: expected [] to deeply equal [ 'runner.drain', 'runner.suspend' ]` — the follow-up launched from the projection taken before the predecessor's close. - Green: `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts -t "rotates an owned warm codex remote runner before checking changed configuration"` (14 passed), `npx vitest run src/services/native-runtime/warm-controller-journal.test.ts` (7 passed). - Real runner: `cd packages/paperclip-runner && npx vitest run src/live/runnerd-stale-controller-journal.test.ts` (stale journal → `controller sequence must be contiguous`; current journal → resumes). - `pnpm --filter @paperclipai/server typecheck`, `pnpm --filter @paperclipai/paperclip-runner typecheck:typescript`; `pnpm verify` selected broader verification (a new runner test file is outside the resolved graphs), so broad coverage comes from required CI. ## Risks - The refresh replaces the root projection only with a checkpoint whose identity matches and whose journal extends the projection; otherwise it holds admission with an existing `native_runner_checkpoint_projection_held:*` code, which is safer than launching a runner that will exit. - The defence refuses launch when a runner is ahead by anything other than a provable settled close. Before this change that launch always ended in a runner exit and a failed run, so the refusal only changes the error code and the moment of failure. - This PR does not change the runner. The runner still exits on an out-of-order new command; a follow-up runner PR will turn that into a rejected command result. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), 1M-token context, tool use (shell, file edits, tests) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
edd00526e7 |
fix(native): park a response_wake with no admitted response instead of self-waking (#20935)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A native agent run ends with a structured result. The status arbiter
turns that result into an issue status and next actions.
> - A result can report `yielded` with `continuation.kind:
"response_wake"`. This means "wake me when somebody responds".
> - Only a direct Board comment gave this kind a passive wait. Every
other `response_wake` became an immediate `issue_status_changed` wake to
the same agent.
> - The agent woke, saw nothing new, reported `response_wake` again, and
woke again. Nothing bounded this loop. It spent provider budget and its
back-to-back runs caused other failures on warm sandboxes.
> - This pull request parks such a result as a visible wait, and puts
declared continuation wakes under the rewake throttle as a backstop.
> - The benefit is that an agent that waits for input stops and costs
nothing until a real response arrives.
## Linked Issues or Issue Description
**What happened?**
Restart-chaos waves on master `683fd1e9a5` (no faults injected) showed
one task run 30 to 66 times, about 12 s apart. Example: wave
`w-20261111T190352-s5101`, task t02 (W2C), 65 runs in 13 minutes. Each
run reported `yielded` with `continuation.kind: "response_wake"` and the
summary "No changes made. This status-change wake does not trigger phase
2". Each run was woken by `issue_status_changed`, and each finalization
committed `live_continuation_registered` with an `enqueue_continuation`
effect of kind `response_wake`.
Root cause: `arbitrateNativeStatus()`
(`server/src/services/native-runtime/status-arbiter.ts`) only treats
`response_wake` as passive when `boardResponseWaitAuthorized` (from
`readNativeBoardResponseWaitSource`), external chat, or plan wait is
true. Otherwise it falls to the generic yielded branch, and
`status-decision-committer.ts` (`enqueue_continuation`) enqueues an
`issue_status_changed` wake to the same agent. `issue_status_changed` is
not in `THROTTLED_ISSUE_REWAKE_REASONS`, and the native committer writes
an `issue.updated` activity row with the run's id for every decision, so
the rewake throttle saw every native run as progress.
**Expected behavior**
A `response_wake` with no admitted response source parks the task. A
comment, an interaction answer or a status change by somebody else wakes
the agent. No path can loop without a bound.
**Steps to reproduce**
Run `tests/restart-chaos/main.ts --env local --adapters codex
--workloads W2,W2Q,W2C --concurrency 10 --tasks 10 --seed 5101
--lifecycle warm --triggers time`. W2/W2C tasks loop after phase 1.
## What Changed
- `status-arbiter.ts`: a `yielded` / `response_wake` result with no
admitted response source and no blocking remaining work now commits
`response_wake_waiting_for_input`, keeps the task `in_progress`, and has
no effects. Policy version is `phase6-v14`. Board, external chat, plan,
governance, dependency, pause and unfinished-work branches are unchanged
and still come first.
- `native-runtime/native-response-wake-wait.ts` (new) and
`recovery/service.ts`: recovery treats that applied decision as a
durable wait while it is the issue's latest decision, the issue is
`in_progress` with the same agent assignee, and no user comment arrived
after the run started. This applies to the stranded-work sweep and to
the paused-agent passive-wait check.
- `execution-projection.ts`: a committed run with this reason shows the
phase `waiting_for_answer`, label "Waiting for a reply", and a next
action that tells the user to comment or change the status. This shows
only while that decision is still the issue's latest decision and the
issue is `in_progress`; an older run shows "Completed".
- Backstop in `issue-rewake-throttle.ts` and `heartbeat/queue.ts`: an
`issue_status_changed` wake that carries a declared `continuationKind`
(the `enqueue_continuation` effect) is now a throttle candidate. The
native status committer's own `issue.updated` / status-decision activity
no longer counts as run progress or as new input. After two consecutive
successful runs with no issue-visible progress, such a wake waits for
the existing escalating cooldown (2 min, doubling, max 30 min). The
native status dispatcher (`dispatchNativeStatusWakeBatch`) does not drop
a throttled intent: it keeps the intent `queued`, reports the cooldown
deadline to the deferred-receipt worker, and dispatches it again when
the cooldown ends. Status wakes without a declared continuation (blocker
owner, notify owner, retry) are unchanged.
- Restart chaos: new enforced universal check U5 (`universal.ts`). A
task with more than 10 runs fails as `U5:run_budget_exceeded`
(`RESTART_CHAOS_U5_MAX_RUNS`). The README records the calibration.
- `doc/architecture/native-status-arbitration.md`: the decision table
and the wait rules.
## Verification
- RED on master (same reason as the waves):
`heartbeat-process-recovery.test.ts` "parks a native response_wake that
no response admitted…" fails with decision `reasonCode:
"live_continuation_registered"` and effect `{ kind:
"enqueue_continuation", continuationKind: "response_wake" }`.
`heartbeat-issue-rewake-throttle.test.ts` "throttles a
same_agent/response_wake/retry continuation…" fails 3/3 because the wake
is admitted.
- GREEN: the integration test finalizes the run, dispatches native
status wakes and runs the stranded-work sweep three times. It asserts
one run total, one wake request, no recovery action, the task
`in_progress`, and the projection "Waiting for a reply".
- `heartbeat-process-recovery.test.ts` "delays a no-progress native
continuation until the rewake cooldown…" finalizes a real `same_agent`
continuation after a no-progress run, dispatches twice (one throttle
receipt, intent stays `queued`, deadline equals `nextAllowedAt`), then
dispatches after the cooldown and gets the run. It fails when the
dispatcher drops the throttled intent.
- Focused suites pass: `heartbeat-process-recovery`,
`heartbeat-issue-rewake-throttle`, `status-arbiter`,
`native-status-arbiter-corpus`, `runtime-mode`, `execution-projection`,
`chat-run-milestone-work`, `heartbeat-dependency-scheduling`,
`heartbeat-run-summary` (904 tests), and all restart-chaos unit tests
(584 tests). `pnpm --filter @paperclipai/server typecheck` passes.
- Universal audit over the 17 saved waves of 2026-11-11: U5 flags 12
tasks, all looping tasks (12 to 66 runs), and 0 tasks on all-pass waves.
- Chaos rerun of local warm seed 5101 (W2, W2Q, W2C): result is added in
a comment on this PR.
## Risks
- Behavioral shift: a model that reports `response_wake` to mean
"continue now" no longer gets an immediate run. The task waits visibly
for a reply. The runner prompt already describes `response_wake` as a
real response wait, and `same_agent` stays available for server-owned
continuations.
- The throttle now applies to native runs. A chain of declared
continuations with no issue-visible output (no comment, document, work
product or other activity) waits for the cooldown after the second such
run. `same_agent` continuations are server-owned (repository
preparation), so this affects only repeated no-output chains.
- Old saved state: decisions committed under `phase6-v13` keep their
recorded effects. A run that finalizes after deploy uses the new policy.
No migration.
- Budget hard-stop behavior is unchanged.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude
|
80b2369f26 |
fix(native): keep remote runnerd exit evidence and report it on failed recovery (#20907)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs on Daytona start a detached `paperclip-runnerd` in the sandbox, and the controller adopts it after a restart > - On staging, recovery fails with `runner_remote_process_identity_unavailable` when the Runner is gone, but nobody can see why it exited > - The launch script sends runnerd's stdout and stderr to `/dev/null`, so only a panic or a graceful error reaches `runnerd.stderr.log`, and the recovery event does not read that file > - This pull request keeps runnerd's stderr in a bounded file in its private diagnostics directory, and adds the exit reason to the `native.remote_runner_recovery_failed` event > - The benefit is that the reason a Runner exited shows in the run log on staging without SSH, which the class R investigation needs ## Linked Issues or Issue Description **What happened?** Staging runs fail with `runner_remote_process_identity_unavailable` after the remote Runner exits (`native.remote_runner_recovery_failed` with `stage: process_verification`, `reason: process_not_running`). The event does not say why the Runner exited. **What did you expect?** The run log names the Runner's exit reason: its durable `recoverableFailure` and diagnostics, and what it wrote to stderr. **Steps to reproduce** Restart-chaos re-wake and stall cells reproduce the refusal. Each failed run log shows only `runnerLifecycle` and `runnerSettled`. Refs #20893 ## What Changed - `REMOTE_RUNNER_CHILD_LAUNCH_SCRIPT` sends runnerd's stderr to `diagnostics/runnerd.process-stderr.log` (mode 0600, in the 0700 directory) instead of `/dev/null`. - At launch, a file above 1 MiB is rotated once to `.1`. - While runnerd runs, a detached trimmer is a disk backstop. Past 8 MiB, it keeps the last 1 MiB. It checks every 15 s and ends when runnerd ends. It is double-forked with its own argv, so it is not a child of runnerd and does not carry runnerd's arguments. runnerd's own exit reason goes to `runnerd.stderr.log`, which is never trimmed. - A symlink or non-file at that path keeps the old `/dev/null` behavior. - Stdout is unchanged (`/dev/null`). Runnerd's PID is still the launch shell's PID, so the identity marker and fingerprint are unchanged. - `remote-runner-recovery-diagnostics.ts` has a read-only `REMOTE_RUNNER_EXIT_EVIDENCE_SCRIPT`. It prints a 2 KiB tail of `runnerd.stderr.log` and of `runnerd.process-stderr.log`. It reads the session diagnostics directory first, then the shared runtime-root directory used by external Runners (Dot, Muse, grok_bot). It never reads through a symlink. Each file keeps its own share of the 4 KiB budget. - When recovery's process verification finds `process_not_running`, the diagnostic adds these fields: - `runnerRecoverableFailure`: a code-shaped string from `runner-state.json`; - `runnerDiagnostics`: the last 4 entries, redacted, 500 characters each; - `runnerExitEvidence`: the redacted file tails, 4 KiB at most. `native.remote_runner_recovery_failed` carries them. - The live process monitor's exit report now puts the durable state summary first, then the file tails. Before, it used one or the other. - `run-failure-diagnostics.ts` is unchanged. It still copies only the closed labels, so no free text reaches failure reports. ## Verification - `pnpm exec vitest run server/src/services/native-runtime/remote-runner-launch-stderr.test.ts`: the launch writer keeps stderr in the file, writes the identity marker with the child's PID, appends across relaunches, rotates an oversized file, trims a growing file while the child runs, and never writes through a symlink. - `pnpm exec vitest run server/src/services/native-runtime/remote-runner-recovery-diagnostics.test.ts`: the evidence script output is bounded and refuses symlinks. Redaction and bounds hold, and the new fields survive the event payload. - `pnpm exec vitest run server/src/services/native-runtime/native-session-executor.test.ts` passes (1374 tests). One expectation was updated for the combined exit report. - `npx tsc --noEmit -p server/tsconfig.json` is clean. - Older runner images: the script is server-side. It adds only `wc`, `tr`, `tail`, `cat`, `mv`, `sleep` and `kill`. The script already needed `node`, `date`, `chmod`, `mv` and `printf`, so any image that launched runnerd before has these tools. The runner content ID is unchanged, so the current Daytona image runs it unmodified. The restart-chaos waves run on that image. ## Risks - Low. A sandbox disk that is already full could make the stderr redirect fail. `exec ... 2>>file` then fails, and the launch reports `runner_remote_process_identity_unavailable` the same way a launch that cannot write its identity marker does. The marker is written first, in the same directory tree. - Stderr content stays in the sandbox. Only a redacted 4 KiB tail goes into the instance's own run log (run-log data path; not Telemetry and not Observability). - No wait is added before `turn.start`. The evidence read runs only on the recovery failure path, with its own 10 s bound. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude Fable 5.2
|
b2779054d4 |
fix(issues): let a native run hand its own task to a person without cancelling itself (#20884)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A task has one assignee. An agent often finishes its work by handing its own task to a person for review. > - The task update route stops the outgoing run before it saves a reassignment. > - Since #16082, a legacy run that hands its own task to a person is exempt, so it finishes its turn. Native runs were not exempt. > - On staging, a native run (ACPX Claude, 11.5 minutes of work) sent `PATCH /api/issues/{id}` to hand its task to a person. The route cancelled that run with `issue_reassigned`. The agent saw `api_transport_failure`. The task moved to the person, but it kept a stale execution blocker ("Saved work recovered. Send a message to continue."). > - The native runtime assumes a running run still owns its task. So widening the exemption alone is not enough: the run then fails after its result is accepted, and its finalizer can act on the person's task. > - This pull request applies the exemption to native runs and makes the native terminal fence, restart recovery and finalizer handle a run that gave its task away. > - The benefit is that an agent can hand its task to a person and finish normally, with no blocker on the person's task. ## Linked Issues or Issue Description Refs #16082 (legacy exemption). Related: #20541 (agent-to-agent self handoff) and #18293 (agent key handoff). Those PRs cover other actors and targets. This PR covers native runs that hand their task to a person. **What happened?** A native agent run handed its own task to a person (`assigneeAgentId: null`, `assigneeUserId`, status `in_review`) through its signed run token. The route cancelled the run that sent the request (`issue_reassigned`, "Cancelled before issue reassignment"). The cancelled native run then went through workspace recovery. The person's task kept an execution blocker with `restore_failed`. **Expected behavior** The run that hands its own task to a person keeps running and finishes its turn. The task rests `in_review` with the person. No execution blocker or recovery action stays on the task. The outgoing run cannot change the task after the hand-off. **Steps to reproduce** 1. Run a native agent on its assigned task. 2. From that run, send `PATCH /api/issues/{id}` with `assigneeAgentId: null`, a person's `assigneeUserId`, and status `in_review`. 3. Before this fix, the route cancels the run with `issue_reassigned`. **Why the exemption was legacy-only** #16082 limited the exemption to the run shape of its incident and kept a verified stop for every other case. The new H2 chaos workload showed why native needed more than the route change. With only the route widened: 1. The run's terminal fence `assertCurrentWakeCommentsRead` requires the run to still own the task. It threw `native_current_wake_comments_binding_changed` after the result was accepted. The run then failed with `native_session_lease_lost`, and its finalizer recorded `workspace_finalization_failed` as an agent-owned recovery on the person's task. 2. A restart after the hand-off left the run `running`, because restart recovery refused it with `issue_execution_lock_changed`. 3. The native finalizer only superseded a `done` decision. A `needs_review` report moved the person's task back to `in_progress` and queued a continuation for the outgoing agent. 4. The REST guard against the outgoing run's later mutations (`issue_handed_off`) applied to legacy runs only. ## What Changed - `server/src/routes/issues.ts`: the signed current-run human self-handoff exemption now applies to native runs. All other checks stay: exact run, agent, company and task, `agent_jwt` source, human recipient. For a native run the route first records the hand-off on the run (`resultJson.humanSelfHandoff`, new `native-human-self-handoff.ts`). Board reassignment, agent keys, and handoffs to an agent or to nobody still stop the outgoing run. - `current-wake-comments.ts`: the terminal fence accepts a run whose recorded hand-off matches the person who now owns the task. Without the record (for example a board reassignment), or after a later reassignment, it still refuses. - `native-restart-recovery.ts`: after a restart, a handed-off run is not resumed. It passes the execution-lock check and is then settled by the existing ownership gate (`task_ownership_or_status_changed`) without changing the person's task, instead of staying `running`. - `status-arbiter.ts` and `native-run-finalizer.ts`: `supersedeHandedOffDecision` keeps the person's task status and strips retries, continuations, owner wakes, recoveries, blockers and reviewer bindings, also from decisions that already preserve the status. Two effects stay: a governed request's interaction, and the workspace-finalization repair, so saved work is not lost. Review runs are not affected. - `server/src/run-task-ownership.ts` (renamed from `legacy-run-task-ownership.ts`): `assertHandedOffRunTaskOwnership` rejects later task mutations from the outgoing legacy or native run, in the route and under the update transaction's locks. - `tests/restart-chaos/`: new opt-in workload `H2`. The agent saves a brief, then hands the task to its creator in review as its last step. The oracle requires the task to rest `in_review` with the creator, every run to settle without `issue_reassigned`, readable non-empty run history, and no execution blocker or active recovery action. - Docs: `doc/execution-semantics.md`, task feature map and agent access map, chaos README. ## Verification - Red on `origin/master` (same tests, master source): the route cancels the native run (500, and 409 `runner_goal_reassignment_stop_unconfirmed`); the finalizer moves the person's task to `in_progress` with a continuation for the outgoing agent; a late `done` from the outgoing native run closes the person's task. - Red chaos wave on master's server (`--server-from` a master checkout, `--adapters claude --workloads H2 --faults K1 --matrix`): 0/2 pass. Both runs ended `cancelled` / `issue_reassigned` (`T:handoff_did_not_cancel_run`). - Green focused tests on this branch: - `native-human-self-handoff.test.ts` (embedded Postgres): hand-off through the real issue service; the run stays `running`; `done`, `needs_review` and failed-turn reports leave the task `in_review` with the person, with `getExecutionBlocker` null and no recovery action; the terminal fence accepts only the recorded hand-off; a late change from the outgoing run fails with `issue_handed_off`; pure policy cases. - `issue-agent-mutation-ownership-routes.test.ts`: native and legacy self-handoffs persist without `cancelRun`, and native records the hand-off before the update. Board, agent key, other run, task, agent and company still stop the run. - Native runtime suite (restart recovery, wake comments, arbiter, stale-owner completion), `issues-service`, `low-trust-red-team-routes`: pass. Server and restart-chaos typecheck pass. `node scripts/check-maps.mjs` passes. - Green chaos waves on this branch (local, `--adapters claude --workloads H2`): - `--matrix --faults K1 --seed 4459`: 2/2 pass (K0 baseline and a K1 crossing). - `--tasks 3 --faults K1 --fault-after-seconds 60` and `--fault-after-seconds 100`: 3/3 pass each. In the first, all three restarts landed after the hand-off while the runs were live; the runs settled and the tasks stayed with the person. - The final waves ran on the pre-rebase head `8a2a3b6a74`. The rebase onto master only merged the chaos workload lists (`types.ts`, `config.ts`, `journeys.ts`). No Daytona wave was run. ## Risks - The outgoing native turn keeps running after the person owns the task. Its native task tools already require the run to own the task, so they refuse (`paperclip_runner_tool_binding_not_authorized`). Its REST mutations are rejected with `issue_handed_off`. - A restart after the hand-off settles the run as failed (`native_restart_recovery_blocked`) instead of resuming it. The person's task is not changed and needs no Retry. - The finalizer rule covers any native run whose task belongs to a person at finalization, not only a self-handoff. Before this change such a run could still move the person's task and queue its own continuation. - Saved state: `resultJson.humanSelfHandoff` is a new optional run key. An older server ignores it, so a run handed off on the new server and finished on an old one fails its fence as today. No schema change. - No API shape change. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic), model ID `claude-opus-5-5`, through Claude Code with tool use and code execution. The session does not expose the context window or reasoning mode. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
Nicky LeachandClaude Opus 5.5
|
d7b04a8b6d |
refactor(runner): rename codex-runtime-adapter to acpx-runtime-adapter (#20938)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Paperclip Runner opens agent sessions through one ACPX runtime adapter > - The adapter began as the Codex ACPX adapter, so its file is `codex-runtime-adapter.ts` > - The adapter now opens sessions for every ACPX agent: Claude, Codex, Pi, Cursor, Grok, Kimi, Copilot and Antigravity > - The Codex name makes readers look for Claude and other agent behavior in the wrong place > - This pull request renames the module to `acpx-runtime-adapter.ts` and changes no behavior > - The benefit is a file name that tells readers what the module does ## Linked Issues or Issue Description No issue exists. Refs #20933, which adds Claude restore behavior to this module. **What existing behavior does this improve?** The name of the shared ACPX runtime adapter module in Paperclip Runner. **Subsystem affected** Paperclip Runner, ACPX driver (`packages/paperclip-runner/src/drivers/acpx/`). **Current behavior** The module that opens ACPX sessions for all agents is named `codex-runtime-adapter.ts`. Its test file, which holds Claude, Pi, Grok, Cursor and Copilot tests, is named `codex-runtime-adapter.test.ts`. **Proposed behavior** The module is `acpx-runtime-adapter.ts` and its test file is `acpx-runtime-adapter.test.ts`. All imports, scripts and docs use the new name. **Reason and benefit** The name matches the module's scope. Reviewers and contributors find agent-specific session behavior faster. **Breaking changes** None. No exported name, behavior or contract changes. ## What Changed - Rename `src/drivers/acpx/codex-runtime-adapter.ts` to `src/drivers/acpx/acpx-runtime-adapter.ts`, and its test file in the same way. Git records both as renames with no content change, except the test file's import path. - Update the imports in `acpx-runtime-sidecar.ts`, `codex-acpx-driver.ts` and four test files. - Update the `dist` import in `scripts/grok-native-smoke.mjs` and the test file key in `scripts/vitest-lane-durations.json`. - Update the links and file names in `doc/RUNNER-INTEGRATION-PLAYBOOK.md`, `doc/architecture/runner-cursor-capabilities.md` and `doc/architecture/runner-rich-acp-capabilities.md`. - Do not change the historical profile identity fixtures under `packages/paperclip-runner/test/fixtures/`. They record the sources of earlier releases with the old path, and tests check them by hash. ## Verification - `npx tsc -p tsconfig.json --noEmit` in `packages/paperclip-runner`: no errors in changed files. The only errors are in `src/eval/workflow-harness.ts`, because `@paperclipai/paperclip-eval-kernel` was not built in the local worktree. - `npx vitest run src/drivers/acpx src/cli/acpx-runtime-sidecar.test.ts`: 93 files, 1859 passed, 9 skipped. - `node --test scripts/contracts/contract-fingerprints.test.mjs`: 11 passed. The pack contract digest reads protocol schemas, the Rust crate and the ACPX patch, not TypeScript sources, so the rename does not change it. - `node --check scripts/grok-native-smoke.mjs`, and `scripts/vitest-lane-durations.json` parses. - `git grep codex-runtime-adapter` finds the old name only in the historical fixtures. - Merge check with #20933: a local merge of that branch into this one completed with no conflicts. Git applied #20933's changes to the renamed files. The merged adapter tests passed (133 passed). ## Risks - Low risk. The change renames files and updates references only. - An open PR that edits the old file path must merge or rebase after this PR. Git rename detection handled #20933 with no conflicts. A PR that adds a new import of `./codex-runtime-adapter.js` fails typecheck until it uses the new path. - `openCodexAcpxRuntime` keeps its name. It is an alias of `openQualifiedAcpxRuntime`, and this PR does not change exported names. - `acpx-runtime-adapter.ts` and its test file were already longer than 1,000 lines. This PR does not add lines to them. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits, local test runs). Extended reasoning was on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1111.0-canary.14 |
||
|
DottaandClaude Opus 5.5
|
7cd5558e41 |
fix(authz): send agents denied a cost-recording route to the Costs page (#20922)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - When an agent is denied a board-only action, the denial names the
board page where a person can do it, and agents relay that link
> - Cost recording retry ("Retry cost recording") is board-only and
lives on the run's row on the Costs page
> - The denial for every `/companies/:id/costs` route named the Budgets
page instead
> - So an agent asked to fix a run's missing cost sent the person to
Budgets, where the control does not exist
> - This pull request maps cost routes to the Costs page and names that
page in the retry route's catalog entry
> - The benefit is that the person lands on the control the agent told
them to use
## Linked Issues or Issue Description
**What happened**
In paid chat eval run 43086418488
(`budgets-costs.repair-cost-recording`), the CEO tried `POST
/api/companies/{companyId}/costs/retry-accounting`, got `403
board_access_required` whose remediation said `[Open
budgets](/<prefix>/activity/budgets)`, and told the person to use the
retry on Budgets. The retry control is on the Costs page.
**Expected behavior**
Board denials for cost routes name `[Open
costs](/<prefix>/activity/costs)`. Budget and budget-incident routes
still name Budgets.
**Steps to reproduce**
1. As an agent, call `POST
/api/companies/{companyId}/costs/retry-accounting`.
2. Read the 403 `remediation`.
3. It links `/<prefix>/activity/budgets`, a page with no cost retry
control.
## What Changed
- `server/src/routes/authz.ts`: `COMPANY_COSTS_PAGE`
(`/<prefix>/activity/costs`); `/companies/:id/costs/*` denials use it.
Budgets and budget incidents keep `COMPANY_BUDGETS_PAGE`.
- `server/src/routes/openapi.ts`: the retry route's description says a
board user retries with Retry cost recording on the run's row on the
Costs page.
- `board-access-denial.test.ts`: regression row for the retry route (red
before the change, green after).
## Verification
- `pnpm exec vitest run src/__tests__/board-access-denial.test.ts
src/__tests__/openapi-routes.test.ts
src/__tests__/agent-refusal-finality.test.ts
src/__tests__/cost-data-quality-routes.test.ts
src/services/native-runtime/runner-api.test.ts` (server): all pass. The
new row fails without the `authz.ts` change.
- End-to-end: the chat eval case `budgets-costs.repair-cost-recording`
grades that the agent's guidance links the Costs page; it will be rerun
after this merges.
## Risks
- Low. Only the link text and target in cost-route denials and one
catalog description change. Agents that previously relayed Budgets now
relay Costs.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
c1edb9ec02 |
fix(native): keep an unreachable remote Runner alive when a failed run closes (#20923)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs on a remote sandbox (Daytona) start a detached `paperclip-runnerd`. After a recoverable failure, the server resumes the same run and reattaches to that Runner > - On staging, `runner_remote_process_identity_unavailable` was the most frequent failure. The tasks stayed blocked until someone clicked Retry > - In 31 of the 34 failed runs the reattachment found the Runner gone with an unsettled `ready` journal (`runnerSettled: false`). The Runner was not lost by accident. The controller's own close stopped it with `SIGKILL`: close could not prove suspension within its 10 s budget, so it force-stopped the Runner in the middle of a turn > - The server then scheduled a same-run reattachment that could never succeed, because the close had just removed the Runner it needed > - This pull request keeps an unreachable remote Runner alive when a failed run closes. A Stop, a cancelled turn and a `stop_cleanup` recovery still stop it > - The benefit is that the same-run reattachment finds the live Runner. The Runner then settles into a `suspended` journal, which the existing relaunch path resumes. The task continues without Retry, and no provider work is replayed ## Linked Issues or Issue Description Refs #20853, #20907 **What happened?** Remote native runs failed with `runner_remote_process_identity_unavailable` and error code `native_continuation_requires_reconciliation`. The tasks were left blocked. In the failed runs the first attempt failed with a recoverable transport error, for example `provider_transport_failed: PRP command turn.start timed out`, `native_session_interrupted` or a server restart. The run log then showed `runner did not prove durable suspension before checkpoint` and `runner shutdown failed: Error: Durable recovery runner timed out.` The second string comes from `waitForProcess`, which sends `SIGKILL` when its deadline passes. The same-run reattachment ran 20–60 s later and reported `process_verification` / `process_not_running` with `runnerLifecycle: ready` and `runnerSettled: false`. Most of these runs had done provider work before the failure (turns and tool calls), so the automatic Retry correctly refused them (`provider_turn_unsettled`). **What did you expect to happen?** When a run fails for a reason that the server resumes on the same run, the close does not destroy the Runner that the reattachment needs. The reattachment adopts it, and the run continues without Retry and without a repeated tool effect. **Steps to reproduce** 1. Start a remote native run and let a turn start. 2. Make the Runner unreachable (a broken transport or a slow event pump), so that the turn fails and the close cannot prove suspension within its budget. 3. The close sends `SIGKILL` to the Runner. The same-run reattachment then fails with `runner_remote_process_identity_unavailable`, and the task is blocked. **Relation to open PRs** - #20853 covers the 3 runs whose journal was a settled `ready` cut (`runnerSettled: true`). This PR does not change that path. - #20907 makes the Runner's exit reason visible. It does not change recovery. This PR does not overlap with it. ## What Changed - `runnerd-codex-transport.ts`: `close(reason, { retainUnsettledRemoteRunner })` accepts a new option. When the option is set, the Runner is remote (a process launcher or an adopted Runner), the close is not a `stop_cleanup`, the Runner is still alive and suspension is not proven, the close does not stop the Runner. It skips the `waitForProcess` kill, the adopted-runner `SIGKILL` and `forceKill`. It still releases the controller route, writes the unsettled checkpoint and throws `NativeSessionCloseUnrecoverableError`, so the session stays quarantined and is never reused. The settlement diagnostic records `runnerRetained: true`, and a new diagnostic records `remote runner left alive unsettled for same-run reattachment`. - `HarnessNativeSession.close` asks for retention only when the session was not explicitly cancelled. After `cancel()` (Stop), it asks for the old behavior, so Stop still stops the Runner. - `CodexHarnessSession.close` passes the option to the transport. `HarnessSession.close` and `CodexAppServerTransport.close` accept the optional field. Callers that do not set it keep the old behavior. - Docs: `doc/execution-semantics.md` and `doc/run-log-events.md` describe the rule and the `runnerRetained` field. How the run continues: the reattaching controller adopts the live Runner. The Runner then completes the `runner.drain` and `runner.suspend` that the failed close queued, and exits with a settled `suspended` journal. That attempt fails as `native_runner_process_exited`, which the server resumes on the same run. The next reattachment finds the process gone with a settled `suspended` journal and relaunches the Runner over it (the existing path from #19745). If the Runner was only slow, it suspends by itself and the first reattachment relaunches. ## Verification - Red, on `master` with only the tests added: `runnerd-codex-transport-remote-retention.test.ts` fails 2 of 5 (`expected 'gone' to be 'alive'`): the close killed the Runner. `harness-driver-backend.test.ts -t "retain an unsettled"` fails 2 of 2. - Green with the fix: `cd packages/paperclip-runner && npx vitest run src/live/runnerd-codex-transport-remote-retention.test.ts src/backends/harness-driver-backend.test.ts`. - The staging shape: a real `paperclip-runnerd` started through a process launcher, a turn in progress, the Runner frozen with `SIGSTOP` during close. The close throws `NativeSessionCloseUnrecoverableError` with `runnerRetained: true`, the Runner stays alive, and its journal stays `ready`. - A second controller attaches through `adoptExistingRunner` (no duplicate launch). The Runner settles into `lifecycle: suspended` with an empty outbox and no pending terminal delivery, then exits. - An explicit cancellation, a caller that does not ask for retention, and a `stop_cleanup` close still stop the Runner. - Also run: `src/backends`, `src/drivers/codex`, `src/drivers/acpx/codex-acpx-driver.test.ts`, `src/drivers/acpx/codex-runtime-adapter.test.ts`, `src/native-session-runtime.test.ts` (1045 tests pass; 2 `spawnSync ETIMEDOUT` under full-parallel load pass alone), and all of `src/live/` (files that timed out under full-parallel load pass at `--maxWorkers=2`). - `npx tsc --noEmit -p packages/paperclip-runner/tsconfig.json` is clean. - `pnpm verify` selects a broad plan (exit 2) because `doc/execution-semantics.md` is a global CI input. CI supplies that coverage. - Not run: a live Daytona run. The fix is in the controller library that runs on the server, so it needs no new Runner image. ## Risks - Medium. A failed close now leaves an unreachable remote Runner alive instead of killing it. This is the same state a restart detach leaves today, and the same-run reattachment or a Stop cleanup owns it. - If the server does not resume the run on the same Runner, the Runner stays alive until something reattaches or the sandbox is released. Remote Runners have no reconnect deadline (`runnerProcessReconnectGraceMs: null`). Before this change they were killed. The close still quarantines the session, so it is never released for reuse. - Stop is unchanged: an explicit cancellation, `settleCancelledTurn` and `stop_cleanup` close without retention, so they still stop the Runner. - Previous-release story: this change adds and changes no persisted state. State that an older release left (a killed Runner with an unsettled `ready` journal) keeps its current handling: the named refusal `runner_remote_process_identity_unavailable` and a working Retry. Tasks already blocked on staging keep their Retry button. A deploy during a run changes nothing, because the decision is made in the closing process. - No wait is added before `turn.start`. The change only affects the failure path of close. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, test runs, read-only staging API reads). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Fable 5.2
|
e8e548e856 |
feat(db): classify migrations for application under the running release (#20913)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - `packages/db` owns the schema and its migrations. There are 467
migrations, and several ship each day
> - A zero-downtime deploy starts the new release while the previous
release still serves the same database
> - Core does not say whether a migration is safe to apply under the
previous release. A deployer must treat each pending migration as a
reason to pause, or a person must review it
> - On stacks that may not pause, releases wait for hours
> - Core owns the schema, so Core must own this knowledge and publish it
as data
> - This pull request adds a conservative classifier, an in-file
override, the published result, and a guard for new migrations
> - The benefit is that a deployer can apply routine compatible
migrations automatically, and incompatible changes become visible at
review time
## Linked Issues or Issue Description
**Subsystem affected**
Database migrations (`packages/db`), the cloud migrator artifact
(`scripts/cloud-migrator-artifacts.mjs`), and the pull-request policy
job.
**Problem or motivation**
Core has no machine-readable statement that a migration can be applied
while the previous release still reads and writes the database. A
deployer cannot tell an added nullable column from a dropped column. It
must pause for both, or refuse the release.
**Proposed solution**
Classify each migration as `online_compatible`, `requires_staging`, or
`unknown`. Publish the result in the migrator manifest. Fail a pull
request when a migration it adds is `unknown` and has no reviewed
override in the file.
**Alternatives considered**
- A hand-kept list in the deployer. It goes stale and it is not reviewed
with the SQL.
- A sibling JSON file in the artifact. It needs a workflow change, and
the attestation of `manifest.json` would not cover it.
**Roadmap alignment**
This is release infrastructure. It adds no product feature.
## What Changed
- `packages/db/src/migration-compatibility.ts`:
`classifyMigrationSql(sql)` returns `{ compatibility, classified,
reasons, override, overrideError }`. It parses statement by statement
(breakpoints, `;`, comments, string and dollar quoting, guarded `DO`
blocks). A statement that it does not recognize is `unknown`. A
migration takes the worst class of its statements.
- Lock and duration hazards have their own reason codes
(`lock_hazard_*`) and classify as `unknown`, not as compatible with a
warning.
- Override: `-- paperclip:migration-compat online_compatible:
<justification>` (or `requires_staging`) in the leading comment block of
the migration file. The justification is mandatory (10 characters or
more). A malformed, repeated, or misplaced directive is an error and
never improves the verdict.
- A table counts as new after a plain `CREATE TABLE`, or after `CREATE
TABLE IF NOT EXISTS` when no earlier migration created that table.
`classifyMigrationSequence` classifies migrations in apply order with
that knowledge.
- `packages/db/scripts/migration-compatibility.ts` (run with `pnpm exec
tsx` in `packages/db`): classify files, print the distribution, print
JSON, or run the guard (`--check`). No new package script.
- Published data: `manifest.json` of the cloud migrator artifact has a
new additive field `migrationCompatibility`. `version` stays `1`. No
existing field changes. The build reads the SQL from the packed
`db.tgz`, so the data describes what the migrator applies. `validate`
(publisher) and `verify` reject data that is malformed or does not match
the packed migrations. The workflow file does not change.
- Guard for new migrations:
`.github/scripts/check-pr-migration-order.mjs` (the existing "Validate
migration ordering against target branch" policy step) now also
classifies the migrations that the pull request adds. `check:migrations`
runs the same guard against the merge base with `origin/master`, so the
`@paperclipai/db` typecheck and build report it locally. `unknown`
without an override fails. `requires_staging` prints a notice and
passes. A new migration that uses `CONCURRENTLY` also fails when it is
not in `MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS`
(`packages/db/src/client.ts`), because the migrator would run it inside
a transaction.
- `doc/DATABASE.md`: new section "Migration compatibility with the
running release" under "Schema change workflow".
Published shape:
```json
"migrationCompatibility": {
"version": 1,
"migrations": [
{
"idx": 468,
"tag": "0468_busy_wendell_rand",
"sqlSha256": "56a6e272…",
"compatibility": "online_compatible",
"classified": "online_compatible",
"override": false,
"reasons": [
{ "code": "add_column", "compatibility": "online_compatible", "message": "Adds a column that is nullable or has a constant default.", "statements": [1] }
]
}
]
}
```
`sqlSha256` is the hash that the migrator records in the migration
history. `overrideJustification` is present only when `override` is
`true`. Entries are in apply order (journal `idx`, then file name; the
shipped journal repeats `idx` 178).
### Distribution over the existing migrations
No existing migration is rewritten or annotated.
| Class | All 467 | Newest 100 |
| --- | --- | --- |
| `online_compatible` | 228 (48.8%) | 67 |
| `requires_staging` | 114 (24.4%) | 12 |
| `unknown` | 125 (26.8%) | 21 |
Migrations per blocking reason (one migration can have several):
| Migrations | Class | Reason |
| --- | --- | --- |
| 74 | `requires_staging` | `create_unique_index_existing_table` |
| 70 | `unknown` | `lock_hazard_create_index` |
| 66 | `unknown` | `lock_hazard_validated_foreign_key` |
| 56 | `unknown` | `lock_hazard_data_backfill` |
| 54 | `unknown` | `drop_constraint` |
| 41 | `requires_staging` | `add_validated_check` |
| 33 | `unknown` | `drop_index` |
| 16 | `unknown` | `do_block_procedural` |
| 15 | `unknown` | `replace_function` |
| 13 | `unknown` | `create_trigger_existing_table` |
| 12 | `unknown` | `data_change_existing_table` |
| 10 | `unknown` | `drop_object`, `lock_hazard_set_not_null`,
`set_default` (each) |
| 8 | `requires_staging` | `drop_column` |
## Verification
- `pnpm --filter @paperclipai/db exec vitest run
src/migration-compatibility.test.ts`: 113 tests pass. They cover a
positive and a negative case for each rule, each reason code, the
override (missing, short, misplaced, and repeated justification are
rejected), unknown by default, the guard, the shipped migrations, the
`CONCURRENTLY` list, and the command line.
- `node --test scripts/cloud-migrator-artifacts.test.mjs
.github/scripts/tests/check-pr-migration-order.test.mjs`: 21 tests pass.
- Real bundle at the first commit of this branch: `preview-artifacts.mjs
pack`, then `cloud-migrator-artifacts.mjs build`, `validate`, and
`verify-install` pass. The manifest lists 467 migrations and is 312 KB.
This run found the repeated journal `idx`, which the second commit
handles.
- `pnpm --filter @paperclipai/db run check:migrations` passes. With
`--check --base HEAD~150` the guard fails on three `unknown` migrations
and prints notices for two `requires_staging` migrations.
- `pnpm verify` exits 2 because `.github/scripts/**` is a global CI
input. Required CI supplies the broad coverage.
- Not run: the publish job (it needs the master-only AWS role), and no
deployer reads the new field yet.
## Risks
- The manifest grows from about 1 KB to about 312 KB. A consumer that
rejects unknown manifest fields would fail. The verifier in this
repository ignores unknown fields; consumers outside this repository
were not inspected.
- From this merge, an open pull request that adds an `unknown` migration
fails the policy step and the `@paperclipai/db` typecheck until the
author rewrites the migration or adds the override. `drizzle-kit` emits
validated foreign keys and plain indexes for existing tables, so this is
common. The error text and `doc/DATABASE.md` give the fix.
- `ADD CONSTRAINT ... NOT VALID` classifies as `online_compatible`.
PostgreSQL still enforces it on new writes, so the author must make sure
that the previous release does not write rows that violate it.
- The classifier reads SQL only. It cannot see that new code writes
values that the previous release cannot read (for example a new enum
value). The `requires_staging` override exists for that case.
- `master` cannot fail the guard: the guard only judges migrations that
are not on the merge base.
## Model Used
- Claude (Anthropic), model ID `claude-fable-5-2`, in Claude Code with
tool use and code execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
fc14f1ade2 |
fix(execution): keep resumed native runs alive past a stale finalization deadline (#20920)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The execution control reconciler bounds how long a run may spend
settling after its provider work ends
> - A run deadline from one attempt stayed on the run row after the run
was handed to a same-run resume or a restart claim
> - The reconciler then failed the resumed, live run with
`execution_finalization_deadline_exceeded` and told a person to inspect
the provider
> - This breaks two product rules: recovery must work without a person,
and users must not get operator instructions
> - This pull request makes a live native owner or a scheduled same-run
resume the owner of the run, and it replaces the operator text with
plain wording and Retry
> - The benefit is that runs that recover after a restart keep running,
and a genuine expiry shows plain words and one working Retry
## Linked Issues or Issue Description
No public issue exists. This is the bug description, in the fields of
`.github/ISSUE_TEMPLATE/bug_report.yml`.
**What happened?**
On a hosted deploy, 11 tasks became blocked with
`execution_finalization_deadline_exceeded` in two hours. Three runs had
the same sequence. The native session failed or the server restarted. A
same-run resume or a restart claim then took the run. The new controller
held a live native lease and prepared the run again. The reconciler then
found the earlier attempt's run deadline and failed the run. The
recorded original failure code was empty, so no real failure caused the
block. The run error told a person to inspect the failed run and verify
that its provider stopped.
**Expected behavior**
A run that a live controller owns, or that has a scheduled same-run
resume, continues. A deadline that really expires shows plain wording
and a working Retry. Users do not get operator instructions.
**Steps to reproduce**
1. Save a running native run with an `executionControlDeadlineAt` in the
past.
2. Give its finalization row a live lease (a restart claim), or phase
`retryable_failure` with no lease (a scheduled resume).
3. Run `reconcileAbandonedExecutionControl`.
4. The run fails with `execution_finalization_deadline_exceeded`, and
its error tells a person to inspect the provider.
**Paperclip version or commit**
`master` at
|
||
|
DottaandClaude Opus 5.5
|
64afd22d82 |
fix(runner): keep the OpenCode MCP bridge on keep-alive so OpenCode keeps its Paperclip tools (#20921)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native OpenCode agents call Paperclip tools through a local MCP
bridge in the runner's OpenCode proxy
> - #20886 made the bridge send `Connection: close` on every response,
as hardening for one stalled parallel tool call that was never
reproduced
> - After #20886 merged, many native OpenCode runs started without any
Paperclip MCP tools
> - Agents then said the Paperclip tools were missing, worked only with
bash and file tools, and never called paperclip_finish
> - This pull request removes the `Connection: close` header and keeps
the delivery diagnostics from #20886
> - The benefit is that OpenCode agents get their Paperclip tools again,
and a recurrence of the original stall still shows in the run log
## Linked Issues or Issue Description
Refs #20886.
**What happened**
In chat eval runs on CI, every OpenCode session before #20886 sent a
first model request of about 34k–36k input tokens (prompt plus the
Paperclip tool schemas). After #20886, about 9 of 25 sessions sent only
about 8k–15k tokens on the same prompts (for example 14,711 for
questions-and-approvals.necessary-question, 15,481 for
cases.cases-disabled, 14,096 for cases.update-case-facts). The prompt
envelope size did not change (about 32.7k characters). Those sessions
made no Paperclip MCP calls: zero tool-delivery records, no
paperclip_finish. The agents' own reasoning said "The paperclip tools
aren't in my toolset". Sessions on a commit without #20886 (
|
||
|
DottaandClaude Opus 5.5
|
c5e6ab5ca8 |
test(evals): retarget instance-settings from the removed Agent Chat toggle to the Cases experiment (#20897)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat catalog eval `instance-operations.instance-settings` checks that a person who asks the chat agent to turn on an instance experiment gets the supported operator path, and that the setting really works > - The case asked "Enable Agent Chat on this instance." Agent Chat is always on since #19360, and the catalog says so (#20169), so the agent correctly answered "already on" and the scripted human could not find the removed toggle > - The premise is obsolete, so this pull request changes it explicitly to the Cases experiment (`enableCases`), which still exists and has a real route gate > - It also replaces a reply-wording regex with structural checks > - The benefit is that the case measures its original promise on today's product ## Linked Issues or Issue Description **What happened** In the full chat-catalog run, the case failed in the human journey: the locator `getByRole('switch', { name: 'Toggle agent chat experimental setting' })` found no element. The agent's reply ("Agent Chat is already enabled ... there's no toggle to change") was correct for the current product. The grader also used reply regexes (`experimental|enableAgentChat`, `operator|admin|human|settings`, and a negative "I have enabled" pattern). **Expected behavior** The case tests an instance experiment a person can still turn on. The agent cannot change instance settings, does not change the setting, and links Experimental settings. The person follows that rendered link, turns the feature on, sees it persist, and the feature's page opens. Grading reads saved state and rendered links, not wording. **Steps to reproduce** 1. Run `chat-features.runner-opencode.local.instance-operations.instance-settings` on master. 2. See the agent answer that Agent Chat is always on. 3. See the human step time out on the missing Agent Chat switch. Refs #19360, #20169. Product companion: #20894 (the denial and the catalog name the Experimental settings page). It is cherry-picked here so the paid run measures it. ## What Changed - Premise change (explicit): query "Enable Agent Chat on this instance." becomes "Turn on Cases for this instance."; context and expected outcome follow. Recorded in the manifest `changedFields`, pinned in the manifest test, inventory digest updated. - Setup: refuses a hidden Cases setting, starts Cases off (cleanup restores the target's value), and probes GET/PATCH experimental settings with the agent's own key (403 `board_access_required`, setting unchanged). - Human journey: checks that Cases is still off after the agent's turn and that `/<prefix>/cases` redirects to the dashboard. Then follows the Experimental settings link that the reply rendered (no typed-URL fallback), turns Cases on, checks it after reload, and opens the Cases page. - Grader: the `supported-human-path` regex check is replaced by `agent-links-experimental-settings` (rendered link to the page, and the person followed it). Saved-state checks cover the off route, the save, persistence and the open Cases page. - `oracleVersion` is now `paperclip.chat-catalog.privileged.instance-settings.v2`. `sourceFiles` follow the code. - Calibration: the valid reply and a reworded reply pass. Each missing state, a missing link, a wrong page, a foreign origin and a literal `<prefix>` link fail. ## Verification - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/chat-catalog-domains/privileged-companions.test.ts tests/runner-e2e/chat-catalog-domains/privileged-budget.test.ts tests/runner-e2e/chat-catalog-contract.test.ts tests/runner-e2e/chat-catalog-consistency.test.ts`: 25 passed. - `node --test tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs`: 10 passed. - `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`: unchanged. - `npx tsc --noEmit -p tests/runner-e2e`: no errors in changed files. - The paid end-to-end run is not done yet. ## Risks - The case now depends on the Cases experiment. If Cases becomes always on or is removed, the case needs another retarget. - The agent can still say something wrong in prose. The grader cannot detect a false "I turned it on" claim without wording checks, so a human reads the reply. The saved state proves that the agent did not change the setting. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
34f71f7d57 |
fix(server): defer a run that meets a busy agent directory instead of failing it (#20829)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Each run of an agent with managed agent files gets a private copy of
the agent directory. Remote copies serialize their prepare, checkpoint,
collect, and release operations through one Postgres advisory lock per
agent, taken with `pg_try_advisory_xact_lock`
> - When the lock is busy, the operation fails at once with "Agent
directory operation is already active"
> - On staging, a deploy restarted the controller while three runs of
the same agent were active. When one run resumed, another run of that
agent was checkpointing its directory. The resume failed, used its last
attempt, and the task became blocked with "Retry or send a message"
> - A run must not fail because another run of the same agent is briefly
using the agent directory
> - This pull request keeps the fail-fast lock and, when it is busy
during preparation, puts the same native run back on the existing
same-run retry schedule a few seconds later
> - The benefit is that lock contention delays a run by seconds instead
of ending it as failed
## Linked Issues or Issue Description
**What happened?**
A task's run failed with `native_continuation_requires_reconciliation`
and the error "Agent directory operation is already active". The task
was blocked and asked the user to retry. The same error ended at least
two other runs of the same agent within one day.
**Steps to reproduce**
1. Use an agent with managed agent files in a remote sandbox.
2. Let one run of the agent checkpoint or collect its agent directory.
3. At the same time, let another run of the same agent prepare (for
example, a same-run resume after a controller restart).
4. The second run fails immediately with "Agent directory operation is
already active".
**Expected behavior**
The second run waits until the first operation is done, then continues.
It does not end as failed.
## What Changed
- `server/src/services/agent-directory-working-copies.ts`: The busy lock
now throws `AgentDirectoryOperationBusyError`, still a 409 with the same
message. Lock behavior is unchanged from `master`: a fail-fast
`pg_try_advisory_xact_lock` held for the life of the holder's
transaction.
- `server/src/services/heartbeat/agent-directory-busy.ts`:
`prepareWithAgentDirectoryDeferral` wraps only instruction preparation
and turns the busy error into `AgentDirectoryPreparationDeferral`
("Another run of this agent is still using its files"). A busy lock met
while collecting files after a turn is not wrapped and keeps its
existing path.
- **Before native admission** (a brand-new run, such as the first run of
a task): `server/src/services/heartbeat/retries.ts`
`finalizeAgentDirectoryBusyDeferral` cancels the run as an expected
control-plane wait (`providerWorkStarted: false`) and schedules an
`agent_directory_busy` retry 5–15 seconds later. The task keeps its
execution lock on the retry. The lane spends no failure retries
(`execution-recovery-attempt.ts`), its count is the retry run's
`scheduledRetryAttempt`, and it stops after 30 attempts.
`pre-dispatch-execution-evidence.ts` classifies this wait like the
existing workspace and AI-connection waits, so retry scheduling does not
ask for reconciliation.
- **A resumed native run**:
`server/src/services/native-runtime/native-controller-recovery.ts`
`deferNativeRunForBusyAgentDirectory` keeps the same run on the existing
same-run retry schedule without spending an attempt. It uses the same
guards as `deferNativeControllerRecovery`: it locks the task, requires
the task's execution run to be this run, requires the current lease
owner, and refuses after a Stop, a startup cancellation or a pending
native cancellation, so those settle normally. Its count is stored in
the run's `contextSnapshot`, which a resume claim does not clear, and it
stops after 30.
- `server/src/services/heartbeat.ts`: routes a preparation deferral to
the same-run path or, before admission, to the scheduled-retry path.
After the bound it follows the normal failure path with the plain
message.
- No schema or migration. Postgres releases a dead holder's advisory
lock with its session. The staging evidence showed a live holder
(another run's checkpoint), not a stale one.
- `doc/DEVELOPING.md`: Documented both paths and the bound.
Earlier revisions of this PR waited on the lock and added idle-session
expiry, keepalive, ownership fences, fenced remote removal and attempt
slots. This revision removes all of them.
### How the directory-wait retry keeps the original wake
The retry no longer replaces the wake. `scheduleBoundedRetryForRun`
takes `preserveWakeContext` for this wait: the retry keeps the original
`wakeReason`, `retryReason` and the whole wake context (interaction,
comment, continuation and admission fields), and its wake request keeps
the original reason. The wait is recorded separately as
`preparationWait: { kind: "agent_directory_busy", attempt, nextAttemptAt
}`, next to the carried `executionRetryAccounting`. Gates that key on
the wake reason therefore treat the retry exactly as they treated the
original wake.
| # | Site | Treatment for the directory wait | Why |
| --- | --- | --- | --- |
| 1 | `heartbeat/retries.ts` `scheduleBoundedRetryForRun` | Changed:
`preserveWakeContext` keeps the original wake context and reason and
adds `preparationWait` | Root cause: the retry overwrote the wake reason
|
| 2 | `run-dispatch/domain/wake-context.ts`
`isResolvedInteractionContinuationWakeContext` /
`PREPARATION_WAIT_RETRY_REASONS` | Special case removed | The original
wake context is kept, so the continuation is classified from it |
| 3 | `heartbeat/retries.ts` `interactionRunnerWait` | Special case
removed | The interaction fields are kept in the context |
| 4 | `run-dispatch/adapters/postgres.ts` scheduled-retry dependency
facts | Changed: a preparation-wait retry whose original wake was
admitted as a dependency-blocked interaction
(`dependencyBlockedInteraction`, recorded at wake time) is admitted the
same way | Without the wait the original wake would have run; this gate
is only met because of the wait |
| 5 | `run-dispatch/domain/wake-context.ts`
`isNonAssigneeWorkspaceBusyRetry` + `heartbeat/retries.ts`
`finalizeAgentDirectoryBusyDeferral` | Kept:
`preparationWaitDeferredWhileAssignee`, derived only from the recorded
admission authority | The scheduled-retry ownership gate has no
wake-reason path; the workspace and AI waits use the same recorded flag
|
| 6 | `heartbeat/retries.ts` explicit user continuation admission |
Kept: admits the `agent_directory_busy` scheduling reason | The
successor needs its own authorization receipt |
| 7 | `execution-recovery-attempt.ts` uncharged lane and
`agentDirectoryWaits` | Kept | Cap and accounting |
| 8 | `pre-dispatch-execution-evidence.ts` (`isPreDispatchResourceWait`,
`isAgentDirectoryPreparationWait`), used by
`legacy-execution-recovery.ts`, `cancelled-native-startup.ts`,
`explicit-native-continuation.ts` | Kept | Describe the cancelled
pre-provider parent run, not the retry's wake |
| 9 | `explicit-native-continuation.ts`
`undeliveredLegacyUserCommentIds`, `run-dispatch/adapters/postgres.ts`
`unstartedUserTurnOrigin` | Kept | The cancelled parent did not consume
the user's comment |
| 10 | `execution-projection.ts` label and saved accounting,
`task-interruption-presentation.ts`, UI toast and chat suppression |
Kept | Presentation of the wait |
| 11 | `run-dispatch/domain/policy.ts` AI-wait lock requirement;
AI-subscription lock recheck; AI wait detection
(`execution-configuration.ts`, `provider-limit-wait.ts`, `heartbeat.ts`,
`ai-connections.ts`, `ai-connection-runtime.ts`, `local-ai-login.ts`) |
Not applicable | AI-subscription specific |
| 12 | `workspace-busy-wait.ts`, `environment-selection.ts`,
`workspace-preparation.ts`, UI workspace-holder banners | Not applicable
| Workspace holder parking; the directory wait is a timed retry |
| 13 | `cancelled-native-startup.ts` paused workspace-retry repair;
`failureRetriesBefore*Wait` | Not applicable | Historical workspace and
AI records only; every directory wait saves its ledger |
| 14 | `failedRunRetryBudgetSpent`; storage-wait exhaustion escalation;
`status-relevance.ts` `isAdmissionWait` | Not applicable | A directory
wait never fails the run, stops at its own bound, and its run has
started |
## Verification
- Original wake kinds after a directory wait
(`heartbeat-agent-directory-busy.test.ts`, real heartbeat): an
assignment runs; a plain user comment on a dependency-blocked task is
answered with `dependencyBlockedInteraction`; an accepted interaction
continuation keeps its parked-task and reassignment gates; a user
message continuation runs with its comment; an accepted non-assignee
mention runs; and a reassignment during preparation does not let the
previous assignee restart. Each retry keeps the original `wakeReason`
and records `preparationWait`. With `preserveWakeContext` off, five of
these fail. Restart recovery keeps its attempt credit
(`native-runner-restart-recovery.integration.test.ts`).
- Non-assignee wake: an accepted mention for an agent other than the
assignee meets a busy directory; its retry records
`preparationWaitDeferredWhileAssignee: false`, survives promotion, runs
and succeeds once the directory is free. Without the shared clause in
`isNonAssigneeWorkspaceBusyRetry`, promotion cancels it.
- Interaction continuations: `heartbeat-agent-directory-busy.test.ts`
runs an accepted interaction continuation into a busy directory. The
retry keeps `mutation`, `interactionId`, `interactionKind` and
`interactionStatus` and is still classified as a continuation. If the
task is moved to todo, or reassigned, while the retry waits, the retry
does not run and the task stays parked or with its new owner. With the
old explicit storage/Runner lists, both cases fail.
- Red/green, new run:
`server/src/__tests__/heartbeat-agent-directory-busy.test.ts` runs a
real heartbeat for a new assignment whose first instruction preparation
meets a busy lock. Without the preparation wrapper, the run ends failed
(red). With it, the first run is cancelled as a wait, an
`agent_directory_busy` retry is scheduled with the task still locked to
it, the retry runs and succeeds, and no run of the task is failed
(green).
- Red/green, same-run: with the deferral as a no-op, the same-run
deferral tests failed (`expected false to be true`); they now pass.
- `native-controller-recovery.integration.test.ts` covers repeated
claim-and-defer cycles of a resumed run: each cycle keeps the attempt
count, cannot be claimed before the delay, stores the count on the run,
and the fourth deferral past a bound of 3 is refused. It also covers
Stop after the busy error, a foreign lease owner and a moved task (each
refused, nothing changed), a run before admission, and that a raw busy
error (as from collection) is never deferred.
- Focused suites: 232/232 passed (`agent-directory-working-copies`,
`instruction-working-copy`, `heartbeat-agent-directory-busy`,
`native-controller-recovery.integration`, `heartbeat-retry-scheduling`).
`tsc --noEmit` for `server` reports no errors in the changed files.
- Staging evidence (read-only): the failed run had a graceful restart at
07:25Z, then same-run resumes. Its third attempt failed at 07:31:08Z.
Another run of the same agent recorded a warm checkpoint receipt at
07:31:37Z. A second run of the same agent failed the same way at
07:59:46Z.
## Risks
- A deferred run waits 5 to 15 seconds per deferral, up to 30 times. If
the holder is still busy after that, the run fails as before with a
plain message.
- Before admission, deferral cancels the run and creates a retry run,
like the existing workspace-busy wait.
## Model Used
- Claude Opus 5.5 (claude-opus-5-5), Anthropic, with tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Fable 5.2
|
a56b648602 |
fix(upgrade-readiness): report restartable runs and a handoff verdict that blocks only on lost work (#20910)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A deployer can replace the server while agent runs continue. Before the swap, it reads the upgrade-readiness report of the old server > - The version 1 report is ready only when every active run is adoptable and no wake admission is in flight > - On a busy instance, new runs arrive each minute. Some runs are always in preparation, so the report is almost never ready and the instance does not upgrade > - A run in preparation has done no agent work. Restart recovery starts it again. A swap loses nothing > - This pull request adds a `handoff` object to the report. It separates recoverable, restartable and blocking runs > - The benefit is that a deployer can wait only for work that a swap would lose ## Linked Issues or Issue Description Refs #20852 (reference only; this change does not depend on it and does not use its version 2 shape). **What happened?** A busy instance had 16 running runs and 1 queued run. Fourteen had a live process. Two or three native runs had not launched a process yet. One was a legacy run. `upgradeReadiness.ready` needs `blockingRuns === 0 && pendingWakes === 0` in one probe. That state almost never occurs, so the deployer never swaps the server. **Expected behavior** The report tells the deployer which runs a swap would lose. Runs that only wait for a launch, and wake admissions in flight, do not block a graceful replacement. **Steps to reproduce** 1. Keep a steady flow of new runs on an instance. 2. Read `GET /api/instance/task-drain?upgradeReadiness=1` repeatedly. 3. See `ready: false` each time, because one or more runs are in preparation or a wake admission is in flight. ## What Changed - `summarizeUpgradeReadiness` returns an additive `handoff` object: `version`, `ready`, `recoverableRuns`, `restartableRuns`, `blockingRuns`, `pendingWakes`, `unsavedWakes`. - `handoff.ready` is true when the instance is not draining, the census is complete, and `handoff.blockingRuns` and `handoff.unsavedWakes` are 0. Saved pending wakes do not change it. - The wake queue tracks each public wake admission until its wake request and queued run commit. `unsavedWakes` is the count of admissions before that commit. Without a valid count, every pending wake is unsaved. - The top-level version 1 fields keep their values and meaning. `ready` and `blockingRuns` are unchanged, so a current consumer sees no looser verdict. - The census query marks a run `restartable` only with all of these: `paperclip_runner` adapter; no process identity; no provider session before or after; no checkpoint header; no dispatched runner task; no execution-start event. It must also be a native run in the `observed` phase with preparation fencing and an unsettled finalization record, or a legacy run in the `preparing` stage under a recorded controller. - All other runs without adoption evidence stay blocking. This includes local adapters, legacy runs with a child process, follow-up runs that carry a provider session, and runs the census could not observe. - The OpenAPI description and `doc/testing-native-remote-continuity.md` describe the new fields and their limits. Why these classes are correct on master: - Restartable native runs: restart recovery claims a run with no checkpoint and no provider evidence as `bootstrap_incomplete` and executes the same run again (`native-restart-recovery.ts` `classifyNativeRunnerRecoveryEvidence`; `heartbeat/recovery.ts` executes each claim). A graceful shutdown suspends the run and notifies recovery. Existing tests: "recovers an unclaimed remote startup once without requiring a provider checkpoint" and "recovers an unclaimed remote startup that a graceful shutdown suspended during preparation". - Restartable legacy runs: process-loss recovery retries a `preparing` run with `providerWorkStarted: false`. Existing test: "continues a chain whose preparations were killed twice and then interrupted again, without a Retry". - Pending wakes: the count is wake admissions in flight in the process, not queue rows. A wake commits its wake request and its queued run in one transaction. After the commit, the next server dispatches the queued run at startup, so a saved wake does not block. Before the commit, nothing is saved. Shutdown waits at most 30 seconds and then exits, so an unsaved wake can be lost and blocks. ## Verification - `cd server && npx vitest run src/services/heartbeat/upgrade-readiness.test.ts src/services/heartbeat/upgrade-readiness-post-result.test.ts`: 28 passed. - Red/green: before the implementation, the 7 new tests failed (6 summary tests, 1 PostgreSQL census test). After it, all pass. - Red/green for unsaved wakes: a test holds the agent row lock while a wake admission waits. Without the fix, `handoff.ready` was true with no saved wake. With the fix, it is false with `unsavedWakes: 1`, and the wake saves after the lock releases. - New cases: only recoverable runs; recoverable plus unlaunched runs plus pending wakes (`handoff.ready` true, top-level `ready` false and `blockingRuns` unchanged); a started run without recovery evidence; an unobserved execution; draining; truncated census. - The PostgreSQL test changes one fact at a time (process identity, provider session, phase, fencing, checkpoint, adapter, each execution-start event) and expects the run to become blocking. - `cd server && npx vitest run src/__tests__/openapi-routes.test.ts src/__tests__/instance-settings-routes.test.ts`: 157 passed (187 with the two readiness files). - `cd server && npx tsc --noEmit`: passed. - `pnpm verify` exited 2 (a documentation path is a global CI input). Required CI supplies the broad coverage. - Not run: a live server replacement. The classification relies on the existing recovery tests named above. ## Risks - Low risk for current consumers. The change is additive and the version 1 fields are unchanged. - `handoff.ready` is an observation at one instant. A wake admission that starts after the probe is unsaved until it commits. - Restart recovery applies its own proofs and a bounded attempt budget before it starts a restartable run again. The report is an observation, not a guarantee for each run. - The probe adds one indexed `not exists` on run events per running row, inside the existing two-second statement timeout. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Fable 5.2 (`claude-fable-5-2`), with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
fb695f7ae6 |
fix(secrets): let an approved binding fill a blank agent config value (#20908)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents can propose binding a company secret to another agent; a person approves it with "Create binding" on a chat card > - Approval refuses when the target config key already holds a different value, so it never overwrites a working credential > - An empty string or empty plain value also counted as "already exists" > - So an agent whose key was cleared could not be repaired through the governed binding card: approval returned 409 > - This pull request treats a blank value like an unset key; non-empty values still conflict > - The benefit is that the supported binding approval repairs a member with a blank credential ## Linked Issues or Issue Description Refs #20879 **What happened** In the `teams.configure-team` chat eval, the CEO proposed binding the company OpenRouter secret to the QA member as `env.OPENROUTER_API_KEY`. QA's saved value was an empty string. Approving that proposal fails with 409 "Agent config path already exists: env.OPENROUTER_API_KEY". **Expected behavior** Approval fills a blank value (empty string or empty plain value) with the secret reference. A non-empty value still blocks the approval. **Steps to reproduce** 1. Save an agent with `adapterConfig.env.TOKEN = ""`. 2. Have an agent propose a binding of a company secret to that agent as `env.TOKEN`. 3. Approve the proposal as the board user. 4. The approval returns 409 and the binding is not created. ## What Changed - `secret-proposals.ts`: `applyBindingApproval` treats `undefined`, `""` and `{ type: "plain", value: "" }` as an unset key for both `env.<KEY>` and other config paths. - `secret-proposals-routes.test.ts`: regression tests for an empty string and an empty plain value. The existing test that a non-empty value conflicts still passes. ## Verification - `pnpm exec vitest run src/__tests__/secret-proposals-routes.test.ts` (in `server/`): 49 passed. - Red check: without the service change, the two new tests fail. ## Risks - Low risk. Only a blank value changes behavior. A blank value carries no credential, so nothing working is replaced. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
7e8901aa60 |
fix(recovery): let Retry and the person's answer continue a held task with a pending decision (#20872)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run on a remote sandbox can lose its Runner process.
Recovery then saves the sandbox files and puts an execution hold on the
task
> - The hold offers Retry. A person can also continue the task with a
new message
> - Many held tasks also have a pending decision: the question that a
parked turn asked, or an older connection request
> - Continuation admission refuses every continuation while a decision
is pending ("decision_pending"). The answer to that decision is not a
message, so admission defers it behind the hold ("execution_recovery")
> - So Retry waits for the answer and the answer waits for the hold. The
task shows a Retry that does nothing, and nothing continues it
> - This pull request lets a person's Retry pass an open question, and
lets the person's own answer continue the task through the hold
> - The benefit is that a task with a lost Runner and a pending decision
always has a control that works
## Linked Issues or Issue Description
**What happened?**
On staging, the same-run retry of six remote native runs found the
Runner process gone (`process_not_running`, lifecycle `ready`). Each run
failed with `runner_remote_process_identity_unavailable`. Saved-work
recovery then recovered the sandbox files, and the task showed "Saved
work recovered. Retry or send a message to continue." with `canRetry:
true`. Four of these tasks also had a pending decision: the question
that the parked turn asked, or an older pending connection request. For
those tasks, Retry is deferred with `decision_pending`, and an answer to
the decision is deferred with `execution_recovery`. No control continues
the task.
**What did you expect to happen?**
Retry starts one new run. The person's answer to the pending decision
also continues the task.
**Steps to reproduce**
1. A remote native run asks the user a question and parks the turn.
2. The Runner process exits. The same-run retry fails with
`runner_remote_process_identity_unavailable`.
3. Saved-work recovery recovers the files. The task offers Retry.
4. Click Retry: the Retry waits with "A pending approval or question
must be resolved before this message can start."
5. Answer the question: the answer waits with "Waiting for execution
recovery. Your message is saved."
Refs #20838, #20853
## What Changed
- `admitExplicitNativeContinuation`
(`server/src/services/explicit-native-continuation.ts`): a task Retry is
no longer held by a plain pending question (`ask_user_questions`) or
connection request (`connection_intent`). Any interaction with approval
semantics still holds Retry: a plan review (`issue_document` target), a
governed tool action, a secret proposal, a connection authorization, and
other confirmations. A pending approval still holds every continuation.
A new message supersedes a plain question, as on master, and waits for a
decision with approval semantics.
- `admitExplicitNativeContinuation`: the person's own resolution of an
interaction (`mutation: "interaction"`, answered/accepted/rejected,
resolved by the same user) is explicit user input. It continues the held
task like a new message: a fresh session, the hold resolved as
`explicitUserContinuation` with the `interactionId`, and
`issue.execution_recovery_settled` activity. It never replays the failed
run. Governed tool-action confirmations are excluded. The existing
`readQueuedInteractionResponse` reads the answer.
- `heartbeat/queue.ts`: both admission calls pass the wake payload to
the admission.
- `buildExecutionContinuation`
(`server/src/services/execution-continuation.ts`): dispatch rechecks the
user authorization. It accepted an interaction receipt only through a
saved interrupt queue, so the answered successor failed with
`continuation_user_authorization_missing`. It now also accepts the
person's own answer: the interaction named by the durable authorization
was resolved by that user and not by an agent, and the authorization
names this successor run. The check does not depend on the run's current
trigger, so a later answer that replaces the trigger cannot invalidate
it.
- New API-level test
`server/src/services/governed-wait-runner-loss.test.ts` with the two
staging shapes (a parked question from the failed run; an older
connection request on a monitor wake). It checks that Retry starts
exactly one run and leaves the decision pending, that the person's
answer starts exactly one run and settles the hold, and that an answer
resolved by a different user keeps the hold.
- `doc/execution-semantics.md`: describes the rule.
## Verification
- `cd server && pnpm exec vitest run
src/services/governed-wait-runner-loss.test.ts`: 5 passed. The answer
tests now also build the successor's continuation as dispatch does and
check that the agent receives the answer.
- Red for dispatch (first head,
|
||
|
Nicky Leach
|
25f0957c24 |
Select the AI Connection sign-in computer automatically (#20906)
## Thinking Path > - Paperclip manages agents and their AI Connections. > - Codex and Claude subscription sign-in can run on a company agent computer. > - Core stores the credential and uses connection permissions to control access. > - The Connections page asks the user to choose a computer just to host sign-in. > - This change lets Core choose an eligible computer when sign-in starts. ## Linked Issues or Issue Description Refs #20410 and #20417. **What existing behavior does this improve?** OpenAI and Anthropic subscription sign-in from Connections with an agent computer as the sign-in environment. **Current behavior** The user must choose a sign-in computer before the provider flow can start. **Proposed behavior** Start sign-in from the provider form. Core chooses a computer in the company and sign-in environment. The session stays bound to that computer and reports when no eligible computer is available. **Reason and benefit** The saved credential and its permissions are independent of the computer that hosts sign-in, so users can proceed without picking an agent. ## What Changed - Select an eligible computer when a managed Codex or Claude login starts without an explicit target. Prefer ready computers and skip active login leases. - Remove the picker and let the browser resume the existing session by environment and connection intent. Continue to honor explicit agent targets in agent settings. Remove the unused picker fallback from the legacy form. - Update the AI Connections guide, the plugin environment contract, and the OpenAPI descriptions. ## Verification - Passed server and UI typechecks. - Passed 317 tests across the login environment, setup-token route, device-login service, and two UI suites. The Postgres test covers eligibility, inherited environments, explicit targeting, occupied computers, and the no-computer response. - Passed 45 OpenAPI route tests. - Reran the affected UI suites on the final revision. All 230 tests passed. - UI tests cover starting from the provider form and resuming both provider flows with a server-selected target. - [CI](https://github.com/paperclipai/paperclip/actions/runs/43086849302) and [Greptile 5/5](https://github.com/paperclipai/paperclip/pull/20906#issuecomment-6805135968) passed on the final commit. - Local verification used test transports. Live provider sign-in remains to be checked in a Cloud composition that contains this change. ## Risks - Two sign-ins that select a computer before either lease is recorded can contend for the same computer. The existing provider admission guard handles the conflict. A retry can select another computer once the lease is visible. - The computer selection uses existing lifecycle and lease data. Plugin admission still validates the live computer. - No schema migration or plugin protocol change is required. ## Model Used OpenAI Codex (GPT-6), with shell execution and local test tooling. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge |
||
|
DottaandClaude Opus 5.5
|
eb5ad4c920 |
fix(cases): say a Cases-disabled denial is final and name the Experimental setting (#20892)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Cases is an experimental feature; an instance admin turns it on in
Experimental settings
> - When Cases is off, every case and workflow route answers 403 "Cases
are disabled" with no other information
> - A chat agent asked to "open a customer case" could not tell whether
the denial was final or what to offer
> - In one eval run it created a project and a task in place of the
case; in another it spent most of its turn searching routes before it
replied
> - This pull request gives the denial a code and a final, actionable
remediation, like the board-only denial
> - The benefit is a fast, truthful reply that names the setting and
lets the person choose
## Linked Issues or Issue Description
**What happened**
In the chat eval cases.cases-disabled ("Open a customer case for this
bug." with Cases off), the agent got 403 "Cases are disabled". In one
run it then created a project and a bug task without asking. In a later
run it searched many routes and instance settings for about 100 seconds
before it replied.
**Expected behavior**
The denial says that Cases is off, that the denial is final, where an
instance admin turns it on, and that the agent must not create a
substitute record before the person chooses. The agent replies at once
and offers the setting or an ordinary task.
**Steps to reproduce**
1. Turn off Cases in Experimental settings.
2. As an agent, call GET /api/companies/{companyId}/cases.
3. See a bare 403 "Cases are disabled" with no code or remediation.
## What Changed
- Rebased on #20894, which already gave the case routes a
`cases_disabled` code, a final remediation and the hidden-settings
check. This PR moves that remediation into one helper used by every case
and workflow path, and adds: no substitute task or project before the
person chooses, and the offer of an ordinary task.
- New `casesDisabledError()` in `server/src/services/cases-disabled.ts`:
same 403 message, plus `code: "cases_disabled"` and a remediation. The
remediation says the denial is final, forbids a substitute task or
project before the person chooses, links `[Experimental
settings](/<prefix>/company/settings/instance/experimental)`, and offers
an ordinary task.
- Native tool results carry only the error message, so the native case
tools, workflow tools and runner tool authority inline the remediation
in the message (`casesDisabledError({ inlineRemediation: true })`).
- Case routes, workflow service, case handoffs, native case tools,
native workflow tools and the runner tool authority use the helper
instead of a bare `forbidden("Cases are disabled")`.
- Tests: `cases-routes.test.ts` checks the code and remediation;
`native-case-tools.integration.test.ts` checks the native message
carries the remediation (red on master, green here).
## Verification
- `npx vitest run src/__tests__/cases-routes.test.ts
src/services/case-handoffs.test.ts
src/services/native-runtime/native-case-tools.integration.test.ts
src/__tests__/heartbeat-case-handoff.test.ts` (166 passed).
- The cases.cases-disabled chat eval is rerun on a temporary branch;
results are recorded in the PR conversation.
## Risks
- Low risk. The status and message do not change. The error body adds
`code` and `remediation`.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
6dc36700b1 |
fix(costs): mark unpriced usage in the agent, model and project cost reports (#20889)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Cost reports show what agents and projects spent
> - A run can have usage with no known price; Paperclip saves it with
costStatus "unpriced" and costCents 0
> - The summary and the provider, biller and user reports count those
events, but the agent, agent-model and project reports did not
> - An agent that read the by-agent-model report saw a native run at
costCents 0 and told the person it cost $0.00
> - This pull request adds unpricedEventCount to those three reports and
tells agents what it means
> - The benefit is that missing charges are reported as unpriced, not as
free
## Linked Issues or Issue Description
**What happened**
In the chat eval budgets-costs.spend-breakdown, the person asked "How
much did Launch cost this week, broken down by agent?". The week
included a real native run whose usage had no known price (data-quality
reason missing_price). The agent read /costs/by-agent-model and
/costs/by-project. Both returned costCents 0 for that usage and no
unpriced count. The reply showed "1 native run ($0.00)" and gave the
total as complete.
**Expected behavior**
The reports say which rows include unpriced usage. The agent says that
usage is unpriced, marks the total incomplete, and links the record.
**Steps to reproduce**
1. Record one priced and one unpriced metered cost event for an agent in
a project.
2. Call GET /api/companies/{companyId}/costs/by-agent, /by-agent-model
and /by-project.
3. See costCents counts only the priced event, and no field shows the
unpriced event.
## What Changed
- `costService.byAgent`, `byAgentModel` and `byProject` return
`unpricedEventCount` (same filter as the summary: unpriced and not
subscription_included).
- Shared types `CostByAgent`, `CostByAgentModel` and `CostByProject` add
`unpricedEventCount`.
- The OpenAPI catalog entries for the cost reports that count unpriced
usage now say that costCents counts priced usage only, that unpriced
usage must be reported as unpriced (never $0.00), and that
/costs/data-quality lists those runs.
- Tests: embedded-Postgres service test (red on master, green here) and
an OpenAPI description test. A UI test fixture adds the new field.
## Verification
- `npx vitest run src/__tests__/costs-service.test.ts` (19 passed). The
new test fails with the master service.
- `npx vitest run src/__tests__/openapi-routes.test.ts
src/services/native-runtime/runner-api.test.ts` (2146 passed).
- `tsc --noEmit` for packages/shared and server.
## Risks
- Low risk. One added aggregate column per report and an added response
field. The UI does not read it yet.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
77757a3a93 |
fix(evals): serve case-facts its own evidence; accept block-on-design's blocked-interaction reply (#20891)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat-feature eval catalog runs real chat journeys and grades the
saved product state
> - Two cases failed in eval run gha-43069861174 because of the eval,
not the product
> - `cases.update-case-facts` gave its investigator the wrong tool
content. `task-coordination.block-on-design` waited for a run count that
the product does not promise
> - A wrong fixture or a wrong wait hides real product results, so these
must be correct
> - This pull request gives the case-facts investigator the content its
case would really serve, and lets block-on-design accept the product's
bounded blocked-interaction reply
> - The benefit is that both cases test their real feature again, and
the checks still reject the failures that matter
## Linked Issues or Issue Description
**What happened**
`cases.update-case-facts`: the shared Website Brief app served the
website prerequisite text ("record a short implementation note and
complete the task") to the case investigator. The investigator said that
the text did not match the case. It then asked the board for the missing
evidence. The task went to `in_review`, so the harness's later board
handoff got a correct 422 ("Resolve task mode, blockers, or review
before handing off").
`task-coordination.block-on-design`: the CEO saved the Design blocker
and also commented on the blocked Website task. The product woke Morgan
in its bounded blocked-interaction mode (`dependencyBlockedInteraction:
true`). Morgan only acknowledged the hold. The pre-approval wait
required exactly one run, so it timed out.
**Expected behavior**
Each fixture gives the agent the facts its own scenario has. Each wait
accepts every behavior that the product contract allows.
**Steps to reproduce**
1. Run `chat-features.runner-opencode.local.cases.update-case-facts`.
Morgan's continuation reads the website brief and asks the board for
evidence. The review handoff gets a 422.
2. Run
`chat-features.runner-opencode.local.task-coordination.block-on-design`.
The CEO comments on Website. Morgan gets a blocked-interaction run. The
first wait times out.
## What Changed
- `startRunBriefGate` takes an optional `briefText`. The default text
does not change for the other cases.
- `cases.update-case-facts` serves "no further reproduction evidence has
been received yet; the affected device is still unconfirmed" after
release (oracle v7).
- `task-coordination.block-on-design` (oracle v8):
- The pre-approval wait accepts Morgan runs only when the server stamped
them as blocked interactions on Website that name the Design blocker,
and they succeeded. Any other Morgan run, a failed run, or a second CEO
run rejects the wait at once.
- New check: before approval, Website has no documents and its status is
not `in_progress`, `in_review` or `done`.
- After approval, exactly one Morgan work run must still produce the
saved result.
- Calibration tests: the valid interaction passes. An unstamped run, a
wrong blocker, a wrong task, a failed run, a missing declared run, a
document saved before approval, and a second Morgan work run each fail.
## Verification
- `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts
--testTimeout=180000
tests/runner-e2e/chat-catalog-domains/runtime-dependency.test.ts
tests/runner-e2e/chat-catalog-domains/integrator-case-facts.test.ts
tests/runner-e2e/chat-catalog-contract.test.ts
tests/runner-e2e/chat-catalog-consistency.test.ts`: 34 passed.
- `npx tsc --noEmit -p tests/runner-e2e`: no errors in the changed
files.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`:
unchanged.
- A paid eval run of the three affected cases is in progress. The PR
comments will have its result.
## Risks
- Low risk. Only eval fixtures, waits and graders change. The other
cases that use `startRunBriefGate` keep the same text.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), extended thinking, tool
use and code execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
951d1b0b1b |
fix(opencode): ask for the empty-response correction with reasoning off (#20890)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner drives OpenCode turns for agents that use OpenCode models, such as DeepSeek through OpenRouter > - Sometimes the model ends a turn inside its reasoning. It reasons that it will reply and call `paperclip_finish`, and then it stops with no answer and no tool call > - The driver sends one correction prompt. The correction used the same reasoning variant, and in the failed runs the model stopped inside its reasoning again > - The run then failed with `opencode_empty_response`, and the person saw "The OpenCode run failed before it completed." > - This pull request sends the correction prompt with the no-reasoning variant, so the final answer comes in the answer channel > - The benefit is fewer failed chat replies, with no extra prompts and no change to the normal turn ## Linked Issues or Issue Description **What happened** A chat reply failed with `opencode_empty_response` in eval run gha-43069861174 (`agent-chat.conference-room`, OpenCode with `openrouter/deepseek/deepseek-v4.1-flash-1017`). The first completion and the one correction each held only a reasoning item (403 and 411 output tokens). Both reasoning items said the model would reply and call `paperclip_finish`. No answer text and no tool call followed. In retained eval evidence, 14 of 879 attempts needed this correction and 3 of those 14 failed. Each failed correction was another reasoning-only completion. **Expected behavior** After an empty completion, the correction gets a final answer or a completion report. **Steps to reproduce** 1. Start an OpenCode turn with reasoning enabled on an OpenRouter model. 2. Let the provider return a completion that holds only reasoning. 3. See that the correction prompt uses `paperclip-default` (reasoning on), so the model can end inside its reasoning again. ## What Changed - `OpenCodeServerDriver` sends the empty-response correction with the `paperclip-no-reasoning` variant. Normal turns keep their own reasoning mode. - `empty-response.test.ts` starts the `reply` and `empty-again` cases with reasoning on. It checks that the first prompt uses `paperclip-default` and the correction uses `paperclip-no-reasoning`. ## Verification - `pnpm exec vitest run src/drivers/opencode/empty-response.test.ts src/drivers/opencode/reasoning-mode.test.ts src/drivers/opencode/interrupt-pending-tool-call.test.ts` (in `packages/paperclip-runner`): 23 passed. Without the driver change, the `reply` and `empty-again` cases fail (red/green). - `node --test tests/runner-e2e/native-completion-source-contract.test.mjs`: 29 passed. - A paid eval run with this change and the related eval fixes is in progress. The PR comments will have its result. One run cannot prove the rate change, because the failure is rare. ## Risks - Low risk. Only the correction prompt changes. A model with no reasoning can give a shorter final answer, but the correction asks only for the final answer and the completion report from work already done. - Providers other than OpenRouter do not get a variant. Their behavior does not change. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), extended thinking, tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
6154b9485e |
fix(authz): name the Experimental settings page when an agent is denied an instance experiment (#20894)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - People ask chat agents to turn on instance experiments, such as
Cases
> - Agents cannot read or change experimental instance settings, which
is correct
> - The denial they get names no page, so the agent cannot tell the
person where to act
> - The case API gate ("Cases are disabled") also names no page
> - This pull request makes both denials, and the API catalog entry,
name the Experimental settings page
> - The benefit is that the agent replies with a usable link instead of
a vague "ask an admin"
## Linked Issues or Issue Description
**What happened**
An agent that asks for `GET` or `PATCH
/api/instance/settings/experimental` gets the generic board-only denial.
That denial says "Link a page only if you already know its route; do not
search for one." The case API, while Cases is off, returns only "Cases
are disabled". The agent has no route to give the person.
**Expected behavior**
The denial is final and names the page where an instance admin turns the
feature on: `[Open Experimental
settings](/<prefix>/company/settings/instance/experimental)`. If the
operator hides that page, the denial does not link it.
**Steps to reproduce**
1. Use an agent API key.
2. Send `PATCH /api/instance/settings/experimental` with
`{"enableCases": true}`.
3. See a 403 with a remediation that names no page.
4. With Cases off, send `GET /api/companies/<id>/cases` with the agent
key.
5. See a 403 "Cases are disabled" with no remediation.
Refs the chat eval `instance-operations.instance-settings` (retargeted
from the removed Agent Chat toggle to Cases).
## What Changed
- `server/src/routes/authz.ts`: the board-only denial map links
Experimental settings for `/instance/settings/experimental`. It does not
link it when `instance.experimental` is hidden.
- `server/src/routes/cases.ts`: the Cases gate keeps the "Cases are
disabled" message. It adds `code: cases_disabled` and a final
remediation that names the same page (same hidden-page guard).
- `server/src/routes/openapi.ts`: the GET and PATCH catalog entries for
experimental settings name the page. They say that agent credentials
cannot read or change these settings. They give Cases (`enableCases`) as
an example.
- Regression tests in `board-access-denial.test.ts`,
`cases-routes.test.ts` and `openapi-routes.test.ts`.
## Verification
- `cd server && pnpm exec vitest run
src/__tests__/board-access-denial.test.ts
src/__tests__/openapi-routes.test.ts
src/__tests__/cases-routes.test.ts`: 107 passed.
- Red check: with the three source files reverted, 4 new assertions fail
(GET and PATCH denial links, the catalog entry, the Cases gate).
- `src/services/native-runtime/runner-api.test.ts` and
`src/__tests__/instance-settings-routes.test.ts`: pass.
## Risks
- Low risk. Only error bodies and catalog text change. Status codes and
the "Cases are disabled" message stay the same.
- The catalog text is static, so it names the page even when an operator
hides it. The denials check the hidden setting.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
a589b74fbf |
fix(chat): name goal links, missing-name questions and component reference pages in chat guidance (#20888)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - People talk to their agents in chat, and the chat guidance tells the
agent how Paperclip works
> - Chat evals showed three gaps in that guidance
> - The agent explained a task's purpose from titles only, because the
guidance gives no goal link and no instruction to read goal descriptions
> - The agent built a full landing page with an invented product name,
because the chat guidance has no rule to ask for a missing name first
> - The agent posted a real question card or drew a text mock when the
person asked to see a card, because the guidance names the design guide
and ux-lab pages only in general
> - This pull request adds three short rules to the chat guidance
> - The benefit is grounded purpose answers with links, one necessary
question instead of placeholder work, and links to the real reference
pages
## Linked Issues or Issue Description
**What happened**
In chat evals (OpenCode runner, DeepSeek flash): (1) "Why are we doing
this website task?" got an answer from titles only, with no goal links
and without the parent goal's stated reason. (2) "Build a landing page
for my new product." produced a full site named with a placeholder; the
agent never asked for the product name. (3) "Show me the standard
question card" made the agent post a real question card. "Show me how
chat looks when a tool request is denied" made the agent draw a text
mock and probe real tool requests.
**Expected behavior**
(1) The agent reads the task, project, goal and parent goals, gives the
reason that the goal descriptions state, and links each record. (2) The
agent asks one free-text question for the missing name and uses the
answer in the brief. (3) The agent links the design guide question card
or the denial ux-lab page and says they are synthetic presentation
fixtures.
**Steps to reproduce**
1. Run the chat evals goals.explain-purpose,
questions-and-approvals.necessary-question,
developer-labs.component-reference and
developer-labs.interaction-fixture on master.
2. Read the reply and tool events of each run.
## What Changed
- Chat guidance: goal link format `/<prefix>/goals/<goalId>`, and a rule
to explain a task's purpose from its saved goal chain (GET
/api/goals/{id}, follow parentId), link every record, and say so when
the task has no goal.
- Chat guidance: ask for a missing name (for example the product name on
a landing page) with `request_human_input` (`questions`, one required
free-text question) before building or delegating; never use a
placeholder name.
- Chat guidance: when the person asks to see a component or chat state,
link the design guide (question card states) or the named `/ux-lab/...`
pages instead of creating a real card or a mock, and say they are
synthetic fixtures.
- Tests: new red/green tests in `agent-conversations.test.ts`; they also
check that the named UI routes exist in `ui/src/App.tsx`.
## Verification
- `npx vitest run src/__tests__/agent-conversations.test.ts` (62
passed); the new tests fail on master because the text is absent.
- `npx vitest run` for heartbeat-context-summary,
heartbeat-chat-task-link, board-access-denial, task-markdown,
current-wake-comments and heartbeat-list tests (119 passed).
- The affected chat evals are rerun on a temporary branch; results are
recorded in the PR conversation.
## Risks
- Low risk. Guidance text only. The chat directive grows by about 1.6
KB.
- The missing-name rule is narrow (a name the work must carry). It must
not cause extra questions for clear requests.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
5c440d8889 |
test(evals): make scripted chat input name its referent (accept-one-proposal, share-private-task) (#20896)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat catalog evals check that agents in chat keep their user
promises
> - Two cases sent a scripted message whose referent was not clear in
the conversation
> - The agent then answered a different question, so the eval did not
test its promise
> - This pull request makes each scripted message name the thing it
refers to, without telling the agent what to do
> - It also replaces a reply keyword check with structural checks
> - The benefit is that the two cases test their real promises again
## Linked Issues or Issue Description
**What happened**
`questions-and-approvals.accept-one-proposal`: the person accepted a
"Create Acorn launch checklist" card. The CEO created the task once and
replied with a new offer: move the task into todo and execute it. The
eval then sent the same bare "Yes, go ahead with that." again. The CEO
read it as a yes to the new offer and moved the task to todo. An
assignment wake made a third run, over the declared bound of two.
`tasks.share-private-task`: the eval sent "Share this private task with
Riley." in the standing CEO chat. In that chat, "this" is the chat task
itself. The CEO tried to share its own chat task and never linked the
private task. The grader also matched reply words (Riley, owner/share
phrases).
**Expected behavior**
A repeated acceptance must refer to the same proposal, so that the eval
can show that one acceptance creates exactly one task. The share request
must name the private task, so that the eval can show that the agent
routes the owner to the task's sharing control and does not change its
audience.
**Steps to reproduce**
1. Run
`chat-features.runner-opencode.local.questions-and-approvals.accept-one-proposal`.
The CEO reply ends with a new offer, and the bare repeat accepts it.
2. Run `chat-features.runner-opencode.local.tasks.share-private-task`.
The CEO acts on its chat task, not on the private task.
## What Changed
- accept-one-proposal: the second human message names the accepted card
by its visible title (`Yes, go ahead with the "Create Acorn launch
checklist" request.`). The grader checks this exact scripted message.
All other checks stay: one task, still in backlog, one decision bound to
the first reply, two runs. Oracle version is now
`paperclip.chat-catalog.accept-one-proposal.v2`.
- share-private-task: the query is now `Share my private task "Private
launch task" with Riley.` The change is recorded in the manifest
`changedFields`, pinned in its test, and the inventory digest is
updated.
- share-private-task: the reply keyword check is removed. The grader
requires a rendered link to the named private task. `verifyResult` has
the owner open that link and open the task's Share control ("Who can
access this task"). Then it confirms that Riley is still denied. The
audience check also rejects a grant to any person who is not the owner.
Oracle version is now `paperclip.chat-catalog.share-private-task.v2`.
- Calibration tests: a bare repeat or a "move it to todo" repeat fails;
a todo task fails; reworded handoffs pass; a link to the chat instead of
the task fails; grants to Riley or to another person fail; public
visibility fails.
## Verification
- `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts` on
the case tests, contract, consistency and every chat-catalog-domains
test that imports the changed modules: 14 files, 285 tests pass.
- `node --test
tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs`: 10 pass.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`:
no change.
- `npx tsc --noEmit -p tests/runner-e2e`: no errors in the changed
files. Other errors come from the local worktree dependency links.
- Offline regrade of the retained share-private-task run: the run fails
the handoff check (it linked only the chat). The same run with a link to
the private task passes.
- Paid eval runs for both cases are pending.
## Risks
- Low risk. Only eval code, the scenario query and catalog metadata
change.
- The share-private-task grader no longer reads the reply. A false
success claim with a correct link and no saved grant is not caught by
code. A reviewer must read the reply in the paid run.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use, extended
reasoning.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
c02de922af |
fix(chat): deliver the person's time zone to chat agents (#20898)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - People talk to agents in chat, and agents create routines and other scheduled work from that chat > - Paperclip does not record the person's time zone, so a chat agent cannot read "every weekday at nine" or "today" in the person's zone > - The agent then saves schedules in UTC, and the person gets the work at the wrong hour > - This pull request saves the browser time zone with the person's personal preferences and gives it to chat agents in the task prompt > - The benefit is that schedules and dates from chat match the person's local time ## Linked Issues or Issue Description **What happened** A person in America/Chicago wrote in chat: "Every weekday at nine in the morning, summarize Launch for me." The agent saved the routine schedule `0 9 * * 1-5` with the UTC time zone. It replied that it used UTC because the person's time zone is not recorded. **Expected behavior** The agent knows the person's time zone. It saves one enabled `0 9 * * 1-5` schedule with the time zone America/Chicago. Its reply names the zone. **Steps to reproduce** 1. Open the board in a browser with the time zone America/Chicago. 2. Open a chat with an agent. 3. Send "Every weekday at nine in the morning, summarize Launch for me." 4. Look at the routine trigger that the agent creates. Its time zone is UTC. ## What Changed - Database: add the nullable `time_zone` column to `user_workspace_preferences` (migration `0468`). This is an instance-wide personal preference by user, as AGENTS.md section 5 permits. - Shared: `WorkspacePreference` has `timeZone`. `updateWorkspacePreferenceSchema` accepts an optional IANA `timeZone` (max 64 characters, checked with `Intl.DateTimeFormat`). `coordinatorWorkspaceEnabled` is now optional. The request must contain at least one field. - Server: the preference service returns `timeZone` and updates only the fields in the request. The activity log entry stays the same. The OpenAPI entry uses the shared schema. - UI: a new `useBrowserTimeZoneSync` hook in `Layout` saves the browser time zone one time when it is different from the saved zone. It is silent and best-effort. It uses the selected company as the audit context. - Prompt: for a conversation (any adapter), the task context reads the chat owner's saved zone. Recovery keeps the zone that the first turn admitted. The chat block adds one line only when the zone is known: `The person's time zone is America/Chicago (from their browser). Read times and dates they give, such as "nine in the morning" or "today", in that zone, and save schedules such as routine triggers with that timezone; say the zone when you report a schedule.` ## Verification - `cd server && npx vitest run src/__tests__/workspace-preferences.test.ts src/__tests__/workspace-preferences-routes.test.ts src/services/heartbeat/task-markdown.test.ts src/__tests__/heartbeat-task-context.test.ts src/__tests__/openapi-routes.test.ts src/services/native-runtime/runner-api.test.ts` - `cd ui && npx vitest run src/hooks/useBrowserTimeZoneSync.test.tsx src/hooks/useCoordinatorWorkspaceEnabled.test.tsx src/components/Layout.test.tsx` - Red/green: without the task-context and task-markdown changes, the two new prompt tests fail. With the changes, they pass. - `pnpm check:token-gates` is clean. `tsc --noEmit` passes for `packages/shared` and `packages/db`. - Manual: open the board in a browser with a non-UTC zone. Then ask a chat agent to create a weekday routine at nine in the morning. The trigger uses the browser zone. ## Risks - Low risk. The migration adds one nullable column. - The PUT body is now more flexible: `coordinatorWorkspaceEnabled` is optional. An empty body still gets a 400 response. - The prompt grows by one line only for conversations with a known zone. - The saved zone changes when the person uses a browser in a different zone. The latest browser wins. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
5a332517da |
fix(evals): start the owned dev instance with a supported run command (#20885)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat eval catalog has a case that checks how chat handles a request to set up a throwaway Paperclip instance > - In this case, a scripted operator creates an owned development instance with the supported CLI after the agent answers > - The operator step always failed, so the eval never measured the agent follow-up > - The fixture started `paperclipai run` with an unsupported `--yes` flag, and it discarded the child output, so the cause was not visible > - This pull request starts the owned instance with a supported command, keeps the child output for diagnosis, and replaces reply keyword checks with structural checks > - The benefit is a valid eval that measures the case promise, and a readable cause if the operator step fails again ## Linked Issues or Issue Description **What happened** The `cli-operations.owned-dev-instance` chat eval failed every time with "Stopped waiting for Actual owned development target readiness: Owned dev server exited before readiness". The agent reply was correct. The scripted operator phase failed. The fixture ignored the child stdio, so the evidence did not contain the cause. **Expected behavior** The fixture starts the owned development instance with a supported CLI command. The instance becomes healthy. The eval then grades cleanup, isolation and the agent follow-up. If the child exits, the error names a bounded and redacted reason. **Steps to reproduce** 1. Start a disposable source instance with an embedded database. 2. Copy the checkout like the fixture does, then run `paperclipai worktree init --instance X --home H --from-config <source config> --server-port P --db-port D --seed-mode minimal`. 3. Run `paperclipai run --config <checkout>/.paperclip/config.json --yes`. It exits 1 with `error: unknown option '--yes'`. 4. Run the same command without `--yes`. It exits with `TS2688: Cannot find type definition file for 'node'` because the copied plugin SDK has no linked `node_modules`. ## What Changed - Remove `--yes` from the owned `paperclipai run` command. `run` has no such flag. It onboards only when the config is missing, and `worktree init` writes the config. - Link the installed `node_modules` of every workspace package into the copied checkout. Before, the fixture linked only six packages, so `run` could not compile the copied plugin SDK. - Write the owned CLI output to a private `0600` log under the attempt temp root, outside the owned root that cleanup deletes. - Add a bounded reason (last 12 lines, 1200 characters at most, terminal codes removed, known secrets and secret shapes redacted) to readiness and CLI failure errors. - Replace the `operator-dev-guidance` reply keyword check with `initial-chat-left-host-unchanged`. The check compares the digests of every Paperclip installation config under the attempt root (agent HOME, workspace and runtime directories) before and after the initial turn. - Change `native-reports-real-target` to require only the exact `host:port` that the human receipt supplied (bare, linked or with the scheme). It no longer matches outcome words. - Bump the case `oracleVersion` to `paperclip.chat-catalog.privileged.owned-dev-instance.v2` in code and in the descriptor. ## Verification - Local reproduction with a disposable source instance: the old command exits with `unknown option '--yes'`. Without `--yes` and with the old links, `run` exits with TS2688. With this change, the owned target reports healthy in 16 to 19 seconds. The seeded canary task reads back from the target. `worktree:cleanup` removes the instance. - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/chat-catalog-domains/privileged-operators.test.ts tests/runner-e2e/chat-catalog-contract.test.ts tests/runner-e2e/chat-catalog-consistency.test.ts`: 17 passed. - Calibration: the retained CI reply and reworded receipt reports pass. A changed host installation, a missing target, a longer port, a different port, an unattributed reply and an empty initial reply fail. - `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD sha> --check`: no change. - `npx tsc --noEmit -p tests/runner-e2e`: no errors in the changed files. Remaining errors come from a local dependency mismatch in unrelated packages. - The paid eval `chat-features.runner-opencode.local.cli-operations.owned-dev-instance` is not run yet. ## Risks - Low risk. The change is in one eval fixture and its unit test. - The installation inventory walks the attempt root. It skips `node_modules` and `.git`, and it stops with an error after 1,000,000 entries. - The initial-turn check does not grade the reply wording. A reply that falsely claims to have created an instance, without host changes, needs human review of the retained reply. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), agentic coding with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
683fd1e9a5 |
test(restart-chaos): cells for staging's no-restart failures (re-wake, Nth recovery step, long turn, K15 without volume) (#20893)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart-chaos harness (`tests/restart-chaos/`) tests that agent work survives control-plane faults > - In one 8-hour staging window, 20 of 25 failed runs happened with no restart at all, so the harness did not see them > - The harness had blind spots: one short task per agent, no wake right after a successful run, no fault in a later recovery step, short turns, and a redeploy that always keeps its instance home > - This pull request adds cells for those shapes, so the harness reproduces these failure classes on master > - The benefit is that fixes for these classes can be proven red and green with the harness, without staging ## Linked Issues or Issue Description **What happened?** On staging, an agent ran 10–13 tasks at once on reused sandboxes. 30–60 s after a long run succeeded, an ordinary wake (a comment or an issue monitor) started the next run of the same task. That run or its recovery failed before any provider work, with errors such as `runner_remote_process_identity_unavailable`, `Controller journal drain timed out` and `journal_tool_input_unanswered`. Other failures happened in a later recovery step (a held agent-directory lock, a missing runtime asset after a deploy without a volume). **Expected behavior** The chaos harness can reproduce each of these shapes on demand, and grades them: no failed attempt, every task `done`, and no task left blocked without a working Retry. **Steps to reproduce** See the commands in the README sections "Re-wake after success", "Fail the Nth recovery step", "Long turn" and "Redeploy without a volume, with overlap". Related: #20844 (session fault SR1). W2C is the configuration change after a successful run; SR1 is the same change while a question is parked. This PR does not add or change `session-faults.ts`. ## What Changed - New workloads W2, W2Q, W2C (`rewake.ts`): phase 1 ends with the task left open; the harness, as the board, wakes the same task the seeded 0–60 s after its run settled. W2 uses a comment or a board-scheduled issue monitor (by nonce). W2Q parks on an unanswered `request_human_input` question and is woken by a comment before the answer. W2C changes the agent's `adapterConfig.idleTimeoutMs` before the wake. Oracle: phase 1 succeeded, the wake was served by a run that succeeded, `no_failed_attempt`, `done`, markers. - `--faults K0`: cells with no fault, for workloads whose shape is the fault. - `--fail-recovery-step <step>[:<nth>]` (`recovery-step-faults.ts`): when the wave's first restart fault starts its replacement, the harness arms one crash point in a new fail mode; the nth hit throws the error of the real step, once. Steps: `runnerd-start` (`spawn paperclip-runnerd EAGAIN`), `teardown` (provider destroy did not finish), `directory-lock` (`Agent directory operation is already active`). Crossed tasks get `recovery_step_failed_once` and `no_failed_attempt`. - Crash points (`packages/shared/src/crash-points.ts`): fail mode in the control file (`<token> <name>:<nth> fail [pid=<n>]`), with an optional call-site error; `pid=` binds the line to one server process. The harness arms the step for the replacement only and disarms it once its receipt lands. New point `agent_directory.operation_lock.before_acquire`. The teardown point moved into its `try`, so fail mode behaves like a failed provider destroy; kill mode is unchanged. All points stay disarmed under `NODE_ENV=production`. The runner package's own crash-point copy is not changed (runner image content ID unchanged). - `--long-turn-minutes <n>` (`long-turn.ts`, 15–60, opt-in): every task's first turn runs rounds of real tool work for n minutes before its workload. Checks `long_turn_rounds` and `long_turn_run_observed`. - K15 `--k15-fresh-home` (Daytona) and `--k15-overlap-seconds <min>-<max>` (`redeploy.ts`, `supervisor.ts`, `edge-proxy.ts`): each deploy generation gets a new, empty instance home and no volume variables; the next generation boots beside the old one behind an edge proxy on the wave's port, takes traffic once ready, and the old one gets SIGTERM after the overlap. - `tests/runner-e2e/source-sdk-public-inputs.json`: refreshed pins for the changed shared crash-point files. - README: one section per new cell, with what it covers. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts` (48 files, 581 tests after rebasing on master, includes the new `shape-cells.test.ts`). - `pnpm exec vitest run packages/shared/src/crash-points.test.ts server/src/__tests__/crash-point-call-sites.test.ts` (fail mode, call sites). - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`. - Chaos waves on master `1d67e309e0` server code (Daytona image `ghcr.io/paperclipai/paperclip-daytona-runner@sha256:99fc2671fc320816407f63cffd3acbd27471621febcd26494fe25bb5d7c5dd77`, content ID `1836c515…`), all RED unless noted: | Cell | Wave | Result on master | Staging class | | --- | --- | --- | --- | | W2/W2Q/W2C, 10 tasks on one Codex agent, `--daytona-warm`, K0 | `w-20261111T163206-s5201` | 9/10 fail. 3 tasks (W2 t02, W2C t03, t06): first fault `native_runner_process_exited`, same-run reattach, `native.remote_runner_recovery_failed` with `stage: process_verification`, `reason: process_not_running`, `runnerLifecycle: ready`, `runnerSettled: false`, terminal `runner_remote_process_identity_unavailable`, "Workspace repair required". 3 tasks (t04, t08, t10): `native_restart_recovery_blocked` (`native_retry_source_superseded`) while its successor ran (R4 `concurrent_runs`). 3 W2Q tasks: the answer's continuation was cancelled `issue_not_in_progress` (status `blocked`), and the task needed Retry. 2 W2C tasks then looped: 60+ succeeded runs in 20 minutes on an `in_progress` task. | D′/T terminal step reproduced with no restart, no governed wait and no PRP timeout (same error code and refusal fields). C+ retry outcome (ownership ambiguous) reproduced as `native_restart_recovery_blocked`. Not reproduced: `Controller journal drain timed out`, `did not durably suspend`, `journal_tool_input_unanswered`. | | Same, local `--lifecycle warm` | `w-20261111T160103-s5101` | 4/10 fail. 3/3 W2Q: continuation cancelled `issue_not_in_progress`, Retry needed. 1 W2C: the same `in_progress` run loop (39+ runs). | The remote-runner classes need a remote runner; locally they do not show. | | `--fail-recovery-step runnerd-start:1`, K1 at 10 s, local A1 | `w-20261111T162432-s5303` | Fail. After K1 the first runnerd start of recovery failed (`spawn paperclip-runnerd EAGAIN`); the run ended `native_provider_terminal_failed` ("previous turn is no longer active", "Automatic recovery stopped") and needed Retry. Control, same seed and K1 without the step fault (`w-20261111T171114-s5303`): pass. | A later recovery step that fails once ends the run instead of trying again (the A-chain shape). | | `runnerd-start:2`, local | `w-20261111T170909-s5304` | Step never reached: recovery started the runner once and succeeded. Grade fails `recovery_step_failed_once` (tested nothing). | — | | `teardown:1`, K1, Daytona A1 | `w-20261111T172048-s5312` | Pass. The teardown failed once and the sweep settled it. | Not reproduced. | | `directory-lock:1`, K2, Daytona A1 | `w-20261111T171247-s5311` | Step never reached by A1 after K2. | Not reproduced; A1 recovery does not collect remote agent files. | | K15 `--k15-fresh-home --k15-overlap-seconds 30-150`, `--storage s3`, Codex and OpenCode, T1/C1 | `w-20261111T173618-s5322` | 1/4 fail (OpenCode T1): `native_continuation_requires_reconciliation` with "Agent directory belongs to a different execution environment"; the Retry runs failed `adapter_failed` "Configured instruction entry is missing"; the task rests `blocked` on a board recovery action, and two sandboxes were left (R6). One Codex T1 task finished before its fault (`fault_crossed_tool`). | A-deploy: a controller-local file missing after a deploy without a volume. Here the missing file is the agent's instruction bundle, not a runtime-context asset; whether staging keeps instruction bundles outside the instance home is not verified. | | `--long-turn-minutes 15`, K1 at 600 s, local A1 | `w-20261111T175505-s5401` | Task pass (10/10 rounds, 992 s span, run 1019 s, K1 crossed mid-turn). The wave exited 2 only because I switched branches in the worktree during the wave. | Shape check only. | ## Risks - Product code changes are test instrumentation only: three `crashPoint` call sites take a fail-mode error, and one new call site was added. Disarmed (always in production) each call is one comparison. Moving the teardown point into its `try` does not change kill mode: the process dies at the same instant. - The new cells are opt-in. Default waves plan the same tasks as before for the same seed. - The W2 cells depend on model behaviour (phase 1 leaves the task open). The oracle records the task status at the wake. - `--k15-fresh-home` drops everything under the instance home. Whether staging keeps agent instruction bundles outside the instance home is not verified (see the K15 result). ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, agentic coding with tool use (shell, file edits, test and chaos runs) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
b96967a97b |
fix(authz): name the environment settings page when an agent is denied an environment route (#20880)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Execution environments are the computers that agents run on. Only a
board user can list, inspect, test or change them.
> - An agent in chat can be asked "Which computer can run Morgan with
Claude Code?". Its first environment read returns 403.
> - The 403 says the denial is final, but it names no page. The agent
cannot link the person to the control that answers the question.
> - Other board-only areas (secrets, plugins, budgets) already name
their page in the denial.
> - This pull request adds the environment settings page to the agent
denial for every environment route, and says that testing and changing
computers are board-only too.
> - The benefit is that the agent can send the person straight to the
correct control, and it does not search for a probe route that does not
exist.
## Linked Issues or Issue Description
**What happened**
An agent asked which computer can run a coding harness called the
environment list or capabilities route. The route returned 403 "Board
access required" with a remediation that named no page ("Link a page
only if you already know its route; do not search for one"). The agent
had no route to give the person.
**Expected behavior**
The denial names the environment settings page as a Markdown link with
the `<prefix>` placeholder, like the denials for secrets and plugins. It
also says that the other environment steps (capabilities, probes,
changes) are board-only, so the agent replies instead of trying them.
When an operator hides environment settings
(`PAPERCLIP_HIDDEN_SETTINGS=instance.environments`), the denial does not
link the page.
**Steps to reproduce**
1. Use an agent API key.
2. Call `GET /api/companies/<companyId>/environments/capabilities`.
3. Read the 403 body. Before this change, `remediation` contains no
link.
## What Changed
- `server/src/routes/authz.ts`: board-only denials on
`/companies/:id/environments...`, `/environments/...` and
`/environment-leases/...` now link `[Open environment
settings](/<prefix>/company/settings/instance/environments)` and add one
sentence that says execution environments are board-only throughout. The
link is left out when the operator hides `instance.environments`.
`/companies/:id/project-environments` (agent-readable) is not matched.
- `server/src/__tests__/board-access-denial.test.ts`: regression tests
for the list, capabilities, probe and lease routes, the scope sentence,
the project-environments exclusion, and the hidden-settings case.
## Verification
- `cd server && pnpm exec vitest run
src/__tests__/board-access-denial.test.ts`: 33 passed. With the old
`authz.ts`, 5 of the new tests fail (red), and they pass with the change
(green).
- `cd server && pnpm exec vitest run
src/__tests__/environment-routes.test.ts
src/__tests__/agent-permissions-routes.test.ts`: 99 and 126 passed.
## Risks
- Low risk. Only the agent-facing 403 remediation text changes. Status
codes, the error code and board behavior do not change.
- The scope sentence is environment-specific, so other denials are
unchanged.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1111.0-canary.13
|
||
|
DottaandClaude Opus 5.5
|
854b4bcac4 |
fix(opencode): report a provider 401 from the hello probe as missing authentication (#20878)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Each agent has a runtime readiness test ("test environment") that
runs a short hello probe through its harness
> - The OpenCode probe sorts failures into model, timeout,
authentication and generic errors
> - OpenRouter rejects an empty or unknown key with HTTP 401 and the
messages "Missing Authentication header" or "User not found."
> - The probe matched authentication only by wording, so these 401
replies became a generic `opencode_hello_probe_failed` error
> - The server maps `*_hello_probe_auth_required` to the missing-account
state, so the user saw a generic failure instead of "connect an account"
> - This pull request classifies the structured APIError status 401 as
`opencode_hello_probe_auth_required`
> - The benefit is that a member without a working account shows the
missing-account state and its fix
## Linked Issues or Issue Description
**What happened**
An OpenCode (native runner) agent with an empty OpenRouter key failed
its readiness test with the generic error `opencode_hello_probe_failed`.
The chat eval `teams.configure-team` could not form its "one member is
missing an account, one is ready" fixture because of this.
**Expected behavior**
The readiness test reports `opencode_hello_probe_auth_required` (warn)
when the provider rejects authentication with HTTP 401, whatever the
provider's wording.
**Steps to reproduce**
1. Configure a native runner OpenCode agent with model
`openrouter/deepseek/deepseek-v4.1-flash-1017` and `OPENROUTER_API_KEY`
set to an empty string.
2. Run the agent's environment test.
3. OpenCode 1.18.34 prints
`{"type":"error","error":{"name":"APIError","data":{"message":"Missing
Authentication header","statusCode":401}}}`.
4. The result is `opencode_hello_probe_failed` (error), not the
missing-authentication check.
## What Changed
- `parse.ts`: add `openCodeErrorStatusCodes`, which reads the HTTP
status from OpenCode error events.
- `test.ts`: treat a 401 status on the hello probe as
`opencode_hello_probe_auth_required`, in addition to the existing
wording match.
- `test.remote.test.ts`: regression tests with the two recorded
OpenRouter 401 outputs, and a negative control (500 and 429 stay generic
failures).
## Verification
- `pnpm exec vitest run
packages/adapters/opencode-local/src/server/test.remote.test.ts
packages/adapters/opencode-local/src/server/parse.test.ts` passes (28
tests).
- Red check: without the classifier change, the two new 401 tests fail.
- `npx tsc --noEmit -p packages/adapters/opencode-local` passes.
- I recorded the probe output from the pinned OpenCode 1.18.34 binary
against OpenRouter with an empty key and with an unknown key.
## Risks
- Low risk. Only a 401 status changes classification. A 401 means the
provider rejected the credential. Other statuses keep their current
result.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
Nicky LeachandPaperclip
|
5dfbb04b85 |
Poll asynchronous workspace provider preparation (#20865)
## Thinking Path > - Paperclip starts agent runs through a workspace provider when a project uses remote storage. > - The provider must prepare the workspace before the agent starts. > - One long worker call ties the checkout to its transport. > - Some checkouts take longer than the transport allows. > - This change lets an opted-in provider accept work and report its deadline. > - Core waits for completion through short status calls and stops at the provider deadline. ## Linked Issues or Issue Description **What happened?** A workspace provider can lose its reply when preparation takes longer than a single transport request. This stops run bootstrap before agent execution. **Expected behavior** Core should wait for a provider-confirmed workspace through short requests. The provider's deadline must bound the wait. **Steps to reproduce** Use a workspace provider that takes longer to prepare a checkout than its transport can remain open. Start a run that requires a new checkout. ## What Changed - Add an optional workspace preparation method to the plugin SDK. - Validate pending receipts and keep the original binding checks on completed replies. - Poll only during active preparation. Retry a lost status reply within the advertised deadline. - Preserve the original operation for providers that do not offer asynchronous preparation. - Reject deadline offers that exceed the existing lifecycle lock budget. Honor accepted offers without shortening them. - Checkout verification uses the existing synchronous RPC with a 10-second deadline and a 12-second transport timeout. [Core verification PR #20876](https://github.com/paperclipai/paperclip/pull/20876) ## Verification - All 63 workspace SDK and Core integration tests passed with temporary local PostgreSQL. The tests cover lost status replies, deadline expiry, oversized offers, and synchronous verification with an async-capable worker. - All 58 runner SDK input, evidence, and public type tests passed with the regenerated source pins. The local server typecheck passed. ## Risks - This change activates only when the provider advertises the method. Existing providers keep their current behavior. - Polling uses one-second intervals during active preparation. It stops at completion, failure, or the provider deadline. - No database migration is required. ## Model Used - OpenAI GPT-6 Codex with local code execution and tests. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
e601f9d2fc |
fix(runner): answer each OpenCode MCP tool call on its own connection (#20886)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The Paperclip runner runs native OpenCode agents. OpenCode calls Paperclip tools through a local MCP bridge in the OpenCode proxy. OpenCode runs on Bun and uses Bun `fetch` for each MCP request. > - In one chat eval, the model made two parallel tool calls. The bridge produced and wrote both results in about 0.55 s. OpenCode received only one of them. > - The other call waited until Bun's default fetch timeout fired, about five minutes later. The chat turn lost five minutes, and the eval missed its deadline. > - The most likely trigger is the reuse of an idle keep-alive socket. We could not reproduce the loss locally, so this pull request is hardening that is based on a hypothesis. > - This pull request makes the bridge answer every request on its own connection. It also records how each tool result left the bridge, so a recurrence can be attributed from the run log. > - The benefit is that a pooled socket can no longer strand a tool result. If a call is stranded again, the run log shows whether the bridge wrote the answer. ## Linked Issues or Issue Description No public issue exists. Related: Refs #18751 (draft native client acceptance diagnostic), Refs #19546 (abandoned queued calls), Refs #20862 (transport event delivery; it does not touch this path). **What happened** In CI run `gha-42925998380` (chat-features, runner-opencode, `cases.create-case`), the CEO chat run used OpenCode 1.18.34 with an OpenRouter DeepSeek model. At 00:01:12.16 the model issued `list_workflows` (MCP id 3) and `list_projects` (MCP id 4) in parallel. The runner emitted the dynamic tool results for id 4 at 00:01:12.468 and for id 3 at 00:01:12.714 (ok, HTTP 200). The bridge writes the HTTP response directly after that event. There is no queue between them. OpenCode completed `list_projects` at 00:01:13.113. It failed `list_workflows` only at 00:06:01.761 with `The operation timed out.` We read the bundled OpenCode source. Its MCP tool call has no timeout that produces that message: the MCP request timeout gives `-32001 Request timed out`, and the configured value is 1 h for durable calls. With the same binary in Bun mode (Bun 1.3.14), a fetch on a reused keep-alive socket that never gets an answer fails after 298 s with `TimeoutError: The operation timed out.` So Bun's default fetch timeout ended the call, and the client never consumed the written answer. The previous call (`list_agents`) answered at about 00:01:06.89, so its socket was idle for about 5.27 s when the parallel pair started. Our thesis is that call 3 reused that pooled socket, and call 4 opened a new one. **Expected behavior** When the bridge has the result of a tool call, the MCP client gets it. The client must not wait for a five-minute transport timeout. **Steps to reproduce** We did not reproduce it locally. This is the only occurrence in the retained evidence. A macOS stress test (Bun 1.3.14 client, Node 26 server with the bridge's default keep-alive, one call, then an idle time of 4.8–7.2 s, then two parallel calls, 8 pairs in parallel) completed 343 trials and about 1,029 requests with 0 lost results. The failing run was on Linux CI with Node 24. ## What Changed - `mcp-bridge.ts`: every bridge response sends `Connection: close`. The MCP client cannot reuse a pooled keep-alive socket for a later request. A loopback connect per request costs much less than the failure mode. - `mcp-bridge.ts`: each `tools/call` records one closed delivery diagnostic. This includes rejected calls (unsupported tool, invalid input, duplicate identity), which record the outcome `error` (`paperclip.opencode.tool-delivery-diagnostic/v1`) when its response closes. The fields are the call id SHA-256, the socket request index, the socket idle ms, the handler ms, the outcome (`result`, `error` or `unanswered`) and `written` (`writableFinished`). It holds no tool names, inputs or results. The limit is 256 records per bridge. - `opencode-proxy-finish-diagnostics.ts`: the proxy forwards these records to the run log as `warning` frames after session binding. It validates the exact key set and value types and drops anything else. - Tests: a new `mcp-bridge-delivery.test.ts` uses a pooling keep-alive client with the incident's call shape (one call, then two parallel calls). New forwarder cases cover the record validation. The existing finish diagnostic assertion now filters by its own schema. ## Verification - Red: with the `Connection: close` line removed, `mcp-bridge-delivery.test.ts` fails with `expected 'keep-alive' to be 'close'`. The pooling client then reuses the socket. - Green: `cd packages/paperclip-runner && pnpm exec vitest run src/drivers/opencode/mcp-bridge-delivery.test.ts src/drivers/opencode/mcp-bridge-diagnostics.test.ts src/drivers/opencode/mcp-bridge.test.ts src/cli/opencode-proxy-finish-diagnostics.test.ts src/cli/opencode-app-server-proxy.test.ts src/cli/opencode-proxy-abandoned-tool-call.test.ts` gives 6 files and 45 tests passed. - The new test asserts that all three results arrive with the correct JSON-RPC ids, that no socket is reused, and that each call records `socketRequestIndex: 1`, `outcome: "result"`, `written: true` without tool names or raw ids. A second test asserts that a client that leaves before its answer records `outcome: "unanswered"`, `written: false`. A third test asserts that a rejected call records `outcome: "error"`, `written: true`. - `tsc -p tsconfig.json --noEmit` for `packages/paperclip-runner` reports no errors in the changed files. Seven existing errors in `codex-runtime-adapter.ts` and `eval/workflow-harness.ts` come from this worktree's linked dependencies. Required CI typecheck is the source of truth. - This does not prove the production fix. A recurrence on CI will show socket reuse, idle time and `written` for the stranded call in the run log. ## Risks - Low risk. Each MCP request now opens a new loopback TCP connection. That adds well under 1 ms per call, and there is no TLS. - The `notifications/cancelled` handling and the abandoned-call abort do not change. The abort still reads `writableFinished` on response close. - One extra run-log warning for each tool call (at most 256 for each bridge). The record holds only a hash, counters and timings. - Previous-release story (upgrade-safe run state): no persisted run, session, launch or pack state changes. A bridge from the previous release keeps its behavior until the proxy restarts. - If the thesis is wrong, the five-minute stall can recur. The new records will then show `written: true` on a fresh socket and point the next investigation at OpenCode or Bun. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
8eabb13ebd |
fix(recovery): retry a system-woken run once when it lost its runner before any turn (#20873)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run on a remote sandbox can lose its Runner process before
its same-run retry reattaches. The run then fails with
`runner_remote_process_identity_unavailable`
> - After saved-work recovery, the server requests one Retry by itself,
but only for user wakes. System wakes (monitors, automatic assignments)
rest on the visible Retry
> - On staging, a monitor wake's run lost its Runner before it started
any provider turn. Nothing it did could be repeated, but the task stayed
blocked until a person clicked Retry
> - This pull request lets that one case continue automatically, once,
under the run's responsible user
> - The benefit is that a system-woken task whose run never reached a
provider turn continues without a person
## Linked Issues or Issue Description
**What happened?**
On staging, a monitor wake (`issue_monitor_due`) started a remote native
run. Its `session.snapshot` command timed out before any turn. The
same-run retry found the Runner process gone (`process_not_running`,
lifecycle `ready`) and the run failed with
`runner_remote_process_identity_unavailable`. The run had no turn event
and no tool event. Saved-work recovery recovered the files, but
`requestInterruptedNativeTurnRetry` refused with `not_user_request`, so
the task stayed blocked on a manual Retry.
**What did you expect to happen?**
When the run proves that it did no provider turn, the server requests
the Retry once by itself, as it does for user wakes.
**Steps to reproduce**
1. A system wake starts a remote native run.
2. The run fails before any turn event, and the same-run retry finds the
Runner gone.
3. Saved-work recovery completes. No Retry is requested; the task waits
for a person.
Refs #20872, #20853
## What Changed
The rule that changed, exactly: in `requestInterruptedNativeTurnRetry`
(`server/src/services/heartbeat/restart-interrupted-turn.ts`), a run
whose wake was not requested by a user used to stop with
`not_user_request` in all cases. Now it requests the task Retry only
when all of these are true:
- the run failed with one of the existing reattachment-loss codes
(unchanged gate);
- a positive durable record proves that no turn was ever issued: the
run's last `native.remote_runner_recovery_failed` event carries
`remoteRunnerRecovery` with `stage: process_verification`, `reason:
process_not_running` and `providerTurnIssued: false`;
- the run also recorded no turn event and no tool event;
- the run has a `responsibleUserId`; the Retry runs under that user;
- the run is not itself a `task_retry` run;
- the task is still open and assigned to the same agent (unchanged
gate).
An absence of run events is never enough: a turn can reach the provider
before its events are stored (Greptile, first review).
- New `withRemoteRunnerTurnProof` (`native-session-executor.ts`): when a
same-run reattachment is refused because the sandbox proved the Runner
process gone (exit 45, not an unreachable sandbox), the executor reads
the run's controller journal. The controller records each turn command
there before it sends it. Only when
`runnerdStateProvesNoRemoteProviderTurn` holds (just
`run.prepare`/`session.open` and maintenance commands such as
`session.snapshot`, and only session-open events) does it add
`providerTurnIssued: false` to the diagnostic. This is the same proof
that restart recovery already uses for `bootstrap_incomplete`.
`RemoteRunnerRecoveryDiagnostic` keeps the field only as `false`.
- Which staging shapes this covers: a run whose journal holds no turn
command. Whether c14bab19 (the `session.snapshot` timeout) qualifies
depends on its controller journal, which staging does not expose; if its
journal held a `turn.start`, it stays on the manual Retry floor.
32376b3a (`turn.submitted`, then a `turn.start` timeout) always stays on
the floor.
- Everything else about system wakes is unchanged: without the journal
proof, with any turn or tool event, or without a responsible user, a
system wake still rests on the visible Retry.
- Bounded: the Retry uses the existing idempotency key
`restart-interrupted-turn:<runId>`, so it is requested at most once per
run. Its successor is a user-attributed `task_retry` run, so if that run
also loses its Runner, the existing `automatic_retry_used` rule leaves
it on the visible Retry. No loop is possible.
- Attribution: `requestTaskRetry` records `issue.retry_requested` with
actor `restart-recovery` on behalf of the run's responsible user.
Admission keeps every gate (saved-work recovery, stop proof, approvals,
pause, budget, ownership).
- Tests in
`server/src/services/workspace-recovery-retry-copyback.test.ts`: a
positive case (monitor wake, Runner proven gone, journal proof),
including the second-loss `automatic_retry_used` check. Negative cases:
no run events but no journal proof (the Greptile shape), a settled turn,
a tool event, no responsible user.
- New
`server/src/services/native-runtime/remote-runner-turn-proof.test.ts`:
the proof is recorded only for a journal of session commands; not for a
`turn.start` command, a committed turn event, an unreadable journal, or
an unreachable sandbox (`probe_transport_failed`).
- `doc/execution-semantics.md`: describes the rule.
## Verification
- `cd server && pnpm exec vitest run
src/services/workspace-recovery-retry-copyback.test.ts`: 83 passed.
- Red on master (product hunk removed): the positive case fails with
`expected { requested: false, … } to match object { requested: true }`.
- Red on the first head (
|
||
|
DottaandClaude Fable 5.2
|
f9171ef621 |
fix(native): retry a throttled post-turn checkpoint for real; wait out storage throttling while saving the provider session (#20871)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - After a native turn in a remote sandbox, closing the session saves
the provider session (the harness backup) to object storage. The next
attempt resumes from it.
> - On staging, object storage throttled that save ("Please reduce your
request rate", S3 `SlowDown`). The close failed, was retried 3 times 1 s
apart, and the session was quarantined. The next attempt of the same run
was refused with `native_session_cleanup_quarantined`.
> - The retries could never succeed: the Runner transport memoizes
`close()`, so every retry returned the first rejected promise and never
ran the checkpoint again. Quarantine recovery and admission recovery
call the same `close()`.
> - Also, the backup upload did not wait out a throttle or outage longer
than one request's own retries.
> - This pull request makes a close that failed only in its checkpoint
run that checkpoint again on the next `close()`, and makes the backup
upload wait out storage unavailability inside the existing bounded
storage wait.
> - The benefit is that a transient storage throttle no longer ends in a
quarantine that refuses the next attempt of the same run.
## Linked Issues or Issue Description
Refs #20772 (made the backup upload resendable after `SlowDown`). Refs
#20165 (per-request throttle backoff). Refs #20833 (earlier steps of the
same failure chain).
**What happened?**
1. A staging Codex run on Daytona finished its turn. The post-turn close
ran the remote checkpoint.
2. The checkpoint failed: `remote checkpoint failed: Please reduce your
request rate.`
3. The Runner retried the close 3 times 1 s apart (`native session
cleanup recovery after close failure (1..3)`), then 3 quarantine
recoveries. All failed within about 5 s.
4. The next attempt of the same run was refused:
`native_session_cleanup_quarantined: prior session cleanup remains
incomplete`.
**Expected behavior**
A throttle slows the save down. A retry of the close saves the session
again. The next attempt of the same run proceeds once the save succeeds.
**Steps to reproduce**
1. Make a transport close whose checkpoint throws once.
2. Call `close()` again.
3. On master the second call returns the first failure without running
the checkpoint again.
4. Separately, answer each harness-backup `PUT` with 9 `503 SlowDown`
responses. On master `persistNativeHarnessBackup` throws `SlowDown:
Please reduce your request rate.`
**Paperclip version or commit**
master at
|