mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-10 12:37:55 +01:00
2a6229161d5aa4440f7a95baf16c4a6eda5ed034
8011
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandClaude Opus 5.5
|
2a6229161d |
fix(authz): do not link fast response denials to connection settings
Address review: the Connections page shows fast response controls only while the experiment is on, so a denied fast response route now gets the general board instruction instead of a link to a page that may lack them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
bc68e5c062 |
fix(authz): link connection settings and stop the page search
Two journeys showed agents searching for a web page after a board denial that named none: the general fallback told them to link the most specific page they knew. It now says to link a page only when its route is already known and not to search for one. Prompt injection prevention, decision model and fast response routes now link the connection settings page that holds those controls. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
9af18eba9d |
fix(authz): link the agent profile tab that holds a denied control
Address review: a denied POST /api/agents/{id}/keys linked the bare
profile, but API keys are only reachable at /api-keys. Map the agent
subresources with their own tab (keys, permissions, budgets, identity and
instructions, configuration) to that tab.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
41399e317a |
fix(agents): say who sets up a Claude Managed Agent profile
An agent asked to set up an Anthropic managed agent read the Paperclip Runner configuration doc, which named Claude Managed but not who creates its profile, and spent its reply window exploring hiring and adapter configuration. The doc and the profile-creation catalog entry now say that a board operator creates and qualifies the profile and that agents cannot. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
e83d9bb529 |
fix(authz): make board-only denials final and actionable for agents
Agents that hit a board-only route got a bare 403 "Board access required" and kept searching the API for another route until the chat attempt ran out. The shared assertBoard denial now tells agent actors the action is board-only by design, that no other route can do it, and names the board page where the person acts when the route maps to one (company settings, secrets, budgets, members, gateways, approvals, agent profile). The HTTP status and error text are unchanged; agents also get code board_access_required. Chat guidance treats a board-only operation as final instead of asking the agent to discover an operator path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
400e73d15c |
test(evals): keep backup-and-restore passable; note the wording gap in code
A declared coverage gap fails the catalog-complete-conditions check, so the case could never pass. Follow the catalog practice for free-form replies instead: grade structure, note in code that wording is not graded, and read a passing run's replies. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
e6f106b8f8 |
fix(api-catalog): describe the real backup and restore paths for agents
The database backup catalog entry had only a summary. An agent asked to back up an instance and check a restore got 403, found no other path and told the person to use an "Instance settings > Backups" page that does not exist. The entry now says who can call it, that the UI only sets backup retention (on /company/settings), that operators can run `paperclipai db:backup`, what the result file is, and that a restore check goes into a separate empty database with psql, never over the live one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
432dacb484 |
test(evals): grade backup-and-restore replies by structure, record the wording gap
Both reply checks were keyword/regex lists over prose. They passed an initial reply that invented an "Instance settings > Backups" page, and they fail on valid rewording. The replies are now graded by the runs and messages they belong to: the canonical request has its own completed reply, and the second run answers the operator's receipt in a saved, attributed reply. Reply truthfulness is recorded as a coverage gap instead of a regex. The saved-state checks for the archive and the separate-target restore are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
9926a13870 |
fix(tool-access): an unscoped tools:use grant does not turn on actions a connection left off
Every new standard agent gets an unscoped tools:use grant. The policy checked explicit grants before profiles, so that default grant allowed every action of every connection installed for the agent, including actions the person left off when they finished the connection. The agent's effective profile showed the action off while the test-as-agent summary and the gateway allowed it. When a profile in effect for the agent governs the tool's connection, only a grant scoped to the action, connection or application now allows it. An unscoped grant still applies to tools that no connection profile governs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
8384a85876 |
test(evals): grade test-as-agent on the product's access summary, not reply keywords
The reply check was a keyword/regex list over prose. It passed a reply that said Morgan's destructive write was allowed while the saved scope was read only, and it fails on valid rewording. The reply check now keeps only structure (one saved reply after the test, no credential disclosure). The observer reads the same test-as-agent access summary the agent reads and requires it to report the read action allowed and the write action off, so a misleading product summary fails the case. A prose claim that contradicts the scope is not graded; that is recorded as a coverage gap rather than a regex. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
54d2b96c2b |
fix(api-catalog): name the Connections page for company model service settings
Fast response, Decision model and Prompt Injection Prevention are set in sections of /company/settings/connections, but their API catalog entries named no page. An agent that could not change Fast response searched the catalog for the settings page, found none and linked /company/settings, which has no Fast response control. The catalog descriptions now name the section, the company-prefixed Connections page and, for Fast response, the instance experiment that hides the section. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
b7c7ac5a33 |
test(evals): grade fast-responses by following the reply's own link to the control
The scripted human typed /company/settings/connections instead of clicking the reply's link, and the reply checks accepted any "settings" mention, so a reply that linked /company/settings (no Fast response section) passed. The person now clicks each rendered reply link and must land on a page that shows the Fast response region; with no such link the act stops before configuring. The reply keyword checks for this case are removed. Other services in the module keep their existing checks and oracle version. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
59076e62c4 |
test(evals): grade hire-denied by the saved state and the rendered hiring route
The setup probe accepted any 403, but an agent key creating a process agent is also rejected by the adapter-config gate, so the fixture passed while the CEO title still allowed hiring through agent-hires. The probe now requires the agents:create denial text. The grader no longer reads the reply's wording: it requires no new agent (any name), no hire or delegation record, and a rendered link to the hire form or the speaking agent's permissions page, which verifyResult clicks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Dotta
|
ec9c732c0b |
Merge remote-tracking branch 'origin/master' into chat-eval/master-candidate
* origin/master: (74 commits) test(chaos): retry a blocked M1 subtask and check the parent's stall wake (#20147) test(runner-e2e): hide the eval harness from local native agents (#20044) fix(agents): honor an explicit hiring denial on CEO agents (#20041) fix(native): roll back an undelivered warm attach rotation once its runner is proven dead (#20146) fix(native): retire a finished per-turn owner past its own quarantined interrupted bootstrap (#20140) fix(github): send gh repo create and fork to the connection's tools (#20142) fix(github): default repository owner to the only installed organization (#20141) fix(runner): an unknown agent id says which id and how to find the right one (#20135) feat(issues): wake the parent once when a child or blocker stalls on the board (#20098) docs(plans): record the measured seed experiment for per-turn uploads (#20139) fix(native): prove a drained remote start had no turn when its controller only read the goal (#20137) fix(native): verify a staged attachment restart on the claimed sandbox lease (#20136) fix(issues): a Retry held by a blocked subtask names it instead of failing silently (#20109) fix(recovery): settle a dead controller's result lease for a run a pause cancelled (#20113) test(chaos): add R1, a code change in a fresh clone of a real repository (#20111) test(chaos): separate workspace provisioning from agent boot; grade outage-created runs as restart (#20119) fix(runner): keep a restart continuation when a stale earlier continuation is reported lost (#20118) fix(runner): resume an ACPX turn interrupted by a deploy instead of refusing the launch digest (#20112) fix(runner): a finish or block waits for held calls before it is judged (#20108) fix(runner): reopen the ACPX session at turn.start when it can be safely reopened (#20110) ... # Conflicts: # packages/paperclip-runner/src/cli/opencode-app-server-proxy.ts # server/src/__tests__/agent-conversations.test.ts # server/src/routes/agents.ts # server/src/services/agent-conversations.ts # tests/runner-e2e/live-fixtures.ts |
||
|
DottaandClaude Opus 5.5
|
0502cc9fac |
test(chaos): retry a blocked M1 subtask and check the parent's stall wake (#20147)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness grades the M1 scenario: a parent task delegates two subtasks across restarts > - When a subtask's run failed recovery, the harness only clicked Retry on the parent. The parent's Retry is correctly held behind the blocked subtask (#20109), so the task waited until its deadline > - A real board user would also click the subtask's Retry. #20098 also added a wake for a parent whose subtask stalls, and the grade did not check it > - This pull request clicks the board Retry on a blocked subtask (graded as a needed Retry) and checks that the parent got exactly one stall wake per stall > - The benefit is that M1 grades the full user path and proves the stall wake from the product's own activity record ## Linked Issues or Issue Description Refs #20098, #20109, #20094 **What happened?** In wave `w-20261109T003126-s9701`, the M1 parent t05 rested blocked behind a blocked codex subtask. The harness clicked Retry on the parent. The product held that Retry ("waiting on a blocked task") as designed. Nothing retried the subtask, and the task waited 38 minutes until the wave deadline. The wave also could not show whether the parent got the new `issue_blocker_stalled` wake. **Expected behavior** The harness acts like the board: it clicks Retry on a blocked subtask that offers one, and it counts that as a needed Retry. The M1 oracle checks the parent's activity for one `issue.blocker_stalled_wake` per stalled subtask. ## What Changed - `observer.ts`: - For M1, the observer checks the parent's subtasks every 10 s while the parent is open, independent of the parent's rest window. - Each blocked subtask is recorded as a stall: the subtask plus its latest recovery action, from `GET /api/issues/:id/recovery-actions`. - When a blocked subtask's interruption offers Retry, the harness posts the board Retry. It clicks once per subtask; a click that was not accepted can be tried again on a later check. - The parent's `issue.blocker_stalled_wake` activity entries are collected as `stallWakes` through `GET /api/issues/:id/activity`. - `grader.ts`: R3 fails with `retry_needed_on_subtask_<n> (<ids>)` when a subtask needed a board Retry, unless the parent's own Retry already failed R3. - `workloads.ts`: the new M1 check `parent_woken_once_per_stall` is added when a subtask stalled. Each (subtask, recovery action) stall must have exactly one matching wake. A stall with an unknown action matches any wake for that subtask. ## Verification - `npx vitest run` in `tests/restart-chaos`: 203 passed. New tests cover: - an observer test: a blocked subtask gets one Retry click, and the parent is still watched; - the R3 subtask-retry grade; - the stall-wake check: woken once, not woken, woken twice, two stalls of one subtask with only one woken, and no stall. - G1, which also has subtasks, is unaffected: the existing governance tests pass. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: clean. ## Risks - The harness now acts on subtasks, which it did not before. It clicks once per subtask and only when the product offers Retry. - The stall-wake check reads activity through the board API. If the API omits these entries for a board user, the check fails visibly instead of passing. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
218eb4ec8b |
test(runner-e2e): hide the eval harness from local native agents (#20044)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Runner E2E evals run real agents against a real Paperclip server, then grade the result > - A local cell runs the product from the same checkout as the eval, as the same OS user > - In run 42472276601 an OpenCode agent read the eval's own scenario YAML and grader from that checkout > - An agent that can read the oracle can steer toward it instead of the user's request, so the eval stops measuring the product > - This pull request hides the eval harness from every local native agent process and removes eval names from fixtures > - The benefit is that a passing chat eval again shows product behavior, not oracle access ## Linked Issues or Issue Description Refs #19614 (the chat catalog eval harness that these graders belong to). **What happened** In run 42472276601, cell `chat-features.runner-opencode.local.access.cli-authorization` (OpenCode, DeepSeek), the agent printed its environment and saw the checkout path in `PATH` and `PAPERCLIP_RUNTIME_NOTIFICATION_HELPER`. OpenCode's `read` and `grep` tools were denied by its `external_directory` policy. The agent then used its `bash` tool with absolute paths, which that policy does not gate. It listed `/home/runner/_work/paperclip/paperclip`, read `docs/cli/*.md` and `cli/src/commands/client/*.ts`, ran `grep -rn` over `tests/runner-e2e`, and found the query in `chat-full-catalog-testops-manifest.json`. It then read `tests/runner-e2e/chat-scenarios/access.yaml`, the manifest entry, and the case grader `chat-catalog-domains/identity-cli-authorization.ts`. Its reasoning said: "This is the exact test case!" Separately, fixture company names contained the case id, for example `Runner E2E chat-features.runner-opencode.local.access.private-resource …`, and an agent noticed that it was in a test. **Expected behavior** The agent under test cannot read the eval's scenarios, fixtures, graders or captured evidence. It keeps everything that it needs: its workspace, its home, the product skills, the product packages and temporary files. The names it can see do not contain case ids, "E2E" or "test". **Steps to reproduce** 1. Dispatch `runner-full-stack-e2e.yml` for any `chat-features.runner-opencode.local.*` cell on master. 2. In the agent run, use bash to execute `grep -rn expected_outcome /home/runner/_work/paperclip/paperclip/tests/runner-e2e/chat-scenarios`. 3. The command returns the scenario expectations. With this change, the command returns `Permission denied` and exit code 2. ## What Changed - `tests/runner-e2e/agent-filesystem-sandbox.py` (new): applies a Linux Landlock domain that grants filesystem access everywhere except the hidden paths, sets `no_new_privs`, checks in-process that the hidden paths are denied, then `execv`s the command with the same PID, argv and environment. - `tests/runner-e2e/agent-filesystem-sandbox.ts` (new): selects the hidden paths (`tests/`, `.manual-private-trusted/`, the private evidence directory, `RUNNER_TEMP`). It never hides the agent's temporary root, the checkout root or the shared temporary directory. It runs a setup probe, logs the result and fails closed in CI. Then it wraps each `paperclip-runnerd` spawn of the E2E server. - Follow-up: run 42445668869 showed that the hidden list missed two places. A fixture worker listed Playwright's compiled copy of the harness and graders with source maps (`/tmp/playwright-transform-cache-1001/`). An agent in another run listed the attempt root's `artifacts-private/`, `control/` and `selected-private-inputs/` under `/tmp`. A sandboxed attempt now creates its root in `/var/tmp`. The reason: Landlock hides a path by leaving its ancestors ungranted, and agents must still create files in `/tmp`. The launcher sets `PWTEST_CACHE_DIR` to a hidden child of that root and removes the default cache. The hidden list covers the four private children of the root, including children that are created after runnerd starts. The probe also checks that the default Playwright cache is not listable. - `tests/runner-e2e/launch.ts`: passes the selection to the server process. The selection applies only to Linux local native cells. It does not apply to Codex cells (Codex's bubblewrap sandbox cannot mount under Landlock), installed-release launches, legacy adapters or Daytona. - `tests/runner-e2e/server-entry.ts`: installs the sandbox before the product starts. The product server itself stays unrestricted because it loads this entry from the harness. - `tests/runner-e2e/fixture-names.ts` (new), `catalog.ts`, `live-fixtures.ts`: fixture company, agent, environment, secret, AI connection and project names and descriptions use a neutral word and an 8-character digest suffix. They do not contain case ids or "Runner E2E". No grader or harness lookup reads these names. - `tests/runner-e2e/agent-filesystem-sandbox.test.ts` (new): tests the selection, the launch rewrite and the server-entry order. On Linux with Landlock, it also tests that a sandboxed `grep` over a fake `tests/` dir is denied while the product file stays readable. - `tests/runner-e2e/SECURITY.md`: documents the boundary and its limits. - The trusted files `manual-private-handoff.ts`, `manual-private-retention.ts` and `manual-private-report-assets.ts` are byte-identical to master. ## Verification - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/agent-filesystem-sandbox.test.ts tests/runner-e2e/live-fixtures.test.ts tests/runner-e2e/catalog.test.ts tests/runner-e2e/codex-ci-sandbox.test.ts tests/runner-e2e/server-entry.test.ts`: 151 passed, 1 skipped. The Landlock test is skipped on macOS. `npx tsc --noEmit -p tests/runner-e2e` shows no errors in the changed files. - Paid run [42480314919](https://github.com/paperclipai/paperclip/actions/runs/42480314919) for `chat-features.runner-opencode.local.agents.change-manager` ran on branch `chat-eval/eval-isolation-verify`, which is this commit cherry-picked onto the chat catalog candidate. The cell passed. Its `server.log` records: - `probe … "harnessFile":"denied","harnessDirectory":"denied","productFile":"readable","temporaryWrite":"allowed"`, and `denied` for each of `…/paperclip/tests`, `…/.manual-private-trusted`, `…/_temp/manual-private-evidence-*/capture`, `…/_temp/manual-private-evidence-*` and `…/_temp`. `"failures":[]`. - `runnerd launched inside the sandbox (pid 3548)`. - The fixture company was `Company cc369a2e`. No captured snapshot contains "Runner E2E". - The probe is a regression check on every local native cell: a missing or ineffective sandbox stops the attempt in CI before an agent starts. ## Risks - Landlock does not hide that a path exists. `stat` of `tests/runner-e2e` still succeeds (the probe logs `harnessDirectoryStat: visible`). Reading and listing are denied. - Directories above a hidden path (for example `/`, `/home/runner` and the checkout root) cannot be listed. New top-level entries in those directories are not accessible after launch. Agent state, `HOME` and temporary files are in `/tmp`, which stays fully granted. - Not covered: Codex cells, legacy adapters, installed-release launches and macOS. An agent that can reach the Docker daemon can still mount the checkout into a container. `ps` still shows the launcher's command line, which includes the case id. The temporary root (`paperclip-runner-e2e-*`) and instance id (`runner-e2e-*`) in the agent's paths still say "e2e". The chat catalog candidate depends on those prefixes, so this PR does not rename them. - Python 3 must be present on Linux runners. Ubuntu runners have it, and the harness already uses it. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), agentic coding with tool use (shell, file edits, GitHub CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
c0fc7608ee |
fix(agents): honor an explicit hiring denial on CEO agents (#20041)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The board controls which agents can hire, with the "Can create
agents" permission on each agent
> - A CEO agent with that permission turned off could still hire,
because the CEO role allowed hiring before the server read the saved
permission
> - In chat, a CEO agent asked to hire then created the agent through
the `hire_agent` tool, against the board's policy
> - This pull request makes the saved permission decide, and uses the
CEO role only for old rows that never saved a value
> - It also tells a denied agent that the denial is final and where a
person can act
> - The benefit is that the permissions page and the server now agree,
and the board can trust a hiring denial
## Linked Issues or Issue Description
**What happened**
A board user turned off "Can create agents" for a CEO agent. Then the
user asked the CEO in chat to hire a QA engineer. The CEO called
`hire_agent` (`POST /api/companies/{companyId}/agent-hires`). The server
returned 201 and created the agent. The legacy creator rule allowed any
agent with `role: "ceo"` before it read `permissions.canCreateAgents`.
**Expected behavior**
An explicit `canCreateAgents: false` denies hiring for every role,
including CEO. The agent gets a 403 that says the denial is final, and
it sends the person to the hire form or to its permissions page.
**Steps to reproduce**
1. Create a company with a CEO agent.
2. On the CEO's permissions page, turn off "Can create agents".
3. In chat with the CEO, send "Hire a QA engineer named Morgan."
4. See that a new agent named Morgan exists.
Found by the chat eval case `agents.hire-denied`. Related: #19614 (chat
eval harness).
## What Changed
- `canCreateAgentsLegacy` (authorization) uses the role-aware permission
normalization. A CEO keeps hiring unless the board recorded a hiring
denial.
- New `ceoHiringDeniedBy` record in `permissions` (`{ actorType,
actorId, at }`). `PATCH /agents/:id/permissions` writes it when an
authorized actor saves a CEO's hiring as off (an earlier record is
kept). The page shows the effective value, so a saved off is the board's
choice. Turning the toggle off also removes an explicit `agents:create`
grant. A saved `canCreateAgents: false` restricts a CEO only together
with this record.
- Reads report the effective value. A CEO row with no value, or with an
unrecorded false, shows hiring on. A CEO with a recorded denial shows it
off.
- Create and hire inputs cannot supply the record. Configuration updates
keep an existing record unless they turn hiring on. Company export keeps
a CEO's recorded denial, and import restores it, stamped by the
importing user.
- `PATCH /agents/:id/permissions`: an agent actor needs `agents:create`
to change `canCreateAgents`. A denied CEO cannot restore hiring for
itself or grant it to another agent.
- `assertCanCreateAgentsForCompany` adds a final, actionable sentence to
an agent's `agents:create` denial on hiring routes only (`POST /agents`,
`POST /agent-hires`, the hiring toggle on `PATCH /permissions`).
- `POST /api/companies/:companyId/imports/apply` (CEO-safe import): an
agent import that would create agents needs an `agents:create` decision
before any write. Before, any CEO agent could create agents this way.
Import also removes an existing `agents:create` grant when it restores a
CEO's denial.
- Chat guidance tells the agent to link `/<prefix>/agents/new` and its
own `/permissions` page when it cannot hire.
- Regression tests: authorization (a recorded denial is honored; an
unrecorded false keeps CEO hiring), normalization, route tests (denial
text, self-escalation, the record is written only when a CEO is turned
off), service tests for the permission and configuration write paths,
chat guidance text.
### Why a recorded denial and not a plain false
These normal paths save `canCreateAgents: false` on a CEO without a
board choice:
- The permissions page posts the full object. Before this change, a
legacy CEO row with `{}` was read as false, so saving any other toggle
stored false.
- A company import that updates an existing agent normalizes the
manifest permissions in the "stored" context. A manifest without the
field stores false. Export drops false values, so this is common.
- Low-trust creation stores false. If the trust preset later changes to
standard, the false remains.
These paths store true or nothing for a CEO: onboarding seed
(`requestHire`, create context: true), teams-catalog and import creation
(create context: true; the bundled catalog sets true only for a CTO),
built-in agents (no CEO roles), migrations and seed data (no writes to
this field).
A plain-false rule would remove hiring from those CEOs. With the record,
existing data keeps today's behavior, and no data migration is
necessary.
## Verification
- `cd server && npx vitest run
src/__tests__/agent-permissions-service.test.ts
src/__tests__/authorization-service.test.ts
src/__tests__/agent-permissions-routes.test.ts
src/__tests__/agent-conversations.test.ts` passes (244 tests).
- `teams-catalog-service`, `company-portability` and `built-in-agents`
tests pass (189 tests).
- Red check: the new authorization test fails without the
`authorization.ts` change.
- `npx tsc --noEmit -p server` passes.
- End to end: the chat eval `agents.hire-denied` on the OpenCode runner.
The baseline run (Actions run 42478870236) created Morgan through
`hire_agent`. With this fix (Actions run 42480936881), the agent got the
403, created no agent, and linked the hire form and its permissions
page. The eval clicked the rendered hire-form link.
## Risks
- Behavior change: a CEO whose hiring the board turns off after this
change can no longer hire.
- A CEO that already has a saved false keeps hiring until a board user
turns it off again on the permissions page. The page now shows such a
CEO as on, so the board can see and change it.
- A request that creates a CEO with `canCreateAgents: false` does not
restrict it. The board must turn hiring off on the permissions page.
- A CEO agent whose hiring is denied can no longer import a package that
creates agents. A board user can still import it.
- Other CEO special cases are not hiring routes: `issues.ts` lets a CEO
link approvals to issues, and `agents.ts` gives a CEO task assignment.
`teams-catalog.ts` and `llms.ts` read the normalized permissions, so
they follow the same effective value.
- The denial text is longer. No existing test compares this route's
exact error text.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
c61f0bbf11 |
test(runner-e2e): keep promisify(execFile) shape when wrapping runnerd launches
Replacing execFile dropped its util.promisify.custom hook, so promisified
calls resolved a string instead of { stdout, stderr } for every command.
Carry the hook over, wrapped the same way, and test a real promisify call.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
0740a07ffb |
test(runner-e2e): hide a worktree's Git store and sandbox every runnerd launch API
- A linked worktree's .git file points into the main repository; hide the worktree gitdir and the shared common directory too. - Wrap spawn, spawnSync, execFile and execFileSync, match runnerd by name whether absolute or PATH-resolved, and refuse a shell launch rather than start runnerd outside the sandbox. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
b54be5c7a9 |
test(runner-e2e): hide the checkout's .git, fail closed on unhidable paths, salt fixture names
Greptile and review findings on #20044: - git show HEAD:tests/... read the graders through the checkout's .git; hide it. - Harness material under the shared temp dir or around agent state was dropped silently; a required (CI) sandbox now refuses to start. - Unsalted name digests were reversible from public catalog ids. - Unit-test the runnerd spawn wrapper directly. - Document that the boundary is filesystem only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
a0b9895bbd |
fix(native): roll back an undelivered warm attach rotation once its runner is proven dead (#20146)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A follow-up turn can attach to the previous turn's warm local runner. The transport first rotates authority: it archives the previous epoch (suspended runner state and settled controller) under `authority-epochs/` and writes a new control plane that holds only the `run.attach` command > - A hard kill in that window, before the runner connects, leaves the root with the new control plane and no `runner/runner-state.json`. Restart recovery proves the warm runner dead and classifies the run `resume_dead_runner` > - The executor saw a root with the run's own identity and no runner state, called its authority indeterminate, and quarantined it. The run failed with `runner_state_identity_mismatch: authority_indeterminate` and needed a Retry > - The attach was never delivered, and the runner that would have received it is proven dead, so it cannot have taken effect. The root's real authority is still the archived previous epoch > - This pull request rolls back such an undelivered rotation, so the follow-up continues from the previous turn > - The benefit is that a restart in the first second of a warm follow-up no longer needs a Retry ## Linked Issues or Issue Description Refs #20140 **What happened?** Local restart-chaos K13 regression storm w-20261108T235957-s9401 (mastercanary/v2026.1109.0-canary.2 |
||
|
DottaandClaude Opus 5.5
|
bd97a3e6f7 |
fix(native): retire a finished per-turn owner past its own quarantined interrupted bootstrap (#20140)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A follow-up turn on a per-turn session must retire the previous turn's runner state before it starts a new runner. Retirement proves the previous owner finished, and it also proves that every quarantined copy of that session root is settled > - If restart recovery interrupts a run's own start (for example a hard kill during `session.open`), recovery proves the half-started bootstrap held no provider work, quarantines that root, and bootstraps the same run again. The quarantined copy keeps the run's identity > - After that run finished, retirement checked the old bootstrap copy as if it were a finished owner. An unfinished `session.open` (or an opened session without a turn) is never "settled", so the follow-up failed with `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate` and needed a Retry > - This pull request skips a quarantined copy of the same owner when the same bootstrap proof that let recovery quarantine it still holds > - The benefit is that a follow-up after a restart-interrupted start continues on its own ## Linked Issues or Issue Description Refs #20137 **What happened?** Local restart-chaos K13 regression storm w-20261108T235957-s9401 (master |
||
|
DottaandClaude Opus 5.5
|
ad8daeadbc |
test(runner-e2e): hide the attempt's private temp dirs and Playwright's compiled graders
Playwright cached the compiled harness and graders, with source maps, in /tmp/playwright-transform-cache-<uid>, and the attempt root under /tmp held artifacts-private, control and selected-private-inputs. The sandbox skipped everything under /tmp, because hiding a child there would stop agents creating files in /tmp. A sandboxed attempt now creates its root in /var/tmp. The launcher points PWTEST_CACHE_DIR at a hidden child of that root and removes the default cache. The hidden list covers the four private children, including ones created after runnerd starts, and the setup probe also checks that the default Playwright cache is not listable. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Devin FoleyandPaperclip
|
546cfee919 |
fix(github): send gh repo create and fork to the connection's tools (#20142)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents with a managed GitHub connection get `git` and `gh` launchers that use the connection's credential > - The launcher gives that credential to any `gh` command, and the broker does not see which command runs > - So `gh repo create` and `gh repo fork` skip the connection's tool switches, ask-first approvals, owner defaults, and audit > - The connection's `create_repository` and `fork_repository` tools apply all of these > - This pull request stops `gh repo create`, `gh repo new`, and `gh repo fork` when a managed credential is in use, and names the tool to use instead > - The benefit is that repository creation follows the connection's rules, whichever path the agent tries first ## Linked Issues or Issue Description Refs #20141 (default owner for `create_repository` and `fork_repository`). **What happened?** An agent with a managed GitHub connection can run `gh repo create` in its shell. The command uses the connection's token, but it does not use the connection's tool rules. If the user turns off or reviews `create_repository` on the connection, the agent can still create a repository with `gh`. The owner default from #20141 also does not apply, so the command can fail with `403 Resource not accessible by integration` when the App is installed only on an organization. **Expected behavior** With a managed connection, repository creation and forks go through the connection's tools. The tool switch, approvals, owner default, and audit apply. **Steps to reproduce** 1. Connect GitHub with a managed connection. 2. Set the connection's `create_repository` tool to off or ask first. 3. In an agent run, run `gh repo create demo --private`. 4. The command runs with the connection's credential and ignores the tool setting. ## What Changed - `packages/adapter-utils/src/github-launcher.ts`: when the broker returns a managed credential, `gh repo create`, `gh repo new`, and `gh repo fork` exit with code 1 before `gh` starts. The message names the `create_repository` or `fork_repository` tool and tells the agent to finish with Git. - `--help` still runs. Other `gh` commands do not change. Executions without a managed credential run these commands as before. - `doc/execution-github-identity.md`: describes the behavior and states that this is guidance, not a security boundary. ## Verification - New test in `packages/adapter-utils/src/github-launcher.test.ts`: "sends repository creation and forks to the connection's tools when a connection is in use". It covers `create`, the `new` alias, `fork`, `--help`, another `gh repo` command, and the case without a connection. It fails without the launcher change. - `npx vitest run src/execution-target.test.ts src/github-launcher-environment.test.ts src/github-launcher.test.ts` in `packages/adapter-utils`: 83 passed. - `npx vitest run src/drivers/acpx/github-launcher.test.ts` in `packages/paperclip-runner`: 3 passed. - `npx vitest run src/services/heartbeat-github-launchers.test.ts` in `server`: 12 passed. - `npx tsc --noEmit -p packages/adapter-utils` passes. ## Risks - Agents that used `gh repo create --source . --push` must now call `create_repository`, then add the remote and push with Git. The message tells them to do this. - If the connection's tool is off for the agent, the agent cannot create a repository. This is the purpose of the tool setting. - This is not a security boundary. Other commands that receive the token, such as `gh api`, can still call the same GitHub endpoints. - Running launchers keep their earlier source until the next run stages new launchers. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (code reading, shell commands, and test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
0e49576fd9 |
fix(github): default repository owner to the only installed organization (#20141)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents use a managed GitHub connection to create repositories and to fork repositories > - The Paperclip GitHub App can be installed only on an organization, and not on the user's personal account > - Then `create_repository` and `fork_repository` fail when `organization` is empty, because GitHub sends them to the personal account > - Agents, approval cards, and connection tests all fail on their first attempt with `403 Resource not accessible by integration` > - This pull request fills an empty `organization` with the only organization where the App is installed > - The benefit is that repository creation works on the first attempt for these connections ## Linked Issues or Issue Description No public issue exists. Refs #19655 (GitHub repair guidance for the same error). **What happened?** A user connected GitHub through the managed connection. The Paperclip GitHub App was installed on one organization and not on the user's personal account. An agent called `create_repository` without `organization`. GitHub returned `POST https://api.github.com/user/repos: 403 Resource not accessible by integration`. The test button on the connection page gave the same error. The agent then stopped and asked the user to create the repository manually. **Expected behavior** When the App can create repositories only in one organization, an empty `organization` uses that organization. The approval card and the test result show that owner. **Steps to reproduce** 1. Install the Paperclip GitHub App on one organization only. Do not install it on the personal account. 2. Connect GitHub with the managed connection. 3. Run `create_repository` with only `name`, from an agent or from the connection test dialog. 4. GitHub returns 403. ## What Changed - Added `server/src/services/tool-access/github-default-owner.ts`. It finds the default organization from the grant's access snapshot. It returns an organization only when there is exactly one installation, and the snapshot marks that owner's repositories as organization repositories. Older snapshots without account types get no default until access is refreshed. - `governedToolArguments` in `tool-gateway.ts` fills an empty `organization` for `create_repository` and `fork_repository` on managed GitHub connections. The change occurs before policy, approval signing, previews, and audit. Thus agent calls, approval cards, and connection tests use the same owner. - The grant lookup runs only for these two tools, and only when `organization` is empty. If the credential is unavailable, the call does not change and fails with its own error at dispatch. - An explicit owner does not change. A fork into its own owner does not change. Token (PAT) connections, personal installations, and multiple organizations keep GitHub's behavior. - `projectedConnectionToolInputSchema` describes the default on the `organization` argument. Agents and the connection test form see where the repository goes. - Updated `doc/connections/GITHUB.md`. ## Verification - `pnpm --filter @paperclipai/server typecheck` passes. - `npx vitest run src/services/tool-access/github-default-owner.test.ts`: new unit tests for the owner rules (single organization, personal installation, several organizations, another person's account, older snapshots, blank values, forks into the same owner, schema text). - New gateway test in `src/__tests__/tool-gateway.test.ts`: "defaults GitHub repository creation to the only installed organization for agents and connection tests". It covers an agent call, an explicit owner, a connection test call, and a personal installation. It fails without the gateway change. - Related suites pass, 393 tests in 9 files: `tool-gateway.test.ts`, `tool-gateway-service.test.ts`, `connection-instructions.test.ts`, `github-access-errors.test.ts`, `github-default-owner.test.ts`, `tool-gateway-discovery-http.test.ts`, `github-operation-credentials.test.ts`, `project-git-authorization.test.ts`, `prompt-injection-gateway.test.ts`. - Not tested against a live GitHub App installation. ## Risks - Low risk. The default applies only when the grant snapshot shows one organization installation and no personal installation. Without the change, these calls fail with 403. - A stale access snapshot can choose the organization after a user installs the App on the personal account. Installation events and **Refresh access** update the snapshot. - An approval that was created before this change, without `organization`, no longer matches the projected arguments. That approval cannot run and must be requested again. Without the change, it fails with 403. - `gh repo create` and `gh repo fork` in agent shells do not use the gateway, so they keep GitHub's personal default. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (code reading, shell commands, and test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
8e3b64a257 |
fix(runner): an unknown agent id says which id and how to find the right one (#20135)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents delegate by creating subtasks through Paperclip's tools. A tool error must tell the agent how to recover, or it improvises > - Restart chaos t06 ended a delegation with four children instead of two > - Evidence showed no replay: the model built an assignee id from two real agents' halves and got a bare `paperclip_runner_agent_not_found` four times > - To debug the error it created "diagnostic test" subtasks without an assignee. Those succeeded and were left behind as duplicates > - This pull request makes the error say which id was not found, which real agents it most resembles, and to call `list_agents` > - The benefit is that an agent corrects the id instead of creating throwaway tasks ## Linked Issues or Issue Description Refs #19831 (durable ownership plan). Restart chaos wave `w-20261108T225520-s9501`, task t06 (claude, M1, K2 drain). **Problem:** parent run 2751fe3c created Part 1 correctly. For Part 2 it passed `assigneeActorId: 0cb62ac2-742d-442c-acc9-d21dd936e716`: Chaos claude's prefix with Chaos codex's suffix. - Four `create_task` calls (keys `part2`, `part2` again, `-v2`, `-backlog`) each returned only `paperclip_runner_agent_not_found`. One of them crossed the K2 drain and failed the same way as the one before it. - The model then created "Part 2 06f119cf test" (`-noassignee`) and "Part 2 06f119cf" (`-title-test`), both described as "diagnostic test - will be removed/ignored". With no assignee they defaulted to the caller. They stayed in backlog. - Only after `list_agents` did it create the real Part 2. - Same-key replays were never the issue: `create_task` stores the `idempotencyKey` as the child's `origin_id`. ## What Changed - `server/src/services/native-runtime/paperclip-runner-tool-authority.ts`: - `create_task` (assignee) and `get_agent` now fail with `paperclip_runner_agent_not_found: no agent in this company has id "<id>". Closest ids: <name> (<id>), …. Call list_agents and copy a valid id exactly; nothing was created or changed.` - Suggestions are up to three agents of the caller's company whose ids share at least eight leading or trailing characters with the missing id. Another company's agents are never named. - The recoverable-read check matches the error code before the `:` details, so a recovered `get_agent` read still returns the error as a tool result. ## Verification Red/green: - `paperclip-runner-tool-authority.test.ts` "names a missing agent id and its closest real agent instead of a bare code (restart chaos t06)". It uses t06's spliced id, with a second agent "Chaos claude" in the company. - On master: the error is the bare code (red). - With this PR: both `create_task` and `get_agent` name the id, `Chaos claude (0cb62ac2-…)` and `list_agents`, and no task is created. - The file passes (42 tests), including the existing foreign-company assignee rejection. `tsc --noEmit -p server` is clean. ## Risks - One extra read of the company's agents, only on the failure path. - The message names agents of the caller's own company, which `list_agents` already returns to it. - No schema or persisted state change. ## Model Used Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code, with tool use (shell, file editing). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above (#20069, #20108) - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes (no user docs affected) - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green (pending) - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups (pending) - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1109.0-canary.1 |
||
|
DottaandClaude Opus 5.5
|
c49ab6b9ee |
feat(issues): wake the parent once when a child or blocker stalls on the board (#20098)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A parent task delegates to children and waits on blockers. Its agent learns of progress only through wakes > - Today only completion (`issue_children_completed`) and resolution (`issue_blockers_resolved`) wake a waiting issue > - In restart chaos `w-20261108T181313-s9701` t05, a blocker stopped on a recovery action that needs the board, and its parent sat silently blocked for 50+ minutes > - This pull request wakes the waiting agent once per such stall, tells it what it can do, and shows the stall on the parent's page with the child's Retry > - The benefit is that a stuck child or blocker no longer silently strands its parent ## Linked Issues or Issue Description Refs #19831 (durable ownership plan; restart chaos campaign). Restart chaos wave `w-20261108T181313-s9701`, task t05 (M1, claude). **Problem** From the kept database: - CHA-8 `blocks` CHA-5. - CHA-8's board-owned recovery action (`native_session_retry_exhausted`) was created at 18:44:02.909 and resolved with outcome `blocked` at 18:44:08, when CHA-8 moved to `blocked` for the board. - CHA-5's only wake in the whole wave was its `issue_assigned` at 18:20:48. Its agent never heard about CHA-8. **Expected behavior** - When a child or blocker stalls on the board, the waiting agent is woken once. It can then re-delegate, escalate, or block with a reason. - The board sees the stall on the waiting task, with the child's Retry. ## What Changed - **Shared:** - `ISSUE_BLOCKER_STALLED_WAKE_REASON` (`issue_blocker_stalled`), `ISSUE_BLOCKER_STALL_RELATIONS`, and the types `IssueBlockerStallRelation` and `IssueBlockerStalledWakeContext`; - `Issue.stalledDependencies`. - **Server, `services/recovery/blocker-stall-wakeups.ts`:** - A stall is an issue that is not done or cancelled, whose latest recovery action needs the board. That means either open (`active`/`escalated`) and board-owned or escalated, or board-owned and resolved as `blocked` while the issue is still `blocked`. The second case is what t05 needed: recovery resolves its action that way when it hands the issue to the board. - Waiters are the parent, and every issue the stalled one `blocks`, when they are open, visible and agent-assigned. - Each waiter gets one wake per stall. The idempotency key `issue_blocker_stalled:{waiter}:{stalled}:{recoveryActionId}` is checked through the indexed `(company_id, payload->>'issueId')` path. A later stall has a new action id, so it wakes again. - Each wake writes activity `issue.blocker_stalled_wake`. - The pass runs from the recovery-action sweep (`reconcileActiveRecoveryActions`) and from the per-issue blocked sweep (`reconcileResolvedDependencyWakeBackstop`, which visits each blocked issue). No new worker. - A waiter's agent that cannot read the stalled issue (`issue:read`, which honors private visibility) is woken without its title, identifier or recovery details; the activity entry withholds them too. - **Database:** migration `0450` adds the partial unique index `agent_wakeup_requests_issue_blocker_stalled_uq` on `(company_id, idempotency_key)` for `issue_blocker_stalled:` keys in delivered or pending statuses, so two concurrent sweeps cannot both deliver a stall. It is built `CONCURRENTLY IF NOT EXISTS` outside a transaction (listed in `MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS`). - **Queue:** `allowsDependencyBlockedWake` (in `run-dispatch/domain/wake-context.ts`) lets a stall wake run while the waiter's blockers are unresolved, at both dependency gates. Without it the wake was parked until the stalled blocker resolved, which is never. Pause holds are unchanged. - **Agent guidance:** the wake payload carries `blockerStall`, and `renderPaperclipWakePrompt` adds a "Stalled dependency" section. It names the task and its recovery action and the board's next action, then the three choices: re-delegate, escalate with a comment, or block with a reason naming the stalled task. - **Other reason lists:** `issue_blocker_stalled` joins the responsible-user wake reasons, the conversation execution-wake reasons (it never drives a chat turn) and the wake diagnostic's known reasons. - **Board surface:** - `GET /issues/:id` returns `stalledDependencies`, readable issues only (`readableIssueEdgeIds`). - `StalledDependencyNotice` shows on the issue's chat notices, next to `IssueBlockedNotice`: each stalled child or blocker, its board next action, a link, and the child's own Retry (`useTaskRetry`), offered only when the child's interruption allows it. ## Verification Red/green: - `server/src/__tests__/issue-blocker-stall-wakeups.test.ts` (14 tests). - On master: no `issue_blocker_stalled` wake is raised. - With this PR: the parent is woken once per stall, and re-woken after a new stall. A blocker's dependent is woken. - The t05 shape (an action resolved as `blocked`) wakes once, and nothing more after a Retry. - No wake for an agent-owned action, a done, unassigned or hidden waiter, or an action resolved another way. - `listStalledDependencies` lists children and blockers. - `heartbeat-dependency-scheduling.test.ts` "admits an issue_blocker_stalled wake while a blocker is unresolved". On master the wake was parked (`null`); once admitted, the run-start gate cancelled it. With this PR it runs with `dependencyBlockedInteraction`. The file: 14 passed. - `packages/adapter-utils` `server-utils.test.ts`: the new prompt test failed on master (no `blockerStall`). The file: 141 passed. - `ui/src/components/StalledDependencyNotice.test.tsx`: 3 passed. `IssueChatThread.test.tsx`: 109 passed. Regression and static checks: - Nearby server suites (recovery durable wait, dependency wake routes, recovery actions, run-preparation): 149 passed. - `tsc --noEmit` passes for `packages/shared`, `server` and `ui/src`. - `pnpm check:token-gates` is clean. Review round 1 (Greptile, Superagent), each with a regression: - A backlog waiter is never woken. The waiter's status and assignee are guarded at enqueue. - One stall per run: no coalescing, so a merged run cannot keep only the last stall's context. - Only a delivered or pending wake ends a stall. A refused one (wake-on-demand off) is retried after 10 minutes, and an agent that takes no wakes is skipped without writing a row. - Company-wide passes resume after the last stall a capped pass reached. The per-issue sweep also checks the issue's own children and blockers, which covers an unassigned stalled blocker. - The dependency gates no longer trust the reason string. `verifyBlockerStallWake` requires a system request, the stalled issue's current stall action, and a real child or blocker edge. An agent-requested wake with the same context is parked (tested). - The notice shows a refused Retry's reason, and a lost response as "Taking longer than usual…", with Retry still available (tested). Review round 2 (Greptile), each with a regression: - Race: two passes claiming one stall leave one delivered row (unique index; the loser's insert is skipped). - The company recovery sweep selects companies whose only stall is a resolved-`blocked` board action, so an unassigned stalled blocker with a todo waiter is reached. - Private stalled issue: the waiter's wake and activity carry no title, identifier or recovery details. - `GET /issues/:id` always returns `stalledDependencies`, so a cleared stall replaces the cached list. - The notice shows a failed child read with a Reload button (tested). - `pnpm --filter @paperclipai/db check:migrations` passes. Review round 3 (Greptile), each with a regression: - A stall wake that fails to enqueue is retried: the recovery-action pass reports `blockerStallWakeFailures` and the dependency pass counts it in `enqueueFailed`, so the worker throws and retries at its backoff (`issue-recovery-work-lifecycle.test.ts`). - The notice drops a stall once the child's refreshed status has moved on (`StalledDependencyNotice.test.tsx`, 7 tests). ## Risks - A deploy wakes the waiters of every issue that is stalled today, once each. That is intended (they are stranded now), and it's bounded at 200 stalls per pass. - A stall wake may run while blockers are unresolved, like a comment wake. It is bounded to one per stall, and pause holds still apply. - The agent skill (`skills/paperclip`, CODEOWNER-owned) does not yet list the new reason. The wake prompt carries the guidance. - One additive index, built concurrently; no persisted run state change. Existing rows cannot violate it before deploy, since no `issue_blocker_stalled:` keys exist yet. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
95d476c57b |
docs(plans): record the measured seed experiment for per-turn uploads (#20139)
## Thinking Path - #20100 proposed sending only the changed files on per-turn follow-ups, with the repository seed held inside Daytona in a snapshot or a volume. It asked for one measured experiment before building. - Snapshot and volume operations were blocked by our Daytona key's permissions. A restricted key with those permissions now exists, so the experiment has run. - This PR records the numbers and the decision in the design doc. ## Linked Issues or Issue Description Refs #20100 **What happened?** The #20100 design's seed-location choice was unmeasured: snapshot build time and volume read speed were both open questions. **Steps to reproduce** 1. Run the SDK timing of direct upload, volume extract, and snapshot build and create on the paperclip seed (58.7 MB gzip) in Daytona `us`, 2 vCPU. **Expected behavior** The design doc states the measured winner, the per-turn saving, the one-time cost, and the work needed to build it. ## What Changed - `doc/plans/2026-11-08-incremental-per-turn-workspace-upload.md`: - The status now reads: measured; snapshot chosen; not built. - New "Experiment results" table: - direct upload: 6.4 s per turn; - volume: 6.8 s per turn, which rules it out; - snapshot: about 2–3.5 s per turn, after a 177 s build per seed generation. - Findings: fork is unsupported, compression is exhausted, and the snapshot builder runs as `daytona` and does not extract archives. - New "Work to build the snapshot seed" list: lifecycle, sandbox creation, verification and fallback, expiry and cleanup, and rebuild policy. ## Verification Docs only. Every Daytona resource the experiment created was deleted, and an account-wide listing confirmed none remained. ## Risks None at runtime. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool use and code execution, via Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
e194e1930b |
fix(native): prove a drained remote start had no turn when its controller only read the goal (#20137)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a graceful drain suspends a remote native run during its start, restart recovery may bootstrap it again only if the run's Postgres controller checkpoint proves that no provider turn exists > - That proof accepted `run.prepare`, `session.open` and session-open events > - A codex controller reads the session goal (`session.goal.get`, a read-only query) right after it opens the session and before it sends the first turn. A drain that lands in that window leaves a third command in the checkpoint > - The proof refused any checkpoint with a third command. Recovery called it `ambiguous_remote_recovery_evidence` and blocked replay, so the task waited for the board > - This pull request lets the proof accept the runner maintenance and read-only session commands that already count as maintenance in the local proof, and reads every recorded command > - The benefit is that a run drained in the first seconds of a per-turn Daytona start bootstraps again on its own ## Linked Issues or Issue Description Refs #20136 **What happened?** Daytona per-turn restart-chaos wave w-20261108T225520-s9501 (master |
||
|
DottaandClaude Opus 5.5
|
6451d2bfbf |
fix(native): verify a staged attachment restart on the claimed sandbox lease (#20136)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A follow-up codex run on a per-turn Daytona sandbox gets a fresh sandbox, with harness state restored from the previous turn's checkpoint, and attaches the previous runner session to the new run through a staged attachment > - A graceful restart during that start suspends the run. The next controller reattaches to the surviving remote runner, after proving it is on the sandbox lease the restart claim names > - The staged-attachment path required the lease to have been acquired as `resumed`. A per-turn follow-up's lease is `created`, so every such restart was refused with `native_runner_staged_attachment_recovery_unproven` > - The run then needed a Retry > - This pull request binds the staged restart to the claimed lease the same way the ordinary remote reattach does: same provider lease, same remote directory, and not a replacement sandbox > - The benefit is that a per-turn follow-up interrupted by a restart continues on its own ## Linked Issues or Issue Description Refs #20103 **What happened?** Daytona per-turn restart-chaos wave w-20261108T225520-s9501 (master |
||
|
DottaandClaude Opus 5.5
|
4420a5adcf |
fix(issues): a Retry held by a blocked subtask names it instead of failing silently (#20109)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The board's Retry is the one action a person takes when a task stops; it must either start work or say why it can't yet > - Restart chaos t05 left a parent task blocked on a blocked subtask. The board clicked Retry twice; each was accepted and nothing happened > - The Retry was saved and the queue deferred it behind the unresolved blocker, which is correct. But the task still looked failed with Retry offered, so the wait was invisible > - This pull request presents a saved Retry that waits only on unfinished blockers as waiting, names the blocked subtask and links it > - The benefit is that the board sees what to act on (the subtask's own Retry) instead of clicking a Retry that cannot run ## Linked Issues or Issue Description Refs #19831 (durable ownership plan). Restart chaos wave `w-20261108T200620-s9701`, task t05 (claude, M1, K4). **Problem:** parent CHA-5 was blocked by subtask CHA-10, which was itself blocked on a board recovery action. The board pressed Retry on CHA-5 at 20:27:46 and 20:40:46. Each `POST /issues/:id/retry` returned 202, and the saved Retry stayed `deferred_issue_execution` behind CHA-10, so no run started. The task's interruption presentation kept saying `failed` with `canRetry: true` and gave no reason. ## What Changed - `server/src/services/task-interruption-presentation.ts`: - For tasks with a saved (deferred, user-requested) Retry, the loader reads the queue's own dependency readiness (`listDependencyReadiness`). That counts cancelled blockers and done blockers whose workspace has not finalized, like the queue does. - When the saved Retry for the latest run is held only by unfinished blockers (no execution wait, no preparation), the task is presented as `working` with `canRetry: false` and `retryWait: { reason: "dependency_blocked", message }`. - The message has three forms: - a blocked blocker: "Retry or unblock it"; - a cancelled blocker: "Remove or replace that dependency to continue", because the queue keeps it unresolved; - otherwise: waiting for the blockers to finish. - The presentation carries no blocker names or ids. Issue reads return it to restricted agents too. - The Retry response (`POST /issues/:id/retry`) already returns this presentation, so the API says the Retry is waiting on dependencies as soon as it is saved. The blockers themselves are in the issue's `blockedBy`, which issue reads already filter per viewer. - `ui/src/components/ExecutionBlockerNotice.tsx` takes `blockers` (the issue's `blockedBy`, from `IssueDetail`). For a `dependency_blocked` wait it lists each readable unfinished blocker with a link and what it needs: "Blocked: open it to Retry" or "Cancelled: remove or replace it". Tokens only; `pnpm check:token-gates` is clean. - No shared type change, and no code comments added. ## Verification Red/green: - `server/src/__tests__/task-interruption-query.test.ts` "explains a saved Retry held by unfinished blockers instead of offering Retry again (restart chaos t05)". It sets up a parent with a failed run, a saved Retry deferred, and a blocked subtask that blocks it. - On master: `{ state: "failed", canRetry: true }`, no reason. - With this PR: `working`, `canRetry: false`, `dependency_blocked`. The presentation has no blocker title or identifier. - A cancelled blocker gets the remove-or-replace message. Once the blocker is done, the dependency wait is gone. - `ui/src/components/ExecutionBlockerNotice.test.tsx`: the notice links the blocked and the cancelled blockers with their actions, skips the done one, and offers no second Retry. Regression: `task-interruption-query` (25 tests) and `ExecutionBlockerNotice` (16 tests) pass, as do the presentation unit tests, workspace-recovery admission, gates and copyback tests, and `heartbeat-dependency-scheduling`. `tsc --noEmit` is clean for `server` and `ui`. ## Risks - One more batched read (dependency readiness plus blocker statuses), only for tasks in the page that have a saved Retry. - Presentation only. Admission is unchanged: the queue still runs the saved Retry when the blockers are done. - A Retry saved before a blocker was added shows the same wait, which is accurate, because the queue holds it too. ## Model Used Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code, with tool use (shell, file editing). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above (#20069) - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes (code comments; no user docs affected) - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green (pending) - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups (pending) - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1109.0-canary.0 |
||
|
DottaandClaude Opus 5.5
|
77168d3367 |
fix(recovery): settle a dead controller's result lease for a run a pause cancelled (#20113)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A budget hard stop pauses the agent, which cancels its running work and waits until every execution has provably stopped > - The shared ownership reconciler (#19933) settles an accepted result's executor lease when its controller is proven dead, so finalization does not wait out the 20-minute lease > - In the restart-chaos H1 wave, the pause cancelled a run 0.36 s after its result was accepted, and a hard kill then took the controller > - The settle only acted on runs that were still `running`, so the dead lease held finalization, the agent stayed in `pausing`, and every later task for it was stranded > - This pull request settles that lease for a cancelled run too, and records the budget pause with its reason > - The benefit is that a pause crossed by a restart completes at once and resume continues normally ## Linked Issues or Issue Description Refs #19933, #20101 **What happened?** Restart-chaos wave w-20261108T210118-s9802 (scenario H1). An agent-scope budget hard stop paused agent "Chaos claude". The pause cancelled run `dfc22298` at 21:02:27.818, 0.36 s after its result was accepted (finalization phase `workspace_finalizing`, executor lease until 21:21:55, controller PID 78157). A K4 SIGKILL followed. After the board's `raise_budget_and_resume`, the agent stayed `status=paused`, `lifecycle_state=pausing`, `lifecycle_holds=[]`, with its lifecycle operation at `attempts: 90`, `hostComplete: false` and no error. The pause host step (`stopInvocationsForAgents` → `agentExecutionsHaveStopped`) returns `pending` while any coordinator holds an owner lease with an accepted result. `settleNativeAcceptedResultLease` required `heartbeat_runs.status = 'running'`, so it returned `lease_no_longer_held` for the cancelled run. The pause also wrote no activity entry. **Expected behavior** A proven-dead controller's accepted-result lease is settled whatever stopped the run, so finalization finishes and the pause completes. The budget pause is logged with its reason. **Steps to reproduce** Pause an agent right after its run's result is accepted, then SIGKILL the server. After restart, the agent stays in `pausing` until the lease expires (up to 20 minutes). ## What Changed - `native-accepted-result-lease-settlement.ts`: settle the lease for a `running` or `cancelled` run. All other guards are unchanged: exact owner compare-and-set, owner proven gone by `proveOwnerGone`, remote copyback excluded, another host never proof. - `agent-lifecycle.ts` `applyAgentBudgetHoldInTransaction`: when the budget hold is added or removed, records `agent.paused` / `agent.resumed` with `reason: "budget"`, the lifecycle state and the holds, in the transaction that applies the hold. Applying the same hold again records nothing. - `budgets-service.test.ts`: its activity-log mock now also stubs `persistActivity`. Not changed, by design: resume still waits for an incomplete pause to finish before resuming. The pause's host step proves the agent's executions stopped before new work starts. In H1 it was blocked only by the dead lease, which is now settled at restart. ## Verification - `heartbeat-native-restart-suspension.test.ts`: new "finalizes an accepted result whose run a pause cancelled before a hard kill took its controller". It fails on master (lease still 20 minutes out) and passes with the fix (lease owner cleared, finalization past `workspace_finalizing`). Whole file: 41 passed. - New `agent-budget-hold-activity.test.ts`: the pause and release entries, with the reason, written once each. - `budgets-service.test.ts` (19), `budgets-history.test.ts`, `company-portability-routes.test.ts`, `agent-lifecycle/__tests__/lifecycle.test.ts`: passed. - `server` typecheck (`scripts/typecheck-layers.mjs`, low-memory): no errors. - Not run: the H1 chaos rerun (Phase 3 schedules it after merge). ## Risks - A cancelled run's accepted result is now finalized promptly after a hard kill instead of after the lease term. The finalizer already handles a run cancelled during finalization; the settle only shortens the wait. - New activity entries on budget pause and release. Run-log and activity only; no Telemetry. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test execution). Extended reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
89265f7312 |
test(chaos): add R1, a code change in a fresh clone of a real repository (#20111)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness proves that real agent tasks survive control-plane restarts > - Its coding workloads run in a tiny chaoscli fixture. Real work happens in large repositories, where workspace setup, copy and restore cost more and fail differently > - The campaign asks for real coding tasks in a fresh Paperclip clone, run under restarts > - This pull request adds the smallest support for that: a `--workspace-repo` option and an R1 workload > - The benefit is restart coverage for code changes in a real repository, graded by running the agent's own test and a hidden case ## Linked Issues or Issue Description Refs #20071, #20089 **What is missing?** The harness can only seed the generated chaoscli fixture as the project workspace. There is no way to run restart chaos against a code change in a real repository. **Expected behavior** An operator can name a git repository. The harness clones it fresh as the workspace, and a workload asks for a real code change in it with an oracle that runs the result. ## What Changed - `--workspace-repo <git repository>`: `seedChaosLab` clones it (shallow, from committed HEAD, `--no-hardlinks`) as the project workspace instead of the chaoscli fixture. Local only. - R1 workload: the agent adds `packages/shared/src/chaos/slug-<nonce>.ts` with `chaosSlug` and a `node:test` file, and runs it. The nonce names keep concurrent tasks from colliding in one repository. - The R1 oracle runs the agent's test with `node --test` and a hidden case against the module through Node's type stripping. - Collect commands can take an `argv` function of the nonce. - Config rules: R1 and `--workspace-repo` go together. B1 and B5 (which need the chaoscli fixture) do not combine with it. R1 is not a default workload. - README section "Code change in a real repository". ## Verification - `npx vitest run` in `tests/restart-chaos`: 195 passed, including new tests for the config pairing, R1 grading with the nonce argv, and a shallow clone of a fixture repository. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: clean. - I checked the oracle commands by hand on Node 26: `node --test` on a `.ts` test and `node --input-type=module -e 'import … from "./…ts"'` both work. - A live R1 wave against a Paperclip clone runs after the current chaos wave frees the harness slot. Its result will be posted here before merge. ## Risks - A shallow clone of a large repository adds seconds to wave seeding. That is one clone per wave. - The oracle uses Node's built-in type stripping. A runtime without it fails `node_test` and `hidden_case`, which shows as an oracle failure, not a false pass. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
42a5d71cb7 |
test(chaos): separate workspace provisioning from agent boot; grade outage-created runs as restart (#20119)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness grades speed against the native speed budgets, and a breach fails the wave > - In the real-repository R1 wave, the claude cold boot p95 was 31 s against 20.5 s, and cold first activity p95 was 37 s against 30 s > - The performance session traced both. The boot breach was the repository's own provisioning script on the first worktree creation (21–24 s), not Paperclip startup. The first-activity breach was a run created during an outage, behind the next fault > - This pull request reports provisioning on its own row, keeps it out of agent boot, and grades runs created during an outage as restart > - The benefit is that a speed breach points at Paperclip startup, not at project setup scripts or injected downtime ## Linked Issues or Issue Description Refs #20106, #20111 **What happened?** Wave `w-20261108T215822-s9805` (R1 in a Paperclip clone) reported a claude cold boot p95 of 31.0 s. The slow run spent 22.8 s in `heartbeat.prepare.workspace_provision`, the repository's `scripts/provision-worktree.sh`. Reuse takes 0.3 s. Its cold first-activity p95 came from run 630088a4, created 4 s after a fault killed its predecessor. That run queued until the next fault cleared, and #20106 still graded it cold because no fault started inside its own window. **Expected behavior** Project provisioning is reported, not charged to agent boot. A run created while a fault's outage is in progress is graded as restart. ## What Changed - `faultBeforeFirstActivity` takes outage windows and returns true when any window, from fault start to ready, overlaps the run's span from creation to its first activity. It used to take fault start instants only. - A run with `retryOfRunId` (the successor of an interrupted run) is graded as restart. - `collectWaveSpeed` reads each run's workspace operations. Provisioning counts only when a `workspace_provision` operation has `metadata.created: true`, meaning first-time creation of the workspace; reuse (about 0.5 s) is left in boot. - `bootWithoutProvisionMs` subtracts that first-time `heartbeat.prepare.workspace_provision` span from agent boot for the wave grade. Per-run boot breaches use it too. - New report-only row `workspace_provision` per adapter and start, shown only when the run created its workspace. It has no budget. - `runMetrics` and the TTFT baseline tool are unchanged. `BaselineRun` gains an optional `workspaceCreated`. ## Verification - `npx vitest run` in `tests/restart-chaos`: 195 passed. New cases cover a run created during an outage, a retry successor, provisioning counted only for `created: true` through `collectWaveSpeed`, the provisioning row and boot without provisioning. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: clean. ## Risks - A slow provisioning step no longer fails the boot budget. It stays visible on its own row, with no budget, because it runs the project's own script. - A run created during any outage, and any retry successor (including a user Retry), is graded as restart even when it started quickly. Restart recovery still measures it. - One extra API read per run (workspace operations) during speed collection. A failed read counts as not created, so boot is never under-charged. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
b43cccbb08 |
fix(runner): keep a restart continuation when a stale earlier continuation is reported lost (#20118)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a restart loses a native run's active turn, the runner reports it as `provider_turn_lost_on_restore` and the controller starts one same-conversation continuation. A later restart that loses the continuation is answered the same way > - With back-to-back restarts, the controller's checkpoint can miss a continuation that the runner already accepted. On the next restore the controller starts a fresh continuation, and the restored runner then reports the unrecorded earlier continuation as lost > - The event consumer treated that stale lost terminal as the run's terminal, so the run failed while its new continuation was already running > - Recovery then refused to replay the failed run, and the task needed a Retry. In an M1 parent/subtask task, the parent's Retry succeeded but the parent stayed `blocked` on that subtask > - This pull request makes the consumer skip a restart-lost terminal for any turn other than the continuation it started > - The benefit is that back-to-back restarts no longer turn a recoverable lost turn into a failed run that needs Retry ## Linked Issues or Issue Description Refs #20107 **What happened?** Restart-chaos K14 wave w-20261108T213600-s9803 (local, an upgrade between builds plus repeated restarts) at |
||
|
DottaandClaude Opus 5.5
|
412d49e48d |
fix(runner): resume an ACPX turn interrupted by a deploy instead of refusing the launch digest (#20112)
## Thinking Path
- The rule for upgrade-safe saved state: work saved by the old build
must resume under the new build.
- The K14 upgrade-mid-run chaos wave (`838e9bf3d2` → `f789f3e2af`) broke
that rule for Claude. The same heartbeat run resumed after the upgrade
restart, and the runner's resume probe (`runner.drain`) failed with
`ACPX durable launch profile digest does not match runner startup`
(`provider_state_incompatible`). t03 and t04's Claude subtask both
needed a board Retry. Codex crossed the upgrade cleanly.
- The ACPX launch profile digest covers the provider-pack authority and
the sidecar artifact hashes, so every deploy changes it.
- #18879 already re-binds a suspended session to the new digest. A
deploy that restarts the runner mid-turn leaves the same run with an
active turn, or with an open session, which is the state a same-build
lost-turn restore leaves behind. Neither was re-bound.
- A same-build restart already recovers both states safely: it reopens
an open session, and recovers an active turn through
`restore_conversation_after_lost_turn`, which is guarded by the
provider-lifetime fence. An upgraded build should take the same path.
## Linked Issues or Issue Description
Refs #18879
**What happened?**
A deploy restarted the runner while a Claude (ACPX) task was mid-run.
When the same run resumed under the new build, restore refused the saved
state with `provider_state_incompatible`. The task stopped and needed a
manual Retry, which started a fresh provider session and dropped the
conversation.
**Steps to reproduce**
1. Start a Claude task on build A and let it reach an active turn.
2. Upgrade to build B, which ships a new provider pack or sidecar build,
with a graceful restart while the turn runs.
3. The same run's resume fails with `ACPX durable launch profile digest
does not match runner startup`.
**Expected behavior**
The resumed run recovers exactly as it would after a same-build restart.
The conversation is restored, or the lost turn is closed through the
normal recovery path, with no Retry.
## What Changed
-
`packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_backend.rs`:
- `rebinds_settled_session` now also re-binds the saved digest for an
**interrupted turn or open session of the same heartbeat run**
(`rebinds_interrupted_turn`).
- The re-bind requires exactly the same-build recovery preconditions: no
accepted semantic result, no pending runtime request, no unconfirmed
provider exit, and a dormant goal. It also keeps the existing
requirements: no candidate admission, and the same sidecar command and
arguments.
- Recovery then runs unchanged. The provider-lifetime fence still
decides whether the conversation is restored or the lost turn is closed.
- **Relocated installs:** an upgrade can install the same launch files
under a new directory, as an `npx` versioned cache, a new image layout
or a new checkout does. The K14 harness moves between worktrees. The
re-bind now accepts a launch of the same shape: the same executable and
sidecar file names, and identical non-path arguments. It then records
this startup's runner-owned paths in the saved descriptor. A renamed
executable or a changed argument stays strict.
- An active turn or open session saved by a *different* run stays
strict, as before. So do suspended sessions with undelivered events, and
every qualified-release binding (ACPX, agent server and command digest).
- Tests (Rust, `acpx_provider_backend`):
-
`interrupted_turn_of_the_same_run_rebinds_to_the_upgraded_runner_launch_profile`:
red on master. An old-digest saved state with an active turn restores,
the digest is re-bound, and identity and command digest are kept.
-
`open_session_of_the_same_run_rebinds_to_the_upgraded_runner_launch_profile`:
restore gets past the saved-state check, to provider start.
- `saved_sessions_rebind_to_an_upgraded_build_installed_at_a_new_path`:
red on master and on the first fix, with the production error. It covers
an interrupted turn of the same run and a settled session attached by
the next run, both moved to a new install directory.
-
`interrupted_turn_after_upgrade_stays_strict_when_lost_turn_recovery_is_unsafe`:
each unsafe guard still fails with `provider_state_incompatible` and
leaves the saved state untouched.
- The existing strict tests for other runs are unchanged and still pass.
- `packages/paperclip-runner/docs/durable-recovery.md`: documents the
extended re-bind.
- `tests/acpx_warm_attachment.rs`: the end-to-end test
`restores_a_turn_lost_to_a_runner_upgrade_under_the_new_launch_profile`,
from the Phase 2 session's parallel fix. It fails on master with the
production error and passes here.
## Verification
- `cargo test -p paperclip-runner-core --lib acpx_provider_backend`: 44
passed.
- The full `paperclip-runner-core` lib suite: 585 passed. The 3 failures
are timing-based `task_memory` kill-monitor tests on macOS; that file is
untouched (last changed 5 days ago). CI runs them on Linux.
- `cargo fmt --check`: clean.
- Live: the first K14 rerun, at
|
||
|
DottaandClaude Opus 5.5
|
15b987d9b8 |
fix(runner): a finish or block waits for held calls before it is judged (#20108)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Restarts must be invisible to a task: an agent that was working
keeps working after the control plane comes back
> - Restart chaos t05 (Daytona warm wave, codex, K4 hold) ended a task
as blocked even though its document save succeeded
> - #20069 refuses a finish or block that the agent decided before it
saw results that had already returned. Here the results had not returned
yet when the block was checked: after the reattach, the held calls and
the later block are handled at the same time
> - This pull request makes a finish or block wait, for a bounded time,
for earlier calls of its turn that have not returned, and then apply the
same check
> - The benefit is that a controller outage during a Paperclip tool call
no longer turns into a false "blocked" task
## Linked Issues or Issue Description
Refs #19831 (durable ownership plan). Restart chaos wave
`w-20261108T192459-s9501`, task t05 (Daytona warm, codex, C1, K4 hold of
182 s, master
canary/v2026.1108.0-canary.24
|
||
|
DottaandClaude Opus 5.5
|
a8bbdf237c |
fix(runner): reopen the ACPX session at turn.start when it can be safely reopened (#20110)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A native Claude ACPX run must survive restarts and suspensions. A
turn stopped for suspension drops the live ACPX session and leaves the
saved lifecycle `prepared`
> - The next `turn.start` checked for a live session before anything
else, so it failed with "ACPX session is not open". That happened even
though the saved session was safe to reopen
> - In restart chaos this made a Claude approval task (G1 t03) need a
Retry after repeated restores
> - This pull request makes `turn.start` reopen a reopenable session
before it starts the turn
> - The benefit is that a suspended or restored Claude run keeps going
on its own, with no Retry
## Linked Issues or Issue Description
Refs #20089
**What happened?**
In a local restart-chaos G1 (approval) wave, Claude ACPX task t03 failed
after repeated restores with `ACPX session is not open` and needed a
Retry. The Retry overwrote the session directory, so the exact sequence
could not be read back. The same error reproduces directly with
`turn.stop` (suspension) followed by `turn.start` on the same session:
the stop clears the live session and sets the lifecycle to `prepared`,
and `turn.start` never reopens it.
**Expected behavior**
When the saved session can be reopened, `turn.start` reopens it, the
same way `session.open` does, and then starts the turn. "Can be
reopened" means: the conversation identity is recorded, the provider
exit is confirmed, the session belongs to this run, and the lifecycle is
`prepared`, `suspended` or `session_open`. A session that was closed on
purpose, or whose provider exit is unconfirmed, still fails.
**Steps to reproduce**
1. `run.prepare`, `session.open`, then `turn.start`.
2. `turn.stop` (suspension stop).
3. `turn.start` again. Before this change, it fails with "ACPX session
is not open".
**Paperclip version or commit**
master at
|
||
|
DottaandClaude Opus 5.5
|
a88e23fa34 |
fix(runner): read an uncommitted turn start as empty history when the event window is complete (#20107)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run must survive repeated control-plane restarts; each
restore reconciles the turn its checkpoint names as active
> - That reconciliation reads the turn's history. When the provider has
no paginated history, the runnerd transport rebuilds it from the
committed runner events
> - After back-to-back hard restarts, the checkpoint can name an
accepted turn whose start was never committed. The runner was killed
after journaling `turn.start` and before the provider emitted anything
> - The rebuild treated every missing start as lost history and failed
with `codex_history_incomplete`, so the task needed a Retry
> - This pull request treats a missing start as empty history when the
committed window is provably complete
> - The benefit is that a run survives back-to-back restores, as it
already survives one
## Linked Issues or Issue Description
Refs #20074
**What happened?**
In restart-chaos wave w-20261108T200620-s9701 (local, K4/K6 + M1,
back-to-back K4 restores), three runs failed on the second restore. Two
were Claude ACPX C1 tasks; the third was an M1 parent with its subtask.
They failed with `native_session_recovery_failed:
HarnessReconciliationError: codex_history_read_failed: thread/items/list
requires supported paginated history; Error: codex_history_incomplete:
requested turn start is outside the retained runner event window`, and
each needed a Retry.
The sequence:
1. The first restore lost the in-flight turn and started a continuation
turn. ACPX echoes the requested turn id, so the checkpoint named it as
active.
2. Six seconds later a second K4 SIGKILL killed the runner. The runner
had journaled that `turn.start` (now `execution_indeterminate`), but the
provider had emitted nothing, so no `turn.started` was committed and the
runner outbox was empty.
3. On the second restore, `thread/items/list` for that turn hit the
guard, although the committed window was complete: source sequences
1–25, nothing evicted.
**Expected behavior**
The second restore treats the never-started turn like the first restore
treated its lost turn. The runner reports it lost on restore, and
recovery starts a continuation with no Retry.
**Steps to reproduce**
1. Have a checkpoint name an accepted turn whose `turn.started` was
never committed. For example, kill the runner after `turn.start` and
before the provider emits anything.
2. Restore the session. Before this change, recovery fails with
`codex_history_incomplete`.
**Paperclip version or commit**
master at
|
||
|
DottaandClaude Opus 5.5
|
83a229ae8c |
test(chaos): grade a fault-hit run as a restart, not a cold start (#20106)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness grades speed against the native speed budgets, and a regression fails the wave > - It grades a run's time to first activity as a cold or warm start, and restart recovery separately > - Fault receipts carry no run ids, so a run that a fault hit before its first activity was graded as a cold start > - In a K4/K6 wave, the two slowest codex cold samples were runs that a K6 SIGKILL hit before their first activity, so the cold p95 breach was not a cold start > - This pull request grades such a run as a restart > - The benefit is that cold and warm rows measure only starts with no fault, and restart cost stays in restart recovery ## Linked Issues or Issue Description Refs #20071, #20094 **What happened?** Wave `w-20261108T200620-s9701` reported a codex cold time-to-first-activity p95 of 29.9 s against a 17 s budget. The two slowest samples were runs that a K6 restart hit before their first activity. **Expected behavior** A run whose first-activity window contains a fault start is graded as a restart, not as a cold or warm start. **Actual behavior** The run was graded as a cold start, and the restart time counted as cold-start latency. ## What Changed - `faultBeforeFirstActivity(run, faultStarts)`: true when a fault started between the run's trace start and the end of its raw first-activity span (or first-token span when no activity span exists). - `collectWaveSpeed` sets such a run's phase to `restart` before it removes outage time. Cold, warm and single-run grading already skip restart runs. The run still counts in restart recovery through the fault's run timing. - A test covers the helper and the exclusion from the cold row. ## Verification - `npx vitest run wave-speed` in `tests/restart-chaos`: 10 passed. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: clean. ## Risks - A run with no first-activity or first-token span counts as hit when any fault starts after its trace start. Such a run has no cold sample to grade anyway. - A fault that starts just before a run's trace start does not count, and outage removal handles it as before. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1108.0-canary.23 |
||
|
DottaandClaude Opus 5.5
|
776f4cb602 |
fix(native): restart an interrupted remote bootstrap on a resumed reusable lease (t06) (#20103)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native run on a remote sandbox must survive a control-plane restart; a bootstrap interrupted before its first turn is restarted on the same run > - With a reusable (warm) sandbox lease, the restart always resumes the same sandbox. The interrupted runner's state is still on disk there: never suspended, and with no provider turn > - The resumed check accepted only a suspended checkpoint or no state at all, so it failed closed with `resumed_not_suspended`, and the task needed a Retry (restart chaos t06) > - The pre-phase had already proven no turn was sent and had stopped the stale runner, but that proof never reached the resumed check > - This pull request carries that proof forward, for this run only, and starts the bootstrap again on the resumed sandbox > - The benefit is that a warm task interrupted during startup continues on its own after the restart ## Linked Issues or Issue Description **What happened?** In restart-chaos wave w-20261108T192459-s9501 (Daytona warm, claude C1), run 9bf86b59's runner reached transport-ready at 19:37:42 UTC. The control plane was then down (another task's K4, then this task's K1). Restart recovery classified the run `bootstrap_incomplete` and restarted it on the same run at 19:41:51. The reusable lease resumed the same sandbox. `inspectRemoteHarnessState` found this run's runner state with `lifecycle` other than `suspended`, and the run failed at 19:41:59 with `runner_ingress_unavailable: runnerd process failed: runner_harness_state_mismatch: resumed_not_suspended`. The task needed a manual Retry. Per-turn and non-reusable leases do not resume, so they take another branch and were unaffected. **Expected behavior** A same-run restart of a bootstrap that provably sent no turn starts the bootstrap again on the resumed sandbox, with no Retry. It must never take over a live provider session or another run's warm session. **Steps to reproduce** 1. Run a task on a reusable (warm) Daytona lease. 2. Kill the control plane after the runner connected and before the first turn, so the run is `bootstrap_incomplete`. 3. Before this change, the restart fails with `resumed_not_suspended` on the resumed lease. **Paperclip version or commit** master at |
||
|
DottaandClaude Opus 5.5
|
c4ba3974a7 |
fix(heartbeat): close run admission before a graceful shutdown closes its listener (t09) (#20102)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A graceful restart (K2) drains the control plane: it closes HTTP admission, suspends running runs, and the next boot resumes them > - Run admission stayed open while the HTTP listener closed. Heartbeat's shutdown flag was set only later, inside hot-restart preparation > - A run that finished in that window released its issue and promoted the queued comment wake. That started a new run that had no callback listener > - A remote warm continuation then failed with `local_runtime_api_listener_unavailable`, and the task needed a Retry (restart chaos t09) > - This pull request closes run admission before the listener closes, so the promoted wake stays queued for the next boot > - The benefit is that a comment sent during a deploy continues on its own after the restart ## Linked Issues or Issue Description Refs #19596 **What happened?** In restart-chaos wave w-20261108T192459-s9501 (Daytona warm, Codex C1, K2), a K2 SIGTERM started a graceful drain at 19:42:52 UTC, and the HTTP listener closed first. At 19:43:03 the running turn finished and released its issue. That promoted the queued comment wake, and `startNextQueuedRunForAgent` started a new run about 11 s into the drain. Backend setup resolved the local runtime API URL, found the listener closed, and threw `local_runtime_api_listener_unavailable`. The remote target is excluded from the closed-listener handoff, so the run failed, and the task waited for a manual Retry. **Expected behavior** No queued run starts once a graceful shutdown has begun. A wake promoted during the drain stays queued, and the next boot starts it with no Retry. **Steps to reproduce** 1. Run a remote warm task and post a comment while its turn runs, so the comment wake queues behind it. 2. Send SIGTERM to the control plane just before the turn finishes. 3. Before this change, a new run starts during the drain and fails with `local_runtime_api_listener_unavailable`. **Paperclip version or commit** master at |
||
|
DottaandClaude Opus 5.5
|
3bf51913ef |
fix(daytona): gzip transport compression for sandbox images without zstd (#20104)
## Thinking Path - Per-turn remote sandboxes are the default, so every remote turn uploads the workspace into a fresh Daytona sandbox. - The Daytona plugin compresses large uploads with zstd, but only when the sandbox has a `zstd` binary. Without one, the upload goes raw: 220 MB for the paperclip seed. - The default Paperclip runner image has zstd, so it is already covered. A sandbox image without zstd is not: for example a user-supplied `config.snapshot` built on plain `debian:bookworm`. Such an image still has gzip. - This adds gzip as a fallback codec for those images. It is a robustness fallback, not a speedup for the default image. ## Linked Issues or Issue Description Refs #20100 **What happened?** On a Daytona sandbox image without a `zstd` binary, every remote turn uploads the workspace seed uncompressed. Measured live with the paperclip seed: raw takes 18.6–28.8 s, against 6.1–16.0 s compressed. **Steps to reproduce** 1. Use a Daytona snapshot whose image has gzip but no zstd. 2. Run a remote task on a large repository and inspect the `transfer` span: there is no `transfer.compression.codec` attribute, and the full raw size is uploaded. **Expected behavior** Large workspace tarballs are compressed in transit whenever the sandbox can decompress them. ## What Changed - `packages/plugins/sandbox-providers/daytona/src/file-sync.ts`: - The existing availability probe (the same round trip as `mkdir`) also reports `gzip`. - Codec choice: zstd when the sandbox has it and Node supports it (unchanged), otherwise gzip, otherwise raw (unchanged). - gzip runs at level 1. On the 220 MB paperclip seed it produces 69.6 MB in 1.26 s on the host; zstd -3 produces 57.6 MB in 0.54 s. - In the sandbox, gzip decompresses with `gzip -dc -- <scratch> > <raw scratch>`. It reuses the existing explicit `chmod`, atomic `mv -f`, failure sweep, private host temp directory, and minimum-saving fallback. - A failed or partial decompress exits 49, the raw scratch is never promoted, and every scratch file is swept. The workspace sentinel and `baselineSha256` checks still run before `turn.start`. - The `transfer.compression.codec` span attribute is now `zstd` or `gzip` (still a closed set). - Tests in `file-sync.test.ts` (red on master): - A real shell promotion on a sandbox that reports no zstd: the upload is `.gz` and under half the size, `gzip -dc` runs, and the target is byte-identical with mode 0600 and no scratch left. - A truncated gzip upload fails the sync, never creates the target, and leaves no scratch. - Codec preference: zstd when both are present, gzip when only gzip is, raw when neither is. ## Verification - `npx vitest run` in `packages/plugins/sandbox-providers/daytona`: all pass. The existing zstd suite runs on this host because it has zstd. - `tsc --noEmit -p packages/plugins/sandbox-providers/daytona`: clean. - Live Daytona run on the runner image, 2 vCPU, 220 MB seed, through the real `performSyncIn` with each codec forced through the probe. The sha256 matched every time. | Codec | Times | |---|---| | raw | 28.8 s, 18.6 s, 20.2 s, 22.8 s | | zstd | 6.9 s, 6.1 s, 16.0 s, 8.5 s | | gzip | 9.8 s, 16.8 s | Network variance is large. Both codecs roughly halve the raw upload or better. ## Risks - **Host CPU:** about 1.3 s of streamed gzip per large upload, on the libuv thread pool, and only when the sandbox lacks zstd. Files under 8 MB, or that save less than 10%, still go raw. - **No compressor at all:** a sandbox without gzip or zstd uploads raw, as today. - **Probe caching:** the probe runs on every sync, on the `mkdir` round trip it already makes, so it adds no latency. It stays uncached, following the existing design's rule that one principal's observation is never reused for another. - **Observability:** one new value, `gzip`, in an existing closed-set span attribute. The endpoint gate is unchanged. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool use and code execution, via Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
f789f3e2af |
fix(environments): a consented environment delete releases restore-source protection so it never stays blocked (#20099)
## Thinking Path - Deleting an environment with consent destroys its kept sandboxes. #20085 added failed runs' restore-source sandboxes (`retain_on_failure`) to that teardown, and the delete dialog now tells the operator that the consented delete destroys those files. - But every teardown path refuses a lease whose sandbox carries a `workspaceRestoreRecovery` marker (`legacyShutdownWorkspaceResourceProtected`). That is correct for ordinary cleanup, since the files may be the only copy. - So the consented delete claimed the restore sources for cleanup, every teardown returned without destroying, and the delete returned 409 `pending_sandbox_cleanup` forever: a refusal the user cannot act on. - The operator's consent is exactly the decision the protection waits for, so the delete releases it explicitly and audits who did, then proceeds. ## Linked Issues or Issue Description Refs #20048, follow-up to #20085 **What happened?** In Daytona warm wave w-20261108T192459-s9501, the harness's consented environment delete returned 409 `pending_sandbox_cleanup` for minutes (it retried every 3 s). Two restore-source leases (8b26e044, bcb897c9) had been claimed for cleanup with `environment_deleted`. The activity log shows a cleanup attempt on every retry, each returning without destroying. On the kept database copy, `legacyShutdownWorkspaceResourceProtected` returns `true` for both, because of their `workspaceRestoreRecovery` marker. Their restore recovery had already succeeded (workspace sync finalized). Phase 3 deleted the two sandboxes by hand. **Steps to reproduce** 1. A failed run leaves a `retain_on_failure` lease with `workspaceRestoreRecovery` metadata (sandbox stopped). 2. `DELETE /api/environments/:id?destroyReusableSandboxLeases=true`. 3. The lease is claimed (`pending_cleanup`), the teardown refuses it, and every retry of the delete returns 409. **Expected behavior** The consented delete destroys the restore source (the operator was told the files go with it) and the environment is deleted. No 409 the user can't act on. ## What Changed - `server/src/services/environment-runtime.ts`: new `releaseRestoreSourcesForEnvironmentDelete({ environmentId, consentedBy })`. For each of the environment's restore-source sandboxes (`retain_on_failure` with `workspaceRestoreRecovery`, any non-terminal status including an earlier `pending_cleanup` claim), it removes the marker from every lease row on that sandbox and records `restoreSourceReleasedByEnvironmentDelete: { consentedBy, releasedAt }`. `destroyReusableSandboxLeasesForEnvironment` takes `consentedBy` and releases first. - `server/src/routes/environments.ts`: a consented delete whose only blockers are pending cleanup or kept sandboxes releases restore protection before the pending-cleanup retry, so leases an earlier attempt left claimed are freed too. It then passes `consentedBy` to the teardown. - Tests: - A restore source now carries the real `workspaceRestoreRecovery` marker, and the consented teardown destroys it. This failed before the fix (`{ destroyed: 0, failed: 1 }`), matching the wave. - The route test asserts the release runs before the pending-cleanup retry. ## Verification - `cd server && npx vitest run src/__tests__/environment-runtime.test.ts src/__tests__/environment-service.test.ts src/__tests__/environment-routes.test.ts`: 347/347 pass. - The restore-source test fails on master with `failed: 1`. ## Risks - **Losing restore files without consent.** The release runs only on the explicit consented delete path (`destroyReusableSandboxLeases=true`), and only when the remaining blockers are pending cleanup or kept sandboxes. It is recorded on every released row. Leases with a pending native export are still skipped by the teardown. - **Ordinary cleanup.** Unchanged: restore protection still holds everywhere else. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool use and code execution, via Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1108.0-canary.22 |
||
|
DottaandClaude Opus 5.5
|
2a16be36ab |
docs(plans): per-turn follow-ups without re-uploading the repository (#20100)
## Thinking Path - Per-turn remote sandboxes stay the default while warm reuse (#20048) is on hold. - On the paperclip clone, a per-turn follow-up spends about 13 s in `workspace_sync.transfer`, though its changes pack in 0.5 s: every fresh sandbox receives the whole 70 MB (gzip) seed again, at about 5 MB/s from the controller. - The coordinator asked for a design that uploads only changed files into a fresh sandbox, with a full-upload fallback, measured against the spans. It also asked to build it only if the implementation is bounded and safe. ## Linked Issues or Issue Description Refs #20048 **What happened?** Per-turn follow-ups on large repositories wait about 13 s re-uploading the full workspace seed even when almost nothing changed (pack 0.54 s, transfer 13.3 s, time to first token about 31 s). **Steps to reproduce** 1. Run `tests/restart-chaos/ttft-baseline.ts --env daytona --workspace-repo <paperclip clone>` with per-turn sandboxes. 2. Compare `workspace_sync.pack` and `workspace_sync.transfer` on the follow-up turn. **Expected behavior** A follow-up uploads only its overlay into a sandbox that already holds the seed, falling back to the full upload when unsure. ## What Changed - `doc/plans/2026-11-08-incremental-per-turn-workspace-upload.md`. It contains: - the measured span table and the size analysis (220 MB archive, 70 MB gzip, about 5 MB/s); - the expected saving: about 12–13 s per follow-up, time to first token about 31 s → about 18 s; - a design keyed by seed generation with a content-hashed manifest, using a Daytona snapshot (preferred) or a Daytona volume to hold the seed; - the overlay-only turn flow, with a verified fallback; - one bounded experiment to choose the seed location before building. - Not built yet: both seed locations need provider capabilities (snapshot build time, volume read speed) that must be measured first, so the implementation isn't bounded until the experiment picks one. ## Verification Docs only. ## Risks None at runtime. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool use and code execution, via Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
1b7254206f |
fix(recovery): replay a preparation a restart interrupted once proof shows no provider work started (#20072)
## Thinking Path - Restarts are product failures: a task interrupted by a restart must resume on its own, never wait for a user Retry. - A run interrupted during preparation, before any provider work, is safe to replay; the agent never touched the workspace. Paperclip already has the proof (`isInterruptedNativePreparation`) and the settle (`settleInterruptedNativeBootstrap`) that replay such a run in the uncharged interrupted-preparation lane. - But a graceful shutdown terminalizes that run inside the draining process, where the proof cannot hold yet: the owning controller is still that process, and its sandbox lease may still be cleaning up. So recovery records `preserve_without_replay_v1` with replay blocked. - When the proof becomes available (the next process boots, and the lease is released with a confirmed stop), nothing re-judges the hold. The only re-evaluation point runs after a pending-cleanup sweep, so the task waits for a manual Retry. - The coordinator set the rule: replay automatically with durable proof of no provider work; anything uncertain stays held. This PR adds the re-evaluation point and no new proof logic. ## Linked Issues or Issue Description Refs #20048 **What happened?** In Daytona warm wave w-20261108T142906-s9501, task t05: scheduled retry c99b3a2e was 2 s into preparation (legacy, stage `preparing`, no runtime mode, no events, no finalization) when the K2 graceful shutdown interrupted it. Terminalization in the draining process recorded `legacy_execution_requires_reconciliation` with `preserve_without_replay_v1` and replay blocked, while its lease was still `pending_cleanup`. The lease was released with a stop receipt 2 s later, and the next process started, but the hold was never re-judged. The harness clicked Retry at 14:44:27. **Steps to reproduce** 1. Start a run that is still in preparation (legacy, `preparing`, adapter `paperclip_runner`). 2. Gracefully shut down the server; terminalization records a preserve-without-replay hold. 3. Let its lease release with a confirmed stop, and restart the server. 4. The hold stays `blocked`; the task needs a manual Retry. **Expected behavior** Once the owning controller is proven gone and every lease is released with a confirmed stop, the ownership reconciler re-judges the hold with the existing proof. With no provider work and every cleanup confirmed, the hold is settled and the preparation is replayed with no attempt charged. Possible provider work, a live controller, or an unfinished cleanup keeps the hold. ## What Changed - New ownership kind `legacy_interrupted_preparation_hold`, in `server/src/services/interrupted-preparation-hold-settlement.ts` and registered in `recovery/ownership-registry.ts`. - **Listing:** legacy runs that are `interrupted` or `failed`, in stage `preparing` with no `runtimeModeResolvedAt`. Each must have a system-recorded `legacy-execution:<run>` hold: resolved/blocked, `preserve_without_replay_v1`, replay blocked, no owning user, and none of `ownerGoneAttestation`, `explicitUserContinuation` or `executionReconciliation`. Every lease must be released with no failed cleanup, so a lease that settles later is listed on the next pass. - **Owner:** the run's recorded controller (`legacyControllerOwnerStamp`). The shared reconciler settles only after `proveOwnerGone` proves it gone. A missing stamp, or one from another host or PID space, is unprovable and stays held. There is no age fallback. - **Settle:** re-checks the existing durable proof (`isInterruptedNativePreparation`: no coordinator, result or execution events; leases released with stop receipts; workspace repair when needed). It then resolves the hold through `settleInterruptedNativeBootstrap` as `false_positive` with replay allowed and continuation pending, delivered through the existing workspace-recovery queue in the uncharged interrupted-preparation lane. Its row locks make the transition happen once; the settle reports settled only when it resolved holds. - `server/src/services/legacy-execution-recovery.ts`: `settleInterruptedNativeBootstrap` takes an optional `proof`, recorded in the `issue.recovery_action_resolved` activity details as `ownerProof`, and an `onHoldsResolved` callback. - `server/src/__tests__/interrupted-preparation-hold-settle.test.ts`, K2-during-preparation shape, run through the reconciliation pass: - settled with a dead controller in this PID space, with the audit entry naming `pid_dead`; - kept on a `turn.started` event, a stamp from another process space, a missing stamp, the current process, a lease still cleaning up, an operator owner-gone attestation, and a user continuation; - two concurrent settles produce one resolution. - `ownership-reconciliation-report.test.ts`: adds the kind to the list of settling kinds. **Previous-release state:** holds written by the previous release (`preserve_without_replay_v1`, replay `blocked`) are what this re-judges, through the same proof. A run without a recorded controller stamp stays held. No saved state changes shape. ## Verification - `cd server && npx vitest run src/__tests__/interrupted-preparation-hold-settle.test.ts`: 9/9 pass. Removing the kind registration makes the settle case fail. - The ownership, retained-orphan and recovery suites (19 files) pass, after adding the kind to the report's settling-kinds list. - Not yet covered: the Daytona warm wave rerun, on measure/remote-warm with this change. ## Risks - **Replaying work that already ran.** Settlement requires both the shared owner proof and the existing durable no-provider-work proof. Anything unprovable stays held. - **Overriding an operator.** Only holds the system recorded are listed; attested, continued or owned holds are never touched. - **Load.** The listing is a filtered query on the 30 s reconciliation pass, gated so a run with an unsettled lease is not polled. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool use and code execution, via Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
88835e624a |
test(chaos): fail M1 fast on a blocked subtask and grade interrupted waves (#20094)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness proves that real agent tasks survive control-plane restarts > - In a K4/K6 wave with the M1 multi-agent scenario, one subtask failed. Its parent waited on it, and the harness kept watching for over 50 minutes > - When the operator stopped the wave, the harness wrote no grade files, so the 17 finished tasks had no grade > - Separately, master's harness test for time to first activity failed after #20077 changed how time is measured from run creation > - This pull request makes M1 fail fast on a blocked subtask, grades the finished tasks of an interrupted wave, and aligns the speed grade with #20077 > - The benefit is shorter waves with a grade for every task, and a green harness suite on master ## Linked Issues or Issue Description Refs #20071, #20077, #20088 **What happened?** Wave `w-20261108T181313-s9701` (local, codex and claude, A1/C1/M1, K4 and K6) ran 71 minutes against a planned 30. Task t05 (M1) delegated CHA-8 and CHA-9. The CHA-8 run failed after a K4 restart (the defect that #20088 fixes), and CHA-8 went to `blocked`. The observer kept waiting for CHA-8 to finish. After SIGINT, the harness threw before grading, so no task had a grade file. Also, `wave-speed.test.ts` fails on master: #20077 changed `runMetrics` to measure from the start of `heartbeat.queue`. **Steps to reproduce** 1. Run an M1 task whose subtask run fails and leaves the subtask `blocked`. 2. See the observer wait until the parent deadline. 3. Send SIGINT to the harness and see that no grade files are written. **Expected behavior** A blocked subtask ends the wait, and the parent is graded as a failure with a `task_child_blocked` timeline entry. An interrupted wave grades every finished task and fails the wave. ## What Changed - `observer.ts`: `childrenPending` treats a `blocked` subtask as settled, so the parent rests and is graded. The timeline records `task_child_blocked` with the subtask ids. - `observer.ts`: `markUnfinished` closes an open task at interrupt as unfinished (`timedOut`, last seen status) and records `task_unfinished_at_interrupt`. - `main.ts`: an interrupt now sets the harness error and grades the wave, instead of throwing before the grade. The wave still fails. - `wave-speed.ts`: `firstActivityMs` uses `runMetrics().firstActivityAfterRunCreatedMs` from #20077, so there is one definition of time from run creation. - `wave-speed.test.ts`: the fixture starts `heartbeat.queue` when the comment wait ends, as in real runs. ## Verification - `npx vitest run` in `tests/restart-chaos`: 172 passed. Before this change, the wave-speed test failed on master. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: clean. ## Risks - A subtask that is `blocked` only for a short time and then resumes would end the parent's watch after the 60-second grace. The grade then shows a failure that a longer wait could avoid. M1 subtasks have no blockers, so `blocked` means a stuck subtask. - An interrupted wave grades tasks while its server is still up. If the server is already down, artifact collection fails, and those tasks are graded as failures. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
c02fd8d427 |
docs(plans): salvage-then-retire design for kept restore-source and done-task warm sandboxes (#20087)
## Thinking Path - Paperclip keeps some remote sandboxes after their run: warm ones for reuse, and failed runs' stopped sandboxes as workspace-restore sources. - In the warm Daytona chaos waves for #20048, the real leaks were restore sources that nothing retires once a user Retry closes their recovery actions. Separately, a done task's suspended warm sandbox is only destroyed when its environment is deleted. - Retiring them early needs a provider-I/O salvage step: compare against the durable save, export anything unsaved to the issue, then destroy. That spans recovery, storage, attachments and teardown. The coordinator asked for a design note so it can be its own reviewed task instead of a rushed patch. ## Linked Issues or Issue Description Refs #20048, #20085, #20086 **What happened?** Closed restore sources and done-task warm sandboxes stay suspended until their environment is deleted, costing storage. No path compares them with the last durable save or exports their unsaved files before retirement. **Steps to reproduce** 1. A run fails; its sandbox is kept as a `retain_on_failure` restore source. 2. The user clicks Retry; a fresh run succeeds and the task is done. 3. The restore-source sandbox is never retired until the environment is deleted. **Expected behavior** After a clean compare, or a confirmed export of anything unsaved to the issue, the sandbox is destroyed. If salvage fails, it stays suspended and is retried. ## What Changed - `doc/plans/2026-11-08-restore-source-salvage-and-warm-retirement.md`: a design for one owned kind (`environment_kept_sandbox_retirement`) on the shared reconciler. It covers listing rules for both cases (24 h default retention for done-task warm sandboxes), a three-stage salvage state machine (compare, export, retire) with a provider-I/O step and receipt table, failure handling that never destroys without proof, and the test plan. ## Verification Docs only; no code or tests changed. ## Risks None at runtime. Additive documentation. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool use and code execution, via Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
d007afbf86 |
fix(native): let a follow-up wait out the previous turn's warm release instead of failing busy (#20068)
## Thinking Path - With a warm lifecycle, one session serves consecutive turns on a task. A follow-up comment wakes the next run, which claims the warm session's instruction copy and reuses the session. - #20048 makes warm the default on remote sandboxes. Its merge condition includes "no restart or follow-up ever leaves a task needing Retry". - In the #20048 Daytona chaos wave, task t12's follow-up run was created 43 ms after the previous run settled. The previous run was still releasing its warm session, so `claimWarmNativeInstructionCopy` saw it busy and failed the run with `native_session_supervisor_busy`. The task then needed a Retry. - The release takes seconds and always finishes, so a follow-up should wait for it rather than fail. A session busy because another run is preparing or executing a turn is a genuine conflict and must still fail at once, as existing tests require. ## Linked Issues or Issue Description Refs #20048 **What happened?** Wave w-20261108T142906-s9501, t12 (claude, warm): run 5b839da4 succeeded at 14:46:36.224. The follow-up run f2bf2f05 was created at 36.267 and failed immediately with `adapter_failed` / `native_session_supervisor_busy`, from `claimWarmNativeInstructionCopy` called by `instruction-working-copy.prepare`. Its automatic retry also failed, and only a user Retry completed the task. **Steps to reproduce** 1. Run a warm native session turn to completion. 2. While that run is still releasing its warm session (post-turn instruction check, release, hand-back), admit the next run on the same session. 3. The next run's instruction-copy claim throws `native_session_supervisor_busy`. **Expected behavior** The follow-up waits for the previous run's release (bounded) and reuses the session. A session held by a run that is still preparing or executing fails admission at once, as before. ## What Changed - `server/src/services/native-runtime/native-session-executor.ts`: - When the provider turn ends, the executor records the session scope as settling (`settlingNativeSessionScopes`). The outer `finally` clears the mark once the run has handed every scope back. - `claimWarmNativeInstructionCopy` waits, polling every 25 ms for at most 30 s (`warmSessionReleaseWait`), while the warm entry is busy and its scope is settling. It then re-reads the entry and runs the existing checks. If the entry is still busy at the deadline, or busy for any other reason, it throws `native_session_supervisor_busy` as before. - No wait happens on the normal path (entry idle), so turn admission adds no latency. - `server/src/services/native-runtime/native-session-executor.test.ts`: two cases. - A follow-up claimed while the previous turn's post-turn check is held: it waits, then succeeds once the release finishes. It fails on master with the wave's error. - The release outlasts the bound: it fails busy. **Previous-release state:** the change touches only in-memory admission state, so no saved run or session state changes. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts src/services/heartbeat/instruction-working-copy.test.ts`: 1301/1301 pass. - With the wait removed (behaviour on master), the "release finishes" case fails with `native_session_supervisor_busy`. - The existing concurrency tests ("fences owned instruction preparation: concurrent", and the settled-release ownership tests) still fail admission at once. - Not yet covered: the Daytona warm wave rerun for #20048. ## Risks - **A wait that never ends.** The wait is bounded at 30 s and happens only while the owner is past its turn, which always reaches the outer `finally`. Every caller of the inner executor runs inside that `try`, and attachment staging keeps the same session scope. - **Hiding a real conflict.** Only a settling owner is waited for. A preparing or executing owner fails at once, as the existing tests assert. - **Scope.** I tried a similar wait at execution admission, but it conflicted with deliberate fences that reject a run during a settled release (the existing "releases settled codex only under its current owner" tests). It is not part of this PR. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool use and code execution, via Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
c3c226d165 |
test(restart-chaos): governance (approval, budget hard stop) and upgrade mid-run scenarios (#20089)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart chaos harness kills and restarts the control plane
during real agent tasks and grades every task
> - It covers run lifecycle, comments, delegation and long tools, but
not two areas that must also survive a restart: governance (approvals
and budget hard stops) and an upgrade in the middle of a run
> - An approval that disappears, runs twice or runs after a rejection,
or a budget pause that dispatches work, breaks governance. An upgrade
that cannot resume saved state breaks the upgrade-safe-saved-state rule
> - This pull request adds the harness support for both scenarios, using
only the paths a real board user and agent use
> - The benefit is that the next chaos waves can test these boundaries
and report failures with evidence
## Linked Issues or Issue Description
Refs #20084, #20067
**What happened?**
The restart chaos harness had no scenario for an approval-gated action
pending across a restart, for a budget hard stop crossed mid-run with a
restart, or for a server upgrade in the middle of a run. These
boundaries were untested.
**Expected behavior**
- **G1:** a pending approval survives K1, K2 or K4. Approving it after
the restart runs the gated action exactly once; rejecting it never runs
it.
- **H1:** a budget hard stop crossed mid-run pauses the company, nothing
dispatches while paused (also across a restart), and resuming continues
the work.
- **K14:** a wave started on an older build and upgraded mid-turn
resumes old-saved state on the new build with no Retry, no lost comments
and no stuck records.
**Steps to reproduce**
1. Run the harness commands in `tests/restart-chaos/README.md`
("Governance across restarts" and "Upgrade mid-run").
2. Each task is graded R1–R6 plus its oracle.
**Paperclip version or commit**
master at
canary/v2026.1108.0-canary.21
|