Commit Graph
621 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 bfa9002a3c test(restart-chaos): reach the stop-and-retain crash point with --daytona-reuse-lease (#20049)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart-chaos crash-point suite kills the server at named
points, then checks that recovery restores every invariant
> - On Daytona, `environment.stop_retain.after_claim_before_stop` was
never reached, because no workload produced a stop-and-retain release
> - A per-turn run on a reusable sandbox lease ends every turn with
`stop_and_reuse`, which releases through exactly that stop-only claim
> - This pull request adds `--daytona-reuse-lease`, so the chaos Daytona
environment keeps reusable sandboxes and the point is reachable
> - The benefit is live coverage of the stop-and-retain crash window

## Linked Issues or Issue Description

This is a follow-up from the Phase 2 durable-ownership data
(`doc/plans/2026-11-07-phase2-reconciler-data.md`, "Not covered").
Related: #19937 (crash-point suite) and #20042 (stop-only claim fence
renewal). No duplicate PR was found.

**What happened?**

The Daytona crash-point run `cp-20261108T104102` reported
`environment.stop_retain.after_claim_before_stop` as `not_reached`. The
point needs a stop-and-retain intent, and the chaos Daytona environment
is created with ephemeral leases.

**Expected behavior**

A flag lets a Daytona wave or crash-point run use reusable leases. A
per-turn run then stops and keeps its sandbox, and the point is reached
and graded.

**Steps to reproduce**

1. Run `node scripts/crash-points.mjs --env daytona --daytona-image
<image> --points environment.stop_retain.after_claim_before_stop
--adapters codex --workloads A1`.
2. Before this PR the point is `not_reached`.
3. With `--daytona-reuse-lease --lifecycle per_turn` it is reached.

**Paperclip version or commit**

`master` at `8475826d2d` (the signed image `c24059f1…` revision).

## What Changed

- `--daytona-reuse-lease` (wave and crash-point suite, Daytona only)
creates the chaos environment with `reuseLease: true`. The crash-point
suite passes the flag through.
- The R6 sandbox sweep counts a **stopped** sandbox behind a **released
`reuse_by_environment`** lease as kept for reuse, not a leak. A later
run on the same workspace resumes it, because reuse accepts released and
retained leases. A running sandbox, or a stopped one behind an ephemeral
lease, is still a leak.
- The crash-point registry prerequisite and the README name the flag.

## Verification

- Live, Daytona, codex A1, from the image revision `8475826d2d`:
- `cp-20261108T132149`: the point was **reached** and settled with no
invariant violations. R6 flagged the stopped reusable sandbox as a leak,
which led to the sweep fix.
- `cp-20261108T133351`, with the sweep fix: **pass**. 0 orphaned
sandboxes, 0 held leases, the environment deleted, and nothing left
running.
- `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`
passes `daytona-sweep`, `scheduler` and `crash-points` (52/52). `tsc -p
tests/restart-chaos/tsconfig.json` is clean.

## Risks

- Test harness only; the registry change is prerequisite text. Without
the flag, waves behave as before.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with
tool use.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 08:07:19 -06:00
DottaandClaude Opus 5.5 c966334ec3 fix(recovery): settle a run-less lease once its stamped holder is proven gone (#20026)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The control plane recovers in-progress records after a server
restart through a shared ownership path
> - Run-less sandbox leases (login, probe, harness verification) were
released only by a five-minute idle sweep
> - That sweep releases by age: it leaves a dead holder's sandbox for
minutes and takes a live login session's sandbox
> - This pull request settles a run-less lease only when its stamped
holder is proven gone
> - The benefit is no stuck sandbox after a restart, and no live session
loses its sandbox

## Linked Issues or Issue Description

Related: #19965 (named the owner of login/probe leases), #19964 (the
run-lease settle this mirrors). No duplicate PR was found.

**What happened?**

The final Daytona chaos wave `w-20261108T083036-s9401` recorded I2
`environment_login_probe_lease` (`pid_dead`) on 2 records. These were
the codex agent's harness-verification leases. A restart killed their
holder. The leases stayed `active` until the five-minute idle sweep
released them. The same sweep releases any run-less lease after five
idle minutes, also when its holder is alive.

**Expected behavior**

A run-less lease is released as soon as its stamped holder is proven
gone. A lease whose holder is alive is never released, however long it
is idle.

**Steps to reproduce**

1. Start a Daytona wave: `node cli/node_modules/tsx/dist/cli.mjs
tests/restart-chaos/main.ts --env daytona --daytona-image <image>`
(K1/K4 faults).
2. Let a fault kill the server while an agent's harness verification
holds its sandbox lease.
3. See the reconciler report I2 `environment_login_probe_lease` until
the idle sweep releases it.

**Paperclip version or commit**

`master` at `8475826d2d`.

## What Changed

- New `settleEnvironmentLoginProbeLease`, registered on
`environment_login_probe_lease`. It runs only for a stamped holder that
is proven gone, and compares and swaps on that exact stamp.
- It moves the lease to `pending_cleanup` with the recovered-orphan
reason, so the cleanup sweep destroys the sandbox in its next tick.
- It leaves the lease alone when a legacy shutdown fence applies, or
when another live lease holds the same provider resource.
- The idle sweep for run-less leases now covers only legacy rows saved
before holders were stamped (no `leaseController` key). Stamped rows are
never released by age.
- `holderUnchanged` is now exported from the run-lease settle so both
settles share it.
- The controller stamp (`leaseController`, and every other
`ControllerProcess` stamp) now records the platform deployment when the
platform names one (Railway: project, environment, service, deployment,
volume). `controllerStamp` passes it to `proveOwnerGone`. A container
redeploy changes the PID space, and the old stamp would read
`different_process_space` (unprovable) forever. Now the existing
`deployment_replaced` proof shows the old holder gone. This also covers
`environment_run_lease_active` and the other kinds built on
`controllerStamp`.
- `currentDeploymentIdentity` in `owner-proof.ts` now delegates to
`currentControllerDeployment`, so there is one parser.
- Data doc: added the final Daytona wave, the six Daytona-only crash
points (4 pass, 2 not reached, no violations), and the cause of the
roughly 450 s codex TTFT. Runs waited behind the agent's harness
verification, which restarted after each fault; boot itself took about
32 s.
- Restart-chaos README: corrected which Daytona crash points a plain
task reaches.

## Verification

- Red, before the change:
`environment-login-probe-lease-ownership.test.ts` failed 3 of 6. No
settle happened, and the idle sweep released 3 live holders' leases.
- Green: `pnpm exec vitest run` passes for
`environment-login-probe-lease-ownership`,
`environment-run-lease-ownership` and
`heartbeat-orphaned-active-lease-sweep` (45/45), and for
`environment-lease-owner-stamps`,
`heartbeat-observed-workspace-preservation`,
`ownership-reconciliation-report`, `heartbeat/recovery` and
`workspace-recovery-warm-handoff`.
- Redeploy: a stamp from a replaced Railway deployment settles with
`deployment_replaced`. A stamp from the same deployment in another PID
space stays `active`. This was red when `controllerStamp` dropped the
deployment.
- Live-owner counter-cases: the current process, another live PID,
another PID space, and a lease re-stamped after the proof all stay
`active`. A read-only pass never settles.
- Not re-run live: the Daytona wave and crash points ran before this
fix. The next Daytona wave should show no
`environment_login_probe_lease` I2.

## Risks

- A stamped run-less lease whose live holder never releases it, because
of a bug in that process, is no longer released after five idle minutes.
It is released once that process exits. This is the intended trade:
never take a live holder's sandbox.
- On an ephemeral server without a platform deployment identity (not
Railway), a stamp from a replaced container is still unprovable. Such a
run-less lease stays `active` until the provider expires the sandbox,
and the reconciler reports it as unprovable. Before this PR, the
five-minute sweep released it by age. This is the same open gap as
unstamped legacy leases on an ephemeral server: there is no operator
action yet. It is listed as a Phase 2 follow-up.
- Legacy unstamped run-less leases keep the age fallback. These are rows
written before #19965, and no current code writes them.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with
tool use.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 06:43:45 -06:00
DottaandClaude Opus 5.5 022c73e475 fix(restart-chaos): retry a failed image copy from an empty directory; share the image cache (#20021)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart-chaos harness tests recovery on Daytona sandboxes
> - The harness extracts the runner from the Daytona image with skopeo
before a wave starts
> - A skopeo blob read that timed out failed the whole run before any
test started
> - This pull request makes the image extraction retry cleanly and share
one cache
> - The benefit is that one network hiccup no longer costs a 20+ minute
run

## Linked Issues or Issue Description

Related: #19544 (Daytona chaos waves without Docker). No duplicate PR
was found.

**What happened?**

The Daytona wave `w-20261108T082021` and the crash-point run
`cp-20261108T101304` stopped before any test ran. skopeo reported `read:
operation timed out` while it copied a blob. The retry then reported
`not a containers image directory, don't want to overwrite`, because the
destination already had partial files. The crash-point suite also used
its own image cache, so it downloaded the image again on each run.

**Expected behavior**

A failed copy retries into an empty directory. Every wave and
crash-point run reuses an image that another run already extracted.

**Steps to reproduce**

1. Run `node scripts/crash-points.mjs --env daytona --daytona-image
<image>` on a slow or unstable network.
2. Let skopeo time out on one blob read.
3. See the harness stop with exit 2 before any point runs.

**Paperclip version or commit**

`master` at `8475826d2d`.

## What Changed

- With a shared cache, two runs can extract the same image at once. The
final cache directory is created only by an atomic rename, and no run
deletes a cache entry. A run that loses the rename adopts the valid
entry. An invalid leftover stays in place, and the run publishes its
extraction to a unique sibling directory.

- `copyImageToDir` runs up to 3 `skopeo copy` attempts. Each attempt
starts from an empty destination directory.
- `defaultDaytonaImageCacheRoot()` is now the one cache for both the
wave harness and the CLI: `tests/restart-chaos/results/daytona-images`
by default, or `RESTART_CHAOS_IMAGE_CACHE` if set. Crash-point runs
reuse an image that a wave already extracted.
- README updated.
- Three more tests: two runs against one cache through a barrier, a
corrupt leftover, and two runs that both meet a corrupt leftover.
- Unit tests use a fake skopeo that behaves like the real one: it
refuses a non-empty destination and leaves partial blobs behind when it
fails.

## Verification

- `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts
tests/restart-chaos/daytona-image.test.ts`: 17/17 passed.
- `tsc -p tests/restart-chaos/tsconfig.json`: no errors in
`tests/restart-chaos`.
- Live run: the six Daytona crash points were rerun with this patch,
reusing the cached image.

## Risks

- Low. Test harness only; no product code changes.
- A shared cache means all runs depend on one receipt per image digest.
Receipts are already written atomically by renaming the staging dir, and
each one is keyed by the immutable digest.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with
tool use.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 06:00:48 -06:00
DottaandClaude Opus 5.5 8475826d2d fix(wake-queue): deliver user comments whose wake a restart lost or a stale continuation dropped (#19992)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A user's follow-up comment on a task must reach the agent exactly
once, through restarts (invariant 4)
> - The final unmasked local chaos storm (96 tasks, 31 restarts) passed
94 tasks. The 2 failures were both lost user comments
> - In t69, a restart killed the server between saving the comment and
writing its wake. In t72, a comment's own run was cancelled as stale,
because the earlier run completed the task first without reading the
comment
> - This pull request adds a reconciler rule that gives a lost comment
its wake, and saves the comment before a stale cancellation
> - The benefit is that a user's message is never silently dropped when
a restart or a completion race hits at the wrong moment

## Linked Issues or Issue Description

Refs #19956 (work-intent outbox), #19944, #19831 (plan, invariant 4).

**What happened?**
In chaos wave `w-20261108T055444-s9401` (local, unmasked, codex and
claude, 30-min K13 storm, master `46bf913fb9`), 94 of 96 tasks passed.
Two failed:
- **t69, R1 `rested_not_done`.** The user's comment was saved at
06:14:59.358 and reopened the task to `todo`. A K1 SIGKILL fired 28 ms
later, before the wake was written. No wake row exists. Recovery skips a
`todo` task whose last run succeeded, so the task sat with no run until
the wave ended.
- **t72, R4 `concurrent_runs`.** A comment posted during a run queued
its own run. That run was claimed at 06:16:04.874. The first run's
completion marked the task `done` 2 ms later. The comment's run was then
cancelled with `continuation_task_ownership_changed` before dispatch,
and the comment was lost.

**Expected behavior**
Each accepted user comment is delivered once, unless the task is
cancelled or reassigned, or a person closes it.

**Steps to reproduce**
`node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts
--replay w-20261108T055444-s9401/t69` (or `.../t72`).

## What Changed

- `recovery/ownership-invariant-report.ts`: a new reconciler-owned I4
rule, `user_comment_without_wake`. It finds a user comment that meets
all of these conditions:
  - it is newer than every run of the task;
- the task is assigned, is not a conversation, has no live run or lock,
and has at least one earlier run;
  - no wake request carries the comment.

A comment younger than 30 s is not reported yet, because it may still
get its normal wake. Its end-of-grace time is returned as `nextDueAt`.
- `native-runtime/native-finalization-reconciler.ts`: the backstop step
is now exported as `applyOwnershipReportBackstops`. For each such
comment it saves the missing outbox receipt (`executionWait: wake_lost`)
and promotes it.
- `work-intent-outbox.ts`: new `saveUndeliveredUserComments`, which the
reconciler rule and the stale-run path both use. It saves only live user
comments that no completed run received. It writes nothing for a task
that is reassigned, is a conversation, or has a status the caller does
not allow.
- `heartbeat/queue.ts`: the saved-receipt pass that already sends
pause-saved messages also sends `wake_lost` receipts. It uses the same
per-comment consumption check and the same chat-input exclusion.
- `heartbeat/stale-comment-continuation.ts` and `heartbeat.ts`: before a
stale continuation is cancelled, its undelivered user comments are saved
as a deferred receipt. The release drain then reopens the `done` task
and delivers them once. This applies only when an agent run that started
before the comment completed the task after it. A task that a person
marks done is left closed.
- `tests/restart-chaos/grader.ts`: R4 ignores runs cancelled before
dispatch (`continuation_task_ownership_changed`) when it looks for
overlapping execution. Those runs never executed. Comment consumption
still counts them.

## Verification

Each focused file ran alone in one process (`--maxWorkers=1`):
- `heartbeat-held-comment-wake.test.ts` (22 pass). This includes two new
cases:
- t69 shape: the reconciler rule fires, the backstop delivers the
comment once, and a second pass does nothing;
- t72 shape: the stale run is cancelled, the task reopens, and the
comment is delivered once.

  Both fail with the rule or the save disabled, and pass with them.
- `ownership-reconciliation-report.test.ts` (23 pass). It includes the
rule's boundaries: a carried comment, a too-young comment (deadline
only) and an already-read comment.
- `tests/restart-chaos/grader.test.ts` (17 pass). The new R4 case fails
with the old grader. A cancelled run that is not a pre-dispatch
cancellation still fails R4.
- Unchanged and passing: comment wake batching (27), refused assignment
wake (3), wake-queue postgres adapter (103), stale queue invalidation
(68), deferred receipt work (45). The first version also reopened a task
that a person had marked done, which broke one deferred-receipt test.
That case is now excluded.
- The server package typecheck and the restart-chaos harness typecheck
pass.
- Same-session chaos context on master: K6 crash loop
`w-20261108T063326-s9402`: 6 of 6 pass. The K4 long-hold wave is still
running. After merge, I'll replay only the targeted fault shapes (t69
and t72).

## Risks

- The reconciler rule runs on each report pass. It is bounded by the
1-hour lookback and the per-kind row cap, and it excludes conversation
tasks, so chat input keeps its own authorization.
- A comment younger than 30 s waits up to the end of its grace period
before the backstop acts.
- The stale-run save reopens a `done` task only for a comment that the
completing run could not have read.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 02:14:45 -06:00
DottaandClaude Opus 5.5 68d5fb5a6e feat(chaos): check I6 and I8 per crash point; ready the Daytona points (#19984)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The on-demand crash-point suite (#19946) kills the control plane at
named durable write boundaries and checks the plan's invariants
> - Its first full run measured I1–I4 and I7, but I5, I6 and I8 were
reported as not measured
> - #19960 (generation stamping) and #19863 (speed budgets) now record
what I6 and I8 need in the run log
> - The six Daytona-only points have not run yet, and three of them
cannot be reached by a plain task
> - This pull request checks I6 and I8 per point, names the I5 gap, and
readies the Daytona points for the coming Daytona wave
> - The benefit is that every invariant is either asserted or a named
gap, and a Daytona run explains any point it cannot reach

## Linked Issues or Issue Description

Refs #19831 (plan, Phase 5). Follows #19946 (the suite). Uses #19960
(event generations) and #19863 (speed budgets).

**Problem**
- The suite reported I5, I6 and I8 as "not measured".
- A Daytona run would report three points as `not_reached` with no
reason.
- A Daytona run without its key or image failed only after a 3-minute
boot.

**Expected behavior**
- I6 and I8 are checked at each point.
- I5 is a named gap with its reason.
- Daytona points say what they need, and a Daytona run checks its inputs
before it boots.

## What Changed

- `tests/restart-chaos/invariants.ts`: `runLogViolations` checks two
invariants from the run log, scoped to the wave's company:
- I6: a run that failed with an error that names a stale or
earlier-generation event;
- I8: each `run.performance.budget_exceeded` event, with the budget
evidence (metric, measured, p95, adapter, start kind).
Both are keyed by run or event id, so the per-point baseline diff
charges each one to the point that produced it. I5 is reported as
`notMeasured` with a reason (`I5_GAP`).
- `packages/shared/src/crash-point-registry.ts`: an optional
`prerequisite` on the three Daytona points a plain task cannot reach.
These are the teardown claim (needs a failed release), the reusable
destroy (needs a `reuse_by_environment` lease), and stop-and-retain.
- `tests/restart-chaos/crash-points.ts`:
- `--list` prints each point's `needs:`, and a `not_reached` result
names it too;
- `--env daytona` refuses to start without `DAYTONA_API_KEY` and an
image.
- Docs: the README and the plan describe the I5/I6/I8 coverage and the
Daytona points.

## Verification

- `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts
tests/restart-chaos/crash-points.test.ts`: 12 passed.
- `cd packages/shared && npx vitest run src/crash-points.test.ts`: 15
passed.
- `node scripts/typecheck-test-projects.mjs tests/restart-chaos` and
`npx tsc --noEmit -p packages/shared` both pass.
- I ran `runLogViolations` against a throwaway Postgres 18 with seeded
rows, then stopped and deleted it. It returned:
  - I8 for the budget breach, with its evidence;
- I6 for the run of this company that failed on
`stale_generation_event`.
It excluded the other company's stale run, the `process_lost` run and a
non-budget event.
- `node scripts/crash-points.mjs --list --env daytona` lists 19 points,
with `needs:` on the three. `--env daytona` with no key or image fails
at once with the missing names.
- No harness run: the harness slot is held for other work. The next
suite run exercises the new checks.

## Risks

- Test harness only; no product code.
- I6 is a proxy: it matches the run's error text and code. A failure
that names a generation for another reason would be reported, with its
evidence, for a person to judge.
- I8 can flag breaches on a loaded host. Each breach carries its
measured and budget values, so a load-caused breach is visible as one.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 00:06:01 -06:00
DottaandClaude Opus 5.5 051834bdc8 feat(chaos): add the on-demand crash-point suite and its manual CI workflow (#19946)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A control-plane restart between two durable writes leaves an
in-progress record owned by a dead process
> - Random restart-chaos waves found these 20 ms–1 s windows only by
luck
> - #19937 added `crashPoint(name)`, and #19938 put it at 20 durable
write boundaries
> - Nothing yet runs a task, kills the server at a chosen point,
restarts it, and checks the plan's invariants
> - This pull request adds that on-demand suite, as a local command and
as a manual CI workflow (not a required check, per Decision 2)
> - The benefit is a repeatable work list: which boundary breaks which
invariant, for Phases 2–4

## Linked Issues or Issue Description

Refs #19831 (plan, Phase 5 and Decision 2). Refs #19861 (report-only
reconciler used for I1–I7). Stacked on #19938 (call sites), which is
stacked on #19937 (helper).

**Problem**
The named crash points cannot be exercised end to end. No command arms
one point, runs real agent tasks across it, restarts, and grades the
result.

**Expected behavior**
`node scripts/crash-points.mjs --points <glob> --kinds <id> --adapters
<list>` runs one wave per selected point and reports each point as pass,
fail (with failing R rules and violated invariants), not reached, or
harness error. A `workflow_dispatch` workflow runs the same thing on
demand.

## What Changed

- **Runtime arming.** `PAPERCLIP_CRASH_POINT_CONTROL=<file>`, in the
shared helper and the runner mirror. Each hit reads `<token>
<name>[:<nth>]`, and a new token resets the count. It is test mode only:
`NODE_ENV=production` ignores it, and the disarmed path is still one
comparison.
- **`tests/restart-chaos/main.ts --crash-point
<a>:<nth>,<b>:<nth>,...`** (config, types, ledger):
  - one booted server and no Vite UI;
- per point: take an invariant baseline, launch the point's batch, arm
the point, then on rest disarm it and check invariants as violations
that are new since the baseline;
  - every task is graded R1–R6 and T;
  - `crash-point.json` (v2) holds one entry per point.
- **`tests/restart-chaos/supervisor.ts`:**
  - every spawn gets the control file;
- a SIGKILL whose receipt names the child's pid is restarted like K1:
the control file is emptied first, the owned tree is killed, and a
replacement starts. A failed tree cleanup fails the fault;
- the isolated instance pins `PAPERCLIP_IN_WORKTREE=false` (a
`.paperclip/.env` above the checkout otherwise suppressed all
scheduling);
  - `RESTART_CHAOS_HEALTH_TIMEOUT_MS`.
- **`tests/restart-chaos/invariants.ts`:**
- the Phase 1 report-only pass with `record: false`: two real-clock
passes for I1 and I2, and one pass 15 minutes ahead for I3, I4 and I7;
  - spool kinds are read from the wave's instance;
  - an unfinished pass is a harness error;
  - each violation carries its wake and run state as evidence.
- **`tests/restart-chaos/crash-points.ts` and
`scripts/crash-points.mjs`:**
- select points by glob, kind and environment, and run them in one wave;
  - a per-point outcome: pass, fail, not_reached or harness_error;
  - `SUMMARY.md` with a violation evidence column.
- **`.github/workflows/crash-points.yml`**, manual only, under the
paid-workflow contract:
  - a credential-free `authorize` job;
  - a `suite` job in `runner-e2e-paid` whose first step reauthorizes;
  - a frozen install with `--ignore-scripts` and no shared cache;
  - only the selected adapters' keys;
  - the workflow is added to both security-test allowlists.
- **Docs:** "Crash-point suite" in `tests/restart-chaos/README.md`, and
"Built (Phase 5)" plus the first work list in the plan.

## Verification

- Focused tests:
- `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts
tests/restart-chaos/crash-points.test.ts`;
- `cd packages/shared && npx vitest run src/crash-points.test.ts`: 15
tests, including runtime arming in a real child process;
- `node --test scripts/__tests__/release-verify-workflow.test.mjs`: 72
tests;
  - `tests/runner-e2e/workflow-security.test.ts`: 19 tests.
- Typecheck: `node scripts/typecheck-test-projects.mjs
tests/restart-chaos` passes.
- Loop line: question "how long is one crash-point loop?" | loop: `node
scripts/crash-points.mjs --points <one>` (codex A1) | elapsed 2.7 min
end to end: boot 9 s, crash 9 s after arming, replacement ready in 6 s,
rest about 2 min | result: one point alone takes about 3 min. Before
this change it took about 15 min (one cold boot per point, Vite UI, host
at load 77).
- Full local run, 14 points, codex A1, 11.8 min in all:
- every point was reached (the server killed itself at each one, and the
replacement came back);
  - 11 pass;
- `workspace.finalize.after_start_before_copyback` and
`workspace.finalize.after_copyback_before_done` fail I2 and R6: a
`workspace_finalize` operation stays running after the run succeeded;
- `wake.claim.after_run_running_before_wakeup_claimed` reports I4. Its
evidence shows the wake failed with "…retrying once", and the retry
delivered the work, so this is the known refusal-regex false positive.
- The 6 Daytona-only points were not run.

## Risks

- Test harness and an on-demand workflow only. No product code changes.
- `.github/**` needs a CODEOWNER review. The workflow uses provider
keys, so it refuses non-default branches and actors that are not on the
allowlist, like `runner-full-stack-e2e.yml`.
- `PAPERCLIP_IN_WORKTREE=false` is pinned in every chaos wave, not only
crash-point waves. A chaos instance is never a worktree dev instance, so
scheduling must not be suppressed.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 23:41:15 -06:00
DottaandClaude Opus 5.5 4aed3056db feat(ui): steer a queued follow-up into the live turn after a 5 s edit window (#19970)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a person comments on a task while the agent works, the comment
waits in a queue until the current run ends, and only the next run reads
it
> - Measured locally, a comment posted during a Codex tool call waited
54–68 s for a reply (median 64 s)
> - Codex and Pi can take a message into the live turn ("Steer"), but
today the person must click Steer on each queued message
> - Restart recovery now delivers a steered message that a restart
interrupted (#19962), so steering is safe to do automatically
> - This pull request steers a queued follow-up into the live turn on
its own after a 5-second window in which the person can edit or cancel
it
> - The benefit is a faster reply to mid-turn comments on Codex and Pi,
with no new decision for the person

## Linked Issues or Issue Description

**Feature**
Send a queued follow-up into the running turn automatically when the
runner supports steering.

**Problem**
A follow-up comment posted during a running turn waits for the whole
turn to end and for a new run to start. Steering is available for Codex
and Pi, but only as a manual button on each queued message.

**Proposed behavior**
After a 5-second edit window, the queued message is steered into the
live turn. During the window the row shows "Sending in Xs" with Edit and
Cancel. If steering is unavailable, the message stays queued as before.
Claude, OpenCode and Cursor keep the queue. Steer and Interrupt stay
available as buttons.

Refs #19962

## What Changed

- `queued-message-auto-steer.ts`: one constant,
`QUEUED_MESSAGE_AUTO_STEER_DELAY_MS = 5_000`, a deadline helper and an
eligibility check. A message steers automatically only when all of these
are true:
- steering is available for the queue, and it is a `paperclip_runner_v1`
queue with a queue id and a target run;
  - the person can edit the message;
- it is not an interaction response or a response that requires a fresh
session;
- it is not optimistic, cancelled, already attempted, or open in the
editor.
- `TaskChatQueuedMessages`: countdown row "Sending in Xs" with Edit and
Cancel in place of Steer and the "…" menu. The deadline is `max(message
changed, first eligible) + delay`, so an old queued message does not
send the moment its turn becomes steerable, and an edit restarts the
window. When the deadline passes, the row uses the existing Steer
action, so the server path, idempotency and fallback stay the same.
Cancel returns the row to the normal Steer button. The countdown is
announced once to screen readers, not on every tick.
- `TaskChatThread` turns it on with the constant and passes the message
being edited, so an open edit never sends. The prop stays off by default
for other callers.
- Storybook: `AutoSteerCountdown` (with a play test for Cancel) and
`AutoSteerCountdownMobile` in "Composer / Steering and queued messages".
- Restart chaos TTFT driver: a `midturn` phase measures comment-to-reply
for a comment posted during a 40-second tool call, with `--midturn-mode
queue|steer` and `--midturn-fault K1|K2|K6`. It records the reply count,
so exactly-once delivery is checked.

## Verification

- `pnpm --filter @paperclipai/ui vitest run
src/components/task-chat/TaskChatQueuedMessages.test.tsx
src/components/TaskChatThread.test.tsx`: 434 passed. Seven new
auto-steer tests use fake timers: countdown and send, Cancel, Edit
restarts the window, an open edit never sends, ineligible queues stay
manual, no instant send for an old message, one attempt per revision.
They fail with the trigger disabled.
- `node scripts/check-token-gates.mjs`: clean.
- Storybook in the browser, desktop and mobile: the first row counts
down and steers ("Message steered into the active turn."). The cancelled
row shows the normal Steer button. On mobile the text truncates and the
row fits.
- Mid-turn latency, local, Codex, comment during a 40-second tool call:
queue 54–68 s (median 64 s), steer 47–61 s (median 50 s). The steer was
acknowledged in 0.8–0.9 s. The remaining time is the tool call and the
model, not the hand-off.
- Restart chaos, Codex, steer mode, faults K1, K2 and K6, 2 samples
each, on #19962 plus this branch: 6 of 6 messages acknowledged, still
unread at the fault, recovered, and answered exactly once (before
#19962: 0 of 6).

## Risks

- A message now leaves the queue after 5 seconds unless the person edits
or cancels it. The window and the countdown make this visible, and
steering is only automatic where the provider supports it.
- Pi: a steer the model already read before a restart can arrive again
after recovery, because Pi reports no read evidence (see #19962).
Automatic steering makes this a little more likely.
- If steering fails, the message stays queued and is delivered by the
next run, as today.
- UI only and a test harness. No schema, API or server change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file editing, browser checks, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 22:59:44 -06:00
DottaandClaude Opus 5.5 f35eb1e5ef fix(observability): judge the TTFT budget from run creation; report the wait behind a live turn (#19941)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Invariant 8 of the durable-ownership plan says turns must start
fast. #19863 added per-run speed budgets that write
`run.performance.budget_exceeded` to the run log.
> - The Phase 2 chaos data (#19927) showed follow-up TTFT about 3x over
the claude budget.
> - The cause is measurement scope. A follow-up comment posted during a
live turn waits for that turn to end, and TTFT is anchored at the
comment. The budgets came from idle follow-ups.
> - This pull request judges the TTFT budget from run creation and
reports the wait separately.
> - The benefit is that budget breaches point at real start-up slowness,
not at time spent behind a turn the user is already watching.

## Linked Issues or Issue Description

Refs #19863 (speed budgets), #19927 (Phase 2 data), #19852 (startup
sub-spans), #19888 (warm default).

**What happened?**
In two local storm waves, C1 follow-up runs breached the claude warm
TTFT budget (p50 27.6–29.9 s against 9.5 s). Their spans show
`comment.to_run_created` at 6–62 s (waiting for the live turn), then
session resume at 1.4–3.8 s and submit-to-first-event at 2–5 s. Codex
without the wait was about 4.8 s, the same as the warm-follow-up
baseline.

**What did you expect?**
The budget should judge start-up latency, which is what its baseline
measured. The wait behind a live turn should be visible but not judged.

**Steps to reproduce**
1. Run a local chaos storm with C1 tasks.
2. Read `run.performance.budget_exceeded` rows with metric `ttft` for
`issue_commented` runs, and compare them with each run's
`comment.to_run_created` span.

## What Changed

- `native-run-trace.ts`: the TTFT budget check measures from run
creation (start of `heartbeat.queue`, or end of the ingress span). The
TTFT span itself does not change.
- `native-speed-budgets.ts`: breaches can carry `inputWaitMs` (TTFT
only).
- `tests/restart-chaos/ttft-baseline-stats.ts` and `ttft-baseline.ts`:
the wait behind a live turn is its own measured step (`inputWait`), next
to the full TTFT and the TTFT from run creation. The TTFT table gains a
"Wait behind live turn" column. The full comment-to-first-token time
stays the user-visible metric.
- `doc/run-log-events.md` and the plan's speed-budget section describe
the rule.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-speed-budgets.test.ts
src/services/native-runtime/native-run-trace.test.ts`: 43 pass, 2 new.
`pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts
tests/restart-chaos/ttft-baseline-stats.test.ts`: 12 pass, 1 new. They
cover a follow-up under budget after a 60 s wait (no breach), a
follow-up over budget (breach with `inputWaitMs`), and queueing after
run creation still judged.
- Server typecheck passes.

## Risks

- Low. This is a run-log-only change (AGENTS.md §5.7): no Telemetry and
no span attributes change.
- A slow wait before run creation is no longer a TTFT budget breach. The
TTFT span still records it, and the breach payload names it. Admission
holds after run creation are still judged.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edits).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 20:27:41 -06:00
DottaandClaude Opus 5.5 17b9e10252 docs(plans): Phase 2 reconciler data; let chaos K4 holds reach 600 s (#19927)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Control-plane restarts leave in-progress records that each subsystem
releases in its own way. The durable-ownership plan replaces those paths
with one shared path.
> - Phase 1 added a report-only reconciler. Its light local waves showed
zero violations, and that proves little.
> - Phase 2 moves kinds to the shared path. It needs data that shows
which kinds really leak, and the speed numbers to protect.
> - This pull request records that data from harsher waves. It also
fixes the harness so that K4 outages past the reconnect grace qualify.
> - The benefit is that Phase 2 moves kinds in order of measured risk,
against a known speed baseline.

## Linked Issues or Issue Description

Refs #19831 (durable-ownership plan), #19861 (report-only reconciler),
#19863 (speed budgets), #19797 (K4 hold cap).

**What is the problem?**
Phase 2 needs per-kind invariant data and speed data from harsh restart
waves. Only light local waves existed. In addition, a K4 hold over about
405 s always failed its fault receipt, so the 600 s cap from #19797
could not be used.

**Where is it documented?**
`doc/plans/2026-11-07-durable-ownership-and-reconciliation.md` (Phase 2
rows) and `tests/restart-chaos/README.md`.

**What should change?**
Record the data in a dated plan doc. Let chaos K4 holds run up to 600 s
and still qualify.

## What Changed

- Added `doc/plans/2026-11-07-phase2-reconciler-data.md`. It covers 5
unmasked local waves (two storms, a crash loop, two K4 runs). It gives
per-kind violation counts with example rows, owner verdicts, TTFT, boot
and restart recovery against the budgets, Phase 3 gate notes, and what
is not covered (Daytona).
- Added a short pointer to that doc in the durable-ownership plan.
- `tests/restart-chaos`: new flag `--k4-hold-max-seconds` (120–600). It
raises the upper end of the seeded K4 hold. The default range (120–300
s) does not change.
- `tests/runner-e2e/controller-outage.ts`: the receipt's ready bound is
now a parameter (`CONTROLLER_READY_BY_MS`, default 420 s). Runner suites
keep 420 s. The chaos supervisor passes hold + 420 s, so a hold past the
Runner's reconnect grace qualifies.

## Verification

- `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`:
12 files, 135 tests pass, including the new flag test.
- `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/controller-outage.test.ts`: 36 pass, including the new
long-hold receipt test.
- `node scripts/typecheck-test-projects.mjs tests/restart-chaos` and
`tests/runner-e2e`: pass.
- Live: the K4 wave at seed 9203 failed 3 of 6 tasks with the old bound
(fault error "did not produce a qualifying replacement receipt"). It
passed 6 of 6 with this fix, with holds of 294–579 s.

## Risks

- Low risk. The changes are test harness code and docs only. The runner
suites keep the 420 s ready bound by default.
- The data is local only. The doc says which kinds the waves do not
reach.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edits). The model ran the chaos waves and wrote
the analysis and code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 19:55:02 -06:00
DottaandClaude Opus 5.5 d157ce223c fix(evals): keep large failed private evidence and the merged report within the private checks (#19932)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat evaluation lane runs each chat case in a paid runner and
keeps the private evidence of each attempt in a bounded private archive
> - A failed attempt keeps its Playwright trace and video, and the
bundle has a 256 MB allowance
> - Playwright's HTML reporter copies each attachment into
`html-report/data/<sha1>.<ext>`, and the same bytes also stay in
`playwright-output/`
> - The bundle holds the original tree and the packaged tree, so a 30 MB
trace used about 125 MB of the allowance before the blob report copies
> - Large failed attempts went over the allowance, and then all of their
private evidence was lost
> - This pull request removes the `playwright-output/` copy only when
the HTML report holds the identical bytes
> - A second gap lost the whole merged campaign report of failed cells:
the report copied trace viewer font, SVG and manifest files to a place
where the trusted seal refuses them
> - This pull request stops that copy, and it names a refused report
carrier with a closed category
> - The benefit is that large failed attempts keep their evidence, the
merged report is kept, and the HTML report and its trace viewer still
open the trace

## Linked Issues or Issue Description

**What happened**
Failed chat evaluation attempts with large Playwright traces lost all of
their private evidence. The bundle went over the 256 MB private evidence
allowance. For example, the failed `tasks.organize-open-work` cell made
a 254 MB bundle. That bundle held the same 30.7 MB trace four times
(`playwright-output/.../trace.zip` and `html-report/data/<sha1>.zip`, in
both the original tree and the packaged tree), plus two blob report
copies. Larger traces went over the limit, and then the transfer refused
the whole bundle.

Also, the merged campaign report of a failed cell was refused with
`refusal: unsupported-packaged-evidence-carrier` (for example
`runs-adapters.inspect-failed-run`, run 42431162923). The cell's
evidence manifest lists its trace viewer files
`html-report/trace/codicon.*.ttf`, `playwright-logo.svg` and
`manifest.webmanifest`. `report.ts` copied them to
`normalized/evidence/<id>/attempt-1/html-report/trace/`. The
generated-report seal accepts trace viewer files only under
`html/trace/`, so it refused these three files and lost the whole
report. I reproduced this from the retained cell archive: a merge of its
blob report passes the seal rule, and the three normalized copies are
the only files that fail it.

**Expected behavior**
A failed attempt keeps its trace, and the trace uses as little of the
allowance as possible. The HTML report and the trace viewer still open
the trace for a person who downloads the bundle. The merged report of a
failed cell is kept. If the seal refuses a carrier, the refusal names
its category, not its path. No privacy, integrity or size check changes.

**Steps to reproduce**
1. Dispatch `runner-full-stack-e2e.yml` with `cadence=manual` for a
private chat case that fails with a large trace, for example
`chat-features.runner-opencode.local.tasks.organize-open-work`.
2. Download `manual-cell-<sha256(executionId)>.zip` from the private
history bucket.
3. List the archive. The trace bytes are in `playwright-output/` and in
`html-report/data/` in both `originals/` and `sealed-results/`.
4. For the report refusal: look at the "Retain complete manual merged
evidence only in restricted storage" step of the same run. It prints
`refusal: unsupported-packaged-evidence-carrier` and no
`manual-report.zip` is retained.

## What Changed

- Add `tests/runner-e2e/private-report-copies.ts` with
`dropDuplicatePlaywrightOutput`. It removes a `playwright-output/` file
only when `html-report/data/<sha1 of content><ext>` is a regular file
(not a symlink) with the same size and the same SHA-256. It writes the
removed paths, their kept paths and digests to
`snapshots/playwright-output-copies.json`.
- The function does nothing when the cell has a framework JSON report
(`framework-report.json` or `snapshots/playwright-framework.json`),
because that report registers `playwright-output/` attachments by path.
It also does nothing when `playwright-output`, `html-report` or
`html-report/data` is missing or is a symlink.
- Call it in the private manual branch of `tests/runner-e2e/launch.ts`,
before packaging and before both seals. Both evidence trees then keep
one HTML report copy of each trace and video.
- The HTML report, its trace viewer assets and the blob report do not
change.
- `tests/runner-e2e/report.ts` does not stage a cell file into
`normalized/evidence/` when its type is one the generated-report seal
refuses (the same `isStandardEvidenceFile("html-report/...")` rule). The
merged report has its own authenticated trace viewer under
`html/trace/`. The cell archive still keeps the files.
- `tests/runner-e2e/manual-private-transfer.ts` adds
`assertGeneratedReportCarriers`, which runs before the report seal. It
throws a fixed message per closed category (`font`, `svg`,
`webmanifest`, `image`, `source map`, `no extension`, `other`, `trace
viewer asset`, `trace viewer bytes`), so the refusal line names, for
example, `unsupported-generated-report-carrier-font`. It never names a
path, and the seal stays the authority.
- No change to the target-checked trusted files
(`manual-private-handoff.ts`, `manual-private-retention.ts`,
`manual-private-report-assets.ts`), so candidate branches do not need an
update. No change to the limits or the secret scans.

## Verification

- `vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/private-report-copies.test.ts
tests/runner-e2e/manual-private-cell-report-assets.test.ts
tests/runner-e2e/private-framework-report.test.ts
tests/runner-e2e/manual-private-handoff.test.ts`: 79 passed.
- The new tests use a real failed Playwright run with the HTML, JUnit
and blob reporters and a real retained trace. They show these things:
- Only the duplicate `playwright-output/` files are removed. The HTML
report and blob report bytes do not change.
- The deduplicated cell passes `packageEvidence`,
`sealStandardOriginalEvidence`, `sealManualResultEvidence`, the trusted
`validateSealedOriginals` and `validateSealedInventory`, the strict
archive pack and unpack, and `stageSealedPrivateCell`.
  - A tampered or symlinked HTML report copy keeps the original file.
  - Framework JSON cells and symlinked data directories are not changed.
- `vitest run --config tests/runner-e2e/vitest.config.ts manual-private
private-report private-refusal private-framework generated-report
report`: 25 files, 311 passed.
- `generated-report-carriers.test.ts` shows each refused carrier gets
its closed category code, the code has no path, and the real seal
refuses the same tree. `report.test.ts` shows the font, SVG and manifest
are not staged and the other files are.
- A new test runs the real launcher private branch from `launch.ts`. It
shows that the sealed originals keep one trace copy, in
`html-report/data/`.
- Live run: I applied the commit to a branch from
`chat-eval/master-candidate` (`chat-eval/case-evidence-dedupe`) and
dispatched
`chat-features.runner-opencode.local.tasks.organize-open-work` (run
42432263920). I will add the bundle size result in a comment.

## Risks

- Low risk. The change runs only in the private manual evidence branch.
It removes a file only after a full content comparison with the kept
copy. If a check fails, the file stays.
- A person who wants the trace must open `html-report/data/<sha1>.zip`
(the HTML report links to it), not `playwright-output/<test>/trace.zip`.
The receipt in `snapshots/playwright-output-copies.json` names the new
location.
- `report.ts` and `manual-private-transfer.ts` run from the master
checkout in the report job, so the report fix takes effect only after
merge. The normalized per-cell HTML report copy no longer has the viewer
font, logo and manifest. The merged report under `html/` and the cell
archive keep a complete viewer.
- The blob report still has a compressed copy of the trace, and the
bundle still holds the original and the packaged tree. A trace over
about 60 MB can still go over the allowance. This change does not fix
that case.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), run in Claude Code with
tool use (shell, file editing, GitHub CLI and AWS CLI).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 19:37:12 -06:00
DottaandClaude Opus 5.5 1de794cb80 fix(evals): name runtime-built evidence refusals by their fixed template (#19920)
## Thinking Path

- #19908 made a refused private evidence transfer print a closed refusal
code instead of one generic line.
- Its first real use printed `refusal: dynamic`: the trusted modules
build some refusals at runtime (secret-leak findings,
publication-boundary limits), and those were deliberately not named
because their text can carry paths or counts.
- The fixed part of each template is safe and is exactly what an
operator needs to know which check refused.

## Linked Issues or Issue Description

**What happened**

After #19908, a refused `runs-adapters.inspect-failed-run` attempt
(workflow run 42422571919) printed `Restricted manual evidence operation
failed (refusal: dynamic)`, so the reason was still unknown.

**Expected behavior**

A refused transfer still keeps message text private, but names which
check refused it with a closed code, so the attempt can be diagnosed.

**Steps to reproduce**

1. Dispatch `runner-full-stack-e2e.yml` manually for a chat cell whose
private evidence fails transfer validation.
2. Open the "Retain complete approved manual originals" step log: it
shows the single fixed failure line with no reason.

## What Changed

- `private-refusal-code.ts`: runtime-built refusals from the trusted
modules map to fixed template codes (`template-secret-leak`,
`template-publication-boundary-exceeded`,
`template-invalid-file-metadata`,
`template-history-expiration-unprovable`). Variable parts are never
printed.
- Tests: each template is named without its variable text, and a source
scan asserts every runtime-built `new Error` in the trusted modules maps
to a named code (not `dynamic`).

## Verification

- `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/private-refusal-code.test.ts
tests/runner-e2e/manual-private-transfer.test.ts`: 20 passed.
- `npx tsc --noEmit -p tests/runner-e2e`: clean.

## Risks

- Template codes reveal which kind of check refused (for example that a
secret-shaped value was found), not where or what. That is the intended
operator signal; the finding itself stays private.

## Model Used

Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code, with
tool use (shell, file editing).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 18:41:39 -06:00
DottaandClaude Opus 5.5 ba3fe09954 fix(evals): name why a private evidence transfer refused, without its text (#19908)
## Thinking Path

- Paid chat evals retain their evidence through a trusted transfer
(`tests/runner-e2e/manual-private-transfer.ts`) that runs from master,
separate from the target checkout.
- When the transfer refuses an attempt, it prints one fixed line,
"Restricted manual evidence operation failed; no public fallback", and
keeps nothing. The closed diagnostic on the target side cannot see this
step either.
- So a refused attempt leaves no evidence and no reason.
`runs-adapters.inspect-failed-run` lost three attempts in a row this
way, and `goals.create-goal-tree` lost one; nobody could tell which
check refused.
- This change keeps the privacy rule (no message text in public logs)
but names the refusal with a closed code.

## Linked Issues or Issue Description

**What happened**

A paid chat-eval attempt whose private evidence the trusted transfer
refused printed only "Restricted manual evidence operation failed; no
public fallback". The receipt showed `attemptedSourceBytes: 0` and
`transfer.status: failed_or_unknown`, and no evidence was kept, so the
reason was unknowable. `runs-adapters.inspect-failed-run` lost three
consecutive attempts this way.

**Expected behavior**

A refused transfer still keeps message text private, but names which
check refused it with a closed code, so the attempt can be diagnosed.

**Steps to reproduce**

1. Dispatch `runner-full-stack-e2e.yml` manually for a chat cell whose
private evidence fails transfer validation.
2. Open the "Retain complete approved manual originals" step log: it
shows the single fixed failure line with no reason.

## What Changed

- New `tests/runner-e2e/private-refusal-code.ts`:
`privateRefusalCode(error)` returns a slug of the message only when it
is a fixed literal `new Error("...")` in the trusted evidence modules
(transfer, handoff, retention, report assets, history, framework report,
history budget, evidence, redaction). Otherwise it returns `fs-<code>`
for Node error codes, `error-type-<Name>` for typed errors,
`non-error-thrown`, or `dynamic`. Message text is never returned.
- `manual-private-transfer.ts`: the failure line now includes `(refusal:
<code>)`.
- Tests: fixed literals are named; dynamic, filesystem, typed and
non-error throws never leak their text.

## Verification

- `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/private-refusal-code.test.ts
tests/runner-e2e/manual-private-transfer.test.ts`: 15 passed.
- `npx tsc --noEmit -p tests/runner-e2e`: clean.
- The three files the transfer requires to match between trusted and
target checkouts (`manual-private-handoff.ts`,
`manual-private-retention.ts`, `manual-private-report-assets.ts`) are
unchanged.

## Risks

- The public log line gains a code. Codes are either fixed literal
messages (already public in this repository's source), Node error codes,
or error type names. Dynamic messages, which can contain paths, are
reported only as `dynamic`.
- Reading the module sources once at failure time adds a small file read
on the error path only.

## Model Used

Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code, with
tool use (shell, file editing).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 16:28:03 -06:00
DottaandClaude Opus 5.5 fc6e1b0346 perf(native): keep local Codex and OpenCode runners warm by default (#19888)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - When someone messages an agent, the turn must start fast. Time to
first token (TTFT) is a first-class metric.
> - The Phase 0 baseline (#19846) found that every local turn launches a
new runner and loads the provider session again, because `per_turn` is
the default lifecycle. That made a follow-up message slower than a new
task.
> - Warm runners keep the runner and provider session alive between
runs. #19874 fixed the defect that broke warm OpenCode follow-ups, and
#19879 bounds an idle warm runner whose controller dies.
> - This pull request makes warm the default for local Codex and
OpenCode agents, and migrates existing agents that only carried the old
creation default.
> - The benefit: follow-up TTFT p50 drops from 12.0 s to 4.7 s (Codex)
and from 21.7 s to 4.2 s (OpenCode), with no new restart-chaos failures.

## Linked Issues or Issue Description

Refs #19846 (TTFT baseline), #19852 (startup sub-spans), #19874 (warm
completion contract), #19879 (warm reconnect grace).

**What existing behavior does this improve?**
The default runner lifecycle for native agents, and how a follow-up
message starts.

**Current behavior**
Agent creation writes `lifecycleMode: "per_turn"` into every native
agent. Every follow-up launches a new runner and restores the provider
session: 4.2 s (Codex) and 16.3 s (OpenCode) at p50 before the turn
starts.

**Proposed behavior**
New native agents store no lifecycle. On a local target, an unset
lifecycle resolves to `warm` with the default 5-minute idle window for
Codex and OpenCode. Other harnesses, remote targets and managed
providers without warm support stay `per_turn`. An explicit choice
always wins. Migration 0443 clears the stored creation default from
existing agents, except where an edit set "Turn by turn" explicitly.

**Reason and benefit**
Measured locally (`doc/plans/2026-11-07-warm-follow-up-ttft.md`). Warm
follow-ups are 2.6–5× faster for Codex and OpenCode, and restart
recovery latency is equal or lower.

## What Changed

- `native-lifecycle.ts`: an unset `agentMode` resolves to warm (5-minute
idle window) for local `codex_app_server` and `opencode_server` agents.
`runtime-selection.ts` passes `localTarget`. Explicit `per_turn` or
`warm` wins. An admitted session keeps its persisted policy. ACPX Claude
stays `per_turn`: it relaunches its provider process on every follow-up
by design, so warm did not shorten its follow-up (10.7 s warm vs 10.5 s
per_turn).
- Creation and configuration writes no longer store `lifecycleMode:
"per_turn"` for Codex, Claude or OpenCode. This covers
`normalizePaperclipRunnerAdapterConfig` (runs on every create and
update), `paperclipRunnerTransitionConfig` (switching to the native
runner), `agent-runner-selection.ts`, `codex-local` `build-config.ts`,
and the simplified new-agent setup (`paperclipRunnerCreationLifecycle`
in `@paperclipai/shared`). Grok, Pi and Cursor keep their explicit
value.
- Agent form: the Runner lifecycle field shows the unset state as
"Automatic". "Turn by turn" and "Warm session" remain.
- Migration `0443_native_warm_follow_up_default.sql`: clears
`lifecycleMode = 'per_turn'` from native Codex, OpenCode and Claude
agents unless `agent_config_revisions` shows an edit that set it to
`per_turn`. Each migrated agent gets an `agent.updated` activity entry
with the old value (`actor_id =
migration:native-warm-follow-up-default`). Creation records no revision,
so a "Turn by turn" pick made at creation cannot be told apart from the
old default. The product owner approved migrating those agents. Claude
agents follow the default, which is `per_turn` today.
- Restart chaos harness and TTFT driver: `--lifecycle
per_turn|warm|default` (saved in the wave plan). `default` removes the
catalog's pinned lifecycle, so a wave exercises the real default path.
- Docs: `doc/DEVELOPING.md` "Runner lifecycle default", and the results
in `doc/plans/2026-11-07-warm-follow-up-ttft.md`, which also documents
baseline finding 4 (Claude's resumed turn-start gap is ACPX's per-turn
process generation, not the provider).

## Verification

- `server`: `native-lifecycle.test.ts` (new default cases: local Codex
and OpenCode warm, Claude, remote, unknown target, unmeasured harness,
managed Claude, admitted per-turn session, explicit choices, non-native
adapter), `agent-runner-selection.test.ts`, `runtime-selection.test.ts`
(335 pass).
- `packages/db`: `native-warm-follow-up-default-migration.test.ts` on
embedded Postgres. Default agents are cleared, the explicit edit is
kept, other harnesses and legacy adapters are untouched, audit rows are
written once, and a rerun is a no-op. Without the explicit-edit filter
the test fails. `check:migrations` passes.
- `codex-local` `build-config.test.ts` (65), `ui`
`config-fields.test.tsx` (40), restart-chaos unit tests (134), `tsc
--noEmit` for `server`.
- `adapter-utils` `paperclip-runner-lifecycle-default.test.ts`:
configuration writes and adapter conversion store no lifecycle for
Codex, OpenCode and Claude, keep `per_turn` for the others, and preserve
explicit choices. It fails without the fix.
- End-to-end default check on this branch (`ttft-baseline.ts --lifecycle
default`, codex and opencode, 2 samples): the agents read back with no
stored lifecycle, and every follow-up reused the runner and provider
session (no `runner.process.launch`). Follow-up TTFT was 3.6–4.8 s. This
check found the per-write `per_turn` injection fixed here.
- Restart chaos (codex and opencode, A1/B1/B5, K1/K2/K6, seed 4242), 8/8
PASS each with no open signatures: forced warm, forced per_turn, and
`--lifecycle default` on this branch (agents created without a
lifecycle).
- `pnpm verify` selects broader CI (harness config is a global input),
so required CI supplies the broad coverage.

## Risks

- Behavior change: local Codex and OpenCode agents keep a runner and
provider process for up to 5 minutes after a run. Instance sleep waits
for that idle window. The process is reaped by the idle timer, by
graceful shutdown, or by the #19879 reconnect grace if its controller
dies.
- Migration: agents whose "Turn by turn" was chosen at creation become
Automatic. The activity entry records the old value; choosing "Turn by
turn" restores it.
- Previous-release story (`rules/upgrade-safe-run-state.mdc`): a task
session keeps the lifecycle policy it was admitted with, so sessions
saved before this change stay `per_turn`. New sessions get the default.
No run, checkpoint or launch state changes shape.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use (shell, file edit, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 14:37:36 -06:00
DottaandClaude Opus 5.5 6cfb69b4b0 docs(plans): Phase 0 ownership inventory and TTFT baseline (#19846)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A control-plane restart can land between two durable writes. It
leaves an in-progress record owned by a dead process. About 69
restart-chaos PRs fixed that shape one subsystem at a time.
> - The durable ownership plan (#19831) replaces those fixes with one
owner stamp, one prover, write-ahead intents and one reconciler. Phase 0
must first list every in-progress record kind and measure how fast turns
start today.
> - Without the inventory, Phase 1 cannot choose one stamp or one proof.
Without the baseline, no later phase can show that it kept turns fast
(invariant 8).
> - This pull request adds the Phase 0 inventory, the TTFT, boot and
restart-recovery baseline, and the run-log-only driver that produced the
baseline.
> - The benefit is one reviewed map of 83 record kinds to plan sections
A–G, and measured p50/p95 budgets per adapter.

## Linked Issues or Issue Description

Refs #19831. Refs #19838 (the TTFT anchor that the baseline uses).

**What is the problem?**
Phase 0 of the durable ownership plan has no inventory of in-progress
record kinds and no speed baseline.

**What is the expected outcome?**
Two reviewed documents in `doc/plans/`, plus a repeatable way to measure
the baseline again in later phases.

**Why does it matter?**
Phase 1 cannot start without them, and every later phase's exit criteria
compare against the baseline.

## What Changed

- `doc/plans/2026-11-07-ownership-inventory.md`:
- 83 durable in-progress record kinds (94 reader rows) across native
runtime, environments, workspaces, accounting and the wake queue;
- each kind with its table and column, writer, states, owner, release
rule, proof, obligation, invariants and plan sections A–G;
- the overlaps, the merge decisions and the zero-downtime upgrade record
kinds.
- `doc/plans/2026-11-07-ttft-baseline.md`:
- TTFT, agent boot, pre-turn steps and restart recovery per adapter
(codex, claude, opencode), local, cold and warm, p50/p95;
- restart recovery from the overnight chaos evidence, the slowest steps
ranked, and the gaps.
- `tests/restart-chaos/ttft-baseline.ts`: an on-demand driver on the
chaos supervisor. It reads the run log only, never Telemetry, and always
stops its embedded Postgres.
- `tests/restart-chaos/ttft-baseline-stats.ts` and its unit tests: pure
percentile, boot, pre-turn ranking and recovery statistics.

No runtime behavior changes.

## Verification

- `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts
tests/restart-chaos/ttft-baseline-stats.test.ts`
- `node scripts/typecheck-test-projects.mjs tests/restart-chaos`
- The baseline numbers come from the driver run named in the report. To
reproduce them, follow the report's "Method" section.
- The owner spot-checked the inventory's file:line claims at
`f2d8a51027`. One example: `processLossRetryCount` has no writer, and
`heartbeat/recovery.ts:2675` is its only reader.

## Risks

- Low risk. This adds documentation and an opt-in test driver. Nothing
runs in CI except the pure stats unit tests.
- The driver spends a small amount of provider credit when someone runs
it by hand.
- The baseline sample is small (6 per cell). The local host was shared
with other worktrees, so the p95 tails include load. The report states
both.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use,
code execution and five parallel read-only subagents for the inventory.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 10:47:31 -06:00
377bc87df1 feat(connectors): show and select personal AI account defaults (#19845)
Show a crown on each saved personal AI provider default. Add a provider-specific menu action to other eligible accounts. Share saved-state queries, block duplicate writes, and keep the saved crown until a write succeeds.

Keep account details available for accessible grants. Show session lookup errors with Retry. Resolve list permissions once per request and index account lookups. Add UI and API regression coverage and account documentation.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Paperclip <Paperclip-Paperclip@users.noreply.github.com>
2026-11-07 10:26:08 -06:00
DottaandPaperclip f2d8a51027 fix(ui): show skill descriptions and preserve composer line height (#19502)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - People write requests in the new-task and task-page composers.
> - Both composers use the shared Markdown editor for skill selection.
> - Skill names alone do not explain which skill to use.
> - Inline skill pills can increase the height of a text line.
> - This pull request shows descriptions in a full-width menu and gives
inline skills a color and icon.
> - People can choose a skill and read the request with normal line
spacing.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Skill selection and inline skill display in the new-task and task-page
composers.

**Subsystem affected**

ui/ — shared Markdown editor and task composer stories.

**Current behavior**

The slash menu shows skill names in a narrow popup. Inline skills use
pill styles that can make a text line too tall.

**Proposed behavior**

Match the slash menu to the active composer width. Put each skill name
and description on one row. Truncate the description with an ellipsis.
Show each inline skill in purple with the Boxes icon and normal text
line height.

**Reason and benefit**

People can compare skills before selection. Inline skills remain easy to
find without changing paragraph spacing.

**Breaking changes**

None. Skill links retain the same Markdown labels and references.
Mention selection and task submission retain their existing behavior.

**Additional context**

Related: #19272 changes the submission key in these composers. The
browser cases here check skill links after task creation and reply
submission. Searches found no duplicate change for this display issue.
This is a small improvement to the shipped skills UI in ROADMAP.md.

## What Changed

- Align the slash menu with the active composer. Observe composer size
changes and keep the menu inside the viewport.
- Show each skill name and description on one row. Keep skills without
descriptions available.
- Use theme tokens for purple skill text and the Boxes icon. Preserve
the surrounding text line height. Include both prefixed and standard
mask declarations, consistent with the other icon rules.
- Add a composer anchor to the task-page surface. Keep the shared editor
responsible for menu placement and skill display.
- Increase the contrast of menu details and the task composer model
label in the covered stories.
- Add task-page stories and extend new-task stories. Cover desktop,
mobile, light and dark themes, keyboard insertion, and Markdown
submission.
- Add 18 browser cases for menu size and alignment, description
truncation, inline line height, and submitted skill references.
- Use the production mobile context in task-page stories. Check mobile
text size, long skill names without descriptions, and the saved new-task
skill URL.

## Verification

- Before the compatibility correction,
`a6371219ad1ce0ba908b63b55b7999e4e683cace`: all 130 focused editor,
mention, destructive-theme, and avatar tests pass across six files. No
existing test was changed.
- Current head `c7091dd5d34897f82b752bf93f94b1026cc7a82f` adds the
prefixed mask declaration requested by review. All 12 affected
theme/avatar tests pass. A fresh Storybook production build and all 18
composer browser cases pass on the identical source tree, with two
workers, traces enabled, and no retries. Token gates and whitespace
checks pass.
- Browser cases check both composers, desktop/mobile text size,
light/dark themes, popup width and alignment after resizing, single-row
ellipsis, names without descriptions, inline paragraph height, and
submitted skill links.
- The preceding head has a fully successful CI run (`42260195469`,
attempt 2). Current-head Greptile is 5/5 and its check completed
successfully. All four review threads are resolved. Current-head CI
[42270466448](https://github.com/paperclipai/paperclip/actions/runs/42270466448)
passed, including typecheck, build, all general and serialized test
shards, all Runner checks, and all eight end-to-end shards. The
aggregate verify gate passed.
- Screenshots for both composers were reviewed and accepted before this
PR was opened. Stories use production components with fixture APIs and
submissions. They do not prove backend persistence or agent execution.
Mobile uses a resized Chromium viewport with production mobile props.
The visual baseline is unpublished, so captures support inspection, not
pixel comparisons.
- CI run `42254435625` on `9193b47d` failed two existing
destructive-theme checks because the new skill-only theme blocks came
before the main blocks. The failure reproduced locally. Moving the
unchanged declarations into the main blocks made both checks pass.
- CI run `42258164662` on `7fab621f` caught one more existing CSS-reader
contract: the avatar test expects `--background` first in `.dark`. That
failure also reproduced locally. The final source puts the four skill
declarations at the ends of the existing blocks. A mechanical comparison
proves every pre-existing stylesheet line and all four skill declaration
values are identical.
- The first run also failed an unchanged Runner credential test during
its first lease acquisition. All 42 tests in that file pass in a local
diagnostic. The original CI cause remains unknown; no Runner fix is
claimed. The second run's Runner package job was cancelled by the
corrective push, so it supplies no passing verdict. The final-head
Runner package check passed, which shows recovery without explaining the
original lease error.
- One `7fab621f` mobile play function failed during keyboard insertion
(17 browser cases passed, one failed). Its saved DOM contains the skill
and an open link dialog without the trailing sentence. The cause remains
unknown. Three focused traced executions, a bounded frame-delay
diagnostic, and a later full 18-case run passed without source changes.
No reliability fix is claimed. The original failure remains recorded
separately from the final-head 18-case pass.
- Final-head CI run `42260195469`, attempt 1, failed one unchanged
runtime startup test in `idle-completed-history.test.ts`. Its HTTP
readiness check received connection refusals for 10 seconds. The same
test passed in 2.35 seconds in the preceding CI run; the only
intervening source change moved four CSS declarations. A focused local
run passed the case (one passed, 74 filtered out). A full-file local
diagnostic was interrupted after about four minutes without a verdict;
it is not counted as a pass. The startup failure cause remains unknown.
No runtime source or test was changed. Failed CI jobs were rerun once on
the same head; attempt 2 passed. This shows recovery, without explaining
the first startup failure.
- The broad local typecheck could not complete because Cargo is absent.
The affected-test selector requests broad CI for the browser test.
Required CI supplies broad typecheck, test, and build coverage. Linux
local checks used Node 25.9.0, pnpm 9.15.4, and Playwright Chromium 151.
- Earlier evidence retains a Storybook build interrupted by SIGTERM, a
dev-server play error during HMR, and a browser attempt that could not
launch before Chromium installation. Later successful runs do not
explain the earlier interruptions.

## Risks

The shared editor also serves other mention types. Their existing
placement path stays in place. Slash menu placement must stay aligned
during viewport and composer size changes. The browser cases cover both.
The mask correction was checked in Chromium; older WebKit was not
executed. Mobile cases use a resized Chromium viewport. They do not test
a physical phone or its virtual keyboard.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, tool use, and code
execution. The runtime does not expose a more specific model ID or
context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-07 08:26:19 -06:00
DottaandPaperclip 27170dca49 fix(workspaces): preserve the read-only capture phase budget (#19803)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Remote workspace saves must preserve the files produced by a
completed turn.
> - When a sandbox disk is full, the existing fallback transfers a
stable source and seals it on the controller.
> - The fallback gives metadata checks and the portable codec the same
two-minute timer.
> - Slow but successful metadata checks can leave the codec only
milliseconds, although normal sandbox capture allows five minutes.
> - This change keeps the metadata budget and gives the local codec the
existing finite capture budget.
> - All source checks, archive checks, ownership checks, and failure
cleanup remain required.

## Linked Issues or Issue Description

Refs #19465. Related: #19364 and #19619. The open diagnostic change in
#19619 records fallback causes; it does not separate these phase
budgets. No duplicate budget fix was found.

**What happened?**

A retained workspace failed five read-only exports at `stage=archive`
with `transport_timeout`; four warnings also named `ABORT_ERR`. The
fallback had already transferred the source and passed the received-tree
and source inventory checks. Its controller codec received the same
two-minute signal used by those checks.

**Expected behavior**

Metadata checks share a finite two-minute budget. Sealing uses the same
finite five-minute policy as normal sandbox capture. The final source
check uses only the unused metadata budget. A changed source, expired
budget, or cancelled codec must not produce a preservation receipt.

**Steps to reproduce**

1. Create a standalone Git fixture with staged changes, an untracked
result, and a confined symlink.
2. Make the initial sandbox helper report the exact disk-full error.
3. Transfer real files and run the actual inventory scripts. Delay a
successful second inventory reply until just before the native
two-minute deadline.
4. Observe that the real portable codec starts with a live signal, then
aborts when that shared timer expires.

## What Changed

- Name and reuse the existing 300-second command policy for controller
capture.
- Retain the aggregate 120-second metadata budget. Preserve its unused
remainder for the final source recheck.
- Keep the exact disk-full error, source comparisons, file/Git checks,
and cleanup behavior.
- Add six real-filesystem and real-codec controls. Document the separate
phase limits and the current absence of a caller abort signal on the
save interface.
- Repair the existing onboarding test's fake-environment mapping with
the same real-agent PATCH mapping used by its sibling. Verify the
persisted agent through a real GET. Production UI code is unchanged.

## Verification

- Native-timer RED on base `35d41f11bb`: a successful source reply
arrived at 119,981.62 ms. Its command took 119,854.47 ms within its
119,873 ms cap. The actual codec started at 119,982.70 ms with a live
signal and aborted at 120,001.84 ms. Its nested cause matched the
retained warning: `stage=archive reason=operation_failed
transport=transport_timeout code=ABORT_ERR`. The source inventory was
unchanged and private staging was removed.
- The real-archive control sealed 10,166,395 bytes and 533 entries in
555.55 ms. It restored HEAD, staged data, untracked data, and the
symlink, and passed Git fsck. The recorded delay is synthetic; it is not
a measurement of the failed hosted workspace or provider throughput.
- The native RED leaves about 18 ms for the final check too. The
repaired code must still refuse that exact exhausted-metadata case; it
is not claimed as a native-clock GREEN.
- The committed fast regressions control phase time and delegate to the
actual inventory commands and portable codec. They cover successful
restore after the old metadata signal expires during sealing,
pre-capture metadata expiry, codec deadline/cancellation, and final
metadata expiry after a real completed archive.
- From `server/`: `node ../node_modules/vitest/vitest.mjs run
src/services/remote-workspace-readonly-capture-budget.test.ts
src/services/remote-workspace-transport.test.ts
src/services/remote-workspace-readonly-inventory.test.ts
src/services/remote-workspace-transfer-timeout.test.ts --fsModuleCache`
— 156 tests passed in 42.30 seconds before removal of a redundant test
duration assertion. The final-head six-case regression also passes after
rebasing onto `fff70fe49e`; only unrelated Runner and E2E files changed
in that rebase.
- Review follow-up `ad565dd984` makes the phase clock fully controlled.
It checks the requested final 2,000 ms budget separately from the real
subprocess's 30-second test limit. All six controls pass in 11.94
seconds, and server TypeScript passes again. The earlier
real-time-plus-offset clock could falsely fail on a loaded machine; that
review finding is fixed.
- Server TypeScript and module boundary checks pass. The first
TypeScript attempt could not resolve the unbuilt plugin SDK. Building
that existing workspace prerequisite resolved the setup errors; no
source workaround was added.
- The first preflight attempt used the host pnpm shim and was stopped.
The second lacked sandbox network access and was stopped. The canonical
cached preflight then completed in 22.1 seconds with the Runner build
restored from cache. A later test launch used the wrong working
directory for a test rewrite; its wrapper was stopped before waiting
through a second native-timer test. None of these interrupted attempts
is a pass.
- Final-head CI first failed one unchanged Runner-support privacy test
at 30 seconds: [job
failure](https://github.com/paperclipai/paperclip/actions/runs/42359560195/job/128365192635).
The other 6,864 tests passed and four were skipped. The failing test and
its report owners match base `fff70fe49e`; the exact case passes locally
unchanged in 2.97 seconds (one pass, 40 unselected). Job cleanup
reported an orphan `unzip`; that does not establish the timeout cause.
GitHub rejected the scoped retry while the parent workflow was still
running, so no retry has started. No old-head retry was started because
the separately proven onboarding fixture repair requires a new head and
a fresh full pipeline. The original Runner failure remains recorded.
- The same CI run failed onboarding case 160 at 60 seconds; its retained
page showed `Environment not found`. The fixture injected an environment
only in browser responses, then sent that fake ID to the real agent
update. The original case passed locally in 37.44 seconds despite
rejected HTTP 422 updates. Adding a real update-success guard produced a
local RED in 26.83 seconds (expected 200, got 422). Reusing the sibling
fixture's fake-ID-to-local-default PATCH mapping produced GREEN in 22.52
seconds. A real GET verifies the same persisted agent ID, name, and
default environment. Existing sign-in, cancellation, and no-hire
assertions are unchanged. CI screenshot/error context and the local
persistence RED remain retained.
- All other product checks on `ad565dd984` passed, including build,
TypeScript, all general and serialized server shards, and native Runner
checks. Final head `58bb4ee2411dfefca462f0e9b5a701d84ce1326e` passes the
fresh pipeline: [CI
run](https://github.com/paperclipai/paperclip/actions/runs/42360927780).
Runner support passes in 3m36s and onboarding shard 5/8 passes in 5m46s,
with no Runner-support source change. The original failures remain part
of this verification history.
- Environment: macOS arm64, Node 26.4.0, real Node/Git/filesystem
operations. No provider or recovery calls, replayed turn, source
mutation, or hosted recovery acceptance was performed. Final-head
required CI and review pass; the exact gate state is recorded below.

- Exact-head final gate: 57 successful checks, three skipped, two
neutral, and no failures. Both required gates (`ci / verify` and `ci /
e2e`) pass. Greptile reviewed this exact head at 5/5, and zero review
threads remain open. The non-required companion `review` job is still
queued with no steps; it has not passed, and GitHub therefore reports
`UNSTABLE` while the PR remains mergeable.

## Risks

- Controller work after transfer can now take up to two minutes for
metadata plus five minutes for sealing. Each phase remains bounded.
Existing entry, repository, configuration, and transfer limits stay in
force.
- The save interface has no caller abort signal. This change adds no
Stop guarantee and does not replace the recovery retry window with an
operation deadline. Existing physical ownership and publication fences
remain in force.
- This proves the timer mechanism with a synthetic slow inventory. It
does not measure the actual dirty workspace size or show that its
archive finishes within five minutes. A guarded file-only recovery and
independent durable readback remain required after normal delivery.
- The onboarding fixture maps only its exact synthetic environment ID.
It still calls the real agent update and read routes. It does not loosen
production environment validation.
- No schema, provider transport, task status, recovery policy, or source
cleanup authority changes.

## Model Used

OpenAI GPT-6 family through Codex, with reasoning, repository tools,
code execution, and regression tests. The exact serving model ID and
context window are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green (both required gates and all
product checks pass; non-required companion review remains queued as
stated above)
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-07 03:58:24 -06:00
DottaandClaude Opus 5.5 6fb6220d96 fix(restart-chaos): rebuild the Daytona plugin when its sources change (#19798)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart chaos harness runs Daytona waves with a local control
plane that installs the bundled Daytona sandbox-provider plugin from
`packages/plugins/sandbox-providers/daytona`
> - That plugin is excluded from the pnpm workspace, so workspace builds
and `scripts/agent-preflight.mjs` never rebuild it
> - The harness built it only when `dist/` was missing. A long-lived
chaos worktree that moved across many master commits kept serving the
plugin it built on day one
> - As a result, plugin-side fixes such as #19696 (reclaim sandboxes
left by a restart during acquire) and #19754 (keep a runner sandbox
alive through the reconnect grace) were not under test in those waves,
while server and runner fixes were
> - This pull request keys the plugin build on a content hash of its
inputs and rebuilds when the hash changes
> - The benefit is that every Daytona wave tests the plugin of the
commit it claims to test

## Linked Issues or Issue Description

**Bug description**
`ensureDaytonaPlugin()` in `tests/restart-chaos/main.ts` returned early
whenever `dist/manifest.js`, `dist/worker.js` and the Daytona SDK
existed.

**Steps to reproduce**
Build the plugin on commit A, check out a later commit B that changes
`packages/plugins/sandbox-providers/daytona/src`, and run a `--env
daytona` wave. The wave installs the commit-A plugin. In one chaos
worktree, `dist/plugin.js` was 20 hours older than `src/plugin.ts` and
lacked `runnerReconnectGraceMs` from #19754.

**Expected behavior**
The harness rebuilds the plugin whenever its sources, lockfile or the
plugin SDK source changed since the last build.

Refs #19544

## What Changed

- `tests/restart-chaos/plugin-build.ts` (new): `pluginBuildInputHash`
hashes the plugin's `package.json`, `pnpm-lock.yaml`, `tsconfig.json`
and `src/**`, plus extra inputs: the plugin SDK `src`, `package.json`
and `tsconfig.json`, and the root `tsconfig.json` and
`tsconfig.base.json` that both packages extend. It excludes
`node_modules` and `dist`. `pluginBuildState` compares that hash with a
stamp in `dist/.restart-chaos-build-input.sha256` and reports a reason.
- `tests/restart-chaos/main.ts`: `ensureDaytonaPlugin` installs, links
the SDK and builds when the state is stale, then writes the stamp. It
records `daytona_plugin_build_state` and `daytona_plugin_built` in the
timeline.
- `tests/restart-chaos/plugin-build.test.ts` (new): an unstamped build
is stale; the stamped build is up to date; a changed source, lockfile or
SDK source is stale; changes to `node_modules` and `dist` are ignored;
missing outputs or dependencies are stale.

Other out-of-workspace packages, checked as requested:
- `pnpm-workspace.yaml` excludes only
`packages/plugins/sandbox-providers/**`.
- The restart chaos local mode (`--env local`) does not install a
sandbox provider.
- CI's runner-e2e "Prepare bundled Daytona plugin" step runs on a fresh
checkout per job, so it cannot see a stale `dist/`.
- The other sandbox providers are not used by these harnesses. The risk
is therefore limited to long-lived local worktrees running Daytona
waves, which this change covers.

## Verification

- `vitest run --config tests/restart-chaos/vitest.config.ts`: 121/121
pass, including the new staleness tests.
- `tsc -p tests/restart-chaos/tsconfig.json`: no errors in the changed
files.
- Manual: in the chaos worktree, `dist/plugin.js` (built 2026-11-06
11:16) lacked `runnerReconnectGraceMs`. After a rebuild at master
35d41f11bb it contains it.

## Risks

- This change touches the test harness only.
- The first wave after this change rebuilds the plugin once in each
worktree, because existing builds have no stamp. That takes a few
seconds.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edits, GitHub CLI). Default context window,
extended thinking.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 03:44:25 -06:00
DottaandClaude Opus 5.5 9cc3760552 test(runner-e2e): allow controller outages up to ten minutes (#19797)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart chaos harness and the runner-e2e recover-controller
cells stop the control plane with `replaceController`
(`tests/runner-e2e/controller-outage.ts`), hold it down, then start it
again
> - `parseControllerOutage` refuses holds longer than 300000 ms
> - Restart chaos needs longer holds. One example is proving that a
Daytona sandbox survives an outage longer than its old 5-minute
auto-stop but inside the 7-minute runner reconnect grace (RC-54)
> - With the cap, such a fault errors before the controller stops, so
the scenario cannot run
> - This pull request raises the test-harness cap to 600000 ms
> - The benefit is that chaos runs can now inject 5–10 minute outages;
production code and qualification contracts are unchanged

## Linked Issues or Issue Description

**Bug description**
A restart chaos replay with a 360000 ms K4 hold failed with "Controller
outage requires SIGTERM/SIGKILL and 0..300000 integer milliseconds". The
controller was never stopped.

**Expected behavior**
The test harness accepts outages of up to 10 minutes. The
recover-controller receipt check keeps its own readiness limit
(`readyAfterMs < 420000`).

Refs #19544

## What Changed

- `tests/runner-e2e/controller-outage.ts`: adds
`MAX_CONTROLLER_OUTAGE_MS = 600_000` and uses it in
`parseControllerOutage` and its error message.
`validControllerOutageReceipt` is unchanged and still requires readiness
inside the 7-minute grace.
- `tests/runner-e2e/controller-outage.test.ts`: the rejection cases now
use 600001. A new test accepts 360000 (SIGKILL) and 600000 (SIGTERM).
The existing five-minute contract test is unchanged.

## Verification

- `vitest run tests/runner-e2e/controller-outage.test.ts
tests/runner-e2e/server-restart-request.test.ts`: 52/52 pass.
- `server-restart-request.test.ts` now uses 600001 as its out-of-range
`controllerOutage`. The separate restart-request `controllerDowntimeMs`
limit (five minutes) is unchanged.

## Risks

- This change is test-only. No product or server code changes.
- A suite that sets `PAPERCLIP_RUNNER_E2E_CONTROLLER_OUTAGE_MS` above
300000 now runs instead of being refused. The recover-controller receipt
check still fails any receipt whose readiness is outside the grace.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edits, GitHub CLI). Default context window,
extended thinking.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 03:11:10 -06:00
github-actions[bot]andlockfile-bot e95dcadbd2 chore(lockfile): refresh pnpm lockfiles (#19773)
## Thinking Path

> - Paperclip uses a dependency lock for reproducible builds.
> - Dependency changes on master need matching lock entries.
> - Paid eval reports also pin a reviewed offline correction.
> - Reports must install before paid model work starts.
> - This automated PR keeps the lock and report proof together.

## Linked Issues or Issue Description

**What happened?**

Dependency manifests on master changed after the last committed lock
refresh.

**Expected behavior**

Frozen installs must match those manifests. Eval reports must retain
matching offline lock proof.

**Steps to reproduce**

Run a frozen install before and after this lock refresh.

## What Changed

- Refresh the dependency lock from the manifests on master.
- Refresh the sandbox provider lockfiles from their manifests on master.
- Include the matching offline report correction and hashes.

## Verification

- Pinned pnpm resolution completed with lifecycle scripts disabled.
- The report proof refresh accepted the reviewed committed baseline.
- The workflow rejected changes outside the lock and report proof files.

## Risks

The lockfile bot merges this PR without CI or review when every added
package version is at least 3 days old on npm. Otherwise the PR waits
for a maintainer, and the daily refresh tries again. The bot merges only
the exact commit this workflow generated from manifests already reviewed
on master. Unknown report lock baselines remain blocked. This update
changes no permissions or paid enable gates.

## Model Used

None — generated by pnpm and the deterministic report-lock updater.

## Checklist

- [x] Thinking path and problem description included
- [x] Generated lock and matching report proof included
- [x] Release-age gate decides between bot merge and maintainer review

Co-authored-by: lockfile-bot <lockfile-bot@users.noreply.github.com>
2026-11-07 07:14:22 +00:00
96bc021edb fix(server): certify settled legacy processes for future retries (#19377)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Bounded retries must preserve current authority, budgets and
deadlines.
> - The tenant can wait for supported future work after the source
process and its cleanup stop.
> - Retained process identity alone cannot prove that cleanup finished.
> - This change records host-owned settlement and checks its exact
identity before a retry can cover its wait.
> - Uncertain process state, changed authority, live claims and cleanup
debt keep the tenant awake.

## Linked Issues or Issue Description

Refs #19001. Related dependencies: #19019 and #18988. Related #19340
covers explicit Native task Retry; this change covers future Legacy
process retries.

Current integrated source: `1645970a9322d1213ba25fc7b52366c285682977`,
normally merged with actual master
`131b566ca4fa866c4348739c75cfe48c805b3398`. The prerequisites above are
merged. This PR targets `master`.

**What happened?**

A future issue, company or review retry can retain its failed source
process identity. The deadline reader could not prove that host cleanup
finished. Supported waits stayed blocked. Ordinary issue retries also
needed an exact source-settlement check, and represented coalesced
requests needed exact input coverage.

**Expected behavior**

A supported future retry covers only its existing wait, under current
company and actor authority, after exact host settlement. Missing,
malformed, stale, cancelled, Native or unfinished source state stays
blocked. No retry gains new execution authority.

**Steps to reproduce**

Fail an ordinary process or fail environment preparation before adapter
entry. Let normal cleanup schedule the existing retry. Inspect its
settlement and deadline, stop and reconstruct the owner, and admit one
useful successor at the saved deadline. Hold cleanup or change authority
to verify that the reader refuses coverage.

## What Changed

- Add nullable internal `legacyProcessSettlement` state through normally
generated migration `0439_cultured_impossible_man`. No backfill or
adapter result creates the record.
- Write exact host settlement after stopped-process and cleanup checks.
Compare the current run and reject live claims or cleanup debt. Optional
persistence failure preserves normal cleanup and keeps sleep coverage
blocked.
- Add a separate preparation receipt for a claimed Legacy run that
failed before any adapter or Native dispatch. Require the same boot, run
times, preparing stage, released claim and completed cleanup. A null PID
alone proves nothing.
- Clear settlement on a new claim, process metadata or Native selection.
Omit the internal record from public serialized data.
- Check exact source lineage and current authority for ordinary issue,
company and taskless retries. Cover completed coalesced requests only
when the admitted retry represents their exact inputs.
- Add the closed `current_review_retry` wait with current reviewer,
stage and existing issue authority checks. Require a nonempty current
stage ID.
- Add actual producer, cold reconstruction, SQL negative, redaction and
optional-persistence controls.
- Repair CI fixture expectations for the additive column, startup mock,
budget owner and popup return URL. Bind the controlled Native comment
server explicitly to loopback.
- Replace temporary Slack clock logging with an exact persisted-deadline
assertion. Wait until that real deadline before the existing single
default poll. Keep both forced polls, business assertions and timing
limits.

## Verification

- On final integrated source `1645970a`, normal Shared/SDK
prerequisites, Runner TypeScript and the entire normal App layered
typecheck pass. Tracked source and lockfile remain unchanged.
- All 26 whole-file settlement, recovery-reader and bootstrap controls
pass. Seven separate real PostgreSQL producer/negative cases pass at
`cf16c6a`; each has one selected passing case and 18 filtered
registrations. Filtered registrations are not acceptance.
- Master then changed the preparation source-lock scope. Its three
intersecting cases were rerun on final `1645970a`: an actual failed
company process, an actual failed ordinary issue process, and held
preparation cleanup. Each preserves its source identity and original
deadline and admits one useful successor. Issue and preparation cases
exercise normal automatic retry; the company case uses the normal
explicit bounded-retry producer.
- The unchanged 26 controls and four other PostgreSQL cases retain their
`cf16c6a` proof after a scoped source comparison. Optional
settlement/preparation persistence failures keep certification blocked
while normal cleanup completes. An entered adapter without process
identity stays uncertified. This is scoped carry, not a claim that every
earlier case ran again on the final head.
- All recorded test owners and controlled groups closed. Final read-only
SQL audits returned no test sessions. One earlier postcase resource
barrier observed a transient unattributed database backend; that
database and original evidence remain preserved. Its later disappearance
is recovery, not an explained cause. No assertion, deadline or test body
was relaxed.
- Migration0439 was generated normally after master history. All
executable SQL and journal history are preserved; the five retained
snapshots are0435–0439. Every Legacy-owned patch and unrelated
actual-master blob/mode is preserved.
- Published exact head `1645970a9322d1213ba25fc7b52366c285682977` passes
all current required checks: 68 successful contexts and eight expected
skips after exhaustive pagination and superseded-run accounting. The
fresh complete Greptile review scores this exact head 5/5. All four
inline review threads are resolved. The latest Ponytail turn stopped
without a verdict; one normal retry was requested, with no active turn
or new finding observed. No old-head score is used. The original
old-base policy failure, its dependent aggregate failure, and the
corrected description-gate failure remain recorded. No full local
repository suite is claimed; remote typecheck, test and build gates
pass.

To repeat the real database coverage, set both
`PAPERCLIP_RECOVERY_DEADLINES_TEST_DATABASE_URL` and
`PAPERCLIP_RECOVERY_DEADLINES_TEST_ADMIN_DATABASE_URL` to the same fresh
dedicated test database and run the selected
`recovery-deadline-acceptance.test.ts` cases with one worker. An unset
URL, skipped case or filtered registration is not acceptance.

The fixtures use controlled local processes and real PostgreSQL.
Reconstruction creates new service owners inside the same test process.
They do not prove deployed Cloud alarms, physical host/database
suspension, new-host recovery or Native capacity settlement. The
original30-second retry remains below the Cloud270-second minimum wake
lead.

The merge projection also includes actual master
`c3180398faca815fed7a62674bd4b0c4e88884ae`, whose common completion path
now sends deferred/recovery notifications. Its unpublished normal merge
tree is `16b028d57150f8caa576e913f38343a365b542dd`. Normal Shared/SDK,
Runner TypeScript and the full App graph pass on that combined source.
One unchanged ordinary-issue acceptance case was rerun there: an actual
failed process retains the original automatic retry deadline, admits one
useful successor, and passes the settlement, authority and no-duplicate
assertions. It passed in 45.055 s; 18 filtered registrations are not
acceptance. All known owners and groups closed, and the independent
final database audit found zero remaining test sessions. The successful
database was removed normally. Recorded worker socket peak is
unavailable for this run; parent-process zero values are not used as
worker measurements. All 29 Legacy-owned paths and every unrelated
master blob and mode are preserved.

Two outside-diff review observations concern existing waiting-chat
compaction and cross-host Native result-lease recovery. Their files are
byte-identical to current master. This focused Legacy change does not
fix or qualify those separate paths.

## Risks

Settlement proves exact local closure. It does not cover live provider
work, Native ownership, cancellation, costs, claims, cleanup debt or
changed authority. Independent checks remain.

Ship normal migration0439 with the reader and writer. Preserve
executable history and the newest five snapshots0435–0439. Generate the
MCP child's next migration normally after this parent.

The prerequisites are merged; the final remote checks must use this
integrated exact head. This proof uses controlled local processes and
external PostgreSQL. It does not establish deployed Cloud alarm
acceptance, physical host or database suspension, cold host recovery, or
R5 Native capacity settlement. The30-second retry is below the Cloud
minimum wake lead.

This focused repair follows the roadmap's bounded recovery and Cloud
deployment work. It adds no new provider policy.

## Model Used

OpenAI `gpt-6.1-sol` assisted with readers, fixtures, source review and
verification through Codex. A coordinating GPT-6-family agent
implemented and reviewed shared lifecycle wiring and integration. Tools
included Git, native TypeScript checks and controlled tests. Exact
served revisions and context limits were not exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used and its available capability
details
- [x] I have checked ROADMAP.md and related work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have linked existing public work and described the issue
- [x] I have not included internal issue IDs or private links
- [x] My branch name describes the change
- [x] Required focused checks pass at their explicitly recorded
source/carry scopes above
- [x] I have added or updated meaningful tests
- [x] I have recorded the internal contract and migration requirements
- [x] I have documented risks and evidence limits
- [x] All required published exact-head Paperclip CI gates are green
- [x] Published exact-head Greptile is 5/5 with no open findings in this
focused diff; separate unchanged master observations are identified
above
- [x] I will address reviewer findings before requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Codex <noreply@openai.com>
2026-11-07 00:56:04 -06:00
DottaandClaude Opus 5.5 efc0b359da fix(ci): rebuild and sign an unsigned reused Daytona e2e image (#19677)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Paid runner E2E campaigns run in `runner-full-stack-e2e.yml`.
Daytona cells need a signed, content-addressed runner image
> - The "Publish verified Daytona image" job reuses `e2e-content-<id>`
when the tag exists, and signs only after a fresh build
> - A run that stops between the push and `cosign sign` leaves an
unsigned tag. Every later run reuses it and fails `cosign verify` with
"no signatures found"
> - That failure skips the whole test matrix, including local cells. The
first Reliability TestOps launch (run 42298071794) ran 0/7 cells for
this reason
> - This pull request reuses a tag only when its signature verifies.
Otherwise it rebuilds, pushes and signs the tag
> - The benefit is that an interrupted run cannot block every later
Daytona campaign

## Linked Issues or Issue Description

Refs #19549

**What happened?**
Run https://github.com/paperclipai/paperclip/actions/runs/42298071794
(TestOps launch of the Reliability plan) failed in "Reuse or publish
immutable image" with `Error: no signatures found` (exit code 10). No
test cell ran. The report shows 7 missing attempts.

**Expected behavior**
A campaign with Daytona cells always produces a signed, verified image,
or it fails for a real build or signing error. A stale unsigned tag must
not block it.

**Steps to reproduce**
1. A run pushes
`ghcr.io/paperclipai/paperclip-daytona-runner:e2e-content-<id>` and
stops before `cosign sign` (for example, a runner shutdown).
2. A later campaign with the same content ID finds the tag and skips the
build.
3. `cosign verify` fails with "no signatures found", and the matrix is
skipped.

**Paperclip version or commit**
master 02a2299173.

## What Changed

- `.github/workflows/runner-full-stack-e2e.yml`: add a `verify_signed`
helper. Reuse an existing content tag only when its digest has this
workflow's signature. If verification fails, warn, rebuild, push and
sign. The final `cosign verify`, logout and config checks are unchanged.
- `tests/runner-e2e/daytona-image.test.ts`: assert that reuse is gated
on signature verification and that the build-and-sign path follows it.

## Verification

- `npx vitest run daytona-image workflow-security` in
`tests/runner-e2e`: 38/38 pass.
- Red before the fix: the new assertion fails on the unchanged workflow
(1 failed, 18 passed in `daytona-image`).
- The workflow YAML parses.
- Live check after merge: relaunch the Reliability TestOps plan. The
image job should warn about the unsigned tag, rebuild it, sign it and
verify it, and then the cells run.

## Risks

- Low risk. The rebuild uses the same trusted inputs and cache, and it
only happens when verification fails. The new digest replaces the
unsigned tag. The identity and issuer checks are unchanged.
- Rebuilding an unsigned tag takes extra time on that one run.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code with tool use
(shell, browser read of the GitHub job log), standard context.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 22:29:11 -06:00
DottaandClaude Opus 5.5 644ee66f5d test(restart-chaos): R5 live tools and processes; fail a wave whose worktree changes mid-wave (#19697)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart chaos harness (`tests/restart-chaos/`) grades every task
on R1–R6; R5 catches silent stalls, including a run that stays `running`
with no live evidence
> - A B5 task runs `bench.sh` for about four minutes with no output, so
a healthy Retry run was graded `R5:silent_dead_run` while its runner
process was alive and its tool was still running
> - Separately, a runner `dist` rebuilt in the harness worktree during a
wave made later restarts run different code and produced a false
signature (RC-32's investigation)
> - This pull request makes R5 treat a run as live while a tool it
started has not ended or while its recorded local runner process still
runs, and fails a wave whose worktree changes mid-wave
> - The benefit is fewer false R5 failures and no silently mixed-code
waves

## Linked Issues or Issue Description

Refs #19629
Refs #19641

**What is the problem?**
R5 flags a quiet but live run (a long tool call, a live runner process)
as `silent_dead_run`. A build, install or branch switch in the harness
worktree during a wave goes unnoticed, though every restart re-runs the
control plane from that worktree.

**What is the expected behavior?**
A run with a tool in flight, or with its recorded local runner process
still alive (same PID and start time), is live for R5. A wave whose
checkout or runner build output changes during the wave fails as a
harness error.

**Why does it matter?**
False R5 failures send fixers after non-bugs, and mixed-code waves
produce signatures that are not product behavior.

## What Changed

- `observer.ts`: the run-event reader counts tool starts minus tool ends
(PRP item ids are not unique per tool call), and remembers the run's
latest local process identity until recovery reports it stopped. Samples
carry `toolInFlight` and `processAlive`. `localProcessAlive` checks `ps`
for the same PID with the same start time, so a recycled PID never
counts. Daytona runs are not checked (their process is remote).
- `grader.ts`: `deadRunningRun` skips a run with a tool in flight or a
live process.
- `types.ts`: `RunSample.toolInFlight` and `RunSample.processAlive`.
- `wave-guard.ts` (new) and `main.ts`: fingerprint `HEAD`, the tracked
diff contents (`git diff --binary HEAD`), untracked file contents, and
size/mtime of `packages/paperclip-runner/dist` and the runnerd binary.
The baseline is taken after the harness's own setup (preflight rebuilds
provider entrypoints). Re-check after each fault and at the end; fail
the wave (`worktree_changed_mid_wave`) if it changed.
- `main.ts`, `scheduler.ts`: `faults.json` and a new `task-faults.json`
(each task's launch time and crossing faults) are written after every
fault. `--replay` falls back to that log when a task has no grade file
(an interrupted or wedged wave), so a storm task replays the faults that
crossed it at the same offsets.
- `README.md`: both grading notes.

## Verification

- `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`:
111 tests pass. New tests cover:
- a B5 run quiet for 4 minutes passes with a tool in flight or a live
process, and fails otherwise
- tool and process liveness come from run events and clear on tool end
and process stop
- `localProcessAlive` accepts this process's real PID and start time and
rejects a different start time
- the guard catches rebuilt dist, a re-edited dirty file, an edited
untracked file and a rebuilt runner binary, in a real temporary git repo
- a storm task with no grade file replays the faults that crossed it at
their offsets from its launch
- `node scripts/typecheck-test-projects.mjs tests/restart-chaos`:
passes.

## Risks

- Low risk, test harness only.
- The R5 exemption narrows R5. A run that is truly stuck while its
process lives (a hung provider) now passes R5 and falls to R1's time
budget instead.
- One `ps` call per running run per poll on local waves.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 22:07:42 -06:00
DottaandClaude Opus 5.5 da850990f8 fix(restart-chaos): grade R4 on overlapping execution; wait out pending cleanup for R6 (#19710)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart chaos harness grades every task on rules R1 to R6. R4
means "no duplicate execution"
> - R4 counted every queued, running or scheduled-retry run in one poll
as an active run, and failed the task when there were two
> - A follow-up comment while a run is in progress normally queues a
wake behind that run. The product cancels the wake when the running run
finishes, so the wake never executes
> - A Daytona wave failed an opencode task for exactly this case, so R4
reported a duplicate-execution failure that did not happen
> - This pull request grades R4 on actual overlapping execution
> - The benefit is that R4 failures again mean that two runs really
executed together, or a comment was consumed twice

## Linked Issues or Issue Description

**Bug description**
`gradeR4` failed a task whenever one poll showed two runs in
`ACTIVE_RUN_STATUSES` (queued, running, scheduled_retry). This includes
a wake queued behind a running run that never starts.

**Steps to reproduce**
Daytona wave `w-20261107T013205-s11`, task t06 (opencode C1 K2):
- Run b4b16a4d is running.
- The C1 follow-up comment queues run c090799a.
- c090799a is cancelled 1 s after b4b16a4d succeeds and never starts.
- Before this change, R4 reported `concurrent_runs: c090799a(queued),
b4b16a4d(running)`.

**Expected behavior**
R4 passes. It fails only when two runs execute at the same time, or when
a side effect (a consumed comment) repeats.

Refs #19544

## What Changed

- `tests/restart-chaos/grader.ts`: `gradeR4` fails when two runs are
`running` in the same sample, or when the started→finished windows of
two started runs overlap. A still-active run's window lasts until the
last sample. Queued and scheduled-retry runs that never started do not
count. The comment-consumed-twice check is unchanged.
- `tests/restart-chaos/grader.test.ts`: a test with the t06 shape (a
queued wake cancelled behind a running run) that passes; queued plus
scheduled_retry without a start passes; overlapping windows fail even
when no single poll saw both running. The existing two-running and
comment tests are kept.
- `tests/restart-chaos/README.md`: the R4 grading note.
- R6 settle for Daytona (`tests/restart-chaos/daytona-sweep.ts`,
`main.ts`). The product retries a `pending_cleanup` lease after
`ENVIRONMENT_CLEANUP_RETRY_MS` (5 min), so a 2-min settle reported
normal cleanup as a leak. R6 now waits `PENDING_CLEANUP_RETRY_MS + 90
s`, capped at 8 min; a test keeps the constant equal to the server's. An
environment DELETE that returns 409 while a sandbox cleanup is pending
is correct product behaviour. It is retried for the same window. If it
is still blocked after that, it is reported as
`environmentDeleteBlocked`, and it fails the wave and keeps the instance
(whose database owns the cleanup work). It is not counted as a harness
error. Tests cover the 409 retry and a real (500) delete failure.

## Verification

- `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`:
109/109 pass. `node scripts/typecheck-test-projects.mjs
tests/restart-chaos`: clean.
- R6 evidence: in wave `w-20261107T015455-s12`, three leases were still
`pending_cleanup` at the 2-min mark, and the environment DELETE returned
409 "Cannot delete this environment while a sandbox cleanup is pending".
- The t06 sample sequence from `w-20261107T013205-s11` is the new test
case.

## Risks

- This change touches the test harness only.
- R4 is now less strict about queued rows. If a queued duplicate starts
executing, its start time makes its window overlap, so R4 still fails.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edits, GitHub CLI). Default context window,
extended thinking.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 21:31:29 -06:00
scotttong f5fb8ada3b Restore chat contacts, agent pins, and scrollable recent navigation (#19685) 2026-11-06 19:17:25 -08:00
DottaandClaude Opus 5.5 3fedfe8763 fix(runner-e2e): wait out a briefly uncertain owned group instead of failing the stop (#19692)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart chaos harness and the runner e2e suites stop and restart
the control plane through one process-tree owner
(`tests/runner-e2e/process-tree-owner.ts`)
> - The owner signals only process groups whose identity it has proven,
and refuses everything when a group's identity becomes uncertain
> - Under load a group can turn over between two `ps` snapshots: the
observed members exit, and a straggler that nobody saw start is left in
the group, reparented away from every owned anchor
> - The stopper treated that as fatal on the first inspection, so a K2
graceful stop at load 28 failed in under a second, and a chaos storm
wave was marked as a harness error although the server came back
> - This pull request makes the stopper re-inspect within its existing
deadline until the uncertainty settles. It still never signals the
uncertain group, and it fails with the original error if the uncertainty
persists
> - The benefit is that chaos waves under load stop failing on a harness
race, without weakening the rule against signaling unproven processes

## Linked Issues or Issue Description

Refs #19470

**What is the problem?**
`stopOwnedProcessTree` fails at once with "Owned process group identity
became uncertain" when, under load, a group's observed members exit and
an unobserved straggler remains for a moment. In a chaos storm this
failed a K2 stop 0.9 s after SIGTERM (load 28), and the wave exited as a
harness error.

**What is the expected behavior?**
The stopper waits (within the same grace or forced deadline) for a later
inspection that settles the group, and it never signals the uncertain
group. If the straggler outlives the deadline, the stop still fails with
the same error.

**Why does it matter?**
A harness-only race must not invalidate a chaos wave, and the fix must
not let the harness signal a process whose ownership is not proven.

## What Changed

- `tests/runner-e2e/server-stop.ts`: the chaos and e2e server stopper
(the actual K2 path) uses the same bounded settle-wait for its graceful
observe/signal and its SIGKILL escalation. Live evidence: a K2 at load
13.9 failed in 0.5 s on an ACPX Claude provider group (launcher, reaper,
module-hook loader, `claude` runtime) that tore itself down during the
drain and was gone 0.4 s later.
- `tests/runner-e2e/process-tree-owner.ts`: the uncertainty error now
names the unowned members (pid, ppid, pgid, kind, start).
`settleUncertainIdentity` is exported and shared. Export the
`UNCERTAIN_GROUP_IDENTITY` message prefix. `stopOwnedProcessTree`
retries the grace signal, the SIGKILL escalation and the stop check
every 100 ms while that error persists, bounded by the existing grace
deadline and one forced window (set before escalation and reused by the
escalation retry, the stop check and the fallback), so a stop never
exceeds gracefulMs + forcedMs. Other errors, and an uncertainty that
outlives the deadline, take the existing fallback path unchanged. The
owner's identity checks do not change, and neither does `signal()`.
- `tests/runner-e2e/process-tree-owner.test.ts`: three tests. An
uncertainty that starts only at SIGKILL escalation fails within one
forced window and never sends SIGKILL. A brief turnover settles and the
stop succeeds without signaling the uncertain group or killing the
direct child. A persistent turnover still rejects with "identity became
uncertain" and never signals.

## Verification

- `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/process-tree-owner.test.ts
tests/runner-e2e/server-stop.test.ts`: 35 pass across
`process-tree-owner.test.ts` and `server-stop.test.ts`, including 5 new
tests; the new brief-turnover tests fail on the old stoppers. With the
old stopper, the first new test fails: it rejects on the first
inspection and then kills the direct child.
- `node scripts/typecheck-test-projects.mjs tests/runner-e2e`: passes.

## Risks

- Low risk, test infrastructure only.
- A stop that hits a persistent uncertainty now fails at the deadline
(up to the grace time) instead of at once. It sends no extra signals.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 20:55:26 -06:00
DottaandClaude Opus 5.5 aca908a165 fix(restart-chaos): stop the timeline echo loop when stdout breaks (#19695)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart chaos harness (`tests/restart-chaos/`) runs waves of
real agent tasks while it restarts the control plane on purpose
> - A wave writes a timeline file and also echoes every timeline entry
to stdout
> - When the process that reads that stdout exits, every echo fails with
EPIPE. The uncaught-exception handler then records the error with
another echo, so the loop never ends
> - In one Daytona wave this wrote 3.2 million entries (2.5 GB) in 30
minutes. It also stopped the wave from grading its tasks
> - This pull request stops the echo after the first stdout error, and
records the stdout loss once
> - The benefit is that a wave keeps running and grading when its log
reader goes away

## Linked Issues or Issue Description

**Bug description**
`Timeline.add` echoes with `console.info`. After stdout breaks, each
echo raises EPIPE as an uncaught exception. The handler in `main.ts`
calls `timeline.add("harness_uncaught_exception")`, and that call echoes
again.

**Steps to reproduce**
Run a wave as a child process that reads its stdout, for example from a
wave-loop script. Kill the parent while the wave runs. `timeline.jsonl`
then grows without bound with `harness_uncaught_exception` / `write
EPIPE` entries.

**Expected behavior**
The wave goes on writing its files (timeline, grades, ledger) and stops
writing to the broken stdout.

Refs #19544

## What Changed

- `tests/restart-chaos/timeline.ts`: a `process.stdout` error listener
and a guard around `console.info` turn off the echo for good after a
broken-pipe error. The timeline file is still written. On the first
entry after the echo stops, the timeline writes one
`harness_stdout_lost` entry (file only) with the error. The file adds
`isBrokenStdoutError` and `markTimelineEchoBroken`.
- `tests/restart-chaos/main.ts`: the uncaught-exception handler passes
broken-stdout errors to `markTimelineEchoBroken`. It does not mark the
wave as interrupted and does not set a harness error.
- `tests/restart-chaos/timeline.test.ts` (new): the error
classification, and the file-but-no-echo behaviour with exactly one
`harness_stdout_lost` entry.

## Verification

- `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`:
102/102 pass.
- `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: clean.
- Incident: wave `w-20261107T001552-s13`. Its timeline had 76 real
entries, then 3,242,762 `harness_uncaught_exception` entries. Those 3.2
million entries came from the loop this pull request removes.

## Risks

- This change touches the test harness only.
- After stdout breaks, the console shows no wave progress, but every
result is still in the wave's files. A different uncaught exception
still stops the wave as before.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edits, GitHub CLI). Default context window,
extended thinking.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 20:06:02 -06:00
Devin FoleyandPaperclip 1aa0c36397 Fix execution after onboarding plan approval (#19698)
## Thinking Path

> - Paperclip manages agents and their work.
> - First-task onboarding turns a user's goal into an approved proposal.
> - The interview offers a plan and a team to carry out that plan.
> - The instructions leave execution after acceptance unclear.
> - An agent can add a plan-only approval boundary and finish without
doing the requested work.
> - This change makes the approval scope explicit and tests both
execution and plan-only outcomes.

## Linked Issues or Issue Description

**What happened?**

An onboarding agent can treat an interview about building a product as a
request for a plan only. It can then retain the approved plan and close
onboarding without creating execution tasks. The interview evaluation
previously accepted that result.

**Expected behavior**

Acceptance of an execution proposal starts its selected work. An
explicit plan-only request stops after planning. User-requested separate
approvals and existing permission and budget controls still apply.

**Steps to reproduce**

1. Choose the onboarding interview and describe a deliverable.
2. Accept the saved plan proposal.
3. Check that the agent starts the approved work instead of closing with
only a plan.

## What Changed

- Clarify execution intent, approval wording, continuation, and
completion in the first-task prompt.
- Require assigned child work and saved output in the interview
acceptance evaluation. Use checkbox acceptance in the regular interview
journey and retain conversational acceptance in the approval-reply
suite.
- Add an explicit plan-only acceptance journey and nine grader
regression checks.
- Update evaluation counts, grading versions, and onboarding
verification guidance.

## Verification

- Passed: all applicable CI checks on `198a6b47cac7`, including
typechecks, the general and serialized server tests, Runner checks,
build, and browser E2E shards. [CI
run](https://github.com/paperclipai/paperclip/actions/runs/42324855376).
- Passed locally: the full Runner support suite, 6,860 tests across 323
files (three skipped), plus 13 onboarding asset tests.
- Passed locally: the full workspace build, workspace package typechecks
(server rechecked after restoring excluded-plugin SDK links), and the
Runner evaluation TypeScript check.
- Passed: map validation and whitespace checks.
- Initial broad local checks failed during checkout preparation because
PostgreSQL library symlinks and runtime/SDK assets were missing.
Restoring the dependency setup made the affected controls pass. The full
root test suite passed in CI; it was not rerun locally after that
repair.
- Live model journeys have not been run. Grader tests do not prove model
compliance.

## Risks

- Prompt behavior needs live qualification across providers. This change
adds no server-side enforcement or permission bypass.
- The new plan-only case adds four local evaluation cells. A full
campaign now has 56 first-task cells.
- Existing proposals that reserve later approval keep that restriction.
This change does not rewrite tenant data or start existing onboarding
tasks.
- No schema migration or deployment is included.

## Model Used

- OpenAI `gpt-6.1-sol`, xhigh reasoning effort, with repository tools
and local test execution. Context window size is not exposed by this
session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (focused and full Runner
support locally; full root coverage passed in CI)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-06 17:56:27 -08:00
DottaandClaude Opus 5.5 d26372dc8a test(restart-chaos): add a result trigger and recovered-run comment-wake guards (#19689)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart chaos harness kills the control plane at chosen moments
and grades whether each task still finishes correctly
> - One wave lost a user's follow-up comment after a kill that landed
after the run accepted its result but before the run settled
> - No existing trigger aims a fault at that moment, so the case was
hard to reproduce on purpose
> - The recovery path that settles such a run had no regression test
that covers the saved comment
> - This pull request adds a `result` trigger to the harness and a
regression test for the recovery path
> - The benefit is that the harness can aim faults at the
result-accepted window, and a change that drops, repeats, or re-wakes a
saved comment fails a test

## Linked Issues or Issue Description

**What happened?**
In a restart-chaos storm wave, a user posted a comment while a native
run was active. A hard kill (K1) landed after the run accepted its
result and before the run settled. The run settled `succeeded` and the
task stayed `done`, but no follow-up run read the comment.

**Steps to reproduce**
The original failure did not reproduce. Three targeted waves ran 12
tasks with K1 at comment time or in the new result window. Every
follow-up comment was consumed exactly once. The new regression test
also passes on current `master`. The test and the trigger guard the path
and do not claim a fix.

**Expected behavior**
A comment posted during a run is consumed exactly once after the run
ends, including when restart recovery settles the run.

## What Changed

- `tests/restart-chaos`: add `--triggers result` (C1 only). K1 fires 0–3
s after the run that received the follow-up proposes or accepts its
result (`run.result.proposed` or `run.result.accepted`). The fault fires
only while that run is still `running`: the observer checks the latest
sample and rereads the runs right before dispatch. A run that settled
during the delay, or a follow-up posted at rest, records a `result`
miss. Add poll-driven unit tests for these cases and a README row.
- `server/src/__tests__/native-run-projection-work.test.ts`: add
embedded Postgres tests in which background recovery, not the executor,
settles the run (projection, environment settlement, and
deferred-receipt workers), with a deferred user comment and an active
local lease left by the killed executor. The tests check:
- The comment is promoted into exactly one follow-up run, and a second
restart adds none.
- A comment the run already received as a same-turn steer is not woken
again. The other saved comment still is.
- A queue whose only comment was steered stays cancelled, and no
follow-up starts.
- A Retry click does not consume the comment a second time. A click that
observed the source is stale. A click that observed the follow-up joins
it.

## Verification

- `cd tests/restart-chaos && npx vitest run --config vitest.config.ts`
(96 tests pass)
- `cd server && npx vitest run
src/__tests__/native-run-projection-work.test.ts` (54 tests pass)
- Live, local, ACPX Claude, Codex and OpenCode: `node
cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env
local --workloads C1 --triggers comment --faults K1 --adapters
codex,claude,opencode --tasks 4 --concurrency 2 --rc06-masked` passed 4
of 4. Two waves with `--triggers result` consumed each follow-up exactly
once. One task in those waves found a separate defect: a kill 1.4 s
after the result commit left the run `running` behind the dead
controller's 20-minute finalization lease. A separate pull request fixes
that defect.

## Risks

- Low risk. These are test and harness changes only. No production code
changes.
- The `result` trigger depends on the observer's poll interval (about 3
s), so the exact kill point within the window varies.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, run as an agent in
Claude Code with tool use (shell, file editing, local test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 19:53:58 -06:00
DottaandPaperclip a0ac81e114 fix(ui): show task image results in compact thumbnail grids (#19626)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Task feeds show agent messages, documents, and files in time order.
> - Each image currently uses a full feed card.
> - A turn with seven screenshots makes the conversation hard to scan.
> - This pull request groups adjacent images from the same run in a
compact grid.
> - Users can scan the results and open each image in the existing
gallery.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Image results in task history and the live task feed.

**Current behavior**

Each image uses a separate full-width card. Native image upload receipt
messages can also separate related screenshots.

**Proposed behavior**

Adjacent images from one run share a wrapping thumbnail grid. New image
results join the grid. Documents, video, and authored messages keep
their positions. The grid opens the existing gallery.

**Reason and benefit**

Users can see a batch of screenshots without scrolling past one large
card for each file.

**Breaking changes**

The feed layout changes. The stored comments, files, APIs, and
permissions do not change.

Related public work: Refs #17228. Refs #8785. Refs #7667. These changes
supplied the artifact cards and media gallery that this pull request
reuses.

## What Changed

- Add one image grouping function for settled history, turn history, and
live results. Group only images with the same known run. Keep the first
row key and each image key when results arrive.
- Add a token-based grid that reuses the media artifact cards and image
gallery.
- Add an optional typed attachment-preparation marker to file
registration and chat-file reuse. Preserve the marker through comment
normalization. Use it with matching image, author, and run links to hide
generated receipts. Keep the exact historical sentence as a fallback for
old receipts.
- Add nine production component stories with seven sample-account
screenshots. Cover mobile, Classic, mixed results, single images,
missing images, streaming, gallery navigation, and the complete feed
projection.
- Add grouping and production-feed regressions. Add 18 browser cases
across light and dark themes.
- Improve contrast for the shared fallback label and gallery counter
after the accessibility checks found low contrast.
- Match alternate original-image paths when opening the gallery. Keep a
separate Open original link beside Download in the gallery.
- Record the feed layout in DESIGN.md.

## Verification

- Final qualification at `09b12381ce85bee6593050bd8a2fa18cbe2c45a6`:
[required CI
passed](https://github.com/paperclipai/paperclip/actions/runs/42317080212/job/128237094513),
including typecheck, build, all test lanes, E2E, and verify. The
snapshot has 58 successful check runs, three conditional skips, one
neutral advisory job, and a successful legacy Snyk status. No pending or
failed checks remain. Greptile and Paperclip Review both score this
exact head 5/5; all review threads are resolved.

- Current-head local checks: 544 focused UI tests passed across 11
files, with one existing skip. UI/shared typechecks and token gates
pass. All 18 focused Storybook browser cases passed after the CSS merge,
with retries disabled.
- Two database integration cases verify file registration and chat-file
reuse receipt markers. All 52 issue-validator tests and six upstream
default-model tests passed.
- Regressions failed before the fixes for unknown-run grouping, receipt
wording, and authored progress text with an image. The repair uses a
typed receipt marker and requires matching author, attachment, image,
and run links before hiding a generated receipt. All five review
findings are resolved.
- Browser checks cover loaded images, wrapping columns, overflow,
missing-image fallback, streaming stability, keyboard gallery
navigation, original-file links, and downloads. Fixtures use production
renderers. These checks do not prove deployed persistence or provider
behavior and do not compare visual baselines.
- Local limits: the preflight helper could not finish because Cargo was
absent. The earlier low-memory server typecheck and dependency-free
selector did not replace final-head CI. The now-passing final CI
supplies broad typecheck, build, server, Runner, and E2E coverage.
- Prior independent CI failures remain recorded: one Slack timer case
passed a single diagnosed rerun; its exact timer cause was not proved. A
later chat shard lost its self-hosted runner without a test assertion
failure. The current revision passes that shard and all applicable
gates. Master conflict resolution retained both image-grid and patch CSS
rules.

## Risks

- Images without a run ID remain separate. Messages and other resource
types end each batch.
- Receipt suppression requires a matching preparation marker or the
exact legacy sentence, plus matching attachment, image work product,
run, and author. Tests preserve human messages and native progress text
with attachments. The optional marker uses existing JSON metadata and
needs no database migration.
- Shared gallery and fallback text now have stronger contrast.

## Model Used

OpenAI GPT-6, through Codex. The agent used reasoning, repository tools,
code execution, and browser tests. The runtime did not expose a more
specific model ID or context window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-06 18:42:22 -06:00
Devin FoleyandPaperclip 2a4a001d37 Preserve GitHub access repair guidance for MCP callers (#19655)
Return GitHub permission denials as failed MCP tool results with trusted,
connection-specific recovery guidance. Explain the personal-account and
organization App installation boundary without replaying writes or bypassing
provider-content checks. Align the repair steps with the Permissions UI.

Cover enabled model checks and malicious provider failures across direct
calls, connection tests and stored approvals. Repair the CI-exposed synthetic
onboarding environment fixture without weakening cancellation assertions.

Verified 311 targeted tests, six onboarding Playwright tests, server
typecheck, module boundaries, full CI and Apex 5/5 with all threads resolved.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-06 16:32:27 -08:00
DottaandPaperclip 752cb8d3da fix(ui): contain patch previews and highlight diff source (#19478)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Tasks show documents and file attachments in cards and file views.
> - Patch source needs clear added and removed lines in each view.
> - Long patch lines can exceed the width of an inline preview.
> - This pull request keeps the preview inside the card and adds patch
colors.
> - Users can read a compact patch preview and open the full source.

## Linked Issues or Issue Description

**What happened?**
Patch text can exceed the inline document box. Patch files and patch
code blocks show plain text in the document view.

**Expected behavior**
Keep source inside its preview. Show added lines in green and removed
lines in red. Keep headers, source markers, copy controls, line numbers,
and downloads.

**Steps to reproduce**
1. Open a document card with a diff code block that has long unbroken
lines.
2. Open a `.patch` or `.diff` attachment in a task tab.
3. Open the same patch in the workspace file view.

**Paperclip version or commit**
Source checkout based on `e82210386f1e`.

**Deployment mode**
Built from source. The browser checks use Storybook fixtures.

Related work: #19269 adds inline diffs for file edits. This change uses
its line classifier for document and file source views.

## What Changed

- Share the unified diff line classifier with file and document views.
Track hunk lengths so later file headers keep their metadata style.
- Add safe patch source rendering for patch files and `diff` or `patch`
code blocks.
- Clip long lines with an ellipsis in compact cards. Wrap the full
source in file views.
- Add a Preview patch action for small `.patch` and `.diff` attachments,
including generic binary MIME types. Opening a large artifact list does
not fetch patch source. Bound the actual response bytes. Reject remote
preview URLs and redirects. Keep download and retry fallbacks.
- Add patch text colors with sufficient contrast in both themes. Keep
source-line selection visible over patch row colors. Copy the original
patch text without layout line breaks.
- Add five production component stories and browser checks at two widths
in both themes.
- Document the focused Storybook browser command.

## Verification

- Final base qualification at
`a120e0ada3f1eee9746f47a10384b5aa95f79824`: normally integrated the
already-merged #19610 model-fixture repair
(`8621bcf53ee2e4bbefb7d193add42dd544ca06e8`), preserving the reviewed
patch preview changes. Official preflight passes (1.8s); all 129 focused
tests pass again (23.67s), with UI typecheck and token gates passing.
Current revision CI and reviews are pending; prior-head review receipts
below are not current-head approval.

- Conflict qualification at `befc954a6116f7292506200b96623b673f9385b5`:
normally merged current master
`c66b4a55efdc5675b5a5146197078b94de6407da`, resolving the append-only
`ui/src/index.css` conflict by retaining both the patch rules and all
current agent identity rules. No feature behavior was changed during
resolution. Official isolated preflight passes. The seven focused
component/parser suites pass 129 tests; UI typecheck and token gates
pass. Fresh head CI and reviews are pending.
- Qualification setup limits: the first focused command used the
repository root instead of the UI working directory, so setup failed
before any test ran. It was corrected to the UI directory and actual
attachment-test path. The first UI typecheck found the unbuilt local
plugin SDK; building that checkout’s SDK and repeating typecheck passes.
These setup attempts are not test/typecheck passes.

- Focused Vitest run: 139 tests passed in eight files. This includes
source preservation, HTML escaping, large patches, tabs, Markdown
wrapping, line numbers, and transcript diff parsing. A 200-card check
confirms no patch fetch occurs before a preview request. CRLF
regressions cover no-newline markers and added lines that start with
`+++`.
- `pnpm --filter @paperclipai/ui typecheck`: passed after the plugin SDK
build prerequisite.
- `pnpm check:token-gates`: passed.
- Storybook catalog check and 30 map/catalog tests: passed.
- `STORYBOOK_TEST_BASE_URL=http://localhost:6006 pnpm exec playwright
test patch-preview.spec.ts
--config=tests/storybook-visual/playwright.config.ts --project=stories
--workers=2 --retries=0`: 20 passed on Linux Chromium. The checks
include accessibility, theme colors, overflow, opening a document,
wrapping source, exact clipboard contents, and the selected source line.
Screenshots were inspected and saved to the task.
- The first browser runs found contrast failures in diff text and source
gutters. Patch text tokens and patch gutter opacity now fix those
failures. The final run passes without retries.
- `pnpm verify`: exit 2. The new browser spec is a global CI input.
Required CI must supply broad coverage.
- The local full Storybook build was killed with exit 137. This is not a
build pass. External screenshot baselines are not published, so no
baseline comparison was run.
- Reviews found selected-row background precedence, file headers in
unified patches without git separators, eager patch loading across a
collection, and CRLF no-newline markers. Each finding has a focused fix
and regression checks. The Storybook hunk count now matches its source
lines.
- Prior feature commit `b3cb9d6d3c0423315540514b1ad68325a64c1cc2`: [CI
typecheck, build, tests, runner checks, and E2E
passed](https://github.com/paperclipai/paperclip/actions/runs/42247974154).
[Security scan
passed](https://github.com/paperclipai/paperclip/runs/127995160859).
[Greptile is
5/5](https://github.com/paperclipai/paperclip/pull/19478#issuecomment-6688482753),
and [Paperclip Review is
5/5](https://github.com/paperclipai/paperclip/runs/127998788846). All
six inline review threads are resolved.

## Risks

- Compact previews omit the end of long lines. The full document and
downloads keep the source.
- Patch colors apply to unified diff syntax. Other patch formats keep
literal text and do not have full format parsing.
- The shared classifier keeps the existing task chat behavior.
Regression tests cover that consumer.
- No database, API, or access rules change.

## Model Used

- OpenAI Codex, GPT-6, with reasoning, tool use, and code execution. The
runtime did not expose a more specific model ID or context window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-07 00:06:50 +00:00
DottaandClaude Opus 5.5 961455b9a0 test(restart-chaos): add the K13 restart storm mode (#19629)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Task recovery restarts agent work after a control-plane restart or a
failed run
> - When automatic recovery runs out or is blocked, the user clicks
Retry
> - Retry then started a fresh isolated attempt cloned from committed
HEAD, so the agent lost its uncommitted files, its task branch and the
worktree Git link
> - A restart must never lose the agent's work
> - This pull request makes Retry keep the task's own workspace when
that is safe, and tells the agent where the old files are when a fresh
attempt is still required
> - The benefit is that a Retry continues the real work instead of
starting over

## Linked Issues or Issue Description

**What happened?**
After `native_restart_recovery_blocked` or an exhausted recovery
generation, a Retry ran in
`isolated-workspace-attempts/<company>/<issue>/<run>/workspace`. That
directory was a detached clone of the base ref. It did not contain the
uncommitted edits from the issue's git worktree. It was not on the task
branch. The agent was not told where the old files were. A restart-chaos
wave graded this as an R2 failure (`pre_fault_files_lost`). A second
defect: when a restart hit the bootstrap of a fresh attempt, preparing
the same run again failed with
`isolated_task_directory_binding_mismatch`, so the Retry could not
finish.

**What did you expect?**
Retry continues in the task's own workspace with its edits and `.git`
intact when no other writer can hold it. When a fresh attempt is really
needed, the old directory stays as it was and the agent gets its path
and its changed files. Preparing the same attempt again after a restart
is idempotent.

**Steps to reproduce**
1. Start a native task in a git worktree. Let the agent write an
uncommitted file.
2. Kill the control plane so recovery ends in
`native_restart_recovery_blocked` or an exhausted generation.
3. Click Retry.
4. The successor runs in a fresh clone without the uncommitted file or
the task branch.

Root cause:
- `server/src/services/fresh-workspace-continuation.ts`
`canSelectFreshWorkspace` returns a fresh intent for every task Retry
that has a system recovery record.
- `server/src/services/explicit-native-continuation.ts` (fresh selection
in the source loop) then always persisted `fresh_isolated_workspace`.
- `server/src/services/fresh-workspace-seed.ts`
`materializeFreshWorkspaceSeed` clones the project source at the base
ref with `--no-local`, checks out a detached commit and removes
`origin`. Uncommitted work and the branch are not carried over.
- RC-35: `server/src/services/heartbeat/workspace-preparation.ts`
rejected any reusable existing workspace while `fresh` was true,
including the attempt directory that the same run had already created
and bound.

## What Changed

- New continuation mode `reuse_issue_workspace`. At Retry admission,
after all stop proofs pass, `retainedIssueWorkspaceForRetry` checks that
the task's bound workspace is the single local directory the failed run
used, owned by the task, not a shared workspace, not under remote
storage authority, with no other lease, failed cleanup or running
workspace operation, a settled recovery generation, no workspace-export
failure, and a readable Git state rooted at that directory. Then the
Retry continues there with a fresh session. Otherwise the existing fresh
attempt is kept.
- The reuse decision consumes the generation's one alternative attempt
(`freshAttemptRunId`), sets the Retry strategy to `continue`, and
records `issue.retry_workspace_reused`.
- Environment selection pins the reused workspace binding and fails with
`retry_workspace_binding_mismatch` if the binding changed before
dispatch.
- Fresh fallback: the run's continuation context gets `priorWorkspace`
(path, branch, bounded list of uncommitted files). Git status is read
with optional locks disabled. The prompt tells the agent to inspect it
and carry over the work. The old directory is not changed.
- RC-35: preparing the same fresh attempt again after a restart reuses
the attempt directory that this run already bound.
- `inspectGitCloseReadiness` is exported for the reuse check.
- `doc/task-recovery.md` and `doc/execution-semantics.md` describe the
new contract.

## Verification

- `server/src/services/retry-workspace-continuity.test.ts` (embedded
Postgres, real git worktree with uncommitted edits):
- exhausted-generation Retry and blocked-recovery Retry continue in the
same worktree, `.git` file and branch intact. Both failed before the fix
(readiness was `fresh_isolated_workspace`).
- The fresh attempt is kept when the workspace is unsafe: live lease,
running workspace operation, corrupt `.git`, missing directory, remote
authority, shared workspace, running export generation.
- `workspace-preparation.test.ts`:
- RC-35 red/green: re-preparing the same fresh attempt reuses its bound
directory and keeps files written before the restart. Another run still
gets `isolated_task_directory_binding_mismatch`.
- The fresh fallback sets `priorWorkspace` with the old path and
`unpublished.txt`.
- `environment-selection.test.ts`: the new mode pins the binding and
rejects a changed binding or cwd.
- `packages/adapter-utils/src/server-utils.test.ts`: the prompt names
the retained directory and its files.
- Regression files run locally: `workspace-preparation`,
`environment-selection`, `task-retry`, `quarantined-task-retry`,
`explicit-native-continuation`, `workspace-recovery`.
- Restart-chaos wave: WAVE_RESULT_PLACEHOLDER

## Risks

- A Retry now writes into the old directory instead of a new one. The
checks above prevent a second writer. A stale binding fails the run with
`retry_workspace_binding_mismatch`; it does not write to another
directory.
- If the cause of the failure was inside the workspace itself, reuse
could repeat it. Workspace-export failures, restore failures and
required workspace recovery keep the fresh attempt.
- `priorWorkspace` is an optional new field in the continuation
envelope. Older adapters ignore it.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 18:01:42 -06:00
Nicky LeachandClaude Opus 5.5 97953e9b59 fix(ci): repair the weekly test-weight refresh and extend it to e2e and Runner lanes (#19650)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Contributors wait for PR CI before merge, so CI wall-clock time sets
the pace of work
> - CI test shards are balanced with per-file durations that a weekly
workflow refreshes
> - The weekly refresh has failed since 2026-11-02, and the e2e weights
have not been measured since 2026-09-15. Today e2e shards 3 and 6 take
7.0–7.7 minutes of tests, and shard 1 takes 4.3 minutes
> - A single file can also grow until it sets the time of its shard, and
nothing tells us about it
> - This pull request repairs the refresh, extends it to the e2e and
Runner package lanes, and warns about test files over 180 seconds
> - The benefit is balanced shards that stay balanced, and early notice
of oversized test files

## Linked Issues or Issue Description

**What existing behavior does this improve?**

The weekly refresh of the test-duration weights that balance CI shards.

**Subsystem affected**

CI: `scripts/ci/refresh-test-durations.mjs`,
`.github/workflows/ci-test-durations.yml`, and the Vitest and Playwright
CI reporters.

**Current behavior**

- The 2026-11-02 refresh (run 41408033686) failed. `gh api …/logs` now
refuses output that contains terminal escape sequences.
- Most successful `cloud-readiness.yml` master runs are admission-only
and have no server jobs. The script took the newest 2 runs, so it would
fail with "No server timing jobs found" on most weeks.
- The e2e weights (`scripts/e2e-shard-durations.json`) have no refresh
and are from 2026-09-15.
- No signal tells us when one test file becomes too slow.

**Proposed behavior**

The refresh runs again, selects master runs that ran the server jobs,
refreshes the e2e and Runner package-lane weights from merged PRs, and
lists oversized files. CI writes a warning for each test file over 180
seconds.

**Reason and benefit**

Unbalanced shards add minutes to every PR. The current e2e weights put
472 seconds on the slowest shard. The new weights put about 300 seconds
on each shard.

**Breaking changes**

None. Weights change only shard placement, never which tests run.

## What Changed

- Add `scripts/ci/gh-api.mjs`. It always passes
`--allow-escape-sequences`, retries without the flag when an older `gh`
does not know it, and strips ANSI codes from logs.
- `scripts/ci/refresh-test-durations.mjs`:
- scans up to 50 master runs and uses the 2 newest that ran the server
jobs;
- adds the `--dry-run`, `--pr-body=<path>` and
`--lanes=server,e2e,runner` options;
- validates every lane before it writes a file. If a merged-PR lane has
fewer than 2 valid runs, it keeps the current weights and writes a
warning.
- Add `scripts/ci/pr-run-durations.mjs`. It uses the final passing
`pr.yml` run of each merged PR, and it validates:
  - the PR is merged into `master`;
  - the merge commit is in the checkout;
  - the run's head SHA is the PR's final head SHA;
  - every shard of the run succeeded.

  It parses Playwright per-spec times and Runner per-file times.
- Add `scripts/ci/oversized-test-files.mjs` (Vitest) and
`scripts/ci/oversized-test-files-playwright.mjs` (Playwright). In GitHub
Actions they write a `::warning` for each test file over
`PAPERCLIP_CI_OVERSIZED_TEST_FILE_SECONDS` (default 180). They never
fail a job. The root and Runner Vitest configs and the e2e Playwright
config load them.
- `scripts/ci/test-timing-reporter.mjs`: move the per-file duration
calculation into an exported `moduleDurationMs()` helper. The behavior
does not change.
- `.github/workflows/ci-test-durations.yml` and
`.github/ci-duration-pr.md`:
  - the bot PR can also update the e2e and Runner manifests;
  - the PR body lists oversized files;
  - the verify step also runs the new tests and the e2e shard tests.
- Refresh `scripts/e2e-shard-durations.json`: 56 of 56 specs measured
(before: 30), from merged PRs #19613, #19191 and #19176. The file now
uses the shared schema with `sources` and `coverage`.

## Verification

- `node --test scripts/ci/test-selection-pr-timings.test.mjs
scripts/ci/test-selection-timings.test.mjs
scripts/__tests__/e2e-shard.test.mjs`: 32/32 pass. These include the e2e
shard balance check with the new weights.
- `node scripts/ci/refresh-test-durations.mjs --dry-run` against real
GitHub data, all lanes:

| Manifest | Measured | Sources |
| --- | --- | --- |
| `general-server-shard-durations.json` | 1488/1534 files | master runs
42296943853, 42294279748 |
| `serialized-shard-durations.json` | 167/167 files | the same two runs
|
| `e2e-shard-durations.json` | 56/56 specs | PR runs 42296374365,
42297700476, 42296994179 |
| Runner `vitest-lane-durations.json` | 310/311 files | the same three
PR runs |

- e2e placement over 8 shards, with the new medians as the real costs:
  - with the old weights, the slowest shard has 472 seconds;
  - with the new weights, every shard has about 300 seconds.
- The dry run lists 12 files over 180 seconds. The largest is
`server/src/services/workspace-recovery.test.ts`, at 636 seconds.
- In this PR, only the e2e weight file changes. The weekly run refreshes
the server weights. The Runner manifest is added by #19646.

## Risks

- The `--allow-escape-sequences` path ran only in unit tests. The local
`gh` did not support the flag, so the real dry run used the fallback.
The next weekly run is the first real test of the flag.
- The changes to the bot-PR shell steps did not run in Actions yet.
- The e2e weights come from 3 PR runs. Real shard times vary more than
the estimate.
- This PR and #19637 both edit the `reporters` lists in
`vitest.config.ts` and `packages/paperclip-runner/vitest.config.ts`. The
second PR to merge needs a small merge resolution.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
code execution. A Claude sub-agent wrote the change, and the parent
session reviewed it.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 15:50:02 -08:00
DottaandClaude Opus 5.5 fb1b458409 fix(runner-e2e): drop the Rust dependency layer that filled the image builder disk (#19669)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Runner Full-Stack E2E runs Daytona cells from a signed,
content-addressed runner image
> - #19545 made the image build faster. One part of it added a Rust
dependency layer that keeps Cargo's target and registry trees in the
builder
> - The first default-branch build after the merge ran out of builder
disk while it exported the registry cache
> - That run pushed the image tag before the failure, so the tag has no
signature, and campaigns that reuse it fail `cosign verify`
> - This pull request removes only the dependency layer and keeps the
parts of #19545 that add no disk
> - The benefit is a working default-branch image build again, a new
content ID that replaces the unsigned tag, and the measured 16.5 to 11.6
minute improvement

## Linked Issues or Issue Description

Refs #19545. Related #18494 (the same builder disk limit).

**What happened?**

The default-branch `image_only` dispatch, run 42286903290 (job
128202722275), built and pushed
`ghcr.io/paperclipai/paperclip-daytona-runner:e2e-content-7154c0dfc9abb297d8d638a982326c1435dc4440c25112afacc559712d4430b3`
(`sha256:2767e91c5279...`). The registry cache export then failed with
`ResourceExhausted: ... no space left on device` (exit 102). `preparing
build cache for export` took 182 s, against 87 s before #19545. The job
stopped before `cosign sign`. The dependency layer retained `target` 852
MB and Cargo registry 333 MB (measured in run 42271236140).

**Expected behavior**

Default-branch image builds publish the image, write the cache, and sign
the digest within the builder disk.

**Steps to reproduce**

Dispatch `runner-full-stack-e2e.yml` from master with `-f
image_only=true` and a Daytona selector on `840838476a`. Read the `Reuse
or publish immutable image` step log.

## What Changed

- `docker/daytona-runner/Dockerfile`: remove the placeholder-target
dependency layer. The runnerd stage is again the single build layer from
before #19545, so its cache key matches the earlier default-branch
cache. Restore and extend the disk comment.
- Keep the content-ID check above the `COPY --from` steps and keep `COPY
--link`. They add no retained data.
- `tests/runner-e2e/README.md`: describe the two remaining layout rules
and why there is no dependency layer.
- The Dockerfile is a content-ID input, so the next build gets a new
tag. No campaign reuses the unsigned tag.

## Verification

- `vitest run tests/runner-e2e/daytona-image.test.ts
tests/runner-e2e/workflow-image-only.test.ts`: 20/20 pass.
- Branch evidence for the kept parts: job 128077687214 took 11.6 min,
against 16.5 min before. The cached-layer download overlapped the
compile.
- After merge, an `image_only` dispatch from master must publish, write
the cache and sign. I did not use a dispatch before merge, because only
a default-branch build writes the cache, and the cache export is what
failed.

## Risks

- Runner source changes again recompile the full Rust dependency graph
(about 5 min). To get that saving back, the builder needs more free disk
first (for example, remove unused preinstalled toolchains in the image
job, or use a larger RunsOn volume). That is a separate change.
- The unsigned tag stays in GHCR. Nothing selects it after this change,
because the content ID changes.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, GitHub CLI). Extended reasoning on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 17:49:39 -06:00
Devin FoleyandPaperclip 8621bcf53e Cancel subscription sign-in requests when leaving agent setup (#19610)
Back during pending subscription sign-in could accept a late response and
reopen agent setup. Abort pending reads, release abandoned sessions, and
ignore stale responses. Wait for cleanup before starting a replacement,
and preserve sessions belonging to another account or environment.

Cover deferred resume/start/prompt responses for both providers, local
login cancellation, quick reselection, and browser request abort. Stabilize
Runner fixtures and normalization assertions exposed by integration CI.

Verified: 483 focused UI tests, six onboarding browser cases, UI typecheck
and token gates, Runner fixture checks, and the complete CI run on
221f2e3eb7. Greptile scored 5/5;
all review threads are resolved. Earlier failures and recovery evidence
remain in the PR description.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-06 14:31:11 -08:00
e482f0f632 fix(server): shed only retryable object-journal reads under load (#19176)
Client polling can exhaust the shared object journal queue and reject log persistence. Bound ordinary object-log and trace reads, keep persistence and internal verification queued, and return retryable 503 responses when reads are busy. Leave feedback exports unsent until every log page is available.

Validation: 72 latest-master affected tests and 10 workflow tests pass locally, with the workflow fixture passing strict TypeScript checks. The preceding broader run passed 280 server tests with two opt-in S3 skips. All applicable checks on d0a937768d are green (56 success, three skipped); exact-head Apex review is 5/5 with no unresolved findings. Background persistence still has no fixed queue bound during a prolonged storage outage.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-06 13:47:25 -08:00
DottaandClaude Opus 5.5 02a2299173 feat(codex): default Codex to gpt-6.1-sol, half the gpt-5.6-sol price (#19623)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Codex agents run through Paperclip Runner or the legacy
`codex_local` harness, and the runner E2E suites run paid Codex cells
> - The shared Codex default and many selections preferred `gpt-5.6-sol`
> - The reviewed OpenAI rate card in
`server/src/services/codex-pricing.ts` (`openai-standard-2026-09-30`)
prices `gpt-6.1-sol` at $2 input / $0.10 cached / $10 output per 1M
tokens, and `gpt-5.6-sol` at $4 / $0.40 / $20
> - So every default run and every paid Codex eval cell cost twice as
much as necessary
> - This pull request moves every place that prefers or selects
`gpt-5.6-sol` to `gpt-6.1-sol`, and keeps historical facts, rate cards
and compatibility fallbacks on `gpt-5.6-sol`
> - The benefit is half the token price for new default Codex agents and
for the Codex E2E and TestOps cells

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The default Codex model for new agents and the Codex model that the
runner E2E suites and TestOps plans select.

**Subsystem affected**
Adapters (`adapter-utils`, `codex-local`), server native runtime
fallback, UI AI connection pools, Paperclip Runner devtools and live
evals, runner E2E catalog and TestOps mappings.

**Current behavior**
New Codex agents without an explicit model store `gpt-5.6-sol`. The
runner E2E `runner-codex` profile, the Reliability `queue-steer` plan
cell, the Product regression TestOps mapping and the ACPX Codex
qualification selection all use `gpt-5.6-sol`.

**Proposed behavior**
They use `gpt-6.1-sol`. Existing agents that store a model keep it.
`gpt-5.6-sol` stays selectable.

**Reason and benefit**
`gpt-6.1-sol` costs half as much per token on the in-repo reviewed rate
card. The current OpenAI pricing page (retrieved 2026-11-06) shows the
same short-context rates and also lists long-context rates of $4 / $0.20
/ $15 for `gpt-6.1-sol` against $8 / $0.80 / $30 for `gpt-5.6-sol`.

**Breaking changes**
None for stored configurations. See Risks.

## What Changed

Changed defaults and selections (all now `gpt-6.1-sol`):

- `PAPERCLIP_RUNNER_DEFAULT_MODELS.codex` in
`packages/adapter-utils/src/paperclip-runner-permissions.ts`. This also
changes `DEFAULT_CODEX_LOCAL_MODEL`,
`resolvePaperclipRunnerModel("codex", …)`, runner config normalization,
`codex_local` → Paperclip Runner conversion, the UI new-agent and
agent-config defaults, the composer default model and the public MCP
effort lookup.
- `codex-local` `models`: the 5.6 entry is now the literal
`gpt-5.6-sol`, because it used to be `DEFAULT_CODEX_LOCAL_MODEL`. The
list order and contents do not change.
- AI connection pool: a new OpenAI member profile uses
`PAPERCLIP_RUNNER_DEFAULT_MODELS.codex` instead of a hard-coded
`gpt-5.6-sol`.
- Paperclip Runner devtools ACPX Codex default and preset, the
`runner-codex-sol` local provider smoke, and the `acpx-codex-sol` live
workflow candidate.
- `tests/acpx-qualification-models.ts` Codex selection (runner E2E and
runner acceptance ACPX Codex profiles).
- Runner E2E: the `runner-codex` profile follows the default, so all
native Codex cells move. This includes the `runner-codex` cells among
the 52 standard `everyday-workflows` cells and the seven Codex
reliability cells. `runner-codex-mini` stays on `gpt-5.4-mini`. The
Reliability plan `queue-steer` `steeringProfile` is now
`['runner-codex', 'gpt-6.1-sol']`. Its executionId
(`everyday-workflows.runner-codex.daytona.queue-steer`), its fullName
and its TestOps case identity do not change. No separate profile is
necessary.
- `product-regression-testops.mjs`: the declared model for every
`runner-codex` case is now `gpt-6.1-sol` (it was already that for
`configured-repository`).
- `tests/runner-e2e/installed-cli.ts`: the credential-free installed
startup proof opens a session on the default model.
- `tests/runner-e2e/scheduled-prices.json`: add a `gpt-6.1-sol` price
(`openai-standard-2026-11-06`, including 30m/12h cache writes and
long-context rates from the OpenAI pricing page). The weekly scheduled
Codex cells need an exact price for their model, or admission fails
closed.
- Storybook: the composer model picker fixtures, the connection pool
stories and host, and the agent settings preview show `gpt-6.1-sol` as
the default.
- Docs: `doc/DEVELOPING.md` (the Codex conversion default) and
`tests/runner-e2e/EVERYDAY-WORKFLOWS.md` (local matrix, reliability
extras, `queue-steer` row).
- Tests that assert the default now expect `gpt-6.1-sol`. New
assertions: the default has the `0.159.0` CLI floor, the compatibility
fallback is floor-free, the `queue-steer` plan cell runs `runner-codex`
on `gpt-6.1-sol`, and all native Codex everyday cells (standard and
reliability) use `gpt-6.1-sol`.

Deliberately kept on `gpt-5.6-sol`:

- Rate cards: `server/src/services/codex-pricing.ts` and
`packages/paperclip-runner/src/evals/model-pricing.ts`. Old receipts
need them. (The runner eval rate card has no `gpt-6.1-sol` entry. An
unpriced candidate is reported as unpriced, not as zero. I did not add a
rate there because that card has its own version and source.)
- The remote Codex CLI compatibility fallback. A new
`CODEX_LOCAL_COMPATIBILITY_FALLBACK_MODEL = "gpt-5.6-sol"` replaces the
default in `codexModelFallbackCandidates`. `gpt-6.1-sol` needs Codex
0.159.0 or newer, so it cannot be the floor-free last resort. Fallback
behavior is unchanged.
- `tests/runner-e2e/provider-usage.ts`: the implicit request
cache-policy attribution stays qualified only for `gpt-5.6-sol`. I did
not verify that policy for `gpt-6.1-sol`. A `gpt-6.1-sol` response that
omits the write-TTL split keeps its cache-write cost unknown, which is
the existing behavior for unqualified models.
- `tests/runner-e2e/scheduled-measurements.json`: historical
`gpt-5.6-sol` measurements. They no longer match the current suite
definitions, so the forecast reports them as historical only. The
forecast tests now build their synthetic cohorts on the current
definition hash.
- Model catalogs that list `gpt-5.6-sol` as available (`codex-local`,
`cursor-local`, `opencode-local`, Speko voice diagnostics, Pi picker
fixture).
- The `gpt-5.6` → `gpt-5.6-sol` alias for old stored configs.
- Dated or evidence docs: `doc/plans/*`, `doc/adapter-model-audit-*`,
`QUALIFICATION-*`, `doc/connections/MEMORY.md`,
`doc/connections/TASK-REVIEWS-VERIFICATION.md`,
`tests/runner-e2e/CHAT-FEATURES.md`, the Slack eval evidence, and the
S49 soak signature text (the soak failure happened on `gpt-5.6-sol`).
- Test fixtures that parse, price, migrate or round-trip an existing
`gpt-5.6-sol` value (Rust runner tests, driver tests, pricing tests,
stored-config tests, stories that show an existing agent).

## Verification

- `npx vitest run` on every server, UI, adapter-utils and codex-local
test file that mentions `gpt-5.6-sol` or the default, plus all of
`packages/adapter-utils` and `packages/adapters/codex-local`: 5,483
passed. Three timeouts under load (`sandbox-managed-runtime`,
`acp.quota`) passed when rerun.
- `packages/paperclip-runner`: `vitest run` for `src/eval`, `src/evals`,
`src/drivers/acpx`, `src/drivers/codex`, `src/cli`, `src/contracts`,
`src/backends`: 2,641 passed.
- `node --test tests/runner-e2e/reliability-testops.test.mjs
tests/runner-e2e/product-regression-testops.test.mjs`: 28 passed.
- `node tests/runner-e2e/run-unit.mjs`: all model, catalog, reliability,
scheduled and TestOps tests pass. Failures remain only in
`copilot-protection-flow`, `denied-target-isolation-flow`,
`pi-controls-flow`, `workflow-target` and `unit-launcher`. These five
also fail on unmodified master in this local environment (filesystem
watch and git environment), and they do not touch models.
- `vitest run --config tests/runner-acceptance/vitest.config.ts`: 19
passed.
- `pnpm test:e2e:runner -- --list --suite everyday-workflows --group
reliability`: the seven `everyday-workflows.runner-codex.daytona.<case>`
cells show `gpt-6.1-sol` with `OPENAI_API_KEY,DAYTONA_API_KEY`.
`--matrix-json` gives 16 reliability cells with the same credentials as
before. `--list --all` shows no `gpt-5.6-sol` cell.
- Typecheck: `adapter-utils`, `adapter-codex-local`, `server`, `ui`,
`paperclip-runner` (TypeScript), and `pnpm test:e2e:runner:typecheck`.
- `pnpm check:token-gates`: clean.
- Not run: paid cells, Rust tests (no Rust change), the browser suites,
and the full `pnpm -r typecheck` / `pnpm build` (CI runs them).

## Risks

- Agents without an explicit model change model on their next save or
normalization, and Paperclip Runner rows with a blank model use
`gpt-6.1-sol` at execution. Stored models, including `gpt-5.6-sol`, are
never rewritten. No migration is necessary.
- `gpt-6.1-sol` needs Codex CLI 0.159.0 or newer with ChatGPT sign-in.
The runner pins `@openai/codex@0.160.0`. On a remote target with an
older image CLI, the existing compatibility check falls back to
`gpt-6-sol` and then `gpt-5.6-sol`, and that probe now runs for
default-model agents too. A legacy `codex_local` agent created with the
new default on a host with an older Codex CLI gets the existing
actionable "upgrade Codex" Test result.
- Qualification: the native Codex profile uses `adapter_constant` model
qualification, so it follows the default. No qualification gate is
dropped. The ACPX Codex qualification selection moves to `gpt-6.1-sol`.
Its next qualification run uses the new model. The provider-usage
cache-policy attribution stays limited to `gpt-5.6-sol`.
- Runner E2E history: the Codex cells change their suite definition
hashes, so earlier `gpt-5.6-sol` results are a separate cohort and do
not count as current.
- The `gpt-6.1-sol` scheduled price uses the file's existing `expiresAt`
(2026-11-19). It needs the same review as the other entries before then.
- No `.github/` change. The Reliability queue-steer cell keeps the
`runner-codex` profile id, so the existing workflow credential and setup
steps still match.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, running in
Claude Code with tool use (shell, file edits, web fetch of the OpenAI
pricing page).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Related: #19604 (commits the Reliability TestOps identities; this PR
changes only `steeringProfile` in the same file).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 15:35:34 -06:00
DottaandClaude Opus 5.5 8479929f62 test(restart-chaos): add an RC-06 mask, wave spend and replay masks (#19559)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart chaos harness (`tests/restart-chaos/`) restarts the
control plane during real agent tasks and grades each task on rules
R1–R6
> - One known product bug (RC-06) leaves a completed run "awaiting
accounting". The company hard stop then refuses all new work, so one
stuck row fails every later task in the wave
> - That masks every other restart bug in the wave. Waves also had no
record of what they spent
> - This pull request adds an opt-in RC-06 mask, records wave spend, and
lets a replay keep or add the mask
> - The benefit is that chaos waves can find new restart bugs while
RC-06 is open, and stay inside a known spend

## Linked Issues or Issue Description

Refs #19477 (the Phase 1 harness this extends).

**What existing behavior does this improve?**
Restart chaos waves. One stuck accounting row (RC-06) hard-stops the
isolated Chaos Lab company. All later tasks then rest blocked "over
budget", and a Retry is refused with "Company cannot start work while
completed runs await accounting". Two waves went 2/6 and 0/6 for this
one reason.

**Subsystem affected**
Test harness only: `tests/restart-chaos/` (config, seed, scheduler,
ledger, supervisor, main, README).

**Current behavior**
- No way to see past RC-06 without disabling the budget.
- Wave spend is not recorded.
- `--replay` rejected any mask flag, and it was not documented that the
saved plan carries the mask.

**Proposed behavior**
- `--rc06-masked` keeps the budget amount and turns off only the hard
stop on the isolated company. Masked tasks are labeled in the ledger and
summary.
- Each wave records its spend.
- A replay keeps the source plan's masks and may add the RC-06 mask.

## What Changed

- `--rc06-masked` (wave mode): after the company is created, the seed
upserts the company budget policy with the same amount and
`hardStopEnabled: false`. It writes `seed_mask_rc06` to the timeline.
The plan stores `options.masks = ["rc06"]`.
- The accounting gate in `budgets.ts` (`observedBlocks`) applies only
when `hardStopEnabled` is true, so the mask lifts it. No product code
changes.
- Ledger lines carry `masks`. `SUMMARY.md` labels the cell and the
latest-wave row `rc06-masked`.
- Wave spend: before the sweep, `main.ts` reads
`/api/companies/:id/costs/summary?period=all` (each wave owns its
company), writes `wave_spend` to the timeline and prints it. If the wave
stops early, cleanup records it once, best effort, before the server
stops.
- Load: #19513 already records `loadAvg1m` on each fault receipt. This
branch first added the same value under another name, and the rebase
removed it. The README now documents `loadAvg1m` and the load-suspect
threshold.
- Replay: `--replay <wave>/<task>` keeps the masks saved in the source
`plan.json`. `--replay … --rc06-masked` adds the RC-06 mask. A replay
never drops a mask.
- README documents the flag, spend, load and the replay rule.
- New unit tests: replay keeps saved masks and the replay flag adds the
mask; the masked seed posts the company policy with the same amount and
`hardStopEnabled: false` and records `seed_mask_rc06`, and an unmasked
seed makes no policy call.

## Verification

- `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`:
71/71 pass (after rebase onto master with #19513).
- `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: pass.
- Live: 4 masked local waves ran with this code (seeds 3307, 4409, 5501,
6619; Codex, ACPX Claude and OpenCode).
- `seed_mask_rc06` appeared and no task was blocked by the accounting
gate, even though `pendingRunCount` was 1–8 at wave end.
  - `wave_spend` was recorded each time (38¢–127¢ per wave).
  - The 1-minute load average was recorded on every fault (15–58).
- Before the mask, the same harness gave 2/6 and 0/6, every failure
RC-06. With the mask, the waves found other restart failures, all since
fixed on master: RC-14, RC-16, RC-20 and RC-21.

## Risks

- Low risk. Test harness only, and the mask is opt-in.
- A masked wave has no product hard stop. The harness still records
spend for each wave, and the operator must watch it. The mask applies
only to the isolated, temporary Chaos Lab company. It never touches
another instance.
- Masked results do not count as RC-06 coverage. The ledger labels them
so they are not mixed with unmasked cells.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edit). Extended reasoning on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 15:27:12 -06:00
DottaandPaperclip 6148fbd31f feat(ui): distinguish GitHub resources in inline Markdown links (#19476)
## Thinking Path

> - Paperclip helps people manage AI agents and their work.
> - Task conversations and documents use the shared Markdown renderer.
> - GitHub repository, pull request, and issue links currently use the
same icon.
> - Long pasted URLs make these references hard to scan in a
conversation.
> - This change identifies each resource from its URL and shows a
compact label with a distinct icon.
> - The exact destination and custom link text stay intact. Connected
status data keeps its existing behavior.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Inline GitHub links in rendered Markdown.

**Current behavior**

Repositories, pull requests, and issues all use the GitHub mark. Pasted
resource links show the full URL.

**Proposed behavior**

Pull requests use the pull request icon. Issues use the issue icon.
Repositories and other GitHub pages keep the brand mark. Pasted resource
URLs show `owner/repository#number` or `owner/repository`. Custom labels
keep their text and formatting.

**Reason and benefit**

Readers can identify GitHub resource types and scan references in a
narrow conversation. Full URLs, query strings, and comment anchors stay
available through the link destination. Link rendering makes no remote
metadata requests.

**Breaking changes**

The visible text of a bare GitHub resource URL becomes compact. The
destination does not change. No API, database, or connection changes are
required.

Searches found no duplicate work for GitHub inline resource
presentation. Related work: [same-origin Markdown
routing](https://github.com/paperclipai/paperclip/pull/17864). This
change preserves that route handling.

## What Changed

- Add a pure GitHub URL classifier with exact host checks and resource
labels.
- Move ordinary URL presentation into `MarkdownUrlLink`. Keep internal
references, workspace links, custom resolvers, and connected metadata in
their existing order.
- Add separate icons and accessible resource names. Preserve authored
Markdown titles and formatted labels.
- Add Playground, Narrow, and Streaming Storybook examples that mount
`MarkdownBody`.
- Add parser and renderer tests, eight Chromium browser cases, and the
browser command in the development guide.
- Consider Open Graph loading. Use the existing connected metadata path
for remote status and titles. Add no arbitrary page fetches.

## Verification

- Focused Vitest coverage: 129 passing tests across nine files for
GitHub parsing, Markdown rendering, connected metadata, document
rendering, and workspace-file rendering.
- `pnpm check:token-gates`.
- `pnpm --filter @paperclipai/ui typecheck`.
- `pnpm --filter @paperclipai/ui build`.
- `pnpm build-storybook`.
- Storybook catalog generation/check and 30 catalog/map tests.
- Site-route regression: Stars, Education, and enterprise pages keep
general GitHub branding and their full URL text. Three parser cases
failed before the fix; parser and renderer cases now pass.
- Streaming regression: three resource cases failed before the fix and
pass after it. The fix preserves the generated word span and custom
formatted labels.
- Chromium: `STORYBOOK_TEST_BASE_URL=http://localhost:6106 pnpm exec
playwright test inline-links.spec.ts
--config=tests/storybook-visual/playwright.config.ts --project=stories`.
- Browser scope: eight passing cases for ordinary/streaming modes,
dark/light at desktop and 390px mobile, completed play assertions, exact
comment destinations, safe external attributes, keyboard focus,
wrapping, and no GitHub requests. Screenshots are saved as task
artifacts; no review images are committed.
- The first browser attempt in the resumed environment could not launch
Chromium because its executable was missing. The browser was installed,
and all eight cases then passed.
- The browser cases use synthetic text and one connected metadata
fixture. They do not prove provider authentication or server
persistence. No visual baseline comparison is claimed; the repository
baseline manifest is unpublished.
- Initial typecheck found two unsupported story assertion options and an
unbuilt SDK dependency in the fresh checkout. The options were removed
and the SDK was built before the successful retry.
- `pnpm verify` returned 2 because the new browser spec requires broad
CI coverage. Focused tests and the browser suite supply local evidence.
Required CI supplies broad coverage.

## Risks

- Compact labels hide URL detail from the visible line. The exact href
and default tooltip preserve that detail.
- GitHub site routes can change. Unknown paths retain the general GitHub
treatment. Lookalike hosts and credential-bearing URLs do not get GitHub
resource treatment.
- Connected external-object status and title rendering keeps priority
over URL-derived presentation.

## Model Used

OpenAI Codex, GPT-6 (`gpt-6` family), with reasoning, tool use, browser
checks, and code execution. The exact deployment ID and context window
are not exposed in this run.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-06 15:26:10 -06:00
DottaandClaude Opus 5.5 38068de553 chore(evals): commit the Reliability TestOps plan and case identities (#19604)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Runner E2E cases run from Allure TestOps plans through
`runner-full-stack-e2e.yml`
> - #19549 added the "Reliability — staging soak" plan. Its selection
fails closed until real TestOps identities are committed
> - Registration run
https://github.com/paperclipai/paperclip/actions/runs/42289820474 on
master (02214b1a3c) completed. It created plan 135 with cases 885–891
and assigned the plan to the standing GitHub job 166
> - This pull request commits those identities into
`reliability-testops.mjs`
> - The benefit is that a TestOps launch of plan 135 now selects the
seven reliability cells

## Linked Issues or Issue Description

Refs #19549
Refs #19601

**What happened?**
The Reliability TestOps plan exists in TestOps, but the repository still
had an empty `reliabilityRegistration`. A launch therefore failed closed
with `product_regression_testops_reliability_unregistered`.

**Expected behavior**
A launch of plan 135 maps each TestOps case ID to its runner E2E cell
and runs it.

**Steps to reproduce**
1. Launch plan 135 from TestOps before this change.
2. The selection fails with `reliability_unregistered`.

## What Changed

- `tests/runner-e2e/reliability-testops.mjs`: set
`reliabilityRegistration` to plan 135 and the seven case IDs from the
registration receipt.

## Verification

- `node --test tests/runner-e2e/reliability-testops.test.mjs
tests/runner-e2e/product-regression-testops.test.mjs`: 28/28 pass.
- `registeredReliabilityCases()` with plan-2 IDs reserved resolves all
seven cells: 885 interrupt-redirect, 886 reassign-midrun, 887
three-turn-revise, 888 long-silent-command, 889 tree-pause-release, 890
queue-steer (Codex), 891 queued-across-restart.
- The registration receipt from run 42289820474 reports `complete: true`
with assignment and parameters created.

## Risks

- Low risk. The change only adds identities. A launch of plan 135 runs
seven paid cells under the existing admission and budget caps. Nothing
runs per PR or on a schedule.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code with tool use
(shell, GitHub CLI), standard context.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 15:22:02 -06:00
DottaandClaude Opus 5.5 586caa5f49 fix(native): hand a shutdown-interrupted startup to the next controller (#19596)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The native Runner path runs an agent turn through a runnerd process
that the control plane owns
> - A restart of the control plane must keep the task going without a
click, if that is safe
> - A graceful shutdown closes the HTTP listener first and then drains
the heartbeat runs
> - A native run that was between provider dispatch and runnerd backend
setup in that gap failed with `local_runtime_api_listener_unavailable`,
so the drain did not see it as running and did not suspend it for the
next boot
> - This pull request makes that failure a restart detach: the run stays
running with its claim, the drain suspends it, and the next controller
adopts it
> - The benefit is that a SIGTERM in this window no longer spends a
failure and a same-run retry, and the task continues by itself

## Linked Issues or Issue Description

Refs #19530 (the same restart window, before provider dispatch).

**What happened?**

In a restart chaos replay (ACPX Claude, local runner), a
restart-recovered run got SIGTERM while
`createRunnerdBackendWithinSessionClaim` built the runnerd backend.
`resolveLocalRuntimeApiUrl()` threw
`local_runtime_api_listener_unavailable` because shutdown had already
closed the listener. The run settled `failed`. The drain then logged
`restartSuspendedRunIds: []`. The next boot used the one bootstrap
retry. A second SIGTERM hit the same window, and the task stopped at a
manual Retry.

**Expected behavior**

A run that this process cannot continue because it is shutting down is
handed to the next controller. It is not settled as failed.

**Steps to reproduce**

1. Start a native Runner task and restart the server (K1) so that
startup recovery resumes the run.
2. Send SIGTERM while the resumed run sets up its runnerd backend.
3. See the run fail with `local_runtime_api_listener_unavailable` and
the drain report no suspended runs.

## What Changed

- `server/src/runtime-api.ts`: `resolveLocalRuntimeApiUrl()` throws the
new `LocalRuntimeApiListenerClosedError` when this process had a
listener and closed it (the `listening` flag is false or the `close`
event fired). The error message stays
`local_runtime_api_listener_unavailable`, so the existing classifiers do
not change. A listener that never started still throws the plain error.
- `server/src/services/native-runtime/native-session-executor.ts`: for a
local Runner, the startup failure handler treats
`LocalRuntimeApiListenerClosedError` like a restart detach. It stops
lease renewal, keeps the claim and throws
`NativeControllerDetachedForRestartError`. Heartbeat already keeps such
a run `running`. A remote Runner keeps the same-run bootstrap retry from
#19462, because startup recovery has no process evidence to adopt a
claimed remote startup
(`native-runner-restart-preparation.integration.test.ts` still covers
that path).
- `doc/execution-semantics.md`: describe the new rule.
- Tests: a heartbeat regression that closes the real runtime API
listener before backend setup, then drains and restarts; a unit test for
the closed versus never-started listener; two `vi.mock` factories keep
the real module exports.

## Verification

- Red before the fix: `pnpm exec vitest run
src/__tests__/heartbeat-native-restart-suspension.test.ts -t listener`
failed with `restartSuspendedRunIds: []` and the run `failed`
(`local_runtime_api_listener_unavailable`, `bootstrap_retry`).
- Green after the fix: the same test passes. The drain suspends the run,
the run stays `running` with `awaiting_runner_reattach`, and the next
controller claims it and reaches the backend once.
- `pnpm exec vitest run
src/__tests__/heartbeat-native-restart-suspension.test.ts
src/http/server.test.ts
src/__tests__/server-startup-feedback-export.test.ts
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/native-recovered-owner-followup.test.ts`:
1333 tests pass. One timing test in `native-session-executor.test.ts`
failed once under host load average ~30 and passed alone.
- `pnpm exec tsc --noEmit -p server/tsconfig.json`: pass.
- Live, restart chaos harness, local ACPX Claude (`claude-sonnet-5`),
`--rc06-masked`, this branch at a35ec8b845 (before the remote-scope
commit 5dc63f24fb, which does not change the local path). The original
replay schedule no longer reached the window, because the task now ends
before the second fault. So I moved the faults earlier: K1 at 10 s, then
K2 as soon as the recovered server is ready.
- K1 + K2, 2 runs: both runs PASS with **R1 pass**. Each drain logged
`restartSuspendedRunIds: [<run>]` (before: `[]`). The next boot
classified the run `resume_dead_runner` and the same run completed. No
`local_runtime_api_listener_unavailable` failure.
- K1 + K2 + K2, 2 runs: each drain suspended the run. The fourth boot
blocked the run with `native_restart_recovery_blocked`, because three
restart claims used the three-attempt recovery budget. Retry then
completed the task (R3 pass). That budget count is a separate open item.
This pull request does not change it.

## Risks

- Low. The new path applies only after this process closed its own
listener, which happens only during shutdown. The run then waits for the
drain or the next boot, as for the other restart detaches.
- If shutdown does not reach the drain (for example SIGKILL after the
listener closes), the run keeps its live claim until the claim expires.
Then startup recovery handles it as a hard restart. This is the same as
any run that is running at a hard kill.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, file edits, GitHub CLI). Extended reasoning on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 15:10:44 -06:00
github-actions[bot]andlockfile-bot e28158b353 chore(lockfile): refresh pnpm lockfiles (#19617)
## Thinking Path

> - Paperclip uses a dependency lock for reproducible builds.
> - Dependency changes on master need matching lock entries.
> - Paid eval reports also pin a reviewed offline correction.
> - Reports must install before paid model work starts.
> - This automated PR keeps the lock and report proof together.

## Linked Issues or Issue Description

**What happened?**

Dependency manifests on master changed after the last committed lock
refresh.

**Expected behavior**

Frozen installs must match those manifests. Eval reports must retain
matching offline lock proof.

**Steps to reproduce**

Run a frozen install before and after this lock refresh.

## What Changed

- Refresh the dependency lock from the manifests on master.
- Refresh the sandbox provider lockfiles from their manifests on master.
- Include the matching offline report correction and hashes.

## Verification

- Pinned pnpm resolution completed with lifecycle scripts disabled.
- The report proof refresh accepted the reviewed committed baseline.
- The workflow rejected changes outside the lock and report proof files.

## Risks

The lockfile bot merges this PR without CI or review when every added
package version is at least 3 days old on npm. Otherwise the PR waits
for a maintainer, and the daily refresh tries again. The bot merges only
the exact commit this workflow generated from manifests already reviewed
on master. Unknown report lock baselines remain blocked. This update
changes no permissions or paid enable gates.

## Model Used

None — generated by pnpm and the deterministic report-lock updater.

## Checklist

- [x] Thinking path and problem description included
- [x] Generated lock and matching report proof included
- [x] Release-age gate decides between bot merge and maintainer review

Co-authored-by: lockfile-bot <lockfile-bot@users.noreply.github.com>
2026-11-06 21:08:43 +00:00
DottaandPaperclip 4d554f0821 test(runner): set default inputs in workflow target fixture (#19602)
## Thinking Path

> - Paperclip manages work performed by AI agents.
> - Its Runner support tests verify workflow authorization and target
pinning.
> - The workflow now passes two Boolean inputs to its shell guard.
> - The target fixture runs that guard with `set -u` but omits those
inputs.
> - Five cases stop on an unset variable before they reach their
assertions.
> - This change supplies the workflow's normal false defaults in the
fixture.
> - The production guard and all target-pinning assertions remain
unchanged.

## Linked Issues or Issue Description

Refs #19537, #19576.

**What happened?**

The required Runner support suite fails five `workflow-target.test.ts`
cases with `IMAGE_ONLY: unbound variable`. The real workflow supplies
`IMAGE_ONLY` and `TESTOPS_REQUESTED`. The test fixture does not supply
either value.

**Expected behavior**

The fixture uses the normal workflow defaults and reaches the target
authorization assertions.

**Steps to reproduce**

Run `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts
workflow-target.test.ts` on master
`4938a930d66846094810cbed45d2fd33c0d14c2d`.

## What Changed

- Set `IMAGE_ONLY` and `TESTOPS_REQUESTED` to `false` in the target
fixture's child environment.

## Verification

- RED: the unchanged current-master fixture had five failures and four
passes. Bash stopped on `IMAGE_ONLY: unbound variable` before target
authorization ran (220ms).
- GREEN: all nine cases pass after the one-line fixture repair (168ms).
Existing checks still reject a custom bounded target and preserve the
queued commit for scheduled and bounded manual runs.
- Local evidence uses the actual test and workflow files with installed
Vitest 5.0.3 and a temporary focused config. It omits the unrelated
provider-bundle global setup. No provider, external service, or paid
workflow runs.
- The [original required CI
failure](https://github.com/paperclipai/paperclip/actions/runs/42288042001/job/128134318743)
has the same five Bash failures. Its evidence is retained; no unchanged
retry was used.
- Whitespace checks pass. Exact head
`f1d178ad483a49f8dd087a1c6d22bf43c06a39e4` has 56 successful checks,
three expected skips, one neutral advisory status and successful Snyk.
No check is pending or failed. The [full Runner support
suite](https://github.com/paperclipai/paperclip/actions/runs/42289536842/job/128139308510),
typecheck, build and [aggregate
verification](https://github.com/paperclipai/paperclip/actions/runs/42289536842/job/128145908574)
pass. Greptile and Paperclip Review both rate this exact head 5/5, with
no findings or unresolved threads.
- Environment: macOS arm64, Node 26.4.0, Vitest 5.0.3.

## Risks

Low risk. This changes only fixture inputs for normal target selection.
It does not change the production workflow guard, branch policy,
credentials, paid execution, timeouts or assertions. Product
documentation needs no update for test-only default inputs.

## Model Used

OpenAI Codex, GPT-6 family, with repository analysis, code editing and
tool execution. The exact runtime model ID and context window are not
exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub references)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes (not
needed: fixture-only default inputs)
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-06 14:52:38 -06:00
DottaandClaude Opus 5.5 02214b1a3c fix(evals): treat a job-less TestOps plan row as unassigned during registration (#19601)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Runner E2E cases run from Allure TestOps plans through
`runner-full-stack-e2e.yml`
> - #19549 added a sibling "Reliability — staging soak" plan and a
one-shot registration workflow
> - The first live `register` dispatch created the seven cases and the
plan, then stopped with `plan_job_conflict`
> - Live TestOps lists a new plan's uncovered cases as one row without a
job, and the code read that row as another job
> - This pull request treats a job-less row as "not assigned"
> - The benefit is that a second `register` dispatch can assign the plan
to the standing GitHub job and finish registration

## Linked Issues or Issue Description

Refs #19549

**What happened?**
The `runner-e2e-testops-registration.yml` `register` run on master
failed with `reliability_testops_registration_plan_job_conflict`. It had
already created the seven cases and the plan. `GET
/api/testplan/<id>/job` for that plan returned `[{"testCasesCount":7}]`.
The row has no `job` field.

**Expected behavior**
Registration treats a plan with only a job-less row as unassigned. It
assigns the plan to the standing Product regression GitHub job and reads
the assignment back.

**Steps to reproduce**
1. Dispatch `runner-e2e-testops-registration.yml` on master with
`operation=register` when no Reliability plan exists.
2. The new plan reports one job-less row.
3. Registration fails with `plan_job_conflict` before the assignment
step.

**Paperclip version or commit**
master at 744056ecc8 and later.

## What Changed

- `tests/runner-e2e/reliability-testops-registration.mjs`: add
`jobAssignments()`. It keeps the array check and ignores rows without a
`job`. Inspect, the pre-assignment conflict check and the
post-assignment read-back use it.
- `tests/runner-e2e/reliability-testops.test.mjs`: the fake TestOps now
returns the live job-less row for a new plan. A new test reproduces the
half-registered plan and checks that `register` assigns the standing job
and that the receipt becomes complete.

## Verification

- `node --test tests/runner-e2e/reliability-testops.test.mjs
tests/runner-e2e/product-regression-testops.test.mjs`: 28/28 pass.
- Red before the fix: with the old registration module, 4 tests fail,
including the new one. Green after: 13/13 in the reliability file.
- Live check after merge: dispatch `register` again. The cases and plan
already exist, so it only assigns the plan to the standing job.

## Risks

- Low risk. The change touches only the one-shot registration helper and
its tests. It does not change provider cells, launch selection or
reporting.
- A plan assigned to another job still fails with `plan_job_conflict`.
The existing test still covers that case.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code with tool use
(shell, browser read of the TestOps API response), standard context.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 14:36:26 -06:00
DottaandClaude Opus 5.5 1634f8b66b test(restart-chaos): run chaos waves against Daytona sandboxes without Docker (#19544)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart chaos harness (`tests/restart-chaos/`) restarts a local
control plane on purpose while agent tasks run, and grades every task on
rules R1 to R6
> - The plan targets agent runs in Daytona sandboxes, but the harness
only supported `--env local`
> - A local controller needs two inputs that match the sandbox image
exactly: the image's Linux `paperclip-runnerd` and its provider pack.
The documented way to get them uses Docker, and Docker is not available
on the operator machine
> - The Daytona org that chaos uses also runs the live Cloud fleet, so
the harness must never change a sandbox it did not create through its
own Paperclip environment
> - This pull request adds `--env daytona`. It extracts both inputs from
the image without Docker, and refuses a controller whose runner source
differs from the image. It seeds a Daytona environment from the
runner-e2e fixtures and adds a read-only Daytona R6 sweep
> - The benefit is that chaos waves can now cross real Daytona sandboxes
with a local, killable control plane

## Linked Issues or Issue Description

**Feature description**
Run restart chaos waves with a local control plane and agent runs in
Daytona sandboxes (plan
`doc/plans/2026-11-06-control-plane-restart-chaos.md`, Phase 1b).

**Problem it solves**
`--env daytona` threw "not wired yet". The controller-side runnerd and
provider pack had to come from `docker cp`, which needs a Docker daemon.

**Proposed solution**
Use `skopeo copy` (no daemon) to fetch the linux/amd64 image into a
directory, walk the OCI layers from top to bottom, and extract only the
two paths. Before the wave starts, compare the image's content-ID label
with this checkout.

Refs #19477

## What Changed

- `tests/restart-chaos/daytona-image.ts`: a streaming tar reader and an
OCI layer walk (top to bottom, `.wh.<name>` and `.wh..wh..opq`
whiteouts, hard links, symlinks written last, no `..` paths). It
extracts `/usr/local/bin/paperclip-runnerd` and
`/opt/paperclip-runner/provider-pack/`. It checks the layer blob
digests, the ELF header (x86-64), the provider-pack manifest schema, and
its `runnerSourceRevision` against the image revision label. It records
digests in `receipt.json`, caches the result by image digest under
`results/daytona-images/`, and deletes the layer blobs. It also runs
standalone.
- Controller/image check: `assertControllerMatchesImage` compares the
image's `io.paperclip.runner.content-id` label with
`computeDaytonaImageContentId` for this checkout. On a mismatch it fails
and prints both IDs and the image's source revision.
- `seed.ts`: for `--env daytona`, it installs the bundled Daytona plugin
(`installFixtureDaytonaProvider`) and waits for provider readiness. It
creates the `daytona` environment fixture from `catalog.ts` with a
`DAYTONA_API_KEY` company secret and runs the native Codex
platform-qualification probe. Then it assigns the agents and the project
to that environment. `--daytona-cpu` sets the sandbox vCPU (default 4, 4
GiB, 10 GiB).
- `daytona-sweep.ts`: a read-only R6 for Daytona. The Daytona client is
a frozen wrapper that exposes only a label-filtered `list`, and no stop,
archive or delete exists. The sweep keeps only rows whose
`paperclip-company-id` and `paperclip-environment-id` both equal the
wave's own UUIDs, and it re-checks every row. It reconciles sandboxes
with the environment's leases:
  - A sandbox archived after its lease was released is expected.
- An orphaned sandbox or an `active`/`pending_cleanup` lease fails R6
for the owning task, or the wave.
- `assertHarnessMayMutate` refuses `paperclip-create-*` names and
`fleet-sbx-*` snapshots.
- Cleanup only deletes the wave's environment through Paperclip and
reports what is still running.
- `config.ts`: `--daytona-image` (or `PAPERCLIP_E2E_DAYTONA_IMAGE`)
needs an immutable digest. `--concurrency × --daytona-cpu` must stay
within `RESTART_CHAOS_DAYTONA_VCPU_BUDGET` (default 40). B5 is
local-only, because its oracle reads the live workspace.
- `main.ts`: Daytona preflight (credentials, image extraction,
content-ID check, Daytona plugin build as CI does). It passes
`PAPERCLIP_RUNNER_REMOTE_BINARY_PATH` and
`PAPERCLIP_RUNNER_REMOTE_PROVIDER_PACK_PATH` to every server start and
writes `sandbox-sweep.json`. Cleanup errors never skip stopping the
controller.
- README: a Daytona section (image matching, image-only CI dispatch,
vCPU budget, fleet safety, R6).
- Unit tests: synthetic tar layers for the tar reader, whiteouts, opaque
directories, overrides, hard links, symlinks and parent replacement.
Further tests cover the content-ID check, sweep reconciliation, exact
wave selection (unlabelled rows and other labels are never selected),
the read-only client, the fleet guard, and the CLI options.

## Verification

- `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`:
62/62 pass, credential-free.
- `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: clean.
- Real extraction without Docker, image `…@sha256:b0c71f4d…`: 42 layers,
about 8 minutes, a 3 GB pack.
- The content-ID check refused the current checkout. The image is
`b8b16e15…` from `7e6d882558`; the checkout is `aec45589…`.
- A checkout of `7e6d882558` computes `b8b16e15…`, so the check compares
like with like.
- Current image:
`ghcr.io/paperclipai/paperclip-daytona-runner@sha256:a84579522192de0393ae24d09e67c4e752df4c701bd315ff54ad9bbca42e2dc0`
(source `5d25117ffa`, content `4e9a221f…`). It matches this branch, and
the receipt shows the content ID match.
- Paid waves, `--env daytona --adapters codex,claude --tasks 3
--concurrency 2 --faults K1 --seed 11`:
- Wave 1 (10 vCPU org, 4 vCPU sandboxes): 0/3. Every create returned
HTTP 400 "Total CPU limit exceeded. Maximum allowed: 10", reproduced
directly with the SDK. This led to the vCPU budget, and the
archived-sandbox fix in R6.
- Wave 2 (same org, 2 vCPU sandboxes): **3/3 PASS**, all
restart-crossing. Codex completed through one working Retry; both Claude
tasks continued automatically. R6: 4 sandboxes archived after release, 0
orphans, 0 held leases.
- Wave 3 (company org, 4 vCPU sandboxes): 0/3. Two product failures were
found and reported to the campaign ledger: an ACPX Claude recovery
blocked on `ambiguous_remote_recovery_evidence` with no Retry, and a
false budget hard stop at 10.75¢ of 3000¢ that paused the company. The
harness changed no sandbox. R6 reported 2 held leases, and the
environment delete returned 409 `pending_sandbox_cleanup` because of
them.
- Fleet-org key check: one sandbox with a unique non-`paperclip-create`
name and label was created, then deleted by the exact ID that create
returned.

## Risks

- This change touches the test harness only, with no product code.
- The extracted provider pack is about 3 GB per image in the gitignored
results directory. Delete old `results/daytona-images/<digest>`
directories by hand.
- The harness never changes Daytona sandboxes. If Paperclip's
environment cleanup is blocked, sandboxes stay until their 60-minute ttl
or a person acts. The harness reports them in `sandbox-sweep.json` and
fails the wave.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edits, GitHub CLI). Default context window,
extended thinking.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 14:26:45 -06:00
DottaandClaude Opus 5.5 b7e28eb2f7 fix(evals): let an allowlisted manual dispatch authorize its private evidence (#19599)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Chat-feature evals run in the trusted `runner-full-stack-e2e`
workflow and keep their evidence in private history
> - A manual chat run needs authority to write that private evidence
> - Today that authority is a stored approval for the exact run ID in
`RUNNER_E2E_MANUAL_PRIVATE_APPROVALS_JSON`
> - The run ID exists only after dispatch, so every manual run stops
until a person writes the record
> - The dispatcher is already checked against
`RUNNER_E2E_ALLOWED_ACTOR_IDS`, so the second step adds friction without
a new decision
> - This pull request lets an allowlisted manual dispatch authorize its
own private evidence, as the TestOps admission already does
> - The benefit is that an allowlisted operator can start a chat eval
and it runs, while scheduled runs still never touch chat evidence

## Linked Issues or Issue Description

Refs #18134, Refs #18170

**What existing behavior does this improve?**
A manual `runner-full-stack-e2e` run with chat-feature IDs fails at
private evidence transfer unless a person first adds an approval row for
that run ID to a repository variable.

**Subsystem affected**
Runner E2E manual workflow private evidence authorization
(`manual-private-transfer.ts`, `runner-full-stack-e2e.yml`).

**Proposed change**
Derive the evidence authority from the dispatch itself, with the same
checks the TestOps chat admission uses.

## What Changed

- New `tests/runner-e2e/manual-dispatch-authority.ts`:
`authorizeManualDispatch` admits private evidence only when all of these
are true:
- The event is `workflow_dispatch` of `runner-full-stack-e2e.yml` on the
default branch, with the `manual` cadence.
  - The run attempt is `1`.
- The actor is in `RUNNER_E2E_ALLOWED_ACTOR_IDS`, the event sender is
that actor, and the triggering actor is the same person.
  - The trusted cell selection equals the dispatch's own `id` input.
- The authority binds the workflow SHA, the target SHA from the
authorize job, the run ID and the cells. Every job derives the same
identity hash.
- `manual-private-transfer.ts`: uses the TestOps authority first, then
the dispatch authority, then the stored approval. With the dispatch
authority, `prepare` records the target launcher digest instead of
comparing it with an approved digest.
- `runner-full-stack-e2e.yml`: the 14 steps that read
`RUNNER_E2E_MANUAL_PRIVATE_APPROVALS_JSON` now set
`PAPERCLIP_MANUAL_DISPATCH_AUTHORITY: v1` and pass
`RUNNER_E2E_ALLOWED_ACTOR_IDS`.
- `doc/manual-e2e-private-evidence.md`: describes the dispatch
authority. The stored-approval shape stays documented for the TestOps
catalog case workflow.
- The new file is separate on purpose. `manual-private-retention.ts` and
the other protocol files must match each target byte for byte, so
existing candidate branches keep working.

## Verification

- `vitest run tests/runner-e2e/manual-dispatch-authority.test.ts
tests/runner-e2e/manual-private-transfer.test.ts
tests/runner-e2e/manual-private-report.test.ts
tests/runner-e2e/manual-private-report-integration.test.ts
tests/runner-e2e/manual-private-policy.test.ts`: 57 tests passed.
- New tests cover one valid dispatch and rejection of: a scheduled run,
a bounded cadence, a non-allowlisted actor, an empty allowlist, a rerun
by another person, a second attempt, a non-default or different workflow
ref, a sender who is not the actor, a missing target SHA, a missing
authority switch, and cells that differ from the dispatch input. The
eight-cell limit still applies.
- A `prepare` test runs the real transfer script: a non-allowlisted
actor produces no outputs, and an allowlisted dispatch produces the
session policy and an identity that records the target launcher digest.
- Negative control: with the previous `manual-private-transfer.ts`, the
new `prepare` test fails.
- Workflow tests that read this workflow: 111 of 116 passed. The 5
failures in `workflow-target.test.ts` (`IMAGE_ONLY: unbound variable`)
also fail on unchanged `master`. `paid-runner-routing.test.mjs` passed
(7 tests).
- I did not run a live dispatch. The first live proof is a manual
dispatch of chat IDs after merge.

## Risks

- The dispatcher now vouches for the target branch. A target branch with
a changed launcher can write private evidence if an allowlisted operator
dispatches it. The protocol files must still match the trusted copies
byte for byte.
- The eight-cell limit is unchanged. The report job has eight fixed read
slots, and each read policy must fit the 2,048-character session policy
limit. Lifting the limit needs a separate change to the report sessions.
- The `runner-e2e-paid` environment still has required reviewers. That
is a repository setting and this change does not touch it.
- The authority uses `RUNNER_E2E_ALLOWED_ACTOR_IDS`, not `CODEOWNERS`.
Keep that variable aligned with the people who may start paid runs.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 14:26:31 -06:00
DottaandClaude Opus 5.5 744056ecc8 feat(evals): run the staging-soak reliability cases from a TestOps plan (#19549)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The runner full-stack E2E lane runs real Paperclip work against real
agents, and TestOps launches its standing plans
> - PR #19455 added seven staging-soak reliability cases, but no TestOps
plan or workflow lane selects them, so they run only by hand with
personal keys
> - These cases catch restart, interrupt, queue and lost-runner
regressions, so they must run from TestOps before a release with the
standard reports
> - This pull request adds a sibling "Reliability — staging soak"
TestOps plan on the existing plan-2 lane, with native OpenCode and
DeepSeek 1017 through OpenRouter as the plan profile for six of seven
cases
> - The benefit is a cheap, repeatable pre-release reliability launch
with the same Allure/TestOps results, retained report and dashboard as
plan 2

## Linked Issues or Issue Description

Refs #19455

**What existing behavior does this improve?**
The staging-soak reliability cases from #19455 run only by a manual
local command.

**Subsystem affected**
Runner full-stack E2E (`tests/runner-e2e/`) and its TestOps lane in
`.github/workflows/runner-full-stack-e2e.yml`.

**Current behavior**
No TestOps plan includes the cases. The workflow cannot select them from
TestOps. Every cell uses native Codex (`gpt-5.6-sol`) or an ACPX facade
on Daytona, which is expensive.

**Proposed behavior**
A TestOps launch of the Reliability plan runs seven cells (six DeepSeek
1017, one native Codex for steering) through the standing plan-2 path
and reports them like plan 2.

**Reason and benefit**
The cases become a first-class, low-cost, pre-release check with
retained evidence. Plan 2 keeps its 20-case qualification semantics.

**Breaking changes**
None. Plan 2 selection, reporting and history identities stay the same.

## What Changed

- **Plan choice: a sibling plan, not an extension of plan 2.** Plan 2 is
bounded at 20 standing IDs. Its `full-run` / `test-plan-2` tag needs a
complete 20-case attempt, and subset runs do not qualify it. Adding
seven cases would change that denominator and the qualification history.
The sibling plan shares the same TestOps GitHub job, selection,
admission, projection, upload and verification code. A launch selects
cases from exactly one plan (`plan_mixed` fails), and each plan has its
own full-plan size (20 or 7).
- `tests/runner-e2e/reliability-testops.mjs` (new): the mapping module.
It has seven cases, full names `paperclip.Product
E2E.everyday-workflows.<case>`, execution IDs, soak signatures and an
empty `reliabilityRegistration`. A partial, duplicate, plan-2 or
malformed registration is rejected. An empty registration makes
selection fail closed before any spend
(`product_regression_testops_reliability_unregistered` for a bootstrap,
`case_mismatch` for a TestOps launch). No TestOps IDs are invented.
- `product-regression-testops.mjs` / `-cli.mjs`: select a plan by its
case identities or by the new bootstrap input. The projection binds each
cell's own environment (`local` or `daytona`) instead of a fixed
`local`. It adds a `soak-<signature>` tag and reports `plan` and a
per-plan `fullPlan`. Plan-2 history IDs, parameters and results do not
change.
- Catalog: `runner-opencode-deepseek-1017` is now one shared fixture
with `supportedEnvironments: ["local", "daytona"]`. The OpenCode server
driver already runs on Daytona for `runner-opencode` (task-workspaces,
core-compatibility, rolling deployment); only the model differs. The
chat-feature suite stays local, and its definition hash does not change.
`everyday-workflows` gains exactly six DeepSeek plan cells (definition
version 12, 52 + 13 + 3 cells).
- Cell matrix (plan), DeepSeek 1017 on native OpenCode:
`interrupt-redirect`, `reassign-midrun` and `tree-pause-release` run on
`local`. `three-turn-revise` runs on Daytona with the per-turn lease
(S49). `long-silent-command` (S52/S52b) and `queued-across-restart`
(S55/U9) run on the Daytona warm lease.
- `queue-steer` is the one exception. It must steer an active turn, and
the OpenCode server driver declares `steering: false`, so the route
answers `steering_unsupported` and the case cannot pass. The plan uses
the existing steering-capable
`everyday-workflows.runner-codex.daytona.queue-steer` cell for it
(Greptile finding). No DeepSeek `queue-steer` cell exists. The other
native Codex Daytona cells and the ACPX facades stay selectable through
`--group reliability`, but they are outside the plan.
- `runner-full-stack-e2e.yml`: adds the `testops_plan` bootstrap input
(`product-regression` by default), sets the plan-specific bootstrap
launch name and tags, and renames two jobs to plan-neutral names. The
OpenCode steps now match `startsWith(matrix.profileId,
'runner-opencode')`, so the DeepSeek profile gets the pinned OpenCode
binary, runner TypeScript, the local Node check and the remote provider
pack. Before this change, the DeepSeek profile could not run in this
workflow. `OPENROUTER_API_KEY`, `OPENAI_API_KEY` (only for
`queue-steer`) and `DAYTONA_API_KEY` come from the same
`matrix.credentialName` and `matrix.environmentId` gates that other
cells use.
- `runner-e2e-testops-registration.yml` +
`reliability-testops-registration.mjs` (new): a trusted, master-only,
allowlisted-actor, idempotent `workflow_dispatch`. `inspect` only reads.
`register` reuses cases with the same full name, creates missing cases
and the plan, and assigns the plan to plan 2's GitHub job with plan 2's
job parameters. It writes a receipt with a ready-to-commit
`reliabilityRegistration` snippet. Only this job gets `ALLURE_TOKEN`,
and it never runs a provider cell.
- **No schedule and not per PR.** The plan runs only from a TestOps
launch or a manual dispatch. It is not in the daily or weekly cron
(`scheduledSelection` excludes it). A weekly run would add a paid
50-minute Daytona cell every week, but these signatures move with
release-risky changes, not with time.
- Docs: `EVERYDAY-WORKFLOWS.md` (matrix, environment reasons, local
coverage limits, registration steps), `README.md`, `doc/evals.md`.

### Registration still needed (requires TestOps credentials)

1. Dispatch **Runner E2E TestOps registration** from master with
`operation=inspect`.
2. Dispatch it with `operation=register`.
3. Commit the `reliabilityRegistration` snippet from the run summary
into `tests/runner-e2e/reliability-testops.mjs` in one reviewed PR.
4. Launch the plan from TestOps. Tag only a complete seven-case attempt
`full-run`.

### What local plan cells cannot reproduce

The local cells do not reproduce the Daytona-only variants: the remote
`workspace_busy` save after Stop (S47), remote stop settlement during
reassignment (S48), and a lost or replaced remote runner during a tree
hold (S50). S51 (lost runner) has no case yet. The explicit-only native
Codex Daytona extras still cover S47, S48 and S50 on a sandbox.

## Verification

- `node --test tests/runner-e2e/product-regression-testops.test.mjs
tests/runner-e2e/reliability-testops.test.mjs`: 26 pass. These tests
cover fail-closed selection, partial or duplicate registration, plan
mixing, environment binding, per-plan denominators, preserved failure
classes, the CLI select → project → verify flow for a seven-case launch,
registration idempotency and job checks, and the workflow inputs and
credential gates.
- `vitest` for `reliability-testops-catalog`,
`product-regression-testops-catalog`, `everyday-reliability` and
`catalog.test`: 98 pass. These tests bind the mapping to real catalog
cells (profile, model, environment, credentials, single attempt). They
also check that the cells stay out of `--all`, the suite, profile and
environment selectors and the daily/weekly cron. Real grader output must
survive projection.
- `pnpm test:e2e:runner:typecheck`: pass.
- `pnpm test:e2e:runner -- --list --suite everyday-workflows --group
reliability` lists 16 cells: 6 DeepSeek plan cells, 7 native Codex cells
(one is the `queue-steer` plan cell) and 3 ACPX extras. `--matrix-json`
for the DeepSeek plan IDs emits `needsDaytona=true` and three Daytona
OpenCode cells with `credentialName=OPENROUTER_API_KEY`.
- Full `pnpm test:e2e:runner:unit`: the remaining failures happen the
same way on an unchanged master checkout. They are in
`workflow-security` (3), `controller-outage`, `operating-receipts`,
`pi-*`, `private-framework-report` and `remote-native-bootstrap`. #19529
tracks that drift. This PR adds no failures.
- Not run: paid cells, a TestOps registration and a TestOps launch. They
need provider and TestOps credentials, which exist only in protected CI
environments. Catalog and projection tests are not live qualification.

## Risks

- The registration script calls TestOps APIs (`POST /api/testcase`,
`/api/testplan`, `/assign`, `/jobparameter`) through the same endpoints
as the chat catalog reconcile. Tests use fakes, not the live API.
`inspect` comes first, and `register` is idempotent.
- DeepSeek 1017 on Daytona has not run before. The first launch is the
acceptance step for those three cells.
- If a later case needs another capability that OpenCode lacks, it must
use a capable profile in the same way as `queue-steer`. The catalog test
pins the single exception.
- The OpenCode workflow conditions now match any `runner-opencode*`
profile. Today this covers `runner-opencode` and
`runner-opencode-deepseek-1017` only.
- If someone commits a bad `reliabilityRegistration`, module load fails
for both plans. A unit test rejects any committed registration that is
not empty or complete.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with tool use
(shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 14:19:58 -06:00
DottaandClaude Opus 5.5 840838476a perf(runner-e2e): cut the Daytona image build critical path (#19545)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner Full-Stack E2E workflow tests the Runner in Daytona
sandboxes from a signed, content-addressed image
> - Every change to an audited Runner input needs a new image before a
Daytona cell can test it
> - The image job took 10 to 16.5 minutes today, and a restart chaos
campaign waits on it after each runner-side fix
> - Measurement shows two serial waits: a full Rust dependency compile
(about 5 of 6.6 minutes) and a 3.5 to 4 minute download of cached base
layers that starts only after the builder stages finish
> - There is also no way to build only the image: every dispatch reaches
a paid gate that the operator must reject
> - This pull request removes both serial waits and adds an `image_only`
dispatch input
> - The benefit is a shorter fix-to-Daytona-test loop, with every image
security check unchanged

## Linked Issues or Issue Description

**What happened?**

The `Publish verified Daytona image` job is on the critical path of the
control-plane restart chaos campaign. Builds today took 16.5 min (run
42240434011), 15 min (42253920727), 10.5 min (42259118276) and 16 min
(42265637650). Run 42264611573 failed: its RunsOn runner lost
communication with GitHub 7.6 minutes into the build step, during the
Rust compile. GitHub has no log for that job.

**Expected behavior**

A Runner source change should not recompile the locked third-party Rust
graph. Cached image layers should download while the builder stages
compile. An operator should be able to publish only the image, without a
paid gate.

**Steps to reproduce**

Read the BuildKit timeline in the logs of job 127970125495 (run
42240434011). See the stage table below.

## What Changed

- `docker/daytona-runner/Dockerfile`: compile the locked Rust dependency
graph in its own layer against placeholder targets. The layer depends
only on `Cargo.toml`, `Cargo.lock` and the crate manifest. The step
deletes the placeholder outputs and fingerprints, so the next step
always compiles the real Runner sources. The final stage still receives
only the stripped executable.
- `docker/daytona-runner/Dockerfile`: move the content-ID /
source-revision check above the two `COPY --from` steps. It does not
depend on either builder stage, so BuildKit now downloads the cached
base and CLI layers in parallel with the compile. Both copies use `COPY
--link`.
- `.github/workflows/runner-full-stack-e2e.yml`: add the boolean
`image_only` dispatch input. `authorize`, `target_lock`, `catalog` and
`daytona_image` run unchanged. Every other campaign job gets `&&
inputs.image_only != true`. `authorize` rejects `image_only` together
with TestOps inputs.
- `tests/runner-e2e/workflow-image-only.test.ts` (new): pin the job
classification, the top-level guard on each campaign job, the guard on
every protected environment, and the TestOps rejection.
- `tests/runner-e2e/README.md`: document the layer layout and the
`image_only` command.

## Verification

Measured on the same RunsOn `m7i-flex.xlarge` (4 vCPU, 16 GiB, native
x64) runner class. "Before" is master job 127970125495 (run
42240434011). "After (cold)" is this branch, job 128077687214 (run
42271236140), with no dependency layer in the registry cache yet. Its
paid gate was rejected.

| Stage | Before | After (cold dependency cache) |
|---|---|---|
| Whole image job | 16.5 min | **11.6 min** |
| Rust dependency graph | 311 s (inside one 399 s step) | 395 s (own
layer; reused from cache after a master build) |
| `paperclip-runner-core` compile | ~87 s | 65 s |
| Cached base and CLI layer download | 236 s, **after** cargo finished |
257 s, **in parallel** with cargo |
| Copy runnerd + provider pack | 20 s + 50 s | 9 s (`--link`) |
| Provider pack stage | 143 s (parallel) | 152 s (parallel) |
| Export + push | 82 s + 10 s | 81 s + 11 s |
| Dependency layer size | n/a | `target` 852 MB + registry 333 MB
(builder stage only) |

With a warm dependency cache, a Runner source change compiles only the
65 s crate step. The critical path then becomes the ~4 min cached-layer
download or the provider pack. The expected job time is about 7 to 8
min. I measure this after merge with an `image_only` dispatch (master
first, to write the cache, then a Rust-only branch change).

Disk: the instance starts with 38.76 GiB free. The dependency layer adds
about 1.2 GB uncompressed to builder storage. The final image does not
change.


- `vitest run tests/runner-e2e/workflow-image-only.test.ts
tests/runner-e2e/workflow-security.test.ts
tests/runner-e2e/daytona-image.test.ts`: the new test and all image
tests pass. Three tests in `workflow-security.test.ts` (`gates every
provider-secret job with stable actor IDs`, `keeps provider credentials
inside explicitly gated paid workflows`, `isolates private archive trust
from workflows that execute target-branch code`) fail on `origin/master`
without this change, because of `chat-catalog-testops-case.yml`. This
pull request does not change them.
- `node --test tests/runner-e2e/product-regression-testops.test.mjs`:
15/15 pass.
- No local Docker build. The Dockerfile is measured with a
branch-targeted CI dispatch whose paid gate was rejected.

## Risks

- The dependency layer keeps the Rust `target` and registry trees in a
builder-stage layer, and default-branch builds export it to the registry
cache. This adds builder disk and cache upload when `Cargo.lock`
changes. The build prints the size (`du -sk target
/usr/local/cargo/registry`).
- The dependency cache helps only after a default-branch build writes
it. Branch builds read it but never write it. This is the existing cache
policy.
- `image_only` changes the workflow, so it works only after merge to
master.
- No check is removed. Lockfile verification, cosign sign and verify,
the anonymous pull check, the provider-pack manifest match, the
default-branch-only cache write and the actor allowlist are unchanged.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, GitHub CLI). Extended reasoning on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 14:15:16 -06:00
Devin FoleyandPaperclip 17901d74ba Fix onboarding account changes after returning from Review (#19557)
Save changed onboarding connections on the existing agent after validation.
Select the chosen personal provider account and serialize default changes so
an abandoned save cannot override a later choice after Back. Preserve runner,
environment, and unrelated agent settings. Reject unsupported managed-account
to legacy-secret switches with an actionable Connect error.

Verified with 327 component tests, three backend credential-selection tests,
nine affected browser tests, typecheck, build, token gates, all CI jobs, and
Greptile 5/5 with no unresolved comments.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-06 11:55:29 -08:00