mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-11 05:11:49 +01:00
358977454ba449f37dee1a2c5244e53be34bd0ce
7418
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandClaude Opus 5.5
|
358977454b |
test(native): cover unknown-outcome recovery and hold settlement without a recorded PID space (#20649)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - #20628 settles a host-tool callback that a server restart orphaned,
and settles the no-replay holds that earlier releases left for a
completed turn
> - Review of #20628 asked for two more cases. That PR merged before the
commit with those cases reached it
> - This pull request adds only those tests
> - The benefit is that the two edge rules of #20628 stay covered
## Linked Issues or Issue Description
Refs #20628
**What happened?**
The review of #20628 named two untested rules. First, a recovery
authority that throws `semantic_tool_outcome_unknown` must settle only
for a callback from an earlier controller, and must rethrow for this
controller's own callback. Second, a coordinator row saved before
`controller_host_hash` existed must still settle its holds through the
stopped controller boot.
**Expected behavior**
Both rules have regression tests.
**Steps to reproduce**
1. Read `recoverToolCallback` and `native_completed_turn_cleanup_hold`
on master.
2. Search the tests for a thrown `semantic_tool_outcome_unknown`, or for
a row with a null `controllerHostHash`. Neither exists.
**Paperclip version or commit**
master at
canary/v2026.1110.0-canary.18
|
||
|
DottaandClaude
|
30403781b9 |
fix(native): keep crediting proven unstarted restarts past the generation limit (#20647)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native run survives a control-plane restart through restart recovery, which resumes the interrupted provider turn > - Restart recovery charges a provider attempt only when the provider did work that could be lost, and it stops after three charged attempts > - From controller generation 12 on, every restart charged an attempt before that work rule was checked > - In a busy restart wave, one long task sees 12 to 15 restarts, most caused by other tasks. A restart that hit before the provider accepted a turn was charged, the budget ran out, and a person had to click Retry > - This pull request keeps the work rule past generation 12 for restarts that a durable record proves did not start the provider, and bounds a crash loop with a count of consecutive restarts without provider progress > - The benefit is that a long task survives many unrelated restarts without a manual Retry, while a run that never makes progress still stops ## Linked Issues or Issue Description Refs #20625 (same restart recovery path). No public issue exists for this case. **What happened?** `restartResumeKeepsAttempt` returned "charge" for every restart once the run reached `NATIVE_PROGRESS_CREDIT_GENERATION_LIMIT` (12) controller generations. It did this before it read the work-based outcome. In a local restart chaos wave (L6, seed 9106, task t03, Claude B5 with RR1) the run reached generation 13. The last restart interrupted a resume whose runner started but whose provider had not accepted a turn. That restart was charged at the last attempt, so the claim failed with `execution_recovery_budget_exhausted` and `native_restart_recovery_blocked`. The task needed a manual Retry. **Expected behavior** A restart that a durable record proves did not start provider work does not spend an attempt, also past generation 12. A crash loop that never makes provider progress still stops automatically. When it stops, the task shows a Retry that works. **Steps to reproduce** Run a native task past controller generation 12. Make the last restart hit after `native.process_start_requested` but before `turn.accepted`, with the provider attempt at 3. The new integration test `does not charge a restart past the generation limit whose resume never started the provider` reproduces the failure before the fix. **Paperclip version or commit** `master` at |
||
|
DottaandClaude Opus 5.5
|
4f33bb3578 |
fix(workspaces): release the Blocked status of a sibling-settled no-replay hold (#20648)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a remote workspace save fails, recovery actions hold the task until the run's files are saved > - #20641 lets a resolved no-replay hold settle when a sibling action of the same run saved the files > - Review found that the settlement did not release the task's Blocked status, and that a settled hold with a historical source stayed selectable > - A new test with the waiting user comment also showed that the worker skipped the second hold of the same run in every pass, so it never settled > - #20641 was merged before these fixes were pushed > - This pull request completes #20641 with those fixes > - The benefit is that the task leaves Blocked and continues without a person ## Linked Issues or Issue Description Refs #20641, Refs #20569 **What happened?** #20641 merged at its first head. The review follow-ups for it were pushed later and are not on master: - A settled no-replay hold that owned the task's Blocked status (`nativeFailureBlock`) left the task `blocked`. - A hold settled by a sibling stayed selectable through `workspaceRecoverySourceEligible` when it named a `historicalWorkspaceSource`. - With saved user input waiting, the export hold stayed selectable and came first in each worker pass. `reconcileRetainedWorkspaces` skipped the other hold of the same run, so the no-replay hold never settled. **Expected behavior** After the save, the no-replay hold settles in the same or next pass. The task returns to `todo` when that hold owns its Blocked status. The settled hold is never selected again. **Steps to reproduce** 1. Use the staging-shape test in `workspace-recovery-capture-policy.test.ts`, with a waiting "try again" comment. 2. Run the worker until nothing is due. 3. On master, the worker never goes idle and the no-replay hold stays `resolved`/`blocked`. ## What Changed - `workspace-recovery-fallback.ts`: new shared `holdOwnsBlockedStatus`, used by `continueFromLastSavedWorkspace` and `settleSiblingRecoveredHold`. A settled no-replay hold that owns the Blocked status returns the task to `todo`. A newer decision's Blocked status is kept. An export hold's status stays with its own result finalization. - `workspace-recovery-source.ts`: `workspaceRecoverySourceEligible` excludes a resolved hold that carries `siblingWorkspaceRecovery`. - `workspace-recovery.ts`: `reconcileRetainedWorkspaces` runs several holds of one run one after another in that run's job, instead of skipping the later ones. - `workspace-recovery-capture-policy.test.ts`: the staging test now has the waiting comment, and a variant with `nativeFailureBlock` and `historicalWorkspaceSource`. It checks `todo`, no re-selection, and no repeat handoff in a later pass. - `doc/execution-semantics.md`: documents these rules. ## Verification - Red, with each rule disabled: the status stayed `blocked`; the settled hold was selected again; the worker never went idle (starvation). - Green: `npx vitest run --root server` on 14 workspace recovery suites (including `workspace-recovery-capture-policy`, `-fallback`, `-admission`, `-source`, `execution-blocker-read`, `preserved-native-response-wait.integration`): all passed. - `npx tsc --noEmit -p server/tsconfig.json`: clean. ## Risks - Several holds of one run now run in sequence in one job. Concurrency per run is unchanged (one job per run). - A no-replay hold releases the Blocked status only when it still owns it. Otherwise the status is kept. - Previous-release story: no new persisted state. Holds saved by earlier releases are handled on the first worker pass after deploy. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude
|
805531cc71 |
fix(native): never leave a cleanup-gated task without Retry or an owner (#20639)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs close their provider session at the end of each attempt.
When close cannot prove that the runner stopped, the runner keeps a
cleanup gate (`native_session_cleanup_quarantined`) in server memory
> - #20572 hid Retry for every gated run and said "Paperclip finishes
the cleanup automatically". Only a remote environment release settles
the gate. A local gate that needs an operator never settles, and a
server restart loses it
> - A restart-chaos wave on master found two tasks that rested blocked
with no Retry, no scheduled retry and no recovery owner. A person could
not continue them
> - This pull request makes the run's own recovery retire the gate when
the host proves the old processes stopped. It also offers Retry whenever
nothing settles the gate, and it makes that Retry work
> - The benefit is that a cleanup-gated task either continues by itself
or shows a Retry that starts. It never rests without both
## Linked Issues or Issue Description
Refs #20572, #20578
**What happened?**
Restart-chaos wave `w-20261110T185144-s9107` (local, claude, fault TM2:
the server is paused with SIGSTOP for about 33 s). Tasks t04 and t05
completed their provider turn. The runner checkpoint owner lapsed while
the server was paused (`native_runner_checkpoint_owner_lost`). The
session close then failed with `native_session_close_unrecoverable:
runner did not durably suspend before checkpoint`. The provider had
drained and no tool call was pending. The runner kept an
operator-required cleanup gate. Thirty seconds later the run's own
same-run recovery verified that the runner process and group had stopped
(`native.local_process_stopped`). The gate still refused the recovery,
and the run ended `failed` with `native_session_cleanup_quarantined`.
The task showed `canRetry: false` and a cleanup wait until the deadline.
Nothing owned that wait.
**Expected behavior**
When the old runner and provider processes are proven stopped, the run's
own recovery continues. When nothing settles the gate, the task offers a
Retry that starts.
**Steps to reproduce**
1. Run restart-chaos L7: `--env local --adapters codex,claude
--workloads A1,B5,C1 --faults TM1,TM2,TM3 --time-age-hours 1
--concurrency 3 --tasks 6 --seed 9107`.
2. Observe a claude TM2 task rest `blocked` with `canRetry=false`,
`retryWait=true` and no scheduled retry.
**Paperclip version or commit**
master at
|
||
|
DottaandClaude
|
37c50f70c2 |
fix(native): continue a turn lost on restore in a proven same-run resume (#20645)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native run keeps its provider conversation when its local runner dies. When the restored runner finds that the active turn was lost, it reports `provider_turn_lost_on_restore` and does not invent a result. > - After a control-plane restart, the admitted restart attempt may send one continuation in that same conversation (#15297). The agent then reconciles unfinished work. The task is not resent. > - A runner can also be lost without a server restart, for example when `runner.drain` times out. The run then resumes itself after the retry delay from its exact checkpoint. That path never allowed the continuation, because the executor passed `resumeInterruptedTurn` only for a `resume_dead_runner` restart claim. > - So the lost turn failed with `native_provider_terminal_failed`, automatic replacement was refused (`uncertain_provider_action`), and the user had to click Retry. > - This pull request allows the same single continuation for that resume, but only when durable records prove the same three facts that a restart claim proves. > - The benefit is that a runner lost mid-turn continues in the same conversation with no Retry. No case that is refused today becomes allowed without that proof. ## Linked Issues or Issue Description Refs #15297, #20625 **What happened?** Restart-chaos wave L7 (seed 9107, local), task codex B5 TM2, run on master after #20625. A TM3 restart resumed the run (attempt 1). During the TM2 freeze the TS runtime lost its runner: `native_session_recovery_failed: runner_local_connect_failed: PRP command runner.drain timed out`. The executor saved `retryable_failure` with `recoveryMode: "exact_checkpoint_resume"`. After the delay, recovery verified the runner process stopped (`native.local_process_stopped`). The same run then resumed as attempt 2 (`native.process_start_requested`, `providerAttempt: 2`). The restored runner reported `turn.failed` with `provider_turn_lost_on_restore` (`recoverable: true`). No continuation was sent. The run failed with `native_provider_terminal_failed`, recovery stopped with `uncertain_provider_action` / `preserve_without_replay_v1`, and the task needed Retry. **Expected behavior** The run sends the one same-conversation continuation, exactly as it does after a restart claim, and finishes without Retry. **Steps to reproduce** `node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env local --adapters codex,claude --workloads A1,B5,C1 --faults TM1,TM2,TM3 --time-age-hours 1 --concurrency 3 --tasks 6 --seed 9107` (timing-dependent: the drain timeout must land during an active turn). ## What Changed - New `server/src/services/native-runtime/scheduled-resume-continuation.ts`. It decides whether a same-run scheduled resume may continue a lost turn. Each of the three proofs comes from a durable record: 1. **The previous runner is stopped.** The latest process record of the run in `heartbeat_run_events` (`native.process_start_requested`, `native.process_identity_recorded`, `native.local_process_stopped`) must be `native.local_process_stopped`. Its `processPid` must equal its `processGroupId`, and neither the pid nor the group may be alive now. This is the existing `readNativeLocalProcessStop` check, read in the claim transaction before this attempt records its own `native.process_start_requested`. 2. **The attempt was admitted under the bounded budget.** The executor's claim moves `native_run_finalizations` from phase `retryable_failure` to `observed`. It charges a provider attempt, and the claim refuses any attempt above 3 (`native_session_retry_exhausted`). The decision also requires `1 <= attempt <= NATIVE_PROVIDER_RECOVERY_ATTEMPT_LIMIT`. 3. **It is the same provider conversation.** The saved `failure_detail` must show `recoveryMode: "exact_checkpoint_resume"`, `recoverable: true`, `providerSessionEstablished: true` and `checkpointExists: true`, with no `successorRunId` or `replacementDenied`. At run time the existing runtime guards still apply: the backend must recover the checkpointed session (`recovered`, not a continuity-break replacement), and the lost turn must equal the checkpoint's `activeTurnId` and the snapshot's interrupted turn. The provider kinds stay codex, acpx and opencode, with no compaction, goal control, held result or pending runtime request. - `native-session-executor.ts`: computes that decision in the coordinator claim. It passes `resumeInterruptedTurn` when either the restart claim is `resume_dead_runner` (unchanged) or the decision allows it. - `doc/execution-semantics.md`: describes the rule and the refused cases. **Cases that stay refused** (each has a test in `scheduled-resume-continuation.test.ts`): - a restart-recovery claim (it keeps its own existing rule); - a remote execution target; - a coordinator that is not `retryable_failure`, including a fresh run; - `recoveryMode` `bootstrap_retry` or `ambiguous_state`, or no saved failure detail; - a failure that is not recoverable; - no established provider session; - no checkpoint; - a recorded replacement (`successorRunId` or `replacementDenied`); - an attempt outside 1..3; - no stop receipt, a stop receipt superseded by a newer launch, a stopped process that is still alive, or a run that never had a process. The runtime-side refusals are unchanged and already covered by `native-session-runtime.test.ts`: real provider errors, historical error-null failures, and no restart permission. ## Verification - `cd server && npx vitest run src/services/native-runtime/scheduled-resume-continuation.test.ts`: 22 tests, covering the decision table and the stop receipt in embedded Postgres. - `cd server && npx tsc --noEmit` passes. - Live: not proven. An L7 wave on this branch (seed 9107) did not reproduce the drain-timeout path, so this pull request has unit and database tests only. In that wave the same cell (codex B5 TM2) needed Retry for a different reason: TM2 aged its restart-recovery claim lease before dispatch, and budget admission refused with `Native recovery no longer owns provider dispatch`. This pull request does not change that path. Four other tasks passed. Claude C1 TM2 rested blocked without a Retry button, which is a separate cleanup-quarantine issue. ## Risks - This allows one more path to send the same-conversation continuation. It is limited to local runs. It needs the same proofs as a restart claim, and it uses the attempt the resume already charged. It does not change the generation cap or `restartResumeKeepsAttempt`. - A rolling deploy reads `failure_detail` and stop events written by earlier releases. Those fields and events already have this shape, so an older saved failure is judged by the same rule. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test runs). Extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
353945c86d |
fix(tool-gateway): let agents create gateway tokens without receiving the value (#20613)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Tool gateways let external MCP clients use a company's tools with a
gateway client token
> - Since delegated connection management (#17737), an agent can create
and revoke gateway client tokens for its responsible person
> - #20581 stopped agents from getting the one-time token value, but it
did this by making token creation person-only
> - The product decision is that agents keep their delegated authority
to create and revoke tokens
> - This pull request restores agent token creation, but the agent never
receives the token value: the person reveals the value once on the
gateway's Tokens page
> - The benefit is that an agent can do the rotation work for the
person, and the secret still never enters chat, run events, logs or tool
output
## Linked Issues or Issue Description
Refs #17737
Refs #20581
**What happened**
After #20581, `POST /api/tool-gateway/gateways/{gatewayId}/tokens`
returns 403 `gateway_token_requires_person` to every agent caller.
Agents lost the delegated authority to create gateway client tokens that
#17737 gave them. Before #20581, an agent that created a token received
the plaintext value in its tool result and could repeat it in chat.
**Expected behavior**
An agent with delegated connection management can create and revoke
gateway client tokens. The agent never receives a token value. The
responsible person gets the value once, on the gateway's Tokens page.
Person callers keep today's behavior.
**Steps to reproduce**
1. Give an agent delegated connection management for a responsible
person with `tools:admin`.
2. As the agent, call `POST
/api/tool-gateway/gateways/{gatewayId}/tokens` (for example through
`call_api`).
3. See 403 `gateway_token_requires_person`.
## What Changed
- Agent create: `POST /api/tool-gateway/gateways/:gatewayId/tokens` from
an agent with delegated authority returns 201 with token metadata only
(`id`, `tokenPrefix`, `expiresAt`, `pendingReveal: true`), plus
`tokensPath`, a `reveal_credential` `requiredUserAction` for the
responsible person, and a `nextStep` instruction. The saved row has no
usable value: its stored hash is of a value that is generated and then
discarded. Gateway authentication also rejects `pendingReveal` rows.
- Person reveal: new person-only `POST
/api/tool-gateway/gateway-tokens/:tokenId/reveal`. It generates the
value, stores its hash, clears `pendingReveal` and returns the value
once. A conditional update makes a second or concurrent reveal return
409. Only the token's responsible person can reveal it (403
`gateway_token_reveal_requires_responsible_person`). Agent callers get
403 `gateway_token_requires_person`. A token revoked before reveal
cannot be revealed.
- Why a reveal that generates the value: no stored plaintext existed to
reveal, and storing one (even encrypted) for later display adds a secret
at rest. Generating the value at reveal time keeps the existing one-time
display and stores only a hash.
- Schema: `tool_mcp_gateway_tokens.pending_reveal boolean not null
default false` (migration `0460_gateway_token_pending_reveal`). Shared
`ToolMcpGatewayToken.pendingReveal`, and `ConnectionRequiredUserAction`
gets `reveal_credential` / `gateway_token`.
- UI: the Tokens page shows a "Waiting for you to reveal" section for
pending tokens with a "Reveal token" button. The value appears in the
existing "New token — copy now" one-time block. Pending tokens show the
status "Not revealed".
- Agent gateway create: the 201 for agent callers also includes
`gatewayPath`, `tokensPath` and a `nextStep` that tells the agent to
link the saved gateway.
- The agent create-token `nextStep` and the catalog say not to revoke
the old token in the same turn: its client keeps working on it until the
person installs the new value.
- The reveal update and its `tool_gateway.token_revealed` activity entry
commit in one transaction. The Tokens page offers Reveal only for tokens
that act for the signed-in person.
- API reference: new `docs/api/tool-gateway-tokens.md` (create, reveal,
revoke) in the docs nav.
- Revoke stays allowed for agents with delegated authority (unchanged
route).
- API catalog: create-gateway, create-token, reveal and revoke
descriptions now say who acts, that agents never receive values, the
page paths, and that the old token is revoked only after the client
works with the new one.
- Agent access map: create-token policy note and a new reveal entry.
- Evals `gateways-profiles.create-gateway` and
`gateways-profiles.rotate-token`:
- Remove the keyword grader (`human|operator|admin|manage`) and the
handoff text checks. Grade saved state, rendered links and secret
absence. A plain reply or a question card is accepted.
- Remove the setup probes that called the create routes as the agent.
Under delegated authority, those probes created a real gateway
("Forbidden agent gateway"), which failed create-gateway before the
agent ran.
- create-gateway: exactly one gateway for the intended profile and
company, created by the agent (or by the person after a handoff); the
person opens it from a rendered link; attribution matches the creator;
any agent-created token is still pending reveal; no token value in chat,
cards or run events.
- rotate-token: after the agent's turn, both setup tokens still work and
any replacement the agent created is pending the person's reveal; the
person follows the rendered Tokens page link and reveals (or issues) the
value in a private browser; the new value works before the old token is
revoked; the unrelated client keeps access; no token value appears
anywhere, including a regex scan for any full `pcgw_<id>.<secret>`
value.
- The changed scenario context and outcomes are recorded in
`chat-full-catalog-testops-manifest.json` `changedFields` and pinned in
the manifest test. The inventory digest, descriptors (`oracleVersion`
v2, `sourceFiles`) and SDK public input pins are updated. The unused
`external-gateway-handoff.ts` module is removed.
- Fixture realism: the create-gateway profile now allows a real read
tool (an empty profile made the agent ask which apps to add instead of
finishing), and the rotation tokens get a neutral owner note instead of
"Disposable rotation fixture" (the agent read that note as an eval
hint).
- Shared private-browser helpers (`external-private-browser.ts`,
`external-private-browser-worker.ts`) get one optional `pendingTokenId`
input and one `revealedPending` receipt field.
## Verification
- Red: on `master` route code, the new integration tests `lets agents
create and revoke gateway client tokens without ever receiving a token
value` and `reveals an agent-created gateway token value only to its
responsible person` fail (403 `gateway_token_requires_person`).
- Green: `cd server && npx vitest run
src/__tests__/connection-management.integration.test.ts -t gateway` (25
passed). The test checks that the agent `call_api` result has no `token`
field and no token-shaped value, that the agent cannot reveal, that the
person reveals once (second reveal 409) and the value authenticates,
that agent revoke works and stops the value, that a token revoked before
reveal never gets a value, and that person create still returns the
value.
- `npx vitest run src/__tests__/tool-gateway.test.ts
src/__tests__/openapi-routes.test.ts
src/services/native-runtime/runner-api.test.ts` (2247 passed).
- `cd ui && npx vitest run
src/pages/apps/gateways/panels/TokensPanel.test.tsx
src/pages/apps/gateways/gateway-helpers.test.ts` (16 passed); `pnpm
check:token-gates` clean.
- Eval graders and catalog: `pnpm exec vitest run --config
tests/runner-e2e/vitest.config.ts` on the gateway oracle tests,
`external.test.ts`, catalog consistency/contract, TestOps
export/retained, phase/clock callers and SDK input/type tests (all
pass); `node --test` on the TestOps manifest, adapter, contract,
streaming and reconcile tests (all pass); `node
tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`
(unchanged).
- `node scripts/check-maps.mjs` passes. `tsc --noEmit` for `server`,
`ui` and `tests/runner-e2e`: no errors in changed files.
- Paid end-to-end run of both eval cases, run
[42872921890](https://github.com/paperclipai/paperclip/actions/runs/42872921890)
on
|
||
|
DottaandPaperclip
|
2150820bd8 |
perf(native): parallelize durability eligibility checks (#20646)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A web server must prove that retained run state is durable before volume-free enrollment. > - The proof reads selected native sessions and validates their existing recovery receipts. > - Serial database checks made a valid inventory take 59.665 seconds. > - The signed preparation request has a 55-second limit. > - This pull request overlaps independent reads within fixed limits. > - The benefit is a faster eligibility check with the same preservation rules. ## Linked Issues or Issue Description Refs #20599 and #20601. A search found no duplicate of this remaining serial-read bottleneck. **What happened?** An actual full startup eligibility check returned eligible with no blockers in 59.665 seconds. Controller checks took 44.134 seconds and harness checks took 8.592 seconds. Signed preparation timed out at 55.002 seconds. **Expected behavior** A durable instance can complete signed preparation within its existing deadline. **Steps to reproduce** 1. Retain many native task sessions with database-backed controller checkpoints and valid filesystem-independence receipts. 2. Run the full startup eligibility check. 3. Observe serial checkpoint and receipt reads consume the signed request budget. ## What Changed - Check the existing terminal-controller metadata and checkpoint-existence predicate within the existing eight-reader source batch. - Validate harness independence receipts with at most four concurrent readers per existing page. - Stop admitting more harness reads after an invalid result or error. Join all started reads before returning or throwing. - Keep large controller, full-checkpoint and legacy archive reads serial. - Add real PostgreSQL regressions for overlap, concurrency limits and failure settlement. ## Verification - Genuine controller regression RED: all 16 checkpoint reads ran, but peak overlap was one. GREEN: 10 focused PostgreSQL controls passed, including bounded overlap, joined failures, active and warm handoffs, and failed-source preservation. The other 45 cases were filtered, not claimed as passes. - Genuine harness regression RED: overlap was one instead of four. GREEN: all three new PostgreSQL controls passed. Six existing authority, replacement and refusal controls also passed. - Final head `c059e031`: the same 10 controller and 9 harness PostgreSQL controls passed after rebasing onto current master and refreshing Runner artifacts. `node node_modules/typescript/bin/tsc --noEmit -p server/tsconfig.json` also passed. The first `pnpm exec` attempt could not start the compiler because package-manager bootstrap fetches failed; it is not counted as a typecheck pass. - `git diff --check` passed. - Actual private-process acceptance with deployed `8e4c5e81` dependencies and only the two changed owners: the complete check passed in 37.596 seconds with no blockers. Nine runs were running before and after; 1,750 sessions were selected. This is 17.4 seconds inside the unchanged request limit. The earlier complete baseline was 59.665 seconds on 1,745 selected sessions, so these are observed timings on a changing dataset, not a controlled exact speedup. No serving process was patched. - The first private candidate check stopped on a native-controller blocker in 18.216 seconds. A targeted follow-up found no blocker without any preservation or business write. Its cause was not identified. It is retained as a nonpass and is not a speedup result. - Current-head Greptile review is 5/5 with no actionable findings. Required CI is running: https://github.com/paperclipai/paperclip/actions/runs/42898013184. ## Risks - The scans now use more simultaneous database reads. Limits remain eight for controller source readers and four for harness proofs. - Invalid or failed proofs still refuse eligibility. All started readers settle before failure leaves the scan. - Previous-release state carries forward unchanged. This changes no stored shape, checkpoint identity, lease rule, schema, deadline or provider protocol. - Large legacy payload reads remain serial, so their existing memory bound is unchanged. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. This is a measured performance defect in the existing enrollment path. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, code execution and parallel agents. The runtime does not expose the exact model variant or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes:` / `Closes:` / `Refs:` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
4398a723c0 |
fix(native): settle tool callbacks and holds that a server restart orphaned (#20628)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs send provider tool calls to the server as durable
semantic inputs. The server runs the tool and saves the result in the
controller journal
> - Close must prove that every admitted tool call has a saved result.
Otherwise it reports `native_session_settlement_incomplete` and the
runner holds a cleanup gate (`native_session_cleanup_quarantined`)
> - A server restart in the middle of a tool call leaves its input
without a result. The new controller asks the tool authority for a
receipt. Tools without a receipt (assigned app tools,
`connections_search`) return nothing, so the input stays pending forever
> - Runs that already failed this way kept no-replay holds for a turn
that had completed. The task stayed blocked with no Retry and no
continuation
> - Restarts are normal. Work that a restart interrupts must settle by
itself, not wait for a person
> - This pull request settles such a callback as interrupted, with an
unknown outcome, when its owner is provably gone, and settles the holds
that earlier releases left once their controller is gone. It never runs
the tool or the turn again
> - The benefit is that close drains after a restart, and tasks that
earlier releases blocked this way continue on their own
## Linked Issues or Issue Description
Refs #20572 (that PR fixed the presentation of this quarantine. Its
Risks section names this runner drain rule as a separate follow-up.)
Refs #20641 (it settles a resolved no-replay hold when a sibling action
saved the run's files. This PR covers a different case: the hold's own
recovery saved the files, and its run's turn completed. Both use the
same recovery receipt, `completedWorkspaceRecoveryCondition`.)
**What happened?**
A remote native run started an assigned app tool call. The server
restarted 3 seconds later. The new server adopted the runner, and the
provider turn completed about 9 minutes later. The tool call never got a
saved result. Close reported `native_session_settlement_incomplete` with
`runnerSuspended: true`, `providerDrained: false` and one pending
callback. The session quarantined, and the same-run recovery was refused
at admission. A second run showed the same shape with a
`connections_search` callback after a later restart.
The first run then had two resolved-but-blocked holds with
`preserve_without_replay_v1` and an unknown action outcome: the
cleanup-quarantine hold, and a continuation hold whose workspace
recovery saved the files. The task view said "Saved work recovered. Send
a message to continue." with no Retry and no continuation. The task
waits on a child task and stays blocked.
**Expected behavior**
After a restart, a tool call without a saved result settles
automatically. A durable receipt is replayed. Without a receipt, the
agent is told that the outcome is unknown. Close proves the drain, and
no cleanup gate is left. Holds that earlier releases left for a
completed turn settle without a person, and the task continues when its
own wait ends.
**Steps to reproduce**
1. Start a native run and call an assigned app tool.
2. Restart the server while the tool call runs.
3. Let the adopted turn complete, then close the session. Close reports
`native_session_settlement_incomplete` and quarantines. The task stays
blocked by the no-replay holds.
**Paperclip version or commit**
master at
|
||
|
DottaandClaude Opus 5.5
|
d06f43b14f |
fix(workspaces): settle a resolved no-replay hold once a sibling saved its files (#20641)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a remote workspace save fails, recovery actions hold the task until the files are saved > - One failed run can have two holds: its export hold, and a no-replay hold that a later "try again" opened for the same run > - The no-replay hold is recorded resolved-but-blocked, and it is the task's execution blocker > - #20569 lets the export hold retry under the new symlink capture policy and save the files > - The sibling settlement that should then release the no-replay hold only looked at active holds, so the task stayed blocked with no working Retry > - This pull request lets the sibling settlement and its discovery include resolved no-replay holds > - The benefit is that the task continues without a person once the files are saved ## Linked Issues or Issue Description Refs #20569 **What happened?** On staging, a native run on a Daytona sandbox failed to save its workspace (`remote_workspace_archive_unsafe_symlink`). Its export hold (`native_workspace_sync_out_retry_exhausted`) was exhausted after 6 attempts and handed to the board. A user comment then opened a second hold for the same run (`native_continuation_requires_reconciliation`, `preserve_without_replay_v1`). That hold also failed 6 times and was recorded `resolved` with outcome `blocked`. The task's execution blocker is this second hold. **Expected behavior** After #20569 is deployed, the worker re-arms the export hold and saves the files. The no-replay hold of the same run then settles, and the task is no longer blocked. **Steps to reproduce** 1. Create both holds with the exact staging evidence (see the new test `stagingIncident`). 2. Run the recovery worker until nothing is due. 3. Before this change, the export hold completes, but the no-replay hold stays `resolved`/`blocked` and `getExecutionBlocker` still returns it with `canRetry: false`. ## What Changed - `workspace-recovery-fallback.ts` (`settleSiblingRecoveredHold`): a resolved no-replay hold (`isRetainedFileHold`) can now settle when a verified receipt of a sibling action saved its run's files. Settling removes `automaticRecovery`, so the hold stops blocking the task. It starts no turn. - `workspace-recovery-fallback-discovery.ts`: the worker discovers such a resolved hold when a sibling of the same run is recovered. - New test in `workspace-recovery-capture-policy.test.ts` with the exact staging shape of both holds: board owner, `active_run_watchdog`, system recovery policy version 1 without a capture policy version, stage `restoring_files`, `export_interrupted` with cause `provider_error`, 6/6 attempts, `historicalWorkspaceSource` and `workspaceRestoreRecovery` lease ids. - `doc/execution-semantics.md`: states that a resolved no-replay hold also settles from a sibling save. ## Verification - Red before the fix: the new test failed. The no-replay hold stayed `resolved`/`blocked`, and it stayed the execution blocker. - Green after the fix: `npx vitest run --root server src/services/workspace-recovery-capture-policy.test.ts` (3 passed). The test drives the worker's own discovery (`nextWorkspaceRecoveryAt` + `reconcileRetainedWorkspaces`) until nothing is due. It checks: the export hold is re-armed once and saves the files; the no-replay hold is `resolved`/`restored` by the sibling; `getExecutionBlocker` returns null; the continuation hook ran when the task was free; no new run was created; the worker goes idle. - Related suites: `workspace-recovery-fallback`, `-admission`, `-stop-proof`, `-source`, `-preparation`, `-preparation-gates`, `execution-blocker-read`, `preserved-native-response-wait.integration`: 409 passed. - `npx tsc --noEmit -p server/tsconfig.json`: clean. - `pnpm verify` exits 2 (global doc input), so CI supplies the broad coverage. ## Risks - A resolved no-replay hold is now released when another action's verified receipt saved the same run's files. The receipt check is unchanged, so an unverified sibling still does not release it. - Settling starts no turn. Waiting input and the run's own result continue the task through normal admission, which still checks every other hold. - Previous-release story: holds saved by earlier releases need no migration. The worker discovers them on its first pass after deploy. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
e3bc4ca798 |
fix(native): settle warm follow-ups that a restart left behind (#20631)
<!-- Write all pull request text in Simplified Technical English (ASD-STE100): short sentences, one instruction per sentence, simple approved vocabulary, and the active voice. --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native warm runs on a remote sandbox continue the saved session of the previous run. Admission first proves what happened to the previous Runner. > - A deploy restarted the staging server while warm Runners were idle. Three follow-up runs then failed before provider work, each with a different refusal, and their tasks stayed blocked. > - The refusals asked for proof that a restart had made impossible. A restart is a product failure, so recovery must settle it automatically and keep the strict checks for real uncertainty. > - This pull request accepts the proof that the restart leaves behind, in three places: the warm owner on a replaced sandbox, the suspended Runner with an undelivered suspend receipt, and the restore hold of a run that saved nothing new. > - The benefit is that these follow-up runs start, and state saved by older releases carries forward without a migration. ## Linked Issues or Issue Description Refs #20567, #20346 **What happened?** On staging, a deploy replaced the webserver process at about 18:04 UTC on 2026-11-10 (lease controller boot `348ec0bb` on one deployment, then boot `3e84c195` on a new deployment that started at 18:04:16). Three runs of one Codex agent on Daytona failed between 18:08 and 18:15: 1. A warm follow-up got a newly created sandbox. The previous warm Runner stayed `ready` in its own retained sandbox. Live adoption denied it with `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate` (`remote-warm-session-recovery.ts:127`), because only a resumed sandbox may be adopted. The per-turn retirement path refused warm owners (`lifecycle_not_per_turn`). 2. A warm follow-up resumed the same sandbox and directory. The previous Runner had been suspended by the shutdown, but the controller stopped before it acknowledged the suspend result. The Runner kept its terminal fence (`pendingTerminalDelivery` for `runner.suspend`). The read refused it with `native_controller_recovery_missing_evidence:runner_not_settled` (`native-session-executor.ts:7032`). 3. A continuation into a fresh workspace failed with `remote_workspace_recovery_receipt_invalid` (`remote-workspace-recovery-attachments.ts:60`). Its predecessor had restored the committed save, failed before provider work, and kept a restore hold with no recovery receipt. Its saved reference named the snapshot it restored. **Expected behavior** After a restart, the next run continues. It retires a finished owner that nobody can adopt, reads a finished suspension, and skips a source that left no unsaved files. Real uncertainty still refuses with a named reason. **Steps to reproduce** 1. Run a warm native Codex turn on a reusable Daytona sandbox and let it commit. 2. Restart the server (deploy) while the Runner is idle. 3. Start the next turn. Get a new sandbox (case 1), or resume the same sandbox after the shutdown suspended the Runner (case 2). 4. For case 3, continue a task whose previous run restored the last save and failed before provider work, into a fresh workspace. **Paperclip version or commit** Staging before `2e3cbe57` (the runs predate that deploy). The gaps remain on `master` at `9b21dcbd`. ## What Changed - `inspectRemoteWarmSessionRecovery` returns `null` when this run's sandbox is `created` or `replacement`. Such a sandbox holds no Runner to adopt. The caller then uses ordinary admission. - `inspectFinishedPerTurnRunnerdOwner` also retires a **warm** owner on a remote sandbox runner when this run's sandbox is `created` or `replacement`. It uses all the existing proofs: terminal run, committed result, released coordinator lease, stopped or fenced controller, settled journal, and quarantine copies. Local warm Runners and resumed sandboxes keep the live path. - `restoreRemoteSuspendedRunnerCheckpoint` accepts a suspended Runner whose only open fence is its own undelivered `runner.suspend` receipt. The fence must name `runner.suspend` with lifecycle `suspended`, it must be the last controller command, and the restored journal must not name that command differently. A provider cleanup, unsent events, a turn's terminal fence or a warm transition still refuse with `runner_not_settled`. A later authorized Runner process reconciles the receipt, as the Runner already does. - `attachPreservedRemoteWorkspaceSnapshots` skips a source with a receipt-less restore hold when its saved reference names the exact committed snapshot that its own succeeded `remote_workspace_restore` loaded at the same epoch. Any other saved snapshot still needs the canonical receipt. - `doc/execution-semantics.md` describes the three rules. ## Verification - Red (unfixed code), each in the exact staging shape: - `native-recovered-owner-followup.test.ts` "retires a finished warm owner left behind by a restart when this run's sandbox is created/replacement" failed with `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate`. - `native-controller-recovery.integration.test.ts` "defers a remote warm continuation…" failed: live adoption rejected a created/replacement sandbox with the same error. - `native-controller-recovery.integration.test.ts` "reads a resumed sandbox's suspended Runner with an undelivered suspend receipt" failed with `native_controller_recovery_missing_evidence:runner_not_settled`. It reads through a sandbox runner that refuses paths outside the workspace base. - `workspace-recovery-preparation.test.ts` "decides a receipt-less restore hold of a source that restored_unchanged" failed with `remote_workspace_recovery_receipt_invalid`. - Green: the same tests pass. Focused files pass: `native-recovered-owner-followup.test.ts` and `native-controller-recovery.integration.test.ts` (104 tests), `workspace-recovery-preparation.test.ts` (85 tests). - Safety cases (strict checks stay strict): - A warm owner on a resumed sandbox is not retired. - A warm owner with an unsettled journal (pending command, unanswered tool input) stays held and names `retirement_refused=journal_…`. - A warm owner whose finalization lease is still held stays held. - A turn terminal fence, a provider cleanup, unsent events, a fence behind a later command, or a journal that names the command differently still fail with `runner_not_settled` and cache nothing. - A restore-hold source that saved another snapshot, whose restore did not succeed, or whose restore was at another epoch still fails with `remote_workspace_recovery_receipt_invalid`. - Neighbouring suites pass: `native-session-executor.test.ts`, `plugin-runner-release-resume.integration.test.ts`, `explicit-native-continuation.test.ts` and `native-harness-filesystem-independence-continuation.integration.test.ts` (1786 tests). The last one asserted the old warm refusal on a created sandbox. It now shows that the warm follow-up resumes the same provider session for the new run. - `npx tsc --noEmit -p server/tsconfig.json` passes. - `pnpm verify` returns exit 2, because a global doc input changed (`doc/execution-semantics.md`). Required CI supplies the broad coverage. ## Risks - Previous-release state: all three rules read state that earlier releases already save (lease outcome, coordinator rows, Runner state in the sandbox, journal commands, run results and workspace operations). Rows saved by older releases carry forward. No migration. - Case 1 starts a new Runner for a warm session whose old Runner may still exist, idle, in its retained sandbox. Its controller is gone and its result is committed, so it can accept no new commands. The provider session follows the existing resume-or-replace policy. - Case 2 trusts a suspend fence only in the exact shape that the Runner writes before it sends the result. Other fences still refuse. - After case 1, a warm follow-up resumes from its own warm checkpoint. That checkpoint carries no replacement policy, so a failed provider resume is still reported as `native_session_recovery_failed` and is not replaced. This is the existing warm behavior. - Case 3 skips a source only when it provably saved nothing beyond a committed snapshot. That snapshot stays in its own workspace. - The three tasks that already failed stay blocked until their next wake or a user message. This PR does not add an automatic continuation for a completed no-replay hold. A follow-up can add that. - `native-runner-restart-recovery.integration.test.ts` (runs only with a runner container) no longer probes a replacement sandbox, because that probe now retires the owner by design. - Not verified live on Daytona. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, read-only browser access to staging). Extended reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude
|
24661683db |
fix(recovery): bootstrap a remote follow-up that a restart cut before its Runner launched (#20642)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - After a control-plane restart, restart recovery classifies each claimed native run: reattach a live Runner, resume from a checkpoint, or bootstrap again when durable records prove no provider work > - A follow-up run on a continued session gets a rebound admission checkpoint. It names the new run, but it keeps the previous turn's provider session ID > - Recovery read that copied provider session as this run's provider work. A follow-up killed after `native.process_start_requested` but before its remote Runner launched was classified `reattach_remote_runner` > - The reattach then had no Runner to find, failed with `native_continuation_requires_reconciliation`, and the task needed Retry > - This pull request stops the copied session from counting as this run's work, and keeps the existing durable proofs as the authority > - The benefit is that these restarts bootstrap again by themselves and do not repeat provider work ## Linked Issues or Issue Description Refs #20561, #20555 **What happened?** Two Daytona restart-chaos tasks needed a Retry click: - Wave `w-20261110T173243-s9202` t02 (claude C1, K2): the follow-up run was killed at `stage.asset.home`, after `native.process_start_requested`. Recovery chose `reattach_remote_runner`. Verification failed with `native.remote_runner_recovery_failed {stage: runner_state, reason: state_read_failed}` (`runner_remote_state_unavailable: exit=1`): `runner-state.json` did not exist because the Runner never launched. The storage fault window (ST7) had ended 30 s before, so it was not the cause. - Wave `w-20261110T174930-s9205` t04 (codex C1, RR1): the scheduled retry of a `process_lost` run was killed at `session.checkpoint.restore`, after `native.process_start_requested`. Recovery chose `reattach_remote_runner`. Verification failed with `native_remote_recovery_lease_mismatch: stage=runner_verification field=identity_run_id`: the local Runner identity belonged to the previous run, because this run's Runner never started. **Expected behavior** A run whose durable records prove that no Runner received a provider turn bootstraps again without a person. **Steps to reproduce** 1. Start a remote follow-up run on a continued session (for example, a comment after a completed turn). 2. Kill the server after `native.process_start_requested` and before the remote Runner launches. 3. Observe `reattach_remote_runner`, a failed verification and a Retry banner. **Paperclip version or commit** master close to `6d6b2d0e` (waves); still reproduced by the new tests on master `90dd864bff`. ## What Changed - `native-restart-recovery.ts`, no controller checkpoint for the run (`remote_runner_never_received_commands`): a follow-up with `sessionIdBefore` qualifies when it carries only its continued-session checkpoint, and a predecessor's settled Runner identity (`carriesInheritedRunnerIdentity`) no longer disqualifies it. A recorded process identity, a Runner-sent event, a provider event or a pending request still keeps the run held. - `native-restart-recovery.ts`, controller checkpoint exists: the existing Postgres proof (`nativeRunnerCheckpointRemoteBootstrapProof`) now also runs when the run carries only a rebound checkpoint whose provider session a settled earlier run of the same session saved (`carriesSettledPredecessorSession`). The copied provider session then does not count as this run's provider evidence. The controller checkpoint stays the authority: any turn command or turn event keeps the run held. - `doc/execution-semantics.md`: describe the rule. ## Verification - `server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts` (real Postgres): - "restarts a follow-up whose controller never saved a checkpoint (s9205 t04)" and the variant "with its predecessor's remote runner identity": red on master (`reattach_remote_runner`), green now. - "restarts a follow-up whose saved controller checkpoint proves no provider turn (s9202 t02)": red on master, green now. - Held cases stay held: own recorded Runner identity, a carried active turn, a provider session that no settled earlier run saved, a `turn.start` command, a projected `turn.started` event. - Whole file: 171 passed. - `npx tsc --noEmit -p server` passes. - The executor's existing `bootstrap_incomplete` path handles both shapes: a scoped root that belongs to the previous run goes through the existing prior-owner verification, and a root of this run restored from the controller checkpoint goes through `runnerdStateProvesIncompleteBootstrap` / `runnerdStateProvesNoRemoteProviderTurn`. ## Risks - A run with any recorded Runner process, Runner-sent event, provider event, turn command or pending request keeps today's reattach or hold. - A bootstrap claim charges one provider attempt, as today. The bounded limit (3) still applies. - No persisted shape changes. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
6892ed3a98 |
test(restart-chaos): save a screenshot and banner HTML when the Retry button wait fails (#20643)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart-chaos harness drives the board UI and clicks Retry the way a person would > - When no visible Retry appears, the harness grades `ui_retry_button_missing` and keeps only the banner text > - That is not enough to tell a product bug (no button, a hidden button, a wrong name) from a harness locator bug after the wave has ended > - This pull request saves a full-page screenshot and the banner and Retry-button DOM when the wait fails > - The benefit is that every `ui_retry_button_missing` can be diagnosed from the wave folder ## Linked Issues or Issue Description Refs #20616 **What happened?** Wave `w-20261110T174930-s9205` t01 graded `ui_retry_button_missing`, but the captured banner text contained "Retry". The harness saved no screenshot or DOM, so nobody could prove whether the page had a hidden duplicate or no actionable button. **Expected behavior** A failed Retry-button wait leaves enough evidence to diagnose it later. **Steps to reproduce** 1. Run a restart-chaos wave with the board UI. 2. Get a task whose Retry wait times out. 3. Observe that the wave folder holds only the banner text. ## What Changed - `tests/restart-chaos/board-ui.ts`: add `saveRetryMissingDiagnostics`. On a failed Retry wait it saves `ui-retry-missing-<issue>-<time>.png` (full page) and a `.json` file with the page URL, each `[role=status]`/`[role=alert]` banner's `outerHTML`, and every Retry-named button's visibility, bounding box, banner membership and HTML. Diagnostics never change the click result. - `tests/restart-chaos/main.ts`: the wave's board helper writes to `<wave>/ui-retry-diagnostics`. - `tests/restart-chaos/observer.ts`, `types.ts`: the task's retry attempt records the saved file paths. - `tests/restart-chaos/board-ui.test.ts`: a test for the saved files. ## Verification - `npx vitest run --config tests/restart-chaos/vitest.config.ts tests/restart-chaos/board-ui.test.ts`: 9 passed (new: "saves a screenshot and the banner's HTML when no visible Retry appears"). - `npx tsc --noEmit -p tests/restart-chaos/tsconfig.json` passes. ## Risks - Harness only; no product code changes. A failed screenshot or file write is ignored and does not change the grade. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
ed5e7493bb |
fix(native): take back a lapsed lease held under the run's own restart claim (#20625)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native run holds an execution lease on its `native_run_finalizations` row. The owner renews it every 5 minutes. After a host sleep (or the restart-chaos time fault TM2), the stored lease looks lapsed although the owner is alive. > - #20474 lets the same owner take back a lapsed lease when no other actor recorded anything on the row. One of the checks is "`recovery_state` is null". > - Restart recovery writes `recovery_state` (`resuming_session`, `awaiting_runner_reattach` or `bootstrap_incomplete`) in the same update that grants the lease. The executor keeps that value until a result is accepted. > - So a run that a restart resumed can never take back its own lapsed lease. Renewal interrupts the turn, and failure settlement is refused with `native_session_lease_lost`. The run fails with `adapter_failed`, recovery holds it with `preserve_without_replay_v1`, and the user must click Retry. > - This pull request counts the claim's own record as untouched when the row still shows exactly that claim and nothing recorded since. > - The benefit is that a resumed run survives a later host sleep with no Retry. Every real takeover still fences the late owner out. ## Linked Issues or Issue Description Refs #20474 **What happened?** The zero-retry restart-chaos matrix (local wave L7, seed 9107: TM1, TM2, TM3 with 1 h aging) had 3 of 6 tasks that needed Retry. All three were Codex runs. In each run, a TM3 kill made restart recovery resume the run (`resume_dead_runner`, generation 2). The runner restored the conversation, reported `provider_turn_lost_on_restore`, and the same-conversation restart continuation started as designed. Then the TM2 fault aged every database time by one hour. The execution lease lapsed while its owner was alive. The run then failed with `native_session_lease_lost` (`adapter_failed`), and recovery stopped with `native_continuation_requires_reconciliation` / `preserve_without_replay_v1`. One run had already proposed its result and completed its turn. The Retry click worked every time. **Expected behavior** A run whose lease lapsed only because time jumped, and that no other actor claimed, continues with no Retry. This must also be true after a restart resumed that run. **Steps to reproduce** `node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env local --adapters codex,claude --workloads A1,B5,C1 --faults TM1,TM2,TM3 --time-age-hours 1 --concurrency 3 --tasks 6 --seed 9107` ## What Changed - `server/src/services/native-runtime/native-execution-lease.ts`: the "untouched" part of `nativeExecutionLeaseHeldCondition` now also accepts the run's own local restart-recovery claim. All of these must be true: - the last `recovery_history` entry is that claim: same controller boot and generation as the row; - its disposition matches the current `recovery_state` (`resume_dead_runner` → `resuming_session`, `reattach_existing_runner` → `awaiting_runner_reattach`, `bootstrap_incomplete` → `bootstrap_incomplete`); - `recovery_request_id` equals the request that claim recorded. Any later actor appends a history entry (shutdown suspension, failure settlement), moves the state (`awaiting_evidence`, `blocked`), or replaces the owner, boot or generation. Each of these still fences the late owner out. All other untouched checks are unchanged. - `native-execution-lease-reacquire.test.ts`: new cases for each claim kind, a requested claim, failure settlement under the claim, and eight fenced variants. - `doc/plans/2026-11-09-explicit-native-lease-claim-tokens.md`: Step 1 lists the new rule. ## Verification - Red/green: `cd server && npx vitest run src/services/native-runtime/native-execution-lease-reacquire.test.ts`. Without the source change, 5 new cases fail (the three claim kinds, the requested claim, and failure settlement). With it, all 37 pass. - Related suites pass: `native-execution-lease.test.ts`, `heartbeat-restart-claim-lapse.test.ts`, `heartbeat-native-orphan-sweep-lease-race.test.ts`, `native-runner-restart-lease-renewal.integration.test.ts`, `native-runner-restart-preparation.integration.test.ts`. - `cd server && npx tsc --noEmit` passes. - Live: the same L7 wave (seed 9107, local) on this branch. The server log has 0 `native_session_lease_lost` (the baseline had 3). The two tasks whose cells failed this way in the baseline (codex C1 TM1, codex C1 TM3) now pass with no Retry. Two other tasks needed Retry for different causes, which this pull request does not change: codex B5 TM2 (a `runner.drain` timeout, then a same-run retry that may not continue a turn lost on restore) and claude C1 TM2 (`ACPX session is not open` on `turn.start` until the retry budget ran out). The L6 wave (seed 9106) also ran on this branch: 5 of 6 passed. Its one failure is `execution_recovery_budget_exhausted` and is not related to leases. ## Risks - The rule widens when an owner may take back its own lapsed lease. It applies only to local runs, before a result, with the same owner token, attempt, boot, pid, process start and generation. A claim that any other actor changed afterwards does not match. - Rows saved by earlier releases carry the same history entry shape, so a run claimed before a deploy is handled the same way. A deploy changes the boot, so the new process never takes back the old process's lease. - No schema change and no migration. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test runs). Extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
2e6850e745 |
perf(task-chat): open tasks from a preloaded cache and cold tasks in one request (#20592)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - People open tasks from task lists, the Inbox and the sidebar many times each hour > - A task's chat shows its first frame only after about seven separate reads finish > - #20355 added one batched `chat-bootstrap` read that returns all of these reads. #20450 used it only for the sidebar's Recents > - The hosted database is CPU-bound, so the UI must open tasks faster and must also send fewer requests > - This pull request uses the bootstrap for task lists, for open intent on any task link, for cold opens and for a hard load of a task URL. It shares one coordination module between all of these paths > - The benefit is that a task opens from cache more often, a cold open sends one chat request instead of about seven, and the first content paint comes sooner ## Linked Issues or Issue Description Refs #20355 (server read that this pull request consumes). Refs #20450 (Recents preload, now on the shared path). **Is your feature request related to a problem?** A click on a task in a task list waited for about seven separate chat reads. The default task list (Coordinator) had no preload at all. A press or a resting hover on a task link sent the same seven reads, and the page then sent a second set. A hard load of a task URL started its chat reads only after the JavaScript bundle and the company list loaded. Sidebar project task links had no open-intent prefetch. **Describe the solution you'd like** Use the single `chat-bootstrap` read everywhere a task is about to open. Let the page's gating reads share a bootstrap that is in flight instead of sending their own requests. Keep the core issue read separate so the header does not wait for the larger read. **Describe alternatives you've considered** Preload 30 rows per list in three requests (the first prototype). Each task costs the server about fifty queries, and a resting hover warms any other row before the click, so this change preloads 10 rows in one request. ## What Changed - `ui/src/lib/issue-chat-bootstrap.ts`: one module decides when to send a bootstrap and lets page reads share it (`readFromIssueChatBootstrap`). Seeding uses the existing `issueChatBootstrapCache` (replace-style reads are refreshed when newer, merge-style reads are filled only when empty, other companies are ignored). It skips tasks that are loading, loaded in the last 30 seconds, or already fresh in cache, under the id or the identifier. It never aborts a request in flight, so a click that unmounts the list still uses the answer. Each request is bound to the account that sent it; a request sent before the first session read is bound to the first account the session reports, and a sign-out revokes it. A page read shares a bootstrap only while its key holds no data (a refresh after a write always reads the server), and waits at most 5 seconds for it. Every ref of a task (id and identifier) is tracked at request start. Entries expire after 30 seconds. - Task lists preload the chat of their top 10 rows in one request, after the list's own reads settle and the browser is idle (`usePreloadTaskChats`, `when-page-settles.ts`): the Coordinator task list (`TaskList` with `preloadCompanyId`), the streamlined and legacy issues lists and the streamlined Inbox. Recents now uses the same path, so a task in both places loads once. A list that stays open preloads each task at most once in five minutes: live events (#20563's lanes) keep a preloaded task's cache current or mark it stale, and a stale key is read from the server, never from an older bootstrap. All seeded keys are per-task keys on the fast live lane, not the ambient lane. - Open intent (press, click, resting hover or focus) on a task with no chat history in cache warms it with one bootstrap request instead of six separate reads (`prefetchIssueDetailForNavigation`). A task seen before still refreshes its cached reads one by one. - A cold open starts one bootstrap. The gating reads (recent comments, activity, runs, input facts, interactions, attachments, live runs, active run) wait for it instead of sending their own requests. The core issue read stays independent. - A hard load of a task URL starts `GET /api/issues/:ref/chat-bootstrap` from the inline preload script, in parallel with the JavaScript bundle. The page adopts that answer. - Sidebar project task links (Coordinator sidebar) get the same hover, press and focus warming as Recents and `IssueLinkQuicklook`. - Tests: unit tests for the coordination module, Recents, `TaskList` preload, sidebar warming and the cold-intent path. A new layout journey checks that a task the Inbox preloaded reveals once at its latest message without reading its chat again. Journeys that hold one separate read now also hold or fail the bootstrap. The Inbox reveal journey held a read that the chat no longer sends on open (it was flaky on master too), so it now holds the recent comment window. - The perf harness gains `PAPERCLIP_ISSUE_PERF_LIST_DWELL_MS` and the README documents it. ## Verification Measurement setup: production UI builds of master (`e01fe7f84a` and the earlier base `8d2bf78323`) and of this branch, served by one local server with `PAPERCLIP_UI_DEV_MIDDLEWARE=false` and embedded PostgreSQL. Only `index.html` changed between variants (both asset sets were present), so the server, database and seeded task were the same. Four rounds of the issue-detail harness, with master and branch alternated in each round: 20 pairs per S1 cell, 40 per S2 cell. The host was shared with other agents (load average 16 to 43), so the spread is large. Content paint (the task is readable), median paired delta, branch minus master: | Scenario | Profile | Pairs | Content paint | Branch faster in | API requests before content (master / branch) | |---|---|---:|---:|---:|---:| | S1 preloaded click (dwell 1500 ms) | unthrottled | 20 | -45 ms | 15/20 | 25 / 18 | | S1 preloaded click (dwell 1500 ms) | Fast 4G, 4x CPU | 20 | -23 ms | 13/20 | 26 / 18 | | S1 click, no dwell | unthrottled | 20 | -51 ms | 15/20 | 25 / 18 | | S1 click, no dwell | Fast 4G, 4x CPU | 20 | -21 ms | 12/20 | 26 / 18 | | S2 cold deep link | unthrottled | 40 | -61 ms | 28/40 | 54 / 40 | | S2 cold deep link | Fast 4G, 4x CPU | 40 | -129 ms | 29/40 | 47 / 39 | Medians (p25 to p75), content paint in ms: S1 dwell unthrottled 295 (283-315) → 248 (217-276). S1 dwell Fast 4G 1133 (1114-1226) → 1093 (853-1176). S1 no dwell unthrottled 295 (277-328) → 260 (233-332). S1 no dwell Fast 4G 984 (842-1196) → 923 (816-1110). S2 unthrottled 1002 (677-1235) → 896 (619-1280). S2 Fast 4G 4643 (4155-5061) → 4516 (4298-4894). Header paint is neutral to slightly later in two cells: S1 dwell unthrottled +39 ms (3/20 faster) and S2 Fast 4G +56 ms (17/40). In those cells the header now paints in the same frame as the content. In the first two rounds, the median gap from header to content fell from 105-157 ms to 29-74 ms unthrottled and from 557 ms to 298 ms for S2 on Fast 4G. Other header cells are -27 to +1 ms. The local server answers each read in a few milliseconds, so the network wait that the preload removes is small here. The request count is the stable result: a cold open sends one chat request instead of about seven. List page: the preload request starts after the first task row is in the DOM (64 ms after it unthrottled, 200-280 ms after it on Fast 4G, page clock). Time to the first visible row, 30-task list: unthrottled master 1863 ms vs branch 1563 ms (6 pairs, branch faster in 5). Fast 4G master 3626 ms vs branch 3727 ms (14 pairs, branch faster in 6, mixed signs). The preload starts after the row paints, so it cannot delay it; the Fast 4G difference is noise. One 10-task bootstrap took 78-162 ms on the server. Checks run: - `pnpm exec vitest run` for the bootstrap, cache, prefetch, Recents, sidebar, quicklook, `TaskList`, issues list, Inbox, issue detail and Coordinator tests: 18 files, 646 tests pass. - `pnpm exec tsc -p ui/tsconfig.json --noEmit`: clean, except missing generated Storybook catalog files that this change does not touch. - `pnpm check:token-gates`: clean. - `tests/e2e/task-loading-frame.spec.ts`, `disconnect-resilience.spec.ts` and `connection-reviews.spec.ts` (production build): 21 passed, 1 skipped. - `tests/perf/issue-detail/layout-stability.spec.ts` and `plan-startup.spec.ts` against the production build: all pass. The Inbox reveal journey and the new preloaded journey passed 4 of 4 repeats each. - `pnpm verify` asks for broader CI coverage because an e2e spec changed. ## Risks - More server work for list views: one bootstrap of up to 10 tasks per list visit (about fifty queries per task), skipped for tasks already loaded in the last 30 seconds. It runs only after the page's own reads settle and the browser is idle, and only while the tab is visible. - A resting hover on a never-opened task now sends one bootstrap instead of about six reads. The server work per task is about the same; the request count is lower. - If a bootstrap fails, each page read falls back to its own request. The fallback is slower than before for that one open (the reads start when the page mounts, not at the press). - The header can paint a few tens of milliseconds later in the same frame as the content when everything is cached. - An answer that arrives after a sign-out or account change is dropped. An answer for another company is ignored. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, Playwright runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude
|
6505538fc5 |
fix(realtime): limit a shared live-event read set to 100 events and 250 ms (#20632)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - The server sends the output of a running agent to the browser on the
company WebSocket.
> - #20624 lets the events that are already in the queue share one set
of permission and redaction reads.
> - That set could serve a full queue of 1,000 events for one second.
> - If access is revoked during that time, the principal still receives
the queued output. The security scan on #20624 reported this.
> - This pull request limits one set to 100 events and 250 ms.
> - The benefit is a small, stated limit on output that a revoked
principal can receive, with the same delivery speed.
## Linked Issues or Issue Description
Refs #20624
**What happened?**
After #20624, one set of permission reads can authorize each event that
was in the queue when the set opened. The queue holds at most 1,000
events or 1 MiB. The set stays in use for one second. If a membership,
an API key or a task grant is revoked in that time, the server still
sends the remaining queued events.
**Expected behavior**
A revocation stops delivery after a small, fixed quantity of queued
output. The documentation states that quantity.
**Steps to reproduce**
1. Open a company WebSocket as a user who can read a private task.
2. Publish 1,000 transcript events for a run of that task in one burst.
3. Revoke the task grant while the server delivers the burst.
4. The server delivers the remaining queued events.
**Paperclip version or commit**
`master` at
|
||
|
DottaandClaude Opus 5.5
|
fd6e71a7ff |
fix(native): restore a warm backup when its source sandbox is idle (#20627)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs on remote sandboxes keep a warm sandbox and a verified harness backup so the next run continues the same provider session > - A run that starts while the previous run's workspace export is still saving gets a new (created) sandbox lease, because reuse skips an exporting sandbox > - When that run prepares the harness, it finds a backup but no recorded reusable lease, and it fails closed with `runner_harness_state_mismatch: backup_without_reusable_lease` > - #20278 accepted a destroyed source and a settled restore hold as retired backup sources, but not an idle warm source that is still `retained` for reuse > - This pull request accepts an idle warm source under the same proofs, claims it so it is never resumed again, and covers the resumed lease that the failed run leaves behind > - The benefit is that these recoveries continue the same session automatically instead of blocking the task ## Linked Issues or Issue Description Refs #20278 **What happened?** A monitor wake started about 90 seconds after the previous run on the same task finished. The previous run's warm sandbox lease was `retained` with policy `reuse_by_environment`, but its workspace export was still saving. Reuse skipped it and created a new sandbox. Workspace realization waited for the export (about 136 s) and restored the saved files. Then harness preparation found the previous run's backup (its lease stamp names this exact manifest, and no provider work followed it). The source lease was not destroyed and was not a restore hold, so the run failed with `native_session_recovery_failed: Error: runner_harness_state_mismatch: backup_without_reusable_lease` and the task was blocked. **What did you expect?** The run continues the same provider session. The idle source sandbox holds exactly the backed-up state, so restoring the backup into the new sandbox is safe. **Steps to reproduce** 1. Finish a warm native run on a remote sandbox. Its workspace export is still saving. 2. Start the next run on the same task before the export settles. 3. Reuse skips the exporting sandbox and creates a new lease. Harness preparation fails with `backup_without_reusable_lease`. 4. A retry resumes that new lease, which has no harness state, and fails with `resumed_unavailable`. ## What Changed - `native-retired-backup-source.ts`: an idle warm source is a third retired backup source. It must be `released` or `retained` with policy `reuse_by_environment`, its own run must be terminal and not the current run, and it must have no cleanup in flight, no restore hold or pending shutdown settlement, and no pending native workspace export. The backup stamp, single-owner chain, and later-provider-work checks still apply. - Before the restore, the idle source is claimed with a compare-and-swap on the checked row. The claim moves its `reusableSandboxLease` scope into a `reusableSandboxLeaseSuperseded` receipt. Reuse never resumes that older state again, and the next acquisition treats it as obsolete and destroys it. If a run resumed or changed the source first, the claim matches no row and the backup is not restored. - `environments.ts`: a cross-run reuse handoff now requires that the source row has no superseded receipt. The supersede claim and the handoff compete on the same row, so an acquisition that selected the source before the claim cannot resume it afterward. - The proof now rejects a backup whose source is the target allocation itself (`targetProviderLeaseId`). - `native-session-executor.ts`: a resumed lease whose sandbox has no harness state at all (`unavailable`) restores the backup under the same proof. This recovers the lease that the failed run left behind, so rows saved by older releases recover after deploy without a migration. - `doc/execution-semantics.md`: documents the idle-source rule, the supersede claim, and the resumed stateless lease. ## Verification - Red before the fix, green after: - `native-session-executor.test.ts` "restores a warm backup into a created lease only when an idle warm source finished its run and its export" failed with `backup_without_reusable_lease`. - "... a resumed lease never held the session and the warm source is idle" failed with `resumed_unavailable`. - `native-retired-backup-source.integration.test.ts` (real Postgres) "authorizes an idle retained/released warm source and claims it as superseded" failed (returned false). - Integration test "refuses a reuse handoff of a source superseded while its provider resume was pending" fails without the handoff guard and passes with it. - Safety rows stay refused: source run still running, workspace export pending, cleanup in flight, restore into its own allocation, stamp names another checkpoint, provider work after the backup, two host leases for one allocation, claim lost to another run, and a resumed lease that lost its own state. Existing destroyed-source and restore-hold rows are unchanged. - `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts src/services/native-runtime/native-retired-backup-source.integration.test.ts src/services/native-runtime/remote-warm-session-recovery.test.ts` passes (1408 tests). - `npx tsc --noEmit -p server/tsconfig.json` passes. - `pnpm verify` exits 2 because a global CI input (`doc/execution-semantics.md`) changed. Required CI supplies the broad coverage. ## Risks - Previous-release story: no persisted shape changes. Lease rows saved by earlier releases carry forward. The new `reusableSandboxLeaseSuperseded` metadata key is additive. - The idle source is destroyed at the next acquisition for its workspace. Its state is the verified backup, which the restored lease now holds, so no work is lost. - Not covered: when the source cannot be proven retired or idle (for example a live owner or later provider work), the run still fails closed with the same named code. An automatic fresh-session fallback for that case is not in this PR. - The root trigger (reuse skips a sandbox while its export saves, then waits for that export) is unchanged. This PR does not change the reuse lookup. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
76c787fffc |
fix(runner): report monitor notes length limits (#20604)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents use task monitors to resume a future check. > - The monitor schema limits notes to 500 characters. > - The field description did not state the limit. Rejection showed only the schema path. > - This pull request adds the limit to the description and to safe length errors. > - Agents can correct notes before they retry a rejected call. ## Linked Issues or Issue Description Related prior fix: #16868. **What happened?** A native `set_task_monitor` call with 501 characters in `monitor.notes` failed with a path that ended in `notes/maxLength`. The error did not state 500. The schema already had `maxLength: 500`, but the field description did not state it. **Expected behavior** State the limit in the field description and in the rejection. Accept 500 characters. Reject 501 characters before dispatch. Preserve a saved monitor when a replacement is invalid. **Steps to reproduce** 1. Supply a future `monitor.nextCheckAt` and 501 characters in `monitor.notes`. 2. Read the native provider error. 3. Before this fix, it shows the constraint path without the maximum. **Paperclip version or commit** Reproduced on `7368a0b15c2f02107778a961144d86e2eb59a952` in a Linux source checkout, with Node 25.9.0 and Rust 1.97.1. ## What Changed - Use one local value for the published notes limit and its description. The server test checks agreement with the persisted shared validator. Keep the standalone Runner package boundary. - Read the numeric maximum from the authorized JSON Schema validation error. Keep submitted values private and preserve the 512-byte feedback bound. - Test the published schema and real bridge dispatch path at 500 and 501 characters. Test saved contents and rejection without monitor changes with real PostgreSQL. - Refresh generated contracts and document the limit. - Sync two SDK proof lock identities after the reviewed master lock refresh in #20615. This fixes the remaining CI proof mismatch. Keep compiler, parser, Zod, and all rejection checks unchanged. ## Verification - Final head: `bac60234ab3a8fc7043a90c7f9c81b02e250428b`. - Red: the server regression failed because the field description omitted `500`. The Rust regression failed because the error omitted `maximum 500 characters`. - Local passes after the final base update: 44 PostgreSQL monitor/scheduler tests, 14 real-source Rust bridge tests, 11 cache/Docker guard tests, and 56 SDK proof tests. Runner TypeScript build and generated contract checks pass. - The focused Rust check compiles the actual source in a small Cargo workspace. The full Rust CI suites also pass on Linux and Windows. - Final-head [PR CI run](https://github.com/paperclipai/paperclip/actions/runs/42881633978) passes. This includes typecheck, build, Runner support, Runner Rust and Vitest, server tests, serialized tests, and all eight e2e shards. - Final-head [Windows Rust run](https://github.com/paperclipai/paperclip/actions/runs/42881633419) passes. - Greptile: 5/5 on the final head, with no open findings. There are 61 successful checks and five conditional skips. The separate Paperclip Review remains queued; its comment says work is paused. No CI or test check is pending or failed. - `pnpm verify` returned exit 2 and selected broader CI. The full local suite was not run. Final-head CI supplies that coverage. Earlier failed attempts were repaired and are not counted as passes. ## Risks - Low risk. The existing 500-character bound, scheduling semantics, and permissions stay in place. - Length feedback uses only a schema path and numeric bound. It does not print submitted text. - The Runner binary and tool contracts must reach an instance for the new guidance to apply. This PR does not deploy or merge. ## Model Used OpenAI Codex, GPT-6, with reasoning, tool use, and code execution. The runtime does not expose a more specific model ID or context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>canary/v2026.1110.0-canary.17 |
||
|
DottaandClaude Opus 5.5
|
38944cff1a |
fix(environments): retry and safely replace a reusable sandbox that will not resume (#20626)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A task that runs on a reusable (warm) cloud sandbox resumes the same sandbox on the next wake > - When a follow-up arrives while the provider is auto-stopping or archiving the idle sandbox, the resume can fail > - Each failed attempt waited the provider's whole five-minute start deadline and then failed with `reusable_sandbox_resume_failed`, even when the workspace was saved and nothing lived only on the sandbox > - Two such runs in a row made a user wait for many minutes and recorded two failed runs > - This pull request bounds the resume, retries a state change inside the run, and replaces the sandbox from the saved snapshot when that is safe > - The benefit is that recovery works automatically and the first turn is not delayed by minutes ## Linked Issues or Issue Description **What happened?** Two consecutive runs failed before provider dispatch with: `Reusable sandbox lease "<id>" could not be resumed (cause=provider_error); the lease was preserved and no replacement was created.` Each run took about 5.5 minutes (320s and 330s) in preparation. The third run resumed the same sandbox from `stopped` in 4.7 seconds after the provider settled. The previous run had ended with a clean final save and a verified harness backup. **Steps to reproduce** 1. Let a task's reusable sandbox go idle so that the provider auto-stops or archives it. 2. Send a follow-up while the provider still reports the sandbox as changing state. 3. The Daytona worker waits in `ensureSandboxStarted` until its 300s start deadline and throws "is still <state> at its start deadline". The host does not classify that as transient and it has no safe fallback, so the run fails. **Expected behavior** The run retries briefly. If the sandbox still cannot start and nothing lives only on it, the run continues on a replacement sandbox that restores the saved workspace. **Root cause** `server/src/services/environment-runtime.ts` (reusable resume block, about line 1820): the resume deadline was the provider's full start deadline (300s plus RPC margin), the transient classifier did not recognize a state transition, and the catch always threw `ReusableSandboxResumeError`. #20212 waits out a state change inside the same 300s start deadline, so it covers the start but not the time budget or the fallback. ## What Changed - Add `server/src/services/reusable-sandbox-resume-replacement.ts`: - `REUSABLE_SANDBOX_RESUME_BUDGET_MS` (90s): the total resume time for one run, retries included. - `sandboxStateTransitionFailure`: "state change in progress" and "is still <state> at its start deadline" are transient. - `retireUnresumableReusableSandboxLease`: replace only when the lease is `released`/`retained` with no live owner, the task workspace is clean with a committed saved snapshot and no other writer, and a `paperclip_runner` session has a verified lease-bound harness backup. Access refusals (401/403) and an unavailable worker stay strict. After the replacement lease (which carries the harness backup) is recorded, the old lease moves to `pending_cleanup` (reason `resume_failed`) in one guarded write, so the existing cleanup sweep destroys the old sandbox and no provider delete runs before the first turn. A failed creation or a restart before that write leaves the old lease `retained` for the next attempt. - `environment-runtime.ts`: cap the resume deadline at the budget, treat a state transition as transient, and call the replacement policy before failing. A replacement records `sandboxLeaseAcquisition: { outcome: "replacement", reason: "resume_failed" }`; workspace preparation then restores the saved snapshot. - `remote-workspace-transport.ts`: add the closed failure label `sandbox_state_transition`, so a strict failure names the cause instead of `provider_error`. - `doc/execution-semantics.md`: document the budget, the in-run retry, the replacement conditions, and the strict cases. - Update the two slow-start tests: the resume RPC timeout is now 90s instead of 330s/150s. A 61s start still resumes the same sandbox. ## Verification - Red before the fix (new file `server/src/__tests__/environment-runtime-reusable-resume-replacement.test.ts`, staging shape: lease created by the real acquisition path, then retained after a clean save with a backup stamp, 300s provider deadline, worker refusal with no status): 3 tests failed with `could not be resumed (cause=provider_error); the lease was preserved and no replacement was created.` - Green after the fix: 10/10 pass, including a case where the replacement creation fails and the next run replaces the sandbox with the backup carried forward. The safe case replaces the sandbox, keeps every resume RPC at or below 90s, makes no synchronous destroy call, and moves the old lease to `pending_cleanup`. A single state-change refusal resumes the same sandbox inside the run. Six strict cases keep the lease and fail with `cause=sandbox_state_transition`: unsaved work, no saved version, an uncommitted saved version, a live owner, no verified backup, and a 403. - `pnpm exec vitest run src/__tests__/environment-runtime.test.ts` (197 pass), `heartbeat-retry-scheduling`, `legacy-execution-recovery`, `remote-workspace-transport` (251 pass). - `tsc -p server/tsconfig.json --noEmit` passes. - `pnpm verify` exits 2 (a doc change is a global CI input). Required CI supplies the broad coverage. - Not run: a live Daytona run on staging. The provider message is kept off the run record, so the exact transition state on staging (stopping or archiving) is inferred from timing, not observed. ## Risks - A strict resume now gives up after 90s instead of 300s. The provider start continues in the background, and the bounded bootstrap retry resumes the same sandbox. A stopped sandbox resumes in seconds and an archived one in about 25s on Daytona, so 90s leaves headroom. - A replacement leaves the old sandbox to the cleanup sweep. If the old start finishes later, the sweep still destroys it. The guarded write only claims a `released`/`retained` lease with no live owner. - Upgrade safety: no new persisted fields are required. Leases saved by older releases are read with the same rule; one without a verified backup stays strict. The replacement reason `resume_failed` already exists in the acquisition outcome. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Devin FoleyandPaperclip
|
8148fc0b1e |
fix(release): fetch draft-notes PR summaries in batches (#20602)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The release workflow drafts the next stable's release notes when a beta publishes (`draft_stable_notes`) > - The draft nests each referenced PR's summary under its commit subject, and it fetches each summary with its own `gh pr view` call > - A beta range now references more than 1,500 PRs, so these serial calls do not finish inside the job's 10-minute timeout > - This pull request fetches the summaries in batches of 50 per GraphQL query > - The benefit is that the job finishes in under a minute and pushes the notes branch again ## Linked Issues or Issue Description **What happened?** The `draft_stable_notes` job of [run 42763789753](https://github.com/paperclipai/paperclip/actions/runs/42763789753) (beta `2026.1110.0-beta.0`) reached its 10-minute timeout. GitHub cancelled it during "Draft stable notes from the published beta", while a `gh` process was still running. The beta itself published and its smoke tests passed. The `release-notes/v2026.1110.0-beta.0` branch was not pushed. **Expected behavior** The job writes the notes draft and pushes the branch for every beta, whatever the size of the commit range. **Steps to reproduce** 1. Publish a beta whose range from the last stable has many PR-referencing commits. This range has 1,586 commits, and 1,583 different PRs. 2. `draft-stable-notes.sh` calls `gh pr view <n>` once for each commit, one after another. 3. The job reaches `timeout-minutes: 10` before the loop ends. **Paperclip version or commit** `09e24dd12c36251dd12af2d73dbc715ce990f922` (beta `2026.1110.0-beta.0`). This was the first beta to run the job after #12124 added the PR summaries; earlier betas skipped it. ## What Changed - `scripts/draft-stable-notes.sh`: - Collects the unique PR numbers first. Fetches their bodies 50 per `gh api graphql` query (aliased `pullRequest(number:)` fields), into a temporary cache that `enrich_pr` reads. - Parses the raw response with `jq`, not with `gh --jq`. When any number in a batch is not a PR (for example, an issue reference), gh exits non-zero and does not apply its `--jq` filter. The other PRs in the batch still have usable data. - The output format does not change. A missing `gh` or `jq`, a failed batch, a cache directory that `mktemp` cannot create, or `DRAFT_NOTES_SKIP_PR_ENRICHMENT=1` still gives bare subjects. - No code comments are added (per `CONTRIBUTING.md`); this description holds the explanation. - `scripts/draft-stable-notes.test.mjs`: - The enrichment tests now use a fake `gh` that answers GraphQL like GitHub. - New tests: batching (120 PRs make 3 queries, and a repeated PR is fetched once); a batch with a non-PR number; a failed batch; a failed `mktemp`; prose-only and empty bodies; the opt-out. - Range tests now run with enrichment disabled, so an authenticated `gh` on the test machine cannot reach GitHub. - `package.json`: adds the test file to `test:release-registry`. These tests did not run in CI before. ## Verification - `node --test scripts/draft-stable-notes.test.mjs`: 17 pass. They also pass with macOS bash 3.2 first on `PATH`. - `node scripts/ci-profile/node-test.mjs scripts/draft-stable-notes.test.mjs scripts/release-lib.test.mjs`: 31 pass. - Red check: the new tests against the old script fail 5 cases (all batching and enrichment cases). The `mktemp` test fails against an unguarded `mktemp` call. - Live run on the real range: `./scripts/draft-stable-notes.sh 2026.1110.0-beta.0` finished in 47 seconds. It wrote 4,596 excerpt lines for 1,583 PRs. - Parity: for 40 randomly sampled PRs from that range, the new excerpts are byte-identical to the old per-PR `gh pr view` output. - `shellcheck`: no new findings. The two remaining notices were there before this change. ## Risks - Low. The change touches only the post-publish draft job, and every failure still gives the bare-subject draft. - `jq` is now needed for the PR summaries. GitHub's Ubuntu runners include it. Without `jq`, the draft has no summaries, the same as without `gh`. - I did not change `release.yml`. The job already checks out `master`, so after this merges, re-running the failed job of run 42763789753 should push the missing notes branch with the fixed script. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) via Claude Code, with tool use (shell, file editing, test runs, read-only GitHub API calls). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
d2832a5a0b |
fix(runner): keep redacted display events inside their schema bounds (#20607)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent runs execute through Paperclip Runner. Its daemon, runnerd,
stores provider events durably before the server receives them
> - Before storing rich display events, such as tool output and provider
notices, runnerd redacts credential-shaped values. It then checks the
result against the event's protocol schema
> - Redaction can make a string longer. A field already at its schema
limit then goes over the limit, runnerd rejects the event, and the
daemon exits
> - The event is replayed after each restart, so every replacement
runner exits the same way. The run loops until something outside it
stops the run
> - This pull request trims a field that redaction pushed past its
schema limit, in both places that redact rich display events, so the
event is accepted and the runner keeps working
> - The benefit is that long tool output that contains a secret no
longer crashes the runner
## Linked Issues or Issue Description
**What happened?**
An agent run read GitHub Actions output and, after a context compaction,
entered a loop. About every 14 seconds runnerd started, restored its
durable PRP session, and exited with `paperclip-runnerd: redacted
display activity exceeds its schema bounds`. This continued for 15
minutes (62 launches) until a server restart ended the run. Every
reattach after that failed the same way, so the task was blocked.
**Expected behavior**
Redaction never makes a valid display event invalid. runnerd stores the
redacted event and continues.
**Steps to reproduce**
1. Produce a tool output longer than 64 KiB that contains a
credential-shaped pair, for example a GitHub Actions log line
`GITHUB_TOKEN: ***`. The ACPX sidecar keeps its last 64 KiB, which is
exactly the `toolExecution.output` limit of 65,536 characters.
2. runnerd redacts the pair to `GITHUB_TOKEN: [REDACTED]`, which is
longer, so `output` now has more than 65,536 characters.
3. `preserve_bounded_display_content` returns `redacted display activity
exceeds its schema bounds`, and runnerd exits.
A probe test with 64 KiB of output that contains `GITHUB_TOKEN: ***`,
`token=abc`, `Authorization: Bearer x`, or `password: hunter2` gets
exactly this error on `master`.
**Paperclip version or commit**
`master` at
|
||
|
Devin FoleyandPaperclip
|
4f8dd0e25a |
fix(runner): stop relaunching a runner that keeps exiting after restore (#20608)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent runs execute through Paperclip Runner. The server-side
transport supervises the runnerd process and restarts it when it exits
unexpectedly
> - That recovery keeps launching replacements until one restores the
durable PRP session, within a grace window
> - Each successful restore starts a fresh grace window. So a runner
that restores and then exits again a few seconds later loops with no
limit
> - In a real run, a deterministic runnerd crash looped this way 62
times in 15 minutes, with no error recorded, until a server restart
ended it
> - This pull request fails the transport after 3 restored runners each
exit within a minute of their restore. The error carries the runner's
last diagnostic
> - The benefit is that a crashing runner fails the run in under a
minute, with the reason recorded, instead of looping silently
## Linked Issues or Issue Description
**What happened?**
After runnerd exited unexpectedly, `#recoverRunnerProcess` launched a
replacement. The replacement authenticated ("runner process restored its
durable PRP session") and then exited about 5 seconds later with the
same diagnostic. `#watchRunner` then started a new recovery with a new
420-second grace window. This repeated every ~14 seconds for 15 minutes,
until the server shut down. The run produced no output and recorded no
failure during that time.
**Expected behavior**
A runner that keeps exiting right after each restore fails the transport
quickly. The failure names the runner's own diagnostic.
**Steps to reproduce**
1. Start a transport with `runnerReconnectGraceMs` set.
2. Kill runnerd. Recovery launches a replacement, which restores the
session.
3. Kill each restored replacement within a few seconds. On `master`,
recovery relaunches every time; the new test shows no failure after 3
restored runners exit.
**Paperclip version or commit**
`master` at
|
||
|
DottaandClaude
|
90dd864bff |
test(restart-chaos): wait for any visible Retry, not the first match (#20616)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart chaos harness checks that a task never needs a human
Retry after a control-plane restart, and that every Retry the board
shows works
> - When a task rests with a Retry offer, the harness opens the task
page and clicks the Retry button like a user
> - The harness waited for the first `Retry` button in document order to
become visible
> - A page can hold an earlier `Retry` button that is in the
accessibility tree but has no box. The harness then waited on that
button for the full timeout and reported `ui_retry_button_missing`,
although the user saw a Retry in the status banner
> - This pull request makes the harness wait for any visible Retry
button, banner first
> - The benefit is that a chaos grade reports a missing Retry only when
the user cannot see one
## Linked Issues or Issue Description
**What happened?**
A Daytona chaos wave graded one task `ui_retry_button_missing`. The
captured status banner text was "Couldn't finish this attempt. Your
message is saved. Retry", so the page showed a Retry button. The harness
waited on `getByRole("button", { name: "Retry", exact: true }).first()`.
That locator selects the first match in document order. When that match
is not visible, the wait times out even when another Retry is visible.
The click loop after the wait already used only visible buttons, but the
wait itself did not.
**Expected behavior**
The harness reports `ui_retry_button_missing` only when no visible Retry
button exists before the timeout. When a visible Retry exists, the
harness clicks it, banner first.
**Steps to reproduce**
1. Serve a page with a zero-size `Retry` button before a visible status
banner that holds a second `Retry` button.
2. Call `BoardUi.clickRetry` for that page.
3. Before this change, the result is `ui_retry_button_missing`. After
it, the harness clicks the banner Retry and sees the retry request.
Separate finding, not changed here: in that wave the Retry was also a
dead end on the server side. The run had failed with
`native_session_cleanup_quarantined`. The server build that ran the wave
still projected `canRetry: true` for that run, and the retry route
refuses that code with 409. Current master already removes that Retry
offer (#20572).
## What Changed
- `tests/restart-chaos/board-ui.ts`: wait on the `Retry` role locator
filtered to visible elements (`filter({ visible: true })`), instead of
the first match in document order. After the wait, re-query the visible
buttons and click them banner first, as before.
- `tests/restart-chaos/board-ui.test.ts`: add a page with an earlier
zero-size `Retry` and a visible banner `Retry`. The test fails before
the change (`ui_retry_button_missing`) and passes after it.
## Verification
- `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts
tests/restart-chaos/board-ui.test.ts
tests/restart-chaos/triggers.test.ts`: 43 passed. With the old
`board-ui.ts`, the new test fails with `ui_retry_button_missing`.
- `pnpm exec tsc --noEmit -p tests/restart-chaos`: passes.
## Risks
- Low risk. The change is test-harness code only. No product code
changes.
- The timeout and the "missing" result do not change. The locator
`visible` filter needs Playwright 1.51 or later; the repo uses 1.62.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
28063474d2 |
fix(ui): show why a queued message edit was not saved (#20583)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - In task chat, a person can edit a queued message before the agent receives it > - The server rejects an edit with 409 when the queue changed in another session > - The composer kept the text, but it showed no reason, so the edit looked stuck > - This pull request shows the reason under the editor > - The benefit is that the person knows the edit was not saved and what to do next ## Linked Issues or Issue Description **What happened** Two sessions edited the same queued message. The older save got `409 queued_comment_revision_conflict`. The composer still said "Editing queued message" and showed no error. **Expected behavior** The composer tells the person that the queue changed in another session, keeps the text, and lets them save again. **Steps to reproduce** 1. Queue a message on a task with a running agent. 2. Open the task in two tabs and start editing the queued message in both. 3. Save in the second tab, then save in the first tab. 4. See no error in the first tab. ## What Changed - `TaskChatComposer` shows a save failure for a queued message edit in its existing `role="alert"` line (`data-testid="task-chat-queued-edit-error"`). The error clears when the edit ends or a new save starts. - `api/errors.ts` has copy for `queued_comment_revision_conflict`, `queued_comment_stale_queue`, `queued_comment_stale_target` and `queued_comment_not_pending`. - The existing queue refresh after a 409 updates the revision, so a second save works. ## Verification - Red then green: `cd ui && npx vitest run src/components/task-chat/TaskChatComposer.test.tsx` (new test `shows why a concurrent queue change rejected the save and keeps the edit` fails on `master`; all 146 pass with the change). - `npx vitest run src/components/TaskChatThread.test.tsx src/api/errors.test.ts` (479 passed). - `pnpm check:token-gates`: all gates clean. - `tsc --noEmit -p ui`: no errors outside unrelated Storybook files. ## Risks - Low risk. UI copy only, no new API. A stale edit now shows a short error line in addition to the "Queued message changed" label. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude
|
070d6f83fd |
perf(realtime): share permission reads among queued live events (#20624)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - A task page shows the output of a running agent as it is produced.
> - The server sends each output event to the browser on the company
WebSocket.
> - Before each send, the server checks permissions and redacts secrets.
Each event did its own sequential database reads.
> - When output arrives faster than those reads complete, the queue
grows. The text reaches the browser later and later, and the server then
closes the connection.
> - This pull request lets the events that are already in the queue
share one set of reads.
> - The benefit is that live output keeps pace with the run, and a
revoked permission still applies to the next event.
## Linked Issues or Issue Description
**What happened?**
On a hosted instance, the live output of a running task sometimes
appeared very slowly. A probe WebSocket on the company event stream
received events 0.5 seconds after their creation time at the start.
After 30 seconds the delay was 23 seconds. The server delivered about 6
events each second. Then it closed the socket with code 1013.
The delivery loop for a connection is serial. For each transcript event
it ran a membership read, a scope decision, a run lookup and a task read
decision. Native run events added a settings read and a secret registry
read. A burst of 200 events issued 1,401 SQL statements.
**Expected behavior**
Live output reaches the browser at the rate the run produces it. The
cost of the permission checks does not grow with the number of events in
a burst. A revoked permission stops delivery immediately.
**Steps to reproduce**
1. Use a deployment that authenticates users, with a database that has a
network round trip.
2. Start several agent runs that produce output at the same time.
3. Open the task page of one run and read the live output.
4. Compare the creation time of each `heartbeat.run.log` or
`heartbeat.run.event` message with its arrival time.
5. The delay grows continuously until the server closes the socket with
code 1013.
**Paperclip version or commit**
`master` at
|
||
|
DottaandClaude Opus 5.5
|
5f34f777ec |
fix(evals): grade 16 chat cases by saved state instead of reply keywords (#20600)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat-feature eval catalog checks that a chat agent does the work a person asks for > - In the latest full-cohort run on master, 16 cases failed although the saved work was correct > - Their graders matched words, one literal value, or one API route in free-form model output > - A grader that fails correct work hides real failures in the noise, and it can also pass wrong work that uses the expected word > - This pull request grades those cases by saved state, audit records, setup literals and rendered page structure, and keeps only narrow checks for false claims > - The benefit is that these cases fail only when the product or agent does the wrong thing ## Linked Issues or Issue Description Refs #18971 (related: company-secret question handoffs) **What happened** The full-cohort chat-eval run on master failed 16 cases where the retained evidence shows correct behaviour. Examples: `companies.rename-company` accepted only `company.updated`, but the supported branding route logs `company.branding_updated`. `recovery.post-stop-message` required a signup button label that matched a regex, and rejected a labelled email form with the button "Start writing free". `secrets.secret-proposal` flagged "the value is delivered when Morgan next runs" as a false delivery claim. Other graders required words such as "uncommitted", "telemetry", "administrator" or the app name that the question did not ask for. **Expected behavior** A grader accepts every supported route and wording that achieves the outcome. It fails when saved state is wrong, when a requested fact is wrong, or when the reply makes a claim that saved state contradicts. **Steps to reproduce** 1. Load the retained observation for one of the cases below from the full-cohort evidence. 2. Grade it with the old oracle. The keyword or route check fails although the saved state is correct. 3. Grade it with the new oracle. It passes. A broken variant (wrong saved state or a contradicting claim) still fails. ## What Changed - `activity.audit-change`: attribution is graded by structure. The reply names the recorded editor, names no other company agent (new setup literal `otherActors`), and states only real transitions. Sentence grammar is no longer matched. - `app-permissions.ask-before-action`: each continuation reply must exist and must link no app record other than the reviewed connection. It no longer must name the app. - `budgets-costs.spend-breakdown`: the reply must point at the record with unresolved usage (task link, identifier or run id) and must not give it a zero or free amount. The words "subscription" and "incomplete" are no longer required. - `cases.browse-cases`: an excluded case may be mentioned in prose. It must not be linked or listed as a result row. - `cases.handoff-same-task`: handoff instructions carry findings forward when they point at the saved `findings` document, reproduce it, or restate it. The premise (a primary Morgan run performs the handoff) is unchanged. - `cases.update-case-facts`: a saved browser value may add clauses that only rule out other platforms ("desktop Safari unaffected"). - `companies.agent-observability`: the reply no longer must name Telemetry and the run log. It must not claim that they are exported. - `companies.rename-company`: both supported name routes (`company.updated`, `company.branding_updated`) are accepted. - `connection-setup.choose-provider`: the answer must exist and must not claim a connected account. The unavailable method no longer must be named; the setup path stays graded by clicking the card or link. - `goals.align-existing-work`: a question card from the answering run counts as the phase-two response. Refusal words and "unchanged" words are no longer required; false change claims still fail. - `recovery.post-stop-message` (and `steering.interrupt-with-direction`, which shares the helper): a visible form with a labelled field and a submit control is a signup action, whatever the button says. - `routines.pause-owned`: the "paused"/"running" words are no longer required. The saved checkpoint proves both; a cancellation claim still fails. - `secrets.secret-proposal` (and `secrets.configure-company-secret`, which shares the helper): a sentence that says when delivery will happen is not a delivery claim. - `skills.explicit-restriction`: the reply no longer must say "administrator". Success claims and invented tool limitations still fail. - `workspaces.inspect-git`: the reply no longer must say "uncommitted". Unchanged HEAD and status prove it; commit or push claims still fail. - Every affected `oracleVersion` is bumped in code and in `chat-catalog-descriptors.json`. No scenario YAML changed. ## Verification - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts --testTimeout=180000 tests/runner-e2e/chat-catalog-domains tests/runner-e2e/chat-catalog-contract.test.ts tests/runner-e2e/chat-catalog-consistency.test.ts`: all pass. - `PAPERCLIP_CHAT_EVAL_RENDERED_ARTIFACT_CANARY=1` rendered-artifact canary with Chromium: pass, including new form cases. - `node --test tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs`: pass. `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`: unchanged pins. - `npx tsc --noEmit -p tests/runner-e2e`: no errors in changed files. - Offline calibration on retained evidence, for each case: the retained correct evidence passes, earlier passing evidence passes where it exists, and a broken variant fails. `spend-breakdown` retained evidence still fails, because its reply shows unresolved usage as `$0.00`. `handoff-same-task` retained evidence still fails on the kept primary-run premise. - Live paid runs: see the comment below. ## Risks - Low risk. The change is limited to eval graders and their unit tests. Product code does not change. - Each grader is more accepting of wording. Saved-state checks and the narrow false-claim checks remain, so a wrong outcome still fails. - `secrets.configure-company-secret` still fails its retained evidence. Its agent asked a two-question card, and the act supports only one-question cards. This pull request changes it only by the shared delivery-claim helper. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
89ec4e7614 |
feat(links): add cached GitHub pull request previews (#20593)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Task conversations contain links to pull requests and other work. > - A URL alone cannot tell the operator whether a PR is open, merged, or blocked by conflicts. > - Fetching every link at render time wastes API calls and can expose private data through a shared cache. > - This pull request adds a demand-driven links service and a GitHub PR adapter. > - Each viewer and company has a separate cache. Repeated links share one request and result. > - Operators get context when they hover or focus a link. Normal links stay useful when a provider is unavailable. ## Linked Issues or Issue Description **Subsystem affected** Task Markdown links, shared link metadata, and the server GitHub API boundary. **Problem or motivation** PR links have compact labels and neutral icons. They do not provide a rich hover card or a common status source across repeated mentions. A background resolver cannot borrow a person's GitHub connection because it has no viewer identity. **Proposed solution** Recognize supported links locally. Request metadata after a delayed hover or keyboard focus. Show a card at the first line of the link, aligned with its icon. Share the result across all mentions in the current viewer and company. Keep the exact original navigation URL. The first adapter supports GitHub pull requests. Open is green, merged is purple, and merge conflicts are orange. Draft, closed, unknown, loading and unavailable states remain quiet. **Alternatives considered** Eager page scans would spend API quota on links the operator does not inspect. Company-wide caching of personal GitHub results would cross the credential boundary. This change uses neither approach. **Roadmap alignment** ROADMAP.md was checked. This extends existing task links and the GitHub resolver. It adds no deployment adapter or new background worker. Searches of open and closed PRs for link previews and PR hover cards found no matching implementation. ## What Changed - Add host-owned link recognition and a normalized preview contract. The service selects an adapter by provider. - Add a board-only company read endpoint. It checks company access before cache reads and sends a private, no-store response. - Use the current viewer's eligible GitHub connection. A configured connection that cannot supply credentials fails closed. With no managed connection, reuse existing company credentials or public access. Instance credentials retain the public-repository check. - Reuse the GitHub external-object resolver for API transport and response parsing. Add title and mergeability to its normalized snapshot without changing its existing state contract. - Coalesce requests by company, viewer and canonical PR URL. Bound cached entries and concurrent work. Cache success for one minute and failures for a short period or the provider retry window. - Add an abortable request deadline. Timed-out work stays in the concurrency budget until it settles. No periodic worker or idle polling is added. - Add a shared browser query and anchored preview card. Repeated Markdown links observe the same result. There are no automatic retries, window-focus refreshes, toasts or connection prompts. - Add ten production-component Storybook examples and real pointer/keyboard browser checks. Document the API and verification route. ## Verification - Linux source checkout: 233 focused tests passed in nine suites. They cover recognition, fixed API destinations, provider states, access failures, credential selection, viewer/company isolation, request coalescing, expiry, cache bounds, concurrency bounds and mounted OpenAPI coverage. - A live public GitHub smoke returned this PR's correct title and open state. Two equivalent URLs shared one real network request. Personal connection access was not part of that smoke. - A controlled HTTP server stalls a real response body. The test proves cancellation while another lookup completes. - UI typecheck and the server layered typecheck passed. The UI production build passed. - Token gates and query policy gates passed. All 30 map/catalog support checks passed. The three new server test files also passed a focused TypeScript check. - All 27 focused browser checks passed on the reviewed source tree. They cover light/dark themes, narrow layouts, open/merged/conflict/draft/closed/loading/unavailable/failed states, fixed positioning during pointer movement, exact navigation and keyboard access. A real browser HTTP interception counts one deferred lookup for two equivalent PR links. - The first browser attempt lacked Chromium. A later combined build/test attempt exhausted the sandbox's 8 GiB memory limit and killed Storybook. Those attempts supply no pass evidence. Chromium is installed, and the final browser checks run with the build stopped. One initial Storybook dependency reload interrupted a streaming check. All eight existing inline-link cases passed on the warmed server, followed by a complete 27-case pass. - Synthetic Storybook pointer events did not exercise the native hover boundary. The assertions now run through real Playwright pointer and keyboard actions. The fixture stories retain the production components. - Review fixes preserve the Draft label when GitHub reports both draft and conflicts. They also preserve neutral link colors in dark mode. The 25 affected service checks pass, including the added combined-state regression. Browser checks compare unknown, draft and closed links with the muted theme color. - The first CI policy run rejected a literal retry opt-out. The code now uses the existing explicit zero-retry helper. A new test-spy type error was also corrected. These failures are fixed; their earlier runs remain failed. - `pnpm verify --explain` returned exit 2 because the browser test path selects broad CI. No full local suite is claimed. The final CI run supplies that coverage. - The previous head `437913c797963aa113bb82e2ae8b836969317ef9` passed CI and received a 5/5 Greptile score. All four review threads are resolved. Master then advanced. The current head `ae2a1c2adbd61a621cca9bdd160561952613df6e` resolves the source-manifest conflict against master `9b21dcbdda01296349f3cfe0a3a6891e79df6130`. Fresh CI and review are running. - The previous CI run found a lost class on decorated PR links and a stale SDK source manifest. Both are fixed. The existing CommentThread integration test and 26 focused link service tests pass. All 45 SDK source/privacy tests pass. All eight source-manifest helper tests pass. The manifest helper changed only source identities and added the two new shared files. Runtime, compiler, parser and library pins stay unchanged. - A later review found that expired timeouts could reuse a settled promise. The service now removes settled requests from its shared-request map and counts unfinished provider work separately. The new regression failed before the fix and passes after it. The concurrency-bound and real stalled-HTTP tests also pass. - UI typecheck and token gates pass after these fixes. The browser rerun needed Chromium in the new session. That first attempt did not exercise the UI. The complete 27-case browser rerun passed before the master merge with retries disabled. All 27 browser cases also passed on the merge revision with retries disabled. - After the master merge, all 84 focused link, OpenAPI and integration tests pass. All 55 SDK source/privacy/manifest tests pass. UI typecheck and token gates pass again. The source manifest retains master's reviewed runtime pins and includes both sets of source changes. GitHub confirms the PR can merge cleanly. The preflight helper rebuilt the shared package and plugin SDK; Cargo is unavailable locally. Final CI supplies Rust coverage. - The latest master update changed the reviewed report-lock policy but left the SDK lock hash at its previous value. The SDK source/evidence checks reproduced the mismatch. This change sets only the SDK lock hash to the authenticated output of the existing master policy and patch. Compiler, parser and Zod pins stay unchanged. All 56 SDK source/evidence/manifest tests now pass. All 84 focused link/API/integration tests, UI typecheck and token gates also pass on the updated source. ## Risks - This cache is process-local. Separate server replicas can make separate GitHub requests. - A cached result can remain visible for its short lifetime after GitHub changes. A later hover refreshes expired data. - GitHub can report unknown mergeability. The UI shows conflicts only when GitHub explicitly reports them. - Personal preview data stays in viewer/company caches. It is not written to company external-object records or persisted browser query storage. - Stories simulate the API. They do not prove a live GitHub login or private-repository access. - No schema migration is required. ## Model Used OpenAI Codex, based on GPT-6. The runtime does not expose the exact serving model ID or context-window limit. Used reasoning, repository inspection, source editing, shell execution and browser tools. No subagents were used. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
161ee3b9a2 |
fix(runner-e2e): refresh SDK public input pins after lockfile refresh (#20618)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner support suite checks the SDK public source pins in `tests/runner-e2e/source-sdk-public-inputs.json` > - The automated lockfile refresh #20615 changed `pnpm-lock.yaml` and `tests/runner-e2e/report-lock.json` > - It did not update the two lockfile values in the SDK pin registry > - So `ci / Runner support suite` and `ci / verify` fail on every PR and on master > - This pull request refreshes those two values > - The benefit is that PR CI is green again without a maintainer override ## Linked Issues or Issue Description Refs #20615. Refs #20550 (the same kind of resync). **What happened?** After #20615, the Runner support suite fails in `source-sdk-public-inputs.test.mjs` ("source-sdk-public-inputs.json is stale for 1 pinned input(s): pnpm-lock.yaml (changed)"). It also fails in `source-sdk-evidence.test.ts` and `source-sdk-public-type.test.ts` ("Reviewed SDK reference policy does not match public proof"). **What did you expect?** The pin registry matches the committed lockfile and the reviewed report-lock policy, and the suite passes. **Steps to reproduce** On master `3bf7af6c77`, run `node tests/runner-e2e/source-sdk-public-inputs.mjs $(git rev-parse HEAD) --check`. It exits 1 with `"changed": true`. ## What Changed - `inputs["pnpm-lock.yaml"]` is now the git blob of master's lockfile (`6d825f97…`). I generated it with `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --write`. - `lockSha256` is now `b3208306…`. This is the `outputSha256` in `tests/runner-e2e/report-lock.json` after #20615, and the sha256 of the committed lockfile. `source-sdk-reviewed-fixture.ts` requires these two values to be equal. - No other values change. ## Verification - `node tests/runner-e2e/source-sdk-public-inputs.mjs $(git rev-parse HEAD) --check` exits 0 with `"changed": false`. - `shasum -a 256 pnpm-lock.yaml` equals `report-lock.json` `outputSha256` and the new `lockSha256`. - CI `ci / Runner support suite` on this PR runs the three failing test files. ## Risks Low risk. The change is test pin data only. It does not change product code, the lockfile, or the report-lock policy. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
42d4499489 |
fix(daytona): force a stuck stop before starting a retained sandbox (#20609)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - On Daytona, a follow-up task resumes the agent's retained sandbox before the agent starts > - The host stops an idle sandbox gracefully. A process that ignores SIGTERM can leave Daytona in `stopping` for many minutes > - Resume waited for that state change until its 300 s start deadline. It never escalated, so each attempt failed after five minutes, and the bounded retries did the same > - On staging, a user comment waited about 21 minutes before the agent started. Two resume attempts each used 299.7 s in the plugin dispatch > - This pull request forces a stuck stop once, then starts the sandbox within the same deadline. It also traces the wait with the observed state > - The benefit is that a stuck stop costs about 30 s instead of failing the run, and the next case is diagnosable from the trace ## Linked Issues or Issue Description **What happened?** A user commented on a task that used a retained Daytona sandbox. The first run and its first retry each failed after about 5 min 20 s with `reusable_sandbox_resume_failed (cause=provider_error)`. The Honeycomb trace shows `execution.plugin_dispatch` at 299.7 s in both. Inside it there is only `lease.lookup` (44 ms) and `lease.channel_cleanup`, and no `lease.start`. So the whole budget went to `waitOutSandboxTransition`, which polled a sandbox that stayed in a transition state. The later attempt, after a control-plane restart, found the sandbox `stopped`. Its `lease.start` took 1.3 s. A transition that ends in `stopped` is a stop. **Expected behavior** A sandbox that stays `stopping` for longer than the host's own graceful-stop budget is forced to stop, then starts. The run does not fail on the start deadline. **Steps to reproduce** 1. Retain a Daytona sandbox whose graceful stop does not finish (Daytona reports `stopping`). 2. Resume or realize the lease. 3. Before this change, the call polls until the 300 s deadline and fails with "is still stopping at its start deadline". ## What Changed - `packages/plugins/sandbox-providers/daytona/src/plugin.ts`: `waitOutSandboxTransition` tracks how long the sandbox stays `stopping`. After `STUCK_STOPPING_FORCE_AFTER_MS` (30 s) it calls `sandbox.stop(seconds, true)` once, then keeps polling. A refused force stop keeps the old wait within the same deadline. - The wait is a `lease.wait_transition` span. The force stop is a `lease.force_stop` child span. - `lease-tracing.ts`: the closed `paperclip.sandbox.startup.state` label also records `starting`, `stopping`, `restoring`, `archiving` and `resizing`. Before, these were all `other`. - `server/src/services/plugin-host-services.ts`: the host admits the two new span names and the new closed state values. - `doc/observability.md`: documents the new spans and states. ## Verification - Red/green: with the plugin change stashed, the three new stuck-`stopping` tests (realize, refused force stop, resume) fail by hanging to the deadline. With the change they pass. The "settles on its own" test confirms that no force stop happens for a normal stop. - `vitest run` in `packages/plugins/sandbox-providers/daytona`: 14 files, 618 tests passed (8 skipped). `tsc --noEmit` passed. - `server/src/__tests__/plugin-host-services-span.test.ts`: red before the allowlist change, 25/25 after. - `pnpm verify` exited 2 (the doc is a global CI input), so CI supplies the broad coverage. - Not verified on a live Daytona sandbox. Hosted acceptance needs the change deployed to staging. ## Risks - The force stop sends SIGKILL to processes in a sandbox that is already stopping and is about to start. The disk persists across a stop, so the retained workspace is kept. A graceful stop that is still running past 30 s is killed. The host's own graceful stop budget is 30 s, so the host had already given up on it. - Daytona can refuse a stop while a state change is in progress. Then the code waits as before, so the result is no worse than today. - The root cause is inferred from the trace (a transition that ends in `stopped`). The new state label confirms or refutes it the next time it happens. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, browser, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
9b21dcbdda |
refactor: remove SPEC docs and repoint spec citations (#20565)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Contributors and agents read the docs under `doc/` before they change code > - `doc/SPEC.md` and `doc/SPEC-implementation.md` were out of date. `doc/SPEC-implementation.md` became an append-only log: in 60 days it got about 1,645 new lines and lost 181, across 140 commits > - #20557 changed `AGENTS.md` so the code and its tests are the source of truth, and the specs are only background > - This pull request removes the two spec docs. It moves only the sections that live code comments cite into focused docs, and it repoints all other references > - The benefit is one less stale document that agents read and append to ## Linked Issues or Issue Description Refs #20557 (merged). This PR is the follow-up that removes the docs. It is based on `master`, which already contains #20557. **Issue type** Outdated content. The docs are out of date and conflict with the code. **Where is the issue?** `doc/SPEC.md`, `doc/SPEC-implementation.md`, and the code comments and docs that cite them. **What's wrong?** `doc/SPEC.md` and `doc/SPEC-implementation.md` are out of date, but agents still read them and append a paragraph with each fix. **Suggested fix** Delete both files. Move the sections that code cites into focused docs. Repoint the references. ## What Changed - Deleted `doc/SPEC.md` and `doc/SPEC-implementation.md`. - Moved these spec sections, because live code comments cite them: - §9.3.1 "Shared default-open issue writes" → new section in `doc/agent-permission-defaults.md`. - §9.8.1 "Issue-thread interaction resolver contract" → `doc/execution-semantics.md`, "Issue-thread interaction resolution". That section already had most of the contract. I added only the missing parts: the `supersedeOnUserComment` default, the canonical policy list, the addressee and cap rules, the rule that surfaces show the server's effective audience, and the cross-issue budget charge. The short policy summary at the top of the section now points to the list, so the definitions occur one time only. - §9.9 "Task Watchdog Authority Contract" → new "Authority contract" section in `doc/TASK-WATCHDOG.md`. It holds the allowed and disallowed mutation lists and the deferred capabilities. The atomic recovery batch, restoration verification, and interaction rules were already in `doc/execution-semantics.md` §11, so the new section links there and does not repeat them. I did not move the "Downstream acceptance criteria" checklist or the sentence about the CTO and Security sign-off. These were implementation-time notes; the tests now hold that coverage. - §9.10 "Company Skill Policy Contract" → new `doc/company-skill-policy.md`. No focused doc had this contract. I did not move the "Required regression tests" phase checklist. - Did not move any other part of the specs. - Repointed code comments that cited spec sections: - `packages/shared/src/issue-write-denial.ts` (§9.3) → `doc/agent-permission-defaults.md` - `server/src/routes/issues.ts` (§9.9) → `doc/TASK-WATCHDOG.md`, "Authority contract" - `server/src/routes/issues.ts` (§9.3, §9.8.1 cross-issue budget) → `doc/agent-permission-defaults.md` - `ui/src/lib/interaction-audience.ts` and `ui/src/components/InteractionGovernancePanel.tsx` (§9.8.1) → `doc/execution-semantics.md` - `ui/src/lib/skill-policy-denial.ts`, `ui/src/pages/CompanySkills.tsx`, `ui/src/pages/CompanySkills.production.tsx`, `ui/src/pages/SkillStudio.tsx` (§9.10) → `doc/company-skill-policy.md` - `server/src/services/recovery/ownership-invariant-report.ts` (monitor deadline wait path) → `doc/execution-semantics.md` §8. That section already lists the one-shot monitor as a valid wait path. - Resynced the git blob pin for `packages/shared/src/issue-write-denial.ts` in `tests/runner-e2e/source-sdk-public-inputs.json`. The comment edit changed the file's hash, and the Runner support suite checks that pin. I used `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --write`; only that one pin changed. - Updated `AGENTS.md` §1 so it does not name the deleted files. - Removed the links to the deleted files from `doc/plugins/PLUGIN_SPEC.md`. - Updated the `doc-maintenance` agent skill (`.agents/skills/doc-maintenance/`) so it no longer audits or stages `doc/SPEC.md`. - Removed or replaced references in `maps/README.md`, `maps/feature-map/README.md`, `maps/storybook-map/README.md`, `doc/PRODUCT.md`, `doc/CLIPHUB.md`, `doc/DEPLOYMENT-MODES.md`, `doc/DEVELOPING.md`, `doc/evals.md`, `doc/spec/agent-runs.md`, `doc/AGENTCOMPANIES_SPEC_INVENTORY.md`, `doc/connections/CONNECTOR-PLAYBOOK.md`, `doc/plugins/ideas-from-opencode.md`, `doc/execution-semantics.md`, `doc/TASK-WATCHDOG.md`, and `.devin/wiki.json`. - In `doc/execution-semantics.md`, the "admission is independent of cleanup and accounting" reference now states the rule inline: only a hard stop, an explicit pause, or an inactive company refuses admission. - Kept these references as they are: dated plans in `doc/plans/` and `docs/plans/`, the dated log `doc/logs/2026-05-24-cli-api-parity-e2e-log.md`, `releases/`, the transcript fixture `ui/src/fixtures/runTranscriptFixtures.ts` (it uses the filename only as sample text), and `docs/specs/external-task-protocol.md` (it refers to the Symphony project's own `SPEC.md`). ## Verification - `git grep -n "SPEC-implementation\|SPEC\.md"` (without `releases/`, and ignoring `PLUGIN_SPEC.md` citations) finds only the kept references listed above. - `grep -rn "§9\.\(3\|8\|9\|10\)" ui/src server/src packages/shared/src` finds no results. - `node scripts/check-maps.mjs`: "Maps aligned: 35 areas, 192 capability IDs". - `pnpm check:token-gates`: "All gates clean". - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/source-sdk-public-type.test.ts tests/runner-e2e/source-sdk-evidence.test.ts`: 45 passed. These failed in CI before the pin resync. - `pnpm verify`: exit 2 ("Broader verification required") because `AGENTS.md` is a global CI input. Required CI supplies the broad coverage for this revision. - All code changes are comments only. No runtime behavior changes. The title uses `refactor:` because comments in source files changed. ## Risks - Low risk. The change removes docs and edits comments only. - A reader can lose context from spec sections that I did not move. The text stays in git history. The code and its tests are the source of truth. - External links to the two deleted files will return 404. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), in Claude Code with tool use (shell, file edits). Standard context window. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (focused checks above; CI supplies broad coverage) - [x] I have added or updated tests where applicable (not applicable: docs and comments only) - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
3bf7af6c77 |
perf(plugins): read the lifecycle inbox from per-resource cursors (#20576)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Plugins get agent, project and agent runtime changes from the `resource_lifecycle_events` journal. Each plugin acknowledges events one resource at a time. > - #20405 made the inbox page read linear in the company journal size. The journal has no retention, so each page still reads every event of the company to find the unacknowledged ones: about 1.7 s per 1M events. > - The delivery worker reads a page again after each acknowledgment. The idle-sleep check also reads the whole journal when all events are acknowledged. `acknowledge` checks every older event of the resource. > - On a hosted instance, the page read ran 181 times in one window at about 1 s each. > - This pull request adds a per-plugin, per-resource cursor. Reads start at the cursor, so they do not read acknowledged history again. > - The benefit is that page, idle-check and acknowledgment cost follow the number of resources and pending events, not the journal size. Delivery order and results stay the same. ## Linked Issues or Issue Description Refs #20405 **What happened?** The plugin lifecycle inbox page query (`select ... from "resource_lifecycle_events" ... DISTINCT ON ...`) ran 181 times at about 1 s each on a hosted database in one window. Each acknowledgment causes another full read of the company journal. The idle-sleep check (`hasPending`) reads the full journal when all events are acknowledged. `acknowledge` reads the full history of the resource. **Expected behavior** A page read, the idle-sleep check and an acknowledgment must not read acknowledged history again. Their cost must not grow with the journal size. **Steps to reproduce** 1. Give a company 400 agents with 2,500 `agent_runtime` events each (1M events). 2. Acknowledge the first 2,000 events of each agent for one plugin. 3. Call `pluginLifecycleInbox(db, pluginId).list(companyId, 100, undefined, ["agent", "project", "agent_runtime"])`. 4. The read takes about 1.7 s. `hasPending` takes about 240 ms. **Paperclip version or commit** `8d2bf78323` (master). **Deployment mode** Hosted (PostgreSQL). Local repro uses embedded PostgreSQL 18. ## What Changed - `packages/db/src/schema/plugin_lifecycle_resource_cursors.ts`: new table `plugin_lifecycle_resource_cursors`. It has one row per plugin, company, resource type and resource id. `acknowledged_through_id` means: every event of that resource at or below this id is acknowledged by the plugin. Plugin and company deletion cascade. - Migration `0462_plugin_lifecycle_resource_cursors`: creates the table and carries existing acknowledgments forward (see Risks for the previous-release story). - `server/src/services/plugin-lifecycle-inbox.ts`: - `list` and `hasPending` use a backward loose index scan on `resource_lifecycle_events_resource_order_idx`. It visits each resource stream once and gets its latest event id. A stream whose latest event is at or below its cursor is settled and is not read again. For other streams, the read gets one head: the unacknowledged creation, else the first unacknowledged transition above the cursor. Then it applies `afterId`, id order and the limit, as before. - `acknowledge` reads only the events between the cursor and the acknowledged event. It still writes the per-event `plugin_lifecycle_acknowledgments` row. It moves the cursor only over a contiguous acknowledged prefix. A creation recorded after a pending transition does not move the cursor. - An event counts as acknowledged when it is at or below the cursor, or when it has an acknowledgment row. Thus rows written by an earlier release, and gaps in old history, keep their current behavior. - `server/src/__tests__/resource-lifecycle-events.test.ts`: new database tests for the migration carry-forward, gaps, late creations, cursor reads, the rollback rows and the plugin-delete cascade. All tests from #20405 still pass without change. - `scripts/benchmark-plugin-lifecycle-inbox.ts`: runs the migration backfill on the fixture and times it. Adds an acknowledgment case. Runs `VACUUM ANALYZE`, because hosted tables are autovacuumed. - `doc/DATABASE.md`: describes the cursor, its read cost and the migration. - The delivery worker (`plugin-lifecycle-delivery.ts`) is not changed. ## Verification Thesis: the cost is a read of the whole company journal (page, idle check) or the whole resource history (acknowledgment). If a per-resource cursor marks the acknowledged prefix, each read can skip that prefix. Then the cost follows the number of resource streams and pending events. The rows must stay equal. Local SQL timings: embedded PostgreSQL 18.1 on Apple Silicon, synthetic data, warm cache, one warmup, 7 samples, baseline and treatment order alternate per sample. The machine was shared with other jobs (load average about 35), so spread is wide. Values are median (min to max) in ms. Rows are equal (`assert.deepEqual`) for every list and probe case. Fixture A: 400 agents x 2,500 runtime events, 2,000 acknowledged per agent. 1,020,440 events, 800,420 acknowledgments. | Case | Before | After | Speedup | | --- | --- | --- | --- | | page of 100, all types | 1,700 (1,633-2,438) | 11.5 (9.5-12.2) | 148x | | page of 100, agent and project | 3.3 (2.5-5.4) | 4.2 (3.6-6.7) | 0.8x (slower) | | idle-sleep check | 238 (174-367) | 2.7 (2.1-3.4) | 88x | | acknowledge one head | 15.3 (9.4-30.1) | 20.3 (12.9-29.9) | neutral (noise) | | migration backfill | - | 9.3 s | - | Fixture B: 40 agents x 25,000 runtime events, 24,000 acknowledged per agent (long history). 1,020,080 events. | Case | Before | After | Speedup | | --- | --- | --- | --- | | page of 100, all types | 1,943 (1,642-4,979) | 4.4 (2.9-5.6) | 444x | | page of 100, agent and project | 3.5 (2.1-5.7) | 4.6 (2.8-12.0) | 0.8x (slower) | | idle-sleep check | 234 (220-243) | 2.8 (1.9-6.5) | 85x | | acknowledge one head | 64.6 (48.5-71.6) | 5.0 (2.6-7.1) | 13x | | migration backfill | - | 4.6 s | - | Fixture C: 5,000 agents x 200 runtime events, 190 acknowledged per agent (many resources). 1,025,040 events. | Case | Before | After | Speedup | | --- | --- | --- | --- | | page of 100, all types | 1,867 (1,649-1,975) | 76.9 (66.2-497) | 24x | | page of 100, agent and project | 10.3 (8.7-12.4) | 21.6 (20.2-23.2) | 0.5x (slower) | | idle-sleep check | 220 (201-298) | 2.8 (2.0-5.3) | 78x | | acknowledge one head | 4.9 (3.4-8.6) | 6.0 (5.2-7.6) | neutral | | migration backfill | - | 15.6 s | - | `EXPLAIN (ANALYZE, BUFFERS)` for the all-types page on fixture A: before 2,488 ms, 13,597 shared hits plus temp files; after 7.0 ms, 7,926 shared hits. Commands: ```sh pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-plugin-lifecycle-inbox.ts --baseline-ref=8d2bf78323 --agents=400 --per-agent=2500 --acked=2000 --samples=7 pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-plugin-lifecycle-inbox.ts --baseline-ref=8d2bf78323 --agents=40 --per-agent=25000 --acked=24000 --samples=7 pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-plugin-lifecycle-inbox.ts --baseline-ref=8d2bf78323 --agents=5000 --per-agent=200 --acked=190 --samples=7 cd server && npx vitest run src/__tests__/resource-lifecycle-events.test.ts src/__tests__/plugin-lifecycle-delivery.test.ts src/__tests__/project-instance-secrets.test.ts src/__tests__/agent-runtime-hooks.test.ts pnpm --filter @paperclipai/db check:migrations ``` - `resource-lifecycle-events.test.ts`, `plugin-lifecycle-delivery.test.ts` and `project-instance-secrets.test.ts`: 86 passed. `agent-runtime-hooks.test.ts`: 22 passed. - `tsc --noEmit -p server`: pass. `check:migrations`: pass. ## Risks - **Schema change.** Migration `0462` adds one table and two foreign keys to `plugins` and `companies`. It adds no index on an existing table. The backfill reads the acknowledgment table once. It took 4.6 s to 15.6 s on 1M-event fixtures with 0.8M to 1M acknowledgments. - **Migration order.** The number `0462` is reserved after open PRs that use `0459` to `0461` (#20571, #20568, #20573). Merge this PR after them. Before merge, rebase onto master and chain the snapshot after the latest migration. - **Previous-release story.** - Upgrade: the backfill stores, per plugin and resource, the end of the contiguous acknowledged prefix. A cursor never passes an unacknowledged event, so no event is lost. Acknowledged events above a gap keep their rows, so no event is delivered again. A test runs the migration statement on rows that only an earlier release wrote and checks that `list` returns the same rows before and after. - Rolling deploy: an earlier release that still runs writes only acknowledgment rows. The new reads honor those rows. - Rollback: the new code still writes every acknowledgment row, so an earlier release sees the same progress. - **Ordering assumption.** The cursor is safe because writers record a resource's events while they hold that resource's row lock (`paperclip_record_agent_runtime` locks the agent row; agent status and project writers document the lock). Thus a later commit for the same resource always has a higher id. A writer that breaks this contract could commit an event below the cursor, and that event would not be delivered. A lower id from a different resource is still delivered (existing test). - **Slower path.** A page that asks only for `agent` and `project` streams is 1 to 11 ms slower. These streams are short, so the old full read was cheap, and the new read visits each stream once. The cost is bounded by the number of resources. - **Many resources.** The page cost grows with the number of resource streams of the requested types. At 5,000 agents a page is about 77 ms, compared with about 1.9 s before. - The hosted planner can choose a different plan. The hosted improvement is not yet measured. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, local PostgreSQL benchmarks). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Nicky LeachandPaperclip
|
19e938415a |
fix(runtime): omit Codex refresh handoffs from backups (#20614)
## Thinking Path > - Managed ACPX Codex retains its final refresh after provider exit so Core can save it. > - Shutdown saves a checkpoint before Core removes the live credential handoff. > - Checkpoint profiles must exclude the handoff while retaining resumable session history. ## Linked Issues or Issue Description Follow-up to #20580 and its credential-backup review finding. Refs #20457. ## What Changed - Exclude `auth-refresh.json` and `auth-refresh.json.tmp` from the shared Codex checkpoint profile, covering native and ACPX homes. - Extend real checkpoint-copy coverage to ACPX Codex, both handoff filenames, and native Codex. Keep the live files and session history intact. ## Verification - A real ACPX checkpoint-copy test failed before the exclusion fix; Grok passed. - Five targeted real-copy/profile tests pass, including after applying the change to current master. - `git diff --check` and local secret/PII diff review pass. - Full regression coverage runs in GitHub CI at the maintainer's request. No live provider credentials were used. ## Risks - Two entries added to an existing exclusion list; no runtime lifecycle, schema, persisted identity, or session format changes. Saved sessions remain readable. Existing backup restoration uses the same exclusion profile. - Provider credential cleanup still follows final writeback; excluding a backup entry does not delete the live handoff. ## Model Used - OpenAI GPT-6 through Codex, with tool-assisted implementation and verification. The runtime does not expose a more specific model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
86cc3e3a11 |
fix(workspaces): save remote workspaces that contain outside symlinks (#20569)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents on a remote sandbox (for example Daytona) save their workspace after each run, and recovery retries a failed save > - The save refused the whole workspace when one symlink was absolute or pointed out of the workspace > - Ordinary agent work makes such links: a headless Chromium profile links `SingletonSocket` by absolute path into `/tmp` > - Recovery then spent its full budget on the same refusal, handed the hold to the board, and the task stayed blocked > - This pull request keeps confined links, records and omits outside links, and retries saves that an older policy exhausted > - The benefit is that these saves succeed and blocked tasks continue without a person ## Linked Issues or Issue Description **What happened?** On staging, a native run (OpenCode on `paperclip_runner`, Daytona sandbox) built Storybook and drove Chromium with `--user-data-dir=<workspace>/.review-chrome`. Chromium wrote `.review-chrome/SingletonSocket -> /tmp/.org.chromium.Chromium.*/SingletonSocket`. The save failed three times with `remote_workspace_archive_unsafe_symlink` (`native_workspace_sync_out_retry_exhausted`). System recovery then tried six more times. It recorded each failure as `export_interrupted` with cause `provider_error`, became `exhausted`, and gave the hold to the board. The task and its parent stayed blocked. **Expected behavior** The save succeeds for an ordinary workspace that contains symlinks. Links inside the workspace stay links. Absolute and escaping links are dropped and recorded, and are never followed. Holds that an earlier release exhausted recover automatically after deploy. **Steps to reproduce** 1. Create a workspace with `node_modules/.bin/tool -> ../tool/cli.js`, an absolute link to a file outside the workspace, and a link `escape -> ../outside/file`. 2. Capture it with `captureRemoteWorkspaceArchive` (or the serialized sandbox helper). 3. Before this change, the capture fails with `remote_workspace_archive_unsafe_symlink`. ## What Changed - `remote-workspace-snapshot-archive.ts`: capture records an absolute or escaping symlink in the manifest `excluded` list and omits it, instead of failing the save. It never follows a link. Confined links are kept. Restore validation is unchanged and still refuses an archive that names an outside link. - Added `REMOTE_WORKSPACE_CAPTURE_POLICY_VERSION = 2`. Each new system recovery generation records `capturePolicyVersion`. Generations without the field count as version 1. - New `workspace-recovery-capture-policy.ts`: an exhausted save whose last failure was in `preserve_files` (`export_interrupted`) under an older capture policy gets one new bounded generation. Restore refusals (`archive_unsafe`) are not re-armed, because restore validation is unchanged. It shares the run-over guard (`sourceRunProvablyOver`) with the stop-proof re-arm. It writes a `capturePolicyRecovery` marker and `issue.workspace_recovery_rearmed` activity with reason `capture_policy_changed`. The new generation has the current version, so the rule cannot repeat for it. - The recovery worker discovers these holds (`capturePolicyRearmPendingCondition` in `nextWorkspaceRecoveryAt` and the obligations query). - Docs: `doc/native-workspace-finalization-recovery.md` (symlink policy) and `doc/execution-semantics.md` (capture-policy re-arm). Previous-release story: archives saved by the previous release never contain outside links, and restore of them is unchanged. Recovery state saved by the previous release has no `capturePolicyVersion`; it is read as version 1 and, if exhausted in `preserve_files`, is re-armed once on the first worker pass after deploy. No migration is needed. A capture already in flight at the cutover uses the helper that its operation installed; the next attempt installs the new helper. ## Verification Red before the fix (same tests, unfixed code): - `remote-workspace-snapshot-archive.test.ts` "keeps confined links and drops outside links" (local and serialized helper): failed with `remote_workspace_archive_unsafe_symlink`. - `remote-workspace-transport.test.ts` "saves a workspace with outside links through the sandbox helper" (split and Daytona one-shot output): failed with `remote_workspace_archive_unsafe_symlink`. - `workspace-recovery-capture-policy.test.ts` "retries once under the new policy": failed, because the worker did not discover the exhausted hold (`nextWorkspaceRecoveryAt` returned null). Green after the fix: - `npx vitest run --root server src/services/remote-workspace-snapshot-archive.test.ts src/services/remote-workspace-transport.test.ts src/services/workspace-recovery-capture-policy.test.ts`: 271 passed. - The recovery test uses the exact staging evidence shape (6 attempts, `export_interrupted`, `provider_error`, board owner, no policy version). It ends `complete` and `recovered`, saves a snapshot, and leaves the source untouched. - Safety cases: the archive does not contain the outside bytes; restore refuses a hand-made archive with an escaping link; a hold exhausted under the current policy is not re-armed and no sandbox resume is attempted. - `npx tsc --noEmit -p server/tsconfig.json`: clean. - `pnpm verify`: see CI for the broad selection. ## Risks - A tracked symlink that is absolute or escaping is now omitted. After restore it shows as a deletion in `git status`. Before, the whole save failed. - On deploy, every hold exhausted in `preserve_files` with `export_interrupted` gets one more bounded recovery generation, including holds that failed for provider reasons. This is one generation per hold per policy version. - The manifest exclusion label stays `paperclip.remote-workspace-exclusions.v1`. Workspaces that the old policy accepted produce the same manifest, so unchanged-snapshot reuse still works. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
8e4c5e81b9 |
perf(issues): read created-from tasks by index and inbox review waits by page (#20573)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The task page shows "Created from this task". The inbox shows blocked-inbox attention for a page of tasks. > - On a hosted database, the created-from list ran about 4,000 times in one afternoon at about 499 ms each. The task page sends it each time a task opens. > - Its filter was an `OR` of three provenance paths over the list's own rows. PostgreSQL could not use an index for it. It read every task of the company and tested each one, with the list's other per-row conditions (such as the chat conversation state subquery). > - The blocked-inbox attention map also read external review waits for every in-review task of the company, not only for the tasks of the page. > - This pull request reads created-from tasks from three index probes, and reads review waits only for the in-review tasks that can change a finding of the page. > - The benefit is less database work on each task open and each inbox page. The responses stay the same. ## Linked Issues or Issue Description Refs #20400 (shares review-wait reads between the attention maps of one request). Refs #20551 (authorizes the review waits of a page with one read; this change narrows the waits that the page asks for, and adds a test that pins the batched read). Refs #13241 (adds the created-from list). **What happened** - `GET /api/companies/:id/issues?createdFromIssueId=…` used `createdFromIssueCondition`. Its filter was `origin_issue_id = source OR (origin_issue_id IS NULL AND origin_run_id IN source runs) OR (… AND EXISTS creation activity of a source run)`. The `OR` stops index use, so the statement read every task of the company. The list's other conditions, which include the chat conversation state subquery, ran for each task before the task was excluded. - `listIssueBlockedInboxAttentionMap` read review waits for every in-review task of the company graph. The caller uses the findings of the page rows only. **Expected behavior** The created-from list reads only the created tasks. The blocked-inbox map reads review waits only for in-review tasks that the liveness classifier can read for a page row. The rows, the order, and the attention values do not change. **Steps to reproduce** 1. Use a company with tens of thousands of tasks, runs, and activity rows. 2. Open a task page. Look at the created-from list statement: it reads the full task set of the company. 3. Open an inbox page with blocked-inbox attention. Look at the review-wait read: it gets the ID of every in-review task of the company. ## What Changed - `server/src/services/issue-creation-origin.ts`: `createdFromIssueCondition` is now `issues.id <> source AND issues.id IN (three branches)`. Each branch starts from an index: - recorded origin task: `issues_company_origin_issue_idx`; - legacy origin run: the source task's runs (`heartbeat_runs_company_origin_issue_idx`), then the new partial index on `issues (company_id, origin_run_id)`; - creation activity: `activity_log_run_id_idx` for the source runs, then the task by primary key. The precedence does not change: a recorded origin task wins, the run path applies only without one, and the activity path applies only without both. The activity path matches only an `entity_id` that is the canonical text of the task ID, as before (it also compares `id::text = entity_id`). - `packages/db`: add `issues_company_legacy_origin_run_idx` on `issues (company_id, origin_run_id) WHERE origin_issue_id IS NULL AND origin_run_id IS NOT NULL`. Migration `0461` creates it with `CREATE INDEX CONCURRENTLY IF NOT EXISTS` and is registered with the other concurrent migrations. `doc/DATABASE.md` documents the index and the benchmark command. - `server/src/services/issues.ts`: add `blockedInboxReviewWaitIssueIds`. It returns the in-review tasks in the blocker closure of the page rows. The blocked-inbox map reads review waits only for these tasks. The liveness classifier reads a wait only for a row or for a blocker that it reaches from a row, and the caller reads only the findings of the rows. The classifier does not change. - Tests: - `issue-created-from-routes.test.ts`: new cases for precedence in both directions, a source-run comment, an activity row whose task has another origin run, a non-canonical and a non-UUID activity entity ID, the source's own creation activity, and other-company rows. - `blocked-inbox-review-wait-scope.test.ts`: the narrowed set for chains, cycles, shared blockers and off-page tasks. The rows get the same findings with the narrowed waits as with the waits of every in-review task, and each narrowed wait is needed. - `issue-list-review-wait-authorization.integration.test.ts`: on real PostgreSQL with two authorized waits, `listExternalReviewWaits` and one issue-list request with blocked-inbox attention each send one bound-run message-link statement. With the per-wait authorization from before #20551, both tests fail (2 statements). - `scripts/benchmark-created-from-issues.ts`: local benchmark. It loads the baseline `issues.ts` and `issue-creation-origin.ts` from a pinned commit, asserts identical complete row payloads, and reports alternating timings and `EXPLAIN (ANALYZE, BUFFERS)`. ## Verification - `pnpm --filter @paperclipai/server exec vitest run src/__tests__/issue-created-from-routes.test.ts src/__tests__/blocked-inbox-review-wait-scope.test.ts src/__tests__/issue-list-review-wait-authorization.integration.test.ts src/__tests__/agent-task-hierarchy-depth.test.ts src/__tests__/issue-blocker-attention.test.ts src/__tests__/issue-list-assignee-filter-routes.test.ts src/__tests__/issues-parent-id-alias.test.ts src/__tests__/batched-chat-authorization.integration.test.ts src/__tests__/issue-liveness.test.ts` passes (126 tests). One timing test in `issue-list-assignee-filter-routes` failed one time under machine load and passed on rerun. - `pnpm --filter @paperclipai/server typecheck` and `pnpm --filter @paperclipai/db check:migrations` pass. - `pnpm verify` asks for broader verification because the change adds a file under `scripts/`. Required CI supplies that coverage. - Benchmark: `pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-created-from-issues.ts --baseline-ref=bd30ef8510` Fixture (synthetic, local PostgreSQL 18.1): one company with 30,000 tasks (10,628 with a recorded origin task, 10,445 with only a legacy origin run), 60,030 runs with wide context snapshots (145 MB), and 261,079 activity rows; a second company with 10,000 tasks and 20,000 runs. The busy source task has 63 created tasks through all three paths, plus decoys. The quiet source task has 3. The UI's page: `sortField=id`, `asc`, `limit=500`, routine executions included. Both versions return identical complete payloads. 9 alternating pairs after 2 warm-up pairs: | Case | List statement before | List statement after | List call before, median (min to max) | List call after, median (min to max) | | --- | --- | --- | --- | --- | | Busy source (63 tasks) | 234.9 ms, 47,437 buffers | 0.90 ms, 437 buffers | 156.0 ms (114.5 to 480.7) | 31.6 ms (19.9 to 62.7) | | Quiet source (3 tasks) | 90.7 ms, 47,413 buffers | 0.25 ms, 46 buffers | 126.1 ms (90.6 to 300.4) | 10.5 ms (7.2 to 26.1) | The list call time includes the enrichment reads (labels, runs, attention), which do not change. Inbox page (50 open tasks, blocked-inbox attention): the review-wait read goes from 2 statements with 1,215 parameters (every in-review task of the company) to 1 statement with 31 parameters. The page time does not change in this fixture (median 801 ms before, 873 ms after, both with a wide spread), because the fixture has no review waits and the other graph reads of the map are most of the time. With real waits, the server does an inherited GitHub review read for each wait it finds, so the gain grows with the number of off-page waits. Migration timing (local PostgreSQL 18.1): `0461` (measured as the same statement under its earlier number) on 500,000 tasks (846 MB, 20 companies, one third with a legacy origin run) takes 1.6 s. The index is 12 MB. ## Risks - The new index adds a small write cost to tasks with only an origin run. New tasks with a source task record `origin_issue_id`, so the index holds mostly older tasks. The migration does not lock writes. - The created-from rewrite must keep the same rows. The new tests and the benchmark payload check cover precedence, company scope, and ID text forms. Visibility and read filters are the list's other conditions and do not change. - The review-wait narrowing depends on the liveness classifier reading waits only for rows and their blocker chains. A unit test compares the findings of the rows with full and narrowed waits. A future classifier change that reads waits of other tasks must update `blockedInboxReviewWaitIssueIds`. The function comment says this. - The local timings are synthetic. Hosted acceptance needs a fresh query-statistics window after deployment. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell, file edits, local PostgreSQL) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1110.0-canary.16 |
||
|
DottaandClaude Opus 5.5
|
bce79f186f |
fix(evals): count a pending question card as the turn's response where the case allows it (#20597)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat catalog evals check that a chat agent answers the person and saves the right records > - An agent can end its turn on a question card; the product then records `interaction_owns_conversation_response` and posts no reply comment > - The privileged settle step and some graders still waited for a reply comment, so valid card-ended turns timed out or failed > - This pull request adds one shared rule for when a pending card counts as the turn's response, and uses it where the case allows a card > - The benefit is that card-ended turns are graded on what the person saw, while cases that need a written reply stay strict ## Linked Issues or Issue Description Refs #20588 (request_human_input version fix found by the same runs). Refs #20590 (chat review cards link their document). **What happened?** In the full chat eval run on master `f4337bb5fb`, agents ended turns on pending question or review cards. `developer-labs.component-reference` timed out in the privileged settle step, which required a reply comment. `goals.align-existing-work` refused the foreign goal in a question card, but the grader read only reply comments. `questions-and-approvals.reject-and-revise` returned its draft in a review card bound to the draft revision, but the grader and the result check required a reply link. **Expected behavior** A pending card counts as the turn's response only when the product decided that the card owns the run's response, the lead's own run created it, a person can answer it, and it is not a governed approval. Each case still grades the content it requires. **Steps to reproduce** Regrade the retained evidence of the cases above with the old graders: they fail on the missing reply comment although the card is on the chat. ## What Changed - New `chat-catalog-domains/turn-response-card.ts`: `turnResponseCards()` returns pending cards from runs whose `presentationDecision.reasonCodes` contains `interaction_owns_conversation_response`. A card must be created by the lead, be human-answerable (a text question, two or more choices, or an ordinary confirmation) and not carry `toolAction`, `secretProposal` or `connectionAuthorization`. - `settlePrivilegedChat` treats such a card as a settled turn. The privileged graders still require their written guidance, so `developer-labs.component-reference` now fails at grading instead of timing out. - `goals.align-existing-work` reads the card from the run woken by the foreign-goal request as that phase's answer. A leaked foreign text, or a card that does not refuse, still fails. Oracle `paperclip.chat-catalog.internal.goals-align.v2`. - `questions-and-approvals.reject-and-revise` accepts a review card bound to the draft's exact saved revision. The result check opens the draft from the card's document link (needs #20590). Oracle `paperclip.chat-catalog.plan-review.v6` for both plan-review cases. - Descriptors updated for oracle versions and `sourceFiles`. ## Verification - `turn-response-card.test.ts`: the retained cards pass; other run, other agent, answered, other company, user-created, non-human resolver, single non-text choice, governed confirmation, and a run that replied all fail. - Offline regrade of retained evidence (master `f4337bb5fb` full-cohort run): `goals.align-existing-work` cross-company check passes with the card and fails without it; `reject-and-revise` `real-local-review-draft` passes with the bound card and fails with no card or a card bound to another revision. - `npx vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/chat-catalog-domains ...clock tests --testTimeout=180000`: 269 files passed (one file needed the plugin SDK build locally, then passed). - Consistency and contract tests pass. `npx tsc --noEmit -p tests/runner-e2e` is clean. SDK public input pins are unchanged. - Paid run [42855262851](https://github.com/paperclipai/paperclip/actions/runs/42855262851) on a temporary branch with this PR, #20588 and #20590 (OpenCode, local): `reject-and-revise` and `revoke-personal-grant` passed; `goals.align-existing-work` failed only `phase-bound-strategic-context-explanation` (keyword check, unchanged here); `developer-labs.component-reference` ended on a question card again and now failed at grading (`existing-question-reference`, `reference-only`) instead of timing out in settle. This run's agents replied in text in the two passing cases, so the card branch of `reject-and-revise` is covered by the offline regrade only. ## Risks - `settlePrivilegedChat` now stops waiting on a card-ended turn. Privileged cases whose graders need a reply fail sooner and with the real failed check; none pass more easily. - Changed oracle versions start new history for `goals.align-existing-work` and both plan-review cases. - The remaining keyword check `phase-bound-strategic-context-explanation` in `goals.align-existing-work` is unchanged and still fails the retained reply; it needs its own fix. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), agentic coding with tool use in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
ed3ffc3bc7 |
fix(ui): let chat review requests open the document they ask about (#20590)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - In chat, an agent can end its turn on a confirmation card that asks
the person to review a saved document
> - The full task view links the card's document, but the compact chat
card showed the document key as plain text unless it was the plan
> - A person who is asked to review a draft in chat could not open the
draft from the card
> - This pull request links a document on the same chat to its
`#document-<key>` panel, the same link a reply uses
> - The benefit is that a review request in chat lets the person read
what they are asked to approve
## Linked Issues or Issue Description
**What happened?**
In a chat eval run on master `f4337bb5fb`
(`questions-and-approvals.reject-and-revise`), the agent saved a `draft`
document and ended its turn on a confirmation card "Review research
notes introduction draft" with `target: { type: "issue_document", key:
"draft", revisionId }`. In the chat composer takeover, the card showed
`draft` as a plain chip with no link.
**Expected behavior**
A confirmation card in chat opens the document it asks the person to
review, like the full card in the task view does.
**Steps to reproduce**
Create a pending `request_confirmation` on a chat task with
`payload.target = { type: "issue_document", key: "draft", revisionId }`.
Open the chat. The takeover shows the `draft` chip without a link.
## What Changed
- `CompactTarget` in `TaskChatCompactInteractionCard.tsx` gives an
`issue_document` target on the same chat the href `#document-<key>`.
Plan targets, targets with an explicit `href`, attachments and documents
on other tasks keep their current behavior.
- Added a `TaskChatThread` test: a pending draft review request in the
takeover links `#document-draft`.
## Verification
- `npx vitest run src/components/TaskChatThread.test.tsx -t "links a
review request"` in `ui`: red before the fix, green after.
- `npx vitest run src/components/TaskChatThread.test.tsx
src/components/coordinator/CoordinatorTaskCommitment.test.tsx
src/lib/issue-thread-interactions.test.ts`: 457 passed.
- `node scripts/check-token-gates.mjs`: all gates clean.
- Paid chat eval on a temporary branch with this change: see the comment
on this PR.
## Risks
- Low risk. Plan review cards are unchanged; their plan preview already
opens the plan.
- No API, schema or stored state changes.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), agentic coding with
tool use in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
github-actions[bot]andlockfile-bot
|
9816d2d25c |
chore(lockfile): refresh pnpm lockfiles (#20615)
## Thinking Path > - Paperclip uses a dependency lock for reproducible builds. > - Dependency changes on master need matching lock entries. > - Paid eval reports also pin a reviewed offline correction. > - Reports must install before paid model work starts. > - This automated PR keeps the lock and report proof together. ## Linked Issues or Issue Description **What happened?** Dependency manifests on master changed after the last committed lock refresh. **Expected behavior** Frozen installs must match those manifests. Eval reports must retain matching offline lock proof. **Steps to reproduce** Run a frozen install before and after this lock refresh. ## What Changed - Refresh the dependency lock from the manifests on master. - Refresh the sandbox provider lockfiles from their manifests on master. - Include the matching offline report correction and hashes. ## Verification - Pinned pnpm resolution completed with lifecycle scripts disabled. - The report proof refresh accepted the reviewed committed baseline. - The workflow rejected changes outside the lock and report proof files. ## Risks The lockfile bot merges this PR without CI or review when every added package version is at least 3 days old on npm. Otherwise the PR waits for a maintainer, and the daily refresh tries again. The bot merges only the exact commit this workflow generated from manifests already reviewed on master. Unknown report lock baselines remain blocked. This update changes no permissions or paid enable gates. ## Model Used None — generated by pnpm and the deterministic report-lock updater. ## Checklist - [x] Thinking path and problem description included - [x] Generated lock and matching report proof included - [x] Release-age gate decides between bot merge and maintainer review Co-authored-by: lockfile-bot <lockfile-bot@users.noreply.github.com> |
||
|
DottaandPaperclip
|
12f70d0d4c |
Keep Slack form custody and receipt transfer durable (#19825)
## Thinking Path > - Paperclip manages work for AI-agent companies. > - Chat connections deliver provider messages and user answers. > - Slack requires a response within its original three-second deadline. > - Forms and their delivery receipts must survive retries and tenant restarts. > - This change uses the existing connector queue and durable form owners. > - Current authorization, exact receipt checks and normal ACK rules stay in force. > - The tenant can settle useful work while keeping the connection enabled. ## Linked Issues or Issue Description Refs #19019 and #19348. Companion Cloud PR: paperclipai/paperclip-cloud#1250. ## What Changed - Preserve form grants, sessions, encrypted evidence, terminal receipts and company-authorized Retry. - Check initial form enrollment before the provider send. Keep a typed preparation wait only while an exact current enrollment obligation exists. - Refund the attempt only when the claimed attempt and credential lease still match and provider invocation has not started. Possible or accepted sends retain uncertain-delivery custody. An ordinary error with the same text still uses the normal five-attempt limit. - Notify the existing publication worker when normal enrollment commits. Reconstruct pending work from durable state after a lost reply or owner restart. - Page settled history before proving idle safety. Keep unfinished, malformed and foreign receipts visible beyond the first 1,000 rows. - Keep Slack callback, answer, receipt and ACK rules in their provider owners. Support body-bound Cloud idle capabilities when a complete catalog exceeds the header budget. - Add focused regression coverage, an enabled quiet/reactivation recipe and explicit fixture port checks. - Keep SDK evidence tests tied to a normally built reviewed reference, with a separate strict refusal for actual unreviewed CI outputs. ## Verification App head: `6284d6881b5dc38a9ee149bcbde35e69a67dd8ca`, including actual master `dff8bed007e01c32f3826144b893ec80603d1335`. Companion Cloud head: `f83b38187dc7e6b2e219fe865dc16d268cc66fdb`, including actual Cloud master `b1dcc3421a91efffee631a61536cf48f357c22fc`. Cloud #1250 is merged at `1d07d81f23d4627ab7b6dcf49176b4128a2f8f9b` after exact-head passing checks and complete Greptile 5/5. Current exact-head CI typecheck, build, tests and aggregate verification are green; the complete edited Greptile review is 5/5 and all 13 review threads and nested comment pages are resolved. - Both latest enrollment findings were reproduced against the preceding product. The pure readiness control returned no wait for the first enrollment. The real-PostgreSQL control observed a provider call while that enrollment was still staged. - The repaired source passes three whole files: 21 readiness controls, 80 real-PostgreSQL milestone controls and 37 tenant-actor middleware controls. All 138 pass with zero excluded or TODO cases. The SQL case reconstructs the worker through seven waits, preserves the attempt budget, recovers a controlled lost enrollment reply, and commits one provider-port call and one exact grant. Replaying the selected attempt does not duplicate them. Authorization, SDK and enrollment-response ports are controlled in that SQL case. - The same files cover ordinary five-attempt exhaustion, accepted and possible sends, stale ownership, malformed bindings, attention states and current actor restrictions. One synthetic mismatched-binding parameter uses generation 2 against current generation 1; it is not evidence for an older-generation control. - The normal server-package typecheck passes at the measured source. It runs the normal Runner TypeScript, protocol, SDK and declaration prerequisites and all seven main App programs. The separate declaration configuration is also recorded. A worker union-narrowing error was found, fixed with an explicit callable/null branch, and verified by that normal compiler. No cast or test relaxation was used. - The normal merge of current master preserves all 78 owned paths and 15,783 unrelated master entries and modes. The three measured test files and their changed feature paths are unchanged. New upstream DB code adds only the normal nonunique concurrent native-session index and its migration classification. Shared and adapter-utils changes alter unrelated prompt or denial wording. The 138-case proof carries for those unchanged behaviors; it is not reported as a rerun after the merge. The normal Shared/SDK ensure, DB build, adapter-utils build and complete server-package check also pass at the current merged head. Their fresh inventory contains 8,143 outputs; canonical SQL and journal copies match. This build did not apply the new index migration. - Earlier real-PostgreSQL retained-history coverage reproduced the missing proof with 1,005 settled rows. Its repair retains the existing two-second/64-query inspection budget, current endpoint bound and exact owner snapshot. The 46 source-bound controls cover unfinished and invalid tails, exact cross-page action references, Retry, rollback and lost receipt responses. - Earlier enabled quiet/reactivation coverage produced ordinary useful work, kept configuration enabled, accepted fresh signed work plus a duplicate, and produced exactly one second useful reply. Its two App-worker windows lasted at least 30 seconds with zero measured PG writes, connection or outbound activity after settlement. That evidence stays tied to its measured revision. A missing worker EXIT marker limits that earlier proof; parent evidence does not replace it. - Earlier three-second callback, terminal ACK, current-question race and complete body-size controls remain source-bound. The 500-owner/1,000-binding body measured 313,875 bytes with a 618-byte JWT; signature tampering failed closed. - Final managed-provider acceptance now passes all five fresh Node boots: A/A, A/AB, B/A rejection, B/AB recovery, B/B final. Two independently configured, challenged, verified, linked and enrolled Slack identities exercise ACK and retirement debt separately. Normal provider factory/materialization delivers the configuration through a controlled external stack-provider boundary. The registered opening returns HTTP 200 in 21 ms within the original 3-second deadline. Two answers persist, normal ACK is accepted, exact retirement and lost acceptance/GC replies recover, and the original canonical snapshot remains unchanged without duplicate provider sends or submissions. - The separate complete committed-preflight case passes. A committed `update_attempted` response is lost before SDK entry; normal service-owner reconstruction after the original lease/deadline recovers the same useful update and one answer without a second trigger. This reconstructs service owners within one App process; the separate five fresh Node boots cover process restart and managed signing-key rotation. - Assertion-time PG socket reports balance 15 opens/15 closes across the five boots and 5/5 in preflight, peak 2 and zero live sockets or unexpected attempts. Worker terminal EXIT markers were missing and are not replaced by parent EXIT markers. These are domain/recovery and assertion-time connection proofs, not a new whole-worker terminal quiet measurement. All known owners/groups closed; independent SQL audits were empty; only successful fixture databases were removed. Earlier failed fixture databases/logs remain retained. No product guard, deadline or assertion was weakened. Exact runtime source and normal outputs are unchanged by the three test files in the current follow-up, so these source-bound proofs carry without repeating those journeys. - The preceding App CI ran 11,312 support cases: 11,289 passed, three SDK-evidence cases failed, and 20 external cases skipped. Normal CI dependency setup resolved six stale Slack patch-hash references and produced an unreviewed lock. The public SDK proof correctly refused that lock; the tests incorrectly assumed the CI output was a reviewed reference. The three-file test repair keeps the protected lock policy, source/compiler/parser/Zod pins, scanner and redaction rules unchanged. It builds an owned reviewed reference using the existing normal Shared/SDK producers for the positive seal test, while a separate negative requires actual unreviewed-build privacy refusal and no retained SDK bytes. It does not relabel the CI build as certified or admit a new lock hash. - All four whole SDK support files pass locally: public type 30, evidence 16, public-input synchronizer 8 and lock preparation 10, totaling 64 with zero excluded/TODO cases in 28.73 seconds. The genuine fresh reference build/positive seal and separate invalidated-reference privacy refusal passed within their original 120-second bounds. Original installed dependencies, SDK outputs/cache and runtime source stayed unchanged; normal global setup produced only its own Runner JavaScript outputs. PostgreSQL was denied throughout, all four worker markers were observed, and the owned process group/private test root closed. The local canonical-lock negative mutates only its own reference; fresh remote CI also passes the actual resolved-CI-lock branch: all 16 evidence and 30 public-type tests pass, including the genuine reviewed-reference positive and the strict actual-unreviewed-build refusal. The upstream synchronizer file runs its current 10 cases in CI. - The first complete CI attempt on this head also exposed two separate 30-second timeouts: one real reporter-artifact test and the `agent-test-environment` module-loading hook. Full original logs remain preserved. Their exact test, sanitizer and evidence source blobs are unchanged from the preceding head; inspection found no deterministic setup cause. After the original run finished, one retry of only the failed jobs passed on the same synthetic merge `168851e210671ebe97e7a778e8cf891807e80502`: Runner support reports 11,304 passes and 20 external skips; serialized shard 6 reports all 395 passes, including all 67 route bodies. No source, timeout or assertion changed. This proves recovery; the original causes remain unexplained. - The final pre-merge tree audit uses actual master `17136a6c25ef6cb7c816a29140a4c48c2c75fb31` (refreshed after the earlier `85dec046` audit). The expected normal squash preserves all 81 owned paths and 15,816 unrelated master entries and modes. Only OpenAPI has edits from both sides, and its complete content composition preserves the new project route and token help alongside this PR's API entries. The shared inbound selection changes preserve the same candidate set; new concurrent indexes, Native/workspace behavior and shutdown changes remain upstream behavior. The scoped form proof uses process agents and does not qualify the new Native/sandbox branches. The latest question-schema change still accepts the explicit version-1 payload exercised here. The last master delta also changes managed AI/Native credential refresh and writeback; those branches are outside the measured process-agent form paths. Existing 8,143 local outputs remain tied to their measured revision; no new whole-tree output equality or deployed suspension is claimed. - Reproduce the public path with `doc/testing/slack-cloud-qualification.md`, the companion Cloud checkout, Node 24/22 and fresh owned fixture databases. Aggregate local `pnpm -r typecheck`, `pnpm test:run` and `pnpm build` were not rerun as one command set. Current exact-head required CI and the complete Greptile 5/5 review now pass; the full edited summaries and all paginated threads were read. Local worker quiet does not prove deployed host or Neon suspension. Photon and Discord remain separate work; this PR completes the described Slack slice. ## Risks Use compatible App and Cloud readers before transferring custody. Signatures prove authenticity; current authorization and exact ownership still control execution. Preserve old targets, secrets and terminal facts until their normal owners release them. Keep the managed seal identity stable while custody remains. The required Slack adapter patch remains pinned. Upstream's concurrent index uses the normal top-level migration path. ## Model Used OpenAI `gpt-6.1-sol` implemented and tested the changes. A GPT-6-based Codex managing agent reviewed and integrated them. The host does not expose the managing agent's exact model ID or context-window sizes. The agents used code execution and test tools. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used, its version and tool capabilities - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have linked public related issues or PRs - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal Paperclip ticket id - [x] I have run the local tests listed above and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation - [x] I have considered and documented risks - [x] All Paperclip CI gates are green for the current head - [x] Greptile is 5/5 with no open findings for the current head - [x] I will address all reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
1805a1d247 |
perf(workspaces): probe workspace run bindings through indexes (#20568)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The terminal workspace sweep checks each Git workspace for recent activity and for live consumers before it expires or moves the workspace > - These checks find the runs of a workspace with one SQL scope, `expirationScope`. The scope used one `OR` over all run bindings, and one branch decoded the context JSON of each run > - PostgreSQL could not use an index for that `OR`. Each check read every run of the company and its large context from TOAST. On a hosted database, each call took 2 to 5 seconds, about 5,200 s in total in one window > - This pull request splits the scope into one index probe per binding and adds one concurrent expression index for the legacy `executionWorkspaceId` context key > - The benefit is that each check reads a few index entries instead of all company runs. The cleanup decisions do not change ## Linked Issues or Issue Description **What existing behavior does this improve?** The workspace activity and cleanup-safety checks: `gitWorkspaceLastActivityAt`, `isGitWorkspaceExpirationEligible`, `workspaceMoveEligible` and the orphan-folder consumer check. All of them use `expirationScope` in `server/src/services/git-workspace-expiration.ts`. **Subsystem affected** Execution workspace cleanup in the server (`git-workspace-expiration.ts`, `isolated-repository-rehome.ts`, `isolated-task-directory-retention.ts` through the shared scope) and the `heartbeat_runs` schema in `packages/db`. **Current behavior** The `relevant_runs` CTE selects company runs where any of these is true: `issue_id` or `native_issue_id` is in the issue tree or the bound issues, the run is a bound issue's checkout or execution run, or the context JSON has a matching `issueId`, `taskId` or `executionWorkspaceId`. The planner does one sequential scan of the company runs and reads each context from TOAST. The cost grows with the run history of the company, not with the size of the workspace. **Proposed behavior** Each binding is a separate index probe. The `UNION` of run IDs goes to a primary-key probe. The SQL gives the same rows. **Reason and benefit** Local SQL measurements show a 34x to 169x decrease in time. The decrease grows with the run history (see Verification). ## What Changed - `expirationScope` builds `relevant_runs` from a `UNION` of index probes, one for each binding. The key sets go in as arrays (`= ANY(ARRAY(...))`). With `IN (subquery)`, the planner chose a hash join over a full scan, because it does not count the cost of reading TOASTed contexts. - The legacy context keys use `context_snapshot ->> 'key'`. This expression matches the existing indexes `heartbeat_runs_company_ctx_issue_created_idx` and `heartbeat_runs_company_ctx_task_created_idx`. For a non-object context, `->>` gives `NULL`, as the strict JSONPath decode did. Such a run can still match through a native binding. - New partial index `heartbeat_runs_company_ctx_execution_workspace_idx` on `(company_id, context_snapshot ->> 'executionWorkspaceId') WHERE context_snapshot ? 'executionWorkspaceId'`. The query repeats the predicate. For an array context, `?` can be true, but `->>` gives `NULL`, so the result does not change. The index is partial so that it serves only this probe. A full index that starts with `company_id` made the planner choose it for an unrelated company-scoped query. Migration 0460 builds it with `CREATE INDEX CONCURRENTLY IF NOT EXISTS`. The migration is in the list of migrations that run outside a transaction. - New tests: a legacy `issueId` or `executionWorkspaceId` binding on a run that is bound to a different issue still blocks cleanup. A plan test with `enable_seqscan = off` checks that no binding scans `heartbeat_runs` and that the three context indexes are used. - The benchmark script `scripts/benchmark-workspace-activity.ts` gets a `--runs-per-workspace` option. Unrelated fixture runs now have distinct context IDs, as real history does. - `doc/DATABASE.md` describes the probes and the new index. ## Verification - `pnpm --filter @paperclipai/server exec vitest run src/__tests__/git-workspace-activity.test.ts src/__tests__/native-harness-backup-expiration.test.ts`: 132 of 132 tests pass. The second file checks that the native session index stays the planner's choice for its query. These include the existing route matrix (root, child, bound, native, context issue, context task, context workspace, checkout, execution run), exactly-once counts, malformed and uppercase contexts, company isolation, controller leases and lease vetoes. - `pnpm --filter @paperclipai/server typecheck` and `pnpm --filter @paperclipai/db check:migrations` pass. - Benchmark (local SQL measurement, PostgreSQL 18.1, synthetic fixture, 8 workspaces per sample, before and after alternate in each iteration). The script compares the complete ordered caller results and the complete `relevant_runs` multisets of the baseline and treatment. They are equal in every run. `pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-workspace-activity.ts --baseline-ref=origin/master --runs-per-workspace=<N>` | Fixture | Check | Before median (min–max) | After median (min–max) | Speedup | | --- | --- | --- | --- | --- | | 4,800 runs, 330 MB | last activity | 523 ms (460–598) | 12 ms (8–15) | 44x | | 4,800 runs, 330 MB | eligibility | 452 ms (432–473) | 13 ms (11–15) | 34x | | 4,800 runs, 330 MB | idle eligibility | 465 ms (442–573) | 12 ms (10–13) | 40x | | 24,000 runs, 1.7 GB | last activity | 2,240 ms (2,214–2,454) | 13 ms (11–14) | 169x | | 24,000 runs, 1.7 GB | eligibility | 2,245 ms (2,097–2,377) | 17 ms (14–19) | 133x | | 24,000 runs, 1.7 GB | idle eligibility | 2,196 ms (2,179–2,381) | 15 ms (13–18) | 143x | Before: one `Seq Scan` on `heartbeat_runs`. After: `heartbeat_runs_company_issue_created_idx`, `heartbeat_runs_company_native_issue_contract_id_uq`, the three context indexes and `heartbeat_runs_pkey`. - Index build time: on a synthetic table with 200,000 runs and 8 KiB contexts (1.8 GB), the full form of this index took 3.7 s to build concurrently and used 15 MB. The partial index reads the same rows and holds only runs that have the key. ## Risks - Cleanup safety: a wrong "allowed" result could delete an active workspace. The scope keeps every binding of the old query. The benchmark checks exact equality of the run sets, and the existing tests check each binding route and each veto. - Migration: the index build reads every run context once. It runs concurrently and does not block writes. Until migration 0460 builds the index, the new query is still correct but the `executionWorkspaceId` probe scans. - Plan choice: the array form makes index probes the cheapest plan for a small tree. A very large issue tree gives a large array, but each element is still one index probe. - The sweep frequency does not change. This pull request reduces only the cost of each check. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Nicky Leach
|
17136a6c25 | fix(runtime): persist managed Runner credential refreshes across execution targets (#20580) | ||
|
DottaandPaperclip
|
85dec04677 |
fix(native): validate active controller durability during eligibility (#20601)
## Thinking Path > - Paperclip controls agents while their work runs outside the web server. > - A web replacement must retain each controller's durable recovery state. > - Modern controllers commit that state to the database before sending work. > - Their local controller file is only an external-persistence identity marker. > - Disk eligibility still required recovery metadata that can arrive after a run finishes. > - This pull request verifies the complete active database authority instead. > - It also bounds concurrent source reads so a large history does not require serial network waits. ## Linked Issues or Issue Description Refs: #20311, #20584, #20599. **What happened?** A live native run can have a complete database controller checkpoint and no terminal session snapshot yet. The disk-preservation check treats its local identity marker as a local-only obligation. The check also resolves selected sources one at a time. **Expected behavior** Accept a complete, bound active recovery authority. Refuse missing, partial, stale or foreign evidence. Keep all execution and enrollment guards. **Steps to reproduce** Use the new active-controller regression. Start a real Runner controller with the database checkpoint writer. Keep the business run active and omit its terminal snapshot metadata. The previous check returns a local obligation even after the full checkpoint commits. ## What Changed - Require the exact external-persistence marker and a complete decoded encrypted checkpoint for active runs. Match run, agent, task, session, Runner and workspace identities. For warm reuse, also permit an older marker only when its full older checkpoint matches that marker and its exact earlier successful run has a committed accepted result in the same company, agent, task, session, Runner and workspace. Keep the current live writer mandatory. Read the marker again after validation. - Accept that proof only with its fenced live writer and task binding, or the existing exact saved recovery snapshot. The saved-snapshot path still requires the full active journal. - Keep newer remote workspace authority strict. A malformed reference cannot fall back to an older prepared reference. Conflicting plugin task authority remains blocked. - Resolve selected sources at most eight at a time for read-only eligibility. Join every started reader before reporting an error. Keep source order, pagination, predicates and all write paths unchanged. Offline migration and prestaging still resolve sources one at a time. - Add real PostgreSQL regressions for active durability, partial and foreign state, expired ownership, previous prepared-reference state, post-result state and reader failure joining. ## Verification - Canonical Node24 preflight passed in74.2s. It built the current Runner inputs and published the cache. No shared Cargo target or hosted database was used. - Genuine active regression RED: the previous check returned true when false was required. The real checkpoint writer had committed the complete authority and replaced the local file with the Runner marker. Treatment passed, including the refusal controls. - Genuine source-reader RED: sixteen retained pointers used one reader at a time. Treatment used at most eight and joined all eight when the first reader failed. With a controlled20ms reader delay and real PostgreSQL selection, the small fixture changed from497.5ms to85.3ms. This is a bounded-concurrency proof, not a production speedup measurement. - All54 cases in the affected controller migration integration file passed in76.07s, with no skips. These include existing failed-controller, archived-controller, inline database and previous-session preservation cases. - Server TypeScript passed. The normal merge of current master kept both changed files byte-for-byte identical to the54-case proof. Final post-merge focused checks passed two of two in14.83s (52 other cases intentionally unselected); server TypeScript and module boundaries passed. Comment-only correction743 removed all new code comments required by CONTRIBUTING. A later warm-reuse regression reproduced the remaining false blocker through the real Runner rotateRunIdentity API. The bounded treatment uses one exact older-run lookup and full old/current checkpoint validation; it adds no Runner format, release or lineage service. Final normal-merge head `c62d6eb1755cf99b6def9563ba69854dba17de42` includes the new0459 baseline. All three focused PostgreSQL cases passed in21.37s (52 existing cases intentionally unselected); fresh server TypeScript and module boundaries passed. The prior54-case regression evidence is retained; no full matrix was repeated. - `git diff --check` passed. Final-head CI passed:56 checks succeeded and4 were skipped; the separate Paperclip Review advisory was still queued. All required test, build, typecheck and ci/verify gates passed on c62. [Final required workflow](https://github.com/paperclipai/paperclip/actions/runs/42868279612). Fresh [Greptile review](https://github.com/paperclipai/paperclip/pull/20601#issuecomment-6774537057) is5/5 on exact head `c62d6eb1755cf99b6def9563ba69854dba17de42`, with no open review threads. ```sh node scripts/agent-preflight.mjs --skip-rust-debug node --import ./cli/node_modules/tsx/dist/loader.mjs packages/db/src/test-postgres-command.ts pnpm exec vitest run server/src/services/native-runtime/native-controller-migration.integration.test.ts --disableConsoleIntercept pnpm --filter @paperclipai/server exec tsc --noEmit ``` ## Risks - This reports preservation evidence. It does not grant attachment, execution, replacement or enrollment authority. The normal live-owner and admission gates still apply. - A partial old or current checkpoint, stale local controller state, identity mismatch or uncertain owner stays blocked. Warm-marker proof also rejects a foreign agent, newer source, later completion, rejected result, per-turn scope or expired current writer. No source, checkpoint, session, lease, archive or provider state is changed by the check. - The eight readers add bounded database concurrency during an explicit eligibility request. There is no new background scan, cache, row filter or timeout change. Live enumeration time still needs deployed measurement. - The plugin task-lease branch uses the producer's environment lease row ID, as native-session-executor and existing diskless adoption coverage require. A direct active-eligibility positive for that plugin branch is a coverage limit; existing mismatch/conflicting-authority checks remain. The advisory found no must-fix defect, and this does not change the current tested revision. - Saved formats and terminal migration behavior remain unchanged. Older prepared workspace references remain supported. A fully saved result can still pass before the run-status update, but its entire active checkpoint must decode. - The observed staging run finished naturally and became valid on the existing terminal path during investigation. That observation is not deployed verification of the new active path. ## Model Used OpenAI GPT-6 (Codex), with reasoning, tool use and code execution. This runtime does not expose a more specific model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
9009fbdedb |
fix(runner): accept request_human_input payload version forms models send (#20588)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents ask people for decisions with the runner's `request_human_input` tool, which creates a question or confirmation card > - The tool schema required `payload.version` to be the number 1 > - The server ignores the version the model sends and always stores version 1 > - Some models send `"version": "1"` as a string, so every call failed with "/payload/version must be equal to constant" > - This pull request accepts `1` or `"1"`, and lets the model omit the version > - The benefit is that the model's natural call creates the card on the first try, and a turn does not end with no card and no reply ## Linked Issues or Issue Description Refs #19405 (adds invocation examples for the same tool; it does not change validation). Refs #20134 (aligns the same tool's text limits). **What happened?** In a chat eval run on master `f4337bb5fb` (OpenCode runner, `deepseek-v4.1-flash`), the agent called `request_human_input` up to six times. Each call failed with `Invalid tool input: /payload/version must be equal to constant`. The agent's reasoning shows it sent `"version": "1"` and could not produce the number. In one case it then made the card with `call_api`. In two other cases the turn lost the card. This happened in `questions-and-approvals.reject-and-revise`, `goals.align-existing-work` and `assistant-connections.revoke-personal-grant`. **Expected behavior** A version field that the server overwrites must not block the call. The tool must still reject an unknown version and a questions card without a question form. **Steps to reproduce** Validate `{ interactionKind: "questions", payload: { version: "1", questionSet: {...} } }` against the live `request_human_input` input schema. Before this change, validation fails at `/payload/version`. ## What Changed - `payload.version` in the tool schema accepts `1` or `"1"` and is optional. Its description says Paperclip stores version 1. - A questions card no longer requires `payload.version`. It still requires `questionSet` or `questions`. - The payload description no longer tells the model to send `version:1`. - Regenerated the semantic tool contracts, the seeded eval catalog digest and the protocol manifest with the existing generator. - Added a regression test: number, string and missing version pass for live and scenario schemas; version `2` and a payload without questions fail. ## Verification - `npx vitest run src/protocol-actions/` in `packages/paperclip-runner`: red before the fix with the exact eval error (`/payload/version must be equal to constant`), green after (47 tests). - Focused runner tests for the tool bridge, semantic tools, mock control plane, codex semantic results, catalog and semantic tools: 231 passed. - `node scripts/generate-semantic-contracts.mjs --check`, `generate-semantic-action-catalog --check`, `generate-protocol-manifest --check`, `generate-replay-goldens --check`: current. - `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`: unchanged. - Paid chat eval on a temporary branch with this change: see the comment on this PR. ## Risks - Low risk. The server already writes `version: 1` into every interaction payload created by this tool (`paperclip-runner-tool-authority.ts`), so no stored shape changes. - Previous-release story: no persisted run, session or launch state changes. Saved tool receipts keep the input they recorded. A retry with a different version form under the same idempotency key is a different input, as before. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), agentic coding with tool use in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
1b7ec13acb |
perf(native): default remote sandbox runs to a warm, reusable sandbox for measured harnesses (#20048)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents can run in remote sandboxes (Daytona), where each turn needs
the task's workspace and a runner inside the sandbox
> - With a default environment, every remote turn got a brand-new
sandbox: the whole workspace was uploaded again and a new runner
started, even for a follow-up comment
> - In a paperclip-sized workspace that made a follow-up take 30 s to
first token, against 8–10 s with a reusable warm sandbox
> - This pull request makes an agent that never chose a lifecycle run
warm on a sandbox for the measured harnesses (Codex, OpenCode, Claude),
so a follow-up keeps its sandbox, runner and workspace
> - The default applies only where warm can work: a sandbox provider
that keeps reusable leases, and a managed AI account that supports a
warm session. Everywhere else the run is per-turn, exactly as before
> - The benefit is fast remote follow-ups by default, without
re-uploading a large repository every turn
## Linked Issues or Issue Description
**What happened?**
On Daytona, with a fresh paperclip clone as the project workspace, a
follow-up comment took 30 s to its first token. The run log showed
`environment: fresh/created` on every follow-up. It spent 13.3 s in
`workspace_sync.transfer`, a full upload into the new sandbox, and 8–9 s
in `runner.session.resume`.
**Steps to reproduce**
1. Create a Daytona environment with the defaults: lifecycle inherited,
lease reuse off.
2. Run a Codex task in a large repository, then post a follow-up.
3. The follow-up gets a new sandbox, uploads the whole workspace again
and starts a new runner.
**Expected behavior**
A follow-up on the same task reuses the live sandbox, its runner and its
workspace (plan rule 2: no full re-upload per turn).
## What Changed
- **One rule for the default.** `defaultsRemoteSandboxRunToWarm`
(`native-runtime/native-warm-capability.ts`) decides the remote-sandbox
warm default for both the lease (environment view) and the runner
(native session policy), in one vocabulary (runtime backend plus ACPX
agent). It is applied only when no lifecycle was chosen anywhere:
admitted session, environment `runnerLifecycleMode`, or agent
`lifecycleMode`. It holds only when:
1. the harness is measured (Codex, OpenCode, ACPX Claude);
2. the sandbox provider keeps reusable leases (Daytona, CreateOS and
boxd declare them; E2B, Modal and the built-in `fake` do not); and
3. with a managed AI account, that harness supports a managed warm
session (`supportsManagedNativeWarmSession`: Codex).
- **The harness resolves as the runtime resolves it.**
`resolveWarmProviderSelection` uses
`resolvePaperclipRunnerProviderProfile`, so an omitted `provider` is
Codex and ACPX without an agent is Claude, in both decisions. A config
the runtime would refuse gets no default.
- **Lease side** (`runner-environment-lifecycle.ts`,
`environment-run-orchestrator.ts`): for an eligible run only, the
orchestrator asks the provider before acquisition
(`providerSupportsReusableLeases` in `environment-runtime.ts`: the
acquisition rule, one registry lookup for a plugin provider, no
worker-ready wait). With support, the run-scoped view sets `reuseLease:
true` and forwards the warm idle window to the provider's auto-stop. The
shared environment is never rewritten. Every other run resolves exactly
as on master, without the probe.
- **Runner side** (`native-lifecycle.ts`, `runtime-selection.ts`): the
native policy applies the same rule to the acquired lease
(`reusableLeaseConfigured && effectiveCapabilities.reusableLeases`). If
acquisition downgrades the lease anyway, the run falls back to per-turn;
a default never produces
`runner_warm_environment_requires_reusable_lease`.
- **Idle window.** The remote default uses the window the environment
view forwards to the provider: an inheriting environment's
`runnerIdleTimeoutMs`, then the agent's `idleTimeoutMs`, then 5 minutes.
The sandbox execution target now carries the environment's window
(`runnerIdleTimeoutMs`, an optional field in `adapter-utils`'s execution
target) so the runner and the provider auto-stop agree. An explicit warm
choice and the local default are unchanged.
- Unchanged: explicit agent choices, environments that pin a lifecycle,
plugin-task environments, admitted sessions (`persistedPolicy`), and
local behavior.
- `tests/restart-chaos`: the vCPU budget check counts `--daytona-inherit
--lifecycle default` sandboxes as retained, because they now default to
warm.
- `doc/plans/2026-11-08-large-workspace-speed.md`: the measurements, the
ranked waits, and the gate.
## Verification
- Unit and integration tests, one per rule:
- Rule 1 (provider gate): `native-lifecycle.test.ts` (ephemeral lease →
per-turn), `runner-environment-lifecycle.test.ts` (no reusable leases →
environment unchanged, no warm window),
`environment-runtime-reusable-lease-probe.test.ts` (`fake`, an
undeclared plugin, a declared plugin whose worker does not verify the
reuse methods, and a plugin without a running worker read false),
`runtime-selection.test.ts` ("defaults an agent without a lifecycle on
...": reusable → warm/keep_running; reuse configured on an unsupported
provider → per_turn/destroy_after_turn; ephemeral → per_turn),
`environment-run-orchestrator.test.ts` (probe result decides
`reuseLease` and the forwarded window).
- Rule 2 (explicit configuration wins): explicit per-turn agent, pinned
environment, admitted session, configured `idleTimeoutMs`, managed
Claude per-turn, managed Codex warm; the orchestrator does not probe
when a lifecycle is explicit.
- Rule 3 (omitted provider): `{}` resolves to Codex in the environment
view; `runtime-selection.test.ts` "agrees with the environment view's
lease for ..." feeds the environment view's lease into selection for
Codex, a provider-less Codex agent, OpenCode, ACPX Claude, ACPX without
an agent, and Copilot (per-turn), and checks both decisions agree. A
plugin-task target stays out of the default.
- The two suites that failed on the previous head
(`runtime-selection.test.ts`,
`workspace-recovery-preparation-dispatch.test.ts`) pass locally.
Locally, 6 connector-skill cases in `runtime-selection.test.ts` fail
with `ENOENT .claude/skills/slack/SKILL.md` because this checkout is a
nested worktree; they do not touch this change and pass in CI.
- End-to-end check of this default on Daytona (before the gate was
added; Daytona keeps reusable leases, so the gate does not change this
path): a user-default environment (lifecycle inherited, lease reuse
unset, `--daytona-inherit`), a fresh paperclip clone, built from the
image's source revision with this change (branch measure/remote-warm).
| | Before (per-turn sandbox) | After (this change) |
|---|---|---|
| Follow-up, Codex | 30.4 s | 7.6 s |
| Follow-up, Claude | about 30 s | 11.0 s |
| First follow-up after an idle K1, Codex | 45.9 s | 20.3 s |
| First follow-up after an idle K1, Claude | 38.3 s | 18.3 s |
The run-log reuse records show the follow-ups reusing the workspace,
environment, runner and provider session, with no
`workspace_sync.transfer`. First turns still upload the workspace once
(14.8 s and 26.1 s). Every sandbox was cleaned up.
- Daytona restart chaos wave (K1/K2/K4 with a mid-run comment,
inheriting environment, agents with no lifecycle, runner image built
from this branch): wave `w-20261110T171438-s20048`, seed 20048, on
|
||
|
DottaandClaude
|
c37eacbee3 |
fix(native): Retry re-finalizes a saved result that ran out of finalization retries (#20578)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs save an accepted result. Then a finalizer applies that
result to the task.
> - Finalization has its own budget of 3 failed attempts (#20555). When
the budget runs out, the coordinator goes to `terminal_failure` with
`native_finalization_retry_exhausted`.
> - For a provider-succeeded run, the task then rests `in_review` with a
board recovery card. The task shows "resolved" and offers no Retry. The
user is stuck.
> - The ordinary Retry is also wrong here: it starts a new provider run,
which repeats work the agent already finished.
> - This pull request adds two things. First, contention failures on
such a result get two longer automatic delays before the board gets the
task. Second, when the state is still exhausted, the task offers Retry,
and that Retry applies the same saved result again.
> - The benefit: fewer tasks need a Retry, and when one does, Retry
works and does no duplicate provider work.
## Linked Issues or Issue Description
Refs #20555 (it names this gap under Risks).
**What happened?**
In restart-chaos wave `w-20261110T044338-s20385`, task t03 ended with a
succeeded run, an accepted result, and a coordinator in
`terminal_failure` / `native_finalization_retry_exhausted`. The issue
stayed `in_review` with a board-owned `active_run_watchdog` card and
`canRetry=false`. #20555 stopped the early exhaustion. A real exhaustion
still leaves the same stuck state. Coordinators that older builds
already left in this state are stuck the same way.
**Expected behavior**
Transient finalization failures recover by themselves when that is safe.
When they do not, the task offers Retry. That Retry re-finalizes the
saved result. It does not start a provider run and does not charge a
provider attempt.
**Steps to reproduce**
1. Let a native run save an accepted result and end `succeeded`.
2. Make each finalization of that result fail until the budget of 3 runs
out.
3. Before this fix, the task presentation is `resolved` with
`canRetry=false`, and the board card has no action that applies the
result. The new test
`server/src/__tests__/native-exhausted-finalization-retry.test.ts`
reproduces this.
**Paperclip version or commit**
master at
|
||
|
Nicky LeachandPaperclip
|
98ce93dd4e |
Choose project storage from compatible environments (#20587)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Projects supply repository files to agent runs through core staging or a storage plugin. > - A storage plugin's default claim silently attached new projects even when their agents used a different environment. > - An environment name or driver type does not establish shared storage access. The existing binding and task-driver contracts do. > - This pull request groups creation choices by that contract and selects storage from the creator's effective environment. > - The benefit is a visible, validated choice and a clear error before an incompatible managed workspace operation. ## Linked Issues or Issue Description Fixes #20459. Related: #20156 covers a separate provider-failure diagnostic. The PR search found no other fix for #20459. ROADMAP.md has no duplicate project-creation change. ## What Changed - Show **Available in** when configured environments require different project storage. Each option names the environments that can use the same storage owner. Hide the selector when all choices are compatible. - Derive compatibility from repository staging and managed binding ownership. Local, SSH, and sandbox environments use core staging. Managed storage requires a registered task driver from the same installed storage plugin. Separate storage plugins remain incompatible. Unknown plugin contracts and the fake sandbox are excluded. - Follow the creating agent's environment, or the person's primary agent, then the instance default and local fallback. Apply managed-only and forced Kubernetes policy. The product has no separate user or company environment-default field today. - Add a company-authorized discovery endpoint and an optional creation `environmentId`. Validate the choice again on save. Store the storage owner without overriding agent run environments. Preserve explicit API storage choices and existing project ownership. - Keep the hidden repository-free private-task project independent of environment availability. Submit the observed storage owner as `expectedStorageProviderPluginId` so a changed contract fails at save without setting a workspace policy. Instance task-isolation defaults stay inherited. - Reject an incompatible managed workspace before a provider operation. Name the storage and selected environment in the error. Add regression tests, API documentation, creation-story fixtures, and minimal corrections to obsolete map text. ## Verification - **102 tests passed across 7 files:** project environment integration, installed storage policy, repository persistence, project environment routes, project creation telemetry routes, OpenAPI, and NewProjectDialog. Run with `node node_modules/vitest/vitest.mjs run server/src/__tests__/project-environments.test.ts server/src/__tests__/project-workspace-provider.integration.test.ts server/src/__tests__/project-repositories-persistence.test.ts server/src/__tests__/project-routes-env.test.ts server/src/__tests__/project-goal-telemetry-routes.test.ts server/src/__tests__/openapi-routes.test.ts ui/src/components/NewProjectDialog.test.tsx --silent`. - The review fixes cover both archived-default private-task cases and changed-storage rejection. With both instance isolation flags enabled, a new core repository project keeps a null policy and resolves to an isolated Git worktree. The complete affected set passes all 102 tests. - The CI query-error policy failure was fixed by using the shared query view. The form and shared query-view suites pass **24 tests**, including cached choices during a transient refetch failure. `node scripts/check-query-error-rendering.mjs --enforce --summary` reports zero findings. - CI exposed stale SDK source hashes and heartbeat fixtures that paired plugin storage with Local. Refreshed exactly four shared-source hashes with the maintained generator (runtime pins unchanged). All **53 SDK privacy/source-proof tests**, including real built SDK output, pass. All **7 corrected managed-storage heartbeat scenarios** pass; the fixtures declare the owning plugin's task environment and substitute transport at the existing execution seam. - Shared and UI TypeScript checks pass. The UI production build passes. `pnpm check:token-gates`, `node scripts/check-maps.mjs`, `node scripts/check-module-boundaries.mjs`, and diff checks pass. - Tested UI states: compatible choices, incompatible groups, default within a group, manual override, create failure, loading, discovery failure and retry, and an unavailable default. Tests use the production form and existing design tokens. - The integration tests use real isolated PostgreSQL on macOS arm64 with Node 26.12.0. They cover user/agent defaults, storage persistence, policy restrictions, company and authentication denials, and rejection before a provider call. Discovery uses saved configuration and makes no worker or provider requests. - Local verification limits: `pnpm -r typecheck` and `pnpm build` stop in unchanged adapter-utils code because the installed ACPX types lack `inheritProcessEnv`. A direct server check also finds an unchanged AI SDK type-version mismatch. The existing heartbeat test `inherits the instance default environment across companies while preserving explicit agent overrides` times out at line 518 on both this change and unmodified master with the same dependencies. The full `pnpm test:run` attempt passed all six chat shards (1,948 tests), then the Git workspace expiration suite reported 14 failures. I stopped that broad run. Two sampled failures reproduce on unchanged master with the same `unknown_worktree_git_files` refusal; the other expiration failures were not independently compared. A later broad workspace-preparation run also hit workspace validation failures and was stopped; the seven corrected fixture scenarios pass independently. These limits do not establish a full local suite pass. Hosted CI remains required. - Initial missing-package and loopback-denial failures were resolved by repairing isolated dependency links, building the SDK, and running database tests with loopback access. No deployed or live-provider execution is claimed. ## Risks - Creation defaults now follow the effective environment. A plugin's `defaultForProjects` claim alone no longer selects storage. This is the intended behavior change. - Compatibility covers repository project storage. It does not guarantee arbitrary host paths, retained-workspace moves, adapter/image support, credentials, or worker readiness. Matching provider configs cannot prove a cross-plugin storage contract. - A person's primary agent supplies the personal default because there is no separate personal environment setting. An unavailable configured default requires a valid explicit choice or a configuration repair. - No migration runs. Existing projects keep their storage owner. This change does not detach existing managed data or provide a storage migration. - Full local verification has the limits listed above. CI and standard Greptile review are pending. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, tool use, and code execution. The exact serving snapshot and context-window size were not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
1c78d023c6 |
perf(native): narrow backup eligibility to selected runs (#20599)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Server replacement checks whether retained native sessions still depend on local harness data. > - This check joins selected task sessions, runs, backup stamps, and durable backup records. > - PostgreSQL can start from backup leases and scan an issue's full run history before applying the selected-run condition. > - A measured instance spent 9.56 seconds in that selector and exceeded an 8-second statement limit. > - This pull request selects the exact current run identities first, then applies the existing lease and backup checks. > - The same receipt validation and eligibility decisions remain in place. ## Linked Issues or Issue Description **What happened?** Startup eligibility timed out while selecting native harness backup obligations. The plan scanned historical runs for each lease before checking the task session's selected run. **Expected behavior** The selector should use the selected run identities without scanning unrelated run history. It must return the same session, scope, company, and receipt flag. **Steps to reproduce** Use a database with retained native task sessions, many historical runs on the same issues, and version-two backup stamps. Run startup eligibility with an 8-second statement timeout. The lease-first plan can exceed that limit. **Paperclip version or commit** The local baseline is `d0c30e69a9cea912db7c5290c5d585fdd63ec228`. **Deployment mode** Self-hosted server with PostgreSQL. ## What Changed - Keep `AS MATERIALIZED` as an intentional join-order fence: build selected task-session and last-run identities before joining backup leases. - Keep only narrow identity columns in that materialized set. Read the JSON receipt flag through the selected run's primary key after the lease and current-backup filters, so profiles for unrelated identities do not widen the intermediate rows. - Preserve company and agent bindings, exact normalized-session text comparison, current-backup exclusion, scope deduplication, ordering, and the 100-row page. - Add real PostgreSQL regressions for full ordered-row equivalence, nullable JSON receipt flags, malformed and uppercase session bindings, cross-company rows, duplicate stamps, page tails, and invalid receipts. - Existing saved state carries forward under the same validation rules. No schema or persisted identity changes are required. ## Verification - The two focused PostgreSQL regressions passed locally. Command: `pnpm exec vitest run server/src/services/native-runtime/native-harness-backup-obligations.integration.test.ts --silent`. - Worktree preflight passed. The focused regressions (2/2) and `node server/scripts/typecheck-layers.mjs` passed on final head `c739353b22` after rebase. - One bounded read-only pair on PostgreSQL 17.11 returned the same 54 complete ordered rows: baseline 9,564 ms, candidate 298 ms (32.1 times faster). The plan changed from lease-to-issue-history scans to selected-run primary-key lookups and a hash join. - A synthetic PostgreSQL 18.1 fixture already chose an efficient baseline plan. Across five alternating warm pairs, its median was 104.6 ms before and 109.4 ms after. This neutral control used 22,800 runs, 14,100 leases, 3,173 task sessions, and 5,460 backups; all 54 complete rows matched. The empty control also matched. - These measurements cover the selector. A later read-only assembled-call probe on a newer serving release substituted only this selector: the selector took 544 ms for 54 rows, while the full eligibility call took 56.1 seconds and still reported a controller migration blocker. Other checks remain outside this PR; this PR does not claim to bring the whole request within its deadline. - Required CI and fresh review results will be recorded here before handoff. ## Risks - Materialization adds a selected-identity pass. Databases that already choose an efficient plan may see little benefit, as the local neutral control shows. - Pagination and receipt validation must stay exact. The regression crosses the 100-row boundary and checks that an invalid final scope still blocks eligibility. ## Model Used OpenAI Codex, GPT-6 family. The exact model ID and context window are unavailable in this session. The agent used reasoning, code editing, shell execution, PostgreSQL diagnostics, and GitHub tools. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
580ec41110 |
fix(workspaces): run private tasks parked in a personal project (#20589)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A private task without a project is parked in its owner's "My private tasks" project > - The spec says an issue-level grant can expose a task without exposing its containing project > - Workspace setup for a run also required project read on the task's project > - So an agent that could read its private task still failed at setup, before any work, when the task was in "My private tasks" > - This pull request accepts task access for that personal project when the project supplies no workspace files > - The benefit is that a person can make a chat or task private and the agent can still work on it, without gaining access to the person's other private tasks ## Linked Issues or Issue Description **What happened** A person made their chat with the CEO agent private. The chat moved into the person's "My private tasks" project. The next CEO run failed at execution setup in about one second with "Project is private and the actor is not an access member." **Expected behavior** The CEO can work in a private chat (or on a private task) that it can read. The CEO does not get access to the person's other private tasks. **Steps to reproduce** 1. Open a chat with an agent and make the chat private (the chat has no project). 2. Send a message to the agent. 3. See the run fail at setup with `deny_project_private` on the person's "My private tasks" project. ## What Changed - `assertTaskWorkspaceExecutionAccess` marks the task's own project as a container when it is not also the selected workspace or source project. - `assertTaskWorkspaceSourceProjectAccess` accepts a `deny_project_private` decision for such a container only when it is a personal "My private tasks" project with no project workspaces. Task read is already authorized earlier in the same check. - Shared private projects, personal projects with workspaces, workspace and source projects, and low-trust boundary denials are not changed. ## Verification - Red then green: `cd server && npx vitest run src/__tests__/agent-project-access.test.ts` (new test `runs a private task in its owner's personal project without granting that project` fails on `master` with the eval's error; 12 passed with the change). The test also checks that the personal project stays unreadable, that a shared private project still needs membership, and that a personal project with a workspace still needs membership. - `npx vitest run src/__tests__/workspace-preparation-history.test.ts src/__tests__/project-workspace-root-provenance.test.ts src/__tests__/task-workspace-child-trust.test.ts src/__tests__/task-workspace-source-copy-privacy.test.ts src/__tests__/privacy-production-review.test.ts` (81 passed). - `tsc --noEmit -p server`: no errors in the changed files. ## Risks - The change relaxes one authorization check. It applies only to the task's own personal project, only after task read is allowed, only for the private-project denial reason, and only when that project has no project workspaces. A project read through any other path is not changed. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
accf22076f |
fix(search): apply task privacy to artifact results (#20595)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Private tasks are visible only to the people and agents that have access > - Company search already filters task results by the caller's task read condition > - Artifact results (agent documents, work products and attachments) in search, and the company artifacts list, did not use that condition > - So a company member could find a private task's document title and snippet through search > - This pull request applies the same task read condition to artifact results > - The benefit is that private task outputs stay private in search and in the artifacts list ## Linked Issues or Issue Description **What happened** An agent saved a document on a private task. A different company member without access to that task searched for a word in the document. The search returned the document as an artifact result with a snippet. **Expected behavior** Search and the artifacts list show artifacts only from tasks the caller can read. **Steps to reproduce** 1. Create a private task and let an agent save a document on it. 2. Sign in as a company member who cannot read the task. 3. Search for a word in the document, or open the company artifacts list. 4. See the private document in the results. ## What Changed - `company-search.ts`: artifact counts and artifact rows apply `options.issueReadCondition`. - `routes/companies.ts`: `GET /companies/:companyId/artifacts` passes the caller's `issueReadSqlCondition` to the artifacts service. ## Verification - Red then green: `cd server && npx vitest run src/__tests__/company-search-service.test.ts -t "private task artifacts"` (fails on `master` with the private document in the outsider's results). - Red then green: `npx vitest run src/__tests__/company-artifacts-service.test.ts -t "caller can read"`. - `npx vitest run src/__tests__/company-search-service.test.ts src/__tests__/company-search-extract-service.test.ts src/__tests__/company-search-extract-routes.test.ts src/__tests__/company-artifacts-service.test.ts` (all pass). - `tsc --noEmit -p server`: no errors in the changed files. ## Risks - Low risk. The read condition is the same one task results use. Callers with full access (local board, instance admin, privacy mode off) get `true` and see no change. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
491dfa1c6d |
fix(tool-gateway): make gateway client token creation person-only (#20581)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Tool gateways let external MCP clients use a company's tools with a gateway client token > - A gateway client token is shown only once, to the caller that creates it > - Since delegated connection management (#17737), an agent can create a gateway client token for its responsible person > - In that case the agent receives the one-time token value, and it can repeat the value in chat > - This pull request makes gateway client token creation person-only > - The benefit is that the one-time token value goes only to the person who creates it in the gateway UI ## Linked Issues or Issue Description Refs #17737 **What happened** A person asked the CEO agent in chat to replace the token of a gateway client. The agent called `POST /api/tool-gateway/gateways/{gatewayId}/tokens` under delegated authority. The response gave the agent the new token value, and the agent put the value in its chat reply. **Expected behavior** An agent does not receive a gateway client token value. A person creates the token on the gateway's Tokens page and gives it to the client. The agent tells the person where to do this. **Steps to reproduce** 1. Give an agent delegated connection management for a responsible person with `tools:admin`. 2. As the agent, call `POST /api/tool-gateway/gateways/{gatewayId}/tokens` (for example through `call_api`). 3. See a 201 response that contains the new token value. ## What Changed - `POST /api/tool-gateway/gateways/:gatewayId/tokens` returns 403 `gateway_token_requires_person` to agent callers before it creates a token. The message is final. It gives the Tokens page path (`/<prefix>/apps/gateways/<gatewayId>/tokens`) and tells the agent not to ask for or repeat token values. The response also has `tokensPath`. - Board (person) callers are not changed. - The API catalog entry for the route now says that it is person-only. - Existing agent-issued tokens are not changed. Runtime issuer and responsible-user checks still apply to them. The integration tests now seed such tokens directly to keep that coverage. ## Verification - Red: the new integration test `never returns gateway client token plaintext to agent callers` fails on `master` (the agent `call_api` result is 201 with the token value). - Green: `cd server && npx vitest run src/__tests__/connection-management.integration.test.ts -t gateway` (24 passed). The test also checks that no token row is created for the agent, and that a person session still gets a 201 with the token. - `npx vitest run src/__tests__/tool-gateway.test.ts src/__tests__/openapi-routes.test.ts` (152 passed). - `npx vitest run src/services/native-runtime/runner-api.test.ts` (2090 passed). - `tsc --noEmit -p server`: no errors in the changed files. ## Risks - Behavior change: agents can no longer create gateway client tokens. A workflow that relied on that now gets a 403 that sends the person to the Tokens page. Agents can still revoke tokens and manage gateways under delegated authority. - Low migration risk: no schema change, and existing tokens keep working. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
d5c1f48180 |
perf(native): index the backup lease lookup by scope digest (#20571)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native harness backup expiration removes old session backups. Before it removes a backup, it finds the environment leases that reference the backup root > - This lookup reads leases by run ID, by the `metadata.nativeHarnessBackup.sessionScopeSha256` digest, or by the identity's own lease ID, in one `OR` query > - The metadata arm has no index. So PostgreSQL reads every lease of the company, and the metadata of each lease, for each lookup > - On a hosted test database, this statement ran 21,736 times at a mean of 158 ms in one window. After other fixes, it still ran 2,364 times at a mean of 121 ms > - This pull request adds a small partial expression index on the digest > - The benefit is that each lookup reads a few index entries instead of all company leases. The query text and the result do not change ## Linked Issues or Issue Description **What existing behavior does this improve?** The lease lookup in `backupDependencies` in `server/src/services/native-runtime/native-harness-backup-expiration.ts`. `reconcileBackup` calls it for each backup root when the expiration worker loads or checks a root. **Subsystem affected** Native harness backup expiration and the `environment_leases` table schema. **Current behavior** The query is `company_id = $1 and (heartbeat_run_id in (...) or metadata->'nativeHarnessBackup'->>'sessionScopeSha256' = $n or id = $m)`. The run arm and the ID arm have indexes. The digest arm has no index. A `BitmapOr` plan needs an index for every arm, so the planner reads all company leases and evaluates the JSON expression on each row. Lease metadata holds sandbox details, so these reads are large. **Proposed behavior** The partial index `environment_leases_native_backup_scope_idx` keys leases by the digest. It holds only leases that have a backup stamp. Every arm of the `OR` now has an index, and the planner combines three index probes. The SQL and the returned lease IDs do not change. **Reason and benefit** This statement was in the top ten by total database time on a loaded hosted database. A local reproduction shows a query that is more than 6,000x faster and reads 9 buffers instead of about 436,000. **Breaking changes** None. The API, the lease-matching rules and the results do not change. ## What Changed - Add the partial expression index `environment_leases_native_backup_scope_idx` on `(metadata->'nativeHarnessBackup'->>'sessionScopeSha256')`, where that value is not null. - Add migration `0459`. It uses `CREATE INDEX CONCURRENTLY IF NOT EXISTS`. The migration runner runs it as a top-level statement, the same as migrations 0454–0458. - Add a regression test. It checks the lease-matching rules: run leases, a lease with only the backup stamp, and the identity's own lease match. Another digest, a non-string stamp, a top-level key with the same name, and the same digest in another company do not match. The test also checks that the custom plan and a forced generic plan of the generated SQL use `BitmapOr` with the new index and no sequential scan. - Add `scripts/benchmark-environment-lease-backup-lookup.ts`. It captures the SQL that the real service generates and measures it with and without the index. - Document the index in `doc/DATABASE.md`. ## Verification Cause and thesis. The cost comes from the one `OR` arm without an index, which forces a scan of all company leases and their metadata. Thesis: an index on the digest lets the planner use a `BitmapOr` of three index probes, so the cost no longer depends on the company's lease count. The results stay identical. The benchmark confirms the thesis. I did not change the SQL. With the index, the existing `OR` query already gets the index plan. A `UNION` rewrite would not help without the index, because the digest arm would still scan the company. Local SQL benchmark: ```sh pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-environment-lease-backup-lookup.ts ``` Fixture (synthetic): one company with 100,000 leases with 4 KiB of incompressible metadata each. 30% of the leases have a backup stamp for another session. A second company has 20,000 leases. The table is 678 MB. The backup root has 3 run leases, 1 lease with only the stamp, and 1 identity lease. The same digest is also in the other company. Embedded PostgreSQL 18 on an Apple laptop, which other work loaded heavily. The hosted database is Neon with different compute, so absolute times are local SQL measurements only. The script measures in the order: no index, index, no index, index. Each measurement has 2 warmups and 11 samples of `EXPLAIN ANALYZE` execution time and of the full `reconcileBackup` call. | Measurement | Without index | With index | | --- | --- | --- | | Lease query execution, run 1 | median 196 ms (138–562 ms) | median 0.010 ms (0.007–0.017 ms) | | Lease query execution, run 2 | median 185 ms (144–295 ms) | median 0.030 ms (0.020–0.045 ms) | | `reconcileBackup` wall time (includes file reads), runs 1 and 2 | median 188 ms and 216 ms | median 1.4 ms and 4.2 ms | | Plan | `Gather > Seq Scan` | `Bitmap Heap Scan > BitmapOr` of `heartbeat_run_idx`, the new index and `pkey` | | Buffers | 365,880 hit + 70,050 read | 9 hit | These numbers are from head `65bd9d7ac6`. The final head only renumbers the migration to 0459 after rebases; the SQL, the index and the service code are the same. Two earlier runs on the same fixture, SQL and index gave 236–425 ms without the index and 0.023–0.033 ms with the index. The machine had heavy load from other work, so the spread without the index is wide. Output check: in all four measurements, the service returned the same 5 lease IDs, and the complete query rows were equal. Index build (`CREATE INDEX CONCURRENTLY` from the migration file): 0.8–2.4 s for the 678 MB table over all runs. Index size: 3.4 MB with 30% of leases stamped. Tests: - `cd server && npx vitest run src/__tests__/native-harness-backup-expiration.test.ts`: 109 passed. The new test failed when I dropped the index before the plan check (confirmed red), and it passes with the index. - `pnpm --filter @paperclipai/db check:migrations`: passed. - `pnpm --filter @paperclipai/db typecheck` and `pnpm --filter @paperclipai/server typecheck`: passed. - `pnpm verify`: it asks for broader coverage (exit 2) because of the new script. See the CI results for this head. ## Risks - Migration time. A concurrent index build reads the full `environment_leases` table and does not block writes. Locally, the build took 0.8–2.4 s for 678 MB. The time grows linearly with the table size. I did not time it on a production-size clone. I expect a few seconds to tens of seconds on a large instance, which is in the 1–2 minute budget. This estimate is not proven. - If the build stops, PostgreSQL can leave an invalid index. The migration runner repairs invalid concurrent indexes when it runs again, and `IF NOT EXISTS` makes a rerun safe. - The index expression must stay textually equal to the query expression. If someone changes the JSON path, the planner cannot use the index. The new test catches this for custom and generic plans. - The index is partial, so ANALYZE does not collect statistics for the expression. The planner uses the default equality estimate. On the large fixture and in the small test, the planner still selects the index plan. A company with very few leases can still get a sequential scan, which is cheap. - Each lease write with a backup stamp writes one more small index entry. The write cost is small. - This change does not reduce how often the worker calls the lookup. It makes each call cheap. The other digest lookups in backup retention (`native-harness-backup-retention.ts`) and session retention checks can also use the index. - Previous-release story: no persisted state changes. The index is derived from existing lease metadata. - The hosted improvement is not verified until deployment. After deployment, compare a new query-statistics window for this statement. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, local PostgreSQL benchmarks). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude
|
7368a0b15c |
fix(recovery): re-queue native runs that fail on a transient database fault before provider work (#20561)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native (Paperclip Runner) runs prepare a task, pass a final dispatch
gate, and only then start a provider.
> - A native run can fail before any provider work starts. Examples are
a Postgres serialization or deadlock error (`40001`, `40P01`) at budget
admission, or a transient database error during setup.
> - Today such a run ends `adapter_failed` or `setup_failed`. `failRun`
writes the "no provider work" record only for legacy runs, and the
bounded retry paths do not retry these codes for native runs. The
release then opens the native hold
(`native_continuation_requires_reconciliation`), and a person must click
Retry.
> - The goal is that 0% of tasks need a Retry click after a restart or a
storage/DB fault. Duplicate provider work is worse than a Retry click,
so the fix must only act when the server can prove no provider work
started.
> - This pull request records that proof for native runs and sends a
transient database failure to the existing uncharged, bounded
`interrupted_preparation` retry lane.
> - The benefit is that these runs re-queue by themselves, and runs with
any sign of provider work keep today's hold.
## Linked Issues or Issue Description
**What happened?**
In restart-chaos waves with database pressure, a native run failed at
budget admission with exhausted serialization retries (`40001`/`40P01`).
The run event history had no `native.process_start_requested` event.
About 15 s later the server recorded "Automatic recovery stopped" with
cause `native_continuation_requires_reconciliation`. The task stayed
blocked until a person clicked Retry.
**Expected behavior**
A native run that failed before any provider work, on a fault that
clears by itself, re-queues automatically. The attempt is bounded and
does not spend the failure budget. A persistent failure still ends
failed, with a working Retry.
**Steps to reproduce**
1. Start a native (Paperclip Runner) task.
2. Make budget admission or a setup database write fail with SQLSTATE
`40001` or `40P01` until its bounded retries are exhausted.
3. See the run end `adapter_failed`/`setup_failed`, the task go to
`blocked`, and a `native_continuation_requires_reconciliation` recovery
action that waits for Retry.
**Paperclip version or commit**
`master` at
|