mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-11 05:11:49 +01:00
85dec046776e0c3ff0f96962f103b91c85730957
7379
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandPaperclip
|
85dec04677 |
fix(native): validate active controller durability during eligibility (#20601)
## Thinking Path > - Paperclip controls agents while their work runs outside the web server. > - A web replacement must retain each controller's durable recovery state. > - Modern controllers commit that state to the database before sending work. > - Their local controller file is only an external-persistence identity marker. > - Disk eligibility still required recovery metadata that can arrive after a run finishes. > - This pull request verifies the complete active database authority instead. > - It also bounds concurrent source reads so a large history does not require serial network waits. ## Linked Issues or Issue Description Refs: #20311, #20584, #20599. **What happened?** A live native run can have a complete database controller checkpoint and no terminal session snapshot yet. The disk-preservation check treats its local identity marker as a local-only obligation. The check also resolves selected sources one at a time. **Expected behavior** Accept a complete, bound active recovery authority. Refuse missing, partial, stale or foreign evidence. Keep all execution and enrollment guards. **Steps to reproduce** Use the new active-controller regression. Start a real Runner controller with the database checkpoint writer. Keep the business run active and omit its terminal snapshot metadata. The previous check returns a local obligation even after the full checkpoint commits. ## What Changed - Require the exact external-persistence marker and a complete decoded encrypted checkpoint for active runs. Match run, agent, task, session, Runner and workspace identities. For warm reuse, also permit an older marker only when its full older checkpoint matches that marker and its exact earlier successful run has a committed accepted result in the same company, agent, task, session, Runner and workspace. Keep the current live writer mandatory. Read the marker again after validation. - Accept that proof only with its fenced live writer and task binding, or the existing exact saved recovery snapshot. The saved-snapshot path still requires the full active journal. - Keep newer remote workspace authority strict. A malformed reference cannot fall back to an older prepared reference. Conflicting plugin task authority remains blocked. - Resolve selected sources at most eight at a time for read-only eligibility. Join every started reader before reporting an error. Keep source order, pagination, predicates and all write paths unchanged. Offline migration and prestaging still resolve sources one at a time. - Add real PostgreSQL regressions for active durability, partial and foreign state, expired ownership, previous prepared-reference state, post-result state and reader failure joining. ## Verification - Canonical Node24 preflight passed in74.2s. It built the current Runner inputs and published the cache. No shared Cargo target or hosted database was used. - Genuine active regression RED: the previous check returned true when false was required. The real checkpoint writer had committed the complete authority and replaced the local file with the Runner marker. Treatment passed, including the refusal controls. - Genuine source-reader RED: sixteen retained pointers used one reader at a time. Treatment used at most eight and joined all eight when the first reader failed. With a controlled20ms reader delay and real PostgreSQL selection, the small fixture changed from497.5ms to85.3ms. This is a bounded-concurrency proof, not a production speedup measurement. - All54 cases in the affected controller migration integration file passed in76.07s, with no skips. These include existing failed-controller, archived-controller, inline database and previous-session preservation cases. - Server TypeScript passed. The normal merge of current master kept both changed files byte-for-byte identical to the54-case proof. Final post-merge focused checks passed two of two in14.83s (52 other cases intentionally unselected); server TypeScript and module boundaries passed. Comment-only correction743 removed all new code comments required by CONTRIBUTING. A later warm-reuse regression reproduced the remaining false blocker through the real Runner rotateRunIdentity API. The bounded treatment uses one exact older-run lookup and full old/current checkpoint validation; it adds no Runner format, release or lineage service. Final normal-merge head `c62d6eb1755cf99b6def9563ba69854dba17de42` includes the new0459 baseline. All three focused PostgreSQL cases passed in21.37s (52 existing cases intentionally unselected); fresh server TypeScript and module boundaries passed. The prior54-case regression evidence is retained; no full matrix was repeated. - `git diff --check` passed. Final-head CI passed:56 checks succeeded and4 were skipped; the separate Paperclip Review advisory was still queued. All required test, build, typecheck and ci/verify gates passed on c62. [Final required workflow](https://github.com/paperclipai/paperclip/actions/runs/42868279612). Fresh [Greptile review](https://github.com/paperclipai/paperclip/pull/20601#issuecomment-6774537057) is5/5 on exact head `c62d6eb1755cf99b6def9563ba69854dba17de42`, with no open review threads. ```sh node scripts/agent-preflight.mjs --skip-rust-debug node --import ./cli/node_modules/tsx/dist/loader.mjs packages/db/src/test-postgres-command.ts pnpm exec vitest run server/src/services/native-runtime/native-controller-migration.integration.test.ts --disableConsoleIntercept pnpm --filter @paperclipai/server exec tsc --noEmit ``` ## Risks - This reports preservation evidence. It does not grant attachment, execution, replacement or enrollment authority. The normal live-owner and admission gates still apply. - A partial old or current checkpoint, stale local controller state, identity mismatch or uncertain owner stays blocked. Warm-marker proof also rejects a foreign agent, newer source, later completion, rejected result, per-turn scope or expired current writer. No source, checkpoint, session, lease, archive or provider state is changed by the check. - The eight readers add bounded database concurrency during an explicit eligibility request. There is no new background scan, cache, row filter or timeout change. Live enumeration time still needs deployed measurement. - The plugin task-lease branch uses the producer's environment lease row ID, as native-session-executor and existing diskless adoption coverage require. A direct active-eligibility positive for that plugin branch is a coverage limit; existing mismatch/conflicting-authority checks remain. The advisory found no must-fix defect, and this does not change the current tested revision. - Saved formats and terminal migration behavior remain unchanged. Older prepared workspace references remain supported. A fully saved result can still pass before the run-status update, but its entire active checkpoint must decode. - The observed staging run finished naturally and became valid on the existing terminal path during investigation. That observation is not deployed verification of the new active path. ## Model Used OpenAI GPT-6 (Codex), with reasoning, tool use and code execution. This runtime does not expose a more specific model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
9009fbdedb |
fix(runner): accept request_human_input payload version forms models send (#20588)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents ask people for decisions with the runner's `request_human_input` tool, which creates a question or confirmation card > - The tool schema required `payload.version` to be the number 1 > - The server ignores the version the model sends and always stores version 1 > - Some models send `"version": "1"` as a string, so every call failed with "/payload/version must be equal to constant" > - This pull request accepts `1` or `"1"`, and lets the model omit the version > - The benefit is that the model's natural call creates the card on the first try, and a turn does not end with no card and no reply ## Linked Issues or Issue Description Refs #19405 (adds invocation examples for the same tool; it does not change validation). Refs #20134 (aligns the same tool's text limits). **What happened?** In a chat eval run on master `f4337bb5fb` (OpenCode runner, `deepseek-v4.1-flash`), the agent called `request_human_input` up to six times. Each call failed with `Invalid tool input: /payload/version must be equal to constant`. The agent's reasoning shows it sent `"version": "1"` and could not produce the number. In one case it then made the card with `call_api`. In two other cases the turn lost the card. This happened in `questions-and-approvals.reject-and-revise`, `goals.align-existing-work` and `assistant-connections.revoke-personal-grant`. **Expected behavior** A version field that the server overwrites must not block the call. The tool must still reject an unknown version and a questions card without a question form. **Steps to reproduce** Validate `{ interactionKind: "questions", payload: { version: "1", questionSet: {...} } }` against the live `request_human_input` input schema. Before this change, validation fails at `/payload/version`. ## What Changed - `payload.version` in the tool schema accepts `1` or `"1"` and is optional. Its description says Paperclip stores version 1. - A questions card no longer requires `payload.version`. It still requires `questionSet` or `questions`. - The payload description no longer tells the model to send `version:1`. - Regenerated the semantic tool contracts, the seeded eval catalog digest and the protocol manifest with the existing generator. - Added a regression test: number, string and missing version pass for live and scenario schemas; version `2` and a payload without questions fail. ## Verification - `npx vitest run src/protocol-actions/` in `packages/paperclip-runner`: red before the fix with the exact eval error (`/payload/version must be equal to constant`), green after (47 tests). - Focused runner tests for the tool bridge, semantic tools, mock control plane, codex semantic results, catalog and semantic tools: 231 passed. - `node scripts/generate-semantic-contracts.mjs --check`, `generate-semantic-action-catalog --check`, `generate-protocol-manifest --check`, `generate-replay-goldens --check`: current. - `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`: unchanged. - Paid chat eval on a temporary branch with this change: see the comment on this PR. ## Risks - Low risk. The server already writes `version: 1` into every interaction payload created by this tool (`paperclip-runner-tool-authority.ts`), so no stored shape changes. - Previous-release story: no persisted run, session or launch state changes. Saved tool receipts keep the input they recorded. A retry with a different version form under the same idempotency key is a different input, as before. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), agentic coding with tool use in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
1b7ec13acb |
perf(native): default remote sandbox runs to a warm, reusable sandbox for measured harnesses (#20048)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents can run in remote sandboxes (Daytona), where each turn needs
the task's workspace and a runner inside the sandbox
> - With a default environment, every remote turn got a brand-new
sandbox: the whole workspace was uploaded again and a new runner
started, even for a follow-up comment
> - In a paperclip-sized workspace that made a follow-up take 30 s to
first token, against 8–10 s with a reusable warm sandbox
> - This pull request makes an agent that never chose a lifecycle run
warm on a sandbox for the measured harnesses (Codex, OpenCode, Claude),
so a follow-up keeps its sandbox, runner and workspace
> - The default applies only where warm can work: a sandbox provider
that keeps reusable leases, and a managed AI account that supports a
warm session. Everywhere else the run is per-turn, exactly as before
> - The benefit is fast remote follow-ups by default, without
re-uploading a large repository every turn
## Linked Issues or Issue Description
**What happened?**
On Daytona, with a fresh paperclip clone as the project workspace, a
follow-up comment took 30 s to its first token. The run log showed
`environment: fresh/created` on every follow-up. It spent 13.3 s in
`workspace_sync.transfer`, a full upload into the new sandbox, and 8–9 s
in `runner.session.resume`.
**Steps to reproduce**
1. Create a Daytona environment with the defaults: lifecycle inherited,
lease reuse off.
2. Run a Codex task in a large repository, then post a follow-up.
3. The follow-up gets a new sandbox, uploads the whole workspace again
and starts a new runner.
**Expected behavior**
A follow-up on the same task reuses the live sandbox, its runner and its
workspace (plan rule 2: no full re-upload per turn).
## What Changed
- **One rule for the default.** `defaultsRemoteSandboxRunToWarm`
(`native-runtime/native-warm-capability.ts`) decides the remote-sandbox
warm default for both the lease (environment view) and the runner
(native session policy), in one vocabulary (runtime backend plus ACPX
agent). It is applied only when no lifecycle was chosen anywhere:
admitted session, environment `runnerLifecycleMode`, or agent
`lifecycleMode`. It holds only when:
1. the harness is measured (Codex, OpenCode, ACPX Claude);
2. the sandbox provider keeps reusable leases (Daytona, CreateOS and
boxd declare them; E2B, Modal and the built-in `fake` do not); and
3. with a managed AI account, that harness supports a managed warm
session (`supportsManagedNativeWarmSession`: Codex).
- **The harness resolves as the runtime resolves it.**
`resolveWarmProviderSelection` uses
`resolvePaperclipRunnerProviderProfile`, so an omitted `provider` is
Codex and ACPX without an agent is Claude, in both decisions. A config
the runtime would refuse gets no default.
- **Lease side** (`runner-environment-lifecycle.ts`,
`environment-run-orchestrator.ts`): for an eligible run only, the
orchestrator asks the provider before acquisition
(`providerSupportsReusableLeases` in `environment-runtime.ts`: the
acquisition rule, one registry lookup for a plugin provider, no
worker-ready wait). With support, the run-scoped view sets `reuseLease:
true` and forwards the warm idle window to the provider's auto-stop. The
shared environment is never rewritten. Every other run resolves exactly
as on master, without the probe.
- **Runner side** (`native-lifecycle.ts`, `runtime-selection.ts`): the
native policy applies the same rule to the acquired lease
(`reusableLeaseConfigured && effectiveCapabilities.reusableLeases`). If
acquisition downgrades the lease anyway, the run falls back to per-turn;
a default never produces
`runner_warm_environment_requires_reusable_lease`.
- **Idle window.** The remote default uses the window the environment
view forwards to the provider: an inheriting environment's
`runnerIdleTimeoutMs`, then the agent's `idleTimeoutMs`, then 5 minutes.
The sandbox execution target now carries the environment's window
(`runnerIdleTimeoutMs`, an optional field in `adapter-utils`'s execution
target) so the runner and the provider auto-stop agree. An explicit warm
choice and the local default are unchanged.
- Unchanged: explicit agent choices, environments that pin a lifecycle,
plugin-task environments, admitted sessions (`persistedPolicy`), and
local behavior.
- `tests/restart-chaos`: the vCPU budget check counts `--daytona-inherit
--lifecycle default` sandboxes as retained, because they now default to
warm.
- `doc/plans/2026-11-08-large-workspace-speed.md`: the measurements, the
ranked waits, and the gate.
## Verification
- Unit and integration tests, one per rule:
- Rule 1 (provider gate): `native-lifecycle.test.ts` (ephemeral lease →
per-turn), `runner-environment-lifecycle.test.ts` (no reusable leases →
environment unchanged, no warm window),
`environment-runtime-reusable-lease-probe.test.ts` (`fake`, an
undeclared plugin, a declared plugin whose worker does not verify the
reuse methods, and a plugin without a running worker read false),
`runtime-selection.test.ts` ("defaults an agent without a lifecycle on
...": reusable → warm/keep_running; reuse configured on an unsupported
provider → per_turn/destroy_after_turn; ephemeral → per_turn),
`environment-run-orchestrator.test.ts` (probe result decides
`reuseLease` and the forwarded window).
- Rule 2 (explicit configuration wins): explicit per-turn agent, pinned
environment, admitted session, configured `idleTimeoutMs`, managed
Claude per-turn, managed Codex warm; the orchestrator does not probe
when a lifecycle is explicit.
- Rule 3 (omitted provider): `{}` resolves to Codex in the environment
view; `runtime-selection.test.ts` "agrees with the environment view's
lease for ..." feeds the environment view's lease into selection for
Codex, a provider-less Codex agent, OpenCode, ACPX Claude, ACPX without
an agent, and Copilot (per-turn), and checks both decisions agree. A
plugin-task target stays out of the default.
- The two suites that failed on the previous head
(`runtime-selection.test.ts`,
`workspace-recovery-preparation-dispatch.test.ts`) pass locally.
Locally, 6 connector-skill cases in `runtime-selection.test.ts` fail
with `ENOENT .claude/skills/slack/SKILL.md` because this checkout is a
nested worktree; they do not touch this change and pass in CI.
- End-to-end check of this default on Daytona (before the gate was
added; Daytona keeps reusable leases, so the gate does not change this
path): a user-default environment (lifecycle inherited, lease reuse
unset, `--daytona-inherit`), a fresh paperclip clone, built from the
image's source revision with this change (branch measure/remote-warm).
| | Before (per-turn sandbox) | After (this change) |
|---|---|---|
| Follow-up, Codex | 30.4 s | 7.6 s |
| Follow-up, Claude | about 30 s | 11.0 s |
| First follow-up after an idle K1, Codex | 45.9 s | 20.3 s |
| First follow-up after an idle K1, Claude | 38.3 s | 18.3 s |
The run-log reuse records show the follow-ups reusing the workspace,
environment, runner and provider session, with no
`workspace_sync.transfer`. First turns still upload the workspace once
(14.8 s and 26.1 s). Every sandbox was cleaned up.
- Daytona restart chaos wave (K1/K2/K4 with a mid-run comment,
inheriting environment, agents with no lifecycle, runner image built
from this branch): wave `w-20261110T171438-s20048`, seed 20048, on
|
||
|
DottaandClaude
|
c37eacbee3 |
fix(native): Retry re-finalizes a saved result that ran out of finalization retries (#20578)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs save an accepted result. Then a finalizer applies that
result to the task.
> - Finalization has its own budget of 3 failed attempts (#20555). When
the budget runs out, the coordinator goes to `terminal_failure` with
`native_finalization_retry_exhausted`.
> - For a provider-succeeded run, the task then rests `in_review` with a
board recovery card. The task shows "resolved" and offers no Retry. The
user is stuck.
> - The ordinary Retry is also wrong here: it starts a new provider run,
which repeats work the agent already finished.
> - This pull request adds two things. First, contention failures on
such a result get two longer automatic delays before the board gets the
task. Second, when the state is still exhausted, the task offers Retry,
and that Retry applies the same saved result again.
> - The benefit: fewer tasks need a Retry, and when one does, Retry
works and does no duplicate provider work.
## Linked Issues or Issue Description
Refs #20555 (it names this gap under Risks).
**What happened?**
In restart-chaos wave `w-20261110T044338-s20385`, task t03 ended with a
succeeded run, an accepted result, and a coordinator in
`terminal_failure` / `native_finalization_retry_exhausted`. The issue
stayed `in_review` with a board-owned `active_run_watchdog` card and
`canRetry=false`. #20555 stopped the early exhaustion. A real exhaustion
still leaves the same stuck state. Coordinators that older builds
already left in this state are stuck the same way.
**Expected behavior**
Transient finalization failures recover by themselves when that is safe.
When they do not, the task offers Retry. That Retry re-finalizes the
saved result. It does not start a provider run and does not charge a
provider attempt.
**Steps to reproduce**
1. Let a native run save an accepted result and end `succeeded`.
2. Make each finalization of that result fail until the budget of 3 runs
out.
3. Before this fix, the task presentation is `resolved` with
`canRetry=false`, and the board card has no action that applies the
result. The new test
`server/src/__tests__/native-exhausted-finalization-retry.test.ts`
reproduces this.
**Paperclip version or commit**
master at
|
||
|
Nicky LeachandPaperclip
|
98ce93dd4e |
Choose project storage from compatible environments (#20587)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Projects supply repository files to agent runs through core staging or a storage plugin. > - A storage plugin's default claim silently attached new projects even when their agents used a different environment. > - An environment name or driver type does not establish shared storage access. The existing binding and task-driver contracts do. > - This pull request groups creation choices by that contract and selects storage from the creator's effective environment. > - The benefit is a visible, validated choice and a clear error before an incompatible managed workspace operation. ## Linked Issues or Issue Description Fixes #20459. Related: #20156 covers a separate provider-failure diagnostic. The PR search found no other fix for #20459. ROADMAP.md has no duplicate project-creation change. ## What Changed - Show **Available in** when configured environments require different project storage. Each option names the environments that can use the same storage owner. Hide the selector when all choices are compatible. - Derive compatibility from repository staging and managed binding ownership. Local, SSH, and sandbox environments use core staging. Managed storage requires a registered task driver from the same installed storage plugin. Separate storage plugins remain incompatible. Unknown plugin contracts and the fake sandbox are excluded. - Follow the creating agent's environment, or the person's primary agent, then the instance default and local fallback. Apply managed-only and forced Kubernetes policy. The product has no separate user or company environment-default field today. - Add a company-authorized discovery endpoint and an optional creation `environmentId`. Validate the choice again on save. Store the storage owner without overriding agent run environments. Preserve explicit API storage choices and existing project ownership. - Keep the hidden repository-free private-task project independent of environment availability. Submit the observed storage owner as `expectedStorageProviderPluginId` so a changed contract fails at save without setting a workspace policy. Instance task-isolation defaults stay inherited. - Reject an incompatible managed workspace before a provider operation. Name the storage and selected environment in the error. Add regression tests, API documentation, creation-story fixtures, and minimal corrections to obsolete map text. ## Verification - **102 tests passed across 7 files:** project environment integration, installed storage policy, repository persistence, project environment routes, project creation telemetry routes, OpenAPI, and NewProjectDialog. Run with `node node_modules/vitest/vitest.mjs run server/src/__tests__/project-environments.test.ts server/src/__tests__/project-workspace-provider.integration.test.ts server/src/__tests__/project-repositories-persistence.test.ts server/src/__tests__/project-routes-env.test.ts server/src/__tests__/project-goal-telemetry-routes.test.ts server/src/__tests__/openapi-routes.test.ts ui/src/components/NewProjectDialog.test.tsx --silent`. - The review fixes cover both archived-default private-task cases and changed-storage rejection. With both instance isolation flags enabled, a new core repository project keeps a null policy and resolves to an isolated Git worktree. The complete affected set passes all 102 tests. - The CI query-error policy failure was fixed by using the shared query view. The form and shared query-view suites pass **24 tests**, including cached choices during a transient refetch failure. `node scripts/check-query-error-rendering.mjs --enforce --summary` reports zero findings. - CI exposed stale SDK source hashes and heartbeat fixtures that paired plugin storage with Local. Refreshed exactly four shared-source hashes with the maintained generator (runtime pins unchanged). All **53 SDK privacy/source-proof tests**, including real built SDK output, pass. All **7 corrected managed-storage heartbeat scenarios** pass; the fixtures declare the owning plugin's task environment and substitute transport at the existing execution seam. - Shared and UI TypeScript checks pass. The UI production build passes. `pnpm check:token-gates`, `node scripts/check-maps.mjs`, `node scripts/check-module-boundaries.mjs`, and diff checks pass. - Tested UI states: compatible choices, incompatible groups, default within a group, manual override, create failure, loading, discovery failure and retry, and an unavailable default. Tests use the production form and existing design tokens. - The integration tests use real isolated PostgreSQL on macOS arm64 with Node 26.12.0. They cover user/agent defaults, storage persistence, policy restrictions, company and authentication denials, and rejection before a provider call. Discovery uses saved configuration and makes no worker or provider requests. - Local verification limits: `pnpm -r typecheck` and `pnpm build` stop in unchanged adapter-utils code because the installed ACPX types lack `inheritProcessEnv`. A direct server check also finds an unchanged AI SDK type-version mismatch. The existing heartbeat test `inherits the instance default environment across companies while preserving explicit agent overrides` times out at line 518 on both this change and unmodified master with the same dependencies. The full `pnpm test:run` attempt passed all six chat shards (1,948 tests), then the Git workspace expiration suite reported 14 failures. I stopped that broad run. Two sampled failures reproduce on unchanged master with the same `unknown_worktree_git_files` refusal; the other expiration failures were not independently compared. A later broad workspace-preparation run also hit workspace validation failures and was stopped; the seven corrected fixture scenarios pass independently. These limits do not establish a full local suite pass. Hosted CI remains required. - Initial missing-package and loopback-denial failures were resolved by repairing isolated dependency links, building the SDK, and running database tests with loopback access. No deployed or live-provider execution is claimed. ## Risks - Creation defaults now follow the effective environment. A plugin's `defaultForProjects` claim alone no longer selects storage. This is the intended behavior change. - Compatibility covers repository project storage. It does not guarantee arbitrary host paths, retained-workspace moves, adapter/image support, credentials, or worker readiness. Matching provider configs cannot prove a cross-plugin storage contract. - A person's primary agent supplies the personal default because there is no separate personal environment setting. An unavailable configured default requires a valid explicit choice or a configuration repair. - No migration runs. Existing projects keep their storage owner. This change does not detach existing managed data or provide a storage migration. - Full local verification has the limits listed above. CI and standard Greptile review are pending. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, tool use, and code execution. The exact serving snapshot and context-window size were not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
1c78d023c6 |
perf(native): narrow backup eligibility to selected runs (#20599)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Server replacement checks whether retained native sessions still depend on local harness data. > - This check joins selected task sessions, runs, backup stamps, and durable backup records. > - PostgreSQL can start from backup leases and scan an issue's full run history before applying the selected-run condition. > - A measured instance spent 9.56 seconds in that selector and exceeded an 8-second statement limit. > - This pull request selects the exact current run identities first, then applies the existing lease and backup checks. > - The same receipt validation and eligibility decisions remain in place. ## Linked Issues or Issue Description **What happened?** Startup eligibility timed out while selecting native harness backup obligations. The plan scanned historical runs for each lease before checking the task session's selected run. **Expected behavior** The selector should use the selected run identities without scanning unrelated run history. It must return the same session, scope, company, and receipt flag. **Steps to reproduce** Use a database with retained native task sessions, many historical runs on the same issues, and version-two backup stamps. Run startup eligibility with an 8-second statement timeout. The lease-first plan can exceed that limit. **Paperclip version or commit** The local baseline is `d0c30e69a9cea912db7c5290c5d585fdd63ec228`. **Deployment mode** Self-hosted server with PostgreSQL. ## What Changed - Keep `AS MATERIALIZED` as an intentional join-order fence: build selected task-session and last-run identities before joining backup leases. - Keep only narrow identity columns in that materialized set. Read the JSON receipt flag through the selected run's primary key after the lease and current-backup filters, so profiles for unrelated identities do not widen the intermediate rows. - Preserve company and agent bindings, exact normalized-session text comparison, current-backup exclusion, scope deduplication, ordering, and the 100-row page. - Add real PostgreSQL regressions for full ordered-row equivalence, nullable JSON receipt flags, malformed and uppercase session bindings, cross-company rows, duplicate stamps, page tails, and invalid receipts. - Existing saved state carries forward under the same validation rules. No schema or persisted identity changes are required. ## Verification - The two focused PostgreSQL regressions passed locally. Command: `pnpm exec vitest run server/src/services/native-runtime/native-harness-backup-obligations.integration.test.ts --silent`. - Worktree preflight passed. The focused regressions (2/2) and `node server/scripts/typecheck-layers.mjs` passed on final head `c739353b22` after rebase. - One bounded read-only pair on PostgreSQL 17.11 returned the same 54 complete ordered rows: baseline 9,564 ms, candidate 298 ms (32.1 times faster). The plan changed from lease-to-issue-history scans to selected-run primary-key lookups and a hash join. - A synthetic PostgreSQL 18.1 fixture already chose an efficient baseline plan. Across five alternating warm pairs, its median was 104.6 ms before and 109.4 ms after. This neutral control used 22,800 runs, 14,100 leases, 3,173 task sessions, and 5,460 backups; all 54 complete rows matched. The empty control also matched. - These measurements cover the selector. A later read-only assembled-call probe on a newer serving release substituted only this selector: the selector took 544 ms for 54 rows, while the full eligibility call took 56.1 seconds and still reported a controller migration blocker. Other checks remain outside this PR; this PR does not claim to bring the whole request within its deadline. - Required CI and fresh review results will be recorded here before handoff. ## Risks - Materialization adds a selected-identity pass. Databases that already choose an efficient plan may see little benefit, as the local neutral control shows. - Pagination and receipt validation must stay exact. The regression crosses the 100-row boundary and checks that an invalid final scope still blocks eligibility. ## Model Used OpenAI Codex, GPT-6 family. The exact model ID and context window are unavailable in this session. The agent used reasoning, code editing, shell execution, PostgreSQL diagnostics, and GitHub tools. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
580ec41110 |
fix(workspaces): run private tasks parked in a personal project (#20589)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A private task without a project is parked in its owner's "My private tasks" project > - The spec says an issue-level grant can expose a task without exposing its containing project > - Workspace setup for a run also required project read on the task's project > - So an agent that could read its private task still failed at setup, before any work, when the task was in "My private tasks" > - This pull request accepts task access for that personal project when the project supplies no workspace files > - The benefit is that a person can make a chat or task private and the agent can still work on it, without gaining access to the person's other private tasks ## Linked Issues or Issue Description **What happened** A person made their chat with the CEO agent private. The chat moved into the person's "My private tasks" project. The next CEO run failed at execution setup in about one second with "Project is private and the actor is not an access member." **Expected behavior** The CEO can work in a private chat (or on a private task) that it can read. The CEO does not get access to the person's other private tasks. **Steps to reproduce** 1. Open a chat with an agent and make the chat private (the chat has no project). 2. Send a message to the agent. 3. See the run fail at setup with `deny_project_private` on the person's "My private tasks" project. ## What Changed - `assertTaskWorkspaceExecutionAccess` marks the task's own project as a container when it is not also the selected workspace or source project. - `assertTaskWorkspaceSourceProjectAccess` accepts a `deny_project_private` decision for such a container only when it is a personal "My private tasks" project with no project workspaces. Task read is already authorized earlier in the same check. - Shared private projects, personal projects with workspaces, workspace and source projects, and low-trust boundary denials are not changed. ## Verification - Red then green: `cd server && npx vitest run src/__tests__/agent-project-access.test.ts` (new test `runs a private task in its owner's personal project without granting that project` fails on `master` with the eval's error; 12 passed with the change). The test also checks that the personal project stays unreadable, that a shared private project still needs membership, and that a personal project with a workspace still needs membership. - `npx vitest run src/__tests__/workspace-preparation-history.test.ts src/__tests__/project-workspace-root-provenance.test.ts src/__tests__/task-workspace-child-trust.test.ts src/__tests__/task-workspace-source-copy-privacy.test.ts src/__tests__/privacy-production-review.test.ts` (81 passed). - `tsc --noEmit -p server`: no errors in the changed files. ## Risks - The change relaxes one authorization check. It applies only to the task's own personal project, only after task read is allowed, only for the private-project denial reason, and only when that project has no project workspaces. A project read through any other path is not changed. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
accf22076f |
fix(search): apply task privacy to artifact results (#20595)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Private tasks are visible only to the people and agents that have access > - Company search already filters task results by the caller's task read condition > - Artifact results (agent documents, work products and attachments) in search, and the company artifacts list, did not use that condition > - So a company member could find a private task's document title and snippet through search > - This pull request applies the same task read condition to artifact results > - The benefit is that private task outputs stay private in search and in the artifacts list ## Linked Issues or Issue Description **What happened** An agent saved a document on a private task. A different company member without access to that task searched for a word in the document. The search returned the document as an artifact result with a snippet. **Expected behavior** Search and the artifacts list show artifacts only from tasks the caller can read. **Steps to reproduce** 1. Create a private task and let an agent save a document on it. 2. Sign in as a company member who cannot read the task. 3. Search for a word in the document, or open the company artifacts list. 4. See the private document in the results. ## What Changed - `company-search.ts`: artifact counts and artifact rows apply `options.issueReadCondition`. - `routes/companies.ts`: `GET /companies/:companyId/artifacts` passes the caller's `issueReadSqlCondition` to the artifacts service. ## Verification - Red then green: `cd server && npx vitest run src/__tests__/company-search-service.test.ts -t "private task artifacts"` (fails on `master` with the private document in the outsider's results). - Red then green: `npx vitest run src/__tests__/company-artifacts-service.test.ts -t "caller can read"`. - `npx vitest run src/__tests__/company-search-service.test.ts src/__tests__/company-search-extract-service.test.ts src/__tests__/company-search-extract-routes.test.ts src/__tests__/company-artifacts-service.test.ts` (all pass). - `tsc --noEmit -p server`: no errors in the changed files. ## Risks - Low risk. The read condition is the same one task results use. Callers with full access (local board, instance admin, privacy mode off) get `true` and see no change. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
491dfa1c6d |
fix(tool-gateway): make gateway client token creation person-only (#20581)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Tool gateways let external MCP clients use a company's tools with a gateway client token > - A gateway client token is shown only once, to the caller that creates it > - Since delegated connection management (#17737), an agent can create a gateway client token for its responsible person > - In that case the agent receives the one-time token value, and it can repeat the value in chat > - This pull request makes gateway client token creation person-only > - The benefit is that the one-time token value goes only to the person who creates it in the gateway UI ## Linked Issues or Issue Description Refs #17737 **What happened** A person asked the CEO agent in chat to replace the token of a gateway client. The agent called `POST /api/tool-gateway/gateways/{gatewayId}/tokens` under delegated authority. The response gave the agent the new token value, and the agent put the value in its chat reply. **Expected behavior** An agent does not receive a gateway client token value. A person creates the token on the gateway's Tokens page and gives it to the client. The agent tells the person where to do this. **Steps to reproduce** 1. Give an agent delegated connection management for a responsible person with `tools:admin`. 2. As the agent, call `POST /api/tool-gateway/gateways/{gatewayId}/tokens` (for example through `call_api`). 3. See a 201 response that contains the new token value. ## What Changed - `POST /api/tool-gateway/gateways/:gatewayId/tokens` returns 403 `gateway_token_requires_person` to agent callers before it creates a token. The message is final. It gives the Tokens page path (`/<prefix>/apps/gateways/<gatewayId>/tokens`) and tells the agent not to ask for or repeat token values. The response also has `tokensPath`. - Board (person) callers are not changed. - The API catalog entry for the route now says that it is person-only. - Existing agent-issued tokens are not changed. Runtime issuer and responsible-user checks still apply to them. The integration tests now seed such tokens directly to keep that coverage. ## Verification - Red: the new integration test `never returns gateway client token plaintext to agent callers` fails on `master` (the agent `call_api` result is 201 with the token value). - Green: `cd server && npx vitest run src/__tests__/connection-management.integration.test.ts -t gateway` (24 passed). The test also checks that no token row is created for the agent, and that a person session still gets a 201 with the token. - `npx vitest run src/__tests__/tool-gateway.test.ts src/__tests__/openapi-routes.test.ts` (152 passed). - `npx vitest run src/services/native-runtime/runner-api.test.ts` (2090 passed). - `tsc --noEmit -p server`: no errors in the changed files. ## Risks - Behavior change: agents can no longer create gateway client tokens. A workflow that relied on that now gets a 403 that sends the person to the Tokens page. Agents can still revoke tokens and manage gateways under delegated authority. - Low migration risk: no schema change, and existing tokens keep working. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
d5c1f48180 |
perf(native): index the backup lease lookup by scope digest (#20571)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native harness backup expiration removes old session backups. Before it removes a backup, it finds the environment leases that reference the backup root > - This lookup reads leases by run ID, by the `metadata.nativeHarnessBackup.sessionScopeSha256` digest, or by the identity's own lease ID, in one `OR` query > - The metadata arm has no index. So PostgreSQL reads every lease of the company, and the metadata of each lease, for each lookup > - On a hosted test database, this statement ran 21,736 times at a mean of 158 ms in one window. After other fixes, it still ran 2,364 times at a mean of 121 ms > - This pull request adds a small partial expression index on the digest > - The benefit is that each lookup reads a few index entries instead of all company leases. The query text and the result do not change ## Linked Issues or Issue Description **What existing behavior does this improve?** The lease lookup in `backupDependencies` in `server/src/services/native-runtime/native-harness-backup-expiration.ts`. `reconcileBackup` calls it for each backup root when the expiration worker loads or checks a root. **Subsystem affected** Native harness backup expiration and the `environment_leases` table schema. **Current behavior** The query is `company_id = $1 and (heartbeat_run_id in (...) or metadata->'nativeHarnessBackup'->>'sessionScopeSha256' = $n or id = $m)`. The run arm and the ID arm have indexes. The digest arm has no index. A `BitmapOr` plan needs an index for every arm, so the planner reads all company leases and evaluates the JSON expression on each row. Lease metadata holds sandbox details, so these reads are large. **Proposed behavior** The partial index `environment_leases_native_backup_scope_idx` keys leases by the digest. It holds only leases that have a backup stamp. Every arm of the `OR` now has an index, and the planner combines three index probes. The SQL and the returned lease IDs do not change. **Reason and benefit** This statement was in the top ten by total database time on a loaded hosted database. A local reproduction shows a query that is more than 6,000x faster and reads 9 buffers instead of about 436,000. **Breaking changes** None. The API, the lease-matching rules and the results do not change. ## What Changed - Add the partial expression index `environment_leases_native_backup_scope_idx` on `(metadata->'nativeHarnessBackup'->>'sessionScopeSha256')`, where that value is not null. - Add migration `0459`. It uses `CREATE INDEX CONCURRENTLY IF NOT EXISTS`. The migration runner runs it as a top-level statement, the same as migrations 0454–0458. - Add a regression test. It checks the lease-matching rules: run leases, a lease with only the backup stamp, and the identity's own lease match. Another digest, a non-string stamp, a top-level key with the same name, and the same digest in another company do not match. The test also checks that the custom plan and a forced generic plan of the generated SQL use `BitmapOr` with the new index and no sequential scan. - Add `scripts/benchmark-environment-lease-backup-lookup.ts`. It captures the SQL that the real service generates and measures it with and without the index. - Document the index in `doc/DATABASE.md`. ## Verification Cause and thesis. The cost comes from the one `OR` arm without an index, which forces a scan of all company leases and their metadata. Thesis: an index on the digest lets the planner use a `BitmapOr` of three index probes, so the cost no longer depends on the company's lease count. The results stay identical. The benchmark confirms the thesis. I did not change the SQL. With the index, the existing `OR` query already gets the index plan. A `UNION` rewrite would not help without the index, because the digest arm would still scan the company. Local SQL benchmark: ```sh pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-environment-lease-backup-lookup.ts ``` Fixture (synthetic): one company with 100,000 leases with 4 KiB of incompressible metadata each. 30% of the leases have a backup stamp for another session. A second company has 20,000 leases. The table is 678 MB. The backup root has 3 run leases, 1 lease with only the stamp, and 1 identity lease. The same digest is also in the other company. Embedded PostgreSQL 18 on an Apple laptop, which other work loaded heavily. The hosted database is Neon with different compute, so absolute times are local SQL measurements only. The script measures in the order: no index, index, no index, index. Each measurement has 2 warmups and 11 samples of `EXPLAIN ANALYZE` execution time and of the full `reconcileBackup` call. | Measurement | Without index | With index | | --- | --- | --- | | Lease query execution, run 1 | median 196 ms (138–562 ms) | median 0.010 ms (0.007–0.017 ms) | | Lease query execution, run 2 | median 185 ms (144–295 ms) | median 0.030 ms (0.020–0.045 ms) | | `reconcileBackup` wall time (includes file reads), runs 1 and 2 | median 188 ms and 216 ms | median 1.4 ms and 4.2 ms | | Plan | `Gather > Seq Scan` | `Bitmap Heap Scan > BitmapOr` of `heartbeat_run_idx`, the new index and `pkey` | | Buffers | 365,880 hit + 70,050 read | 9 hit | These numbers are from head `65bd9d7ac6`. The final head only renumbers the migration to 0459 after rebases; the SQL, the index and the service code are the same. Two earlier runs on the same fixture, SQL and index gave 236–425 ms without the index and 0.023–0.033 ms with the index. The machine had heavy load from other work, so the spread without the index is wide. Output check: in all four measurements, the service returned the same 5 lease IDs, and the complete query rows were equal. Index build (`CREATE INDEX CONCURRENTLY` from the migration file): 0.8–2.4 s for the 678 MB table over all runs. Index size: 3.4 MB with 30% of leases stamped. Tests: - `cd server && npx vitest run src/__tests__/native-harness-backup-expiration.test.ts`: 109 passed. The new test failed when I dropped the index before the plan check (confirmed red), and it passes with the index. - `pnpm --filter @paperclipai/db check:migrations`: passed. - `pnpm --filter @paperclipai/db typecheck` and `pnpm --filter @paperclipai/server typecheck`: passed. - `pnpm verify`: it asks for broader coverage (exit 2) because of the new script. See the CI results for this head. ## Risks - Migration time. A concurrent index build reads the full `environment_leases` table and does not block writes. Locally, the build took 0.8–2.4 s for 678 MB. The time grows linearly with the table size. I did not time it on a production-size clone. I expect a few seconds to tens of seconds on a large instance, which is in the 1–2 minute budget. This estimate is not proven. - If the build stops, PostgreSQL can leave an invalid index. The migration runner repairs invalid concurrent indexes when it runs again, and `IF NOT EXISTS` makes a rerun safe. - The index expression must stay textually equal to the query expression. If someone changes the JSON path, the planner cannot use the index. The new test catches this for custom and generic plans. - The index is partial, so ANALYZE does not collect statistics for the expression. The planner uses the default equality estimate. On the large fixture and in the small test, the planner still selects the index plan. A company with very few leases can still get a sequential scan, which is cheap. - Each lease write with a backup stamp writes one more small index entry. The write cost is small. - This change does not reduce how often the worker calls the lookup. It makes each call cheap. The other digest lookups in backup retention (`native-harness-backup-retention.ts`) and session retention checks can also use the index. - Previous-release story: no persisted state changes. The index is derived from existing lease metadata. - The hosted improvement is not verified until deployment. After deployment, compare a new query-statistics window for this statement. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, local PostgreSQL benchmarks). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude
|
7368a0b15c |
fix(recovery): re-queue native runs that fail on a transient database fault before provider work (#20561)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native (Paperclip Runner) runs prepare a task, pass a final dispatch
gate, and only then start a provider.
> - A native run can fail before any provider work starts. Examples are
a Postgres serialization or deadlock error (`40001`, `40P01`) at budget
admission, or a transient database error during setup.
> - Today such a run ends `adapter_failed` or `setup_failed`. `failRun`
writes the "no provider work" record only for legacy runs, and the
bounded retry paths do not retry these codes for native runs. The
release then opens the native hold
(`native_continuation_requires_reconciliation`), and a person must click
Retry.
> - The goal is that 0% of tasks need a Retry click after a restart or a
storage/DB fault. Duplicate provider work is worse than a Retry click,
so the fix must only act when the server can prove no provider work
started.
> - This pull request records that proof for native runs and sends a
transient database failure to the existing uncharged, bounded
`interrupted_preparation` retry lane.
> - The benefit is that these runs re-queue by themselves, and runs with
any sign of provider work keep today's hold.
## Linked Issues or Issue Description
**What happened?**
In restart-chaos waves with database pressure, a native run failed at
budget admission with exhausted serialization retries (`40001`/`40P01`).
The run event history had no `native.process_start_requested` event.
About 15 s later the server recorded "Automatic recovery stopped" with
cause `native_continuation_requires_reconciliation`. The task stayed
blocked until a person clicked Retry.
**Expected behavior**
A native run that failed before any provider work, on a fault that
clears by itself, re-queues automatically. The attempt is bounded and
does not spend the failure budget. A persistent failure still ends
failed, with a working Retry.
**Steps to reproduce**
1. Start a native (Paperclip Runner) task.
2. Make budget admission or a setup database write fail with SQLSTATE
`40001` or `40P01` until its bounded retries are exhausted.
3. See the run end `adapter_failed`/`setup_failed`, the task go to
`blocked`, and a `native_continuation_requires_reconciliation` recovery
action that waits for Retry.
**Paperclip version or commit**
`master` at
|
||
|
DottaandClaude
|
b5f62b3384 |
docs(rules): correct the scope of the API route and company-scope rules (#20596)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - `rules/*.mdc` holds the architecture rules that reviewers and agents apply to a change > - Pull request #20585 added two rules: API route expectations and company-scope exceptions > - Review found that both rules state more than is true for the current code > - A rule that is too broad makes reviewers flag correct code and can make an agent add a wrong check > - This pull request corrects the scope of both rules and fixes one stale pointer > - The benefit is rules that match the code they describe ## Linked Issues or Issue Description Refs #20585 **Issue type** Incorrect contributor documentation. **Where is the issue?** `rules/api-route-expectations.mdc`, `rules/company-scope-exceptions.mdc`, `AGENTS.md` section 8, and `.agents/skills/honeycomb-tracing/SKILL.md`. **What's wrong?** - The API route rule said that every route checks company access. Instance-level routes such as `health.ts`, `liveness.ts`, `ready.ts`, `public-mcp.ts` and `org-chart-svg.ts` do not, and must not. - The API route rule told reviewers to flag each status code outside a list of seven. `429` and `503` are correct in some routes. - The company-scope rule said that it lists the only instance-wide data. `user_sidebar_preferences` is instance-wide and was not in the list. - The company-scope rule said that it applies to new files that its `globs` cannot match. - The `honeycomb-tracing` skill pointed to `AGENTS.md` for the Telemetry workflow. That text is now in `rules/data-paths.mdc`. **Suggested fix** Limit each rule to what is true, and repoint the skill. ## What Changed - The API route rule and `AGENTS.md` section 8 now apply to routes that read or write company data. - The API route rule names the instance-level routes and says not to add a company check to them. - The status code list is now a list of common codes. Reviewers flag an incorrect code, not an unlisted code. - The company-scope rule lists the sidebar company order as an instance-wide exception, in the text, the `globs`, and the Greptile scope. - The company-scope rule says "known data" and no longer claims to match new files. - The `honeycomb-tracing` skill points to `rules/data-paths.mdc`. ## Verification - Read the two rules and confirm that each named file exists. A local script checked the `globs` and the route names. - Confirm that `.greptile/files.json` is valid JSON and that its scope matches the rule `globs`. - No source code changed. No test suite ran locally. ## Risks - Low risk. The change is documentation only. - The list of instance-level routes can go out of date when a route is added. ## Model Used - Claude (Anthropic), in Claude Code with tool use. - `claude-fable-5-2`. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
88c1500fac |
docs(agents): add a core rule that points to the module-boundaries rule (#20598)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - `AGENTS.md` section 5 holds the core engineering rules that apply to every change > - The section had no rule about which module owns a behavior > - The repository already has a repository-wide rule for this, `rules/module-boundaries.mdc` > - Agents see that rule only as one line in the list of rule headers > - This pull request adds a core rule that states the principle and links to that rule > - The benefit is that module ownership has the same weight as the other core rules ## Linked Issues or Issue Description Refs #20585 **Issue type** Missing contributor documentation. **Where is the issue?** `AGENTS.md` section 5. **What's wrong?** - The core engineering rules did not say that provider-specific behavior stays with the module that owns it. - The core engineering rules did not point to `rules/module-boundaries.mdc`. **Suggested fix** Add a core rule that states the principle and links to the existing rule. ## What Changed - Added rule 4 to `AGENTS.md` section 5, "Keep behavior with its owning module". The rule links to `rules/module-boundaries.mdc`. - An earlier commit on this branch added a copy of the pstack `principle-boundary-discipline` skill. A later commit removed it. The net change is `AGENTS.md` only. ## Verification - Open `AGENTS.md` section 5 and follow the link in rule 4. - Run `git diff origin/master --stat`. Only `AGENTS.md` changes. - No source code changed. No test suite ran locally. ## Risks - Low risk. The change is documentation only. ## Model Used - Claude (Anthropic), in Claude Code with tool use. - `claude-fable-5-2`. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
6ed6b0754b |
perf(recovery): probe recovery source runs and stale-continuation runs by key (#20584)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The workspace recovery worker saves the files of failed runs. After each pass, it reads the time of its next deadline (`nextWorkspaceRecoveryAt`) > - That read finds the source run, its finalizer and its leases with `id::text = evidence->>'runId'`. A text comparison cannot use a key. Each exhausted hold read every run, finalizer and lease of its company > - On a hosted staging database this statement used about 4 s per call (21 calls in the sample window). The cost grows with exhausted holds multiplied by the company's run count > - The ownership reconciler's stale-continuation check (`checkStaleDroppedComments`) has the run lookup shape that #20412 fixed in the lost-wake check: one OR across the legacy context and the native column > - This pull request compares the run id as a canonical uuid, and it gives the stale-continuation check the per-binding run probes from #20412 > - The benefit is the same recovery decisions with index-driven work: 11 s to 93 ms for the deadline read on a representative local fixture ## Linked Issues or Issue Description Refs #20412 **What happened?** The workspace recovery deadline read was one of the most expensive statements that remained on a hosted staging database after the fixes in #20393, #20405, #20386, #20412, #20391 and #20400. Query statistics showed about 4 s per call. The statement text starts with `select case when (issue_recovery_actions.…`. The statement evaluates the retained-source fallback and stop-proof conditions for each recovery action. These conditions read the source run (`heartbeat_runs`), its finalizer (`native_run_finalizations`) and its leases (`environment_leases`) with `x.id::text = evidence->>'runId'`. The cast prevents the use of the primary key. PostgreSQL reads all rows of the company for each exhausted hold. The worker's discovery (`retainedWorkspaceCandidates`) uses the same conditions. The stale-continuation check matched a task's runs with `(context_snapshot->>'issueId' = issue OR native_issue_id = issue)`. With the OR, PostgreSQL can join runs only by company. It also hashed every wake request of the company for the stale-wake test. **Expected behavior** Each condition uses a key lookup. The worker and the reconciler make the same decisions for the same saved state. **Steps to reproduce** 1. Seed a company with a deep run history and a few thousand workspace recovery actions. Let a few hundred be exhausted holds that stay active. 2. Call `nextWorkspaceRecoveryAt`. 3. The call takes seconds, and its time grows with holds multiplied by runs. The benchmark below shows this. **Paperclip version or commit** Baseline: `8d2bf78323`. **Deployment mode** Server with PostgreSQL. The local proof uses disposable PostgreSQL 18.1. Staging uses hosted PostgreSQL. ## What Changed - `workspace-recovery-fallback-discovery.ts`: the source run, finalizer and lease probes compare `id = canonicalExecutionUuid(evidence->>'runId')`. The helper returns null unless the text is the canonical lowercase uuid form. Thus the probe matches exactly the rows that `id::text = evidence->>'runId'` matched. The helper already exists in `execution-identity.ts`. - `nextWorkspaceRecoveryAt` and worker discovery use these conditions. Their SQL changes only in these probes. - `checkStaleDroppedComments` reads in two stages. The first stage keeps comments that a stale-continuation cancellation dropped and that no open wake carries. The second stage applies the run probes. The run probes use the `taskRuns` helper from #20412, which is now shared by both checks. The stale-wake lookup ends with `offset 0`, so it stays a per-comment index probe. - The final `order by c.created_at limit N`, the projection and the delivery check after the query do not change. - Add `scripts/benchmark-recovery-status-checks.ts`. It loads the changed files from an explicit baseline commit and from the working tree. Before it records time, it compares every changed condition for every recovery action, the complete service results and the positional SQL rows. - Add tests: each run binding in the stale-continuation check (native, legacy, live run on either binding, wrong agent, wrong company, failed run, run outside the comment time, closed by a person, open wake). Also a test that a hold with a non-canonical saved run id is not discovered, as before. - Document both checks and the benchmark in `doc/DATABASE.md`. - No schema change, index, migration, or new startup work. ## Verification - `pnpm exec vitest run src/__tests__/ownership-reconciliation-report.test.ts src/services/workspace-recovery-fallback.test.ts src/services/workspace-recovery-stop-proof.test.ts src/__tests__/heartbeat-held-comment-wake.test.ts --maxWorkers=1` (from `server/`). - `pnpm --filter @paperclipai/server typecheck`. - `pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-recovery-status-checks.ts --baseline-ref=8d2bf78323 --samples=7` Benchmark setup: PostgreSQL 18.1 (embedded, local), `shared_buffers` 128 MB, `work_mem` 4 MB. All data is synthetic, with md5-hex filler. Each fixture has two companies (full size and a quarter). Two warmup rounds, then 7 timed samples for each arm, in alternating order. Each sample runs the real service function and asserts the same result. These are local SQL measurements on a shared machine, so the spread is wide. Workspace recovery deadline (`nextWorkspaceRecoveryAt`). About 8% of actions are exhausted holds that stay active. Some actions save a non-canonical (uppercase) run id. Before timing, the benchmark compares six conditions (fallback pending, due time, due; re-arm pending, due time, due) for every action of the fixture. | Fixture | Runs / actions / pending holds | Baseline median (min–max) | Treatment median (min–max) | Shared buffers | | --- | --- | --- | --- | --- | | small | 6,500 / 255 / 36 | 50.9 ms (47–76) | 6.8 ms (6–8) | 19,133 → 1,075 | | representative | 78,750 / 3,813 / 588 | 11,098 ms (8,277–12,790) | 93 ms (35–145) | 3,493,417 → 24,902 | | deep history | 191,250 / 3,813 / 588 | 40,146 ms (31,190–52,521) | 40 ms (36–230) | 13,491,792 → 26,911 | Stale-continuation check (`checkStaleDroppedComments`). The fixture is pessimistic: more than half of recent comments on done tasks have a stale cancelled wake. The table shows statement time from `EXPLAIN ANALYZE` with default settings, and one statement run with `enable_nestloop = off` (the company-only hash plan). The result is checked as equal in both modes. | Fixture | Runs / recent comments / reported | Statement, baseline → treatment | Hash plan, baseline → treatment | | --- | --- | --- | --- | | small | 3,792 / 76 / 24 | 3.2 → 1.5 ms | 5.5 → 3.8 ms | | representative | 75,406 / 1,125 / 200 (limit) | 34 → 26 ms | 66 → 25 ms | | busy hour | 152,031 / 5,625 / 200 (limit) | 328 → 153 ms | 2,302 → 165 ms | - Neutral path: the complete stale-continuation service call stays about the same (busy hour 750 → 655 ms median). The per-comment delivery check after the query (`deliveredQueuedCommentIds`) is most of that time. This change does not touch it. ## Risks - Low risk. Every predicate keeps the same logic. Only the probe shape and the evaluation order change. The benchmark compares each changed condition for every recovery action, including non-canonical saved run ids, and the complete results of both checks. - Previous-release story: no saved state, schema or identity changes. Holds saved by older releases are read with the same rules, so they carry forward unchanged. - Rows with the same comment `created_at` have no defined order, as before. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, local PostgreSQL). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1110.0-canary.15 |
||
|
DottaandClaude Opus 5.5
|
be0d100a8b |
fix(native): settle cleanup-gate refusals after a completed turn without a dead-end Retry (#20572)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs execute a provider turn, then close the provider session
and prove it stopped
> - When close cannot prove that the old runner, provider and tools
stopped, the runner holds a cleanup gate
(`native_session_cleanup_quarantined`) that refuses the next session on
that task
> - A run finished its provider turn and saved its work. A server
restart had left one tool callback without a saved result, so close
could not prove the drain, and the gate refused the run's own same-run
recovery 30 seconds later
> - The task then showed "Couldn't finish this attempt" with a Retry
button, but the retry route refuses this error code with 409. The saved
error text told an operator to act
> - This pull request makes the task projection, the execution blocker,
the executor and the retry route agree, and replaces the operator text
with plain wording
> - The benefit is that finished work no longer looks like a failed
attempt, and the UI never offers a Retry that the server refuses
## Linked Issues or Issue Description
**What happened?**
A remote native run completed its provider turn and saved its comments
and documents. A server restart during the turn left one semantic tool
callback without a saved result. Session close then reported
`native_session_settlement_incomplete` (runner suspended, provider drain
unproven), and the runner kept an operator-required cleanup gate. The
same-run disposition recovery was refused at admission, and the run
ended `failed` with `native_session_cleanup_quarantined`.
**Expected behavior**
The task should show its real state (here, waiting on a child task). It
should not show a failure banner or a Retry that the server refuses. No
text should ask an administrator to act.
**Steps to reproduce**
1. Start a remote native run and restart the server while a tool
callback runs.
2. Let the resumed turn complete without a semantic result, so a
same-run recovery is scheduled.
3. Observe the refusal at admission, the "Couldn't finish this attempt"
banner, and a Retry that returns 409.
**Paperclip version or commit**
master at
|
||
|
DottaandClaude Opus 5.5
|
d82e5efeb9 |
perf(ui): cut repeated reads while a task is open (#20563)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The board UI keeps an open task current with a shared company
websocket and with polling fallbacks
> - In a busy company, one open task sent 35 to 60 API requests in 17
seconds, and each request waits on the database
> - Most of these requests were repeated reads: task summaries 22 times,
the task run list 11 times, task browsers 6 times, and the 60 KB agent
list 6 times
> - The cause is not the open task. Other agents' run and task events
invalidate company-wide lists on every 300 ms batch, and the task view
polls runs and browsers on fixed intervals even while the websocket is
open
> - This pull request spaces company-wide refreshes, and it lets the
websocket replace run and browser polling while it is connected
> - The benefit is about 79% fewer reads from an open task in a busy
company, with the same polling fallback when the connection is down
## Linked Issues or Issue Description
Refs #17608
Refs #9627
**What happened?**
An open task in a busy company sent the same reads again and again. In
one 17-second visit, the UI requested `issues/summaries` 22 times, the
task live runs 11 times, the task browsers 6 times, and the company
agent list 6 times. The company live runs and the task's child lists
(`parentId`, `createdFromIssueId`, `descendantOf`) were requested 2 to 3
times each.
**Expected behavior**
A busy company must not make an open task refetch company-wide lists on
every event batch. While the live connection is open, the UI must use
events, not fixed fast intervals, for task runs and browsers. The UI
must keep its polling fallback when the connection is closed.
**Steps to reproduce**
1. Open a task in a company where other agents run work.
2. Keep the task open for 17 seconds.
3. Count the API requests in the browser network panel.
**Paperclip version or commit**
Based on master at
|
||
|
DottaandClaude Opus 5.5
|
8e332ccb5e |
fix(evals): name stale SDK pins and fix four chat-eval harness defects (#20579)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat feature eval catalog (`tests/runner-e2e`) runs real chat journeys against a real server and grades the saved state, the run's tool events and the rendered reply > - The full-cohort run on master `f4337bb5fb` had 12 failures that come from the harness, not from the product or the model > - Six plugin and secrets cells stopped in 7–9 s, before the server started, because the SDK public source pins were stale > - Four more cells failed on harness timing, a first-visit prompt, or a read receipt that ignored a dedicated tool > - This pull request fixes those four harness defects and adds a cheap guard that names stale SDK pins > - The benefit is that these cells now report on the product and the model again ## Linked Issues or Issue Description Refs #19614, #20550, #20453, #20417, #20483 **What happened** On master `f4337bb5fb`, these chat eval cells failed for harness reasons: - `plugins.configure-plugin`, `plugins.disable-plugin`, `plugins.install-plugin`, `plugins.use-plugin-tool`, `routines.plugin-managed-routine` and `secrets.least-scope-delivery` stopped before the server started. The source SDK privacy scan refused, because `tests/runner-e2e/source-sdk-public-inputs.json` did not match `packages/shared/src`. #20550 and #20362 already resynced the pins on master. The pins are current at `ce1ee9e89f`. - `agent-chat.track-delegated-work` and `navigation-preferences.edit-profile`: the reply was correct and had the right link. The case read the rendered links immediately after the runs settled, before the reply rendered, so it recorded no links. - `inbox-search.archive-done` and `inbox-search.restore-archive`: setup completes five tasks, so the GitHub star prompt from #20453 opened over the chat and blocked the composer. - `status-cards.explain-card`: the follow-up run read the deleted source with `get_task` and received a 404. The reply said so. `apiReadStatuses` counted only `call_api` reads, so the grader recorded no unavailable read. Why the existing pin check did not stop the drift: `ci / Runner support suite` runs the source SDK tests and is part of the required `ci / verify`. On #19614 it passed at 03:43Z on a merge ref from before #20417 (06:12Z) and #20483 (08:14Z). #19614 merged at 14:02Z without a new CI run on current master. After that, the suite failed on every PR until #20550. So the check exists and fires; the gap was a stale green result, and the failure message was opaque (`SDK public source proof unavailable`). **Expected behavior** - A change to a pinned SDK input without a pin refresh fails with the changed files and the command that fixes it. - Cases read rendered links only after the saved reply renders. - The eval browser is a returning user's browser that has already seen the star prompt. The product does not change. - A dedicated task read tool (`get_task`, `list_task_documents`, `read_task_document`, also when namespaced) counts as a read of the record it names, with the status its receipt reports. **Steps to reproduce** 1. Run `chat-features.runner-opencode.local.<case>` for one of the cases above on master `f4337bb5fb`. 2. Read `result.json` and `snapshots/chat-catalog-observation.json` in the evidence bundle. 3. See the failures described above. ## What Changed - `tests/runner-e2e/source-sdk-public-inputs.mjs`: add `staleSdkPublicInputs(root)`. It compares the registry with `git ls-tree HEAD` for every pinned input (the roots and build inputs that the pins file lists), and returns each changed, unpinned or removed file. - `tests/runner-e2e/source-sdk-public-inputs.test.mjs`: a test fails when the committed pins do not match the committed tree. The message names the files and gives the `--write` command. The Runner support suite runs it for every `packages/**` change. A synthetic test covers changed, unpinned and removed inputs. - `tests/runner-e2e/chat-catalog-domains/runtime-handoffs.ts` and `identity-profile.ts`: call the existing `waitForRenderedReply` before they read rendered links. - `tests/runner-e2e/chat-returning-browser.ts` (new) and `chat-catalog-flow.ts`: inside the flow's failure boundary, before setup, mark `paperclip.github-star-prompt.seen` in the eval browser. An init script covers later documents of the app origin, and the open page is marked directly. A test checks that the key equals `STAR_PROMPT_SEEN_KEY` in `ui/src/components/GitHubStarPrompt.tsx`. - `tests/runner-e2e/chat-catalog-domains/api-read-receipts.ts`: bind dedicated task read tools by the digest of the `call_api` input that the server builds for them (`taskBridgeApiRequest`). The result must carry the same `apiOperationId`. Other tools, other records, mismatched receipts and unpaired results do not count. - Bump `oracleVersion` in code and descriptor for every case that uses `api-read-receipts.ts` (shared grader change) and for `navigation-preferences.edit-profile`. Add `internal-comment-reference.ts` to the source files of the two cases that now import it. ## Verification - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts` (whole config): 11301 passed; 1 local-only failure in `controller-replacement-smoke.test.ts` (server spawn, no changed imports). - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts --testTimeout=180000 tests/runner-e2e/chat-catalog-domains tests/runner-e2e/chat-catalog-contract.test.ts tests/runner-e2e/chat-catalog-consistency.test.ts tests/runner-e2e/chat-returning-browser.test.ts` - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts source-sdk` - `node tests/runner-e2e/source-sdk-public-inputs.mjs $(git rev-parse HEAD) --check` reports `changed: false`. - Red checks: with the `f4337bb5fb` registry, the new pin test fails and names 15 inputs. Without the new waits, the two rendered-reply tests fail. A regrade of the retained `explain-card` run events now gives the follow-up run `unavailable: true, live: false` and the first run `live: true`. - Paid runs of the affected cells (`chat-features.runner-opencode.local.*`): - Run 42852698879 (head `d54c04d5f0`): `track-delegated-work`, `restore-archive`, `explain-card`, `configure-plugin` and `plugin-managed-routine` pass. I read each reply and saved state. `track-delegated-work` now records the rendered `/COM/issues/COM-2` link. `explain-card` records `laterUnavailable: true, laterLive: false` from the follow-up `get_task` 404. - `plugins.install-plugin` now starts its server (the pin refusal is gone). It fails a later grading check, `truthful-installation-handoff`: the reply sends the person to the plugin settings page but does not name the package. This is outside this PR. - `inbox-search.archive-done`: the star prompt no longer blocks the composer, and the query is sent. The agent then archives the 501 Done tasks one at a time (about 2 s each, 57 in the 240 s window), because only the per-task archive route exists. This is outside this PR. - `navigation-preferences.edit-profile` did not start in that run (the AWS credentials step timed out). Run 42857020103 (head `2f0b80aa84`): pass. The agent write gets 401, the reply links `/COM/company/settings/instance/profile`, and the person follows the rendered link and saves the name. ## Risks - Low risk. Test harness only; no product code changes. - The receipt change counts more reads as record reads. It only adds dedicated reads bound to the exact record id and operation, so a grader that needs a read can pass on a genuine `get_task` read. The affected cases have new oracle versions. - The star prompt is not tested by any chat catalog case. Product e2e tests for the prompt do not use this flow. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file editing, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude
|
84007892e9 |
refactor: move AGENTS.md section detail to docs and rules, add a bug-fixing section (#20585)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - `AGENTS.md` is the instruction file that each human and AI contributor reads before a change > - Pull request #20557 shortened the file, but several sections still held detail that applies to only some changes > - The scoped-rules section told agents to read each matching rule in full, and most changes match five rules > - The core engineering rules mixed three product invariants with process conventions > - This pull request moves that detail to the docs and to scoped rules, and adds a section on how to fix a bug > - The benefit is a 211-line file that states what applies to every change and points to the rest ## Linked Issues or Issue Description Refs #20557 **Issue type** Outdated or unclear contributor documentation. **Where is the issue?** `AGENTS.md` at the repository root, and `rules/`. **What's wrong?** - The scoped-rules section made agents read the full body of each matching rule before a change. The rule bodies total about 47 KB. - Sections 4, 6, 8, 9 and 10 repeated detail that belongs in `doc/DEVELOPING.md`, `doc/DATABASE.md`, a route rule, `DESIGN.md` and `CONTRIBUTING.md`. - Section 4 had a database reset command for `data/pglite`. That directory does not exist. - Section 5 had seven rules. Four of them were process conventions or path-specific detail. - The file did not say how to fix a bug. **Suggested fix** Make the rule headers the index. Move section detail to the document or rule that owns it. Keep only the three invariants in section 5. Add a bug-fixing section. ## What Changed - The scoped-rules section now tells agents to list the rule headers with one `grep` command. Agents read a rule body only for a design decision in its area, when the description is not sufficient, or when a reviewer cites it. - `rules/README.md` says that each `description` must state the required behavior in one sentence. - `rules/pull-request-review.mdc` has a description that states its requirements. - Section 4 (dev setup) and section 11 (pull requests) now point to `doc/DEVELOPING.md` and `CONTRIBUTING.md`. - The schema change workflow moved from section 6 to `doc/DATABASE.md`. - The API and auth expectations moved from section 8 to the new `rules/api-route-expectations.mdc`. - Section 9 links to `DESIGN.md` for design rules. `AGENTS.md` no longer tells contributors to run `pnpm check:token-gates`. - Added section 10, "Fixing a Bug": reproduce the error, write a failing test, write the fix, see the test pass, and verify the fix. - Section 5 now has three rules: company scope, contract synchronization, and the control-plane invariants. - The company-scope exceptions moved to the new `rules/company-scope-exceptions.mdc`. - Removed the strategic-docs rule. The plan-doc convention moved to the repo map. The artifact rule moved to section 7. - Removed rule 7. Five code comments that cited `AGENTS.md §5.7` now cite `rules/data-paths.mdc`. - Regenerated the SDK public source pin, because one of those comments is in `packages/shared/src/crash-points.ts`. - Added both new rules to `.greptile/files.json`. - The API route rule applies only to routes that read or write company data. Its status code list is a list of common codes. - The company-scope rule lists the sidebar company order as an instance-wide exception. - The `honeycomb-tracing` skill now points to `rules/data-paths.mdc` for the Telemetry workflow. ## Verification - Open `AGENTS.md` and confirm that each link resolves. A local script checked this. - Run `grep -E '^(description|globs|alwaysApply):' rules/*.mdc` and confirm that each rule prints a description. - Run `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`. It passes locally. - `packages/shared/src/crash-points.test.ts` passed locally before the rebase onto `master`. - Five source files changed in comments only. No other test suite ran locally. ## Risks - An agent that reads only rule descriptions can miss a detail in a rule body. Reviewers still apply the full rules. - Pull requests and the definition of done are now sections 11 and 12. Sections 1 to 9 keep their numbers. - Dated plan documents under `doc/plans/` still cite `AGENTS.md §5.7`. They are historical records. - Pull request #20565 also changes `AGENTS.md`. The second one to merge needs a conflict fix in section 1 and section 2. ## Model Used - Claude (Anthropic), in Claude Code with tool use. - `claude-fable-5-2`. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
d0c30e69a9 |
fix(storage): abort a snapshot upload a restart cut off once its save attempt is superseded (#20542)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Remote workspaces save their files as encrypted snapshots in object
storage. Snapshots over 16 MiB use S3 multipart uploads.
> - A server restart can cut off a snapshot upload in the middle of an
`UploadPart` request.
> - The server aborts a multipart upload only in-process, on an error or
an abort signal. A SIGKILL can never do that. On SIGTERM, the drain step
timed out while the store held the part, and the process exited without
`AbortMultipartUpload`.
> - The incomplete upload then stays in the bucket. Nothing records its
upload id and nothing finds it again, so incomplete uploads pile up and
the operator pays for their parts.
> - This pull request aborts the incomplete uploads of each save attempt
when that attempt can no longer win, with no polling and no schema
change.
> - The benefit is that no incomplete upload outlives its save attempt.
## Linked Issues or Issue Description
**What happened?**
A restart-chaos wave (Daytona, R1 workload, storage fault ST4,
`--storage-trigger snapshot-part`, faults K1 and K2, master
`fea4530a30`) passed all 3 tasks. All 3 committed snapshots read back
intact. But the storage audit found 3 incomplete multipart uploads. Each
one was the exact upload that a restart cut off in the middle of
`UploadPart`. The retried save then published a different attempt of the
same snapshot.
**Expected behavior**
When a save attempt can no longer become the snapshot's object, its
incomplete multipart uploads are aborted. The storage audit shows 0
incomplete uploads after the wave.
**Steps to reproduce**
1. Run a snapshot save larger than 16 MiB against S3.
2. Kill the server (SIGKILL), or send SIGTERM while the store holds an
`UploadPart` request.
3. Let the save retry and publish.
4. List the bucket's multipart uploads. The cut-off attempt's upload is
still there.
**Paperclip version or commit**
master `fea4530a30` and later.
**Relevant logs or output**
`shutdown step timed out; continuing shutdown without it
{"step":"delivery_work_stop","durationMs":20001}`, then `Paperclip did
not stop within 30000ms; sending SIGKILL`.
## What Changed
- The S3 storage provider and the storage service get
`abortIncompleteUploads(objectKey)`. It lists the multipart uploads
under that exact key (`ListMultipartUploads` with the key as the
prefix), skips other keys that only share the prefix, and aborts each
one. `NoSuchUpload` counts as already done. The local-disk provider has
no multipart uploads and does nothing.
- Save publication: the save journals every attempt's object key in
`metadata.uploads` before any remote I/O. When one attempt moves the
snapshot from `pending` to `uploaded`, no other attempt can win any
more. The winning process now aborts the incomplete uploads of every
other journaled attempt key. Each abort is best effort and bounded to 5
s. A snapshot with one attempt sends no extra request.
- A losing uploader that is still live: if its upload fails after
another attempt has published the snapshot, it now returns the published
snapshot (after it verifies it) instead of the error. So an abort from
the winner never fails a save that is still running in another process.
- Snapshot retention: before it deletes a retired snapshot's objects
(this includes saves that #20502 retires as abandoned), retention aborts
the incomplete uploads of each attempt key. A store that refuses the
listing (for example a missing `s3:ListBucketMultipartUploads`
permission) logs a warning and does not block the deletion.
- Abandoned saves: when the ownership reconciler retires an abandoned
save (`remote_snapshot_save` and `remote_snapshot_ledger` settlement,
from #20502), it now aborts the incomplete uploads of every journaled
attempt key after its transaction commits. The owner run is terminal and
the save work is settled, so no live uploader can own those uploads.
This is the durable path for SIGKILL: the reconciler pass after boot
settles the dead save and cleans its uploads.
- SIGTERM drain: the multipart upload code keeps an in-process set of
running uploads. Shutdown aborts them right away when
`delivery_work_stop` times out. It also aborts them after the heartbeat
run drain, before the finalizer and workspace-recovery waits, after a 5
s grace for them to finish. It waits at most 10 s for the
`AbortMultipartUpload` calls. Shutdown does nothing extra when no upload
runs.
- No polling and no periodic sweep. All triggers are events that already
exist: publication, abandonment settlement, retention of a retired
snapshot, and shutdown.
Previous-release story (`rules/upgrade-safe-run-state.mdc`): this adds
no persisted state and changes no saved shape. A snapshot that the
previous release saved with cut-off attempts in `metadata.uploads`
carries forward. Its incomplete uploads are aborted when the new release
publishes a retry of that snapshot, or when retention retires the
snapshot.
## Verification
- Red before, green after (all fail on master, pass with this change):
- `server/src/services/remote-workspace-snapshot-storage.test.ts`:
"aborts the incomplete upload of an attempt a restart cut off once a
retried save publishes" and "adopts the published save when another
uploader aborts this attempt's upload".
- `server/src/services/remote-workspace-snapshot-retention.test.ts`:
"aborts each attempt's incomplete multipart upload before deleting a
retired snapshot's objects". A second test proves that a refused listing
does not block retention.
- `server/src/__tests__/storage-s3-multipart-shutdown.test.ts`: file and
streamed uploads whose part the store holds are aborted at shutdown, and
the provider aborts only the exact key across listing pages.
- `server/src/__tests__/remote-snapshot-ledger-settlement.test.ts`:
"aborts the incomplete multipart uploads of every attempt when the
reconciler retires an abandoned save" (both settlement kinds; a key
outside the snapshot's prefix is skipped).
- `tests/restart-chaos/storage-orphan-uploads.test.ts` (MinIO): the real
provider aborts both cut-off uploads of one attempt key, and keeps a
sibling attempt and a longer key that shares the prefix.
- `cd server && npx vitest run src/__tests__/storage-s3 src/storage
src/__tests__/remote-snapshot-ledger-settlement.test.ts
src/services/remote-workspace-snapshot
src/services/remote-workspace-transport.test.ts
src/__tests__/remote-workspace-file-resources.test.ts`: 19 files, 517
tests pass; with `src/services/recovery`, `src/__tests__/ownership*` and
`heartbeat-workspace-busy.test.ts`, 32 files and 580 tests pass.
- `cd server && npx tsc --noEmit` and `node
scripts/typecheck-test-projects.mjs tests/restart-chaos` pass.
- Live proof, Daytona R1, ST4, `--storage-trigger snapshot-part`, K1/K2,
seed 1033, the evidence wave's flags, image
`paperclip-daytona-runner@sha256:05a4a092…` (controllerMatch: match):
- Before (master `fea4530a30`): `w-20261110T130541-s1033`: 3/3 pass, **3
incomplete multipart uploads**.
- `w-20261110T134623-s1033` on an earlier revision of this PR (publish +
retention hooks only): 2 incomplete uploads remained. Those saves were
abandoned, not retried, so neither hook ran. This led to the settlement
hook.
- After (head `8bc8fc66d0`): `w-20261110T141144-s1033`: 3/3 tasks pass,
exit 0, every fired restart proved it cut the held `UploadPart`, the
reconciler settled 3 abandoned saves, shutdown logged one in-flight
abort, and the storage audit shows **0 incomplete multipart uploads**
and 3 committed snapshots that read back intact.
## Risks
- A live uploader in another process can lose its incomplete upload when
a different attempt publishes first. That uploader now adopts the
published snapshot, so its save still succeeds.
- Narrow gap: an uploader that recorded its attempt but had not yet sent
`CreateMultipartUpload` when the winner published, and that then crashes
during its upload, leaves an incomplete upload until retention retires
the snapshot. An S3 lifecycle rule for incomplete uploads stays a good
backstop.
- Stores without `ListMultipartUploads` permission log a warning and
keep the previous behavior.
- Shutdown can take up to 10 s longer, and only when an upload is still
running after the drain.
## Model Used
Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
Nicky LeachandPaperclip
|
2e3cbe57a4 |
fix(native): preserve parked Runners across controller restarts (#20586)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native execution parks a released sandbox Runner for a later turn. > - The warm idle timer detaches Core and leaves Runner retention to the host. > - Controller shutdown instead stops recently parked Runners through checkpoint cleanup. > - This makes reuse depend on whether the idle timer fired before shutdown. > - This pull request uses the same detach operation for idle expiry and shutdown. > - The benefit is consistent Runner retention without a new retention or recovery mechanism. ## Linked Issues or Issue Description Fixes: #20012. Related: #20005 unified cached and adopted process reuse. Closed #19975 included this shutdown change within a broader retention proposal. ## What Changed - Detach confirmed released entries during controller shutdown. Keep busy, unreleased, and close-marked entries on the existing cleanup path. - Share the detach operation with idle expiry and cached-owner handoff. - Record that a local handle has started detaching. Later cleanup cannot fall back to stopping its Runner after a partial detach failure. - Preserve explicit environment deletion, release proof, admission, and process-reuse checks. - Add Codex and Claude restart, detach-failure, and environment-deletion regressions. Add real Runner adoption cases after the restart sweep, including managed GitHub rebinding. - Document the lifetime rule and make the shutdown warning cover both detach and checkpoint failures. ## Verification - Final source revision: `e4bb6ecbd7`. Local environment: macOS arm64, Node 26.12.0. - New controller-restart and detach-failure tests fail against unchanged master: 4 expected failures. Old shutdown calls close or reports success instead of reporting detach failure. - Focused executor tests: 22 passed. - Full native session executor suite: 1,328 passed. - `cd server && node scripts/typecheck-layers.mjs`: passed all source and test layers. - `node scripts/check-module-boundaries.mjs` and `git diff --check`: passed. - Manual diff review and added-line credential/PII scan: passed. - `cd server && node ../node_modules/vitest/vitest.mjs run src/services/native-runtime/plugin-runner-release-resume.integration.test.ts --bail=1 --reporter=verbose`: 14 passed on this revision. This includes new Codex and Claude restart cases and managed GitHub rebinding. - Failure history: an earlier concurrent run had 4 passes and 10 failures. Its first failure was the existing Claude cached case exceeding 60 seconds. Later cases encountered a leftover prototype spy, recursive calls, and startup timeouts. No source changed before the passing full rerun. A clean worktree of unchanged master (`e01fe7f84a`) passed all 11 original cases. The first timeout's exact cause remains unknown; the later pass does not establish that cause. - Standard Greptile review: 5/5 on `e4bb6ecbd7bd965b5f56507d2c42c580e8a1d547`, with no actionable findings. - `node scripts/local-verify.mjs --explain` returned exit 2 because the execution-semantics document requires broad CI. Required CI will supply broad typecheck, test, and build coverage. - A hosted deployment and an actual controller process replacement have not been run. The added integration cases exercise the shutdown sweep and a new adopting transport against a real Runner. ## Risks - SIGINT and SIGTERM preserve confirmed parked Runners, including ordinary service stops. The host still owns eviction and retention. - Detach failure retains a fenced local owner and reports failure. Core does not infer permission to stop the Runner from that failure. - Reuse remains conditional on current admission, process configuration, and host availability. - No persisted schema, checkpoint format, launch identity, or migration changes. State from the previous release uses the existing adoption or verified restore paths. ## Model Used - OpenAI Codex, GPT-6, with reasoning, repository inspection, tool use, and local test execution. The exact deployment identifier and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
765ccd8691 |
perf(chat): index chat-admission checks by company and payload issue id (#20562)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Chat connections save each external message as a `chat_actions` row
before the server admits it as an agent wakeup.
> - The server checks if an issue holds chat input that it did not
admit. Recovery skips such an issue. Manual task start refuses it.
> - On a hosted instance, the recovery read with this check used the
most total database time: 488 calls at about 1 s each. The manual-start
check ran 155 times at about 0.9 s.
> - The check matches `chat_actions.payload->>'issueId'` against the
issue. No index covers this value, so each call reads the full
`chat_actions` table and its payloads.
> - This pull request adds one concurrent partial index for the check.
The check SQL does not change.
> - The benefit is less background database load, so request reads wait
less. The results stay the same.
## Linked Issues or Issue Description
**What existing behavior does this improve?**
The chat-admission check `unadmittedChatWakeupCondition`. Recovery uses
it in the per-issue candidate read. Manual task start uses it in the
held-input check.
**Subsystem affected**
Server chat admission, issue recovery, manual task start, and the
database schema.
**Current behavior**
The check finds `inbound_wakeup` actions by `company_id` and
`payload->>'issueId'`. Two nested checks find `failed_run_retry` actions
and later `inbound_wakeup` actions on the same company and payload issue
ID. `chat_actions` has no index on `company_id` or on the payload issue
ID. Each call reads the full table and reads each payload. On a hosted
instance, the two statements used about 630 s of database time in one
statistics window.
**Proposed behavior**
The check and its nested checks read a small set of rows from one index.
The admission result does not change.
**Reason and benefit**
The hosted database had many concurrent active queries. Request reads
waited behind background work such as this check.
**Breaking changes**
None. The migration adds an index only.
## What Changed
- Migration `0458_chat_admission_issue_index` builds
`chat_actions_company_issue_kind_idx` with `CREATE INDEX CONCURRENTLY IF
NOT EXISTS`. It runs through the existing top-level statement path,
which repairs an invalid index on retry.
- Keys: `(company_id, payload->>'issueId', kind, status)`.
- Partial: `kind IN ('inbound_wakeup', 'failed_run_retry')`.
- One index serves the outer input lookup, the failed-run retry lookup,
and the later-input lookup. All three match on the same company and
payload issue ID.
- The Drizzle schema declares the index. The snapshot chain moves to
`0458` and keeps five snapshots.
- New regression test
`server/src/__tests__/chat-admission-condition.test.ts` pins the
admission semantics on real PostgreSQL:
- Pending and failed input without a receipt holds the issue. Received,
processed and cancelled input does not. Other action kinds do not.
- A skipped receipt, a receipt for another agent, or a receipt in
another company does not admit input. A failed or cancelled run still
counts as admission.
- The issue ID is compared as company-scoped text. Uppercase, array,
null, other-issue and other-company values do not match.
- Only an admitted retry in the same conversation and issue that lists
the action releases skipped input.
- Only later admitted input in the same conversation, issue and session
generation releases a failed input. Pending input stays held.
- The check reads `chat_actions` only through the new index.
- `scripts/benchmark-chat-admission-index.ts` measures both caller
statements with and without the index and fails if the rows differ.
- `scripts/benchmark-chat-admission-index-migration.ts` times the
migration through the normal migration runner.
- `doc/DATABASE.md` documents the index and the benchmark commands.
## Verification
Local SQL measurements on synthetic data. PostgreSQL 18.1 (embedded),
default settings. The fixture has 100,995 `chat_actions` rows (662 MB):
60,000 GitHub actions with 6 KiB incompressible payloads, 2,000 issues
with 20 admitted inbound messages each (3 KiB payloads), and 1,000
failed-run retries. Each case has 2 warmups and 9 samples of `EXPLAIN
ANALYZE` execution time. Runs alternate: no index, index, no index,
index.
| Caller statement | Case | No index, median (range) | Index, median
(range) |
| --- | --- | --- | --- |
| Manual-start held check | quiet issue | 3,093 ms (2,150–3,519) / 3,586
ms (2,375–11,235) | 0.036 ms (0.034–0.052) / 0.040 ms (0.033–0.047) |
| Manual-start held check | admitted input | 3,555 ms (3,310–8,080) /
5,205 ms (3,353–9,525) | 0.049 ms / 0.052 ms |
| Manual-start held check | unadmitted input | 3,002 ms (2,864–3,221) /
4,179 ms (3,200–6,585) | 0.051 ms / 0.044 ms |
| Manual-start held check | failed, later admitted | 3,586 ms / 3,722 ms
| 0.066 ms / 0.054 ms |
| Manual-start held check | admitted retry | 2,733 ms / 4,953 ms | 0.049
ms / 0.053 ms |
| Recovery candidate read | quiet issue | 43 ms (41–48) / 96 ms (44–219)
| 0.047 ms / 0.055 ms |
| Recovery candidate read | admitted input | 203 ms (48–575) / 174 ms
(44–314) | 0.055 ms / 0.058 ms |
| Recovery candidate read | unadmitted input | 66 ms / 144 ms | 0.053 ms
/ 0.054 ms |
- Buffers per call fall from about 40,000 shared hits to 5–23.
- Locally, the recovery read without the index uses a hashed full scan
of `chat_actions`. The held check uses a full scan per call. On the
hosted instance both took about 1 s per call. The local fixture is
smaller than the hosted table.
- Correctness: complete positional rows match across all four runs for
all six cases (quiet, admitted, unadmitted, failed then admitted,
retried, other-company decoy). The script asserts the expected held
result for each case.
- Migration timing: 600,001 `chat_actions` rows (1,759 MB), 300,000 of
them in the index scope. First run through `applyPendingMigrations`: 4.2
s. Rerun after the history row is removed: 0.5 s. Concurrent build
alone: 3.4 s. Index size: 30 MB.
Commands:
```sh
pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-chat-admission-index.ts
OTHER_PAYLOAD_KIB=2 pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-chat-admission-index-migration.ts
cd server && npx vitest run src/__tests__/chat-admission-condition.test.ts # 6 passed
pnpm --filter @paperclipai/db check:migrations # passed
pnpm --filter @paperclipai/server typecheck # passed
```
## Risks
- The migration builds one index concurrently. It does not block writes.
On the 1.76 GB local fixture it took about 4 s. A larger hosted table
takes longer, but the build does not lock out reads or writes.
- Writes to `inbound_wakeup` and `failed_run_retry` actions maintain one
more index. Status updates on these rows were already not HOT updates,
because other partial indexes use `status`.
- The planner has no expression statistics for a partial index. The new
test pins that the check uses the index on an analyzed table with
unrelated rows.
- The check SQL does not change, so admission behavior does not change.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use and code execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
96ab5e3d38 |
fix(recovery): read a retained Runner from its own allocation and continue unverifiable no-replay holds from the last save (#20567)
<!-- Write all pull request text in Simplified Technical English (ASD-STE100): short sentences, one instruction per sentence, simple approved vocabulary, and the active voice. --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native warm runs on a remote sandbox continue a saved session in the next run. The next run first reads the retained Runner lifecycle from the sandbox. > - #20346 abandons the workspace epoch of a run that died during preparation. The next run then restores the last save into a new directory of the same sandbox. > - The retained-sandbox binding check required the old and the new directory to be the same. The warm follow-up failed with `native_controller_recovery_missing_evidence:retained_sandbox_binding` before provider work. > - The failed run's no-replay hold then refused its files with `allocation_ownership_ambiguous`. The task stayed blocked. The text said that Retry "succeeds only after Paperclip is updated", so no click could help. > - This pull request binds the Runner read to the directory that the retained lease recorded, and lets the existing retained-source fallback settle these holds from the last save. > - The benefit is that a moved workspace continues its session, and a hold that cannot be verified continues automatically instead of dead-ending. ## Linked Issues or Issue Description Refs #20346 **What happened?** On staging, a warm native continuation ran about 10 minutes after a deploy. The run before it died during preparation (database deadlock), so its workspace epoch was abandoned. The continuation restored the last save into a new allocation directory of the same Daytona sandbox. It then failed with `native_controller_recovery_missing_evidence:retained_sandbox_binding`, because the retained lease of the committed source run recorded the old directory. The failure recorded a no-replay hold (`native_continuation_requires_reconciliation`, resolved with outcome `blocked`). Workspace recovery refused the source with `allocation_ownership_ambiguous`, because the sandbox's reusable-lease chain had predecessors in another directory and a non-native preparation-only run. The task stayed blocked with "Retry runs the check again and succeeds only after Paperclip is updated." **Expected behavior** The continuation reads the retained Runner state from the directory where that Runner wrote it, and continues. When retained files cannot be verified, the task continues from the last committed save without a person. **Steps to reproduce** 1. Run a warm native session on a reusable remote sandbox and let it commit. 2. Fail the next run during preparation, so that its epoch is abandoned (#20346). 3. Start the next warm continuation. It restores the last save into a new allocation directory and fails with `retained_sandbox_binding`. 4. See the hold refuse its source with `allocation_ownership_ambiguous` and stay blocked. **Paperclip version or commit** Staging `c874cfa4dacc` (includes #20346). ## What Changed - `assertNativeRetainedSandboxBinding` now returns the directory that the retained lease recorded. It accepts a different target directory only when both are canonical sibling allocations of the same execution workspace under the sandbox's `.paperclip-workspaces` (`<workspaceId>` or `<workspaceId>-<epoch>-<leaseId>`). Same sandbox (`providerLeaseId`), environment, run, issue and committed owner are still required. Other directories stay refused. - `restoreRemoteSuspendedRunnerCheckpoint` reads the Runner state from that recorded directory. A moved source must be suspended and settled. A Runner left in the old allocation is never adopted live. - The retained-source fallback (`continueFromLastSavedWorkspace`) and its discovery now also cover the native no-replay hold (`native_continuation_requires_reconciliation`, active or resolved with outcome `blocked`, replay blocked for the same run). Rows saved by earlier releases are discovered without migration. - The fallback owns the Blocked status of such a hold only while the recorded `nativeFailureBlock` status version still matches the task. Otherwise it keeps the newer block and starts no continuation. - The refusal text no longer says "succeeds only after Paperclip is updated", and it promises no fallback. - `doc/execution-semantics.md` describes both rules. ## Verification - Red (unfixed code): `native-controller-recovery.integration.test.ts` "binds a retained Runner read to the source's directory after the workspace moved within the same sandbox" failed with `native_controller_recovery_missing_evidence:retained_sandbox_binding` at `native-controller-recovery.ts:145`, the staging error. - Red (unfixed code): `workspace-recovery-fallback.test.ts` "continues a resolved-but-blocked native no-replay hold from the last save, discovered by the worker" failed: the worker found no due work for the exact staging row. The newer-block safety case and the message case in `system-recovery-state.test.ts` also failed. - Provider boundary: a new test drives `restoreRemoteSuspendedRunnerCheckpoint` through a sandbox runner that refuses paths outside the sandbox workspace base. The suspended case reads `runner-state.json` from the source directory, never the moved root. The ready case is refused with `runner_not_settled` and caches nothing. - Green: `native-controller-recovery.integration.test.ts`, `workspace-recovery-fallback.test.ts` and `system-recovery-state.test.ts` pass (45 tests). The neighbouring suites `workspace-recovery-retry-copyback`, `workspace-recovery-preparation-gates` and `workspace-recovery-stop-proof` pass (135 tests). - Safety cases: another workspace, another parent, a non-canonical path or name (`-0-`, trailing text), `/unrelated` and another sandbox stay refused. A hold whose task moved to a newer Blocked decision keeps the block and starts no continuation. Exactly one continuation is delivered. Native replacement is denied for the abandoned source, so no second executor starts. - `npx tsc --noEmit -p server/tsconfig.json` passes. `pnpm verify` returns exit 2 (a global doc input changed), so required CI supplies the broad coverage. ## Risks - Previous-release state: rows saved by earlier releases carry forward without migration. The binding reads existing lease metadata. The fallback discovers existing resolved-but-blocked holds by their existing evidence fields. - The binding now accepts a sibling allocation directory. The read still binds to the exact sandbox, run, environment and committed owner, and it reads only the source's recorded directory. A non-suspended Runner in a moved allocation is refused, not adopted. - The fallback abandons the unverified epoch and continues from the last committed save with a fresh session. In the staging shape the run did no provider work after it restored that save, so no work is lost. Unverified files are attached to the task when they can be read, as before. - Not verified live: a real warm Codex resume after a moved allocation on Daytona. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, read-only browser access to staging). Extended reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
e01fe7f84a |
perf(workspaces): reuse duplicate cohort source archives (#20574)
Capture identical live sources once within a migration cohort and reuse the completed archive for per-workspace encryption and upload. Bound shared scratch to one capture key per root and retain existing source, journal, and publication guards. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
99edcd28a7 |
perf(attention): read each retention candidate kind in its own bounded branch (#20566)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The attention retention worker archives old "Needs you" items, such as open decisions, pending questions, failed runs and items in review > - To find its work, the worker reads a page of retention candidates. It reads one page per changed scope, and one single-source read for each check of a due item > - On a hosted stack this candidate read ran 2,835 times at about 1 s per call, 4,201 s of database time in one afternoon window > - The read built every candidate of every kind before it applied the keyset limit. Its company and source filters used the `param is null or column = param` form. Under a generic prepared plan these filters cannot use an index. The failed-run branch also matched retry receipts with a bound LIKE parameter, so it could not use the partial index `heartbeat_run_events_retry_exhausted_idx` > - This pull request reads each kind in its own branch, puts every filter into that branch, and limits each branch to one page > - The benefit is a bounded read for each page, and a primary-key read for each single-source check, with the same rows, order and cursors ## Linked Issues or Issue Description **What existing behavior does this improve?** The attention retention candidate scan in `server/src/services/attention-retention.ts`. **Subsystem affected** Server: attention retention background work (database reads only). **Current behavior** Each page reads all candidates of all selected kinds, then sorts them and keeps 100. Optional filters use `($n is null or column = $n)`. A generic prepared plan cannot use an index for these filters. The single-source check compares `id::text`, so it cannot use the primary key. The retry receipt LIKE pattern is a parameter, so the plan cannot use the partial receipt index. **Proposed behavior** Each kind is one branch with its company, source, issue and keyset filters inside it and `limit 100`. Kinds before the cursor are not read. A source id filter uses the uuid primary key. The receipt pattern is a SQL literal that matches the partial index predicate. The page result does not change. Refs #20386 (adds the partial receipt index that this branch now uses). ## What Changed - `server/src/services/attention-retention.ts`: replace the single union scan with `candidateQuery`. Each source kind is a branch that has its own filters, retention join and one-page limit. The outer query still orders by `(source_kind, source_id)` and keeps 100 rows. - Only add a filter when the scope sets it. There is no `param is null or ...` form now. - A single-source read compares `id = $uuid`. It returns no rows for text that is not the canonical lowercase uuid form. The former `id::text = $text` comparison also never matched such text. - Skip the kinds that come before the keyset cursor kind. `ATTENTION_RETENTION_KIND_ORDER` holds the kind order, and a test checks it against the database collation. - The failed-run branch matches `'Bounded retry exhausted%'` as a literal, so it reads `heartbeat_run_events_retry_exhausted_idx`. - Export `listAttentionRetentionCandidates` for the test and the benchmark. The service uses it without a change in behavior. - New test `server/src/__tests__/attention-retention-candidates.test.ts`. It runs on real PostgreSQL and compares every page, row and cursor with the former scan (kept in the test as an oracle). It covers: one company, all companies, one kind, the blocker kind (which ignores retention), an unknown kind, issue scopes (including uppercase and malformed ids), exact source ids (uppercase, braces, no dashes, empty and malformed text return nothing), a `null` source id from a decoded scope, other-company isolation, inactive companies, kept and archived retention, and failed runs without a lifecycle receipt. - New benchmark `scripts/benchmark-attention-retention-candidates.ts`. It loads the baseline service from git, captures the SQL that each version sends for the same scopes, and times both under custom and generic plans. Each round alternates the order. It fails if the rows differ. - No migration. The existing indexes cover each branch: company and status indexes on each source table, the issues company/status index, and the partial receipt index from #20386. ## Verification - `cd server && npx vitest run src/__tests__/attention-retention-candidates.test.ts`: 10 passed. - `cd server && npx vitest run src/__tests__/attention-retention-work.test.ts`: see the note under Risks. - `cd server && npx tsc --noEmit`: pass. - Benchmark (local SQL measurement only): `pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-attention-retention-candidates.ts --baseline-ref=8d2bf78323` Embedded PostgreSQL 18.1. Fixture: 12 companies (1 large and 11 at 1/5 size), 38,400 issues, 64,000 runs, 1.29 M run events, 6,400 decisions, 9,600 interactions, 4,800 recovery actions, 7,534 retention rows. Seven samples for each scope, after two warmups, with `plan_cache_mode` forced to custom and then to generic. Rows were equal in every sample. | Plan | Scope | Rows | Before median (min–max) | After median (min–max) | Speedup | EXPLAIN shared blocks before → after | | --- | --- | --- | --- | --- | --- | --- | | generic | single source (check path) | 1 | 4.82 ms (3.81–5.88) | 0.19 ms (0.12–0.66) | 25× | 3,491 → 10 | | generic | company failed runs | 81 | 45.0 ms (40.5–51.7) | 2.9 ms (2.3–4.6) | 15× | 3,752 → 3,752 | | generic | company page 1 | 100 | 39.4 ms (30.1–63.7) | 7.6 ms (5.8–14.3) | 5.2× | 4,396 → 4,719 | | generic | company page 2 | 100 | 33.3 ms (31.2–38.1) | 6.9 ms (5.2–10.3) | 4.8× | 4,237 → 4,712 | | generic | company issue scope | 4 | 11.8 ms (9.8–24.0) | 4.1 ms (3.4–6.4) | 2.9× | 4,243 → 4,225 | | generic | all companies page 1 | 100 | 107.2 ms (100.2–116.0) | 29.3 ms (18.8–64.4) | 3.7× | 13,657 → 14,672 | | custom | single source (check path) | 1 | 2.68 ms (2.01–5.33) | 0.81 ms (0.65–1.50) | 3.3× | 3,491 → 10 | | custom | company failed runs | 81 | 5.7 ms (3.9–15.8) | 4.6 ms (3.2–7.1) | 1.2× | 3,752 → 3,752 | | custom | company page 1 | 100 | 11.7 ms (9.2–13.3) | 10.7 ms (9.4–19.5) | 1.1× (neutral) | 4,396 → 4,719 | | custom | company page 2 | 100 | 11.2 ms (7.6–14.2) | 10.1 ms (8.5–12.0) | 1.1× (neutral) | 4,237 → 4,712 | | custom | company issue scope | 4 | 10.9 ms (9.1–13.3) | 8.4 ms (6.4–10.4) | 1.3× | 4,243 → 4,225 | | custom | all companies page 1 | 100 | 21.3 ms (17.4–25.9) | 24.4 ms (20.3–72.0) | 0.9× (neutral) | 13,688 → 14,672 | - The timing connection uses named prepared statements, as the server does by default. The benchmark checks `pg_prepared_statements` before it collects samples, so the samples reuse the parsed statement. - The driver uses named prepared statements by default, so after five runs PostgreSQL can switch a statement to a generic plan. The generic rows are the relevant case for a busy server. Custom plans were already good for most scopes, and the change is neutral there. - Plan after the change (company page, generic): the failed-run branch reads the partial receipt index (13 blocks), then probes `heartbeat_runs` by primary key. The single-source check reads each primary key and the retention unique index (10 blocks in all). ## Risks - Low behavior risk. The test compares the new pages with the former query on the same data, including cursors in the middle of a kind and at kind boundaries. - Kind skipping depends on the order of kind names. Every kind name is different from the others before its first underscore, so C and linguistic collations give the same order. A test checks the order against the database. - The benchmark fixture is smaller than the hosted database. The hosted cost (about 1 s per call) is larger than the local baseline. I did not measure the hosted improvement. It needs a fresh query-statistics window after deploy. - On this loaded local machine, one full run of `attention-retention-work.test.ts` had 3 tests time out at 30 s (no-run issue finalization, no-run workspace finalization, and interaction create/withdraw). The same 3 tests pass when run alone. CI is the authority for this suite. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic. Agentic coding with tool use (shell, file edits, local PostgreSQL). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
faaae003a9 |
perf(chat): find the conversation head from open delivery work (#20564)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - External chat messages go into a durable delivery queue. Each
conversation drains in provider order.
> - Each drain step reads the head of one conversation. The head query
filters one endpoint's deliveries by state and thread.
> - On a hosted instance, this query ran about 10,000 times at about 300
ms each. The database was busy, and request reads waited behind
background work.
> - PostgreSQL read every processed delivery of the endpoint and its
`normalized_event` value to find a few open rows.
> - This pull request finds the endpoint's open delivery IDs first and
fetches only those rows.
> - The benefit is less background database load. The selected head and
the delivery order do not change.
## Linked Issues or Issue Description
**What existing behavior does this improve?**
The inbound conversation head query
(`earliestOpenConversationDelivery`).
**Subsystem affected**
Server chat inbound delivery queue.
**Current behavior**
The query has `endpoint_id = $1 and (state in ('received', 'retry',
'processing') or (state = 'processed' and <pending inbound wake>))`, a
thread filter on `normalized_event`, and a sort on `normalized_event`
keys. No index matches the `or`. PostgreSQL scans all deliveries of the
endpoint. For each row it reads `normalized_event` to test the thread.
It also reads all of `chat_actions` to build the pending-wake set. A
busy endpoint keeps all of its processed history, so the cost grows with
that history.
**Proposed behavior**
The query selects `id in (<open delivery IDs of the endpoint>)`. That
set is a `union all` of two small reads:
- received, retry and processing deliveries of the endpoint, through the
existing `chat_deliveries_work_idx`;
- processed deliveries that an issued or processing `inbound_wakeup`
action of the same endpoint (and company) still owns.
PostgreSQL fetches only those rows by primary key, then applies the same
reaction, thread and order rules.
**Reason and benefit**
The cost no longer grows with processed history. On a local fixture with
60,000 processed deliveries, the head query reads 1,309 shared buffers
instead of 262,630.
**Breaking changes**
None. There is no schema change, migration, API change, timer or poll
change.
Related work: #17971 changed the pending-wake predicate of the same
queue. #20400 added the index for the other chat delivery statement in
the same report (`select state ...
normalized_event->'githubAutomatic'->'context'->>'deliveryId'`). That
statement already uses the #20400 index on master, so this pull request
does not change it.
## What Changed
- `delivery-ledger.ts`: add `openConversationDeliveryIds(endpointId)`.
It returns the endpoint's open delivery IDs. It selects the same set as
the old state predicate for that endpoint.
- `conversation-queue.ts`: the conversation head query uses `id in
(openConversationDeliveryIds(endpointId))` in place of the state `or`.
The thread filter, reaction exclusion, order, limit and lifecycle
dependency read do not change.
- `inbound-queue-discovery.test.ts`: add a real PostgreSQL test. It
compares the new set with the old predicate on three endpoints. It
covers each state, issued and processing wakes, duplicate wakes, wakes
of non-processed deliveries, processed, other-kind and null-delivery
actions, an action on a different endpoint of the same company, and
reactions. It also checks that the real captured head query returns the
oracle's first row.
- `benchmark-inbound-queue-discovery.ts`: add
`--endpoint-history=<rows>` (a busy endpoint with 4 KiB payloads) and
`--only=<fixture>`. Add a work budget: the treatment head query must
visit at most 100 delivery rows on the history fixtures.
- `doc/DATABASE.md`: describe the head query and the benchmark command.
## Verification
Local SQL measurements. PostgreSQL 17.10 (Postgres.app, macOS arm64),
disposable database, synthetic data, default settings with 256 MB shared
buffers. The host was under heavy load from other work, so timings are
noisy. Shared buffers and visited rows are stable. Each arm has two
warmups and five alternating samples (median [min..max]). Baseline:
`origin/master` at `8d2bf78323`.
| Fixture | Query | Baseline | Treatment | Shared buffers | Delivery
rows visited | Output |
| --- | --- | ---: | ---: | ---: | ---: | --- |
| 60,000 processed, 4 KiB | head | 290.09 ms [220.21..554.19] | 7.34 ms
[5.85..7.68] | 262,630 → 1,309 | 60,002 → 4 | Equal |
| 20,000 processed, 4 KiB | head | 276.02 ms [211.77..322.12] | 34.61 ms
[4.45..194.15] | 67,618 → 486 | 20,002 → 4 | Equal |
| 1,500 processed, 32 KiB | head | 15.88 ms [13.66..21.79] | 2.29 ms
[1.64..7.75] | 10,602 → 80 | 1,502 → 4 | Equal |
| 20 processed | head | 1.49 ms [0.92..3.66] | 1.85 ms [1.01..1.97] | 2
→ 45 | 22 → 968 | Equal |
- The deadline and ready selectors do not change. Their buffers are
identical in both arms; their timings move only with host noise.
- The small control is neutral. On a near-empty table, PostgreSQL picks
a sequential scan for the open-state branch. This costs about 45
buffers.
- The 60,000-row run passes `--require-work-budget`. The baseline
exceeds the budget (`baselineExceedsWorkBudget: true`). This is the
red/green proof. There is no timing threshold in CI.
- On the treatment, the remaining head cost is the scan of
`chat_actions` for pending wakes of the endpoint. The baseline also
reads all of `chat_actions` for its hashed subplan.
- `pnpm exec vitest run src/__tests__/inbound-queue-discovery.test.ts`
(server): 6 tests pass, including the new equivalence test.
- `chat-channels.integration.test.ts` filtered with `-t
"reorder|orders|FIFO|conversation drain|lease renewal|receipt
order|edit|lifecycle"`: 148 assembled chat tests pass. They cover Slack,
Teams, Telegram, Discord and GitHub provider-order drains, head-of-line
barriers, lease loss, lifecycle edits and wake retries.
- `pnpm exec tsc --noEmit -p tsconfig.json` (server) passes.
Reproduce:
```sh
pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-inbound-queue-discovery.ts --baseline-ref=8d2bf78323 --endpoint-history=60000 --only=endpoint-history --require-work-budget
```
Set `PAPERCLIP_TEST_POSTGRES_ADMIN_URL` to use an existing disposable
PostgreSQL server. The benchmark uses synthetic data only.
## Risks
- Query plans depend on data and settings. These local numbers do not
prove hosted request latency. A follow-up statistics window after deploy
must confirm the gain.
- The pending-wake branch still scans the endpoint's `chat_actions`. A
partial index on pending `inbound_wakeup` actions would remove that
scan. I did not add it, to avoid a migration.
- Low behavior risk. The new set equals the old predicate for each
endpoint, and the order expression is unchanged. Tests compare both on
the same rows.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell,
file edit) in Claude Code. Extended reasoning was used.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandPaperclip
|
dff8bed007 |
fix(ui): show chat activity immediately during admission (#20559)
## Thinking Path > - Paperclip helps people manage AI agents and their work. > - Chat uses a shared runner turn to show activity after Send. > - The turn currently shows Queued while the server admits the request. > - People need immediate activity feedback during this short wait. > - This change uses the existing Working header and Thinking activity during ordinary admission. > - Specific server holds stay visible so people can see setup or maintenance delays. ## Linked Issues or Issue Description **What existing behavior does this improve?** The activity indicator in task chat and agent chat after Send. **Current behavior** Ordinary admission shows Queued in the header and activity row. It changes to Working and Thinking when execution starts. **Proposed behavior** Show Working and Thinking from message submission through ordinary queued admission. Keep the same avatar, name, and activity row in place. Preserve specific server waits and the existing queue controls. **Subsystem affected** ui/ — React board UI. ## What Changed - Use the default activity presentation when a queued turn has no specific server hold. - Retain maintenance, agent setup, server restart, previous-run, provider, and preparation labels. - Omit the elapsed-time suffix until a start time is known. This prevents a frozen Working for 0s header before a run exists. - Update component tests for submission, acknowledgement, reassignment, and runtime discovery on desktop and mobile. - Update the activity guidance in DESIGN.md. - Keep this behavior in the shared runner turn. No API, queue, or execution contract changes are needed. ## Verification - Chat component suites: 582 passed, 1 existing skip. - Command: `pnpm exec vitest run ui/src/components/task-chat/TaskChatRunnerTurn.test.tsx ui/src/components/TaskChatThread.test.tsx ui/src/components/IssueChatThread.test.tsx`. - `pnpm check:token-gates`: passed. - `pnpm verify`: exit 2. DESIGN.md is a global CI input, so the selector requires broader CI coverage. The three focused chat suites passed again on final commit `e85b8e43b007288f3ef41e0a42e801d5d2c41e54`. Required CI passed on this revision. See the CI result below. - The earlier CI attempt on `796fac80a06769d67af5b899fb186a875bfb228d` failed the unchanged Rust test `process_lineage::journal::tests::reaping_stops_a_dead_runners_provider_and_its_escaped_tool_job` with `unattributed_orphan`. This UI change does not alter its compiled inputs. The cause is not yet established. The current revision passed the Rust check. The earlier failure cause remains unknown. [Failed job](https://github.com/paperclipai/paperclip/actions/runs/42843806417/job/129926132490). - The workspace preflight installed dependencies and built shared and plugin SDK outputs. Its unrelated Runner build could not start because cargo is unavailable. The current revision passed CI typecheck and build. - Both current-head reviews are 5/5 with no remaining findings. There are no unresolved review threads. - Current-head typecheck, build, general tests, serialized server tests, all eight E2E shards, and Rust checks passed. [CI run](https://github.com/paperclipai/paperclip/actions/runs/42845383745). - The Runner support suite reported one 30-second timeout in `private-reporter-artifacts.test.ts`; 11,291 other tests passed. Both tests in that file passed locally in isolation (1.64 seconds for the full command). The one rerun passed in CI attempt 2. The timeout cause remains unknown. [Failed job](https://github.com/paperclipai/paperclip/actions/runs/42845383745/job/129932351496). - Final CI result: attempt 2 passed on `e85b8e43b007288f3ef41e0a42e801d5d2c41e54`. Typecheck, build, tests, Rust checks, and all eight E2E shards passed. [CI attempt 2](https://github.com/paperclipai/paperclip/actions/runs/42845383745/attempts/2). ## Risks - Low risk. This changes presentation only. A queued turn can show Thinking before provider execution starts. - Specific server holds remain visible. Scheduling, permissions, and saved message controls do not change. ## Model Used - OpenAI GPT-6 through Codex, with reasoning, tool use, and code execution. The runtime does not expose a more specific deployment model ID or context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
56aa160a4e |
perf(native): index heartbeat runs by company and native session (#20560)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs share a native session. Retention, recovery, resume and archive code reads all runs of one session from `heartbeat_runs`. > - On a hosted instance, the native retention lookup ran about 67,000 times in one day at about 22 ms, and about 10,000 times at about 11 ms in a later window. > - No index has `native_session_id`. Each read walks the agent's or the company's run history and filters the rows. > - This pull request adds one concurrent partial index on `(company_id, native_session_id)`. The SQL does not change. > - The benefit is less background database load. Request reads wait less, and the results stay the same. ## Linked Issues or Issue Description **What existing behavior does this improve?** The native session-run lookups. The largest caller is `sessionRetentionBlocker` in native harness backup and checkpoint retention. It selects `id, status, finished_at`, an incomplete-recovery flag and the issue ID for one company, agent and native session. **Subsystem affected** Server native runtime (retention, recovery, resume and archive reads) and the database schema. **Current behavior** `heartbeat_runs` has no index on `native_session_id`. PostgreSQL reads the whole run history of the company (or of the agent through `heartbeat_runs_company_agent_started_idx`) and filters by session. The cost grows with the run history, not with the size of the session. **Proposed behavior** The index finds the few runs of one session directly. Every caller already filters by `company_id` and `native_session_id`, so one index serves the agent-scoped and the issue-scoped reads. **Reason and benefit** This statement family used about 1,500 s of database time in one day on a busy hosted database. The local proof below shows 12.3 ms to 0.04 ms per call. ## What Changed - `packages/db`: add `heartbeat_runs_company_native_session_idx` on `(company_id, native_session_id) WHERE native_session_id IS NOT NULL`. Migration `0457_native_session_run_index` builds it with `CREATE INDEX CONCURRENTLY IF NOT EXISTS`. The migration is registered as a top-level-statement migration, the same way as 0454–0456. - Add a regression test in `native-harness-backup-expiration.test.ts`. It captures the real retention lookup and proves that its plan can use the index. It also pins the company and session scope: a live run of another company with the same session ID does not protect a checkpoint. A live run of the same agent in another session does not protect it. A live run of the same session does protect it. - Add `scripts/benchmark-native-session-run-index.ts`. It runs the real checkpoint retention pass (dry run), captures the exact SQL, and replays it with and without the index. It also times the migration through the normal migration runner. - `doc/DATABASE.md`: describe the index and the benchmark command. ## Verification Local SQL measurements, synthetic data. PostgreSQL 18.1 (embedded test server). Fixture: 100,000 runs (50,000 of the measured agent, 50,000 of a second agent), 4 runs per session, 4 KiB incompressible `runner_profile_json` per run, 638 MB table. Order: no index, index, no index, index. 15 `EXPLAIN (ANALYZE, BUFFERS)` samples after 3 warmups. | Pass | Plan | Median execution | Min / max | Shared buffers | | --- | --- | --- | --- | --- | | No index (1) | Seq Scan | 12.30 ms | 10.33 / 35.27 ms | 3,607 | | Index (2) | Index Scan using `heartbeat_runs_company_native_session_idx` | 0.036 ms | 0.021 / 0.115 ms | 38 | | No index (3) | Seq Scan | 12.38 ms | 10.24 / 26.14 ms | 3,607 | | Index (4) | Index Scan using `heartbeat_runs_company_native_session_idx` | 0.043 ms | 0.022 / 0.098 ms | 38 | - Correctness: the service SQL is the same in every pass. The returned rows (all columns) and the retention result are equal in every pass. The script fails on any difference. - Migration: the first run through the migration runner took 525 ms on the 638 MB fixture. The rerun took 273 ms and skipped the existing index. A direct concurrent build took 145–221 ms. Index size: 1.9 MB. - `pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-native-session-run-index.ts` - `npx vitest run src/__tests__/native-harness-backup-expiration.test.ts` (server): 108 passed. - `pnpm --filter @paperclipai/db check:migrations`: passed. Five snapshots remain. - `pnpm verify` exits 2 (a new script under `scripts/` is a global CI input). Required CI supplies the broad coverage. ## Risks - Low risk. The migration adds one index concurrently and does not lock writes. The SQL of every caller is unchanged. - The index adds a small write cost to `heartbeat_runs` inserts and to updates of `native_session_id`. Rows with a null session are not indexed. - Persisted run state: no saved run, session or checkpoint shape changes. State saved by the previous release carries forward unchanged. - Hosted latency is not measured yet. The numbers above are local SQL measurements. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edits, local PostgreSQL). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
10bc6431da |
test(restart-chaos): measure TTFT in a real repository workspace, after an idle restart, and on Daytona (#20047)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos TTFT driver measures time to first token, but only with a tiny fixture workspace and only locally > - Dotta asked whether speed is good with a workspace like the paperclip repository itself, and that had never been measured > - This pull request lets the driver run in a real repository workspace, measure the first follow-up after an idle restart, run on Daytona (per-turn or warm reusable sandboxes), and report each run's workspace operations > - The benefit is a repeatable large-workspace speed measurement; its first run found a full workspace re-upload on every remote turn ## Linked Issues or Issue Description **What happened?** The TTFT driver seeded only a tiny fixture repository, created tasks outside any project, ran only locally, and could not show workspace operation durations. So large-workspace speed (worktree creation, provision, remote upload) was never measured. **Steps to reproduce** 1. Run `ttft-baseline.ts` with any options. 2. Every task runs without a project workspace and on the local host. 3. Nothing in the report shows worktree, provision or sync cost. **Expected behavior** The driver can measure a real repository workspace, locally and on Daytona, and names the workspace operations each run waited on. ## What Changed - `--workspace-repo <path>`: the seed uses that repository as the project's primary workspace (git_worktree isolation), and tasks are created in the project. - A `postfault` phase: K1 while idle, then a follow-up on each task; reported in its own table. - `--env daytona`, `--daytona-image`, `--daytona-cpu`, `--daytona-warm` (the reusable warm Daytona fixture). The Daytona controller preparation moved to `daytona-controller.ts`, and every sandbox is cleaned up at the end; incomplete cleanup fails the run. - Each run's workspace operations (`/api/heartbeat-runs/:id/workspace-operations`) are saved, and printed ranked by duration. ## Verification - `tsc -p tests/restart-chaos`: no errors in harness files. - Used for the large-workspace measurement, a fresh paperclip clone on local and Daytona (per-turn and warm). First turn / follow-up: local 31–34 s / 7 s; Daytona per-turn 58–63 s / 30 s; Daytona warm 45–51 s / 8–10 s. Every Daytona sandbox was cleaned up (environment deleted, none remaining). ## Risks - Test harness only; no product code. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file editing, test execution, Daytona measurements). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
ce1ee9e89f |
fix: stop prompts from encouraging subtasks (#20362)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents read runtime prompts, tool help, skills, and role instructions. > - Several of these texts tell agents to create child tasks as a normal next step. > - One accepted-plan prompt even forbids work on the current task. > - Task parentage is useful, but it does not need a prompt that encourages task creation. > - This pull request removes those prompts and keeps the task APIs available. ## Linked Issues or Issue Description Refs #10642. This change adjusts prompt wording; it does not add delegation-cycle guards. **Current behavior** Prompts tell agents to create children after plan approval, use children for long work, and create a task just to pass context. Onboarding requires a child even for a single request. **Proposed behavior** Continue work on the current task. Keep explicit assignments, optional task creation, parentage, dependencies, and approval gates. Use the separate-task recipes when the user requests that structure or when continuing existing work. **Reason and benefit** Reduce needless task trees without removing subtask support. Keep the patch limited to wording, existing test expectations, and generated catalog data. ## What Changed - Align accepted-plan prompts and remove direct child-creation advice from runtime and error text. - Make task tool descriptions neutral. Keep their fields, defaults, and handlers. - Update the main skill, onboarding, CEO instructions, company templates, specialist workflows, and matching guides. - Keep historical release snapshots intact. Refresh the explicitly seeded schema fixture's catalog digest; the server still verifies historical release snapshot hashes. - Refresh the affected generated catalogs, capability inventory, semantic contracts, and fixture manifest. ## Verification - Final tested head: `c6e304d0c105cdbfbe0b2ab0a233307fcd91a04b`. - All 66 check runs are complete: 60 pass, one is neutral, and five are skipped by CI conditions. Snyk also passes. No checks fail or remain pending. - [PR CI](https://github.com/paperclipai/paperclip/actions/runs/42846226554) passes typecheck, build, server/general tests, all eight E2E shards, Runner support, Runner Rust, and the aggregate verify gate. [Windows](https://github.com/paperclipai/paperclip/actions/runs/42846226546) and [Docker Runner](https://github.com/paperclipai/paperclip/actions/runs/42846226108) also pass. - [Greptile](https://github.com/paperclipai/paperclip/pull/20362#issuecomment-6751419835) and [Paperclip review](https://github.com/paperclipai/paperclip/pull/20362#issuecomment-6751454010) give this exact head 5/5. All six inline threads are resolved. Neither current summary has findings. - Merged master `bf0220d34a156c81853755c3d1365ddc527037de`. Resolved three generated-file conflicts with the canonical generators. - Local verification passes: 473 focused prompt, continuation, UI, catalog, runtime and native-execution tests; Runner TypeScript build; generated metadata; UI token gates; and diff checks. - The prior head `c22018cb0f` failed three SDK evidence tests because two source hashes were stale. Reproduced all three failures locally. The existing generator refreshed only the hashes for the denial-copy file and its test. All 53 affected evidence/source-pin tests then passed, including private-value rejection. No scanner logic, assertions, or runtime dependency pins changed. Current CI confirms the support suite passes. - A prior commit failed the Rust process reaping test. Its cause remains unproven. No Rust code or assertions changed. Later Rust runs pass; this does not establish the cause of the earlier failure. - Local preflight built shared/plugin SDK packages, then stopped because `cargo` is absent. Current-head CI supplies Rust and full build coverage. `pnpm verify --explain` selects broad required CI; full local suites were not repeated. ## Risks - Wording changes guide the model; they do not enforce a ban on subtasks. - Existing installed skill versions and saved custom instructions need their normal refresh process. This PR does not migrate company configuration. - Existing approved task assignments and approval gates remain in effect. ## Model Used - OpenAI GPT-6, through Codex, with reasoning, tool use, and code execution. The exact routed model ID and context-window size were not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
79e28dc0d0 |
fix(runner): record pending calls at sidecar shutdown (#20272)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The ACPX sidecar carries task tools between a provider and its controller. > - A closed controller pipe rejects all pending tools with `sidecar stdin closed`. > - Current logs do not record the first shutdown trigger or the pending calls at that point. > - The sampled runs record a server SIGTERM before the 300-second tool idle timeouts and later pipe failures. The exact pipe-close cause remains unresolved. > - This pull request adds one bounded shutdown record before cleanup removes the evidence. > - Operators can use it to compare controller shutdown with pending tool calls. ## Linked Issues or Issue Description **What happened?** Several task document reads failed together with `sidecar stdin closed`. In two sampled staging traces, earlier task tools reached a 300-second idle timeout. The sidecar source maps the closure error to pending-call rejection during shutdown. It does not record the trigger or the pending call population. Pattern: [Task reads lose their tool connection](https://www.braintrust.dev/app/Paperclip/p/Paperclip%20Staging/patterns/5544e6f3-9e33-4661-95c3-5849c9a8f2b6). **Expected behavior** Record the local shutdown reason and a bounded snapshot of pending calls before cleanup removes them. Keep tool cancellation and authorization unchanged. **Steps to reproduce** 1. Start the sidecar with pending task calls. 2. Close controller stdin, or send SIGTERM or SIGINT. 3. Inspect the sidecar event stream and stderr. **Paperclip version or commit** Baseline: `33a23e024697552a4a639e49957a3443c75a5bd5`. ### Hypothesis and thesis for a fix The provisional hypothesis is that controller teardown causes the batch failures after an earlier stall. The source proves how EOF rejects pending tools. The traces establish ordering. They do not prove why the controller closed stdin or why the earlier calls stalled. Run logs now confirm an earlier server shutdown in both examples. One later recovery could not find the provider; the other later ended at a durable receipt limit. The first failed settlement and exact pipe-close trigger remain unresolved. The later example already used a provider revision containing #19365 and #19290, so those earlier fixes do not alone explain all occurrences. This is a **diagnostic-only change**. The sidecar owns the local shutdown signal and its pending waiters. Record that state once at the first shutdown request. Do not add retries or change deadlines without evidence. This supports each provider that uses this sidecar. It preserves company/task/run authority, approval checks, tool admission and cleanup ownership. It does not resolve or claim recovery of the original read stalls. ## What Changed - Emit `sidecar_shutdown_requested` through the existing diagnostic stream and stderr before shutdown can remove waiters. - Record pending tool, input, permission and completion-normalization counts. Include at most 16 oldest tool calls with elapsed time and a SHA-256 call ID. Exclude arguments, results, operation names and raw call IDs. - Add real sidecar process tests for EOF and both shutdown signals. Check successful resolution, cancelled calls, pending-call rejection, provider cleanup and sidecar exit with code 0 and no signal before fallback cleanup. - Document the record and its limits. Saved session formats and previous-release behavior stay unchanged. ## Verification - Candidate: `85a929eef87d5f775bbed9f2c98409432c86c702`. The baseline replay uses an isolated detached checkout at the baseline SHA with the exact candidate regression file. - Baseline red: the three process regressions failed because the expected shutdown record was absent. The sidecar initialized, handled calls and shut down before each assertion. - Unchanged process regressions pass with the implementation. The focused sidecar and snapshot suites passed: 117 tests, one unrelated macOS candidate-admission test skipped on Linux x64. - `cd packages/paperclip-runner && pnpm exec vitest run src/cli/acpx-runtime-sidecar.test.ts src/cli/acpx-sidecar-shutdown.test.ts` - `cd packages/paperclip-runner && node --test test/acpx-sidecar-contract.test.mjs`: 16 passed. - `cd packages/paperclip-runner && pnpm exec tsc -p tsconfig.json --noEmit`: passed after building the local eval-kernel dependency. - `git diff --check`: passed. - These tests use the real sidecar pipe and process boundary with a controlled provider host. Independent outcome and cleanup files prove rejection and process cleanup. They do not exercise a live authenticated Claude/staging run or prove deployed recovery. - The local preflight could not build Rust because this host has no Cargo. CI must supply the applicable Rust/build checks. At head `85a929eef87d5f775bbed9f2c98409432c86c702`, all 61 check runs are complete: 58 passed and three conditional jobs skipped. The legacy Snyk status passed. Greptile is 5/5 with both threads resolved. [CI result](https://github.com/paperclipai/paperclip/actions/runs/42630335496). ## Risks - The controller can stop reading before it receives the diagnostic frame. Stderr is an additional path, not a delivery guarantee. - EOF records a local pipe closure. It cannot identify the upstream controller cause by itself. - Hashed IDs require offline hashing of the existing tool-call journal for correlation. At most 16 samples are retained; counts include the full pending population. - No persistent state, retry, cancellation or authorization contract changes. ## Model Used OpenAI Codex, GPT-6, with reasoning, tool use and code execution. The runtime does not expose a more specific model ID or context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
e5abcbf7a9 |
fix(native): release a restart claim that lapsed during a failed preparation (#20529)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - After a restart, native restart recovery claims an unfinished run
and prepares it again on the same heartbeat run.
> - The restart claim holds a lease on the run's native coordinator row.
For a remote claim with durable authority, the lease is 60 s.
> - When that preparation fails, the failure path first preserves the
remote workspace. Then it settles the run.
> - When preparation outlived the claim lease, preservation took the
run's own lapsed claim for another owner. It threw
`NativeWorkspaceFinalizationBusyError`, and the execution returned
without settling. The run stayed `running` with no event until a later
scan reclaimed it (4.5 min in one wave, never in another).
> - This pull request releases exactly that lapsed claim and asks
restart recovery to decide on the run right away. It also logs the cause
when the failure path defers.
> - The benefit is that a failed restart-claim preparation is reclaimed
or blocked visibly within seconds, instead of hanging.
## Linked Issues or Issue Description
Refs #20514 (the same restart-chaos investigation).
**What happened?**
In restart-chaos wave `w-20261110T101517-s11936` (Daytona R1, K2, ST1),
run `9d39550a` got a `bootstrap_incomplete` restart claim (generation
3). Its `heartbeat.prepare.run_output` step waited 141 s through an ST1
storage outage. The 60 s claim lease lapsed meanwhile, and the begin
failed with `run_log_writer_authority_lost` when storage came back. The
inner failure path then called `preserveNativeFailedWorkspace`. Its
`ownsClaim` check required an unexpired lease, so the run's own claim
counted as another owner and the call threw
`NativeWorkspaceFinalizationBusyError`.
`preserveNativeWorkspaceOnFailure` set `nativeOwnershipHeld` and the
execution returned before `runCompletion.failRun`. No failure was
recorded, and nothing was logged. The run stayed `running` until a later
scan reclaimed it 4.5 minutes later. In a second wave no such scan came,
and the run stayed `running` for 30 minutes.
**Expected behavior**
A failed preparation must settle or retry, or be claimed promptly by its
next owner. It must not stay `running` with no record.
**Steps to reproduce**
1. Claim a remote native run for restart recovery.
2. Keep its preparation busy past the 60 s claim lease, for example with
a storage outage during `runOutput.begin`.
3. Let preparation fail. The run stays `running` with no failure event.
**Paperclip version or commit**
`master` at
|
||
|
DottaandClaude
|
c58356003f |
fix(native): give run finalization its own retry budget (#20555)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs save an accepted result. Then a finalizer applies that
result to the issue.
> - After a control-plane restart, recovery can resume the same run.
Each resume uses one provider attempt (`coordinator.attempt`).
> - The finalizer used the same `coordinator.attempt` value as its own
retry budget. After three resumes, `attempt` was 3. So the first
transient finalization failure made the coordinator `terminal_failure`.
> - The result was then lost: the issue stayed `in_review` and no Retry
was available. This breaks the rule that a restart must not leave a task
stuck.
> - This pull request gives finalization its own retry count. Provider
attempts no longer use up finalization retries.
> - The benefit: a resumed run that finishes its turn also finishes its
task, and finalization stays bounded.
## Linked Issues or Issue Description
**What happened?**
In restart-chaos wave `w-20261110T044338-s20385`, task t03 (local,
Codex, three K1 SIGKILL restarts, `--db-pressure 4`), recovery resumed
run `d5d0e839` three times (`providerAttempt: 3`). The resumed turn
completed and the server logged `run.result.accepted`, and the run ended
`succeeded`. The first finalization hit `NativeStatusRaceError` three
times under row contention. `recordRetryableFailure` then compared
`coordinator.attempt` (3) with its limit of 3 and wrote
`terminal_failure` / `native_finalization_retry_exhausted`. The run's
`execution.nextAction` is "Reassess against the latest authoritative
issue status version.", which only the `status_cas_exhausted` branch
writes. The second finalize call 2 s later got
`native_finalization_terminal_failure` from `claimCoordinator`. The
issue stayed `in_review` with `canRetry=false`.
**Expected behavior**
A transient finalization failure must stay retryable. The finalizer must
apply the saved result without another provider turn. Finalization must
still have a limit.
**Steps to reproduce**
1. Resume a native run until its coordinator `attempt` is 3, for example
with three restarts.
2. Let the resumed run save an accepted result.
3. Make the first finalization fail with a retryable error.
4. Before this fix, the coordinator goes directly to `terminal_failure`.
The new test
`server/src/__tests__/native-finalization-retry-budget.test.ts`
reproduces this order.
**Paperclip version or commit**
The wave ran at e2b0290213. The bug is still on master at
|
||
|
DottaandClaude Opus 5.5
|
bd30ef8510 |
perf(chat): authorize the waits of an issue page with one read (#20551)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents can work on tasks that start from an external chat, such as Slack. When the agent waits for the next chat message, the task shows "in review" with an external response wait. > - Issue lists and inbox attention show these waits. To show a wait, the server checks that the chat conversation is still authorized: the messages, the destination, the endpoint, and each person who wrote. > - The server did this check one time for each waiting task, and it started all the checks at the same time. A page with 19 waits sent 19 copies of the message-link statement at the same moment, each on its own pool connection, and then 19 copies of each policy read. > - On a hosted database, 19 identical copies of this statement were active at the same moment, about 670 ms each. They were 19 of the 20 active queries. > - This pull request reads the messages of all waits of a page with one statement and shares the policy reads between the waits. Each wait gets the same authorization result as before. > - The benefit is fewer statements and fewer connections for each issue page, so one page cannot fill the pool with copies of one read. ## Linked Issues or Issue Description Refs #20400 (shares wait reads between the callers of one request; this change removes the per-wait fan-out inside that shared read). Refs #18966 (reuses the prepared message-link statement; this change keeps that statement for single runs). **What happened** `listExternalReviewWaits` ran `authorizeChatConversationForBoundRun` one time for each waiting task, inside `Promise.all`. Each call sends the message-link statement, then a resource read, and three reads for each person. A page with N waits sent N copies of each statement at the same time, on up to N pool connections. **Expected behavior** One issue page sends one message-link statement for all of its waits. A resource or a person that is common to many waits is read one time. The authorization result of each wait does not change. **Steps to reproduce** 1. Put many tasks of one company in review with an external chat response wait. 2. Open the issue list or the inbox. 3. Look at the active statements of the database. One copy of the message-link statement runs for each wait. ## What Changed - `server/src/services/native-runtime/chat-attachment-reuse.ts`: - Split `authorizeChatConversationForBoundRun` into steps: read the bound provider and wake comments, read the message rows, and apply the policy to the rows (`authorizeBoundRunLinks`). The single-run path uses the same prepared statement, the same lock modes and the same policy code as before. - Move the message-link join and its policy filters to one query builder, `boundRunLinksQuery`. The single-run statement and the batch statement both use it. - Add `authorizeChatConversationsForBoundRuns(db, companyId, runs)`. It is read-only and takes no locks. It reads the messages of all runs with one statement, then gives each run only the rows that the single-run statement returns for that run (same issue, agent, provider and comments). The runs share each distinct resource and principal read. Each run gets its conversation, or `null` for a policy denial. Other errors are thrown, as before. - Add `isBoundChatAuthorizationDenial` for the three policy denial errors. - `server/src/services/native-runtime/external-review-wait.ts`: `listExternalReviewWaits` uses the batch authorization. The inherited GitHub review-wait reads do not change. - `server/src/__tests__/batched-chat-authorization.integration.test.ts`: new tests on real PostgreSQL. For each case, the batch result must equal the result of separate single-run reads. The cases include each denial: a missing or foreign comment, no comments, no execution marker, a wrong provider, agent or issue, a person without a link or without a membership, a viewer member, a revoked link, a disabled or unavailable destination, a paused endpoint, a closed conversation, an unprocessed delivery, a delivery without a person, and a verifying endpoint. A test also makes sure that one person's decision never applies to another person in the same batch, and that a batch sends one message-link statement. - `scripts/benchmark-chat-bound-authorization.ts`: a local benchmark that loads the pre-fix modules from a baseline commit, checks that both versions return the same waits, and measures them. ## Verification - `pnpm --filter @paperclipai/server exec vitest run src/__tests__/batched-chat-authorization.integration.test.ts src/__tests__/prepared-chat-authorization.integration.test.ts` passes (10 tests). - Mutation check: when I removed the issue filter from the per-run row selection, or shared one principal decision for all people of an endpoint, the new tests failed (1 and 3 failures). - Related suites pass: `heartbeat-reviewed-chat-binding`, `external-chat-wait`, `preserved-native-response-wait`, `chat-attachment-reuse` and `external-review-wait` (392 tests). - `pnpm verify` asks for broader verification because the change adds a file under `scripts/`. Required CI supplies that coverage. - Benchmark: `pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-chat-bound-authorization.ts --baseline-ref=f4337bb5fb` (the measured pre-fix commit) Benchmark fixture (synthetic, local PostgreSQL 18.1, pool of 10): one company, one Slack endpoint, 20 tasks waiting for a chat response from 5 people, 4 of those waits denied (the person is a suspended member), and 2,000 earlier conversations with 50,000 processed deliveries (6 KiB events, 419 MB). Both versions return the same 16 waits. 15 alternating pairs after warm-up: | One issue page (20 waits) | Before | After | | --- | --- | --- | | Statements sent | 101 | 18 | | Message-link statements | 20 | 1 | | Server time of the message-link statements | 0.50 ms (sum of 20) | 0.13 ms | | Page time, median (min to max) | 8.86 ms (6.94 to 12.05) | 4.05 ms (3.62 to 5.54) | | 8 pages at the same time, median (min to max) | 54.18 ms (48.36 to 89.24) | 14.99 ms (12.60 to 19.59) | ## Risks - The batch must give each wait the same result as a separate read. The batch statement uses the same join and policy filters. It then selects each run's rows with the equality filters of the single-run statement. The policy code is the same function for both paths. The tests compare both paths for each denial case. - Shared reads: the batch reads each resource and each person one time. All reads are plain reads with no locks, as before. A policy change that occurs during one page read can now apply to all waits of the page at the same time, instead of to some of them. - An invalid wake comment id in one wait makes the whole page read fail. This was already true: the failure of one wait rejected the whole `Promise.all`. - The single-run path (attachment reuse, status decisions, run preparation and others) keeps its prepared statement and lock modes. The order of the conditions in its WHERE clause changed; the conditions are the same. - Not proven: why each copy took about 670 ms on the hosted database. Locally one copy takes about 0.03 ms with the expected index plan. The likely causes are the burst of copies on a small compute and cold pages. Verify after deployment that the statement runs at most one time for each issue page request. - No migration and no schema change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude
|
b004519fde |
docs(agents): shorten AGENTS.md and move data-path detail to a scoped rule (#20557)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - `AGENTS.md` is the instruction file that each human and AI contributor reads before a change > - The file had 311 lines, and most of the text applied to only a few areas of the repository > - The file told each contributor to read five documents before every change and to treat `doc/SPEC-implementation.md` as the build contract > - The file listed 4 of the 12 rules in `rules/`, so the list was already out of date > - This pull request states contributor ownership, makes the code and tests the source of truth, and moves path-specific detail into a scoped rule > - The benefit is a shorter file that agents load on every session, with detail that loads only for the files it applies to ## Linked Issues or Issue Description **Issue type** Outdated or unclear contributor documentation. **Where is the issue?** `AGENTS.md` at the repository root. **What's wrong?** - The file told contributors to read five documents in order before each change. - The file made `doc/SPEC-implementation.md` the build contract and part of the definition of done. The maintainer considers that document outdated. - The file kept a manual list of scoped rules. The list named 4 of the 12 rules. - Rule 7 (the three data paths) used about 40 lines. The rule applies only to telemetry, observability, and run-log files. - The agent preflight instruction was in the verification section, not in the setup section. - The design system section had no number and came after the definition of done. **Suggested fix** Keep in `AGENTS.md` only the text that applies to every change. Move path-specific text to `rules/` and point to `doc/` for detail. ## What Changed - Added an ownership statement at the top of `AGENTS.md`. - Section 1 now says that the code and tests are the source of truth. The spec documents are background. - Removed the spec from the definition of done and from rule 4. - Section 2 is now a table that maps a type of change to the document to read. - Removed the manual rule list. The instruction to read each matching rule in `rules/*.mdc` stays. - Moved agent preflight to section 4. The database reset step now says that it deletes local data. - Rule 7 is now a short summary. The full text is in the new `rules/data-paths.mdc`. - Added `rules/data-paths.mdc` to `.greptile/files.json` with the same scope as the rule globs. - `CONTRIBUTING.md` now links to the new rule. - Section 7 is now the default command and six labelled rules. Maps and Storybook are one subsection. - The design system text is now a subsection of section 9. - Section numbers did not change, because code comments cite `AGENTS.md §5.7` and `§6`. ## Verification - Open `AGENTS.md` and confirm that each link and each path resolves. A local script checked this. - Compare `rules/data-paths.mdc` with the old rule 7 text. The definitions and review levels are the same. - Confirm that `.greptile/files.json` is valid JSON and that the new scope matches the rule globs. - No source code changed. No test suite ran locally. ## Risks - Contributors no longer treat the spec as a requirement. A change can now differ from the spec without an update to the spec. - An agent that does not read scoped rules gets only the summary of the three data paths. The summary keeps the review level for each path. - Two Storybook instructions now live only in `maps/README.md` and `maps/storybook-map/README.md`. ## Model Used - Claude (Anthropic), in Claude Code with tool use. - `claude-opus-5-5` added the ownership line and did the first review. - `claude-fable-5-2` wrote the restructure and this pull request. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Dotta
|
38cc6b2a74 |
fix(ui): restore pinned sidebar links above More (#20369)
Restore the mixed pinned navigation list after Inbox and before More, retaining saved order, agent pins, drag controls, and unpin actions. Verified with focused tests, UI checks, browser interaction checks, and passing PR CI. Greptile: 5/5. |
||
|
DottaandPaperclip
|
bf0220d34a |
perf(workspaces): migrate with one archive and no readback (#20554)
## Thinking Path > - Paperclip coordinates agents and keeps their workspace files. > - A legacy workspace must move from a local volume to object storage. > - The importer creates an archive, downloads and restores it, and captures the source again. > - Parent imports also download and recapture children that already have committed snapshots. > - This repeated work makes the one-time migration slow. > - This change copies each source file once and trusts the upload acknowledgement. > - Existing source locks and atomic publication still own the migration. ## Linked Issues or Issue Description **What existing behavior does this improve?** The one-time migration of legacy local and Git workspaces to durable snapshots. **Current behavior** Each workspace is captured twice and restored as a test during migration. Parent imports also read back already imported child snapshots. Capture hashes source files before writing them and scans them again afterward. **Proposed behavior** Build one archive from one source-file read, encrypt it, upload it, and publish its reference. Trust the upload acknowledgement. Reuse journaled uploads after an interrupted process. Retain committed child snapshots by database reference. **Reason and benefit** Remove repeated source reads, temporary copies, complete downloads, and recursive child verification. The tests assert one capture, one upload, and zero migration downloads. **Breaking changes** The archive format stays compatible. File hashes required by current readers are computed inline during the copy. There is no preliminary file-hashing pass. Existing readers can cold-restore the result. Related work: #19817 and #20447. ## What Changed - Add single-pass capture that fills fixed-length checksum fields in the existing archive header after copying. - Remove migration readback, test extraction, fresh-source recapture, and Git integrity scans. - Compute the encrypted archive checksum while writing it. - Reuse acknowledged and journaled uploads without rebuilding or downloading them. - Retain already imported children through their database bindings. Remove the recursive payload verification cache. - Keep company checks, source and writer locks, recovery bindings, journal identities, cancellation, and atomic publication. - Bind archived repository intent paths to the operation without reconstructing an archive to check them. - Update the migration contract and the affected tests. ## Verification - Node 24.22.2 and pnpm 9.15.4 on macOS arm64. - At |
||
|
DottaandClaude Opus 5.5
|
d0057ee384 |
test(restart-chaos): fire a planned restart at native start (#20538)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart-chaos harness (`tests/restart-chaos`) restarts the
server during real agent runs and grades the recovery.
> - Several native recovery fixes only apply when a restart lands after
the run has started its native runner. Examples: a restart claim whose
lease lapses in a storage outage (#20529), and a K2 seconds after native
start on a remote sandbox.
> - On Daytona R1, workspace preparation takes 100-250 s and varies a
lot under storage faults. A time-based fault and `--storage-trigger
run-log` both land while the run is still preparing. In three waves,
none reached native start.
> - This pull request adds `--native-trigger`, which fires the planned
fault a set delay after the run's own `native.process_start_requested`
event or first provider event.
> - The benefit is that a wave can reliably restart the server at native
start and prove those recovery paths live.
## Linked Issues or Issue Description
Refs #20529, Refs #20514
**What happened?**
Waves `w-20261110T113240-s11936`, `w-20261110T120901-s11937` and
`w-20261110T124555-s11936` (Daytona R1, K2) were meant to restart the
server after native start. Every K2 landed during remote workspace
preparation, before native runtime selection. The drain took the legacy
interrupt path, so the native recovery path under test never ran.
**Expected behavior**
A wave can fire its planned fault a known number of seconds after the
run starts its native runner.
**Steps to reproduce**
1. Run a Daytona R1 wave with `--faults K2 --fault-after-seconds 110`,
optionally with `--storage-faults ST1`.
2. Read `server.log`. The graceful drain lists the runs under
`interruptedRunIds`, not `restartSuspendedRunIds`.
**Paperclip version or commit**
`master` at
|
||
|
DottaandClaude Opus 5.5
|
db0d523bc8 |
fix(recovery): start a Retry after a restart cut off a relaunch whose earlier Runners were proven stopped (#20537)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a run fails, a person presses Retry and expects a new run to start > - Before a native Retry starts, Paperclip must prove that the previous run's Runner and provider stopped, so two providers never work on the same task > - When the server is killed after it asked for a Runner launch but before it recorded the Runner's process, there is no PID to check. Paperclip then proves the stop from two facts: no Runner process for the run exists on this host, and the run never saved Runner state > - A run that already crossed restarts did save Runner state, from its earlier Runners. Those Runners were proven stopped, with stop receipts. The proof still refused, because it only asked whether any state belonged to the run > - The Retry returned 202, but the task then waited for a board operator to confirm the stop (`confirm_run_stopped`, `process_identity_missing`), until the task timed out > - This pull request lets the proof accept Runner state that only the earlier, proven-stopped Runners could have written > - The benefit is that a Retry after a restart-interrupted relaunch starts right away, with no operator step ## Linked Issues or Issue Description **Subsystem affected** Server native-run stop proof for a Retry or a new message (`server/src/services/native-local-process-stop.ts`, `unidentifiedLocalRunnerProvenAbsent` in `server/src/services/native-runtime/native-session-executor.ts`). **Problem or motivation** Restart-chaos wave `w-20261110T111245-s701`, task t03 (local, codex, C1, K1/K2, `--storage s3 --storage-faults ST7`): run `f1ffeed3` crossed three restarts. Its event history is: 1. Launch request, process identity, stop receipt (first Runner, killed by K1) 2. Launch request, process identity, stop receipt (second Runner, killed by K1) 3. Launch request at 11:13:54.999, then a K1 kill before the Runner's identity was recorded 4. A fourth controller resumed the run, a K2 drain interrupted its preparation, and the run failed (`adapter_failed`, fixed separately by #20514) The person's Retry at 11:14:37 returned 202. Admission reached `recordUnidentifiedLocalRunnerStop`. No Runner process for the run existed, but the Runner state root held `control-plane-state.json` with this run's id, written by Runners 1 and 2. The proof refused, the issue was saved as `waiting_for_user` / `confirm_run_stopped` / `process_identity_missing`, and the wake stayed deferred until the task timed out about 15 minutes later. The harness graded `R3:retry_no_new_run`. **Proposed solution** A Runner persists its command journal (`runner/runner-state.json`) before any command takes effect, and a provider starts only from a command. Its `runner/process-lineage/` writer marker is written before any supervised process starts. So a launch whose Runner never wrote either file after the launch request cannot own a provider. - The launch request event now records `requestedAt`, taken on the launching host just before any backend may spawn. - `recordUnidentifiedLocalRunnerStop` passes that time on only when the latest process event before the request is a stop receipt (every earlier Runner of the run was proven stopped) or there is none. - `unidentifiedLocalRunnerProvenAbsent` then accepts saved state of this run only when `runner/runner-state.json` (if present) and every `runner/process-lineage/` entry were last modified before `requestedAt`. A file time that falls on a whole second (a coarse-timestamp filesystem) must predate the request by 2 s. The existing checks stay: no `--run-id` Runner process on this host, the launching host matches, and no remote lease. **Previous-release story** A launch request saved by the previous release has no `requestedAt`. It keeps today's proof: it is accepted only without saved Runner state of the run, otherwise the board shows the operator confirmation. Nothing is migrated or rewritten. **Alternatives considered** - Treating a proven-gone controller as proof that its Runner stopped. Local Runners outlive a controller restart (graceful drains reattach them), so a gone controller says nothing about its Runner or provider. - Running `paperclip-runnerd reap-lineage` for each writer marker. It needs a Runner PID and binary on the hot Retry path, and a Runner whose journal write failed has no marker to find. ## What Changed - `native-session-executor.ts`: the `native.process_start_requested` payload adds `requestedAt`. `unidentifiedLocalRunnerProvenAbsent` takes `earlierRunnersStoppedBefore` and accepts this run's saved state only when the Runner journal and lineage entries predate it (`runnerStateUnchangedSince`). - `native-local-process-stop.ts`: `recordUnidentifiedLocalRunnerStop` reads the last two process events and passes the request time only after a stop receipt (or no earlier launch). - `explicit-native-continuation.test.ts`: new embedded-Postgres case. A native run with a proven-stopped Runner, saved Runner state, and a later unidentified relaunch: the Retry is admitted and the stop is recorded. It stays blocked (`process_identity_missing`) when the Runner journal or a lineage entry changed after the request, when the request has no `requestedAt` (previous release), when the earlier Runner has no stop receipt, when a whole-second file time is within 2 s of the request, or when a `--run-id` Runner process is alive. A whole-second time 3.5 s before the request is admitted. ## Verification - Red/green: `cd server && npx vitest run src/services/explicit-native-continuation.test.ts -t "proven stopped"`. On master the `proven` case fails (admission returns null); with the fix all 8 cases pass. - Related suites: `explicit-native-continuation.test.ts`, `issue-confirm-run-stopped-routes.test.ts`, and the unidentified-Runner and stop cases of `native-session-executor.test.ts`. - Live, the s701 seed with the original wave's flags (`--env local --adapters codex,claude --workloads A1,C1 --faults K1,K2 --triggers time,result --concurrency 4 --tasks 4 --seed 701 --storage s3 --storage-faults ST7 --keep`): - `w-20261110T131243-s701` (final head) and `w-20261110T125259-s701`, master plus this fix plus #20514 merged locally: 4/4 pass each, every task crossed a restart, no R3 signature, 0 speed breaches. - `w-20261110T125616-s701`, master plus this fix without #20514: t01 and t02 pass. t03 and t04 hit the drain-during-preparation failure that #20514 fixes (`R3:retry_needed`). The Retry started a new run each time and both tasks finished (`retry_no_new_run` is gone). New launch requests carry `requestedAt`. - The restart timings of these replays did not land in the exact gap between a launch request and its process identity. The red/green test covers that window. ## Risks - Low. The change only widens one fail-closed proof, and only for launch requests saved by this release, after a stop receipt, with Runner files older than the request. The proof compares file times with a time taken on the same host. A backwards wall-clock step between the request and a Runner's first write could hide that write. The `--run-id` process check still has to pass, so that Runner must also be gone. - The Retry path adds one `lstat` per file in a small directory. It adds no wait and no work before `turn.start`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
faae6b919f |
fix(runner): explain task scope on missing tool inputs (#19774)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents read saved task documents through native tools.
> - Active-task tools use the task bound to the run. Other task tools
require a task UUID.
> - Agents sometimes call the second set with only a document key.
> - The validator rejects the call, but gives no instruction about the
active-task tool.
> - This pull request adds scope instructions to the tool schemas and
returns them for missing required fields.
> - Agents can correct a rejected call. Task selection and permission
checks stay in place.
## Linked Issues or Issue Description
**What happened?**
Two observed document reads sent only `{"key":"sentry-case"}` to
`read_task_document`. Each returned `Invalid tool input: / must have
required property 'taskId'`. Each later retry with a task UUID returned
without an error.
Pattern: [Task reads fail when agents omit
taskId](https://www.braintrust.dev/app/Paperclip/p/Paperclip%20Staging/patterns/703d8c29-6a40-4aa2-b2c6-e2c5c36b5657).
Related PR: #18824 changed document key validation. It does not address
task scope.
**Expected behavior**
The tool contract must state which task it reads. A missing-field error
must include the schema owner's recovery instructions. A retry must
still pass validation and task permission checks.
**Steps to reproduce**
1. Call `read_task_document` through the MCP bridge with only a document
key.
2. Inspect the error sent to the caller.
3. Retry with an authorized task UUID. Confirm the returned document
belongs to that task.
### Hypothesis and thesis for a fix
Hypothesis: generic tool descriptions contribute to confusion between
active-task and explicit-task tools. An agent can also ignore a correct
schema. These tests do not measure the rate of model mistakes.
Thesis: Paperclip owns task scope instructions. Its five explicit-task
tools now describe the required UUID and the active-task alternative.
The two MCP transports return direct missing-property descriptions from
the admitted schema. The transports do not infer task IDs or apply
Paperclip task policy.
Proof: the same HTTP MCP regression fails on baseline
`f673bd3cb5b5b65ec51781e37355e168c3fe8ae4` and passes with the fix. It
still rejects before handler dispatch. An explicit retry reaches the
requested task. An isolated real API and PostgreSQL test checks the
saved revision, active-task selection, malformed UUIDs, and
cross-company denial through both transports.
## What Changed
- Add UUID and active-task guidance to `get_task`, `comment_on_task`,
and the three explicit-task document tools.
- Return bounded, distinct descriptions for up to three direct missing
required fields. Keep ordinary validation errors unchanged.
- Share this formatter between the Runner and OpenCode MCP bridges.
- Add transport regressions, real route and database checks, and tool
documentation.
## Verification
- Baseline red: both transport regressions return only the
required-property error and fail the recovery instruction assertion.
- Candidate green: 58 Runner and OpenCode transport and error formatter
tests pass on Linux with Node 24.22.2.
- Final-head server verification passed all 115 tests: 113 document-key
checks and two real API integration cases.
- The two local integration cases use real MCP HTTP, agent authority,
API routes, and disposable PostgreSQL. They read a selected other task
and its saved revision, read the active task separately, reject missing
IDs for all five tools, reject malformed UUIDs, and return 404 for a
foreign company task.
- Runner TypeScript typecheck and build pass.
- The full server typecheck passed on head
`03a37e9e50896416cfd45be33bf6dbe3e13169df` with
`PAPERCLIP_TYPECHECK_LOW_MEMORY=1 node scripts/typecheck-layers.mjs`.
The first run ended without a compiler diagnostic. Final-head CI passes
and Greptile is 5/5.
- Full preflight stops because cargo is absent. Rust checks were not run
locally.
- One existing delayed-body fetch test failed with a Node 25 fetch
runtime error. All transport tests pass on Node 24. The first
integration attempt used the wrong fixture return shape; the test was
corrected before the passing run.
- CI follow-up: the Runner test program reported four TS18046 errors
because `Response.json()` returns `unknown` under CI typings. Both tests
now declare the response type. `pnpm exec tsc -p tsconfig.test.json
--noEmit` passes, and both transport suites pass again (56 tests). Final
head `ffccebca19fe9e21a097829adc020679ffa7d86c` passes CI and Greptile
5/5.
- Review follow-up: the foreign task now has a saved document with a
distinct body. Both integration cases passed again after this test-only
change. The read must return 404 and must not return that body.
Current-head CI passes and Greptile is 5/5.
- Final CI:
https://github.com/paperclipai/paperclip/actions/runs/42352822739 . All
required jobs passed for `ffccebca19fe9e21a097829adc020679ffa7d86c`.
Greptile is 5/5 with no unresolved review thread.
- Real local API behavior is verified. External model behavior and
deployed recovery are not verified. No merge or deployment is included.
## Risks
- Missing-field errors now contain schema descriptions. The text is
limited to direct properties in the admitted schema, at most three
distinct descriptions and 1,000 characters each.
- Nested schemas and references do not get recovery text. Their
validation errors remain intact.
- Clearer instructions can help callers recover. They do not guarantee
that a model will choose the correct tool.
## Model Used
- OpenAI GPT-6 via Codex, with tool use and code execution. The exact
deployment model ID and context window are not exposed to this session.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Claude <noreply@anthropic.com>
|
||
|
DottaandPaperclip
|
cac82ed4d9 |
fix(ui): show parent tasks in responsive breadcrumbs (#20366)
## Thinking Path > - Paperclip helps people manage AI agents and their work. > - Tasks can have a parent task. > - The task header does not show that parent in its navigation path. > - On a phone, the header removes the task path and shows only the current title. > - Long parent and current titles must share the available space. > - This pull request adds a parent link and limits each displayed task title to 48 characters. > - People can move to the parent and keep the current task number in view. ## Linked Issues or Issue Description **What existing behavior does this improve?** The navigation path in a task header. **Current behavior** The global path omits the parent. The phone header shows only the current task. Long parent titles cannot share the row safely. **Proposed behavior** Show the immediate parent before the current task. Limit each displayed title to 48 Unicode code points, with an ellipsis. Limit the parent to 40% of the row. On phones, remove the collection link and parent number so both titles and the current task number fit. **Reason and benefit** People can see the task context and open its parent. Full titles stay in hover text and accessible names. Private parents have no link. Related: #9302 adds project navigation. This change adds task parent navigation. ## What Changed - Add the immediate parent from the existing authorized ancestor response to the global task path. - Make parent and current titles shrink. Apply a 48-character display limit without changing stored titles. - Keep the current task number visible on phones. Keep private parent labels noninteractive. - Add ten Storybook scenarios with the production header. Cover linked parents, long titles, phone widths, tablet width, dark theme, standalone tasks, Mine source, and private parents on desktop and phones. - Add regression tests and document the layout rules in DESIGN.md. - Keep the source collection and filters when people follow the parent link. Cover route state, stored state and legacy query links. - Use the existing noninteractive lock chip for private parents. Hide its parent number on phones. Refresh its control key when an open task changes screen size. Both regressions failed before their fixes and pass after them. - Refresh the SDK public source registry after the master update. The existing helper keeps all runtime and dependency pins unchanged. This repairs three support tests that failed on stale master source identities. ## Verification - Focused UI tests: `pnpm exec vitest run ui/src/components/BreadcrumbBar.test.tsx ui/src/pages/IssueDetail.test.tsx` — 280 passed. - UI typecheck: `pnpm --filter @paperclipai/ui typecheck` — passed. - `pnpm build-storybook` — passed. - `pnpm check:token-gates` — passed. - Catalog checks and 30 map/catalog tests — passed. - Chromium: all ten scenarios rendered with no page errors. Story play functions finished. Checked 1440, 768, 390 and 320 pixel widths, keyboard parent navigation, private parent links, title limits and horizontal overflow. - Private-parent browser check: resize an open task from 1440 to 320 pixels and back. The chip changes from the parent number to Private and back. It remains noninteractive. - Screenshots were captured in light and dark themes. They are task evidence and are not committed. - This verifies the header with fixture data and the page wiring with mocked APIs. It does not claim live backend authorization or a pixel baseline comparison. The static server returns 404 for optional shell context reads. After keyboard navigation, destination task reads also return 404 because the preview has no task backend. The header renders and all play checks pass. - Case discovery and OpenAPI tests: 47 passed. - Current head: `e7a1ada02e75e9b2cbd5f6076ad6c113954f3a4b`. [CI run 42834603680](https://github.com/paperclipai/paperclip/actions/runs/42834603680) passed. All 61 head check runs are complete: 58 passed and three conditional jobs were skipped. The legacy Snyk status passed. [Greptile](https://github.com/paperclipai/paperclip/pull/20366#issuecomment-6751938924) reviewed this head at 5/5 with no actionable findings. All four inline review threads are resolved. - After the master update: all 280 UI tests, UI typecheck, token gates, Storybook build, 30 catalog tests and ten Chromium stories pass. All six screenshots match the saved images byte for byte. - SDK and process support checks: 105 tests pass. The source pin helper has eight passing tests. The three SDK failures reproduced before the registry refresh and passed after it. The helper check reports no remaining source drift and no runtime pin changes. - The earlier case discovery failure reproduced before the summary correction and passed after it. All server shards passed on the next head. That CI run failed one unchanged Rust process-reaping test: its owned tool stopped, but two unattributed orphan processes blocked verification. The earlier CI run passed the same Rust inputs. Cross-test process interference is possible; the historical log does not prove the cause. The next CI run passed Rust. That run failed three SDK checks on stale master source pins and one process inspection check. The source pin repair is in this head. The process check passes locally, but the earlier process inspection failure remains unexplained. - The same prior run failed the cohort migration resume test. Its upload barrier releases after five seconds; maximum uploads were seven and three in the two attempts, below eight. Three server files also exceeded the 180-second file budget. These test inputs come from master. A local attempt could not reproduce that CI assertion because its child Git command failed in this execution environment. Fresh CI 42834603680 passed these paths, including every server shard and Runner support. No test or process safety check was weakened. ## Risks - Breadcrumb titles display less text. Hover text and accessible names retain the full title. - The global path shows the immediate parent. People can follow parent links through a deeper task tree. - The shared breadcrumb component also serves other pages. Existing header and agent-switcher tests pass. - No schema, API behavior or stored title change. - SDK source identities must match the integrated source. The registry change uses the existing helper and preserves all runtime and dependency pins. ## Model Used - OpenAI Codex, based on GPT-6. The exact runtime model ID and context window size are not exposed. Used reasoning, repository tools and Chromium verification. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude
|
8d2bf78323 |
test(runner-e2e): resync the SDK public source pins with master (#20550)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner support suite checks that the plugin SDK's public types come only from reviewed source inputs, pinned by git blob hash in `tests/runner-e2e/source-sdk-public-inputs.json` > - #19614 landed those pins, generated against an older master; since then `package.json` and ten files under `packages/shared/src` changed > - So `source-sdk-public-type.test.ts` (2 tests) and `source-sdk-evidence.test.ts` (1 test) fail the `ci / Runner support suite` check on every PR merge ref > - This pull request regenerates the source pins with the script made for that job > - The benefit is a green Runner support suite for every open PR again ## Linked Issues or Issue Description **What happened?** Since #19614 merged, `ci / Runner support suite` fails on every PR (for example #20047 and #20538) with `SDK public source proof unavailable` and `Source SDK evidence privacy check failed`. **Steps to reproduce** 1. On master at |
||
|
DottaandClaude Opus 5.5
|
902f376796 |
fix(recovery): keep a remote restart claim's lease through slow preparation (#20540)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - After a restart, a remote run resumes on the same heartbeat run under a restart claim: a lease on its native coordinator row, 60 s for a remote claim (`REMOTE_CONTROLLER_LEASE_MS`) > - Preparation then runs before the native executor takes its own lease renewal. It includes the run-log begin and the workspace restore, which can each wait out a storage outage for minutes (#20299) > - Preparation renewed the claim only for `reattach_remote_runner`. A `bootstrap_incomplete` or resume claim kept its 60 s lease with nothing renewing it > - So a preparation wait longer than a minute let the lease lapse. The run failed at its next writer-authority check (`run_log_writer_authority_lost`) the moment storage came back > - This PR renews every remote restart claim during preparation. When the same owner's executor claims at the next attempt, it takes over that renewal > - The benefit is that a resumed remote run survives a long storage outage during preparation instead of failing as soon as storage returns ## Linked Issues or Issue Description **What happened?** In restart-chaos waves on Daytona R1 with `--storage s3 --storage-faults ST1` and K2 at +110 s, a resumed `bootstrap_incomplete` run's `heartbeat.prepare.run_output` ended `failed` right after the outage ended: - `w-20261110T101517-s11936` t01, run `9d39550a`. The claim was at 10:19:55 and the outage ended at 10:22:18.15. `run_output` failed at 10:22:18.17, after 141 s. - `w-20261110T081647-s11936` t02, run `9da10424`. The claim was at 08:21:29 and the outage ended at 08:23:30. During the outage the journal claim fails on its head `GET` before it checks authority. The first attempt after the outage reaches `claimFrom`'s `assertAuthority` (`run-log-authority.ts`), which requires a live coordinator lease. That lease expired 60 s after the claim. **Why not just widen the condition?** For a `bootstrap_incomplete` claim, the executor's own claim moves the coordinator attempt to N+1 (`nextNativeProviderAttempt`). `startNativeSessionExecutionLeaseRenewal` shares one renewal per lease owner and threw `native_session_renewal_binding_changed` when the attempt differed. That is presumably why preparation renewal was limited to reattach claims. **Related:** #20460 (live status while a run waits out a storage outage). #20514 settles a failure that this lapse used to leave silent. ## What Changed - `heartbeat/recovery.ts`: start the preparation lease renewal for every remote restart claim (`scope.remote`), not only `reattach_remote_runner`. - `native-runtime/native-execution-lease.ts`: - When the same owner's executor starts its renewal at a higher attempt, it takes over the shared renewal, rebinding the attempt and `onLost`. A lower attempt or any other binding change still throws. - A renewal that failed under the old binding is not counted as a loss. - A failed renewal is not a loss when the coordinator shows the same owner already at a higher attempt (the executor's claim committed while the renewal was in flight). The renewal deadline still ends ownership if no executor takes over. - Tests: one recovery case and three lease-renewal cases (below). ## Verification - `cd server && npx vitest run src/services/native-runtime/native-execution-lease.test.ts src/services/heartbeat/recovery.test.ts src/services/native-runtime/native-execution-lease-reacquire.test.ts src/services/native-runtime/native-runner-restart-lease-renewal.integration.test.ts src/services/native-runtime/native-runner-restart-preparation.integration.test.ts`: 143 pass. `npx tsc --noEmit`: pass. - Red/green, recovery test "keeps a remote bootstrap restart claim through preparation longer than its first lease, then hands it to the next attempt": the claim's first lease ends 4 s in. Preparation runs past it and the run-log writer authority still passes. Then the executor claims attempt 2 and its renewal keeps extending the lease with no detach. - Red with master's `recovery.ts`: the lease is never extended. - Red with only the `recovery.ts` change: `native_session_renewal_binding_changed`. - Green with both. - Red/green, lease unit tests: handover to the next attempt (and refusal of a lower one); a failure under the old binding is not a loss; a failure while the coordinator shows the same owner at a higher attempt is not a loss. All three fail on master. - Live wave: results in a comment below. ## Risks - Medium-low. - The preparation renewal now runs for remote `bootstrap_incomplete` and `resume_dead_runner` claims too. It renews the same coordinator row under the same CAS conditions reattach claims already use. - If preparation truly loses the lease (another owner takes it), its `onLost` still runs (`detachNativeSessionsForRestart`) and authority checks still fence writes. - The new non-loss case applies only when the coordinator names this exact owner at a higher attempt. The renewal deadline still bounds it. - Previous-release state: this persists nothing new. A claim saved by the previous release resumes unchanged; its preparation is simply renewed now. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1110.0-canary.13 |
||
|
DottaandClaude
|
94cfb84150 |
fix(scripts): build the plugin SDK in the agent worktree preflight (#20473)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Contributors and agents prepare each new git worktree with `node
scripts/agent-preflight.mjs`
> - The server imports `@paperclipai/plugin-sdk` through its built
`dist`. The preflight did not build that `dist`.
> - On a fresh worktree, the server and the restart-chaos harness stop
at startup with `ERR_MODULE_NOT_FOUND` for
`server/node_modules/@paperclipai/plugin-sdk/dist/index.js`
> - The SDK exports `dist` on purpose: plugin workers load it under
plain Node. CI, `paperclipai run` and the typecheck scripts already
build it with `scripts/ensure-plugin-build-deps.mjs`.
> - This pull request makes the preflight run the same builder as a step
> - The benefit is that one preflight command gives a worktree that can
start the server and the harness
## Linked Issues or Issue Description
**What happened?**
After `node scripts/agent-preflight.mjs` on a fresh worktree, `node
cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts ...` stops
at startup. The error is `ERR_MODULE_NOT_FOUND` for
`server/node_modules/@paperclipai/plugin-sdk/dist/index.js`, imported
from `server/src/services/reusable-sandbox-resume-error.ts`. The
workaround is `pnpm --filter @paperclipai/plugin-sdk build`.
**Expected behavior**
The preflight prepares everything that the server needs to start. The
server and the harness import their modules without a manual build.
**Steps to reproduce**
1. Make a new worktree from `origin/master`.
2. Run `node scripts/agent-preflight.mjs`.
3. See that `packages/plugins/sdk/dist` does not exist.
4. From `server/`, import
`./src/services/reusable-sandbox-resume-error.ts` with tsx. The import
fails with `ERR_MODULE_NOT_FOUND`.
**Paperclip version or commit**
`origin/master` at
|
||
|
|
f4337bb5fb |
test(evals): land the chat catalog eval harness on master (#19614)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat-feature evals measure whether agents complete real chat requests through the product > - The chat catalog eval harness and its repairs live on the `chat-eval/master-candidate` branch, not on master > - Today's work made 139 of 242 chat evals pass in verified single-case live runs. Each product fix landed on master as its own pull request > - The eval-side changes (setups, graders, fault seams, evidence capture) still exist only on the candidate > - This pull request lands the harness on master. Master stays the source of truth for product behavior > - The benefit is one tree: evals run against master commits, and product fixes reach the evals as soon as they merge ## Linked Issues or Issue Description Refs #19441, Refs #19599, Refs #20270, Refs #20334, Refs #20342, Refs #20344, Refs #20354, Refs #20360, Refs #20373, Refs #20382, Refs #20409 **What existing behavior does this improve?** The chat catalog eval harness and the product it evaluates live on a long-running branch outside master. Eval fixes do not reach master, so master-based eval runs use old setups and graders. **Subsystem affected** `tests/runner-e2e` chat catalog harness, plus a small set of product changes that exist only on the candidate. **Proposed change** Land the harness on master. Every product fix from this eval work already merged to master as its own pull request. Conflicts keep master's product behavior. ## What Changed - Most of the diff (808 of 851 files) is under `tests/runner-e2e/`: chat-catalog domains, scenarios, descriptors, the TestOps manifest, private evidence tooling, and their tests. - Graders now check saved state, setup literals and rendered or clicked links. Many keyword and regex checks on free-form replies are removed. Each changed grader has a bumped `oracleVersion` and calibration tests (a valid reply, a reworded valid reply, and broken variants that must fail). - The shared link helper `hasLink` no longer treats a list link (for example `/COM/issues?view=mine`) as a link to every task without a URL key. - Setup fixes build real state through supported product routes: routine history, agent grants through the board grant route, budget holds with a real cap, and a storage-layer archive fault for `inbox-search.archive-partial-failure`. - Browser boundary journeys (sign-in, chat-disabled, onboarding) now write the standard pass evidence files. - Retargeted cases, approved by the maintainer: `agent-chat.chat-disabled` (Conference Room gate), `agents.restore-instructions` (no saved instruction history), `access.manage-membership` (member permission edits stay board-only). - The private evidence trusted files (`manual-private-handoff.ts`, `manual-private-retention.ts`, `manual-private-report-assets.ts`) are byte-identical to master. - Product files outside `tests/` (43 files, +1,328 −49) carry candidate-only changes that master does not yet have. Most are tests. The main code changes are in `request-human-input.ts`, `heartbeat-run-summary.ts`, `native-completion-feedback.ts`, `native-run-finalizer.ts`, `openapi.ts`, `secrets.ts` and `IssueChatLongThreadPerf.tsx`. ## Verification - `pnpm vitest run chat-catalog-domains chat-catalog-contract.test.ts chat-catalog-consistency.test.ts unit-launcher.test.mjs --testTimeout=180000` in `tests/runner-e2e`: 269 files pass, 3,550 tests pass. - `node --test chat-full-catalog-testops-manifest.test.mjs`: 10 pass. - `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`: no change. - `tsc` for `server` and `tests/runner-e2e`: clean. - Live single-case verification: 139 of 242 cases passed on paid runs, and a person read the saved evidence for each. These are single-case runs, not one full-cohort run. ## Risks - Large diff. Most of it is test-only, but the 43 non-test files change product code paths. Review those files first. - Some graders are now less strict on reply wording. A person must read replies for meaning on those cases. - 42 cases need disposable outside accounts (Notion, Slack, GitHub, Daytona and others) and stay blocked. `access.invite-teammate` stays failing until local agent isolation lands, because the runner's durable state keeps a live invite token. - The paid workflow compares the trusted private-evidence files with master. They are unchanged, so this risk is low. ## Model Used Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution, in Claude Code. Subagents used the same model. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
60b9147e7c |
fix(native): accept harness diagnostics in a remote session-opened bootstrap proof (#20543)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on a remote sandbox (Daytona) keep their controller
state in a Postgres checkpoint. The controller writes every command to
that checkpoint before it sends the command.
> - After a restart, recovery uses that record to prove that a drained
remote bootstrap reached no provider turn. If so, it bootstraps the run
again on the same heartbeat run.
> - A graceful restart seconds after native start often catches a runner
that opened its session and emitted ACPX `harness.diagnostic` events,
but was never sent a turn.
> - The remote no-turn proof did not accept `harness.diagnostic`, so
recovery blocked these runs as `ambiguous_remote_recovery_evidence` and
the task needed a manual Retry.
> - This pull request accepts `harness.diagnostic` in that proof. The
proof's command check still requires that no turn command was ever
issued.
> - The benefit is that a restart just after native start re-bootstraps
the run automatically.
## Linked Issues or Issue Description
Refs #20538 (the harness trigger that reproduced it)
**What happened?**
Restart-chaos wave `w-20261110T091639-s11936` t02, run `c4fc7f03`
(Daytona R1, K2): a K2 landed 13 s after native start. Recovery returned
`ambiguous_remote_recovery_evidence`, and the run blocked with
`native_restart_recovery_blocked` (R3 retry_needed). I reproduced it
with a kept database in `w-20261110T132656-s11936` t02, run `ecd98d55`,
using `--native-trigger native-start+5` from #20538. Decoded controller
checkpoint:
- commands: `run.prepare`, `session.open`, `session.goal.get` (no turn
command);
- committed events: `session.started`, `session.capabilities.updated`,
`session.goal.snapshot`, `harness.diagnostic`, `harness.diagnostic`,
`session.goal.snapshot`;
- `ackedSourceSeq` 6, no `sessionCheckpoint`, no projected provider
events.
`nativeRunnerCheckpointRemoteBootstrapProof` returned `null` only
because `harness.diagnostic` was not in `SESSION_OPEN_EVENT_TYPES`.
**Expected behavior**
Recovery classifies the run `bootstrap_incomplete`
(`remote_session_opened_without_turn`) and bootstraps it again.
**Steps to reproduce**
1. Run a Daytona R1 codex wave with `--faults K2 --native-trigger
native-start+5`.
2. The suspended run is blocked with
`ambiguous_remote_recovery_evidence`.
**Paperclip version or commit**
`master` at
|
||
|
DottaandClaude Opus 5.5
|
7329eaf353 |
perf(ui): open sidebar Recents tasks from cache (#20450)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The board UI sidebar has a Recents list of the tasks a person opened
last
> - The Recents list renders from a local snapshot, but its links had no
warm-up. A click waited for the task-detail code and about nine
first-frame reads
> - Task lists (Inbox, Tasks) already warm a task on hover, press and
click, and #20355 added a batched `chat-bootstrap` read that had no UI
consumer
> - The hosted database is CPU-bound, so a fix must add at most one read
per page load and must not compete with the page's own reads
> - This pull request gives Recents the same open-intent prefetch as
task lists, warms the task-detail code, and preloads the top ten recent
tasks with one `chat-bootstrap` request after the page is idle
> - The benefit is that a click on a recent task shows its chat from
cache, without a wait for the API
## Linked Issues or Issue Description
Refs #20355 (server read this PR consumes). Builds on #20389, #20398 and
#20419.
**Is your feature request related to a problem?**
Opening a task from the sidebar's Recents was slower than opening it
from a task list. The Recents links had no prefetch. The task page
waited for its first-frame reads after the click.
**Describe the solution you'd like**
Recents links use the task lists' open-intent prefetch. When the page is
idle, one batched request loads the first frame of the most recent tasks
into the task page's caches.
**Describe alternatives you've considered**
Per-task prefetch of every recent task. That makes up to ten times nine
requests per page load, which the hosted database cannot afford.
**Additional context**
I searched open PRs and remote branches for "chat-bootstrap" and
"instant task opens". The only related work is the server-side branches
that merged as #20355. There is no other in-flight UI consumer of
`chat-bootstrap`.
## What Changed
- `ui/src/hooks/useIssueNavigationPrefetch.ts`: the open-intent handlers
from `IssueLinkQuicklook` move into one hook (light warm on pass-over;
`history: true` on a resting hover or focus, a mouse or pen press, and a
click). `IssueLinkQuicklook` now uses the hook. Its behavior does not
change.
- `ui/src/components/RecentTaskNavItem.tsx`: one Recents link wrapper
that applies the hook. The classic list (`SidebarRecentTasks`), the
Coordinator task rows and `DesignSidebar` use it. `SidebarNavItem`
accepts the handlers through a new optional `linkHandlers` prop.
- Classic Recents links now open `/issues/<identifier>` (Coordinator
already did). The old UUID link remounted the task page after a redirect
to the identifier route.
- `ui/src/hooks/useRecentTaskOpenWarmup.ts`: when Recents shows a task,
it fires the task-detail code warm-up (`useWarmTaskDetailCode`). After
two frames, a visible tab, no fetching queries and an idle callback, it
sends one `issues/chat-bootstrap?ids=` request for the ten most recent
tasks. It skips tasks whose first frame is cached and fresh. It sends no
request for an empty list or a hidden tab. It sends at most one request
per account per page load. Unmount or an account change aborts the
request.
- `ui/src/lib/issueChatBootstrapCache.ts`: `seedIssueChatBootstrap`
writes each field under the keys the task page reads, for the requested
ref, the id and the identifier: core issue, recent comments, comment
input facts (one batch, through the new `seedIssueIdBatch`), recent
activity, the recent run window (through the new shared
`buildRecentIssueRunWindow`), interactions, attachments, live runs and
active run. An explicit `null` active run is stored. A missing field is
not stored. It does not replace data that is as new or newer, data with
a read in flight, or data with a pending invalidation. Keys whose page
merges observed history (comments, activity, runs, input facts) are
filled only when empty. Items for another company are ignored.
- The seed fills the `?view=core` key, not the full detail key.
`seedIssueDetailCache` is not used: the core payload cannot stand in for
the detail read's related data, and seeding it there would suppress that
read.
- `issuesApi.chatBootstrap` client method.
- `tests/perf/collections/recent-task-open.spec.ts` and its README
section.
## Verification
Local disposable instance (`local_trusted`, production UI build,
embedded PostgreSQL, collection fixture plus 12 comments on each of 8
recent tasks, streamlined UI with the Coordinator sidebar). Journey:
open Projects with 8 Recents, wait for network idle plus 3 s, click a
recent task, measure until its latest comment is on screen. Each run
uses a fresh browser context (empty HTTP cache and storage apart from
Recents). Median of 5 runs, same server and fixture for both arms:
| Arm | Click to first chat frame | Idle page API requests |
| --- | --- | --- |
| `master` (
|
||
|
DottaandClaude Opus 5.5
|
c9226de30b |
fix(heartbeat): leave a preparing run to the next controller after a graceful drain (#20514)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The heartbeat service runs each agent task. A graceful shutdown
(SIGTERM) drains running native runs and hands them to the next server.
> - The drain detaches the native runner session and releases the run's
lease with the recovery state `awaiting_runner_reattach`.
> - An in-process execution can still be in run preparation at that
moment, for example while the run-log journal claim waits on a slow or
unavailable object store.
> - When that execution then lost writer authority
(`run_log_writer_authority_lost`), the generic failure path settled the
run as `adapter_failed`. The next server then saw
`native_continuation_requires_reconciliation`, and replay stayed
blocked.
> - This pull request makes a preparing execution that a restart
detached stop quietly, the same way a detaching native session already
does.
> - The benefit is that a slow preparation step plus a graceful restart
no longer turns into a failed run that needs a manual retry.
## Linked Issues or Issue Description
**What happened?**
In restart-chaos wave `w-20261110T092411-s701` (local, A1/C1, K1/K2, ST7
read-after-write lag), three of four tasks failed with R3
`retry_needed`. For run `218a0425`, the K2 SIGTERM drain wrote
`native.recovery.transition` with `awaiting_runner_reattach` and
released the lease. About 3 s later, the old server's in-process
execution, which was still in `runOutput.begin`, got
`run_log_writer_authority_lost`. `runCompletion.failRun` then settled
the run `adapter_failed` with a durable settlement. The next server
found `native_continuation_requires_reconciliation` and did not replay
the run.
**Expected behavior**
The drain hands the run to the next server. The old execution must not
settle it. A real authority loss outside a drain must still fail the
run.
**Steps to reproduce**
1. Start a native run recovery, and hold it in preparation (slow store,
ST1 outage wait, or ST7 lag).
2. Send SIGTERM so `drainRunningRunsForShutdown` suspends the run.
3. Let preparation continue and check writer authority. The run settles
`adapter_failed`.
**Paperclip version or commit**
`master` at
|
||
|
DottaandClaude Opus 5.5
|
15a734d068 |
fix(heartbeat): show a storage outage wait as live status, then record it once (#20460)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - In cloud mode a run reads and writes file storage on its critical path: it starts its run log (`s3_segments`), restores its saved workspace, and saves it again > - Since #20299 a run waits out a storage outage there for up to 5 minutes instead of failing > - That wait was silent. The board showed "running" with no sign of life, so a storage wait looked like a dead run (restart-chaos R5 `silent_dead_run`) > - The first version of this PR wrote a run event on every wait. That writes the run row (`next_event_seq`, `updated_at`) while the run-log claim is still in progress, and it covered only the run-log begin > - This PR shows the wait as live run status instead, writes nothing during the wait, and records one lifecycle event when the wait ends. It covers every critical-path wait in the run's storage scope > - The benefit is that a storage outage shows on the board as waiting, the run's history keeps one honest line about it, and the claim path is left alone ## Linked Issues or Issue Description Refs #20299 (wait out a store outage on a run's critical path). **What happened?** - `w-20261110T043533-s11936` (Daytona R1, ST1 194 s, K2): run `d49f58d8` waited 186 s in `heartbeat.prepare.run_output` with no event. The grader failed it as `R5:silent_dead_run`. - `w-20261110T105903-s11936`: the restore download (`storage.snapshot.download`) waited 182 s with no sign of life. **Expected:** the board shows the run waiting for file storage, and nothing about the notice can disturb run-log ownership. **On the failed `run_output` seen on builds of the first version** (`w-20261110T101517-s11936` t01 run `9d39550a`, `w-20261110T081647-s11936` t02 run `9da10424`): the evidence points to an expired restart lease, not to the notice's row writes. Neither the writer-authority check (`run-log-authority.ts`) nor the journal claim compares `next_event_seq` or `updated_at`. A remote `bootstrap_incomplete` restart claim gets a 60 s lease (`REMOTE_CONTROLLER_LEASE_MS`), and preparation renews it only for `reattach_remote_runner` claims. In t01 the claim was at 10:19:55, the outage ended at 10:22:18.15, and `run_output` failed at 10:22:18.17, on the first authority check after the store came back. That fix ships separately as its own PR. ## What Changed - `storage/storage-unavailable.ts`: - The run's storage scope can carry a wait observer (`reportStorageOutageWaitsTo`). - `retryThroughStorageOutage` tells it about each backoff (`waiting`). The operation that owns the wait reports one `ended` (`recovered` true or false), so nested waits do not double-report. - `storageOutageWaitFor(purpose, work)` names the step. - An observer that throws never changes the wait. `endStorageOutageWaitOn` keeps the observer. - `heartbeat/storage-wait-notice.ts` (new): - Each wait sets live runtime status (phase `waiting_for_storage`, "Waiting for file storage before starting the run log (45s so far)") and publishes `heartbeat.run.progress`. Nothing goes to the database. - When the wait ends, it writes one `lifecycle` warn event: "Waited Ns for file storage before …", with payload `reason: "storage_outage_wait"`, `purpose`, `waitedMs` and `recovered`. - `heartbeat.ts`: installs the notice next to `endStorageOutageWaitOn` (one line). - Purposes: - run-log begin: "starting the run log"; - `loadWorkspaceSnapshotArchive`: "restoring the workspace"; - `uploadWorkspaceSnapshot`: "saving the workspace". - `packages/shared`: adds `waiting_for_storage` to `HeartbeatRunStatusPhase`. - `tests/restart-chaos/live-status-evidence.ts` (new): R5 also counts the board's live run status (`/api/issues/:id/live-runs` `currentStatusUpdatedAt` / `lastEventAt`) as evidence that a running run is alive. Wiring it into `observer.ts` takes one import and one changed line. - `doc/run-log-events.md`: documents the live status and the one event. - Removed: the first version's per-wait run event. ## Verification - `cd server && npx vitest run src/services/heartbeat/run-output.test.ts src/__tests__/storage-outage-wait.test.ts src/services/remote-workspace-snapshot-storage.test.ts src/services/object-byte-journal.test.ts src/__tests__/storage-s3-credential-rotation.test.ts`: 103 pass. - Red/green: with the observer disabled (master behaviour), these new tests fail. With this change they pass: - run-output "shows the wait as live status and writes nothing to the run while its log claim waits": no run-row write and no event during the claim's waits; the live status is `waiting_for_storage`; the claim's writer-authority check still passes; one lifecycle event afterwards; - the helper's observer tests (one end per wait; gave-up reporting; a throwing observer; the Stop signal keeps the observer); - snapshot storage "tells the run it is waiting for storage while a save or a restore waits out an outage". - `PAPERCLIP_IN_WORKTREE=false npx vitest run --config tests/restart-chaos/vitest.config.ts tests/restart-chaos/live-status-evidence.test.ts tests/restart-chaos/observer.test.ts tests/restart-chaos/triggers.test.ts`: 65 pass. `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: pass. `cd server && npx tsc --noEmit`: pass. - Live restart-chaos waves: results in a comment below. ## Risks - Low. - The retry policy, its budget and Stop handling are unchanged. - During a wait, the only new effect is an in-memory status and a live event. - Afterwards, one event per wait, written without blocking the run. - Runtime status is per server process. A board on another instance sees the waiting line only after the closing event. - Shared journal work (`runSharedStorageWork`) keeps its own scope without the observer. Its waits surface through the enclosing run-scoped wait, as before. - Previous-release state: this persists no new run or session state. A run saved by the previous release resumes unchanged. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
bf514759f9 |
fix(run-log): answer 404 for an object run log that was never written (#20539)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Each heartbeat run keeps a run log, and the board reads it through
`GET /api/heartbeat-runs/:id/log`
> - A run can store its log in object storage (`s3_segments`)
> - When a run fails before its log is written (for example in
`runOutput.begin`), the object store throws a plain `Error("Object run
log not found")`
> - The route turns that plain error into a 500, so the board shows a
server failure for a log that simply does not exist
> - The local file store already answers 404 `Run log not found` for the
same case, and the UI treats a 404 as an unavailable log
> - This pull request makes the object store throw the same 404
> - The benefit is that a missing object run log looks like a missing
log, and a real storage failure still answers 500
## Linked Issues or Issue Description
**What happened?**
`GET /api/heartbeat-runs/:id/log` answered 500 `Internal server error`
for an `s3_segments` run whose log was never written. The server error
was `Object run log not found` from the object run-log store.
**Expected behavior**
The route answers 404 `Run log not found`, the same as the local file
store. The transcript view already treats a 404 for a finished run as a
missing log, and the log viewer already treats it as an unavailable log.
A real storage failure still answers 500.
**Steps to reproduce**
1. Use S3 storage for run logs.
2. Make a run fail before its run log is written (for example in
`runOutput.begin`).
3. Read `GET /api/heartbeat-runs/<run id>/log`. It answers 500.
**Paperclip version or commit**
`master` at
|
||
|
DottaandClaude
|
c874cfa4da |
test(restart-chaos): anchor two-server speed samples on the primary's restart (#20535)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart-chaos harness grades each wave's speed, including how fast runs recover after a server restart > - Two-server faults (MS1–MS3, #20518) record their `readyAt` after the second server stops > - The speed grade used that time as a restart-recovery anchor, also for faults that never restart the primary > - Wave `w-20261110T115048-s302` passed every task but exited 1 on four such samples > - This pull request anchors two-server samples on the primary's own restart > - The benefit is that a two-server wave fails speed only for a real recovery delay ## Linked Issues or Issue Description Refs #20518 (the two-server faults). **What happened?** In wave `w-20261110T115048-s302` (Codex, B5, MS2/MS3, 4 tasks) every task passed, but the speed grade counted 4 single-run breaches on `restart_recovery_control_plane` (p50 10.9 s, p95 12.8 s, budget p95 4.0 s), one per fault. - MS2 never restarts the primary (it freezes and thaws it), yet it produced a restart-recovery sample. - MS3's sample started at the fault's `readyAt`, after the second server's 33–43 s tail. The primary's replacement had been ready since 1794311949160 (f02) and 1794312728179 (f04), 43.4 s and 33.6 s earlier. - So each sample measured the B5 run's next provider event after the second server stopped, while `bench.sh` ran quietly. **Expected behavior** MS1 and MS2 produce no restart-recovery sample. MS3's sample starts when the primary's replacement is ready. ## What Changed - `wave-speed.ts`: - New `primaryRestartReadyAt(fault)`: a non-MS fault keeps its `readyAt`; MS1 gives none; MS2 gives none, unless the primary exited after the thaw and was replaced (then the replacement's `healthyAt`); MS3 gives `multiServer.primaryRestart.readyAt`. - `faultRuns` (recovery samples and the retry "restart" phase) uses that anchor and skips faults without one. - `outageWindows` removes only the time the primary could not serve: none for MS1, the freeze for MS2, and for MS3 from the second server's readiness (when the primary is told to stop) to the replacement's readiness. Before, the second server's whole window was removed from run timings. - `collectWaveSpeed` sets a run's restart phase only from faults that restarted the primary. A freeze still subtracts its downtime, but a run it hit before its first activity stays in its cold or warm budget (review finding). - `wave-speed.test.ts`: six tests built from s302's saved receipts, including two `collectWaveSpeed` cases (a run frozen before its first activity; a retry after an MS2 freeze). - README: one note on how the speed grade treats MS faults. Test-only. No longer freeze and no lease change, as agreed. ## Verification - Red against master `wave-speed.ts`: all 6 new tests fail. Green with the change. - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 38 files, 480 tests pass. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - Offline re-grade of s302 with the fix, from its saved `faults.json` and run timings (`t0*.grade.json`): - Restart-recovery samples go from 4 to 2. The MS2 samples (f01, f03) are gone. - The two MS3 samples are anchored at 1794311949160 (f02) and 1794312728179 (f04), 43.4 s and 33.6 s earlier than before. - Outage removed from run timings: 110.0 s and 113.3 s for the MS2 freezes, 5.6 s and 6.5 s for the MS3 restarts. - No s302 run was hit before its first activity (each freeze came about 68 s into its run), so no run changes phase. - The MS3 latency itself needs the run's event list, which a wave does not save for passing tasks. The observer's saved samples show the run's first recorded activity 0.5 s (f02) and 1.6 s (f04) after the replacement was ready. The next provider event comes at or after that, so this is a lower bound, not the graded value. - `restart_recovery_control_plane` has n = 2 < 5, so its row reports "insufficient sample". The wave's speed verdict now depends only on whether each MS3 sample is under 4.0 s, which this offline data cannot confirm. - No new wave (as agreed). ## Risks - Low. Test-harness code only. - After MS1/MS2 a retry of a failed run is graded as an ordinary warm run, since the primary never restarted. - MS1 overlap and the MS2 thaw tail no longer count as outage, so run timings in those windows are graded in full. The primary served during them. - A primary replaced after an MS2 thaw is graded like any restart. ## Model Used - Anthropic Claude, model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edit, code execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>canary/v2026.1110.0-canary.12 |