mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-13 15:55:46 +01:00
353945c86daedc13de0f23bdf8baf1e35f718e71
356
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandClaude Opus 5.5
|
353945c86d |
fix(tool-gateway): let agents create gateway tokens without receiving the value (#20613)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Tool gateways let external MCP clients use a company's tools with a
gateway client token
> - Since delegated connection management (#17737), an agent can create
and revoke gateway client tokens for its responsible person
> - #20581 stopped agents from getting the one-time token value, but it
did this by making token creation person-only
> - The product decision is that agents keep their delegated authority
to create and revoke tokens
> - This pull request restores agent token creation, but the agent never
receives the token value: the person reveals the value once on the
gateway's Tokens page
> - The benefit is that an agent can do the rotation work for the
person, and the secret still never enters chat, run events, logs or tool
output
## Linked Issues or Issue Description
Refs #17737
Refs #20581
**What happened**
After #20581, `POST /api/tool-gateway/gateways/{gatewayId}/tokens`
returns 403 `gateway_token_requires_person` to every agent caller.
Agents lost the delegated authority to create gateway client tokens that
#17737 gave them. Before #20581, an agent that created a token received
the plaintext value in its tool result and could repeat it in chat.
**Expected behavior**
An agent with delegated connection management can create and revoke
gateway client tokens. The agent never receives a token value. The
responsible person gets the value once, on the gateway's Tokens page.
Person callers keep today's behavior.
**Steps to reproduce**
1. Give an agent delegated connection management for a responsible
person with `tools:admin`.
2. As the agent, call `POST
/api/tool-gateway/gateways/{gatewayId}/tokens` (for example through
`call_api`).
3. See 403 `gateway_token_requires_person`.
## What Changed
- Agent create: `POST /api/tool-gateway/gateways/:gatewayId/tokens` from
an agent with delegated authority returns 201 with token metadata only
(`id`, `tokenPrefix`, `expiresAt`, `pendingReveal: true`), plus
`tokensPath`, a `reveal_credential` `requiredUserAction` for the
responsible person, and a `nextStep` instruction. The saved row has no
usable value: its stored hash is of a value that is generated and then
discarded. Gateway authentication also rejects `pendingReveal` rows.
- Person reveal: new person-only `POST
/api/tool-gateway/gateway-tokens/:tokenId/reveal`. It generates the
value, stores its hash, clears `pendingReveal` and returns the value
once. A conditional update makes a second or concurrent reveal return
409. Only the token's responsible person can reveal it (403
`gateway_token_reveal_requires_responsible_person`). Agent callers get
403 `gateway_token_requires_person`. A token revoked before reveal
cannot be revealed.
- Why a reveal that generates the value: no stored plaintext existed to
reveal, and storing one (even encrypted) for later display adds a secret
at rest. Generating the value at reveal time keeps the existing one-time
display and stores only a hash.
- Schema: `tool_mcp_gateway_tokens.pending_reveal boolean not null
default false` (migration `0460_gateway_token_pending_reveal`). Shared
`ToolMcpGatewayToken.pendingReveal`, and `ConnectionRequiredUserAction`
gets `reveal_credential` / `gateway_token`.
- UI: the Tokens page shows a "Waiting for you to reveal" section for
pending tokens with a "Reveal token" button. The value appears in the
existing "New token — copy now" one-time block. Pending tokens show the
status "Not revealed".
- Agent gateway create: the 201 for agent callers also includes
`gatewayPath`, `tokensPath` and a `nextStep` that tells the agent to
link the saved gateway.
- The agent create-token `nextStep` and the catalog say not to revoke
the old token in the same turn: its client keeps working on it until the
person installs the new value.
- The reveal update and its `tool_gateway.token_revealed` activity entry
commit in one transaction. The Tokens page offers Reveal only for tokens
that act for the signed-in person.
- API reference: new `docs/api/tool-gateway-tokens.md` (create, reveal,
revoke) in the docs nav.
- Revoke stays allowed for agents with delegated authority (unchanged
route).
- API catalog: create-gateway, create-token, reveal and revoke
descriptions now say who acts, that agents never receive values, the
page paths, and that the old token is revoked only after the client
works with the new one.
- Agent access map: create-token policy note and a new reveal entry.
- Evals `gateways-profiles.create-gateway` and
`gateways-profiles.rotate-token`:
- Remove the keyword grader (`human|operator|admin|manage`) and the
handoff text checks. Grade saved state, rendered links and secret
absence. A plain reply or a question card is accepted.
- Remove the setup probes that called the create routes as the agent.
Under delegated authority, those probes created a real gateway
("Forbidden agent gateway"), which failed create-gateway before the
agent ran.
- create-gateway: exactly one gateway for the intended profile and
company, created by the agent (or by the person after a handoff); the
person opens it from a rendered link; attribution matches the creator;
any agent-created token is still pending reveal; no token value in chat,
cards or run events.
- rotate-token: after the agent's turn, both setup tokens still work and
any replacement the agent created is pending the person's reveal; the
person follows the rendered Tokens page link and reveals (or issues) the
value in a private browser; the new value works before the old token is
revoked; the unrelated client keeps access; no token value appears
anywhere, including a regex scan for any full `pcgw_<id>.<secret>`
value.
- The changed scenario context and outcomes are recorded in
`chat-full-catalog-testops-manifest.json` `changedFields` and pinned in
the manifest test. The inventory digest, descriptors (`oracleVersion`
v2, `sourceFiles`) and SDK public input pins are updated. The unused
`external-gateway-handoff.ts` module is removed.
- Fixture realism: the create-gateway profile now allows a real read
tool (an empty profile made the agent ask which apps to add instead of
finishing), and the rotation tokens get a neutral owner note instead of
"Disposable rotation fixture" (the agent read that note as an eval
hint).
- Shared private-browser helpers (`external-private-browser.ts`,
`external-private-browser-worker.ts`) get one optional `pendingTokenId`
input and one `revealedPending` receipt field.
## Verification
- Red: on `master` route code, the new integration tests `lets agents
create and revoke gateway client tokens without ever receiving a token
value` and `reveals an agent-created gateway token value only to its
responsible person` fail (403 `gateway_token_requires_person`).
- Green: `cd server && npx vitest run
src/__tests__/connection-management.integration.test.ts -t gateway` (25
passed). The test checks that the agent `call_api` result has no `token`
field and no token-shaped value, that the agent cannot reveal, that the
person reveals once (second reveal 409) and the value authenticates,
that agent revoke works and stops the value, that a token revoked before
reveal never gets a value, and that person create still returns the
value.
- `npx vitest run src/__tests__/tool-gateway.test.ts
src/__tests__/openapi-routes.test.ts
src/services/native-runtime/runner-api.test.ts` (2247 passed).
- `cd ui && npx vitest run
src/pages/apps/gateways/panels/TokensPanel.test.tsx
src/pages/apps/gateways/gateway-helpers.test.ts` (16 passed); `pnpm
check:token-gates` clean.
- Eval graders and catalog: `pnpm exec vitest run --config
tests/runner-e2e/vitest.config.ts` on the gateway oracle tests,
`external.test.ts`, catalog consistency/contract, TestOps
export/retained, phase/clock callers and SDK input/type tests (all
pass); `node --test` on the TestOps manifest, adapter, contract,
streaming and reconcile tests (all pass); `node
tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`
(unchanged).
- `node scripts/check-maps.mjs` passes. `tsc --noEmit` for `server`,
`ui` and `tests/runner-e2e`: no errors in changed files.
- Paid end-to-end run of both eval cases, run
[42872921890](https://github.com/paperclipai/paperclip/actions/runs/42872921890)
on
|
||
|
Devin FoleyandPaperclip
|
8148fc0b1e |
fix(release): fetch draft-notes PR summaries in batches (#20602)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The release workflow drafts the next stable's release notes when a beta publishes (`draft_stable_notes`) > - The draft nests each referenced PR's summary under its commit subject, and it fetches each summary with its own `gh pr view` call > - A beta range now references more than 1,500 PRs, so these serial calls do not finish inside the job's 10-minute timeout > - This pull request fetches the summaries in batches of 50 per GraphQL query > - The benefit is that the job finishes in under a minute and pushes the notes branch again ## Linked Issues or Issue Description **What happened?** The `draft_stable_notes` job of [run 42763789753](https://github.com/paperclipai/paperclip/actions/runs/42763789753) (beta `2026.1110.0-beta.0`) reached its 10-minute timeout. GitHub cancelled it during "Draft stable notes from the published beta", while a `gh` process was still running. The beta itself published and its smoke tests passed. The `release-notes/v2026.1110.0-beta.0` branch was not pushed. **Expected behavior** The job writes the notes draft and pushes the branch for every beta, whatever the size of the commit range. **Steps to reproduce** 1. Publish a beta whose range from the last stable has many PR-referencing commits. This range has 1,586 commits, and 1,583 different PRs. 2. `draft-stable-notes.sh` calls `gh pr view <n>` once for each commit, one after another. 3. The job reaches `timeout-minutes: 10` before the loop ends. **Paperclip version or commit** `09e24dd12c36251dd12af2d73dbc715ce990f922` (beta `2026.1110.0-beta.0`). This was the first beta to run the job after #12124 added the PR summaries; earlier betas skipped it. ## What Changed - `scripts/draft-stable-notes.sh`: - Collects the unique PR numbers first. Fetches their bodies 50 per `gh api graphql` query (aliased `pullRequest(number:)` fields), into a temporary cache that `enrich_pr` reads. - Parses the raw response with `jq`, not with `gh --jq`. When any number in a batch is not a PR (for example, an issue reference), gh exits non-zero and does not apply its `--jq` filter. The other PRs in the batch still have usable data. - The output format does not change. A missing `gh` or `jq`, a failed batch, a cache directory that `mktemp` cannot create, or `DRAFT_NOTES_SKIP_PR_ENRICHMENT=1` still gives bare subjects. - No code comments are added (per `CONTRIBUTING.md`); this description holds the explanation. - `scripts/draft-stable-notes.test.mjs`: - The enrichment tests now use a fake `gh` that answers GraphQL like GitHub. - New tests: batching (120 PRs make 3 queries, and a repeated PR is fetched once); a batch with a non-PR number; a failed batch; a failed `mktemp`; prose-only and empty bodies; the opt-out. - Range tests now run with enrichment disabled, so an authenticated `gh` on the test machine cannot reach GitHub. - `package.json`: adds the test file to `test:release-registry`. These tests did not run in CI before. ## Verification - `node --test scripts/draft-stable-notes.test.mjs`: 17 pass. They also pass with macOS bash 3.2 first on `PATH`. - `node scripts/ci-profile/node-test.mjs scripts/draft-stable-notes.test.mjs scripts/release-lib.test.mjs`: 31 pass. - Red check: the new tests against the old script fail 5 cases (all batching and enrichment cases). The `mktemp` test fails against an unguarded `mktemp` call. - Live run on the real range: `./scripts/draft-stable-notes.sh 2026.1110.0-beta.0` finished in 47 seconds. It wrote 4,596 excerpt lines for 1,583 PRs. - Parity: for 40 randomly sampled PRs from that range, the new excerpts are byte-identical to the old per-PR `gh pr view` output. - `shellcheck`: no new findings. The two remaining notices were there before this change. ## Risks - Low. The change touches only the post-publish draft job, and every failure still gives the bare-subject draft. - `jq` is now needed for the PR summaries. GitHub's Ubuntu runners include it. Without `jq`, the draft has no summaries, the same as without `gh`. - I did not change `release.yml`. The job already checks out `master`, so after this merges, re-running the failed job of run 42763789753 should push the missing notes branch with the fixed script. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) via Claude Code, with tool use (shell, file editing, test runs, read-only GitHub API calls). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
5f34f777ec |
fix(evals): grade 16 chat cases by saved state instead of reply keywords (#20600)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat-feature eval catalog checks that a chat agent does the work a person asks for > - In the latest full-cohort run on master, 16 cases failed although the saved work was correct > - Their graders matched words, one literal value, or one API route in free-form model output > - A grader that fails correct work hides real failures in the noise, and it can also pass wrong work that uses the expected word > - This pull request grades those cases by saved state, audit records, setup literals and rendered page structure, and keeps only narrow checks for false claims > - The benefit is that these cases fail only when the product or agent does the wrong thing ## Linked Issues or Issue Description Refs #18971 (related: company-secret question handoffs) **What happened** The full-cohort chat-eval run on master failed 16 cases where the retained evidence shows correct behaviour. Examples: `companies.rename-company` accepted only `company.updated`, but the supported branding route logs `company.branding_updated`. `recovery.post-stop-message` required a signup button label that matched a regex, and rejected a labelled email form with the button "Start writing free". `secrets.secret-proposal` flagged "the value is delivered when Morgan next runs" as a false delivery claim. Other graders required words such as "uncommitted", "telemetry", "administrator" or the app name that the question did not ask for. **Expected behavior** A grader accepts every supported route and wording that achieves the outcome. It fails when saved state is wrong, when a requested fact is wrong, or when the reply makes a claim that saved state contradicts. **Steps to reproduce** 1. Load the retained observation for one of the cases below from the full-cohort evidence. 2. Grade it with the old oracle. The keyword or route check fails although the saved state is correct. 3. Grade it with the new oracle. It passes. A broken variant (wrong saved state or a contradicting claim) still fails. ## What Changed - `activity.audit-change`: attribution is graded by structure. The reply names the recorded editor, names no other company agent (new setup literal `otherActors`), and states only real transitions. Sentence grammar is no longer matched. - `app-permissions.ask-before-action`: each continuation reply must exist and must link no app record other than the reviewed connection. It no longer must name the app. - `budgets-costs.spend-breakdown`: the reply must point at the record with unresolved usage (task link, identifier or run id) and must not give it a zero or free amount. The words "subscription" and "incomplete" are no longer required. - `cases.browse-cases`: an excluded case may be mentioned in prose. It must not be linked or listed as a result row. - `cases.handoff-same-task`: handoff instructions carry findings forward when they point at the saved `findings` document, reproduce it, or restate it. The premise (a primary Morgan run performs the handoff) is unchanged. - `cases.update-case-facts`: a saved browser value may add clauses that only rule out other platforms ("desktop Safari unaffected"). - `companies.agent-observability`: the reply no longer must name Telemetry and the run log. It must not claim that they are exported. - `companies.rename-company`: both supported name routes (`company.updated`, `company.branding_updated`) are accepted. - `connection-setup.choose-provider`: the answer must exist and must not claim a connected account. The unavailable method no longer must be named; the setup path stays graded by clicking the card or link. - `goals.align-existing-work`: a question card from the answering run counts as the phase-two response. Refusal words and "unchanged" words are no longer required; false change claims still fail. - `recovery.post-stop-message` (and `steering.interrupt-with-direction`, which shares the helper): a visible form with a labelled field and a submit control is a signup action, whatever the button says. - `routines.pause-owned`: the "paused"/"running" words are no longer required. The saved checkpoint proves both; a cancellation claim still fails. - `secrets.secret-proposal` (and `secrets.configure-company-secret`, which shares the helper): a sentence that says when delivery will happen is not a delivery claim. - `skills.explicit-restriction`: the reply no longer must say "administrator". Success claims and invented tool limitations still fail. - `workspaces.inspect-git`: the reply no longer must say "uncommitted". Unchanged HEAD and status prove it; commit or push claims still fail. - Every affected `oracleVersion` is bumped in code and in `chat-catalog-descriptors.json`. No scenario YAML changed. ## Verification - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts --testTimeout=180000 tests/runner-e2e/chat-catalog-domains tests/runner-e2e/chat-catalog-contract.test.ts tests/runner-e2e/chat-catalog-consistency.test.ts`: all pass. - `PAPERCLIP_CHAT_EVAL_RENDERED_ARTIFACT_CANARY=1` rendered-artifact canary with Chromium: pass, including new form cases. - `node --test tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs`: pass. `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`: unchanged pins. - `npx tsc --noEmit -p tests/runner-e2e`: no errors in changed files. - Offline calibration on retained evidence, for each case: the retained correct evidence passes, earlier passing evidence passes where it exists, and a broken variant fails. `spend-breakdown` retained evidence still fails, because its reply shows unresolved usage as `$0.00`. `handoff-same-task` retained evidence still fails on the kept primary-run premise. - Live paid runs: see the comment below. ## Risks - Low risk. The change is limited to eval graders and their unit tests. Product code does not change. - Each grader is more accepting of wording. Saved-state checks and the narrow false-claim checks remain, so a wrong outcome still fails. - `secrets.configure-company-secret` still fails its retained evidence. Its agent asked a two-question card, and the act supports only one-question cards. This pull request changes it only by the shared delivery-claim helper. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
89ec4e7614 |
feat(links): add cached GitHub pull request previews (#20593)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Task conversations contain links to pull requests and other work. > - A URL alone cannot tell the operator whether a PR is open, merged, or blocked by conflicts. > - Fetching every link at render time wastes API calls and can expose private data through a shared cache. > - This pull request adds a demand-driven links service and a GitHub PR adapter. > - Each viewer and company has a separate cache. Repeated links share one request and result. > - Operators get context when they hover or focus a link. Normal links stay useful when a provider is unavailable. ## Linked Issues or Issue Description **Subsystem affected** Task Markdown links, shared link metadata, and the server GitHub API boundary. **Problem or motivation** PR links have compact labels and neutral icons. They do not provide a rich hover card or a common status source across repeated mentions. A background resolver cannot borrow a person's GitHub connection because it has no viewer identity. **Proposed solution** Recognize supported links locally. Request metadata after a delayed hover or keyboard focus. Show a card at the first line of the link, aligned with its icon. Share the result across all mentions in the current viewer and company. Keep the exact original navigation URL. The first adapter supports GitHub pull requests. Open is green, merged is purple, and merge conflicts are orange. Draft, closed, unknown, loading and unavailable states remain quiet. **Alternatives considered** Eager page scans would spend API quota on links the operator does not inspect. Company-wide caching of personal GitHub results would cross the credential boundary. This change uses neither approach. **Roadmap alignment** ROADMAP.md was checked. This extends existing task links and the GitHub resolver. It adds no deployment adapter or new background worker. Searches of open and closed PRs for link previews and PR hover cards found no matching implementation. ## What Changed - Add host-owned link recognition and a normalized preview contract. The service selects an adapter by provider. - Add a board-only company read endpoint. It checks company access before cache reads and sends a private, no-store response. - Use the current viewer's eligible GitHub connection. A configured connection that cannot supply credentials fails closed. With no managed connection, reuse existing company credentials or public access. Instance credentials retain the public-repository check. - Reuse the GitHub external-object resolver for API transport and response parsing. Add title and mergeability to its normalized snapshot without changing its existing state contract. - Coalesce requests by company, viewer and canonical PR URL. Bound cached entries and concurrent work. Cache success for one minute and failures for a short period or the provider retry window. - Add an abortable request deadline. Timed-out work stays in the concurrency budget until it settles. No periodic worker or idle polling is added. - Add a shared browser query and anchored preview card. Repeated Markdown links observe the same result. There are no automatic retries, window-focus refreshes, toasts or connection prompts. - Add ten production-component Storybook examples and real pointer/keyboard browser checks. Document the API and verification route. ## Verification - Linux source checkout: 233 focused tests passed in nine suites. They cover recognition, fixed API destinations, provider states, access failures, credential selection, viewer/company isolation, request coalescing, expiry, cache bounds, concurrency bounds and mounted OpenAPI coverage. - A live public GitHub smoke returned this PR's correct title and open state. Two equivalent URLs shared one real network request. Personal connection access was not part of that smoke. - A controlled HTTP server stalls a real response body. The test proves cancellation while another lookup completes. - UI typecheck and the server layered typecheck passed. The UI production build passed. - Token gates and query policy gates passed. All 30 map/catalog support checks passed. The three new server test files also passed a focused TypeScript check. - All 27 focused browser checks passed on the reviewed source tree. They cover light/dark themes, narrow layouts, open/merged/conflict/draft/closed/loading/unavailable/failed states, fixed positioning during pointer movement, exact navigation and keyboard access. A real browser HTTP interception counts one deferred lookup for two equivalent PR links. - The first browser attempt lacked Chromium. A later combined build/test attempt exhausted the sandbox's 8 GiB memory limit and killed Storybook. Those attempts supply no pass evidence. Chromium is installed, and the final browser checks run with the build stopped. One initial Storybook dependency reload interrupted a streaming check. All eight existing inline-link cases passed on the warmed server, followed by a complete 27-case pass. - Synthetic Storybook pointer events did not exercise the native hover boundary. The assertions now run through real Playwright pointer and keyboard actions. The fixture stories retain the production components. - Review fixes preserve the Draft label when GitHub reports both draft and conflicts. They also preserve neutral link colors in dark mode. The 25 affected service checks pass, including the added combined-state regression. Browser checks compare unknown, draft and closed links with the muted theme color. - The first CI policy run rejected a literal retry opt-out. The code now uses the existing explicit zero-retry helper. A new test-spy type error was also corrected. These failures are fixed; their earlier runs remain failed. - `pnpm verify --explain` returned exit 2 because the browser test path selects broad CI. No full local suite is claimed. The final CI run supplies that coverage. - The previous head `437913c797963aa113bb82e2ae8b836969317ef9` passed CI and received a 5/5 Greptile score. All four review threads are resolved. Master then advanced. The current head `ae2a1c2adbd61a621cca9bdd160561952613df6e` resolves the source-manifest conflict against master `9b21dcbdda01296349f3cfe0a3a6891e79df6130`. Fresh CI and review are running. - The previous CI run found a lost class on decorated PR links and a stale SDK source manifest. Both are fixed. The existing CommentThread integration test and 26 focused link service tests pass. All 45 SDK source/privacy tests pass. All eight source-manifest helper tests pass. The manifest helper changed only source identities and added the two new shared files. Runtime, compiler, parser and library pins stay unchanged. - A later review found that expired timeouts could reuse a settled promise. The service now removes settled requests from its shared-request map and counts unfinished provider work separately. The new regression failed before the fix and passes after it. The concurrency-bound and real stalled-HTTP tests also pass. - UI typecheck and token gates pass after these fixes. The browser rerun needed Chromium in the new session. That first attempt did not exercise the UI. The complete 27-case browser rerun passed before the master merge with retries disabled. All 27 browser cases also passed on the merge revision with retries disabled. - After the master merge, all 84 focused link, OpenAPI and integration tests pass. All 55 SDK source/privacy/manifest tests pass. UI typecheck and token gates pass again. The source manifest retains master's reviewed runtime pins and includes both sets of source changes. GitHub confirms the PR can merge cleanly. The preflight helper rebuilt the shared package and plugin SDK; Cargo is unavailable locally. Final CI supplies Rust coverage. - The latest master update changed the reviewed report-lock policy but left the SDK lock hash at its previous value. The SDK source/evidence checks reproduced the mismatch. This change sets only the SDK lock hash to the authenticated output of the existing master policy and patch. Compiler, parser and Zod pins stay unchanged. All 56 SDK source/evidence/manifest tests now pass. All 84 focused link/API/integration tests, UI typecheck and token gates also pass on the updated source. ## Risks - This cache is process-local. Separate server replicas can make separate GitHub requests. - A cached result can remain visible for its short lifetime after GitHub changes. A later hover refreshes expired data. - GitHub can report unknown mergeability. The UI shows conflicts only when GitHub explicitly reports them. - Personal preview data stays in viewer/company caches. It is not written to company external-object records or persisted browser query storage. - Stories simulate the API. They do not prove a live GitHub login or private-repository access. - No schema migration is required. ## Model Used OpenAI Codex, based on GPT-6. The runtime does not expose the exact serving model ID or context-window limit. Used reasoning, repository inspection, source editing, shell execution and browser tools. No subagents were used. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
42d4499489 |
fix(daytona): force a stuck stop before starting a retained sandbox (#20609)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - On Daytona, a follow-up task resumes the agent's retained sandbox before the agent starts > - The host stops an idle sandbox gracefully. A process that ignores SIGTERM can leave Daytona in `stopping` for many minutes > - Resume waited for that state change until its 300 s start deadline. It never escalated, so each attempt failed after five minutes, and the bounded retries did the same > - On staging, a user comment waited about 21 minutes before the agent started. Two resume attempts each used 299.7 s in the plugin dispatch > - This pull request forces a stuck stop once, then starts the sandbox within the same deadline. It also traces the wait with the observed state > - The benefit is that a stuck stop costs about 30 s instead of failing the run, and the next case is diagnosable from the trace ## Linked Issues or Issue Description **What happened?** A user commented on a task that used a retained Daytona sandbox. The first run and its first retry each failed after about 5 min 20 s with `reusable_sandbox_resume_failed (cause=provider_error)`. The Honeycomb trace shows `execution.plugin_dispatch` at 299.7 s in both. Inside it there is only `lease.lookup` (44 ms) and `lease.channel_cleanup`, and no `lease.start`. So the whole budget went to `waitOutSandboxTransition`, which polled a sandbox that stayed in a transition state. The later attempt, after a control-plane restart, found the sandbox `stopped`. Its `lease.start` took 1.3 s. A transition that ends in `stopped` is a stop. **Expected behavior** A sandbox that stays `stopping` for longer than the host's own graceful-stop budget is forced to stop, then starts. The run does not fail on the start deadline. **Steps to reproduce** 1. Retain a Daytona sandbox whose graceful stop does not finish (Daytona reports `stopping`). 2. Resume or realize the lease. 3. Before this change, the call polls until the 300 s deadline and fails with "is still stopping at its start deadline". ## What Changed - `packages/plugins/sandbox-providers/daytona/src/plugin.ts`: `waitOutSandboxTransition` tracks how long the sandbox stays `stopping`. After `STUCK_STOPPING_FORCE_AFTER_MS` (30 s) it calls `sandbox.stop(seconds, true)` once, then keeps polling. A refused force stop keeps the old wait within the same deadline. - The wait is a `lease.wait_transition` span. The force stop is a `lease.force_stop` child span. - `lease-tracing.ts`: the closed `paperclip.sandbox.startup.state` label also records `starting`, `stopping`, `restoring`, `archiving` and `resizing`. Before, these were all `other`. - `server/src/services/plugin-host-services.ts`: the host admits the two new span names and the new closed state values. - `doc/observability.md`: documents the new spans and states. ## Verification - Red/green: with the plugin change stashed, the three new stuck-`stopping` tests (realize, refused force stop, resume) fail by hanging to the deadline. With the change they pass. The "settles on its own" test confirms that no force stop happens for a normal stop. - `vitest run` in `packages/plugins/sandbox-providers/daytona`: 14 files, 618 tests passed (8 skipped). `tsc --noEmit` passed. - `server/src/__tests__/plugin-host-services-span.test.ts`: red before the allowlist change, 25/25 after. - `pnpm verify` exited 2 (the doc is a global CI input), so CI supplies the broad coverage. - Not verified on a live Daytona sandbox. Hosted acceptance needs the change deployed to staging. ## Risks - The force stop sends SIGKILL to processes in a sandbox that is already stopping and is about to start. The disk persists across a stop, so the retained workspace is kept. A graceful stop that is still running past 30 s is killed. The host's own graceful stop budget is 30 s, so the host had already given up on it. - Daytona can refuse a stop while a state change is in progress. Then the code waits as before, so the result is no worse than today. - The root cause is inferred from the trace (a transition that ends in `stopped`). The new state label confirms or refutes it the next time it happens. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, browser, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
9b21dcbdda |
refactor: remove SPEC docs and repoint spec citations (#20565)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Contributors and agents read the docs under `doc/` before they change code > - `doc/SPEC.md` and `doc/SPEC-implementation.md` were out of date. `doc/SPEC-implementation.md` became an append-only log: in 60 days it got about 1,645 new lines and lost 181, across 140 commits > - #20557 changed `AGENTS.md` so the code and its tests are the source of truth, and the specs are only background > - This pull request removes the two spec docs. It moves only the sections that live code comments cite into focused docs, and it repoints all other references > - The benefit is one less stale document that agents read and append to ## Linked Issues or Issue Description Refs #20557 (merged). This PR is the follow-up that removes the docs. It is based on `master`, which already contains #20557. **Issue type** Outdated content. The docs are out of date and conflict with the code. **Where is the issue?** `doc/SPEC.md`, `doc/SPEC-implementation.md`, and the code comments and docs that cite them. **What's wrong?** `doc/SPEC.md` and `doc/SPEC-implementation.md` are out of date, but agents still read them and append a paragraph with each fix. **Suggested fix** Delete both files. Move the sections that code cites into focused docs. Repoint the references. ## What Changed - Deleted `doc/SPEC.md` and `doc/SPEC-implementation.md`. - Moved these spec sections, because live code comments cite them: - §9.3.1 "Shared default-open issue writes" → new section in `doc/agent-permission-defaults.md`. - §9.8.1 "Issue-thread interaction resolver contract" → `doc/execution-semantics.md`, "Issue-thread interaction resolution". That section already had most of the contract. I added only the missing parts: the `supersedeOnUserComment` default, the canonical policy list, the addressee and cap rules, the rule that surfaces show the server's effective audience, and the cross-issue budget charge. The short policy summary at the top of the section now points to the list, so the definitions occur one time only. - §9.9 "Task Watchdog Authority Contract" → new "Authority contract" section in `doc/TASK-WATCHDOG.md`. It holds the allowed and disallowed mutation lists and the deferred capabilities. The atomic recovery batch, restoration verification, and interaction rules were already in `doc/execution-semantics.md` §11, so the new section links there and does not repeat them. I did not move the "Downstream acceptance criteria" checklist or the sentence about the CTO and Security sign-off. These were implementation-time notes; the tests now hold that coverage. - §9.10 "Company Skill Policy Contract" → new `doc/company-skill-policy.md`. No focused doc had this contract. I did not move the "Required regression tests" phase checklist. - Did not move any other part of the specs. - Repointed code comments that cited spec sections: - `packages/shared/src/issue-write-denial.ts` (§9.3) → `doc/agent-permission-defaults.md` - `server/src/routes/issues.ts` (§9.9) → `doc/TASK-WATCHDOG.md`, "Authority contract" - `server/src/routes/issues.ts` (§9.3, §9.8.1 cross-issue budget) → `doc/agent-permission-defaults.md` - `ui/src/lib/interaction-audience.ts` and `ui/src/components/InteractionGovernancePanel.tsx` (§9.8.1) → `doc/execution-semantics.md` - `ui/src/lib/skill-policy-denial.ts`, `ui/src/pages/CompanySkills.tsx`, `ui/src/pages/CompanySkills.production.tsx`, `ui/src/pages/SkillStudio.tsx` (§9.10) → `doc/company-skill-policy.md` - `server/src/services/recovery/ownership-invariant-report.ts` (monitor deadline wait path) → `doc/execution-semantics.md` §8. That section already lists the one-shot monitor as a valid wait path. - Resynced the git blob pin for `packages/shared/src/issue-write-denial.ts` in `tests/runner-e2e/source-sdk-public-inputs.json`. The comment edit changed the file's hash, and the Runner support suite checks that pin. I used `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --write`; only that one pin changed. - Updated `AGENTS.md` §1 so it does not name the deleted files. - Removed the links to the deleted files from `doc/plugins/PLUGIN_SPEC.md`. - Updated the `doc-maintenance` agent skill (`.agents/skills/doc-maintenance/`) so it no longer audits or stages `doc/SPEC.md`. - Removed or replaced references in `maps/README.md`, `maps/feature-map/README.md`, `maps/storybook-map/README.md`, `doc/PRODUCT.md`, `doc/CLIPHUB.md`, `doc/DEPLOYMENT-MODES.md`, `doc/DEVELOPING.md`, `doc/evals.md`, `doc/spec/agent-runs.md`, `doc/AGENTCOMPANIES_SPEC_INVENTORY.md`, `doc/connections/CONNECTOR-PLAYBOOK.md`, `doc/plugins/ideas-from-opencode.md`, `doc/execution-semantics.md`, `doc/TASK-WATCHDOG.md`, and `.devin/wiki.json`. - In `doc/execution-semantics.md`, the "admission is independent of cleanup and accounting" reference now states the rule inline: only a hard stop, an explicit pause, or an inactive company refuses admission. - Kept these references as they are: dated plans in `doc/plans/` and `docs/plans/`, the dated log `doc/logs/2026-05-24-cli-api-parity-e2e-log.md`, `releases/`, the transcript fixture `ui/src/fixtures/runTranscriptFixtures.ts` (it uses the filename only as sample text), and `docs/specs/external-task-protocol.md` (it refers to the Symphony project's own `SPEC.md`). ## Verification - `git grep -n "SPEC-implementation\|SPEC\.md"` (without `releases/`, and ignoring `PLUGIN_SPEC.md` citations) finds only the kept references listed above. - `grep -rn "§9\.\(3\|8\|9\|10\)" ui/src server/src packages/shared/src` finds no results. - `node scripts/check-maps.mjs`: "Maps aligned: 35 areas, 192 capability IDs". - `pnpm check:token-gates`: "All gates clean". - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/source-sdk-public-type.test.ts tests/runner-e2e/source-sdk-evidence.test.ts`: 45 passed. These failed in CI before the pin resync. - `pnpm verify`: exit 2 ("Broader verification required") because `AGENTS.md` is a global CI input. Required CI supplies the broad coverage for this revision. - All code changes are comments only. No runtime behavior changes. The title uses `refactor:` because comments in source files changed. ## Risks - Low risk. The change removes docs and edits comments only. - A reader can lose context from spec sections that I did not move. The text stays in git history. The code and its tests are the source of truth. - External links to the two deleted files will return 404. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), in Claude Code with tool use (shell, file edits). Standard context window. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (focused checks above; CI supplies broad coverage) - [x] I have added or updated tests where applicable (not applicable: docs and comments only) - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
bce79f186f |
fix(evals): count a pending question card as the turn's response where the case allows it (#20597)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat catalog evals check that a chat agent answers the person and saves the right records > - An agent can end its turn on a question card; the product then records `interaction_owns_conversation_response` and posts no reply comment > - The privileged settle step and some graders still waited for a reply comment, so valid card-ended turns timed out or failed > - This pull request adds one shared rule for when a pending card counts as the turn's response, and uses it where the case allows a card > - The benefit is that card-ended turns are graded on what the person saw, while cases that need a written reply stay strict ## Linked Issues or Issue Description Refs #20588 (request_human_input version fix found by the same runs). Refs #20590 (chat review cards link their document). **What happened?** In the full chat eval run on master `f4337bb5fb`, agents ended turns on pending question or review cards. `developer-labs.component-reference` timed out in the privileged settle step, which required a reply comment. `goals.align-existing-work` refused the foreign goal in a question card, but the grader read only reply comments. `questions-and-approvals.reject-and-revise` returned its draft in a review card bound to the draft revision, but the grader and the result check required a reply link. **Expected behavior** A pending card counts as the turn's response only when the product decided that the card owns the run's response, the lead's own run created it, a person can answer it, and it is not a governed approval. Each case still grades the content it requires. **Steps to reproduce** Regrade the retained evidence of the cases above with the old graders: they fail on the missing reply comment although the card is on the chat. ## What Changed - New `chat-catalog-domains/turn-response-card.ts`: `turnResponseCards()` returns pending cards from runs whose `presentationDecision.reasonCodes` contains `interaction_owns_conversation_response`. A card must be created by the lead, be human-answerable (a text question, two or more choices, or an ordinary confirmation) and not carry `toolAction`, `secretProposal` or `connectionAuthorization`. - `settlePrivilegedChat` treats such a card as a settled turn. The privileged graders still require their written guidance, so `developer-labs.component-reference` now fails at grading instead of timing out. - `goals.align-existing-work` reads the card from the run woken by the foreign-goal request as that phase's answer. A leaked foreign text, or a card that does not refuse, still fails. Oracle `paperclip.chat-catalog.internal.goals-align.v2`. - `questions-and-approvals.reject-and-revise` accepts a review card bound to the draft's exact saved revision. The result check opens the draft from the card's document link (needs #20590). Oracle `paperclip.chat-catalog.plan-review.v6` for both plan-review cases. - Descriptors updated for oracle versions and `sourceFiles`. ## Verification - `turn-response-card.test.ts`: the retained cards pass; other run, other agent, answered, other company, user-created, non-human resolver, single non-text choice, governed confirmation, and a run that replied all fail. - Offline regrade of retained evidence (master `f4337bb5fb` full-cohort run): `goals.align-existing-work` cross-company check passes with the card and fails without it; `reject-and-revise` `real-local-review-draft` passes with the bound card and fails with no card or a card bound to another revision. - `npx vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/chat-catalog-domains ...clock tests --testTimeout=180000`: 269 files passed (one file needed the plugin SDK build locally, then passed). - Consistency and contract tests pass. `npx tsc --noEmit -p tests/runner-e2e` is clean. SDK public input pins are unchanged. - Paid run [42855262851](https://github.com/paperclipai/paperclip/actions/runs/42855262851) on a temporary branch with this PR, #20588 and #20590 (OpenCode, local): `reject-and-revise` and `revoke-personal-grant` passed; `goals.align-existing-work` failed only `phase-bound-strategic-context-explanation` (keyword check, unchanged here); `developer-labs.component-reference` ended on a question card again and now failed at grading (`existing-question-reference`, `reference-only`) instead of timing out in settle. This run's agents replied in text in the two passing cases, so the card branch of `reject-and-revise` is covered by the offline regrade only. ## Risks - `settlePrivilegedChat` now stops waiting on a card-ended turn. Privileged cases whose graders need a reply fail sooner and with the real failed check; none pass more easily. - Changed oracle versions start new history for `goals.align-existing-work` and both plan-review cases. - The remaining keyword check `phase-bound-strategic-context-explanation` in `goals.align-existing-work` is unchanged and still fails the retained reply; it needs its own fix. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), agentic coding with tool use in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
github-actions[bot]andlockfile-bot
|
9816d2d25c |
chore(lockfile): refresh pnpm lockfiles (#20615)
## Thinking Path > - Paperclip uses a dependency lock for reproducible builds. > - Dependency changes on master need matching lock entries. > - Paid eval reports also pin a reviewed offline correction. > - Reports must install before paid model work starts. > - This automated PR keeps the lock and report proof together. ## Linked Issues or Issue Description **What happened?** Dependency manifests on master changed after the last committed lock refresh. **Expected behavior** Frozen installs must match those manifests. Eval reports must retain matching offline lock proof. **Steps to reproduce** Run a frozen install before and after this lock refresh. ## What Changed - Refresh the dependency lock from the manifests on master. - Refresh the sandbox provider lockfiles from their manifests on master. - Include the matching offline report correction and hashes. ## Verification - Pinned pnpm resolution completed with lifecycle scripts disabled. - The report proof refresh accepted the reviewed committed baseline. - The workflow rejected changes outside the lock and report proof files. ## Risks The lockfile bot merges this PR without CI or review when every added package version is at least 3 days old on npm. Otherwise the PR waits for a maintainer, and the daily refresh tries again. The bot merges only the exact commit this workflow generated from manifests already reviewed on master. Unknown report lock baselines remain blocked. This update changes no permissions or paid enable gates. ## Model Used None — generated by pnpm and the deterministic report-lock updater. ## Checklist - [x] Thinking path and problem description included - [x] Generated lock and matching report proof included - [x] Release-age gate decides between bot merge and maintainer review Co-authored-by: lockfile-bot <lockfile-bot@users.noreply.github.com> |
||
|
DottaandPaperclip
|
12f70d0d4c |
Keep Slack form custody and receipt transfer durable (#19825)
## Thinking Path > - Paperclip manages work for AI-agent companies. > - Chat connections deliver provider messages and user answers. > - Slack requires a response within its original three-second deadline. > - Forms and their delivery receipts must survive retries and tenant restarts. > - This change uses the existing connector queue and durable form owners. > - Current authorization, exact receipt checks and normal ACK rules stay in force. > - The tenant can settle useful work while keeping the connection enabled. ## Linked Issues or Issue Description Refs #19019 and #19348. Companion Cloud PR: paperclipai/paperclip-cloud#1250. ## What Changed - Preserve form grants, sessions, encrypted evidence, terminal receipts and company-authorized Retry. - Check initial form enrollment before the provider send. Keep a typed preparation wait only while an exact current enrollment obligation exists. - Refund the attempt only when the claimed attempt and credential lease still match and provider invocation has not started. Possible or accepted sends retain uncertain-delivery custody. An ordinary error with the same text still uses the normal five-attempt limit. - Notify the existing publication worker when normal enrollment commits. Reconstruct pending work from durable state after a lost reply or owner restart. - Page settled history before proving idle safety. Keep unfinished, malformed and foreign receipts visible beyond the first 1,000 rows. - Keep Slack callback, answer, receipt and ACK rules in their provider owners. Support body-bound Cloud idle capabilities when a complete catalog exceeds the header budget. - Add focused regression coverage, an enabled quiet/reactivation recipe and explicit fixture port checks. - Keep SDK evidence tests tied to a normally built reviewed reference, with a separate strict refusal for actual unreviewed CI outputs. ## Verification App head: `6284d6881b5dc38a9ee149bcbde35e69a67dd8ca`, including actual master `dff8bed007e01c32f3826144b893ec80603d1335`. Companion Cloud head: `f83b38187dc7e6b2e219fe865dc16d268cc66fdb`, including actual Cloud master `b1dcc3421a91efffee631a61536cf48f357c22fc`. Cloud #1250 is merged at `1d07d81f23d4627ab7b6dcf49176b4128a2f8f9b` after exact-head passing checks and complete Greptile 5/5. Current exact-head CI typecheck, build, tests and aggregate verification are green; the complete edited Greptile review is 5/5 and all 13 review threads and nested comment pages are resolved. - Both latest enrollment findings were reproduced against the preceding product. The pure readiness control returned no wait for the first enrollment. The real-PostgreSQL control observed a provider call while that enrollment was still staged. - The repaired source passes three whole files: 21 readiness controls, 80 real-PostgreSQL milestone controls and 37 tenant-actor middleware controls. All 138 pass with zero excluded or TODO cases. The SQL case reconstructs the worker through seven waits, preserves the attempt budget, recovers a controlled lost enrollment reply, and commits one provider-port call and one exact grant. Replaying the selected attempt does not duplicate them. Authorization, SDK and enrollment-response ports are controlled in that SQL case. - The same files cover ordinary five-attempt exhaustion, accepted and possible sends, stale ownership, malformed bindings, attention states and current actor restrictions. One synthetic mismatched-binding parameter uses generation 2 against current generation 1; it is not evidence for an older-generation control. - The normal server-package typecheck passes at the measured source. It runs the normal Runner TypeScript, protocol, SDK and declaration prerequisites and all seven main App programs. The separate declaration configuration is also recorded. A worker union-narrowing error was found, fixed with an explicit callable/null branch, and verified by that normal compiler. No cast or test relaxation was used. - The normal merge of current master preserves all 78 owned paths and 15,783 unrelated master entries and modes. The three measured test files and their changed feature paths are unchanged. New upstream DB code adds only the normal nonunique concurrent native-session index and its migration classification. Shared and adapter-utils changes alter unrelated prompt or denial wording. The 138-case proof carries for those unchanged behaviors; it is not reported as a rerun after the merge. The normal Shared/SDK ensure, DB build, adapter-utils build and complete server-package check also pass at the current merged head. Their fresh inventory contains 8,143 outputs; canonical SQL and journal copies match. This build did not apply the new index migration. - Earlier real-PostgreSQL retained-history coverage reproduced the missing proof with 1,005 settled rows. Its repair retains the existing two-second/64-query inspection budget, current endpoint bound and exact owner snapshot. The 46 source-bound controls cover unfinished and invalid tails, exact cross-page action references, Retry, rollback and lost receipt responses. - Earlier enabled quiet/reactivation coverage produced ordinary useful work, kept configuration enabled, accepted fresh signed work plus a duplicate, and produced exactly one second useful reply. Its two App-worker windows lasted at least 30 seconds with zero measured PG writes, connection or outbound activity after settlement. That evidence stays tied to its measured revision. A missing worker EXIT marker limits that earlier proof; parent evidence does not replace it. - Earlier three-second callback, terminal ACK, current-question race and complete body-size controls remain source-bound. The 500-owner/1,000-binding body measured 313,875 bytes with a 618-byte JWT; signature tampering failed closed. - Final managed-provider acceptance now passes all five fresh Node boots: A/A, A/AB, B/A rejection, B/AB recovery, B/B final. Two independently configured, challenged, verified, linked and enrolled Slack identities exercise ACK and retirement debt separately. Normal provider factory/materialization delivers the configuration through a controlled external stack-provider boundary. The registered opening returns HTTP 200 in 21 ms within the original 3-second deadline. Two answers persist, normal ACK is accepted, exact retirement and lost acceptance/GC replies recover, and the original canonical snapshot remains unchanged without duplicate provider sends or submissions. - The separate complete committed-preflight case passes. A committed `update_attempted` response is lost before SDK entry; normal service-owner reconstruction after the original lease/deadline recovers the same useful update and one answer without a second trigger. This reconstructs service owners within one App process; the separate five fresh Node boots cover process restart and managed signing-key rotation. - Assertion-time PG socket reports balance 15 opens/15 closes across the five boots and 5/5 in preflight, peak 2 and zero live sockets or unexpected attempts. Worker terminal EXIT markers were missing and are not replaced by parent EXIT markers. These are domain/recovery and assertion-time connection proofs, not a new whole-worker terminal quiet measurement. All known owners/groups closed; independent SQL audits were empty; only successful fixture databases were removed. Earlier failed fixture databases/logs remain retained. No product guard, deadline or assertion was weakened. Exact runtime source and normal outputs are unchanged by the three test files in the current follow-up, so these source-bound proofs carry without repeating those journeys. - The preceding App CI ran 11,312 support cases: 11,289 passed, three SDK-evidence cases failed, and 20 external cases skipped. Normal CI dependency setup resolved six stale Slack patch-hash references and produced an unreviewed lock. The public SDK proof correctly refused that lock; the tests incorrectly assumed the CI output was a reviewed reference. The three-file test repair keeps the protected lock policy, source/compiler/parser/Zod pins, scanner and redaction rules unchanged. It builds an owned reviewed reference using the existing normal Shared/SDK producers for the positive seal test, while a separate negative requires actual unreviewed-build privacy refusal and no retained SDK bytes. It does not relabel the CI build as certified or admit a new lock hash. - All four whole SDK support files pass locally: public type 30, evidence 16, public-input synchronizer 8 and lock preparation 10, totaling 64 with zero excluded/TODO cases in 28.73 seconds. The genuine fresh reference build/positive seal and separate invalidated-reference privacy refusal passed within their original 120-second bounds. Original installed dependencies, SDK outputs/cache and runtime source stayed unchanged; normal global setup produced only its own Runner JavaScript outputs. PostgreSQL was denied throughout, all four worker markers were observed, and the owned process group/private test root closed. The local canonical-lock negative mutates only its own reference; fresh remote CI also passes the actual resolved-CI-lock branch: all 16 evidence and 30 public-type tests pass, including the genuine reviewed-reference positive and the strict actual-unreviewed-build refusal. The upstream synchronizer file runs its current 10 cases in CI. - The first complete CI attempt on this head also exposed two separate 30-second timeouts: one real reporter-artifact test and the `agent-test-environment` module-loading hook. Full original logs remain preserved. Their exact test, sanitizer and evidence source blobs are unchanged from the preceding head; inspection found no deterministic setup cause. After the original run finished, one retry of only the failed jobs passed on the same synthetic merge `168851e210671ebe97e7a778e8cf891807e80502`: Runner support reports 11,304 passes and 20 external skips; serialized shard 6 reports all 395 passes, including all 67 route bodies. No source, timeout or assertion changed. This proves recovery; the original causes remain unexplained. - The final pre-merge tree audit uses actual master `17136a6c25ef6cb7c816a29140a4c48c2c75fb31` (refreshed after the earlier `85dec046` audit). The expected normal squash preserves all 81 owned paths and 15,816 unrelated master entries and modes. Only OpenAPI has edits from both sides, and its complete content composition preserves the new project route and token help alongside this PR's API entries. The shared inbound selection changes preserve the same candidate set; new concurrent indexes, Native/workspace behavior and shutdown changes remain upstream behavior. The scoped form proof uses process agents and does not qualify the new Native/sandbox branches. The latest question-schema change still accepts the explicit version-1 payload exercised here. The last master delta also changes managed AI/Native credential refresh and writeback; those branches are outside the measured process-agent form paths. Existing 8,143 local outputs remain tied to their measured revision; no new whole-tree output equality or deployed suspension is claimed. - Reproduce the public path with `doc/testing/slack-cloud-qualification.md`, the companion Cloud checkout, Node 24/22 and fresh owned fixture databases. Aggregate local `pnpm -r typecheck`, `pnpm test:run` and `pnpm build` were not rerun as one command set. Current exact-head required CI and the complete Greptile 5/5 review now pass; the full edited summaries and all paginated threads were read. Local worker quiet does not prove deployed host or Neon suspension. Photon and Discord remain separate work; this PR completes the described Slack slice. ## Risks Use compatible App and Cloud readers before transferring custody. Signatures prove authenticity; current authorization and exact ownership still control execution. Preserve old targets, secrets and terminal facts until their normal owners release them. Keep the managed seal identity stable while custody remains. The required Slack adapter patch remains pinned. Upstream's concurrent index uses the normal top-level migration path. ## Model Used OpenAI `gpt-6.1-sol` implemented and tested the changes. A GPT-6-based Codex managing agent reviewed and integrated them. The host does not expose the managing agent's exact model ID or context-window sizes. The agents used code execution and test tools. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used, its version and tool capabilities - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have linked public related issues or PRs - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal Paperclip ticket id - [x] I have run the local tests listed above and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation - [x] I have considered and documented risks - [x] All Paperclip CI gates are green for the current head - [x] Greptile is 5/5 with no open findings for the current head - [x] I will address all reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandPaperclip
|
98ce93dd4e |
Choose project storage from compatible environments (#20587)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Projects supply repository files to agent runs through core staging or a storage plugin. > - A storage plugin's default claim silently attached new projects even when their agents used a different environment. > - An environment name or driver type does not establish shared storage access. The existing binding and task-driver contracts do. > - This pull request groups creation choices by that contract and selects storage from the creator's effective environment. > - The benefit is a visible, validated choice and a clear error before an incompatible managed workspace operation. ## Linked Issues or Issue Description Fixes #20459. Related: #20156 covers a separate provider-failure diagnostic. The PR search found no other fix for #20459. ROADMAP.md has no duplicate project-creation change. ## What Changed - Show **Available in** when configured environments require different project storage. Each option names the environments that can use the same storage owner. Hide the selector when all choices are compatible. - Derive compatibility from repository staging and managed binding ownership. Local, SSH, and sandbox environments use core staging. Managed storage requires a registered task driver from the same installed storage plugin. Separate storage plugins remain incompatible. Unknown plugin contracts and the fake sandbox are excluded. - Follow the creating agent's environment, or the person's primary agent, then the instance default and local fallback. Apply managed-only and forced Kubernetes policy. The product has no separate user or company environment-default field today. - Add a company-authorized discovery endpoint and an optional creation `environmentId`. Validate the choice again on save. Store the storage owner without overriding agent run environments. Preserve explicit API storage choices and existing project ownership. - Keep the hidden repository-free private-task project independent of environment availability. Submit the observed storage owner as `expectedStorageProviderPluginId` so a changed contract fails at save without setting a workspace policy. Instance task-isolation defaults stay inherited. - Reject an incompatible managed workspace before a provider operation. Name the storage and selected environment in the error. Add regression tests, API documentation, creation-story fixtures, and minimal corrections to obsolete map text. ## Verification - **102 tests passed across 7 files:** project environment integration, installed storage policy, repository persistence, project environment routes, project creation telemetry routes, OpenAPI, and NewProjectDialog. Run with `node node_modules/vitest/vitest.mjs run server/src/__tests__/project-environments.test.ts server/src/__tests__/project-workspace-provider.integration.test.ts server/src/__tests__/project-repositories-persistence.test.ts server/src/__tests__/project-routes-env.test.ts server/src/__tests__/project-goal-telemetry-routes.test.ts server/src/__tests__/openapi-routes.test.ts ui/src/components/NewProjectDialog.test.tsx --silent`. - The review fixes cover both archived-default private-task cases and changed-storage rejection. With both instance isolation flags enabled, a new core repository project keeps a null policy and resolves to an isolated Git worktree. The complete affected set passes all 102 tests. - The CI query-error policy failure was fixed by using the shared query view. The form and shared query-view suites pass **24 tests**, including cached choices during a transient refetch failure. `node scripts/check-query-error-rendering.mjs --enforce --summary` reports zero findings. - CI exposed stale SDK source hashes and heartbeat fixtures that paired plugin storage with Local. Refreshed exactly four shared-source hashes with the maintained generator (runtime pins unchanged). All **53 SDK privacy/source-proof tests**, including real built SDK output, pass. All **7 corrected managed-storage heartbeat scenarios** pass; the fixtures declare the owning plugin's task environment and substitute transport at the existing execution seam. - Shared and UI TypeScript checks pass. The UI production build passes. `pnpm check:token-gates`, `node scripts/check-maps.mjs`, `node scripts/check-module-boundaries.mjs`, and diff checks pass. - Tested UI states: compatible choices, incompatible groups, default within a group, manual override, create failure, loading, discovery failure and retry, and an unavailable default. Tests use the production form and existing design tokens. - The integration tests use real isolated PostgreSQL on macOS arm64 with Node 26.12.0. They cover user/agent defaults, storage persistence, policy restrictions, company and authentication denials, and rejection before a provider call. Discovery uses saved configuration and makes no worker or provider requests. - Local verification limits: `pnpm -r typecheck` and `pnpm build` stop in unchanged adapter-utils code because the installed ACPX types lack `inheritProcessEnv`. A direct server check also finds an unchanged AI SDK type-version mismatch. The existing heartbeat test `inherits the instance default environment across companies while preserving explicit agent overrides` times out at line 518 on both this change and unmodified master with the same dependencies. The full `pnpm test:run` attempt passed all six chat shards (1,948 tests), then the Git workspace expiration suite reported 14 failures. I stopped that broad run. Two sampled failures reproduce on unchanged master with the same `unknown_worktree_git_files` refusal; the other expiration failures were not independently compared. A later broad workspace-preparation run also hit workspace validation failures and was stopped; the seven corrected fixture scenarios pass independently. These limits do not establish a full local suite pass. Hosted CI remains required. - Initial missing-package and loopback-denial failures were resolved by repairing isolated dependency links, building the SDK, and running database tests with loopback access. No deployed or live-provider execution is claimed. ## Risks - Creation defaults now follow the effective environment. A plugin's `defaultForProjects` claim alone no longer selects storage. This is the intended behavior change. - Compatibility covers repository project storage. It does not guarantee arbitrary host paths, retained-workspace moves, adapter/image support, credentials, or worker readiness. Matching provider configs cannot prove a cross-plugin storage contract. - A person's primary agent supplies the personal default because there is no separate personal environment setting. An unavailable configured default requires a valid explicit choice or a configuration repair. - No migration runs. Existing projects keep their storage owner. This change does not detach existing managed data or provide a storage migration. - Full local verification has the limits listed above. CI and standard Greptile review are pending. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, tool use, and code execution. The exact serving snapshot and context-window size were not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
8e332ccb5e |
fix(evals): name stale SDK pins and fix four chat-eval harness defects (#20579)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat feature eval catalog (`tests/runner-e2e`) runs real chat journeys against a real server and grades the saved state, the run's tool events and the rendered reply > - The full-cohort run on master `f4337bb5fb` had 12 failures that come from the harness, not from the product or the model > - Six plugin and secrets cells stopped in 7–9 s, before the server started, because the SDK public source pins were stale > - Four more cells failed on harness timing, a first-visit prompt, or a read receipt that ignored a dedicated tool > - This pull request fixes those four harness defects and adds a cheap guard that names stale SDK pins > - The benefit is that these cells now report on the product and the model again ## Linked Issues or Issue Description Refs #19614, #20550, #20453, #20417, #20483 **What happened** On master `f4337bb5fb`, these chat eval cells failed for harness reasons: - `plugins.configure-plugin`, `plugins.disable-plugin`, `plugins.install-plugin`, `plugins.use-plugin-tool`, `routines.plugin-managed-routine` and `secrets.least-scope-delivery` stopped before the server started. The source SDK privacy scan refused, because `tests/runner-e2e/source-sdk-public-inputs.json` did not match `packages/shared/src`. #20550 and #20362 already resynced the pins on master. The pins are current at `ce1ee9e89f`. - `agent-chat.track-delegated-work` and `navigation-preferences.edit-profile`: the reply was correct and had the right link. The case read the rendered links immediately after the runs settled, before the reply rendered, so it recorded no links. - `inbox-search.archive-done` and `inbox-search.restore-archive`: setup completes five tasks, so the GitHub star prompt from #20453 opened over the chat and blocked the composer. - `status-cards.explain-card`: the follow-up run read the deleted source with `get_task` and received a 404. The reply said so. `apiReadStatuses` counted only `call_api` reads, so the grader recorded no unavailable read. Why the existing pin check did not stop the drift: `ci / Runner support suite` runs the source SDK tests and is part of the required `ci / verify`. On #19614 it passed at 03:43Z on a merge ref from before #20417 (06:12Z) and #20483 (08:14Z). #19614 merged at 14:02Z without a new CI run on current master. After that, the suite failed on every PR until #20550. So the check exists and fires; the gap was a stale green result, and the failure message was opaque (`SDK public source proof unavailable`). **Expected behavior** - A change to a pinned SDK input without a pin refresh fails with the changed files and the command that fixes it. - Cases read rendered links only after the saved reply renders. - The eval browser is a returning user's browser that has already seen the star prompt. The product does not change. - A dedicated task read tool (`get_task`, `list_task_documents`, `read_task_document`, also when namespaced) counts as a read of the record it names, with the status its receipt reports. **Steps to reproduce** 1. Run `chat-features.runner-opencode.local.<case>` for one of the cases above on master `f4337bb5fb`. 2. Read `result.json` and `snapshots/chat-catalog-observation.json` in the evidence bundle. 3. See the failures described above. ## What Changed - `tests/runner-e2e/source-sdk-public-inputs.mjs`: add `staleSdkPublicInputs(root)`. It compares the registry with `git ls-tree HEAD` for every pinned input (the roots and build inputs that the pins file lists), and returns each changed, unpinned or removed file. - `tests/runner-e2e/source-sdk-public-inputs.test.mjs`: a test fails when the committed pins do not match the committed tree. The message names the files and gives the `--write` command. The Runner support suite runs it for every `packages/**` change. A synthetic test covers changed, unpinned and removed inputs. - `tests/runner-e2e/chat-catalog-domains/runtime-handoffs.ts` and `identity-profile.ts`: call the existing `waitForRenderedReply` before they read rendered links. - `tests/runner-e2e/chat-returning-browser.ts` (new) and `chat-catalog-flow.ts`: inside the flow's failure boundary, before setup, mark `paperclip.github-star-prompt.seen` in the eval browser. An init script covers later documents of the app origin, and the open page is marked directly. A test checks that the key equals `STAR_PROMPT_SEEN_KEY` in `ui/src/components/GitHubStarPrompt.tsx`. - `tests/runner-e2e/chat-catalog-domains/api-read-receipts.ts`: bind dedicated task read tools by the digest of the `call_api` input that the server builds for them (`taskBridgeApiRequest`). The result must carry the same `apiOperationId`. Other tools, other records, mismatched receipts and unpaired results do not count. - Bump `oracleVersion` in code and descriptor for every case that uses `api-read-receipts.ts` (shared grader change) and for `navigation-preferences.edit-profile`. Add `internal-comment-reference.ts` to the source files of the two cases that now import it. ## Verification - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts` (whole config): 11301 passed; 1 local-only failure in `controller-replacement-smoke.test.ts` (server spawn, no changed imports). - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts --testTimeout=180000 tests/runner-e2e/chat-catalog-domains tests/runner-e2e/chat-catalog-contract.test.ts tests/runner-e2e/chat-catalog-consistency.test.ts tests/runner-e2e/chat-returning-browser.test.ts` - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts source-sdk` - `node tests/runner-e2e/source-sdk-public-inputs.mjs $(git rev-parse HEAD) --check` reports `changed: false`. - Red checks: with the `f4337bb5fb` registry, the new pin test fails and names 15 inputs. Without the new waits, the two rendered-reply tests fail. A regrade of the retained `explain-card` run events now gives the follow-up run `unavailable: true, live: false` and the first run `live: true`. - Paid runs of the affected cells (`chat-features.runner-opencode.local.*`): - Run 42852698879 (head `d54c04d5f0`): `track-delegated-work`, `restore-archive`, `explain-card`, `configure-plugin` and `plugin-managed-routine` pass. I read each reply and saved state. `track-delegated-work` now records the rendered `/COM/issues/COM-2` link. `explain-card` records `laterUnavailable: true, laterLive: false` from the follow-up `get_task` 404. - `plugins.install-plugin` now starts its server (the pin refusal is gone). It fails a later grading check, `truthful-installation-handoff`: the reply sends the person to the plugin settings page but does not name the package. This is outside this PR. - `inbox-search.archive-done`: the star prompt no longer blocks the composer, and the query is sent. The agent then archives the 501 Done tasks one at a time (about 2 s each, 57 in the 240 s window), because only the per-task archive route exists. This is outside this PR. - `navigation-preferences.edit-profile` did not start in that run (the AWS credentials step timed out). Run 42857020103 (head `2f0b80aa84`): pass. The agent write gets 401, the reply links `/COM/company/settings/instance/profile`, and the person follows the rendered link and saves the name. ## Risks - Low risk. Test harness only; no product code changes. - The receipt change counts more reads as record reads. It only adds dedicated reads bound to the exact record id and operation, so a grader that needs a read can pass on a genuine `get_task` read. The affected cases have new oracle versions. - The star prompt is not tested by any chat catalog case. Product e2e tests for the prompt do not use this flow. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file editing, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude
|
84007892e9 |
refactor: move AGENTS.md section detail to docs and rules, add a bug-fixing section (#20585)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - `AGENTS.md` is the instruction file that each human and AI contributor reads before a change > - Pull request #20557 shortened the file, but several sections still held detail that applies to only some changes > - The scoped-rules section told agents to read each matching rule in full, and most changes match five rules > - The core engineering rules mixed three product invariants with process conventions > - This pull request moves that detail to the docs and to scoped rules, and adds a section on how to fix a bug > - The benefit is a 211-line file that states what applies to every change and points to the rest ## Linked Issues or Issue Description Refs #20557 **Issue type** Outdated or unclear contributor documentation. **Where is the issue?** `AGENTS.md` at the repository root, and `rules/`. **What's wrong?** - The scoped-rules section made agents read the full body of each matching rule before a change. The rule bodies total about 47 KB. - Sections 4, 6, 8, 9 and 10 repeated detail that belongs in `doc/DEVELOPING.md`, `doc/DATABASE.md`, a route rule, `DESIGN.md` and `CONTRIBUTING.md`. - Section 4 had a database reset command for `data/pglite`. That directory does not exist. - Section 5 had seven rules. Four of them were process conventions or path-specific detail. - The file did not say how to fix a bug. **Suggested fix** Make the rule headers the index. Move section detail to the document or rule that owns it. Keep only the three invariants in section 5. Add a bug-fixing section. ## What Changed - The scoped-rules section now tells agents to list the rule headers with one `grep` command. Agents read a rule body only for a design decision in its area, when the description is not sufficient, or when a reviewer cites it. - `rules/README.md` says that each `description` must state the required behavior in one sentence. - `rules/pull-request-review.mdc` has a description that states its requirements. - Section 4 (dev setup) and section 11 (pull requests) now point to `doc/DEVELOPING.md` and `CONTRIBUTING.md`. - The schema change workflow moved from section 6 to `doc/DATABASE.md`. - The API and auth expectations moved from section 8 to the new `rules/api-route-expectations.mdc`. - Section 9 links to `DESIGN.md` for design rules. `AGENTS.md` no longer tells contributors to run `pnpm check:token-gates`. - Added section 10, "Fixing a Bug": reproduce the error, write a failing test, write the fix, see the test pass, and verify the fix. - Section 5 now has three rules: company scope, contract synchronization, and the control-plane invariants. - The company-scope exceptions moved to the new `rules/company-scope-exceptions.mdc`. - Removed the strategic-docs rule. The plan-doc convention moved to the repo map. The artifact rule moved to section 7. - Removed rule 7. Five code comments that cited `AGENTS.md §5.7` now cite `rules/data-paths.mdc`. - Regenerated the SDK public source pin, because one of those comments is in `packages/shared/src/crash-points.ts`. - Added both new rules to `.greptile/files.json`. - The API route rule applies only to routes that read or write company data. Its status code list is a list of common codes. - The company-scope rule lists the sidebar company order as an instance-wide exception. - The `honeycomb-tracing` skill now points to `rules/data-paths.mdc` for the Telemetry workflow. ## Verification - Open `AGENTS.md` and confirm that each link resolves. A local script checked this. - Run `grep -E '^(description|globs|alwaysApply):' rules/*.mdc` and confirm that each rule prints a description. - Run `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`. It passes locally. - `packages/shared/src/crash-points.test.ts` passed locally before the rebase onto `master`. - Five source files changed in comments only. No other test suite ran locally. ## Risks - An agent that reads only rule descriptions can miss a detail in a rule body. Reviewers still apply the full rules. - Pull requests and the definition of done are now sections 11 and 12. Sections 1 to 9 keep their numbers. - Dated plan documents under `doc/plans/` still cite `AGENTS.md §5.7`. They are historical records. - Pull request #20565 also changes `AGENTS.md`. The second one to merge needs a conflict fix in section 1 and section 2. ## Model Used - Claude (Anthropic), in Claude Code with tool use. - `claude-fable-5-2`. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
ce1ee9e89f |
fix: stop prompts from encouraging subtasks (#20362)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents read runtime prompts, tool help, skills, and role instructions. > - Several of these texts tell agents to create child tasks as a normal next step. > - One accepted-plan prompt even forbids work on the current task. > - Task parentage is useful, but it does not need a prompt that encourages task creation. > - This pull request removes those prompts and keeps the task APIs available. ## Linked Issues or Issue Description Refs #10642. This change adjusts prompt wording; it does not add delegation-cycle guards. **Current behavior** Prompts tell agents to create children after plan approval, use children for long work, and create a task just to pass context. Onboarding requires a child even for a single request. **Proposed behavior** Continue work on the current task. Keep explicit assignments, optional task creation, parentage, dependencies, and approval gates. Use the separate-task recipes when the user requests that structure or when continuing existing work. **Reason and benefit** Reduce needless task trees without removing subtask support. Keep the patch limited to wording, existing test expectations, and generated catalog data. ## What Changed - Align accepted-plan prompts and remove direct child-creation advice from runtime and error text. - Make task tool descriptions neutral. Keep their fields, defaults, and handlers. - Update the main skill, onboarding, CEO instructions, company templates, specialist workflows, and matching guides. - Keep historical release snapshots intact. Refresh the explicitly seeded schema fixture's catalog digest; the server still verifies historical release snapshot hashes. - Refresh the affected generated catalogs, capability inventory, semantic contracts, and fixture manifest. ## Verification - Final tested head: `c6e304d0c105cdbfbe0b2ab0a233307fcd91a04b`. - All 66 check runs are complete: 60 pass, one is neutral, and five are skipped by CI conditions. Snyk also passes. No checks fail or remain pending. - [PR CI](https://github.com/paperclipai/paperclip/actions/runs/42846226554) passes typecheck, build, server/general tests, all eight E2E shards, Runner support, Runner Rust, and the aggregate verify gate. [Windows](https://github.com/paperclipai/paperclip/actions/runs/42846226546) and [Docker Runner](https://github.com/paperclipai/paperclip/actions/runs/42846226108) also pass. - [Greptile](https://github.com/paperclipai/paperclip/pull/20362#issuecomment-6751419835) and [Paperclip review](https://github.com/paperclipai/paperclip/pull/20362#issuecomment-6751454010) give this exact head 5/5. All six inline threads are resolved. Neither current summary has findings. - Merged master `bf0220d34a156c81853755c3d1365ddc527037de`. Resolved three generated-file conflicts with the canonical generators. - Local verification passes: 473 focused prompt, continuation, UI, catalog, runtime and native-execution tests; Runner TypeScript build; generated metadata; UI token gates; and diff checks. - The prior head `c22018cb0f` failed three SDK evidence tests because two source hashes were stale. Reproduced all three failures locally. The existing generator refreshed only the hashes for the denial-copy file and its test. All 53 affected evidence/source-pin tests then passed, including private-value rejection. No scanner logic, assertions, or runtime dependency pins changed. Current CI confirms the support suite passes. - A prior commit failed the Rust process reaping test. Its cause remains unproven. No Rust code or assertions changed. Later Rust runs pass; this does not establish the cause of the earlier failure. - Local preflight built shared/plugin SDK packages, then stopped because `cargo` is absent. Current-head CI supplies Rust and full build coverage. `pnpm verify --explain` selects broad required CI; full local suites were not repeated. ## Risks - Wording changes guide the model; they do not enforce a ban on subtasks. - Existing installed skill versions and saved custom instructions need their normal refresh process. This PR does not migrate company configuration. - Existing approved task assignments and approval gates remain in effect. ## Model Used - OpenAI GPT-6, through Codex, with reasoning, tool use, and code execution. The exact routed model ID and context-window size were not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude
|
8d2bf78323 |
test(runner-e2e): resync the SDK public source pins with master (#20550)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner support suite checks that the plugin SDK's public types come only from reviewed source inputs, pinned by git blob hash in `tests/runner-e2e/source-sdk-public-inputs.json` > - #19614 landed those pins, generated against an older master; since then `package.json` and ten files under `packages/shared/src` changed > - So `source-sdk-public-type.test.ts` (2 tests) and `source-sdk-evidence.test.ts` (1 test) fail the `ci / Runner support suite` check on every PR merge ref > - This pull request regenerates the source pins with the script made for that job > - The benefit is a green Runner support suite for every open PR again ## Linked Issues or Issue Description **What happened?** Since #19614 merged, `ci / Runner support suite` fails on every PR (for example #20047 and #20538) with `SDK public source proof unavailable` and `Source SDK evidence privacy check failed`. **Steps to reproduce** 1. On master at |
||
|
|
f4337bb5fb |
test(evals): land the chat catalog eval harness on master (#19614)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat-feature evals measure whether agents complete real chat requests through the product > - The chat catalog eval harness and its repairs live on the `chat-eval/master-candidate` branch, not on master > - Today's work made 139 of 242 chat evals pass in verified single-case live runs. Each product fix landed on master as its own pull request > - The eval-side changes (setups, graders, fault seams, evidence capture) still exist only on the candidate > - This pull request lands the harness on master. Master stays the source of truth for product behavior > - The benefit is one tree: evals run against master commits, and product fixes reach the evals as soon as they merge ## Linked Issues or Issue Description Refs #19441, Refs #19599, Refs #20270, Refs #20334, Refs #20342, Refs #20344, Refs #20354, Refs #20360, Refs #20373, Refs #20382, Refs #20409 **What existing behavior does this improve?** The chat catalog eval harness and the product it evaluates live on a long-running branch outside master. Eval fixes do not reach master, so master-based eval runs use old setups and graders. **Subsystem affected** `tests/runner-e2e` chat catalog harness, plus a small set of product changes that exist only on the candidate. **Proposed change** Land the harness on master. Every product fix from this eval work already merged to master as its own pull request. Conflicts keep master's product behavior. ## What Changed - Most of the diff (808 of 851 files) is under `tests/runner-e2e/`: chat-catalog domains, scenarios, descriptors, the TestOps manifest, private evidence tooling, and their tests. - Graders now check saved state, setup literals and rendered or clicked links. Many keyword and regex checks on free-form replies are removed. Each changed grader has a bumped `oracleVersion` and calibration tests (a valid reply, a reworded valid reply, and broken variants that must fail). - The shared link helper `hasLink` no longer treats a list link (for example `/COM/issues?view=mine`) as a link to every task without a URL key. - Setup fixes build real state through supported product routes: routine history, agent grants through the board grant route, budget holds with a real cap, and a storage-layer archive fault for `inbox-search.archive-partial-failure`. - Browser boundary journeys (sign-in, chat-disabled, onboarding) now write the standard pass evidence files. - Retargeted cases, approved by the maintainer: `agent-chat.chat-disabled` (Conference Room gate), `agents.restore-instructions` (no saved instruction history), `access.manage-membership` (member permission edits stay board-only). - The private evidence trusted files (`manual-private-handoff.ts`, `manual-private-retention.ts`, `manual-private-report-assets.ts`) are byte-identical to master. - Product files outside `tests/` (43 files, +1,328 −49) carry candidate-only changes that master does not yet have. Most are tests. The main code changes are in `request-human-input.ts`, `heartbeat-run-summary.ts`, `native-completion-feedback.ts`, `native-run-finalizer.ts`, `openapi.ts`, `secrets.ts` and `IssueChatLongThreadPerf.tsx`. ## Verification - `pnpm vitest run chat-catalog-domains chat-catalog-contract.test.ts chat-catalog-consistency.test.ts unit-launcher.test.mjs --testTimeout=180000` in `tests/runner-e2e`: 269 files pass, 3,550 tests pass. - `node --test chat-full-catalog-testops-manifest.test.mjs`: 10 pass. - `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`: no change. - `tsc` for `server` and `tests/runner-e2e`: clean. - Live single-case verification: 139 of 242 cases passed on paid runs, and a person read the saved evidence for each. These are single-case runs, not one full-cohort run. ## Risks - Large diff. Most of it is test-only, but the 43 non-test files change product code paths. Review those files first. - Some graders are now less strict on reply wording. A person must read replies for meaning on those cases. - 42 cases need disposable outside accounts (Notion, Slack, GitHub, Daytona and others) and stay blocked. `access.invite-teammate` stays failing until local agent isolation lands, because the runner's durable state keeps a live invite token. - The paid workflow compares the trusted private-evidence files with master. They are unchanged, so this risk is low. ## Model Used Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution, in Claude Code. Subagents used the same model. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandClaude Opus 5.5
|
ff4ba7d319 |
perf(runner): run launch artifacts in place from a trusted read-only root (#20428)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Rust Runner (`paperclip-runnerd`) starts provider sidecars and proxies from qualified launch artifacts: a Node runtime and the scripts it runs > - Since #12699, Runner copies each artifact into a sealed memfd and checks its SHA-256 on every verification. This defends against a path swap or an in-place rewrite between the check and the exec > - On a hosted agent computer (4 vCPU), the copy and digest of the 127 MB Node takes about 0.26–0.30 s warm and about 1.2 s right after a wake. This is on the path from `session.open` to the sidecar spawn, and again on every warm `run.attach`. Each Runner also holds about 125 MiB of shmem for the copy > - On that computer, the artifacts live on the Runner ROM: a read-only erofs image that the deploy pins by digest. The threat that the memfd defends against does not exist there. On local hosts the artifacts are writable, so the memfd must stay > - This pull request adds an explicit, path-scoped, default-off `--trusted-launch-root <dir>`. Runner runs an artifact in place only when its open descriptor proves that it is on an immutable mount under that root. Everything else keeps the sealed, hashed snapshot > - The benefit is a shorter `session.open` and `run.attach` on agent computers, and about 125 MiB less shmem per Runner, with no change where the flag is absent ## Linked Issues or Issue Description **What existing behavior does this improve?** Launch-artifact verification in `paperclip-runnerd` (`qualified_launch.rs`). Its callers are the ACPX `verified_transport` (every `session_config`, including the park rebind) and the OpenCode and Cursor Cloud proxy launches. **Subsystem affected** Rust Runner (`packages/paperclip-runner/runner/crates/runner-core`). **Current behavior** Every verification opens the artifact, copies all bytes into a sealed memfd, and hashes them. Node is 127 MB. The cost is described in the Thinking Path. **Proposed behavior** When the host passes `--trusted-launch-root /opt/paperclip/runner` and the root passes its checks, Runner holds each admitted artifact open in place and executes it through `/proc/self/fd/N`. It does not copy or re-hash it. Without the flag, nothing changes. **Reason and benefit** See the Thinking Path. Refs #12699 (introduced the sealed memfd). Refs #20423 (overlap, see below). ## What Changed - `qualified_launch.rs`: new `TrustedLaunchRoot` and `set_trusted_launch_root` (process-global, set once at startup, like `--task-memory-cgroup`). - **Root checks at startup.** The path must be absolute, already canonical (no symlink or `..`), a directory, and on a read-only, exec-permitted erofs or squashfs mount. If a check fails, runnerd logs a warning and keeps today's behavior. - **Per-artifact checks, on the open descriptor.** After today's checks (regular file, not a symlink, not group- or world-writable, open-time identity), the descriptor must be a regular file on the root's device. Its kernel path (`readlink /proc/self/fd/N`) must lie under the root by path components, so `/opt/paperclip/runner2` does not match. Its mount must pass `fstatvfs` (`ST_RDONLY` set, `ST_NOEXEC` clear) and `fstatfs` (erofs or squashfs magic). If any check fails, Runner logs the reason and uses the sealed snapshot. - **Admitted artifacts** become `VerifiedProcessArtifact::in_place_on_trusted_root` (crate-private). The held file goes through the existing `inherited_artifact` path, so the exec is `/proc/self/fd/N` and `PAPERCLIP_VERIFIED_RUNTIME_EXECUTABLE` still names a descriptor that nested launches can reopen. - New `verify_assigned_executable_launch_artifact` never uses the root. `stdio_mcp.rs` now calls it, so host-assigned stdio MCP binaries always keep the sealed snapshot. - `paperclip-runnerd.rs`: parse `--trusted-launch-root <dir>`. Older runnerd builds ignore unknown flags, so a host can pass it before every Runner understands it. - The trusted path is Linux-only. On macOS and Windows, `TrustedLaunchRoot::open` always refuses. - `tests/runner-e2e/SECURITY.md`: documents how the Kimi and Pi provider-death fixtures behave under a trusted root. ### Decision: the trusted path drops the digest The trusted path does not hash the artifact, not even once per process: - The ROM image is pinned by digest at deploy time. - On the agent computer, the Launcher computes the expected `--acpx-sidecar-command-sha256` and `--acpx-sidecar-script-sha256` from the same ROM files immediately before it starts Runner. A re-hash of an immutable file can only repeat that result. - One hash per process would not help `session.open`, because each task starts a new Runner process. - The mount check is the guard: erofs and squashfs have no write path, so the bytes cannot change while the descriptor is open. ### Fixtures that assert the sealed memfd - Untrusted-mode assertions are unchanged. The fixtures do not pass the flag. - Kimi (`kimi-provider-fault.py` `checked_parent_descriptor`): with a trusted root, the parent descriptor is the ROM's Node file (one link). The existing linked-descriptor branch handles it: same inode as `/proc/<pid>/exe`, and the complete bytes hashed against `nodeSha256`. That branch never accepts the file as a memfd. The existing test `test_linked_descriptor_keeps_regular_single_link_checks_without_memfd_claim` covers it. - Pi provider-death admission binds `/proc/self/fd/3|7` to the native distribution snapshot's Node. The TypeScript ACPX closure copies that file whether or not a trusted root is set, so the flag does not affect it. ### Overlap with #20423 #20423 (draft) moves the same copy and hash to Runner boot and reuses the result. Where this flag is set (Cloud agent computers), there is nothing left to precompute, so #20423 is mostly redundant there. #20423 still helps hosts without a trusted root (local and sandbox Runners). This PR does not change #20423. ## Verification All runs are on Linux (aarch64, Rust 1.97.1, in a container): - `cargo fmt --manifest-path runner/Cargo.toml --all -- --check` - `cargo check --manifest-path runner/Cargo.toml --locked --workspace` - `cargo test -p paperclip-runner-core --lib -- qualified_launch trusted_root commonjs`: 12 passed. The new tests are: - `an_artifact_under_an_immutable_trusted_root_runs_in_place_without_a_digest`: trusted-root hit, with no memfd, the same inode as the source, and no re-hash. - `an_artifact_outside_the_trusted_root_keeps_the_sealed_snapshot`: a path outside the root is sealed, and a wrong digest still fails. - `a_sibling_with_the_root_as_a_name_prefix_is_outside_the_root`: `rom2/` does not match `rom`. - `a_root_that_is_not_read_only_falls_back_to_the_sealed_snapshot`: the real probe refuses a writable root at startup. A per-artifact mount failure falls back, and a wrong digest still fails. - `the_real_mount_probe_rejects_a_writable_filesystem` - `symlink_and_parent_escapes_from_the_root_fall_back`: a directory symlink out of the root, a `..` path out of the root, and a final-component symlink (still refused). - `a_root_must_be_absolute_canonical_and_a_directory` - `without_a_trusted_root_launch_artifacts_are_sealed_and_hashed`: flag absent, behavior unchanged. - `host_assigned_executables_ignore_the_trusted_root` - `trusted_root_artifacts_execute_the_held_file_and_reopen_it_for_nested_launches`: Runner execs from the held file, and a child re-execs `$PAPERCLIP_VERIFIED_RUNTIME_EXECUTABLE`. Both report the source file as `/proc/self/exe`. - `cargo test -p paperclip-runner-core --test stdio_mcp --test process_supervisor`: 4 and 8 passed. - Ad hoc, not committed: a privileged container with a real loop-mounted squashfs. The real `fstatvfs`/`fstatfs` probe admitted a `ro` mount and refused a `ro,noexec` mount. The unit tests replace only that mount probe, because CI cannot mount filesystems. - Not verified: a run on an agent computer with the real erofs ROM, and the latency and memory savings there. The Cloud Launcher change that passes the flag is a separate follow-up. ## Risks - **This change needs review from a security owner before merge.** It removes the sealed copy and the per-launch digest for admitted artifacts. That relies on the host's assertion that the root is a digest-pinned immutable image, plus the descriptor checks above. - **Misconfiguration fails closed.** A wrong, writable, `noexec`, non-canonical or missing root only loses the speedup, and runnerd logs a warning. There is no global "trust everything" switch, and Runner never selects a root from the filesystem it finds. - **Residual trust.** On an admitted file, Runner no longer detects a ROM image that differs from what the launch profile expected. On the agent computer, the Launcher derives the expected digests from the same ROM. Another host that passes the flag must pin its image the same way. An attacker with root in the guest (who could write the backing block device or remount) is out of scope; such an attacker can already control runnerd. - **Allowlist.** Only erofs and squashfs are accepted, because they cannot be written through any mount. A read-only bind mount of a writable filesystem is refused. A ROM that changes to another filesystem type will silently fall back to the sealed path (slower, still safe). - No migration. Behavior is unchanged when the flag is absent. Older runnerd builds ignore the flag. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, container builds). Extended reasoning was on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
f7c85bca18 |
test(runner-e2e): delete each runner E2E instance when it ends and expire kept instances after 6 hours (#20246)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness boots a full temporary Paperclip instance per wave: embedded Postgres, workspaces and caches, often 5–13 GB > - At wave end it deleted that instance only when the wave passed with no harness error and no `--keep` > - Almost every chaos wave has at least one failing task, so almost every instance stayed on disk > - Over Nov 6–9, 359 `paperclip-runner-e2e-*` directories filled the machine's disk (ENOSPC), which stopped a running storm wave > - This pull request deletes the instance at the end of every wave, smoke and baseline run unless it is explicitly kept or its processes could not be stopped, and expires kept instances after 6 hours > - The benefit is that chaos waves can run back to back without filling the disk ## Linked Issues or Issue Description **What happened?** `main.ts` kept the temporary root whenever `keep || anyFailure || harnessError`. A failing task is the normal output of a chaos wave, so instances accumulated until the disk was full. The codex K13 storm w-20261109T113925-s9403 died after about a minute with every task cut off. **Steps to reproduce** 1. Run any wave in which at least one task fails. 2. The `paperclip-runner-e2e-*` directory under the OS temp dir stays behind, with no expiry. **Expected behavior** The instance is removed when the wave ends. Evidence is already in `results/`. Only an explicit `--keep`, or a stop failure that may leave processes running, keeps it, and kept instances expire. **Actual behavior** Every failing wave leaks its instance permanently. ## What Changed - New shared `tests/runner-e2e/instance-retention.ts`: - `releaseInstance` deletes the root unless `--keep` was passed or the processes could not be stopped. In those cases it writes a `.paperclip-runner-e2e-kept.json` marker (owner, reason, time) and sets the directory to 0700. An optional `retain` list keeps small evidence files and removes everything else. - `sweepExpiredKeptInstances` removes `paperclip-runner-e2e-*` directories that carry the marker, are older than 6 hours, and have no live `postmaster.pid` anywhere inside them, and that no live process command line references. Each root is made removable first, and one failed delete does not stop the sweep. Unmarked directories are never touched, so existing directories from older runs stay until someone decides on them. `recovery_state` markers never expire. - `createRunnerE2ETemporaryRoot` runs the sweep, so every caller gets it. - Callers: - A wave whose Daytona cleanup is still pending (environment DELETE blocked, sandboxes still running after cleanup, or cleanup errors) keeps its instance as `recovery_state`, because its database owns that cleanup. - The wave releases its instance before writing the ledger and summary, so a failed report write cannot skip cleanup. - Restart chaos `main.ts` (crash-point suite included) releases the instance in its final cleanup, which also runs on failure and interrupts. The crash-point task-count check moved above supervisor creation so that error no longer leaks a fresh root. - `ttft-baseline.ts` releases after stopping the supervisor, including on SIGINT and SIGTERM. - The rolling-deployment and controller-replacement smokes now delete their roots, keeping only the evidence report, logs and screenshots. If cleanup is unconfirmed, the root is kept with a marker. - `launch.ts` marks its unconfirmed-remote-cleanup root as `recovery_state`, which never expires, and its `PAPERCLIP_RUNNER_E2E_KEEP_FAILED_PRIVATE` root as `keep`. Its existing deletion path is unchanged, and the file gets one line shorter. - README: the instance lifecycle and `--keep`. ## Verification - `npx vitest run --config tests/runner-e2e/vitest.config.ts`: 328 files, 6915 passed. The new retention tests cover: - deleting a finished instance, and retaining only listed evidence; - keeping it with a marker and mode 0700; - sweeping only marked, expired instances whose Postgres isn't running, while sparing fresh, live, recovery-state, unmarked and unrelated directories; - a stale `postmaster.pid`; - sweeping on `createRunnerE2ETemporaryRoot`. The create-sweep test failed before `createRunnerE2ETemporaryRoot` ran the sweep. - `npx vitest run --config tests/restart-chaos/vitest.config.ts`: 221 passed. - `node scripts/typecheck-test-projects.mjs tests/runner-e2e` and `tests/restart-chaos`: clean. - No wave or smoke was run, because the disk is still nearly full. ## Risks - A failed wave's instance is no longer available for post-mortem DB inspection unless the wave ran with `--keep`. Evidence bundles in `results/` are unchanged. - The sweep relies on `postmaster.pid`. An instance whose Postgres was started outside the harness, without that file, could be removed after 6 hours. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
347d21fcbd |
test(runner-e2e): scope each agent's sandbox to its own run's saved state (#20253)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The runner E2E chat evals run real agents against a local Paperclip
instance on a CI host
> - PR #20044 hides the eval harness from local native agents with a
Linux Landlock domain around each runnerd launch
> - That domain is the same for every run in an attempt, so an eval
agent can still read another run's saved state
> - Recent chat-eval runs show agents that read other runs' OpenCode
session databases, another run's run log, the instance `pg_wal`,
`control-plane-state.json`, and `/proc/<pid>/cmdline` of harness
processes
> - This pull request scopes each runnerd launch to its own run and
moves the selected case id off the command line
> - The benefit is that an eval agent cannot learn about the case or
other runs from files outside its own run
## Linked Issues or Issue Description
Refs #20044
**What happened**
In local native OpenCode chat-eval cells, worker agents read files that
belong to other runs or to the harness:
- other runs' OpenCode sqlite session databases under
`runtime/paperclip-runner/durable-sessions/` (a status-cards run)
- `/proc/<pid>/cmdline` of harness processes, which shows harness paths
and the case id (`routines.pause-owned`)
- another run's run-log ndjson, and a listing of the instance
`db/pg_wal` (`routines.pause-owned`)
- a listing of `control-plane-state.json`
(`runs-adapters.list-model-options`)
The Landlock sandbox from #20044 hides only the harness (`tests/`, the
Git store, private attempt directories). All runnerd launches share one
hidden set, so the instance's per-run directories stay readable.
**Expected behavior**
An agent process sees only its own run's durable session, OpenCode
session and provider trace. Other runs' sessions, run logs and traces,
the instance database files, secrets and the shared control-plane state
in the attempt instance are denied. The case id is not on any harness
command line.
**Steps to reproduce**
1. Dispatch a local native OpenCode chat-features cell that has two
agents (for example a status-cards case).
2. In the worker's shell, run `ls
<instance>/runtime/paperclip-runner/durable-sessions/*/` and `ls
<instance>/db/pg_wal`.
3. Run `cat /proc/<launcher pid>/cmdline`.
4. Before this change, all three succeed and show other runs' files and
`--id chat-features...`.
## What Changed
- `agent-filesystem-sandbox.py`: new `--expose PATH` option. It grants a
path beneath a hidden directory. Landlock does not check traversal, so
the run uses its own entry while the parent stays unlistable. The helper
refuses a missing exposure, an exposure outside every hidden path, or
one that would reveal a hidden path. It fails the launch unless every
exposed path is readable.
- `agent-filesystem-sandbox.ts`: the launcher passes the instance root
in the sandbox configuration. `runnerdLaunchSandbox` adds per-launch
hidden paths: `durable-sessions/`, `runner/`, `control-plane/`,
`opencode/` under `runtime/paperclip-runner/`, and `data/run-logs/`,
`data/provider-traces/`, `data/backups/`, `db/`, `secrets/`. It exposes
only the launch's own durable-session directory (from `--state-dir`),
Codex runner state directory, OpenCode session directory (from
`--session-id`, named as the OpenCode driver names it) and provider
trace file.
- The wrapper creates the exposed entries, and the shared instance
directories that the product creates on first use, before runnerd
starts. It logs the run-scoped hidden and exposed paths for each launch.
- New `probeRunScopedSandbox`: at server start, a sandboxed shell runs
against a fixture instance. Another run's session, OpenCode database,
runner state, run log, trace, `pg_wal`, secrets, control-plane state and
the server's `/proc/<pid>/environ` must be denied. The run's own
session, OpenCode directory, ACPX directory and trace must be writable.
In CI any failure stops the attempt. The result goes to `server.log`.
- `selectors.ts` and `launch.ts`: the launcher reads the selected cell
from `PAPERCLIP_RUNNER_E2E_SELECTED_ID` and removes it from its
environment.
- `runner-full-stack-e2e.yml` (Run paid cell) and
`chat-catalog-testops-case.yml`: pass the cell in that variable instead
of `--id`. A target branch that does not read the variable still gets
`--id`.
- `SECURITY.md`: documents the run scope, the `/proc` limit and what
stays readable.
## Verification
- `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/agent-filesystem-sandbox.test.ts
tests/runner-e2e/workflow-security.test.ts
tests/runner-e2e/manual-private-workflow.test.ts
tests/runner-e2e/private-trust-workflow-security.test.ts` passes on
macOS. The Landlock host tests (including the new fixture-instance probe
test) run only on Linux CI.
- `npx tsc --noEmit -p tests/runner-e2e` shows no errors in the changed
files.
- Paid run
[42611358428](https://github.com/paperclipai/paperclip/actions/runs/42611358428):
`chat-features.runner-opencode.local.agents.change-manager` on the same
commit applied to `chat-eval/master-candidate` (
|
||
|
DottaandClaude Opus 5.5
|
0d80b74c21 |
test(chaos): treat a crash-restart EPERM as a warning once the server pid is dead (#20232)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness crashes the local Paperclip server with SIGKILL (K1, K13 storms) and grades how tasks survive > - `crashForRestart` sends SIGKILL to every process group the server owns > - On macOS, `killpg` can report EPERM for a group whose members are still exiting > - When that happens the fault is recorded as failed, and the wave exits 2, even though the server process itself is dead > - This pull request tolerates EPERM in the crash path once the server pid is confirmed dead, and logs it as a warning > - The benefit is that a wave grades the product, not a cleanup race in the harness ## Linked Issues or Issue Description **What happened?** A local Codex K13 storm at concurrency 6 (wave w-20261109T084023-s9403, 112 tasks) exited 2 because faults f04 and f09 failed with `kill EPERM`. The server pids of both faults had died, and the tasks recovered normally. **Steps to reproduce** 1. Run a long K13 storm on macOS with high concurrency. 2. Occasionally the SIGKILL of the server's process groups races with members that are exiting. **Expected behavior** If the server process is dead, the crash completes and the remaining cleanup continues; the EPERM is recorded as a warning. **Actual behavior** The fault fails with `kill EPERM` and the wave reports an error. ## What Changed - `crashForRestart` catches EPERM from the group SIGKILL. It then sends SIGKILL to the server pid directly if needed and waits for it to exit within the forced timeout. - If the server exited, it re-observes the tree and sends SIGKILL to each remaining group on its own, so one group's EPERM does not leave the others to drain gracefully. It logs a warning naming the server pid and any group that still refused SIGKILL, then continues with the normal post-kill cleanup. - If the server is still alive, or the error is not EPERM, the error is rethrown as before. - Four tests: EPERM with a dead server finishes with a warning; a detached descendant is still killed without a graceful drain when the first group kill reports EPERM; EPERM with a surviving server fails; a non-EPERM error fails. The failure tests stop their owners in `finally`. ## Verification - `npx vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/server-stop.test.ts`: the dead-server test fails before the change (`kill EPERM`), the descendant test fails against the first revision (the descendant drains on SIGTERM), and all 22 pass after it. - `node scripts/typecheck-test-projects.mjs tests/runner-e2e`: clean. ## Risks - A group that is still running and returns EPERM is no longer reported in the crash step itself. The cleanup that follows still checks every observed group, and it fails if any group survives. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
218eb4ec8b |
test(runner-e2e): hide the eval harness from local native agents (#20044)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Runner E2E evals run real agents against a real Paperclip server, then grade the result > - A local cell runs the product from the same checkout as the eval, as the same OS user > - In run 42472276601 an OpenCode agent read the eval's own scenario YAML and grader from that checkout > - An agent that can read the oracle can steer toward it instead of the user's request, so the eval stops measuring the product > - This pull request hides the eval harness from every local native agent process and removes eval names from fixtures > - The benefit is that a passing chat eval again shows product behavior, not oracle access ## Linked Issues or Issue Description Refs #19614 (the chat catalog eval harness that these graders belong to). **What happened** In run 42472276601, cell `chat-features.runner-opencode.local.access.cli-authorization` (OpenCode, DeepSeek), the agent printed its environment and saw the checkout path in `PATH` and `PAPERCLIP_RUNTIME_NOTIFICATION_HELPER`. OpenCode's `read` and `grep` tools were denied by its `external_directory` policy. The agent then used its `bash` tool with absolute paths, which that policy does not gate. It listed `/home/runner/_work/paperclip/paperclip`, read `docs/cli/*.md` and `cli/src/commands/client/*.ts`, ran `grep -rn` over `tests/runner-e2e`, and found the query in `chat-full-catalog-testops-manifest.json`. It then read `tests/runner-e2e/chat-scenarios/access.yaml`, the manifest entry, and the case grader `chat-catalog-domains/identity-cli-authorization.ts`. Its reasoning said: "This is the exact test case!" Separately, fixture company names contained the case id, for example `Runner E2E chat-features.runner-opencode.local.access.private-resource …`, and an agent noticed that it was in a test. **Expected behavior** The agent under test cannot read the eval's scenarios, fixtures, graders or captured evidence. It keeps everything that it needs: its workspace, its home, the product skills, the product packages and temporary files. The names it can see do not contain case ids, "E2E" or "test". **Steps to reproduce** 1. Dispatch `runner-full-stack-e2e.yml` for any `chat-features.runner-opencode.local.*` cell on master. 2. In the agent run, use bash to execute `grep -rn expected_outcome /home/runner/_work/paperclip/paperclip/tests/runner-e2e/chat-scenarios`. 3. The command returns the scenario expectations. With this change, the command returns `Permission denied` and exit code 2. ## What Changed - `tests/runner-e2e/agent-filesystem-sandbox.py` (new): applies a Linux Landlock domain that grants filesystem access everywhere except the hidden paths, sets `no_new_privs`, checks in-process that the hidden paths are denied, then `execv`s the command with the same PID, argv and environment. - `tests/runner-e2e/agent-filesystem-sandbox.ts` (new): selects the hidden paths (`tests/`, `.manual-private-trusted/`, the private evidence directory, `RUNNER_TEMP`). It never hides the agent's temporary root, the checkout root or the shared temporary directory. It runs a setup probe, logs the result and fails closed in CI. Then it wraps each `paperclip-runnerd` spawn of the E2E server. - Follow-up: run 42445668869 showed that the hidden list missed two places. A fixture worker listed Playwright's compiled copy of the harness and graders with source maps (`/tmp/playwright-transform-cache-1001/`). An agent in another run listed the attempt root's `artifacts-private/`, `control/` and `selected-private-inputs/` under `/tmp`. A sandboxed attempt now creates its root in `/var/tmp`. The reason: Landlock hides a path by leaving its ancestors ungranted, and agents must still create files in `/tmp`. The launcher sets `PWTEST_CACHE_DIR` to a hidden child of that root and removes the default cache. The hidden list covers the four private children of the root, including children that are created after runnerd starts. The probe also checks that the default Playwright cache is not listable. - `tests/runner-e2e/launch.ts`: passes the selection to the server process. The selection applies only to Linux local native cells. It does not apply to Codex cells (Codex's bubblewrap sandbox cannot mount under Landlock), installed-release launches, legacy adapters or Daytona. - `tests/runner-e2e/server-entry.ts`: installs the sandbox before the product starts. The product server itself stays unrestricted because it loads this entry from the harness. - `tests/runner-e2e/fixture-names.ts` (new), `catalog.ts`, `live-fixtures.ts`: fixture company, agent, environment, secret, AI connection and project names and descriptions use a neutral word and an 8-character digest suffix. They do not contain case ids or "Runner E2E". No grader or harness lookup reads these names. - `tests/runner-e2e/agent-filesystem-sandbox.test.ts` (new): tests the selection, the launch rewrite and the server-entry order. On Linux with Landlock, it also tests that a sandboxed `grep` over a fake `tests/` dir is denied while the product file stays readable. - `tests/runner-e2e/SECURITY.md`: documents the boundary and its limits. - The trusted files `manual-private-handoff.ts`, `manual-private-retention.ts` and `manual-private-report-assets.ts` are byte-identical to master. ## Verification - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/agent-filesystem-sandbox.test.ts tests/runner-e2e/live-fixtures.test.ts tests/runner-e2e/catalog.test.ts tests/runner-e2e/codex-ci-sandbox.test.ts tests/runner-e2e/server-entry.test.ts`: 151 passed, 1 skipped. The Landlock test is skipped on macOS. `npx tsc --noEmit -p tests/runner-e2e` shows no errors in the changed files. - Paid run [42480314919](https://github.com/paperclipai/paperclip/actions/runs/42480314919) for `chat-features.runner-opencode.local.agents.change-manager` ran on branch `chat-eval/eval-isolation-verify`, which is this commit cherry-picked onto the chat catalog candidate. The cell passed. Its `server.log` records: - `probe … "harnessFile":"denied","harnessDirectory":"denied","productFile":"readable","temporaryWrite":"allowed"`, and `denied` for each of `…/paperclip/tests`, `…/.manual-private-trusted`, `…/_temp/manual-private-evidence-*/capture`, `…/_temp/manual-private-evidence-*` and `…/_temp`. `"failures":[]`. - `runnerd launched inside the sandbox (pid 3548)`. - The fixture company was `Company cc369a2e`. No captured snapshot contains "Runner E2E". - The probe is a regression check on every local native cell: a missing or ineffective sandbox stops the attempt in CI before an agent starts. ## Risks - Landlock does not hide that a path exists. `stat` of `tests/runner-e2e` still succeeds (the probe logs `harnessDirectoryStat: visible`). Reading and listing are denied. - Directories above a hidden path (for example `/`, `/home/runner` and the checkout root) cannot be listed. New top-level entries in those directories are not accessible after launch. Agent state, `HOME` and temporary files are in `/tmp`, which stays fully granted. - Not covered: Codex cells, legacy adapters, installed-release launches and macOS. An agent that can reach the Docker daemon can still mount the checkout into a container. `ps` still shows the launcher's command line, which includes the case id. The temporary root (`paperclip-runner-e2e-*`) and instance id (`runner-e2e-*`) in the agent's paths still say "e2e". The chat catalog candidate depends on those prefixes, so this PR does not rename them. - Python 3 must be present on Linux runners. Ubuntu runners have it, and the harness already uses it. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), agentic coding with tool use (shell, file edits, GitHub CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
051834bdc8 |
feat(chaos): add the on-demand crash-point suite and its manual CI workflow (#19946)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A control-plane restart between two durable writes leaves an in-progress record owned by a dead process > - Random restart-chaos waves found these 20 ms–1 s windows only by luck > - #19937 added `crashPoint(name)`, and #19938 put it at 20 durable write boundaries > - Nothing yet runs a task, kills the server at a chosen point, restarts it, and checks the plan's invariants > - This pull request adds that on-demand suite, as a local command and as a manual CI workflow (not a required check, per Decision 2) > - The benefit is a repeatable work list: which boundary breaks which invariant, for Phases 2–4 ## Linked Issues or Issue Description Refs #19831 (plan, Phase 5 and Decision 2). Refs #19861 (report-only reconciler used for I1–I7). Stacked on #19938 (call sites), which is stacked on #19937 (helper). **Problem** The named crash points cannot be exercised end to end. No command arms one point, runs real agent tasks across it, restarts, and grades the result. **Expected behavior** `node scripts/crash-points.mjs --points <glob> --kinds <id> --adapters <list>` runs one wave per selected point and reports each point as pass, fail (with failing R rules and violated invariants), not reached, or harness error. A `workflow_dispatch` workflow runs the same thing on demand. ## What Changed - **Runtime arming.** `PAPERCLIP_CRASH_POINT_CONTROL=<file>`, in the shared helper and the runner mirror. Each hit reads `<token> <name>[:<nth>]`, and a new token resets the count. It is test mode only: `NODE_ENV=production` ignores it, and the disarmed path is still one comparison. - **`tests/restart-chaos/main.ts --crash-point <a>:<nth>,<b>:<nth>,...`** (config, types, ledger): - one booted server and no Vite UI; - per point: take an invariant baseline, launch the point's batch, arm the point, then on rest disarm it and check invariants as violations that are new since the baseline; - every task is graded R1–R6 and T; - `crash-point.json` (v2) holds one entry per point. - **`tests/restart-chaos/supervisor.ts`:** - every spawn gets the control file; - a SIGKILL whose receipt names the child's pid is restarted like K1: the control file is emptied first, the owned tree is killed, and a replacement starts. A failed tree cleanup fails the fault; - the isolated instance pins `PAPERCLIP_IN_WORKTREE=false` (a `.paperclip/.env` above the checkout otherwise suppressed all scheduling); - `RESTART_CHAOS_HEALTH_TIMEOUT_MS`. - **`tests/restart-chaos/invariants.ts`:** - the Phase 1 report-only pass with `record: false`: two real-clock passes for I1 and I2, and one pass 15 minutes ahead for I3, I4 and I7; - spool kinds are read from the wave's instance; - an unfinished pass is a harness error; - each violation carries its wake and run state as evidence. - **`tests/restart-chaos/crash-points.ts` and `scripts/crash-points.mjs`:** - select points by glob, kind and environment, and run them in one wave; - a per-point outcome: pass, fail, not_reached or harness_error; - `SUMMARY.md` with a violation evidence column. - **`.github/workflows/crash-points.yml`**, manual only, under the paid-workflow contract: - a credential-free `authorize` job; - a `suite` job in `runner-e2e-paid` whose first step reauthorizes; - a frozen install with `--ignore-scripts` and no shared cache; - only the selected adapters' keys; - the workflow is added to both security-test allowlists. - **Docs:** "Crash-point suite" in `tests/restart-chaos/README.md`, and "Built (Phase 5)" plus the first work list in the plan. ## Verification - Focused tests: - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts tests/restart-chaos/crash-points.test.ts`; - `cd packages/shared && npx vitest run src/crash-points.test.ts`: 15 tests, including runtime arming in a real child process; - `node --test scripts/__tests__/release-verify-workflow.test.mjs`: 72 tests; - `tests/runner-e2e/workflow-security.test.ts`: 19 tests. - Typecheck: `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - Loop line: question "how long is one crash-point loop?" | loop: `node scripts/crash-points.mjs --points <one>` (codex A1) | elapsed 2.7 min end to end: boot 9 s, crash 9 s after arming, replacement ready in 6 s, rest about 2 min | result: one point alone takes about 3 min. Before this change it took about 15 min (one cold boot per point, Vite UI, host at load 77). - Full local run, 14 points, codex A1, 11.8 min in all: - every point was reached (the server killed itself at each one, and the replacement came back); - 11 pass; - `workspace.finalize.after_start_before_copyback` and `workspace.finalize.after_copyback_before_done` fail I2 and R6: a `workspace_finalize` operation stays running after the run succeeded; - `wake.claim.after_run_running_before_wakeup_claimed` reports I4. Its evidence shows the wake failed with "…retrying once", and the retry delivered the work, so this is the known refusal-regex false positive. - The 6 Daytona-only points were not run. ## Risks - Test harness and an on-demand workflow only. No product code changes. - `.github/**` needs a CODEOWNER review. The workflow uses provider keys, so it refuses non-default branches and actors that are not on the allowlist, like `runner-full-stack-e2e.yml`. - `PAPERCLIP_IN_WORKTREE=false` is pinned in every chaos wave, not only crash-point waves. A chaos instance is never a worktree dev instance, so scheduling must not be suppressed. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
17b9e10252 |
docs(plans): Phase 2 reconciler data; let chaos K4 holds reach 600 s (#19927)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Control-plane restarts leave in-progress records that each subsystem releases in its own way. The durable-ownership plan replaces those paths with one shared path. > - Phase 1 added a report-only reconciler. Its light local waves showed zero violations, and that proves little. > - Phase 2 moves kinds to the shared path. It needs data that shows which kinds really leak, and the speed numbers to protect. > - This pull request records that data from harsher waves. It also fixes the harness so that K4 outages past the reconnect grace qualify. > - The benefit is that Phase 2 moves kinds in order of measured risk, against a known speed baseline. ## Linked Issues or Issue Description Refs #19831 (durable-ownership plan), #19861 (report-only reconciler), #19863 (speed budgets), #19797 (K4 hold cap). **What is the problem?** Phase 2 needs per-kind invariant data and speed data from harsh restart waves. Only light local waves existed. In addition, a K4 hold over about 405 s always failed its fault receipt, so the 600 s cap from #19797 could not be used. **Where is it documented?** `doc/plans/2026-11-07-durable-ownership-and-reconciliation.md` (Phase 2 rows) and `tests/restart-chaos/README.md`. **What should change?** Record the data in a dated plan doc. Let chaos K4 holds run up to 600 s and still qualify. ## What Changed - Added `doc/plans/2026-11-07-phase2-reconciler-data.md`. It covers 5 unmasked local waves (two storms, a crash loop, two K4 runs). It gives per-kind violation counts with example rows, owner verdicts, TTFT, boot and restart recovery against the budgets, Phase 3 gate notes, and what is not covered (Daytona). - Added a short pointer to that doc in the durable-ownership plan. - `tests/restart-chaos`: new flag `--k4-hold-max-seconds` (120–600). It raises the upper end of the seeded K4 hold. The default range (120–300 s) does not change. - `tests/runner-e2e/controller-outage.ts`: the receipt's ready bound is now a parameter (`CONTROLLER_READY_BY_MS`, default 420 s). Runner suites keep 420 s. The chaos supervisor passes hold + 420 s, so a hold past the Runner's reconnect grace qualifies. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 12 files, 135 tests pass, including the new flag test. - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/controller-outage.test.ts`: 36 pass, including the new long-hold receipt test. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` and `tests/runner-e2e`: pass. - Live: the K4 wave at seed 9203 failed 3 of 6 tasks with the old bound (fault error "did not produce a qualifying replacement receipt"). It passed 6 of 6 with this fix, with holds of 294–579 s. ## Risks - Low risk. The changes are test harness code and docs only. The runner suites keep the 420 s ready bound by default. - The data is local only. The doc says which kinds the waves do not reach. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits). The model ran the chaos waves and wrote the analysis and code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
d157ce223c |
fix(evals): keep large failed private evidence and the merged report within the private checks (#19932)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat evaluation lane runs each chat case in a paid runner and
keeps the private evidence of each attempt in a bounded private archive
> - A failed attempt keeps its Playwright trace and video, and the
bundle has a 256 MB allowance
> - Playwright's HTML reporter copies each attachment into
`html-report/data/<sha1>.<ext>`, and the same bytes also stay in
`playwright-output/`
> - The bundle holds the original tree and the packaged tree, so a 30 MB
trace used about 125 MB of the allowance before the blob report copies
> - Large failed attempts went over the allowance, and then all of their
private evidence was lost
> - This pull request removes the `playwright-output/` copy only when
the HTML report holds the identical bytes
> - A second gap lost the whole merged campaign report of failed cells:
the report copied trace viewer font, SVG and manifest files to a place
where the trusted seal refuses them
> - This pull request stops that copy, and it names a refused report
carrier with a closed category
> - The benefit is that large failed attempts keep their evidence, the
merged report is kept, and the HTML report and its trace viewer still
open the trace
## Linked Issues or Issue Description
**What happened**
Failed chat evaluation attempts with large Playwright traces lost all of
their private evidence. The bundle went over the 256 MB private evidence
allowance. For example, the failed `tasks.organize-open-work` cell made
a 254 MB bundle. That bundle held the same 30.7 MB trace four times
(`playwright-output/.../trace.zip` and `html-report/data/<sha1>.zip`, in
both the original tree and the packaged tree), plus two blob report
copies. Larger traces went over the limit, and then the transfer refused
the whole bundle.
Also, the merged campaign report of a failed cell was refused with
`refusal: unsupported-packaged-evidence-carrier` (for example
`runs-adapters.inspect-failed-run`, run 42431162923). The cell's
evidence manifest lists its trace viewer files
`html-report/trace/codicon.*.ttf`, `playwright-logo.svg` and
`manifest.webmanifest`. `report.ts` copied them to
`normalized/evidence/<id>/attempt-1/html-report/trace/`. The
generated-report seal accepts trace viewer files only under
`html/trace/`, so it refused these three files and lost the whole
report. I reproduced this from the retained cell archive: a merge of its
blob report passes the seal rule, and the three normalized copies are
the only files that fail it.
**Expected behavior**
A failed attempt keeps its trace, and the trace uses as little of the
allowance as possible. The HTML report and the trace viewer still open
the trace for a person who downloads the bundle. The merged report of a
failed cell is kept. If the seal refuses a carrier, the refusal names
its category, not its path. No privacy, integrity or size check changes.
**Steps to reproduce**
1. Dispatch `runner-full-stack-e2e.yml` with `cadence=manual` for a
private chat case that fails with a large trace, for example
`chat-features.runner-opencode.local.tasks.organize-open-work`.
2. Download `manual-cell-<sha256(executionId)>.zip` from the private
history bucket.
3. List the archive. The trace bytes are in `playwright-output/` and in
`html-report/data/` in both `originals/` and `sealed-results/`.
4. For the report refusal: look at the "Retain complete manual merged
evidence only in restricted storage" step of the same run. It prints
`refusal: unsupported-packaged-evidence-carrier` and no
`manual-report.zip` is retained.
## What Changed
- Add `tests/runner-e2e/private-report-copies.ts` with
`dropDuplicatePlaywrightOutput`. It removes a `playwright-output/` file
only when `html-report/data/<sha1 of content><ext>` is a regular file
(not a symlink) with the same size and the same SHA-256. It writes the
removed paths, their kept paths and digests to
`snapshots/playwright-output-copies.json`.
- The function does nothing when the cell has a framework JSON report
(`framework-report.json` or `snapshots/playwright-framework.json`),
because that report registers `playwright-output/` attachments by path.
It also does nothing when `playwright-output`, `html-report` or
`html-report/data` is missing or is a symlink.
- Call it in the private manual branch of `tests/runner-e2e/launch.ts`,
before packaging and before both seals. Both evidence trees then keep
one HTML report copy of each trace and video.
- The HTML report, its trace viewer assets and the blob report do not
change.
- `tests/runner-e2e/report.ts` does not stage a cell file into
`normalized/evidence/` when its type is one the generated-report seal
refuses (the same `isStandardEvidenceFile("html-report/...")` rule). The
merged report has its own authenticated trace viewer under
`html/trace/`. The cell archive still keeps the files.
- `tests/runner-e2e/manual-private-transfer.ts` adds
`assertGeneratedReportCarriers`, which runs before the report seal. It
throws a fixed message per closed category (`font`, `svg`,
`webmanifest`, `image`, `source map`, `no extension`, `other`, `trace
viewer asset`, `trace viewer bytes`), so the refusal line names, for
example, `unsupported-generated-report-carrier-font`. It never names a
path, and the seal stays the authority.
- No change to the target-checked trusted files
(`manual-private-handoff.ts`, `manual-private-retention.ts`,
`manual-private-report-assets.ts`), so candidate branches do not need an
update. No change to the limits or the secret scans.
## Verification
- `vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/private-report-copies.test.ts
tests/runner-e2e/manual-private-cell-report-assets.test.ts
tests/runner-e2e/private-framework-report.test.ts
tests/runner-e2e/manual-private-handoff.test.ts`: 79 passed.
- The new tests use a real failed Playwright run with the HTML, JUnit
and blob reporters and a real retained trace. They show these things:
- Only the duplicate `playwright-output/` files are removed. The HTML
report and blob report bytes do not change.
- The deduplicated cell passes `packageEvidence`,
`sealStandardOriginalEvidence`, `sealManualResultEvidence`, the trusted
`validateSealedOriginals` and `validateSealedInventory`, the strict
archive pack and unpack, and `stageSealedPrivateCell`.
- A tampered or symlinked HTML report copy keeps the original file.
- Framework JSON cells and symlinked data directories are not changed.
- `vitest run --config tests/runner-e2e/vitest.config.ts manual-private
private-report private-refusal private-framework generated-report
report`: 25 files, 311 passed.
- `generated-report-carriers.test.ts` shows each refused carrier gets
its closed category code, the code has no path, and the real seal
refuses the same tree. `report.test.ts` shows the font, SVG and manifest
are not staged and the other files are.
- A new test runs the real launcher private branch from `launch.ts`. It
shows that the sealed originals keep one trace copy, in
`html-report/data/`.
- Live run: I applied the commit to a branch from
`chat-eval/master-candidate` (`chat-eval/case-evidence-dedupe`) and
dispatched
`chat-features.runner-opencode.local.tasks.organize-open-work` (run
42432263920). I will add the bundle size result in a comment.
## Risks
- Low risk. The change runs only in the private manual evidence branch.
It removes a file only after a full content comparison with the kept
copy. If a check fails, the file stays.
- A person who wants the trace must open `html-report/data/<sha1>.zip`
(the HTML report links to it), not `playwright-output/<test>/trace.zip`.
The receipt in `snapshots/playwright-output-copies.json` names the new
location.
- `report.ts` and `manual-private-transfer.ts` run from the master
checkout in the report job, so the report fix takes effect only after
merge. The normalized per-cell HTML report copy no longer has the viewer
font, logo and manifest. The merged report under `html/` and the cell
archive keep a complete viewer.
- The blob report still has a compressed copy of the trace, and the
bundle still holds the original and the packaged tree. A trace over
about 60 MB can still go over the allowance. This change does not fix
that case.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), run in Claude Code with
tool use (shell, file editing, GitHub CLI and AWS CLI).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
1de794cb80 |
fix(evals): name runtime-built evidence refusals by their fixed template (#19920)
## Thinking Path - #19908 made a refused private evidence transfer print a closed refusal code instead of one generic line. - Its first real use printed `refusal: dynamic`: the trusted modules build some refusals at runtime (secret-leak findings, publication-boundary limits), and those were deliberately not named because their text can carry paths or counts. - The fixed part of each template is safe and is exactly what an operator needs to know which check refused. ## Linked Issues or Issue Description **What happened** After #19908, a refused `runs-adapters.inspect-failed-run` attempt (workflow run 42422571919) printed `Restricted manual evidence operation failed (refusal: dynamic)`, so the reason was still unknown. **Expected behavior** A refused transfer still keeps message text private, but names which check refused it with a closed code, so the attempt can be diagnosed. **Steps to reproduce** 1. Dispatch `runner-full-stack-e2e.yml` manually for a chat cell whose private evidence fails transfer validation. 2. Open the "Retain complete approved manual originals" step log: it shows the single fixed failure line with no reason. ## What Changed - `private-refusal-code.ts`: runtime-built refusals from the trusted modules map to fixed template codes (`template-secret-leak`, `template-publication-boundary-exceeded`, `template-invalid-file-metadata`, `template-history-expiration-unprovable`). Variable parts are never printed. - Tests: each template is named without its variable text, and a source scan asserts every runtime-built `new Error` in the trusted modules maps to a named code (not `dynamic`). ## Verification - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/private-refusal-code.test.ts tests/runner-e2e/manual-private-transfer.test.ts`: 20 passed. - `npx tsc --noEmit -p tests/runner-e2e`: clean. ## Risks - Template codes reveal which kind of check refused (for example that a secret-shaped value was found), not where or what. That is the intended operator signal; the finding itself stays private. ## Model Used Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code, with tool use (shell, file editing). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
ba3fe09954 |
fix(evals): name why a private evidence transfer refused, without its text (#19908)
## Thinking Path
- Paid chat evals retain their evidence through a trusted transfer
(`tests/runner-e2e/manual-private-transfer.ts`) that runs from master,
separate from the target checkout.
- When the transfer refuses an attempt, it prints one fixed line,
"Restricted manual evidence operation failed; no public fallback", and
keeps nothing. The closed diagnostic on the target side cannot see this
step either.
- So a refused attempt leaves no evidence and no reason.
`runs-adapters.inspect-failed-run` lost three attempts in a row this
way, and `goals.create-goal-tree` lost one; nobody could tell which
check refused.
- This change keeps the privacy rule (no message text in public logs)
but names the refusal with a closed code.
## Linked Issues or Issue Description
**What happened**
A paid chat-eval attempt whose private evidence the trusted transfer
refused printed only "Restricted manual evidence operation failed; no
public fallback". The receipt showed `attemptedSourceBytes: 0` and
`transfer.status: failed_or_unknown`, and no evidence was kept, so the
reason was unknowable. `runs-adapters.inspect-failed-run` lost three
consecutive attempts this way.
**Expected behavior**
A refused transfer still keeps message text private, but names which
check refused it with a closed code, so the attempt can be diagnosed.
**Steps to reproduce**
1. Dispatch `runner-full-stack-e2e.yml` manually for a chat cell whose
private evidence fails transfer validation.
2. Open the "Retain complete approved manual originals" step log: it
shows the single fixed failure line with no reason.
## What Changed
- New `tests/runner-e2e/private-refusal-code.ts`:
`privateRefusalCode(error)` returns a slug of the message only when it
is a fixed literal `new Error("...")` in the trusted evidence modules
(transfer, handoff, retention, report assets, history, framework report,
history budget, evidence, redaction). Otherwise it returns `fs-<code>`
for Node error codes, `error-type-<Name>` for typed errors,
`non-error-thrown`, or `dynamic`. Message text is never returned.
- `manual-private-transfer.ts`: the failure line now includes `(refusal:
<code>)`.
- Tests: fixed literals are named; dynamic, filesystem, typed and
non-error throws never leak their text.
## Verification
- `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/private-refusal-code.test.ts
tests/runner-e2e/manual-private-transfer.test.ts`: 15 passed.
- `npx tsc --noEmit -p tests/runner-e2e`: clean.
- The three files the transfer requires to match between trusted and
target checkouts (`manual-private-handoff.ts`,
`manual-private-retention.ts`, `manual-private-report-assets.ts`) are
unchanged.
## Risks
- The public log line gains a code. Codes are either fixed literal
messages (already public in this repository's source), Node error codes,
or error type names. Dynamic messages, which can contain paths, are
reported only as `dynamic`.
- Reading the module sources once at failure time adds a small file read
on the error path only.
## Model Used
Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code, with
tool use (shell, file editing).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
9cc3760552 |
test(runner-e2e): allow controller outages up to ten minutes (#19797)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness and the runner-e2e recover-controller cells stop the control plane with `replaceController` (`tests/runner-e2e/controller-outage.ts`), hold it down, then start it again > - `parseControllerOutage` refuses holds longer than 300000 ms > - Restart chaos needs longer holds. One example is proving that a Daytona sandbox survives an outage longer than its old 5-minute auto-stop but inside the 7-minute runner reconnect grace (RC-54) > - With the cap, such a fault errors before the controller stops, so the scenario cannot run > - This pull request raises the test-harness cap to 600000 ms > - The benefit is that chaos runs can now inject 5–10 minute outages; production code and qualification contracts are unchanged ## Linked Issues or Issue Description **Bug description** A restart chaos replay with a 360000 ms K4 hold failed with "Controller outage requires SIGTERM/SIGKILL and 0..300000 integer milliseconds". The controller was never stopped. **Expected behavior** The test harness accepts outages of up to 10 minutes. The recover-controller receipt check keeps its own readiness limit (`readyAfterMs < 420000`). Refs #19544 ## What Changed - `tests/runner-e2e/controller-outage.ts`: adds `MAX_CONTROLLER_OUTAGE_MS = 600_000` and uses it in `parseControllerOutage` and its error message. `validControllerOutageReceipt` is unchanged and still requires readiness inside the 7-minute grace. - `tests/runner-e2e/controller-outage.test.ts`: the rejection cases now use 600001. A new test accepts 360000 (SIGKILL) and 600000 (SIGTERM). The existing five-minute contract test is unchanged. ## Verification - `vitest run tests/runner-e2e/controller-outage.test.ts tests/runner-e2e/server-restart-request.test.ts`: 52/52 pass. - `server-restart-request.test.ts` now uses 600001 as its out-of-range `controllerOutage`. The separate restart-request `controllerDowntimeMs` limit (five minutes) is unchanged. ## Risks - This change is test-only. No product or server code changes. - A suite that sets `PAPERCLIP_RUNNER_E2E_CONTROLLER_OUTAGE_MS` above 300000 now runs instead of being refused. The recover-controller receipt check still fails any receipt whose readiness is outside the grace. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits, GitHub CLI). Default context window, extended thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
github-actions[bot]andlockfile-bot
|
e95dcadbd2 |
chore(lockfile): refresh pnpm lockfiles (#19773)
## Thinking Path > - Paperclip uses a dependency lock for reproducible builds. > - Dependency changes on master need matching lock entries. > - Paid eval reports also pin a reviewed offline correction. > - Reports must install before paid model work starts. > - This automated PR keeps the lock and report proof together. ## Linked Issues or Issue Description **What happened?** Dependency manifests on master changed after the last committed lock refresh. **Expected behavior** Frozen installs must match those manifests. Eval reports must retain matching offline lock proof. **Steps to reproduce** Run a frozen install before and after this lock refresh. ## What Changed - Refresh the dependency lock from the manifests on master. - Refresh the sandbox provider lockfiles from their manifests on master. - Include the matching offline report correction and hashes. ## Verification - Pinned pnpm resolution completed with lifecycle scripts disabled. - The report proof refresh accepted the reviewed committed baseline. - The workflow rejected changes outside the lock and report proof files. ## Risks The lockfile bot merges this PR without CI or review when every added package version is at least 3 days old on npm. Otherwise the PR waits for a maintainer, and the daily refresh tries again. The bot merges only the exact commit this workflow generated from manifests already reviewed on master. Unknown report lock baselines remain blocked. This update changes no permissions or paid enable gates. ## Model Used None — generated by pnpm and the deterministic report-lock updater. ## Checklist - [x] Thinking path and problem description included - [x] Generated lock and matching report proof included - [x] Release-age gate decides between bot merge and maintainer review Co-authored-by: lockfile-bot <lockfile-bot@users.noreply.github.com> |
||
|
DottaandClaude Opus 5.5
|
efc0b359da |
fix(ci): rebuild and sign an unsigned reused Daytona e2e image (#19677)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Paid runner E2E campaigns run in `runner-full-stack-e2e.yml`.
Daytona cells need a signed, content-addressed runner image
> - The "Publish verified Daytona image" job reuses `e2e-content-<id>`
when the tag exists, and signs only after a fresh build
> - A run that stops between the push and `cosign sign` leaves an
unsigned tag. Every later run reuses it and fails `cosign verify` with
"no signatures found"
> - That failure skips the whole test matrix, including local cells. The
first Reliability TestOps launch (run 42298071794) ran 0/7 cells for
this reason
> - This pull request reuses a tag only when its signature verifies.
Otherwise it rebuilds, pushes and signs the tag
> - The benefit is that an interrupted run cannot block every later
Daytona campaign
## Linked Issues or Issue Description
Refs #19549
**What happened?**
Run https://github.com/paperclipai/paperclip/actions/runs/42298071794
(TestOps launch of the Reliability plan) failed in "Reuse or publish
immutable image" with `Error: no signatures found` (exit code 10). No
test cell ran. The report shows 7 missing attempts.
**Expected behavior**
A campaign with Daytona cells always produces a signed, verified image,
or it fails for a real build or signing error. A stale unsigned tag must
not block it.
**Steps to reproduce**
1. A run pushes
`ghcr.io/paperclipai/paperclip-daytona-runner:e2e-content-<id>` and
stops before `cosign sign` (for example, a runner shutdown).
2. A later campaign with the same content ID finds the tag and skips the
build.
3. `cosign verify` fails with "no signatures found", and the matrix is
skipped.
**Paperclip version or commit**
master
|
||
|
DottaandClaude Opus 5.5
|
3fedfe8763 |
fix(runner-e2e): wait out a briefly uncertain owned group instead of failing the stop (#19692)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness and the runner e2e suites stop and restart the control plane through one process-tree owner (`tests/runner-e2e/process-tree-owner.ts`) > - The owner signals only process groups whose identity it has proven, and refuses everything when a group's identity becomes uncertain > - Under load a group can turn over between two `ps` snapshots: the observed members exit, and a straggler that nobody saw start is left in the group, reparented away from every owned anchor > - The stopper treated that as fatal on the first inspection, so a K2 graceful stop at load 28 failed in under a second, and a chaos storm wave was marked as a harness error although the server came back > - This pull request makes the stopper re-inspect within its existing deadline until the uncertainty settles. It still never signals the uncertain group, and it fails with the original error if the uncertainty persists > - The benefit is that chaos waves under load stop failing on a harness race, without weakening the rule against signaling unproven processes ## Linked Issues or Issue Description Refs #19470 **What is the problem?** `stopOwnedProcessTree` fails at once with "Owned process group identity became uncertain" when, under load, a group's observed members exit and an unobserved straggler remains for a moment. In a chaos storm this failed a K2 stop 0.9 s after SIGTERM (load 28), and the wave exited as a harness error. **What is the expected behavior?** The stopper waits (within the same grace or forced deadline) for a later inspection that settles the group, and it never signals the uncertain group. If the straggler outlives the deadline, the stop still fails with the same error. **Why does it matter?** A harness-only race must not invalidate a chaos wave, and the fix must not let the harness signal a process whose ownership is not proven. ## What Changed - `tests/runner-e2e/server-stop.ts`: the chaos and e2e server stopper (the actual K2 path) uses the same bounded settle-wait for its graceful observe/signal and its SIGKILL escalation. Live evidence: a K2 at load 13.9 failed in 0.5 s on an ACPX Claude provider group (launcher, reaper, module-hook loader, `claude` runtime) that tore itself down during the drain and was gone 0.4 s later. - `tests/runner-e2e/process-tree-owner.ts`: the uncertainty error now names the unowned members (pid, ppid, pgid, kind, start). `settleUncertainIdentity` is exported and shared. Export the `UNCERTAIN_GROUP_IDENTITY` message prefix. `stopOwnedProcessTree` retries the grace signal, the SIGKILL escalation and the stop check every 100 ms while that error persists, bounded by the existing grace deadline and one forced window (set before escalation and reused by the escalation retry, the stop check and the fallback), so a stop never exceeds gracefulMs + forcedMs. Other errors, and an uncertainty that outlives the deadline, take the existing fallback path unchanged. The owner's identity checks do not change, and neither does `signal()`. - `tests/runner-e2e/process-tree-owner.test.ts`: three tests. An uncertainty that starts only at SIGKILL escalation fails within one forced window and never sends SIGKILL. A brief turnover settles and the stop succeeds without signaling the uncertain group or killing the direct child. A persistent turnover still rejects with "identity became uncertain" and never signals. ## Verification - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/process-tree-owner.test.ts tests/runner-e2e/server-stop.test.ts`: 35 pass across `process-tree-owner.test.ts` and `server-stop.test.ts`, including 5 new tests; the new brief-turnover tests fail on the old stoppers. With the old stopper, the first new test fails: it rejects on the first inspection and then kills the direct child. - `node scripts/typecheck-test-projects.mjs tests/runner-e2e`: passes. ## Risks - Low risk, test infrastructure only. - A stop that hits a persistent uncertainty now fails at the deadline (up to the grace time) instead of at once. It sends no extra signals. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Devin FoleyandPaperclip
|
1aa0c36397 |
Fix execution after onboarding plan approval (#19698)
## Thinking Path > - Paperclip manages agents and their work. > - First-task onboarding turns a user's goal into an approved proposal. > - The interview offers a plan and a team to carry out that plan. > - The instructions leave execution after acceptance unclear. > - An agent can add a plan-only approval boundary and finish without doing the requested work. > - This change makes the approval scope explicit and tests both execution and plan-only outcomes. ## Linked Issues or Issue Description **What happened?** An onboarding agent can treat an interview about building a product as a request for a plan only. It can then retain the approved plan and close onboarding without creating execution tasks. The interview evaluation previously accepted that result. **Expected behavior** Acceptance of an execution proposal starts its selected work. An explicit plan-only request stops after planning. User-requested separate approvals and existing permission and budget controls still apply. **Steps to reproduce** 1. Choose the onboarding interview and describe a deliverable. 2. Accept the saved plan proposal. 3. Check that the agent starts the approved work instead of closing with only a plan. ## What Changed - Clarify execution intent, approval wording, continuation, and completion in the first-task prompt. - Require assigned child work and saved output in the interview acceptance evaluation. Use checkbox acceptance in the regular interview journey and retain conversational acceptance in the approval-reply suite. - Add an explicit plan-only acceptance journey and nine grader regression checks. - Update evaluation counts, grading versions, and onboarding verification guidance. ## Verification - Passed: all applicable CI checks on `198a6b47cac7`, including typechecks, the general and serialized server tests, Runner checks, build, and browser E2E shards. [CI run](https://github.com/paperclipai/paperclip/actions/runs/42324855376). - Passed locally: the full Runner support suite, 6,860 tests across 323 files (three skipped), plus 13 onboarding asset tests. - Passed locally: the full workspace build, workspace package typechecks (server rechecked after restoring excluded-plugin SDK links), and the Runner evaluation TypeScript check. - Passed: map validation and whitespace checks. - Initial broad local checks failed during checkout preparation because PostgreSQL library symlinks and runtime/SDK assets were missing. Restoring the dependency setup made the affected controls pass. The full root test suite passed in CI; it was not rerun locally after that repair. - Live model journeys have not been run. Grader tests do not prove model compliance. ## Risks - Prompt behavior needs live qualification across providers. This change adds no server-side enforcement or permission bypass. - The new plan-only case adds four local evaluation cells. A full campaign now has 56 first-task cells. - Existing proposals that reserve later approval keep that restriction. This change does not rewrite tenant data or start existing onboarding tasks. - No schema migration or deployment is included. ## Model Used - OpenAI `gpt-6.1-sol`, xhigh reasoning effort, with repository tools and local test execution. Context window size is not exposed by this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (focused and full Runner support locally; full root coverage passed in CI) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
fb1b458409 |
fix(runner-e2e): drop the Rust dependency layer that filled the image builder disk (#19669)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Runner Full-Stack E2E runs Daytona cells from a signed, content-addressed runner image > - #19545 made the image build faster. One part of it added a Rust dependency layer that keeps Cargo's target and registry trees in the builder > - The first default-branch build after the merge ran out of builder disk while it exported the registry cache > - That run pushed the image tag before the failure, so the tag has no signature, and campaigns that reuse it fail `cosign verify` > - This pull request removes only the dependency layer and keeps the parts of #19545 that add no disk > - The benefit is a working default-branch image build again, a new content ID that replaces the unsigned tag, and the measured 16.5 to 11.6 minute improvement ## Linked Issues or Issue Description Refs #19545. Related #18494 (the same builder disk limit). **What happened?** The default-branch `image_only` dispatch, run 42286903290 (job 128202722275), built and pushed `ghcr.io/paperclipai/paperclip-daytona-runner:e2e-content-7154c0dfc9abb297d8d638a982326c1435dc4440c25112afacc559712d4430b3` (`sha256:2767e91c5279...`). The registry cache export then failed with `ResourceExhausted: ... no space left on device` (exit 102). `preparing build cache for export` took 182 s, against 87 s before #19545. The job stopped before `cosign sign`. The dependency layer retained `target` 852 MB and Cargo registry 333 MB (measured in run 42271236140). **Expected behavior** Default-branch image builds publish the image, write the cache, and sign the digest within the builder disk. **Steps to reproduce** Dispatch `runner-full-stack-e2e.yml` from master with `-f image_only=true` and a Daytona selector on `840838476a`. Read the `Reuse or publish immutable image` step log. ## What Changed - `docker/daytona-runner/Dockerfile`: remove the placeholder-target dependency layer. The runnerd stage is again the single build layer from before #19545, so its cache key matches the earlier default-branch cache. Restore and extend the disk comment. - Keep the content-ID check above the `COPY --from` steps and keep `COPY --link`. They add no retained data. - `tests/runner-e2e/README.md`: describe the two remaining layout rules and why there is no dependency layer. - The Dockerfile is a content-ID input, so the next build gets a new tag. No campaign reuses the unsigned tag. ## Verification - `vitest run tests/runner-e2e/daytona-image.test.ts tests/runner-e2e/workflow-image-only.test.ts`: 20/20 pass. - Branch evidence for the kept parts: job 128077687214 took 11.6 min, against 16.5 min before. The cached-layer download overlapped the compile. - After merge, an `image_only` dispatch from master must publish, write the cache and sign. I did not use a dispatch before merge, because only a default-branch build writes the cache, and the cache export is what failed. ## Risks - Runner source changes again recompile the full Rust dependency graph (about 5 min). To get that saving back, the builder needs more free disk first (for example, remove unused preinstalled toolchains in the image job, or use a larger RunsOn volume). That is a separate change. - The unsigned tag stays in GHCR. Nothing selects it after this change, because the content ID changes. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, GitHub CLI). Extended reasoning on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Devin FoleyandPaperclip
|
8621bcf53e |
Cancel subscription sign-in requests when leaving agent setup (#19610)
Back during pending subscription sign-in could accept a late response and
reopen agent setup. Abort pending reads, release abandoned sessions, and
ignore stale responses. Wait for cleanup before starting a replacement,
and preserve sessions belonging to another account or environment.
Cover deferred resume/start/prompt responses for both providers, local
login cancellation, quick reselection, and browser request abort. Stabilize
Runner fixtures and normalization assertions exposed by integration CI.
Verified: 483 focused UI tests, six onboarding browser cases, UI typecheck
and token gates, Runner fixture checks, and the complete CI run on
|
||
|
|
e482f0f632 |
fix(server): shed only retryable object-journal reads under load (#19176)
Client polling can exhaust the shared object journal queue and reject log persistence. Bound ordinary object-log and trace reads, keep persistence and internal verification queued, and return retryable 503 responses when reads are busy. Leave feedback exports unsent until every log page is available.
Validation: 72 latest-master affected tests and 10 workflow tests pass locally, with the workflow fixture passing strict TypeScript checks. The preceding broader run passed 280 server tests with two opt-in S3 skips. All applicable checks on
|
||
|
DottaandClaude Opus 5.5
|
02a2299173 |
feat(codex): default Codex to gpt-6.1-sol, half the gpt-5.6-sol price (#19623)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Codex agents run through Paperclip Runner or the legacy
`codex_local` harness, and the runner E2E suites run paid Codex cells
> - The shared Codex default and many selections preferred `gpt-5.6-sol`
> - The reviewed OpenAI rate card in
`server/src/services/codex-pricing.ts` (`openai-standard-2026-09-30`)
prices `gpt-6.1-sol` at $2 input / $0.10 cached / $10 output per 1M
tokens, and `gpt-5.6-sol` at $4 / $0.40 / $20
> - So every default run and every paid Codex eval cell cost twice as
much as necessary
> - This pull request moves every place that prefers or selects
`gpt-5.6-sol` to `gpt-6.1-sol`, and keeps historical facts, rate cards
and compatibility fallbacks on `gpt-5.6-sol`
> - The benefit is half the token price for new default Codex agents and
for the Codex E2E and TestOps cells
## Linked Issues or Issue Description
**What existing behavior does this improve?**
The default Codex model for new agents and the Codex model that the
runner E2E suites and TestOps plans select.
**Subsystem affected**
Adapters (`adapter-utils`, `codex-local`), server native runtime
fallback, UI AI connection pools, Paperclip Runner devtools and live
evals, runner E2E catalog and TestOps mappings.
**Current behavior**
New Codex agents without an explicit model store `gpt-5.6-sol`. The
runner E2E `runner-codex` profile, the Reliability `queue-steer` plan
cell, the Product regression TestOps mapping and the ACPX Codex
qualification selection all use `gpt-5.6-sol`.
**Proposed behavior**
They use `gpt-6.1-sol`. Existing agents that store a model keep it.
`gpt-5.6-sol` stays selectable.
**Reason and benefit**
`gpt-6.1-sol` costs half as much per token on the in-repo reviewed rate
card. The current OpenAI pricing page (retrieved 2026-11-06) shows the
same short-context rates and also lists long-context rates of $4 / $0.20
/ $15 for `gpt-6.1-sol` against $8 / $0.80 / $30 for `gpt-5.6-sol`.
**Breaking changes**
None for stored configurations. See Risks.
## What Changed
Changed defaults and selections (all now `gpt-6.1-sol`):
- `PAPERCLIP_RUNNER_DEFAULT_MODELS.codex` in
`packages/adapter-utils/src/paperclip-runner-permissions.ts`. This also
changes `DEFAULT_CODEX_LOCAL_MODEL`,
`resolvePaperclipRunnerModel("codex", …)`, runner config normalization,
`codex_local` → Paperclip Runner conversion, the UI new-agent and
agent-config defaults, the composer default model and the public MCP
effort lookup.
- `codex-local` `models`: the 5.6 entry is now the literal
`gpt-5.6-sol`, because it used to be `DEFAULT_CODEX_LOCAL_MODEL`. The
list order and contents do not change.
- AI connection pool: a new OpenAI member profile uses
`PAPERCLIP_RUNNER_DEFAULT_MODELS.codex` instead of a hard-coded
`gpt-5.6-sol`.
- Paperclip Runner devtools ACPX Codex default and preset, the
`runner-codex-sol` local provider smoke, and the `acpx-codex-sol` live
workflow candidate.
- `tests/acpx-qualification-models.ts` Codex selection (runner E2E and
runner acceptance ACPX Codex profiles).
- Runner E2E: the `runner-codex` profile follows the default, so all
native Codex cells move. This includes the `runner-codex` cells among
the 52 standard `everyday-workflows` cells and the seven Codex
reliability cells. `runner-codex-mini` stays on `gpt-5.4-mini`. The
Reliability plan `queue-steer` `steeringProfile` is now
`['runner-codex', 'gpt-6.1-sol']`. Its executionId
(`everyday-workflows.runner-codex.daytona.queue-steer`), its fullName
and its TestOps case identity do not change. No separate profile is
necessary.
- `product-regression-testops.mjs`: the declared model for every
`runner-codex` case is now `gpt-6.1-sol` (it was already that for
`configured-repository`).
- `tests/runner-e2e/installed-cli.ts`: the credential-free installed
startup proof opens a session on the default model.
- `tests/runner-e2e/scheduled-prices.json`: add a `gpt-6.1-sol` price
(`openai-standard-2026-11-06`, including 30m/12h cache writes and
long-context rates from the OpenAI pricing page). The weekly scheduled
Codex cells need an exact price for their model, or admission fails
closed.
- Storybook: the composer model picker fixtures, the connection pool
stories and host, and the agent settings preview show `gpt-6.1-sol` as
the default.
- Docs: `doc/DEVELOPING.md` (the Codex conversion default) and
`tests/runner-e2e/EVERYDAY-WORKFLOWS.md` (local matrix, reliability
extras, `queue-steer` row).
- Tests that assert the default now expect `gpt-6.1-sol`. New
assertions: the default has the `0.159.0` CLI floor, the compatibility
fallback is floor-free, the `queue-steer` plan cell runs `runner-codex`
on `gpt-6.1-sol`, and all native Codex everyday cells (standard and
reliability) use `gpt-6.1-sol`.
Deliberately kept on `gpt-5.6-sol`:
- Rate cards: `server/src/services/codex-pricing.ts` and
`packages/paperclip-runner/src/evals/model-pricing.ts`. Old receipts
need them. (The runner eval rate card has no `gpt-6.1-sol` entry. An
unpriced candidate is reported as unpriced, not as zero. I did not add a
rate there because that card has its own version and source.)
- The remote Codex CLI compatibility fallback. A new
`CODEX_LOCAL_COMPATIBILITY_FALLBACK_MODEL = "gpt-5.6-sol"` replaces the
default in `codexModelFallbackCandidates`. `gpt-6.1-sol` needs Codex
0.159.0 or newer, so it cannot be the floor-free last resort. Fallback
behavior is unchanged.
- `tests/runner-e2e/provider-usage.ts`: the implicit request
cache-policy attribution stays qualified only for `gpt-5.6-sol`. I did
not verify that policy for `gpt-6.1-sol`. A `gpt-6.1-sol` response that
omits the write-TTL split keeps its cache-write cost unknown, which is
the existing behavior for unqualified models.
- `tests/runner-e2e/scheduled-measurements.json`: historical
`gpt-5.6-sol` measurements. They no longer match the current suite
definitions, so the forecast reports them as historical only. The
forecast tests now build their synthetic cohorts on the current
definition hash.
- Model catalogs that list `gpt-5.6-sol` as available (`codex-local`,
`cursor-local`, `opencode-local`, Speko voice diagnostics, Pi picker
fixture).
- The `gpt-5.6` → `gpt-5.6-sol` alias for old stored configs.
- Dated or evidence docs: `doc/plans/*`, `doc/adapter-model-audit-*`,
`QUALIFICATION-*`, `doc/connections/MEMORY.md`,
`doc/connections/TASK-REVIEWS-VERIFICATION.md`,
`tests/runner-e2e/CHAT-FEATURES.md`, the Slack eval evidence, and the
S49 soak signature text (the soak failure happened on `gpt-5.6-sol`).
- Test fixtures that parse, price, migrate or round-trip an existing
`gpt-5.6-sol` value (Rust runner tests, driver tests, pricing tests,
stored-config tests, stories that show an existing agent).
## Verification
- `npx vitest run` on every server, UI, adapter-utils and codex-local
test file that mentions `gpt-5.6-sol` or the default, plus all of
`packages/adapter-utils` and `packages/adapters/codex-local`: 5,483
passed. Three timeouts under load (`sandbox-managed-runtime`,
`acp.quota`) passed when rerun.
- `packages/paperclip-runner`: `vitest run` for `src/eval`, `src/evals`,
`src/drivers/acpx`, `src/drivers/codex`, `src/cli`, `src/contracts`,
`src/backends`: 2,641 passed.
- `node --test tests/runner-e2e/reliability-testops.test.mjs
tests/runner-e2e/product-regression-testops.test.mjs`: 28 passed.
- `node tests/runner-e2e/run-unit.mjs`: all model, catalog, reliability,
scheduled and TestOps tests pass. Failures remain only in
`copilot-protection-flow`, `denied-target-isolation-flow`,
`pi-controls-flow`, `workflow-target` and `unit-launcher`. These five
also fail on unmodified master in this local environment (filesystem
watch and git environment), and they do not touch models.
- `vitest run --config tests/runner-acceptance/vitest.config.ts`: 19
passed.
- `pnpm test:e2e:runner -- --list --suite everyday-workflows --group
reliability`: the seven `everyday-workflows.runner-codex.daytona.<case>`
cells show `gpt-6.1-sol` with `OPENAI_API_KEY,DAYTONA_API_KEY`.
`--matrix-json` gives 16 reliability cells with the same credentials as
before. `--list --all` shows no `gpt-5.6-sol` cell.
- Typecheck: `adapter-utils`, `adapter-codex-local`, `server`, `ui`,
`paperclip-runner` (TypeScript), and `pnpm test:e2e:runner:typecheck`.
- `pnpm check:token-gates`: clean.
- Not run: paid cells, Rust tests (no Rust change), the browser suites,
and the full `pnpm -r typecheck` / `pnpm build` (CI runs them).
## Risks
- Agents without an explicit model change model on their next save or
normalization, and Paperclip Runner rows with a blank model use
`gpt-6.1-sol` at execution. Stored models, including `gpt-5.6-sol`, are
never rewritten. No migration is necessary.
- `gpt-6.1-sol` needs Codex CLI 0.159.0 or newer with ChatGPT sign-in.
The runner pins `@openai/codex@0.160.0`. On a remote target with an
older image CLI, the existing compatibility check falls back to
`gpt-6-sol` and then `gpt-5.6-sol`, and that probe now runs for
default-model agents too. A legacy `codex_local` agent created with the
new default on a host with an older Codex CLI gets the existing
actionable "upgrade Codex" Test result.
- Qualification: the native Codex profile uses `adapter_constant` model
qualification, so it follows the default. No qualification gate is
dropped. The ACPX Codex qualification selection moves to `gpt-6.1-sol`.
Its next qualification run uses the new model. The provider-usage
cache-policy attribution stays limited to `gpt-5.6-sol`.
- Runner E2E history: the Codex cells change their suite definition
hashes, so earlier `gpt-5.6-sol` results are a separate cohort and do
not count as current.
- The `gpt-6.1-sol` scheduled price uses the file's existing `expiresAt`
(2026-11-19). It needs the same review as the other entries before then.
- No `.github/` change. The Reliability queue-steer cell keeps the
`runner-codex` profile id, so the existing workflow credential and setup
steps still match.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, running in
Claude Code with tool use (shell, file edits, web fetch of the OpenAI
pricing page).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Related: #19604 (commits the Reliability TestOps identities; this PR
changes only `steeringProfile` in the same file).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
38068de553 |
chore(evals): commit the Reliability TestOps plan and case identities (#19604)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Runner E2E cases run from Allure TestOps plans through
`runner-full-stack-e2e.yml`
> - #19549 added the "Reliability — staging soak" plan. Its selection
fails closed until real TestOps identities are committed
> - Registration run
https://github.com/paperclipai/paperclip/actions/runs/42289820474 on
master (
|
||
|
DottaandClaude Opus 5.5
|
586caa5f49 |
fix(native): hand a shutdown-interrupted startup to the next controller (#19596)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The native Runner path runs an agent turn through a runnerd process that the control plane owns > - A restart of the control plane must keep the task going without a click, if that is safe > - A graceful shutdown closes the HTTP listener first and then drains the heartbeat runs > - A native run that was between provider dispatch and runnerd backend setup in that gap failed with `local_runtime_api_listener_unavailable`, so the drain did not see it as running and did not suspend it for the next boot > - This pull request makes that failure a restart detach: the run stays running with its claim, the drain suspends it, and the next controller adopts it > - The benefit is that a SIGTERM in this window no longer spends a failure and a same-run retry, and the task continues by itself ## Linked Issues or Issue Description Refs #19530 (the same restart window, before provider dispatch). **What happened?** In a restart chaos replay (ACPX Claude, local runner), a restart-recovered run got SIGTERM while `createRunnerdBackendWithinSessionClaim` built the runnerd backend. `resolveLocalRuntimeApiUrl()` threw `local_runtime_api_listener_unavailable` because shutdown had already closed the listener. The run settled `failed`. The drain then logged `restartSuspendedRunIds: []`. The next boot used the one bootstrap retry. A second SIGTERM hit the same window, and the task stopped at a manual Retry. **Expected behavior** A run that this process cannot continue because it is shutting down is handed to the next controller. It is not settled as failed. **Steps to reproduce** 1. Start a native Runner task and restart the server (K1) so that startup recovery resumes the run. 2. Send SIGTERM while the resumed run sets up its runnerd backend. 3. See the run fail with `local_runtime_api_listener_unavailable` and the drain report no suspended runs. ## What Changed - `server/src/runtime-api.ts`: `resolveLocalRuntimeApiUrl()` throws the new `LocalRuntimeApiListenerClosedError` when this process had a listener and closed it (the `listening` flag is false or the `close` event fired). The error message stays `local_runtime_api_listener_unavailable`, so the existing classifiers do not change. A listener that never started still throws the plain error. - `server/src/services/native-runtime/native-session-executor.ts`: for a local Runner, the startup failure handler treats `LocalRuntimeApiListenerClosedError` like a restart detach. It stops lease renewal, keeps the claim and throws `NativeControllerDetachedForRestartError`. Heartbeat already keeps such a run `running`. A remote Runner keeps the same-run bootstrap retry from #19462, because startup recovery has no process evidence to adopt a claimed remote startup (`native-runner-restart-preparation.integration.test.ts` still covers that path). - `doc/execution-semantics.md`: describe the new rule. - Tests: a heartbeat regression that closes the real runtime API listener before backend setup, then drains and restarts; a unit test for the closed versus never-started listener; two `vi.mock` factories keep the real module exports. ## Verification - Red before the fix: `pnpm exec vitest run src/__tests__/heartbeat-native-restart-suspension.test.ts -t listener` failed with `restartSuspendedRunIds: []` and the run `failed` (`local_runtime_api_listener_unavailable`, `bootstrap_retry`). - Green after the fix: the same test passes. The drain suspends the run, the run stays `running` with `awaiting_runner_reattach`, and the next controller claims it and reaches the backend once. - `pnpm exec vitest run src/__tests__/heartbeat-native-restart-suspension.test.ts src/http/server.test.ts src/__tests__/server-startup-feedback-export.test.ts src/services/native-runtime/native-session-executor.test.ts src/services/native-runtime/native-recovered-owner-followup.test.ts`: 1333 tests pass. One timing test in `native-session-executor.test.ts` failed once under host load average ~30 and passed alone. - `pnpm exec tsc --noEmit -p server/tsconfig.json`: pass. - Live, restart chaos harness, local ACPX Claude (`claude-sonnet-5`), `--rc06-masked`, this branch at |
||
|
github-actions[bot]andlockfile-bot
|
e28158b353 |
chore(lockfile): refresh pnpm lockfiles (#19617)
## Thinking Path > - Paperclip uses a dependency lock for reproducible builds. > - Dependency changes on master need matching lock entries. > - Paid eval reports also pin a reviewed offline correction. > - Reports must install before paid model work starts. > - This automated PR keeps the lock and report proof together. ## Linked Issues or Issue Description **What happened?** Dependency manifests on master changed after the last committed lock refresh. **Expected behavior** Frozen installs must match those manifests. Eval reports must retain matching offline lock proof. **Steps to reproduce** Run a frozen install before and after this lock refresh. ## What Changed - Refresh the dependency lock from the manifests on master. - Refresh the sandbox provider lockfiles from their manifests on master. - Include the matching offline report correction and hashes. ## Verification - Pinned pnpm resolution completed with lifecycle scripts disabled. - The report proof refresh accepted the reviewed committed baseline. - The workflow rejected changes outside the lock and report proof files. ## Risks The lockfile bot merges this PR without CI or review when every added package version is at least 3 days old on npm. Otherwise the PR waits for a maintainer, and the daily refresh tries again. The bot merges only the exact commit this workflow generated from manifests already reviewed on master. Unknown report lock baselines remain blocked. This update changes no permissions or paid enable gates. ## Model Used None — generated by pnpm and the deterministic report-lock updater. ## Checklist - [x] Thinking path and problem description included - [x] Generated lock and matching report proof included - [x] Release-age gate decides between bot merge and maintainer review Co-authored-by: lockfile-bot <lockfile-bot@users.noreply.github.com> |
||
|
DottaandPaperclip
|
4d554f0821 |
test(runner): set default inputs in workflow target fixture (#19602)
## Thinking Path > - Paperclip manages work performed by AI agents. > - Its Runner support tests verify workflow authorization and target pinning. > - The workflow now passes two Boolean inputs to its shell guard. > - The target fixture runs that guard with `set -u` but omits those inputs. > - Five cases stop on an unset variable before they reach their assertions. > - This change supplies the workflow's normal false defaults in the fixture. > - The production guard and all target-pinning assertions remain unchanged. ## Linked Issues or Issue Description Refs #19537, #19576. **What happened?** The required Runner support suite fails five `workflow-target.test.ts` cases with `IMAGE_ONLY: unbound variable`. The real workflow supplies `IMAGE_ONLY` and `TESTOPS_REQUESTED`. The test fixture does not supply either value. **Expected behavior** The fixture uses the normal workflow defaults and reaches the target authorization assertions. **Steps to reproduce** Run `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts workflow-target.test.ts` on master `4938a930d66846094810cbed45d2fd33c0d14c2d`. ## What Changed - Set `IMAGE_ONLY` and `TESTOPS_REQUESTED` to `false` in the target fixture's child environment. ## Verification - RED: the unchanged current-master fixture had five failures and four passes. Bash stopped on `IMAGE_ONLY: unbound variable` before target authorization ran (220ms). - GREEN: all nine cases pass after the one-line fixture repair (168ms). Existing checks still reject a custom bounded target and preserve the queued commit for scheduled and bounded manual runs. - Local evidence uses the actual test and workflow files with installed Vitest 5.0.3 and a temporary focused config. It omits the unrelated provider-bundle global setup. No provider, external service, or paid workflow runs. - The [original required CI failure](https://github.com/paperclipai/paperclip/actions/runs/42288042001/job/128134318743) has the same five Bash failures. Its evidence is retained; no unchanged retry was used. - Whitespace checks pass. Exact head `f1d178ad483a49f8dd087a1c6d22bf43c06a39e4` has 56 successful checks, three expected skips, one neutral advisory status and successful Snyk. No check is pending or failed. The [full Runner support suite](https://github.com/paperclipai/paperclip/actions/runs/42289536842/job/128139308510), typecheck, build and [aggregate verification](https://github.com/paperclipai/paperclip/actions/runs/42289536842/job/128145908574) pass. Greptile and Paperclip Review both rate this exact head 5/5, with no findings or unresolved threads. - Environment: macOS arm64, Node 26.4.0, Vitest 5.0.3. ## Risks Low risk. This changes only fixture inputs for normal target selection. It does not change the production workflow guard, branch policy, credentials, paid execution, timeouts or assertions. Product documentation needs no update for test-only default inputs. ## Model Used OpenAI Codex, GPT-6 family, with repository analysis, code editing and tool execution. The exact runtime model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub references) - [x] My branch name describes the change and contains no internal Paperclip ticket id - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes (not needed: fixture-only default inputs) - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
02214b1a3c |
fix(evals): treat a job-less TestOps plan row as unassigned during registration (#19601)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Runner E2E cases run from Allure TestOps plans through
`runner-full-stack-e2e.yml`
> - #19549 added a sibling "Reliability — staging soak" plan and a
one-shot registration workflow
> - The first live `register` dispatch created the seven cases and the
plan, then stopped with `plan_job_conflict`
> - Live TestOps lists a new plan's uncovered cases as one row without a
job, and the code read that row as another job
> - This pull request treats a job-less row as "not assigned"
> - The benefit is that a second `register` dispatch can assign the plan
to the standing GitHub job and finish registration
## Linked Issues or Issue Description
Refs #19549
**What happened?**
The `runner-e2e-testops-registration.yml` `register` run on master
failed with `reliability_testops_registration_plan_job_conflict`. It had
already created the seven cases and the plan. `GET
/api/testplan/<id>/job` for that plan returned `[{"testCasesCount":7}]`.
The row has no `job` field.
**Expected behavior**
Registration treats a plan with only a job-less row as unassigned. It
assigns the plan to the standing Product regression GitHub job and reads
the assignment back.
**Steps to reproduce**
1. Dispatch `runner-e2e-testops-registration.yml` on master with
`operation=register` when no Reliability plan exists.
2. The new plan reports one job-less row.
3. Registration fails with `plan_job_conflict` before the assignment
step.
**Paperclip version or commit**
master at
|
||
|
DottaandClaude Opus 5.5
|
b7e28eb2f7 |
fix(evals): let an allowlisted manual dispatch authorize its private evidence (#19599)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Chat-feature evals run in the trusted `runner-full-stack-e2e` workflow and keep their evidence in private history > - A manual chat run needs authority to write that private evidence > - Today that authority is a stored approval for the exact run ID in `RUNNER_E2E_MANUAL_PRIVATE_APPROVALS_JSON` > - The run ID exists only after dispatch, so every manual run stops until a person writes the record > - The dispatcher is already checked against `RUNNER_E2E_ALLOWED_ACTOR_IDS`, so the second step adds friction without a new decision > - This pull request lets an allowlisted manual dispatch authorize its own private evidence, as the TestOps admission already does > - The benefit is that an allowlisted operator can start a chat eval and it runs, while scheduled runs still never touch chat evidence ## Linked Issues or Issue Description Refs #18134, Refs #18170 **What existing behavior does this improve?** A manual `runner-full-stack-e2e` run with chat-feature IDs fails at private evidence transfer unless a person first adds an approval row for that run ID to a repository variable. **Subsystem affected** Runner E2E manual workflow private evidence authorization (`manual-private-transfer.ts`, `runner-full-stack-e2e.yml`). **Proposed change** Derive the evidence authority from the dispatch itself, with the same checks the TestOps chat admission uses. ## What Changed - New `tests/runner-e2e/manual-dispatch-authority.ts`: `authorizeManualDispatch` admits private evidence only when all of these are true: - The event is `workflow_dispatch` of `runner-full-stack-e2e.yml` on the default branch, with the `manual` cadence. - The run attempt is `1`. - The actor is in `RUNNER_E2E_ALLOWED_ACTOR_IDS`, the event sender is that actor, and the triggering actor is the same person. - The trusted cell selection equals the dispatch's own `id` input. - The authority binds the workflow SHA, the target SHA from the authorize job, the run ID and the cells. Every job derives the same identity hash. - `manual-private-transfer.ts`: uses the TestOps authority first, then the dispatch authority, then the stored approval. With the dispatch authority, `prepare` records the target launcher digest instead of comparing it with an approved digest. - `runner-full-stack-e2e.yml`: the 14 steps that read `RUNNER_E2E_MANUAL_PRIVATE_APPROVALS_JSON` now set `PAPERCLIP_MANUAL_DISPATCH_AUTHORITY: v1` and pass `RUNNER_E2E_ALLOWED_ACTOR_IDS`. - `doc/manual-e2e-private-evidence.md`: describes the dispatch authority. The stored-approval shape stays documented for the TestOps catalog case workflow. - The new file is separate on purpose. `manual-private-retention.ts` and the other protocol files must match each target byte for byte, so existing candidate branches keep working. ## Verification - `vitest run tests/runner-e2e/manual-dispatch-authority.test.ts tests/runner-e2e/manual-private-transfer.test.ts tests/runner-e2e/manual-private-report.test.ts tests/runner-e2e/manual-private-report-integration.test.ts tests/runner-e2e/manual-private-policy.test.ts`: 57 tests passed. - New tests cover one valid dispatch and rejection of: a scheduled run, a bounded cadence, a non-allowlisted actor, an empty allowlist, a rerun by another person, a second attempt, a non-default or different workflow ref, a sender who is not the actor, a missing target SHA, a missing authority switch, and cells that differ from the dispatch input. The eight-cell limit still applies. - A `prepare` test runs the real transfer script: a non-allowlisted actor produces no outputs, and an allowlisted dispatch produces the session policy and an identity that records the target launcher digest. - Negative control: with the previous `manual-private-transfer.ts`, the new `prepare` test fails. - Workflow tests that read this workflow: 111 of 116 passed. The 5 failures in `workflow-target.test.ts` (`IMAGE_ONLY: unbound variable`) also fail on unchanged `master`. `paid-runner-routing.test.mjs` passed (7 tests). - I did not run a live dispatch. The first live proof is a manual dispatch of chat IDs after merge. ## Risks - The dispatcher now vouches for the target branch. A target branch with a changed launcher can write private evidence if an allowlisted operator dispatches it. The protocol files must still match the trusted copies byte for byte. - The eight-cell limit is unchanged. The report job has eight fixed read slots, and each read policy must fit the 2,048-character session policy limit. Lifting the limit needs a separate change to the report sessions. - The `runner-e2e-paid` environment still has required reviewers. That is a repository setting and this change does not touch it. - The authority uses `RUNNER_E2E_ALLOWED_ACTOR_IDS`, not `CODEOWNERS`. Keep that variable aligned with the people who may start paid runs. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
744056ecc8 |
feat(evals): run the staging-soak reliability cases from a TestOps plan (#19549)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The runner full-stack E2E lane runs real Paperclip work against real agents, and TestOps launches its standing plans > - PR #19455 added seven staging-soak reliability cases, but no TestOps plan or workflow lane selects them, so they run only by hand with personal keys > - These cases catch restart, interrupt, queue and lost-runner regressions, so they must run from TestOps before a release with the standard reports > - This pull request adds a sibling "Reliability — staging soak" TestOps plan on the existing plan-2 lane, with native OpenCode and DeepSeek 1017 through OpenRouter as the plan profile for six of seven cases > - The benefit is a cheap, repeatable pre-release reliability launch with the same Allure/TestOps results, retained report and dashboard as plan 2 ## Linked Issues or Issue Description Refs #19455 **What existing behavior does this improve?** The staging-soak reliability cases from #19455 run only by a manual local command. **Subsystem affected** Runner full-stack E2E (`tests/runner-e2e/`) and its TestOps lane in `.github/workflows/runner-full-stack-e2e.yml`. **Current behavior** No TestOps plan includes the cases. The workflow cannot select them from TestOps. Every cell uses native Codex (`gpt-5.6-sol`) or an ACPX facade on Daytona, which is expensive. **Proposed behavior** A TestOps launch of the Reliability plan runs seven cells (six DeepSeek 1017, one native Codex for steering) through the standing plan-2 path and reports them like plan 2. **Reason and benefit** The cases become a first-class, low-cost, pre-release check with retained evidence. Plan 2 keeps its 20-case qualification semantics. **Breaking changes** None. Plan 2 selection, reporting and history identities stay the same. ## What Changed - **Plan choice: a sibling plan, not an extension of plan 2.** Plan 2 is bounded at 20 standing IDs. Its `full-run` / `test-plan-2` tag needs a complete 20-case attempt, and subset runs do not qualify it. Adding seven cases would change that denominator and the qualification history. The sibling plan shares the same TestOps GitHub job, selection, admission, projection, upload and verification code. A launch selects cases from exactly one plan (`plan_mixed` fails), and each plan has its own full-plan size (20 or 7). - `tests/runner-e2e/reliability-testops.mjs` (new): the mapping module. It has seven cases, full names `paperclip.Product E2E.everyday-workflows.<case>`, execution IDs, soak signatures and an empty `reliabilityRegistration`. A partial, duplicate, plan-2 or malformed registration is rejected. An empty registration makes selection fail closed before any spend (`product_regression_testops_reliability_unregistered` for a bootstrap, `case_mismatch` for a TestOps launch). No TestOps IDs are invented. - `product-regression-testops.mjs` / `-cli.mjs`: select a plan by its case identities or by the new bootstrap input. The projection binds each cell's own environment (`local` or `daytona`) instead of a fixed `local`. It adds a `soak-<signature>` tag and reports `plan` and a per-plan `fullPlan`. Plan-2 history IDs, parameters and results do not change. - Catalog: `runner-opencode-deepseek-1017` is now one shared fixture with `supportedEnvironments: ["local", "daytona"]`. The OpenCode server driver already runs on Daytona for `runner-opencode` (task-workspaces, core-compatibility, rolling deployment); only the model differs. The chat-feature suite stays local, and its definition hash does not change. `everyday-workflows` gains exactly six DeepSeek plan cells (definition version 12, 52 + 13 + 3 cells). - Cell matrix (plan), DeepSeek 1017 on native OpenCode: `interrupt-redirect`, `reassign-midrun` and `tree-pause-release` run on `local`. `three-turn-revise` runs on Daytona with the per-turn lease (S49). `long-silent-command` (S52/S52b) and `queued-across-restart` (S55/U9) run on the Daytona warm lease. - `queue-steer` is the one exception. It must steer an active turn, and the OpenCode server driver declares `steering: false`, so the route answers `steering_unsupported` and the case cannot pass. The plan uses the existing steering-capable `everyday-workflows.runner-codex.daytona.queue-steer` cell for it (Greptile finding). No DeepSeek `queue-steer` cell exists. The other native Codex Daytona cells and the ACPX facades stay selectable through `--group reliability`, but they are outside the plan. - `runner-full-stack-e2e.yml`: adds the `testops_plan` bootstrap input (`product-regression` by default), sets the plan-specific bootstrap launch name and tags, and renames two jobs to plan-neutral names. The OpenCode steps now match `startsWith(matrix.profileId, 'runner-opencode')`, so the DeepSeek profile gets the pinned OpenCode binary, runner TypeScript, the local Node check and the remote provider pack. Before this change, the DeepSeek profile could not run in this workflow. `OPENROUTER_API_KEY`, `OPENAI_API_KEY` (only for `queue-steer`) and `DAYTONA_API_KEY` come from the same `matrix.credentialName` and `matrix.environmentId` gates that other cells use. - `runner-e2e-testops-registration.yml` + `reliability-testops-registration.mjs` (new): a trusted, master-only, allowlisted-actor, idempotent `workflow_dispatch`. `inspect` only reads. `register` reuses cases with the same full name, creates missing cases and the plan, and assigns the plan to plan 2's GitHub job with plan 2's job parameters. It writes a receipt with a ready-to-commit `reliabilityRegistration` snippet. Only this job gets `ALLURE_TOKEN`, and it never runs a provider cell. - **No schedule and not per PR.** The plan runs only from a TestOps launch or a manual dispatch. It is not in the daily or weekly cron (`scheduledSelection` excludes it). A weekly run would add a paid 50-minute Daytona cell every week, but these signatures move with release-risky changes, not with time. - Docs: `EVERYDAY-WORKFLOWS.md` (matrix, environment reasons, local coverage limits, registration steps), `README.md`, `doc/evals.md`. ### Registration still needed (requires TestOps credentials) 1. Dispatch **Runner E2E TestOps registration** from master with `operation=inspect`. 2. Dispatch it with `operation=register`. 3. Commit the `reliabilityRegistration` snippet from the run summary into `tests/runner-e2e/reliability-testops.mjs` in one reviewed PR. 4. Launch the plan from TestOps. Tag only a complete seven-case attempt `full-run`. ### What local plan cells cannot reproduce The local cells do not reproduce the Daytona-only variants: the remote `workspace_busy` save after Stop (S47), remote stop settlement during reassignment (S48), and a lost or replaced remote runner during a tree hold (S50). S51 (lost runner) has no case yet. The explicit-only native Codex Daytona extras still cover S47, S48 and S50 on a sandbox. ## Verification - `node --test tests/runner-e2e/product-regression-testops.test.mjs tests/runner-e2e/reliability-testops.test.mjs`: 26 pass. These tests cover fail-closed selection, partial or duplicate registration, plan mixing, environment binding, per-plan denominators, preserved failure classes, the CLI select → project → verify flow for a seven-case launch, registration idempotency and job checks, and the workflow inputs and credential gates. - `vitest` for `reliability-testops-catalog`, `product-regression-testops-catalog`, `everyday-reliability` and `catalog.test`: 98 pass. These tests bind the mapping to real catalog cells (profile, model, environment, credentials, single attempt). They also check that the cells stay out of `--all`, the suite, profile and environment selectors and the daily/weekly cron. Real grader output must survive projection. - `pnpm test:e2e:runner:typecheck`: pass. - `pnpm test:e2e:runner -- --list --suite everyday-workflows --group reliability` lists 16 cells: 6 DeepSeek plan cells, 7 native Codex cells (one is the `queue-steer` plan cell) and 3 ACPX extras. `--matrix-json` for the DeepSeek plan IDs emits `needsDaytona=true` and three Daytona OpenCode cells with `credentialName=OPENROUTER_API_KEY`. - Full `pnpm test:e2e:runner:unit`: the remaining failures happen the same way on an unchanged master checkout. They are in `workflow-security` (3), `controller-outage`, `operating-receipts`, `pi-*`, `private-framework-report` and `remote-native-bootstrap`. #19529 tracks that drift. This PR adds no failures. - Not run: paid cells, a TestOps registration and a TestOps launch. They need provider and TestOps credentials, which exist only in protected CI environments. Catalog and projection tests are not live qualification. ## Risks - The registration script calls TestOps APIs (`POST /api/testcase`, `/api/testplan`, `/assign`, `/jobparameter`) through the same endpoints as the chat catalog reconcile. Tests use fakes, not the live API. `inspect` comes first, and `register` is idempotent. - DeepSeek 1017 on Daytona has not run before. The first launch is the acceptance step for those three cells. - If a later case needs another capability that OpenCode lacks, it must use a capable profile in the same way as `queue-steer`. The catalog test pins the single exception. - The OpenCode workflow conditions now match any `runner-opencode*` profile. Today this covers `runner-opencode` and `runner-opencode-deepseek-1017` only. - If someone commits a bad `reliabilityRegistration`, module load fails for both plans. A unit test rejects any committed registration that is not empty or complete. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with tool use (shell, file edits, local test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
840838476a |
perf(runner-e2e): cut the Daytona image build critical path (#19545)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner Full-Stack E2E workflow tests the Runner in Daytona sandboxes from a signed, content-addressed image > - Every change to an audited Runner input needs a new image before a Daytona cell can test it > - The image job took 10 to 16.5 minutes today, and a restart chaos campaign waits on it after each runner-side fix > - Measurement shows two serial waits: a full Rust dependency compile (about 5 of 6.6 minutes) and a 3.5 to 4 minute download of cached base layers that starts only after the builder stages finish > - There is also no way to build only the image: every dispatch reaches a paid gate that the operator must reject > - This pull request removes both serial waits and adds an `image_only` dispatch input > - The benefit is a shorter fix-to-Daytona-test loop, with every image security check unchanged ## Linked Issues or Issue Description **What happened?** The `Publish verified Daytona image` job is on the critical path of the control-plane restart chaos campaign. Builds today took 16.5 min (run 42240434011), 15 min (42253920727), 10.5 min (42259118276) and 16 min (42265637650). Run 42264611573 failed: its RunsOn runner lost communication with GitHub 7.6 minutes into the build step, during the Rust compile. GitHub has no log for that job. **Expected behavior** A Runner source change should not recompile the locked third-party Rust graph. Cached image layers should download while the builder stages compile. An operator should be able to publish only the image, without a paid gate. **Steps to reproduce** Read the BuildKit timeline in the logs of job 127970125495 (run 42240434011). See the stage table below. ## What Changed - `docker/daytona-runner/Dockerfile`: compile the locked Rust dependency graph in its own layer against placeholder targets. The layer depends only on `Cargo.toml`, `Cargo.lock` and the crate manifest. The step deletes the placeholder outputs and fingerprints, so the next step always compiles the real Runner sources. The final stage still receives only the stripped executable. - `docker/daytona-runner/Dockerfile`: move the content-ID / source-revision check above the two `COPY --from` steps. It does not depend on either builder stage, so BuildKit now downloads the cached base and CLI layers in parallel with the compile. Both copies use `COPY --link`. - `.github/workflows/runner-full-stack-e2e.yml`: add the boolean `image_only` dispatch input. `authorize`, `target_lock`, `catalog` and `daytona_image` run unchanged. Every other campaign job gets `&& inputs.image_only != true`. `authorize` rejects `image_only` together with TestOps inputs. - `tests/runner-e2e/workflow-image-only.test.ts` (new): pin the job classification, the top-level guard on each campaign job, the guard on every protected environment, and the TestOps rejection. - `tests/runner-e2e/README.md`: document the layer layout and the `image_only` command. ## Verification Measured on the same RunsOn `m7i-flex.xlarge` (4 vCPU, 16 GiB, native x64) runner class. "Before" is master job 127970125495 (run 42240434011). "After (cold)" is this branch, job 128077687214 (run 42271236140), with no dependency layer in the registry cache yet. Its paid gate was rejected. | Stage | Before | After (cold dependency cache) | |---|---|---| | Whole image job | 16.5 min | **11.6 min** | | Rust dependency graph | 311 s (inside one 399 s step) | 395 s (own layer; reused from cache after a master build) | | `paperclip-runner-core` compile | ~87 s | 65 s | | Cached base and CLI layer download | 236 s, **after** cargo finished | 257 s, **in parallel** with cargo | | Copy runnerd + provider pack | 20 s + 50 s | 9 s (`--link`) | | Provider pack stage | 143 s (parallel) | 152 s (parallel) | | Export + push | 82 s + 10 s | 81 s + 11 s | | Dependency layer size | n/a | `target` 852 MB + registry 333 MB (builder stage only) | With a warm dependency cache, a Runner source change compiles only the 65 s crate step. The critical path then becomes the ~4 min cached-layer download or the provider pack. The expected job time is about 7 to 8 min. I measure this after merge with an `image_only` dispatch (master first, to write the cache, then a Rust-only branch change). Disk: the instance starts with 38.76 GiB free. The dependency layer adds about 1.2 GB uncompressed to builder storage. The final image does not change. - `vitest run tests/runner-e2e/workflow-image-only.test.ts tests/runner-e2e/workflow-security.test.ts tests/runner-e2e/daytona-image.test.ts`: the new test and all image tests pass. Three tests in `workflow-security.test.ts` (`gates every provider-secret job with stable actor IDs`, `keeps provider credentials inside explicitly gated paid workflows`, `isolates private archive trust from workflows that execute target-branch code`) fail on `origin/master` without this change, because of `chat-catalog-testops-case.yml`. This pull request does not change them. - `node --test tests/runner-e2e/product-regression-testops.test.mjs`: 15/15 pass. - No local Docker build. The Dockerfile is measured with a branch-targeted CI dispatch whose paid gate was rejected. ## Risks - The dependency layer keeps the Rust `target` and registry trees in a builder-stage layer, and default-branch builds export it to the registry cache. This adds builder disk and cache upload when `Cargo.lock` changes. The build prints the size (`du -sk target /usr/local/cargo/registry`). - The dependency cache helps only after a default-branch build writes it. Branch builds read it but never write it. This is the existing cache policy. - `image_only` changes the workflow, so it works only after merge to master. - No check is removed. Lockfile verification, cosign sign and verify, the anonymous pull check, the provider-pack manifest match, the default-branch-only cache write and the actor allowlist are unchanged. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, GitHub CLI). Extended reasoning on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Nicky LeachandClaude Opus 5.5
|
79c724b292 |
ci: run the Runner support suite as a required PR check (#19537)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The Runner support suite (`pnpm test:e2e:runner:unit`) checks the code behind paid Runner runs: workflow security policy, evidence oracles, cost receipts and controller recovery. > - Only a manual Docker Runner check ran this suite. PR CI did not run it. > - So changes on master broke 16 of its tests, and nobody saw it. #19527 and #19529 fix those tests. > - This pull request runs the suite in PR CI and makes the `verify` check require it. > - The benefit is that a change that breaks the suite fails on its own pull request, before merge. ## Linked Issues or Issue Description Refs #19527 and #19529 (both merged; they repair the suite). Also accounts for #19533, which added the sandbox provider job. **Merge order:** #19527 and #19529 have merged, so the suite now passes on master. This PR can merge. **What existing behavior does this improve?** PR CI for `pr-trusted.yml`. **Current behavior** `pnpm test:e2e:runner:unit` runs only in `docker-runner-check.yml`, by manual dispatch. Master changes broke 12–13 tests by Nov 3 and 16 by Nov 6, with no signal on any pull request. **Proposed behavior** A required "Runner support suite" job runs the suite on pull requests. **Reason and benefit** Drift fails on the pull request that causes it. Authors update deliberate pins (profile versions, suite hashes, workflow policy) in the same pull request. ## What Changed - `pr-trusted.yml`: new job `verify_runner_e2e_support` ("Runner support suite"). It uses the same steps as other trusted jobs: checkout without credentials, setup-node directly before pnpm, the npm overrides, the shared `setup-ci-dependencies` restore, and CI profiling. Then it runs `pnpm test:e2e:runner:unit`. It has no secrets. Timeout: 20 minutes. - `pr-trusted.yml`, job `verify`: the job now needs the new job and checks its result. A failed, cancelled or wrongly skipped run fails `verify`. - `turbo.json`: new routing task `ci:runner-e2e-support`. Its inputs are `server/**`, `cli/**`, `packages/**` and `ui/**`. Changes to `tests/**`, `scripts/**` and `.github/**` already force full CI, so they also run the job. - `.github/ci-task-matrix.json`: one lane for the new job. - Tests: `ci-plan.test.mjs`, `ci-profile.test.mjs`, `e2e-shard.test.mjs` and `workflow-security.test.ts` now include the new job. A UI component change now selects this one Runner task. The suite imports UI modules and checks that catalog-listed UI files stay tracked in git, so UI changes can break it. - CI profiling: when `PAPERCLIP_CI_PROFILE` is on, the suite's Vitest config adds `scripts/ci-profile/vitest-reporter.mjs`, and `run-unit.mjs` runs the Node tests through `scripts/ci-profile/node-test.mjs`. So the job uploads per-test timings, like the root and Runner package suites. The launcher loads the profiling helper only after its "Vitest is not installed" check. - `tests/runner-e2e/README.md`: says that PR CI runs the suite, and that deliberate pin changes go in the same pull request. ## Verification - `node --test .github/scripts/tests/*.test.mjs scripts/ci/*.test.mjs scripts/__tests__/e2e-shard.test.mjs scripts/__tests__/release-verify-workflow.test.mjs scripts/__tests__/runner-release-crate-cache.test.mjs`: 830 passed, 3 failed. The same 3 fail on the base branch without this change. They need local tools that are not on this Mac. - A local run with `PAPERCLIP_CI_PROFILE=true` wrote per-test and per-file timing records for Vitest and for Node. Without profiling, output and exit codes do not change. Both `unit-launcher.test.mjs` tests pass. - The aggregate test now covers the new job: `verify` rejects it when it fails, is cancelled, or is skipped while selected. - The pnpm bootstrap policy tests in `workflow-security.test.ts` pass with 10 setups in `pr-trusted.yml`. - On this PR's merged tree (`80b526d697`, master plus this PR): `pnpm test:e2e:runner:unit` passed, Vitest 320 files (6,830 tests) and node:test 373, in 58s. It also passes on master `4c1f629c31`. - Suite runtime on a 10-core Mac: 43–63s. With `--maxWorkers=4`: Vitest 118s, about 490s of CPU in total. On a 4-vCPU GitHub runner I expect about 3–4 minutes for the tests and 1–2 minutes for setup. The job runs in parallel with the longer test shards, so it should not lengthen CI. - `pr.yml` calls `pr-trusted.yml@master`. This PR's own CI does not run the new job. The first real CI timing comes from the first pull request after merge. ## Risks - More runner minutes: one extra job on most pull requests (all except documentation-only ones). - The suite has not run on Linux CI recently: old `docker-runner-check` logs have expired. A Linux-only failure would show up on the first pull requests after merge. The job can be removed from the `verify` list quickly if needed. - #19229 removes `version: 9.15.4` from the pnpm steps in this file. Whichever PR merges second needs a small rebase. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), by Anthropic, in Claude Code with tool use (shell, file edits, git, GitHub CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Nicky LeachandClaude Opus 5.5
|
4c1f629c31 |
test(evals): repair the Runner support suite after master drift (#19529)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The Runner support suite (`pnpm test:e2e:runner:unit`) tests the code behind paid Runner qualification: workflow security policy, evidence oracles, cost receipts and controller recovery. > - PR CI does not run this suite. Only a manual `docker-runner-check.yml` run does. > - So deliberate changes on master broke its tests, and nobody saw it. There were 12–13 failures on Nov 3 and 16 on Nov 6. > - This pull request finds the master change behind each failure and fixes the side that is wrong. No security check becomes weaker. > - The benefit is that the suite passes on master again (with #19527), so it can become a required check. ## Linked Issues or Issue Description Refs #18516, #18625, #18983, #18325, #19265, #18994, #17009, #19195. Companion PR: #19527 (the one workflow fix). **What happened?** At master `543aa4d331`, `pnpm test:e2e:runner:unit` had 16 failures in 8 files. At `98ebeefbc9` I saw 15 in 7 files. The two embedded-Postgres files passed in every run I made, and the 256MiB packaging test timed out once. **Expected behavior** The suite passes on master. Each test still checks the rule it was written for. **Steps to reproduce** Run `pnpm install`, then `pnpm test:e2e:runner:unit` on master. ## What Changed Each failure, its cause, and the fix: - **`workflow-security.test.ts`: chat catalog workflows hold provider keys and use `runner-e2e-private-paid`.** This is a deliberate policy. #18516 admitted guarded, pinned-source chat TestOps cells, with special checks. #18625 removed the pilot workflows and their checks. It did not move the checks to the full-catalog workflows from #18542 and #18647, which use the same trust model. The fix adds `chat-catalog-testops.yml` and `chat-catalog-testops-case.yml` to the provider-key allowlist. It also brings back the #18516 checks, adapted to the caller plus reusable-workflow layout: - The cell is `workflow_call` only. Its single caller is the master-only dispatcher. - The trusted guard runs before the pinned candidate checkout. `verify-selection` runs before each provider key. - The paid job never sees `ALLURE_TOKEN`. The report job runs in `runner-e2e-private-history` and has no candidate checkout. The upload job has no private environment and no secret except `ALLURE_TOKEN`. - Provider keys appear only in the two caller jobs. Those jobs need `authorize, select, build` and pass the guard's `target_sha`. - I mutation-tested this: an `ALLURE_TOKEN` in the paid job fails, and a second caller workflow fails. - **`workflow-security.test.ts`: `ref: ${{ github.sha }}` 6 vs 4, `persist-credentials: false` 12 vs 10.** #18983 added two trusted TestOps jobs (`testops_selection`, `publish_testops`). Both check out reviewed code next to the TestOps token. This is the safe direction. The counts are updated. A new check makes sure both jobs check out only `github.sha`, do not install the candidate lockfile, and use no secret except `ALLURE_TOKEN`. - **`pi-file-evidence.test.ts` (4 tests): "one observed provider validation execution".** #19265 deliberately changed the ACPX tool normalizer to keep the latest concrete title. Pi's shell call now goes from `bash` to the exact command. The ACPX sidecar uses this normalizer, so the oracle would also reject real Pi runs. The oracle now accepts one change only, from `bash` to the exact validation command, with no change back. It also counts every completed `execute` call, whatever its name. This is stricter than before: a refined title cannot hide a second shell command. New mutations cover a regressed name, a different command, and an extra call that has its own command name. The retained 44-row real stream still passes. - **`pi-controls-evidence.test.ts`: Pi profile 37.** #18325 deliberately reset Pi to profile 37. The pin and the `pi-native` suite hash are updated. I checked that the old hash comes back exactly when the version is set to 36. So the version is the only change. - **`controller-outage.test.ts`: 1 continuity cell, not 2.** #18994 deliberately limited this lane to providers with managed warm sessions. The test now expects exactly `runner-codex`. - **`operating-receipts.test.ts` (3 tests): "Operating overhead allowance needs review".** This is a time bomb from #17009. The fixture allowance was valid from Oct 1 until Nov 1, 2026. `admitAttempt` reads the real clock. The allowance window is limited to 31 days, so a longer fixture is not possible. These tests now pin only `Date` to the fixture time. Timers and the filesystem stay real. I ran the full suite with the clock moved to Dec 15, 2026, and found no other date-bound failures. - **New file `private-trust-workflow-security.test.ts`.** The private archive trust test, the chat catalog checks and the full-stack TestOps job checks moved here from `workflow-security.test.ts`. That file would have crossed 1,000 lines. It is now 970 lines. The checks did not change. - **`remote-native-bootstrap.test.ts`: cannot load `@paperclipai/plugin-sdk`.** Since #19195, server code under test imports the built plugin SDK. `run-unit.mjs` now runs `scripts/ensure-plugin-build-deps.mjs` first, as `test:run:general` already does. This step is cached. - **`private-framework-report.test.ts`: 256MiB packaging test timed out.** It takes about 10s alone and more than 30s with the whole suite running in parallel. It now has a 120s budget. - **`controller-database.test.ts`, `controller-recovery-observer.integration.test.ts`: no change.** Embedded Postgres started and both files passed in all my runs. The earlier failures look like local resource problems. ## Verification - Combined check (this branch plus #19527, on master `5d25117ffa`): `pnpm test:e2e:runner:unit` — Vitest 317 files passed (6,815 tests, 3 skipped), node:test 373 passed, 0 failed. It took 63s on a 10-core Mac. - `pnpm test:e2e:runner:typecheck` — passed. - This branch alone fails only the two pnpm-setup tests that #19527 fixes. - Focused files: `pi-file-evidence` 75 passed, `pi-controls-evidence` 72 passed, `operating-receipts` 12 passed, `workflow-security` 14 of 16 passed (the other 2 pass with #19527), `private-trust-workflow-security` 3 passed, `remote-native-bootstrap` 71 passed. ## Risks - Low risk. Only test files, one evidence oracle under `tests/runner-e2e/`, and the suite launcher change. No product code changes. - The Pi oracle change affects how paid Pi file qualification is graded. It accepts the old and new title shapes, and it is stricter about extra shell calls. A live Pi qualification run would confirm it on real Rust-normalized events. I did not run one, because it is paid. - The chat catalog policy is the same as #18516. Under that design, the pinned candidate source runs in `runner-e2e-private-paid` with `id-token: write`. The guards above limit who can start it and which source runs. This PR does not change that trust model. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), by Anthropic, in Claude Code with tool use (shell, file edits, git, GitHub CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
7afb57c022 |
test(runner-e2e): repair restart-cell harness drift (#19538)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner Product E2E suite proves that agent work survives a control-plane restart > - A restart chaos campaign found three restart cells that fail in the test harness, not in the product > - A harness failure hides the real restart result, so the campaign cannot classify these cells > - This pull request repairs the harness drift in those cells and names the one failure the harness cannot prevent > - The benefit is that these cells now test restart behavior, or fail with a clear cause ## Linked Issues or Issue Description Refs #14811 (connector setup became one screen). Refs #19475 (completion-update restart fixture). **What happened?** 1. `lifecycle-baseline.runner-codex.local.tool-review-restart` clicked a "Save and continue" button. #14811 removed that button. The cell failed during setup. After that fix, the cell also failed after the restart: the expanded review card does not survive the post-restart reload, so "Approve & run" was not visible. 2. `local-session-integrity.runner-acpx-claude.local.structured-question-restart-resume` failed with `page.goto: net::ERR_ABORTED` after the controller restart. The old document's Vite client reloads when the replacement server starts, and this aborts the test navigation. The code tolerated only `TimeoutError`. 3. `completion-updates.runner-acpx-claude.local.handoff-completion-restart` waited for the source agent to run an instructed wait command. Claude replied without the command (2/2 attempts), so the restart never occurred. The cell then failed with a generic timeout. **Expected behavior** Each cell reaches its restart boundary and tests restart behavior. If the boundary cannot be reached, the cell names the cause. **Steps to reproduce** Run `pnpm test:e2e:runner -- --max-parallel 1 --max-automatic-retries 0 --id <cell>` for each cell above on `master`. ## What Changed - `tests/runner-e2e/connection-reviews.ts`: accept the one-screen "Check link" step and the older "Save and continue" step. This uses the same regex approach as `connection-flow.ts`. - `tests/runner-e2e/runner.spec.ts` (tool review): click "Review request" after the restart reload, on the document that acts on the card. - `tests/runner-e2e/chat-restart.ts`: add `gotoAfterServerRestart`. It retries only `net::ERR_ABORTED`, within the caller's deadline and at most 5 attempts. It still accepts `TimeoutError`. It rethrows all other errors. The structured-question restart uses it. The existing sentinel and durable-state assertions still prove the new document. - `tests/runner-e2e/completion-updates.ts` and `completion-update-flow.ts`: add `completionRestartGateSkipped` and `completionRestartGateTimeoutDetail`. The gate poll stops at once with `restart boundary not reached (agent skipped gate)` when no completion turn is active and the turn that carries this task's Done facts (`chatCompletionUpdates`, as in `completionDelivery`) succeeded or published its reply before the gate reports `waiting`. A failed or cancelled turn is not blamed on the agent, because delivery can retry it; the timeout detail names it instead. The post-restart chat navigation now uses `gotoAfterServerRestart`. - A fixture-level gate is not possible without a product hook. The native run finalizer publishes the completion reply inside its own transaction, through the issue service closure. The isolated-server entrypoint can patch only class prototypes and HTTP responses, and the reply uses no tool call or HTTP request. - `tests/runner-e2e/README.md`: document the skipped-gate failure. - Docs: use `pnpm --filter @paperclipai/paperclip-runner run build:runner-binaries` in `tests/runner-e2e/README.md`, `tests/restart-chaos/README.md`, `tests/restart-chaos/main.ts` and `doc/DEVELOPING.md`. - No product code changes. ## Verification - Unit: `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/chat-restart.test.ts tests/runner-e2e/completion-updates.test.ts tests/runner-e2e/connection-reviews.test.ts` (36 passed). - Typecheck: `node scripts/typecheck-test-projects.mjs tests/runner-e2e` passed. - Live cells, local, `--max-parallel 1 --max-automatic-retries 0`, macOS, load average 35–65: - `lifecycle-baseline.runner-codex.local.tool-review-restart`: PASS (first run found the review-card reload drift, fixed in the second commit). - `local-session-integrity.runner-acpx-claude.local.structured-question-restart-resume`: PASS. A first attempt failed before the restart with `PRP command session.open timed out` at load ~36 (environment). - `completion-updates.runner-acpx-claude.local.handoff-completion-restart`: FAIL as designed, twice (before and after the review fix), with `restart boundary not reached (agent skipped gate): completion turn … ended succeeded without the reference gate reporting waiting`. Claude skipped the wait both times, so this cell still does not exercise a restart. ## Risks - Low risk. Test harness and docs only. - The navigation retry could hide a real abort. It is limited to `net::ERR_ABORTED`, 5 attempts and the deadline, and the test still asserts that a new document loaded. - The ACPX Claude completion restart cell still does not exercise a restart when the agent skips the gate. It now reports this cause by name. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edit, browser test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Devin FoleyandPaperclip
|
fd4b197f83 |
fix(ui): hide runner selection during agent setup (#19517)
## Thinking Path > - Paperclip lets people manage AI agents for work. > - Agent setup connects a harness, model, account, and computer. > - The system already selects a qualified runner for a new agent. > - Setup still asks users to select that internal runner. > - The runner dropdown can be the only field in Advanced. > - This PR removes the dropdown and hides an empty Advanced section. > - Users keep their saved execution settings and see fewer setup controls. ## Linked Issues or Issue Description **What existing behavior does this improve?** Agent creation, onboarding, built-in agent setup, company import, and agent configuration. **Subsystem affected** The UI, setup error messages, tests, and Storybook fixtures. **Current behavior** Users can select a runner from a dropdown. Some setup screens place only that dropdown in Advanced. Errors can direct users to that control. **Proposed behavior** Remove runner dropdowns from setup and configuration. Use the existing automatic selection for new agents. Retain saved runner settings. Show Advanced only when other fields remain. **Reason and benefit** Users can configure agents without an extra internal execution choice. Setup also avoids an empty Advanced section. **Breaking changes** Users can no longer select a runner through these UI dropdowns. The API and CLI retain their runner options. Existing agents keep their saved execution settings. **Additional context** Related: [Default new agents to the qualified runner for their harness](https://github.com/paperclipai/paperclip/pull/15422). This PR changes the existing setup UI. It adds no harness or runtime default. The master merge retains the custom gateway placement from [#19510](https://github.com/paperclipai/paperclip/pull/19510). ## What Changed - Remove the shared runner dropdown and its setup/configuration callers. - Hide Advanced when it has no remaining fields. Keep command and argument settings where supported. Preserve custom gateway setup under Advanced on the Claude and OpenAI API key paths. - Keep setup failures visible and keep the retry action outside the removed control. - Update errors and documentation that referred to the removed dropdown. - Update component tests and Storybook fixtures for the new UI states. - Make Legacy first-task evals resume an API-configured setup record. Record that path separately from fresh automatic onboarding. ## Verification - Latest master integration (`187c0f6d6e5`): 276 onboarding/provider-connection tests passed across six files. UI typecheck and build passed. Token gates, query-error rendering checks, and the Storybook catalog check passed. The conflict resolution retains master's bounded connection tests and cancellation while removing the obsolete runner instruction. The first test command used the wrong working directory and failed before collecting tests; the corrected invocation from `ui/` passed. - Earlier branch verification passed: full workspace build and typecheck, 1,037 focused UI/service tests, 49 heartbeat provider contract tests, eval typecheck, and 30 map/catalog tests. - The full local `pnpm test:run` run passed all six chat integration shards (1,931 tests). It was stopped for the master merge before the complete suite finished. The [complete CI run](https://github.com/paperclipai/paperclip/actions/runs/42264662642) passed on the previous head `72e1699`. - Current head `a097c8369d079c839e78661a2206cfa62c5210e0`: [full CI](https://github.com/paperclipai/paperclip/actions/runs/42273168268) passed. PR checks total 55 passed and 3 skipped, with none failed or pending. Greptile reviewed this head at 5/5 with no actionable findings or unresolved review threads. - Live browser/provider evals have not run. The Legacy eval change has no live qualification claim. ## Risks - Removing the dropdown changes the setup UI. Existing API and CLI runner choices remain available. - Conditional Advanced sections must retain valid command and argument fields. - Legacy first-task evals now prove setup resumption. They do not prove the fresh-agent automatic default. - No schema change or data migration is included. ## Model Used OpenAI `gpt-6.1-sol`, through Codex in T3 Code. Reasoning effort: `xhigh`. The model used file editing, shell commands, and test tools. The harness does not expose the context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
187c0f6d6e |
fix(runner-e2e): prepare the artifact oracle for every suite that runs it (#19518)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner Full-Stack E2E workflow runs paid Daytona restart cells for the control-plane restart chaos campaign > - Manual run 42243751291 failed `daytona-restart-continuity.runner-codex.daytona.recover-controller` before task creation: `docker version` timed out after 10 seconds in the artifact oracle preflight > - The everyday flow requires the Python artifact oracle by case alone (`requiresEverydayArtifactOracle(caseId)`). The workflow step that starts Docker and pulls the oracle image also filters by suite, so suites that reuse `recover-controller` skip it > - On a cold RunsOn host the first `docker version` takes about 20 seconds (the everyday cell in the same run shows this), so the 10-second preflight fails when the workflow step did not run first > - This pull request gates the oracle step by case only, which matches the runtime requirement > - The benefit is that the Daytona restart cell reaches task creation instead of failing in setup ## Linked Issues or Issue Description **What happened?** Run https://github.com/paperclipai/paperclip/actions/runs/42243751291 job `daytona-restart-continuity.runner-codex.daytona.recover-controller` failed with `Artifact sandbox qualification failed before task creation: ... docker version ... timed out after 10 seconds`. The job log shows no "Prepare pinned Python artifact oracle image" step. **Expected behavior** Every cell whose flow runs the artifact oracle prepares Docker and the pinned oracle image before paid execution. **Steps to reproduce** Enumerate `runnerMatrix` cells where `requiresEverydayArtifactOracle(task.id)` is true. The cells in `daytona-restart-continuity`, `rolling-web-deployment`, `rolling-web-deployment-warm` and `remote-workspace-storage` do not match the old workflow condition. ## What Changed - `.github/workflows/runner-full-stack-e2e.yml`: the "Prepare pinned Python artifact oracle image" step is now gated by `matrix.caseId` only. The suite filter is removed. - `tests/runner-e2e/codex-ci-sandbox.test.ts`: the oracle gate test now checks every catalog cell, not only `everyday-workflows` cells, and rejects a suite filter. - `tests/runner-e2e/workflow-security.test.ts`: the pinned condition string follows the new gate. ## Verification - Red: with only the test change, `npx vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/codex-ci-sandbox.test.ts` fails on `prepares the artifact oracle for exactly the stories that execute downloads`. - Green: with the workflow change, the same command passes (13/13). - `tests/runner-e2e/workflow-security.test.ts` locally shows the same 5 failures before and after this change (`pr-trusted.yml` step counts and `chat-catalog-testops-case.yml` credential checks). They do not involve this step. - Live proof: a narrow manual dispatch after merge. ## Risks - Low risk. Cells in four more suites now run a Docker version check, a pinned image pull and an inspect before paid execution. This adds about 20 to 30 seconds per cell. The image is pinned by digest. The step has no secrets. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, GitHub CLI, file edit). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Nicky LeachandClaude Opus 5.5
|
35280516c0 |
fix(ci): set up pnpm in the Vitest selection job like other trusted jobs (#19527)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Trusted CI workflows set up pnpm in a fixed way. Node comes first, and npm audit, fund and update-notifier calls are off. > - A policy test in the Runner support suite checks every pnpm setup in `pr-trusted.yml` for this shape. > - PR #18348 added a pnpm setup to the new Vitest selection job. That setup did not have the npm overrides, and setup-node was not directly before it. > - PR CI does not run the Runner support suite. So the policy test failed on master, and nobody saw it. > - This pull request gives the selection job the same pnpm setup as the other eight jobs. > - The benefit is that every trusted pnpm setup follows one rule again, and the policy test passes. ## Linked Issues or Issue Description Refs #18348 (added the job). Refs #18625 (its description noted the "nine bootstraps where eight are expected" failure). Related: #19229 also edits the pnpm steps in `pr-trusted.yml`. **What happened?** `pnpm test:e2e:runner:unit` fails on master in two tests: "keeps pnpm bootstrap registry telemetry out of trusted workflow setup" and "installs a modern Node runtime before every trusted pnpm bootstrap". `pr-trusted.yml` has nine pnpm setups. The ninth, in `test_selection`, has no `NPM_CONFIG_AUDIT`, `NPM_CONFIG_FUND` or `NPM_CONFIG_UPDATE_NOTIFIER` override. The step directly before it is the eligibility check, not setup-node. **Expected behavior** Every trusted pnpm setup has the three overrides, and setup-node is the step directly before it. **Steps to reproduce** Run `pnpm test:e2e:runner:unit` on master. Read the two failures above. ## What Changed - `pr-trusted.yml`, job `test_selection`: add "Setup Node.js for pnpm bootstrap" directly before "Setup pnpm". Both steps use the existing eligibility condition. The pnpm step now sets the three npm overrides. Both actions keep their pinned SHAs, with version comments added. - `tests/runner-e2e/workflow-security.test.ts`: the expected count of pnpm setups is now set per workflow: 8 for `runner-full-stack-e2e.yml` and 9 for `pr-trusted.yml`. The checks on each setup do not change. ## Verification - On this branch, the two pnpm tests in `workflow-security.test.ts` pass. Before this change, they failed with "expected 8, got 9". - The other three failures in that file are fixed separately in the companion PR that repairs the rest of the suite. - `node --test scripts/ci/test-selection-workflow.test.mjs`: 4 passed. - The workflow parses as YAML. The `test_selection` steps are now: checkout, setup-node, eligibility check, setup-node for pnpm, setup pnpm, restore dependencies, select, upload, publish. - `pr.yml` calls `pr-trusted.yml@master`. So this PR's own CI runs the master copy of this workflow. The change runs in CI only after merge. ## Risks - Low risk. The new setup-node step runs again with the same Node 24 version. It adds about one second when selection is eligible. When selection is not eligible, both steps are skipped, as before. - `test_selection` has `continue-on-error: true`. If the new steps fail, the original test matrix still runs. - #19229 removes `version: 9.15.4` from every pnpm step in this file. If both PRs merge, the second one needs a small rebase on this step. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), by Anthropic, in Claude Code with tool use (shell, file edits, git, GitHub CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
a434edf991 |
fix(runner-e2e): treat macOS EPERM from a zombie-only process group as exited (#19470)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The runner E2E suite restarts the Paperclip server to prove that agent work recovers after a controller restart > - The restart supervisor in `tests/runner-e2e` owns the server process tree and escalates to SIGKILL when graceful stop takes too long > - On macOS, `kill(-pgid)` returns EPERM, not ESRCH, when every member of the group has exited but is not reaped yet > - The supervisor treated that EPERM as fatal, so a server that exited at the same moment as the SIGKILL escalation aborted the restart with "Owned tree cleanup incomplete: Error: kill EPERM" > - This pull request makes the supervisor re-inspect the process table on EPERM and skip the group only when no running member remains > - The benefit is that restart recovery cases no longer fail on a harmless exit race on macOS, while a real permission failure still fails cleanup ## Linked Issues or Issue Description No public issue exists. This describes the bug. **What happened?** The runner E2E cell `everyday-workflows.runner-codex.local.recover-controller` failed on macOS with failure class `cleanup_failure`: "Stopped waiting for isolated server restart ...: Owned tree cleanup incomplete: Error: kill EPERM". The server log shows that the graceful stop took 30 seconds, the supervisor sent SIGKILL, and the server exited with code 0 in the same instant. The restart did not start, so the case never checked recovery. **Steps to reproduce** 1. On macOS, fork a child that calls `setpgid(0, 0)` and exits. Do not reap it. 2. Call `killpg(child, SIGKILL)`. The kernel returns EPERM (errno 1), not ESRCH. After the reap, it returns ESRCH. 3. In the runner E2E supervisor, any group that becomes zombie-only between the `ps` inspection and the signal causes the same EPERM, which the code rethrows. **Expected behavior** A group whose members have all exited counts as stopped, the same as ESRCH. EPERM stays fatal while a running member remains. **Environment** macOS 26.5 (Darwin 25.5.0), arm64, Node 26.4. The behavior is the same with and without the Claude Code sandbox. ## What Changed - `tests/runner-e2e/process-tree-owner.ts`: on EPERM from a group signal, read a fresh process table. Skip the group (as with ESRCH) only when it has no running (non-zombie) member. Otherwise rethrow. `signalSnapshot` is now async. - `tests/runner-e2e/process-tree-owner.test.ts`: two regressions. One covers a group that becomes zombie-only when signaled. One proves that EPERM stays fatal while a running member remains. - `tests/runner-e2e/README.md`: document the EPERM rule next to the existing zombie rule. ## Verification - Red before the fix: `node node_modules/vitest/vitest.mjs run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/process-tree-owner.test.ts` failed with `promise rejected "Error: kill EPERM"`. - Green after the fix: `process-tree-owner.test.ts`, `server-stop.test.ts`, and `process-tree.test.ts` pass (35 tests). `pnpm test:e2e:runner:typecheck` passes. - Kernel behavior: a Python fork probe on this Mac shows that `killpg` on an unreaped zombie group returns EPERM, both inside and outside the sandbox. - Live cell: `pnpm test:e2e:runner -- --max-automatic-retries 0 --id everyday-workflows.runner-codex.local.recover-controller` on macOS. The fixed revision passed. The server stop escalated to SIGKILL, the replacement server started, and the run recovered. See the comment below for the second rerun. ## Risks - Low risk. The change is in the test harness only. - EPERM is ignored only after a fresh inspection shows that no running process holds that group ID. A process group ID cannot be reused while zombie members still hold it, so the supervisor does not signal or skip an unrelated group. - The extra `ps` read runs only on EPERM. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). Default context window. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
2b2004bf50 |
test(runner-e2e): gate completion briefs inside the run's own task directory (#19475)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agent Chat returns delegated-task completions to the chat that started them, and the runner E2E `completion-updates` suite checks this with real providers > - The restart cell (`completion-updates.runner-codex.local.handoff-completion-restart`) failed before its server restart with "Timed out waiting for worker waiting while originating chat is idle" > - Since project-optional task workspaces (#15749), a task in a project without a workspace runs in its isolated task directory. A native chat runs in its isolated conversation directory. Native Codex can read only that directory. > - The fixture still wrote its brief wait scripts into the managed project workspace and the agent home. Codex got EPERM on both scripts, wrote the note from the task record, and finished. The worker never reached the gate, so the setup poll timed out. > - This pull request makes the waits name only cwd-relative files and publishes each brief into the run's own isolated task directory > - The benefit is that the restart cell reaches its real boundary again (durable Done, source provider held at the gate, server restart) and tests restart recovery instead of a stale fixture path ## Linked Issues or Issue Description **What happened?** The runner E2E cell `completion-updates.runner-codex.local.handoff-completion-restart` timed out in setup. The run ledger shows both agents report `permission_denied` (EPERM) for `node .../wait-for-brief-*.cjs`. The worker run cwd was `<instance>/isolated-workspaces/<company>/<issue>`. The fixture wrote the script to the managed project workspace and the agent home. **What did you expect to happen?** The worker waits at the brief gate while the chat is idle. Then the cell releases the brief, holds the completion reply at its gate, and restarts the server. **Steps to reproduce** `pnpm test:e2e:runner -- --max-automatic-retries 0 --id completion-updates.runner-codex.local.handoff-completion-restart` on master `6cafcaf89b`. **Classification** Test defect. The failure happens before the restart. Workspace isolation is the intended product behavior. ## What Changed - `tests/runner-e2e/chat-stories.ts`: add `workspaceChatBrief(nonce, timeoutMs)`. It returns a bounded `node -e` command that reads and writes only cwd-relative files, plus `at(dir)` to find the gate and ready files in a workspace. - `tests/runner-e2e/completion-update-flow.ts`: the worker and completion instructions use that command. The fixture reads readiness from, and publishes briefs to, `isolated-workspaces/<company>/<issue>` for each delegated task and the source chat. The setup poll state now records the worker run cwd. Cleanup releases every wait that is still open. - `tests/runner-e2e/chat-stories.test.ts`: regression test. It runs the command from a temporary cwd, checks that the command names no host path, and checks the nonce and timeout bounds. It was red before the helper existed. - `tests/runner-e2e/catalog.ts`: bump the `completion-updates` definition to version 28 and set `workerBriefWorkspace: "run-isolated-task-directory"`. - `tests/runner-e2e/README.md`: describe the new brief location. ## Verification - `npx vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/chat-stories.test.ts tests/runner-e2e/completion-updates.test.ts tests/runner-e2e/catalog.test.ts`: 108 passed. - Live cell, native Codex `gpt-5.6-sol`, local runner: `pnpm test:e2e:runner -- --max-automatic-retries 0 --id completion-updates.runner-codex.local.handoff-completion-restart`. Results are in the Verification comment below. - Two earlier local attempts in this worktree failed in setup with `harness_test_failed`. The worktree did not have a built `paperclip-runnerd`. After `pnpm --filter @paperclipai/paperclip-runner build`, the cell passed. ## Risks - Low risk. The change is in test fixtures only. No product code changes. - The idle, busy, and multiple cells share this flow. They now also use the isolated task directory. That is the same directory the product selects for these tasks. - The agent must run the inline command from its cwd. The instructions say this, and the command fails closed (no gate) if the agent runs it elsewhere. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). Default context window. Extended reasoning on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |