Commit Graph
6613 Commits
Author SHA1 Message Date
Dotta 1bae706393 test(evals): scope private runtime metadata to declared cases 2026-10-26 20:18:06 -05:00
Dotta 55eddc4313 test(evals): add public unavailable-model native failure prerequisite 2026-10-26 20:16:04 -05:00
Dotta 5196e94ad2 test(evals): wire repository fixtures to selected private inputs 2026-10-26 20:15:45 -05:00
Dotta 9d93100f84 test(e2e): bind campaign eligibility to measured product identities 2026-10-26 20:10:00 -05:00
Dotta c80f926546 test(evals): add real case kickoff and saved card generation checks 2026-10-26 20:09:27 -05:00
Dotta 2e21d32864 test(evals): accept substantive checklist paraphrases 2026-10-26 20:09:04 -05:00
Dotta 831a45d9c6 test(e2e): cover real company packages and conflict-safe import 2026-10-26 20:09:01 -05:00
Nicky Leach e61ee4b005 fix(runner): preserve Claude user settings trust in HOME (#17826) canary/v2026.1027.0-canary.2 2026-10-26 18:02:15 -07:00
Dotta e7468282f5 test(evals): author nine case routine card and skill scenarios 2026-10-26 20:01:46 -05:00
Dotta 98bd794052 test(e2e): verify opt-in native agent trace delivery 2026-10-26 20:00:11 -05:00
Dotta 775d063d1e test(e2e): author human company setup and real first website 2026-10-26 19:56:45 -05:00
Dotta ad0907afb9 test(evals): refuse incomplete binary secret scans 2026-10-26 19:54:36 -05:00
Dotta 589ed1a30c test(evals): cover unavailable MCP setup recovery 2026-10-26 19:52:11 -05:00
Dotta 9767bca5fc test(e2e): cover governed hire revisions and budget incidents 2026-10-26 19:52:07 -05:00
Dotta 010e5c51a7 test(evals): verify operator synthetic plugin installation 2026-10-26 19:48:23 -05:00
Dotta 35f601cec2 test(e2e): expose explicit agent access grant setup blockers 2026-10-26 19:46:56 -05:00
Dotta d54e55b708 test(evals): cover admitted gateway client profile access 2026-10-26 19:46:13 -05:00
DottaandPaperclip 8f67a700a4 test(evals): qualify owned published sources for two operator chat cases
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-26 19:45:12 -05:00
Dotta 3344de11b3 test(e2e): author real company model service controls 2026-10-26 19:41:42 -05:00
DottaandPaperclip 5f8a3c8eec test(evals): erase selected secrets outside retained diagnostics
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-26 19:41:13 -05:00
Dotta 99513b6c48 test(evals): author real bounded CLI device authorization journey 2026-10-26 19:38:30 -05:00
Dotta 15989bc518 test(e2e): bind plan decisions and draft revision provenance 2026-10-26 19:37:30 -05:00
Dotta 4b54b90b90 test(evals): scope external inputs and close probe credential lifetimes 2026-10-26 19:33:54 -05:00
Dotta 5cc4d6f1fa test(e2e): cover missing links and personal announcements 2026-10-26 19:32:09 -05:00
DottaandPaperclip 9be9c65e85 test(evals): isolate declared case credentials and private provider inputs
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-26 19:31:15 -05:00
Dotta dbabcc4e0d test(evals): add real disposable plugin management journeys 2026-10-26 19:29:28 -05:00
Dotta 62c6745930 test(e2e): author real identity onboarding boundaries 2026-10-26 19:28:56 -05:00
Dotta 5da3ff9ce0 test(evals): verify restoration across a real concurrent edit 2026-10-26 19:24:13 -05:00
Nicky Leach baec3abd67 Authorize background project Git with explicit Connector consent (#17798) canary/v2026.1027.0-canary.1 2026-10-26 17:22:49 -07:00
Dotta 747eb78c7b test(evals): scan secret outputs privately and cover catalog quarantine 2026-10-26 19:22:29 -05:00
Dotta a1859e0e67 test(e2e): reconcile finite catalog authoring checkpoint 2026-10-26 19:16:40 -05:00
Dotta 50775edc48 test(evals): bind conversational approvals and race real question answers 2026-10-26 19:14:54 -05:00
Dotta 4cde887b01 Exercise property edits under actual later execution ownership denial 2026-10-26 19:14:03 -05:00
Dotta b33f3b8b64 test(e2e): author bounded privileged operator journeys 2026-10-26 19:12:54 -05:00
Dotta 24016a4ac2 Add actual two-principal notification and explicit mention journey 2026-10-26 19:09:46 -05:00
Dotta 51bd1c8138 test(evals): finish stale approvals and one-action tool review coverage 2026-10-26 19:07:50 -05:00
Dotta 483d5f2b9b test(evals): close question continuation and sidebar scope gaps 2026-10-26 19:00:15 -05:00
Dotta 0b69bdef99 Add historical final-artifact search through public migration evidence 2026-10-26 18:59:55 -05:00
Dotta f232f03d0e test(evals): exercise human CLI authorization and revocation 2026-10-26 18:54:07 -05:00
Dotta 874035eca4 Add genuine generated website delivery and retained revision journeys 2026-10-26 18:51:39 -05:00
Dotta 0f95be3f07 test(evals): author real expired-session browser boundary 2026-10-26 18:48:58 -05:00
Dotta 4c4408b85d Use immutable production attachment origin for completion evidence 2026-10-26 18:46:48 -05:00
Dotta 6a28921e39 test(evals): cover human profile and company lifecycle journeys 2026-10-26 18:44:07 -05:00
Dotta 80ba3b2d35 Add real stale assignment read and execution ownership conflict journey 2026-10-26 18:43:30 -05:00
Dotta b1f30d7d51 eval: add real operator companions and finite disabled-chat boundary 2026-10-26 18:39:51 -05:00
Dotta 6e5df53427 Add genuine interrupted execution and continuation history journey 2026-10-26 18:39:50 -05:00
Dotta ba04d3761b test(evals): retain identity probe credentials for one attempt 2026-10-26 18:37:03 -05:00
Dotta 55ad3f3498 Add run-bound agent feedback and authorized human commentary journey 2026-10-26 18:35:40 -05:00
Dotta 25328a9b17 test(evals): verify workspace closure retains promised work and scope 2026-10-26 18:34:14 -05:00
6874fed05c feat(adapters): refresh model catalogs and sandbox CLI pins (Oct 23 audit) (#17809)
<!-- entire-trail-link-start -->
https://entire.com/gh/paperclipai/-/trails/26/changes/paperclip/26
<!-- entire-trail-link-end -->

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents run on coding harnesses. Each adapter has a model catalog,
and sandboxes contain pinned CLIs.
> - Upstream providers released new CLI versions and new models. OpenAI
retired some Codex models for ChatGPT sign-in.
> - A stale catalog hides valid models. A misleading error makes
operators upgrade a CLI that is already current.
> - This pull request adds verified OpenCode model IDs, names Codex
ChatGPT retirements in the environment Test, and refreshes the sandbox
CLI pins.
> - The benefit is that operators can select current models and get the
correct reason when a retired model fails.

## Linked Issues or Issue Description

Refs #16708 (adds Claude Haiku 5.5 to `claude_local`; this PR does not
duplicate it).
Refs #17245 (moves the Codex runtime to 0.163.0; this PR does not change
Codex pins).

**What happened?**

OpenAI retired `gpt-5.5` (October 14, 2026) and `gpt-5.4` /
`gpt-5.4-mini` (August 31, 2026) from Codex with ChatGPT sign-in. A
saved agent with one of these models and ChatGPT sign-in now fails. The
legacy environment Test says that the plan, the rollout, or an old CLI
can be the cause. The Paperclip Runner Test gives generic account and
runtime advice. The real cause is the retirement.

**Expected behavior**

Both Tests give the retirement date and the documented replacement.
API-key users keep the models.

**Proposed change**

Record the verified retirements. Add the new verified OpenCode IDs.
Refresh the sandbox CLI pins with verified digests. Record the audit.

## What Changed

- Codex: add a table of models retired from Codex with ChatGPT sign-in
(`gpt-5.5` → `gpt-6.1-sol`, or `gpt-6-luna` on Free and Go; `gpt-5.4` →
`gpt-6-sol`; `gpt-5.4-mini` → `gpt-6-luna`). The shared
`codexChatGptRetirementHint` builds the hint. The legacy CLI Test
(`codexHelloProbeModelRejectionCheck`) and the Paperclip Runner Test
(`testNativeRunnerAuthentication`) both use it when the backend rejects
one of these models. For other failures, the Paperclip Runner Test hint
stays the same. The picker keeps the models for API-key users.
- OpenCode: add `anthropic/claude-haiku-5-5` and `google/gemini-4-argon`
to the remote fallback catalog. The Paperclip Runner OpenCode picker
also uses this catalog.
- Evaluation image sandbox layer (`docker/daytona-runner`):

  | CLI | Before | After |
  | --- | --- | --- |
  | Grok CLI | 1.0.46 | 1.0.70 |
  | Gemini CLI | 0.62.0 | 0.65.0 |
  | Kimi Code | 2.1.1 | 2.3.3 |
  | Cursor CLI | 2026.10.01-e373342 | 2026.10.22-27dcf5f |
  | GitHub CLI | 2.102.0 | 2.103.3 |

- Add `doc/adapter-model-audit-2026-10-23.md` with sources,
compatibility findings, and deferred items.

No model in this PR needs a newer Codex, Claude, or OpenCode runtime
than the current pins. Claude Haiku 5.5 needs Claude Code 2.1.293 or
later; #16708 records that floor, and this PR does not add it.

## Verification

- `vitest run src/index.test.ts src/server/test.remote.test.ts
src/server/acp.cli-version.test.ts` in `packages/adapters/codex-local`:
41 passed.
- `vitest run src/services/native-runtime/setup-readiness.test.ts` in
`server`: 98 passed. These include the two new Paperclip Runner Test
cases: a retired model shows the retirement hint, and a model that is
not retired keeps the generic hint. The retirement case fails on the
earlier code. Run with the host's empty `ANTHROPIC_BASE_URL` unset; see
the baseline note in the audit doc.
- `vitest run` in `packages/adapters/opencode-local`: 87 passed.
- `vitest run src/__tests__/adapter-models.test.ts` in `server`: 27
passed.
- `tsc --noEmit` in `packages/adapters/codex-local` and
`packages/adapters/opencode-local`: passed.
- Cursor archive SHA-256
`85296944e4600a83c4b764f578d1502bf9c416db200d33a593e8f3ecabfdc294` is
computed from the versioned Linux x64 tarball. The extracted binary
reports `2026.10.22-27dcf5f`. On October 26, 2026,
`https://cursor.com/install` still resolves this version.
- GitHub CLI SHA-256
`4c806978ffce1cec70853302a56d70ac6da4f19bae7adca4e1a8ab00633dd91d`
matches `gh_2.103.3_checksums.txt`.
- Grok CLI 1.0.70 and Kimi Code 2.3.3 installed from npm report those
versions. The flags that Paperclip uses (`-m`, `--effort`, `grok
models`; `-m`, `-p`, `--output-format`, `acp`) are present in `--help`.
- `git merge-tree` with #16708 (head `3a06740e9`) and #17245 (head
`41f9c611f`): this PR shares no files with either PR. The two PRs can
merge in any order.
- Not run in the authoring environment: the full `pnpm -r typecheck` and
`pnpm build` (no Rust toolchain), the full server `tsc` (memory limit),
and the image build. CI runs these.

## Risks

- Low risk for catalogs: the IDs are additive suggestions. No saved
selection, default, or spending setting changes.
- The Paperclip Runner Test changes only the hint text, and only for a
ChatGPT rejection of a retired model. Check codes and pass/fail results
do not change.
- Sandbox CLI pins:
- Kimi Code 2.2.0 and 2.3.0 remove the `kaos` MCP executor and require
qualified plugin skill names. Paperclip uses neither. Kimi Code 2.3.2
stops automatic swarms in `kimi -p` and ACP sessions.
- Gemini CLI 0.65.0 makes workspace settings read-only in untrusted
folders. Paperclip writes home settings and sets
`GEMINI_CLI_TRUST_WORKSPACE` for remote runs.
- xAI publishes no Grok CLI changelog. Grok 1.0.70 behavior changes are
verified only against the CLI reference and `--help`.
- The pins apply only after a new image is built and promoted. The fleet
sandbox image definition must get the same pins.
- Rollback: revert this PR.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell,
web research, repository search).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Paperclip <noreply@paperclip.ing>
canary/v2026.1027.0-canary.0
2026-10-26 16:32:13 -07:00