Files
PaperClipAI/tests
DottaandPaperclip 46414fcfa4 test(evals): pin timeline and governed Company approval at native cap two (#19381)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The chat catalog uses a fixed source and explicit execution
conditions.
> - A correct timeline response failed because its completed edit and
attached command result were not recognized.
> - Company credential setup can require a real human approval after the
secure save.
> - The controller must declare that optional approval and its run
bounds before it admits the source.
> - This pull request pins the reviewed source and caps native catalog
overlap at two.
> - The next run can test these conditions without treating local
controls as native acceptance.

## Linked Issues or Issue Description

Refs: #19357, #19086, #18971, #19093.

This follows the prior fixed-source catalog pin. It changes only
controller metadata, admission, tests, and load policy. The complete
catalog and separate core feature PRs remain unmerged.

## What Changed

- Pin immutable source `fe547be84826b4611d3ba9ee1a7d7aa0bd0a2843`, tree
`a23723ea5244e6ce8cee11e9c4b476acb6978fcd`.
- Advance only timeline v10 to v11 and Company v11 to v12. Add the
declared Company proposal helper to its source closure.
- Declare Company minimum two and maximum three native runs. Retain two
base human inputs and permit at most one independently verified UI
approval. Retain its original 480000ms attempt.
- Carry the literal Company-only approval marker from the descriptor
into the manifest. Reject foreign markers, changed bounds, and a
mismatch with static source task metadata.
- Cap native catalog execution at two workers. Keep the external lane
serial and retain zero retries.
- Keep all244 requests, contexts, expected outcomes, model, clocks,
credentials and other case bounds unchanged.

## Verification

- Source:421 focused controls across seven suites pass. Full runner E2E
TypeScript check passes.
- Explicit model-free approval UI canary passes one test with three
local scenarios. It checks accepted approval, wrong human, and failed
response. The first sandbox attempt stopped at Chrome launch; the
allowed execution passed.
- Controller:40 manifest, adapter and reconciliation controls pass.
- SDK506 inputs remain byte-identical to the accepted registry
`efa70621e9fb42aa185f7ee616a1a726aa3eccb367e67e28817cedf26ef2ade8`.
- Launcher and trusted private evidence wire remain byte-identical.
Source contract/flow accounting changes are explicit.
- Source descriptor SHA256:
`50e90d8c02ad57f444e27b522efe694db9734d47bdbc3a058475af77bac8b2ca`.
- Controller manifest SHA256:
`0447315601ed21ba15d6ea0ecd7fb89bc79ad6262db2594cd12ac774aa8d6c21`.
- Review the two changed rows. The next planned experiment selects
timeline and Company once at cap2 before a full ten-case run. No native
run on this source has passed yet.

- Exact controller head `d49dfc59ab7fddfe1a1de7e81bdb9c4e13400f30`:
normal CI has56 successful checks and three configured skips. Protected
`ci / verify` and `ci / e2e` both pass. Greptile5/5 and zero open
threads are verified for this head.

## Risks

- Original timeline5056 and Company5057 remain FAILED. Timeline5056
supplied the correct account; the local revised check passes its
diagnostic only. Company5057 saved the secret but left the binding
approval pending.
- Company native3 completion within480s is unproved. This change does
not raise its clock or claim to fix latency.
- Two-worker execution is a changed load condition. It does not prove a
concurrency cause, reliability, or eight-worker qualification.
- Secret journeys require independent semantic safety review before pass
credit. The documented closing-comma warning and arbitrary
metadata-prefill classifier limitations remain follow-ups in the
functional secret PR; a native failure is not waived.
- Candidate core PRs remain unmerged. The full-catalog196/244 admission
gate remains fixed. Local controls do not qualify all244 journeys.
- Local checks use Node26.4.0. They do not replace actual AWS runtime
qualification.

- The separate Paperclip Review app remains queued while its assigned
task waits for maintenance. It has no completed assessment. GitHub
reports `UNSTABLE` and `MERGEABLE`; this is not an all-checks PASS
claim. No review retry or staging mutation was performed.

## Model Used

OpenAI Codex, GPT-6.1 Sol (`gpt-6.1-sol`), with reasoning, repository
tools and code execution. The host does not expose an exact
context-window size. No model or paid provider was called for local
controls.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-05 19:06:42 -06:00
..