Files
PaperClipAI/tests
Dotta d8b0eae687 test(evals): pin completed timeline command account for full-ten measurement (#19395)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The chat catalog needs fixed source and explicit conditions for each
measurement.
> - The latest timeline run delivered a complete account within its
original deadline.
> - Its assertion missed ordinary exitCode notation and a command with
an omitted subject inside a named worker bullet.
> - The reviewed source corrects those bounded textual checks while
retaining independent history and command proof.
> - This pull request pins that immutable source for one full-ten
measurement.
> - Original failed grades and unresolved Company timing remain visible.

## Linked Issues or Issue Description

Refs: #19392, #19086, #19388.

This follows the prior source pin. The latest timeline response
correctly described requirements, a real failed command, saved pause,
human reassignment, repair and successful verification. Its original
grade still failed one narrative assertion. This pin carries the
reviewed correction without merging the full catalog or its production
overlays.

## What Changed

- Pin source `936c3bbe0e83ca15616254f23d9023b2065fc5fb`, tree
`e5c64d67935497164b098e31e995f87ba7875d06`.
- Advance only the timeline oracle from v11 to v12 in the candidate
snapshot and manifest.
- Retain all 244 current queries, contexts, expected outcomes, model,
credentials, clocks and human/native accounting.
- Record one planned full-ten bounded measurement at native cap 2 and
zero automatic retries.
- Change only the manifest, its target assertion and the exact candidate
descriptor snapshot.

## Verification

- I verified the immutable source branch readback before I published
this pin.
- The new narrative control fails on the parent. The fixed source passes
300 timeline, native-proof and declaration controls, including all 242
lazy declarations with zero fixture callbacks.
- The response from result 5094 is accepted by the local diagnostic. The
original grade is retained as FAILED; this is not a new native PASS.
- Fresh full runner E2E types and the SDK guard over 506 inputs pass in
the qualified source graph. An earlier isolated worktree type attempt
had dependency-alias errors; the clean source qualification supersedes
that setup gap.
- Controller: `node --test
tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs
tests/runner-e2e/chat-full-catalog-testops-adapter.test.mjs
tests/runner-e2e/chat-full-catalog-testops-reconcile.test.mjs` — 40
controls pass.
- The snapshot equals the committed source descriptor bytes. All 244
rows compare equal to the prior pin except the timeline oracle version.
- Source descriptor SHA256:
`19185aa5298c93d159c3d8547661fd6ef22185e74be1c18b8497bae2ef8051c7`.
- Controller manifest SHA256:
`822494b170139a8204ff36b33891e2f4960ac0aba6800e9856f2d25dd24de7c6`.
- Launcher, protected evidence wire and the accepted 506 SDK inputs
remain unchanged. The source SDK registry stays
`efa70621e9fb42aa185f7ee616a1a726aa3eccb367e67e28817cedf26ef2ade8`.

- Exact pin head `31f47fcdfb600e6b15b35ed50c2a447d03681f06` has 56
successful checks and three configured skips. Both protected checks
pass. Greptile is 5/5 for this head with zero open threads. Paperclip
Review completed neutral at 5/5. GitHub reports CLEAN and MERGEABLE.
These checks do not qualify the unrun native experiment.

## Risks

- Original timeline result 5094 remains FAILED. All three native runs
succeeded and the account was complete in 248,876 ms, but the original
narrative check failed. Local calibration does not rewrite that grade.
- Original Company result 5093 ended without first-CEO completion. Its
model/runtime timing cause remains unresolved. This source makes no
Company runtime or latency fix claim.
- The full-ten measurement is unrun. It is a bounded experiment, not
proof of deterministic timing, reliability or an all-ten pass.
- Company retains two base human inputs plus at most one independently
verified UI approval, minimum two and maximum three native runs, and
480,000 ms. Timeline retains three native runs, one human input and
720,000 ms. No condition is waived.
- Secret journeys still require independent semantic safety review.
Metadata shape alone does not establish that a proposed name is
non-sensitive.
- The interaction-accept and API-search overlays remain separate,
reviewed, unmerged core PRs. This pin does not merge them or the full
catalog. The 196/244 admission gate stays fixed.

## Model Used

OpenAI Codex, GPT-6.1 Sol (`gpt-6.1-sol`), with reasoning, repository
tools and code execution. The host does not expose an exact
context-window size. No model or paid provider was called for local
controls.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-11-05 21:13:17 -06:00
..