Commit Graph
6 Commits
Author SHA1 Message Date
DottaandPaperclip 76ee5d0886 fix(evals): preserve singleton reports and pin reviewed chat repairs (#18749)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The chat eval lane runs selected cases from one immutable candidate.
> - The finalizer must authenticate every original projection and
publication.
> - The pinned artifact action places one matched artifact at the
download root.
> - The finalizer expects named child directories, so a valid one-case
run fails before final coverage verification.
> - This PR supports both valid layouts and binds one reviewed repair
candidate.
> - The lane keeps exact selection, source, digest and publication
checks.

## Linked Issues or Issue Description

Refs: #18740, #18736, #18742, #18734, #18717.

This PR is a proposed replacement for the source pin in #18740. That PR
remains open. The evaluated candidate contains all five repairs from
that prior cut and five further reviewed repair conditions. The product
PRs remain unmerged. Merge authority for this replacement is pending;
this PR does not dispatch a run.

A completed one-case run has a verified original PASS and a verified
publication. Its final aggregate fails because the pinned artifact
action flattens a single matched artifact. The same path assumption also
affects the later publication verifier. The final checks must still run;
the repair must preserve their identity and digest checks.

## What Changed

- Bind the finite candidate `ab2019e976076ec0eb60b026844f3d9211cb0208`
through the manifest and exact descriptor snapshot.
- Retain the original 244 case identities and the explicit cumulative
definition overrides. Keep the historical definition source separate
from the current candidate.
- Add the reviewed source dependencies for real hire-review readiness
and the strict later-human-intent review overlay from unmerged #18742.
Preserve all original clocks, queries and model settings. Keep the
earlier Studio144 accounting change explicit.
- Document one current target and identify the prior Source5 cut as
unmeasured.
- Preserve named singleton download destinations for both bundles and
publications. Authenticate every bundle before gathering its safe
projection. Reject mixed, foreign, missing and symbolic-link layouts.
Accept only the producer's exact closed once-marker schema. Do not
export or upload results again.

## Verification

- Candidate controls: 217 pure checks in 13 files; 242 real factory
declarations with zero fixture callbacks; 42 admission controls;
unchanged 506-input SDK registry; focused actual database-row type
check.
- Combined trusted pin and layout controls: 124 Node checks in seven
real files, plus source-byte closure for all 244 singleton evidence
policies. The unchanged parent reproduces both layout failures. Actual
producer-marker, multi-case and malformed-layout controls pass.
- The two actual PostgreSQL route regressions on core #18742 head
`61996a96dd09e90a5788e8711a94fc5ae631f425` ran and passed. That core PR
remains unmerged. This is not chat-case acceptance.
- The independent layout logic peer and the whole final eight-path PR
source peer passed. Current-head CI completed with 48 successful jobs
and one skip. Both required checks passed. All 60 exact-head checks
completed: 55 success, two neutral and three skipped. The separate
Windows Rust check passed. Greptile and Paperclip Review both scored
5/5, with no unresolved threads.
- The optional raw-order note is [disposed with source
evidence](https://github.com/paperclipai/paperclip/pull/18749#issuecomment-6598680442).
The admitted selector writes canonical order before hashing. A changed
raw order fails closed before copying. The checked code remains
unchanged.
- Full local candidate typecheck and build are not claimed. Installed
workspace dependency gaps prevent that full local check. The existing
remote source and build preflight must pass before a native query.
- No new model run or TestOps upload was used to prepare this PR. Prior
attempts and grades stay unchanged.

## Risks

A permissive layout repair could hide missing or foreign publications.
Exact selected names, singleton counts, source identities, hashes and
final readback checks must remain enforced. The layout helper must
reject symbolic links and mixed shapes.

The active default-branch ruleset requires CODEOWNERS review for the
workflow. Normal merge is currently blocked on that review. This PR does
not enable auto-merge, change branch rules or dispatch an eval.

The candidate contains explicit unmerged product overlays. It is not a
core release or a full-catalog qualification. The other five SDK cases
remain held. SDK115 completion remains unknown. Provider quota failures
are not fixed by the skill observer change. No credentials, caps, runner
permissions or workflow admission references are added.

## Model Used

OpenAI `gpt-6.1-sol`, with code execution, tool use and independent
agent source review. The host does not expose the exact context window.
No model eval was run to prepare this PR.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes:` / `Closes:`
/ `Refs:` OR (b) described the issue in-PR following the relevant issue
template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub links)
- [x] My branch name describes the change and contains no internal
ticket id
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-03 06:03:25 -06:00
Dotta 2aeceb963d fix(evals): pin reviewed Source4 chat repair candidate (#18736)
## Thinking Path

> - Paperclip manages AI agents and their work.
> - The chat catalog checks real user journeys through a fixed
candidate.
> - The current candidate retains several known fixture and observer
defects.
> - A new source must bind each repaired declaration before paid work
starts.
> - This pull request pins the reviewed Source4 candidate and its exact
descriptor snapshot.
> - The next measurements can test those repairs without changing old
results or relaxing private evidence guards.

## Linked Issues or Issue Description

Refs #18726. Related core changes remain separate and unmerged: #18717
and #18734.

**What happened?**

The active lane still uses Source3. It cannot test the accepted primary,
app review, CLI, timeline, plan review, hire fixture and SDK input
repairs. Old results must remain historical.

**Expected behavior**

The trusted lane must bind the new immutable candidate, all 244 case
identities, exact current declarations and private source hashes. It
must retain the original definition provenance and explain changed
conditions.

**Steps to reproduce**

1. Compare the current manifest with candidate
`51ef450754370c8a8f2cbe4bd18e1536bbc93966`.
2. Check each of its 242 declarations against the checked descriptor
snapshot.
3. Run the focused manifest, contract and reconciliation controls. Run
the selected real cases only after normal source delivery and a fresh
execution guard.

## What Changed

- Pin candidate `51ef450754370c8a8f2cbe4bd18e1536bbc93966` and its exact
descriptor snapshot. Keep all 244 identities and aliases.
- Record cumulative original-to-current field overrides. Keep the
original `de094` definition provenance and case limits. Studio144
retains its earlier explicit 2..3 accounting revision.
- Record the reviewed SDK expected build-lock correction and the
candidate-only full proposal disclosure from unmerged #18734. Keep
compiler/parser/Zod identities, privacy guards and the other SDK holds.
- Update the static owning controls and identify current versus
historical sources in the lane guide. Runtime JavaScript, declarations,
workflow permissions and private transfer code are unchanged in this PR.

## Verification

- 47 focused Node controls passed: contract, manifest and idempotent
reconciliation.
- Candidate controls: 293 pure tests across 16 files passed. All 242
actual factory declarations and 42 admission controls passed without
fixture callbacks.
- The final candidate SDK registry check matched all 506 inputs. The
actual Git launcher, checked snapshot, three private protocol files and
all 244 singleton evidence policies matched.
- The candidate product disclosure is byte-identical to the reviewed
#18734 file. Its token gate passed. The separate core change retains 115
React controls.
- Independent full PR and integration source review passed before
publication.
- Full local workspace typecheck, application build and model execution
were not run for this pin. The existing workspace dependency limitations
remain recorded. Latest-head CI `41586212647` passed on attempt 1: 48
passed and 1 skipped. Both required checks passed. Greptile reviewed the
exact head at 5/5 with no open threads.

## Risks

- These are new candidate measurements. A source-qualified repair does
not prove a user journey passed.
- SDK115 must first pass real admission with its original
300-second/one-native limit. The other five SDK cases and plugin558
canary certificates remain held.
- The CLI parser rejects unresolved placeholders and foreign values. It
executes the actual returned commands without substitutions. The old
failed command remains failed.
- The core guidance and review-card PRs remain unmerged. This pin
changes neither core merge authority nor the previous full-catalog
acceptance gate.
- A running Source3 launch remains pinned to its immutable controller
and candidate. The next model wave must wait for terminal accounting and
cleanup guards.

## Model Used

OpenAI `gpt-6.1-sol`, with code execution, tool use and independent
agent source review. The host does not expose the exact context window.
No model eval was run to prepare this PR.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-11-03 00:04:17 -06:00
DottaandPaperclip 4461905b92 fix(evals): bind repair launcher and Source3 candidate contracts (#18726)
## Thinking Path

> - Paperclip manages AI agents and their work.
> - The private chat catalog tests real user requests through TestOps
and AWS.
> - Each test must bind its immutable candidate and private evidence
owner before it receives model credentials.
> - The previous source pin changed the candidate launcher but kept the
old launcher digest in the full-catalog evidence policy.
> - The private guard rejected the 23-case launch before any original
test or model-key step began.
> - This pull request binds the full-catalog launcher to its reviewed
candidate and selects the next reviewed repair source.
> - The result is an explicit source contract for new measurements, with
the old grades and privacy gates preserved.

## Linked Issues or Issue Description

Refs #18721. Related: #18710 and #18717. The core guidance PR #18717
remains unmerged. A duplicate search found no open PR for this launcher
binding defect.

**What happened?**

The full-catalog evidence policy used the legacy pilot launcher digest
after the candidate launcher changed. All first eight case jobs failed
the private binding step. Their original Execute and model-key steps
were skipped. The operator cancelled the remaining jobs once. This was
an operational failure, not a feature result.

**Expected behavior**

Every singleton policy must use the launcher digest from the same
reviewed manifest as the immutable full-catalog target. A final source
check must compare the digest and private protocol files with the exact
Git target before submission.

**Steps to reproduce**

Compare the old evidencePolicy digest with launch.ts from target
36dff2e. The old owning regression fails. The corrected full-lane policy
emits the digest of launch.ts from target 43c44ccc. The legacy pilot
digest remains unchanged.

## What Changed

- Pin the full catalog to candidate
`43c44ccc823a8c972b247fa7159281887e6f5255` and its exact launcher
digest. Keep the strict private prepare guard and pilot digest
unchanged.
- Bind the actual 242-row candidate descriptor snapshot and the explicit
changed definition fields. Preserve all 244 case IDs, aliases and
original definition provenance.
- Select the reviewed source repairs for skill import guidance,
immutable skill versions, app review navigation, authorized app testing,
adapter prerequisites, the isolated announcement fixture and bounded
diagnostics. These are new candidate measurements.
- Record the one native-accounting change: Studio144 has a 2..3 envelope
and requires three genuine settled source-bound turns. Its query, model,
human count and 540-second case limit stay fixed. All other native
bounds and all original case clocks stay fixed.
- Keep the SDK scanner, compiler/parser identities and canary
certificates unchanged. The public SDK source-input registry is
synchronized; this does not certify the runtime or release held SDK
cases.
- Update the current-target documentation and preserve dated source
history.

## Verification

- The old full-catalog policy fails the new regression with the stale
launcher digest. The corrected policy passes for all 244 singleton
declarations.
- Current CI tests hash the checked exact candidate launcher source
bytes and bind their target and Git blob OID. The previous test accepted
an advanced target with a stale launcher; the corrected test rejects
that mutation. The candidate remains unchanged.
- 62 focused Node controls pass across the manifest, API adapter,
reconciliation and trusted contract owners.
- The source-bound final-GO helper reads the exact Git target launcher
and all three private protocol files. It verifies every singleton policy
against those bytes. All 244 pass.
- The unchanged actual private prepare command passes with the exact
candidate bytes. Mutated launcher bytes, a missing launcher and a
wrong-target policy fail before outputs. No service or provider
credentials are passed in these controls.
- Candidate verification: 294 pure controls, eight Git/FS updater
controls, actual 242 factory declarations with zero fixture callbacks,
and 42 admission/legacy controls pass. SDK registry check reports no
change on the final candidate.
- Prior head6bda80f passed PR CI41577494646 on attempt1:48 jobs passed
and1 intentionally skipped. Both required checks and Greptile5/5 passed.
Two late review findings then required declaration synchronization and
removal of an unused exported helper. Final head09719566 then passed PR
CI41579310890 on attempt1:48 jobs passed and1 intentionally skipped.
Both required checks and exact-head Greptile5/5 pass. The whole ten-file
diff received independent review after both late findings were fixed.
- One actual TypeScript declaration consumer reproduces
missing-export/property errors on the prior declarations and passes
after synchronization (installed TypeScript6.0.3; isolated consumer
only). The finalGO keeps direct exact-source comparisons; no unused
guard export remains.
- Local full graph typecheck/build and native browser/model execution
were not run for this trusted metadata change. Workspace dependency
output prevents a clean local full typecheck claim. Latest-head CI is
required. No new feature PASS is claimed from these controls.

## Risks

- This advances a private evaluation candidate. It does not merge the
candidate's core guidance changes to master. Existing result history and
the catalog qualification gate remain unchanged.
- The new Studio144 accounting and app492 context/outcome are explicit
changed conditions. They must not be credited as passes under the
original definitions.
- SDK input synchronization preserves all scanner guards. Runtime canary
certificates still need real qualification; the other five SDK cases
remain held.
- The isolated announcement fixture contains one public synthetic JSON
object. Production announcement feeds and the global index are
unchanged.
- The cancelled 23-case launch must be reported as operational BROKEN
rows with unknown original grades. It must not be retried under the
unchanged source.

## Model Used

OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) assisted with source integration,
code, tests and review. The model used tool execution and repository
inspection. The host did not report the context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-02 22:54:16 -06:00
Dotta fea6163811 test(evals): pin synchronized chat repair declarations and definitions (#18721)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The chat catalog runs a fixed candidate through the trusted TestOps
lane.
> - The first repair source exposed fixture, observation and
evidence-readiness defects.
> - Changed declarations also need matching static runtime descriptors.
> - This pull request pins one corrected candidate and records each
changed definition.
> - New measurements keep their source and original outcomes separate
from past results.

## Linked Issues or Issue Description

Refs: #18710, #18693, #18680, #18647.

**What happened?**

The first repair candidate left five static descriptors out of sync with
its actual declarations. The real no-callback admission check rejects
them. A five-case attempt was cancelled before any case or model step.
The earlier pin note overstated remote coverage: importing the catalog
did not load every actual factory.

**Expected behavior**

Every repaired declaration must pass the real factory admission check
before a new batch. The trusted pin must record exact source and
definition changes. Existing case identities, prior measurements and the
full-catalog pass gate must remain intact.

**Steps to reproduce**

Run `node --import ./cli/node_modules/tsx/dist/loader.mjs
tests/runner-e2e/chat-catalog-consistency.ts` on the old candidate. It
rejects five declarations. The corrected candidate admits all 242
declarations without invoking a fixture. The existing legacy-cell
scoring and selection controls also pass.

## What Changed

- Pin candidate `36dff2eab3dfb6de6efb4fd2ce6c1a58f8a5f383`. Its
[candidate-only
delta](https://github.com/paperclipai/paperclip/compare/9ab0f951a5b5990c266ef4456610e0a0775aa334...36dff2eab3dfb6de6efb4fd2ce6c1a58f8a5f383)
remains separate from master application code.
- Record the real descriptor digest and all 19 declaration-field changes
across 18 cases. Check them against the exact candidate static
descriptor snapshot, including setup policies. The runtime loader still
rejects drift.
- Keep original definition values and record 19 manifest-field changes
across 15 cases. These include the clarified Fast response request,
reviewed goal/skill navigation outcomes, and scoped oracle/source
ownership changes.
- Keep unchanged all 244 ordered identities, model, case clocks,
native-run bounds, private evidence rules, workflows and reporting
authority. The original ledger and historical results are unchanged.
- Require actual factory admission and legacy controls before each
repair-batch GO. SDK audit capture remains private diagnosis only; it
does not waive scanner rejection or qualify the other SDK cases.

## Verification

- Candidate focused controls: 561 passed across 32 pure test files.
- Real lazy factory admission: 242 declarations passed, with zero
fixture callbacks. The prior source failed 18 declarations before
descriptor sync. The two-file admission and legacy scoring suite passed
all 42 controls.
- Trusted manifest, reconciliation and adapter controls: 38 passed.
Contract and dispatch controls: 23 passed.
- Independent source reviews cover the repair slices, descriptor sync
and this metadata pin.
- `git diff --check` passed.
- No local app, DB, native or model run was started for this metadata
PR. Full harness typecheck/build is not qualified: existing workspace
dependency errors remain outside these pure checks. The established
credential-free remote Runner build is still required before paid
queries. It does not typecheck the full harness graph.
- Initial head 90d4ecf passed first-attempt CI (48 passed, 1 skipped)
and Greptile 5/5. A later review found stale current-target prose and an
unchecked setup-policy provenance assertion. The corrected head adds the
exact candidate descriptor snapshot, computes its digest, and compares
every declared field. A wrong-policy mutant accepted by the parent is
now rejected, as is a stale digest. Correct metadata controls pass all
38 cases. Latest-head CI41572232196 passed48 jobs with1 skip after one
approved failed-only recovery. The first attempt failed one browser
cleanup cancellation with PostgreSQL row-lock timeout; the lock holder
remains unknown. No server or timeout change was made. All46 successful
executions kept their original full step and execution timestamps. Only
the failed browser shard and dependent e2e summary reran. Both required
checks passed, current-head Greptile is5/5 and all review threads are
resolved. An optional review may remain queued; no claim is made that
all optional checks finished. No corrected-candidate feature pass is
claimed.

## Risks

The candidate changes selected setup, observation and oracle conditions.
Each new result must retain its exact source and declared changes; it
cannot replace an old failed result. The Fast response request and
goal/skill outcome changes are explicit new definitions. Static TestOps
prose may still show the original definition until a separate supported
metadata reconciliation. SDK scanner findings remain unknown until the
private audit is captured; the scanner and evidence bounds remain
enforced.

## Model Used

OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) with reasoning, source editing and
tool use. The exact context window is not exposed by this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either linked existing issues or described the issue in-PR
following the issue template
- [x] I have not referenced internal or instance-local Paperclip issues
or links
- [x] My branch name describes the change and contains no internal
Paperclip ticket id
- [x] I have run focused tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All required Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-11-02 21:55:33 -06:00
DottaandPaperclip e5e22148e4 test(evals): pin the reviewed chat catalog fixture repairs (#18710)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The chat catalog uses a trusted TestOps workflow and an immutable
candidate source.
> - The retained first pass found fixture failures before real feature
work could start.
> - Candidate repairs must use the same source checks as the original
lane.
> - This pull request advances only the fixed catalog target and its
source note.
> - New AWS measurements keep the original case queries, limits, graders
and reporting path.

## Linked Issues or Issue Description

Refs: #18542, #18647, #18680, #18693.

**What happened?**

The full-catalog lane remains pinned to the original candidate. A repair
candidate cannot run through the trusted TestOps selection without a
reviewed pin change.

**Expected behavior**

TestOps must select the reviewed immutable repair source. It must keep
the existing case identities, model, clocks, private evidence controls
and original grading.

**Steps to reproduce**

Inspect the full-catalog target in the manifest. The trusted contract
uses that exact commit for selection, source checks, build and export.

## What Changed

- Advance the immutable full-catalog target to
[`9ab0f951a5b5990c266ef4456610e0a0775aa334`](https://github.com/paperclipai/paperclip/commit/9ab0f951a5b5990c266ef4456610e0a0775aa334).
[Candidate-only
delta](https://github.com/paperclipai/paperclip/compare/3b66c3c144360ddf32714a0742fd774b8ceafb16...9ab0f951a5b5990c266ef4456610e0a0775aa334).
- Preserve all original queries and limits. Record company observer
source ownership and the secret-entry v2 oracle as explicit metadata
overrides against the original definition source.
- Explain SDK staging, scoped human membership, original-clock
observation and exact controller-secret baseline conditions. Preserve
the original failed measurements.
- Leave the application and held core fixes off master. This change does
not copy the candidate app or catalog implementation to master.

## Verification

- Candidate integration: 167 focused pure tests passed for SDK custody,
membership, feature and privacy controls.
- Trusted target consumers: 46 manifest, full-catalog adapter and
dispatch contract tests passed.
- The candidate has no diff to the original YAML queries or descriptor
bytes.
- `git diff --check` passed.
- Local full typecheck, application build, DB and model execution were
not run for this two-file pin change. Worker typecheck attempts hit
existing missing workspace dependencies and are not recorded as passes.
The trusted AWS source check and build must pass before any native
query. It imports the real candidate catalog and checks every descriptor
before provider access. It does not typecheck all harness TypeScript
files.
- Initial CI41563164012 passed48 jobs with1 skip. A later review
requested the exact target assertion; the valid-SHA mutant is now
rejected. Corrected-head CI41564795118 passed48 jobs with1 skip
on2226f093. Both required checks passed. Greptile and the follow-up
source review are5/5 with no unresolved threads. No new candidate case
result is claimed here.

## Risks

A new fixture condition can change a measured outcome. Each result
records the actual candidate and trusted workflow revision. Original
failed measurements remain unchanged. The model, case bounds,
independent feature requirements, private custody, reporting identities
and automatic retry count remain fixed. This pin applies to future
full-catalog runs, so operators must not infer that every catalog case
has been qualified. The inherited source-ref label is not source
authority; the exact checked commit is authoritative.

## Model Used

OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) with reasoning, source editing and
tool use. The exact context window is not exposed by this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes:` / `Closes:`
/ `Refs:` OR (b) described the issue in-PR following the relevant issue
template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub references)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id
- [x] I have run focused tests locally and they pass
- [x] I have added or updated tests where applicable; the existing
target consumers cover this data-only change
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-02 20:28:06 -06:00
DottaandPaperclip 774ddc7271 test(evals): add the full chat catalog to TestOps (#18542)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Chat feature evaluations check real user requests through the
product.
> - The YAML catalog has 244 scenarios, but the shipped TestOps
completion route selects only three.
> - The preserved catalog already supplies the execution mappings and
independent oracles.
> - A larger TestOps plan needs exact identities, bounded selection and
source-bound private evidence.
> - This pull request adds that plan route without merging the catalog
app into master or replacing its grades.
> - Users can run the full catalog or a selected subset and see the
actual outcome and prerequisites.

## Linked Issues or Issue Description

Refs: #18516 and #18515. This extends their completion pilot and reuses
its trusted reporting path.

**What existing behavior does this improve?**
The existing three-case TestOps completion workflow.

**Current behavior**
The remaining catalog scenarios cannot be selected through that TestOps
job. The one-case backlog plan has a separate strict singleton contract.

**Proposed behavior**
A separate candidate-scoped plan maps all 244 scenarios. Existing case
aliases, jobs, plans and result history stay intact. Registration is
distinct from a measured pass.

## What Changed

- Add a source-derived 244-row manifest. It binds the 242 domain
descriptors and two legacy tasks to the unchanged YAML queries,
contexts, outcomes and original bounds. The JSON snapshot avoids
importing the unmerged domain tree into master.
- Reconcile exact case identities and a dedicated full-catalog plan.
Reuse legacy aliases, including the existing backlog case, without
renaming them. Preserve history and stop on conflicting aliases or
uncertain writes.
- Extend the shared completion adapter with a separate trusted
full-catalog workflow context. Return a compact observed binding that
fits one 48 KB protected variable. Keep the three-case pilot contract
intact.
- Add a fixed-source manual workflow. Small selections run serially.
Larger selections allow eight independent workers and one serial
external-input worker. Each worker and report reader keeps a singleton
approval and exact-key AWS session. The eight-cell private authority
limit is unchanged.
- Validate and merge metadata-only singleton projections before one
protected Allure upload and complete paginated readback. Preserve failed
originals and cleanup failures beside passed results. Keep screenshots
and raw evidence private.
- Discard a changing full-catalog page-count snapshot and retry within
the same observer bounds. Malformed page fields still fail at once. A
pass still needs one consistent exact selection and all final evidence
checks.
- Keep workflow source and app source separate. The full lane builds
candidate `3b66c3c144360ddf32714a0742fd774b8ceafb16`. Its only changes
from the existing `de0941835cc8f0d01c982574ee258e110babdbeb` pin are
finite TestOps host admission and its offline test. This is not
current-master app qualification.
- Give full selections above three cases a bounded metadata-read
allowance above the existing 120-second deadline ceiling. Readback has
two upper bounds: at most twelve polls with five-second pauses, and a
size-derived deadline of at most 374 seconds, whichever exhausts first.
There is no guaranteed minimum wait or processing SLA. Small selections
and pilot deadlines stay unchanged. Final identity, grade and evidence
predicates stay intact.
- Document the real readiness limits and setup path. Five coverage gaps,
six evidence exceptions, eighteen additional-credential declarations and
thirty-seven external-input declarations overlap. They are not new pass
claims.

## Verification

- Related Node controls passed: pilot API and dispatch, 244-row
manifest, idempotent reconciliation, full selection, three-page result
readback, source guards and partition/output-size bounds. The latest ten
adapter controls include the actual registration receipt's variable-size
limit and the real bootstrap facade's closed failure codes and privacy
checks.
- A deterministic 244-result replay with one second per HTTP response
failed at the old deadline and now verifies after 127 virtual seconds. A
three-second response replay still rejects at 375 virtual seconds. The
full-lane allowance accounts for paginated reads and four-way result
batches; it is not a guaranteed network SLA. All 55 related API/adapter
controls pass. They also reproduce the changing-page-count failure
before the fix, then verify the consistent snapshot. Permanently
changing counts fail after twelve polls. A malformed numeric field still
fails before a retry.
- Owning export controls: 26 passed. They include all 244 synthetic
carriers, mixed passed/failed originals, missing and foreign
projections, privacy filtering and exact source binding.
- Candidate host owner: 31 offline controls passed. The new
finite-catalog case failed before admission changed.
- Preserve two actual local regressions: canonical scenario provenance
failed before its one-line fix; the full registration binding exceeded
the variable limit before compact output. Both now pass.
- Initial local harness setup attempts failed before test execution. The
final narrow owner configuration avoids provider-bundle setup.
- Full repository tests, build and typecheck were not run locally.
Normal required PR CI provides the remote checks.
- Owning TypeScript no-emit, actionlint, paid-workflow policy with
unsafe controls, and diff checks passed.
- Local tests used Node 26.4.0. These are synthetic source controls. No
local app, database, native build or provider qualification ran.
- CI
[41307571776](https://github.com/paperclipai/paperclip/actions/runs/41307571776)
on `23e518be` recorded one 15-second timeout in
`attention-retention-work.test.ts:153`. All other ordinary leaves
passed. The test used a fixed JavaScript date but its retention deadline
comes from PostgreSQL `updatedAt`. The exact failing await and original
SQL timestamp were not captured. The corrected pre-sync head’s server
shard (2/12) passed on AWS in CI 41309030928. Current master
independently fixed the same clock assumption in #18543; conflict
resolution adopts that whole upstream file. No local PostgreSQL run or
timeout increase was used.
- Current-head normal CI
[41463567187](https://github.com/paperclipai/paperclip/actions/runs/41463567187)
passes all required checks on
`fc6c2309cfc7d1e5a83f0a597c7fa1d2beb987e1`, with Greptile 5/5 and all
review threads resolved. The final diff has 23 catalog files. Current
master’s whole retention test is adopted; the redundant earlier test
change is absent. The valid changing-page snapshot regression passes
with all 55 related API/adapter controls. No new TestOps import or live
evaluation has run for this change. After delivery, use metadata-only
bootstrap and a three-case normal TestOps selection to verify the new
route. Do not claim a new 244-case measurement.

## Risks

- The app target remains the preserved catalog candidate. Each result
records its target and workflow revisions. No old campaign ordinal,
acceptance gate or core fix is changed.
- Some mapped cases need private account configuration or have known
source/evidence gaps. Their original checks and evidence rules can still
fail. A registration does not establish readiness or success.
- Whole-plan execution can take hours and spend provider and AWS money.
Original per-case limits and zero retries remain. Unknown setup usage is
not free. This PR does not run all 244 cases.
- The raw private `SOURCE_REF` field retains the prior
integration-lineage label. It is not a verified checked-out branch. The
actual candidate checkout and all authority checks use immutable
`3b66c3c144360ddf32714a0742fd774b8ceafb16`; public projections set
`source.ref` to null.
- TestOps has no atomic bulk upsert. Reconciliation retains partial
progress and performs no automatic resend after an uncertain response.
- The trusted inventory keeps a 12 MiB limit per public carrier file.
The larger parser ceiling does not promise admission above that limit.
The existing 244-case fixture produces a 1,359,465-byte normalized
carrier; this is not a guarantee for future payload sizes.
- Raw evidence remains under the existing private retention owner.
Public GitHub artifacts and TestOps attachments contain only allowlisted
metadata and availability receipts.
- The new workflow needs the exact master entry in the existing runner
group's selected-workflow list before a live run. No repository or
runner-group visibility is broadened by this source change.

## Model Used

OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) assisted with implementation and
independent source review, using code and tools. The host does not
report the context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either linked existing issues or described the issue in-PR
following the relevant issue template
- [x] I have not referenced internal or instance-local Paperclip issues
or links
- [x] My branch name describes the change and contains no internal
Paperclip ticket id
- [x] I have run the relevant local controls and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-02 08:58:44 -06:00