Files
PaperClipAI/tests
DottaandPaperclip 4c314117d2 fix(evals): bind active image identity and isolate routing checks (#17978)
## Thinking Path

> - Paperclip manages agent work through Runner.
> - Product E2E uses a content hash to identify its Daytona image.
> - The hash must include the active provider identity declaration.
> - The active Copilot fixture advanced past the manually listed image
inputs.
> - The runner routing test also captured later YAML steps as shell
commands.
> - This change binds the active fixture and stops shell extraction at
the next step.
> - Operators can verify image identity and routing against the current
source.

## Linked Issues or Issue Description

Refs #17937 and #17947. The related Copilot work includes #15563. This
change repairs the shared image contract; it does not add a Copilot
qualification claim.

**What happened?**
Two image tests fail because the active Copilot identity fixture is
absent from the image hash inputs. Four routing tests execute later
workflow YAML as shell commands after checkout and dependency steps were
added to the authorization job.

**Expected behavior**
The image hash must change when the active identity fixture changes. The
routing tests must execute only the selected routing step.

**Steps to reproduce**
Run the Daytona image and paid runner routing unit files on current
master. Six assertions fail before this patch. The same failures occur
on the clean 6b9e5f3a baseline.

**Paperclip version or commit**
Rebased onto master at 9b7d5b35eb.

**Deployment mode**
Source checkout. These tests make no provider requests.

## What Changed

- Read the active Copilot fixture version from the source-owned profile
manifest.
- Include that manifest and its active fixture in the image inputs. Keep
the historical entries and remove duplicate paths.
- Stop routing shell extraction at the next workflow step or job.
- Remove an obsolete comment that contradicted the current Docker build
context.

## Verification

- Focused image and routing tests: 18 passed after rebase.
- Existing image tests independently read the identity generator and
verify that changed fixture bytes change the image hash.
- Before this patch, six of those 18 tests fail on the unchanged source.
- Full Product support suite before rebase: 4,514 passed, one failed,
three skipped. The remaining macOS embedded Postgres bootstrap failure
also occurs on the clean baseline.
- The shared installed Runner workspace exports are stale in this local
checkout. Fresh CI remains the typecheck and build gate.
- No workflow, provider credentials, paid campaign, spending limit, or
schedule is changed.

## Risks

- The input set changes the image content ID. A fresh immutable image is
required for that content.
- The active fixture version must remain synchronized with the profile
manifest. The existing independent generator check fails if they drift.
- The extraction still depends on the workflow's current indentation.
The tests fail if the routing step cannot be found.

## Model Used

OpenAI Codex, GPT-6 based, with tool execution and extended reasoning.
The exact serving model ID and context window are not exposed in this
session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-27 23:29:43 -05:00
..