Jannik/PaperClipAI
1
0 Fork 0
mirror of https://github.com/paperclipai/paperclip.git synced 2026-11-03 17:51:58 +01:00
Code Issues Packages Projects Releases Wiki Activity
6,122 Commits 1,392 Branches 2,228 Tags
0748b9006273dae7ef8c303905ca0ffa0221a708
Commit Graph
9 Commits
Author SHA1 Message Date
Dotta a6afe3e230 Fix TestOps launch preflight search encoding (#18620) 
## Thinking Path

> - Paperclip manages AI agents that do real work.
> - Product evals verify the saved outcome of that work.
> - TestOps plans select and dispatch these evals.
> - Operator launchers check a unique launch name before they request
work.
> - TestOps expects an encoded filter, but the launchers send plain name
text.
> - This change uses the vendor filter format in the shared TestOps API
owner.
> - The existing checks can then prevent duplicate launch requests.

## Linked Issues or Issue Description

Refs #18516, #18608, and #18542. The completion pilot registered
successfully after #18608. Its optional API launcher then stopped with
HTTP 400 in [run
41477551241](https://github.com/paperclipai/paperclip/actions/runs/41477551241).
It stopped before the launch mutation marker. No evaluation was
dispatched by that request.

The launch search API uses Base64 of UTF-8 JSON filters. The launchers
sent plain name text. I confirmed the format in the deployed public
[vendor filter
codec](https://paperclip.testops.cloud/es-testops-CICoDnkU.js) and its
launch UI. The encoded name filter displayed the existing pilot launch.
The TestOps UI separately dispatched [the genuine three-case
pilot](https://github.com/paperclipai/paperclip/actions/runs/41478906153);
this PR does not create another run or claim that pilot passed.

## What Changed

- Add the typed name-filter encoder to the existing TestOps API module.
- Use it in the backlog and completion launch preflight requests. The
full-catalog route uses the same completion adapter.
- Preserve the exact launch name, nonce, selection, authorization,
once-only markers, and no-resend behavior.
- Add protocol controls through both real launch functions. Document the
wire format.

## Verification

- Red: both new consumer controls reproduce the HTTP 400 boundary with
plain name text. The other 109 API controls pass.
- Green: all 111 API controls pass after the fix. All 166 related API,
pilot contract, and full-catalog controls pass with Node 24.22.2.
- The first broad local invocation lacked the `yaml` dependency in the
new worktree. Its failure log is retained. Reusing the existing
installed dependencies allowed all 166 controls to run and pass without
further source changes.
- The vendor-format query displayed the existing uniquely named pilot
launch in the authenticated TestOps UI. No launch POST or model call was
used for this read-only verification.
- Diff check passes. All required CI checks pass on exact head
`b55bdc6753fa042e4c2465e7b9b366b8126b4730` ([run
41480643021](https://github.com/paperclipai/paperclip/actions/runs/41480643021)).
Both current-head reviews are 5/5, with zero unresolved findings. No
TypeScript, compiled product, workflow, grader, model, clock, retry, or
evidence-boundary changes.

## Risks

The vendor can change its search format. The format is kept in the
existing provider-specific API module, and both launch consumers have
protocol controls. Existing duplicate detection and uncertain-response
refusal remain in place. The already-running UI pilot is independent of
this fix.

## Model Used

OpenAI GPT-6 through Codex assisted with reasoning, repository
inspection, code execution, and native accessibility tool use. The host
does not expose the exact model revision or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either linked existing issues or described the issue in-PR
following the relevant issue template
- [x] I have not referenced internal or instance-local Paperclip issues
or links
- [x] My branch name describes the change and contains no internal
Paperclip ticket id
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
 2026-11-02 10:40:23 -06:00
Dotta 9203ab52f0 fix(evals): bind complete unassigned TestOps plan buckets (#18608) 
## Thinking Path

> - Paperclip manages agents that do real work.
> - Product evals check the saved outcome of that work.
> - TestOps plans select the exact automated cases to run.
> - A new plan can report its unassigned cases as a job-statistics
bucket.
> - The pilot registration guard rejects that bucket as a foreign job.
> - This change binds only a complete, known unassigned bucket and
records assignment diagnostics before writes.

## Linked Issues or Issue Description

Refs #18516 and #18542. The metadata-only bootstrap in
[41464514278](https://github.com/paperclipai/paperclip/actions/runs/41464514278)
created the dedicated plan and verified its GitHub job definition, then
stopped at `plan_job_mismatch` before job or case-assignment mutations.
No model work ran.

The retained receipt did not include the assignment rows. The live
failure's exact shape remains unconfirmed. The API's job-statistics
carrier permits an absent job. The old guard rejects a complete
unassigned bucket. This PR covers that case and records the actual rows
in the next closed inspection.

## What Changed

- Accept one closed unassigned bucket with the exact selected-case
count. Support an explicit null job or an omitted optional job field.
- Assign the exact selected cases once. Keep the existing owned
assignment without assigning it twice.
- Reject foreign jobs, mixed buckets, wrong counts and unknown
unassigned fields before mutations.
- Retain only bounded assignment kinds, numeric job IDs and counts in
the existing observation receipt.
- Merge current master, including the full-catalog adapter from #18542.
Retain its context-aware plan and job checks alongside the new pre-write
assignment inspection. The helper consumes that context’s expected case
count.
- Document the inspection and binding rules. The helper takes the
expected count from its owner. This does not import or run the
full-catalog route from #18542.

## Verification

- Red: both complete unassigned carriers fail on the old guard; the
other 70 dispatch/API controls pass.
- Green: all 72 dispatch/API controls passed before the master sync.
After the sync, all 104 related pilot and full-catalog Node controls
pass. They include ordinary registration, no duplicate assignment,
foreign-job rejection, incomplete and unknown bucket rejection, launch
fences, uncertain-response protection and result readback.
- Diff check passes. No TypeScript or compiled product code changes.
Full repository checks were not run locally. All required CI checks pass
on current head `a8bec6418065a7faae28c8daab3efdfc9b184797` ([run
41473510972](https://github.com/paperclipai/paperclip/actions/runs/41473510972)).
Both current-head reviews are 5/5 with no unresolved threads. Before the
master sync, one unchanged server timing test failed on attempt one and
passed on the only targeted rerun; that original failure remains in CI
attempt one. No rerun was needed on the current head.
- Live acceptance still requires another metadata-only inspection,
followed by a genuine three-case TestOps launch. This PR does not claim
a new passing measurement.

## Risks

An overly broad unassigned rule could replace an existing job. The new
rule permits only one known bucket with the exact selected-case count
and no foreign job. All final binding checks still apply. Unknown
assignment shapes stop before writes. Original case queries, model,
clocks, graders, evidence rules and retries are unchanged.

## Model Used

OpenAI GPT-6 through Codex assisted with repository inspection, code
execution and tool use. The host does not expose the exact model
revision or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either linked existing issues or described the issue in-PR
following the relevant issue template
- [x] I have not referenced internal or instance-local Paperclip issues
or links
- [x] My branch name describes the change and contains no internal
Paperclip ticket id
- [x] I have run the relevant local controls and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
 2026-11-02 09:56:36 -06:00
DottaandPaperclip 774ddc7271 test(evals): add the full chat catalog to TestOps (#18542) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Chat feature evaluations check real user requests through the
product.
> - The YAML catalog has 244 scenarios, but the shipped TestOps
completion route selects only three.
> - The preserved catalog already supplies the execution mappings and
independent oracles.
> - A larger TestOps plan needs exact identities, bounded selection and
source-bound private evidence.
> - This pull request adds that plan route without merging the catalog
app into master or replacing its grades.
> - Users can run the full catalog or a selected subset and see the
actual outcome and prerequisites.

## Linked Issues or Issue Description

Refs: #18516 and #18515. This extends their completion pilot and reuses
its trusted reporting path.

**What existing behavior does this improve?**
The existing three-case TestOps completion workflow.

**Current behavior**
The remaining catalog scenarios cannot be selected through that TestOps
job. The one-case backlog plan has a separate strict singleton contract.

**Proposed behavior**
A separate candidate-scoped plan maps all 244 scenarios. Existing case
aliases, jobs, plans and result history stay intact. Registration is
distinct from a measured pass.

## What Changed

- Add a source-derived 244-row manifest. It binds the 242 domain
descriptors and two legacy tasks to the unchanged YAML queries,
contexts, outcomes and original bounds. The JSON snapshot avoids
importing the unmerged domain tree into master.
- Reconcile exact case identities and a dedicated full-catalog plan.
Reuse legacy aliases, including the existing backlog case, without
renaming them. Preserve history and stop on conflicting aliases or
uncertain writes.
- Extend the shared completion adapter with a separate trusted
full-catalog workflow context. Return a compact observed binding that
fits one 48 KB protected variable. Keep the three-case pilot contract
intact.
- Add a fixed-source manual workflow. Small selections run serially.
Larger selections allow eight independent workers and one serial
external-input worker. Each worker and report reader keeps a singleton
approval and exact-key AWS session. The eight-cell private authority
limit is unchanged.
- Validate and merge metadata-only singleton projections before one
protected Allure upload and complete paginated readback. Preserve failed
originals and cleanup failures beside passed results. Keep screenshots
and raw evidence private.
- Discard a changing full-catalog page-count snapshot and retry within
the same observer bounds. Malformed page fields still fail at once. A
pass still needs one consistent exact selection and all final evidence
checks.
- Keep workflow source and app source separate. The full lane builds
candidate `3b66c3c144360ddf32714a0742fd774b8ceafb16`. Its only changes
from the existing `de0941835cc8f0d01c982574ee258e110babdbeb` pin are
finite TestOps host admission and its offline test. This is not
current-master app qualification.
- Give full selections above three cases a bounded metadata-read
allowance above the existing 120-second deadline ceiling. Readback has
two upper bounds: at most twelve polls with five-second pauses, and a
size-derived deadline of at most 374 seconds, whichever exhausts first.
There is no guaranteed minimum wait or processing SLA. Small selections
and pilot deadlines stay unchanged. Final identity, grade and evidence
predicates stay intact.
- Document the real readiness limits and setup path. Five coverage gaps,
six evidence exceptions, eighteen additional-credential declarations and
thirty-seven external-input declarations overlap. They are not new pass
claims.

## Verification

- Related Node controls passed: pilot API and dispatch, 244-row
manifest, idempotent reconciliation, full selection, three-page result
readback, source guards and partition/output-size bounds. The latest ten
adapter controls include the actual registration receipt's variable-size
limit and the real bootstrap facade's closed failure codes and privacy
checks.
- A deterministic 244-result replay with one second per HTTP response
failed at the old deadline and now verifies after 127 virtual seconds. A
three-second response replay still rejects at 375 virtual seconds. The
full-lane allowance accounts for paginated reads and four-way result
batches; it is not a guaranteed network SLA. All 55 related API/adapter
controls pass. They also reproduce the changing-page-count failure
before the fix, then verify the consistent snapshot. Permanently
changing counts fail after twelve polls. A malformed numeric field still
fails before a retry.
- Owning export controls: 26 passed. They include all 244 synthetic
carriers, mixed passed/failed originals, missing and foreign
projections, privacy filtering and exact source binding.
- Candidate host owner: 31 offline controls passed. The new
finite-catalog case failed before admission changed.
- Preserve two actual local regressions: canonical scenario provenance
failed before its one-line fix; the full registration binding exceeded
the variable limit before compact output. Both now pass.
- Initial local harness setup attempts failed before test execution. The
final narrow owner configuration avoids provider-bundle setup.
- Full repository tests, build and typecheck were not run locally.
Normal required PR CI provides the remote checks.
- Owning TypeScript no-emit, actionlint, paid-workflow policy with
unsafe controls, and diff checks passed.
- Local tests used Node 26.4.0. These are synthetic source controls. No
local app, database, native build or provider qualification ran.
- CI
[41307571776](https://github.com/paperclipai/paperclip/actions/runs/41307571776)
on `23e518be` recorded one 15-second timeout in
`attention-retention-work.test.ts:153`. All other ordinary leaves
passed. The test used a fixed JavaScript date but its retention deadline
comes from PostgreSQL `updatedAt`. The exact failing await and original
SQL timestamp were not captured. The corrected pre-sync head’s server
shard (2/12) passed on AWS in CI 41309030928. Current master
independently fixed the same clock assumption in #18543; conflict
resolution adopts that whole upstream file. No local PostgreSQL run or
timeout increase was used.
- Current-head normal CI
[41463567187](https://github.com/paperclipai/paperclip/actions/runs/41463567187)
passes all required checks on
`fc6c2309cfc7d1e5a83f0a597c7fa1d2beb987e1`, with Greptile 5/5 and all
review threads resolved. The final diff has 23 catalog files. Current
master’s whole retention test is adopted; the redundant earlier test
change is absent. The valid changing-page snapshot regression passes
with all 55 related API/adapter controls. No new TestOps import or live
evaluation has run for this change. After delivery, use metadata-only
bootstrap and a three-case normal TestOps selection to verify the new
route. Do not claim a new 244-case measurement.

## Risks

- The app target remains the preserved catalog candidate. Each result
records its target and workflow revisions. No old campaign ordinal,
acceptance gate or core fix is changed.
- Some mapped cases need private account configuration or have known
source/evidence gaps. Their original checks and evidence rules can still
fail. A registration does not establish readiness or success.
- Whole-plan execution can take hours and spend provider and AWS money.
Original per-case limits and zero retries remain. Unknown setup usage is
not free. This PR does not run all 244 cases.
- The raw private `SOURCE_REF` field retains the prior
integration-lineage label. It is not a verified checked-out branch. The
actual candidate checkout and all authority checks use immutable
`3b66c3c144360ddf32714a0742fd774b8ceafb16`; public projections set
`source.ref` to null.
- TestOps has no atomic bulk upsert. Reconciliation retains partial
progress and performs no automatic resend after an uncertain response.
- The trusted inventory keeps a 12 MiB limit per public carrier file.
The larger parser ceiling does not promise admission above that limit.
The existing 244-case fixture produces a 1,359,465-byte normalized
carrier; this is not a guarantee for future payload sizes.
- Raw evidence remains under the existing private retention owner.
Public GitHub artifacts and TestOps attachments contain only allowlisted
metadata and availability receipts.
- The new workflow needs the exact master entry in the existing runner
group's selected-workflow list before a live run. No repository or
runner-group visibility is broadened by this source change.

## Model Used

OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) assisted with implementation and
independent source review, using code and tools. The host does not
report the context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either linked existing issues or described the issue in-PR
following the relevant issue template
- [x] I have not referenced internal or instance-local Paperclip issues
or links
- [x] My branch name describes the change and contains no internal
Paperclip ticket id
- [x] I have run the relevant local controls and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-11-02 08:58:44 -06:00
Dotta e122478368 test(evals): run and record the chat completion pilot through TestOps (#18516) 
## Thinking Path

> - Paperclip evals must verify real user outcomes.
> - A passing manual pilot does not make its cases runnable from
TestOps.
> - The existing TestOps adapter owns one separate backlog case.
> - The three-case completion pilot needs its own exact selection and
reporting path.
> - This workflow binds a real TestOps job run to reviewed catalog
source and AWS execution.
> - Results retain their original verdict, checks, source and private
evidence.

## Linked Issues or Issue Description

Refs #18364, #18377, #18515 and #18509. This is the main-side companion
to merged #18515. The immutable evaluated source is its actual
integration-branch squash merge,
`de0941835cc8f0d01c982574ee258e110babdbeb`; its tree matches the
reviewed catalog head. The host admission remains inert until this
main-only workflow exists. The broader catalog stays off master.

The existing TestOps connection can run only the separate backlog cell.
The project-work, project-defaults and instance-restart cases currently
require manual campaign dispatch and private retrieval. This change
creates a dedicated reusable TestOps plan and records fresh execution
outcomes there. Source checks and registration do not establish a new
live pass.

## What Changed

- Add a default-branch-only workflow with exact TestOps selection,
immutable source/build outputs, serial AWS execution and zero eval
retries. No source/model/budget selector is accepted.
- Register the three exact reusable cases and a separate plan with the
existing reviewed GitHub integration. Preserve opaque job definitions.
Validate the observed single Branch/defaultValue carrier, PATCH only
enable/integration, and use the job’s default branch without rewriting
its definition or adding plan overrides. Fence the initial operator
launch by actual GitHub run number and nonce; normal TestOps Run remains
repeatable.
- Reuse the existing private-evidence authority, scanners, bounds and
exact-key S3 policies. Authenticated selection supplies only this
original attempt's exact source and case identity. The three trusted
protocol files are unchanged.
- Skip report retrieval after an unsuccessful build or skipped
execution; retain evidence and failed grades when evaluation actually
ran.
- Check out trusted main after the target in both build and execution
jobs, and reject any target tree containing the reserved trusted-helper
path. A real temporary-Git-tree regression exercises the actual YAML
guard.
- Keep reporting credentials in separate protected jobs. Export only
closed verdict/check/accounting metadata and private-evidence receipts
to TestOps. Verify final PNGs privately and record their checksums; keep
all screenshot bytes in protected AWS storage and out of public GitHub
artifacts. Preserve failed grades and failed cleanup.
- Verify actual same-run case results and complete result-owned
attachment metadata within a fixed processing window. Missing or
inherited attachments never qualify. Keep source/result links and stable
case identities.

## Verification

- 68 dispatch, selection, registration and readback controls pass. A
checkout-boundary regression fails on the old ordering, accepts an
ordinary target and rejects a candidate-owned helper tree before trusted
checkout. They cover real workflow environment wiring, actual GitHub
metadata guards, exact subset selection, wrong sources/cases/jobs,
replays and attachment readiness.
- 38 exporter controls pass, including the existing backlog exporter and
the unchanged private-evidence authority consumer. Red/green controls
prove project work accepts the catalog’s 1–3 run range in both export
gates, while 0/4 runs are rejected; the actual pinned-source control
verifies all three cases’ run bounds. Cancelled, failed and scheduled
TestOps job stages are rejected even with complete results. The three
new cases retain all 35 independent checks; private diagnostics and all
image bytes are excluded from the public artifact. Extra media files and
directories are rejected.
- The actual checked-in workflow environment is exercised against the
actual pinned source matrix and host function for all three cases before
provider access. Local fixture controls make zero provider calls and do
not claim live AWS qualification.
- All 71 workflow-policy controls pass. Actionlint and owning typecheck
pass. Relevant TypeScript build dependencies pass.
- Full support suite: 5,827 passed, 13 failed, four skipped. A clean
unchanged-master baseline reproduces all 13 failures in the same five
files. They include fixture expectation drift, operating-receipt
fixtures, a profile pin, pnpm bootstrap counts and local database
startup. This change updates the exact TestOps workflow registries; the
two affected boundary controls now pass. Baseline failures remain
retained and are not presented as a green full support suite.
- The catalog dependency passed CI and current-head 5/5 review, has no
unresolved threads, and is merged. All current-head CI checks pass; both
Greptile and Paperclip reviews are 5/5, with no unresolved threads. No
new TestOps mutation, provider call or passing launch is claimed yet. A
fresh three-case TestOps launch and verified server-side results remain
the delivery gate.

## Risks

This is a paid execution and reporting boundary. The new workflow
accepts only trusted default-branch actor identities and a verified
project/plan/job/case selection. Its evaluated source is immutable and
separate from main reporting code. The full catalog is not promoted.
Screenshots, raw evidence and hidden provider data stay in protected
storage. Public GitHub artifacts carry only the closed metadata
projection. TestOps receives checks and private-evidence
availability/checksum receipts. Unknown setup or AWS accounting stays
unknown. One passing launch will not establish full-catalog reliability.

## Model Used

OpenAI GPT-6 through Codex, with repository inspection, code execution
and tool use. Exact hosted revision and context-window size are not
exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
 2026-11-01 06:09:47 -06:00
DottaandPaperclip d06c5e0eff fix(evals): preserve opaque TestOps parameters (#18394) 
## Thinking Path

> - Paperclip verifies real agent work through Product E2E tests.
> - The fixed Chat feature workflow uses TestOps to select and report
one case.
> - TestOps exposes job parameters as ID/value pairs.
> - The schema does not establish that those IDs identify global
environment variables.
> - The reader used that unsupported relationship to infer parameter
names.
> - This pull request keeps IDs opaque, preserves the strict allowed
values, and records closed read diagnostics.

## Linked Issues or Issue Description

Refs: #18322, #18375, #18385.

**What happened?**
A zero-model bootstrap returned HTTP 404 during context readback. The
exact failed GET was not retained and remains unknown. Schema review
also found no established relationship between job-parameter IDs and
global environment-variable IDs, although the reader queried that
namespace to label parameters.

**Expected behavior**
Read only the existing job-run, launch, and job records. Preserve the
job's exact permitted ID/value vector without inventing parameter names.
Record which fixed GET class failed, its positive resource ID, state,
and available HTTP status without raw responses or URLs.

**Steps to reproduce**
Use the actual API helpers with a job-parameter ID that has no global
environment-variable definition. The new pure controls reject an
unexpected definition lookup and verify the opaque vector and
diagnostics without a backend or model call.

**Paperclip version or commit**
This change starts from `8595efc20bda3ce20a9fbbf34a5d156a6fcf6d57` after
#18385.

**Deployment mode**
GitHub Actions metadata setup in the protected TestOps reporting
environment.

## What Changed

- Remove the unsupported global environment-variable lookup from
job-parameter handling.
- Preserve an exact positive ID and `refs/heads/master` value. Use the
fixed workflow's existing `evaluate` default.
- Treat an operation parameter as such only when its ID matches the
protected configured ID and its value is allowed. Keep rejecting empty,
duplicate, conflicting, or unknown vectors before writes.
- Project only an opaque ID with a fixed-default-ref kind and value, or
an unresolved kind with the value withheld.
- Add closed diagnostics for the existing job-run, launch, and job GETs.
A successful call keeps HTTP status null because the API abstraction
does not expose the status.
- Sync one reporter-test assertion and document opaque IDs. Reporter
runtime and workflow source are unchanged.

## Verification

- Four new actual-helper controls fail on the parent.
- `node --test tests/runner-e2e/chat-feature-testops-api.test.mjs
tests/runner-e2e/testops-chat-reporter.test.mjs`: 58 affected
pure/mock/FS controls pass on Node 26.4.0.
- Three MJS syntax checks and `git diff --check` pass. TypeScript noEmit
was not run for this MJS/documentation change.
- Two intermediate exact-projection assertion failures were retained
before the assertions were synced to the new diagnostic contract.
- [Normal Node 24
CI](https://github.com/paperclipai/paperclip/actions/runs/41137009579)
passes on exact head `3c256fb8f7559093c5fb063170e26c01b6e8a0a4`: 48 jobs
passed and one skipped. Both required `ci / e2e` and `ci / verify` pass.
- Greptile reviewed this exact head at 5/5 with no actionable findings
or unresolved threads. At this verification cut, the optional
Contributor trust check was still in progress; it is not an effective
required status context.
- No actual CLI, API, provider, model, app, database, build, install, or
additional bootstrap was run to validate this change locally.

## Risks

The original HTTP 404 route remains unknown. Removing an unsupported
schema assumption does not prove that it caused that failure. The live
job mapping and dispatch remain unqualified. The strict source, actor,
selection, run-number, nonce, once-only send, private custody, case,
model, grader, and clock checks are unchanged. Unknown metadata still
stops. This PR adds no endpoint, workflow mode, permission, credential
path, or passing result.

## Model Used

OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) with reasoning, code editing, tool
use, and pure test review. The runtime does not expose the exact
context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-30 21:41:05 -05:00
DottaandPaperclip 8595efc20b fix(evals): support official TestOps job metadata (#18385) 
## Thinking Path

> - Paperclip manages AI agents and verifies real work through Product
E2E tests.
> - The fixed Chat feature workflow uses TestOps to select and report
one backlog case.
> - Official registration exposes a workflow-query URL and a Branch
parameter.
> - The existing reader accepts only workflow-file or numeric URLs, and
the admin launcher requires an operation parameter.
> - These assumptions reject valid official metadata before the feature
can run.
> - This pull request supports the exact observed form while preserving
source, selection, and once-only execution checks.

## Linked Issues or Issue Description

Refs: #18322, #18375.

**What happened?**
The second zero-model bootstrap stopped with a generic job-mismatch
code. Ordinary TestOps UI inspection then showed an official
workflow-query URL outside the accepted literals and a Branch-only job.
The exact first failed predicate in that bootstrap remains unknown.

**Expected behavior**
Accept that exact fixed-workflow URL. Preserve the existing master
Branch value when the job has no operation parameter. Keep rejecting
foreign URLs, unknown parameters, wrong branches, and conflicting
operation IDs before mutation.

**Steps to reproduce**
Pass the official query URL and Branch-only metadata through the
existing bootstrap, selection, and admin-launch owners. The new pure
controls exercise those actual helpers without a CLI, backend, or model
call.

**Paperclip version or commit**
This change starts from `0e15c92073a09e50df99190a37f69a86286b989b` after
#18375.

**Deployment mode**
GitHub Actions and the protected TestOps reporting environment.

## What Changed

- Share one exact URL predicate across bootstrap, evaluation selection,
and admin preparation. Add only the observed official workflow-query
literal.
- Support Branch-only, operation-only, and Branch-plus-operation jobs
with strict definition IDs, names, values, and duplicate checks.
- Preserve the exact master Branch value. Set an existing operation
parameter to `evaluate`; otherwise use the fixed workflow's existing
default.
- Use the same checked parameter vector for job and plan writes and
their readbacks.
- Report distinct fixed bootstrap predicate codes without response
values.
- Document when the protected operation-parameter ID is required.
- Align one existing Cloud-maintenance test with the inherited
managed-Cloud S3 policy. Preserve the same-version local-storage proof
in the parent, use valid synthetic Cloud settings in the child, and
check the expected provider mismatch without S3 requests. Retain the
no-seed and unchanged persisted-row assertions.

## Verification

- URL controls: three failures on the parent; all pass with the
correction.
- Branch controls: two failures and one correct rejection on the
predecessor; all pass with the correction.
- `node --test tests/runner-e2e/chat-feature-testops-api.test.mjs`: 40
owning pure/mock controls pass on Node 26.4.0.
- Two MJS syntax checks and `git diff --check` pass. TypeScript noEmit
was not run for this MJS/documentation change.
- The first [normal CI
run](https://github.com/paperclipai/paperclip/actions/runs/41131094974)
failed only in the managed-skill Cloud fixture and its dependent
verification gate: 46 jobs passed, two failed, and one skipped. The
tested merge included a new upstream policy that rejects the test's
local-disk Cloud configuration. The exact caught runtime exception
remains unknown.
- Three pure controls using the actual policy validator pass: the old
invalid Cloud fixture rejects, the new synthetic S3 fixture admits, and
an invalid master key rejects. An initial tsx wrapper IPC permission
error executed zero controls and remains a separate setup failure.
- The PostgreSQL/maintenance-child test was not run locally. On new-head
AWS CI, `managed-skill-content.test.ts` passed all 17 tests in server
shard 9.
- [Normal
CI](https://github.com/paperclipai/paperclip/actions/runs/41133876636)
is green on exact head `b76d5dd40d7711f1b5f6b6316b1114c0aae02d57`: 48
jobs passed and one skipped. Both required `ci / e2e` and `ci / verify`
pass.
- Greptile reviewed this exact head at 5/5 with no actionable findings
or unresolved review threads.
- No actual CLI, backend, model, provider, app, database, build, or
bootstrap was run to validate this correction locally.

## Risks

The exact original bootstrap predicate is still unknown. This fixes a
demonstrated metadata false negative and preserves the prior failed run.
Actual backend parameter types and integration dispatch still need
verification. Unknown metadata stops before the once marker and writes.
This PR changes no workflow, credential, case, model, grader, clock, or
private-evidence policy. It does not claim a passing feature result.

## Model Used

OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) with reasoning, code editing, tool
use, and pure test review. The runtime does not expose the exact
context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-30 17:04:59 -05:00
DottaandPaperclip 9e1735ec17 test(evals): add a fixed TestOps AWS chat feature route (#18322) 
## Thinking Path

> - Paperclip lets people manage AI agents and inspect their saved work.
> - Chat feature evaluations test a real request against saved tasks,
audit records, links and reload behavior.
> - The current suite has Codex and OpenCode cells but no dedicated
runnable DeepSeek 1017 backlog plan in TestOps.
> - TestOps must select the real case before the workflow can access a
model key.
> - This PR adds one fixed, default-branch AWS route for native OpenCode
and DeepSeek 1017.
> - The route keeps original evidence private and sends only a verified
result and approved screenshots to TestOps.

## Linked Issues or Issue Description

**Subsystem affected**

Product E2E evaluation definitions and GitHub Actions delivery.

**Problem or motivation**

A dedicated TestOps plan cannot run the existing backlog chat journey
with native OpenCode. A result must prove saved work and cleanup rather
than report that a model call completed.

**Proposed solution**

Add one OpenCode backlog cell and one fixed public workflow. TestOps
selection validation runs in a token-isolated job before builds or
inference. The workflow retains private originals and reports the bound
outcome to the selected TestOps case.

**Alternatives considered**

A private-repository wrapper would need a new cross-repository GitHub
write credential. The new route uses the existing TestOps GitHub
integration instead. Existing manual candidate approval paths remain
separate.

**Roadmap alignment**

This extends the existing feature evaluation infrastructure. It changes
no production feature, global runtime default or model default.

Current main #18319 supplies the v2 grader, delegation queue accounting
and final screenshot capture; these improvements are preserved. Related
product work: #16814 and #17126 concern backlog handling. This PR
changes neither product implementation and does not claim those changes
are required.

## What Changed

- Add exactly one native OpenCode backlog cell with DeepSeek 1017 under
the separate `runner-opencode-deepseek-1017` profile. Preserve all four
existing Codex and production OpenCode cells and their models.
- Add a default-branch-only workflow with fixed evaluation, zero-result
registration and one plan-launch modes. Require GitHub actor checks and
authenticated singleton selection before inference.
- Add a separate shipped-main evidence authority. Reuse the existing
strict private storage, scanners, inventories and scoped AWS sessions.
- Bind sanitized results, generated JUnit and producer-approved
screenshots to the exact source, run, attempt and case. Keep failed
grades and cleanup failures failed.
- Add TestOps API, CLI and result adapters with pure controls and normal
unit-test wiring. Keep the reporting token out of the evaluated process.
- Verify the current server case result and attachment metadata after
upload, without masking an earlier failed upload or evaluation.
- Apply only the existing reviewed lock correction, verify unchanged
tracked dependency inputs and bind the lock receipt across the
compiled-output handoff before frozen installation.
- Bind the initial administrator launch to one approved workflow run
number and live GitHub source/actor metadata before TestOps
authentication. Use bounded REST case reconciliation and only the
reviewed existing GitHub integration.
- Make two existing CI tests observe their intended state
deterministically: a fake clock for copy-feedback lifetime and a
completed-pass barrier for retry-cancellation batches. Production
behavior and the original assertions remain unchanged.

The two test-only fixes are intentionally included because their
observation boundaries blocked this delivery’s required normal CI. They
preserve every original assertion and change no production behavior. The
original failed runs remain recorded, and the exact causes of those
observations remain unknown.

## Verification

- 157 focused profile, catalog, oracle, filesystem admission, export,
private-path and workflow controls passed on the current-main
composition. The existing stock OpenCode models and the new exact
DeepSeek 1017 selection are checked separately.
- 44 API, CLI, filesystem and post-upload readback controls passed,
including foreign selection, changed hashes, unsafe files, failed
outcomes, expired readback and no-resend behavior.
- The new exact private session and trace layout controls produced four
genuine parent failures and pass with the selected-path extension. The
frozen 244-case candidate approval registry remains unchanged.
- The lock guard has seven filesystem controls plus seven existing
canonical-owner controls. The unrestricted old workflow failed its
actual policy calibration; the corrected frozen-install route passed.
- Eval-owning TypeScript checks, workflow lint and Git whitespace checks
passed. The separate UI test correction was checked with the focused
jsdom unit suite. The server test barrier has pure wrapper and real
scheduler controls; its DB-backed route was not run locally. No
additional local typecheck/build was run for these test-only changes.
- Full native builds, provider execution, app/database behavior and
TestOps registration are reserved for GitHub Actions/AWS. They have not
run locally.
- The initial normal CI run failed the new-workflow policy allowlist and
a heartbeat scheduling assertion. The policy fix has an actual parent
red/current green control and rejects five unsafe route variants.
Heartbeat source is unchanged and its failure cause remains unknown. The
original run is retained.
- The launch-fence actual caller fails against the published parent and
passes the five own-run contradiction controls before authentication.
- The prior `a5aa359d` normal CI completed successfully. The complete
`f9e1f289` run passed all leaf jobs except an existing copy-feedback UI
assertion, plus its dependent verification gate. The four owning UI
files were unchanged from the earlier passing head and actual tested
merge; the exact scheduling cause remains unknown.
- The correction changes only that test to actual React `act` and a
scoped fake clock. All eight component tests pass; early-hide and
never-clear source variants each fail the intended lifetime assertion.
Production clipboard/card behavior is unchanged.
- The `2f27f3fc` run passed the corrected UI shard and every leaf job
except the existing retry-cancellation intermediate-count assertion,
plus its dependent verification gate. The seven relevant owners match
the earlier passing run and historical source comparison; the exact CI
scheduling cause remains unknown.
- The test-only barrier observes a persisted completed worker pass
before asserting the unchanged 20-item cap and two-company fairness,
then releases it for the original 29-item drain and 180-second idle
check. Two extracted-wrapper controls and three actual scheduler
controls pass. The real DB-backed test awaits AWS qualification.
- On `836e80ac`, both repaired test shards passed in AWS. The original
attempt retained a separate E2E cleanup HTTP 500 from a PostgreSQL lock
timeout and one runner shutdown. No production repair or precise
contention/shutdown cause is claimed.
- Exactly one targeted retry of E2E shard 3 and its aggregate gate
passed in attempt 2; exactly one recovery of the interrupted server
shard and its verification gate passed in attempt 3. Other passed jobs
were carried forward. The final normal run is successful. No
whole-workflow retry or evaluation/model retry was launched.
- Both literal required checks (`ci / verify` and `ci / e2e`) are
successful on the exact final source `836e80ac`. Greptile is 5/5 and no
review threads remain open. Actual TestOps registration and the
OpenCode/DeepSeek feature result remain unqualified until their separate
executions.

## Risks

- Actual zero-result registration, tenant API shapes and
public-repository dispatch access must be checked before the first
evaluation. Missing or ambiguous metadata stops the route.
- The real OpenCode/DeepSeek backlog result and Allure outcome remain
unqualified until that run. No passing result is seeded.
- One human query, one accounted work run, 120-second turn, 420-second
attempt, 300-second settled-state bound and zero retries remain
unchanged. Setup usage is outside work accounting and stays unknown.
- The reporting token was configured through the separately reviewed
protected credential bootstrap. It is unavailable to AWS evaluation
jobs.
- Metadata reconciliation is capped at ten pages, 1,000 cases and four
parallel detail reads. Missing or inconsistent metadata stops before
inference. The initial administrator launch counter must be created once
after registration; it does not restrict normal TestOps Run actions.
- Existing manual workflows, historical evaluation grades and candidate
approval records remain unchanged. The original CI failures are
preserved; the test corrections do not establish retrospective causes.

## Model Used

OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) with reasoning and tool use. The
runtime did not expose its context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-30 15:08:16 -05:00
DottaandPaperclip 1511c92965 fix(evals): account for superseded delegation wakes (#18319) 
## Thinking Path

> - Paperclip lets people ask agents to delegate work and return the
result.
> - The chat feature eval checks task creation, worker execution, saved
output, and delivery to the source chat.
> - A task can first belong to the CEO and then move to Morgan.
> - Paperclip correctly cancels the old assignment wake before it
starts.
> - The eval treats that queue record as a failed provider run and stops
before Morgan works.
> - This change accounts for that record separately and preserves the
complete outcome checks.
> - The report also keeps partial check results when an attempt fails.

## Linked Issues or Issue Description

**What happened?**

The delegation journey stopped on an unstarted queue record with
`issue_assignee_changed`. The CEO had created a task, assigned Morgan,
saved the brief, and returned a task link. The test stopped before
worker execution. Its report showed no individual checks even though the
saved observation contained partial results. [Original public
measurement](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/chat-progress-39e-112-latest-121-measurements-with-screenshots-40649033672/?view=case&case=chat-features.delegate-announcement).

**Expected behavior**

Keep the obsolete queue record and its events. Continue observing the
intended worker. Pass only after three successful native work runs, a
current Morgan-authored announcement, a completed task, automatic
source-chat delivery, and browser readback.

**Steps to reproduce**

1. Ask the CEO to have Morgan write and save a launch announcement, then
return it to chat.
2. Create the task under the CEO, then assign Morgan before the queued
CEO run starts.
3. Observe the cancelled old assignment. The previous eval aborts at
that point.

**Paperclip version or commit**

Based on `6da112ab40d7b3cc0bcb79e08ddbb9b915a268a4` from
`origin/master`.

**Deployment mode**

Disposable Product E2E fixtures. The live check uses the trusted
default-branch workflow and AWS fleet.

Related contribution: #17794 introduced these two journeys. The open-PR
search found no separate fix for this queue-accounting defect.

## What Changed

- Recognize only explicit dispatch-gate reassignment cancellations with
a null start time and no native issue. Verify the company, prior CEO
owner, and target task now assigned to Morgan before excluding that
record from work-run counts.
- Keep every queue record in the full observation and final ledger.
Retain its events with an explicit reason for the absent provider log.
- Let the feature flow own its complete saved-state and browser oracle.
Return only actual work runs to the common provider checks.
- Record saved-state checks on failures as well as successes. Capture
the required `final-state.png` after the feature-specific browser
readback.
- Add two explicit local OpenCode cells alongside the two existing Codex
subscription cells. Use the current production-qualified model. The
earlier linked campaign used a different model, so this is a new
measurement.
- Advance the grader and suite versions. Keep the original request,
output requirements, 120-second run cap, five-minute observation bound,
budgets, and single-attempt policy.

## Verification

- Red/green regressions cover queue accounting, missing-log collection,
partial checks on evidence failure, and the required final screenshot.
Flow tests invoke the actual polling callbacks: restoring the original
callback reproduces both early-stop errors. Started cancellations still
fail.
- All 87 focused flow/grading/queue tests pass. Catalog/report checks
and the 137 additional Node checks pass. Eval typechecking passes on
final commit `0a6e11408c4470683417e6bf7965c73cec2a9a60`.
- All current CI tests/builds pass. Greptile is 5/5 on the final commit,
with no unresolved review threads.
- The final commit adds only flow test coverage. Its registered
definition digest equals live target
`e4e0be875ee12c134fe8dd2ce15a88bfcbb100ef`:
`ac6bba21d4b9cb86088be5d00df6442af90268186d6e94c9bea7d611d0488563`.
- Replaying the original saved observation still fails for missing
Morgan execution and delivery. Its stale queue record no longer counts
as provider work.

| Operation | Selected coverage | Observed result |
| --- | --- | --- |
|
[41071710105](https://github.com/paperclipai/paperclip/actions/runs/41071710105),
`b8dff816` | Backlog and delegation | Refused at the exact
private-retention gate before fixtures or providers. Preserved without
feature coverage. |
|
[41073708150](https://github.com/paperclipai/paperclip/actions/runs/41073708150),
`39be9e45` | Backlog and delegation | Both passed their full
product/browser oracles. Retained grades correctly fail missing
`final-state.png`; the feature flow had saved the image under another
name. Fixed in `e4e0be87`. Each original archive remains private and
intact; later workflow concurrency superseded only the report
aggregation. |
|
[41076854978](https://github.com/paperclipai/paperclip/actions/runs/41076854978),
`e4e0be87` | Backlog and delegation | Backlog passed in 92.576 seconds,
with a 47.388-second native run, complete required artifacts and passed
cleanup. Delegation was interrupted by a runner-host shutdown at
15:19:23 UTC; all later evidence-upload steps were skipped. Its private
archive is absent, so behavior, usage and cost remain unverified. |
|
[41080065077](https://github.com/paperclipai/paperclip/actions/runs/41080065077),
`0a6e1140` | Delegation only | Passed on the final commit in 240.786
seconds. All 14 product checks plus browser readback passed; exact
private identity, current worker output, Done state, correlated
source-chat reply, required screenshot and cleanup verified. The three
native runs took 42–60 seconds each. Passing control was not repeated.
One-cell workflow and report enforcement passed. |

The completed delegation in `41073708150` passed all 14 product checks
plus browser readback in 222.593 seconds. Morgan saved the current
two-sentence announcement, the task reached Done, and the CEO
automatically delivered that saved text to the original chat. Its three
successful native runs took 49.117, 40.060 and 39.107 seconds. The
failed required screenshot remains a failed result in that measurement.

All live operations use explicit IDs, concurrency one, no automatic
retries, the unchanged 120-second per-run cap, and scoped private
evidence retention bound to the exact workflow, target, original run,
selector and launcher digest. These narrow measurements do not qualify
the full catalog or establish repeated-run reliability.

The broad local support suite passed 5,139 tests, with two skipped. Two
database tests hit exhausted host semaphore capacity after the installed
Postgres package's missing library symlinks were repaired. Their recheck
reported `could not create semaphores: No space left on device`. No host
IPC resources or kernel settings were changed. The corresponding full CI
tests passed on a fresh host.

## Risks

- A broad cancellation exception could hide failed work. This exception
requires the dispatch gate's exact receipt, a null start time, the
expected company and prior owner, and the actual reassigned task.
Started and unproved cancellations still fail. Backlog creation permits
no task wake.
- New manual OpenCode cells can spend provider money only when
explicitly selected. Default scheduled selection is unchanged.
- This changes eval interpretation and evidence collection. It does not
change product scheduling or task ownership.

## Model Used

OpenAI Codex, GPT-6 family, with code execution and tool use. The exact
model ID and context-window size are not exposed to this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-30 10:40:00 -05:00
DottaandPaperclip 2ee36a4c0a test: add chat feature scenarios and real task workflow evals (#17794) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agent Chat lets a person assign work through a conversation.
> - Feature and API maps do not prove that a natural request produces
the right saved result.
> - Product E2E already owns isolated fixtures, browser runs, evidence
and cleanup.
> - This pull request adds a chat scenario catalog and two real task
workflows in that framework.
> - Independent checks verify the task, its author, the saved output and
the reply in chat.
> - The benefit is a small executable foundation for broader chat
coverage.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

It adds real task creation and delegation coverage for Agent Chat.

**Current behavior**

Existing maps describe features and agent access. Existing chat evals
cover specific protocols and handoffs. They do not connect a broad
human-request catalog to independent saved-state checks.

**Proposed behavior**

Keep 244 planned scenarios across 35 feature areas. Promote two requests
into the explicit-only `chat-features` suite. One saves a backlog task
without starting work. The other delegates an announcement and returns
the saved result to the original chat.

**Reason and benefit**

A successful tool call or confident reply does not prove the requested
result exists. These evals check public API state, attribution and
browser access after a real model run.

**Breaking changes**

No API or data schema changes. Chat instructions now require actual
relative task links. The local E2E private-state scanner recognizes
Codex account ownership metadata while continuing to reject credentials
and model-visible identity leaks.

**Additional context**

Related prior work: #13969 and #13762. This improves verification of the
existing Agent Chat roadmap item. It does not add a new core product
feature.

## What Changed

- Add YAML scenarios with capability IDs, starting context, user
requests and expected outcomes. Validate all area files against their
schema, global ID uniqueness and exact feature-map capability coverage.
- Add two local native Codex subscription cells. Send the exact catalog
request once through Chromium.
- Seed only prerequisites through public APIs. Use shipped agent
instructions and explicit disposable budget policies for unpriced
subscription usage.
- Check task ownership, creation audit, no premature execution, current
worker-authored output, automatic completion delivery and usable links
after reload.
- Keep private Codex account ownership metadata distinct from bearer
credentials. Scan rollout message content, all tokens and published
evidence as before.
- Teach normal chat instructions to use relative task links instead of
invented hosts. Update tests, maps and the runbook.
- Repair existing E2E contract drift: include the active Copilot
identity in the image digest and update Pi profile expectations. Keep
global catalog cardinality in its owning catalog test.

Product E2E owns the new flow and grader. Its existing launch and report
paths remain in use. The shared scanner change is limited to
owner-private temporary Codex runtime metadata. Other callers retain the
full secret scan. No UI, endpoint, schema or permission behavior
changes.

## Verification

- `node scripts/check-maps.mjs` passed: 35 areas and 191 capability IDs.
- Product E2E typecheck and focused tests passed before rebase: 225
Vitest tests and 137 Node tests. Conversation service tests passed
40/40. Five existing runner turn-binding regressions passed.
- Historical live measurement used real Chromium, server, database and
native Codex with no automatic retries. The initial same-definition
batch passed 5/6 cells. Backlog passed 3/3; delegation passed 2/3. One
CEO startup failed because the runner did not receive a provider turn
identity. The cause is unresolved.
- An unchanged follow-up passed delegation three times and backlog
twice. Combined evidence is backlog 5/5 and delegation 5/6. Each attempt
passed cleanup. Later passes do not erase the earlier failure or prove a
runner repair.
- Earlier failed revisions exposed an invented task hostname, private
account metadata false positives and missing funding for unpriced
subscription usage. Those led to the specific changes above; the failed
records remain retained locally.
- The runbook records campaign identifiers and definition provenance.
Generated result artifacts are not committed. Historical live results
predate this branch rebase; fresh PR-source live evidence and final
validation are recorded below.
- Full workspace typecheck and production build passed on the rebased
branch. Product E2E typecheck passed.
- The full Product E2E support suite first reproduced four baseline
failures: two missing active Copilot image-identity inputs and two stale
Pi/catalog expectations. The narrow contract fixes above resolved them.
The full rerun passed 4,282 Vitest tests plus 137 Node tests, with two
skipped. Image checks verify mutation changes the hash, unchanged input
stays stable, and an input closure without Copilot is unaffected.
Original failed logs remain retained.
- The repo-wide local Vitest run has reported two failures in unchanged
service files. The Workspace connection test passed an isolated recheck
and its full file passed 469/469 unchanged. The recovery test hit a
15-second timeout; its full file passed 89/89 unchanged. The run later
hit explicit PostgreSQL `semget` resource exhaustion and was interrupted
after those observed failures; no final aggregate counts or full local
pass are claimed. Its complete available log is retained. No assertion
or timeout was weakened. Matching remote server CI passed on the same
source.
- On clean PR head `39ad80b523ce194c90aafafd077d48dd9a36b4c5`, backlog
passed in `local-2026-10-26T19-24-13-014Z`. The parallel delegation
fixture failed during PostgreSQL initdb, before browser or model
execution; its failed cleanup grade is retained. A disposable standalone
initdb probe then passed. No semaphore sets or kernel settings were
changed before that probe/recheck. After confirming no remaining fixture
process, its private temporary files were removed.
- One separate delegation recheck passed in
`local-2026-10-26T19-29-24-800Z`, with one attempt, zero automatic
retries and successful cleanup. Both passing current-source cells use
definition
`3f2c9ba2d03e3dcddde47ddab3b43bc52de977f25e256fceae0a24c9128e204b`. The
earlier bootstrap failure and the historical provider startup failure
remain distinct. These local results omit automatic source-SHA metadata;
the clean checkout SHA is recorded here from the execution session.
- The later broad local run reported `could not create semaphores: No
space left on device`. A bounded cleanup removed 32 current-user
PostgreSQL sets with 17 members, marker 537, a dead recorded creator and
zero waiters. No live process or kernel limit was changed by that
cleanup. The test process group was subsequently stopped; no broad rerun
is planned. This direct later resource failure does not prove the
precise cause of the earlier live initdb failure.
- Review follow-up adds only static catalog validation and documentation
of the existing five-minute journey bound. The executable definition is
unchanged. Focused catalog/oracle tests passed 28/28; no redundant live
run is claimed.
- Final head `30fcb36ac9ae1f0270a589dbfafcec91ca6a03f3` passed all
required and underlying CI checks in [run
40391402347](https://github.com/paperclipai/paperclip/actions/runs/40391402347),
including `ci / verify` and `ci / e2e`. Greptile and Paperclip Review
both scored this head 5/5, with all review threads resolved. The static
follow-up leaves every executable definition input byte-identical to the
live-tested source at `39ad80b`.

Normal squash auto-merge is enabled. Actual merge is pending eligible
code-owner approval for the owned dependency manifest; no protection is
bypassed.

## Risks

- Live model behavior can vary. One missing turn-start event remains
unexplained; this PR does not claim provider reliability or broad
catalog qualification.
- Subscription costs are unpriced. Explicit fixture funding admits that
usage. Run counts and deadlines remain bounded, but the dollar limit
cannot cap unknown cost.
- Account identity exceptions apply only to known private Codex runtime
files. Token reuse, unknown files and model-visible rollout content
remain blocking findings.
- Deterministic checks establish saved state, output structure, required
brief facts and access. They do not score all prose quality. Initial
samples also require human inspection.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, code execution, browser and
repository tools. The precise model revision and context window are not
exposed to this session. Live eval agents used `gpt-5.6-sol`; that is
the evaluated model, not a claim about the authoring model.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run local tests; focused suites pass and the interrupted
full-run failures are documented above
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-26 16:19:39 -05:00