Files
PaperClipAI/tests
Dotta b68f69a801 test(evals): pin author self-use and credential guard corrections (#19357)
## Thinking Path

> - Paperclip manages agents and their work.
> - The protected chat catalog selects an immutable evaluated source.
> - The measured parent produced seven independently reviewed passes out
of ten.
> - The authoring observer rejected valid author-only use, and two
credential wording guards needed stricter boundaries.
> - The reviewed source corrects those observers and declares each
affected oracle version.
> - This PR pins that source without changing the original questions or
run limits.

## Linked Issues or Issue Description

Refs #19346, #19095, #19093.

**What happened?**

The measured authoring response described real saved author-only use.
The old observer required zero assignments and wrongly rejected an
audited author-only assignment. The credential scanner also needed to
retain an unsafe comma-separated destination and mixed
reference/password requests. These are evaluation-layer corrections. The
parent timeline and company runs did not complete; no causal latency fix
is claimed here.

**Expected behavior**

Select the reviewed immutable observer source. Keep all original
scenario identities, questions, conditions, actor/resource proof,
counts, clocks, credentials, model, permissions and zero retries
unchanged.

**Steps to reproduce**

Run the source authoring and shared credential consumer controls.
Compare the committed candidate descriptor snapshot with source
`d1ab241cac0db7f700fadaaf62ffb3d2b51cf0c3`.

## What Changed

- Scope the private Company creation observer to the exact Create secret
dialog. Its successful close can open a separate details Sheet without
making the observer wait for that Sheet.
- Admit blank or bounded non-secret metadata defaults only for the
Company saved-secret-name question variant. Verify the same canonical
payload in independent human accounting.
- Pin source `d1ab241cac0db7f700fadaaf62ffb3d2b51cf0c3`, tree
`c4efaef62405cbb37236ad01db1187cf8a156868`.
- Store exact descriptor SHA256
`60db015337e807ff53a376b6db11c375a239097b224d2253dad68ecd0b0f61f3`.
- Update only seven oracle-version leaves: authoring v5, company v11,
personal v12, and four shared external consumers v6. Source-file lists
remain fixed.
- Retain provenance anchors, historical conditions and failures. Update
the single expected-target assertion. Exactly three metadata/test files
change.

## Verification

- 41 manifest, adapter, reconciliation and CLI controls pass on this
actual pin projection.
- Qualified source: one actual-worker offline browser canary passes all
three creation, failed-POST and missing-form mechanisms. Full E2E types
pass on this successor. Reused unchanged proof: 286 distinct controls
across authoring, Company saved-name question, human accounting and
shared-consumer/declaration suites pass. Full E2E typecheck passes. All
242 declarations load with zero fixture callbacks.
- SDK506 inputs and registry `efa70621` remain unchanged. Launcher
SHA256 `2b9a6953`, private transport interfaces, production/UI/CLI
source, lockfile and workflows remain unchanged from the measured
source.
- All 244 YAML queries, context and expected outcomes are byte-identical
to the parent. All descriptor native/human counts, clocks,
authentication, setup policy and source-file lists remain fixed.
- The parent source has seven measured semantic passes out of ten. The
new authoring and credential oracle versions are native-unrun. Existing
failed grades remain failed.
- The isolated company concurrency diagnostic
[run42124035785](https://github.com/paperclipai/paperclip/actions/runs/42124035785)
is a separate measurement. This pin does not alter concurrency or claim
to fix native noncompletion.
- Repo-wide build/tests were not repeated for this three-file metadata
update. Normal PR CI remains required.

## Risks

- The name character set does not prove secrecy. The saved-key-name
variant can admit a credential-shaped synthetic default; this general
classifier limit remains unfixed and tracked in #19093. These are finite
experimental journeys only. Independent review must reject any actual
credential-like or unknown-value prefill without clear
public-metadata/proposed-name grounding, even if autograding is green.
If safety cannot be established, the journey cannot receive PASS credit.
Secure UI save, human answer, canonical payload/hash, audit, binding and
byte-leak checks remain mandatory.
- Earlier CI includes three explicit runner-shutdown cancellations and
one deferred-observer test assertion at line573. Those outcomes remain
retained and unresolved; the Company dialog repair makes no claim to fix
them. New exact-head CI is required.
- The Company dialog mismatch was established from production source and
an actual-worker model-free regression before the next secure-save step.
The prior isolated attempt did not reach private entry, so it is not
evidence of an observed private-save failure.
- A valid review warning remains as a documented follow-up in #19093:
safe closing comma exclusions such as “only there” can be falsely
rejected by the credential wording guard. The pin authorizes bounded
experimental journeys only. If a valid actual handoff hits that guard,
its failed grade is retained; no regex failure is waived or counted as
PASS. Independent semantic review remains required for both secret
handoffs.
- Isolated Company original5055 remains FAILED. A real succeeded native
took255949ms and produced a valid saved-name question, but the observer
rejected its proposed non-secret metadata default. Secure entry and the
second native were not reached. This successor repairs that assertion
boundary only.
- The next selected74/466/578 run is an explicit bounded
lower-concurrency functional experiment at the existing cap2, same
model, original clocks and zero retries. It is not evidence of a latency
fix or reliability improvement.
- Local observer proof is not a new native PASS. Timeline and company
noncompletion remain unresolved, and no clock increase is included.
- A secure handoff still requires independent semantic review before
PASS credit. Actual saved resources, owner/audit evidence, provider
outcomes and privacy gates remain required.
- The source catalog remains separate from master. This PR does not
merge the core or full catalog feature changes into master.

## Model Used

OpenAI Codex, `gpt-6.1-sol`, with repository tools and code execution.
The host does not expose the context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-11-05 17:57:18 -06:00
..