Jannik/PaperClipAI
1
0 Fork 0
mirror of https://github.com/paperclipai/paperclip.git synced 2026-11-04 11:01:16 +01:00
Code Issues Packages Projects Releases Wiki Activity

fix(evals): pin reviewed Source4 chat repair candidate (#18736)

Browse Source 
## Thinking Path

> - Paperclip manages AI agents and their work.
> - The chat catalog checks real user journeys through a fixed
candidate.
> - The current candidate retains several known fixture and observer
defects.
> - A new source must bind each repaired declaration before paid work
starts.
> - This pull request pins the reviewed Source4 candidate and its exact
descriptor snapshot.
> - The next measurements can test those repairs without changing old
results or relaxing private evidence guards.

## Linked Issues or Issue Description

Refs #18726. Related core changes remain separate and unmerged: #18717
and #18734.

**What happened?**

The active lane still uses Source3. It cannot test the accepted primary,
app review, CLI, timeline, plan review, hire fixture and SDK input
repairs. Old results must remain historical.

**Expected behavior**

The trusted lane must bind the new immutable candidate, all 244 case
identities, exact current declarations and private source hashes. It
must retain the original definition provenance and explain changed
conditions.

**Steps to reproduce**

1. Compare the current manifest with candidate
`51ef450754370c8a8f2cbe4bd18e1536bbc93966`.
2. Check each of its 242 declarations against the checked descriptor
snapshot.
3. Run the focused manifest, contract and reconciliation controls. Run
the selected real cases only after normal source delivery and a fresh
execution guard.

## What Changed

- Pin candidate `51ef450754370c8a8f2cbe4bd18e1536bbc93966` and its exact
descriptor snapshot. Keep all 244 identities and aliases.
- Record cumulative original-to-current field overrides. Keep the
original `de094` definition provenance and case limits. Studio144
retains its earlier explicit 2..3 accounting revision.
- Record the reviewed SDK expected build-lock correction and the
candidate-only full proposal disclosure from unmerged #18734. Keep
compiler/parser/Zod identities, privacy guards and the other SDK holds.
- Update the static owning controls and identify current versus
historical sources in the lane guide. Runtime JavaScript, declarations,
workflow permissions and private transfer code are unchanged in this PR.

## Verification

- 47 focused Node controls passed: contract, manifest and idempotent
reconciliation.
- Candidate controls: 293 pure tests across 16 files passed. All 242
actual factory declarations and 42 admission controls passed without
fixture callbacks.
- The final candidate SDK registry check matched all 506 inputs. The
actual Git launcher, checked snapshot, three private protocol files and
all 244 singleton evidence policies matched.
- The candidate product disclosure is byte-identical to the reviewed
#18734 file. Its token gate passed. The separate core change retains 115
React controls.
- Independent full PR and integration source review passed before
publication.
- Full local workspace typecheck, application build and model execution
were not run for this pin. The existing workspace dependency limitations
remain recorded. Latest-head CI `41586212647` passed on attempt 1: 48
passed and 1 skipped. Both required checks passed. Greptile reviewed the
exact head at 5/5 with no open threads.

## Risks

- These are new candidate measurements. A source-qualified repair does
not prove a user journey passed.
- SDK115 must first pass real admission with its original
300-second/one-native limit. The other five SDK cases and plugin558
canary certificates remain held.
- The CLI parser rejects unresolved placeholders and foreign values. It
executes the actual returned commands without substitutions. The old
failed command remains failed.
- The core guidance and review-card PRs remain unmerged. This pin
changes neither core merge authority nor the previous full-catalog
acceptance gate.
- A running Source3 launch remains pinned to its immutable controller
and candidate. The next model wave must wait for terminal accounting and
cleanup guards.

## Model Used

OpenAI `gpt-6.1-sol`, with code execution, tool use and independent
agent source review. The host does not expose the exact context window.
No model eval was run to prepare this PR.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
This commit is contained in:
Dotta authored and GitHub committed 2026-11-03 00:04:17 -06:00
1 parent 32f93cf9dd
commit 2aeceb963d
4 files changed
+313 -76
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

tests/runner-e2e/CHAT-FEATURES.md
+14 -5
View File
@@ -217,7 +217,7 @@ overlap. Exact requirements and limitations appear in each case's preconditions.
A declared credential does not establish whether its runtime reference exists.
This lane runs the app and fixtures from candidate
`43c44ccc823a8c972b247fa7159281887e6f5255`. It retains the evaluated app from
`51ef450754370c8a8f2cbe4bd18e1536bbc93966`. It retains the evaluated app from
`3b66c3c144360ddf32714a0742fd774b8ceafb16`. The first repair source added:
owned SDK staging, public membership for three incidental human fixtures,
company settings observation inside its original case deadline, and exact
@@ -239,8 +239,8 @@ change the previous 196-of-244 acceptance gate.
Native execution uses OpenCode and
`openrouter/deepseek/deepseek-v4.1-flash-1017` on the existing AWS fleet. Each case
keeps its original query, oracle, run counts, clocks, budgets and zero automatic
retries. Browser boundaries and the external assistant retain their original
uses the manifest-declared query, oracle, counts, clocks and budgets with zero
automatic retries; explicit reviewed condition changes are recorded below. Browser boundaries and the external assistant retain their original
surfaces and may have zero native work runs. The external companion keeps its own
declared provider requirement. Setup inference and unpriced usage are not free.
@@ -304,12 +304,21 @@ The manifest keeps the original definition source and values, with explicit scop
Workflow, model, case clocks, native-run bounds, credentials, protected reporting and case identities do not change. The full harness TypeScript graph has not been qualified by local pure tests; source import/admission and the existing credential-free remote Runner build remain required before native queries. Static TestOps descriptions may still show the original definition until a separate idempotent metadata reconciliation at a safe source boundary.
### Repair source 3 (2026-11-03; candidate only)
### Historical repair source 3 (2026-11-03; candidate only)
The current target is the canonical revision above. It adds the reviewed skill import guidance, version-aware Studio sample checks, scoped app review/test and team-auth prerequisites, and the synthetic announcement-dismissal input. The core guidance PR remains unmerged; candidate-owned catalogs carry that explicit runtime/tool-description change. App testing declares its changed setup context and outcome. The exact human query is preserved.
Historical target `43c44ccc823a8c972b247fa7159281887e6f5255` added the reviewed skill import guidance, version-aware Studio sample checks, scoped app review/test and team-auth prerequisites, and the synthetic announcement-dismissal input. The core guidance PR remains unmerged; candidate-owned catalogs carry that explicit runtime/tool-description change. App testing declares its changed setup context and outcome. The exact human query is preserved.
Studio test 144 alone changes native accounting from minimum/maximum 2/2 to 2/3. Acceptance requires all three genuine settled turns: initial CEO, actual Studio worker, and exactly one source-bound completion delivery. The original 540-second case clock, controller, one human request and all other case limits remain fixed. The manifest records each previous/current field against the original definition source; no old grade or catalog pass gate is rewritten.
The SDK public-input registry is generated from the committed candidate tree. Compiler, parser, Zod, lock runtime identities, scanner and canary build certificates are unchanged. This does not establish the original SDK detector or a plugin pass. Plugin 558 remains held until genuine canary build evidence is reauthenticated. The public announcement fixture is one reviewed synthetic 443-byte JSON under an isolated staging prefix; production announcements and the global index are untouched. Each original worker process still selects one case, so only the dismissal case enables that input.
The full-catalog private evidence policy binds the launcher digest from the same reviewed manifest as the immutable candidate target. Every final GO reads `launch.ts` from that exact Git object and validates its digest against every selected singleton policy before submission. The legacy pilot launcher digest remains separately pinned.
### Repair source 4 (2026-11-03; candidate only)
The current target is the canonical revision above. This source adds remaining-original-deadline observation for primary guidance; independently expected normalized MCP results; fixed owned timeline-file reads through fixture board authority; finite supported CLI profile guidance with no identifier substitutions and fresh board/API/company/human readback; real completion-turn observation for plan review; and the reviewed true-price hire counterpart. Original failed attempts remain unchanged.
The app-review journey now requires inspection of the exact full redacted arguments and audit hash in the normal shared review card, then verifies that authorized provider dispatch equals that proposal. The 17-line product disclosure is an explicit candidate-only overlay from unmerged core PR #18734; no backend permission or approval rule changes. The independent core guidance PR #18717 also remains unmerged.
SDK public-source eligibility explicitly changes its expected restored lock from stale c7f230 to the authenticated credential-free build-input SHA256 `54a81b2ed3f1f85eaa085c034560ac248e384db34fbc7aba8c5a3846c29b3ce8`. Compiler/parser/Zod identities and all privacy/AST/custody guards remain enforced. This correction is not a plugin pass: case 115 must qualify with its original 300-second/one-native limits, and the other five SDK cases, including plugin 558's canary certificates, remain held. All case clocks and the controller model remain unchanged; Studio144's earlier explicit 2..3 accounting revision remains the only native-bound change.
tests/runner-e2e/chat-full-catalog-testops-manifest.json
+245 -58
View File
@@ -1,6 +1,6 @@
{
"schema": "paperclip.chat-full-catalog-testops-manifest/v1",
"targetSha": "43c44ccc823a8c972b247fa7159281887e6f5255",
"targetSha": "51ef450754370c8a8f2cbe4bd18e1536bbc93966",
"metadataSourceSha": "de0941835cc8f0d01c982574ee258e110babdbeb",
"profile": "runner-opencode",
"model": "openrouter/deepseek/deepseek-v4.1-flash-1017",
@@ -170,9 +170,9 @@
],
"repairMetadata": {
"baseTargetSha": "3b66c3c144360ddf32714a0742fd774b8ceafb16",
"previousTargetSha": "36dff2eab3dfb6de6efb4fd2ce6c1a58f8a5f383",
"previousTargetSha": "43c44ccc823a8c972b247fa7159281887e6f5255",
"definitionSourceSha": "de0941835cc8f0d01c982574ee258e110babdbeb",
"candidateDescriptorSha256": "04205b930fc0f8c93dee5d99f9a9531ae9799067e9c9b67e28ed8f694a43d403",
"candidateDescriptorSha256": "54760e1dca998c2ac2b6546f6daf5a13f399d2e9b125a465810b8fadf66c35d8",
"changedDefinitionFiles": [
{
"path": "tests/runner-e2e/chat-scenarios/companies.yaml",
@@ -191,7 +191,7 @@
"sha256": "18fbb9419bbdacb46eacdaa62dfa99feb9024faf2d1eef829aa70d43952575a9"
}
],
"conditions": "New candidate measurements only. Original 244 definitions, attempt history and old pass gate remain immutable. SDK audit custody remains diagnostic; scan rejection is unchanged. Explicit field overrides below retain original definition values. 144 alone declares reviewedmin2/max3 accounting and requires actual3 settled source-bound turns. All original clocks/models/human counts and all other nativebounds remain. No workflow/credential/reporting authority expansion.",
"conditions": "New candidate measurements only; original definitions and all failed attempts remain historical. Source4 repairs remaining original observation, normalized-result and owned fixture authorization/profile boundaries, exact full-proposal approval evidence and optional real completion observation. Product487 disclosure is a reviewed candidate-only overlay from UNMERGED18734. SDK privacy/AST guards are unchanged; expected restored lock now binds authenticated BUILD-input54a81 rather than stale c7f230, explicitly a new runtime eligibility condition. Only144 retains its earlier reviewed2..3 native-accounting revision with actual3 accepted; all original clocks/model/human counts and all other native bounds stay fixed. No core merge, workflow/credential/permission expansion or regrade",
"changedFields": [
{
"key": "agents.choose-primary",
@@ -252,6 +252,12 @@
"chat-scenarios/app-permissions.yaml"
]
},
{
"key": "app-permissions.ask-before-action",
"field": "oracleVersion",
"previous": "paperclip.chat-catalog.external.v1",
"current": "paperclip.chat-catalog.app-review.v2"
},
{
"key": "app-permissions.test-as-agent",
"field": "sourceFiles",
@@ -286,7 +292,7 @@
"key": "app-permissions.test-as-agent",
"field": "oracleVersion",
"previous": "paperclip.chat-catalog.external.v1",
"current": "paperclip.chat-catalog.app-test.v2"
"current": "paperclip.chat-catalog.app-test.v3"
},
{
"key": "app-permissions.test-as-agent",
@@ -300,6 +306,34 @@
"previous": "The agent routes to the supported test-as-agent control with the intended actor and action identified. The result reports actual effective access and provider outcome. A successful board test is described as that test, not proof that Morgan's next runtime has the tool admitted or that the CEO can impersonate Morgan.",
"current": "The canonical native request performs one supported, authorized test of the selected connection's sample-page read as Morgan and reports the actual result. Persisted source and target provenance plus the independent provider read prove the outcome. Existing caller and target grants remain unchanged; the test does not start Morgan's runtime, widen permissions, use another credential or create unrelated work. The successful test is not proof that Morgan's next runtime has the tool admitted."
},
{
"key": "cli-operations.choose-cli-context",
"field": "sourceFiles",
"previous": [
"chat-catalog-domains/privileged-support.ts",
"chat-catalog-domains/privileged-cli.ts",
"chat-scenarios/cli-operations.yaml",
"chat-catalog-domains/privileged-cli-support.ts",
"../../cli/src/index.ts"
],
"current": [
"chat-catalog-domains/privileged-support.ts",
"chat-catalog-domains/privileged-cli.ts",
"chat-scenarios/cli-operations.yaml",
"chat-catalog-domains/privileged-cli-support.ts",
"../../cli/src/index.ts",
"../../cli/src/commands/client/context.ts",
"../../cli/src/commands/client/company.ts",
"../../cli/src/commands/client/auth.ts",
"../../cli/src/client/context.ts"
]
},
{
"key": "cli-operations.choose-cli-context",
"field": "oracleVersion",
"previous": "paperclip.chat-catalog.privileged.v1",
"current": "paperclip.chat-catalog.cli-context.v2"
},
{
"key": "companies.agent-observability",
"field": "sourceFiles",
@@ -433,12 +467,64 @@
"previous": "paperclip.chat-catalog.identity.v1",
"current": "paperclip.chat-catalog.answer-delivery.v2"
},
{
"key": "questions-and-approvals.formal-hire-revision",
"field": "oracleVersion",
"previous": "paperclip.chat-catalog.hire-revision.native1017.v2",
"current": "paperclip.chat-catalog.hire-revision.native1017.v3"
},
{
"key": "questions-and-approvals.multi-part-question",
"field": "oracleVersion",
"previous": "paperclip.chat-catalog.identity.v1",
"current": "paperclip.chat-catalog.required-question.v2"
},
{
"key": "questions-and-approvals.reject-and-revise",
"field": "sourceFiles",
"previous": [
"chat-catalog-domains/identity-plan-review.ts",
"chat-catalog-domains/identity-support.ts",
"chat-catalog-domains/internal-support.ts",
"chat-scenarios/questions-and-approvals.yaml"
],
"current": [
"chat-catalog-domains/identity-plan-review.ts",
"chat-catalog-domains/identity-support.ts",
"chat-catalog-domains/internal-support.ts",
"chat-scenarios/questions-and-approvals.yaml",
"../../server/src/services/chat-completion-delivery.ts"
]
},
{
"key": "questions-and-approvals.reject-and-revise",
"field": "oracleVersion",
"previous": "paperclip.chat-catalog.identity.v1",
"current": "paperclip.chat-catalog.plan-review.v2"
},
{
"key": "questions-and-approvals.review-plan",
"field": "sourceFiles",
"previous": [
"chat-catalog-domains/identity-plan-review.ts",
"chat-catalog-domains/identity-support.ts",
"chat-catalog-domains/internal-support.ts",
"chat-scenarios/questions-and-approvals.yaml"
],
"current": [
"chat-catalog-domains/identity-plan-review.ts",
"chat-catalog-domains/identity-support.ts",
"chat-catalog-domains/internal-support.ts",
"chat-scenarios/questions-and-approvals.yaml",
"../../server/src/services/chat-completion-delivery.ts"
]
},
{
"key": "questions-and-approvals.review-plan",
"field": "oracleVersion",
"previous": "paperclip.chat-catalog.identity.v1",
"current": "paperclip.chat-catalog.plan-review.v2"
},
{
"key": "secrets.configure-company-secret",
"field": "oracleVersion",
@@ -710,10 +796,10 @@
]
},
{
"key": "app-permissions.test-as-agent",
"key": "app-permissions.ask-before-action",
"field": "oracleVersion",
"previous": "paperclip.chat-catalog.external.v1",
"current": "paperclip.chat-catalog.app-test.v2"
"current": "paperclip.chat-catalog.app-review.v2"
},
{
"key": "app-permissions.test-as-agent",
@@ -745,6 +831,40 @@
"chat-scenarios/app-permissions.yaml"
]
},
{
"key": "app-permissions.test-as-agent",
"field": "oracleVersion",
"previous": "paperclip.chat-catalog.external.v1",
"current": "paperclip.chat-catalog.app-test.v3"
},
{
"key": "cli-operations.choose-cli-context",
"field": "sourceFiles",
"previous": [
"chat-catalog-domains/privileged-support.ts",
"chat-catalog-domains/privileged-cli.ts",
"chat-scenarios/cli-operations.yaml",
"chat-catalog-domains/privileged-cli-support.ts",
"../../cli/src/index.ts"
],
"current": [
"chat-catalog-domains/privileged-support.ts",
"chat-catalog-domains/privileged-cli.ts",
"chat-scenarios/cli-operations.yaml",
"chat-catalog-domains/privileged-cli-support.ts",
"../../cli/src/index.ts",
"../../cli/src/commands/client/context.ts",
"../../cli/src/commands/client/company.ts",
"../../cli/src/commands/client/auth.ts",
"../../cli/src/client/context.ts"
]
},
{
"key": "cli-operations.choose-cli-context",
"field": "oracleVersion",
"previous": "paperclip.chat-catalog.privileged.v1",
"current": "paperclip.chat-catalog.cli-context.v2"
},
{
"key": "companies.agent-observability",
"field": "sourceFiles",
@@ -825,12 +945,6 @@
"previous": "paperclip.chat-catalog.internal.v1",
"current": "paperclip.chat-catalog.goal-hierarchy.v2"
},
{
"key": "navigation-preferences.dismiss-announcement",
"field": "oracleVersion",
"previous": "paperclip.chat-catalog.identity.v1",
"current": "paperclip.chat-catalog.announcement-dismissal.v2"
},
{
"key": "navigation-preferences.dismiss-announcement",
"field": "sourceFiles",
@@ -860,18 +974,76 @@
"../../server/src/routes/announcements.ts"
]
},
{
"key": "navigation-preferences.dismiss-announcement",
"field": "oracleVersion",
"previous": "paperclip.chat-catalog.identity.v1",
"current": "paperclip.chat-catalog.announcement-dismissal.v2"
},
{
"key": "questions-and-approvals.duplicate-answer",
"field": "oracleVersion",
"previous": "paperclip.chat-catalog.identity.v1",
"current": "paperclip.chat-catalog.answer-delivery.v2"
},
{
"key": "questions-and-approvals.formal-hire-revision",
"field": "oracleVersion",
"previous": "paperclip.chat-catalog.hire-revision.native1017.v2",
"current": "paperclip.chat-catalog.hire-revision.native1017.v3"
},
{
"key": "questions-and-approvals.multi-part-question",
"field": "oracleVersion",
"previous": "paperclip.chat-catalog.identity.v1",
"current": "paperclip.chat-catalog.required-question.v2"
},
{
"key": "questions-and-approvals.reject-and-revise",
"field": "sourceFiles",
"previous": [
"chat-catalog-domains/identity-plan-review.ts",
"chat-catalog-domains/identity-support.ts",
"chat-catalog-domains/internal-support.ts",
"chat-scenarios/questions-and-approvals.yaml"
],
"current": [
"chat-catalog-domains/identity-plan-review.ts",
"chat-catalog-domains/identity-support.ts",
"chat-catalog-domains/internal-support.ts",
"chat-scenarios/questions-and-approvals.yaml",
"../../server/src/services/chat-completion-delivery.ts"
]
},
{
"key": "questions-and-approvals.reject-and-revise",
"field": "oracleVersion",
"previous": "paperclip.chat-catalog.identity.v1",
"current": "paperclip.chat-catalog.plan-review.v2"
},
{
"key": "questions-and-approvals.review-plan",
"field": "sourceFiles",
"previous": [
"chat-catalog-domains/identity-plan-review.ts",
"chat-catalog-domains/identity-support.ts",
"chat-catalog-domains/internal-support.ts",
"chat-scenarios/questions-and-approvals.yaml"
],
"current": [
"chat-catalog-domains/identity-plan-review.ts",
"chat-catalog-domains/identity-support.ts",
"chat-catalog-domains/internal-support.ts",
"chat-scenarios/questions-and-approvals.yaml",
"../../server/src/services/chat-completion-delivery.ts"
]
},
{
"key": "questions-and-approvals.review-plan",
"field": "oracleVersion",
"previous": "paperclip.chat-catalog.identity.v1",
"current": "paperclip.chat-catalog.plan-review.v2"
},
{
"key": "secrets.configure-company-secret",
"field": "oracleVersion",
@@ -889,12 +1061,6 @@
]
}
},
{
"key": "skills.author-skill",
"field": "oracleVersion",
"previous": "paperclip.chat-catalog.integrator.v1",
"current": "paperclip.chat-catalog.skill-authoring.v2"
},
{
"key": "skills.author-skill",
"field": "sourceFiles",
@@ -913,6 +1079,12 @@
"chat-catalog-domains/identity-support.ts"
]
},
{
"key": "skills.author-skill",
"field": "oracleVersion",
"previous": "paperclip.chat-catalog.integrator.v1",
"current": "paperclip.chat-catalog.skill-authoring.v2"
},
{
"key": "skills.enable-for-agent",
"field": "sourceFiles",
@@ -1021,24 +1193,6 @@
"chat-scenarios/skills.yaml"
]
},
{
"key": "skills.studio-test",
"field": "expectedRunCount",
"previous": 2,
"current": 3
},
{
"key": "skills.studio-test",
"field": "minimumRunCount",
"previous": null,
"current": 2
},
{
"key": "skills.studio-test",
"field": "oracleVersion",
"previous": "paperclip.chat-catalog.integrator.v1",
"current": "paperclip.chat-catalog.skill-studio.v2"
},
{
"key": "skills.studio-test",
"field": "sourceFiles",
@@ -1059,6 +1213,24 @@
"chat-scenarios/skills.yaml"
]
},
{
"key": "skills.studio-test",
"field": "oracleVersion",
"previous": "paperclip.chat-catalog.integrator.v1",
"current": "paperclip.chat-catalog.skill-studio.v2"
},
{
"key": "skills.studio-test",
"field": "expectedRunCount",
"previous": 2,
"current": 3
},
{
"key": "skills.studio-test",
"field": "minimumRunCount",
"previous": null,
"current": 2
},
{
"key": "steering.stop-versus-pause",
"field": "setupPolicy",
@@ -1103,12 +1275,6 @@
]
}
},
{
"key": "teams.configure-team",
"field": "oracleVersion",
"previous": "paperclip.chat-catalog.external.v1",
"current": "paperclip.chat-catalog.team-configure.v2"
},
{
"key": "teams.configure-team",
"field": "sourceFiles",
@@ -1134,6 +1300,12 @@
"chat-flow.ts",
"chat-scenarios/teams.yaml"
]
},
{
"key": "teams.configure-team",
"field": "oracleVersion",
"previous": "paperclip.chat-catalog.external.v1",
"current": "paperclip.chat-catalog.team-configure.v2"
}
],
"runtimeConditions": {
@@ -1163,18 +1335,27 @@
"condition": "Reviewed synthetic staging input only; per-worker singleton550 enables it; production feed and global index unchanged."
},
"sdkPublicProof": {
"previousRegistrySha256": "4330369fd34812273db11762c82ce4c427bae2353991eae354e7923e229b4a4a",
"registrySha256": "018b5b15450e84f63920575de3b5903f7838678ef66a1e17aa050e0b21b40319",
"previousRegistrySha256": "018b5b15450e84f63920575de3b5903f7838678ef66a1e17aa050e0b21b40319",
"registrySha256": "32c7971e1d9c9d31cd6791045b0265f0475c8f10932ff6dd34681681b6a89342",
"inputCount": 506,
"runtimeCompilerParserZodLockIdentitiesUnchanged": true,
"scannerAndCanaryCertificatesUnchanged": true,
"actual115DetectorUnknown": true,
"plugin558StillHeld": true
"plugin558StillHeld": true,
"runtimeCompilerParserZodIdentitiesUnchanged": true,
"previousExpectedLockSha256": "c7f23021335b60da0cff1dd4c75cf8b948333ef3165d7d1703eb79a589edb798",
"expectedLockSha256": "54a81b2ed3f1f85eaa085c034560ac248e384db34fbc7aba8c5a3846c29b3ce8",
"historical43Finding": "detector3 unresolved_context/publicSource null; owning proof rejected stale expected lock",
"newCandidate115Unqualified": true
},
"toolProposal487": {
"corePr": 18734,
"coreHead": "17236e1736bbecdf04eeeee87ccb794c72d03da1",
"corePRUnmerged": true,
"condition": "Candidate-only exact shared-card bounded redacted full-argument/audit disclosure; no backend authority change. Actual human review/provider dispatch joined by oracle v2."
}
},
"candidateLauncherSnapshot": {
"path": "fixtures/chat-repair-candidate-launcher.txt",
"targetSha": "43c44ccc823a8c972b247fa7159281887e6f5255",
"targetSha": "51ef450754370c8a8f2cbe4bd18e1536bbc93966",
"sourcePath": "tests/runner-e2e/launch.ts",
"gitBlobOid": "dba1467e211611f37937e9e1fc4c0c5be4329cbc"
}
@@ -2792,7 +2973,7 @@
"questions-and-approvals/tool-review"
],
"checks": null,
"oracleVersion": "paperclip.chat-catalog.external.v1",
"oracleVersion": "paperclip.chat-catalog.app-review.v2",
"sourceFiles": [
"chat-catalog-domains/external-app-review.ts",
"chat-catalog-domains/external-provider.ts",
@@ -3057,7 +3238,7 @@
"app-permissions/test-action"
],
"checks": null,
"oracleVersion": "paperclip.chat-catalog.app-test.v2",
"oracleVersion": "paperclip.chat-catalog.app-test.v3",
"sourceFiles": [
"chat-catalog-domains/external-app-test.ts",
"chat-catalog-domains/external-provider.ts",
@@ -4569,13 +4750,17 @@
"cli-operations/context"
],
"checks": null,
"oracleVersion": "paperclip.chat-catalog.privileged.v1",
"oracleVersion": "paperclip.chat-catalog.cli-context.v2",
"sourceFiles": [
"chat-catalog-domains/privileged-support.ts",
"chat-catalog-domains/privileged-cli.ts",
"chat-scenarios/cli-operations.yaml",
"chat-catalog-domains/privileged-cli-support.ts",
"../../cli/src/index.ts"
"../../cli/src/index.ts",
"../../cli/src/commands/client/context.ts",
"../../cli/src/commands/client/company.ts",
"../../cli/src/commands/client/auth.ts",
"../../cli/src/client/context.ts"
],
"screenshot": "final-state.png",
"evidenceKind": "private-final-state-png",
@@ -9230,7 +9415,7 @@
"agents/hire"
],
"checks": null,
"oracleVersion": "paperclip.chat-catalog.hire-revision.native1017.v2",
"oracleVersion": "paperclip.chat-catalog.hire-revision.native1017.v3",
"sourceFiles": [
"chat-catalog-domains/identity-hire-revision.ts",
"chat-catalog-domains/identity-hire-model-choice.ts",
@@ -9358,12 +9543,13 @@
"questions-and-approvals/confirmations"
],
"checks": null,
"oracleVersion": "paperclip.chat-catalog.identity.v1",
"oracleVersion": "paperclip.chat-catalog.plan-review.v2",
"sourceFiles": [
"chat-catalog-domains/identity-plan-review.ts",
"chat-catalog-domains/identity-support.ts",
"chat-catalog-domains/internal-support.ts",
"chat-scenarios/questions-and-approvals.yaml"
"chat-scenarios/questions-and-approvals.yaml",
"../../server/src/services/chat-completion-delivery.ts"
],
"screenshot": "final-state.png",
"evidenceKind": "private-final-state-png",
@@ -9402,12 +9588,13 @@
"agent-chat/handoff"
],
"checks": null,
"oracleVersion": "paperclip.chat-catalog.identity.v1",
"oracleVersion": "paperclip.chat-catalog.plan-review.v2",
"sourceFiles": [
"chat-catalog-domains/identity-plan-review.ts",
"chat-catalog-domains/identity-support.ts",
"chat-catalog-domains/internal-support.ts",
"chat-scenarios/questions-and-approvals.yaml"
"chat-scenarios/questions-and-approvals.yaml",
"../../server/src/services/chat-completion-delivery.ts"
],
"screenshot": "final-state.png",
"evidenceKind": "private-final-state-png",
tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs
+39 -4
View File
@@ -102,8 +102,8 @@ test('foreign identity, duplicated case and changed run-bound metadata are rejec
test('repair source metadata binds the corrected candidate and explicit changed definitions', () => {
const repair = manifest.source.repairMetadata;
assert.equal(repair.baseTargetSha, '3b66c3c144360ddf32714a0742fd774b8ceafb16');
assert.equal(repair.previousTargetSha, '36dff2eab3dfb6de6efb4fd2ce6c1a58f8a5f383');
assert.equal(manifest.targetSha, '43c44ccc823a8c972b247fa7159281887e6f5255');
assert.equal(repair.previousTargetSha, '43c44ccc823a8c972b247fa7159281887e6f5255');
assert.equal(manifest.targetSha, '51ef450754370c8a8f2cbe4bd18e1536bbc93966');
assert.equal(repair.definitionSourceSha, manifest.metadataSourceSha);
assert.equal(repair.candidateDescriptorSnapshot, 'fixtures/chat-repair-candidate-descriptors.json');
const descriptorBytes = readFileSync(new URL(repair.candidateDescriptorSnapshot, import.meta.url));
@@ -120,6 +120,10 @@ test('repair source metadata binds the corrected candidate and explicit changed
"app-permissions.ask-before-action",
"sourceFiles"
],
[
"app-permissions.ask-before-action",
"oracleVersion"
],
[
"app-permissions.test-as-agent",
"sourceFiles"
@@ -136,6 +140,14 @@ test('repair source metadata binds the corrected candidate and explicit changed
"app-permissions.test-as-agent",
"expectedOutcome"
],
[
"cli-operations.choose-cli-context",
"sourceFiles"
],
[
"cli-operations.choose-cli-context",
"oracleVersion"
],
[
"companies.agent-observability",
"sourceFiles"
@@ -176,10 +188,30 @@ test('repair source metadata binds the corrected candidate and explicit changed
"questions-and-approvals.duplicate-answer",
"oracleVersion"
],
[
"questions-and-approvals.formal-hire-revision",
"oracleVersion"
],
[
"questions-and-approvals.multi-part-question",
"oracleVersion"
],
[
"questions-and-approvals.reject-and-revise",
"sourceFiles"
],
[
"questions-and-approvals.reject-and-revise",
"oracleVersion"
],
[
"questions-and-approvals.review-plan",
"sourceFiles"
],
[
"questions-and-approvals.review-plan",
"oracleVersion"
],
[
"secrets.configure-company-secret",
"oracleVersion"
@@ -241,8 +273,8 @@ test('repair source metadata binds the corrected candidate and explicit changed
assert.deepEqual(row(change.key)[change.field], change.current);
assert.notDeepEqual(change.current, change.previous);
}
assert.equal(repair.descriptorChanges.length, 30);
assert.equal(new Set(repair.descriptorChanges.map(change => change.key)).size, 23);
assert.equal(repair.descriptorChanges.length, 38);
assert.equal(new Set(repair.descriptorChanges.map(change => change.key)).size, 27);
assert.deepEqual([...new Set(repair.descriptorChanges.map(change => change.field))].sort(), ['expectedRunCount', 'minimumRunCount', 'oracleVersion', 'setupPolicy', 'sourceFiles']);
for (const change of repair.descriptorChanges) {
assert.notDeepEqual(change.current, change.previous);
@@ -269,4 +301,7 @@ test('repair source metadata binds the corrected candidate and explicit changed
assert.equal(repair.runtimeConditions.toolGuidance106.corePRUnmerged, true);
assert.equal(repair.runtimeConditions.sdkPublicProof.scannerAndCanaryCertificatesUnchanged, true);
assert.equal(repair.runtimeConditions.sdkPublicProof.plugin558StillHeld, true);
assert.equal(repair.runtimeConditions.sdkPublicProof.expectedLockSha256, "54a81b2ed3f1f85eaa085c034560ac248e384db34fbc7aba8c5a3846c29b3ce8");
assert.equal(repair.runtimeConditions.sdkPublicProof.runtimeCompilerParserZodIdentitiesUnchanged, true);
assert.equal(repair.runtimeConditions.toolProposal487.corePRUnmerged, true);
});
tests/runner-e2e/fixtures/chat-repair-candidate-descriptors.json
+15 -9
View File
@@ -822,7 +822,7 @@
"../../ui/src/components/task-chat/TaskChatComposer.tsx",
"chat-scenarios/app-permissions.yaml"
],
"oracleVersion": "paperclip.chat-catalog.external.v1",
"oracleVersion": "paperclip.chat-catalog.app-review.v2",
"expectedRunCount": 6,
"expectedHumanMessageCount": 3,
"minimumRunCount": 6,
@@ -956,7 +956,7 @@
"../../server/src/services/tool-approval-execution.ts",
"chat-scenarios/app-permissions.yaml"
],
"oracleVersion": "paperclip.chat-catalog.app-test.v2",
"oracleVersion": "paperclip.chat-catalog.app-test.v3",
"expectedRunCount": 1,
"turnTimeoutMs": 120000,
"attemptTimeoutMs": 300000,
@@ -1823,9 +1823,13 @@
"chat-catalog-domains/privileged-cli.ts",
"chat-scenarios/cli-operations.yaml",
"chat-catalog-domains/privileged-cli-support.ts",
"../../cli/src/index.ts"
"../../cli/src/index.ts",
"../../cli/src/commands/client/context.ts",
"../../cli/src/commands/client/company.ts",
"../../cli/src/commands/client/auth.ts",
"../../cli/src/client/context.ts"
],
"oracleVersion": "paperclip.chat-catalog.privileged.v1",
"oracleVersion": "paperclip.chat-catalog.cli-context.v2",
"coverageGaps": [],
"expectedRunCount": 1,
"turnTimeoutMs": 120000,
@@ -4194,7 +4198,7 @@
"chat-catalog-domains/internal-support.ts",
"chat-scenarios/questions-and-approvals.yaml"
],
"oracleVersion": "paperclip.chat-catalog.hire-revision.native1017.v2",
"oracleVersion": "paperclip.chat-catalog.hire-revision.native1017.v3",
"expectedRunCount": 1,
"turnTimeoutMs": 120000,
"attemptTimeoutMs": 300000,
@@ -4239,9 +4243,10 @@
"chat-catalog-domains/identity-plan-review.ts",
"chat-catalog-domains/identity-support.ts",
"chat-catalog-domains/internal-support.ts",
"chat-scenarios/questions-and-approvals.yaml"
"chat-scenarios/questions-and-approvals.yaml",
"../../server/src/services/chat-completion-delivery.ts"
],
"oracleVersion": "paperclip.chat-catalog.identity.v1",
"oracleVersion": "paperclip.chat-catalog.plan-review.v2",
"expectedRunCount": 3,
"minimumRunCount": 1,
"turnTimeoutMs": 120000,
@@ -4260,9 +4265,10 @@
"chat-catalog-domains/identity-plan-review.ts",
"chat-catalog-domains/identity-support.ts",
"chat-catalog-domains/internal-support.ts",
"chat-scenarios/questions-and-approvals.yaml"
"chat-scenarios/questions-and-approvals.yaml",
"../../server/src/services/chat-completion-delivery.ts"
],
"oracleVersion": "paperclip.chat-catalog.identity.v1",
"oracleVersion": "paperclip.chat-catalog.plan-review.v2",
"expectedRunCount": 3,
"minimumRunCount": 2,
"turnTimeoutMs": 120000,