mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-09 07:10:34 +01:00
test(evals): pin timeline and governed Company approval at native cap two (#19381)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The chat catalog uses a fixed source and explicit execution conditions. > - A correct timeline response failed because its completed edit and attached command result were not recognized. > - Company credential setup can require a real human approval after the secure save. > - The controller must declare that optional approval and its run bounds before it admits the source. > - This pull request pins the reviewed source and caps native catalog overlap at two. > - The next run can test these conditions without treating local controls as native acceptance. ## Linked Issues or Issue Description Refs: #19357, #19086, #18971, #19093. This follows the prior fixed-source catalog pin. It changes only controller metadata, admission, tests, and load policy. The complete catalog and separate core feature PRs remain unmerged. ## What Changed - Pin immutable source `fe547be84826b4611d3ba9ee1a7d7aa0bd0a2843`, tree `a23723ea5244e6ce8cee11e9c4b476acb6978fcd`. - Advance only timeline v10 to v11 and Company v11 to v12. Add the declared Company proposal helper to its source closure. - Declare Company minimum two and maximum three native runs. Retain two base human inputs and permit at most one independently verified UI approval. Retain its original 480000ms attempt. - Carry the literal Company-only approval marker from the descriptor into the manifest. Reject foreign markers, changed bounds, and a mismatch with static source task metadata. - Cap native catalog execution at two workers. Keep the external lane serial and retain zero retries. - Keep all244 requests, contexts, expected outcomes, model, clocks, credentials and other case bounds unchanged. ## Verification - Source:421 focused controls across seven suites pass. Full runner E2E TypeScript check passes. - Explicit model-free approval UI canary passes one test with three local scenarios. It checks accepted approval, wrong human, and failed response. The first sandbox attempt stopped at Chrome launch; the allowed execution passed. - Controller:40 manifest, adapter and reconciliation controls pass. - SDK506 inputs remain byte-identical to the accepted registry `efa70621e9fb42aa185f7ee616a1a726aa3eccb367e67e28817cedf26ef2ade8`. - Launcher and trusted private evidence wire remain byte-identical. Source contract/flow accounting changes are explicit. - Source descriptor SHA256: `50e90d8c02ad57f444e27b522efe694db9734d47bdbc3a058475af77bac8b2ca`. - Controller manifest SHA256: `0447315601ed21ba15d6ea0ecd7fb89bc79ad6262db2594cd12ac774aa8d6c21`. - Review the two changed rows. The next planned experiment selects timeline and Company once at cap2 before a full ten-case run. No native run on this source has passed yet. - Exact controller head `d49dfc59ab7fddfe1a1de7e81bdb9c4e13400f30`: normal CI has56 successful checks and three configured skips. Protected `ci / verify` and `ci / e2e` both pass. Greptile5/5 and zero open threads are verified for this head. ## Risks - Original timeline5056 and Company5057 remain FAILED. Timeline5056 supplied the correct account; the local revised check passes its diagnostic only. Company5057 saved the secret but left the binding approval pending. - Company native3 completion within480s is unproved. This change does not raise its clock or claim to fix latency. - Two-worker execution is a changed load condition. It does not prove a concurrency cause, reliability, or eight-worker qualification. - Secret journeys require independent semantic safety review before pass credit. The documented closing-comma warning and arbitrary metadata-prefill classifier limitations remain follow-ups in the functional secret PR; a native failure is not waived. - Candidate core PRs remain unmerged. The full-catalog196/244 admission gate remains fixed. Local controls do not qualify all244 journeys. - Local checks use Node26.4.0. They do not replace actual AWS runtime qualification. - The separate Paperclip Review app remains queued while its assigned task waits for maintenance. It has no completed assessment. GitHub reports `UNSTABLE` and `MERGEABLE`; this is not an all-checks PASS claim. No review retry or staging mutation was performed. ## Model Used OpenAI Codex, GPT-6.1 Sol (`gpt-6.1-sol`), with reasoning, repository tools and code execution. The host does not expose an exact context-window size. No model or paid provider was called for local controls. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed
1 parent
7440cfc9b1
commit
46414fcfa4
11 files changed
+165
-30
No files matched your search
@@ -261,11 +261,17 @@ Initial setup uses the existing reporting environment:
|
||||
evaluation after an uncertain request. Initial route verification uses a small
|
||||
selection and does not qualify all 244 cases.
|
||||
|
||||
Each worker executes one case. Selections of up to three cases allow two cases
|
||||
at once. Larger selections allow eight independent cases and one external-input
|
||||
case at once. External-input cases remain serial because they can share QA
|
||||
resources. A full run can take hours. Its elapsed time depends on setup and fleet
|
||||
capacity; the declared case clocks are not a completion-time guarantee.
|
||||
Each worker executes one case. Native cases allow two workers at once for any
|
||||
selection size. Selections of up to three cases that include external input keep
|
||||
the native lane at one worker. The separate external-input lane remains serial
|
||||
because those cases can share QA resources. A larger mixed selection can therefore
|
||||
run two native cases and one external-input case at once.
|
||||
|
||||
This lower-load configuration changes scheduling, not case clocks, model settings
|
||||
or zero automatic retries. It does not qualify eight-way native execution or
|
||||
establish a cause for earlier noncompletion. A full run can take hours. Its elapsed
|
||||
time depends on setup and fleet capacity; declared case clocks are not a
|
||||
completion-time guarantee.
|
||||
|
||||
Each finished worker starts its own protected reader and safe exporter. A
|
||||
separate reporting job uploads that case and verifies its result and two JSON
|
||||
@@ -335,3 +341,13 @@ The canonical candidate above retains Source5 and adds initial personal-secret a
|
||||
The reviewed product overlay from unmerged core PR #18742 at `61996a96dd09e90a5788e8711a94fc5ae631f425` allows a new Ask-first review for a strictly authenticated later same-person human intent following a declined request, including the normal reopened-comment source. Repeating the same intent stays denied; this grants no approval or execution authority. Both PostgreSQL route regressions ran and passed on that core head. The candidate preserves the compatible gateway and actual database row contracts; this is not a core merge or a qualified chat pass. Core PRs #18717, #18734 and #18742 remain unmerged. SDK115 completion remains unknown; the other five SDK cases remain held.
|
||||
|
||||
For one selected case, the trusted controller downloads each exact bundle and publication artifact name into an explicit matching child directory. For multiple cases, it keeps separate named artifact directories. The gather command authenticates the frozen selection, every prepared bundle and its hashes before it copies the safe projections. Final coverage still authenticates the original publication receipts and verifies TestOps results without another export or upload. Empty, mixed, foreign and symbolic-link layouts fail closed. The producer's optional once-upload marker has an exact closed schema; it does not replace a verified publication receipt.
|
||||
|
||||
|
||||
The optional Company-secret binding approval is declared only by
|
||||
`secrets.configure-company-secret` with `optionalCompanySecretBindingApproval: true`.
|
||||
Its two base human inputs may be followed by one independently verified UI approval
|
||||
(maximum three actual human inputs). Its native bounds are explicitly 2..3 and its
|
||||
original 480000ms attempt, model and zero automatic retries remain unchanged.
|
||||
The controller copies this static marker from the candidate descriptor and checks
|
||||
it against source task metadata; metadata extraction never executes fixture hooks.
|
||||
This conditional accounting does not pre-complete approval or qualify a live pass.
|
||||
@@ -37,7 +37,7 @@ export function executionPartition(selection, context, jobRunId) {
|
||||
const size = [matrix, native, external, selection.cases.map(row => row.executionId)]
|
||||
.reduce((sum, value) => sum + JSON.stringify(value).length * 2, 0);
|
||||
if (size > 1_000_000) reject('matrix_size');
|
||||
return { matrix, native, external, parallelism: small ? external.include.length ? '1' : '2' : '8' };
|
||||
return { matrix, native, external, parallelism: small && external.include.length ? '1' : '2' };
|
||||
}
|
||||
async function readOwned(file, root, limit) {
|
||||
const absolute = path.resolve(file), base = await realpath(root);
|
||||
|
||||
@@ -10,7 +10,7 @@ export interface PilotCase {
|
||||
executionId: string;
|
||||
fullName: string;
|
||||
}
|
||||
export type CatalogCase = Omit<PilotCase, "checks"> & { checks: number | null };
|
||||
export type CatalogCase = Omit<PilotCase, "checks"> & { checks: number | null; optionalCompanySecretBindingApproval?: true; expectedHumanMessageCount?: 2 };
|
||||
export const cases: readonly Readonly<PilotCase>[];
|
||||
export const targetSha: string;
|
||||
export const launchSha256: string;
|
||||
|
||||
@@ -3,6 +3,7 @@ import { mkdtemp, readFile, realpath, rm } from 'node:fs/promises';
|
||||
import path from 'node:path';
|
||||
import { pathToFileURL } from 'node:url';
|
||||
import { parse } from 'yaml';
|
||||
import { validateCompanySecretBindingApprovalSource } from './chat-full-catalog-testops-manifest.mjs';
|
||||
import { model, catalogDefinition, catalogWorkflow } from './chat-completion-testops-contract.mjs';
|
||||
|
||||
export function paidWorker(definition, called) {
|
||||
@@ -55,6 +56,7 @@ export async function verifyHostContract(workspace, temporaryRoot, env = process
|
||||
|| execution.task.attemptTimeoutMs.local !== descriptor.attemptTimeoutMs
|
||||
|| execution.suite.definitionMetadata?.companyAndAgentBudgetCents !== 500
|
||||
|| execution.suite.definitionMetadata?.maximumAttemptsPerCell !== 1)) throw new Error('Pinned source selection mismatch');
|
||||
if (full) validateCompanySecretBindingApprovalSource(descriptor, execution.task);
|
||||
const controlEnv = { GITHUB_ACTIONS: 'true', GITHUB_REPOSITORY: 'paperclipai/paperclip', GITHUB_REF: 'refs/heads/master',
|
||||
GITHUB_WORKFLOW_REF: `paperclipai/paperclip/${workflow}@refs/heads/master`, GITHUB_RUN_ID: '1', GITHUB_RUN_ATTEMPT: '1',
|
||||
PAPERCLIP_RUNNER_E2E_SOURCE_SHA: targetSha, PAPERCLIP_CHAT_COMPLETION_TESTOPS: paid.env.PAPERCLIP_CHAT_COMPLETION_TESTOPS,
|
||||
|
||||
@@ -120,7 +120,7 @@ test('full catalog source admission is independent from the old three-case workf
|
||||
});
|
||||
|
||||
test('native and external partitions preserve exact selections, bounded overlapping small runs and the GitHub output bound', () => {
|
||||
for (const rows of [selected, selected.slice(0, 3), selected.filter(row => row.externalInput), [selected[0], selected.find(row => row.externalInput)]]) {
|
||||
for (const rows of [selected, selected.filter(row => !row.externalInput).slice(0, 10), selected.slice(0, 3), selected.filter(row => row.externalInput), [selected[0], selected.find(row => row.externalInput)]]) {
|
||||
const input = { ...selection, cases: rows }, partition = executionPartition(input, context, '77');
|
||||
const merged = [...partition.native.include, ...partition.external.include].map(row => row.executionId);
|
||||
assert.equal(new Set(merged).size, rows.length);
|
||||
@@ -129,7 +129,7 @@ test('native and external partitions preserve exact selections, bounded overlapp
|
||||
assert.equal(partition.parallelism, rows.some(row => row.externalInput) ? '1' : '2');
|
||||
assert.equal(partition.external.include.length, rows.filter(row => row.externalInput).length);
|
||||
} else {
|
||||
assert.equal(partition.parallelism, '8'); assert.ok(partition.external.include.every(row => row.external));
|
||||
assert.equal(partition.parallelism, '2'); assert.ok(partition.external.include.every(row => row.external));
|
||||
}
|
||||
assert.ok([partition.matrix, partition.native, partition.external, rows.map(row => row.executionId)]
|
||||
.reduce((sum, value) => sum + JSON.stringify(value).length * 2, 0) < 1_000_000);
|
||||
|
||||
@@ -4,3 +4,6 @@ export const catalogLauncherSha256: string;
|
||||
export const catalogManifestSha256: string;
|
||||
export const fullCatalogPlanName: string;
|
||||
export const fullCatalogCases: readonly Readonly<CatalogCase>[];
|
||||
export function companySecretBindingApprovalMetadata(declaration: Record<string, unknown>):
|
||||
{ optionalCompanySecretBindingApproval?: true; expectedHumanMessageCount?: 2 };
|
||||
export function validateCompanySecretBindingApprovalSource(row: Record<string, unknown>, task: Record<string, unknown>): void;
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"schema": "paperclip.chat-full-catalog-testops-manifest/v1",
|
||||
"targetSha": "d1ab241cac0db7f700fadaaf62ffb3d2b51cf0c3",
|
||||
"targetSha": "fe547be84826b4611d3ba9ee1a7d7aa0bd0a2843",
|
||||
"metadataSourceSha": "de0941835cc8f0d01c982574ee258e110babdbeb",
|
||||
"profile": "runner-opencode",
|
||||
"model": "openrouter/deepseek/deepseek-v4.1-flash-1017",
|
||||
@@ -172,7 +172,7 @@
|
||||
"baseTargetSha": "3b66c3c144360ddf32714a0742fd774b8ceafb16",
|
||||
"previousTargetSha": "ab2019e976076ec0eb60b026844f3d9211cb0208",
|
||||
"definitionSourceSha": "de0941835cc8f0d01c982574ee258e110babdbeb",
|
||||
"candidateDescriptorSha256": "60db015337e807ff53a376b6db11c375a239097b224d2253dad68ecd0b0f61f3",
|
||||
"candidateDescriptorSha256": "50e90d8c02ad57f444e27b522efe694db9734d47bdbc3a058475af77bac8b2ca",
|
||||
"changedDefinitionFiles": [
|
||||
{
|
||||
"path": "tests/runner-e2e/chat-scenarios/companies.yaml",
|
||||
@@ -199,7 +199,7 @@
|
||||
"sha256": "91a9ddc4f5357f19a2cdc73ff8a090c741d14780e4ed502c856ebd557986f7e1"
|
||||
}
|
||||
],
|
||||
"conditions": "Historical qualification conditions (retained; statements that all requests or sourceFiles remain fixed refer to those earlier successors): New candidate measurements only. Original definitions, prior failed attempts and source6 ab201 remain historical. This target retains that catalog and integrates merged runtime fixes #18711, #18713 and #18718, with the prior-run restoration condition adapted to its existing owner. It adds the reviewed previous-question locator, attributed human-only denial and conditional-outcome oracle, and actor-sensitive timeline narrative/recorded-reassignment fixes. Closed finish diagnostics retain an effective schema digest, hashed call correlation and bounded type/category enums only. Historical MCP timeout and continuation-shape causes remain unresolved; no new all-ten PASS is claimed. Queries, model, human/native counts and original clocks remain fixed; the earlier Studio144 explicit 2..3 accounting stays. Core PRs18717/18734/18742 are now merged. SDK115 completion remains UNKNOWN and the other five SDK cases remain held. Full244 runtime catalog is not merged to master and the196/244 gate is unchanged. This successor binds the seven reviewed scenario repairs plus framework evidence: canonical skill navigation, original announcement phase clock, attributed timeline context, bound hire revision history, credential warning and configuration-question admission, completed team visibility, and canonical private framework evidence. Twelve declaration fields across eleven cases change only oracle versions and skill source-file provenance. All original requests, setup, human/native bounds, model and clocks remain fixed. Historical failed attempts and missing-original UNKNOWN grades remain unchanged; no uniform-ten PASS is claimed. The existing SDK506/efa qualification is reused only for byte-identical registered source inputs. This next successor adds reviewed question-surface observation, grounded scenario grading and safe credential-guidance boundaries after the retained uniform attempt. Eleven existing oracle-version leaves advance; all sourceFiles declarations, requests, setup, counts, clocks, model, launcher and private protocol helpers remain fixed. Component qualification does not attest a new uniform-ten pass.\n\nPrevious successor conditions (retained): This successor measures five reviewed repairs at the retained original model and bounds. Exactly one request is clarified: multipart53 now explicitly asks to walk through the visible form; its original broad request and failures4623/4717 remain historical. The previous inventory0cf530 is retained and the disclosed clarified inventory is814bb363. Other243 catalog requests are unchanged. The company-secret second human message now references the actual saved non-secret metadata instead of repeating the initial request; native continuation must remain attributed to the actual agent. Five oracle-version leaves and four source-file additions change; launcher, private helpers, SDK506/efa, identities, counts, clocks and the196/244 gate remain fixed. All five successor journeys remain unrun; component proof is not a PASS. For Team610, independently saved canonical skill/version/member proof establishes recovery; no required recovery-success phrase is imposed. A contradictory current denial still fails. If the required skill remains missing, truthful disclosure of that condition remains required.\n\nPrevious retained successor conditions: This target adds the reviewed personal secure-entry confirmation handoff, the typographic team partial-warning observer correction, and the company-secret authorization fix from unmerged core PR #19325. Personal v10 retains two declared logical human actions: one canonical chat request plus one independently verified same-human secure-save confirmation handoff. Raw chat comments remain one. Team v5 accepts the actual named-skill warning with a typographic apostrophe; saved-state, repeat, runtime, link, and false-success checks remain required. The company route returns scoped 403 guidance before malformed-body validation and preserves board-only creation. Its effect on the unfinished native turn is unproven. The three affected journeys are unrun on this target. Existing failed attempts remain failed. All queries, expected outcomes, native and human bounds, clocks, profile, model, launcher, private protocol, SDK506/efa, and the 196/244 admission gate remain fixed, including the recorded case53 query clarification. Local component proof is not native feature qualification or main/core merge credit.\n\nPrevious retained successor conditions: This target repairs personal secure-entry observation and truthful human acknowledgment, company secure-entry language and ordinary visible menu navigation, team current-skill claim attribution, and native completion feedback that asks for usable product links. Personal v11 expands visible Details before entry when needed, binds the exact value-enLine truncated
|
||||
"conditions": "Historical qualification conditions (retained; statements that all requests or sourceFiles remain fixed refer to those earlier successors): New candidate measurements only. Original definitions, prior failed attempts and source6 ab201 remain historical. This target retains that catalog and integrates merged runtime fixes #18711, #18713 and #18718, with the prior-run restoration condition adapted to its existing owner. It adds the reviewed previous-question locator, attributed human-only denial and conditional-outcome oracle, and actor-sensitive timeline narrative/recorded-reassignment fixes. Closed finish diagnostics retain an effective schema digest, hashed call correlation and bounded type/category enums only. Historical MCP timeout and continuation-shape causes remain unresolved; no new all-ten PASS is claimed. Queries, model, human/native counts and original clocks remain fixed; the earlier Studio144 explicit 2..3 accounting stays. Core PRs18717/18734/18742 are now merged. SDK115 completion remains UNKNOWN and the other five SDK cases remain held. Full244 runtime catalog is not merged to master and the196/244 gate is unchanged. This successor binds the seven reviewed scenario repairs plus framework evidence: canonical skill navigation, original announcement phase clock, attributed timeline context, bound hire revision history, credential warning and configuration-question admission, completed team visibility, and canonical private framework evidence. Twelve declaration fields across eleven cases change only oracle versions and skill source-file provenance. All original requests, setup, human/native bounds, model and clocks remain fixed. Historical failed attempts and missing-original UNKNOWN grades remain unchanged; no uniform-ten PASS is claimed. The existing SDK506/efa qualification is reused only for byte-identical registered source inputs. This next successor adds reviewed question-surface observation, grounded scenario grading and safe credential-guidance boundaries after the retained uniform attempt. Eleven existing oracle-version leaves advance; all sourceFiles declarations, requests, setup, counts, clocks, model, launcher and private protocol helpers remain fixed. Component qualification does not attest a new uniform-ten pass.\n\nPrevious successor conditions (retained): This successor measures five reviewed repairs at the retained original model and bounds. Exactly one request is clarified: multipart53 now explicitly asks to walk through the visible form; its original broad request and failures4623/4717 remain historical. The previous inventory0cf530 is retained and the disclosed clarified inventory is814bb363. Other243 catalog requests are unchanged. The company-secret second human message now references the actual saved non-secret metadata instead of repeating the initial request; native continuation must remain attributed to the actual agent. Five oracle-version leaves and four source-file additions change; launcher, private helpers, SDK506/efa, identities, counts, clocks and the196/244 gate remain fixed. All five successor journeys remain unrun; component proof is not a PASS. For Team610, independently saved canonical skill/version/member proof establishes recovery; no required recovery-success phrase is imposed. A contradictory current denial still fails. If the required skill remains missing, truthful disclosure of that condition remains required.\n\nPrevious retained successor conditions: This target adds the reviewed personal secure-entry confirmation handoff, the typographic team partial-warning observer correction, and the company-secret authorization fix from unmerged core PR #19325. Personal v10 retains two declared logical human actions: one canonical chat request plus one independently verified same-human secure-save confirmation handoff. Raw chat comments remain one. Team v5 accepts the actual named-skill warning with a typographic apostrophe; saved-state, repeat, runtime, link, and false-success checks remain required. The company route returns scoped 403 guidance before malformed-body validation and preserves board-only creation. Its effect on the unfinished native turn is unproven. The three affected journeys are unrun on this target. Existing failed attempts remain failed. All queries, expected outcomes, native and human bounds, clocks, profile, model, launcher, private protocol, SDK506/efa, and the 196/244 admission gate remain fixed, including the recorded case53 query clarification. Local component proof is not native feature qualification or main/core merge credit.\n\nPrevious retained successor conditions: This target repairs personal secure-entry observation and truthful human acknowledgment, company secure-entry language and ordinary visible menu navigation, team current-skill claim attribution, and native completion feedback that asks for usable product links. Personal v11 expands visible Details before entry when needed, binds the exact value-enLine truncated
|
||||
"changedFields": [
|
||||
{
|
||||
"key": "activity.work-timeline",
|
||||
@@ -236,7 +236,7 @@
|
||||
"key": "activity.work-timeline",
|
||||
"field": "oracleVersion",
|
||||
"previous": "paperclip.chat-catalog.internal.v1",
|
||||
"current": "paperclip.chat-catalog.work-timeline.v10"
|
||||
"current": "paperclip.chat-catalog.work-timeline.v11"
|
||||
},
|
||||
{
|
||||
"key": "activity.work-timeline",
|
||||
@@ -620,6 +620,7 @@
|
||||
],
|
||||
"current": [
|
||||
"chat-catalog-domains/external-company-secret.ts",
|
||||
"chat-catalog-domains/external-secret-proposal.ts",
|
||||
"chat-catalog-domains/external-secret-question.ts",
|
||||
"chat-catalog-domains/external-secret-route.ts",
|
||||
"chat-catalog-domains/external-company-secret-menu.ts",
|
||||
@@ -643,7 +644,7 @@
|
||||
"key": "secrets.configure-company-secret",
|
||||
"field": "oracleVersion",
|
||||
"previous": "paperclip.chat-catalog.external.v1",
|
||||
"current": "paperclip.chat-catalog.company-secret-controller-baseline.v11"
|
||||
"current": "paperclip.chat-catalog.company-secret-controller-baseline.v12"
|
||||
},
|
||||
{
|
||||
"key": "secrets.personal-secret",
|
||||
@@ -950,6 +951,24 @@
|
||||
"field": "originalQuery",
|
||||
"previous": "Help me choose the details for the launch page before you build it.",
|
||||
"current": "Walk me through the launch-page question form so I can choose the audience and tone and enter my own headline before you build anything."
|
||||
},
|
||||
{
|
||||
"key": "secrets.configure-company-secret",
|
||||
"field": "maximumRuns",
|
||||
"previous": 2,
|
||||
"current": 3
|
||||
},
|
||||
{
|
||||
"key": "secrets.configure-company-secret",
|
||||
"field": "optionalCompanySecretBindingApproval",
|
||||
"previous": null,
|
||||
"current": true
|
||||
},
|
||||
{
|
||||
"key": "secrets.configure-company-secret",
|
||||
"field": "expectedHumanMessageCount",
|
||||
"previous": null,
|
||||
"current": 2
|
||||
}
|
||||
],
|
||||
"candidateDescriptorSnapshot": "fixtures/chat-repair-candidate-descriptors.json",
|
||||
@@ -989,7 +1008,7 @@
|
||||
"key": "activity.work-timeline",
|
||||
"field": "oracleVersion",
|
||||
"previous": "paperclip.chat-catalog.internal.v1",
|
||||
"current": "paperclip.chat-catalog.work-timeline.v10"
|
||||
"current": "paperclip.chat-catalog.work-timeline.v11"
|
||||
},
|
||||
{
|
||||
"key": "agents.choose-primary",
|
||||
@@ -1343,6 +1362,7 @@
|
||||
],
|
||||
"current": [
|
||||
"chat-catalog-domains/external-company-secret.ts",
|
||||
"chat-catalog-domains/external-secret-proposal.ts",
|
||||
"chat-catalog-domains/external-secret-question.ts",
|
||||
"chat-catalog-domains/external-secret-route.ts",
|
||||
"chat-catalog-domains/external-company-secret-menu.ts",
|
||||
@@ -1366,7 +1386,7 @@
|
||||
"key": "secrets.configure-company-secret",
|
||||
"field": "oracleVersion",
|
||||
"previous": "paperclip.chat-catalog.external.v1",
|
||||
"current": "paperclip.chat-catalog.company-secret-controller-baseline.v11"
|
||||
"current": "paperclip.chat-catalog.company-secret-controller-baseline.v12"
|
||||
},
|
||||
{
|
||||
"key": "secrets.personal-secret",
|
||||
@@ -1722,6 +1742,24 @@
|
||||
"field": "oracleVersion",
|
||||
"previous": "paperclip.chat-catalog.external.v1",
|
||||
"current": "paperclip.chat-catalog.team-install.v6"
|
||||
},
|
||||
{
|
||||
"key": "secrets.configure-company-secret",
|
||||
"field": "expectedRunCount",
|
||||
"previous": 2,
|
||||
"current": 3
|
||||
},
|
||||
{
|
||||
"key": "secrets.configure-company-secret",
|
||||
"field": "optionalCompanySecretBindingApproval",
|
||||
"previous": null,
|
||||
"current": true
|
||||
},
|
||||
{
|
||||
"key": "secrets.configure-company-secret",
|
||||
"field": "minimumRunCount",
|
||||
"previous": null,
|
||||
"current": 2
|
||||
}
|
||||
],
|
||||
"runtimeConditions": {
|
||||
@@ -1855,7 +1893,7 @@
|
||||
},
|
||||
"candidateLauncherSnapshot": {
|
||||
"path": "fixtures/chat-repair-candidate-launcher.txt",
|
||||
"targetSha": "d1ab241cac0db7f700fadaaf62ffb3d2b51cf0c3",
|
||||
"targetSha": "fe547be84826b4611d3ba9ee1a7d7aa0bd0a2843",
|
||||
"sourcePath": "tests/runner-e2e/launch.ts",
|
||||
"gitBlobOid": "c88df15f509b2ac8638af826647a97d45f16bd1f"
|
||||
}
|
||||
@@ -2398,7 +2436,7 @@
|
||||
"activity/timeline"
|
||||
],
|
||||
"checks": null,
|
||||
"oracleVersion": "paperclip.chat-catalog.work-timeline.v10",
|
||||
"oracleVersion": "paperclip.chat-catalog.work-timeline.v11",
|
||||
"sourceFiles": [
|
||||
"chat-catalog-domains/internal-definition.ts",
|
||||
"chat-catalog-domains/internal-support.ts",
|
||||
@@ -11170,7 +11208,7 @@
|
||||
"fullName": "paperclip.Product E2E.chat-features.secrets.configure-company-secret",
|
||||
"aliases": [],
|
||||
"minimumRuns": 2,
|
||||
"maximumRuns": 2,
|
||||
"maximumRuns": 3,
|
||||
"turnTimeoutMs": 0,
|
||||
"declaredTurnTimeoutMs": 120000,
|
||||
"attemptTimeoutMs": 480000,
|
||||
@@ -11190,9 +11228,10 @@
|
||||
"connection-setup/credentials"
|
||||
],
|
||||
"checks": null,
|
||||
"oracleVersion": "paperclip.chat-catalog.company-secret-controller-baseline.v11",
|
||||
"oracleVersion": "paperclip.chat-catalog.company-secret-controller-baseline.v12",
|
||||
"sourceFiles": [
|
||||
"chat-catalog-domains/external-company-secret.ts",
|
||||
"chat-catalog-domains/external-secret-proposal.ts",
|
||||
"chat-catalog-domains/external-secret-question.ts",
|
||||
"chat-catalog-domains/external-secret-route.ts",
|
||||
"chat-catalog-domains/external-company-secret-menu.ts",
|
||||
@@ -11214,7 +11253,9 @@
|
||||
"screenshot": "final-state.png",
|
||||
"evidenceKind": "private-final-state-png",
|
||||
"evidenceReadiness": "original-private-evidence-validation-required",
|
||||
"boundaryTarget": null
|
||||
"boundaryTarget": null,
|
||||
"optionalCompanySecretBindingApproval": true,
|
||||
"expectedHumanMessageCount": 2
|
||||
},
|
||||
{
|
||||
"key": "secrets.least-scope-delivery",
|
||||
|
||||
@@ -8,6 +8,26 @@ const surfaces = new Set(['agent-chat', 'browser-boundary', 'scoped-external-ass
|
||||
const nonnegative = value => Number.isSafeInteger(value) && value >= 0;
|
||||
const text = value => typeof value === 'string' && value.trim().length > 0;
|
||||
|
||||
export function companySecretBindingApprovalMetadata(declaration) {
|
||||
if (!Object.hasOwn(declaration, 'optionalCompanySecretBindingApproval')) return {};
|
||||
if (declaration.optionalCompanySecretBindingApproval !== true
|
||||
|| declaration.scenarioId !== 'secrets.configure-company-secret'
|
||||
|| declaration.expectedRunCount !== 3 || declaration.minimumRunCount !== 2
|
||||
|| declaration.expectedHumanMessageCount !== 2 || declaration.attemptTimeoutMs !== 480000) {
|
||||
throw new Error('chat_full_catalog_company_approval_invalid');
|
||||
}
|
||||
return { optionalCompanySecretBindingApproval: true, expectedHumanMessageCount: 2 };
|
||||
}
|
||||
|
||||
export function validateCompanySecretBindingApprovalSource(row, task) {
|
||||
const declared = companySecretBindingApprovalMetadata({ ...row, scenarioId: row.key,
|
||||
expectedRunCount: row.maximumRuns, minimumRunCount: row.minimumRuns });
|
||||
const actual = companySecretBindingApprovalMetadata({ ...task, scenarioId: task.id,
|
||||
minimumRunCount: task.minimumExpectedRunCount ?? task.expectedRunCount,
|
||||
attemptTimeoutMs: task.attemptTimeoutMs?.local });
|
||||
if (JSON.stringify(actual) !== JSON.stringify(declared)) throw new Error('Pinned source approval mismatch');
|
||||
}
|
||||
|
||||
export function validateFullCatalogManifest(value) {
|
||||
if (value?.schema !== 'paperclip.chat-full-catalog-testops-manifest/v1'
|
||||
|| !/^[a-f0-9]{64}$/.test(value.launcherSha256 ?? '')
|
||||
@@ -15,6 +35,8 @@ export function validateFullCatalogManifest(value) {
|
||||
|| value.profile !== 'runner-opencode' || value.model !== 'openrouter/deepseek/deepseek-v4.1-flash-1017'
|
||||
|| !Array.isArray(value.cases) || value.cases.length !== 244) throw new Error('chat_full_catalog_manifest_invalid');
|
||||
for (const row of value.cases) {
|
||||
companySecretBindingApprovalMetadata({ ...row, scenarioId: row.key,
|
||||
expectedRunCount: row.maximumRuns, minimumRunCount: row.minimumRuns });
|
||||
const task = legacy.has(row.key) ? row.key.slice(6) : row.key;
|
||||
if (!/^[a-z0-9-]+\.[a-z0-9-]+$/.test(row.key ?? '') || row.canonicalScenario !== row.key || row.scenario !== task
|
||||
|| row.area !== row.key.split('.')[0] || !text(row.name) || !text(row.title)
|
||||
|
||||
@@ -3,7 +3,7 @@ import assert from 'node:assert/strict';
|
||||
import { createHash } from 'node:crypto';
|
||||
import { readFileSync, readdirSync } from 'node:fs';
|
||||
import { createRequire } from 'node:module';
|
||||
import { catalogTargetSha, catalogManifestSha256, fullCatalogPlanName, fullCatalogCases, validateFullCatalogManifest } from './chat-full-catalog-testops-manifest.mjs';
|
||||
import { catalogTargetSha, catalogManifestSha256, fullCatalogPlanName, fullCatalogCases, validateFullCatalogManifest, companySecretBindingApprovalMetadata, validateCompanySecretBindingApprovalSource } from './chat-full-catalog-testops-manifest.mjs';
|
||||
|
||||
const { parseDocument } = createRequire(import.meta.url)('yaml');
|
||||
const bytes = readFileSync(new URL('./chat-full-catalog-testops-manifest.json', import.meta.url));
|
||||
@@ -103,7 +103,7 @@ test('repair source metadata binds the corrected candidate and explicit changed
|
||||
const repair = manifest.source.repairMetadata;
|
||||
assert.equal(repair.baseTargetSha, '3b66c3c144360ddf32714a0742fd774b8ceafb16');
|
||||
assert.equal(repair.previousTargetSha, 'ab2019e976076ec0eb60b026844f3d9211cb0208');
|
||||
assert.equal(manifest.targetSha, 'd1ab241cac0db7f700fadaaf62ffb3d2b51cf0c3');
|
||||
assert.equal(manifest.targetSha, 'fe547be84826b4611d3ba9ee1a7d7aa0bd0a2843');
|
||||
assert.equal(repair.definitionSourceSha, manifest.metadataSourceSha);
|
||||
assert.equal(repair.candidateDescriptorSnapshot, 'fixtures/chat-repair-candidate-descriptors.json');
|
||||
const descriptorBytes = readFileSync(new URL(repair.candidateDescriptorSnapshot, import.meta.url));
|
||||
@@ -319,7 +319,10 @@ test('repair source metadata binds the corrected candidate and explicit changed
|
||||
[
|
||||
"questions-and-approvals.multi-part-question",
|
||||
"originalQuery"
|
||||
]
|
||||
],
|
||||
['secrets.configure-company-secret', 'maximumRuns'],
|
||||
['secrets.configure-company-secret', 'optionalCompanySecretBindingApproval'],
|
||||
['secrets.configure-company-secret', 'expectedHumanMessageCount']
|
||||
]);
|
||||
for (const change of repair.changedFields) {
|
||||
assert.deepEqual(row(change.key)[change.field], change.current);
|
||||
@@ -330,9 +333,9 @@ test('repair source metadata binds the corrected candidate and explicit changed
|
||||
assert.equal(repair.runtimeConditions.multipart53QuestionClarification.unchangedOtherCatalogQueries, 243);
|
||||
assert.equal(repair.runtimeConditions.multipart53QuestionClarification.previousInventoryDigest, '0cf530c11449508e91b5ce664d9375fc8e76466fa908255a1af0ff55169d0220');
|
||||
assert.equal(repair.runtimeConditions.multipart53QuestionClarification.inventoryDigest, '814bb3633268cb0488563aed51fce2f987bd47650ddfb5a3302d0354292c2323');
|
||||
assert.equal(repair.descriptorChanges.length, 49);
|
||||
assert.equal(repair.descriptorChanges.length, 52);
|
||||
assert.equal(new Set(repair.descriptorChanges.map(change => change.key)).size, 33);
|
||||
assert.deepEqual([...new Set(repair.descriptorChanges.map(change => change.field))].sort(), ['expectedRunCount', 'minimumRunCount', 'oracleVersion', 'setupPolicy', 'sourceFiles']);
|
||||
assert.deepEqual([...new Set(repair.descriptorChanges.map(change => change.field))].sort(), ['expectedRunCount', 'minimumRunCount', 'optionalCompanySecretBindingApproval', 'oracleVersion', 'setupPolicy', 'sourceFiles']);
|
||||
for (const change of repair.descriptorChanges) {
|
||||
assert.notDeepEqual(change.current, change.previous);
|
||||
const declaration = declarations.find(value => value.scenarioId === change.key);
|
||||
@@ -350,10 +353,13 @@ test('repair source metadata binds the corrected candidate and explicit changed
|
||||
assert.equal(value.oracleVersion, declaration.oracleVersion);
|
||||
assert.equal(value.minimumRuns, declaration.minimumRunCount ?? declaration.expectedRunCount);
|
||||
assert.equal(value.maximumRuns, declaration.expectedRunCount);
|
||||
const approval = companySecretBindingApprovalMetadata(declaration);
|
||||
assert.deepEqual(Object.fromEntries(Object.keys(approval).map(key => [key, value[key]])), approval);
|
||||
assert.equal(Object.hasOwn(value, 'optionalCompanySecretBindingApproval'), Object.hasOwn(approval, 'optionalCompanySecretBindingApproval'));
|
||||
assert.equal(value.declaredTurnTimeoutMs, declaration.turnTimeoutMs);
|
||||
assert.equal(value.attemptTimeoutMs, declaration.attemptTimeoutMs);
|
||||
}
|
||||
assert.deepEqual(repair.changedFields.filter(change => ['minimumRuns', 'maximumRuns'].includes(change.field)), [{ key: 'skills.studio-test', field: 'maximumRuns', previous: 2, current: 3 }]);
|
||||
assert.deepEqual(repair.changedFields.filter(change => ['minimumRuns', 'maximumRuns'].includes(change.field)), [{ key: 'skills.studio-test', field: 'maximumRuns', previous: 2, current: 3 }, { key: 'secrets.configure-company-secret', field: 'maximumRuns', previous: 2, current: 3 }]);
|
||||
assert.deepEqual(repair.runtimeConditions.studio144Accounting, { key: 'skills.studio-test', previousMinimumRuns: 2, previousMaximumRuns: 2, minimumRuns: 2, maximumRuns: 3, requiredActualAcceptedRuns: 3, originalAttemptTimeoutMs: 540000, humanMessages: 1, condition: 'Initial CEO + real Studio worker + exactly one source-bound completion delivery;2-run no-delivery positive is rejected.' });
|
||||
assert.equal(repair.runtimeConditions.toolGuidance106.corePRUnmerged, false);
|
||||
assert.equal(repair.runtimeConditions.sdkPublicProof.scannerAndCanaryCertificatesUnchanged, true);
|
||||
@@ -375,3 +381,44 @@ test('repair source metadata binds the corrected candidate and explicit changed
|
||||
"actualPostAssertionPGCasesPassed": 2
|
||||
});
|
||||
});
|
||||
|
||||
|
||||
test('Company approval metadata is projected without executing a fixture and rejects foreign or changed bounds', () => {
|
||||
const declaration = { scenarioId: 'secrets.configure-company-secret', optionalCompanySecretBindingApproval: true,
|
||||
expectedRunCount: 3, minimumRunCount: 2, expectedHumanMessageCount: 2, attemptTimeoutMs: 480000,
|
||||
setup() { throw new Error('Fixture must not execute'); } };
|
||||
assert.deepEqual(companySecretBindingApprovalMetadata(declaration),
|
||||
{ optionalCompanySecretBindingApproval: true, expectedHumanMessageCount: 2 });
|
||||
assert.deepEqual(companySecretBindingApprovalMetadata({ scenarioId: 'other' }), {});
|
||||
const future = structuredClone(manifest);
|
||||
Object.assign(future.cases.find(value => value.key === declaration.scenarioId),
|
||||
companySecretBindingApprovalMetadata(declaration), { minimumRuns: 2, maximumRuns: 3 });
|
||||
assert.equal(validateFullCatalogManifest(future), future);
|
||||
for (const mutation of [
|
||||
{ scenarioId: 'secrets.personal-secret' }, { optionalCompanySecretBindingApproval: false },
|
||||
{ expectedRunCount: 2 }, { minimumRunCount: 3 }, { expectedHumanMessageCount: 3 }, { attemptTimeoutMs: 480001 },
|
||||
]) assert.throws(() => companySecretBindingApprovalMetadata({ ...declaration, ...mutation }), /company_approval_invalid/);
|
||||
for (const mutation of [
|
||||
{ key: 'secrets.personal-secret' }, { optionalCompanySecretBindingApproval: false },
|
||||
{ maximumRuns: 2 }, { minimumRuns: 3 }, { expectedHumanMessageCount: 3 }, { attemptTimeoutMs: 480001 },
|
||||
]) {
|
||||
const invalid = structuredClone(future);
|
||||
Object.assign(invalid.cases.find(value => value.key === declaration.scenarioId), mutation);
|
||||
assert.throws(() => validateFullCatalogManifest(invalid), /company_approval_invalid/);
|
||||
}
|
||||
});
|
||||
|
||||
|
||||
test('source admission joins the optional Company approval and base humans to actual static task metadata', () => {
|
||||
const row = { key: 'secrets.configure-company-secret', optionalCompanySecretBindingApproval: true,
|
||||
minimumRuns: 2, maximumRuns: 3, expectedHumanMessageCount: 2, attemptTimeoutMs: 480000 };
|
||||
const task = { id: row.key, optionalCompanySecretBindingApproval: true, minimumExpectedRunCount: 2,
|
||||
expectedRunCount: 3, expectedHumanMessageCount: 2, attemptTimeoutMs: { local: 480000 } };
|
||||
assert.doesNotThrow(() => validateCompanySecretBindingApprovalSource(row, task));
|
||||
for (const mutation of [{ optionalCompanySecretBindingApproval: undefined }, { expectedHumanMessageCount: 3 },
|
||||
{ id: 'secrets.personal-secret' }, { expectedRunCount: 2 }, { minimumExpectedRunCount: 3 }]) {
|
||||
assert.throws(() => validateCompanySecretBindingApprovalSource(row, { ...task, ...mutation }));
|
||||
}
|
||||
const unmarked = { ...row }; delete unmarked.optionalCompanySecretBindingApproval;
|
||||
assert.throws(() => validateCompanySecretBindingApprovalSource(unmarked, task), /approval mismatch/);
|
||||
});
|
||||
@@ -31,6 +31,7 @@ function desiredMetadata(row) {
|
||||
`Full-catalog lane targets candidate ${catalogTargetSha}; other lanes keep their own actual source; target is recorded per result. This registration does not qualify current master or record a PASS.`,
|
||||
`Assumed context:\n${row.originalContext}`,
|
||||
`Required credential names: ${row.requiredCredentials.join(', ') || 'None declared for this case; the controller profile credential is separate.'}`,
|
||||
...(row.optionalCompanySecretBindingApproval ? ['Company secret binding uses two base human inputs plus at most one source-bound, accepted UI approval (maximum three actual human inputs). Native runs: minimum two, maximum three; original 480000ms attempt remains.'] : []),
|
||||
`External input contract: ${row.externalInput == null ? 'None declared.' : JSON.stringify(row.externalInput)}`,
|
||||
`Installed-source requirement: ${row.installedSource ?? 'None declared.'}`,
|
||||
`Source evidence kind: ${row.evidenceKind}`,
|
||||
|
||||
@@ -258,7 +258,7 @@
|
||||
"../../server/src/services/native-runtime/status-decision-committer.ts",
|
||||
"../../server/src/services/work-timeline.ts"
|
||||
],
|
||||
"oracleVersion": "paperclip.chat-catalog.work-timeline.v10",
|
||||
"oracleVersion": "paperclip.chat-catalog.work-timeline.v11",
|
||||
"coverageGaps": [],
|
||||
"expectedRunCount": 3,
|
||||
"minimumRunCount": 3,
|
||||
@@ -4891,6 +4891,7 @@
|
||||
"scenarioId": "secrets.configure-company-secret",
|
||||
"sourceFiles": [
|
||||
"chat-catalog-domains/external-company-secret.ts",
|
||||
"chat-catalog-domains/external-secret-proposal.ts",
|
||||
"chat-catalog-domains/external-secret-question.ts",
|
||||
"chat-catalog-domains/external-secret-route.ts",
|
||||
"chat-catalog-domains/external-company-secret-menu.ts",
|
||||
@@ -4909,9 +4910,11 @@
|
||||
"chat-flow.ts",
|
||||
"chat-scenarios/secrets.yaml"
|
||||
],
|
||||
"oracleVersion": "paperclip.chat-catalog.company-secret-controller-baseline.v11",
|
||||
"expectedRunCount": 2,
|
||||
"oracleVersion": "paperclip.chat-catalog.company-secret-controller-baseline.v12",
|
||||
"expectedRunCount": 3,
|
||||
"minimumRunCount": 2,
|
||||
"expectedHumanMessageCount": 2,
|
||||
"optionalCompanySecretBindingApproval": true,
|
||||
"turnTimeoutMs": 120000,
|
||||
"attemptTimeoutMs": 480000,
|
||||
"authentication": "human-session",
|
||||
|
||||
Reference in new issue
Block a user