mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-04 11:01:16 +01:00
test(evals): pin synchronized chat repair declarations and definitions (#18721)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - The chat catalog runs a fixed candidate through the trusted TestOps
lane.
> - The first repair source exposed fixture, observation and
evidence-readiness defects.
> - Changed declarations also need matching static runtime descriptors.
> - This pull request pins one corrected candidate and records each
changed definition.
> - New measurements keep their source and original outcomes separate
from past results.
## Linked Issues or Issue Description
Refs: #18710, #18693, #18680, #18647.
**What happened?**
The first repair candidate left five static descriptors out of sync with
its actual declarations. The real no-callback admission check rejects
them. A five-case attempt was cancelled before any case or model step.
The earlier pin note overstated remote coverage: importing the catalog
did not load every actual factory.
**Expected behavior**
Every repaired declaration must pass the real factory admission check
before a new batch. The trusted pin must record exact source and
definition changes. Existing case identities, prior measurements and the
full-catalog pass gate must remain intact.
**Steps to reproduce**
Run `node --import ./cli/node_modules/tsx/dist/loader.mjs
tests/runner-e2e/chat-catalog-consistency.ts` on the old candidate. It
rejects five declarations. The corrected candidate admits all 242
declarations without invoking a fixture. The existing legacy-cell
scoring and selection controls also pass.
## What Changed
- Pin candidate `36dff2eab3dfb6de6efb4fd2ce6c1a58f8a5f383`. Its
[candidate-only
delta](https://github.com/paperclipai/paperclip/compare/9ab0f951a5b5990c266ef4456610e0a0775aa334...36dff2eab3dfb6de6efb4fd2ce6c1a58f8a5f383)
remains separate from master application code.
- Record the real descriptor digest and all 19 declaration-field changes
across 18 cases. Check them against the exact candidate static
descriptor snapshot, including setup policies. The runtime loader still
rejects drift.
- Keep original definition values and record 19 manifest-field changes
across 15 cases. These include the clarified Fast response request,
reviewed goal/skill navigation outcomes, and scoped oracle/source
ownership changes.
- Keep unchanged all 244 ordered identities, model, case clocks,
native-run bounds, private evidence rules, workflows and reporting
authority. The original ledger and historical results are unchanged.
- Require actual factory admission and legacy controls before each
repair-batch GO. SDK audit capture remains private diagnosis only; it
does not waive scanner rejection or qualify the other SDK cases.
## Verification
- Candidate focused controls: 561 passed across 32 pure test files.
- Real lazy factory admission: 242 declarations passed, with zero
fixture callbacks. The prior source failed 18 declarations before
descriptor sync. The two-file admission and legacy scoring suite passed
all 42 controls.
- Trusted manifest, reconciliation and adapter controls: 38 passed.
Contract and dispatch controls: 23 passed.
- Independent source reviews cover the repair slices, descriptor sync
and this metadata pin.
- `git diff --check` passed.
- No local app, DB, native or model run was started for this metadata
PR. Full harness typecheck/build is not qualified: existing workspace
dependency errors remain outside these pure checks. The established
credential-free remote Runner build is still required before paid
queries. It does not typecheck the full harness graph.
- Initial head 90d4ecf passed first-attempt CI (48 passed, 1 skipped)
and Greptile 5/5. A later review found stale current-target prose and an
unchecked setup-policy provenance assertion. The corrected head adds the
exact candidate descriptor snapshot, computes its digest, and compares
every declared field. A wrong-policy mutant accepted by the parent is
now rejected, as is a stale digest. Correct metadata controls pass all
38 cases. Latest-head CI41572232196 passed48 jobs with1 skip after one
approved failed-only recovery. The first attempt failed one browser
cleanup cancellation with PostgreSQL row-lock timeout; the lock holder
remains unknown. No server or timeout change was made. All46 successful
executions kept their original full step and execution timestamps. Only
the failed browser shard and dependent e2e summary reran. Both required
checks passed, current-head Greptile is5/5 and all review threads are
resolved. An optional review may remain queued; no claim is made that
all optional checks finished. No corrected-candidate feature pass is
claimed.
## Risks
The candidate changes selected setup, observation and oracle conditions.
Each new result must retain its exact source and declared changes; it
cannot replace an old failed result. The Fast response request and
goal/skill outcome changes are explicit new definitions. Static TestOps
prose may still show the original definition until a separate supported
metadata reconciliation. SDK scanner findings remain unknown until the
private audit is captured; the scanner and evidence bounds remain
enforced.
## Model Used
OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) with reasoning, source editing and
tool use. The exact context window is not exposed by this session.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either linked existing issues or described the issue in-PR
following the issue template
- [x] I have not referenced internal or instance-local Paperclip issues
or links
- [x] My branch name describes the change and contains no internal
Paperclip ticket id
- [x] I have run focused tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All required Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
This commit is contained in:
Dotta authored and GitHub committed
1 parent
1e3ab690b0
commit
fea6163811
4 files changed
+6792
-31
No files matched your search
@@ -217,8 +217,8 @@ overlap. Exact requirements and limitations appear in each case's preconditions.
|
||||
A declared credential does not establish whether its runtime reference exists.
|
||||
|
||||
This lane runs the app and fixtures from candidate
|
||||
`9ab0f951a5b5990c266ef4456610e0a0775aa334`. It retains the evaluated app from
|
||||
`3b66c3c144360ddf32714a0742fd774b8ceafb16` with four bounded fixture repairs:
|
||||
`36dff2eab3dfb6de6efb4fd2ce6c1a58f8a5f383`. It retains the evaluated app from
|
||||
`3b66c3c144360ddf32714a0742fd774b8ceafb16`. The first repair source added:
|
||||
owned SDK staging, public membership for three incidental human fixtures,
|
||||
company settings observation inside its original case deadline, and exact
|
||||
controller-secret baseline handling for secure human entry. The last repair
|
||||
@@ -227,7 +227,8 @@ secret, Morgan-only binding and privacy checks remain required.
|
||||
|
||||
The manifest retains the original `de0941835cc8f0d01c982574ee258e110babdbeb`
|
||||
definition provenance and records each changed row field in `source.repairMetadata`.
|
||||
Queries, case limits, models and the other feature predicates remain unchanged.
|
||||
Case limits and models remain unchanged. The later repair source records its
|
||||
query and outcome changes explicitly; see Repair source 2 below.
|
||||
Original failed measurements stay historical. Trusted selection, evidence transfer
|
||||
and publishing use the workflow's immutable master revision. Each result records
|
||||
both revisions.
|
||||
@@ -294,3 +295,11 @@ and credentials do not enter the public artifact path. Additional credentials
|
||||
reach only cases that declare them. External inputs use the existing selected-case
|
||||
private-input owner, disposable-resource checks and oracle commitments.
|
||||
Unconfigured inputs remain setup blockers.
|
||||
|
||||
### Repair source 2 (candidate only)
|
||||
|
||||
The catalog now pins candidate `36dff2eab3dfb6de6efb4fd2ce6c1a58f8a5f383`, succeeding `9ab0f951a5b5990c266ef4456610e0a0775aa334`. It carries the reviewed fixture, scoped observation, version-readback and evidence-readiness repairs. The checked static descriptor snapshot is byte-exact to this candidate and its digest is verified. Runtime descriptors are synchronized with the actual declarations; all 242 lazy factories must pass the real no-callback consistency check, together with the two legacy-cell controls, before each new repair-batch GO. The previous candidate's mismatched metadata caused launch 1690 to be cancelled before case/model work; it is an operational attempt, not a feature result.
|
||||
|
||||
The manifest keeps the original definition source and values, with explicit scoped changes: Fast response names the requested product feature; goal-tree and authored-skill outcomes use the product-supported optional-owner and ordinary navigation contracts; question delivery/validation and company-secret setup declare their reviewed oracle versions. Each new measurement binds this candidate and the exact changed definition; earlier grades and the 244-case gate are unchanged. SDK scan-audit capture improves private failure diagnosis only: rejection, evidence requirements and privacy bounds remain in force, and the other SDK cases stay held until the finding is understood.
|
||||
|
||||
Workflow, model, case clocks, native-run bounds, credentials, protected reporting and case identities do not change. The full harness TypeScript graph has not been qualified by local pure tests; source import/admission and the existing credential-free remote Runner build remain required before native queries. Static TestOps descriptions may still show the original definition until a separate idempotent metadata reconciliation at a safe source boundary.
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"schema": "paperclip.chat-full-catalog-testops-manifest/v1",
|
||||
"targetSha": "9ab0f951a5b5990c266ef4456610e0a0775aa334",
|
||||
"targetSha": "36dff2eab3dfb6de6efb4fd2ce6c1a58f8a5f383",
|
||||
"metadataSourceSha": "de0941835cc8f0d01c982574ee258e110babdbeb",
|
||||
"profile": "runner-opencode",
|
||||
"model": "openrouter/deepseek/deepseek-v4.1-flash-1017",
|
||||
@@ -170,8 +170,304 @@
|
||||
],
|
||||
"repairMetadata": {
|
||||
"baseTargetSha": "3b66c3c144360ddf32714a0742fd774b8ceafb16",
|
||||
"previousTargetSha": "9ab0f951a5b5990c266ef4456610e0a0775aa334",
|
||||
"definitionSourceSha": "de0941835cc8f0d01c982574ee258e110babdbeb",
|
||||
"candidateDescriptorSha256": "5b098a30ed54c6021698b33b434145ca2e934ff2bb9c1e0738a8d8e5dc622797",
|
||||
"changedDefinitionFiles": [
|
||||
{
|
||||
"path": "tests/runner-e2e/chat-scenarios/companies.yaml",
|
||||
"sha256": "fbaf17ac279d92127de4b58a3605edc5c7954dabfb050f49be210e1f85775d1d"
|
||||
},
|
||||
{
|
||||
"path": "tests/runner-e2e/chat-scenarios/goals.yaml",
|
||||
"sha256": "b140b0e652a9945b7ead9e38e5bc01791f7dc9cd39b1f2b6f292544d50632dae"
|
||||
},
|
||||
{
|
||||
"path": "tests/runner-e2e/chat-scenarios/skills.yaml",
|
||||
"sha256": "51f975d62540e77a48539821f443c6ab134c8ffb63e4c04bb2678a655de84a69"
|
||||
}
|
||||
],
|
||||
"conditions": "New candidate measurements only. Original 244 definitions, attempt history and old pass gate remain immutable. SDK audit custody remains diagnostic; scan rejection is unchanged. Explicit field overrides below retain original definition values. No model, clock, run-bound, workflow, credential or reporting authority expansion.",
|
||||
"changedFields": [
|
||||
{
|
||||
"key": "agents.choose-primary",
|
||||
"field": "sourceFiles",
|
||||
"previous": [
|
||||
"chat-catalog-domains/runtime-primary.ts",
|
||||
"chat-catalog-domains/runtime-support.ts",
|
||||
"chat-scenarios/agents.yaml"
|
||||
],
|
||||
"current": [
|
||||
"chat-catalog-domains/runtime-primary.ts",
|
||||
"chat-catalog-domains/runtime-support.ts",
|
||||
"chat-catalog-domains/identity-support.ts",
|
||||
"chat-scenarios/agents.yaml"
|
||||
]
|
||||
},
|
||||
{
|
||||
"key": "companies.agent-observability",
|
||||
"field": "sourceFiles",
|
||||
"previous": [
|
||||
"chat-catalog-domains/identity-observability.ts",
|
||||
"chat-catalog-domains/identity-private-browser.ts",
|
||||
"chat-catalog-domains/identity-private-browser-worker.ts",
|
||||
"chat-catalog-domains/identity-wizard.ts",
|
||||
"chat-catalog-domains/identity-support.ts",
|
||||
"chat-catalog-domains/internal-support.ts",
|
||||
"chat-scenarios/companies.yaml"
|
||||
],
|
||||
"current": [
|
||||
"chat-catalog-domains/identity-observability.ts",
|
||||
"chat-catalog-domains/identity-company-observer.ts",
|
||||
"chat-catalog-domains/identity-private-browser.ts",
|
||||
"chat-catalog-domains/identity-private-browser-worker.ts",
|
||||
"chat-catalog-domains/identity-wizard.ts",
|
||||
"chat-catalog-domains/identity-support.ts",
|
||||
"chat-catalog-domains/internal-support.ts",
|
||||
"chat-scenarios/companies.yaml"
|
||||
]
|
||||
},
|
||||
{
|
||||
"key": "companies.configure-decision-model",
|
||||
"field": "sourceFiles",
|
||||
"previous": [
|
||||
"chat-catalog-domains/identity-company-models.ts",
|
||||
"chat-catalog-domains/identity-support.ts",
|
||||
"chat-catalog-domains/internal-support.ts",
|
||||
"chat-scenarios/companies.yaml"
|
||||
],
|
||||
"current": [
|
||||
"chat-catalog-domains/identity-company-models.ts",
|
||||
"chat-catalog-domains/identity-company-observer.ts",
|
||||
"chat-catalog-domains/identity-support.ts",
|
||||
"chat-catalog-domains/internal-support.ts",
|
||||
"chat-scenarios/companies.yaml"
|
||||
]
|
||||
},
|
||||
{
|
||||
"key": "companies.fast-responses",
|
||||
"field": "sourceFiles",
|
||||
"previous": [
|
||||
"chat-catalog-domains/identity-company-models.ts",
|
||||
"chat-catalog-domains/identity-support.ts",
|
||||
"chat-catalog-domains/internal-support.ts",
|
||||
"chat-scenarios/companies.yaml"
|
||||
],
|
||||
"current": [
|
||||
"chat-catalog-domains/identity-company-models.ts",
|
||||
"chat-catalog-domains/identity-company-observer.ts",
|
||||
"chat-catalog-domains/identity-support.ts",
|
||||
"chat-catalog-domains/internal-support.ts",
|
||||
"chat-scenarios/companies.yaml"
|
||||
]
|
||||
},
|
||||
{
|
||||
"key": "companies.fast-responses",
|
||||
"field": "originalQuery",
|
||||
"previous": "Make chat acknowledge my requests faster.",
|
||||
"current": "Help me enable the company's Fast response feature so chat acknowledges my requests faster."
|
||||
},
|
||||
{
|
||||
"key": "companies.prompt-injection-guard",
|
||||
"field": "sourceFiles",
|
||||
"previous": [
|
||||
"chat-catalog-domains/identity-company-models.ts",
|
||||
"chat-catalog-domains/identity-support.ts",
|
||||
"chat-catalog-domains/internal-support.ts",
|
||||
"chat-scenarios/companies.yaml"
|
||||
],
|
||||
"current": [
|
||||
"chat-catalog-domains/identity-company-models.ts",
|
||||
"chat-catalog-domains/identity-company-observer.ts",
|
||||
"chat-catalog-domains/identity-support.ts",
|
||||
"chat-catalog-domains/internal-support.ts",
|
||||
"chat-scenarios/companies.yaml"
|
||||
]
|
||||
},
|
||||
{
|
||||
"key": "goals.create-goal-tree",
|
||||
"field": "oracleVersion",
|
||||
"previous": "paperclip.chat-catalog.internal.v1",
|
||||
"current": "paperclip.chat-catalog.goal-hierarchy.v2"
|
||||
},
|
||||
{
|
||||
"key": "goals.create-goal-tree",
|
||||
"field": "expectedOutcome",
|
||||
"previous": "The saved goals have the requested levels, valid parent relationship and appropriate ownership or an explicit missing-owner question. Chat links to the hierarchy and does not invent progress metrics. A retry reuses the intended goals rather than creating another parallel tree.",
|
||||
"current": "The saved goals have the requested levels and a valid same-company parent relationship. Ownership may remain unassigned when none was requested; any assigned owner belongs to the company. The saved hierarchy is inspectable through ordinary Goals navigation and public readback, without invented progress metrics. A retry reuses the intended goals rather than creating another parallel tree."
|
||||
},
|
||||
{
|
||||
"key": "questions-and-approvals.duplicate-answer",
|
||||
"field": "oracleVersion",
|
||||
"previous": "paperclip.chat-catalog.identity.v1",
|
||||
"current": "paperclip.chat-catalog.answer-delivery.v2"
|
||||
},
|
||||
{
|
||||
"key": "questions-and-approvals.multi-part-question",
|
||||
"field": "oracleVersion",
|
||||
"previous": "paperclip.chat-catalog.identity.v1",
|
||||
"current": "paperclip.chat-catalog.required-question.v2"
|
||||
},
|
||||
{
|
||||
"key": "secrets.configure-company-secret",
|
||||
"field": "oracleVersion",
|
||||
"previous": "paperclip.chat-catalog.external.v1",
|
||||
"current": "paperclip.chat-catalog.company-secret-controller-baseline.v2"
|
||||
},
|
||||
{
|
||||
"key": "skills.author-skill",
|
||||
"field": "sourceFiles",
|
||||
"previous": [
|
||||
"chat-catalog-domains/integrator-support.ts",
|
||||
"chat-catalog-domains/internal-support.ts",
|
||||
"chat-catalog-domains/integrator-skills.ts",
|
||||
"chat-scenarios/skills.yaml"
|
||||
],
|
||||
"current": [
|
||||
"chat-catalog-domains/integrator-support.ts",
|
||||
"chat-catalog-domains/internal-support.ts",
|
||||
"chat-catalog-domains/integrator-skills.ts",
|
||||
"chat-scenarios/skills.yaml",
|
||||
"chat-catalog-domains/integrator-skill-versions.ts",
|
||||
"chat-catalog-domains/identity-support.ts"
|
||||
]
|
||||
},
|
||||
{
|
||||
"key": "skills.author-skill",
|
||||
"field": "oracleVersion",
|
||||
"previous": "paperclip.chat-catalog.integrator.v1",
|
||||
"current": "paperclip.chat-catalog.skill-authoring.v2"
|
||||
},
|
||||
{
|
||||
"key": "skills.author-skill",
|
||||
"field": "expectedOutcome",
|
||||
"previous": "A writable company skill captures the checklist with clear instructions and a saved revision. Chat links to the skill so the person can inspect or edit it. The agent preserves the actual checklist instead of merely posting a suggestion or pretending creation automatically enables the skill for every agent.",
|
||||
"current": "A writable company skill captures the checklist with clear instructions and a saved revision. The person can discover it through ordinary Skills/My Skills navigation, open its editable Studio content and reload the same saved revision. No particular chat-link format is required. The agent preserves the actual checklist instead of merely posting a suggestion or pretending creation automatically enables the skill for every agent."
|
||||
},
|
||||
{
|
||||
"key": "skills.enable-for-agent",
|
||||
"field": "sourceFiles",
|
||||
"previous": [
|
||||
"chat-catalog-domains/integrator-skill-enable.ts",
|
||||
"chat-catalog-domains/identity-support.ts",
|
||||
"chat-catalog-domains/internal-definition.ts",
|
||||
"chat-catalog-domains/internal-support.ts",
|
||||
"chat-catalog-domains/integrator-support.ts",
|
||||
"chat-scenarios/skills.yaml",
|
||||
"../../server/src/routes/agents.ts",
|
||||
"../../server/src/routes/company-skill-policy.ts",
|
||||
"../../server/src/services/company-skills.ts",
|
||||
"../../packages/shared/src/validators/skill-policy.ts"
|
||||
],
|
||||
"current": [
|
||||
"chat-catalog-domains/integrator-skill-versions.ts",
|
||||
"chat-catalog-domains/integrator-skill-enable.ts",
|
||||
"chat-catalog-domains/identity-support.ts",
|
||||
"chat-catalog-domains/internal-definition.ts",
|
||||
"chat-catalog-domains/internal-support.ts",
|
||||
"chat-catalog-domains/integrator-support.ts",
|
||||
"chat-scenarios/skills.yaml",
|
||||
"../../server/src/routes/agents.ts",
|
||||
"../../server/src/routes/company-skill-policy.ts",
|
||||
"../../server/src/services/company-skills.ts",
|
||||
"../../packages/shared/src/validators/skill-policy.ts"
|
||||
]
|
||||
},
|
||||
{
|
||||
"key": "skills.explicit-restriction",
|
||||
"field": "sourceFiles",
|
||||
"previous": [
|
||||
"chat-catalog-domains/integrator-support.ts",
|
||||
"chat-catalog-domains/internal-support.ts",
|
||||
"chat-catalog-domains/integrator-skills.ts",
|
||||
"chat-scenarios/skills.yaml"
|
||||
],
|
||||
"current": [
|
||||
"chat-catalog-domains/integrator-support.ts",
|
||||
"chat-catalog-domains/internal-support.ts",
|
||||
"chat-catalog-domains/integrator-skills.ts",
|
||||
"chat-scenarios/skills.yaml",
|
||||
"chat-catalog-domains/integrator-skill-versions.ts"
|
||||
]
|
||||
},
|
||||
{
|
||||
"key": "skills.find-and-install",
|
||||
"field": "sourceFiles",
|
||||
"previous": [
|
||||
"chat-catalog-domains/integrator-skill-imports.ts",
|
||||
"chat-catalog-domains/integrator-skill-source.ts",
|
||||
"chat-catalog-domains/integrator-skills.ts",
|
||||
"chat-catalog-domains/integrator-support.ts",
|
||||
"chat-catalog-domains/internal-support.ts",
|
||||
"chat-scenarios/skills.yaml"
|
||||
],
|
||||
"current": [
|
||||
"chat-catalog-domains/integrator-skill-versions.ts",
|
||||
"chat-catalog-domains/integrator-skill-imports.ts",
|
||||
"chat-catalog-domains/integrator-skill-source.ts",
|
||||
"chat-catalog-domains/integrator-skills.ts",
|
||||
"chat-catalog-domains/integrator-support.ts",
|
||||
"chat-catalog-domains/internal-support.ts",
|
||||
"chat-scenarios/skills.yaml"
|
||||
]
|
||||
},
|
||||
{
|
||||
"key": "skills.fork-read-only",
|
||||
"field": "sourceFiles",
|
||||
"previous": [
|
||||
"chat-catalog-domains/integrator-skill-imports.ts",
|
||||
"chat-catalog-domains/integrator-skill-source.ts",
|
||||
"chat-catalog-domains/integrator-skills.ts",
|
||||
"chat-catalog-domains/integrator-support.ts",
|
||||
"chat-catalog-domains/internal-support.ts",
|
||||
"chat-scenarios/skills.yaml"
|
||||
],
|
||||
"current": [
|
||||
"chat-catalog-domains/integrator-skill-versions.ts",
|
||||
"chat-catalog-domains/integrator-skill-imports.ts",
|
||||
"chat-catalog-domains/integrator-skill-source.ts",
|
||||
"chat-catalog-domains/integrator-skills.ts",
|
||||
"chat-catalog-domains/integrator-support.ts",
|
||||
"chat-catalog-domains/internal-support.ts",
|
||||
"chat-scenarios/skills.yaml"
|
||||
]
|
||||
},
|
||||
{
|
||||
"key": "skills.review-update",
|
||||
"field": "sourceFiles",
|
||||
"previous": [
|
||||
"chat-catalog-domains/integrator-skill-update.ts",
|
||||
"chat-catalog-domains/integrator-skill-source.ts",
|
||||
"chat-catalog-domains/integrator-support.ts",
|
||||
"chat-catalog-domains/internal-support.ts",
|
||||
"chat-scenarios/skills.yaml"
|
||||
],
|
||||
"current": [
|
||||
"chat-catalog-domains/integrator-skill-versions.ts",
|
||||
"chat-catalog-domains/integrator-skill-update.ts",
|
||||
"chat-catalog-domains/integrator-skill-source.ts",
|
||||
"chat-catalog-domains/integrator-support.ts",
|
||||
"chat-catalog-domains/internal-support.ts",
|
||||
"chat-scenarios/skills.yaml"
|
||||
]
|
||||
}
|
||||
],
|
||||
"descriptorChanges": [
|
||||
{
|
||||
"key": "agents.choose-primary",
|
||||
"field": "sourceFiles",
|
||||
"previous": [
|
||||
"chat-catalog-domains/runtime-primary.ts",
|
||||
"chat-catalog-domains/runtime-support.ts",
|
||||
"chat-scenarios/agents.yaml"
|
||||
],
|
||||
"current": [
|
||||
"chat-catalog-domains/runtime-primary.ts",
|
||||
"chat-catalog-domains/runtime-support.ts",
|
||||
"chat-catalog-domains/identity-support.ts",
|
||||
"chat-scenarios/agents.yaml"
|
||||
]
|
||||
},
|
||||
{
|
||||
"key": "companies.agent-observability",
|
||||
"field": "sourceFiles",
|
||||
@@ -246,20 +542,218 @@
|
||||
"chat-scenarios/companies.yaml"
|
||||
]
|
||||
},
|
||||
{
|
||||
"key": "goals.create-goal-tree",
|
||||
"field": "oracleVersion",
|
||||
"previous": "paperclip.chat-catalog.internal.v1",
|
||||
"current": "paperclip.chat-catalog.goal-hierarchy.v2"
|
||||
},
|
||||
{
|
||||
"key": "questions-and-approvals.duplicate-answer",
|
||||
"field": "oracleVersion",
|
||||
"previous": "paperclip.chat-catalog.identity.v1",
|
||||
"current": "paperclip.chat-catalog.answer-delivery.v2"
|
||||
},
|
||||
{
|
||||
"key": "questions-and-approvals.multi-part-question",
|
||||
"field": "oracleVersion",
|
||||
"previous": "paperclip.chat-catalog.identity.v1",
|
||||
"current": "paperclip.chat-catalog.required-question.v2"
|
||||
},
|
||||
{
|
||||
"key": "secrets.configure-company-secret",
|
||||
"field": "oracleVersion",
|
||||
"previous": "paperclip.chat-catalog.external.v1",
|
||||
"current": "paperclip.chat-catalog.company-secret-controller-baseline.v2"
|
||||
},
|
||||
{
|
||||
"key": "secrets.personal-secret",
|
||||
"field": "setupPolicy",
|
||||
"previous": null,
|
||||
"current": {
|
||||
"readiness": "preserve-scenario",
|
||||
"invariantCheckIds": [
|
||||
"current-owner-missing-required-personal-value"
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "skills.author-skill",
|
||||
"field": "sourceFiles",
|
||||
"previous": [
|
||||
"chat-catalog-domains/integrator-support.ts",
|
||||
"chat-catalog-domains/internal-support.ts",
|
||||
"chat-catalog-domains/integrator-skills.ts",
|
||||
"chat-scenarios/skills.yaml"
|
||||
],
|
||||
"current": [
|
||||
"chat-catalog-domains/integrator-support.ts",
|
||||
"chat-catalog-domains/internal-support.ts",
|
||||
"chat-catalog-domains/integrator-skills.ts",
|
||||
"chat-scenarios/skills.yaml",
|
||||
"chat-catalog-domains/integrator-skill-versions.ts",
|
||||
"chat-catalog-domains/identity-support.ts"
|
||||
]
|
||||
},
|
||||
{
|
||||
"key": "skills.author-skill",
|
||||
"field": "oracleVersion",
|
||||
"previous": "paperclip.chat-catalog.integrator.v1",
|
||||
"current": "paperclip.chat-catalog.skill-authoring.v2"
|
||||
},
|
||||
{
|
||||
"key": "skills.enable-for-agent",
|
||||
"field": "sourceFiles",
|
||||
"previous": [
|
||||
"chat-catalog-domains/integrator-skill-enable.ts",
|
||||
"chat-catalog-domains/identity-support.ts",
|
||||
"chat-catalog-domains/internal-definition.ts",
|
||||
"chat-catalog-domains/internal-support.ts",
|
||||
"chat-catalog-domains/integrator-support.ts",
|
||||
"chat-scenarios/skills.yaml",
|
||||
"../../server/src/routes/agents.ts",
|
||||
"../../server/src/routes/company-skill-policy.ts",
|
||||
"../../server/src/services/company-skills.ts",
|
||||
"../../packages/shared/src/validators/skill-policy.ts"
|
||||
],
|
||||
"current": [
|
||||
"chat-catalog-domains/integrator-skill-versions.ts",
|
||||
"chat-catalog-domains/integrator-skill-enable.ts",
|
||||
"chat-catalog-domains/identity-support.ts",
|
||||
"chat-catalog-domains/internal-definition.ts",
|
||||
"chat-catalog-domains/internal-support.ts",
|
||||
"chat-catalog-domains/integrator-support.ts",
|
||||
"chat-scenarios/skills.yaml",
|
||||
"../../server/src/routes/agents.ts",
|
||||
"../../server/src/routes/company-skill-policy.ts",
|
||||
"../../server/src/services/company-skills.ts",
|
||||
"../../packages/shared/src/validators/skill-policy.ts"
|
||||
]
|
||||
},
|
||||
{
|
||||
"key": "skills.explicit-restriction",
|
||||
"field": "sourceFiles",
|
||||
"previous": [
|
||||
"chat-catalog-domains/integrator-support.ts",
|
||||
"chat-catalog-domains/internal-support.ts",
|
||||
"chat-catalog-domains/integrator-skills.ts",
|
||||
"chat-scenarios/skills.yaml"
|
||||
],
|
||||
"current": [
|
||||
"chat-catalog-domains/integrator-support.ts",
|
||||
"chat-catalog-domains/internal-support.ts",
|
||||
"chat-catalog-domains/integrator-skills.ts",
|
||||
"chat-scenarios/skills.yaml",
|
||||
"chat-catalog-domains/integrator-skill-versions.ts"
|
||||
]
|
||||
},
|
||||
{
|
||||
"key": "skills.find-and-install",
|
||||
"field": "sourceFiles",
|
||||
"previous": [
|
||||
"chat-catalog-domains/integrator-skill-imports.ts",
|
||||
"chat-catalog-domains/integrator-skill-source.ts",
|
||||
"chat-catalog-domains/integrator-skills.ts",
|
||||
"chat-catalog-domains/integrator-support.ts",
|
||||
"chat-catalog-domains/internal-support.ts",
|
||||
"chat-scenarios/skills.yaml"
|
||||
],
|
||||
"current": [
|
||||
"chat-catalog-domains/integrator-skill-versions.ts",
|
||||
"chat-catalog-domains/integrator-skill-imports.ts",
|
||||
"chat-catalog-domains/integrator-skill-source.ts",
|
||||
"chat-catalog-domains/integrator-skills.ts",
|
||||
"chat-catalog-domains/integrator-support.ts",
|
||||
"chat-catalog-domains/internal-support.ts",
|
||||
"chat-scenarios/skills.yaml"
|
||||
]
|
||||
},
|
||||
{
|
||||
"key": "skills.fork-read-only",
|
||||
"field": "sourceFiles",
|
||||
"previous": [
|
||||
"chat-catalog-domains/integrator-skill-imports.ts",
|
||||
"chat-catalog-domains/integrator-skill-source.ts",
|
||||
"chat-catalog-domains/integrator-skills.ts",
|
||||
"chat-catalog-domains/integrator-support.ts",
|
||||
"chat-catalog-domains/internal-support.ts",
|
||||
"chat-scenarios/skills.yaml"
|
||||
],
|
||||
"current": [
|
||||
"chat-catalog-domains/integrator-skill-versions.ts",
|
||||
"chat-catalog-domains/integrator-skill-imports.ts",
|
||||
"chat-catalog-domains/integrator-skill-source.ts",
|
||||
"chat-catalog-domains/integrator-skills.ts",
|
||||
"chat-catalog-domains/integrator-support.ts",
|
||||
"chat-catalog-domains/internal-support.ts",
|
||||
"chat-scenarios/skills.yaml"
|
||||
]
|
||||
},
|
||||
{
|
||||
"key": "skills.review-update",
|
||||
"field": "sourceFiles",
|
||||
"previous": [
|
||||
"chat-catalog-domains/integrator-skill-update.ts",
|
||||
"chat-catalog-domains/integrator-skill-source.ts",
|
||||
"chat-catalog-domains/integrator-support.ts",
|
||||
"chat-catalog-domains/internal-support.ts",
|
||||
"chat-scenarios/skills.yaml"
|
||||
],
|
||||
"current": [
|
||||
"chat-catalog-domains/integrator-skill-versions.ts",
|
||||
"chat-catalog-domains/integrator-skill-update.ts",
|
||||
"chat-catalog-domains/integrator-skill-source.ts",
|
||||
"chat-catalog-domains/integrator-support.ts",
|
||||
"chat-catalog-domains/internal-support.ts",
|
||||
"chat-scenarios/skills.yaml"
|
||||
]
|
||||
},
|
||||
{
|
||||
"key": "steering.stop-versus-pause",
|
||||
"field": "setupPolicy",
|
||||
"previous": {
|
||||
"readiness": "preserve-scenario",
|
||||
"invariantCheckIds": [
|
||||
"lead-ready",
|
||||
"two-real-native-workers",
|
||||
"independent-child-pause-budget"
|
||||
]
|
||||
},
|
||||
"current": {
|
||||
"readiness": "preserve-scenario",
|
||||
"funding": "preserve-scenario",
|
||||
"invariantCheckIds": [
|
||||
"lead-ready",
|
||||
"two-real-native-workers",
|
||||
"independent-child-pause-budget",
|
||||
"active-branches-funded"
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "task-coordination.pause-tree",
|
||||
"field": "setupPolicy",
|
||||
"previous": {
|
||||
"readiness": "preserve-scenario",
|
||||
"invariantCheckIds": [
|
||||
"lead-ready",
|
||||
"two-real-native-workers",
|
||||
"independent-child-pause-budget"
|
||||
]
|
||||
},
|
||||
"current": {
|
||||
"readiness": "preserve-scenario",
|
||||
"funding": "preserve-scenario",
|
||||
"invariantCheckIds": [
|
||||
"lead-ready",
|
||||
"two-real-native-workers",
|
||||
"independent-child-pause-budget",
|
||||
"active-branches-funded"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"conditions": [
|
||||
"Initialize owned SDK evidence staging before retention.",
|
||||
"Use real public scoped membership only for three incidental human fixtures.",
|
||||
"Company settings observation uses the original setup-inclusive deadline; no larger case cap.",
|
||||
"Company secret setup and v2 oracle permit only exact immutable controller-bootstrap secret references."
|
||||
],
|
||||
"originalMeasurements": "Preserved. These are new candidate conditions, not retroactive passes."
|
||||
"candidateDescriptorSnapshot": "fixtures/chat-repair-candidate-descriptors.json"
|
||||
}
|
||||
},
|
||||
"cases": [
|
||||
@@ -1262,6 +1756,7 @@
|
||||
"sourceFiles": [
|
||||
"chat-catalog-domains/runtime-primary.ts",
|
||||
"chat-catalog-domains/runtime-support.ts",
|
||||
"chat-catalog-domains/identity-support.ts",
|
||||
"chat-scenarios/agents.yaml"
|
||||
],
|
||||
"screenshot": "final-state.png",
|
||||
@@ -4088,7 +4583,7 @@
|
||||
],
|
||||
"externalInput": null,
|
||||
"installedSource": null,
|
||||
"originalQuery": "Make chat acknowledge my requests faster.",
|
||||
"originalQuery": "Help me enable the company's Fast response feature so chat acknowledges my requests faster.",
|
||||
"expectedOutcome": "The agent explains the offered fast-response setting and directs the human to its configuration and test surface. The journey checks a real acknowledgement and its failure or usage record. A quick acknowledgement is not presented as completion of the underlying task or evidence that the worker started.",
|
||||
"originalContext": "The fast-response experiment is available but disabled; settings and tests require a human connection manager.",
|
||||
"capabilities": [
|
||||
@@ -6236,13 +6731,13 @@
|
||||
"externalInput": null,
|
||||
"installedSource": null,
|
||||
"originalQuery": "Set a company goal of reaching a thousand signups, with a team goal for launching the website.",
|
||||
"expectedOutcome": "The saved goals have the requested levels, valid parent relationship and appropriate ownership or an explicit missing-owner question. Chat links to the hierarchy and does not invent progress metrics. A retry reuses the intended goals rather than creating another parallel tree.",
|
||||
"expectedOutcome": "The saved goals have the requested levels and a valid same-company parent relationship. Ownership may remain unassigned when none was requested; any assigned owner belongs to the company. The saved hierarchy is inspectable through ordinary Goals navigation and public readback, without invented progress metrics. A retry reuses the intended goals rather than creating another parallel tree.",
|
||||
"originalContext": "The company has a growth mission and no matching quarterly goal; the agent may create goals.",
|
||||
"capabilities": [
|
||||
"goals/hierarchy"
|
||||
],
|
||||
"checks": null,
|
||||
"oracleVersion": "paperclip.chat-catalog.internal.v1",
|
||||
"oracleVersion": "paperclip.chat-catalog.goal-hierarchy.v2",
|
||||
"sourceFiles": [
|
||||
"chat-catalog-domains/internal-definition.ts",
|
||||
"chat-catalog-domains/internal-support.ts",
|
||||
@@ -8261,7 +8756,7 @@
|
||||
"questions-and-approvals/continuation"
|
||||
],
|
||||
"checks": null,
|
||||
"oracleVersion": "paperclip.chat-catalog.identity.v1",
|
||||
"oracleVersion": "paperclip.chat-catalog.answer-delivery.v2",
|
||||
"sourceFiles": [
|
||||
"chat-catalog-domains/identity-confirmation-workflows.ts",
|
||||
"chat-catalog-domains/identity-support.ts",
|
||||
@@ -8347,7 +8842,7 @@
|
||||
"questions-and-approvals/questions"
|
||||
],
|
||||
"checks": null,
|
||||
"oracleVersion": "paperclip.chat-catalog.identity.v1",
|
||||
"oracleVersion": "paperclip.chat-catalog.required-question.v2",
|
||||
"sourceFiles": [
|
||||
"chat-catalog-domains/identity-support.ts",
|
||||
"chat-catalog-domains/internal-support.ts",
|
||||
@@ -9854,18 +10349,20 @@
|
||||
"externalInput": null,
|
||||
"installedSource": null,
|
||||
"originalQuery": "Turn our website review checklist into a reusable skill.",
|
||||
"expectedOutcome": "A writable company skill captures the checklist with clear instructions and a saved revision. Chat links to the skill so the person can inspect or edit it. The agent preserves the actual checklist instead of merely posting a suggestion or pretending creation automatically enables the skill for every agent.",
|
||||
"expectedOutcome": "A writable company skill captures the checklist with clear instructions and a saved revision. The person can discover it through ordinary Skills/My Skills navigation, open its editable Studio content and reload the same saved revision. No particular chat-link format is required. The agent preserves the actual checklist instead of merely posting a suggestion or pretending creation automatically enables the skill for every agent.",
|
||||
"originalContext": "No matching skill exists; the CEO may create company skills.",
|
||||
"capabilities": [
|
||||
"skills/authoring"
|
||||
],
|
||||
"checks": null,
|
||||
"oracleVersion": "paperclip.chat-catalog.integrator.v1",
|
||||
"oracleVersion": "paperclip.chat-catalog.skill-authoring.v2",
|
||||
"sourceFiles": [
|
||||
"chat-catalog-domains/integrator-support.ts",
|
||||
"chat-catalog-domains/internal-support.ts",
|
||||
"chat-catalog-domains/integrator-skills.ts",
|
||||
"chat-scenarios/skills.yaml"
|
||||
"chat-scenarios/skills.yaml",
|
||||
"chat-catalog-domains/integrator-skill-versions.ts",
|
||||
"chat-catalog-domains/identity-support.ts"
|
||||
],
|
||||
"screenshot": "final-state.png",
|
||||
"evidenceKind": "private-final-state-png",
|
||||
@@ -9905,6 +10402,7 @@
|
||||
"checks": null,
|
||||
"oracleVersion": "paperclip.chat-catalog.skill-enable.standard-config.v2",
|
||||
"sourceFiles": [
|
||||
"chat-catalog-domains/integrator-skill-versions.ts",
|
||||
"chat-catalog-domains/integrator-skill-enable.ts",
|
||||
"chat-catalog-domains/identity-support.ts",
|
||||
"chat-catalog-domains/internal-definition.ts",
|
||||
@@ -9957,7 +10455,8 @@
|
||||
"chat-catalog-domains/integrator-support.ts",
|
||||
"chat-catalog-domains/internal-support.ts",
|
||||
"chat-catalog-domains/integrator-skills.ts",
|
||||
"chat-scenarios/skills.yaml"
|
||||
"chat-scenarios/skills.yaml",
|
||||
"chat-catalog-domains/integrator-skill-versions.ts"
|
||||
],
|
||||
"screenshot": "final-state.png",
|
||||
"evidenceKind": "private-final-state-png",
|
||||
@@ -9996,6 +10495,7 @@
|
||||
"checks": null,
|
||||
"oracleVersion": "paperclip.chat-catalog.integrator.v1",
|
||||
"sourceFiles": [
|
||||
"chat-catalog-domains/integrator-skill-versions.ts",
|
||||
"chat-catalog-domains/integrator-skill-imports.ts",
|
||||
"chat-catalog-domains/integrator-skill-source.ts",
|
||||
"chat-catalog-domains/integrator-skills.ts",
|
||||
@@ -10041,6 +10541,7 @@
|
||||
"checks": null,
|
||||
"oracleVersion": "paperclip.chat-catalog.integrator.v1",
|
||||
"sourceFiles": [
|
||||
"chat-catalog-domains/integrator-skill-versions.ts",
|
||||
"chat-catalog-domains/integrator-skill-imports.ts",
|
||||
"chat-catalog-domains/integrator-skill-source.ts",
|
||||
"chat-catalog-domains/integrator-skills.ts",
|
||||
@@ -10085,6 +10586,7 @@
|
||||
"checks": null,
|
||||
"oracleVersion": "paperclip.chat-catalog.integrator.v1",
|
||||
"sourceFiles": [
|
||||
"chat-catalog-domains/integrator-skill-versions.ts",
|
||||
"chat-catalog-domains/integrator-skill-update.ts",
|
||||
"chat-catalog-domains/integrator-skill-source.ts",
|
||||
"chat-catalog-domains/integrator-support.ts",
|
||||
|
||||
@@ -22,9 +22,13 @@ test('the manifest joins all244 source YAML scenarios without importing a domain
|
||||
assert.deepEqual(fullCatalogCases.map(value => value.key).sort(), scenarios.map(value => value.id).sort());
|
||||
for (const scenario of scenarios) {
|
||||
const actual = row(scenario.id);
|
||||
assert.equal(actual.originalQuery, scenario.query);
|
||||
const override = field => manifest.source.repairMetadata.changedFields.find(change => change.key === scenario.id && change.field === field);
|
||||
for (const [field, original] of [['originalQuery', scenario.query], ['expectedOutcome', scenario.expected_outcome]]) {
|
||||
const change = override(field);
|
||||
if (change) { assert.equal(change.previous, original); assert.equal(actual[field], change.current); }
|
||||
else assert.equal(actual[field], original);
|
||||
}
|
||||
assert.equal(actual.originalContext, scenario.context);
|
||||
assert.equal(actual.expectedOutcome, scenario.expected_outcome);
|
||||
assert.ok(!Object.hasOwn(actual, 'status') && !Object.hasOwn(actual, 'passed'));
|
||||
}
|
||||
});
|
||||
@@ -96,23 +100,46 @@ test('foreign identity, duplicated case and changed run-bound metadata are rejec
|
||||
});
|
||||
|
||||
|
||||
test('repair source metadata preserves original definitions and declares scoped observer and oracle changes', () => {
|
||||
test('repair source metadata binds the corrected candidate and explicit changed definitions', () => {
|
||||
const repair = manifest.source.repairMetadata;
|
||||
assert.equal(repair.baseTargetSha, '3b66c3c144360ddf32714a0742fd774b8ceafb16');
|
||||
assert.equal(manifest.targetSha, '9ab0f951a5b5990c266ef4456610e0a0775aa334');
|
||||
assert.equal(repair.previousTargetSha, '9ab0f951a5b5990c266ef4456610e0a0775aa334');
|
||||
assert.equal(manifest.targetSha, '36dff2eab3dfb6de6efb4fd2ce6c1a58f8a5f383');
|
||||
assert.equal(repair.definitionSourceSha, manifest.metadataSourceSha);
|
||||
assert.equal(repair.candidateDescriptorSnapshot, 'fixtures/chat-repair-candidate-descriptors.json');
|
||||
const descriptorBytes = readFileSync(new URL(repair.candidateDescriptorSnapshot, import.meta.url));
|
||||
assert.equal(createHash('sha256').update(descriptorBytes).digest('hex'), repair.candidateDescriptorSha256);
|
||||
const declarations = JSON.parse(descriptorBytes);
|
||||
assert.equal(declarations.length, 242);
|
||||
assert.equal(new Set(declarations.map(value => value.scenarioId)).size, 242);
|
||||
assert.deepEqual(repair.changedFields.map(value => [value.key, value.field]), [
|
||||
['agents.choose-primary', 'sourceFiles'],
|
||||
['companies.agent-observability', 'sourceFiles'], ['companies.configure-decision-model', 'sourceFiles'],
|
||||
['companies.fast-responses', 'sourceFiles'], ['companies.prompt-injection-guard', 'sourceFiles'],
|
||||
['companies.fast-responses', 'sourceFiles'], ['companies.fast-responses', 'originalQuery'],
|
||||
['companies.prompt-injection-guard', 'sourceFiles'],
|
||||
['goals.create-goal-tree', 'oracleVersion'], ['goals.create-goal-tree', 'expectedOutcome'],
|
||||
['questions-and-approvals.duplicate-answer', 'oracleVersion'], ['questions-and-approvals.multi-part-question', 'oracleVersion'],
|
||||
['secrets.configure-company-secret', 'oracleVersion'],
|
||||
['skills.author-skill', 'sourceFiles'], ['skills.author-skill', 'oracleVersion'], ['skills.author-skill', 'expectedOutcome'],
|
||||
['skills.enable-for-agent', 'sourceFiles'], ['skills.explicit-restriction', 'sourceFiles'],
|
||||
['skills.find-and-install', 'sourceFiles'], ['skills.fork-read-only', 'sourceFiles'], ['skills.review-update', 'sourceFiles'],
|
||||
]);
|
||||
for (const change of repair.changedFields) {
|
||||
assert.deepEqual(row(change.key)[change.field], change.current);
|
||||
if (change.field === 'sourceFiles') {
|
||||
assert.deepEqual(change.current.filter(value => value !== 'chat-catalog-domains/identity-company-observer.ts'), change.previous);
|
||||
} else {
|
||||
assert.equal(change.previous, 'paperclip.chat-catalog.external.v1');
|
||||
assert.equal(change.current, 'paperclip.chat-catalog.company-secret-controller-baseline.v2');
|
||||
}
|
||||
assert.notDeepEqual(change.current, change.previous);
|
||||
}
|
||||
assert.equal(repair.descriptorChanges.length, 19);
|
||||
assert.equal(new Set(repair.descriptorChanges.map(change => change.key)).size, 18);
|
||||
assert.deepEqual([...new Set(repair.descriptorChanges.map(change => change.field))].sort(), ['oracleVersion', 'setupPolicy', 'sourceFiles']);
|
||||
for (const change of repair.descriptorChanges) {
|
||||
assert.notDeepEqual(change.current, change.previous);
|
||||
const declaration = declarations.find(value => value.scenarioId === change.key);
|
||||
assert.ok(declaration);
|
||||
assert.deepEqual(change.current, declaration[change.field]);
|
||||
if (Object.hasOwn(row(change.key), change.field)) assert.deepEqual(row(change.key)[change.field], change.current);
|
||||
}
|
||||
assert.deepEqual(repair.changedDefinitionFiles.map(value => value.path), [
|
||||
'tests/runner-e2e/chat-scenarios/companies.yaml', 'tests/runner-e2e/chat-scenarios/goals.yaml', 'tests/runner-e2e/chat-scenarios/skills.yaml',
|
||||
]);
|
||||
assert.ok(repair.changedDefinitionFiles.every(value => /^[a-f0-9]{64}$/.test(value.sha256)));
|
||||
});
|
||||
File diff suppressed because it is too large.
Load diff
Reference in new issue
Block a user