Files
Dotta b93ad538b6 test(runner): add question adapter conformance (#12409)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The runner maps provider questions to one versioned Paperclip
contract.
> - Codex and ACPX now perform that mapping through separate adapters.
> - Separate adapter tests do not prove that both paths preserve the
same user-visible form.
> - Fixture validation must also match production behavior for optional
answers and unsupported patterns.
> - This pull request adds shared fixtures, validation, and
cross-adapter conformance checks.
> - The benefit is a reusable question contract for later providers
without enabling a new runtime.

## Linked Issues or Issue Description

Refs #12408

This pull request builds on the Codex ACPX question bridge merged in
#12408. It adds fixture and generator checks for the existing Codex and
ACPX question adapters. It does not add another provider or production
execution path.

## What Changed

- Add a canonical ACPX form fixture and native response. Mark the
equivalent Codex fixture field as required.
- Add shared validation for question IDs, option IDs, answer modes,
required answers, text bounds, numeric bounds, and response shapes.
- Evaluate fixture-only regular expressions in a bounded child process.
Reject patterns that cannot finish safely.
- Validate ACPX fixtures against a manifest-side mirror of the
production form projection. Reject free-text ACPX patterns, but ignore
patterns on enumerated option fields.
- Accept explicit empty optional answers and omit them from the
projected ACPX response, which matches the production parser.
- Validate every question fixture during manifest generation and
regenerate the checked-in manifest.
- Add cross-adapter tests that compare user-visible presentation while
preserving provider-owned IDs and provider-specific response conversion.
- Add negative regressions for malformed forms, invalid responses,
unsafe patterns, special property names, and projection drift.

## Verification

- Replay base: `4fe3189f0256873a359d2d53c209076919fd1c3b` (`master`
after #12408 merged).
- Exact replay head: `0532e7dfbb5a246033ffeef55a0c0013fdab07f1`.
- Stable patch ID for the intended seven-file delta:
`28154d86b2c37e0e8d442419e26703584852f67e`.
- The intended pull request delta contains exactly these seven files:
  - `packages/paperclip-runner/protocol/fixtures/questions/acpx.json`
  - `packages/paperclip-runner/protocol/fixtures/questions/codex.json`
  - `packages/paperclip-runner/protocol/manifest.json`
  - `packages/paperclip-runner/scripts/generate-protocol-manifest.mjs`
  - `packages/paperclip-runner/scripts/protocol-contract.mjs`
-
`packages/paperclip-runner/src/contracts/question-adapter-conformance.test.ts`
  - `packages/paperclip-runner/test/protocol-contract.test.mjs`
- The intended combined delta is 1,211 additions and 20 deletions.
- This change does not add a dependency, lockfile update, migration,
workflow, server route, UI change, documentation file, or production
runtime change.
- GitHub Actions run `33352004952` passed the complete matrix on retry
at the unchanged exact head, including protocol/package verification,
build, typecheck/release-registry, general and serialized server suites,
canary, and all e2e shards.
- Superagent, Socket, Snyk, contributor-trust, policy, and PR-review
checks pass on the exact replay head.
- Greptile reviewed the exact replay head at 5/5 with no blocking
finding and zero unresolved review threads.
- No local test result is claimed. GitHub Actions is the authoritative
verification environment for the replayed revision.

## Risks

This change has low runtime risk because it changes fixtures, generator
validation, generated metadata, and tests only. Fixture pattern checks
run in a child process with a one-second timeout and a bounded output
buffer. The ACPX gate intentionally rejects free-text patterns because
the production adapter has no bounded expression engine. It
intentionally permits an explicit empty optional answer because
production omits that answer from the native response. A validation
mismatch can block manifest generation, but it cannot change server
selection, direct adapters, or task-page behavior.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

OpenAI Codex with GPT-5.6, extended reasoning, repository tool use, and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-08-30 22:09:03 -05:00

76 lines
1.9 KiB
JSON

{
"schema": "paperclip.question_adapter_fixture.v1",
"adapter": "acpx",
"nativeRequest": {
"method": "elicitation/create",
"params": {
"mode": "form",
"message": "Deployment input",
"requestedSchema": {
"type": "object",
"title": "Deployment input",
"required": ["environment"],
"properties": {
"environment": {
"type": "string",
"title": "Environment",
"description": "Where should we deploy?",
"oneOf": [
{
"const": "staging",
"title": "Staging",
"description": "Deploy to staging first."
},
{
"const": "production",
"title": "Production",
"description": "Deploy directly to production."
}
]
}
}
}
}
},
"canonicalQuestionSet": {
"schema": "paperclip.question_set.v1",
"title": "Deployment input",
"submitLabel": "Submit answers",
"questions": [
{
"id": "field-1-environment-ba5285161ba6",
"header": "Environment",
"prompt": "Where should we deploy?",
"required": true,
"answerMode": "single_select",
"options": [
{
"id": "option-1",
"label": "Staging",
"description": "Deploy to staging first."
},
{
"id": "option-2",
"label": "Production",
"description": "Deploy directly to production."
}
]
}
]
},
"canonicalResponse": {
"schema": "paperclip.question_response.v1",
"answers": {
"field-1-environment-ba5285161ba6": {
"selectedOptionIds": ["option-1"]
}
}
},
"nativeResponse": {
"action": "accept",
"content": {
"environment": "staging"
}
}
}