Fix TestOps launch preflight search encoding (#18620)

## Thinking Path

> - Paperclip manages AI agents that do real work.
> - Product evals verify the saved outcome of that work.
> - TestOps plans select and dispatch these evals.
> - Operator launchers check a unique launch name before they request
work.
> - TestOps expects an encoded filter, but the launchers send plain name
text.
> - This change uses the vendor filter format in the shared TestOps API
owner.
> - The existing checks can then prevent duplicate launch requests.

## Linked Issues or Issue Description

Refs #18516, #18608, and #18542. The completion pilot registered
successfully after #18608. Its optional API launcher then stopped with
HTTP 400 in [run
41477551241](https://github.com/paperclipai/paperclip/actions/runs/41477551241).
It stopped before the launch mutation marker. No evaluation was
dispatched by that request.

The launch search API uses Base64 of UTF-8 JSON filters. The launchers
sent plain name text. I confirmed the format in the deployed public
[vendor filter
codec](https://paperclip.testops.cloud/es-testops-CICoDnkU.js) and its
launch UI. The encoded name filter displayed the existing pilot launch.
The TestOps UI separately dispatched [the genuine three-case
pilot](https://github.com/paperclipai/paperclip/actions/runs/41478906153);
this PR does not create another run or claim that pilot passed.

## What Changed

- Add the typed name-filter encoder to the existing TestOps API module.
- Use it in the backlog and completion launch preflight requests. The
full-catalog route uses the same completion adapter.
- Preserve the exact launch name, nonce, selection, authorization,
once-only markers, and no-resend behavior.
- Add protocol controls through both real launch functions. Document the
wire format.

## Verification

- Red: both new consumer controls reproduce the HTTP 400 boundary with
plain name text. The other 109 API controls pass.
- Green: all 111 API controls pass after the fix. All 166 related API,
pilot contract, and full-catalog controls pass with Node 24.22.2.
- The first broad local invocation lacked the `yaml` dependency in the
new worktree. Its failure log is retained. Reusing the existing
installed dependencies allowed all 166 controls to run and pass without
further source changes.
- The vendor-format query displayed the existing uniquely named pilot
launch in the authenticated TestOps UI. No launch POST or model call was
used for this read-only verification.
- Diff check passes. All required CI checks pass on exact head
`b55bdc6753fa042e4c2465e7b9b366b8126b4730` ([run
41480643021](https://github.com/paperclipai/paperclip/actions/runs/41480643021)).
Both current-head reviews are 5/5, with zero unresolved findings. No
TypeScript, compiled product, workflow, grader, model, clock, retry, or
evidence-boundary changes.

## Risks

The vendor can change its search format. The format is kept in the
existing provider-specific API module, and both launch consumers have
protocol controls. Existing duplicate detection and uncertain-response
refusal remain in place. The already-running UI pilot is independent of
this fix.

## Model Used

OpenAI GPT-6 through Codex assisted with reasoning, repository
inspection, code execution, and native accessibility tool use. The host
does not expose the exact model revision or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either linked existing issues or described the issue in-PR
following the relevant issue template
- [x] I have not referenced internal or instance-local Paperclip issues
or links
- [x] My branch name describes the change and contains no internal
Paperclip ticket id
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
This commit is contained in:
Dotta authored and GitHub committed 2026-11-02 10:40:23 -06:00
1 parent f8a888a072
commit a6afe3e230
5 files changed
+45 -3

No files matched your search

+6
View File
@@ -239,6 +239,12 @@ Initial setup uses the existing workflow operations:
independent checks, private-evidence receipts and cleanup. Inspect screenshots
through authenticated private-history retrieval. Retain earlier failed attempts.
Launch preflight searches use TestOps' typed name filter, encoded as Base64 of
UTF-8 JSON in the `search` query parameter. Plain name text returns HTTP 400.
The backlog and completion launchers share this encoding; the full-catalog
launcher uses the completion adapter. Existing nonce and once-only guards still
apply before a plan-run request.
The runtime accepts only `operation`, `ALLURE_JOB_RUN_ID` and reserved username
metadata. It rejects arbitrary source/model/budget selectors, branch workflows,
replays and foreign cases. All secret-bearing jobs recheck actual GitHub run and
@@ -1,4 +1,4 @@
import { testopsClient, endpoint } from './chat-feature-testops-api.mjs';
import { testopsClient, endpoint, launchNameSearch } from './chat-feature-testops-api.mjs';
import { cases, planName, positive, reject, validateSelection, catalogDefinition } from './chat-completion-testops-contract.mjs';
import { reconcileFullCatalog } from './chat-full-catalog-testops-reconcile.mjs';
@@ -225,7 +225,7 @@ export async function launchPilotPlan(api, env, context, workflowId, markOnce) {
if (!Array.isArray(assignments) || assignments.length !== 1 || assignments[0].job?.id !== binding.jobId
|| assignments[0].testCasesCount !== cases.length) reject('plan_job_mismatch');
const name = `${planName} · ${env.CHAT_COMPLETION_TESTOPS_LAUNCH_NONCE}`;
const existing = pageRows(await api(`/api/launch?${new URLSearchParams({ projectId: '1', search: name, page: '0', size: '100' })}`));
const existing = pageRows(await api(`/api/launch?${new URLSearchParams({ projectId: '1', search: launchNameSearch(name), page: '0', size: '100' })}`));
if (existing.length) reject('launch_already_requested');
await markOnce('launch_plan');
const launched = await api(`/api/testplan/${binding.planId}/run`, 'POST', { launchName: name });
@@ -128,6 +128,22 @@ test('the fenced launcher requests one exact three-case TestOps launch', async (
launchName: `${planName} · ${operatorEnv.CHAT_COMPLETION_TESTOPS_LAUNCH_NONCE}` } }]);
});
test('completion launch preflight sends the typed UTF-8 name filter required by TestOps', async () => {
const f = mutationApi({ assignments: [{ job, testCasesCount: 3 }] });
const api = async (route, ...args) => {
if (route.startsWith('/api/launch?')) {
const wire = new URL(route, 'https://paperclip.testops.cloud').searchParams.get('search');
let filters;
try { filters = JSON.parse(Buffer.from(wire, 'base64').toString('utf8')); }
catch { throw new Error('TestOps HTTP 400: typed launch search required'); }
assert.deepEqual(filters, [{ id: 'name', type: 'string', value: `${planName} · ${operatorEnv.CHAT_COMPLETION_TESTOPS_LAUNCH_NONCE}` }]);
}
return f.api(route, ...args);
};
await launchPilotPlan(api, operatorEnv, context, 999, async () => {});
assert.equal(f.calls.filter(row => row.method === 'POST').length, 1);
});
for (const changed of [{ GITHUB_RUN_NUMBER: '6' }, { CHAT_COMPLETION_TESTOPS_LAUNCH_NONCE: 'not-a-nonce' }, { CHAT_COMPLETION_TESTOPS_LAUNCH_RUN_NUMBER: '' }]) {
test(`launcher rejects its invalid fence before requests: ${Object.keys(changed)[0]}`, async () => {
const f = mutationApi();
@@ -2,6 +2,10 @@ export const endpoint = 'https://paperclip.testops.cloud';
export const planName = 'Chat feature evals';
export const fullName = 'paperclip.Product E2E.chat-features.create-backlog';
export const aliases = [fullName, 'paperclip.Product E2E.chat-features.tasks.create-backlog'];
export function launchNameSearch(name) {
// TestOps search uses the UI's typed filter carrier, not plain search text.
return Buffer.from(JSON.stringify([{ id: 'name', type: 'string', value: name }]), 'utf8').toString('base64');
}
class SetupFailure extends Error {
constructor(code, status = null) { super(code); this.code = code; this.status = status; }
}
@@ -348,7 +352,7 @@ export async function launchChatPlan(api, env, markOnce, admission) {
const assignments = await api(`/api/testplan/${known.planId}/job`);
if (!Array.isArray(assignments) || assignments.length > 1 || assignments.some(row => row.job?.id !== known.jobId || row.testCasesCount !== 1)) fail('chat_testops_plan_job_mismatch');
const launchName = `${planName} · ${nonce}`;
const existing = rows(await api(`/api/launch?${new URLSearchParams({ projectId: '1', search: launchName, page: '0', size: '100' })}`));
const existing = rows(await api(`/api/launch?${new URLSearchParams({ projectId: '1', search: launchNameSearch(launchName), page: '0', size: '100' })}`));
if (existing.length) {
if (existing.length !== 1 || existing[0].name !== launchName || !id(existing[0].id)) fail('chat_testops_nonce_conflict');
return { schema: 'paperclip.chat-feature-testops-launch/v1', ...source, nonce, planId: known.planId,
@@ -196,6 +196,22 @@ test('launch configures only the exact registered public job/plan then issues on
assert.ok(!server.calls.some(row => /jobrun.*rerun|dispatches/.test(row.route)));
});
test('backlog launch preflight sends the typed UTF-8 name filter required by TestOps', async () => {
const server = launchServer();
const api = async (route, ...args) => {
if (route.startsWith('/api/launch?')) {
const wire = new URL(route, 'https://paperclip.testops.cloud').searchParams.get('search');
let filters;
try { filters = JSON.parse(Buffer.from(wire, 'base64').toString('utf8')); }
catch { throw new Error('TestOps HTTP 400: typed launch search required'); }
assert.deepEqual(filters, [{ id: 'name', type: 'string', value: `Chat feature evals · ${launchEnv.CHAT_TESTOPS_LAUNCH_NONCE}` }]);
}
return server.api(route, ...args);
};
await launchChatPlan(api, launchEnv, async () => {}, await fence());
assert.equal(server.requests(), 1);
});
test('observed launch nonce prevents a second run request across separate local entry attempts', async () => {
const server = launchServer(); await launchChatPlan(server.api, launchEnv, async () => {}, await fence());
const result = await launchChatPlan(server.api, launchEnv, () => assert.fail('existing launch must not mark or write'), await fence());