mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-03 17:51:58 +01:00
Fix TestOps launch preflight search encoding (#18620)
## Thinking Path > - Paperclip manages AI agents that do real work. > - Product evals verify the saved outcome of that work. > - TestOps plans select and dispatch these evals. > - Operator launchers check a unique launch name before they request work. > - TestOps expects an encoded filter, but the launchers send plain name text. > - This change uses the vendor filter format in the shared TestOps API owner. > - The existing checks can then prevent duplicate launch requests. ## Linked Issues or Issue Description Refs #18516, #18608, and #18542. The completion pilot registered successfully after #18608. Its optional API launcher then stopped with HTTP 400 in [run 41477551241](https://github.com/paperclipai/paperclip/actions/runs/41477551241). It stopped before the launch mutation marker. No evaluation was dispatched by that request. The launch search API uses Base64 of UTF-8 JSON filters. The launchers sent plain name text. I confirmed the format in the deployed public [vendor filter codec](https://paperclip.testops.cloud/es-testops-CICoDnkU.js) and its launch UI. The encoded name filter displayed the existing pilot launch. The TestOps UI separately dispatched [the genuine three-case pilot](https://github.com/paperclipai/paperclip/actions/runs/41478906153); this PR does not create another run or claim that pilot passed. ## What Changed - Add the typed name-filter encoder to the existing TestOps API module. - Use it in the backlog and completion launch preflight requests. The full-catalog route uses the same completion adapter. - Preserve the exact launch name, nonce, selection, authorization, once-only markers, and no-resend behavior. - Add protocol controls through both real launch functions. Document the wire format. ## Verification - Red: both new consumer controls reproduce the HTTP 400 boundary with plain name text. The other 109 API controls pass. - Green: all 111 API controls pass after the fix. All 166 related API, pilot contract, and full-catalog controls pass with Node 24.22.2. - The first broad local invocation lacked the `yaml` dependency in the new worktree. Its failure log is retained. Reusing the existing installed dependencies allowed all 166 controls to run and pass without further source changes. - The vendor-format query displayed the existing uniquely named pilot launch in the authenticated TestOps UI. No launch POST or model call was used for this read-only verification. - Diff check passes. All required CI checks pass on exact head `b55bdc6753fa042e4c2465e7b9b366b8126b4730` ([run 41480643021](https://github.com/paperclipai/paperclip/actions/runs/41480643021)). Both current-head reviews are 5/5, with zero unresolved findings. No TypeScript, compiled product, workflow, grader, model, clock, retry, or evidence-boundary changes. ## Risks The vendor can change its search format. The format is kept in the existing provider-specific API module, and both launch consumers have protocol controls. Existing duplicate detection and uncertain-response refusal remain in place. The already-running UI pilot is independent of this fix. ## Model Used OpenAI GPT-6 through Codex assisted with reasoning, repository inspection, code execution, and native accessibility tool use. The host does not expose the exact model revision or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either linked existing issues or described the issue in-PR following the relevant issue template - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal Paperclip ticket id - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge
This commit is contained in:
1 parent
f8a888a072
commit
a6afe3e230
5 files changed
+45
-3
No files matched your search
@@ -239,6 +239,12 @@ Initial setup uses the existing workflow operations:
|
||||
independent checks, private-evidence receipts and cleanup. Inspect screenshots
|
||||
through authenticated private-history retrieval. Retain earlier failed attempts.
|
||||
|
||||
Launch preflight searches use TestOps' typed name filter, encoded as Base64 of
|
||||
UTF-8 JSON in the `search` query parameter. Plain name text returns HTTP 400.
|
||||
The backlog and completion launchers share this encoding; the full-catalog
|
||||
launcher uses the completion adapter. Existing nonce and once-only guards still
|
||||
apply before a plan-run request.
|
||||
|
||||
The runtime accepts only `operation`, `ALLURE_JOB_RUN_ID` and reserved username
|
||||
metadata. It rejects arbitrary source/model/budget selectors, branch workflows,
|
||||
replays and foreign cases. All secret-bearing jobs recheck actual GitHub run and
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import { testopsClient, endpoint } from './chat-feature-testops-api.mjs';
|
||||
import { testopsClient, endpoint, launchNameSearch } from './chat-feature-testops-api.mjs';
|
||||
import { cases, planName, positive, reject, validateSelection, catalogDefinition } from './chat-completion-testops-contract.mjs';
|
||||
|
||||
import { reconcileFullCatalog } from './chat-full-catalog-testops-reconcile.mjs';
|
||||
@@ -225,7 +225,7 @@ export async function launchPilotPlan(api, env, context, workflowId, markOnce) {
|
||||
if (!Array.isArray(assignments) || assignments.length !== 1 || assignments[0].job?.id !== binding.jobId
|
||||
|| assignments[0].testCasesCount !== cases.length) reject('plan_job_mismatch');
|
||||
const name = `${planName} · ${env.CHAT_COMPLETION_TESTOPS_LAUNCH_NONCE}`;
|
||||
const existing = pageRows(await api(`/api/launch?${new URLSearchParams({ projectId: '1', search: name, page: '0', size: '100' })}`));
|
||||
const existing = pageRows(await api(`/api/launch?${new URLSearchParams({ projectId: '1', search: launchNameSearch(name), page: '0', size: '100' })}`));
|
||||
if (existing.length) reject('launch_already_requested');
|
||||
await markOnce('launch_plan');
|
||||
const launched = await api(`/api/testplan/${binding.planId}/run`, 'POST', { launchName: name });
|
||||
|
||||
@@ -128,6 +128,22 @@ test('the fenced launcher requests one exact three-case TestOps launch', async (
|
||||
launchName: `${planName} · ${operatorEnv.CHAT_COMPLETION_TESTOPS_LAUNCH_NONCE}` } }]);
|
||||
});
|
||||
|
||||
test('completion launch preflight sends the typed UTF-8 name filter required by TestOps', async () => {
|
||||
const f = mutationApi({ assignments: [{ job, testCasesCount: 3 }] });
|
||||
const api = async (route, ...args) => {
|
||||
if (route.startsWith('/api/launch?')) {
|
||||
const wire = new URL(route, 'https://paperclip.testops.cloud').searchParams.get('search');
|
||||
let filters;
|
||||
try { filters = JSON.parse(Buffer.from(wire, 'base64').toString('utf8')); }
|
||||
catch { throw new Error('TestOps HTTP 400: typed launch search required'); }
|
||||
assert.deepEqual(filters, [{ id: 'name', type: 'string', value: `${planName} · ${operatorEnv.CHAT_COMPLETION_TESTOPS_LAUNCH_NONCE}` }]);
|
||||
}
|
||||
return f.api(route, ...args);
|
||||
};
|
||||
await launchPilotPlan(api, operatorEnv, context, 999, async () => {});
|
||||
assert.equal(f.calls.filter(row => row.method === 'POST').length, 1);
|
||||
});
|
||||
|
||||
for (const changed of [{ GITHUB_RUN_NUMBER: '6' }, { CHAT_COMPLETION_TESTOPS_LAUNCH_NONCE: 'not-a-nonce' }, { CHAT_COMPLETION_TESTOPS_LAUNCH_RUN_NUMBER: '' }]) {
|
||||
test(`launcher rejects its invalid fence before requests: ${Object.keys(changed)[0]}`, async () => {
|
||||
const f = mutationApi();
|
||||
|
||||
@@ -2,6 +2,10 @@ export const endpoint = 'https://paperclip.testops.cloud';
|
||||
export const planName = 'Chat feature evals';
|
||||
export const fullName = 'paperclip.Product E2E.chat-features.create-backlog';
|
||||
export const aliases = [fullName, 'paperclip.Product E2E.chat-features.tasks.create-backlog'];
|
||||
export function launchNameSearch(name) {
|
||||
// TestOps search uses the UI's typed filter carrier, not plain search text.
|
||||
return Buffer.from(JSON.stringify([{ id: 'name', type: 'string', value: name }]), 'utf8').toString('base64');
|
||||
}
|
||||
class SetupFailure extends Error {
|
||||
constructor(code, status = null) { super(code); this.code = code; this.status = status; }
|
||||
}
|
||||
@@ -348,7 +352,7 @@ export async function launchChatPlan(api, env, markOnce, admission) {
|
||||
const assignments = await api(`/api/testplan/${known.planId}/job`);
|
||||
if (!Array.isArray(assignments) || assignments.length > 1 || assignments.some(row => row.job?.id !== known.jobId || row.testCasesCount !== 1)) fail('chat_testops_plan_job_mismatch');
|
||||
const launchName = `${planName} · ${nonce}`;
|
||||
const existing = rows(await api(`/api/launch?${new URLSearchParams({ projectId: '1', search: launchName, page: '0', size: '100' })}`));
|
||||
const existing = rows(await api(`/api/launch?${new URLSearchParams({ projectId: '1', search: launchNameSearch(launchName), page: '0', size: '100' })}`));
|
||||
if (existing.length) {
|
||||
if (existing.length !== 1 || existing[0].name !== launchName || !id(existing[0].id)) fail('chat_testops_nonce_conflict');
|
||||
return { schema: 'paperclip.chat-feature-testops-launch/v1', ...source, nonce, planId: known.planId,
|
||||
|
||||
@@ -196,6 +196,22 @@ test('launch configures only the exact registered public job/plan then issues on
|
||||
assert.ok(!server.calls.some(row => /jobrun.*rerun|dispatches/.test(row.route)));
|
||||
});
|
||||
|
||||
test('backlog launch preflight sends the typed UTF-8 name filter required by TestOps', async () => {
|
||||
const server = launchServer();
|
||||
const api = async (route, ...args) => {
|
||||
if (route.startsWith('/api/launch?')) {
|
||||
const wire = new URL(route, 'https://paperclip.testops.cloud').searchParams.get('search');
|
||||
let filters;
|
||||
try { filters = JSON.parse(Buffer.from(wire, 'base64').toString('utf8')); }
|
||||
catch { throw new Error('TestOps HTTP 400: typed launch search required'); }
|
||||
assert.deepEqual(filters, [{ id: 'name', type: 'string', value: `Chat feature evals · ${launchEnv.CHAT_TESTOPS_LAUNCH_NONCE}` }]);
|
||||
}
|
||||
return server.api(route, ...args);
|
||||
};
|
||||
await launchChatPlan(api, launchEnv, async () => {}, await fence());
|
||||
assert.equal(server.requests(), 1);
|
||||
});
|
||||
|
||||
test('observed launch nonce prevents a second run request across separate local entry attempts', async () => {
|
||||
const server = launchServer(); await launchChatPlan(server.api, launchEnv, async () => {}, await fence());
|
||||
const result = await launchChatPlan(server.api, launchEnv, () => assert.fail('existing launch must not mark or write'), await fence());
|
||||
|
||||
Reference in new issue
Block a user