mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-22 11:54:35 +01:00
feat: let agents name prompt-only tasks (#14761)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Users create tasks with a title and a description. > - A required title adds work when the prompt already explains the request. > - An agent can name the task once it reads that request. > - This pull request accepts prompt-only tasks and starts them with a short prompt slice. > - A scoped title tool lets the assigned agent replace that slice early without changing execution state. > - A live browser eval checks the real agent call, saved title, audit entry, and preservation of user titles. ## Linked Issues or Issue Description **Subsystem affected** Cross-cutting: task creation, shared contracts, database, server, runner tools, and board UI. **Problem or motivation** Users must currently write a title before they can submit a detailed task prompt. The agent has enough context to write a useful title itself. **Proposed solution** Make the title optional when a description is present. Save the first 120 characters of the normalized prompt as a provisional title. Ask the assigned agent to call `set_task_title` early. Use an atomic provisional-title guard to preserve titles supplied or edited by users. Keep explicit titles supported. Related: #14543 and #14556 concern empty-title submission. This change intentionally enables that submission when a prompt is present, instead of requiring a title. ## What Changed - Add the `titleNeedsGeneration` field with an idempotent migration. Keep existing titles unchanged. - Add `PUT /api/issues/:id/title` and the native and legacy `set_task_title` tool. Enforce company access, active-run ownership, shared, bounded retry receipts across native/HTTP calls, and transactional audit logging. Refresh external-object links after commit, with the same feature gate and plugin detectors as ordinary title edits. - Add early naming guidance in Standard, Ask, and Plan task context. Preserve the description, status, and assignment. - Allow prompt-only root and child task creation, plus draft restoration in the New Task dialog. Keep user titles supported. - Add an opt-in Product E2E suite for prompt-only Standard and Ask tasks, plus an explicit-title control. It checks actual provider calls within the first five tools, persisted state, audit attribution, and the reloaded UI. - Preserve a closed vocabulary of API key maintenance phrases in declared prose while rejecting opaque credential suffixes. Add one bounded naming retry after wording is rejected, without treating the rejected call as a saved title. - Repair the native cleanup receipt check exposed during full verification: accept matching input digests, retain legacy input checks, and reject conflicting receipts. ## Verification - Live Product E2E on `f43478473800e3a46b85c5ee79677efdb15108e7`: **3/3 passed** with native Codex `gpt-5.4-mini`, first attempts only, automatic retries disabled. Standard and Ask each saved “Rotate expired API key” on their first tool call, with matching persisted state and a single same-run audit entry. The explicit-title control retained its user title with zero title writes. All three verified the reloaded browser UI. - Campaign: `local-2026-09-30T21-30-11-021Z`. Earlier failed campaigns are retained separately; they exposed credential-prose handling and prompted the naming recovery fix. No failed result was regraded or deleted. - Reproduce with `pnpm test:e2e:runner -- --id task-titles.runner-codex-mini.local.prompt-title-standard --id task-titles.runner-codex-mini.local.prompt-title-ask --id task-titles.runner-codex-mini.local.preserve-explicit-title --max-automatic-retries 0` and an authorized provider key. - Full `pnpm -r typecheck` and `pnpm build` passed on the latest commit. The runner build used the configured external eval source tree. - Product E2E unit suite: **61 files, 818 tests passed**; E2E typecheck and UI token gates passed. - Title API/native regressions cover prompt-only and explicit child creation, user edits, ownership/company isolation, external reference refresh, cross-surface retry replay, and the 64-key limit without receipt eviction. All passed. Prompt-context coverage: **44 tests passed**. - Rust credential regressions: **35 tests passed**, including benign maintenance qualifiers and opaque credential rejection in every declared prose field. Catalog/report reconciliation: **28 tests passed**. Native recovery: **560 tests passed**. - Broad local `pnpm test:run`: **14,555 tests passed** in the general server group; two suites failed to initialize embedded PostgreSQL and the existing 40,000-file Git streaming stress test exceeded its 300-second macOS timeout. All three suites then passed in isolation (**5 tests passed**) without code or timeout changes. The original full local command exited nonzero and is not being represented as a clean full run. - Latest-head GitHub checks are green: **53 passed, 4 skipped, zero failed or pending**, including all test shards and the canary packaging dry run. Greptile reviewed the same commit at **5/5**, with zero unresolved review threads. ## Risks - The additive database field must reach the server and UI together. The migration uses `IF NOT EXISTS` and defaults existing tasks to a final title. - Title generation depends on the assigned agent running. Tasks without a run keep their provisional title. - Live qualification covers the native Codex path in Standard and Ask modes. API/legacy and Plan behavior have deterministic coverage. - The credential-prose exception validates the entire suffix against a closed maintenance vocabulary. Unknown suffixes, assignments, quoted values, credential prefixes, and diagnostics retain strict checks. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, tool use, and code execution. The exact deployment ID and context window are not exposed in this session. The live eval uses the native Codex `gpt-5.4-mini` profile. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
d6d67b00d3
commit
018993140f
63 files changed
+2279
-226
No files matched your search
@@ -135,6 +135,26 @@ Adding a task expands its suite's matrix. Update the suite's intentional size,
|
||||
the complete-catalog size, and credential-free unit tests in the same change.
|
||||
Paid tests never silently skip a missing credential or unsupported artifact.
|
||||
|
||||
## Prompt-only task title fixtures
|
||||
|
||||
`task-titles.ts` defines a bounded ordinary writing request and an independent
|
||||
title oracle. Its `single_turn` cases leave the title field empty or supply an
|
||||
explicit control title. The harness captures the exact browser creation response
|
||||
instead of searching by a title that the agent may already have changed. It
|
||||
never patches the title itself. Normal production instructions own the early
|
||||
naming behavior; fixture prompts and agent instruction bundles contain no naming
|
||||
hints. Existing company/secret/environment/agent registry dependencies are reused,
|
||||
with 500-cent company and agent budgets and normal instance teardown.
|
||||
|
||||
Keep the call input, successful result, execution receipt, saved task, and
|
||||
agent/run-attributed audit correlated. Missing evidence must fail. The first-five
|
||||
tool-call bound counts calls in the initial provider run, including discovery.
|
||||
The title must describe API-key rotation without requiring one exact wording.
|
||||
The control must retain its title throughout, not merely restore it at the end.
|
||||
The source digest versions the grader and request in catalog metadata. See
|
||||
[Automatic task titles](README.md#automatic-task-titles) for live selectors,
|
||||
coverage limits, evidence, and calibration.
|
||||
|
||||
## New Paperclip object fixtures
|
||||
|
||||
The explicit-only `lifecycle-baseline` suite reuses this registry and existing
|
||||
|
||||
@@ -878,6 +878,38 @@ environment, task, matcher, or future Paperclip object fixture.
|
||||
See [SECURITY.md](./SECURITY.md) before enabling paid dispatch, the runner
|
||||
group, or permanent public history in this public repository.
|
||||
|
||||
## Automatic task titles
|
||||
|
||||
The explicit-only `task-titles` suite creates tasks in Chromium with an empty
|
||||
title field. Its ordinary writing request contains no naming or tool directions:
|
||||
the production execution prompt must cause the agent to call `set_task_title`.
|
||||
Six local cells cover standard and Ask naming plus an explicit-title preservation
|
||||
control on native Codex and Codex Mini. Each expects one provider run, has a
|
||||
six-minute attempt deadline, and applies 500-cent company and agent budget caps.
|
||||
|
||||
The oracle captures the browser's original POST and creation response, then
|
||||
requires a descriptive replacement title, cleared generation marker, unchanged
|
||||
description/assignee, a correlated successful title call within the first five
|
||||
tool calls of the initial run and before completion, and a matching agent/run
|
||||
audit entry. Reloading the task must show the saved title and requested answer.
|
||||
The control rejects even a temporary rewrite of a user-supplied title. A model's
|
||||
claim that it renamed the task cannot pass. These cells require the dedicated
|
||||
native tool; they do not qualify legacy/API-fallback naming or planning mode.
|
||||
|
||||
```sh
|
||||
pnpm test:e2e:runner -- --list --suite task-titles
|
||||
pnpm test:e2e:runner -- --id task-titles.runner-codex-mini.local.prompt-title-standard --max-automatic-retries 0
|
||||
```
|
||||
|
||||
Only `OPENAI_API_KEY` is required; no Docker artifact oracle or Daytona is used.
|
||||
Use one exact ID for a live smoke run. The existing harness owns fixture setup,
|
||||
cleanup, source/model provenance, billing, failure classification, screenshots,
|
||||
and reports. All runs in the isolated fixture are included in billing/cleanup.
|
||||
Private attempt evidence adds `snapshots/task-title-creation.json` and
|
||||
`snapshots/task-title.json`; public screenshots use the existing marked task
|
||||
route and final-state capture. `task-titles.test.ts` calibrates correct evidence
|
||||
against missing, fabricated, late, misattributed, and overwritten-title outcomes.
|
||||
|
||||
## Everyday user-story evals
|
||||
|
||||
See [EVERYDAY-WORKFLOWS.md](EVERYDAY-WORKFLOWS.md) for the explicit-only native-runner stories and their canonical Evalbook importer. These cells do not expand scheduled `--all` runs.
|
||||
|
||||
@@ -136,10 +136,10 @@ describe("runner E2E catalog", () => {
|
||||
expect(localIntegrityTasks).toHaveLength(2);
|
||||
expect(openRouterBreadthTasks).toHaveLength(3);
|
||||
expect(runnerSuites.map((suite) => suite.expectedMatrixSize)).toEqual([
|
||||
6, 30, 3, 16, 16, 2, 8, 46, 23, 47, 20, 52, 28, 18, 6, 6, 12, 10, 48, 16, 10, 2, 1, 1,
|
||||
6, 30, 3, 16, 16, 2, 6, 8, 46, 23, 47, 20, 52, 28, 18, 6, 6, 12, 10, 48, 16, 10, 2, 1, 1,
|
||||
]);
|
||||
expect(validateRunnerCatalog()).toHaveLength(427);
|
||||
expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(427);
|
||||
expect(validateRunnerCatalog()).toHaveLength(433);
|
||||
expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(433);
|
||||
expect(
|
||||
runnerMatrix.filter((entry) => entry.suite.id === "core-compatibility"),
|
||||
).toHaveLength(48);
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
import { chatConfirmationTasks } from "./chat-cases.js";
|
||||
import { instructionPersistenceTask } from "./instruction-persistence.js";
|
||||
import { apiResponseReadingTask } from "./api-response-reading.js";
|
||||
import { taskTitleTasks, taskTitleDefinitionDigest, TASK_TITLE_BUDGET_CENTS } from "./task-titles.js";
|
||||
import { blockerTasks, blockerProfile } from "./blocker-cases.js";
|
||||
import { accountingTasks } from "./accounting-cases.js";
|
||||
import { continuationTasks } from "./continuation-cases.js";
|
||||
@@ -1138,6 +1139,17 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [
|
||||
tasks: [apiResponseReadingTask], expectedMatrixSize: 2,
|
||||
definitionMetadata: { version: 1, grading: "hidden-evidence-exact-copy-and-api-tool-events", scheduling: "explicit-only" },
|
||||
},
|
||||
{
|
||||
id: "task-titles", label: "Automatic task titles", manualOnly: true,
|
||||
description: "A real agent names a prompt-only task early using production guidance and preserves a supplied title.",
|
||||
groups: ["native", "local"], environments: [localEnvironment],
|
||||
profiles: everydayProfiles.filter(profile => profile.provider === "codex").map(profile => ({
|
||||
...profile,
|
||||
buildAgent(input) { return { ...profile.buildAgent(input), budgetMonthlyCents: TASK_TITLE_BUDGET_CENTS }; },
|
||||
})),
|
||||
tasks: taskTitleTasks, expectedMatrixSize: 6,
|
||||
definitionMetadata: { version: 1, gradingDigest: taskTitleDefinitionDigest, instructions: "production", grading: "initial-response-early-correlated-tool-and-agent-audit", providerRuns: 1, budgetMonthlyCents: TASK_TITLE_BUDGET_CENTS, scheduling: "explicit-only" },
|
||||
},
|
||||
{
|
||||
id: "continuation-accounting", label: "Continuation accounting baseline", manualOnly: true,
|
||||
description: "Structured productive steps, bounded repair, restart and late gates; comments cannot buy more attempts.",
|
||||
@@ -1486,6 +1498,7 @@ export function validateRunnerCatalog(): MatrixExecution[] {
|
||||
...lifecycleLiveTasks,
|
||||
...continuationTasks,
|
||||
...everydayTasks,
|
||||
...taskTitleTasks,
|
||||
...runnerTasks,
|
||||
...localIntegrityTasks,
|
||||
...openRouterBreadthTasks,
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
import path from "node:path";
|
||||
import { isManagedHiringCase } from "./chat-cases.js";
|
||||
import { FixtureRegistry } from "./fixture-registry.js";
|
||||
import { TASK_TITLE_BUDGET_CENTS } from "./task-titles.js";
|
||||
import { stageGrokSubscriptionFixture } from "./grok-subscription-fixture.js";
|
||||
import type { RunnerApi } from "./api.js";
|
||||
import type {
|
||||
@@ -134,7 +135,7 @@ export async function setupLiveFixtures(input: {
|
||||
return api.post<CompanyRecord>("/api/companies", {
|
||||
name: `Runner E2E ${execution.id} ${input.executionNonce}`,
|
||||
description: "Ephemeral paid full-stack runner acceptance fixture",
|
||||
budgetMonthlyCents: 0,
|
||||
budgetMonthlyCents: execution.suite.id === "task-titles" ? TASK_TITLE_BUDGET_CENTS : 0,
|
||||
});
|
||||
},
|
||||
async teardown() {
|
||||
|
||||
@@ -12,6 +12,7 @@ import type { Issue } from "../../packages/shared/src/types/issue.js";
|
||||
import { lifecycleLiveCase, gradeLifecycleRepair } from "./lifecycle-live-cases.js";
|
||||
import { runContinuationFlow } from "./continuation-flow.js";
|
||||
import { runEverydayFlow } from "./everyday-flow.js";
|
||||
import { gradeTaskTitle } from "./task-titles.js";
|
||||
import { runContextIntegrityFlow } from "./context-integrity-flow.js";
|
||||
import { createTaskThroughUi, submitTaskReply } from "./user-actions.js";
|
||||
|
||||
@@ -549,11 +550,12 @@ for (const execution of executions) {
|
||||
const title = execution.task.buildTitle(nonce);
|
||||
let prompt = execution.task.buildPrompt(nonce);
|
||||
let apiResponseSourceId: string | undefined;
|
||||
let titleCreation: { issue: unknown; submitted: unknown } | undefined;
|
||||
let lifecycleBlockerId: string | null = null;
|
||||
const credentials = credentialValues();
|
||||
const secrets = normalizedSecrets(Object.values(credentials));
|
||||
const api = new RunnerApi(request);
|
||||
const companyRunFlow = ["blocker_guidance", "continuation_accounting", "continuation", "context_integrity", "agent_chat", "everyday_workflow", "first_task", "instruction_persistence"].includes(execution.task.flow);
|
||||
const companyRunFlow = execution.suite.id === "task-titles" || ["blocker_guidance", "continuation_accounting", "continuation", "context_integrity", "agent_chat", "everyday_workflow", "first_task", "instruction_persistence"].includes(execution.task.flow);
|
||||
const consoleDiagnostics: Array<Record<string, unknown>> = [];
|
||||
const networkDiagnostics: Array<Record<string, unknown>> = [];
|
||||
const pageLifecycleDiagnostics: Array<Record<string, unknown>> = [];
|
||||
@@ -991,6 +993,14 @@ for (const execution of executions) {
|
||||
if (execution.task.flow === "governed_tool_review") {
|
||||
reviewProvider = await setupConnectionReview({ page, api, prefix: issuePrefix, companyId: fixtures.company.id, agentId: fixtures.agent.id, marker });
|
||||
}
|
||||
// Capture the actual creation response: a generated title can change before
|
||||
// a later list query, and the initial provisional state is part of the oracle.
|
||||
const titleCreationResponse = execution.suite.id === "task-titles"
|
||||
? page.waitForResponse(response => response.request().method() === "POST"
|
||||
&& new URL(response.url()).pathname === `/api/companies/${fixtures!.company.id}/issues`,
|
||||
{ timeout: Math.max(1, startedAtMs + deadlineMs - Date.now()) })
|
||||
.catch((cause: unknown) => new Error("Could not capture task creation response", { cause }))
|
||||
: null;
|
||||
turnSubmissionTimesMs.push(
|
||||
await createTaskThroughUi({
|
||||
page,
|
||||
@@ -1004,7 +1014,14 @@ for (const execution of executions) {
|
||||
);
|
||||
|
||||
const deadlineAt = startedAtMs + deadlineMs;
|
||||
issue = await pollUntil({
|
||||
if (titleCreationResponse) {
|
||||
const response = await titleCreationResponse;
|
||||
if (response instanceof Error) throw response;
|
||||
expect(response.status()).toBe(201);
|
||||
issue = await response.json() as IssueRecord;
|
||||
titleCreation = { issue, submitted: response.request().postDataJSON() };
|
||||
await writeSanitizedJson(snapshotsDir, "task-title-creation.json", titleCreation, secrets);
|
||||
} else issue = await pollUntil({
|
||||
label: `UI-created issue ${title}`,
|
||||
deadlineAt,
|
||||
load: async () => {
|
||||
@@ -2016,6 +2033,17 @@ for (const execution of executions) {
|
||||
await writeSanitizedJson(snapshotsDir, "api-response-pagination.json", paging, secrets);
|
||||
if (!paging.passed) invariantFailures.push(`Bounded response paging was not proven: ${paging.failure}`);
|
||||
}
|
||||
if (execution.suite.id === "task-titles") {
|
||||
const titleEvidence = gradeTaskTitle({
|
||||
initial: titleCreation?.issue, submitted: titleCreation?.submitted,
|
||||
final: issue, prompt, explicitTitle: title, agentId: fixtures.agent.id,
|
||||
runId: selectedRuns[0]!.id,
|
||||
events: runEventsByRun.find(captured => captured.runId === selectedRuns[0]!.id)?.events ?? [],
|
||||
activity: await api.get<unknown[]>(`/api/issues/${issue.id}/activity`),
|
||||
});
|
||||
await writeSanitizedJson(snapshotsDir, "task-title.json", titleEvidence, secrets);
|
||||
invariantFailures.push(...titleEvidence.checks.filter(check => !check.passed).map(check => `${check.id}: ${check.detail}`));
|
||||
}
|
||||
|
||||
const context = record(run.contextSnapshot);
|
||||
const environmentContext = record(context.paperclipEnvironment);
|
||||
@@ -2535,12 +2563,17 @@ for (const execution of executions) {
|
||||
const visibleAgentReplies = page
|
||||
.getByTestId("task-chat-thread")
|
||||
.getByTestId("task-chat-agent-bubble");
|
||||
if (execution.suite.id === "task-titles") {
|
||||
await expect(page.getByRole("heading", { name: issue.title.trim(), exact: true })).toBeVisible();
|
||||
}
|
||||
if (execution.suite.id === "api-response-reading") {
|
||||
// File delivery renders a card instead of an exact summary bubble.
|
||||
// Prove the visible link points to the same independently checked bytes.
|
||||
const proofLink = page.getByRole("link", { name: "Open api-response-proof.txt", exact: true }).first();
|
||||
await expect(proofLink).toBeVisible({ timeout: 30_000 });
|
||||
await expect(proofLink).toHaveAttribute("href", `/api/attachments/${downloadedResponseProof!.attachmentId}/content`);
|
||||
} else if (execution.suite.id === "task-titles") {
|
||||
await expect(visibleAgentReplies.filter({ hasText: marker }).last()).toBeVisible({ timeout: 30_000 });
|
||||
} else if (execution.task.flow === "warm_three_turn") {
|
||||
// Prove the persisted user-facing response is visible, independently
|
||||
// of the byte-for-byte workspace checks and lease continuity checks.
|
||||
|
||||
@@ -0,0 +1,86 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { runnerMatrix, runnerSuites } from "./catalog.js";
|
||||
import { parseRunnerSelectors, selectRunnerExecutions } from "./selectors.js";
|
||||
import { discoverReportCatalog } from "./report-catalog.js";
|
||||
import { gradeTaskTitle, taskTitlePrompt, taskTitleTasks, TASK_TITLE_EARLY_CALL_LIMIT, type TaskTitleEvidenceInput } from "./task-titles.js";
|
||||
|
||||
const prompt = taskTitlePrompt("fixture");
|
||||
const provisional = prompt.replace(/\s+/g, " ").slice(0, 120);
|
||||
const title = "API key rotation checklist";
|
||||
const event = (eventType: string, payload: unknown) => ({ eventType, payload: { prpEvent: { payload } } });
|
||||
const start = (id: string, name: string, input: unknown = {}) => event("item.started", { item: { id, name, type: "tool_use", input } });
|
||||
function evidence(): TaskTitleEvidenceInput {
|
||||
return {
|
||||
prompt, explicitTitle: "", agentId: "agent", runId: "run",
|
||||
submitted: { description: prompt },
|
||||
initial: { id: "task", title: provisional, titleNeedsGeneration: true, description: prompt },
|
||||
final: { id: "task", title, titleNeedsGeneration: false, description: prompt, assigneeAgentId: "agent" },
|
||||
events: [
|
||||
start("context", "get_task_context"),
|
||||
start("naming", "set_task_title", { title, onlyIfProvisional: true, idempotencyKey: "initial-title" }),
|
||||
event("item.completed", { item: { id: "naming", tool_use_id: "naming", type: "tool_result", result: { id: "task", title, changed: true, titleNeedsGeneration: false } } }),
|
||||
event("tool.execution.completed", { name: "set_task_title", executionId: "naming", status: "completed" }),
|
||||
start("finish", "paperclip_finish"),
|
||||
],
|
||||
activity: [{ entityId: "task", entityType: "issue", action: "issue.updated", actorType: "agent", actorId: "agent", runId: "run", details: { title, previous: { title: provisional } } }],
|
||||
};
|
||||
}
|
||||
|
||||
describe("task title Product E2E", () => {
|
||||
it("registers six explicit-only local cells, with production instructions and no naming hints in the request", () => {
|
||||
const suite = runnerSuites.find(entry => entry.id === "task-titles")!;
|
||||
const selected = selectRunnerExecutions(parseRunnerSelectors(["--suite", suite.id]));
|
||||
expect(selected).toHaveLength(6);
|
||||
expect(selected.every(entry => entry.environment.id === "local" && entry.profile.provider === "codex")).toBe(true);
|
||||
expect(selectRunnerExecutions(parseRunnerSelectors(["--all"])).some(entry => entry.suite.id === suite.id)).toBe(false);
|
||||
expect(suite.definitionMetadata).toMatchObject({ instructions: "production", providerRuns: 1, budgetMonthlyCents: 500 });
|
||||
for (const task of taskTitleTasks) {
|
||||
expect(task).toMatchObject({ expectedRunCount: 1, flow: "single_turn" });
|
||||
expect(task.buildPrompt("fixture")).not.toMatch(/title|set_task|\/api\/|finish_task|paperclip_finish/i);
|
||||
}
|
||||
const discovered = discoverReportCatalog({ catalog: runnerMatrix, expected: selected.map(entry => entry.id), results: [] });
|
||||
expect(discovered.filter(entry => entry.suite.id === suite.id)).toHaveLength(6);
|
||||
});
|
||||
|
||||
it("accepts early, correlated tool evidence plus the same agent/run's persisted title mutation", () => {
|
||||
expect(gradeTaskTitle(evidence())).toMatchObject({ passed: true, initialTitle: provisional, finalTitle: title });
|
||||
});
|
||||
|
||||
it.each([
|
||||
["missing evidence", (input: TaskTitleEvidenceInput) => { input.events = []; input.activity = []; }],
|
||||
["a missing creation request", (input: TaskTitleEvidenceInput) => { input.submitted = undefined; }],
|
||||
["a narrated call", (input: TaskTitleEvidenceInput) => { input.events = [event("message", { text: `set_task_title(${title}) succeeded` })]; }],
|
||||
["a stale prompt slice", (input: TaskTitleEvidenceInput) => { (input.final as Record<string, unknown>).title = provisional; }],
|
||||
["a generic new title", (input: TaskTitleEvidenceInput) => { (input.final as Record<string, unknown>).title = "Complete the requested task"; }],
|
||||
["a title supplied by the fixture", (input: TaskTitleEvidenceInput) => { input.submitted = { title: provisional, description: prompt }; }],
|
||||
["missing provisional marker", (input: TaskTitleEvidenceInput) => { (input.initial as Record<string, unknown>).titleNeedsGeneration = false; }],
|
||||
["a late title call", (input: TaskTitleEvidenceInput) => { input.events = [...Array.from({ length: TASK_TITLE_EARLY_CALL_LIMIT }, (_, i) => start(`read-${i}`, "get_task_context")), ...input.events]; }],
|
||||
["naming after five shell tools", (input: TaskTitleEvidenceInput) => { input.events = [...Array.from({ length: TASK_TITLE_EARLY_CALL_LIMIT }, (_, i) => event("tool.execution.started", { executionId: `shell-${i}`, name: "shell" })), ...input.events]; }],
|
||||
["naming after completion", (input: TaskTitleEvidenceInput) => { input.events = [start("already-finished", "paperclip_finish"), ...input.events]; }],
|
||||
["an unmatched result", (input: TaskTitleEvidenceInput) => { input.events = input.events.filter(row => row.eventType !== "item.completed"); }],
|
||||
["no successful execution receipt", (input: TaskTitleEvidenceInput) => { input.events = input.events.filter(row => row.eventType !== "tool.execution.completed"); }],
|
||||
["another run's mutation", (input: TaskTitleEvidenceInput) => { (input.activity[0] as Record<string, unknown>).runId = "other-run"; }],
|
||||
["a board-authored mutation", (input: TaskTitleEvidenceInput) => { (input.activity[0] as Record<string, unknown>).actorType = "user"; }],
|
||||
["a different task", (input: TaskTitleEvidenceInput) => { (input.final as Record<string, unknown>).id = "other-task"; }],
|
||||
["a truncated prompt", (input: TaskTitleEvidenceInput) => { (input.final as Record<string, unknown>).description = provisional; }],
|
||||
] as const)("rejects %s", (_label, corrupt) => {
|
||||
const input = evidence(); corrupt(input);
|
||||
expect(gradeTaskTitle(input).passed).toBe(false);
|
||||
});
|
||||
|
||||
it("preserves an explicit title and rejects a rename even if the original is later restored", () => {
|
||||
const input = evidence();
|
||||
input.explicitTitle = "My chosen title";
|
||||
input.submitted = { title: input.explicitTitle, description: prompt };
|
||||
input.initial = { ...(input.initial as object), title: input.explicitTitle, titleNeedsGeneration: false };
|
||||
input.final = { ...(input.final as object), title: input.explicitTitle };
|
||||
input.events = [start("context", "get_task_context"), start("finish", "paperclip_finish")];
|
||||
const forbiddenWrite = input.activity[0]; input.activity = [];
|
||||
expect(gradeTaskTitle(input).passed).toBe(true);
|
||||
const calls = input.events; input.events = [];
|
||||
expect(gradeTaskTitle(input).passed).toBe(false);
|
||||
input.events = calls;
|
||||
input.activity = [forbiddenWrite];
|
||||
expect(gradeTaskTitle(input).passed).toBe(false);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,126 @@
|
||||
import { createHash } from "node:crypto";
|
||||
import { readFileSync } from "node:fs";
|
||||
import type { RunnerTaskFixture } from "./types.js";
|
||||
|
||||
export const TASK_TITLE_BUDGET_CENTS = 500;
|
||||
export const TASK_TITLE_EARLY_CALL_LIMIT = 5;
|
||||
export const taskTitleDefinitionDigest = createHash("sha256").update(readFileSync(new URL(import.meta.url))).digest("hex");
|
||||
|
||||
// Deliberately no title/tool/API instructions: production guidance must cause naming.
|
||||
export function taskTitlePrompt(nonce: string) {
|
||||
return [
|
||||
"I am preparing a short internal handover for a teammate who is taking over a routine maintenance job tomorrow. Keep the answer practical and brief.",
|
||||
"Give me a three-item checklist for safely rotating an expired API key: create its replacement, update the consuming service, and verify access before revoking the old key.",
|
||||
`Include receipt CHECKLIST-${nonce} in your answer. This is writing only; do not access a real account, change credentials, or create files.`,
|
||||
].join("\n\n");
|
||||
}
|
||||
|
||||
export const taskTitleTasks: readonly RunnerTaskFixture[] = ([
|
||||
["prompt-title-standard", "Name a prompt-only task", "standard"],
|
||||
["prompt-title-ask", "Name a prompt-only Ask task", "ask"],
|
||||
["preserve-explicit-title", "Preserve the user's title", "standard"],
|
||||
] as const).map(([id, label, workMode]) => ({
|
||||
id, label, groups: [], workMode,
|
||||
flow: "single_turn", expectedRunCount: 1,
|
||||
attemptTimeoutMs: { local: 6 * 60_000, daytona: 6 * 60_000 },
|
||||
expectedTerminalState: { issue: "done", run: "succeeded" },
|
||||
buildTitle: nonce => id === "preserve-explicit-title" ? `Teammate handover ${nonce}` : "",
|
||||
buildPrompt: taskTitlePrompt,
|
||||
buildVisibleMarker: nonce => `CHECKLIST-${nonce}`,
|
||||
buildMatchers: (nonce, execution) => [
|
||||
{ kind: "message_contains", expected: `CHECKLIST-${nonce}` },
|
||||
{ kind: "issue_status", expected: "done" },
|
||||
{ kind: "run_status", expected: "succeeded" },
|
||||
{ kind: "runtime_mode", expected: execution.profile.expectedRuntimeMode },
|
||||
{ kind: "environment", expected: execution.environment.id },
|
||||
],
|
||||
}));
|
||||
|
||||
const record = (value: unknown): Record<string, unknown> =>
|
||||
value !== null && typeof value === "object" && !Array.isArray(value) ? value as Record<string, unknown> : {};
|
||||
|
||||
export interface TaskTitleEvidenceInput {
|
||||
initial: unknown;
|
||||
final: unknown;
|
||||
submitted: unknown;
|
||||
prompt: string;
|
||||
explicitTitle: string;
|
||||
agentId: string;
|
||||
runId: string;
|
||||
events: readonly { eventType?: string; payload?: unknown }[];
|
||||
activity: readonly unknown[];
|
||||
}
|
||||
|
||||
/** Grade real call/result/receipt evidence and persisted state, never model prose. */
|
||||
export function gradeTaskTitle(input: TaskTitleEvidenceInput) {
|
||||
const initial = record(input.initial), final = record(input.final), submitted = record(input.submitted);
|
||||
const provisional = input.prompt.trim().replace(/\s+/g, " ").slice(0, 120);
|
||||
const checks: Array<{ id: string; passed: boolean; detail: string }> = [];
|
||||
const check = (id: string, passed: boolean, detail: string) => checks.push({ id, passed, detail });
|
||||
const starts = new Map<string, { name: string; input: Record<string, unknown>; position: number; index: number }>();
|
||||
const results = new Map<string, Record<string, unknown>>();
|
||||
const completions = new Map<string, number>();
|
||||
let position = 0;
|
||||
input.events.forEach((event, index) => {
|
||||
const payload = record(record(record(event.payload).prpEvent).payload);
|
||||
const item = record(payload.item);
|
||||
if (event.eventType === "tool.execution.started" && typeof payload.executionId === "string" && !starts.has(payload.executionId)) {
|
||||
starts.set(payload.executionId, { name: String(payload.name ?? ""), input: {}, position: ++position, index });
|
||||
}
|
||||
if (event.eventType === "item.started" && item.type === "tool_use" && typeof item.id === "string" && typeof item.name === "string") {
|
||||
const prior = starts.get(item.id);
|
||||
starts.set(item.id, { name: item.name, input: record(item.input), position: prior?.position ?? ++position, index: prior?.index ?? index });
|
||||
}
|
||||
if (event.eventType === "item.completed" && item.type === "tool_result" && typeof item.id === "string"
|
||||
&& (item.tool_use_id === undefined || item.tool_use_id === item.id)) results.set(item.id, record(item.result));
|
||||
if (event.eventType === "tool.execution.completed" && payload.name === "set_task_title" && payload.status === "completed" && typeof payload.executionId === "string") {
|
||||
completions.set(payload.executionId, index);
|
||||
}
|
||||
});
|
||||
const calls = [...starts].filter(([, call]) => call.name === "set_task_title");
|
||||
const titleWrites = input.activity.map(record).filter(row => row.action === "issue.updated"
|
||||
&& row.entityType === "issue" && row.entityId === initial.id && typeof record(row.details).title === "string");
|
||||
check("creation-request", submitted.description === input.prompt
|
||||
&& (input.explicitTitle ? submitted.title === input.explicitTitle : !Object.hasOwn(submitted, "title")), "Browser submitted the full prompt with the requested title, or omitted the title entirely.");
|
||||
check("provider-tool-evidence", starts.size > 0 && Boolean(input.runId), "Provider tool evidence from the initial run is present; narration alone is insufficient.");
|
||||
check("initial-title", initial.title === (input.explicitTitle || provisional)
|
||||
&& initial.titleNeedsGeneration === !Boolean(input.explicitTitle), "Creation response proves the initial title and server-owned generation marker before the agent runs.");
|
||||
check("same-task-and-description", typeof initial.id === "string" && final.id === initial.id
|
||||
&& initial.description === input.prompt && final.description === input.prompt
|
||||
&& final.assigneeAgentId === input.agentId, "The original task, assignment, and complete prompt survive naming.");
|
||||
check("generation-settled", final.titleNeedsGeneration === false, "The stored task no longer needs a generated title.");
|
||||
if (input.explicitTitle) {
|
||||
check("explicit-title-preserved", final.title === input.explicitTitle && titleWrites.length === 0
|
||||
&& calls.every(([id]) => results.get(id)?.changed === false), "No title-changing audit or tool result may overwrite a supplied title, even temporarily.");
|
||||
} else {
|
||||
const title = typeof final.title === "string" ? final.title : "";
|
||||
check("descriptive-title", title.trim() === title && title.length >= 8 && title.length <= 120
|
||||
&& title !== provisional && !input.prompt.startsWith(title) && !title.includes("CHECKLIST-")
|
||||
&& /(rotat|replac|renew|expir)/i.test(title) && /\b(api|keys?|credentials?)\b/i.test(title), "A concise title describes the key-rotation request instead of copying the prompt prefix or answer receipt.");
|
||||
const terminalIndex = input.events.findIndex(event => event.eventType === "run.terminal");
|
||||
const finishIndex = Math.min(
|
||||
[...starts.values()].find(call => ["paperclip_finish", "finish_task"].includes(call.name))?.index ?? input.events.length,
|
||||
terminalIndex < 0 ? input.events.length : terminalIndex,
|
||||
);
|
||||
const proven = calls.filter(([id, call]) => {
|
||||
const result = results.get(id);
|
||||
const completedAt = completions.get(id);
|
||||
return result !== undefined && call.position <= TASK_TITLE_EARLY_CALL_LIMIT && call.input.onlyIfProvisional === true
|
||||
&& typeof call.input.idempotencyKey === "string" && call.input.idempotencyKey.trim().length > 0
|
||||
&& call.input.title === title && result.id === initial.id && result.title === title
|
||||
&& result.changed === true && result.titleNeedsGeneration === false
|
||||
&& completedAt !== undefined && completedAt > call.index && completedAt < finishIndex;
|
||||
});
|
||||
check("early-title-tool", proven.length > 0, `A correlated set_task_title call must succeed among the first ${TASK_TITLE_EARLY_CALL_LIMIT} tools of the initial run, before completion.`);
|
||||
check("agent-title-audit", titleWrites.length === 1 && titleWrites[0]?.actorType === "agent"
|
||||
&& titleWrites[0]?.actorId === input.agentId && titleWrites[0]?.runId === input.runId
|
||||
&& record(titleWrites[0]?.details).title === title
|
||||
&& record(record(titleWrites[0]?.details).previous).title === provisional, "The same assigned agent and initial run must own the single persisted title mutation.");
|
||||
}
|
||||
return {
|
||||
schema: "paperclip.task-title-eval.v1", passed: checks.every(entry => entry.passed), checks,
|
||||
initialTitle: initial.title, finalTitle: final.title,
|
||||
titleCalls: calls.map(([id, call]) => ({ id, position: call.position, completed: completions.has(id), result: results.get(id) ?? null })),
|
||||
titleWrites,
|
||||
};
|
||||
}
|
||||
Reference in new issue
Block a user