feat: let agents name prompt-only tasks (#14761)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Users create tasks with a title and a description.
> - A required title adds work when the prompt already explains the
request.
> - An agent can name the task once it reads that request.
> - This pull request accepts prompt-only tasks and starts them with a
short prompt slice.
> - A scoped title tool lets the assigned agent replace that slice early
without changing execution state.
> - A live browser eval checks the real agent call, saved title, audit
entry, and preservation of user titles.

## Linked Issues or Issue Description

**Subsystem affected**

Cross-cutting: task creation, shared contracts, database, server, runner
tools, and board UI.

**Problem or motivation**

Users must currently write a title before they can submit a detailed
task prompt. The agent has enough context to write a useful title
itself.

**Proposed solution**

Make the title optional when a description is present. Save the first
120 characters of the normalized prompt as a provisional title. Ask the
assigned agent to call `set_task_title` early. Use an atomic
provisional-title guard to preserve titles supplied or edited by users.
Keep explicit titles supported.

Related: #14543 and #14556 concern empty-title submission. This change
intentionally enables that submission when a prompt is present, instead
of requiring a title.

## What Changed

- Add the `titleNeedsGeneration` field with an idempotent migration.
Keep existing titles unchanged.
- Add `PUT /api/issues/:id/title` and the native and legacy
`set_task_title` tool. Enforce company access, active-run ownership,
shared, bounded retry receipts across native/HTTP calls, and
transactional audit logging. Refresh external-object links after commit,
with the same feature gate and plugin detectors as ordinary title edits.
- Add early naming guidance in Standard, Ask, and Plan task context.
Preserve the description, status, and assignment.
- Allow prompt-only root and child task creation, plus draft restoration
in the New Task dialog. Keep user titles supported.
- Add an opt-in Product E2E suite for prompt-only Standard and Ask
tasks, plus an explicit-title control. It checks actual provider calls
within the first five tools, persisted state, audit attribution, and the
reloaded UI.
- Preserve a closed vocabulary of API key maintenance phrases in
declared prose while rejecting opaque credential suffixes. Add one
bounded naming retry after wording is rejected, without treating the
rejected call as a saved title.
- Repair the native cleanup receipt check exposed during full
verification: accept matching input digests, retain legacy input checks,
and reject conflicting receipts.

## Verification

- Live Product E2E on `f43478473800e3a46b85c5ee79677efdb15108e7`: **3/3
passed** with native Codex `gpt-5.4-mini`, first attempts only,
automatic retries disabled. Standard and Ask each saved “Rotate expired
API key” on their first tool call, with matching persisted state and a
single same-run audit entry. The explicit-title control retained its
user title with zero title writes. All three verified the reloaded
browser UI.
- Campaign: `local-2026-09-30T21-30-11-021Z`. Earlier failed campaigns
are retained separately; they exposed credential-prose handling and
prompted the naming recovery fix. No failed result was regraded or
deleted.
- Reproduce with `pnpm test:e2e:runner -- --id
task-titles.runner-codex-mini.local.prompt-title-standard --id
task-titles.runner-codex-mini.local.prompt-title-ask --id
task-titles.runner-codex-mini.local.preserve-explicit-title
--max-automatic-retries 0` and an authorized provider key.
- Full `pnpm -r typecheck` and `pnpm build` passed on the latest commit.
The runner build used the configured external eval source tree.
- Product E2E unit suite: **61 files, 818 tests passed**; E2E typecheck
and UI token gates passed.
- Title API/native regressions cover prompt-only and explicit child
creation, user edits, ownership/company isolation, external reference
refresh, cross-surface retry replay, and the 64-key limit without
receipt eviction. All passed. Prompt-context coverage: **44 tests
passed**.
- Rust credential regressions: **35 tests passed**, including benign
maintenance qualifiers and opaque credential rejection in every declared
prose field. Catalog/report reconciliation: **28 tests passed**. Native
recovery: **560 tests passed**.
- Broad local `pnpm test:run`: **14,555 tests passed** in the general
server group; two suites failed to initialize embedded PostgreSQL and
the existing 40,000-file Git streaming stress test exceeded its
300-second macOS timeout. All three suites then passed in isolation (**5
tests passed**) without code or timeout changes. The original full local
command exited nonzero and is not being represented as a clean full run.
- Latest-head GitHub checks are green: **53 passed, 4 skipped, zero
failed or pending**, including all test shards and the canary packaging
dry run. Greptile reviewed the same commit at **5/5**, with zero
unresolved review threads.

## Risks

- The additive database field must reach the server and UI together. The
migration uses `IF NOT EXISTS` and defaults existing tasks to a final
title.
- Title generation depends on the assigned agent running. Tasks without
a run keep their provisional title.
- Live qualification covers the native Codex path in Standard and Ask
modes. API/legacy and Plan behavior have deterministic coverage.
- The credential-prose exception validates the entire suffix against a
closed maintenance vocabulary. Unknown suffixes, assignments, quoted
values, credential prefixes, and diagnostics retain strict checks.

## Model Used

OpenAI Codex, based on GPT-6, with reasoning, tool use, and code
execution. The exact deployment ID and context window are not exposed in
this session. The live eval uses the native Codex `gpt-5.4-mini`
profile.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed 2026-09-30 16:48:24 -05:00
1 parent d6d67b00d3
commit 018993140f
63 files changed
+2279 -226

No files matched your search

+20
View File
@@ -135,6 +135,26 @@ Adding a task expands its suite's matrix. Update the suite's intentional size,
the complete-catalog size, and credential-free unit tests in the same change.
Paid tests never silently skip a missing credential or unsupported artifact.
## Prompt-only task title fixtures
`task-titles.ts` defines a bounded ordinary writing request and an independent
title oracle. Its `single_turn` cases leave the title field empty or supply an
explicit control title. The harness captures the exact browser creation response
instead of searching by a title that the agent may already have changed. It
never patches the title itself. Normal production instructions own the early
naming behavior; fixture prompts and agent instruction bundles contain no naming
hints. Existing company/secret/environment/agent registry dependencies are reused,
with 500-cent company and agent budgets and normal instance teardown.
Keep the call input, successful result, execution receipt, saved task, and
agent/run-attributed audit correlated. Missing evidence must fail. The first-five
tool-call bound counts calls in the initial provider run, including discovery.
The title must describe API-key rotation without requiring one exact wording.
The control must retain its title throughout, not merely restore it at the end.
The source digest versions the grader and request in catalog metadata. See
[Automatic task titles](README.md#automatic-task-titles) for live selectors,
coverage limits, evidence, and calibration.
## New Paperclip object fixtures
The explicit-only `lifecycle-baseline` suite reuses this registry and existing
+32
View File
@@ -878,6 +878,38 @@ environment, task, matcher, or future Paperclip object fixture.
See [SECURITY.md](./SECURITY.md) before enabling paid dispatch, the runner
group, or permanent public history in this public repository.
## Automatic task titles
The explicit-only `task-titles` suite creates tasks in Chromium with an empty
title field. Its ordinary writing request contains no naming or tool directions:
the production execution prompt must cause the agent to call `set_task_title`.
Six local cells cover standard and Ask naming plus an explicit-title preservation
control on native Codex and Codex Mini. Each expects one provider run, has a
six-minute attempt deadline, and applies 500-cent company and agent budget caps.
The oracle captures the browser's original POST and creation response, then
requires a descriptive replacement title, cleared generation marker, unchanged
description/assignee, a correlated successful title call within the first five
tool calls of the initial run and before completion, and a matching agent/run
audit entry. Reloading the task must show the saved title and requested answer.
The control rejects even a temporary rewrite of a user-supplied title. A model's
claim that it renamed the task cannot pass. These cells require the dedicated
native tool; they do not qualify legacy/API-fallback naming or planning mode.
```sh
pnpm test:e2e:runner -- --list --suite task-titles
pnpm test:e2e:runner -- --id task-titles.runner-codex-mini.local.prompt-title-standard --max-automatic-retries 0
```
Only `OPENAI_API_KEY` is required; no Docker artifact oracle or Daytona is used.
Use one exact ID for a live smoke run. The existing harness owns fixture setup,
cleanup, source/model provenance, billing, failure classification, screenshots,
and reports. All runs in the isolated fixture are included in billing/cleanup.
Private attempt evidence adds `snapshots/task-title-creation.json` and
`snapshots/task-title.json`; public screenshots use the existing marked task
route and final-state capture. `task-titles.test.ts` calibrates correct evidence
against missing, fabricated, late, misattributed, and overwritten-title outcomes.
## Everyday user-story evals
See [EVERYDAY-WORKFLOWS.md](EVERYDAY-WORKFLOWS.md) for the explicit-only native-runner stories and their canonical Evalbook importer. These cells do not expand scheduled `--all` runs.
+3 -3
View File
@@ -136,10 +136,10 @@ describe("runner E2E catalog", () => {
expect(localIntegrityTasks).toHaveLength(2);
expect(openRouterBreadthTasks).toHaveLength(3);
expect(runnerSuites.map((suite) => suite.expectedMatrixSize)).toEqual([
6, 30, 3, 16, 16, 2, 8, 46, 23, 47, 20, 52, 28, 18, 6, 6, 12, 10, 48, 16, 10, 2, 1, 1,
6, 30, 3, 16, 16, 2, 6, 8, 46, 23, 47, 20, 52, 28, 18, 6, 6, 12, 10, 48, 16, 10, 2, 1, 1,
]);
expect(validateRunnerCatalog()).toHaveLength(427);
expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(427);
expect(validateRunnerCatalog()).toHaveLength(433);
expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(433);
expect(
runnerMatrix.filter((entry) => entry.suite.id === "core-compatibility"),
).toHaveLength(48);
+13
View File
@@ -1,6 +1,7 @@
import { chatConfirmationTasks } from "./chat-cases.js";
import { instructionPersistenceTask } from "./instruction-persistence.js";
import { apiResponseReadingTask } from "./api-response-reading.js";
import { taskTitleTasks, taskTitleDefinitionDigest, TASK_TITLE_BUDGET_CENTS } from "./task-titles.js";
import { blockerTasks, blockerProfile } from "./blocker-cases.js";
import { accountingTasks } from "./accounting-cases.js";
import { continuationTasks } from "./continuation-cases.js";
@@ -1138,6 +1139,17 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [
tasks: [apiResponseReadingTask], expectedMatrixSize: 2,
definitionMetadata: { version: 1, grading: "hidden-evidence-exact-copy-and-api-tool-events", scheduling: "explicit-only" },
},
{
id: "task-titles", label: "Automatic task titles", manualOnly: true,
description: "A real agent names a prompt-only task early using production guidance and preserves a supplied title.",
groups: ["native", "local"], environments: [localEnvironment],
profiles: everydayProfiles.filter(profile => profile.provider === "codex").map(profile => ({
...profile,
buildAgent(input) { return { ...profile.buildAgent(input), budgetMonthlyCents: TASK_TITLE_BUDGET_CENTS }; },
})),
tasks: taskTitleTasks, expectedMatrixSize: 6,
definitionMetadata: { version: 1, gradingDigest: taskTitleDefinitionDigest, instructions: "production", grading: "initial-response-early-correlated-tool-and-agent-audit", providerRuns: 1, budgetMonthlyCents: TASK_TITLE_BUDGET_CENTS, scheduling: "explicit-only" },
},
{
id: "continuation-accounting", label: "Continuation accounting baseline", manualOnly: true,
description: "Structured productive steps, bounded repair, restart and late gates; comments cannot buy more attempts.",
@@ -1486,6 +1498,7 @@ export function validateRunnerCatalog(): MatrixExecution[] {
...lifecycleLiveTasks,
...continuationTasks,
...everydayTasks,
...taskTitleTasks,
...runnerTasks,
...localIntegrityTasks,
...openRouterBreadthTasks,
+2 -1
View File
@@ -1,6 +1,7 @@
import path from "node:path";
import { isManagedHiringCase } from "./chat-cases.js";
import { FixtureRegistry } from "./fixture-registry.js";
import { TASK_TITLE_BUDGET_CENTS } from "./task-titles.js";
import { stageGrokSubscriptionFixture } from "./grok-subscription-fixture.js";
import type { RunnerApi } from "./api.js";
import type {
@@ -134,7 +135,7 @@ export async function setupLiveFixtures(input: {
return api.post<CompanyRecord>("/api/companies", {
name: `Runner E2E ${execution.id} ${input.executionNonce}`,
description: "Ephemeral paid full-stack runner acceptance fixture",
budgetMonthlyCents: 0,
budgetMonthlyCents: execution.suite.id === "task-titles" ? TASK_TITLE_BUDGET_CENTS : 0,
});
},
async teardown() {
+35 -2
View File
@@ -12,6 +12,7 @@ import type { Issue } from "../../packages/shared/src/types/issue.js";
import { lifecycleLiveCase, gradeLifecycleRepair } from "./lifecycle-live-cases.js";
import { runContinuationFlow } from "./continuation-flow.js";
import { runEverydayFlow } from "./everyday-flow.js";
import { gradeTaskTitle } from "./task-titles.js";
import { runContextIntegrityFlow } from "./context-integrity-flow.js";
import { createTaskThroughUi, submitTaskReply } from "./user-actions.js";
@@ -549,11 +550,12 @@ for (const execution of executions) {
const title = execution.task.buildTitle(nonce);
let prompt = execution.task.buildPrompt(nonce);
let apiResponseSourceId: string | undefined;
let titleCreation: { issue: unknown; submitted: unknown } | undefined;
let lifecycleBlockerId: string | null = null;
const credentials = credentialValues();
const secrets = normalizedSecrets(Object.values(credentials));
const api = new RunnerApi(request);
const companyRunFlow = ["blocker_guidance", "continuation_accounting", "continuation", "context_integrity", "agent_chat", "everyday_workflow", "first_task", "instruction_persistence"].includes(execution.task.flow);
const companyRunFlow = execution.suite.id === "task-titles" || ["blocker_guidance", "continuation_accounting", "continuation", "context_integrity", "agent_chat", "everyday_workflow", "first_task", "instruction_persistence"].includes(execution.task.flow);
const consoleDiagnostics: Array<Record<string, unknown>> = [];
const networkDiagnostics: Array<Record<string, unknown>> = [];
const pageLifecycleDiagnostics: Array<Record<string, unknown>> = [];
@@ -991,6 +993,14 @@ for (const execution of executions) {
if (execution.task.flow === "governed_tool_review") {
reviewProvider = await setupConnectionReview({ page, api, prefix: issuePrefix, companyId: fixtures.company.id, agentId: fixtures.agent.id, marker });
}
// Capture the actual creation response: a generated title can change before
// a later list query, and the initial provisional state is part of the oracle.
const titleCreationResponse = execution.suite.id === "task-titles"
? page.waitForResponse(response => response.request().method() === "POST"
&& new URL(response.url()).pathname === `/api/companies/${fixtures!.company.id}/issues`,
{ timeout: Math.max(1, startedAtMs + deadlineMs - Date.now()) })
.catch((cause: unknown) => new Error("Could not capture task creation response", { cause }))
: null;
turnSubmissionTimesMs.push(
await createTaskThroughUi({
page,
@@ -1004,7 +1014,14 @@ for (const execution of executions) {
);
const deadlineAt = startedAtMs + deadlineMs;
issue = await pollUntil({
if (titleCreationResponse) {
const response = await titleCreationResponse;
if (response instanceof Error) throw response;
expect(response.status()).toBe(201);
issue = await response.json() as IssueRecord;
titleCreation = { issue, submitted: response.request().postDataJSON() };
await writeSanitizedJson(snapshotsDir, "task-title-creation.json", titleCreation, secrets);
} else issue = await pollUntil({
label: `UI-created issue ${title}`,
deadlineAt,
load: async () => {
@@ -2016,6 +2033,17 @@ for (const execution of executions) {
await writeSanitizedJson(snapshotsDir, "api-response-pagination.json", paging, secrets);
if (!paging.passed) invariantFailures.push(`Bounded response paging was not proven: ${paging.failure}`);
}
if (execution.suite.id === "task-titles") {
const titleEvidence = gradeTaskTitle({
initial: titleCreation?.issue, submitted: titleCreation?.submitted,
final: issue, prompt, explicitTitle: title, agentId: fixtures.agent.id,
runId: selectedRuns[0]!.id,
events: runEventsByRun.find(captured => captured.runId === selectedRuns[0]!.id)?.events ?? [],
activity: await api.get<unknown[]>(`/api/issues/${issue.id}/activity`),
});
await writeSanitizedJson(snapshotsDir, "task-title.json", titleEvidence, secrets);
invariantFailures.push(...titleEvidence.checks.filter(check => !check.passed).map(check => `${check.id}: ${check.detail}`));
}
const context = record(run.contextSnapshot);
const environmentContext = record(context.paperclipEnvironment);
@@ -2535,12 +2563,17 @@ for (const execution of executions) {
const visibleAgentReplies = page
.getByTestId("task-chat-thread")
.getByTestId("task-chat-agent-bubble");
if (execution.suite.id === "task-titles") {
await expect(page.getByRole("heading", { name: issue.title.trim(), exact: true })).toBeVisible();
}
if (execution.suite.id === "api-response-reading") {
// File delivery renders a card instead of an exact summary bubble.
// Prove the visible link points to the same independently checked bytes.
const proofLink = page.getByRole("link", { name: "Open api-response-proof.txt", exact: true }).first();
await expect(proofLink).toBeVisible({ timeout: 30_000 });
await expect(proofLink).toHaveAttribute("href", `/api/attachments/${downloadedResponseProof!.attachmentId}/content`);
} else if (execution.suite.id === "task-titles") {
await expect(visibleAgentReplies.filter({ hasText: marker }).last()).toBeVisible({ timeout: 30_000 });
} else if (execution.task.flow === "warm_three_turn") {
// Prove the persisted user-facing response is visible, independently
// of the byte-for-byte workspace checks and lease continuity checks.
+86
View File
@@ -0,0 +1,86 @@
import { describe, expect, it } from "vitest";
import { runnerMatrix, runnerSuites } from "./catalog.js";
import { parseRunnerSelectors, selectRunnerExecutions } from "./selectors.js";
import { discoverReportCatalog } from "./report-catalog.js";
import { gradeTaskTitle, taskTitlePrompt, taskTitleTasks, TASK_TITLE_EARLY_CALL_LIMIT, type TaskTitleEvidenceInput } from "./task-titles.js";
const prompt = taskTitlePrompt("fixture");
const provisional = prompt.replace(/\s+/g, " ").slice(0, 120);
const title = "API key rotation checklist";
const event = (eventType: string, payload: unknown) => ({ eventType, payload: { prpEvent: { payload } } });
const start = (id: string, name: string, input: unknown = {}) => event("item.started", { item: { id, name, type: "tool_use", input } });
function evidence(): TaskTitleEvidenceInput {
return {
prompt, explicitTitle: "", agentId: "agent", runId: "run",
submitted: { description: prompt },
initial: { id: "task", title: provisional, titleNeedsGeneration: true, description: prompt },
final: { id: "task", title, titleNeedsGeneration: false, description: prompt, assigneeAgentId: "agent" },
events: [
start("context", "get_task_context"),
start("naming", "set_task_title", { title, onlyIfProvisional: true, idempotencyKey: "initial-title" }),
event("item.completed", { item: { id: "naming", tool_use_id: "naming", type: "tool_result", result: { id: "task", title, changed: true, titleNeedsGeneration: false } } }),
event("tool.execution.completed", { name: "set_task_title", executionId: "naming", status: "completed" }),
start("finish", "paperclip_finish"),
],
activity: [{ entityId: "task", entityType: "issue", action: "issue.updated", actorType: "agent", actorId: "agent", runId: "run", details: { title, previous: { title: provisional } } }],
};
}
describe("task title Product E2E", () => {
it("registers six explicit-only local cells, with production instructions and no naming hints in the request", () => {
const suite = runnerSuites.find(entry => entry.id === "task-titles")!;
const selected = selectRunnerExecutions(parseRunnerSelectors(["--suite", suite.id]));
expect(selected).toHaveLength(6);
expect(selected.every(entry => entry.environment.id === "local" && entry.profile.provider === "codex")).toBe(true);
expect(selectRunnerExecutions(parseRunnerSelectors(["--all"])).some(entry => entry.suite.id === suite.id)).toBe(false);
expect(suite.definitionMetadata).toMatchObject({ instructions: "production", providerRuns: 1, budgetMonthlyCents: 500 });
for (const task of taskTitleTasks) {
expect(task).toMatchObject({ expectedRunCount: 1, flow: "single_turn" });
expect(task.buildPrompt("fixture")).not.toMatch(/title|set_task|\/api\/|finish_task|paperclip_finish/i);
}
const discovered = discoverReportCatalog({ catalog: runnerMatrix, expected: selected.map(entry => entry.id), results: [] });
expect(discovered.filter(entry => entry.suite.id === suite.id)).toHaveLength(6);
});
it("accepts early, correlated tool evidence plus the same agent/run's persisted title mutation", () => {
expect(gradeTaskTitle(evidence())).toMatchObject({ passed: true, initialTitle: provisional, finalTitle: title });
});
it.each([
["missing evidence", (input: TaskTitleEvidenceInput) => { input.events = []; input.activity = []; }],
["a missing creation request", (input: TaskTitleEvidenceInput) => { input.submitted = undefined; }],
["a narrated call", (input: TaskTitleEvidenceInput) => { input.events = [event("message", { text: `set_task_title(${title}) succeeded` })]; }],
["a stale prompt slice", (input: TaskTitleEvidenceInput) => { (input.final as Record<string, unknown>).title = provisional; }],
["a generic new title", (input: TaskTitleEvidenceInput) => { (input.final as Record<string, unknown>).title = "Complete the requested task"; }],
["a title supplied by the fixture", (input: TaskTitleEvidenceInput) => { input.submitted = { title: provisional, description: prompt }; }],
["missing provisional marker", (input: TaskTitleEvidenceInput) => { (input.initial as Record<string, unknown>).titleNeedsGeneration = false; }],
["a late title call", (input: TaskTitleEvidenceInput) => { input.events = [...Array.from({ length: TASK_TITLE_EARLY_CALL_LIMIT }, (_, i) => start(`read-${i}`, "get_task_context")), ...input.events]; }],
["naming after five shell tools", (input: TaskTitleEvidenceInput) => { input.events = [...Array.from({ length: TASK_TITLE_EARLY_CALL_LIMIT }, (_, i) => event("tool.execution.started", { executionId: `shell-${i}`, name: "shell" })), ...input.events]; }],
["naming after completion", (input: TaskTitleEvidenceInput) => { input.events = [start("already-finished", "paperclip_finish"), ...input.events]; }],
["an unmatched result", (input: TaskTitleEvidenceInput) => { input.events = input.events.filter(row => row.eventType !== "item.completed"); }],
["no successful execution receipt", (input: TaskTitleEvidenceInput) => { input.events = input.events.filter(row => row.eventType !== "tool.execution.completed"); }],
["another run's mutation", (input: TaskTitleEvidenceInput) => { (input.activity[0] as Record<string, unknown>).runId = "other-run"; }],
["a board-authored mutation", (input: TaskTitleEvidenceInput) => { (input.activity[0] as Record<string, unknown>).actorType = "user"; }],
["a different task", (input: TaskTitleEvidenceInput) => { (input.final as Record<string, unknown>).id = "other-task"; }],
["a truncated prompt", (input: TaskTitleEvidenceInput) => { (input.final as Record<string, unknown>).description = provisional; }],
] as const)("rejects %s", (_label, corrupt) => {
const input = evidence(); corrupt(input);
expect(gradeTaskTitle(input).passed).toBe(false);
});
it("preserves an explicit title and rejects a rename even if the original is later restored", () => {
const input = evidence();
input.explicitTitle = "My chosen title";
input.submitted = { title: input.explicitTitle, description: prompt };
input.initial = { ...(input.initial as object), title: input.explicitTitle, titleNeedsGeneration: false };
input.final = { ...(input.final as object), title: input.explicitTitle };
input.events = [start("context", "get_task_context"), start("finish", "paperclip_finish")];
const forbiddenWrite = input.activity[0]; input.activity = [];
expect(gradeTaskTitle(input).passed).toBe(true);
const calls = input.events; input.events = [];
expect(gradeTaskTitle(input).passed).toBe(false);
input.events = calls;
input.activity = [forbiddenWrite];
expect(gradeTaskTitle(input).passed).toBe(false);
});
});
+126
View File
@@ -0,0 +1,126 @@
import { createHash } from "node:crypto";
import { readFileSync } from "node:fs";
import type { RunnerTaskFixture } from "./types.js";
export const TASK_TITLE_BUDGET_CENTS = 500;
export const TASK_TITLE_EARLY_CALL_LIMIT = 5;
export const taskTitleDefinitionDigest = createHash("sha256").update(readFileSync(new URL(import.meta.url))).digest("hex");
// Deliberately no title/tool/API instructions: production guidance must cause naming.
export function taskTitlePrompt(nonce: string) {
return [
"I am preparing a short internal handover for a teammate who is taking over a routine maintenance job tomorrow. Keep the answer practical and brief.",
"Give me a three-item checklist for safely rotating an expired API key: create its replacement, update the consuming service, and verify access before revoking the old key.",
`Include receipt CHECKLIST-${nonce} in your answer. This is writing only; do not access a real account, change credentials, or create files.`,
].join("\n\n");
}
export const taskTitleTasks: readonly RunnerTaskFixture[] = ([
["prompt-title-standard", "Name a prompt-only task", "standard"],
["prompt-title-ask", "Name a prompt-only Ask task", "ask"],
["preserve-explicit-title", "Preserve the user's title", "standard"],
] as const).map(([id, label, workMode]) => ({
id, label, groups: [], workMode,
flow: "single_turn", expectedRunCount: 1,
attemptTimeoutMs: { local: 6 * 60_000, daytona: 6 * 60_000 },
expectedTerminalState: { issue: "done", run: "succeeded" },
buildTitle: nonce => id === "preserve-explicit-title" ? `Teammate handover ${nonce}` : "",
buildPrompt: taskTitlePrompt,
buildVisibleMarker: nonce => `CHECKLIST-${nonce}`,
buildMatchers: (nonce, execution) => [
{ kind: "message_contains", expected: `CHECKLIST-${nonce}` },
{ kind: "issue_status", expected: "done" },
{ kind: "run_status", expected: "succeeded" },
{ kind: "runtime_mode", expected: execution.profile.expectedRuntimeMode },
{ kind: "environment", expected: execution.environment.id },
],
}));
const record = (value: unknown): Record<string, unknown> =>
value !== null && typeof value === "object" && !Array.isArray(value) ? value as Record<string, unknown> : {};
export interface TaskTitleEvidenceInput {
initial: unknown;
final: unknown;
submitted: unknown;
prompt: string;
explicitTitle: string;
agentId: string;
runId: string;
events: readonly { eventType?: string; payload?: unknown }[];
activity: readonly unknown[];
}
/** Grade real call/result/receipt evidence and persisted state, never model prose. */
export function gradeTaskTitle(input: TaskTitleEvidenceInput) {
const initial = record(input.initial), final = record(input.final), submitted = record(input.submitted);
const provisional = input.prompt.trim().replace(/\s+/g, " ").slice(0, 120);
const checks: Array<{ id: string; passed: boolean; detail: string }> = [];
const check = (id: string, passed: boolean, detail: string) => checks.push({ id, passed, detail });
const starts = new Map<string, { name: string; input: Record<string, unknown>; position: number; index: number }>();
const results = new Map<string, Record<string, unknown>>();
const completions = new Map<string, number>();
let position = 0;
input.events.forEach((event, index) => {
const payload = record(record(record(event.payload).prpEvent).payload);
const item = record(payload.item);
if (event.eventType === "tool.execution.started" && typeof payload.executionId === "string" && !starts.has(payload.executionId)) {
starts.set(payload.executionId, { name: String(payload.name ?? ""), input: {}, position: ++position, index });
}
if (event.eventType === "item.started" && item.type === "tool_use" && typeof item.id === "string" && typeof item.name === "string") {
const prior = starts.get(item.id);
starts.set(item.id, { name: item.name, input: record(item.input), position: prior?.position ?? ++position, index: prior?.index ?? index });
}
if (event.eventType === "item.completed" && item.type === "tool_result" && typeof item.id === "string"
&& (item.tool_use_id === undefined || item.tool_use_id === item.id)) results.set(item.id, record(item.result));
if (event.eventType === "tool.execution.completed" && payload.name === "set_task_title" && payload.status === "completed" && typeof payload.executionId === "string") {
completions.set(payload.executionId, index);
}
});
const calls = [...starts].filter(([, call]) => call.name === "set_task_title");
const titleWrites = input.activity.map(record).filter(row => row.action === "issue.updated"
&& row.entityType === "issue" && row.entityId === initial.id && typeof record(row.details).title === "string");
check("creation-request", submitted.description === input.prompt
&& (input.explicitTitle ? submitted.title === input.explicitTitle : !Object.hasOwn(submitted, "title")), "Browser submitted the full prompt with the requested title, or omitted the title entirely.");
check("provider-tool-evidence", starts.size > 0 && Boolean(input.runId), "Provider tool evidence from the initial run is present; narration alone is insufficient.");
check("initial-title", initial.title === (input.explicitTitle || provisional)
&& initial.titleNeedsGeneration === !Boolean(input.explicitTitle), "Creation response proves the initial title and server-owned generation marker before the agent runs.");
check("same-task-and-description", typeof initial.id === "string" && final.id === initial.id
&& initial.description === input.prompt && final.description === input.prompt
&& final.assigneeAgentId === input.agentId, "The original task, assignment, and complete prompt survive naming.");
check("generation-settled", final.titleNeedsGeneration === false, "The stored task no longer needs a generated title.");
if (input.explicitTitle) {
check("explicit-title-preserved", final.title === input.explicitTitle && titleWrites.length === 0
&& calls.every(([id]) => results.get(id)?.changed === false), "No title-changing audit or tool result may overwrite a supplied title, even temporarily.");
} else {
const title = typeof final.title === "string" ? final.title : "";
check("descriptive-title", title.trim() === title && title.length >= 8 && title.length <= 120
&& title !== provisional && !input.prompt.startsWith(title) && !title.includes("CHECKLIST-")
&& /(rotat|replac|renew|expir)/i.test(title) && /\b(api|keys?|credentials?)\b/i.test(title), "A concise title describes the key-rotation request instead of copying the prompt prefix or answer receipt.");
const terminalIndex = input.events.findIndex(event => event.eventType === "run.terminal");
const finishIndex = Math.min(
[...starts.values()].find(call => ["paperclip_finish", "finish_task"].includes(call.name))?.index ?? input.events.length,
terminalIndex < 0 ? input.events.length : terminalIndex,
);
const proven = calls.filter(([id, call]) => {
const result = results.get(id);
const completedAt = completions.get(id);
return result !== undefined && call.position <= TASK_TITLE_EARLY_CALL_LIMIT && call.input.onlyIfProvisional === true
&& typeof call.input.idempotencyKey === "string" && call.input.idempotencyKey.trim().length > 0
&& call.input.title === title && result.id === initial.id && result.title === title
&& result.changed === true && result.titleNeedsGeneration === false
&& completedAt !== undefined && completedAt > call.index && completedAt < finishIndex;
});
check("early-title-tool", proven.length > 0, `A correlated set_task_title call must succeed among the first ${TASK_TITLE_EARLY_CALL_LIMIT} tools of the initial run, before completion.`);
check("agent-title-audit", titleWrites.length === 1 && titleWrites[0]?.actorType === "agent"
&& titleWrites[0]?.actorId === input.agentId && titleWrites[0]?.runId === input.runId
&& record(titleWrites[0]?.details).title === title
&& record(record(titleWrites[0]?.details).previous).title === provisional, "The same assigned agent and initial run must own the single persisted title mutation.");
}
return {
schema: "paperclip.task-title-eval.v1", passed: checks.every(entry => entry.passed), checks,
initialTitle: initial.title, finalTitle: final.title,
titleCalls: calls.map(([id, call]) => ({ id, position: call.position, completed: completions.has(id), result: results.get(id) ?? null })),
titleWrites,
};
}