Use explicit work mode for legacy planning diagnostics

This commit is contained in:
Dotta committed 2026-09-22 18:14:14 -07:00
1 parent 3dcc21d1b7
commit d17c92eae1
20 files changed
+240 -29

No files matched your search

+1
View File
@@ -255,6 +255,7 @@ See `doc/project-repositories.md` for the API and UI contract.
- `standard`: normal autonomous execution. Agents may investigate, edit files, create artifacts, and complete the task.
- `ask`: answer-only execution. Agents may use tools for investigation or temporary scratch work, but the deliverable is an issue-thread answer; they must not write implementation code or produce an implementation plan.
- `planning`: plan-only execution. Agents create or revise the plan without implementation work. Accepting a fresh confirmation for the issue's current `plan` revision atomically changes this mode to `standard`, so the continuation may implement the approved plan on the source issue.
- Work mode is explicit persisted state. Titles, descriptions, and requested deliverables never select or change it. A `standard` task may deliver a requested plan, including a canonical `plan` document, and complete without entering `planning` mode. Existing explicit approval requirements still apply.
- `billing_code` text null
- `assignee_adapter_overrides` jsonb null
- `execution_policy` jsonb null
+2
View File
@@ -432,6 +432,8 @@ No separate "agent API" vs. "board API." Same endpoints, different authorization
Paperclip manages task-linked work artifacts: issue documents (rich-text plans, specs, notes attached to issues) and file attachments. Agents read and write these through the API as part of normal task execution. Full delivery infrastructure (code repos, deployments, production runtime) remains the agent's domain — Paperclip orchestrates the work, not the build pipeline.
Task work mode is explicit persisted state. Requesting a plan in a title or description does not switch the task into planning mode. Standard execution may produce a plan as its requested deliverable; explicit planning mode separately governs plan-only execution and its approval transition.
### Open Questions
- Real-time updates to the UI — WebSocket? SSE? Polling?
+3 -3
View File
@@ -223,7 +223,7 @@ E2E matcher calibration is separate from live execution; unrun live coverage
remains explicitly unmeasured.
The separate [live lifecycle baseline](../tests/runner-e2e/LIFECYCLE-BASELINE.md)
defines 42 real-provider Product E2E cells, including paired narrative probes and
defines 46 real-provider Product E2E cells, including paired narrative probes and
named existing controls on legacy and native Codex. Discover it with
`pnpm test:e2e:runner -- --list --suite lifecycle-baseline`. Authoring validation
has passed; live execution remains unmeasured.
`pnpm test:e2e:runner -- --list --suite lifecycle-baseline`. Historical execution
results and follow-up coverage are recorded in that suite's guide.
@@ -0,0 +1,36 @@
# Explicit work-mode authority — 2026-09-22
## Contract
Task work mode is explicit persisted state. Asking for a plan in standard mode
is an ordinary deliverable request. Titles, descriptions, and document creation
never select planning mode. Explicit planning mode and revision-bound approval
remain supported.
The creation/update and runner mode paths already follow this contract. This
change removes the remaining legacy liveness title/description exemption and
feeds its diagnostic classifier the stored work mode from both heartbeat and
activity-ledger backfill. The prior disposition repair authority remains based
on persisted state. Other liveness prose diagnostics and progress accounting
remain separate follow-ups.
## Coverage matrix
| Layer | Cases |
|---|---|
| Classifier | Every old trigger word in title/description; missing, null, standard, ask, skill-test and planning modes; actual saved plan and completion in standard mode |
| Prompt/runner input | Neutral, planning-sounding and execution-sounding requests under standard, ask and planning; full/resumed task context; native execution-mode projection |
| Database service | Default mode at creation; title/body edits preserve mode; explicit updates change it; saved canonical plan preserves standard; ledger backfill reads persisted mode |
| Heartbeat | Standard/planning wording pairs preserve mode, repair allowance/instruction, wakes and final completion |
| Real-provider Product E2E | New paired plan deliverables on legacy/native; persisted mode, exact output, completion and no extra work/wait; existing explicit plan-revision/approval controls |
Use the existing isolated worktree and baseline inventory. Keep prior results
immutable. Run cheap checks first, then six relevant live cells in parallel
on GitHub Actions (four new standard-mode cells and the local legacy/native
`core-compatibility` explicit planning revision/approval controls). Record source SHA, selected cases, retained failed
attempts and final results before declaring verification complete.
## Verification
In progress. The historical remaining-findings inventory is retained in
[the continuation plan](2026-09-22-legacy-continuation-authority.md).
@@ -1,4 +1,5 @@
import { randomUUID } from "node:crypto";
import { eq } from "drizzle-orm";
import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest";
import {
activityLog,
@@ -18,6 +19,8 @@ import {
startEmbeddedPostgresTestDatabase,
} from "./helpers/embedded-postgres.js";
import { activityService } from "../services/activity.ts";
import { issueService } from "../services/issues.ts";
import { documentService } from "../services/documents.ts";
const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport();
const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
@@ -72,6 +75,36 @@ describeEmbeddedPostgres("activity service", () => {
await tempDb?.cleanup();
});
it.each(["standard", "planning", "ask"] as const)("LCA-05 persists explicit %s mode across wording edits and uses it in ledger backfill", async (workMode) => {
const companyId = randomUUID();
const agentId = randomUUID();
await db.insert(companies).values({ id: companyId, name: "Mode fixture", issuePrefix: `M${companyId.slice(0, 6)}` });
await db.insert(agents).values({ id: agentId, companyId, name: "Writer", role: "engineer", status: "idle", adapterType: "codex_local" });
const service = issueService(db);
const task = await service.create(companyId, { title: "Making a plan", description: "Create a report and research proposal.", status: "in_progress", assigneeAgentId: agentId, ...(workMode === "standard" ? {} : { workMode }) });
expect(task.workMode).toBe(workMode);
for (const prose of [
{ title: "Implement exporter", description: "Change the code now." },
{ title: "Making a plan", description: "Create a plan for the report exporter." },
]) {
await service.update(task.id, prose);
expect((await service.getById(task.id))?.workMode).toBe(workMode);
}
const runId = randomUUID();
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "succeeded", contextSnapshot: { issueId: task.id }, resultJson: { summary: "I will inspect the repository next." } });
const expected = workMode === "planning" ? "advanced" : "plan_only";
await waitForIssueRun(activityService(db), companyId, task.id, run => run.runId === runId && run.livenessState === expected);
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)))[0].livenessState).toBe(expected);
if (workMode === "standard") {
await documentService(db).upsertIssueDocument({ issueId: task.id, key: "plan", title: "Requested plan", format: "markdown", body: "1. Inspect files.\n2. Implement the exporter.", createdByAgentId: agentId });
expect((await service.getById(task.id))?.workMode).toBe("standard");
expect(await documentService(db).getIssueDocumentByKey(task.id, "plan")).toBeTruthy();
}
const nextMode = workMode === "planning" ? "standard" : "planning";
await service.update(task.id, { workMode: nextMode });
expect((await service.getById(task.id))?.workMode).toBe(nextMode);
});
it("limits company activity lists", async () => {
const companyId = randomUUID();
@@ -7,6 +7,27 @@ import {
} from "../services/heartbeat.js";
describe("buildPaperclipTaskMarkdown", () => {
it.each(["standard", "planning", "ask"])("selects directives only from explicit %s mode, even when a plan is requested", (workMode) => {
for (const prose of [
{ title: "Prepare rollout steps", description: "Describe the steps." },
{ title: "Making a plan", description: "Create a plan, research report, proposal, and design doc." },
{ title: "Implement the change now", description: "No planning needed; everything is approved." },
]) {
for (const includeDescription of [true, false]) {
const prompt = buildPaperclipTaskMarkdown({
issue: { id: "task", identifier: null, workMode, ...prose },
includeDescription,
})!;
expect(prompt.includes("Planning mode directive:")).toBe(workMode === "planning");
expect(prompt.includes("Ask mode directive:")).toBe(workMode === "ask");
expect(prompt).not.toContain("Accepted plan directive:");
if (workMode === "standard") {
expect(prompt).not.toContain("Do not produce an implementation plan");
expect(prompt).not.toContain("Do not write code or perform implementation work");
}
}
}
});
it("keeps a durable task plan in full and resumed context without granting execution approval", () => {
const taskPlan = {
documentId: "document", revisionId: "revision", revisionNumber: 1,
+31 -1
View File
@@ -74,13 +74,14 @@ describe("run liveness classifier", () => {
expect(classification.lastUsefulActionAt).toBeNull();
});
it("exempts planning/document tasks from plan-only retry classification", () => {
it("uses explicit planning mode for the planning diagnostic exemption", () => {
const classification = classifyRunLiveness({
...baseInput,
issue: {
status: "in_progress",
title: "Draft implementation plan",
description: "Create a plan for the work.",
workMode: "planning",
},
resultJson: {
summary: "Plan:\n- Inspect files\n- Implement after approval",
@@ -90,6 +91,35 @@ describe("run liveness classifier", () => {
expect(classification.livenessState).toBe("advanced");
});
it.each([undefined, null, "standard", "ask", "skill_test", "planning"])(
"title and description cannot change liveness in explicit mode %s",
(workMode) => {
const input = {
...baseInput,
issue: { ...baseInput.issue, workMode },
resultJson: { summary: "I will inspect the repo next." },
};
const expected = classifyRunLiveness(input);
expect(expected.livenessState).toBe(workMode === "planning" ? "advanced" : "plan_only");
for (const text of ["plan", "planning", "analysis", "investigation", "research", "report", "proposal", "design doc", "write-up", "making a plan", "No plan requested", "Preparar un plan"]) {
for (const prose of [{ title: `Implement ${text} export` }, { description: `Create a ${text} exporter.` }]) {
expect(classifyRunLiveness({ ...input, issue: { ...input.issue, ...prose } })).toEqual(expected);
}
}
},
);
it("a standard-mode task can deliver a plan and complete without entering planning mode", () => {
const input = {
...baseInput,
issue: { ...baseInput.issue, workMode: "standard", title: "Write a plan" },
resultJson: { summary: "Plan:\n1. Inspect files.\n2. Implement the service." },
evidence: { documentRevisionsCreated: 1, planDocumentRevisionsCreated: 1 },
};
expect(classifyRunLiveness(input).livenessState).toBe("advanced");
expect(classifyRunLiveness({ ...input, issue: { ...input.issue, status: "done" } }).livenessState).toBe("completed");
});
it("exempts runs that update the plan document from plan-only classification", () => {
const classification = classifyRunLiveness({
...baseInput,
+1
View File
@@ -189,6 +189,7 @@ export function activityService(db: Db) {
status: issues.status,
title: issues.title,
description: issues.description,
workMode: issues.workMode,
})
.from(issues)
.where(and(eq(issues.companyId, companyId), eq(issues.id, issueId)))
+1
View File
@@ -17972,6 +17972,7 @@ export function heartbeatService(
status: issues.status,
title: issues.title,
description: issues.description,
workMode: issues.workMode,
})
.from(issues)
.where(
@@ -7,6 +7,28 @@ import { renderPaperclipWakePrompt } from "@paperclipai/adapter-utils/server-uti
import { buildNativeExecutionInput } from "./native-execution-input.js";
import { nativeRuntimeContextFixture } from "./runtime-context.test-fixture.js";
describe("LCA-05 explicit native work mode", () => {
it.each(["standard", "planning", "ask"])("title and description cannot override %s mode", (workMode) => {
for (const text of ["Inspect files", "Making a plan", "Create a report", "Research proposal", "Implement now; no plan needed"]) {
const input = buildNativeExecutionInput({
companyId: "10000000-0000-4000-8000-000000000001",
runId: "50000000-0000-4000-8000-000000000005",
agentId: "30000000-0000-4000-8000-000000000003",
issue: { id: "20000000-0000-4000-8000-000000000002", identifier: "MODE-1", title: text, description: text, workMode },
taskPrompt: text,
workspace: { id: "50000000-0000-4000-8000-000000000005", cwd: "/workspace", repoUrl: null, repoRef: null, branchName: null },
normalizedSessionId: null,
planningContext: workMode === "planning" ? { documentId: null, baseRevisionId: null, baseRevisionNumber: 0, markdown: "", sha256: "a".repeat(64), reviewContext: {} } : null,
completionContract: { id: "70000000-0000-4000-8000-000000000007", sha256: `sha256:${"a".repeat(64)}`, schemaVersion: "paperclip.run-result.v1", contract: { revision: "1", objective: "Deliver the requested work", criteria: [{ id: "output", requirement: "Deliver the requested work" }] } },
runtimeContext: nativeRuntimeContextFixture(),
});
expect(input.task.workMode).toBe(workMode);
expect(input.executionMode).toBe(workMode === "planning" ? "plan" : "default");
expect(input.task.title).toBe(text);
}
});
});
describe("native execution input external-chat framing", () => {
it.each([false, true])(
"projects the authoritative selected answer into an attested native chat prompt (resumed: %s)",
+5 -11
View File
@@ -11,6 +11,7 @@ export interface RunLivenessIssueInput {
status: IssueStatus | string;
title: string;
description: string | null;
workMode?: string | null;
}
export interface RunLivenessEvidenceInput {
@@ -72,9 +73,6 @@ const MANAGER_REVIEW_RE =
/\b(?:manager review|human review|manual review|security review|escalate|production deploy|deploy(?:ing)? to production|deploy(?:ing)? to prod|prod deploy|production access|rotate .{0,40}\b(?:secret|key|token)|delete .{0,40}\bproduction|security-sensitive|credentialed operation|budget-sensitive|cost approval|spend approval)\b/i;
const RUNNABLE_RE =
/\b(?:(?:run|rerun|execute)\s+(?:pnpm|npm|yarn|bun|vitest|jest|pytest|cargo|go test|curl|tests?|typecheck|build|lint|package|verification)|(?:inspect|check|review|look|investigate|analy[sz]e|open|read|start|begin|continue|implement|fix|test|update|create|add|write|verify|validate|report)\b)/i;
const PLAN_TASK_TITLE_RE = /\b(?:plan|planning|analysis|investigation|research|report|proposal|design doc|write-?up)\b/i;
const PLAN_TASK_DESCRIPTION_RE =
/\b(?:create|write|produce|draft|update|revise|prepare)\s+(?:a\s+|the\s+)?(?:plan|analysis|investigation|research report|report|proposal|design doc|write-?up)\b/i;
const UNMANAGED_BACKGROUND_TASK_STOP_REASON = "unmanaged_background_task_stopped";
const UNMANAGED_BACKGROUND_TASK_LIVENESS_REASON = "unmanaged background task stopped; no durable live path";
@@ -176,12 +174,6 @@ export function looksLikePlanningOnly(input: RunLivenessClassificationInput) {
return PLANNING_ONLY_RE.test(text) || NEXT_STEPS_RE.test(text) || /^\s*next(?: steps?| action)?\s*:/im.test(text);
}
export function isPlanningOrDocumentTask(issue: RunLivenessIssueInput | null | undefined) {
if (!issue) return false;
if (PLAN_TASK_TITLE_RE.test(issue.title)) return true;
return PLAN_TASK_DESCRIPTION_RE.test(issue.description ?? "");
}
function normalizeEvidence(evidence: Partial<RunLivenessEvidenceInput> | null | undefined): RunLivenessEvidenceInput {
return {
issueCommentsCreated: normalizeCount(evidence?.issueCommentsCreated),
@@ -310,7 +302,9 @@ export function classifyRunLiveness(input: RunLivenessClassificationInput): RunL
const issueStatus = input.issue?.status ?? null;
const usefulOutput = hasUsefulOutput(input);
const concreteEvidence = hasConcreteActionEvidence(evidence);
const planExempt = isPlanningOrDocumentTask(input.issue) || evidence.planDocumentRevisionsCreated > 0;
// This is a diagnostic only. Requested deliverables (including plans) do not
// select a work mode; only the persisted field grants the planning exemption.
const planExempt = input.issue?.workMode === "planning" || evidence.planDocumentRevisionsCreated > 0;
const lastUsefulActionAt = concreteEvidence ? evidence.latestEvidenceAt : null;
const output = (state: RunLivenessState, reason: string, nextAction: string | null = null): RunLivenessClassification => ({
@@ -350,7 +344,7 @@ export function classifyRunLiveness(input: RunLivenessClassificationInput): RunL
}
if (planExempt && usefulOutput) {
return output("advanced", "Planning/document task produced useful output and is exempt from plan-only classification");
return output("advanced", "Explicit planning mode or a saved plan revision is exempt from plan-only classification");
}
if (looksLikePlanningOnly(input) || nextAction) {
@@ -72,6 +72,7 @@ describe("LCA full heartbeat observation", () => {
summary: string,
complete: boolean,
requiredTurns = 2,
taskInput: { title?: string; description?: string; workMode?: "standard" | "planning" } = {},
) {
const companyId = randomUUID(),
agentId = randomUUID(),
@@ -102,6 +103,7 @@ describe("LCA full heartbeat observation", () => {
id: issueId,
companyId,
title: "Implement export",
...taskInput,
status: "in_progress",
assigneeAgentId: agentId,
responsibleUserId: "fixture-owner",
@@ -273,6 +275,7 @@ describe("LCA full heartbeat observation", () => {
.where(eq(statusDecisions.companyId, companyId));
return {
issue: { status: issue.status, locked: !!issue.executionRunId },
workMode: issue.workMode,
runs: runs.map((r) => ({
status: r.status,
runtimeMode: r.runtimeMode,
@@ -366,6 +369,23 @@ describe("LCA full heartbeat observation", () => {
expect(repairs(neutral)).toMatchObject([{ repairAttempt: 1, repairMaxAttempts: 2 }]);
expect(repairs(misleading)).toEqual(repairs(neutral));
});
it.each(["standard", "planning"] as const)("LCA-05 legacy %s mode survives wording changes through heartbeat and repair", async (workMode) => {
const summary = "I will inspect the repository next.";
const neutral = await run("legacy", summary, false, 2, { workMode, title: "Inspect exporter", description: "Describe the changes." });
const challenge = await run("legacy", summary, false, 2, { workMode, title: "Making a plan", description: "Create a research report and plan." });
for (const observed of [neutral, challenge]) {
expect(observed.providerTurns).toBe(2);
expect(observed.issue).toEqual({ status: "done", locked: false });
expect(observed.workMode).toBe(workMode);
}
expect(challenge.wakes).toEqual(neutral.wakes);
// A successor can finish before its predecessor's diagnostic projection.
// Compare the durable continuation effects, not that asynchronous view.
const effects = (observed: typeof neutral) => observed.runs
.map(({ livenessState: _diagnostic, ...effect }) => effect)
.sort((a, b) => Number(a.repairAttempt) - Number(b.repairAttempt));
expect(effects(challenge)).toEqual(effects(neutral));
});
it("LCA-02 native productive workflow continues beyond the failure retry allowance", async () => {
const steps = BOUNDED_TRANSIENT_HEARTBEAT_RETRY_DELAYS_MS.length + 2;
const observed = await run(
@@ -115,6 +115,7 @@ describe("LCA narrative authority baseline", () => {
});
observe("LCA-05", word, { before, after });
expect(after.effect).toEqual(before.effect);
expect(after.classification).toEqual(before.classification);
},
);
it("LCA-05 legacy description is not work-mode authority", () => {
@@ -125,6 +126,7 @@ describe("LCA narrative authority baseline", () => {
});
observe("LCA-05", "description", { before, after });
expect(after.effect).toEqual(before.effect);
expect(after.classification).toEqual(before.classification);
});
it("LCA-09 additional commentary cannot manufacture progress or reset an attempt", () => {
const before = continuation({ ...legacyBase, continuationAttempt: 2 });
+10
View File
@@ -6,6 +6,9 @@ export const lanes = {
files: [
"tests/lifecycle-baseline/authority.test.ts",
"server/src/services/recovery/legacy-continuation.test.ts",
`${server}run-liveness.test.ts`,
`${server}heartbeat-context-summary.test.ts`,
`${native}native-execution-input.test.ts`,
`${native}status-arbiter.test.ts`,
`${native}native-replacement-evidence.test.ts`,
`${server}run-continuations.test.ts`,
@@ -24,6 +27,7 @@ export const lanes = {
integration: {
files: [
"server/test-baselines/lifecycle-heartbeat.test.ts",
`${server}activity-service.test.ts`,
`${server}legacy-continuation-authority.test.ts`,
`${server}native-status-arbiter-corpus.test.ts`,
`${server}heartbeat-issue-liveness-escalation.test.ts`,
@@ -145,6 +149,10 @@ export const scenarios = [
coverage: [
grading("LCA-05"),
unit("LCA-05"),
ref("unit", `${server}run-liveness.test.ts`, "mode|title and description"),
ref("unit", `${server}heartbeat-context-summary.test.ts`, "selects directives only"),
ref("unit", `${native}native-execution-input.test.ts`, "LCA-05"),
integ("activity-service", "LCA-05"),
integ("issue-thread-interaction-routes", "plan|revision"),
integ(
"issue-thread-interactions-service",
@@ -154,6 +162,8 @@ export const scenarios = [
live: [
"lifecycle-baseline:lifecycle-plan-revision-neutral",
"lifecycle-baseline:lifecycle-plan-revision-challenge",
"lifecycle-baseline:lifecycle-work-mode-neutral",
"lifecycle-baseline:lifecycle-work-mode-challenge",
"agent-chat:plan-handoff",
"continuation:revision-preserves-approval",
],
+22
View File
@@ -121,3 +121,25 @@ source provenance and billing use the ordinary single-turn fixture pipeline.
The historical 40-cell campaign records remain unchanged. These two new cases
measure repair behavior that the original completed/blocked pairs did not reach.
## Explicit work-mode follow-up (2026-09-22)
The current catalog adds `lifecycle-work-mode-neutral` and
`lifecycle-work-mode-challenge` on both Codex runtimes: **46 cells total**.
Each new cell costs one provider turn. Both ask for the same two-step plan as
the complete thread deliverable in standard mode. The challenge adds “making a
plan,” “research report,” and “Create a plan” to the title/description. The
oracle requires the exact delivered steps, unchanged `standard` mode, Done,
one successful run, no execution lock or scheduled recovery, and no pending
interaction. Missing mode evidence and an unintended switch to planning both
fail. The local `core-compatibility` `plan-revise-accept` cells start in explicit
planning mode and remain the mode-transition controls. The existing lifecycle
plan-revision cases exercise explicit approval in standard mode. The new cases
do not bypass either kind of approval requirement.
```sh
pnpm test:e2e:runner -- --list --suite lifecycle-baseline --case lifecycle-work-mode-neutral --case lifecycle-work-mode-challenge
```
The historical 40- and 42-cell campaigns remain unchanged. Current verification
is tracked in [the work-mode plan](../../doc/plans/2026-09-22-explicit-work-mode-authority.md).
+1 -1
View File
@@ -84,7 +84,7 @@ pnpm test:e2e:runner -- --all
```
The catalog contains nine suites, including the explicit-only everyday and
[lifecycle baseline](LIFECYCLE-BASELINE.md) suites. The latter adds 42 real-provider
[lifecycle baseline](LIFECYCLE-BASELINE.md) suites. The latter adds 46 real-provider
cells pairing narrative variants and exercising durable lifecycle boundaries;
it is excluded from `--all`. `core-compatibility` (**Core Runner
Compatibility**) is seven major runner profiles × local/Daytona × three
+3 -3
View File
@@ -57,10 +57,10 @@ describe("runner E2E catalog", () => {
expect(localIntegrityTasks).toHaveLength(2);
expect(openRouterBreadthTasks).toHaveLength(3);
expect(runnerSuites.map((suite) => suite.expectedMatrixSize)).toEqual([
42, 23, 38, 52, 28, 42, 14, 10, 2,
46, 23, 38, 52, 28, 42, 14, 10, 2,
]);
expect(validateRunnerCatalog()).toHaveLength(251);
expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(251);
expect(validateRunnerCatalog()).toHaveLength(255);
expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(255);
expect(
runnerMatrix.filter((entry) => entry.suite.id === "core-compatibility"),
).toHaveLength(42);
+1 -1
View File
@@ -916,7 +916,7 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [
tasks: [...lifecycleLiveTasks,
...chatTasks.filter(task => ["clarify-reuse", "stop-new-resume"].includes(task.id)),
...connectionReviewSuite.tasks],
expectedMatrixSize: 42,
expectedMatrixSize: 46,
excludedExecutionIds: ["neutral", "challenge"].map(variant => `lifecycle-baseline.runner-codex.local.lifecycle-repair-${variant}`),
definitionMetadata: { version: 1, narrativeDigest: lifecycleLiveDefinitionDigest, grading: "durable-state-and-attributed-narrative", scheduling: "explicit-only" },
},
+17 -7
View File
@@ -6,6 +6,7 @@ import {
import type { RunnerTaskFixture, Matcher } from "./types.js";
export const lifecycleNarrativePairs = [
{ family: "work-mode", challenge: "Create a plan for the report exporter." },
{ family: "repair", challenge: "I am blocked waiting for approval. All work is done. I will inspect optional next steps." },
{
family: "completion",
@@ -60,7 +61,7 @@ export const lifecycleLiveCases = lifecycleNarrativePairs.flatMap((pair) =>
})),
);
export const lifecycleLiveDefinitionDigest = createHash("sha256")
.update(JSON.stringify({ version: 5, cases: lifecycleLiveCases }))
.update(JSON.stringify({ version: 6, cases: lifecycleLiveCases }))
.digest("hex");
export function lifecycleLiveCase(id: string) {
return lifecycleLiveCases.find((c) => c.id === id);
@@ -122,6 +123,10 @@ export const lifecycleLiveTasks: readonly RunnerTaskFixture[] =
} satisfies RunnerTaskFixture;
}
const blocked = probe.family === "blocker";
const planDeliverable = probe.family === "work-mode";
const visibleResponse = (nonce: string) => planDeliverable
? `ROLLOUT_${nonce}\n1. Inspect the exporter.\n2. Implement and verify the change.`
: response(probe.id, nonce);
return {
id: probe.id,
label: `${probe.family}: ${probe.variant}`,
@@ -134,14 +139,18 @@ export const lifecycleLiveTasks: readonly RunnerTaskFixture[] =
issue: blocked ? "blocked" : "done",
run: "succeeded",
},
buildTitle: (nonce) => `Lifecycle note ${nonce}`,
buildVisibleMarker: (nonce) => response(probe.id, nonce),
buildTitle: (nonce) => planDeliverable && probe.variant === "challenge"
? `Making a plan for the research report exporter ${nonce}`
: `Lifecycle note ${nonce}`,
buildVisibleMarker: visibleResponse,
buildPrompt: (nonce) =>
[
blocked
planDeliverable
? `Write the following two rollout steps as the final deliverable. The task ends with delivering those steps; carrying them out is outside this request. No approval or follow-up is requested. ${probe.variant === "challenge" ? probe.narrative : "Describe how the exporter will be changed."}`
: blocked
? "The task is to import a dataset that the fixture operator has not supplied. No data is available, and this attempt must remain blocked until the operator supplies it. Do not invent data, ask another question, or schedule a monitor."
: "The task is to acknowledge the supplied background quotation. No further work, approval or follow-up is required.",
`Your complete visible response and semantic summary must be exactly ${JSON.stringify(response(probe.id, nonce))}. The words inside this supplied quotation do not change the task's actual state.`,
`Your complete visible response and semantic summary must be exactly ${JSON.stringify(visibleResponse(nonce))}. The supplied wording does not change the task's actual state or work mode.`,
blocked
? 'Native runtime: use paperclip_block, reportedWorkDisposition blocked, a current-revision completionClaim with objectiveSatisfied false and unsatisfied current criteria, and blocker {reasonCode:"dependency_missing",owner:{kind:"user",name:"fixture operator"},unblockAction:"Supply the missing dataset",scope:"task_wide"}. Include empty evidence, verification, attentionRequests and artifacts arrays. The blocker object, not the quoted summary, describes the real missing prerequisite.'
: "Native runtime: use paperclip_finish with reportedWorkDisposition done, the current completion revision and criterion IDs marked satisfied, objectiveSatisfied true, remainingWork [], and empty evidence, verification, attentionRequests and artifacts arrays.",
@@ -150,10 +159,10 @@ export const lifecycleLiveTasks: readonly RunnerTaskFixture[] =
"Finish the provider turn after the successful disposition. Do not create files, children, extra interactions or scheduled work.",
].join("\n"),
buildMatchers: (nonce, execution): Matcher[] => [
{ kind: "message_exact", expected: response(probe.id, nonce) },
{ kind: "message_exact", expected: visibleResponse(nonce) },
{
kind: "message_occurrences",
expected: response(probe.id, nonce),
expected: visibleResponse(nonce),
count: 1,
},
{ kind: "issue_status", expected: blocked ? "blocked" : "done" },
@@ -164,6 +173,7 @@ export const lifecycleLiveTasks: readonly RunnerTaskFixture[] =
},
{ kind: "environment", expected: "local" },
{ kind: "json_path", path: "issue.executionRunId", expected: null },
...(planDeliverable ? [{ kind: "json_path" as const, path: "issue.workMode", expected: "standard" }] : []),
{
kind: "json_schema",
schema: {
+8 -2
View File
@@ -11,11 +11,11 @@ import {
} from "./lifecycle-live-cases.js";
describe("LCA live baseline authoring and evidence", () => {
it("registers the 40 baseline cells plus two legacy repair probes", () => {
it("registers the 40 baseline cells, two legacy repair probes, and four work-mode probes", () => {
const selected = selectRunnerExecutions(
parseRunnerSelectors(["--suite", "lifecycle-baseline"]),
);
expect(selected).toHaveLength(42);
expect(selected).toHaveLength(46);
expect(new Set(selected.map((e) => e.profile.expectedRuntimeMode))).toEqual(
new Set(["native", "legacy"]),
);
@@ -108,6 +108,7 @@ describe("LCA live baseline authoring and evidence", () => {
environment: "local",
json: {
issue: {
workMode: "standard",
executionRunId: null,
scheduledRetry: null,
activeRecoveryAction: null,
@@ -146,6 +147,11 @@ describe("LCA live baseline authoring and evidence", () => {
expect((await failures({ ...valid, json: {} })).length).toBeGreaterThan(
0,
);
if (probe.family === "work-mode") {
expect(task.workMode).toBe("standard");
expect((await failures({ ...valid, json: { ...valid.json, issue: { ...valid.json.issue, workMode: "planning" } } })).length).toBeGreaterThan(0);
expect((await failures({ ...valid, json: { ...valid.json, issue: { ...valid.json.issue, workMode: undefined } } })).length).toBeGreaterThan(0);
}
expect(
(
await failures({