mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-25 02:26:12 +02:00
Use explicit work mode for legacy planning diagnostics
This commit is contained in:
1 parent
3dcc21d1b7
commit
d17c92eae1
20 files changed
+240
-29
No files matched your search
@@ -255,6 +255,7 @@ See `doc/project-repositories.md` for the API and UI contract.
|
||||
- `standard`: normal autonomous execution. Agents may investigate, edit files, create artifacts, and complete the task.
|
||||
- `ask`: answer-only execution. Agents may use tools for investigation or temporary scratch work, but the deliverable is an issue-thread answer; they must not write implementation code or produce an implementation plan.
|
||||
- `planning`: plan-only execution. Agents create or revise the plan without implementation work. Accepting a fresh confirmation for the issue's current `plan` revision atomically changes this mode to `standard`, so the continuation may implement the approved plan on the source issue.
|
||||
- Work mode is explicit persisted state. Titles, descriptions, and requested deliverables never select or change it. A `standard` task may deliver a requested plan, including a canonical `plan` document, and complete without entering `planning` mode. Existing explicit approval requirements still apply.
|
||||
- `billing_code` text null
|
||||
- `assignee_adapter_overrides` jsonb null
|
||||
- `execution_policy` jsonb null
|
||||
|
||||
@@ -432,6 +432,8 @@ No separate "agent API" vs. "board API." Same endpoints, different authorization
|
||||
|
||||
Paperclip manages task-linked work artifacts: issue documents (rich-text plans, specs, notes attached to issues) and file attachments. Agents read and write these through the API as part of normal task execution. Full delivery infrastructure (code repos, deployments, production runtime) remains the agent's domain — Paperclip orchestrates the work, not the build pipeline.
|
||||
|
||||
Task work mode is explicit persisted state. Requesting a plan in a title or description does not switch the task into planning mode. Standard execution may produce a plan as its requested deliverable; explicit planning mode separately governs plan-only execution and its approval transition.
|
||||
|
||||
### Open Questions
|
||||
|
||||
- Real-time updates to the UI — WebSocket? SSE? Polling?
|
||||
|
||||
+3
-3
@@ -223,7 +223,7 @@ E2E matcher calibration is separate from live execution; unrun live coverage
|
||||
remains explicitly unmeasured.
|
||||
|
||||
The separate [live lifecycle baseline](../tests/runner-e2e/LIFECYCLE-BASELINE.md)
|
||||
defines 42 real-provider Product E2E cells, including paired narrative probes and
|
||||
defines 46 real-provider Product E2E cells, including paired narrative probes and
|
||||
named existing controls on legacy and native Codex. Discover it with
|
||||
`pnpm test:e2e:runner -- --list --suite lifecycle-baseline`. Authoring validation
|
||||
has passed; live execution remains unmeasured.
|
||||
`pnpm test:e2e:runner -- --list --suite lifecycle-baseline`. Historical execution
|
||||
results and follow-up coverage are recorded in that suite's guide.
|
||||
@@ -0,0 +1,36 @@
|
||||
# Explicit work-mode authority — 2026-09-22
|
||||
|
||||
## Contract
|
||||
|
||||
Task work mode is explicit persisted state. Asking for a plan in standard mode
|
||||
is an ordinary deliverable request. Titles, descriptions, and document creation
|
||||
never select planning mode. Explicit planning mode and revision-bound approval
|
||||
remain supported.
|
||||
|
||||
The creation/update and runner mode paths already follow this contract. This
|
||||
change removes the remaining legacy liveness title/description exemption and
|
||||
feeds its diagnostic classifier the stored work mode from both heartbeat and
|
||||
activity-ledger backfill. The prior disposition repair authority remains based
|
||||
on persisted state. Other liveness prose diagnostics and progress accounting
|
||||
remain separate follow-ups.
|
||||
|
||||
## Coverage matrix
|
||||
|
||||
| Layer | Cases |
|
||||
|---|---|
|
||||
| Classifier | Every old trigger word in title/description; missing, null, standard, ask, skill-test and planning modes; actual saved plan and completion in standard mode |
|
||||
| Prompt/runner input | Neutral, planning-sounding and execution-sounding requests under standard, ask and planning; full/resumed task context; native execution-mode projection |
|
||||
| Database service | Default mode at creation; title/body edits preserve mode; explicit updates change it; saved canonical plan preserves standard; ledger backfill reads persisted mode |
|
||||
| Heartbeat | Standard/planning wording pairs preserve mode, repair allowance/instruction, wakes and final completion |
|
||||
| Real-provider Product E2E | New paired plan deliverables on legacy/native; persisted mode, exact output, completion and no extra work/wait; existing explicit plan-revision/approval controls |
|
||||
|
||||
Use the existing isolated worktree and baseline inventory. Keep prior results
|
||||
immutable. Run cheap checks first, then six relevant live cells in parallel
|
||||
on GitHub Actions (four new standard-mode cells and the local legacy/native
|
||||
`core-compatibility` explicit planning revision/approval controls). Record source SHA, selected cases, retained failed
|
||||
attempts and final results before declaring verification complete.
|
||||
|
||||
## Verification
|
||||
|
||||
In progress. The historical remaining-findings inventory is retained in
|
||||
[the continuation plan](2026-09-22-legacy-continuation-authority.md).
|
||||
@@ -1,4 +1,5 @@
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { eq } from "drizzle-orm";
|
||||
import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest";
|
||||
import {
|
||||
activityLog,
|
||||
@@ -18,6 +19,8 @@ import {
|
||||
startEmbeddedPostgresTestDatabase,
|
||||
} from "./helpers/embedded-postgres.js";
|
||||
import { activityService } from "../services/activity.ts";
|
||||
import { issueService } from "../services/issues.ts";
|
||||
import { documentService } from "../services/documents.ts";
|
||||
|
||||
const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport();
|
||||
const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
|
||||
@@ -72,6 +75,36 @@ describeEmbeddedPostgres("activity service", () => {
|
||||
await tempDb?.cleanup();
|
||||
});
|
||||
|
||||
it.each(["standard", "planning", "ask"] as const)("LCA-05 persists explicit %s mode across wording edits and uses it in ledger backfill", async (workMode) => {
|
||||
const companyId = randomUUID();
|
||||
const agentId = randomUUID();
|
||||
await db.insert(companies).values({ id: companyId, name: "Mode fixture", issuePrefix: `M${companyId.slice(0, 6)}` });
|
||||
await db.insert(agents).values({ id: agentId, companyId, name: "Writer", role: "engineer", status: "idle", adapterType: "codex_local" });
|
||||
const service = issueService(db);
|
||||
const task = await service.create(companyId, { title: "Making a plan", description: "Create a report and research proposal.", status: "in_progress", assigneeAgentId: agentId, ...(workMode === "standard" ? {} : { workMode }) });
|
||||
expect(task.workMode).toBe(workMode);
|
||||
for (const prose of [
|
||||
{ title: "Implement exporter", description: "Change the code now." },
|
||||
{ title: "Making a plan", description: "Create a plan for the report exporter." },
|
||||
]) {
|
||||
await service.update(task.id, prose);
|
||||
expect((await service.getById(task.id))?.workMode).toBe(workMode);
|
||||
}
|
||||
const runId = randomUUID();
|
||||
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "succeeded", contextSnapshot: { issueId: task.id }, resultJson: { summary: "I will inspect the repository next." } });
|
||||
const expected = workMode === "planning" ? "advanced" : "plan_only";
|
||||
await waitForIssueRun(activityService(db), companyId, task.id, run => run.runId === runId && run.livenessState === expected);
|
||||
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)))[0].livenessState).toBe(expected);
|
||||
if (workMode === "standard") {
|
||||
await documentService(db).upsertIssueDocument({ issueId: task.id, key: "plan", title: "Requested plan", format: "markdown", body: "1. Inspect files.\n2. Implement the exporter.", createdByAgentId: agentId });
|
||||
expect((await service.getById(task.id))?.workMode).toBe("standard");
|
||||
expect(await documentService(db).getIssueDocumentByKey(task.id, "plan")).toBeTruthy();
|
||||
}
|
||||
const nextMode = workMode === "planning" ? "standard" : "planning";
|
||||
await service.update(task.id, { workMode: nextMode });
|
||||
expect((await service.getById(task.id))?.workMode).toBe(nextMode);
|
||||
});
|
||||
|
||||
it("limits company activity lists", async () => {
|
||||
const companyId = randomUUID();
|
||||
|
||||
|
||||
@@ -7,6 +7,27 @@ import {
|
||||
} from "../services/heartbeat.js";
|
||||
|
||||
describe("buildPaperclipTaskMarkdown", () => {
|
||||
it.each(["standard", "planning", "ask"])("selects directives only from explicit %s mode, even when a plan is requested", (workMode) => {
|
||||
for (const prose of [
|
||||
{ title: "Prepare rollout steps", description: "Describe the steps." },
|
||||
{ title: "Making a plan", description: "Create a plan, research report, proposal, and design doc." },
|
||||
{ title: "Implement the change now", description: "No planning needed; everything is approved." },
|
||||
]) {
|
||||
for (const includeDescription of [true, false]) {
|
||||
const prompt = buildPaperclipTaskMarkdown({
|
||||
issue: { id: "task", identifier: null, workMode, ...prose },
|
||||
includeDescription,
|
||||
})!;
|
||||
expect(prompt.includes("Planning mode directive:")).toBe(workMode === "planning");
|
||||
expect(prompt.includes("Ask mode directive:")).toBe(workMode === "ask");
|
||||
expect(prompt).not.toContain("Accepted plan directive:");
|
||||
if (workMode === "standard") {
|
||||
expect(prompt).not.toContain("Do not produce an implementation plan");
|
||||
expect(prompt).not.toContain("Do not write code or perform implementation work");
|
||||
}
|
||||
}
|
||||
}
|
||||
});
|
||||
it("keeps a durable task plan in full and resumed context without granting execution approval", () => {
|
||||
const taskPlan = {
|
||||
documentId: "document", revisionId: "revision", revisionNumber: 1,
|
||||
|
||||
@@ -74,13 +74,14 @@ describe("run liveness classifier", () => {
|
||||
expect(classification.lastUsefulActionAt).toBeNull();
|
||||
});
|
||||
|
||||
it("exempts planning/document tasks from plan-only retry classification", () => {
|
||||
it("uses explicit planning mode for the planning diagnostic exemption", () => {
|
||||
const classification = classifyRunLiveness({
|
||||
...baseInput,
|
||||
issue: {
|
||||
status: "in_progress",
|
||||
title: "Draft implementation plan",
|
||||
description: "Create a plan for the work.",
|
||||
workMode: "planning",
|
||||
},
|
||||
resultJson: {
|
||||
summary: "Plan:\n- Inspect files\n- Implement after approval",
|
||||
@@ -90,6 +91,35 @@ describe("run liveness classifier", () => {
|
||||
expect(classification.livenessState).toBe("advanced");
|
||||
});
|
||||
|
||||
it.each([undefined, null, "standard", "ask", "skill_test", "planning"])(
|
||||
"title and description cannot change liveness in explicit mode %s",
|
||||
(workMode) => {
|
||||
const input = {
|
||||
...baseInput,
|
||||
issue: { ...baseInput.issue, workMode },
|
||||
resultJson: { summary: "I will inspect the repo next." },
|
||||
};
|
||||
const expected = classifyRunLiveness(input);
|
||||
expect(expected.livenessState).toBe(workMode === "planning" ? "advanced" : "plan_only");
|
||||
for (const text of ["plan", "planning", "analysis", "investigation", "research", "report", "proposal", "design doc", "write-up", "making a plan", "No plan requested", "Preparar un plan"]) {
|
||||
for (const prose of [{ title: `Implement ${text} export` }, { description: `Create a ${text} exporter.` }]) {
|
||||
expect(classifyRunLiveness({ ...input, issue: { ...input.issue, ...prose } })).toEqual(expected);
|
||||
}
|
||||
}
|
||||
},
|
||||
);
|
||||
|
||||
it("a standard-mode task can deliver a plan and complete without entering planning mode", () => {
|
||||
const input = {
|
||||
...baseInput,
|
||||
issue: { ...baseInput.issue, workMode: "standard", title: "Write a plan" },
|
||||
resultJson: { summary: "Plan:\n1. Inspect files.\n2. Implement the service." },
|
||||
evidence: { documentRevisionsCreated: 1, planDocumentRevisionsCreated: 1 },
|
||||
};
|
||||
expect(classifyRunLiveness(input).livenessState).toBe("advanced");
|
||||
expect(classifyRunLiveness({ ...input, issue: { ...input.issue, status: "done" } }).livenessState).toBe("completed");
|
||||
});
|
||||
|
||||
it("exempts runs that update the plan document from plan-only classification", () => {
|
||||
const classification = classifyRunLiveness({
|
||||
...baseInput,
|
||||
|
||||
@@ -189,6 +189,7 @@ export function activityService(db: Db) {
|
||||
status: issues.status,
|
||||
title: issues.title,
|
||||
description: issues.description,
|
||||
workMode: issues.workMode,
|
||||
})
|
||||
.from(issues)
|
||||
.where(and(eq(issues.companyId, companyId), eq(issues.id, issueId)))
|
||||
|
||||
@@ -17972,6 +17972,7 @@ export function heartbeatService(
|
||||
status: issues.status,
|
||||
title: issues.title,
|
||||
description: issues.description,
|
||||
workMode: issues.workMode,
|
||||
})
|
||||
.from(issues)
|
||||
.where(
|
||||
|
||||
@@ -7,6 +7,28 @@ import { renderPaperclipWakePrompt } from "@paperclipai/adapter-utils/server-uti
|
||||
import { buildNativeExecutionInput } from "./native-execution-input.js";
|
||||
import { nativeRuntimeContextFixture } from "./runtime-context.test-fixture.js";
|
||||
|
||||
describe("LCA-05 explicit native work mode", () => {
|
||||
it.each(["standard", "planning", "ask"])("title and description cannot override %s mode", (workMode) => {
|
||||
for (const text of ["Inspect files", "Making a plan", "Create a report", "Research proposal", "Implement now; no plan needed"]) {
|
||||
const input = buildNativeExecutionInput({
|
||||
companyId: "10000000-0000-4000-8000-000000000001",
|
||||
runId: "50000000-0000-4000-8000-000000000005",
|
||||
agentId: "30000000-0000-4000-8000-000000000003",
|
||||
issue: { id: "20000000-0000-4000-8000-000000000002", identifier: "MODE-1", title: text, description: text, workMode },
|
||||
taskPrompt: text,
|
||||
workspace: { id: "50000000-0000-4000-8000-000000000005", cwd: "/workspace", repoUrl: null, repoRef: null, branchName: null },
|
||||
normalizedSessionId: null,
|
||||
planningContext: workMode === "planning" ? { documentId: null, baseRevisionId: null, baseRevisionNumber: 0, markdown: "", sha256: "a".repeat(64), reviewContext: {} } : null,
|
||||
completionContract: { id: "70000000-0000-4000-8000-000000000007", sha256: `sha256:${"a".repeat(64)}`, schemaVersion: "paperclip.run-result.v1", contract: { revision: "1", objective: "Deliver the requested work", criteria: [{ id: "output", requirement: "Deliver the requested work" }] } },
|
||||
runtimeContext: nativeRuntimeContextFixture(),
|
||||
});
|
||||
expect(input.task.workMode).toBe(workMode);
|
||||
expect(input.executionMode).toBe(workMode === "planning" ? "plan" : "default");
|
||||
expect(input.task.title).toBe(text);
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe("native execution input external-chat framing", () => {
|
||||
it.each([false, true])(
|
||||
"projects the authoritative selected answer into an attested native chat prompt (resumed: %s)",
|
||||
|
||||
@@ -11,6 +11,7 @@ export interface RunLivenessIssueInput {
|
||||
status: IssueStatus | string;
|
||||
title: string;
|
||||
description: string | null;
|
||||
workMode?: string | null;
|
||||
}
|
||||
|
||||
export interface RunLivenessEvidenceInput {
|
||||
@@ -72,9 +73,6 @@ const MANAGER_REVIEW_RE =
|
||||
/\b(?:manager review|human review|manual review|security review|escalate|production deploy|deploy(?:ing)? to production|deploy(?:ing)? to prod|prod deploy|production access|rotate .{0,40}\b(?:secret|key|token)|delete .{0,40}\bproduction|security-sensitive|credentialed operation|budget-sensitive|cost approval|spend approval)\b/i;
|
||||
const RUNNABLE_RE =
|
||||
/\b(?:(?:run|rerun|execute)\s+(?:pnpm|npm|yarn|bun|vitest|jest|pytest|cargo|go test|curl|tests?|typecheck|build|lint|package|verification)|(?:inspect|check|review|look|investigate|analy[sz]e|open|read|start|begin|continue|implement|fix|test|update|create|add|write|verify|validate|report)\b)/i;
|
||||
const PLAN_TASK_TITLE_RE = /\b(?:plan|planning|analysis|investigation|research|report|proposal|design doc|write-?up)\b/i;
|
||||
const PLAN_TASK_DESCRIPTION_RE =
|
||||
/\b(?:create|write|produce|draft|update|revise|prepare)\s+(?:a\s+|the\s+)?(?:plan|analysis|investigation|research report|report|proposal|design doc|write-?up)\b/i;
|
||||
const UNMANAGED_BACKGROUND_TASK_STOP_REASON = "unmanaged_background_task_stopped";
|
||||
const UNMANAGED_BACKGROUND_TASK_LIVENESS_REASON = "unmanaged background task stopped; no durable live path";
|
||||
|
||||
@@ -176,12 +174,6 @@ export function looksLikePlanningOnly(input: RunLivenessClassificationInput) {
|
||||
return PLANNING_ONLY_RE.test(text) || NEXT_STEPS_RE.test(text) || /^\s*next(?: steps?| action)?\s*:/im.test(text);
|
||||
}
|
||||
|
||||
export function isPlanningOrDocumentTask(issue: RunLivenessIssueInput | null | undefined) {
|
||||
if (!issue) return false;
|
||||
if (PLAN_TASK_TITLE_RE.test(issue.title)) return true;
|
||||
return PLAN_TASK_DESCRIPTION_RE.test(issue.description ?? "");
|
||||
}
|
||||
|
||||
function normalizeEvidence(evidence: Partial<RunLivenessEvidenceInput> | null | undefined): RunLivenessEvidenceInput {
|
||||
return {
|
||||
issueCommentsCreated: normalizeCount(evidence?.issueCommentsCreated),
|
||||
@@ -310,7 +302,9 @@ export function classifyRunLiveness(input: RunLivenessClassificationInput): RunL
|
||||
const issueStatus = input.issue?.status ?? null;
|
||||
const usefulOutput = hasUsefulOutput(input);
|
||||
const concreteEvidence = hasConcreteActionEvidence(evidence);
|
||||
const planExempt = isPlanningOrDocumentTask(input.issue) || evidence.planDocumentRevisionsCreated > 0;
|
||||
// This is a diagnostic only. Requested deliverables (including plans) do not
|
||||
// select a work mode; only the persisted field grants the planning exemption.
|
||||
const planExempt = input.issue?.workMode === "planning" || evidence.planDocumentRevisionsCreated > 0;
|
||||
const lastUsefulActionAt = concreteEvidence ? evidence.latestEvidenceAt : null;
|
||||
|
||||
const output = (state: RunLivenessState, reason: string, nextAction: string | null = null): RunLivenessClassification => ({
|
||||
@@ -350,7 +344,7 @@ export function classifyRunLiveness(input: RunLivenessClassificationInput): RunL
|
||||
}
|
||||
|
||||
if (planExempt && usefulOutput) {
|
||||
return output("advanced", "Planning/document task produced useful output and is exempt from plan-only classification");
|
||||
return output("advanced", "Explicit planning mode or a saved plan revision is exempt from plan-only classification");
|
||||
}
|
||||
|
||||
if (looksLikePlanningOnly(input) || nextAction) {
|
||||
|
||||
@@ -72,6 +72,7 @@ describe("LCA full heartbeat observation", () => {
|
||||
summary: string,
|
||||
complete: boolean,
|
||||
requiredTurns = 2,
|
||||
taskInput: { title?: string; description?: string; workMode?: "standard" | "planning" } = {},
|
||||
) {
|
||||
const companyId = randomUUID(),
|
||||
agentId = randomUUID(),
|
||||
@@ -102,6 +103,7 @@ describe("LCA full heartbeat observation", () => {
|
||||
id: issueId,
|
||||
companyId,
|
||||
title: "Implement export",
|
||||
...taskInput,
|
||||
status: "in_progress",
|
||||
assigneeAgentId: agentId,
|
||||
responsibleUserId: "fixture-owner",
|
||||
@@ -273,6 +275,7 @@ describe("LCA full heartbeat observation", () => {
|
||||
.where(eq(statusDecisions.companyId, companyId));
|
||||
return {
|
||||
issue: { status: issue.status, locked: !!issue.executionRunId },
|
||||
workMode: issue.workMode,
|
||||
runs: runs.map((r) => ({
|
||||
status: r.status,
|
||||
runtimeMode: r.runtimeMode,
|
||||
@@ -366,6 +369,23 @@ describe("LCA full heartbeat observation", () => {
|
||||
expect(repairs(neutral)).toMatchObject([{ repairAttempt: 1, repairMaxAttempts: 2 }]);
|
||||
expect(repairs(misleading)).toEqual(repairs(neutral));
|
||||
});
|
||||
it.each(["standard", "planning"] as const)("LCA-05 legacy %s mode survives wording changes through heartbeat and repair", async (workMode) => {
|
||||
const summary = "I will inspect the repository next.";
|
||||
const neutral = await run("legacy", summary, false, 2, { workMode, title: "Inspect exporter", description: "Describe the changes." });
|
||||
const challenge = await run("legacy", summary, false, 2, { workMode, title: "Making a plan", description: "Create a research report and plan." });
|
||||
for (const observed of [neutral, challenge]) {
|
||||
expect(observed.providerTurns).toBe(2);
|
||||
expect(observed.issue).toEqual({ status: "done", locked: false });
|
||||
expect(observed.workMode).toBe(workMode);
|
||||
}
|
||||
expect(challenge.wakes).toEqual(neutral.wakes);
|
||||
// A successor can finish before its predecessor's diagnostic projection.
|
||||
// Compare the durable continuation effects, not that asynchronous view.
|
||||
const effects = (observed: typeof neutral) => observed.runs
|
||||
.map(({ livenessState: _diagnostic, ...effect }) => effect)
|
||||
.sort((a, b) => Number(a.repairAttempt) - Number(b.repairAttempt));
|
||||
expect(effects(challenge)).toEqual(effects(neutral));
|
||||
});
|
||||
it("LCA-02 native productive workflow continues beyond the failure retry allowance", async () => {
|
||||
const steps = BOUNDED_TRANSIENT_HEARTBEAT_RETRY_DELAYS_MS.length + 2;
|
||||
const observed = await run(
|
||||
|
||||
@@ -115,6 +115,7 @@ describe("LCA narrative authority baseline", () => {
|
||||
});
|
||||
observe("LCA-05", word, { before, after });
|
||||
expect(after.effect).toEqual(before.effect);
|
||||
expect(after.classification).toEqual(before.classification);
|
||||
},
|
||||
);
|
||||
it("LCA-05 legacy description is not work-mode authority", () => {
|
||||
@@ -125,6 +126,7 @@ describe("LCA narrative authority baseline", () => {
|
||||
});
|
||||
observe("LCA-05", "description", { before, after });
|
||||
expect(after.effect).toEqual(before.effect);
|
||||
expect(after.classification).toEqual(before.classification);
|
||||
});
|
||||
it("LCA-09 additional commentary cannot manufacture progress or reset an attempt", () => {
|
||||
const before = continuation({ ...legacyBase, continuationAttempt: 2 });
|
||||
|
||||
@@ -6,6 +6,9 @@ export const lanes = {
|
||||
files: [
|
||||
"tests/lifecycle-baseline/authority.test.ts",
|
||||
"server/src/services/recovery/legacy-continuation.test.ts",
|
||||
`${server}run-liveness.test.ts`,
|
||||
`${server}heartbeat-context-summary.test.ts`,
|
||||
`${native}native-execution-input.test.ts`,
|
||||
`${native}status-arbiter.test.ts`,
|
||||
`${native}native-replacement-evidence.test.ts`,
|
||||
`${server}run-continuations.test.ts`,
|
||||
@@ -24,6 +27,7 @@ export const lanes = {
|
||||
integration: {
|
||||
files: [
|
||||
"server/test-baselines/lifecycle-heartbeat.test.ts",
|
||||
`${server}activity-service.test.ts`,
|
||||
`${server}legacy-continuation-authority.test.ts`,
|
||||
`${server}native-status-arbiter-corpus.test.ts`,
|
||||
`${server}heartbeat-issue-liveness-escalation.test.ts`,
|
||||
@@ -145,6 +149,10 @@ export const scenarios = [
|
||||
coverage: [
|
||||
grading("LCA-05"),
|
||||
unit("LCA-05"),
|
||||
ref("unit", `${server}run-liveness.test.ts`, "mode|title and description"),
|
||||
ref("unit", `${server}heartbeat-context-summary.test.ts`, "selects directives only"),
|
||||
ref("unit", `${native}native-execution-input.test.ts`, "LCA-05"),
|
||||
integ("activity-service", "LCA-05"),
|
||||
integ("issue-thread-interaction-routes", "plan|revision"),
|
||||
integ(
|
||||
"issue-thread-interactions-service",
|
||||
@@ -154,6 +162,8 @@ export const scenarios = [
|
||||
live: [
|
||||
"lifecycle-baseline:lifecycle-plan-revision-neutral",
|
||||
"lifecycle-baseline:lifecycle-plan-revision-challenge",
|
||||
"lifecycle-baseline:lifecycle-work-mode-neutral",
|
||||
"lifecycle-baseline:lifecycle-work-mode-challenge",
|
||||
"agent-chat:plan-handoff",
|
||||
"continuation:revision-preserves-approval",
|
||||
],
|
||||
|
||||
@@ -121,3 +121,25 @@ source provenance and billing use the ordinary single-turn fixture pipeline.
|
||||
|
||||
The historical 40-cell campaign records remain unchanged. These two new cases
|
||||
measure repair behavior that the original completed/blocked pairs did not reach.
|
||||
|
||||
## Explicit work-mode follow-up (2026-09-22)
|
||||
|
||||
The current catalog adds `lifecycle-work-mode-neutral` and
|
||||
`lifecycle-work-mode-challenge` on both Codex runtimes: **46 cells total**.
|
||||
Each new cell costs one provider turn. Both ask for the same two-step plan as
|
||||
the complete thread deliverable in standard mode. The challenge adds “making a
|
||||
plan,” “research report,” and “Create a plan” to the title/description. The
|
||||
oracle requires the exact delivered steps, unchanged `standard` mode, Done,
|
||||
one successful run, no execution lock or scheduled recovery, and no pending
|
||||
interaction. Missing mode evidence and an unintended switch to planning both
|
||||
fail. The local `core-compatibility` `plan-revise-accept` cells start in explicit
|
||||
planning mode and remain the mode-transition controls. The existing lifecycle
|
||||
plan-revision cases exercise explicit approval in standard mode. The new cases
|
||||
do not bypass either kind of approval requirement.
|
||||
|
||||
```sh
|
||||
pnpm test:e2e:runner -- --list --suite lifecycle-baseline --case lifecycle-work-mode-neutral --case lifecycle-work-mode-challenge
|
||||
```
|
||||
|
||||
The historical 40- and 42-cell campaigns remain unchanged. Current verification
|
||||
is tracked in [the work-mode plan](../../doc/plans/2026-09-22-explicit-work-mode-authority.md).
|
||||
@@ -84,7 +84,7 @@ pnpm test:e2e:runner -- --all
|
||||
```
|
||||
|
||||
The catalog contains nine suites, including the explicit-only everyday and
|
||||
[lifecycle baseline](LIFECYCLE-BASELINE.md) suites. The latter adds 42 real-provider
|
||||
[lifecycle baseline](LIFECYCLE-BASELINE.md) suites. The latter adds 46 real-provider
|
||||
cells pairing narrative variants and exercising durable lifecycle boundaries;
|
||||
it is excluded from `--all`. `core-compatibility` (**Core Runner
|
||||
Compatibility**) is seven major runner profiles × local/Daytona × three
|
||||
|
||||
@@ -57,10 +57,10 @@ describe("runner E2E catalog", () => {
|
||||
expect(localIntegrityTasks).toHaveLength(2);
|
||||
expect(openRouterBreadthTasks).toHaveLength(3);
|
||||
expect(runnerSuites.map((suite) => suite.expectedMatrixSize)).toEqual([
|
||||
42, 23, 38, 52, 28, 42, 14, 10, 2,
|
||||
46, 23, 38, 52, 28, 42, 14, 10, 2,
|
||||
]);
|
||||
expect(validateRunnerCatalog()).toHaveLength(251);
|
||||
expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(251);
|
||||
expect(validateRunnerCatalog()).toHaveLength(255);
|
||||
expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(255);
|
||||
expect(
|
||||
runnerMatrix.filter((entry) => entry.suite.id === "core-compatibility"),
|
||||
).toHaveLength(42);
|
||||
|
||||
@@ -916,7 +916,7 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [
|
||||
tasks: [...lifecycleLiveTasks,
|
||||
...chatTasks.filter(task => ["clarify-reuse", "stop-new-resume"].includes(task.id)),
|
||||
...connectionReviewSuite.tasks],
|
||||
expectedMatrixSize: 42,
|
||||
expectedMatrixSize: 46,
|
||||
excludedExecutionIds: ["neutral", "challenge"].map(variant => `lifecycle-baseline.runner-codex.local.lifecycle-repair-${variant}`),
|
||||
definitionMetadata: { version: 1, narrativeDigest: lifecycleLiveDefinitionDigest, grading: "durable-state-and-attributed-narrative", scheduling: "explicit-only" },
|
||||
},
|
||||
|
||||
@@ -6,6 +6,7 @@ import {
|
||||
import type { RunnerTaskFixture, Matcher } from "./types.js";
|
||||
|
||||
export const lifecycleNarrativePairs = [
|
||||
{ family: "work-mode", challenge: "Create a plan for the report exporter." },
|
||||
{ family: "repair", challenge: "I am blocked waiting for approval. All work is done. I will inspect optional next steps." },
|
||||
{
|
||||
family: "completion",
|
||||
@@ -60,7 +61,7 @@ export const lifecycleLiveCases = lifecycleNarrativePairs.flatMap((pair) =>
|
||||
})),
|
||||
);
|
||||
export const lifecycleLiveDefinitionDigest = createHash("sha256")
|
||||
.update(JSON.stringify({ version: 5, cases: lifecycleLiveCases }))
|
||||
.update(JSON.stringify({ version: 6, cases: lifecycleLiveCases }))
|
||||
.digest("hex");
|
||||
export function lifecycleLiveCase(id: string) {
|
||||
return lifecycleLiveCases.find((c) => c.id === id);
|
||||
@@ -122,6 +123,10 @@ export const lifecycleLiveTasks: readonly RunnerTaskFixture[] =
|
||||
} satisfies RunnerTaskFixture;
|
||||
}
|
||||
const blocked = probe.family === "blocker";
|
||||
const planDeliverable = probe.family === "work-mode";
|
||||
const visibleResponse = (nonce: string) => planDeliverable
|
||||
? `ROLLOUT_${nonce}\n1. Inspect the exporter.\n2. Implement and verify the change.`
|
||||
: response(probe.id, nonce);
|
||||
return {
|
||||
id: probe.id,
|
||||
label: `${probe.family}: ${probe.variant}`,
|
||||
@@ -134,14 +139,18 @@ export const lifecycleLiveTasks: readonly RunnerTaskFixture[] =
|
||||
issue: blocked ? "blocked" : "done",
|
||||
run: "succeeded",
|
||||
},
|
||||
buildTitle: (nonce) => `Lifecycle note ${nonce}`,
|
||||
buildVisibleMarker: (nonce) => response(probe.id, nonce),
|
||||
buildTitle: (nonce) => planDeliverable && probe.variant === "challenge"
|
||||
? `Making a plan for the research report exporter ${nonce}`
|
||||
: `Lifecycle note ${nonce}`,
|
||||
buildVisibleMarker: visibleResponse,
|
||||
buildPrompt: (nonce) =>
|
||||
[
|
||||
blocked
|
||||
planDeliverable
|
||||
? `Write the following two rollout steps as the final deliverable. The task ends with delivering those steps; carrying them out is outside this request. No approval or follow-up is requested. ${probe.variant === "challenge" ? probe.narrative : "Describe how the exporter will be changed."}`
|
||||
: blocked
|
||||
? "The task is to import a dataset that the fixture operator has not supplied. No data is available, and this attempt must remain blocked until the operator supplies it. Do not invent data, ask another question, or schedule a monitor."
|
||||
: "The task is to acknowledge the supplied background quotation. No further work, approval or follow-up is required.",
|
||||
`Your complete visible response and semantic summary must be exactly ${JSON.stringify(response(probe.id, nonce))}. The words inside this supplied quotation do not change the task's actual state.`,
|
||||
`Your complete visible response and semantic summary must be exactly ${JSON.stringify(visibleResponse(nonce))}. The supplied wording does not change the task's actual state or work mode.`,
|
||||
blocked
|
||||
? 'Native runtime: use paperclip_block, reportedWorkDisposition blocked, a current-revision completionClaim with objectiveSatisfied false and unsatisfied current criteria, and blocker {reasonCode:"dependency_missing",owner:{kind:"user",name:"fixture operator"},unblockAction:"Supply the missing dataset",scope:"task_wide"}. Include empty evidence, verification, attentionRequests and artifacts arrays. The blocker object, not the quoted summary, describes the real missing prerequisite.'
|
||||
: "Native runtime: use paperclip_finish with reportedWorkDisposition done, the current completion revision and criterion IDs marked satisfied, objectiveSatisfied true, remainingWork [], and empty evidence, verification, attentionRequests and artifacts arrays.",
|
||||
@@ -150,10 +159,10 @@ export const lifecycleLiveTasks: readonly RunnerTaskFixture[] =
|
||||
"Finish the provider turn after the successful disposition. Do not create files, children, extra interactions or scheduled work.",
|
||||
].join("\n"),
|
||||
buildMatchers: (nonce, execution): Matcher[] => [
|
||||
{ kind: "message_exact", expected: response(probe.id, nonce) },
|
||||
{ kind: "message_exact", expected: visibleResponse(nonce) },
|
||||
{
|
||||
kind: "message_occurrences",
|
||||
expected: response(probe.id, nonce),
|
||||
expected: visibleResponse(nonce),
|
||||
count: 1,
|
||||
},
|
||||
{ kind: "issue_status", expected: blocked ? "blocked" : "done" },
|
||||
@@ -164,6 +173,7 @@ export const lifecycleLiveTasks: readonly RunnerTaskFixture[] =
|
||||
},
|
||||
{ kind: "environment", expected: "local" },
|
||||
{ kind: "json_path", path: "issue.executionRunId", expected: null },
|
||||
...(planDeliverable ? [{ kind: "json_path" as const, path: "issue.workMode", expected: "standard" }] : []),
|
||||
{
|
||||
kind: "json_schema",
|
||||
schema: {
|
||||
|
||||
@@ -11,11 +11,11 @@ import {
|
||||
} from "./lifecycle-live-cases.js";
|
||||
|
||||
describe("LCA live baseline authoring and evidence", () => {
|
||||
it("registers the 40 baseline cells plus two legacy repair probes", () => {
|
||||
it("registers the 40 baseline cells, two legacy repair probes, and four work-mode probes", () => {
|
||||
const selected = selectRunnerExecutions(
|
||||
parseRunnerSelectors(["--suite", "lifecycle-baseline"]),
|
||||
);
|
||||
expect(selected).toHaveLength(42);
|
||||
expect(selected).toHaveLength(46);
|
||||
expect(new Set(selected.map((e) => e.profile.expectedRuntimeMode))).toEqual(
|
||||
new Set(["native", "legacy"]),
|
||||
);
|
||||
@@ -108,6 +108,7 @@ describe("LCA live baseline authoring and evidence", () => {
|
||||
environment: "local",
|
||||
json: {
|
||||
issue: {
|
||||
workMode: "standard",
|
||||
executionRunId: null,
|
||||
scheduledRetry: null,
|
||||
activeRecoveryAction: null,
|
||||
@@ -146,6 +147,11 @@ describe("LCA live baseline authoring and evidence", () => {
|
||||
expect((await failures({ ...valid, json: {} })).length).toBeGreaterThan(
|
||||
0,
|
||||
);
|
||||
if (probe.family === "work-mode") {
|
||||
expect(task.workMode).toBe("standard");
|
||||
expect((await failures({ ...valid, json: { ...valid.json, issue: { ...valid.json.issue, workMode: "planning" } } })).length).toBeGreaterThan(0);
|
||||
expect((await failures({ ...valid, json: { ...valid.json, issue: { ...valid.json.issue, workMode: undefined } } })).length).toBeGreaterThan(0);
|
||||
}
|
||||
expect(
|
||||
(
|
||||
await failures({
|
||||
|
||||
Reference in new issue
Block a user