mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-06 23:20:53 +01:00
## Thinking Path > - Paperclip manages agent work through tasks, projects, and routines. > - Daily QA must exercise the staging app through normal browser and API surfaces. > - A schedule must create real work and retain its result and history. > - Pausing that schedule must prevent the next invocation. > - The routine journey in #18652 passed in QA, while its separate completion journey has not passed. > - This PR delivers the routine test independently and preserves its measured flow and assertions. ## Linked Issues or Issue Description Refs: #18652. This is the routine-only delivery from that PR. The original completion implementation and its failed attempts remain on #18652. ## What Changed - Add the target-independent `routine-scheduled-run` journey with one native run and seven required checks. - Start from the measured routine source `ac4efb7c6f4782ea806661683c656fd2b4cc3189`. Keep its request, seven success assertions, one native-run limit, and 65-second pause check. - Bound each observed GET by the original remaining journey deadline. Preserve the normal API timeout for other callers. Keep the original routine error if final trigger disabling also fails, save a closed secondary receipt, and fail an otherwise successful journey when disabling fails. - Narrow the shared output-reader type to its one-argument GET capability without changing its runtime code. - Register only this case in the catalog, types, feature associations, dispatcher, and definition hash. Keep the strict catalog expectation in sync. - Retain the real Routines component controls and the existing project-flow coverage for the shared browser helper. - Document resource cleanup and the 65-second pause observation. Preserve the current routine map content. ## Verification - The original source passed [QA staging run 41567574185](https://pages.paperclip.ing/evals-qa-smoke-41567574185-1/). TestOps launch 1657 / result 3431 maps to case 718. All seven checks and five phases passed. One scheduler-bound native run, saved output attribution, the full pause window, and cleanup were verified. - Current source `8f62137f130f59fd3e39276e3d60185ddff24517` passed [fresh QA staging run 41582706162](https://pages.paperclip.ing/evals-qa-smoke-41582706162-1/) against Core `09162fe893b464dd2ccdafd14bd3ef1611202a86`. TestOps launch 1855. All seven checks, five phases, and all three workflow jobs passed. The run used one scheduler-bound native turn and observed 65.352 seconds after pause and completion. Canonical evidence collection and normal cleanup completed, with no active operations and the owned agent terminal. - This fresh pass covers the reviewed deadline and failure-reporting changes in the current source. The earlier ac4 pass and the byte-preserving b940 extraction remain historical records under their original conditions; neither failed attempts nor prior source conditions are regraded. - Causal controls: six assertions failed on the old behavior, then all 62 focused controls passed after the repair. Local: 834 related support tests, 41 shared chat/output-reader tests, and 26 unchanged real component controls passed. Runner E2E typecheck, recursive typecheck, build, maps, token gates, and `git diff --check` passed. The temporary output-reader type mismatch exposed by typecheck was corrected by narrowing the capability type. - The first restricted build/typecheck invocations failed because local tsx IPC was denied. A restricted support run also lacked the report CLI output. The same source passed with normal owned local IPC allowed; the original logs remain retained. - The original head's hosted e2e shard 2/8 failed at the unchanged mobile preference frame-count test: expected more than five frames, observed five. The cause is unknown. Its original log remains retained; it is not labeled a proven flake. No CI rerun was requested. - Current-head full CI run 41580423918, attempt 1, passed. The fresh review is 5/5 with zero unresolved threads. The broad local suite was not repeated. The earlier full local run recorded in #18643 had four failures and did not reach 164 serialized groups; this PR does not claim an all-local pass. - Prior failed routine attempts remain recorded. They exposed composer focus, collapsed schedule controls, pending browser events, and Markdown serialization problems. The measured later pass does not regrade those attempts. ## Risks - The shared browser-event helper also serves task creation and project creation. Its regression tests and existing consumers remain included. - The staging pass qualifies the captured source and configuration. The coordinator retains source merging, daily pinning, and plan selection. - The repository test and the unpassed completion test are not registered by this PR. ## Model Used OpenAI GPT-6 through Codex, with tool use and code execution. The exact deployment ID and context window are not exposed by this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub issue/PR references) --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
142 lines
11 KiB
TypeScript
142 lines
11 KiB
TypeScript
import { mkdtemp, readFile, rm } from "node:fs/promises";
|
|
import os from "node:os";
|
|
import path from "node:path";
|
|
import { createHash } from "node:crypto";
|
|
import { afterEach, expect, it, vi } from "vitest";
|
|
import { runScheduledRoutine } from "./routine-schedule-flow.js";
|
|
import type { JourneyContext, JourneyEvidence, TargetSession } from "./types.js";
|
|
|
|
// Browser rendering is covered by the real Routines component tests. This entry
|
|
// control keeps the API reads and the production attribution grade intact.
|
|
vi.mock("@playwright/test", () => ({ expect: Object.assign(() => ({ toBeVisible: async () => {} }), {
|
|
poll: (load: () => Promise<unknown>) => ({ toBe: async (value: unknown) => expect(await load()).toBe(value) }),
|
|
}) }));
|
|
type Row = Record<string, any>;
|
|
const directories: string[] = [];
|
|
afterEach(async () => { await Promise.all(directories.splice(0).map(directory => rm(directory, { recursive: true, force: true }))); });
|
|
|
|
async function fixture(fault: "body" | "document-author" | "revision-run" | "creation-description" | "creation-project" | null = null, disableFailure = false) {
|
|
const campaignId = "journey-1-campaign", expectedBody = `scheduled-${campaignId}\nSchedule check complete.`;
|
|
const agent = { id: "agent", name: "Owned QA agent", provider: "opencode", model: "configured-model", runtimeMode: "native" };
|
|
const project = { id: "project", companyId: "company", name: `Journey QA routine project ${campaignId}` };
|
|
const trigger = { id: "trigger", companyId: "company", routineId: "routine", kind: "schedule", cronExpression: "* * * * *",
|
|
timezone: "America/Chicago", enabled: false, updatedAt: new Date(60_000).toISOString() };
|
|
const invocation = { id: "invocation", companyId: "company", routineId: "routine", triggerId: "trigger", source: "schedule",
|
|
status: "completed", linkedIssueId: "task", routineRevisionId: "routine-revision", triggeredAt: new Date(61_000).toISOString() };
|
|
const task = { id: "task", identifier: "QA-1", companyId: "company", projectId: "project", assigneeAgentId: agent.id,
|
|
status: "done", originKind: "routine_execution", originId: "routine", originRunId: "invocation" };
|
|
const run = { id: "native-run", companyId: "company", issueId: task.id, agentId: agent.id, status: "succeeded", runtimeMode: "native",
|
|
contextSnapshot: { paperclipEnvironment: { id: "environment" } },
|
|
runnerProfileJson: { nativeExecutionInput: { provider: { kind: agent.provider, model: agent.model } } } };
|
|
const body = fault === "body" ? expectedBody.slice(0, -1) : expectedBody;
|
|
const document = { id: "document", companyId: "company", issueId: task.id, latestRevisionId: "revision", body,
|
|
createdByAgentId: fault === "document-author" ? "foreign-agent" : agent.id, createdByUserId: null };
|
|
const revision = { id: "revision", documentId: document.id, body, createdByAgentId: agent.id,
|
|
createdByRunId: fault === "revision-run" ? "foreign-run" : run.id };
|
|
const reads: string[] = [];
|
|
let instructions = "", pastes = 0, triggerCreated = false, responseResolver: ((value: unknown) => void) | undefined;
|
|
const routine = () => ({ id: "routine", companyId: "company", title: `Journey QA scheduled routine ${campaignId}`,
|
|
projectId: fault === "creation-project" ? "foreign-project" : project.id, assigneeAgentId: agent.id,
|
|
createdByUserId: "user", createdByAgentId: null,
|
|
description: fault === "creation-description" ? instructions.replaceAll("`", "\\`") : instructions,
|
|
concurrencyPolicy: "always_enqueue", triggers: triggerCreated ? [trigger] : [] });
|
|
const locator = (name = ""): any => ({ first() { return this; }, getByRole: (_role: string, options: { name?: string } = {}) => locator(options.name),
|
|
getByPlaceholder: () => locator(), locator: () => ({
|
|
fill: async (value: string) => { instructions = value; }, focus: async () => {},
|
|
// The real MDX browser discriminator covers serialization. Keep this unit
|
|
// editor boundary separate from the production API reads and grade.
|
|
evaluate: async (_action: unknown, text?: string) => {
|
|
if (text === undefined) return false;
|
|
pastes += 1; instructions = text;
|
|
},
|
|
}),
|
|
fill: async () => {}, click: async () => {
|
|
if (name === "Enable trigger") { trigger.enabled = true; responseResolver?.({ status: () => 200, json: async () => ({ ...trigger }) }); }
|
|
if (name === "Pause trigger") trigger.enabled = false;
|
|
if (name === "Create routine") responseResolver?.({ status: () => 201, json: async () => routine() });
|
|
} });
|
|
const api = { origin: "https://qa.example.test", async post(route: string) {
|
|
if (route === "/api/companies/company/projects") return project;
|
|
expect(route).toBe("/api/routines/routine/triggers"); triggerCreated = true; return { trigger: { ...trigger } };
|
|
}, async patch(route: string, input: Row) {
|
|
if (route === "/api/routine-triggers/trigger") {
|
|
if (disableFailure) throw new Error("Private transport error must not replace attribution");
|
|
trigger.enabled = input.enabled;
|
|
}
|
|
},
|
|
async get(route: string) {
|
|
reads.push(route);
|
|
if (route === "/api/routines/routine") return routine();
|
|
if (route === "/api/routines/routine/runs") return [invocation];
|
|
if (route === "/api/issues/task") return task;
|
|
if (route === "/api/issues/task/runs") return [{ runId: run.id }];
|
|
if (route === "/api/heartbeat-runs/native-run") return run;
|
|
if (route === "/api/issues/task/comments?order=asc" || route === "/api/issues/task/interactions") return [];
|
|
if (route === "/api/issues/task/documents/scheduled-summary") return document;
|
|
if (route === "/api/issues/task/documents/scheduled-summary/revisions/revision") return revision;
|
|
throw new Error(`Unexpected fixture read ${route}`);
|
|
} };
|
|
const page = { getByRole: (_role: string, options: { name?: string } = {}) => locator(options.name), getByPlaceholder: () => locator(),
|
|
async goto(url: string) { if (url.endsWith("/routines/routine/runs")) throw new Error("Stop the positive control before the real pause window"); },
|
|
waitForResponse: () => new Promise(resolve => { responseResolver = resolve; }) };
|
|
const context = { api, page, company: { id: "company", issuePrefix: "QA" }, user: { id: "user" }, agent, environmentId: "environment" } as unknown as JourneyContext;
|
|
const directory = await mkdtemp(path.join(os.tmpdir(), "routine-attribution-")); directories.push(directory);
|
|
const evidence = { checks: [] } as unknown as JourneyEvidence;
|
|
return { expectedBody, body, instructions: () => instructions, pastes: () => pastes, evidence, trigger, directory, reads,
|
|
run: () => runScheduledRoutine(context, { async own() {} } as unknown as TargetSession, campaignId, evidence,
|
|
{ directory, deadlineAt: Date.now() + 10_000 }) };
|
|
}
|
|
|
|
it.each(["creation-description", "creation-project"] as const)("retains closed %s evidence before the unchanged creation failure", async fault => {
|
|
const f = await fixture(fault);
|
|
await expect(f.run()).rejects.toMatchObject({ code: "routine_browser_created" });
|
|
const receipt = JSON.parse(await readFile(path.join(f.directory, "scheduled-routine-creation.json"), "utf8"));
|
|
expect(receipt.matches).toEqual({ company: true, title: true, project: fault !== "creation-project", assignee: true,
|
|
userCreator: true, agentCreatorNull: true, description: fault !== "creation-description", concurrency: true, zeroTriggers: true });
|
|
expect(receipt.expectedDescription.sha256).toBe(createHash("sha256").update(f.instructions()).digest("hex"));
|
|
expect(receipt.savedDescription.backslashCount).toBe(fault === "creation-description" ? 6 : 0);
|
|
expect(JSON.stringify(receipt)).not.toContain("Schedule check complete");
|
|
expect(f.evidence.checks).toEqual([{ id: "routine-browser-created", passed: false }]);
|
|
expect(f.reads.some(route => route.includes("heartbeat-runs"))).toBe(false);
|
|
});
|
|
|
|
it.each(["body", "document-author", "revision-run"] as const)("retains closed %s attribution evidence before preserving the failed grade", async fault => {
|
|
const f = await fixture(fault);
|
|
await expect(f.run()).rejects.toMatchObject({ code: "routine_result_attributed" });
|
|
const receipt = JSON.parse(await readFile(path.join(f.directory, "scheduled-routine-result.json"), "utf8"));
|
|
expect(receipt.matches).toMatchObject({ documentExactBody: fault !== "body", documentAuthor: fault !== "document-author", revisionRun: fault !== "revision-run" });
|
|
expect(Object.values(receipt.matches).every(Boolean)).toBe(false);
|
|
expect(receipt.documentBody.sha256).toBe(createHash("sha256").update(f.body).digest("hex"));
|
|
expect(receipt.revision.createdByRunId).toBe(fault === "revision-run" ? "foreign-run" : "native-run");
|
|
expect(JSON.stringify(receipt)).not.toContain("Schedule check complete");
|
|
expect(f.evidence.checks.at(-1)).toEqual({ id: "routine-result-attributed", passed: false });
|
|
expect(f.trigger.enabled).toBe(false);
|
|
});
|
|
|
|
it("keeps the real journey attribution failure when final trigger disabling also fails", async () => {
|
|
const f = await fixture("body", true);
|
|
await expect(f.run()).rejects.toMatchObject({ code: "routine_result_attributed" });
|
|
expect(f.evidence.checks.at(-1)).toEqual({ id: "routine-result-attributed", passed: false });
|
|
const receipt = JSON.parse(await readFile(path.join(f.directory, "scheduled-routine-cleanup-failure.json"), "utf8"));
|
|
expect(receipt).toMatchObject({ code: "routine_trigger_disable_failed", primaryFailurePreserved: true });
|
|
expect(JSON.stringify(receipt)).not.toContain("Private transport");
|
|
});
|
|
|
|
it("sends the unchanged exact two-line body as a delimited instruction and accepts valid attribution", async () => {
|
|
const f = await fixture();
|
|
await expect(f.run()).rejects.toMatchObject({ code: "routine_open_history_failed" });
|
|
const block = f.instructions().match(/```text\n([\s\S]*?)\n```/);
|
|
expect(block?.[1]).toBe(f.expectedBody);
|
|
expect(f.instructions()).toContain("including the final period");
|
|
expect(f.instructions()).toContain("document body.\n\n```text\n");
|
|
expect(f.instructions()).toContain("\n```\n\nThen finish the task.");
|
|
expect(f.pastes()).toBe(1);
|
|
const creation = JSON.parse(await readFile(path.join(f.directory, "scheduled-routine-creation.json"), "utf8"));
|
|
expect(Object.values(creation.matches).every(Boolean)).toBe(true);
|
|
expect(creation.savedDescription.sha256).toBe(createHash("sha256").update(f.instructions()).digest("hex"));
|
|
const receipt = JSON.parse(await readFile(path.join(f.directory, "scheduled-routine-result.json"), "utf8"));
|
|
expect(Object.values(receipt.matches).every(Boolean)).toBe(true);
|
|
expect(f.evidence.checks.at(-1)).toEqual({ id: "routine-result-attributed", passed: true });
|
|
expect(f.trigger.enabled).toBe(false);
|
|
});
|