mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-25 02:26:12 +02:00
DottaandPaperclip
6d46851920
## Thinking Path > - Paperclip manages agents and the work they perform. > - Product E2E checks verify real user paths through those agents. > - A recorded GitHub review-bot walkthrough had no repeatable registered case. > - App setup, automatic triggering, publication, and task completion need separate evidence. > - This pull request adds an explicit attended journey to the existing catalog and report tools. > - Reusable setup profiles and independent assertions preserve the same outcome across compatible targets. ## Linked Issues or Issue Description Refs: #17262. This case remains an explicit attended attached-target flow. It does not replace the shared deployment-target or paid-operations framework. **What existing behavior does this improve?** The Product E2E catalog can now select and retain evidence for GitHub review-bot setup and its first automatic review. **Current behavior** Recorded walkthroughs cannot be rerun as a fixed registered case. A published review alone can hide a failed run or unfinished task. **Proposed behavior** Create and install an owned App, save automatic new-PR settings, create a defective PR, and require correlated review delivery, an inline finding, a consistent check, a successful run, and a Done task. Retain failures and confirm owned-resource cleanup. **Reason and benefit** The case makes a useful external integration reproducible without treating setup, unit checks, or report rendering as proof of a completed live journey. ## What Changed - Add two explicit attended cells for an existing configured agent in local or Daytona execution. - Declare hosting, account, connection, and credential prerequisites through private setup bindings. - Add fixed browser controls, ownership checks, independent grading, and cleanup. - Package declared screenshots through the existing sanitizer, manifest, validator, and dashboard. - Reject operations-managed execution until budgeted routing is qualified. - Document hosted QA credential setup and evidence limits. ## Verification - Focused catalog, oracle, report, connection, and ownership checks: 127 tests passed. - Inherited-environment preflight regression failed against the previous commit and passed after the fix. Actual-run environment verification remains required. - Harness TypeScript check and full workspace typecheck passed. Full build passed. The local full-test attempt was interrupted after repeated embedded PostgreSQL startup failures; its exit code of zero does not establish a pass; Exact-head Linux CI passed the full test, typecheck, build, Runner, serialized, browser, and canary gates on `abbddeade848edefc8de75f11cc9008b2cd52764`. - A real hosted attempt published a rated review and inline finding. It failed when its native session timed out before the task reached Done. Cleanup passed. Full live qualification remains open. - Credential scan found no known credential values in the authored files or retained report corpus. ## Risks - The full case creates external test resources and uses the configured provider. It requires an explicit selection and current task authority. It is excluded from default and CI matrices. - Time and run limits are not a hard dollar cap. Missing dollar cost stays unknown. Fixture setup does not weaken budget policy. - Fresh personal OAuth, unattended authentication, runtime skill-read proof, and cross-host qualification are separate claims. - The selected live fixture needs environment/lifecycle recovery before it can adopt a longer provider deadline. ## Model Used OpenAI GPT-6 family through Codex, with reasoning, code execution, and browser tools. The exact deployment model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used, including the available version and capability details - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR following the enhancement template - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal ticket ID - [x] I have run focused tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation - [x] I have considered and documented risks - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open findings - [x] I will address all Greptile and reviewer comments before merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
52 lines
4.0 KiB
TypeScript
52 lines
4.0 KiB
TypeScript
import { createHash } from "node:crypto";
|
|
import { mkdtemp, readFile, rm, stat, writeFile } from "node:fs/promises";
|
|
import os from "node:os";
|
|
import path from "node:path";
|
|
import { afterEach, expect, it } from "vitest";
|
|
import { runnerMatrix } from "./catalog.js";
|
|
import { writeGitHubReviewReport } from "./github-review-report.js";
|
|
import { validateRetainedRunnerResult } from "./result-validation.js";
|
|
import type { RunnerE2EResult } from "./types.js";
|
|
|
|
const temporary: string[] = [];
|
|
afterEach(async () => { await Promise.all(temporary.splice(0).map(root => rm(root, { recursive: true, force: true }))); });
|
|
const pixel = Buffer.from("iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/x8AAwMCAO+jBv0AAAAASUVORK5CYII=", "base64");
|
|
function fixture(): RunnerE2EResult {
|
|
const cell = runnerMatrix.find(row => row.id === "github-review-journey.configured-agent.local.setup-and-first-review")!;
|
|
return { schema: "paperclip.runner-e2e.result/v2", executionId: cell.id, suiteId: cell.suite.id, suiteDefinitionHash: cell.suiteDefinitionHash,
|
|
profileId: cell.profile.id, environmentId: "local", caseId: cell.task.id, provider: "opencode", model: "configured-model", runtimeMode: "native",
|
|
attempt: 1, status: "failed", error: "native_session_interrupted", cleanup: "passed", durationMs: 1000,
|
|
startedAt: "2026-10-24T00:00:00Z", finishedAt: "2026-10-24T00:00:01Z",
|
|
githubReviewJourney: { version: 1, outcome: "failed", phase: "review_execution_completion", setupId: "qa", hosting: "paperclip-cloud", targetCommit: "a".repeat(40), assisted: true,
|
|
dependencies: [], checkpoints: { github_delivery: true }, cleanup: { endpoint: true, app: true, branch: true, pullRequest: true },
|
|
budgetPolicyRead: "verified", budgetPolicies: [{ scopeType: "agent", amount: 100, hardStopEnabled: true, unpricedUsagePolicy: "allow" }],
|
|
execution: { reviewId: "review", issueId: "issue", runId: "run", runStatus: "failed", runtimeMode: "native", provider: "opencode", model: "configured-model" } },
|
|
screenshots: [{ id: "failure", label: "Failed native completion", file: "failure.png", sha256: createHash("sha256").update(pixel).digest("hex") }] };
|
|
}
|
|
|
|
it("packages an honest failed journey for the existing report pipeline without admitting other private files", async () => {
|
|
const output = await mkdtemp(path.join(os.tmpdir(), "github-report-test-")); temporary.push(output);
|
|
await writeFile(path.join(output, "failure.png"), pixel);
|
|
await writeFile(path.join(output, "auth.png"), pixel);
|
|
const result = await writeGitHubReviewReport({ output, campaignId: "retained-failure", result: fixture(), resources: { branch: "owned", appName: "owned", runIds: ["run"] }, catalog: runnerMatrix, secrets: [] });
|
|
expect(result.status).toBe("failed");
|
|
expect(() => validateRetainedRunnerResult(result)).not.toThrow();
|
|
const manifest = JSON.parse(await readFile(path.join(output, "evidence/evidence-manifest.json"), "utf8"));
|
|
expect(manifest).toMatchObject({ leaks: [], missing: [] });
|
|
expect(manifest.files).toContain("snapshots/failure.png");
|
|
expect(manifest.files).toContain("snapshots/journey-resources.json");
|
|
await expect(stat(path.join(output, "evidence/auth.png"))).rejects.toThrow();
|
|
expect(await readFile(path.join(output, "evidence/snapshots/failure.png"))).toEqual(pixel);
|
|
const campaign = JSON.parse(await readFile(path.join(output, "campaign.json"), "utf8"));
|
|
expect(campaign).toMatchObject({ passed: 0, failed: 1 });
|
|
const html = await readFile(path.join(output, "dashboard.html"), "utf8");
|
|
expect(html).toContain("evidence/snapshots/failure.png");
|
|
});
|
|
|
|
it("refuses an altered screenshot rather than retaining its claimed digest", async () => {
|
|
const output = await mkdtemp(path.join(os.tmpdir(), "github-report-test-")); temporary.push(output);
|
|
await writeFile(path.join(output, "failure.png"), "altered");
|
|
await expect(writeGitHubReviewReport({ output, campaignId: "altered", result: fixture(), resources: { branch: "owned", appName: "owned", runIds: [] }, catalog: runnerMatrix, secrets: [] }))
|
|
.rejects.toThrow("journey_screenshot_digest_mismatch");
|
|
});
|