Files
PaperClipAI/tests/runner-e2e/public-summary-image.ts
T
Dotta bcc6fe7a44 fix(runner): restore multi-turn remote sessions (#12840)
## Thinking Path

> - Paperclip manages AI agents and their work.
> - The runner executes agent turns on local and remote providers.
> - A remote per-turn session must save its state before Paperclip
releases its sandbox.
> - The session runtime returned after 100 milliseconds while the remote
checkpoint still ran.
> - The next turn also checked the local state path instead of the
verified remote backup.
> - This pull request waits for the bounded remote close and accepts
only a verified suspended backup.
> - The benefit is reliable multi-turn execution without weaker identity
checks.

## Linked Issues or Issue Description

**What happened?**

A successful remote agent turn released its sandbox before the runner
saved the verified continuation backup. The next turn failed with
`runner_state_identity_mismatch`.

**Expected behavior**

Paperclip must finish the bounded remote checkpoint before it releases
the sandbox. A later turn must validate and restore the digest-matched
suspended backup.

**Steps to reproduce**

1. Run a native ACPX Claude Plan test in a non-reusable Daytona sandbox.
2. Reject the first plan to start a second turn.
3. Observe that the second turn fails before provider execution.

**Paperclip version or commit**

The failure reproduced at `13775a90b078ff64872f50961ea1b83d575e7bc6`.

**Deployment mode**

GitHub Actions with a Daytona sandbox.

## What Changed

- Wait for the internally bounded remote runner close and checkpoint
before the host returns.
- Preserve the existing short cleanup bound for other providers.
- Validate remote continuation lifecycle from a complete digest-verified
backup when local runner state is absent.
- Keep corrupt, non-suspended, mismatched, and unverified state
fail-closed.
- Make native Plan completion and accepted-Plan wake prompts
deterministic.

## Verification

- A prior 45-cell local campaign passed 44 cells. The only failure was
the OpenCode Plan prompt variance fixed here.
- A focused OpenCode local Plan rerun passed.
- ACPX Claude Daytona message and question cells passed.
- Focused regressions cover delayed checkpoint close and verified remote
backup lifecycle.
- GitHub Build and the focused ACPX Claude Daytona Plan cell will
validate this exact head.

## Risks

Remote runnerd sessions now wait for their internally bounded
close/checkpoint path before returning; generic provider cleanup retains
the existing 100 millisecond bound. Durable run success still cannot be
reversed. The environment release guard still blocks sandbox destruction
when no verified backup stamp exists.

## Model Used

OpenAI Codex, GPT-5.6, extended reasoning, with code execution and
GitHub Actions inspection.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal task
id
- [ ] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open findings
- [ ] I will address all Greptile and reviewer comments before
requesting merge
2026-09-05 06:25:06 -05:00

160 lines
6.1 KiB
TypeScript

import path from "node:path";
import { mkdir } from "node:fs/promises";
import { chromium } from "@playwright/test";
import { runnerMatrix } from "./catalog.js";
import type { RunnerE2ECampaign } from "./types.js";
const MAX_PUBLIC_EXECUTION_DURATION_MS = 24 * 60 * 60 * 1_000;
function html(value: unknown) {
return String(value ?? "")
.replaceAll("&", "&")
.replaceAll("<", "&lt;")
.replaceAll(">", "&gt;")
.replaceAll('"', "&quot;")
.replaceAll("'", "&#39;");
}
function durationLabel(durationMs: number) {
const seconds = Math.round(durationMs / 1_000);
if (seconds < 60) return `${seconds}s`;
const minutes = Math.floor(seconds / 60);
if (minutes < 60) return `${minutes}m ${seconds % 60}s`;
return `${Math.floor(minutes / 60)}h ${minutes % 60}m`;
}
export function renderPublicCampaignSummary(campaign: RunnerE2ECampaign) {
const catalogById = new Map(
runnerMatrix.map((execution) => [execution.id, execution]),
);
const expectedIds = [
...new Set(
campaign.expected.filter((executionId) => catalogById.has(executionId)),
),
];
const expectedIdSet = new Set(expectedIds);
const resultById = new Map(
campaign.results
.filter((result) => expectedIdSet.has(result.executionId))
.map((result) => [result.executionId, result]),
);
const isPassed = (executionId: string) => {
const result = resultById.get(executionId);
return result?.status === "passed" && result.cleanup === "passed";
};
const passed = expectedIds.filter(isPassed).length;
const durationMs = [...resultById.values()].reduce(
(total, result) =>
total +
(Number.isFinite(result.durationMs) && result.durationMs >= 0
? Math.min(result.durationMs, MAX_PUBLIC_EXECUTION_DURATION_MS)
: 0),
0,
);
const suites = [
...new Map(
runnerMatrix.map((execution) => [
execution.suite.id,
execution.suite.label,
]),
),
].map(([suiteId, label]) => {
const selectedIds = expectedIds.filter(
(executionId) => catalogById.get(executionId)?.suite.id === suiteId,
);
return {
label,
selected: selectedIds.length,
passed: selectedIds.filter(isPassed).length,
};
});
const rows = suites
.filter((suite) => suite.selected > 0)
.map(
(suite) => `<div class="suite">
<span>${html(suite.label)}</span>
<strong>${suite.passed}/${suite.selected}</strong>
<em>${suite.passed === suite.selected ? "Passed" : "Needs attention"}</em>
</div>`,
)
.join("");
return `<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<style>
* { box-sizing: border-box; }
body { width: 1400px; min-height: 780px; margin: 0; padding: 70px; color: #11110f; background: #f4f1e8; font: 24px/1.35 ui-sans-serif, system-ui, sans-serif; }
header { display: flex; align-items: center; justify-content: space-between; padding-bottom: 42px; border-bottom: 2px solid #cbc6b8; }
.brand { display: flex; align-items: center; gap: 16px; font-weight: 750; letter-spacing: -.03em; }
.mark { width: 44px; height: 44px; display: grid; place-items: center; border: 3px solid #11110f; border-radius: 50%; font-size: 24px; }
.campaign { color: #5f5b52; font: 600 18px/1.3 ui-monospace, monospace; }
main { padding-top: 54px; }
.eyebrow { margin: 0 0 10px; color: #676157; font: 700 16px/1.2 ui-monospace, monospace; letter-spacing: .09em; text-transform: uppercase; }
h1 { margin: 0; font-size: 66px; line-height: 1; letter-spacing: -.055em; }
.metrics { display: grid; grid-template-columns: repeat(3, 1fr); gap: 18px; margin: 46px 0; }
.metric { padding: 26px; border: 2px solid #cbc6b8; border-radius: 18px; background: #fffdf7; }
.metric strong, .metric span { display: block; }
.metric strong { font: 750 44px/1 ui-monospace, monospace; }
.metric span { margin-top: 10px; color: #676157; font-size: 16px; text-transform: uppercase; letter-spacing: .07em; }
.suites { display: grid; gap: 12px; }
.suite { display: grid; grid-template-columns: 1fr 120px 190px; align-items: center; padding: 17px 22px; border-top: 1px solid #cbc6b8; }
.suite strong { font: 700 22px/1 ui-monospace, monospace; }
.suite em { color: #676157; font-size: 17px; font-style: normal; text-align: right; }
footer { margin-top: 44px; color: #676157; font-size: 15px; }
</style>
</head>
<body>
<header><div class="brand"><span class="mark">P</span><span>Paperclip</span></div><div class="campaign">Trusted history publication</div></header>
<main>
<p class="eyebrow">Runner full-stack E2E</p>
<h1>Campaign summary</h1>
<section class="metrics">
<div class="metric"><strong>${passed}/${expectedIds.length}</strong><span>Known executions passed</span></div>
<div class="metric"><strong>${expectedIds.length - passed}</strong><span>Failed or incomplete</span></div>
<div class="metric"><strong>${html(durationLabel(durationMs))}</strong><span>Total test time</span></div>
</section>
<section class="suites">${rows}</section>
<footer>Generated from fixed catalog labels and sanitized numeric/status fields. Provider output is never rendered here.</footer>
</main>
</body>
</html>`;
}
export async function writePublicCampaignSummaryImage(
campaign: RunnerE2ECampaign,
output: string,
) {
await mkdir(path.dirname(output), { recursive: true });
const browser = await chromium.launch({
headless: true,
env: {
LANG: "C.UTF-8",
PATH: process.env.PATH ?? "",
TMPDIR: process.env.RUNNER_TEMP ?? "/tmp",
},
});
try {
const context = await browser.newContext({
viewport: { width: 1400, height: 900 },
deviceScaleFactor: 1,
});
await context.setOffline(true);
await context.route("**/*", (route) => route.abort());
const page = await context.newPage();
await page.setContent(renderPublicCampaignSummary(campaign), {
waitUntil: "domcontentloaded",
});
await page.screenshot({
path: output,
type: "png",
fullPage: true,
animations: "disabled",
});
await context.close();
} finally {
await browser.close();
}
}