mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-25 09:41:04 +01:00
## Thinking Path > - Paperclip manages agents and the work they perform. > - Product E2E tests exercise real agents, tools, setup and cleanup. > - Repeated runs need retained costs and traces, including failed attempts. > - A passing retry must not hide earlier spending or feedback. > - Recurring work also needs admission limits, private reports and failure detection. > - This change supplies those controls while keeping paid activation behind existing gates. ## Linked Issues or Issue Description Refs #16996 and #17158. This is the consolidated landing of the reviewed eval operations stack: #17019, #17024, #17025, #17027, #17030, #17033, #17035, #17038, #17043, #17044, #17048, #17050, #17051, #17092, #17096, #17100, #17107, #17109, #17112, #17115, #17117, #17121, #17123, #17127, #17130, #17133, #17134, #17137, #17140, #17143, #17154. The repository permits squash merges only. A local rehearsal showed that squashing each slice into master causes repeated conflicts in dependent slices. This PR preserves their commits and lands their combined implementation in one reviewed integration. Superseded slices will close after this merge. ## What Changed - Retain every attempt's cost receipt, including setup, failures, interruptions and retries. Separate reported amounts, estimates, missing costs and reserved capacity in reports and dashboards. - Add bounded request admission, exact model pricecards, gateway error accounting, cache policy and reconciliation evidence. Refuse unsupported routes and requests that cannot fit the allowance. - Integrate Braintrust gateway traces and OTLP correlation, with secret filtering and experiment imports. Record phase timing, including setup, feedback capture and cleanup. - Collect complaint and suggestion evidence after each attempt, including recovery after failures. Retain explicit review dispositions independently from the product grade. - Add gated daily and weekly selections, measured forecasts, setup/readiness checks, wall-time limits and retained monitor notifications. Include declared non-model overhead in admission while reporting the existing subscription separately. - Keep recurring history private, bound uploads and retention, and use isolated archive environments. Keep public publication an explicit operation. Report and publication jobs install only the committed lockfile, with scripts disabled and isolated stores; stale pins fail closed. - Integrate current master, preserve its Product E2E journey billing and privacy checks, and test that shared setup receipts do not double-count estimated agent usage. Preserve upstream fixture repairs. ## Verification - Complete integration baseline `0649a5ce7544ab20825c2183d2f421157db49f24` includes master `a3e1d915a7110a4e115b46396a3f79e6dfb4d05e` and every original stack head as ancestors. - On that integration baseline, `pnpm test:e2e:runner:unit`: 4,012 Vitest tests passed, two skipped, across 193 files; all 137 Node source-contract tests passed. - Focused billing and Product E2E journey contract checks: 24 tests passed. They cover estimates, unknown receipts, single-owner setup costs, browser-only execution and private publication guards. - Monitor and workflow wiring: all 96 Node tests passed. `node scripts/check-token-gates.mjs`: all gates clean. - `pnpm test:e2e:runner:typecheck` passes after the standard Runner TypeScript build refreshed local declarations. The first attempt rejected the new master `runner_prepared_launch` union member because this checkout retained an older generated declaration; the source union already contained that member. - Current head `492b0708a1c531a023d9ac10f0f3dc6383cc1754` removes dependency re-resolution from archive-authorized reporting jobs after the security scan flagged it. All 15 workflow-security tests pass after updating the old two-install assertion. - Hosted [CI run 39965460180](https://github.com/paperclipai/paperclip/actions/runs/39965460180) passes on the current repaired head, including typecheck, build, general tests, Runner checks, browser shards and canary dry run. Greptile is 5/5 on this exact head with no unresolved threads. The security scan passes after the fixed lockfile-resolution thread was resolved. - Paperclip Review returned a terminal neutral check but an incomplete 0-file assessment on the repaired head: its platform blocked both diff responses as prompt-injection content. This is not a successful code review. No bypass or fabricated pass was attempted; current-head Greptile, security and hosted CI are independent gates. - No paid test, reservation release or recurring-run activation accompanies this landing. Live route qualification and invoice reconciliation remain separate evidence. <details> <summary>Retained earlier validation failures</summary> Earlier broad local runs had two 15-second test timeouts (CLI poll cadence and Retry validation), one embedded-database setup timeout, and later PostgreSQL semaphore exhaustion. The exact timed-out tests passed separately without larger timeouts; the broad-run causes were not established. Those local failures remain distinct from subsequent hosted CI results. The independent #17158 fix records its Muse fixture collision, its UUID correction and hosted verification. Local Muse execution was also blocked by macOS embedded-PostgreSQL startup. No historical failure is treated as a successful paid qualification. </details> ## Risks - Admission is a local control, not a provider invoice cap. Missing provider receipts or resource charges retain unknown amounts and holds; estimates do not become billed dollars. - The bounded path admits only documented local suites and provider routes. Ordinary legacy or separate Runner protocol campaigns have different controls. Merging this code does not qualify additional routes. - Recurrence still needs the existing enable gates and current qualification/allowance evidence. Archive permissions and resources must match the documented private-environment contract. - Legacy reports may lack complete attempt or phase evidence. Reports preserve that limitation and do not infer zero usage from missing data. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools and code execution. The exact deployment variant and context-window size are not exposed by this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (targeted checks; full-suite host limitation disclosed above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
346 lines
13 KiB
TypeScript
346 lines
13 KiB
TypeScript
import { createHash } from "node:crypto";
|
|
import { lstat, readdir, readFile, readlink } from "node:fs/promises";
|
|
import path from "node:path";
|
|
import { fileURLToPath, pathToFileURL } from "node:url";
|
|
|
|
const repositoryRoot = path.resolve(import.meta.dirname, "../..");
|
|
|
|
export const DAYTONA_IMAGE_CONTENT_SCHEMA =
|
|
"paperclip-daytona-runner-image-content/v5";
|
|
export const DAYTONA_IMAGE_PLATFORM = "linux/amd64";
|
|
export const DAYTONA_IMAGE_DOCKERFILE_PATH = "docker/daytona-runner/Dockerfile";
|
|
|
|
// This mirrors the explicit repository-local build inputs copied by
|
|
// docker/daytona-runner/Dockerfile. Broad package-tree COPYs are forbidden by
|
|
// the contract test so development-only files cannot silently enter the image
|
|
// without first changing this content-identity contract.
|
|
export const DAYTONA_IMAGE_INPUT_PATHS = [
|
|
".dockerignore",
|
|
".npmrc",
|
|
"docker/daytona-runner/Dockerfile",
|
|
"package.json",
|
|
"patches",
|
|
"pnpm-lock.yaml",
|
|
"pnpm-workspace.yaml",
|
|
"scripts/link-plugin-dev-sdk.mjs",
|
|
"tsconfig.base.json",
|
|
"packages/paperclip-eval-kernel/package.json",
|
|
"packages/paperclip-eval-kernel/src",
|
|
"packages/paperclip-eval-kernel/tsconfig.json",
|
|
"packages/paperclip-runner/scripts/provision-grok.mjs",
|
|
"packages/paperclip-runner/package.json",
|
|
"packages/paperclip-runner/cursor-distributions.json",
|
|
"packages/paperclip-runner/cursor-contract.json",
|
|
"packages/paperclip-runner/protocol",
|
|
"packages/paperclip-runner/runner/Cargo.lock",
|
|
"packages/paperclip-runner/runner/Cargo.toml",
|
|
"packages/paperclip-runner/runner/crates",
|
|
"packages/paperclip-runner/scripts/acpx-sidecar-contract.mjs",
|
|
"packages/paperclip-runner/scripts/build-provider-pack.mjs",
|
|
"packages/paperclip-runner/scripts/provider-pack-executable-shims.mjs",
|
|
"packages/paperclip-runner/scripts/build-node-startup-timeout.mjs",
|
|
"packages/paperclip-runner/scripts/candidate-provider-pack.mjs",
|
|
"packages/paperclip-runner/scripts/build-copilot-distribution.mjs",
|
|
"packages/paperclip-runner/scripts/copilot-inner-distribution.mjs",
|
|
"packages/paperclip-runner/scripts/copilot-profile-identity.mjs",
|
|
"packages/paperclip-runner/scripts/materialize-copilot-binary.mjs",
|
|
"packages/paperclip-runner/scripts/install-copilot-assets.mjs",
|
|
"packages/paperclip-runner/test/fixtures/copilot-profile-v12-identity.json",
|
|
"packages/paperclip-runner/test/fixtures/copilot-profile-v14-identity.json",
|
|
"packages/paperclip-runner/test/fixtures/copilot-profile-v21-identity.json",
|
|
"packages/paperclip-runner/test/fixtures/copilot-profile-v24-identity.json",
|
|
"packages/paperclip-runner/test/fixtures/copilot-profile-v25-identity.json",
|
|
"packages/paperclip-runner/test/fixtures/copilot-profile-v26-identity.json",
|
|
"packages/paperclip-runner/test/fixtures/copilot-profile-v35-identity.json",
|
|
"packages/paperclip-runner/test/fixtures/copilot-profile-v40-identity.json",
|
|
"packages/paperclip-runner/test/fixtures/copilot-profile-v42-identity.json",
|
|
"packages/paperclip-runner/test/fixtures/copilot-profile-v46-identity.json",
|
|
"packages/paperclip-runner/test/fixtures/copilot-profile-v47-identity.json",
|
|
"packages/paperclip-runner/test/fixtures/copilot-profile-v48-identity.json",
|
|
"packages/paperclip-runner/test/fixtures/copilot-profile-v49-identity.json",
|
|
"packages/paperclip-runner/test/fixtures/copilot-profile-v50-identity.json",
|
|
"packages/paperclip-runner/test/fixtures/copilot-profile-v51-identity.json",
|
|
"packages/paperclip-runner/test/fixtures/copilot-profile-v52-identity.json",
|
|
"packages/paperclip-runner/test/fixtures/copilot-profile-v54-identity.json",
|
|
"packages/paperclip-runner/test/fixtures/copilot-profile-v55-identity.json",
|
|
"packages/paperclip-runner/test/fixtures/copilot-profile-v57-identity.json",
|
|
"packages/paperclip-runner/test/fixtures/copilot-profile-v58-identity.json",
|
|
"packages/paperclip-runner/scripts/materialize-cursor-distribution.mjs",
|
|
"packages/paperclip-runner/scripts/cursor-runtime-patch.mjs",
|
|
"packages/paperclip-runner/scripts/build-verified-provider-entrypoints.mjs",
|
|
"packages/paperclip-runner/scripts/provider-pack-runtime-tree.mjs",
|
|
"packages/paperclip-runner/scripts/generate-acpx-sidecar-contract.mjs",
|
|
"packages/paperclip-runner/scripts/generate-protocol-schema-module.mjs",
|
|
"packages/paperclip-runner/scripts/materialize-pi-distribution.mjs",
|
|
"packages/paperclip-runner/scripts/pi-distribution",
|
|
"packages/paperclip-runner/scripts/cursor-native-usage.mjs",
|
|
"packages/paperclip-runner/scripts/cursor-native-usage-source.json",
|
|
"packages/paperclip-runner/scripts/provision-cursor.mjs",
|
|
"packages/paperclip-runner/cursor-distributions.json",
|
|
"packages/paperclip-runner/cursor-contract.json",
|
|
// Pi runtime/extension/Node/closure pins are included by the src tree below.
|
|
"packages/paperclip-runner/src",
|
|
"packages/paperclip-runner/styles.css",
|
|
"packages/paperclip-runner/tsconfig.json",
|
|
"packages/paperclip-runner/tsconfig.surfaces.json",
|
|
] as const;
|
|
|
|
const ignoredRunnerDevelopmentDirectoryPaths = new Set([
|
|
"packages/paperclip-runner/devtools",
|
|
"packages/paperclip-runner/docs",
|
|
"packages/paperclip-runner/examples",
|
|
"packages/paperclip-runner/test",
|
|
"packages/paperclip-runner/test-fixtures",
|
|
"packages/paperclip-runner/test-support",
|
|
]);
|
|
|
|
const runnerDocumentationFilePattern = /\.md$/;
|
|
const runnerTestFilePattern = /\.(?:spec|test)\.(?:[cm]?[jt]sx?)$/;
|
|
const runnerRustIntegrationTestPathPattern =
|
|
/^packages\/paperclip-runner\/runner\/crates\/[^/]+\/tests(?:\/|$)/;
|
|
const runnerSmokeScriptPattern =
|
|
/^packages\/paperclip-runner\/scripts\/[^/]+-smoke\.mjs$/;
|
|
|
|
export interface DaytonaImageContentOptions {
|
|
repositoryRoot?: string;
|
|
inputPaths?: readonly string[];
|
|
platform?: string;
|
|
baseImages?: readonly string[];
|
|
frontendDigest?: string;
|
|
/** Must match the Docker PAPERCLIP_RUNNER_CANDIDATE_PROVIDERS build argument. */
|
|
candidateProviders?: readonly string[];
|
|
}
|
|
|
|
export function normalizedDaytonaCandidateProviders(values: readonly string[]): string[] {
|
|
const selected = [...values].sort();
|
|
if (selected.some(value => !["cursor", "copilot", "pi"].includes(value))
|
|
|| new Set(selected).size !== selected.length) {
|
|
throw new Error("Daytona candidate providers must be distinct known ACP profiles");
|
|
}
|
|
return selected;
|
|
}
|
|
|
|
function compareNames(left: string, right: string): number {
|
|
return left < right ? -1 : left > right ? 1 : 0;
|
|
}
|
|
|
|
function normalizedRelativePath(value: string): string {
|
|
return value.split(path.sep).join("/");
|
|
}
|
|
|
|
function shouldIgnore(relativePath: string): boolean {
|
|
return relativePath.split("/").includes("node_modules");
|
|
}
|
|
|
|
function shouldIgnoreRunnerDevelopmentInput(relativePath: string): boolean {
|
|
if (!relativePath.startsWith("packages/paperclip-runner/")) return false;
|
|
if (ignoredRunnerDevelopmentDirectoryPaths.has(relativePath)) return true;
|
|
return (
|
|
runnerDocumentationFilePattern.test(relativePath) ||
|
|
runnerTestFilePattern.test(relativePath) ||
|
|
runnerRustIntegrationTestPathPattern.test(relativePath) ||
|
|
runnerSmokeScriptPattern.test(relativePath)
|
|
);
|
|
}
|
|
|
|
function updateRecord(
|
|
hash: ReturnType<typeof createHash>,
|
|
kind: string,
|
|
relativePath: string,
|
|
payload = "",
|
|
): void {
|
|
hash.update(kind);
|
|
hash.update("\0");
|
|
hash.update(relativePath);
|
|
hash.update("\0");
|
|
hash.update(payload);
|
|
hash.update("\0");
|
|
}
|
|
|
|
function assertPinnedBaseImage(reference: string): void {
|
|
if (!/@sha256:[0-9a-f]{64}$/.test(reference)) {
|
|
throw new Error(
|
|
`Daytona image base must use an immutable sha256 digest: ${reference}`,
|
|
);
|
|
}
|
|
}
|
|
|
|
function assertPinnedFrontendDigest(digest: string): void {
|
|
if (!/^sha256:[0-9a-f]{64}$/.test(digest)) {
|
|
throw new Error(
|
|
`Daytona Dockerfile frontend must use an immutable sha256 digest: ${digest}`,
|
|
);
|
|
}
|
|
}
|
|
|
|
export function extractDaytonaDockerfileFrontendDigest(
|
|
dockerfile: string,
|
|
): string {
|
|
const firstLine = dockerfile.split(/\r?\n/, 1)[0] ?? "";
|
|
const match = /^#\s*syntax=\S+@(sha256:[0-9a-f]{64})\s*$/.exec(firstLine);
|
|
if (!match) {
|
|
throw new Error(
|
|
"Daytona Dockerfile first line must pin its syntax frontend to an immutable sha256 digest",
|
|
);
|
|
}
|
|
const digest = match[1]!;
|
|
assertPinnedFrontendDigest(digest);
|
|
return digest;
|
|
}
|
|
|
|
export function extractDaytonaBaseImages(dockerfile: string): string[] {
|
|
const baseImages: string[] = [];
|
|
const stageAliases = new Set<string>();
|
|
|
|
for (const line of dockerfile.split(/\r?\n/)) {
|
|
if (!/^\s*FROM\s/i.test(line)) continue;
|
|
const match = /^\s*FROM(?:\s+--\S+)*\s+(\S+)(?:\s+AS\s+(\S+))?\s*$/i.exec(
|
|
line,
|
|
);
|
|
if (!match) {
|
|
throw new Error(`Cannot parse Daytona Dockerfile FROM line: ${line}`);
|
|
}
|
|
|
|
const reference = match[1]!;
|
|
if (!stageAliases.has(reference)) {
|
|
assertPinnedBaseImage(reference);
|
|
baseImages.push(reference);
|
|
}
|
|
if (match[2]) stageAliases.add(match[2]);
|
|
}
|
|
|
|
if (baseImages.length === 0) {
|
|
throw new Error("Daytona Dockerfile does not declare a base image");
|
|
}
|
|
return baseImages;
|
|
}
|
|
|
|
async function resolveBaseImages(
|
|
root: string,
|
|
explicitBaseImages?: readonly string[],
|
|
): Promise<string[]> {
|
|
const baseImages = explicitBaseImages
|
|
? [...explicitBaseImages]
|
|
: extractDaytonaBaseImages(
|
|
await readFile(path.join(root, DAYTONA_IMAGE_DOCKERFILE_PATH), "utf8"),
|
|
);
|
|
if (baseImages.length === 0) {
|
|
throw new Error("Daytona image content identity requires a base image");
|
|
}
|
|
for (const reference of baseImages) assertPinnedBaseImage(reference);
|
|
return baseImages.sort(compareNames);
|
|
}
|
|
|
|
async function resolveFrontendDigest(
|
|
root: string,
|
|
explicitFrontendDigest?: string,
|
|
): Promise<string> {
|
|
const digest =
|
|
explicitFrontendDigest ??
|
|
extractDaytonaDockerfileFrontendDigest(
|
|
await readFile(path.join(root, DAYTONA_IMAGE_DOCKERFILE_PATH), "utf8"),
|
|
);
|
|
assertPinnedFrontendDigest(digest);
|
|
return digest;
|
|
}
|
|
|
|
async function hashEntry(
|
|
hash: ReturnType<typeof createHash>,
|
|
root: string,
|
|
relativePath: string,
|
|
): Promise<void> {
|
|
const normalizedPath = normalizedRelativePath(relativePath);
|
|
if (
|
|
shouldIgnore(normalizedPath) ||
|
|
shouldIgnoreRunnerDevelopmentInput(normalizedPath)
|
|
) {
|
|
return;
|
|
}
|
|
|
|
const absolutePath = path.resolve(root, relativePath);
|
|
const relativeFromRoot = path.relative(root, absolutePath);
|
|
if (
|
|
relativeFromRoot.startsWith(`..${path.sep}`) ||
|
|
relativeFromRoot === ".." ||
|
|
path.isAbsolute(relativeFromRoot)
|
|
) {
|
|
throw new Error(
|
|
`Daytona image input escapes repository root: ${relativePath}`,
|
|
);
|
|
}
|
|
|
|
const stats = await lstat(absolutePath);
|
|
if (stats.isDirectory()) {
|
|
const entries = await readdir(absolutePath, { withFileTypes: true });
|
|
entries.sort((left, right) => compareNames(left.name, right.name));
|
|
for (const entry of entries) {
|
|
await hashEntry(hash, root, path.join(relativePath, entry.name));
|
|
}
|
|
return;
|
|
}
|
|
if (stats.isSymbolicLink()) {
|
|
updateRecord(hash, "symlink", normalizedPath, await readlink(absolutePath));
|
|
return;
|
|
}
|
|
if (stats.isFile()) {
|
|
const executable =
|
|
(stats.mode & 0o111) === 0 ? "non-executable" : "executable";
|
|
updateRecord(hash, "file", normalizedPath, executable);
|
|
hash.update(await readFile(absolutePath));
|
|
hash.update("\0");
|
|
return;
|
|
}
|
|
throw new Error(`Unsupported Daytona image input type: ${relativePath}`);
|
|
}
|
|
|
|
export async function computeDaytonaImageContentId(
|
|
options: DaytonaImageContentOptions = {},
|
|
): Promise<string> {
|
|
const root = path.resolve(options.repositoryRoot ?? repositoryRoot);
|
|
const inputPaths = [
|
|
...(options.inputPaths ?? DAYTONA_IMAGE_INPUT_PATHS),
|
|
].sort(compareNames);
|
|
const hash = createHash("sha256");
|
|
updateRecord(
|
|
hash,
|
|
"contract",
|
|
DAYTONA_IMAGE_CONTENT_SCHEMA,
|
|
options.platform ?? DAYTONA_IMAGE_PLATFORM,
|
|
);
|
|
updateRecord(
|
|
hash,
|
|
"dockerfile-frontend",
|
|
await resolveFrontendDigest(root, options.frontendDigest),
|
|
);
|
|
for (const baseImage of await resolveBaseImages(root, options.baseImages)) {
|
|
updateRecord(hash, "base-image", baseImage);
|
|
}
|
|
for (const candidate of normalizedDaytonaCandidateProviders(options.candidateProviders ?? [])) {
|
|
updateRecord(hash, "candidate-provider", candidate);
|
|
}
|
|
for (const inputPath of inputPaths) {
|
|
await hashEntry(hash, root, inputPath);
|
|
}
|
|
return hash.digest("hex");
|
|
}
|
|
|
|
const invokedPath = process.argv[1] ? path.resolve(process.argv[1]) : null;
|
|
if (invokedPath && import.meta.url === pathToFileURL(invokedPath).href) {
|
|
const arguments_ = process.argv.slice(2);
|
|
const candidateFlag = arguments_[0];
|
|
if (arguments_.length > 1 || (candidateFlag !== undefined && !candidateFlag.startsWith("--candidate-providers="))) {
|
|
throw new Error("Expected only --candidate-providers=cursor,copilot,pi");
|
|
}
|
|
const candidateProviders = candidateFlag?.slice("--candidate-providers=".length).split(",").filter(Boolean) ?? [];
|
|
computeDaytonaImageContentId({ candidateProviders })
|
|
.then((contentId) => process.stdout.write(`${contentId}\n`))
|
|
.catch((error: unknown) => {
|
|
process.stderr.write(
|
|
`${error instanceof Error ? error.message : String(error)}\n`,
|
|
);
|
|
process.exitCode = 1;
|
|
});
|
|
}
|
|
|
|
export const DAYTONA_IMAGE_CONTENT_SCRIPT = fileURLToPath(import.meta.url);
|