fix(evals): retain local controller cost origin (#17990)

## Thinking Path

> - Paperclip manages AI agents and their work.
> - Its eval journals retain model costs and operating reservations
under one cap.
> - The operating policy records the controller's execution origin.
> - The schema currently accepts only GitHub-hosted and AWS Fleet
origins.
> - A local controller must therefore use an inaccurate label or fail
admission.
> - This PR adds the local origin and keeps the hosted-only schedule
gate.

## Linked Issues or Issue Description

Refs #17986. No duplicate open PR or issue was found in the
operating-overhead/local search.

**What existing behavior does this improve?**

The Product E2E operating allowance and its retained journal policy.

**Current behavior**

At master `40c1ab36cd`, an unscheduled local controller cannot set
`runner: "local"`. The generic allowance validator rejects it. Local
qualification can use the same accounting controls but cannot record its
origin correctly.

**Proposed behavior**

Accept the local origin in the generic allowance. Keep all component,
price, cap, expiry, and receipt rules. The scheduled validator still
requires `github-hosted` and rejects a local allowance.

**Reason and benefit**

Local qualification can retain an accurate execution origin without
claiming that its reservations are billed GitHub charges.

## What Changed

- Add `local` to the operating allowance's runner type and validator.
- Exercise local journal creation, model admission, retained overhead,
restart, and reconciliation in the existing boundary test.
- Check that a local allowance fails Product scheduled admission.
- Pin new hosted Runner controller admission to `github-hosted`.
Preserve original labels when reading historical manifests; those reads
cannot authorize a new worker.
- Exercise the hosted-origin rejection and historical/local reads in
both source and copied portable runtimes.
- Document the local label and its accounting limits.

## Verification

- Before the fix: the new local persistence case and schedule-boundary
assertion failed; eight tests passed.
- Focused ledger, overhead, operations, and scheduled finalization
tests: 71 passed across five files.
- Controlled Runner source and portable tests: 86 passed. The new
hosted-origin regression failed before the fix.
- Full Product support suite at the first commit: 4,524 passed, three
skipped, and one retained embedded PostgreSQL bootstrap failure. The
same database failure was previously reproduced on a clean baseline.
- Separate Node test phase on the host: 137 passed.
- Actual Linux controller: database lifecycle tests pass. After
installing its missing `zip` utility, the full Product suite has 4,527
passes and one Copilot parent-version coverage failure. That same
failure reproduces on clean master `40c1ab36cd` with the same Linux
filesystem and dependencies (14 passes, one failure). Copilot files are
unchanged. The initial seven ZIP-related failures and the watcher
failure remain retained.
- The final fresh Linux semantic check passes: `GOMEMLIMIT=2GiB GOGC=50
GOMAXPROCS=1 node node_modules/typescript/bin/tsc --singleThreaded -p
tests/runner-e2e/tsconfig.json`. The parallel compiler OOM is retained.
No semantic checks were omitted.
- Both fresh reviews are 5/5, with the addressed thread resolved. Hosted
Build first failed during unchanged Pi setup prefetch with an operation
timeout. One job rerun recovered; the original failure remains retained.
All 58 latest-head checks are now acceptable.
- No paid model requests, sandbox creation, schedule changes, or
historical journal edits.

## Risks

- The caller supplies this origin. It must describe the actual
controller.
- `local` does not prove zero compute cost, provide a default allowance,
or release a hold.
- Both maintained hosted admission paths require the GitHub-hosted
controller origin. A previously accepted hosted policy with an AWS/local
label now fails new admission. Historical reads preserve the original
evidence.

## Model Used

- OpenAI GPT-6 through Codex, with tool use and code execution. The
exact serving model ID and context window are not exposed in this
session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed 2026-10-28 02:05:36 -05:00
1 parent 40c1ab36cd
commit cff331f6ca
7 files changed
+37 -8

No files matched your search

+7
View File
@@ -512,6 +512,13 @@ existing Braintrust subscription with a reference to that policy and current
plan evidence. Any newly incurred subscription or capacity charge needs its own
incremental allocation; the existing fee must remain visible separately.
Unscheduled local controllers use `runner: "local"` to retain their actual
execution origin. The same component, cap, expiry and receipt rules apply. This
label supplies no default zero charge and releases no reservation. Scheduled
allowances still require `github-hosted`; a local allowance fails that gate. The
separate hosted Runner controller enforces the same origin when admitting work.
Historical manifest reads preserve their original label; they grant no new admission.
The catalog checks that this allowance matches the selected runner, covers at
least the next four hours, and leaves room for a full conservative model-request
reservation. This happens before the build and test runners start. The workflow passes
+3 -2
View File
@@ -17,11 +17,11 @@ const currentAllowance = (time = Date.now()): EvalOperatingOverhead => ({ ...all
const runner = "ubuntu-latest";
const modelIdentity = { executionId: "claude.local.fixture", attemptId: "attempt-1", phase: "agent" as const };
it("keeps operating overhead inside the same cap through model completion and a journal restart", async () => {
it.each(["aws-fleet", "local"] as const)("keeps %s overhead inside the same cap through model completion and a journal restart", async runnerKind => {
const directory = await mkdtemp(path.join(tmpdir(), "eval-overhead-"));
const file = path.join(directory, "ledger.jsonl");
const limits: EvalSpendLimits = { campaignId: "fixture", maxUsd: "20", deadlineAt: "2099-01-01T00:00:00Z",
maxRequests: 10, maxConcurrent: 1, maxAttemptsPerExecution: 1, overhead: currentAllowance() };
maxRequests: 10, maxConcurrent: 1, maxAttemptsPerExecution: 1, overhead: { ...currentAllowance(), runner: runnerKind } };
try {
const ledger = await openEvalCostLedger(file, limits);
try {
@@ -89,6 +89,7 @@ it("requires fresh reviewed allowances tied to the actual CI runner before sched
expect(() => scheduledOperatingOverhead(undefined, runner, "20", now)).toThrow("Missing reviewed");
expect(() => scheduledOperatingOverhead("not json", runner, "20", now)).toThrow("JSON");
expect(() => scheduledOperatingOverhead(JSON.stringify(allowance()), runner, "20", now)).toThrow("runner");
expect(() => scheduledOperatingOverhead(JSON.stringify({ ...hosted, runner: "local" }), runner, "20", now)).toThrow("runner");
expect(() => scheduledOperatingOverhead(raw, "runs-on/fleet=paperclip-public-pr-x64/env=public-ci", "20", now)).toThrow("runner");
expect(() => scheduledOperatingOverhead(raw, "unrecognized", "20", now)).toThrow("runner");
expect(() => scheduledOperatingOverhead(raw, runner, "20", Date.parse("2026-10-19T23:59:59Z"))).toThrow("needs review");
+2 -2
View File
@@ -6,7 +6,7 @@ export interface EvalOperatingOverhead {
version: 1;
accountingReadiness?: "model_requests";
reference: string;
runner: "github-hosted" | "aws-fleet";
runner: "github-hosted" | "aws-fleet" | "local";
verifiedAt: string;
expiresAt: string;
components: Array<{ component: typeof EVAL_OVERHEAD_COMPONENTS[number]; maxUsd: string; reference: string; provisionalBound?: EvalCostBoundEvidence }>;
@@ -30,7 +30,7 @@ export function operatingReservationBound(allowance: EvalOperatingOverhead, comp
export function validateOperatingOverhead(value: EvalOperatingOverhead, totalUsd: string): bigint {
object(value, ["version", "accountingReadiness", "reference", "runner", "verifiedAt", "expiresAt", "components"]);
if (value.accountingReadiness !== undefined && value.accountingReadiness !== "model_requests") throw new Error("Invalid accounting readiness policy");
if (value.version !== 1 || !reference(value.reference) || !["github-hosted", "aws-fleet"].includes(value.runner)
if (value.version !== 1 || !reference(value.reference) || !["github-hosted", "aws-fleet", "local"].includes(value.runner)
|| typeof value.verifiedAt !== "string" || typeof value.expiresAt !== "string"
|| !Number.isFinite(Date.parse(value.verifiedAt)) || !Number.isFinite(Date.parse(value.expiresAt))
|| Date.parse(value.expiresAt) <= Date.parse(value.verifiedAt)