mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The on-demand crash-point suite (#19946) kills the control plane at named durable write boundaries and checks the plan's invariants > - Its first full run measured I1–I4 and I7, but I5, I6 and I8 were reported as not measured > - #19960 (generation stamping) and #19863 (speed budgets) now record what I6 and I8 need in the run log > - The six Daytona-only points have not run yet, and three of them cannot be reached by a plain task > - This pull request checks I6 and I8 per point, names the I5 gap, and readies the Daytona points for the coming Daytona wave > - The benefit is that every invariant is either asserted or a named gap, and a Daytona run explains any point it cannot reach ## Linked Issues or Issue Description Refs #19831 (plan, Phase 5). Follows #19946 (the suite). Uses #19960 (event generations) and #19863 (speed budgets). **Problem** - The suite reported I5, I6 and I8 as "not measured". - A Daytona run would report three points as `not_reached` with no reason. - A Daytona run without its key or image failed only after a 3-minute boot. **Expected behavior** - I6 and I8 are checked at each point. - I5 is a named gap with its reason. - Daytona points say what they need, and a Daytona run checks its inputs before it boots. ## What Changed - `tests/restart-chaos/invariants.ts`: `runLogViolations` checks two invariants from the run log, scoped to the wave's company: - I6: a run that failed with an error that names a stale or earlier-generation event; - I8: each `run.performance.budget_exceeded` event, with the budget evidence (metric, measured, p95, adapter, start kind). Both are keyed by run or event id, so the per-point baseline diff charges each one to the point that produced it. I5 is reported as `notMeasured` with a reason (`I5_GAP`). - `packages/shared/src/crash-point-registry.ts`: an optional `prerequisite` on the three Daytona points a plain task cannot reach. These are the teardown claim (needs a failed release), the reusable destroy (needs a `reuse_by_environment` lease), and stop-and-retain. - `tests/restart-chaos/crash-points.ts`: - `--list` prints each point's `needs:`, and a `not_reached` result names it too; - `--env daytona` refuses to start without `DAYTONA_API_KEY` and an image. - Docs: the README and the plan describe the I5/I6/I8 coverage and the Daytona points. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts tests/restart-chaos/crash-points.test.ts`: 12 passed. - `cd packages/shared && npx vitest run src/crash-points.test.ts`: 15 passed. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` and `npx tsc --noEmit -p packages/shared` both pass. - I ran `runLogViolations` against a throwaway Postgres 18 with seeded rows, then stopped and deleted it. It returned: - I8 for the budget breach, with its evidence; - I6 for the run of this company that failed on `stale_generation_event`. It excluded the other company's stale run, the `process_lost` run and a non-budget event. - `node scripts/crash-points.mjs --list --env daytona` lists 19 points, with `needs:` on the three. `--env daytona` with no key or image fails at once with the missing names. - No harness run: the harness slot is held for other work. The next suite run exercises the new checks. ## Risks - Test harness only; no product code. - I6 is a proxy: it matches the run's error text and code. A failure that names a generation for another reason would be reported, with its evidence, for a person to judge. - I8 can flag breaches on a loaded host. Each breach carries its measured and budget values, so a load-caused breach is visible as one. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
183 lines
11 KiB
TypeScript
183 lines
11 KiB
TypeScript
/**
|
||
* Plan invariants I1–I8 for one wave (durable ownership plan, "Invariants").
|
||
*
|
||
* Runs the Phase 1 report-only ownership reconciler against the wave's own
|
||
* database after every task rested. Nothing is written: `record: false`.
|
||
*
|
||
* I1, I2 two passes at the real clock: a record with no owner stamp (I1),
|
||
* or whose owner was proven gone on both passes and is still in
|
||
* progress (I2: "settled within one reconciler pass").
|
||
* I3, I4, I7 one more pass with the clock moved past every grace window
|
||
* (accounting closure 10 min, next action 5 min). The tasks are at
|
||
* rest, so anything still waiting then would wait forever.
|
||
* I6 run-log proxy: a run that failed on a stale-generation event.
|
||
* Since #19960 an earlier generation's event is history and never
|
||
* fails the current turn (runner code `stale_generation_event`).
|
||
* I8 run-log check: every `run.performance.budget_exceeded` event the
|
||
* speed budgets wrote (TTFT, agent boot, restart recovery; #19863).
|
||
* I5 not measured: see I5_GAP. R3 (one Retry at most) bounds it.
|
||
*
|
||
* I6 and I8 are company-wide scans keyed by run or event id, so the per-point
|
||
* baseline diff in main.ts charges each one to the point that produced it.
|
||
*/
|
||
import { createDb, type Db } from "../../packages/db/src/index.js";
|
||
import {
|
||
createOwnershipReportState,
|
||
runOwnershipReconciliationPass,
|
||
type InvariantViolation,
|
||
} from "../../server/src/services/recovery/ownership-invariant-report.js";
|
||
|
||
export type PlanInvariant = "I1" | "I2" | "I3" | "I4" | "I5" | "I6" | "I7" | "I8";
|
||
|
||
export type SuiteRule = InvariantViolation["rule"] | "I6_generation_fencing" | "I8_fast_turns";
|
||
|
||
/** A violation with the state of its record and run, so a failure is diagnosable after the instance is gone. */
|
||
export type EvidencedViolation = Omit<InvariantViolation, "rule"> & { rule: SuiteRule; evidence?: Record<string, unknown> };
|
||
|
||
/**
|
||
* Why I5 (work-based budgets) is not asserted. A charge is decided per restart
|
||
* claim from the attempt's event window (provider-attempt-outcome.ts, #19948)
|
||
* and recorded only indirectly in the finalization's bounded recovery
|
||
* history (progressAttempt, attemptKeptForProgress). Asserting it needs either
|
||
* an explicit per-claim charge record or reading that history per run. The
|
||
* cheap check that exists is R3, and the pre-spawn and pre-accept points
|
||
* (native.process_start.*, runner.process.*, runner.turn_start.*) recover
|
||
* without a Retry today.
|
||
*/
|
||
export const I5_GAP = "Charge decisions are not recorded per restart claim; needs an explicit charge record (section F follow-up). "
|
||
+ "R3 bounds it: no point needed a Retry.";
|
||
|
||
export interface InvariantCheck {
|
||
violations: EvidencedViolation[];
|
||
byInvariant: Partial<Record<PlanInvariant, number>>;
|
||
inProgress: Record<string, { inProgress: number; gone: number; alive: number; unprovable: number }>;
|
||
notMeasured: Partial<Record<PlanInvariant, string>>;
|
||
errors: string[];
|
||
}
|
||
|
||
/** Grace windows the shifted pass must exceed (ownership-invariant-report.ts). */
|
||
export const GRACE_SHIFT_MS = 15 * 60_000;
|
||
|
||
export const invariantOf = (violation: Pick<EvidencedViolation, "rule">): PlanInvariant => violation.rule.slice(0, 2) as PlanInvariant;
|
||
|
||
export function summarizeViolations(violations: readonly Pick<EvidencedViolation, "rule">[]): Partial<Record<PlanInvariant, number>> {
|
||
const counts: Partial<Record<PlanInvariant, number>> = {};
|
||
for (const violation of violations) counts[invariantOf(violation)] = (counts[invariantOf(violation)] ?? 0) + 1;
|
||
return counts;
|
||
}
|
||
|
||
/**
|
||
* Local-disk kinds (usage-receipt and orphan-cleanup spools) resolve their
|
||
* directories from the environment when listed. Point them at the wave's
|
||
* instance, not the operator's, for the duration of the check.
|
||
*/
|
||
async function withWaveInstance<T>(instance: { paperclipHome: string; instanceId: string }, run: () => Promise<T>): Promise<T> {
|
||
const keys = ["PAPERCLIP_HOME", "PAPERCLIP_INSTANCE_ID", "SANDBOX_ORPHAN_CLEANUP_SPOOL_DIR"] as const;
|
||
const saved = Object.fromEntries(keys.map(key => [key, process.env[key]]));
|
||
process.env.PAPERCLIP_HOME = instance.paperclipHome;
|
||
process.env.PAPERCLIP_INSTANCE_ID = instance.instanceId;
|
||
delete process.env.SANDBOX_ORPHAN_CLEANUP_SPOOL_DIR;
|
||
try { return await run(); } finally {
|
||
for (const key of keys) {
|
||
if (saved[key] === undefined) delete process.env[key]; else process.env[key] = saved[key];
|
||
}
|
||
}
|
||
}
|
||
|
||
const clip = (value: unknown) => typeof value === "string" ? value.slice(0, 300) : value ?? null;
|
||
|
||
type Row = Record<string, unknown>;
|
||
const text = (value: unknown) => (typeof value === "string" ? value : "");
|
||
|
||
/** I6 and I8 from the run log of `companyId` (all companies when null). */
|
||
export async function runLogViolations(db: Db, companyId: string | null): Promise<EvidencedViolation[]> {
|
||
const breaches = await db.$client.unsafe(
|
||
`select e.id::text as id, e.company_id::text as "companyId", e.run_id::text as "runId", r.issue_id::text as "issueId", e.payload
|
||
from heartbeat_run_events e join heartbeat_runs r on r.id = e.run_id
|
||
where e.event_type = 'run.performance.budget_exceeded' and ($1::text is null or e.company_id::text = $1)`, [companyId]) as Row[];
|
||
const stale = await db.$client.unsafe(
|
||
`select id::text as id, company_id::text as "companyId", issue_id::text as "issueId", status, error_code as "errorCode", error
|
||
from heartbeat_runs
|
||
where status in ('failed', 'timed_out', 'interrupted')
|
||
and (coalesce(error_code, '') ~* 'generation' or coalesce(error, '') ~* 'stale_generation|earlier generation')
|
||
and ($1::text is null or company_id::text = $1)`, [companyId]) as Row[];
|
||
return [
|
||
...breaches.map(row => {
|
||
const payload = (row.payload ?? {}) as Record<string, unknown>;
|
||
return { rule: "I8_fast_turns" as const, kind: "speed_budget", recordId: text(row.id), companyId: text(row.companyId) || null,
|
||
runId: text(row.runId) || null, issueId: text(row.issueId) || null, reason: `${text(payload.metric) || "unknown"}_over_p95`,
|
||
evidence: { budget: { metric: payload.metric, measuredMs: payload.measuredMs, budgetP95Ms: payload.budgetP95Ms,
|
||
adapter: payload.adapter, startKind: payload.startKind } } };
|
||
}),
|
||
...stale.map(row => ({ rule: "I6_generation_fencing" as const, kind: "run", recordId: text(row.id),
|
||
companyId: text(row.companyId) || null, runId: text(row.id), issueId: text(row.issueId) || null,
|
||
reason: "run_failed_on_earlier_generation_event" })),
|
||
];
|
||
}
|
||
|
||
async function withEvidence(db: Db, violations: EvidencedViolation[]): Promise<EvidencedViolation[]> {
|
||
if (!violations.length) return violations;
|
||
const wakeIds = violations.filter(violation => violation.kind === "wake_request" || violation.kind === "wake_request_claimed")
|
||
.map(violation => violation.recordId);
|
||
const runIds = [...new Set(violations.flatMap(violation => violation.runId ? [violation.runId] : []))];
|
||
const wakes = (wakeIds.length ? await db.$client.unsafe(
|
||
"select id::text, status, reason, error, run_id::text as \"runId\" from agent_wakeup_requests where id::text = any($1)", [wakeIds]) : []) as Row[];
|
||
const runs = (runIds.length ? await db.$client.unsafe(
|
||
"select id::text, status, error_code as \"errorCode\", error from heartbeat_runs where id::text = any($1)", [runIds]) : []) as Row[];
|
||
const wakeById = new Map(wakes.map(wake => [wake.id, wake]));
|
||
const runById = new Map(runs.map(run => [run.id, run]));
|
||
return violations.map(violation => {
|
||
const wake = wakeById.get(violation.recordId);
|
||
const run = violation.runId ? runById.get(violation.runId) : undefined;
|
||
const evidence = {
|
||
...(wake ? { wake: { status: wake.status, reason: clip(wake.reason), error: clip(wake.error), runId: wake.runId } } : {}),
|
||
...(run ? { run: { status: run.status, errorCode: run.errorCode, error: clip(run.error) } } : {}),
|
||
};
|
||
return Object.keys(evidence).length ? { ...violation, evidence } : violation;
|
||
});
|
||
}
|
||
|
||
export async function checkPlanInvariants(input: {
|
||
connectionString: string; companyId: string | null; paperclipHome: string; instanceId: string; now?: Date;
|
||
}): Promise<InvariantCheck> {
|
||
const db = createDb(input.connectionString);
|
||
const errors: string[] = [];
|
||
try {
|
||
const now = input.now ?? new Date();
|
||
const pass = (state: ReturnType<typeof createOwnershipReportState>, at: Date) =>
|
||
withWaveInstance(input, () => runOwnershipReconciliationPass(db, { state, now: at, record: false }));
|
||
const state = createOwnershipReportState();
|
||
// I2 needs the same record proven gone on both passes: a kind that failed
|
||
// to list on the first pass has only one proof, so report it as incomplete.
|
||
const first = await pass(state, now);
|
||
for (const kind of first.kinds) if (kind.error) errors.push(`${kind.kind} (first I1/I2 pass): ${kind.error}`);
|
||
if (first.retryNeeded) errors.push("the first I1/I2 pass did not finish (retryNeeded)");
|
||
const second = await pass(state, new Date(now.getTime() + 1_000));
|
||
// A failed I3/I4/I7 query is only logged by the pass and sets retryNeeded.
|
||
// Retry once; a pass that still could not finish is an incomplete check.
|
||
let shifted = await pass(createOwnershipReportState(), new Date(now.getTime() + GRACE_SHIFT_MS));
|
||
if (shifted.retryNeeded) shifted = await pass(createOwnershipReportState(), new Date(now.getTime() + GRACE_SHIFT_MS));
|
||
for (const kind of [...second.kinds, ...shifted.kinds]) if (kind.error) errors.push(`${kind.kind}: ${kind.error}`);
|
||
if (second.retryNeeded) errors.push("the I1/I2 pass did not finish (retryNeeded)");
|
||
if (shifted.retryNeeded) errors.push("the I3/I4/I7 pass did not finish after one retry (retryNeeded)");
|
||
const ownCompany = (violation: InvariantViolation) => input.companyId === null || violation.companyId === null
|
||
|| violation.companyId === input.companyId;
|
||
const fromClock = second.violations.filter(violation => violation.rule === "I1_owned" || violation.rule === "I2_settled");
|
||
const fromShift = shifted.violations.filter(violation => violation.rule !== "I1_owned" && violation.rule !== "I2_settled");
|
||
let runLog: EvidencedViolation[] = [];
|
||
try { runLog = await runLogViolations(db, input.companyId); } catch (error) {
|
||
errors.push(`I6/I8 run-log check: ${error instanceof Error ? error.message : String(error)}`);
|
||
}
|
||
const violations = await withEvidence(db, [...fromClock, ...fromShift].filter(ownCompany));
|
||
violations.push(...runLog);
|
||
const inProgress = Object.fromEntries(second.kinds.filter(kind => kind.inProgress > 0)
|
||
.map(kind => [kind.kind, { inProgress: kind.inProgress, ...kind.verdicts }]));
|
||
return {
|
||
violations, byInvariant: summarizeViolations(violations), inProgress, errors,
|
||
notMeasured: { I5: I5_GAP },
|
||
};
|
||
} finally {
|
||
await db.$client.end({ timeout: 5 }).catch(() => undefined);
|
||
}
|
||
}
|