mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-28 20:21:50 +01:00
<!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/105/changes/paperclip/105 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip needs evidence that an admitted execution survives a main-server replacement. > - Runner mechanics and server adoption belong to #17097 and #17439. > - A healthy HTTP endpoint does not prove recovery of the original execution. > - Qualification must identify the serving artifact, natural ownership, authenticated attachment and resumed processing separately. > - It must also retain failed runs and measure normal-operation costs. > - This PR extends the existing qualification machinery and acceptance record. ## Linked Issues or Issue Description **What existing behavior does this improve?** Operators can assess upgrade continuity from phase evidence tied to exact artifacts and profiles. **Current behavior** A same-artifact restart or synthetic executor pass cannot establish different-artifact upgrade and rollback support. Idle SQL gates also do not measure active-run capacity. **Proposed behavior** Use immutable A/B artifacts and strict phase receipts through the existing supervisor. Preserve failed evidence and add measured before/after workload costs. Builds on merged #17097 and #17439. The base is now `master`. #17456 follows for review convenience. This PR lands qualification machinery and the evidence record. The remaining hosted gates stay open for deployment qualification; merging this machinery does not claim those experiments passed. ## What Changed - Extend the main-server supervisor and installed-artifact verification for actual serving revisions, forward migrations and compatible rollback. - Distinguish HTTP readiness, natural ownership, authenticated attachment, resumed processing and backlog settlement. - Retain the original reconnect grace and execution identity checks. Never substitute a new task or edited lease for continuity. - Reuse real-harness and Daytona qualification paths, with exact profile and artifact provenance. - Bind current Copilot attestation inputs in the Daytona image fingerprint. - Keep one acceptance record, including both failed file-only stress phases and the final passing bounded-pool comparison. ## Verification After retargeting to master, all 158 focused qualification cases across 13 files pass, with E2E typecheck. The first 12-file batch has four report-lock test failures and 133 passes. Master #17811 had committed the exact already-approved output lock, while that fixture assumed HEAD must be the uncorrected input. The fixture now checks both pinned forms, executes the committed-lock path, and reconstructs the pinned input to retain all rejection tests. The production helper, policy hashes, patch and dependency resolution are unchanged. The corrected batch explicitly includes both process-preservation suites; the prior 146-case receipt has its own source and selection. Current candidate `8f1d6ec02f` integrates master `09ad9adce2`, including the merged server slice. Its diff contains only qualification code and documentation. The [qualification record](https://github.com/paperclipai/paperclip/blob/8f1d6ec02f5bf6c90e7b27da8156953cabd9a180/doc/plans/2026-10-22-hot-upgrade-qualification.md) is the acceptance ledger and retains source-attributed failures and hosted limits. On the eighth integration, all 146 cases across 12 focused qualification files pass, including report-lock preparation, workflow security and workflow targeting. E2E typecheck passes. The earlier 118-case result remains attributed to its prior source. The first sandbox invocation has 117 passes and one failure because process inspection was denied during inert fixture cleanup. The permitted repeat passes all 118 and a targeted process check finds no remaining fixture daemon. The earlier 96-case receipt remains source-attributed in the ledger. The latest two review regressions first fail: an intermediate replacement can cite another runner/session despite a valid final step, and malformed receipt JSON aborts evidence packaging. Each replacement now binds run/company/issue/runner/session, and corrupt receipts retain explicit invalid metadata plus sanitized diagnostics while other logs and receipts still package. These checks prove qualification machinery with injected supervisor operations, configuration subprocesses and real filesystem packaging. They do not qualify a hosted harness. Three request-reader cases, 14 artifact/configuration/cache/history cases, and three lost-partial-receipt cases fail before their respective corrections. The final collector preserves sanitized startup, readiness and recovery phase receipts through packaging and temporary-root deletion. Earlier embedded-Postgres initialization, report-catalog drift, CI scheduling/cleanup failures and fixture collisions remain recorded with their corrections or limits. The previous Greptile service error is retained as historical review evidence, not a current review pass. The record includes prior source `9bb9d3417e`'s actual Linux contention/fairness successes and the ordinary-warm regression proof from the server slice. The historical 300-session component campaign passes four after phases and retains earlier burst failures. Those component costs do not establish hosted capacity. The hosted API-only pair has zero agents/viewers, fails both absolute latency gates and the candidate memory gate, and does not complete the active-agent/viewer matrix. The strict ten-agent same-artifact restart remains attributed to its original artifact. It does not qualify this updated stack. Actual A→B/B→A, the normal drain-free Cloud rollout, and other hosted harnesses remain open. No new provider turn, deployment or hosted qualification was run during merge preparation. Prior source `57d3767eac` jobs 122590643944, 122590643983 and 122590644161 ended after runner shutdown/cancellation without final Vitest results. The shutdown cause is unproven. No retry was issued before this source was superseded by the required server recovery fix. These attempts remain failures/incomplete evidence. Fresh exact-head CI, legacy status, completed Greptile 5/5, resolved feedback and normal merge eligibility remain required. ## Risks Different-artifact A→B/B→A, the normal drain-free Cloud rollout path, other hosted harnesses and the complete concurrency/viewer matrix remain open. Frozen branch builds are development-build evidence. This PR owns qualification and evidence; production defects must be fixed in their earlier owning slice. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools and code execution. The runtime does not expose a more specific model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
61 lines
3.2 KiB
TypeScript
61 lines
3.2 KiB
TypeScript
import { mkdtemp, rm, writeFile } from "node:fs/promises";
|
|
import { tmpdir } from "node:os";
|
|
import { join } from "node:path";
|
|
import { afterEach, beforeEach, expect, it } from "vitest";
|
|
import { runnerMatrix } from "./catalog.js";
|
|
import { controllerOutageForExecution } from "./controller-outage.js";
|
|
import { readRestartRequest } from "./server-restart-request.js";
|
|
|
|
let root: string;
|
|
let requestPath: string;
|
|
beforeEach(async () => {
|
|
root = await mkdtemp(join(tmpdir(), "paperclip-supervisor-request-"));
|
|
requestPath = join(root, "restart.json");
|
|
});
|
|
afterEach(async () => { await rm(root, { recursive: true, force: true }); });
|
|
|
|
const cells = ["everyday-workflows", "daytona-restart-continuity"].flatMap(suite =>
|
|
["codex", "acpx"].map(provider => {
|
|
const execution = runnerMatrix.find(row => row.suite.id === suite && row.task.id === "recover-controller"
|
|
&& row.environment.id === "daytona" && row.profile.generation === "native" && row.profile.provider === provider);
|
|
if (!execution) throw new Error(`Missing real ${suite}/${provider} continuity cell`);
|
|
return execution;
|
|
}));
|
|
|
|
it.each(cells)("accepts the real supervisor request file for $id", async execution => {
|
|
const outage = controllerOutageForExecution(execution, {})!;
|
|
await writeFile(requestPath, JSON.stringify({ requestId: "actual-restart-request",
|
|
discardControllerCache: true, controllerOutage: outage }));
|
|
await expect(readRestartRequest(requestPath, [execution.id])).resolves.toMatchObject({
|
|
requestId: "actual-restart-request", discardControllerCache: true, controllerOutage: outage,
|
|
});
|
|
});
|
|
|
|
it("rejects cache replacement for unselected, local and legacy cells", async () => {
|
|
await writeFile(requestPath, JSON.stringify({ requestId: "unsupported-restart", discardControllerCache: true }));
|
|
const local = runnerMatrix.find(row => row.environment.id === "local")!;
|
|
const legacy = runnerMatrix.find(row => row.profile.generation === "legacy")!;
|
|
expect(local).toBeDefined(); expect(legacy).toBeDefined();
|
|
for (const ids of [[], ["not-a-cell"], [local.id], [legacy.id]]) {
|
|
await expect(readRestartRequest(requestPath, ids)).rejects.toThrow(/cache replacement requires/i);
|
|
}
|
|
});
|
|
|
|
it.each([
|
|
{ controllerOutage: { signal: "SIGKILL", holdMs: 125_000 }, discardControllerCache: false },
|
|
{ controllerOutage: { signal: "SIGKILL", holdMs: 300_001 }, discardControllerCache: true },
|
|
{ controllerOutage: { signal: "SIGKILL", holdMs: 125_000 }, discardControllerCache: true, preserveRunner: {} },
|
|
{ controllerOutage: { signal: "SIGKILL", holdMs: 125_000 }, discardControllerCache: true, crashAfterResult: {} },
|
|
{ recoveryBinding: {} },
|
|
{ controllerDowntimeMs: -1 },
|
|
])("keeps invalid and conflicting fault boundaries rejected: %j", async fault => {
|
|
await writeFile(requestPath, JSON.stringify({ requestId: "conflicting-restart", ...fault }));
|
|
await expect(readRestartRequest(requestPath, [cells[0].id])).rejects.toThrow();
|
|
});
|
|
|
|
it("ignores absent and incomplete atomic request files", async () => {
|
|
await expect(readRestartRequest(requestPath, [cells[0].id])).resolves.toBeNull();
|
|
await writeFile(requestPath, "{");
|
|
await expect(readRestartRequest(requestPath, [cells[0].id])).resolves.toBeNull();
|
|
});
|