mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-01 23:43:16 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The web server owns request handling and remote execution coordination. > - Deployment must start a ready replacement while the old server still serves. > - An outage test does not prove that two live servers can overlap safely. > - This pull request adds a separate Product E2E cell for that boundary. > - The existing run, file, recovery, and retained-session checks still apply. ## Linked Issues or Issue Description Refs #18518. The runtime prerequisite is merged; this PR qualifies its ephemeral server and deployment contracts. ## What Changed - Add the explicit `rolling-web-deployment` Codex Daytona workflow. - Start independent A/B/C homes and compiled builds behind a test ingress. - Wait for readiness before routing, then drain and delete the old home. - Observe active authority before handoff and use the existing read-only recovery oracle. - Require ordinary reads and saves to succeed throughout replacement. Read back every accepted save exactly once. - Preserve the five-minute outage suites and their existing validators. - Add a model-free supervisor smoke and an unready replacement control. - Require graceful exit before deleting an old home; forced cleanup cannot satisfy the oracle. - Preserve a durable Stop through lease release/expiry until the recovered execution owner acknowledges it. Permit only fenced reattachment of the original execution for Stop cleanup; submit no new turn. - Exclude events saved by A during shutdown from evidence of B making progress. ## Verification - E2E TypeScript check passed. - Affected harness checks: 143 passed. Catalog, fixture and report checks: 120 passed. - Actual supervisor smoke passed two replacements: 436 reads, 45 saves, zero request failures, and confirmed owned resource cleanup. - B became ready in 18.7 seconds; C became ready in 109.2 seconds. A prior C timeout remains failed evidence. - Fresh browser proof: 11 checks passed, 246 reads, 50 acknowledged saves, zero failures, 1,058ms reconnect, both old homes deleted and cleanup confirmed. - Stop lease-gap regression: three PostgreSQL delivery tests passed with a simulated provider edge. Current server TypeScript build/vendoring passed. - Full harness: 5,814 passed, 17 failed, four skipped. Two shutdown races passed the focused file; two report timeouts passed an isolated 42-test rerun. Onboarding/workflow-security/Pi/date-expired fixtures are unchanged on inspected master; the host embedded PostgreSQL fixture remains a setup limitation. This is not a green full local suite. - CI at `172509b8be` passed all gates. Greptile is 5/5 with every review thread resolved. The earlier access-service race failure passed its isolated regression and the subsequent full CI. - Final review corrections: six focused executor cases, three PostgreSQL command-delivery cases, 19 harness cases and server/E2E typechecks passed. - Three live attempts failed before agent creation: two harness setup gaps were fixed, then provider authentication failed. The saved OpenAI key returns 401; a refresh is pending. The Daytona probe reported cleanup uncertain, and no sandbox with that attempt label was found through the authenticated staging manager. No live model execution is claimed. - Live Runner overlap, released-container qualification and hosted deployment proof remain pending. Merging this opt-in qualification lane does not claim those results. ## Risks - The source lane shares workspace dependencies and does not establish container packaging correctness. - The live workflow needs an explicitly owned S3 bucket, PostgreSQL, pinned remote image and provider credentials. - Qualification cleanup stops all owned processes after failure. A failed replacement must not be reported as a pass. ## Model Used OpenAI Codex, GPT-6. Tool use, code execution and repository review. The runtime does not expose a more specific model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
121 lines
8.5 KiB
TypeScript
121 lines
8.5 KiB
TypeScript
import { isRollingWebDeployment } from "./controller-rolling-deployment.js";
|
|
import { readFile } from "node:fs/promises";
|
|
import { runnerMatrix } from "./catalog.js";
|
|
import { parseAcpxCandidateAdmission, type AcpxCandidateAdmission } from "../../packages/paperclip-runner/src/drivers/acpx/candidate-admission.js";
|
|
import { QUALIFIED_ACPX_PROFILES } from "../../packages/paperclip-runner/src/drivers/acpx/qualified-profiles.js";
|
|
import { KIMI_DISTRIBUTION_CLOSURE_SHA256 } from "../../packages/paperclip-runner/src/drivers/acpx/kimi-closure-pins.js";
|
|
import type { MatrixExecution } from "./types.js";
|
|
import { controllerOutageForExecution, parseControllerOutage, type ControllerOutage } from "./controller-outage.js";
|
|
import { parseRecoveryBinding, type RecoveryBinding } from "./controller-recovery-observer.js";
|
|
import { parseFinalizationCrashBinding, type FinalizationCrashBinding } from "./finalization-crash-boundary.js";
|
|
import { parseRestartRunnerIdentity, type RestartRunnerIdentity } from "./process-tree-owner.js";
|
|
|
|
export interface RestartRequest {
|
|
requestId: string;
|
|
kimiCandidateAdmission?: AcpxCandidateAdmission;
|
|
controllerDowntimeMs?: number;
|
|
preserveRunner?: RestartRunnerIdentity;
|
|
discardControllerCache?: boolean;
|
|
crashAfterResult?: FinalizationCrashBinding;
|
|
controllerOutage?: ControllerOutage;
|
|
rollingDeployment?: true;
|
|
recoveryBinding?: RecoveryBinding;
|
|
}
|
|
|
|
export function kimiCandidateAdmissionForExecution(execution: MatrixExecution, companyId: string, now = Date.now()): AcpxCandidateAdmission | undefined {
|
|
if (execution.suite.id !== "kimi-qualification" || execution.profile.qualificationCandidate !== "kimi"
|
|
|| QUALIFIED_ACPX_PROFILES.kimi.qualificationStatus !== "pending") return undefined;
|
|
if (!execution.suite.manualOnly || execution.profile.id !== "runner-acpx-kimi" || !/^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/.test(companyId)) {
|
|
throw new Error("Kimi qualification requires an exact disposable company and manual profile");
|
|
}
|
|
const target = execution.environment.expectedExecutionTarget.kind === "remote" ? "linux-x64" : `${process.platform}-${process.arch}`;
|
|
if (target !== "darwin-arm64" && target !== "linux-x64") throw new Error("Kimi fixture host is unsupported");
|
|
const [platform, architecture] = target.split("-");
|
|
return parseAcpxCandidateAdmission({ schema: "paperclip.acpx-candidate-admission.v1", companyId, agent: "kimi",
|
|
model: execution.profile.model, profileDigest: QUALIFIED_ACPX_PROFILES.kimi.commandDigest,
|
|
nativeClosureDigest: `sha256:${KIMI_DISTRIBUTION_CLOSURE_SHA256[target]}`, platform, architecture,
|
|
expiresAt: now + 60 * 60 * 1000 }, { companyId, agent: "kimi", model: execution.profile.model }, now);
|
|
}
|
|
|
|
function selectedKimiCandidateAdmission(value: unknown, executionIds: readonly string[]): AcpxCandidateAdmission {
|
|
const executions = executionIds.map(id => runnerMatrix.find(row => row.id === id));
|
|
if (!executions.length || executions.some(row => !row || row.suite.id !== "kimi-qualification"
|
|
|| !row.suite.manualOnly || row.profile.id !== "runner-acpx-kimi" || row.profile.qualificationCandidate !== "kimi")) {
|
|
throw new Error("Kimi authority staging requires an explicit Kimi qualification selection");
|
|
}
|
|
const record = value as Partial<AcpxCandidateAdmission> | null;
|
|
if (!record || typeof record.companyId !== "string" || !/^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/.test(record.companyId)) throw new Error("Invalid Kimi fixture company");
|
|
const admission = parseAcpxCandidateAdmission(value, { companyId: record.companyId, agent: "kimi", model: executions[0]!.profile.model });
|
|
if (!executions.some(row => row!.environment.expectedExecutionTarget.kind === "remote"
|
|
? admission.platform === "linux" && admission.architecture === "x64"
|
|
: admission.platform === process.platform && admission.architecture === process.arch)) throw new Error("Kimi fixture target does not match the selected execution");
|
|
return admission;
|
|
}
|
|
|
|
export async function readRestartRequest(restartRequestPath: string, executionIds: readonly string[]): Promise<RestartRequest | null> {
|
|
let encoded: string;
|
|
try {
|
|
encoded = await readFile(restartRequestPath, "utf8");
|
|
} catch (error) {
|
|
if ((error as NodeJS.ErrnoException).code === "ENOENT") return null;
|
|
throw error;
|
|
}
|
|
let value: unknown;
|
|
try {
|
|
value = JSON.parse(encoded);
|
|
} catch {
|
|
// The writer may not have completed its atomic replacement yet.
|
|
return null;
|
|
}
|
|
if (!value || typeof value !== "object" || Array.isArray(value)) return null;
|
|
const requestId = (value as { requestId?: unknown }).requestId;
|
|
if (
|
|
typeof requestId !== "string" ||
|
|
!/^[A-Za-z0-9._:-]{1,200}$/.test(requestId)
|
|
) {
|
|
return null;
|
|
}
|
|
const kimiAuthority = (value as { kimiCandidateAdmission?: unknown }).kimiCandidateAdmission;
|
|
const kimiCandidateAdmission = kimiAuthority === undefined ? undefined : selectedKimiCandidateAdmission(kimiAuthority, executionIds);
|
|
if (kimiCandidateAdmission && Object.keys(value).some(key => !["requestId", "kimiCandidateAdmission", "discardControllerCache"].includes(key))) {
|
|
throw new Error("Kimi startup authority staging cannot combine another restart operation");
|
|
}
|
|
if (kimiCandidateAdmission && (value as { discardControllerCache?: unknown }).discardControllerCache === true) {
|
|
throw new Error("Kimi startup authority staging cannot replace controller state");
|
|
}
|
|
const preserveRunner = (value as { preserveRunner?: unknown }).preserveRunner;
|
|
const controllerDowntimeMs = (value as { controllerDowntimeMs?: unknown }).controllerDowntimeMs;
|
|
if (controllerDowntimeMs !== undefined && (!Number.isSafeInteger(controllerDowntimeMs)
|
|
|| Number(controllerDowntimeMs) < 0 || Number(controllerDowntimeMs) > 300_000)) {
|
|
throw new Error("Controller downtime must be an integer between zero and five minutes");
|
|
}
|
|
const rollingDeployment = (value as { rollingDeployment?: unknown }).rollingDeployment;
|
|
if (rollingDeployment !== undefined && (rollingDeployment !== true || executionIds.length !== 1
|
|
|| !executionIds.every(id => { const execution = runnerMatrix.find(row => row.id === id); return execution && isRollingWebDeployment(execution); }))) {
|
|
throw new Error("Rolling deployment requires its explicit isolated cell");
|
|
}
|
|
const discardControllerCache = (value as { discardControllerCache?: unknown }).discardControllerCache === true;
|
|
if (discardControllerCache && !executionIds.some(id => {
|
|
const execution = runnerMatrix.find(row => row.id === id);
|
|
return execution !== undefined && (controllerOutageForExecution(execution, {}) !== undefined || isRollingWebDeployment(execution));
|
|
})) throw new Error("Controller cache replacement requires an explicit remote controller-outage cell");
|
|
const crashAfterResult = (value as { crashAfterResult?: unknown }).crashAfterResult;
|
|
const controllerOutage = (value as { controllerOutage?: unknown }).controllerOutage;
|
|
const recoveryBinding = (value as { recoveryBinding?: unknown }).recoveryBinding;
|
|
if (recoveryBinding !== undefined && controllerOutage === undefined && !rollingDeployment) throw new Error("Recovery observation requires a controller outage");
|
|
if (rollingDeployment && (!discardControllerCache || controllerOutage !== undefined || controllerDowntimeMs !== undefined
|
|
|| preserveRunner !== undefined || crashAfterResult !== undefined)) throw new Error("Rolling deployment cannot combine an outage or another fault");
|
|
if (controllerOutage !== undefined && (!discardControllerCache || preserveRunner !== undefined || crashAfterResult !== undefined)) {
|
|
throw new Error("Controller outage requires the explicit remote cache-replacement cell without another crash boundary");
|
|
}
|
|
if (crashAfterResult !== undefined && (preserveRunner !== undefined || discardControllerCache)) {
|
|
throw new Error("A finalization crash cannot preserve a running provider or discard its controller cache");
|
|
}
|
|
return { requestId, discardControllerCache, ...(rollingDeployment ? { rollingDeployment: true as const } : {}), ...(kimiCandidateAdmission ? { kimiCandidateAdmission } : {}), ...(controllerDowntimeMs === undefined ? {} : { controllerDowntimeMs: Number(controllerDowntimeMs) }),
|
|
...(recoveryBinding === undefined ? {} : { recoveryBinding: parseRecoveryBinding(recoveryBinding) }),
|
|
...(controllerOutage === undefined ? {} : { controllerOutage: parseControllerOutage(controllerOutage) }),
|
|
...(preserveRunner === undefined ? {} : { preserveRunner: parseRestartRunnerIdentity(preserveRunner) }),
|
|
...(crashAfterResult === undefined ? {} : { crashAfterResult: parseFinalizationCrashBinding(crashAfterResult) }) };
|
|
|
|
}
|