mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-04 02:32:07 +01:00
fix(server): wake native workspace finalization from committed work (#17734)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs can accept a result before workspace copy-back finishes. > - The periodic orphan pass repeatedly tries this recovery, including cases that require operator action. > - Copy-back needs exact ownership evidence. A generic timer cannot grant that authority. > - This pull request wakes workspace recovery from committed changes and local owner completion. > - Saved retries use the existing scheduler. Empty work, completed barriers, and operator holds become quiet. ## Linked Issues or Issue Description Refs #17717. This is slice 52 of the recurring-server sleep stack, targeting master after parent #17717 merged at `4c56702e06c698cc10d4a86514669aba0f4d9a9b`. Related work: #17709 (status reassessment), #17687 (unsafe export recovery), and #16478 (workspace repair holds). This PR covers native workspace copy-back and validation. Result commit/projection and resource settlement remain separate recovery duties. It does not claim full-instance sleep. ## What Changed - Register a company/run/task-scoped workspace worker. Reconstruct accepted, unfinished work from existing records. Use saved retry and lease deadlines. - Notify from both accepted-result writers (control-plane and PRP coordinator), workspace receipts, coordinator changes, operator confirmation, retained workspace repair, and relevant task updates. Notifications follow the outer transaction. - Distinguish active local ownership, busy external ownership, pending Cloud evidence, operator-required evidence, and changed bindings. Only concrete external ownership/evidence waits receive a scoped retry. - Preserve advisory locks, signed stop proof, exact binding checks, physical callback joins, and existing failure policy. Join physical operations during shutdown. - Respect the existing heartbeat execution-suppression policy before discovery and execution; wake suppressed work on settings activation. - Skip physical workspace recovery in the enabled periodic orphan pass. Keep direct/startup reconciliation and the remaining result/cleanup recovery paths. - Keep routine company-only finalization hints quiet. Ordinary work requires issue/run scope; explicit settings and owner-release events keep their company-wide contract. - Preserve the parent's entire Runner/package tree, profile 80 and all historical identities. Add the dated obligation, producer, lifecycle and limitation audit. ## Verification - Current local source: `4059b24e3ea5c623be54ade18ec9b6ab78eb0289`, prepared by a normal merge of the current parent. The sole manual conflict keeps ordinary workspace fanout inside the existing issue/run guard. All 1,198 parent-only, 15 owned-only and 10 automatic shared paths are preserved. The complete package tree matches the parent byte-for-byte. - Controlled regression: the original workspace-only unscoped fanout performs three candidate SELECTs for routine company hints. The same zero-query assertion passes with scoped fanout. A real accepted result then produces one successful workspace settlement and returns to quiet. The failed baseline remains in the audit. - The first complete local group passes **1,693 cases in 14 files**, including all 19 workspace-worker cases. It covers both accepted-result producers, physical ownership, export retry/failure, finalization recovery, arbiter corpus, issue recovery actions, control-plane port, workspace recovery, heartbeat recovery, startup, retained cleanup and native execution. - Separate process-recovery and external-chat response files qualify the changed recovery/settlement paths without sharing their general test database with other files. Both pass in full: 410 process-recovery cases in 162.79 seconds and 273 external-chat cases in 53.79 seconds. The total is **2,376 distinct passes in 16 complete files**, with no skips. Normal merge `5c46c52873158529736b8349e4d44658865108ee` adopts the parent squash with this entire tested child tree unchanged. - Production server typechecking, build, module boundaries, migration order and source-profile checks pass. Thirteen dedicated PostgreSQL 17 fixtures use the unchanged migration 413 schema. Embedded-only paths use the existing recovery helper with fresh per-call fixtures. All temporary imports are restored byte-exact. No test assertion or timeout is relaxed. - Runner validation belongs to the unchanged parent package tree: 130 passing installation/compatibility/transport cases, 22 installed ACPX/Grok contract cases and 26 native-provider Rust cases. One existing opt-in closed Pi snapshot case is skipped because its distribution root is not configured. These are source-scoped checks, with no live provider qualification claim. - The owner audit at `doc/plans/2026-10-26-native-workspace-finalization-sleep-audit.md` preserves earlier failures and their disposition. Current-head acceptance: all 57 checks are clean, including protected CI and security. Greptile gives 5/5 on `7df5c9c44ea9e171be3c61b4430d3727b10f3337` with no actionable findings. Earlier scores do not substitute for this result. No Ponytail verdict is present under the authorized missing-review policy. - No claim of a full local repository suite, deployed app/database sleep, expanded Cloud wake coverage or measured cost savings is made. ## Risks - Supported wake producers are in-process application services. This adds no cross-replica or independent SQL-writer transport. - A real unresolved export still needs recovery. Busy owners and pending Cloud stop evidence retain a 30-second scoped check. No timeout authorizes copy-back. - Result projection and resource release still use their existing recovery. Some existing cleanup failures remain best effort; this PR does not infer cleanup success from a workspace barrier. - Cloud alarm coverage and host-sleep safety remain unchanged. No deployed suspension or cost savings have been measured. - Parent #17717 is merged. Current base is master at `4c56702e06c698cc10d4a86514669aba0f4d9a9b`. Fresh current-head CI and review must pass before this slice merges. ## Model Used OpenAI Codex, GPT-6 family, with repository tools and code execution. The exact serving model ID and context-window size are not exposed in this session. No subagent was used for this slice. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com>
This commit is contained in:
26 files changed
+987
-66
No files matched your search
@@ -2728,7 +2728,7 @@ describeEmbeddedPostgres("issue recovery actions", () => {
|
||||
const recovery = recoveryService(db, { enqueueWakeup: vi.fn(async () => null) });
|
||||
let firstAction: typeof issueRecoveryActions.$inferSelect | undefined;
|
||||
for (let tick = 0; tick < 3; tick += 1) {
|
||||
expect(await withNativeWorkspaceFinalizationOwnership({ db, companyId, runId }, copyback)).toEqual({ acquired: false });
|
||||
expect(await withNativeWorkspaceFinalizationOwnership({ db, companyId, runId }, copyback)).toEqual({ acquired: false, reason: "owner_unverified" });
|
||||
const detail = await request(app).get(`/api/issues/${sourceIssueId}`).expect(200);
|
||||
expect(detail.body.activeRecoveryAction).toMatchObject({
|
||||
cause: "native_workspace_finalization_owner_unverified", status: "active", attemptCount: 1,
|
||||
|
||||
@@ -0,0 +1,317 @@
|
||||
import { heartbeatService } from "../services/heartbeat.js";
|
||||
import { instanceSettingsService } from "../services/instance-settings.js";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import fs from "node:fs/promises";
|
||||
import { mkdtemp, rm } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { setTimeout as delay } from "node:timers/promises";
|
||||
import { and, eq } from "drizzle-orm";
|
||||
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { agents, companies, completionContracts, createDb, heartbeatRuns, issues, nativeRunFinalizations,
|
||||
nativeRunResults, issueRecoveryActions, workspaceOperations, type Db } from "@paperclipai/db";
|
||||
import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js";
|
||||
import { issueService } from "../services/issues.js";
|
||||
import { nativeWorkspaceFinalizationWork } from "../services/native-runtime/native-workspace-finalization-work.js";
|
||||
import { withNativeWorkspaceFinalizationOwnership } from "../services/native-runtime/native-workspace-finalization-ownership.js";
|
||||
import { resumeNativeWorkspaceAfterOwnerStop } from "../services/native-runtime/native-workspace-owner-recovery.js";
|
||||
import * as platformProof from "../services/native-runtime/native-workspace-platform-proof.js";
|
||||
import { reconcileNativeFinalizations } from "../services/native-runtime/native-finalization-reconciler.js";
|
||||
import { NativeRunCoordinatorStore } from "../services/native-runtime/native-run-coordinator-store.js";
|
||||
import { PaperclipControlPlanePort } from "../services/native-runtime/paperclip-control-plane-port.js";
|
||||
import { notifyNativeFinalizationWork } from "../services/native-finalization-notifications.js";
|
||||
import { idleWorkSnapshot } from "../services/task-admission.js";
|
||||
import { CONTROL_PLANE_CONFORMANCE_RESULT, CONTROL_PLANE_CONFORMANCE_TERMINAL } from "../vendor/paperclip-runner/testing.js";
|
||||
import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
|
||||
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
|
||||
|
||||
describe("workspace finalization scheduled lifecycle", () => {
|
||||
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined;
|
||||
let db: ReturnType<typeof createDb>;
|
||||
let coordinator: ReturnType<typeof createDeliveryWorkCoordinator> | undefined;
|
||||
let worker: ReturnType<ReturnType<typeof nativeWorkspaceFinalizationWork>["register"]> | undefined;
|
||||
let root: string;
|
||||
const priorLogs = process.env.WORKSPACE_OPERATION_LOG_BASE_PATH;
|
||||
const companyId = randomUUID(), agentId = randomUUID();
|
||||
let admitted = true;
|
||||
const errors: unknown[] = [];
|
||||
const settled = vi.fn(async () => {});
|
||||
beforeAll(async () => {
|
||||
root = await mkdtemp(join(tmpdir(), "workspace-finalization-work-"));
|
||||
process.env.WORKSPACE_OPERATION_LOG_BASE_PATH = join(root, "logs");
|
||||
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_WORKSPACE_FINALIZATION_WORK_TEST_DATABASE_URL");
|
||||
if (!external) temporary = await startEmbeddedPostgresTestDatabase("workspace-finalization-work-");
|
||||
db = createDb(external ?? temporary!.connectionString);
|
||||
await db.insert(companies).values({ id: companyId, name: "Workspace finalization", issuePrefix: `WF${randomUUID().slice(0, 6)}` });
|
||||
await db.insert(agents).values({ id: agentId, companyId, name: "Workspace worker", adapterType: "paperclip_runner" });
|
||||
}, 30_000);
|
||||
beforeEach(() => {
|
||||
admitted = true; errors.length = 0; settled.mockClear();
|
||||
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true });
|
||||
});
|
||||
afterEach(async () => {
|
||||
worker?.stop(); await coordinator?.stop(); worker = undefined; coordinator = undefined;
|
||||
vi.useRealTimers(); vi.restoreAllMocks();
|
||||
await db.delete(nativeRunFinalizations).where(eq(nativeRunFinalizations.companyId, companyId));
|
||||
expect(errors).toEqual([]);
|
||||
});
|
||||
afterAll(async () => {
|
||||
await db?.$client.end({ timeout: 1 }); await temporary?.cleanup(); await rm(root, { recursive: true, force: true });
|
||||
if (priorLogs === undefined) delete process.env.WORKSPACE_OPERATION_LOG_BASE_PATH;
|
||||
else process.env.WORKSPACE_OPERATION_LOG_BASE_PATH = priorLogs;
|
||||
});
|
||||
async function start() {
|
||||
coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => admitted, canReconcile: () => admitted,
|
||||
onError: error => errors.push(error) });
|
||||
worker = nativeWorkspaceFinalizationWork(db, { onWorkspaceSettled: settled }).register(coordinator, () => admitted);
|
||||
await worker.ready;
|
||||
}
|
||||
async function flush() {
|
||||
await vi.waitFor(async () => {
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
expect(idleWorkSnapshot().active).toBe(0);
|
||||
expect(coordinator!.nextWakeAt()).toBeNull();
|
||||
}, { timeout: 15_000 });
|
||||
}
|
||||
async function quiet() {
|
||||
const previous = db.$client.options.debug;
|
||||
const queries: string[] = [];
|
||||
db.$client.options.debug = (_id, query) => { queries.push(query); };
|
||||
try { await vi.advanceTimersByTimeAsync(120_000); await delay(10); expect(queries).toEqual([]); }
|
||||
finally { db.$client.options.debug = previous; }
|
||||
}
|
||||
async function seed() {
|
||||
const runId = randomUUID(), issueId = randomUUID(), contractId = randomUUID(), sourceInstanceId = randomUUID();
|
||||
await db.insert(issues).values({ id: issueId, companyId, title: "Preserve completed work", status: "in_progress", assigneeAgentId: agentId });
|
||||
await db.insert(completionContracts).values({ id: contractId, companyId, issueId, revision: 1,
|
||||
schemaVersion: "paperclip.completion-contract.v1", policyVersion: "workspace-worker-v1", risk: "standard",
|
||||
completionAuthority: "server_arbiter", incompleteCriteriaPolicy: "preserve_non_terminal",
|
||||
contractJson: { revision: "workspace-worker-v1", objective: "Finish", criteria: [{ id: "objective", requirement: "Complete the task" }] },
|
||||
canonicalSha256: contractId, createdByActorType: "system", createdByActorId: "test" });
|
||||
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, runtimeMode: "native", status: "running",
|
||||
nativeIssueId: issueId, nativeSessionId: runId, runnerInstanceId: sourceInstanceId,
|
||||
completionContractId: contractId, completionContractSha256: contractId, contextSnapshot: { issueId } });
|
||||
await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, issueId));
|
||||
|
||||
await db.insert(workspaceOperations).values({ companyId, issueId, heartbeatRunId: runId,
|
||||
phase: "workspace_finalize", status: "failed", cwd: root });
|
||||
return { companyId, issueId, runId, agentId, sessionId: runId, completionContractId: contractId,
|
||||
completionContractSha256: contractId, sourceInstanceId, controlPlaneSourceInstanceId: "workspace-worker-test" };
|
||||
}
|
||||
async function complete(binding: Awaited<ReturnType<typeof seed>>, writer: Db = db,
|
||||
producer: "control_plane" | "coordinator" = "control_plane") {
|
||||
const port = producer === "control_plane" ? new PaperclipControlPlanePort(writer, binding)
|
||||
: new NativeRunCoordinatorStore(writer, { ...binding, normalizedSessionId: binding.sessionId,
|
||||
runnerSourceInstanceId: binding.sourceInstanceId,
|
||||
completionContractRevision: CONTROL_PLANE_CONFORMANCE_RESULT.completionClaim.contractRevision,
|
||||
completionContractCriterionIds: CONTROL_PLANE_CONFORMANCE_RESULT.completionClaim.criteria.map(item => item.criterionId) });
|
||||
await port.completeRun({ result: CONTROL_PLANE_CONFORMANCE_RESULT, terminal: CONTROL_PLANE_CONFORMANCE_TERMINAL,
|
||||
callerResultId: `${binding.runId}:result` });
|
||||
}
|
||||
async function successful(runId: string) {
|
||||
return db.select().from(workspaceOperations).where(and(eq(workspaceOperations.heartbeatRunId, runId),
|
||||
eq(workspaceOperations.phase, "workspace_finalize"), eq(workspaceOperations.status, "succeeded")));
|
||||
}
|
||||
it.each(["restore", "worktree"])("honors %s suppression through the public heartbeat registration", async mode => {
|
||||
const binding = await seed(); await complete(binding);
|
||||
const runtimeEnv = mode === "restore" ? { PAPERCLIP_DATABASE_RESTORE_IN_PROGRESS: "true" }
|
||||
: { PAPERCLIP_IN_WORKTREE: "true", PAPERCLIP_INSTANCE_ID: "workspace-suppression-test" };
|
||||
const settings = instanceSettingsService(db, { runtimeEnv });
|
||||
if (mode === "worktree") await settings.updateExperimental({ enableWorktreeRunExecution: false });
|
||||
const heartbeat = heartbeatService(db, { runtimeEnv });
|
||||
coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => admitted, canReconcile: () => admitted,
|
||||
onError: error => errors.push(error) });
|
||||
worker = heartbeat.registerNativeWorkspaceFinalizationWork(coordinator, () => admitted);
|
||||
await worker.ready; await flush(); await quiet();
|
||||
expect(await successful(binding.runId)).toHaveLength(0);
|
||||
if (mode === "worktree") {
|
||||
try {
|
||||
await settings.updateExperimental({ enableWorktreeRunExecution: true });
|
||||
await flush(); expect(await successful(binding.runId)).toHaveLength(1); await quiet();
|
||||
} finally { await settings.updateExperimental({ enableWorktreeRunExecution: false }); }
|
||||
}
|
||||
});
|
||||
it.each(["control_plane", "coordinator"] as const)("is quiet with no work and retained history, then wakes on the first %s accepted result", async producer => {
|
||||
await start(); await flush(); await quiet();
|
||||
const binding = await seed();
|
||||
await complete(binding, db, producer); await flush();
|
||||
expect(await successful(binding.runId)).toHaveLength(1);
|
||||
expect(await db.select().from(nativeRunResults).where(eq(nativeRunResults.runId, binding.runId))).toHaveLength(1);
|
||||
expect(settled).not.toHaveBeenCalled();
|
||||
await quiet();
|
||||
worker!.stop(); await coordinator!.stop(); await start(); await flush(); await quiet();
|
||||
expect(await successful(binding.runId)).toHaveLength(1);
|
||||
});
|
||||
it("ignores routine company-only finalization hints and still wakes on accepted work", async () => {
|
||||
await start(); await flush(); await quiet();
|
||||
const previous = db.$client.options.debug;
|
||||
const queries: string[] = [];
|
||||
db.$client.options.debug = (_id, query) => {
|
||||
if (/from\s+"native_run_finalizations"/i.test(query)) queries.push(query);
|
||||
};
|
||||
try {
|
||||
for (let index = 0; index < 3; index += 1) {
|
||||
await db.transaction(async tx => { await notifyNativeFinalizationWork(tx, { companyId }); });
|
||||
await flush();
|
||||
}
|
||||
expect(queries).toEqual([]);
|
||||
} finally { db.$client.options.debug = previous; }
|
||||
const binding = await seed();
|
||||
await complete(binding); await flush();
|
||||
expect(await successful(binding.runId)).toHaveLength(1);
|
||||
await quiet();
|
||||
});
|
||||
|
||||
it("lets result reconciliation use a successful barrier despite a later stale failure", async () => {
|
||||
const binding = await seed(); await complete(binding);
|
||||
await db.insert(workspaceOperations).values({ companyId, heartbeatRunId: binding.runId, issueId: binding.issueId,
|
||||
phase: "workspace_finalize", status: "succeeded", cwd: root });
|
||||
await db.insert(workspaceOperations).values({ companyId, heartbeatRunId: binding.runId, issueId: binding.issueId,
|
||||
phase: "workspace_finalize", status: "failed", cwd: root, createdAt: new Date(Date.now() + 1000) });
|
||||
await start(); await flush(); await quiet();
|
||||
await reconcileNativeFinalizations(db, [binding.runId], { skipNativeWorkspaceFinalization: true });
|
||||
const [result] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, binding.runId));
|
||||
expect(result.phase).toBe("committed");
|
||||
expect(await successful(binding.runId)).toHaveLength(1);
|
||||
expect(await db.select().from(workspaceOperations).where(eq(workspaceOperations.heartbeatRunId, binding.runId))).toHaveLength(3);
|
||||
await flush(); await quiet();
|
||||
});
|
||||
it.each([
|
||||
["commit", "control_plane"], ["rollback", "control_plane"],
|
||||
["commit", "coordinator"], ["rollback", "coordinator"],
|
||||
] as const)("respects the outer transaction's %s for the real %s result producer", async (outcome, producer) => {
|
||||
await start(); await flush();
|
||||
const binding = await seed();
|
||||
const transaction = db.transaction(async tx => {
|
||||
await complete(binding, tx as unknown as Db, producer);
|
||||
await vi.advanceTimersByTimeAsync(5000);
|
||||
expect(await successful(binding.runId)).toHaveLength(0);
|
||||
if (outcome === "rollback") throw new Error("rollback fixture");
|
||||
});
|
||||
if (outcome === "rollback") await expect(transaction).rejects.toThrow("rollback fixture"); else await transaction;
|
||||
await flush();
|
||||
expect(await successful(binding.runId)).toHaveLength(outcome === "commit" ? 1 : 0);
|
||||
await quiet();
|
||||
});
|
||||
it.each(["startup", "first result"])("retains work during %s drain and resumes without a poll", async phase => {
|
||||
admitted = false;
|
||||
const binding = await seed();
|
||||
if (phase === "startup") await complete(binding);
|
||||
await start();
|
||||
if (phase === "first result") await complete(binding);
|
||||
await vi.advanceTimersByTimeAsync(120_000);
|
||||
expect(await successful(binding.runId)).toHaveLength(0);
|
||||
await quiet();
|
||||
admitted = true; await vi.advanceTimersByTimeAsync(30_000); await flush();
|
||||
expect(await successful(binding.runId)).toHaveLength(1); await quiet();
|
||||
});
|
||||
it("reconstructs a durable retry deadline and runs only when it is due", async () => {
|
||||
const binding = await seed(); await complete(binding);
|
||||
const due = new Date(Date.now() + 60_000);
|
||||
await db.update(nativeRunFinalizations).set({ phase: "retryable_failure", nextAttemptAt: due })
|
||||
.where(eq(nativeRunFinalizations.runId, binding.runId));
|
||||
await start();
|
||||
expect(coordinator!.nextWakeAt()).toBe(due.getTime());
|
||||
await vi.advanceTimersByTimeAsync(30_000);
|
||||
expect(await successful(binding.runId)).toHaveLength(0);
|
||||
worker!.stop(); await coordinator!.stop(); await start();
|
||||
expect(coordinator!.nextWakeAt()).toBe(due.getTime());
|
||||
await vi.advanceTimersByTimeAsync(30_000); await flush();
|
||||
expect(await successful(binding.runId)).toHaveLength(1); await quiet();
|
||||
});
|
||||
it("waits quietly for the actual local owner to release", async () => {
|
||||
const binding = await seed(); await complete(binding);
|
||||
await withNativeWorkspaceFinalizationOwnership({ db, companyId, runId: binding.runId }, async () => {
|
||||
await start(); await flush(); await quiet();
|
||||
expect(await successful(binding.runId)).toHaveLength(0);
|
||||
});
|
||||
await flush(); expect(await successful(binding.runId)).toHaveLength(1); await quiet();
|
||||
});
|
||||
it("joins the actual copyback callback during shutdown", async () => {
|
||||
const binding = await seed(); await complete(binding);
|
||||
let enter!: () => void, release!: () => void;
|
||||
const entered = new Promise<void>(resolve => { enter = resolve; });
|
||||
const held = new Promise<void>(resolve => { release = resolve; });
|
||||
const stat = fs.stat;
|
||||
vi.spyOn(fs, "stat").mockImplementation(async (...args) => {
|
||||
if (args[0] === root) { enter(); await held; }
|
||||
return stat(...args);
|
||||
});
|
||||
const starting = start();
|
||||
try {
|
||||
await entered;
|
||||
let stopped = false;
|
||||
const stopping = coordinator!.stop().then(() => { stopped = true; });
|
||||
await delay(20);
|
||||
expect(stopped).toBe(false);
|
||||
expect(await successful(binding.runId)).toHaveLength(0);
|
||||
release(); await starting; await stopping;
|
||||
expect(stopped).toBe(true); expect(await successful(binding.runId)).toHaveLength(1);
|
||||
await quiet();
|
||||
} finally { release(); await starting; }
|
||||
});
|
||||
it("retains an unverifiable owner without recurring SQL or physical writes", async () => {
|
||||
const binding = await seed(); await complete(binding);
|
||||
const owner = { token: randomUUID(), hostname: "another-controller", pid: 1234, processStartedAt: new Date().toISOString() };
|
||||
await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeWorkspaceFinalizationOwner: owner } }).where(eq(heartbeatRuns.id, binding.runId));
|
||||
await start(); await flush(); await quiet();
|
||||
expect(await successful(binding.runId)).toHaveLength(0);
|
||||
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, binding.runId)))[0].runnerProfileJson?.nativeWorkspaceFinalizationOwner).toEqual(owner);
|
||||
|
||||
await db.transaction(async tx => { await notifyNativeFinalizationWork(tx, binding); });
|
||||
await flush(); await quiet();
|
||||
expect(await successful(binding.runId)).toHaveLength(0);
|
||||
const [action] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, binding.issueId));
|
||||
await resumeNativeWorkspaceAfterOwnerStop({ ...binding, db, actionId: action.id, ownerToken: owner.token,
|
||||
controllerAndCopybackStopped: true, stopEvidence: "The platform confirms this exact deployment and all copyback processes stopped.", actorId: "operator" });
|
||||
await flush(); expect(await successful(binding.runId)).toHaveLength(1); await quiet();
|
||||
});
|
||||
|
||||
it("retries a concrete Cloud stop verification and disarms after verified copyback", async () => {
|
||||
const binding = await seed(); await complete(binding);
|
||||
const deployment = { provider: "railway" as const, projectId: "p", environmentId: "e", serviceId: "s", deploymentId: "old" };
|
||||
const owner = { token: randomUUID(), hostname: "another-controller", pid: 1234, processStartedAt: new Date().toISOString(), worker: deployment };
|
||||
await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeWorkspaceFinalizationOwner: owner } }).where(eq(heartbeatRuns.id, binding.runId));
|
||||
const probe = vi.spyOn(platformProof, "requestWorkspaceWorkerStopProof").mockResolvedValueOnce({ status: "pending" })
|
||||
.mockResolvedValue({ status: "verified", proof: { v: 1, iss: "paperclip-cloud", aud: "paperclip-workspace-worker-stop/v1",
|
||||
sub: "stack", runId: binding.runId, ownerToken: owner.token, worker: deployment, stopped: true, iat: 1, exp: 61 } });
|
||||
await start();
|
||||
expect(probe).toHaveBeenCalledOnce();
|
||||
expect(coordinator!.nextWakeAt()).not.toBeNull();
|
||||
await vi.advanceTimersByTimeAsync(15_000);
|
||||
expect(probe).toHaveBeenCalledOnce(); expect(await successful(binding.runId)).toHaveLength(0);
|
||||
await vi.advanceTimersByTimeAsync(15_000); await flush();
|
||||
expect(probe).toHaveBeenCalledTimes(2); expect(await successful(binding.runId)).toHaveLength(1); await quiet();
|
||||
});
|
||||
|
||||
it("stays quiet after a task binding changes and wakes when the task is restored", async () => {
|
||||
const binding = await seed(); await complete(binding);
|
||||
const deployment = { provider: "railway" as const, projectId: "p", environmentId: "e", serviceId: "s", deploymentId: "old" };
|
||||
const owner = { token: randomUUID(), hostname: "another-controller", pid: 1234, processStartedAt: new Date().toISOString(), worker: deployment };
|
||||
await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeWorkspaceFinalizationOwner: owner } }).where(eq(heartbeatRuns.id, binding.runId));
|
||||
await db.update(issues).set({ status: "done" }).where(eq(issues.id, binding.issueId));
|
||||
const probe = vi.spyOn(platformProof, "requestWorkspaceWorkerStopProof").mockResolvedValue({ status: "verified", proof: {
|
||||
v: 1, iss: "paperclip-cloud", aud: "paperclip-workspace-worker-stop/v1", sub: "stack", runId: binding.runId,
|
||||
ownerToken: owner.token, worker: deployment, stopped: true, iat: 1, exp: 61,
|
||||
} });
|
||||
await start(); await flush(); await quiet();
|
||||
expect(probe).toHaveBeenCalledOnce(); expect(await successful(binding.runId)).toHaveLength(0);
|
||||
await issueService(db).update(binding.issueId, { status: "in_progress" });
|
||||
await flush(); expect(await successful(binding.runId)).toHaveLength(1); await quiet();
|
||||
});
|
||||
it("records a failed validation once, honors its saved retry, and settles after repair", async () => {
|
||||
const binding = await seed();
|
||||
await db.update(workspaceOperations).set({ cwd: join(root, "absent") }).where(eq(workspaceOperations.heartbeatRunId, binding.runId));
|
||||
await complete(binding); await start();
|
||||
const [failed] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, binding.runId));
|
||||
expect(failed.phase).toBe("retryable_failure"); expect(failed.nextAttemptAt).not.toBeNull();
|
||||
await vi.waitFor(async () => {
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
expect(coordinator!.nextWakeAt()).toBe(failed.nextAttemptAt!.getTime());
|
||||
});
|
||||
expect(await successful(binding.runId)).toHaveLength(0);
|
||||
await db.update(workspaceOperations).set({ cwd: root }).where(eq(workspaceOperations.heartbeatRunId, binding.runId));
|
||||
await vi.advanceTimersByTimeAsync(Math.max(0, failed.nextAttemptAt!.getTime() - Date.now())); await flush();
|
||||
expect(await successful(binding.runId)).toHaveLength(1); await quiet();
|
||||
});
|
||||
});
|
||||
@@ -118,6 +118,7 @@ const {
|
||||
registerScheduledRetryCancellationWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })),
|
||||
registerStoppedInstructionCollectionWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })),
|
||||
registerRetainedNativeSessionCleanupWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve(), stop: vi.fn() })),
|
||||
registerNativeWorkspaceFinalizationWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve(), stop: vi.fn() })),
|
||||
registerNativeStatusReassessmentWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })),
|
||||
registerNativeReviewResponseWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })),
|
||||
registerNativeResponsePresentationWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve(), stop: vi.fn() })),
|
||||
@@ -574,6 +575,17 @@ describe("startServer feedback export wiring", () => {
|
||||
}
|
||||
});
|
||||
|
||||
it.each([false, true])("registers workspace finalization with its shutdown subscription only when maintenance is enabled (%s)", async enabled => {
|
||||
loadConfigMock.mockReturnValue(buildTestConfig({ heartbeatSchedulerEnabled: enabled }));
|
||||
await startServer(); await settleStartupHousekeeping();
|
||||
expect(heartbeatServiceMock.registerNativeWorkspaceFinalizationWork).toHaveBeenCalledTimes(enabled ? 1 : 0);
|
||||
if (enabled) {
|
||||
expect(heartbeatServiceMock.registerNativeWorkspaceFinalizationWork.mock.calls[0][1]()).toBe(true);
|
||||
const worker = heartbeatServiceMock.registerNativeWorkspaceFinalizationWork.mock.results[0].value;
|
||||
expect(fakeServer.on).toHaveBeenCalledWith("close", worker.stop);
|
||||
}
|
||||
});
|
||||
|
||||
it.each([false, true])("registers retained cleanup with its shutdown subscription only when maintenance is enabled (%s)", async enabled => {
|
||||
loadConfigMock.mockReturnValue(buildTestConfig({ heartbeatSchedulerEnabled: enabled }));
|
||||
await startServer(); await settleStartupHousekeeping();
|
||||
@@ -1046,7 +1058,7 @@ describe("startServer feedback export wiring", () => {
|
||||
intervalCallback?.();
|
||||
await new Promise<void>((resolve) => setImmediate(resolve));
|
||||
expect(heartbeatServiceMock.sweepStaleIssueLocks).toHaveBeenCalledTimes(2);
|
||||
expect(heartbeatServiceMock.reapOrphanedRuns).toHaveBeenLastCalledWith({ staleThresholdMs: 300_000, skipScheduledRetryCancellation: true, skipCapturedInstructionCommit: true, skipInstructionDirectoryCleanup: true, skipStoppedInstructionCollection: true, skipCancelledWorkspacePreservation: true, skipObsoletePolicyReviews: true, skipAutomaticCompletionReviews: true, skipUnsafeWorkspaceExports: true, skipNativePendingEffects: true, skipNativeResponsePresentation: true, skipNativeReviewResponse: true, skipNativeStatusReassessment: true, skipRetainedNativeSessionCleanup: true });
|
||||
expect(heartbeatServiceMock.reapOrphanedRuns).toHaveBeenLastCalledWith({ staleThresholdMs: 300_000, skipScheduledRetryCancellation: true, skipCapturedInstructionCommit: true, skipInstructionDirectoryCleanup: true, skipStoppedInstructionCollection: true, skipCancelledWorkspacePreservation: true, skipObsoletePolicyReviews: true, skipAutomaticCompletionReviews: true, skipUnsafeWorkspaceExports: true, skipNativePendingEffects: true, skipNativeResponsePresentation: true, skipNativeReviewResponse: true, skipNativeStatusReassessment: true, skipRetainedNativeSessionCleanup: true, skipNativeWorkspaceFinalization: true });
|
||||
expect(heartbeatServiceMock.reconcileCostAccounting).toHaveBeenCalledTimes(1);
|
||||
expect(retiredDetector).not.toHaveBeenCalled();
|
||||
} finally {
|
||||
@@ -1202,19 +1214,21 @@ describe("startServer feedback export wiring", () => {
|
||||
}
|
||||
});
|
||||
|
||||
it("becomes ready while retained cleanup is held but joins it at shutdown", async () => {
|
||||
it.each(["retained cleanup", "workspace finalization"] as const)("becomes ready while %s is held but joins it at shutdown", async owner => {
|
||||
loadConfigMock.mockReturnValue(buildTestConfig({ heartbeatSchedulerEnabled: true }));
|
||||
let releaseCleanup!: () => void;
|
||||
const ready = new Promise<void>(resolve => { releaseCleanup = resolve; });
|
||||
const stop = vi.fn();
|
||||
heartbeatServiceMock.registerRetainedNativeSessionCleanupWork.mockReturnValueOnce({ ready, stop });
|
||||
const register = owner === "retained cleanup" ? heartbeatServiceMock.registerRetainedNativeSessionCleanupWork
|
||||
: heartbeatServiceMock.registerNativeWorkspaceFinalizationWork;
|
||||
register.mockReturnValueOnce({ ready, stop });
|
||||
const timer = vi.spyOn(globalThis, "setInterval").mockImplementation(() => 1 as unknown as ReturnType<typeof setInterval>);
|
||||
let started = false;
|
||||
const starting = startServer().then(server => { started = true; return server; });
|
||||
let stopping: Promise<void> | undefined;
|
||||
let stopped = false;
|
||||
try {
|
||||
await vi.waitFor(() => expect(heartbeatServiceMock.registerRetainedNativeSessionCleanupWork).toHaveBeenCalledOnce());
|
||||
await vi.waitFor(() => expect(register).toHaveBeenCalledOnce());
|
||||
await settleStartupHousekeeping();
|
||||
expect(started).toBe(true);
|
||||
expect(getStartupRecoveryState().phase).toBe("ready");
|
||||
|
||||
Reference in new issue
Block a user