diff --git a/doc/plans/2026-10-31-zero-downtime-web-deploys.md b/doc/plans/2026-10-31-zero-downtime-web-deploys.md index bc9747378cd..71e4bf658b2 100644 --- a/doc/plans/2026-10-31-zero-downtime-web-deploys.md +++ b/doc/plans/2026-10-31-zero-downtime-web-deploys.md @@ -1,9 +1,10 @@ # Zero-downtime web deployments -Date: 2026-10-31. Status (2026-11-01 14:54 UTC): existing-volume staging enrollment +Date: 2026-10-31. Status (2026-11-01 15:39 UTC): existing-volume staging enrollment and manual sleep/wake preserve saved files and read/save behavior. The full local -active-agent journey, natural database idling and ten-deploy staging campaign -remain unqualified. No production canary yet. +active-agent journey now passes on `cc23019691`; hosted browser continuity and +the ten-deploy staging campaign remain unqualified. Natural database autosuspension +passes after abandoned test-resource cleanup. No production canary yet. Core #18543 merged as `7751fc3107` after 55 successful CI checks, Greptile 5/5 and no unresolved threads. It supplies runtime admission, immutable-cache @@ -56,7 +57,51 @@ readiness. All 33 readiness, managed-skill and durable-file tests pass, with ser typecheck. The failed attempt remains failed evidence; its child credential, bucket and all three provider sandboxes were cleaned and checked. Review also replaced the whole agent-file manifest read with a scalar SQL digest projection. -The next live attempt must exercise both replacements and later continuation. +PR #18552 merged as `c0550e7010` with all required CI green, Greptile 5/5, +and its scalar-query review findings resolved. The real `a15a12651c` attempt +passed both web replacements: 765 reads, 78 saves and 78 verified saves with +zero HTTP probe failures. The active execution survived once; the first three +turns retained the native/provider session and accepted terminal contract. +The fourth continuation FAILED: the retained Runner was safely suspended with +an empty outbox and no pending terminal, but warm recovery attempted live-idle +reattachment. Its provider state and diagnostics are preserved; all three owned +sandboxes were independently confirmed absent. The next bounded fix reuses the +existing per-turn suspended-receipt restoration for warm admission after cache +loss, preserving physical sandbox binding and ordinary admission fences. +A real PostgreSQL/Linux Runner regression reproduces the exact live error before +the fix. Recovery then exposed a second stale-cache read: source reconciliation +tried to stage new launch assets that the transport deliberately had not created. +The existing remote process launcher now passes its structured launch identity +to asset preparation, so reconciliation preserves the previous authority's +retained launch files. Both new regressions pass: three turns preserve one +provider thread after two cache deletions, including a failed remote state read +and retry. All ten Linux continuity scenarios and 1,067 surrounding boundary tests pass. +The exact `cc23019691` server build passes. PR #18554 has Greptile 5/5; CI and +its second review identified a narrower dispatch boundary: warm receipt +inspection must leave every non-suspended state to the existing live-adoption +checks. A new preliminary-read regression fails before that refinement. The +refinement plus both suspension regressions pass in a fresh Linux fixture, and +server typecheck passes. The first retry reused a container whose deliberately +failed pre-fix case left its live Runner on the ingress port; that infrastructure +failure is preserved. The local wrapper now creates and cleans one container per +invocation. This does not weaken the existing idle check or claim support for +adopting a persistently backpressured Runner. + +The live Codex/Daytona attempt at that clean source PASSED on its first attempt, +with no automatic retries: 518 reads, 53 saves and 53 verified saves across both +replacements, zero probe failures, both complete predecessor homes deleted. +The active execution resumes under the replacement and completes once. All four +turns use the same native/provider session with accepted native terminal records. +The final turn runs the saved project and returns the independently checked +output. The original Git commit, untracked workspace reference, prior downloads +and revised files survive. Browser reconnection remains explicitly unqualified +in this API-led continuation case and must pass in the hosted campaign. +Child credential, bucket and local test-home cleanup passed. Independent provider +verification found two archived setup sandboxes left by the harness; those exact +owned residuals were deleted separately and independently confirmed absent, +preserving the failed automatic-cleanup verification. The product journey pass does not prove automatic fixture cleanup. The attempt's +reported-plus-estimated model/runtime cost is about $0.243. Its immutable evidence +is `zdt-live-2026-11-01T15-24-43-429Z` under the private live-evidence directory. Review also identified save quotas incorrectly applied during cache recovery. Recovery uses the existing inbound remote projection and preserves path/link @@ -77,7 +122,13 @@ this is migration continuity evidence, not uninterrupted old-tab asset loading. Three eight-minute natural database-idle observations failed. A separate bounded SQL attribution window saw no other app-database sessions; it is diagnostic stimulus, -not idle proof, and does not establish the source of provider activity. The third window, bracketed by closed SQL counter snapshots, recorded 36 transactions and small read deltas on environment setup, leases, environments and plugins; no app-row writes. The source is not yet attributed. Automatic +not idle proof, and does not establish the source of provider activity. The third window, bracketed by closed SQL counter snapshots, recorded 36 transactions and small read deltas on environment setup, leases, environments and plugins; no app-row writes. A later scoped diagnostic found an abandoned ephemeral setup sandbox still +archived at the provider despite a historical cleanup-success row. The normal +verification worker revisited it every five minutes. After deleting that exact +owned sandbox, the product independently stamped its destroyed-resource proof; +no historical row or cleanup evidence was manually rewritten. The fresh provider-metadata-only observation passed: the same single volume-free +web deployment remained serving while Neon automatically suspended after its +five-minute idle timeout. No app HTTP or SQL was used during the observation. Automatic idle sleep was correctly refused because this development fixture is not eligible for the free/demo policy. Manual sleep with force disabled passed: the registry reports sleeping, Railway has no active web deployment or attached volume, and @@ -151,9 +202,9 @@ Measure one-time setup separately from repeated attempts. Keep failed evidence. | Item | State | Next action | | --- | --- | --- | | Prior prerequisites | #18403 and all five storage workstreams merged | Retain the ephemeral filesystem contract | -| Core implementation | #18518, #18526, #18529, #18530, #18531 and #18533 merged; local packaged migration, browser overlap, cold restoration and durable command proofs recorded below | Complete live remote execution, accounting and later continuation qualification | +| Core implementation | #18518, #18526, #18529, #18530, #18531, #18533, #18543 and #18552 merged; #18554 reviewed at 5/5 with CI pending; local packaged migration, browser overlap, cold restoration and durable command proofs recorded below | Publish and qualify the exact fixed release after CI/review merge | | Cloud implementation | #1096, #1098, #1099, #1100, #1101 and #1102 merged; #1102 serving on staging | Verify repeated Railway overlap behavior after Core recovery qualification | -| Staging | Original enrollment complete; migrated data and signing key preserved; volume detached and retained | Diagnose idle behavior, verify sleep/wake and repair, then qualify ten app-only replacements | +| Staging | Original enrollment complete; migrated data and signing key preserved; volume detached and retained | Sleep/wake and natural database idle pass; verify repair and ten app-only replacements with browser continuity | | Production | No canary mutation performed | Proceed only after the staging and active-agent acceptance gates pass | Keep PRs, tested revisions, artifact identities, evidence paths, deployment targets diff --git a/server/src/__tests__/helpers/remote-warm-controller.test-worker.ts b/server/src/__tests__/helpers/remote-warm-controller.test-worker.ts index 3e952c66126..160be9a352e 100644 --- a/server/src/__tests__/helpers/remote-warm-controller.test-worker.ts +++ b/server/src/__tests__/helpers/remote-warm-controller.test-worker.ts @@ -1,5 +1,5 @@ import { createDb } from "@paperclipai/db"; -import { executePaperclipNativeSession } from "../../services/native-runtime/native-session-executor.js"; +import { closeIdleWarmNativeSessionsForRestart, executePaperclipNativeSession } from "../../services/native-runtime/native-session-executor.js"; import { dockerCommandRunner } from "./docker-command-runner.js"; // Separate TypeScript fixture import time from actual controller admission. @@ -9,17 +9,34 @@ process.once("message", async (input: { connectionString: string; container: string; ingressUrl: string; + suspendAfterTurn?: boolean; + transientBackpressureInspection?: boolean; options: Parameters[0]; }) => { const target = input.options.runnerExecutionTarget; if (target?.kind !== "remote") throw new Error("Expected remote worker target"); try { + const runner = dockerCommandRunner(input.container); + let inspectionOverridden = false; + const execute = runner.execute.bind(runner); + runner.execute = async request => { + const result = await execute(request); + if (input.transientBackpressureInspection && !inspectionOverridden && result.exitCode === 0 + && request.args?.some(arg => arg.includes("base64 <") && arg.includes("runner-state.json"))) { + // Only the preliminary read reports transient pressure. The subsequent + // live-adoption owner must still inspect the actual Runner state. + const state = JSON.parse(Buffer.from(result.stdout.replace(/\s+/g, ""), "base64").toString("utf8")); + inspectionOverridden = true; + return { ...result, stdout: Buffer.from(JSON.stringify({ ...state, lifecycle: "backpressure" })).toString("base64") }; + } + return result; + }; const result = await executePaperclipNativeSession({ ...input.options, db: createDb(input.connectionString), runnerExecutionTarget: { ...target, - runner: dockerCommandRunner(input.container), + runner, getRunnerIngressEndpoint: async ({ path }) => { const endpoint = { kind: "authenticated_websocket" as const, @@ -31,7 +48,11 @@ process.once("message", async (input: { }, onLog: async (stream, text) => { process.send?.({ log: { stream, text } }); }, }); - process.send?.({ result }); + if (input.suspendAfterTurn) { + const closed = await closeIdleWarmNativeSessionsForRestart(); + if (closed.closed !== 1 || closed.busy || closed.failed) throw new Error("Warm fixture failed to suspend"); + } + process.send?.({ result: { ...result, inspectionOverridden } }); } catch (error) { process.send?.({ error: error instanceof Error ? error.stack : String(error) }); } diff --git a/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts b/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts index 82b90639a47..29e67300b86 100644 --- a/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts +++ b/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts @@ -358,10 +358,12 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( expect(next.kind).toBe("bootstrap_incomplete"); }); - it.skipIf(!process.env.PAPERCLIP_TEST_RUNNER_CONTAINER).each(["warm", "hard", "legacy", "changed", "same-server", "diskless", "diskless-per-turn", "diskless-per-turn-read-retry"])("preserves remote warm continuity across completed turns: %s", async (scenario) => { + it.skipIf(!process.env.PAPERCLIP_TEST_RUNNER_CONTAINER).each(["warm", "hard", "legacy", "changed", "same-server", "diskless", "diskless-per-turn", "diskless-per-turn-read-retry", "diskless-warm-suspended", "diskless-warm-suspended-read-retry", "diskless-live-inspection"])("preserves remote warm continuity across completed turns: %s", async (scenario) => { const hardRestart = !["warm", "same-server"].includes(scenario); const diskless = scenario.startsWith("diskless"); const perTurn = scenario.startsWith("diskless-per-turn"); + const warmSuspended = scenario.startsWith("diskless-warm-suspended"); + const resumedAfterSuspension = perTurn || warmSuspended; const container = process.env.PAPERCLIP_TEST_RUNNER_CONTAINER!; const tunnel = await createRunnerTestTlsTunnel({ port: Number(process.env.PAPERCLIP_TEST_RUNNER_PORT), key: process.env.PAPERCLIP_TEST_RUNNER_TLS_KEY!, cert: process.env.NODE_EXTRA_CA_CERTS! }); @@ -425,7 +427,7 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( onLog: async (_stream: "stdout" | "stderr", text: string) => { logs.push(text); }, }; let controller: ReturnType | undefined; - const executeInController = (executionOptions: Parameters[0]) => new Promise<{ exitCode: number | null }>((resolveResult, reject) => { + const executeInController = (executionOptions: Parameters[0]) => new Promise<{ exitCode: number | null; inspectionOverridden?: boolean }>((resolveResult, reject) => { controller = spawn(process.execPath, ["--import", createRequire(import.meta.url).resolve("tsx"), resolve("server/src/__tests__/helpers/remote-warm-controller.test-worker.ts")], { stdio: ["ignore", "pipe", "pipe", "ipc"], env: process.env }); let timer = setTimeout(() => { controller?.kill("SIGKILL"); reject(new Error("Controller fixture imports exceeded 120 seconds")); }, 120_000); @@ -437,7 +439,7 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( if (message.ready) { clearTimeout(timer); timer = setTimeout(() => { controller?.kill("SIGKILL"); reject(new Error(`Warm controller admission exceeded 60 seconds: ${logs.join("").slice(-3_000)}`)); }, 60_000); - controller!.send?.({ connectionString: temporary.connectionString, container, ingressUrl: `wss://127.0.0.1:${tunnel.port}`, + controller!.send?.({ connectionString: temporary.connectionString, container, suspendAfterTurn: warmSuspended, transientBackpressureInspection: scenario === "diskless-live-inspection" && executionOptions.execution.binding.runId !== fixture.runId, ingressUrl: `wss://127.0.0.1:${tunnel.port}`, options: { ...executionOptions, db: undefined, onLog: undefined, runnerExecutionTarget: { ...executionOptions.runnerExecutionTarget, runner: undefined } } }); } if (message.log) logs.push(message.log.text); @@ -520,7 +522,7 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( const snapshot = await runner.execute({ command: "sh", args: ["-c", `cat '${remoteCwd}'/.paperclip-runtime/paperclip-runner/sessions/*/runner/runner-state.json`] }); expect(snapshot.exitCode).toBe(0); const saved = JSON.parse(snapshot.stdout); - expect(saved.lifecycle).toBe(perTurn ? "suspended" : "ready"); + expect(saved.lifecycle).toBe(resumedAfterSuspension ? "suspended" : "ready"); const processBefore = await runner.execute({ command: "sh", args: ["-c", `cat '${remoteCwd}'/.paperclip-runtime/paperclip-runner/sessions/*/runner/runner-process.identity`] }); expect(processBefore.exitCode).toBe(0); const historyBefore = await runner.execute({ command: "sh", args: ["-c", `cat '${remoteCwd}'/.paperclip-runtime/paperclip-runner/sessions/*/filesystem/codex-home/fake-codex-state.json`] }); @@ -551,7 +553,7 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( (controller as ReturnType).kill("SIGKILL"); await exited; controller = undefined; - if (!perTurn) await expect(executePaperclipNativeSession({ ...nextOptions, + if (!resumedAfterSuspension) await expect(executePaperclipNativeSession({ ...nextOptions, runnerExecutionTarget: { ...nextOptions.runnerExecutionTarget, sandboxLeaseAcquisition: { outcome: "replacement", providerLeaseId: "unrelated-sandbox" } }, })).rejects.toThrow("prior_owner_terminal_state_indeterminate"); @@ -589,7 +591,7 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( ...(perTurn ? { sessionCheckpoint: admission.checkpoint } : {}) } }).where(eq(heartbeatRuns.id, row.id)); }; await admitContinuation(nextOptions.execution); - if (scenario === "diskless-per-turn-read-retry") { + if (scenario.endsWith("-read-retry")) { let rejectedReads = 0; await expect(executePaperclipNativeSession({ ...nextOptions, runnerExecutionTarget: { ...nextOptions.runnerExecutionTarget, runner: { ...runner, @@ -608,6 +610,7 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( } const second = diskless ? await executeInController(nextOptions) : await executePaperclipNativeSession(nextOptions); expect(second.exitCode).toBe(0); + if (scenario === "diskless-live-inspection") expect("inspectionOverridden" in second && second.inspectionOverridden).toBe(true); const historyAfter = await runner.execute({ command: "sh", args: ["-c", `cat '${remoteCwd}'/.paperclip-runtime/paperclip-runner/sessions/*/filesystem/codex-home/fake-codex-state.json`] }); const providerAfter = JSON.parse(historyAfter.stdout); expect(providerAfter.threadId).toBe(providerBefore.threadId); @@ -617,10 +620,10 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( expect(calls.stdout.split("\n").filter(line => line === "thread/start")).toHaveLength(1); const providerStarts = calls.stdout.split("\n").filter(line => line === "process-start").length; expect(providerStarts).toBeGreaterThanOrEqual(1); - expect(providerStarts).toBeLessThanOrEqual(perTurn || ["legacy", "changed"].includes(scenario) ? 3 : hardRestart ? 2 : 1); + expect(providerStarts).toBeLessThanOrEqual(resumedAfterSuspension || ["legacy", "changed"].includes(scenario) ? 3 : hardRestart ? 2 : 1); const processAfter = await runner.execute({ command: "sh", args: ["-c", `cat '${remoteCwd}'/.paperclip-runtime/paperclip-runner/sessions/*/runner/runner-process.identity`] }); expect(processAfter.exitCode).toBe(0); - if (perTurn || ["legacy", "changed"].includes(scenario)) expect(processAfter.stdout).not.toBe(processBefore.stdout); + if (resumedAfterSuspension || ["legacy", "changed"].includes(scenario)) expect(processAfter.stdout).not.toBe(processBefore.stdout); else expect(processAfter.stdout).toBe(processBefore.stdout); await finalizeNativeRun({ db: fixture.db, runId: nextRunId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true }); if (diskless && controller) { @@ -659,9 +662,9 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( const finalCalls = await runner.execute({ command: "cat", args: [`${remoteCwd}/provider-calls.log`] }); expect(finalCalls.stdout.split("\n").filter(line => line === "turn/start")).toHaveLength(3); expect(finalCalls.stdout.split("\n").filter(line => line === "thread/start")).toHaveLength(1); - expect(finalCalls.stdout.split("\n").filter(line => line === "process-start")).toHaveLength(providerStarts + (perTurn ? 1 : 0)); + expect(finalCalls.stdout.split("\n").filter(line => line === "process-start")).toHaveLength(providerStarts + (resumedAfterSuspension ? 1 : 0)); const finalProcess = await runner.execute({ command: "sh", args: ["-c", `cat '${remoteCwd}'/.paperclip-runtime/paperclip-runner/sessions/*/runner/runner-process.identity`] }); - if (perTurn) expect(finalProcess.stdout).not.toBe(processAfter.stdout); + if (resumedAfterSuspension) expect(finalProcess.stdout).not.toBe(processAfter.stdout); else expect(finalProcess.stdout).toBe(processAfter.stdout); expect(await closeIdleWarmNativeSessionsForRestart()).toEqual({ closed: perTurn ? 0 : 1, busy: 0, failed: 0 }); const suspended = await runner.execute({ command: "sh", args: ["-c", `cat '${remoteCwd}'/.paperclip-runtime/paperclip-runner/sessions/*/runner/runner-state.json`] }); diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index 452a290b23d..3837dc0e906 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -5881,12 +5881,21 @@ async function restorePerTurnRunnerdController(input: { root, nativeSessionKey(input.execution)); if (!restored && !identity) throw new Error("native_runner_checkpoint_missing"); } + await restoreRemoteSuspendedRunnerCheckpoint(input, root); +} + +/** The controller checkpoint and Runner lifecycle have separate authorities. + * Rehydrate a missing suspension receipt before choosing live warm adoption. */ +async function restoreRemoteSuspendedRunnerCheckpoint(input: { + db: Db; execution: NativeExecutionInput; + runnerExecutionTarget?: AdapterExecutionTarget | null; +}, root: string): Promise { const target = input.runnerExecutionTarget; const prior = readRunnerdDurableIdentity(root); // PRP and lifecycle evidence have different owners. A failed remote read can // leave a valid PRP projection, so its presence must not suppress this retry. // Never overwrite contradictory local Runner evidence with a remote receipt. - if (target.transport !== "sandbox" || target.sandboxLeaseAcquisition?.outcome !== "resumed" + if (target?.kind !== "remote" || target.transport !== "sandbox" || target.sandboxLeaseAcquisition?.outcome !== "resumed" || !durableIdentityMatchesSession(prior, input.execution) || prior.runId === input.execution.binding.runId || runnerdAuthorityLifecycle(root, prior) !== "absent") return; let state: Record; @@ -5903,6 +5912,9 @@ async function restorePerTurnRunnerdController(input: { createHash("sha256").update(nativeSessionKey(input.execution)).digest("hex"), "runner"); state = await readRemoteRunnerState({ runner: target.runner, stateDirectory }); } + // A running warm source keeps the existing live-adoption identity, idle, + // controller-fencing and physical-process checks. Only suspension is cached. + if (input.execution.session.lifecyclePolicy.mode === "warm" && state.lifecycle !== "suspended") return; if (state.lifecycle !== "suspended" || !Array.isArray(state.outbox) || state.outbox.length !== 0 || state.pendingTerminalDelivery != null || state.pendingProviderCleanup != null || state.warmTransition != null) { throw new Error("native_controller_recovery_missing_evidence:runner_not_settled"); @@ -5924,7 +5936,10 @@ async function restoreWarmRunnerdController(input: { const identity = readRunnerdDurableIdentity(root); const reference = identity ?? readRunnerdDurableIdentity(root, true); if (!idleWarmNativeSessionOwnerRunId(input) && (!reference || !existsSync(nativeSessionCheckpointPath(input.execution)))) { - if (await restoreNativeControllerProjections(input.db, input.execution, root)) return; + if (await restoreNativeControllerProjections(input.db, input.execution, root)) { + await restoreRemoteSuspendedRunnerCheckpoint(input, root); + return; + } } if (!durableIdentityMatchesSession(reference, input.execution)) return; const retainedRunId = idleWarmNativeSessionOwnerRunId(input); @@ -5945,11 +5960,13 @@ async function restoreWarmRunnerdController(input: { issueId = prior.issueId; } } - if (identity?.runId === runId) return; - const restored = await restoreNativeRunnerCheckpoint(input.db, - { companyId: input.execution.binding.companyId, issueId, runId }, root, nativeSessionKey(input.execution)); - // A legacy file-backed root still follows its existing prior-owner checks. - if (!restored && !identity) throw new Error("native_runner_checkpoint_missing"); + if (identity?.runId !== runId) { + const restored = await restoreNativeRunnerCheckpoint(input.db, + { companyId: input.execution.binding.companyId, issueId, runId }, root, nativeSessionKey(input.execution)); + // A legacy file-backed root still follows its existing prior-owner checks. + if (!restored && !identity) throw new Error("native_runner_checkpoint_missing"); + } + if (!retainedRunId) await restoreRemoteSuspendedRunnerCheckpoint(input, root); } async function restoreNativeControllerProjections(db: Db, execution: NativeExecutionInput, root: string): Promise { @@ -11884,7 +11901,7 @@ export function createRemoteRunnerProcessLauncher(input: { stateDirectory: string; diagnosticsDirectory: string; runnerInstanceId: string; - ensureArtifact?: () => Promise; + ensureArtifact?: (spec: RunnerProcessLaunchSpec) => Promise; requireProcessStartFingerprint?: boolean; onSpawn?: (meta: { pid: number; @@ -11970,11 +11987,11 @@ export function createRemoteRunnerProcessLauncher(input: { if (input.trace) { await input.trace.measure( "runner.runtime.stage", - input.ensureArtifact, + () => input.ensureArtifact!(spec), { parentName: "runner.session.startup" }, ); } else { - await input.ensureArtifact(); + await input.ensureArtifact(spec); } } if (pendingSignal) return cancelledBeforeDispatch(); @@ -13804,7 +13821,11 @@ async function createRunnerdBackendWithinSessionClaim( stateDirectory: remoteStateDirectory, diagnosticsDirectory: posix.join(remoteSessionRoot!, "diagnostics"), runnerInstanceId: input.runnerInstanceId, - ensureArtifact: ensureRemoteRunner, + // Source recovery relaunches the previous authority against its + // retained provider home. The transport intentionally has not created + // this turn's launch assets until that source is reconciled. + ensureArtifact: spec => ensureRemoteRunner( + !spec.nativeRunner || spec.nativeRunner.identity.runId === input.execution.binding.runId), onSpawn: input.onSpawn, onLog: input.onLog, trace: input.trace,