perf(server): reach readiness without waiting for backlog-sized recovery passes (#21014)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server process starts, recovers saved work, and then reports
ready on `/api/ready`. The deploy health gate waits for that signal.
> - On a busy staging workspace, one boot took 85 s to bind the listener
and 4.4 minutes to report ready. Small, idle stacks take 20–90 s.
> - Two parts of that time do not need to block readiness. The server
published several hundred UI assets one at a time before it bound the
listener. After the listener bound, it waited for the first pass of each
periodic recovery queue, and those passes grow with the backlog.
> - The volume-free data gate, which can refuse a start, ran last. By
then the server had already dispatched queued runs that a refusal would
abandon.
> - This pull request moves the refusal gate to the start, publishes the
assets 8 at a time with replayable bodies, and runs the backlog-sized
first passes after readiness, in their original order.
> - The benefit is a faster time from container start to ready, and no
work admitted before a gate that can refuse to start.

## Linked Issues or Issue Description

No public issue exists. This is the problem description (enhancement
template).

**Is your enhancement related to a problem? Please describe.**
Server boot time grows with the saved backlog. These timestamps come
from the log of one staging boot (release 87fe4d9793). The container
started at 01:16:56. The listener bound at 01:18:21, 85 s after start.
The fresh queued-run dispatch completed at 01:18:25. The full queued-run
dispatch pass completed at 01:20:14. The volume-free data gate ran at
01:21:20. Readiness came about 4.4 minutes after start. During startup
the log showed hundreds of `An error was encountered in a non-retryable
streaming request.` lines.

**Describe the solution you'd like**
Readiness waits only for the correctness gates and for the adoption of
old runs. Work that the periodic queues already retry runs after
readiness. A gate that can refuse to start runs before the server admits
any work.

**Describe alternatives you've considered**
- Move all post-listen recovery to the background. We rejected this.
Native recovery can fail closed and must still fail the start. It also
adopts old runs.
- Skip asset re-publication when the image is unchanged. We rejected
this. The collision check (never overwrite different bytes) needs a read
per asset.

**Additional context**
Startup map (line numbers are from `origin/master` 82e0393598):
- Blocks the listener: migrations and the embedded database
(`server/src/index.ts:369-741`); config and identity (`:753-774`);
backfills (`:821-833`, where
`sweepSupersededPendingRequestConfirmations` scans every pending
confirmation); `qualifyCloudConfigurationForCurrentImage` (`:956`,
synchronous package hashing); `createApp` (`:964`), which awaited serial
UI asset publication (`server/src/app.ts:1110`,
`server/src/retained-ui-assets.ts:69-105`).
- Blocks readiness after the listener: first passes of tool reviews,
gateway sessions and delivery workers (`index.ts:1529-1558`); the
terminal workspace reaper (`:1652`); native recovery and hot-restart
adoption (`:1707-1779`); the first passes of 17 recovery queues
(`:1782-1890`), including the full queued-run dispatch (`:1805-1811`);
backup registration (`:1979-1999`); the data gate and package
qualification (`:2020-2026`).
- Grows with the backlog: the queue first passes (queued runs, pending
deliveries, watchdogs, native sessions and results), the terminal
workspace sweep, the native recovery scan, and the confirmation sweep.
UI asset publication grows with the build (hundreds of assets), not the
backlog.

## What Changed

- **Volume-free data gate runs first.** `assertEphemeralServerData` now
runs right after `claimVolumeFreeControllerState` and the runtime config
load. That is before `createApp`, before the listener binds, and before
any queued-run dispatch. It used to run at the end of startup. The
package qualification gate stays at the end, because bundled plugin
installs can change its inputs.
- **UI asset publication is faster but still a gate before work.**
`createApp` starts the publication while it builds its routes. It awaits
the result before its first job, queue or run starts, which is the same
point as on master. A colliding build still refuses to start before it
admits any work (Greptile review).
- **Publication sends Buffer bodies, 8 at a time.** Each asset is read
once into a Buffer. The S3 client can replay a Buffer body; it cannot
replay a stream. The new code sends no stream bodies, so it no longer
reaches the SDK path that logs "non-retryable streaming request". After
the first failure, no new assets start. A collision found on one asset
outranks a storage outage on another (Greptile review).
- **Backlog-sized first passes run after readiness.** The 17 queue
registrations after native recovery (residual recovery through
stopped-instruction collection) now run in a tracked background
sequence, in the same order. Each pass is isolated. A pass that fails is
logged with its name, and the later passes, the housekeeping and the
scheduler still run. A queue whose first pass failed stays registered,
and the coordinator retries it. Native recovery and hot-restart adoption
still complete before ready. Idle sleep stays blocked while the sequence
runs, and shutdown waits for it like other scheduler work.
- **Queued turns keep the right status.** A new flag,
`startupQueuedRunPassOutstanding()`, stays true until the first full
queued-run pass ends. Until then the execution projection still shows
"Waiting for server restart". The flag clears even when that pass fails,
because the dispatch queue then owns those turns.
- **Startup timeline log.** At readiness the server logs the duration of
each startup step (`startup timeline at readiness`). It logs again when
the background passes finish. No OpenTelemetry collector is needed.
- **Benchmark.** A new script,
`scripts/benchmark-retained-ui-assets.ts`, measures publication against
storage with a fixed round-trip latency. It can fail when time exceeds a
budget (`RETAINED_UI_BENCH_MAX_MS`).

Why each moved step is safe:
- **Data gate (earlier).** Its inputs are the migrated database and the
owner marker. `claimVolumeFreeControllerState` still runs first.
Interaction with #20952 and #20990, which are both in this branch: those
changes made the gate tolerate
`native_harness_backups_require_migration` and
`local_instruction_copies_require_settlement` when the filesystem
carries the volume-free owner marker. That covered the case where this
boot's own dispatch and recovery wrote such run state before the late
gate ran. With the gate first, no work of this boot can write run state
before the gate: the app, its queues, run admission, native recovery and
dispatch all start after it. A test checks this order. So the gate sees
only state that a previous process left, and the owner-marker tolerance
still decides that state. This is strictly safer. Nothing relied on the
gate running after recovery: the only work that could clear a blocker
was background work that the old position never awaited. Effect: a
refused start no longer leaves dispatched runs behind.
- **UI asset publication.** The await point is unchanged, so the
gate-before-work order matches master. Only the time it takes changes.
- **Recovery queue first passes (after ready).** Each queue owns its
work and retries it on deadlines, commit wakes, and intervals. A first
pass that ends after ready behaves like a later pass. Interrupted runs
are still classified and adopted by native recovery before ready, and
they resume without a Retry. No wait is added before a turn starts:
turns can start sooner. The HTTP API already served requests during
recovery, because readiness never gated requests. Observable changes:
- The deploy gate passes earlier. A queue that throws while it registers
is now logged instead of failing the start. Each queue's `ready` already
absorbed run errors.
- `startup queued-run dispatch pass complete` can now appear after
`Server startup recovery complete`.

Left alone (safety unclear or not measured):
- The first passes of tool review recovery, gateway sessions, the
delivery workers, the GitHub workers, and the terminal workspace reaper.
These run before native recovery, and I did not prove the ordering safe.
- `qualifyCloudConfigurationForCurrentImage` before `createApp`. It
hashes plugin packages synchronously; its result is only a warning
there.
- The confirmation sweep and principal access backfill before the
listener. They scan whole tables.

## Verification

- Red/green: with `server/src/index.ts`, `app.ts` and
`retained-ui-assets.ts` reverted to master, these new tests fail. With
the change they pass:
  - `server-startup-feedback-export.test.ts`:
- refuses ineligible volume-free data before the app, the listener or
any queued-run dispatch
- reports ready while the first queued-run dispatch pass still runs,
then finishes it and the passes after it
- runs every deferred recovery pass after readiness when one throws,
logs it, and keeps the failed queue registered (fails on the first
revision of this PR, passes now)
  - `retained-ui-assets.test.ts`:
    - re-publishes with bounded parallel, replayable writes
- refuses a colliding build even when another asset's storage outage
fails first (fails with `failure ??=`, passes now)
- Regression guards that also pass: "checks volume-free data once,
before any of this boot's own startup work can write run state" and
"stops starting new assets after one collides".
- Focused suites pass after merging `origin/master`:
`server-startup-feedback-export` (131 tests), `retained-ui-assets`,
`execution-projection`, `liveness`, `heartbeat-startup-fresh-dispatch`,
`ephemeral-server-startup-run-state`, `ephemeral-server-data`,
`ephemeral-migration`, `ephemeral-server-contract` and
`static-index-html`. `tsc --noEmit -p server/tsconfig.json` is clean.
- Benchmark (measured locally, 400 synthetic assets, 25 ms simulated
round trip): `pnpm --filter @paperclipai/server exec tsx
../scripts/benchmark-retained-ui-assets.ts`

  | | serial (before) | 8 in flight (after) |
  | --- | --- | --- |
  | first publish | 10.5 s | 1.3 s |
  | restart re-publish | 21.3 s | 2.7 s |

This time is still before the listener, as on master, but it is about 8
times shorter and overlaps route construction.
- Local boot (measured, empty instance, embedded PostgreSQL, 3 boots
each, a shared and noisy machine):
  - listener to ready: before 385–427 ms; after 212–303 ms.
- Spawn to listener was 5.5–12.8 s in both cases. Module load and the
embedded database dominate it, and it does not differ.
- Not measured: the hosted saving. From the staging log, the inferred
saving is the queued-run pass and the passes after it (about 175 s:
01:18:25 to 01:21:20), plus most of the serial asset publication before
the listener. Confirm with the new `startup timeline at readiness` log
line after deploy.

## Risks

- **Medium: readiness semantics change.** The deploy gate now passes
before the first recovery passes finish. The old server can leave
earlier while the new server still dispatches its backlog. Queued turns
show "Waiting for server restart" until the dispatch pass ends, and the
queues retry on their own.
- **Low: earlier data gate.** It can refuse a start that the old order
would have admitted only if background work cleared a blocker first.
That was a race before this change.
- **Low: parallel publication against throttling storage.** Eight
requests at a time, with the existing per-asset backoff (0.5–8 s). A
persistent outage still lets the server start on local files.
- No migrations. No API contract changes.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edit, subagents for code search). Extended
reasoning on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Related: #21010 (retry a throttled asset upload).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-12 05:35:47 -06:00
1 parent 2e593bf3cd
commit b853cc28ae
9 files changed
+532 -146

No files matched your search

+69
View File
@@ -0,0 +1,69 @@
/** Shared UI asset publication at startup, against storage with a fixed
* round-trip latency. Local and synthetic; never targets real storage.
* Run with pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-retained-ui-assets.ts
* Options (env): RETAINED_UI_BENCH_ASSETS (default 400), RETAINED_UI_BENCH_RTT_MS (default 25),
* RETAINED_UI_BENCH_MAX_MS: fail when the default re-publication exceeds this budget. */
import assert from "node:assert/strict";
import { promises as fs } from "node:fs";
import os from "node:os";
import path from "node:path";
import { Readable } from "node:stream";
import { RETAINED_UI_ASSET_PUBLISH_CONCURRENCY, retainedUiAssets } from "../server/src/retained-ui-assets.js";
import type { StorageProvider } from "../server/src/storage/types.js";
const assets = Number(process.env.RETAINED_UI_BENCH_ASSETS ?? 400);
const rttMs = Number(process.env.RETAINED_UI_BENCH_RTT_MS ?? 25);
const budgetMs = process.env.RETAINED_UI_BENCH_MAX_MS ? Number(process.env.RETAINED_UI_BENCH_MAX_MS) : null;
function latencyStorage() {
const objects = new Map<string, Buffer>();
let requests = 0, streamBodies = 0;
const roundTrip = () => { requests++; return new Promise(resolve => setTimeout(resolve, rttMs)); };
const provider: StorageProvider = {
id: "s3",
putObject: async () => { throw new Error("Expected conditional publication"); },
async putObjectConditional(input) {
await roundTrip();
if (!Buffer.isBuffer(input.body)) streamBodies++;
if (objects.has(input.objectKey)) throw Object.assign(new Error("At least one of the pre-conditions you specified did not hold"), { name: "PreconditionFailed", $metadata: { httpStatusCode: 412 } });
const chunks: Buffer[] = [];
if (Buffer.isBuffer(input.body)) chunks.push(input.body);
else for await (const chunk of input.body) chunks.push(Buffer.from(chunk));
objects.set(input.objectKey, Buffer.concat(chunks));
return { etag: '"bench"' };
},
async getObject(input) {
await roundTrip();
const bytes = objects.get(input.objectKey);
if (!bytes) throw Object.assign(new Error("Object missing"), { status: 404 });
return { stream: Readable.from([bytes]), contentLength: bytes.length, etag: '"bench"' };
},
async headObject(input) { await roundTrip(); return { exists: objects.has(input.objectKey) }; },
async deleteObject(input) { objects.delete(input.objectKey); },
};
return { provider, stats: () => ({ requests, streamBodies }), reset: () => { requests = 0; streamBodies = 0; } };
}
const directory = await fs.mkdtemp(path.join(os.tmpdir(), "retained-ui-bench-"));
try {
for (let index = 0; index < assets; index++) {
await fs.writeFile(path.join(directory, `chunk-${index}.${index.toString(16)}.js`), `export const chunk${index} = ${JSON.stringify("x".repeat(2_000 + (index % 50) * 400))};\n`);
}
const results: Record<string, unknown>[] = [];
for (const concurrency of [1, RETAINED_UI_ASSET_PUBLISH_CONCURRENCY]) {
const storage = latencyStorage();
const retained = retainedUiAssets(storage.provider, { concurrency });
for (const scenario of ["first_publish", "restart_republish"] as const) {
storage.reset();
const started = performance.now();
await retained.publish(directory);
results.push({ concurrency, scenario, ms: Math.round(performance.now() - started), ...storage.stats() });
}
}
console.table(results);
const restart = results.find(row => row.concurrency === RETAINED_UI_ASSET_PUBLISH_CONCURRENCY && row.scenario === "restart_republish")!;
assert.equal(restart.streamBodies, 0, "publication must send replayable Buffer bodies");
if (budgetMs !== null) assert.ok((restart.ms as number) <= budgetMs, `restart re-publication took ${restart.ms} ms, over the ${budgetMs} ms budget`);
} finally {
await fs.rm(directory, { recursive: true, force: true });
}
@@ -20,9 +20,8 @@ function storage() {
id: "s3", putObject: async () => { throw new Error("Expected conditional publication"); },
async putObjectConditional(input) {
if (objects.has(input.objectKey)) throw new Error("Precondition failed");
const chunks: Buffer[] = [];
for await (const chunk of input.body) chunks.push(Buffer.from(chunk));
objects.set(input.objectKey, Buffer.concat(chunks)); return { etag: '"fixture-etag"' };
if (!Buffer.isBuffer(input.body)) throw new Error("Expected a replayable Buffer body");
objects.set(input.objectKey, Buffer.from(input.body)); return { etag: '"fixture-etag"' };
},
async getObject(input) {
const bytes = objects.get(input.objectKey);
@@ -78,11 +77,11 @@ it("retries an asset the storage throttled instead of failing the publication",
const put = provider.putObjectConditional!.bind(provider);
let refusals = 3;
provider.putObjectConditional = async input => {
if (refusals > 0) { refusals--; for await (const _chunk of input.body) { /* drain as a real client would */ } throw slowDown(); }
if (refusals > 0) { refusals--; throw slowDown(); }
return put(input);
};
const waits: number[] = [];
await retainedUiAssets(provider, { sleep: async ms => { waits.push(ms); } }).publish(root);
await retainedUiAssets(provider, { sleep: async ms => { waits.push(ms); }, concurrency: 1 }).publish(root);
expect(waits).toEqual([500, 1_000, 2_000]);
expect(objects.get("ui-assets/v1/chunk-a.js")?.toString()).toBe("alpha");
expect(objects.get("ui-assets/v1/chunk-b.js")?.toString()).toBe("beta");
@@ -142,3 +141,57 @@ it("retries an existing asset whose saved bytes storage could not serve, without
expect(isTransientStorageError(Object.assign(new Error("The operation was aborted"), { name: "AbortError" }))).toBe(true);
expect(isTransientStorageError(Object.assign(new Error("Object missing"), { status: 404 }))).toBe(false);
});
// Staging, 2026-11-11: a busy workspace took 85 s to bind its listener. Its
// restart re-published several hundred already-published assets one at a time
// (a refused conditional write, then a verifying read, per asset), and each
// refused stream upload logged "An error was encountered in a non-retryable
// streaming request." Publication now sends replayable Buffer bodies and keeps
// a bounded number of assets in flight.
it("re-publishes an already-published build with bounded parallel, replayable writes", async () => {
const root = await directory(), { provider } = storage();
for (let index = 0; index < 40; index++) await fs.writeFile(path.join(root, `chunk-${index}.js`), `export const n = ${index};`);
await retainedUiAssets(provider).publish(root);
const put = provider.putObjectConditional!.bind(provider), get = provider.getObject.bind(provider);
let inFlight = 0, peak = 0, streamBodies = 0;
const latency = async <T,>(work: () => Promise<T>) => {
inFlight++; peak = Math.max(peak, inFlight);
try { await new Promise(resolve => setTimeout(resolve, 5)); return await work(); } finally { inFlight--; }
};
provider.putObjectConditional = input => { if (!Buffer.isBuffer(input.body)) streamBodies++; return latency(() => put(input)); };
provider.getObject = input => latency(() => get(input));
await retainedUiAssets(provider).publish(root);
expect(streamBodies).toBe(0);
expect(peak).toBeGreaterThan(1);
expect(peak).toBeLessThanOrEqual(8);
});
it("refuses a colliding build even when another asset's storage outage fails first", async () => {
const root = await directory(), { provider, objects } = storage();
await fs.writeFile(path.join(root, "chunk-a.js"), "alpha");
await fs.writeFile(path.join(root, "chunk-b.js"), "beta");
objects.set("ui-assets/v1/chunk-b.js", Buffer.from("different bytes"));
const put = provider.putObjectConditional!.bind(provider), get = provider.getObject.bind(provider);
provider.putObjectConditional = async input => {
if (input.objectKey.endsWith("chunk-a.js")) throw Object.assign(new Error("Please reduce your request rate."), { name: "SlowDown", $metadata: { httpStatusCode: 503 } });
return put(input);
};
// The collision is found only after the outage on chunk-a has exhausted its retries.
provider.getObject = async input => {
if (input.objectKey.endsWith("chunk-b.js")) await new Promise(resolve => setTimeout(resolve, 100));
return get(input);
};
const retained = retainedUiAssets(provider, { sleep: async () => {}, concurrency: 2 });
await expect(publishRetainedUiAssetsAtStartup(retained, root, () => {})).rejects.toThrow("collides");
});
it("stops starting new assets after one collides, and still refuses the build", async () => {
const root = await directory(), { provider, objects } = storage();
for (let index = 0; index < 30; index++) await fs.writeFile(path.join(root, `chunk-${String(index).padStart(2, "0")}.js`), `v${index}`);
objects.set("ui-assets/v1/chunk-00.js", Buffer.from("different bytes"));
let writes = 0;
const put = provider.putObjectConditional!.bind(provider);
provider.putObjectConditional = input => { writes++; return put(input); };
await expect(retainedUiAssets(provider, { concurrency: 2 }).publish(root)).rejects.toThrow("collides");
expect(writes).toBeLessThan(30);
});
@@ -8,7 +8,7 @@ import { joinRequestRetentionService } from "../services/join-request-retention.
import { approvalRetentionService } from "../services/approval-retention.js";
import { createSecretProposalsService } from "../services/secret-proposals.js";
import { logger } from "../middleware/logger.js";
import { getStartupRecoveryState } from "../startup-recovery-state.js";
import { getStartupRecoveryState, startupQueuedRunPassOutstanding } from "../startup-recovery-state.js";
import { idleWorkSnapshot } from "../services/task-admission.js";
import { readIdleSleepSafety } from "../services/idle-sleep-safety.js";
import * as idleLocalWork from "../services/idle-local-work.js";
@@ -47,6 +47,33 @@ const cloudMigrationMocks = vi.hoisted(() => ({ migrate: vi.fn(async () => null)
vi.mock("../services/cloud-configuration-migration.js", () => ({ migrateCloudConfiguration: cloudMigrationMocks.migrate }));
vi.mock("../services/cloud-configuration-qualification.js", () => ({ qualifyCloudConfigurationForCurrentImage: cloudMigrationMocks.qualify }));
vi.mock("../telemetry.js", () => ({ initTelemetry: cloudRuntimeStartupMocks.telemetry, getTelemetryClient: vi.fn(() => null) }));
// A volume-free start (a deployment listen URL) with its data gate and the
// cross-controller channels it opens. Off unless a test opts in.
const volumeFreeMocks = vi.hoisted(() => ({
config: vi.fn((): string | null => null),
data: vi.fn(async (_db: unknown) => undefined),
claim: vi.fn(async (_volumeFree: boolean) => "claimed" as const),
}));
vi.mock("../services/ephemeral-server-contract.js", async importOriginal => ({
...await importOriginal<typeof import("../services/ephemeral-server-contract.js")>(),
assertEphemeralServerConfig: volumeFreeMocks.config,
assertEphemeralServerData: volumeFreeMocks.data,
}));
vi.mock("../services/native-runtime/volume-free-controller-state.js", async importOriginal => ({
...await importOriginal<typeof import("../services/native-runtime/volume-free-controller-state.js")>(),
claimVolumeFreeControllerState: volumeFreeMocks.claim,
}));
vi.mock("../services/deployment-notifications.js", () => ({
deploymentNotifications: vi.fn(() => ({ ready: async () => {}, markReady: vi.fn(), subscribeCompany: vi.fn(() => () => {}), stop: async () => {} })),
}));
vi.mock("../services/native-runtime/native-cancellation-delivery.js", async importOriginal => ({
...await importOriginal<typeof import("../services/native-runtime/native-cancellation-delivery.js")>(),
registerNativeCancellationDelivery: vi.fn(() => ({ ready: Promise.resolve(), stop: async () => {} })),
}));
vi.mock("../services/native-runtime/native-runtime-response-delivery.js", async importOriginal => ({
...await importOriginal<typeof import("../services/native-runtime/native-runtime-response-delivery.js")>(),
registerNativeRuntimeResponseDelivery: vi.fn(() => ({ ready: Promise.resolve(), stop: async () => {} })),
}));
// Producer transaction/notification behavior is covered by the real PostgreSQL
// GitHub worker suite; this startup seam supplies a lightweight database mock.
@@ -690,6 +717,7 @@ beforeEach(() => {
describe("startServer feedback export wiring", () => {
beforeEach(() => {
vi.clearAllMocks();
volumeFreeMocks.config.mockReturnValue(null);
deliveryWorkErrors.length = 0;
process.env.PAPERCLIP_DECISION_SIGNING_SECRET = "fedcba9876543210fedcba9876543210";
process.env.PAPERCLIP_AGENT_JWT_SECRET = "0123456789abcdef0123456789abcdef";
@@ -1952,6 +1980,108 @@ describe("startServer feedback export wiring", () => {
expect(fakeServer.close).not.toHaveBeenCalled();
});
// Staging, 2026-11-11: a busy volume-free server dispatched queued runs at
// 01:18:25 and ran its data eligibility gate at 01:21:20, 4.4 minutes after
// the container started. A refusal then abandoned work it had started.
it("refuses ineligible volume-free data before the app, the listener or any queued-run dispatch", async () => {
loadConfigMock.mockReturnValue(buildTestConfig({ heartbeatSchedulerEnabled: true }));
volumeFreeMocks.config.mockReturnValue("postgres://paperclip@db.internal:5432/paperclip");
volumeFreeMocks.data.mockRejectedValueOnce(new Error("Volume-free server data is not eligible: agent_files_require_migration"));
await expect(startServer()).rejects.toThrow("agent_files_require_migration");
expect(volumeFreeMocks.claim).toHaveBeenCalledWith(true);
expect(volumeFreeMocks.claim.mock.invocationCallOrder[0]).toBeLessThan(volumeFreeMocks.data.mock.invocationCallOrder[0]!);
expect(createAppMock).not.toHaveBeenCalled();
expect(heartbeatServiceFactoryMock).not.toHaveBeenCalled();
expect(fakeServer.listen).not.toHaveBeenCalled();
expect(deliveryWorkRegisterMock).not.toHaveBeenCalled();
});
// Staging, 2026-11-11: the gate's findings included run state this boot's
// own dispatch and recovery had just written. Run first, the gate sees only
// state a previous process left: no queue, run admission, recovery, or
// dispatch of this boot can write before it.
it("checks volume-free data once, before any of this boot's own startup work can write run state", async () => {
loadConfigMock.mockReturnValue(buildTestConfig({ heartbeatSchedulerEnabled: true }));
volumeFreeMocks.config.mockReturnValue("postgres://paperclip@db.internal:5432/paperclip");
await expect(startServer()).resolves.toBeDefined();
await settleStartupHousekeeping();
expect(volumeFreeMocks.data).toHaveBeenCalledOnce();
const gate = volumeFreeMocks.data.mock.invocationCallOrder[0]!;
expect(volumeFreeMocks.claim.mock.invocationCallOrder[0]).toBeLessThan(gate);
for (const writer of [createAppMock, heartbeatServiceFactoryMock, deliveryWorkRegisterMock, fakeServer.listen,
heartbeatServiceMock.recoverNativeRunsAfterRestart, heartbeatServiceMock.registerRunDispatchWork]) {
expect(writer).toHaveBeenCalled();
expect(Math.min(...writer.mock.invocationCallOrder)).toBeGreaterThan(gate);
}
expect(getStartupRecoveryState().phase).toBe("ready");
});
it("runs every deferred recovery pass after readiness when one throws, logs it, and keeps the failed queue registered", async () => {
loadConfigMock.mockReturnValue(buildTestConfig({ heartbeatSchedulerEnabled: true, heartbeatSchedulerIntervalMs: 30000 }));
const failure = new Error("residual recovery store unavailable");
heartbeatServiceMock.registerResidualRecoveryWork.mockImplementationOnce(() => ({ ready: Promise.reject(failure) }));
heartbeatServiceMock.registerActiveRunSilenceWork.mockImplementationOnce(() => { throw new Error("silence registration broke"); });
let releaseDispatch!: () => void;
const dispatchPass = new Promise<void>((_resolve, reject) => { releaseDispatch = () => reject(new Error("dispatch pass failed")); });
heartbeatServiceMock.registerRunDispatchWork.mockImplementationOnce(() => ({ ready: dispatchPass,
retries: { reconciliationSnapshot: () => ({ revision: "startup-retries:1", reconciled: true }), stop: vi.fn(async () => {}) } }));
const timer = vi.spyOn(globalThis, "setInterval").mockImplementation(() => 1 as unknown as ReturnType<typeof setInterval>);
const starting = startServer();
try {
await starting;
expect(getStartupRecoveryState().phase).toBe("ready");
expect(startupQueuedRunPassOutstanding()).toBe(true);
releaseDispatch();
// The queued-run pass failed, and "Waiting for server restart" still clears.
await vi.waitFor(() => expect(startupQueuedRunPassOutstanding()).toBe(false));
await vi.waitFor(() => expect(heartbeatServiceMock.registerStoppedInstructionCollectionWork).toHaveBeenCalledOnce());
// Every pass after the failures still ran.
for (const later of [heartbeatServiceMock.registerIssueStateRecoveryWork, heartbeatServiceMock.registerDeferredReceiptWork,
heartbeatServiceMock.registerProviderLimitResumeWork, heartbeatServiceMock.registerScheduledRetryCancellationWork,
heartbeatServiceMock.registerNativePendingEffectWork]) expect(later).toHaveBeenCalledOnce();
// The failed queues were registered with the coordinator, which retries them.
expect(heartbeatServiceMock.registerResidualRecoveryWork).toHaveBeenCalledOnce();
expect(heartbeatServiceMock.registerRunDispatchWork).toHaveBeenCalledOnce();
const failures = vi.mocked(logger.error).mock.calls.filter(([fields]) =>
typeof fields === "object" && fields !== null && "startupRecoveryPass" in fields);
expect(failures.map(([fields]) => (fields as { startupRecoveryPass: string }).startupRecoveryPass))
.toEqual(["ResidualRecovery", "ActiveRunSilence", "RunDispatch"]);
expect(fakeServer.close).not.toHaveBeenCalled();
} finally { releaseDispatch(); await starting.catch(() => {}); timer.mockRestore(); }
});
// Staging, 2026-11-11: the first full queued-run dispatch pass of a busy
// workspace took 109 s, and the passes after it another 66 s, all before
// /api/ready answered. Those queues retry on their own; readiness no longer
// waits for their first pass.
it("reports ready while the first queued-run dispatch pass still runs, then finishes it and the passes after it", async () => {
loadConfigMock.mockReturnValue(buildTestConfig({ heartbeatSchedulerEnabled: true, heartbeatSchedulerIntervalMs: 30000 }));
let releaseDispatch!: () => void;
const dispatchPass = new Promise<void>(resolve => { releaseDispatch = resolve; });
heartbeatServiceMock.registerRunDispatchWork.mockImplementationOnce(() => ({ ready: dispatchPass,
retries: { reconciliationSnapshot: () => ({ revision: "startup-retries:1", reconciled: true }), stop: vi.fn(async () => {}) } }));
const timer = vi.spyOn(globalThis, "setInterval").mockImplementation(() => 1 as unknown as ReturnType<typeof setInterval>);
const starting = startServer();
try {
await vi.waitFor(() => expect(getStartupRecoveryState().phase).toBe("ready"), { timeout: 2_000 });
await starting;
// Native recovery, which adopts old runs, still finished before ready.
expect(heartbeatServiceMock.recoverNativeRunsAfterRestart).toHaveBeenCalledOnce();
// A queued turn still says it waits for this server's recovery.
expect(startupQueuedRunPassOutstanding()).toBe(true);
expect(heartbeatServiceMock.registerScheduledRetryCancellationWork).not.toHaveBeenCalled();
// Idle sleep stays blocked while the pass runs.
expect(idleWorkSnapshot().active).toBeGreaterThan(0);
releaseDispatch();
await vi.waitFor(() => expect(startupQueuedRunPassOutstanding()).toBe(false));
await vi.waitFor(() => expect(heartbeatServiceMock.registerStoppedInstructionCollectionWork).toHaveBeenCalledOnce());
expect(heartbeatServiceMock.registerScheduledRetryCancellationWork).toHaveBeenCalledOnce();
} finally { releaseDispatch(); await starting.catch(() => {}); timer.mockRestore(); }
});
it("closes the bound listener when native startup recovery fails", async () => {
loadConfigMock.mockReturnValue(buildTestConfig({
heartbeatSchedulerEnabled: true,
+11 -2
View File
@@ -1110,8 +1110,12 @@ export async function createApp(
);
if (process.env.PAPERCLIP_SHARED_UI_ASSETS === "true") {
const retained = retainedUiAssets(createStorageProviderFromConfig(loadStorageConfig()));
await publishRetainedUiAssetsAtStartup(retained, path.join(uiDist, "assets"), error =>
logger.error({ err: error }, "Shared UI assets could not be published because storage is unavailable; serving this build from local files"));
// Publish while the routes below are built; the outcome is awaited
// before the first background work starts. The promise settles to the
// failure instead of rejecting, so it is never an unhandled rejection.
app.locals.retainedUiAssetsPublication = trackIdleWork(publishRetainedUiAssetsAtStartup(retained, path.join(uiDist, "assets"), error =>
logger.error({ err: error }, "Shared UI assets could not be published because storage is unavailable; serving this build from local files"))
.then(() => null, (error: unknown) => ({ error })));
staticUi.use("/assets", retained.serve);
}
// Serve root/index through the same runtime HTML transform as SPA routes.
@@ -1251,6 +1255,11 @@ export async function createApp(
app.use(staticUi);
app.use(errorHandler);
// A build whose asset names collide with different published bytes refuses
// to start, so the outcome is known before any job, queue, or run starts.
const retainedUiAssetsPublication = await (app.locals.retainedUiAssetsPublication as Promise<{ error: unknown } | null> | undefined);
if (retainedUiAssetsPublication) throw retainedUiAssetsPublication.error;
jobCoordinator.start();
const deliveryWork = createDeliveryWorkCoordinator({
owner: db,
+160 -106
View File
@@ -134,7 +134,8 @@ import { initializeCloudRuntimeConfig } from "./services/cloud-runtime-config.js
import { loadCloudBootstrapConfig, loadConfig, type Config } from "./config.js";
import { logger } from "./middleware/logger.js";
import { startEventLoopStallMonitor } from "./event-loop-stall-monitor.js";
import { setStartupRecoveryPhase } from "./startup-recovery-state.js";
import { setStartupQueuedRunPassPending, setStartupRecoveryPhase } from "./startup-recovery-state.js";
import { startupTimeline } from "./startup-timeline.js";
import {
StartupRefusalError,
migrationRefusalError,
@@ -749,6 +750,7 @@ async function startServerWithDatabaseTeardown(
await Promise.all([...clients.map((client) => endDatabaseClient(client, 5)), closeLivenessDb(db, 5), closeCheckpointDb(db, 5), closeFileStorageDb(db, 5)]);
};
startupDatabase.close = closeDatabaseClients;
startupTimeline.mark("database");
setSentryRunLinkResolver(createSentryRunLinkResolver(db));
await migrateCloudConfiguration(db as any);
await initializeAdapterSettings(db as any);
@@ -771,6 +773,18 @@ async function startServerWithDatabaseTeardown(
if (restoredCloudRuntimeIdentity) config = loadConfig();
if (isCloudManagedInstance()) initializeRuntimeConfigConsumers(config);
// The volume-free data gate can refuse to start. Its inputs are the migrated
// database and the owner marker claimed above, so it runs here: before the
// app, its delivery queues and plugins start, before the listener binds, and
// before any queued run is dispatched or any old run adopted. A refused start
// then abandons no work. It used to run minutes later, after startup had
// already dispatched queued runs. No startup step it skipped could make an
// ineligible database eligible: the migrations that clear its blockers are
// maintenance commands, and the background passes startup starts were never
// awaited before it.
if (deploymentListenUrl !== null) await assertEphemeralServerData(db as any);
startupTimeline.mark("volume_free_data_gate");
const isWarmStandby = await createCloudWarmStandby(db as any);
if (config.deploymentMode === "local_trusted" && !isLoopbackHost(config.host)) {
@@ -831,6 +845,7 @@ async function startServerWithDatabaseTeardown(
if (confirmationSweep.expired > 0) {
logger.info(confirmationSweep, "Expired pending confirmations superseded by newer agent requests");
}
startupTimeline.mark("startup_backfills");
if (config.deploymentMode === "authenticated") {
const {
createBetterAuthHandler,
@@ -1007,6 +1022,7 @@ async function startServerWithDatabaseTeardown(
managedPluginAutoInstall,
});
startupTimeline.mark("app_created");
if (listenPort !== requestedListenPort) {
logger.warn(`Requested port is busy; using next free port (requestedPort=${requestedListenPort}, selectedPort=${listenPort})`);
}
@@ -1097,6 +1113,7 @@ async function startServerWithDatabaseTeardown(
});
});
startupListenerBound = true;
startupTimeline.mark("listening");
// A blocked event loop lapses every controller lease and freezes every
// request, so record each long block with the deltas that name its class.
const eventLoopStallMonitor = startEventLoopStallMonitor({
@@ -1556,6 +1573,7 @@ async function startServerWithDatabaseTeardown(
: app.locals.toolGateway.cleanupExpiredSessions()).catch((err: unknown) => logger.error({ err }, "startup gateway token cleanup failed"));
await timeStartupRecovery("delivery_workers", () => Promise.all([budgetEnforcementWorker, accountingWorker, fastResponseWorker, ...deliveryWorkers].map(worker => worker.ready)));
await Promise.all([githubEventWorker.ready, githubContinuityWorker.ready]);
startupTimeline.mark("delivery_first_passes");
if (heartbeat) {
const secretProposals = createSecretProposalsService(db as any);
@@ -1778,116 +1796,139 @@ async function startServerWithDatabaseTeardown(
trackHeartbeatSchedulerWork(startupHeartbeatRecovery);
await startupHeartbeatRecovery;
}
startupTimeline.mark("native_recovery");
if (config.heartbeatSchedulerEnabled) await heartbeat.registerResidualRecoveryWork(deliveryWork,
() => !isIdleTaskDrainActive() && !isWarmStandby() && !heartbeatSchedulerStopped).ready;
// The first passes of the periodic recovery queues below (residual
// recovery, the full queued-run dispatch, watchdogs, native settlement)
// scale with saved work: minutes on a busy workspace. They need not finish
// before the server serves requests correctly, so they run after readiness,
// in this order, as tracked scheduler work:
// - Every gate that can refuse a start (the volume-free data gate, native
// recovery failing closed) and the adoption of old runs (native recovery,
// hot-restart adoption) still complete before ready.
// - Each queue is the owner of its work and is already retried, woken by
// commits, and re-run on its deadline; a first pass that finishes after
// ready is the same as any later pass.
// - Until the queued-run pass finishes, a queued turn still says it waits
// for the server to finish recovery (see startup-recovery-state.ts).
// - Idle sleep stays blocked while the passes run (tracked work), and
// shutdown waits for them like any scheduler work.
const runStartupRecoveryPasses = async () => {
if (!config.heartbeatSchedulerEnabled) return;
// Each pass is isolated: one that throws is logged and the passes after
// it still run. A pass's queue stays registered and retries on its own
// schedule (retry interval, deadlines, commit wakes).
const pass = async (name: string, work: () => unknown) => {
if (heartbeatSchedulerStopped) return;
try { await work(); }
catch (err) { logger.error({ err, startupRecoveryPass: name }, "startup recovery pass failed; its queue retries on its own schedule"); }
};
await pass("ResidualRecovery", () => heartbeat.registerResidualRecoveryWork(deliveryWork,
() => !isIdleTaskDrainActive() && !isWarmStandby() && !heartbeatSchedulerStopped).ready);
// Report-only ownership reconciliation (durable ownership plan, Phase 1).
// Its first pass runs in the background so it never delays readiness.
if (config.heartbeatSchedulerEnabled) void registerOwnershipReconciliationWork(db as any, deliveryWork,
void registerOwnershipReconciliationWork(db as any, deliveryWork,
() => !isIdleTaskDrainActive() && !isWarmStandby() && !heartbeatSchedulerStopped,
{ enabled: async () => !(await heartbeat.resolveSchedulingSuppression()).suppressed }).ready
.catch(err => logger.warn({ err }, "ownership reconciliation report pass failed"));
if (config.heartbeatSchedulerEnabled) await heartbeat.registerActiveRunSilenceWork(deliveryWork,
() => !isIdleTaskDrainActive() && !isWarmStandby() && !heartbeatSchedulerStopped).ready;
if (config.heartbeatSchedulerEnabled) {
const work = heartbeat.registerIssueStateRecoveryWork(deliveryWork,
() => !isIdleTaskDrainActive() && !isWarmStandby() && !heartbeatSchedulerStopped);
idleDeadlineOwners.watchdogs = bindIdleDeadlineOwner(work.watchdogs.reconciliationSnapshot,
heartbeat.readIdleWatchdogSchedule, () => ({ coverage: { kind: "watchdogs" } }));
await work.ready;
}
if (config.heartbeatSchedulerEnabled) await heartbeat.registerDeferredReceiptWork(deliveryWork,
() => !isIdleTaskDrainActive() && !isWarmStandby() && !heartbeatSchedulerStopped).ready;
await pass("ActiveRunSilence", () => heartbeat.registerActiveRunSilenceWork(deliveryWork,
() => !isIdleTaskDrainActive() && !isWarmStandby() && !heartbeatSchedulerStopped).ready);
await pass("IssueStateRecovery", async () => {
const work = heartbeat.registerIssueStateRecoveryWork(deliveryWork,
() => !isIdleTaskDrainActive() && !isWarmStandby() && !heartbeatSchedulerStopped);
idleDeadlineOwners.watchdogs = bindIdleDeadlineOwner(work.watchdogs.reconciliationSnapshot,
heartbeat.readIdleWatchdogSchedule, () => ({ coverage: { kind: "watchdogs" } }));
await work.ready;
});
await pass("DeferredReceipt", () => heartbeat.registerDeferredReceiptWork(deliveryWork,
() => !isIdleTaskDrainActive() && !isWarmStandby() && !heartbeatSchedulerStopped).ready);
// A connected conversation's Retry needs the chat callback createApp registered.
if (config.heartbeatSchedulerEnabled) await heartbeat.registerProviderLimitResumeWork(deliveryWork,
await pass("ProviderLimitResume", () => heartbeat.registerProviderLimitResumeWork(deliveryWork,
() => !isIdleTaskDrainActive() && !isWarmStandby() && !heartbeatSchedulerStopped,
() => budgetResumeDelivery.prepareChatRetry).ready;
if (config.heartbeatSchedulerEnabled) {
const work = heartbeat.registerRunDispatchWork(deliveryWork,
() => !isIdleTaskDrainActive() && !isWarmStandby() && !heartbeatSchedulerStopped);
idleDeadlineOwners.retries = bindIdleDeadlineOwner(work.retries.reconciliationSnapshot, readIdleRetrySchedule,
snapshot => ({ coverage: { kind: "scheduled_retries", runIds: snapshot.runIds, wakeIds: snapshot.wakeIds }, issueWaits: snapshot.issueWaits }));
await work.ready;
logger.info({ processUptimeMs: Math.round(process.uptime() * 1000) }, "startup queued-run dispatch pass complete");
}
if (config.heartbeatSchedulerEnabled) await heartbeat.registerScheduledRetryCancellationWork(deliveryWork,
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready;
() => budgetResumeDelivery.prepareChatRetry).ready);
await pass("RunDispatch", async () => {
const work = heartbeat.registerRunDispatchWork(deliveryWork,
() => !isIdleTaskDrainActive() && !isWarmStandby() && !heartbeatSchedulerStopped);
idleDeadlineOwners.retries = bindIdleDeadlineOwner(work.retries.reconciliationSnapshot, readIdleRetrySchedule,
snapshot => ({ coverage: { kind: "scheduled_retries", runIds: snapshot.runIds, wakeIds: snapshot.wakeIds }, issueWaits: snapshot.issueWaits }));
try { await work.ready; }
// A failed first pass still ends the "waiting for server restart"
// status: the dispatch queue now owns those turns and retries them.
finally { setStartupQueuedRunPassPending(false); }
startupTimeline.mark("queued_run_pass");
logger.info({ processUptimeMs: Math.round(process.uptime() * 1000) }, "startup queued-run dispatch pass complete");
});
await pass("ScheduledRetryCancellation", () => heartbeat.registerScheduledRetryCancellationWork(deliveryWork,
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready);
if (config.heartbeatSchedulerEnabled) await heartbeat.registerCapturedInstructionCommitWork(deliveryWork,
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready;
if (config.heartbeatSchedulerEnabled) await heartbeat.registerInstructionDirectoryCleanupWork(deliveryWork,
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready;
if (config.heartbeatSchedulerEnabled) {
const worker = heartbeat.registerNativeWorkspaceFinalizationWork(deliveryWork,
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby());
unsubscribeNativeWorkspaceFinalization = worker.stop;
server.on("close", worker.stop);
trackHeartbeatSchedulerWork(worker.ready);
}
if (config.heartbeatSchedulerEnabled) {
const worker = heartbeat.registerRetainedNativeSessionCleanupWork(deliveryWork,
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby());
unsubscribeRetainedNativeSessionCleanup = worker.stop;
server.on("close", worker.stop);
trackHeartbeatSchedulerWork(worker.ready);
}
if (config.heartbeatSchedulerEnabled) {
const worker = heartbeat.registerExternalConversationSettlementWork(deliveryWork,
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby());
trackHeartbeatSchedulerWork(worker.ready);
}
if (config.heartbeatSchedulerEnabled) {
const worker = heartbeat.registerNativeRecoveryNoticeWork(deliveryWork,
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby());
trackHeartbeatSchedulerWork(worker.ready);
}
if (config.heartbeatSchedulerEnabled) {
const worker = heartbeat.registerNativeEnvironmentSettlementWork(deliveryWork,
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby());
trackHeartbeatSchedulerWork(worker.ready);
}
if (config.heartbeatSchedulerEnabled) {
const worker = heartbeat.registerNativeRunProjectionWork(deliveryWork,
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby());
trackHeartbeatSchedulerWork(worker.ready);
}
if (config.heartbeatSchedulerEnabled) {
const worker = heartbeat.registerNativeResultCommitWork(deliveryWork,
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby());
trackHeartbeatSchedulerWork(worker.ready);
}
if (config.heartbeatSchedulerEnabled) await heartbeat.registerNativeStatusReassessmentWork(deliveryWork,
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready;
if (config.heartbeatSchedulerEnabled) await heartbeat.registerNativeReviewResponseWork(deliveryWork,
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready;
if (config.heartbeatSchedulerEnabled) {
const worker = heartbeat.registerNativeResponsePresentationWork(deliveryWork,
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby());
unsubscribeNativeResponsePresentation = worker.stop;
server.on("close", worker.stop);
trackHeartbeatSchedulerWork(worker.ready);
}
if (config.heartbeatSchedulerEnabled) await heartbeat.registerNativePendingEffectWork(deliveryWork,
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready;
if (config.heartbeatSchedulerEnabled) await heartbeat.registerUnsafeWorkspaceExportWork(deliveryWork,
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready;
if (config.heartbeatSchedulerEnabled) await heartbeat.registerAutomaticCompletionReviewWork(deliveryWork,
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready;
if (config.heartbeatSchedulerEnabled) await heartbeat.registerObsoletePolicyReviewWork(deliveryWork,
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready;
if (config.heartbeatSchedulerEnabled) await heartbeat.registerCancelledWorkspacePreservationWork(deliveryWork,
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready;
if (config.heartbeatSchedulerEnabled) await heartbeat.registerStoppedInstructionCollectionWork(deliveryWork,
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready;
await pass("CapturedInstructionCommit", () => heartbeat.registerCapturedInstructionCommitWork(deliveryWork,
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready);
await pass("InstructionDirectoryCleanup", () => heartbeat.registerInstructionDirectoryCleanupWork(deliveryWork,
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready);
await pass("NativeWorkspaceFinalization", async () => {
const worker = heartbeat.registerNativeWorkspaceFinalizationWork(deliveryWork,
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby());
unsubscribeNativeWorkspaceFinalization = worker.stop;
server.on("close", worker.stop);
trackHeartbeatSchedulerWork(worker.ready);
});
await pass("RetainedNativeSessionCleanup", async () => {
const worker = heartbeat.registerRetainedNativeSessionCleanupWork(deliveryWork,
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby());
unsubscribeRetainedNativeSessionCleanup = worker.stop;
server.on("close", worker.stop);
trackHeartbeatSchedulerWork(worker.ready);
});
await pass("ExternalConversationSettlement", async () => {
const worker = heartbeat.registerExternalConversationSettlementWork(deliveryWork,
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby());
trackHeartbeatSchedulerWork(worker.ready);
});
await pass("NativeRecoveryNotice", async () => {
const worker = heartbeat.registerNativeRecoveryNoticeWork(deliveryWork,
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby());
trackHeartbeatSchedulerWork(worker.ready);
});
await pass("NativeEnvironmentSettlement", async () => {
const worker = heartbeat.registerNativeEnvironmentSettlementWork(deliveryWork,
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby());
trackHeartbeatSchedulerWork(worker.ready);
});
await pass("NativeRunProjection", async () => {
const worker = heartbeat.registerNativeRunProjectionWork(deliveryWork,
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby());
trackHeartbeatSchedulerWork(worker.ready);
});
await pass("NativeResultCommit", async () => {
const worker = heartbeat.registerNativeResultCommitWork(deliveryWork,
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby());
trackHeartbeatSchedulerWork(worker.ready);
});
await pass("NativeStatusReassessment", () => heartbeat.registerNativeStatusReassessmentWork(deliveryWork,
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready);
await pass("NativeReviewResponse", () => heartbeat.registerNativeReviewResponseWork(deliveryWork,
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready);
await pass("NativeResponsePresentation", async () => {
const worker = heartbeat.registerNativeResponsePresentationWork(deliveryWork,
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby());
unsubscribeNativeResponsePresentation = worker.stop;
server.on("close", worker.stop);
trackHeartbeatSchedulerWork(worker.ready);
});
await pass("NativePendingEffect", () => heartbeat.registerNativePendingEffectWork(deliveryWork,
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready);
await pass("UnsafeWorkspaceExport", () => heartbeat.registerUnsafeWorkspaceExportWork(deliveryWork,
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready);
await pass("AutomaticCompletionReview", () => heartbeat.registerAutomaticCompletionReviewWork(deliveryWork,
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready);
await pass("ObsoletePolicyReview", () => heartbeat.registerObsoletePolicyReviewWork(deliveryWork,
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready);
await pass("CancelledWorkspacePreservation", () => heartbeat.registerCancelledWorkspacePreservationWork(deliveryWork,
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready);
await pass("StoppedInstructionCollection", () => heartbeat.registerStoppedInstructionCollectionWork(deliveryWork,
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready);
startupTimeline.mark("recovery_passes");
};
const runStartupHousekeeping = async () => {
const step = (name: string, work: () => Promise<unknown>) =>
@@ -1965,9 +2006,21 @@ async function startServerWithDatabaseTeardown(
snapshot => ({ issueWaits: snapshot.issueWaits }));
trackHeartbeatSchedulerWork(Promise.all([scheduled.timers.ready, scheduled.monitors.ready, scheduled.routines.ready]));
};
trackHeartbeatSchedulerWork(runStartupHousekeeping().then(() => {
if (!heartbeatSchedulerStopped) startHeartbeatScheduler();
}));
// Same order as before: recovery passes, then housekeeping, then the
// scheduler. A pass that fails is logged and the passes after it, the
// housekeeping and the scheduler still run, as after a failed housekeeping
// step. A queue whose first pass failed stays registered and retries.
setStartupQueuedRunPassPending(true);
trackHeartbeatSchedulerWork(runStartupRecoveryPasses()
.catch((err: unknown) => { logger.error({ err }, "startup recovery passes failed; their queues retry on their own schedule"); })
.finally(() => setStartupQueuedRunPassPending(false))
.then(() => {
startupTimeline.log("startup recovery passes complete");
return runStartupHousekeeping();
})
.then(() => {
if (!heartbeatSchedulerStopped) startHeartbeatScheduler();
}));
} else {
// The heartbeat scheduler is disabled, but the orphan-sandbox cleanup sweep
// is still required. A failed acquire can leak a paid provider sandbox, so
@@ -2018,7 +2071,6 @@ async function startServerWithDatabaseTeardown(
}
if (deploymentListenUrl !== null) {
await assertEphemeralServerData(db as any);
const qualification = await qualifyCloudConfigurationForCurrentImage(db as any);
if (qualification && !qualification.configurationPackagesEligible) {
throw new Error(`Volume-free server packages are not eligible: ${qualification.blockers.map(blocker => blocker.code).join(", ")}`);
@@ -2026,6 +2078,8 @@ async function startServerWithDatabaseTeardown(
}
markIdleStartupComplete({ scheduledBackups: config.databaseBackupEnabled });
setStartupRecoveryPhase("ready");
startupTimeline.mark("ready");
startupTimeline.log("startup timeline at readiness");
deploymentHints?.markReady();
logger.info(`Server startup recovery complete on ${config.host}:${listenPort}`);
void systemdNotify(["--ready", `--status=Listening on ${config.host}:${listenPort}`]).then((notified) => {
+60 -30
View File
@@ -1,5 +1,4 @@
import { createHash } from "node:crypto";
import { createReadStream } from "node:fs";
import fs from "node:fs/promises";
import path from "node:path";
import { pipeline } from "node:stream/promises";
@@ -59,51 +58,82 @@ export async function publishRetainedUiAssetsAtStartup(
}
}
/** Assets published at once: enough to hide per-request latency, small enough
* that a throttling bucket sees a steady stream rather than a burst. */
export const RETAINED_UI_ASSET_PUBLISH_CONCURRENCY = 8;
/** Public, immutable build assets only. Publishing before readiness lets an
* older tab fetch its chunks after the last old container has disappeared.
* Objects have no automatic expiry: retain them at least 30 days after the
* last serving deployment, including rollback, before operator reclamation. */
export function retainedUiAssets(provider: StorageProvider, options: { sleep?: (ms: number) => Promise<void> } = {}): { publish(directory: string): Promise<void>; serve: RequestHandler } {
export function retainedUiAssets(provider: StorageProvider, options: { sleep?: (ms: number) => Promise<void>; concurrency?: number } = {}): { publish(directory: string): Promise<void>; serve: RequestHandler } {
const sleep = options.sleep ?? ((ms: number) => new Promise<void>(resolve => setTimeout(resolve, ms)));
const concurrency = Math.max(1, Math.floor(options.concurrency ?? RETAINED_UI_ASSET_PUBLISH_CONCURRENCY));
if (provider.id !== "s3" || !provider.putObjectConditional) throw new Error("Retained UI assets require conditional shared object storage");
async function publishAsset(filename: string, objectKey: string) {
// Build assets are small: read each once into a Buffer. The storage client
// can replay a Buffer body; it cannot replay a stream, and every refused
// stream upload, including the 412 each already-published asset answers
// on a restart, logged a "non-retryable streaming request" warning.
const bytes = await fs.readFile(filename);
const expected = createHash("sha256").update(bytes).digest("hex");
// Storage throttles a burst: retry the whole asset instead of failing startup.
for (let attempt = 0; ; attempt++) {
const signal = AbortSignal.timeout(30_000);
try {
try {
await provider.putObjectConditional!({ objectKey, body: bytes, contentLength: bytes.length, contentType: "application/octet-stream", ifNoneMatch: "*", signal });
} catch (error) {
// An existing key or uncertain upload may be accepted only if its
// complete bytes match. Never overwrite a colliding build asset.
// A read that storage could not serve proves nothing either way:
// keep it, so the asset is tried again instead of being judged.
const existing = await provider.getObject({ objectKey, signal })
.catch(readError => { throw isTransientStorageError(readError) ? readError : error; });
if (await digest(existing.stream) !== expected) throw new Error("Static build asset name collides with different published bytes");
}
return;
} catch (error) {
if (!isTransientStorageError(error) || attempt >= publishRetryDelaysMs.length) throw error;
await sleep(publishRetryDelaysMs[attempt]!);
}
}
}
async function publish(directory: string) {
const assets: Array<{ filename: string; objectKey: string }> = [];
async function visit(relativeDirectory: string) {
for (const entry of await fs.readdir(path.join(directory, relativeDirectory), { withFileTypes: true })) {
const relative = path.posix.join(relativeDirectory, entry.name);
if (entry.isDirectory()) { await visit(relative); continue; }
if (relative.endsWith(".map")) continue;
if (!entry.isFile() || !assetPath(relative)) throw new Error("Unsupported static build asset");
const filename = path.join(directory, relative);
const objectKey = `${prefix}${relative}`;
// A build has hundreds of assets and storage throttles a burst. The
// upload body is a stream, so the client cannot replay it: retry the
// whole asset here, with a fresh stream, instead of failing startup.
for (let attempt = 0; ; attempt++) {
const signal = AbortSignal.timeout(30_000);
const contentLength = (await fs.stat(filename)).size;
const expected = await digest(createReadStream(filename, { signal }));
const body = createReadStream(filename, { signal });
try {
try {
await provider.putObjectConditional!({ objectKey, body, contentLength, contentType: "application/octet-stream", ifNoneMatch: "*", signal });
} catch (error) {
// An existing key or uncertain upload may be accepted only if its
// complete bytes match. Never overwrite a colliding build asset.
// A read that storage could not serve proves nothing either way:
// keep it, so the asset is tried again instead of being judged.
const existing = await provider.getObject({ objectKey, signal })
.catch(readError => { throw isTransientStorageError(readError) ? readError : error; });
if (await digest(existing.stream) !== expected) throw new Error("Static build asset name collides with different published bytes");
}
break;
} catch (error) {
if (!isTransientStorageError(error) || attempt >= publishRetryDelaysMs.length) throw error;
await sleep(publishRetryDelaysMs[attempt]!);
} finally { body.destroy(); }
}
assets.push({ filename: path.join(directory, relative), objectKey: `${prefix}${relative}` });
}
}
await visit("");
// Each asset is an independent conditional write. A build has hundreds, so
// a bounded pool replaces hundreds of serial round trips with a few dozen.
// The first failure stops new assets from starting; ones in flight finish,
// and a collision among them outranks an outage.
let next = 0;
const failure: { found: boolean; error: unknown } = { found: false, error: null };
const worker = async () => {
while (!failure.found && next < assets.length) {
const asset = assets[next++]!;
try { await publishAsset(asset.filename, asset.objectKey); }
// A collision refuses the build; an outage only lets it start on local
// files. A storage outage on one asset must never hide a collision
// found on another.
catch (error) {
if (!failure.found || (isTransientStorageError(failure.error) && !isTransientStorageError(error))) {
failure.found = true;
failure.error = error;
}
}
}
};
await Promise.all(Array.from({ length: Math.min(concurrency, assets.length) }, worker));
if (failure.found) throw failure.error;
}
const serve: RequestHandler = async (req, res, next) => {
if (req.method !== "GET" && req.method !== "HEAD") { next(); return; }
+2 -2
View File
@@ -18,7 +18,7 @@ import { executionFailureRetryCount } from "./execution-recovery-attempt.js";
import { retryIdempotentDatabaseOperation } from "../database-retry.js";
import { workspaceRecoveryDisplay } from "./workspace-recovery-receipt.js";
import { readTaskDrain } from "./task-admission.js";
import { getStartupRecoveryState } from "../startup-recovery-state.js";
import { getStartupRecoveryState, startupQueuedRunPassOutstanding } from "../startup-recovery-state.js";
import { hasPendingRunCleanup } from "./execution-blocker.js";
import { isAgentAwaitingSetup } from "../modules/agent-lifecycle/index.js";
import { NATIVE_RESPONSE_WAKE_WAITING_REASON } from "./native-runtime/status-arbiter.js";
@@ -532,7 +532,7 @@ export function projectExecution(
// Process-local, like the drain: while this server starts, recovers, or
// shuts down, its full queued-run pass has not run (or no longer runs).
const serverPhase = getStartupRecoveryState().phase;
if (run.status === "queued" && serverPhase !== "ready") {
if (run.status === "queued" && startupQueuedRunPassOutstanding()) {
projection.cause = "server_restarting";
projection.nextAction = serverPhase === "draining"
? "The server is restarting for an update. This turn starts automatically on the replacement server."
+15
View File
@@ -2,6 +2,7 @@ export type StartupRecoveryPhase = "starting" | "recovering" | "ready" | "draini
let phase: StartupRecoveryPhase = "ready";
let updatedAt = new Date().toISOString();
let queuedRunPassPending = false;
export function setStartupRecoveryPhase(next: StartupRecoveryPhase): void {
phase = next;
@@ -15,7 +16,21 @@ export function getStartupRecoveryState(): {
return { phase, updatedAt };
}
/** The server reports ready before its first full queued-run dispatch pass
* finishes; that pass scales with the backlog. Until it does, a queued turn
* still waits on this server's recovery, not on anything the user can change. */
export function setStartupQueuedRunPassPending(pending: boolean): void {
queuedRunPassPending = pending;
}
/** True while this process has not yet run its full queued-run pass: before
* ready, while draining, or after ready until the first pass finishes. */
export function startupQueuedRunPassOutstanding(): boolean {
return phase !== "ready" || queuedRunPassPending;
}
export function resetStartupRecoveryStateForTests(): void {
phase = "ready";
updatedAt = new Date().toISOString();
queuedRunPassPending = false;
}
+26
View File
@@ -0,0 +1,26 @@
import { logger } from "./middleware/logger.js";
export type StartupTimelineMark = { step: string; atMs: number; durationMs: number };
/** Boot-to-ready milestones, measured from process start. One log line at
* readiness (and one when the background recovery passes finish) names the
* step that took the time, without an OpenTelemetry collector. */
export function createStartupTimeline(uptimeMs: () => number = () => process.uptime() * 1000) {
const marks: StartupTimelineMark[] = [];
return {
mark(step: string) {
const atMs = Math.round(uptimeMs());
const previous = marks.at(-1)?.atMs ?? 0;
marks.push({ step, atMs, durationMs: atMs - previous });
},
marks: () => marks.map(mark => ({ ...mark })),
log(message: string) {
logger.info({
processUptimeMs: Math.round(uptimeMs()),
startupSteps: Object.fromEntries(marks.map(mark => [mark.step, mark.durationMs])),
}, message);
},
};
}
export const startupTimeline = createStartupTimeline();