mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-04 19:16:16 +01:00
fix(ci): make Cloud readiness reruns and fixtures reliable (#18823)
## Thinking Path > - Paperclip manages agent work and preserves task state. > - Cloud readiness verifies an exact source revision before release. > - Its workspace tests must wait for execution and database cleanup to finish. > - Two attempts hit the same ten-second polling limit and read a running row. > - The second attempt also lost AWS runners and exceeded a fixture cleanup limit. > - This pull request waits for owned execution, bounds client shutdown, and gives verification reruns an independent runner pool. > - Assertions, source identity and required verification gates stay in place. ## Linked Issues or Issue Description **What happened?** [Cloud readiness run 41684102136](https://github.com/paperclipai/paperclip/actions/runs/41684102136) failed twice on master `367f29478d14003cc5c0f161b7ce8f3164993634`. Attempt 1 failed `heartbeat-workspace-busy.test.ts:473`: expected `succeeded`, received `running`. Attempt 2 failed the same helper in the inherited-workspace resume case at line 744. The helper polls for ten seconds, then returns the active row as if the wait finished. Attempt 2 also exceeded the 30-second `workspace-preparation.test.ts` teardown limit. Its client close had no deadline, and database plus directory cleanup shared the default hook budget despite the fixture's existing 90-second startup budget. Other attempt-2 failures show runner shutdown or lost communication. Several shutdown signals arrived at 18:05:17 UTC. Read-only AWS inspection confirms that all eight infrastructure-failed instances carry `runs-on-workflow-job-interrupted=true`. Five have explicit `Server.SpotInstanceTermination` reasons, including the server-1 runner that lost communication. Three were terminated through the fleet's shutdown path. CloudTrail records the Rust runner's interruption tag at 18:03:37 UTC and its termination by the RunsOn fleet role at 18:05:31 UTC, despite EC2 showing a user-initiated termination. Repeating the entire workflow reused the same Spot fleet. **Expected behavior** Wait for owned heartbeat execution before reading final state or scheduled retries. Bound database-client shutdown and allow the fixture's cleanup cost class to finish. Rerun verification on GitHub-hosted runners after an interrupted fleet attempt. Preserve all checks and report prior failures. **Steps to reproduce** 1. Delay the test adapter beyond the helper's ten-second polling window. 2. Invoke the ordinary local shared-workspace run or resume the deferred sandbox run. 3. Observe the old helper return `running` before the execution completes. 4. Evaluate the old runner expressions with `github.run_attempt = 2`; they still choose the interrupted AWS pool. **Paperclip version or commit** Base: `367f29478d14003cc5c0f161b7ce8f3164993634`. **Deployment mode** GitHub Actions Cloud readiness, with disposable PostgreSQL fixtures. Related: #11293 fixed the earlier gap between cancellation and retry-row insertion. This change waits for the complete execution instead. Duplicate searches covered the workspace test, retention fixture and runner shutdown reports. ## What Changed - Replace both workspace-test polling helpers with the existing heartbeat completion barrier. Keep final status, retry and coordination assertions. - Bound workspace-preparation client shutdown to one second. Apply the existing embedded-PostgreSQL fixture budget to its teardown. - Use AWS only for the first exact-source attempt. Route later verification, chaos and cloud-migrator planning/build attempts to GitHub-hosted runners. The publisher keeps its existing routing. The final branch also incorporates upstream #18821's CODEOWNER routing fix. - Add runner-expression coverage for numeric, string, missing and later attempt values. Document retry routing. ## Verification - The original workflow's failed assertions and interrupted jobs remain linked above. A local two-suite run also passed all 157 assertions and failed only the original 30-second teardown hook. - All 341 workflow contracts pass. After the master synchronization, all 347 routing and CI-planner controls pass. Actionlint passes for the three changed workflows. - A matched delayed-adapter comparison uses disposable PostgreSQL 17 databases, the same 11-second adapter delay, and the same 90-second experimental outer budget. The old helper fails with `running`; the completion barrier passes. Production test deadlines stay unchanged. - Earlier experiments hit a 30-second experimental outer limit and the macOS host's shared-memory limit during native database startup. The completed comparison uses task-owned databases on an existing test server. Those earlier attempts are not counted as passes. - All 31 workspace-busy cases pass with normal test deadlines. The 126-case workspace fixture also passes with its 90-second teardown budget. - Final-head cleanup measurements: main client close 7 ms, owned database cleanup 20,938 ms, and workspace-home removal 826 ms. A measured fixture held 124,494 PostgreSQL files (208 MiB) after repeated table resets. The measured database cleanup alone consumes most of the old 30-second hook budget. The separate baseline reproduced a timeout at that hook; the fixture now uses its established 90-second cost class. - Repository typecheck and build pass on synchronized head `2aceb3abd144a569efddabed3b7e0550ea23ceb2`. - All 157 focused cases pass on the synchronized branch, including complete PostgreSQL 18 fixture shutdown. [Full hosted CI](https://github.com/paperclipai/paperclip/actions/runs/41697804471) passes on the final head, including the required verification and E2E aggregates, all server/chat/workspace groups, Runner lanes and all eight browser shards. The separate GitHub quality-review workflow also passes. All 60 reported checks are terminal: 54 pass, three are informational neutral results, and three are expected skips. There are no failures or pending checks. The complete local `pnpm test:run` command was not run; targeted fixtures already encountered this host's shared-memory limit. Hosted CI supplies the full Vitest and browser matrix. - Greptile and Paperclip Review both give synchronized head `2aceb3abd144a569efddabed3b7e0550ea23ceb2` 5/5. There are no open review threads. - Module-boundary and whitespace checks pass. ## Risks - Hosted reruns can have different cache hit rates and queue times. They execute the same source and verification commands. - The first attempt retains the existing AWS Spot fleet policy. AWS evidence establishes several interruption-triggered failures; later verification attempts use an independent pool. This does not change the fleet's external capacity policy. - The completion barrier waits for owned background execution. A hung execution still fails the test's outer deadline. - The larger teardown budget remains finite. PostgreSQL's bounded stop protocol and cleanup errors still fail the hook. - No production execution, database schema or provider contract changes. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, repository inspection, code edits and test execution. The runtime does not expose the exact serving model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed
1 parent
83b5f7d486
commit
4a38ff9c92
7 files changed
+32
-39
No files matched your search
@@ -7,7 +7,7 @@ const fleet = "runs-on/fleet=paperclip-post-merge-x64/env=public-ci";
|
||||
const sha = "a".repeat(40);
|
||||
const base = {
|
||||
repository: "paperclipai/paperclip", repository_id: "1170821064",
|
||||
ref: "refs/heads/master", event_name: "push", sha,
|
||||
ref: "refs/heads/master", event_name: "push", sha, run_attempt: 1,
|
||||
};
|
||||
const expectedJobs = {
|
||||
"cloud-readiness.yml": [],
|
||||
@@ -58,6 +58,11 @@ for (const [file, expectedNames] of Object.entries(expectedJobs)) {
|
||||
cases.push({ name: "preview of master", inputs: { channel: "preview" } });
|
||||
cases.push({ name: "stable release", inputs: { channel: "stable" } });
|
||||
}
|
||||
cases.push({ name: "first attempt as a string", github: { run_attempt: "1" }, expected: fleet });
|
||||
cases.push({ name: "rerun after runner interruption", github: { run_attempt: 2 } });
|
||||
cases.push({ name: "later rerun", github: { run_attempt: 3 } });
|
||||
cases.push({ name: "rerun attempt as a string", github: { run_attempt: "2" } });
|
||||
cases.push({ name: "missing attempt", github: { run_attempt: undefined } });
|
||||
for (const { name, github = {}, inputs: overrides = {}, enabled = "true", expected = "ubuntu-latest" } of cases) {
|
||||
test(`${file}/${job}: ${name}`, () => {
|
||||
const context = { github: { ...defaultContext, ...github }, inputs: { ...inputs, ...overrides }, vars: { AWS_POST_MERGE_CI_ENABLED: enabled } };
|
||||
|
||||
@@ -34,7 +34,7 @@ jobs:
|
||||
typecheck:
|
||||
if: ${{ !inputs.runner_binaries_only }}
|
||||
name: Typecheck
|
||||
runs-on: ${{ vars.AWS_POST_MERGE_CI_ENABLED == 'true' && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.sha != '' && inputs.ref == github.sha && 'runs-on/fleet=paperclip-post-merge-x64/env=public-ci' || 'ubuntu-latest' }}
|
||||
runs-on: ${{ vars.AWS_POST_MERGE_CI_ENABLED == 'true' && github.run_attempt == 1 && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.sha != '' && inputs.ref == github.sha && 'runs-on/fleet=paperclip-post-merge-x64/env=public-ci' || 'ubuntu-latest' }}
|
||||
timeout-minutes: 20
|
||||
permissions:
|
||||
contents: read
|
||||
@@ -120,7 +120,7 @@ jobs:
|
||||
general_tests:
|
||||
if: ${{ !inputs.runner_binaries_only }}
|
||||
name: General tests (${{ matrix.group_label }})
|
||||
runs-on: ${{ vars.AWS_POST_MERGE_CI_ENABLED == 'true' && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.sha != '' && inputs.ref == github.sha && 'runs-on/fleet=paperclip-post-merge-x64/env=public-ci' || 'ubuntu-latest' }}
|
||||
runs-on: ${{ vars.AWS_POST_MERGE_CI_ENABLED == 'true' && github.run_attempt == 1 && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.sha != '' && inputs.ref == github.sha && 'runs-on/fleet=paperclip-post-merge-x64/env=public-ci' || 'ubuntu-latest' }}
|
||||
timeout-minutes: 20
|
||||
permissions:
|
||||
contents: read
|
||||
@@ -232,7 +232,7 @@ jobs:
|
||||
serialized_tests:
|
||||
if: ${{ !inputs.runner_binaries_only }}
|
||||
name: Serialized tests (${{ matrix.shard_label }})
|
||||
runs-on: ${{ vars.AWS_POST_MERGE_CI_ENABLED == 'true' && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.sha != '' && inputs.ref == github.sha && 'runs-on/fleet=paperclip-post-merge-x64/env=public-ci' || 'ubuntu-latest' }}
|
||||
runs-on: ${{ vars.AWS_POST_MERGE_CI_ENABLED == 'true' && github.run_attempt == 1 && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.sha != '' && inputs.ref == github.sha && 'runs-on/fleet=paperclip-post-merge-x64/env=public-ci' || 'ubuntu-latest' }}
|
||||
timeout-minutes: 20
|
||||
permissions:
|
||||
contents: read
|
||||
@@ -282,7 +282,7 @@ jobs:
|
||||
runner_workflow_evals:
|
||||
if: ${{ !inputs.runner_binaries_only }}
|
||||
name: Runner workflow eval scorer contract
|
||||
runs-on: ${{ vars.AWS_POST_MERGE_CI_ENABLED == 'true' && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.sha != '' && inputs.ref == github.sha && 'runs-on/fleet=paperclip-post-merge-x64/env=public-ci' || 'ubuntu-latest' }}
|
||||
runs-on: ${{ vars.AWS_POST_MERGE_CI_ENABLED == 'true' && github.run_attempt == 1 && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.sha != '' && inputs.ref == github.sha && 'runs-on/fleet=paperclip-post-merge-x64/env=public-ci' || 'ubuntu-latest' }}
|
||||
timeout-minutes: 10
|
||||
permissions:
|
||||
contents: read
|
||||
@@ -313,7 +313,7 @@ jobs:
|
||||
verify_paperclip_runner:
|
||||
if: ${{ !inputs.runner_binaries_only }}
|
||||
name: Verify Paperclip Runner (${{ matrix.lane }})
|
||||
runs-on: ${{ vars.AWS_POST_MERGE_CI_ENABLED == 'true' && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.sha != '' && inputs.ref == github.sha && 'runs-on/fleet=paperclip-post-merge-x64/env=public-ci' || 'ubuntu-latest' }}
|
||||
runs-on: ${{ vars.AWS_POST_MERGE_CI_ENABLED == 'true' && github.run_attempt == 1 && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.sha != '' && inputs.ref == github.sha && 'runs-on/fleet=paperclip-post-merge-x64/env=public-ci' || 'ubuntu-latest' }}
|
||||
timeout-minutes: 20
|
||||
permissions:
|
||||
contents: read
|
||||
@@ -523,7 +523,7 @@ jobs:
|
||||
build:
|
||||
if: ${{ !inputs.runner_binaries_only }}
|
||||
name: Build
|
||||
runs-on: ${{ vars.AWS_POST_MERGE_CI_ENABLED == 'true' && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.sha != '' && inputs.ref == github.sha && 'runs-on/fleet=paperclip-post-merge-x64/env=public-ci' || 'ubuntu-latest' }}
|
||||
runs-on: ${{ vars.AWS_POST_MERGE_CI_ENABLED == 'true' && github.run_attempt == 1 && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.sha != '' && inputs.ref == github.sha && 'runs-on/fleet=paperclip-post-merge-x64/env=public-ci' || 'ubuntu-latest' }}
|
||||
timeout-minutes: 20
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
@@ -94,11 +94,11 @@ jobs:
|
||||
# falls back to GitHub-hosted runners, including raced merge dispatches.
|
||||
name: Check preview artifacts
|
||||
if: github.ref == 'refs/heads/master' && github.event_name == 'workflow_dispatch' && (inputs.channel == 'preview' || inputs.channel == 'cloud-migrator') && !inputs.dry_run
|
||||
runs-on: ${{ vars.AWS_POST_MERGE_CI_ENABLED == 'true' && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.event_name == 'workflow_dispatch' && inputs.channel == 'cloud-migrator' && github.sha != '' && inputs.source_ref == github.sha && 'runs-on/fleet=paperclip-post-merge-x64/env=public-ci' || 'ubuntu-latest' }}
|
||||
runs-on: ${{ vars.AWS_POST_MERGE_CI_ENABLED == 'true' && github.run_attempt == 1 && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.event_name == 'workflow_dispatch' && inputs.channel == 'cloud-migrator' && github.sha != '' && inputs.source_ref == github.sha && 'runs-on/fleet=paperclip-post-merge-x64/env=public-ci' || 'ubuntu-latest' }}
|
||||
permissions:
|
||||
contents: read
|
||||
# Preserve the previous hosted default; only AWS needs the Fleet limit.
|
||||
timeout-minutes: ${{ vars.AWS_POST_MERGE_CI_ENABLED == 'true' && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.event_name == 'workflow_dispatch' && inputs.channel == 'cloud-migrator' && github.sha != '' && inputs.source_ref == github.sha && 10 || 360 }}
|
||||
timeout-minutes: ${{ vars.AWS_POST_MERGE_CI_ENABLED == 'true' && github.run_attempt == 1 && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.event_name == 'workflow_dispatch' && inputs.channel == 'cloud-migrator' && github.sha != '' && inputs.source_ref == github.sha && 10 || 360 }}
|
||||
outputs:
|
||||
image: ${{ steps.plan.outputs.image }}
|
||||
packages: ${{ steps.plan.outputs.packages }}
|
||||
@@ -122,7 +122,7 @@ jobs:
|
||||
name: Build preview migrator
|
||||
needs: plan_preview
|
||||
if: needs.plan_preview.outputs.packages == 'true'
|
||||
runs-on: ${{ vars.AWS_POST_MERGE_CI_ENABLED == 'true' && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.event_name == 'workflow_dispatch' && inputs.channel == 'cloud-migrator' && github.sha != '' && inputs.source_ref == github.sha && 'runs-on/fleet=paperclip-post-merge-x64/env=public-ci' || 'ubuntu-latest' }}
|
||||
runs-on: ${{ vars.AWS_POST_MERGE_CI_ENABLED == 'true' && github.run_attempt == 1 && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.event_name == 'workflow_dispatch' && inputs.channel == 'cloud-migrator' && github.sha != '' && inputs.source_ref == github.sha && 'runs-on/fleet=paperclip-post-merge-x64/env=public-ci' || 'ubuntu-latest' }}
|
||||
timeout-minutes: 30
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
@@ -20,8 +20,8 @@ concurrency:
|
||||
jobs:
|
||||
chaos_and_recovery:
|
||||
name: Restart, replay, trace, and recovery faults
|
||||
runs-on: ${{ vars.AWS_POST_MERGE_CI_ENABLED == 'true' && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.sha != '' && inputs.ref == github.sha && 'runs-on/fleet=paperclip-post-merge-x64/env=public-ci' || 'ubuntu-latest' }}
|
||||
timeout-minutes: ${{ vars.AWS_POST_MERGE_CI_ENABLED == 'true' && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.sha != '' && inputs.ref == github.sha && 40 || 45 }}
|
||||
runs-on: ${{ vars.AWS_POST_MERGE_CI_ENABLED == 'true' && github.run_attempt == 1 && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.sha != '' && inputs.ref == github.sha && 'runs-on/fleet=paperclip-post-merge-x64/env=public-ci' || 'ubuntu-latest' }}
|
||||
timeout-minutes: ${{ vars.AWS_POST_MERGE_CI_ENABLED == 'true' && github.run_attempt == 1 && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.sha != '' && inputs.ref == github.sha && 40 || 45 }}
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
|
||||
@@ -382,6 +382,13 @@ config; an explicit shared test database retains one partition per job.
|
||||
The Runner transport lease-expiry and cold-admission regressions live in their
|
||||
own file so those real-clock waits overlap other files in the existing pool.
|
||||
|
||||
Exact-source release verification uses the post-merge AWS fleet on the first
|
||||
attempt. Reruns of its verification and chaos jobs use GitHub-hosted runners.
|
||||
Exact-source cloud-migrator planning and package builds use the same policy.
|
||||
This gives interrupted jobs an independent recovery path. It keeps the same
|
||||
source, commands, assertions and required gates. A failed test still needs a fix;
|
||||
a successful rerun does not explain an earlier failure.
|
||||
|
||||
The two PR Runner Vitest jobs separate package tests from native server tests.
|
||||
The first runs the complete package inventory; the second runs the complete
|
||||
native server group. The existing `--shard=1/2` and `--shard=2/2` commands route
|
||||
|
||||
@@ -209,33 +209,14 @@ describeEmbeddedPostgres("shared-workspace run serialization", () => {
|
||||
await db.delete(companies);
|
||||
}
|
||||
|
||||
async function waitForRunToLeaveActiveStates(runId: string, timeoutMs = 10_000) {
|
||||
const deadline = Date.now() + timeoutMs;
|
||||
while (Date.now() < deadline) {
|
||||
const run = await heartbeat.getRun(runId);
|
||||
if (run && !["queued", "running"].includes(run.status)) return run;
|
||||
await new Promise((resolve) => setTimeout(resolve, 50));
|
||||
}
|
||||
return await heartbeat.getRun(runId);
|
||||
async function waitForRunToLeaveActiveStates(runId: string) {
|
||||
await heartbeat.drainActiveRunExecutions();
|
||||
return heartbeat.getRun(runId);
|
||||
}
|
||||
|
||||
// A deferral does two writes in order: first it cancels the original run,
|
||||
// then it inserts the scheduled-retry row. waitForRunToLeaveActiveStates
|
||||
// returns after the first write, so a read of the retry row can land before
|
||||
// the second write and find nothing. Poll until the retry row exists so the
|
||||
// retry-row assertions never observe the gap between the two writes.
|
||||
async function waitForRetryRun(originalRunId: string, timeoutMs = 10_000) {
|
||||
const deadline = Date.now() + timeoutMs;
|
||||
while (Date.now() < deadline) {
|
||||
const retryRun = await db
|
||||
.select()
|
||||
.from(heartbeatRuns)
|
||||
.where(eq(heartbeatRuns.retryOfRunId, originalRunId))
|
||||
.then((rows) => rows[0] ?? null);
|
||||
if (retryRun) return retryRun;
|
||||
await new Promise((resolve) => setTimeout(resolve, 50));
|
||||
}
|
||||
return await db
|
||||
async function waitForRetryRun(originalRunId: string) {
|
||||
await heartbeat.drainActiveRunExecutions();
|
||||
return db
|
||||
.select()
|
||||
.from(heartbeatRuns)
|
||||
.where(eq(heartbeatRuns.retryOfRunId, originalRunId))
|
||||
|
||||
@@ -70,11 +70,11 @@ describe.skipIf(!support.supported && !externalDatabaseUrl)("heartbeat workspace
|
||||
if (external) await db.execute(sql`truncate table environments restart identity cascade`);
|
||||
}, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS);
|
||||
afterAll(async () => {
|
||||
await db?.$client.end();
|
||||
await db?.$client.end({ timeout: 1 });
|
||||
await database?.cleanup();
|
||||
vi.unstubAllEnvs();
|
||||
if (home) await rm(home, { recursive: true, force: true });
|
||||
});
|
||||
}, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS);
|
||||
beforeEach(async () => {
|
||||
persistenceBefore = new Map(persistenceKeys.map(key => [key, process.env[key]]));
|
||||
vi.restoreAllMocks();
|
||||
|
||||
Reference in new issue
Block a user