From 4265cb3a2b8056bed0e2583ac7b88aa706f1622f Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Thu, 8 Oct 2026 14:33:10 -0700 Subject: [PATCH 1/5] fix(ci): cache native server integration builds in release verification (#15619) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Release verification must run the same server integration coverage as PR verification. > - Three server suites use real Rust Runner binaries. > - PR checks already run them with the shared Rust dependency cache, but release checks cold-build them inside ordinary server test shards. > - A cold Dot Runner build can consume its entire setup deadline before any test runs. > - This pull request gives release verification a required cached native integration lane and preserves every test. ## Linked Issues or Issue Description **What happened?** On master commit `1881894973a2b25838d8abed9bd8aeebc3af4441`, [Cloud readiness run 37837810707](https://github.com/paperclipai/paperclip/actions/runs/37837810707) failed in server shard 7. Dot Runner's `cargo build --release` exceeded its 300-second child-process deadline. The other 1,640 tests in that shard passed. The failed setup then tried to remove an undefined temporary path and emitted a second error. Cargo output was captured as an opaque buffer, which obscured build progress. **What did you expect to happen?** Build fixture binaries before tests in a lane with the existing Rust dependency cache. Run all native integration tests and make their result required for release readiness. A setup failure should keep its original diagnostic. **Steps to reproduce** Run release verification on a clean runner. The existing `general-server-without-chat` group retains the three Cargo-backed suites outside the PR workflow. The Dot suite can time out during a cold release build. Run the new workflow and partition tests against the previous source to reproduce five routing and coverage failures deterministically. **Version / commit** Observed at `1881894973a2b25838d8abed9bd8aeebc3af4441`; this change is based on `ed6abbf158b`. **Deployment mode** GitHub Actions Release and Cloud readiness verification. No application runtime or deployment action changes. Related work: #15581 also edits Dot integration tests for onboarding behavior. It does not repair release test routing. Searches found no open PR for this failure. ## What Changed - Add an explicit server test group that excludes the dedicated chat and native suites. Preserve the existing PR and local test groups. - Run all three native server suites in one required matrix lane with the existing trusted Rust dependency cache. - Build debug and release fixture binaries in a visible step with a 10-minute limit before tests start. Keep Cargo freshness checks and the existing test deadlines. - Stream Cargo diagnostics and clean up safely when Dot setup stops before creating its temporary directory. - Test complete, non-overlapping partitions for Release, Cloud readiness, other callers, and existing PR/local groups. Check cache restrictions, build order, and the source verification dependency. ## Verification - Passed 44 workflow and partition tests: `node --test scripts/__tests__/run-vitest-stable-shard.test.mjs scripts/__tests__/release-verify-workflow.test.mjs`. - The same tests fail in five relevant cases against the previous workflow and selector. They pass after this correction. - An independent review repeated all 44 tests successfully. - Passed `actionlint .github/workflows/release-verify.yml`, `git diff --check`, and a secret scan. - Passed all 381 workflow policy tests: `node --test '.github/scripts/tests/*.test.mjs'`. - Passed full `pnpm build` in a clean worktree. - Built debug and release fixture binaries, then passed all 32 tests in the three real native integration suites: `pnpm test:run:general -- --group general-server-native-runner` (49.5 seconds, no skips). - Passed full `pnpm -r typecheck`. - Injected a synthetic Cargo setup failure. The suite reports that failure without the secondary undefined-path cleanup error. - Exact-head CI completed: 53 successful checks, including all server shards, both native Runner lanes, build, typecheck, browser tests, and the canary dry run. Two optional Storybook checks were skipped. - Started the duplicate local `pnpm test:run` aggregate and stopped it after exact-head CI passed. No completed local aggregate result is claimed. All test processes owned by that run exited. - Greptile scored the final head `ffe5ab5a28af2dfea9db1c1952c652f070bf52f8` at 5/5 with no review threads. - `Superagent Supply Chain Scan` is neutral, not passed: it only supports exact dependency pin replacements and cannot verify structural edits to `.github/workflows/release-verify.yml`. It reported no annotations. Independent source review, Greptile, actionlint, and the workflow policy tests cover this structural change. - GitHub still requires a code-owner review for the workflow files. There are no merge conflicts. ## Risks - Adds one CI matrix job, which increases concurrent runner demand. The existing cache writer and trust restrictions stay in place. - Missing dependencies still compile from the lockfile. A build that exceeds its step limit fails visibly; failed tests still block source verification. - Workflow execution uses the new lane after merge. PR tests validate the workflow contract and run the existing native test lane. - The supply chain scanner cannot analyze this workflow structure. Its neutral result is an explicit coverage limit; it is not counted as a successful scan. - No runtime, schema, deployment, publication, credential, or test-timeout changes. ## Model Used OpenAI Codex, based on GPT-6, with repository inspection, code execution, and independent agent review. The exact deployed model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- .../tests/release-runner-cache.test.mjs | 11 +++-- .github/workflows/release-verify.yml | 42 ++++++++++++------ doc/cloud-build-readiness.md | 11 +++-- .../scripts/run-pr-vitest-lane.mjs | 8 ++-- .../release-verify-workflow.test.mjs | 28 +++++++++++- .../run-vitest-stable-shard.test.mjs | 22 ++++++++-- scripts/run-vitest-stable.mjs | 44 ++++++++++--------- server/src/__tests__/dot-runner.test.ts | 14 +++++- 8 files changed, 130 insertions(+), 50 deletions(-) diff --git a/.github/scripts/tests/release-runner-cache.test.mjs b/.github/scripts/tests/release-runner-cache.test.mjs index 96bba1f3250..b6f478122e6 100644 --- a/.github/scripts/tests/release-runner-cache.test.mjs +++ b/.github/scripts/tests/release-runner-cache.test.mjs @@ -38,14 +38,19 @@ test("parallel lanes cover check:all exactly once and never bypass verification" const scripts = JSON.parse(readFileSync(new URL("../../../packages/paperclip-runner/package.json", import.meta.url))).scripts; const checks = [...runner.matchAll(/^ checks: (.+)$/gm)].flatMap(([, value]) => value.split(" ")); assert.deepEqual(checks, scripts["check:all"].split(" && ").map((command) => command.replace(/^pnpm run /, ""))); - assert.deepEqual([...runner.matchAll(/^ - lane: (.+)$/gm)].map(([, value]) => value), ["protocol", "rust"]); + assert.deepEqual([...runner.matchAll(/^ - lane: (.+)$/gm)].map(([, value]) => value), ["protocol", "rust", "server-integration"]); assert.match(runner, /fail-fast: false/); assert.doesNotMatch(runner, /max-parallel: 1|^ needs:|continue-on-error:/m); const verify = runner.split(" - name: Verify Paperclip Runner\n")[1].split(" - name: Warm debug")[0]; assert.match(verify, /RUNNER_CHECKS: \$\{\{ matrix.checks \}\}/); assert.match(verify, /set -euo pipefail/); assert.match(verify, /for check in \$RUNNER_CHECKS; do\s+pnpm --filter @paperclipai\/paperclip-runner "\$check"\s+done/); - assert.doesNotMatch(verify, /if:|cache-hit/); + const verifyIf = verify.match(/^\s*if: \$\{\{ (.+) \}\}$/m)?.[1]; + assert.equal(verifyIf, "matrix.lane != 'server-integration'"); + for (const lane of ["protocol", "rust", "server-integration"]) { + assert.equal(runInNewContext(verifyIf, { matrix: { lane } }), lane !== "server-integration"); + } + assert.doesNotMatch(verify, /cache-hit/); assert.doesNotMatch(runner, /id-token: write|packages: write|secrets: inherit/); }); @@ -57,7 +62,7 @@ test("only the trusted Rust lane writes, and warms both build profiles before sa assert.match(warm, /run: pnpm --filter @paperclipai\/paperclip-runner build:rust/); const sha = "a".repeat(40); const base = { repository: "paperclipai/paperclip", event_name: "push", ref: "refs/heads/master", sha }; - for (const lane of ["protocol", "rust"]) { + for (const lane of ["protocol", "rust", "server-integration"]) { for (const [overrides, ref, trusted] of [ [{}, sha, true], [{ event_name: "pull_request", ref: "refs/pull/1/merge" }, sha, false], diff --git a/.github/workflows/release-verify.yml b/.github/workflows/release-verify.yml index 76e5cd28bb3..492b7e62b47 100644 --- a/.github/workflows/release-verify.yml +++ b/.github/workflows/release-verify.yml @@ -107,45 +107,46 @@ jobs: matrix: include: # Split the long chat file by collected test locations, and balance - # the remaining server files across ten runners. Normal PR/local - # invocations retain their complete general-server group. - - group: general-server-without-chat + # the remaining server files across ten runners. Rust-backed server + # suites run in the cached Runner lane below. Local invocations retain + # their complete general-server group. + - group: general-server-without-chat-or-native-runner group_label: server (1/10) shard_index: 0 shard_count: 10 - - group: general-server-without-chat + - group: general-server-without-chat-or-native-runner group_label: server (2/10) shard_index: 1 shard_count: 10 - - group: general-server-without-chat + - group: general-server-without-chat-or-native-runner group_label: server (3/10) shard_index: 2 shard_count: 10 - - group: general-server-without-chat + - group: general-server-without-chat-or-native-runner group_label: server (4/10) shard_index: 3 shard_count: 10 - - group: general-server-without-chat + - group: general-server-without-chat-or-native-runner group_label: server (5/10) shard_index: 4 shard_count: 10 - - group: general-server-without-chat + - group: general-server-without-chat-or-native-runner group_label: server (6/10) shard_index: 5 shard_count: 10 - - group: general-server-without-chat + - group: general-server-without-chat-or-native-runner group_label: server (7/10) shard_index: 6 shard_count: 10 - - group: general-server-without-chat + - group: general-server-without-chat-or-native-runner group_label: server (8/10) shard_index: 7 shard_count: 10 - - group: general-server-without-chat + - group: general-server-without-chat-or-native-runner group_label: server (9/10) shard_index: 8 shard_count: 10 - - group: general-server-without-chat + - group: general-server-without-chat-or-native-runner group_label: server (10/10) shard_index: 9 shard_count: 10 @@ -297,6 +298,7 @@ jobs: checks: check:eval-kernel check:protocol - lane: rust checks: check:runner check:api-authority + - lane: server-integration steps: - name: Checkout repository @@ -373,14 +375,28 @@ jobs: # dependencies; never restore installed executables from cargo/bin. cache-workspace-crates: false cache-bin: false - # Both lanes restore the existing dependency cache. Only the Rust + # All lanes restore the existing dependency cache. Only the Rust # lane saves it, after warming both release and debug dependencies. save-if: ${{ matrix.lane == 'rust' && github.repository == 'paperclipai/paperclip' && github.event_name == 'push' && github.ref == 'refs/heads/master' && inputs.ref == github.sha }} - name: Install dependencies run: pnpm install --no-frozen-lockfile + - name: Build native server test binaries + if: ${{ matrix.lane == 'server-integration' }} + timeout-minutes: 10 + working-directory: packages/paperclip-runner + run: | + set -euo pipefail + pnpm build:rust + cargo build --release --manifest-path runner/Cargo.toml --locked -p paperclip-runner-core --bin paperclip-runnerd --bin fake-codex-app-server + + - name: Run native server integration suites + if: ${{ matrix.lane == 'server-integration' }} + run: pnpm test:run:general -- --group general-server-native-runner + - name: Verify Paperclip Runner + if: ${{ matrix.lane != 'server-integration' }} env: RUNNER_CHECKS: ${{ matrix.checks }} run: | diff --git a/doc/cloud-build-readiness.md b/doc/cloud-build-readiness.md index df143f11eca..18611b833b0 100644 --- a/doc/cloud-build-readiness.md +++ b/doc/cloud-build-readiness.md @@ -33,9 +33,14 @@ test jobs on standard runners. Measure queue time to assess the timing gain. Release verification spreads the general server suites across ten standard hosted runners, with the long chat suite split separately across three jobs. Each server -job still runs one test worker. The partition covers every suite exactly once; -normal PR and local test groups keep their existing shape. More jobs increase -concurrent runner demand, so compare queue time as well as test duration. +job still runs one test worker. The three Rust-backed server suites run in a +separate Runner lane with the shared dependency cache. That lane builds both +debug and release test binaries in a visible, bounded step before Vitest starts; +the suites still check Cargo freshness and run against the current source. A +cache miss builds from source. Native test failures block source verification. +The partition covers every suite exactly once; normal PR and local test groups +keep their existing shape. More jobs increase concurrent runner demand, so compare +queue time as well as test duration. All release verification installs, including the Runner scorer and chaos evals, allow pnpm to refresh an outdated lockfile. Contributor PRs leave lockfile updates diff --git a/packages/paperclip-runner/scripts/run-pr-vitest-lane.mjs b/packages/paperclip-runner/scripts/run-pr-vitest-lane.mjs index e383bd90246..cc49d2a94e2 100644 --- a/packages/paperclip-runner/scripts/run-pr-vitest-lane.mjs +++ b/packages/paperclip-runner/scripts/run-pr-vitest-lane.mjs @@ -15,10 +15,10 @@ // Contract, mirrored in scripts/run-vitest-stable.mjs (prWorkflowName) and // pinned by scripts/__tests__/run-vitest-stable-shard.test.mjs: // - Only the PR workflow (pr.yml, whose GITHUB_WORKFLOW the reusable -// pr-trusted.yml jobs inherit) excludes the suite from the server shards, -// and only there does this wrapper run it. Any other caller — local runs, -// release-verify.yml — keeps the suite in the server group, so a renamed -// workflow degrades to the slower covered path instead of losing coverage. +// pr-trusted.yml jobs inherit) excludes these suites from the existing +// without-chat group, and only there does this wrapper run them. Other +// callers keep that group complete. Release verification instead selects +// an explicit partition and its own required cached native lane. // - The suite runs on the lane whose --shard=N/M has N === M (or an unsharded // invocation), so exactly one PR lane carries it. import { spawnSync } from "node:child_process"; diff --git a/scripts/__tests__/release-verify-workflow.test.mjs b/scripts/__tests__/release-verify-workflow.test.mjs index 293280c2d14..98717aa8afa 100644 --- a/scripts/__tests__/release-verify-workflow.test.mjs +++ b/scripts/__tests__/release-verify-workflow.test.mjs @@ -244,10 +244,10 @@ test("release verify workflow covers the same split test surface as stable PR ve assert.match(buildJob, /persist-credentials: false/); assert.doesNotMatch(buildJob, /cache: pnpm/); - for (const group of ["general-server-without-chat", "general-chat", "general-workspaces-a", "general-workspaces-b"]) { + for (const group of ["general-server-without-chat-or-native-runner", "general-chat", "general-workspaces-a", "general-workspaces-b"]) { assert.match(verifyWorkflow, new RegExp(`group: ${group}`)); } - for (const [group, count] of [["general-server-without-chat", 10], ["general-chat", 3]]) { + for (const [group, count] of [["general-server-without-chat-or-native-runner", 10], ["general-chat", 3]]) { const rows = [...verifyWorkflow.matchAll(new RegExp(`group: ${group}\\n\\s+group_label: [^\\n]+\\n\\s+shard_index: (\\d+)\\n\\s+shard_count: (\\d+)`, "g"))]; assert.deepEqual(rows.map((row) => [Number(row[1]), Number(row[2])]), Array.from({ length: count }, (_, index) => [index, count])); @@ -271,6 +271,30 @@ test("release verify workflow covers the same split test surface as stable PR ve assert.match(verifyWorkflow, /pnpm test:run:serialized -- --shard-index/); }); +test("release verification builds native test binaries in a required Rust-cached lane", () => { + const workflow = readWorkflow("release-verify.yml"); + const runnerJob = workflow.match(/ verify_paperclip_runner:\n[\s\S]*?(?=\n [A-Za-z0-9_-]+:|$)/)?.[0] ?? ""; + assert.equal((runnerJob.match(/- lane: server-integration/g) ?? []).length, 1); + assert.doesNotMatch(runnerJob, /continue-on-error/); + assert.match(runnerJob, /timeout-minutes: 20/); + assert.match(runnerJob, /shared-key: release-runner-v2/); + assert.match(runnerJob, /cache-workspace-crates: false/); + assert.match(runnerJob, /cache-bin: false/); + assert.match(runnerJob, /save-if: \$\{\{ matrix\.lane == 'rust'/); + assert.match(runnerJob, /Build native server test binaries\n\s+if: \$\{\{ matrix\.lane == 'server-integration' \}\}\n\s+timeout-minutes: 10/); + assert.match(runnerJob, /pnpm build:rust\n\s+cargo build --release --manifest-path runner\/Cargo\.toml --locked -p paperclip-runner-core --bin paperclip-runnerd --bin fake-codex-app-server/); + assert.match(runnerJob, /Run native server integration suites\n\s+if: \$\{\{ matrix\.lane == 'server-integration' \}\}\n\s+run: pnpm test:run:general -- --group general-server-native-runner/); + assert.ok(runnerJob.indexOf("Cache Runner Rust dependencies") < runnerJob.indexOf("Build native server test binaries")); + assert.ok(runnerJob.indexOf("Build native server test binaries") < runnerJob.indexOf("Run native server integration suites")); + + // The reusable workflow result includes every matrix child. Its caller must + // await that result without an override that can certify a failed native lane. + const cloud = readWorkflow("cloud-readiness.yml"); + assert.match(cloud, /verify:[\s\S]*?uses: \.\/\.github\/workflows\/release-verify\.yml/); + assert.match(cloud, /source_verified:[\s\S]*?needs: \[verify\]/); + assert.doesNotMatch(cloud, /continue-on-error|always\(\)/); +}); + test("Runner eval workflows pin actions and gate paid live execution", () => { const actionPinWorkflows = [ readWorkflow("release-verify.yml"), diff --git a/scripts/__tests__/run-vitest-stable-shard.test.mjs b/scripts/__tests__/run-vitest-stable-shard.test.mjs index d4fcb7a0d55..a044c67c151 100644 --- a/scripts/__tests__/run-vitest-stable-shard.test.mjs +++ b/scripts/__tests__/run-vitest-stable-shard.test.mjs @@ -319,9 +319,8 @@ test("12 PR without-chat shards plus the dedicated chat and native-runner lanes assert.ok(defaultRun.generalServerSuiteCount === full.generalServerSuiteCount); }); -// Mirrors release-verify.yml (10 shards, called by the Release and Cloud -// readiness workflows) and local runs: no Rust-cached vitest lane exists -// there, so the native-runner suite must stay in the server shards. +// The legacy group and local default must remain complete when a caller has +// not selected a separate native lane. for (const [caller, envOverrides] of [["Release", { GITHUB_WORKFLOW: "Release" }], ["no ambient workflow", {}]]) { test(`10 without-chat shards under ${caller} keep the native-runner suite and cover the server group with chat alone`, () => { const full = dryRunJson(["--mode", "general", "--group", "general-server", "--shard-index", "0", "--shard-count", "1"], envOverrides); @@ -339,6 +338,23 @@ for (const [caller, envOverrides] of [["Release", { GITHUB_WORKFLOW: "Release" } }); } +// Release verification selects its dedicated lane explicitly. Do not infer it +// from the caller name: previews and future workflow callers need the same cover. +for (const caller of ["Release", "Cloud readiness", "another caller"]) { + test(`release server, chat, and native lanes cover every server suite once under ${caller}`, () => { + const env = { GITHUB_WORKFLOW: caller }; + const full = dryRunJson(["--mode", "general", "--group", "general-server", "--shard-index", "0", "--shard-count", "1"], env); + const files = Array.from({ length: 10 }, (_, index) => dryRunJson([ + "--mode", "general", "--group", "general-server-without-chat-or-native-runner", + "--shard-index", String(index), "--shard-count", "10", + ], env)).flatMap((shard) => shard.selectedGeneralServerSuites); + const native = dryRunJson(["--mode", "general", "--group", "general-server-native-runner"], env); + const combined = [...files, chatSuitePath, ...native.selectedGeneralServerSuites]; + assert.equal(new Set(combined).size, combined.length, "lanes must not overlap"); + assert.deepEqual(combined.sort(), full.selectedGeneralServerSuites.sort()); + }); +} + test("the native-runner lane runs exactly the cargo-dependent suites", () => { const lane = dryRunJson(["--mode", "general", "--group", "general-server-native-runner"]); assert.deepEqual(lane.selectedGeneralServerSuites, [ diff --git a/scripts/run-vitest-stable.mjs b/scripts/run-vitest-stable.mjs index 1d98b848af0..dd00f145cda 100644 --- a/scripts/run-vitest-stable.mjs +++ b/scripts/run-vitest-stable.mjs @@ -76,6 +76,7 @@ const generalModeName = "general"; const allModeName = "all"; const generalServerGroupName = "general-server"; const generalServerWithoutChatGroupName = "general-server-without-chat"; +const generalServerWithoutChatOrNativeRunnerGroupName = "general-server-without-chat-or-native-runner"; const generalChatGroupName = "general-chat"; const generalServerNativeRunnerGroupName = "general-server-native-runner"; const chatSuite = "server/src/__tests__/chat-channels.integration.test.ts"; @@ -92,10 +93,9 @@ const nativeRunnerSuites = [ // In the PR workflow (pr.yml, the caller of pr-trusted.yml — reusable // workflows inherit the caller's GITHUB_WORKFLOW), the last Verify Paperclip // Runner vitest shard runs the native-runner group instead, because those -// lanes restore the shared release-runner-v1 Rust cache (see +// lanes restore the shared release-runner-v2 Rust cache (see // packages/paperclip-runner/scripts/run-pr-vitest-lane.mjs). Every other -// caller — local runs, release-verify.yml under the Release and Cloud -// readiness workflows — keeps the suite in the server shards, so a renamed or +// caller of the same group keeps the suite in the server shards, so a renamed or // unknown workflow degrades to today's slower-but-covered behavior rather // than dropping the suite. const prWorkflowName = "PR"; @@ -103,6 +103,18 @@ const nativeRunnerSuiteRunsInRustCachedLane = process.env.GITHUB_WORKFLOW === pr const withoutChatExcludedSuites = nativeRunnerSuiteRunsInRustCachedLane ? [chatSuite, ...nativeRunnerSuites] : [chatSuite]; +// Release verification selects this explicit group and runs the omitted suites +// in its cached Runner job. Local callers keep the complete default group. +const generalServerGroups = [ + generalServerGroupName, + generalServerWithoutChatGroupName, + generalServerWithoutChatOrNativeRunnerGroupName, +]; +function excludedServerSuites(groupName) { + if (groupName === generalServerWithoutChatOrNativeRunnerGroupName) return [chatSuite, ...nativeRunnerSuites]; + if (groupName === generalServerWithoutChatGroupName) return withoutChatExcludedSuites; + return []; +} const generalWorkspacesAGroupName = "general-workspaces-a"; const generalWorkspacesBGroupName = "general-workspaces-b"; const generalWorkspacesAProjects = ["@paperclipai/ui", "paperclipai"]; @@ -111,6 +123,7 @@ const generalGroupNames = [generalServerGroupName, generalWorkspacesAGroupName, const allowedGeneralGroupNames = [ ...generalGroupNames, generalServerWithoutChatGroupName, + generalServerWithoutChatOrNativeRunnerGroupName, generalChatGroupName, generalServerNativeRunnerGroupName, ]; @@ -290,7 +303,7 @@ function parseCliOptions(argv) { const shardAllowed = mode === serializedModeName || (mode === generalModeName && - ([generalServerGroupName, generalServerWithoutChatGroupName, generalChatGroupName, generalWorkspacesAGroupName].includes(group))); + ([...generalServerGroups, generalChatGroupName, generalWorkspacesAGroupName].includes(group))); if (!shardAllowed && shardIndex !== null) { fail( "--shard-index/--shard-count are only valid with serialized mode or a shardable general server/chat/workspaces-a group.", @@ -448,14 +461,9 @@ function runGeneralGroup(routeTests, groupName, shardIndex = null, shardCount = ); return; } - if (groupName === generalServerGroupName || groupName === generalServerWithoutChatGroupName) { - // In the PR workflow the without-chat group also leaves the native-runner - // suite to the Rust-cached vitest lane; the full general-server group - // (local runs) keeps both. - const withoutChat = groupName === generalServerWithoutChatGroupName; - const files = withoutChat - ? generalServerTestFiles.filter((file) => !withoutChatExcludedSuites.includes(file)) - : generalServerTestFiles; + if (generalServerGroups.includes(groupName)) { + const excludedSuites = excludedServerSuites(groupName); + const files = generalServerTestFiles.filter((file) => !excludedSuites.includes(file)); if (shardCount !== null && shardCount > 1) { const shardFiles = selectGeneralServerShard( files, @@ -483,10 +491,8 @@ function runGeneralGroup(routeTests, groupName, shardIndex = null, shardCount = } const excludeRouteArgs = routeTests.flatMap((file) => ["--exclude", file.serverPath]); - if (withoutChat) { - for (const suite of withoutChatExcludedSuites) { - excludeRouteArgs.push("--exclude", suite.replace(/^server\//, "")); - } + for (const suite of excludedSuites) { + excludeRouteArgs.push("--exclude", suite.replace(/^server\//, "")); } runVitest( [ @@ -584,12 +590,10 @@ if (options.dryRun) { options.mode === generalModeName && options.group === generalServerNativeRunnerGroupName ? nativeRunnerSuites : options.mode === generalModeName && - [generalServerGroupName, generalServerWithoutChatGroupName].includes(options.group) && + generalServerGroups.includes(options.group) && options.shardCount !== null ? selectGeneralServerShard( - options.group === generalServerWithoutChatGroupName - ? generalServerTestFiles.filter((file) => !withoutChatExcludedSuites.includes(file)) - : generalServerTestFiles, + generalServerTestFiles.filter((file) => !excludedServerSuites(options.group).includes(file)), options.shardIndex, options.shardCount, generalServerShardDurations, diff --git a/server/src/__tests__/dot-runner.test.ts b/server/src/__tests__/dot-runner.test.ts index c6b586db09b..fe7caa79c85 100644 --- a/server/src/__tests__/dot-runner.test.ts +++ b/server/src/__tests__/dot-runner.test.ts @@ -37,7 +37,7 @@ describe("durable Dot Runner integration", () => { let root: string; beforeAll(async () => { const runnerRoot = fileURLToPath(new URL("../../../packages/paperclip-runner/", import.meta.url)); - execFileSync("cargo", ["build", "--release", "--locked", "--manifest-path", join(runnerRoot, "runner/Cargo.toml"), "-p", "paperclip-runner-core", "--bin", "paperclip-runnerd"], { cwd: runnerRoot, stdio: "pipe", timeout: 300000 }); + execFileSync("cargo", ["build", "--release", "--locked", "--manifest-path", join(runnerRoot, "runner/Cargo.toml"), "-p", "paperclip-runner-core", "--bin", "paperclip-runnerd"], { cwd: runnerRoot, stdio: "inherit", timeout: 300000 }); temporary = await startEmbeddedPostgresTestDatabase("paperclip-dot-runner-"); db = createDb(temporary.connectionString); root = await mkdtemp(join(tmpdir(), "paperclip-dot-state-")); @@ -47,7 +47,17 @@ describe("durable Dot Runner integration", () => { vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", randomBytes(32).toString("base64")); await instanceSettingsService(db).updateExperimental({ enablePublicMcp: true, enableOpenAiDot: true, enableNativeRunner: false }); }, 360000); - afterAll(async () => { await temporary?.cleanup(); await rm(root, { recursive: true, force: true }); vi.unstubAllEnvs(); }); + afterAll(async () => { + try { + await temporary?.cleanup(); + } finally { + try { + if (root) await rm(root, { recursive: true, force: true }); + } finally { + vi.unstubAllEnvs(); + } + } + }); it("requires each persisted prerequisite for pairing and new work without relying on the retired environment flag", async () => { const settings = instanceSettingsService(db); From 9b624a110e281badc54cbdf37ddb792cdc1690ff Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Thu, 8 Oct 2026 15:58:55 -0700 Subject: [PATCH 2/5] fix: checkpoint database backups before idle sleep (#15629) ## Thinking Path > - Paperclip coordinates agent work and preserves company state. > - Hosted instances can stop compute when admission and all work sources are quiet. > - The periodic database backup timer currently blocks every automatic sleep. > - Disabling backups would remove recovery points. > - This change offers an explicit final-backup checkpoint before sleep. > - A verified archive and restart marker preserve recovery while compute is stopped. ## Linked Issues or Issue Description **What happened?** An otherwise idle instance with automatic backups enabled always reports background work. Operators cannot reclaim idle compute without disabling those backups. The work inventory also queries the database before checking known local blockers. **Expected behavior** An operator can enable checkpoint mode. The server may authorize sleep only after all other work is quiet and a fresh backup is verified under the same owned hold. Backups resume on restart with the existing retention policy. **Steps to reproduce** Acquire a bounded idle drain on an instance with no application work and automatic backups enabled. Read its owned idle safety report. It reports present even when all other work is clear. With this change and `PAPERCLIP_DB_BACKUP_IDLE_CHECKPOINT_ENABLED=1`, a successful final backup can permit sleep; backup failures still refuse it. **Paperclip version or commit** Reproduced from master at 4265cb3a2b. **Deployment mode** Hosted single-process instances with persistent backup storage and serialized sleep/wake operations. Searched open backup and sleep PRs and issues. Related: #15522 and #15599 establish owned idle holds and plugin drains. #15358 proposes general backup health/retention validation; this change verifies only opted-in sleep checkpoints and their restart catch-up. ## What Changed - Add an off-by-default checkpoint flag. Keep automatic backups enabled. - Reject known local blockers before database work. - Finish a new backup during each owned safety check, including final revalidation. - Verify the complete gzip archive and sync its file and directory before pruning older recovery points. - Use unique checkpoint filenames so repeated checks cannot replace an earlier verified archive. - Keep backup work counted through dump, validation and persistence. Changed owners, expired holds, concurrent work and errors refuse sleep. - Persist a restart marker before the dump. Attempt a fresh backup on restart before idle eligibility, then resume the normal timer. Only a verified fresh backup clears the marker. - Restrict managed checkpoint reads to verified Cloud control actors. Tenant instance-admin elevation does not grant backup authority. - Keep archive verification outside engine fallback so failures retain their original cause. - Document that recurring duplicate archives pause while asleep. The final checkpoint and historical archives remain on persistent storage; retention does not prune while asleep. ## Verification - Focused idle, spool, backup and route checks: 137 tests passed, including restoration into a separate PostgreSQL database. - Backup library and checkpoint tests: 15 passed, including rejection before historical backups can be pruned. - `pnpm -r typecheck` and `pnpm build` passed. Follow-up server typecheck and database build cover the verification callback. - Review follow-up: 51 backup/restore/startup tests, 122 route/safety tests, 32 signed-control tests, and server typecheck passed. Startup cases exercise both flag states and retry after failure. The final test-cleanup adjustment passes all 35 startup tests. - The full local run hit four ancestor-directory skill-fixture failures and one tool-route failure. All five passed from an isolated checkout at 949210 (3 targeted chat/tool checks and the complete 39-test email suite). The redundant local run was stopped after final-head hosted CI passed. No full local pass is claimed. - Added-line secret/private-reference review and `git diff --check` passed. - [Final-head hosted CI](https://github.com/paperclipai/paperclip/actions/runs/37853380365) passed, including typecheck, build, all general and serialized test shards, all eight E2E shards and the canary dry run. Apex is 5/5 on 5b1de388ef, with no new findings and all review threads resolved. The branch is conflict-free. Live staging activation awaits merge and a canonical image; no production configuration or deployment changed. ## Risks - The flag defaults off. This is for a single process whose host controls every writer and preserves its backup volume across sleep. Independent database writers require a different backup owner. - Safety reads perform a backup only after a valid owned drain and all other checks pass. Slow backups can outlive the caller timeout; they remain counted until their work settles and cannot authorize sleep after hold expiry. - Initial and final reads each take a fresh backup. Hosts must allow sufficient request time while retaining shutdown headroom. - Retention settings are unchanged. Sleeping instances stop producing duplicate hourly archives and retain their last checkpoint. Filesystem data, encryption keys and off-provider disaster recovery remain separate responsibilities. - No schema migration. Disable the checkpoint flag to restore the blanket backup blocker; automatic backups and an existing restart marker remain active. ## Model Used OpenAI Codex, GPT-6. Used reasoning, repository inspection, code editing and command execution. The runtime did not expose an exact model revision or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR following the bug report template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal identifier - [x] Focused and isolated local checks pass; final-head hosted CI is green (local full-run limitation documented above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation - [x] I have considered and documented risks - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all review comments before requesting merge --------- Co-authored-by: Paperclip --- doc/DEVELOPING.md | 3 + doc/idle-sleep-safety.md | 43 ++++++++- packages/db/src/backup-lib.test.ts | 34 ++++++- packages/db/src/backup-lib.ts | 29 ++++-- .../__tests__/idle-database-backup.test.ts | 92 +++++++++++++++++++ server/src/__tests__/idle-local-work.test.ts | 2 + .../src/__tests__/idle-sleep-safety.test.ts | 75 ++++++++++++++- .../instance-settings-routes.test.ts | 44 ++++++++- .../server-startup-feedback-export.test.ts | 68 +++++++++++++- server/src/app.ts | 3 +- server/src/index.ts | 20 +++- .../src/routes/instance-database-backups.ts | 2 +- server/src/routes/instance-settings.ts | 16 +++- server/src/services/idle-database-backup.ts | 52 +++++++++++ server/src/services/idle-local-work.ts | 6 +- server/src/services/idle-sleep-safety.ts | 17 +++- 16 files changed, 476 insertions(+), 30 deletions(-) create mode 100644 server/src/__tests__/idle-database-backup.test.ts create mode 100644 server/src/services/idle-database-backup.ts diff --git a/doc/DEVELOPING.md b/doc/DEVELOPING.md index 9736a2e7be1..84dfd41b1e5 100644 --- a/doc/DEVELOPING.md +++ b/doc/DEVELOPING.md @@ -1687,6 +1687,9 @@ Environment overrides: - `PAPERCLIP_DB_BACKUP_ENABLED=true|false` - `PAPERCLIP_DB_BACKUP_INTERVAL_MINUTES=` +- `PAPERCLIP_DB_BACKUP_IDLE_CHECKPOINT_ENABLED=1` enables verified final backups + for owned idle sleep, with restart catch-up. Off by default; see + [idle sleep safety](idle-sleep-safety.md) for the hosting and storage contract. - `PAPERCLIP_DB_BACKUP_RETENTION_DAYS=` - `PAPERCLIP_DB_BACKUP_DIR=/absolute/or/~/path` - `PAPERCLIP_DB_BACKUP_MAX_AGE_HOURS=` controls the `/api/health` diff --git a/doc/idle-sleep-safety.md b/doc/idle-sleep-safety.md index 61f0977f044..71bd5a549e2 100644 --- a/doc/idle-sleep-safety.md +++ b/doc/idle-sleep-safety.md @@ -52,11 +52,50 @@ counted after readiness until its writes settle. Scheduler work already in flight remains counted until its promise settles. Idle holds pause new scheduler admissions. Database backup promises remain -counted through success or failure. Configured periodic backups also block -sleep until a host owns a durable wake schedule for them. A generation counter invalidates a +counted through success or failure. Configured periodic backups block sleep by +default. The opt-in checkpoint mode below preserves a final backup instead of +running the timer while asleep. A generation counter invalidates a scan if tracked work both starts and finishes during inspection. Reports remain unknown until startup recovery and HTTP tracking are installed. +### Backup checkpoints for idle instances + +`PAPERCLIP_DB_BACKUP_IDLE_CHECKPOINT_ENABLED=1` opts an instance with automatic +backups enabled into this mode. Keep its database and persistent backup volume +attached across sleep. All writers must use the instance's admission protocol; +this mode cannot cover independent writers to the same database. +On managed instances, checkpoint-enabled safety reads require a verified signed +Cloud control actor. Tenant instance-admin elevation does not grant platform +backup authority. Self-hosted instance administrators retain operator access. + +Each owned safety read first checks local work, plugin drains and durable work. +Known local blockers return before opening a database transaction. If every +other check passes, the server takes a fresh backup with the existing backup +engine and the retention policy from Instance Settings. It reads the full gzip +archive, checks its trailer and nonempty content, and syncs the archive and +directory before returning `none`. The final pre-stop revalidation takes another +fresh backup. Reads can therefore take longer; the host must bound its request +and still leave enough lease time for a confirmed provider stop. + +There is no cached backup exemption. Failed, concurrent or untracked backups, +other work during the scan, changed owners and expired holds cannot authorize +sleep. Client timeouts do not clear the backup's in-flight token. Slow backups +may retain an awake instance until a later sleep attempt. + +A durable `.idle-backup-wake-required` marker is written before the dump. +No recurring duplicate archives are produced while the instance is asleep. +The last pre-sleep archive and existing history stay on persistent storage; +no retention pruning runs during sleep. On restart, the server attempts a fresh +backup before enabling idle eligibility, then resumes the normal interval. +Only a verified new backup clears the marker. Failure keeps it for the next +scheduled retry. Disabling automatic backups explicitly still disables this +startup backup. Disabling only checkpoint mode restores the sleep blocker and +continues to honor an existing restart marker. + +This does not replace filesystem backups, encryption-key recovery or off-provider +disaster recovery. Validate restoration of a staging checkpoint before enabling +the flag on customer instances. + The report inspects accounting and sandbox-cleanup spools directly. Any entry, including malformed JSON, temporary files and failed write probes, blocks sleep. Read errors and invalid spool paths return unknown. Recovery readers that skip diff --git a/packages/db/src/backup-lib.test.ts b/packages/db/src/backup-lib.test.ts index b6851534991..de3166714cc 100644 --- a/packages/db/src/backup-lib.test.ts +++ b/packages/db/src/backup-lib.test.ts @@ -1,7 +1,7 @@ import fs from "node:fs"; import os from "node:os"; import path from "node:path"; -import { gunzipSync } from "node:zlib"; +import { gunzipSync, gzipSync } from "node:zlib"; import { afterEach, describe, expect, it } from "vitest"; import postgres from "postgres"; import { createBufferedTextFileWriter, runDatabaseBackup, runDatabaseRestore } from "./backup-lib.js"; @@ -75,6 +75,38 @@ describe("createBufferedTextFileWriter", () => { }); describeEmbeddedPostgres("runDatabaseBackup", () => { + it("preserves a post-pg_dump verification error without falling back or pruning", async () => { + const connectionString = await createTempDatabase(); + const backupDir = createTempDir("checkpoint-pg-dump-"); + const executable = path.join(backupDir, "pg-dump-fixture"); + fs.writeFileSync(executable, "#!/bin/sh\nprintf 'SELECT 1;\\n'\n", { mode: 0o700 }); + const history = path.join(backupDir, "paperclip-history.sql.gz"); + const historyBytes = gzipSync("SELECT 'history';"); + fs.writeFileSync(history, historyBytes); + fs.utimesSync(history, new Date(0), new Date(0)); + const previous = process.env.PAPERCLIP_PG_DUMP_PATH; + process.env.PAPERCLIP_PG_DUMP_PATH = executable; + const failure = new Error("fixture archive fsync failed"); + let verified = 0; + try { + await expect(runDatabaseBackup({ + connectionString, backupDir, + retention: { dailyDays: 7, weeklyWeeks: 4, monthlyMonths: 1 }, + verifyBeforePrune: async archive => { + verified++; + expect(gunzipSync(fs.readFileSync(archive)).toString()).toBe("SELECT 1;\n"); + throw failure; + }, + })).rejects.toBe(failure); + expect(verified).toBe(1); + expect(fs.readFileSync(history)).toEqual(historyBytes); + expect(fs.readdirSync(backupDir).filter(name => name.endsWith(".sql.gz"))).toEqual(["paperclip-history.sql.gz"]); + } finally { + if (previous === undefined) delete process.env.PAPERCLIP_PG_DUMP_PATH; + else process.env.PAPERCLIP_PG_DUMP_PATH = previous; + } + }); + it("preserves identity generation, sequence options and progress in JavaScript backups", async () => { const source = await createTempDatabase(); const target = await createSiblingDatabase(source, "identity_restore_target"); diff --git a/packages/db/src/backup-lib.ts b/packages/db/src/backup-lib.ts index a95ac3f1be8..fdf849de266 100644 --- a/packages/db/src/backup-lib.ts +++ b/packages/db/src/backup-lib.ts @@ -6,6 +6,7 @@ import { open as openFile } from "node:fs/promises"; import { pipeline } from "node:stream/promises"; import { createGunzip, createGzip } from "node:zlib"; import postgres from "postgres"; +import { randomUUID } from "node:crypto"; export type BackupRetentionPolicy = { dailyDays: number; @@ -28,6 +29,8 @@ export type RunDatabaseBackupOptions = { excludeTables?: string[]; nullifyColumns?: Record; backupEngine?: "auto" | "pg_dump" | "javascript"; + /** Optional durability gate. A rejected archive must not prune history. */ + verifyBeforePrune?: (backupFile: string) => Promise; }; export type RunDatabaseBackupResult = { @@ -542,13 +545,17 @@ export async function runDatabaseBackup(opts: RunDatabaseBackupOptions): Promise await sql.end(); }; mkdirSync(opts.backupDir, { recursive: true }); - const sqlFile = resolve(opts.backupDir, `${filenamePrefix}-${timestamp()}.sql`); + // Initial and final sleep checks can run within the same second. A failed + // second checkpoint must never overwrite the first verified recovery point. + const checkpointSuffix = opts.verifyBeforePrune ? `-${randomUUID()}` : ""; + const sqlFile = resolve(opts.backupDir, `${filenamePrefix}-${timestamp()}${checkpointSuffix}.sql`); const backupFile = `${sqlFile}.gz`; const writer = createBufferedTextFileWriter(sqlFile); try { if (backupEngine === "pg_dump" || (backupEngine === "auto" && canUsePgDump)) { await sql`SELECT 1`; + let dumped = false; try { await closeSql(); await runPgDumpBackup({ @@ -556,14 +563,7 @@ export async function runDatabaseBackup(opts: RunDatabaseBackupOptions): Promise backupFile, connectTimeout, }); - await writer.abort(); - const sizeBytes = statSync(backupFile).size; - const prunedCount = pruneOldBackups(opts.backupDir, retention, filenamePrefix); - return { - backupFile, - sizeBytes, - prunedCount, - }; + dumped = true; } catch (error) { if (existsSync(backupFile)) { try { unlinkSync(backupFile); } catch { /* ignore */ } @@ -575,6 +575,16 @@ export async function runDatabaseBackup(opts: RunDatabaseBackupOptions): Promise sql = postgres(opts.connectionString, { max: 1, connect_timeout: connectTimeout }); sqlClosed = false; } + if (dumped) { + // Only an engine failure can select the JavaScript fallback. Once + // pg_dump succeeds, archive verification or persistence failures + // must retain their original error and must not reuse a closed writer. + await writer.abort(); + const sizeBytes = statSync(backupFile).size; + await opts.verifyBeforePrune?.(backupFile); + const prunedCount = pruneOldBackups(opts.backupDir, retention, filenamePrefix); + return { backupFile, sizeBytes, prunedCount }; + } } await sql`SELECT 1`; @@ -1056,6 +1066,7 @@ export async function runDatabaseBackup(opts: RunDatabaseBackupOptions): Promise unlinkSync(sqlFile); const sizeBytes = statSync(backupFile).size; + await opts.verifyBeforePrune?.(backupFile); const prunedCount = pruneOldBackups(opts.backupDir, retention, filenamePrefix); return { diff --git a/server/src/__tests__/idle-database-backup.test.ts b/server/src/__tests__/idle-database-backup.test.ts new file mode 100644 index 00000000000..07e794c9205 --- /dev/null +++ b/server/src/__tests__/idle-database-backup.test.ts @@ -0,0 +1,92 @@ +import { mkdtemp, readFile, rm, symlink, utimes, writeFile } from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; +import { gzipSync } from "node:zlib"; +import { afterEach, beforeEach, describe, expect, it } from "vitest"; +import { sql } from "drizzle-orm"; +import { createDb, ensurePostgresDatabase, getEmbeddedPostgresTestSupport, runDatabaseBackup, runDatabaseRestore, startEmbeddedPostgresTestDatabase } from "@paperclipai/db"; +import { clearIdleBackupWakeRequired, idleBackupWakeMarker, markIdleBackupWakeRequired, verifyIdleDatabaseBackup } from "../services/idle-database-backup.js"; + +let directory: string; +beforeEach(async () => { directory = await mkdtemp(path.join(os.tmpdir(), "idle-backup-")); }); +afterEach(async () => { await rm(directory, { recursive: true, force: true }); }); + +describe("idle database backup persistence", () => { + it("keeps a restart marker until a fresh valid backup is complete", async () => { + const archive = path.join(directory, "backup.sql.gz"); + await markIdleBackupWakeRequired(directory); + await writeFile(archive, gzipSync("CREATE TABLE fixture (id integer);\n")); + await verifyIdleDatabaseBackup(archive); + // Verification on the sleep path must not remove the wake obligation. + expect(await readFile(idleBackupWakeMarker(directory), "utf8")).toContain("restarts"); + await clearIdleBackupWakeRequired(directory); + await expect(readFile(idleBackupWakeMarker(directory))).rejects.toMatchObject({ code: "ENOENT" }); + expect(await readFile(archive)).toEqual(gzipSync("CREATE TABLE fixture (id integer);\n")); + }); + + it.each([ + ["empty file", Buffer.alloc(0)], ["empty SQL", gzipSync("")], + ["truncated gzip", gzipSync("SELECT 1;").subarray(0, -4)], + ["plain text", Buffer.from("SELECT 1;")], + ])("rejects %s and retains the restart obligation", async (_label, contents) => { + await markIdleBackupWakeRequired(directory); + const archive = path.join(directory, "invalid.sql.gz"); + await writeFile(archive, contents); + await expect(verifyIdleDatabaseBackup(archive)).rejects.toThrow(); + expect(await readFile(idleBackupWakeMarker(directory), "utf8")).toContain("restarts"); + }); + + it("rejects a missing archive and symlinked archive or marker", async () => { + const archive = path.join(directory, "backup.sql.gz"); + await expect(verifyIdleDatabaseBackup(archive)).rejects.toThrow(); + const target = path.join(directory, "target"); + await writeFile(target, gzipSync("SELECT 1;")); + await symlink(target, archive); + await expect(verifyIdleDatabaseBackup(archive)).rejects.toThrow(); + await symlink(target, idleBackupWakeMarker(directory)); + await expect(markIdleBackupWakeRequired(directory)).rejects.toThrow(); + expect(await readFile(target)).toEqual(gzipSync("SELECT 1;")); + }); +}); + +const support = await getEmbeddedPostgresTestSupport(); +describe.skipIf(!support.supported)("idle backup restore", () => { + it("restores the last pre-sleep state into a separate database", async () => { + const source = await startEmbeddedPostgresTestDatabase("idle-backup-restore-"); + const sourceDb = createDb(source.connectionString); + const targetUrl = new URL(source.connectionString); + targetUrl.pathname = "/idle_restore"; + try { + await sourceDb.execute(sql`CREATE TABLE backup_fixture (id integer PRIMARY KEY, value text NOT NULL)`); + await sourceDb.execute(sql`INSERT INTO backup_fixture VALUES (1, 'last committed state')`); + await markIdleBackupWakeRequired(directory); + const result = await runDatabaseBackup({ + connectionString: source.connectionString, backupDir: directory, + retention: { dailyDays: 14, weeklyWeeks: 4, monthlyMonths: 6 }, + verifyBeforePrune: verifyIdleDatabaseBackup, + }); + await verifyIdleDatabaseBackup(result.backupFile); + await ensurePostgresDatabase(source.connectionString, "idle_restore"); + await runDatabaseRestore({ connectionString: targetUrl.toString(), backupFile: result.backupFile }); + const target = createDb(targetUrl.toString()); + expect(await target.execute(sql`SELECT id, value FROM backup_fixture`)).toEqual([{ id: 1, value: "last committed state" }]); + const checkpointBytes = await readFile(result.backupFile); + expect(await readFile(idleBackupWakeMarker(directory), "utf8")).toContain("restarts"); + // A failed verification must not delete existing recovery points, even + // when their age would make them eligible for the configured pruning. + const history = path.join(directory, "paperclip-history.sql.gz"); + const historyBytes = gzipSync("SELECT 'historical recovery point';"); + await writeFile(history, historyBytes); + await utimes(history, new Date(0), new Date(0)); + await expect(runDatabaseBackup({ + connectionString: source.connectionString, backupDir: directory, + backupEngine: "javascript", retention: { dailyDays: 7, weeklyWeeks: 4, monthlyMonths: 1 }, + verifyBeforePrune: async () => { throw new Error("verification failed"); }, + })).rejects.toThrow("verification failed"); + expect(await readFile(history)).toEqual(historyBytes); + expect(await readFile(result.backupFile)).toEqual(checkpointBytes); + } finally { + await source.cleanup(); + } + }, 60_000); +}); diff --git a/server/src/__tests__/idle-local-work.test.ts b/server/src/__tests__/idle-local-work.test.ts index 51e9dae011f..666c58d0b20 100644 --- a/server/src/__tests__/idle-local-work.test.ts +++ b/server/src/__tests__/idle-local-work.test.ts @@ -40,10 +40,12 @@ describe("idle cleanup inspection", () => { expect(await readIdleLocalWork()).toBe("none"); markIdleStartupComplete({ scheduledBackups: true }); expect(await readIdleLocalWork()).toBe("present"); + expect(await readIdleLocalWork({ backupCheckpoint: true })).toBe("none"); markIdleStartupComplete({ scheduledBackups: false }); await fs.mkdir(custom); await fs.writeFile(path.join(custom, "receipt.tmp"), "incomplete"); expect(await readIdleLocalWork()).toBe("present"); + expect(await readIdleLocalWork({ backupCheckpoint: true })).toBe("present"); await fs.rm(custom, { recursive: true }); expect(await readIdleLocalWork()).toBe("none"); }); diff --git a/server/src/__tests__/idle-sleep-safety.test.ts b/server/src/__tests__/idle-sleep-safety.test.ts index 832673aca58..66a023dd9b9 100644 --- a/server/src/__tests__/idle-sleep-safety.test.ts +++ b/server/src/__tests__/idle-sleep-safety.test.ts @@ -1,4 +1,4 @@ -import { beginIdleTrackedWork } from "../services/task-admission.js"; +import { beginIdleTrackedWork, idleWorkSnapshot } from "../services/task-admission.js"; import { randomUUID } from "node:crypto"; import { eq, sql } from "drizzle-orm"; import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest"; @@ -22,6 +22,73 @@ const ownerId = "d0b833f4-4098-42de-8420-1907f3aa4895"; const owned = () => ({ ...held(), ownerId }); const emptyLocal = async () => "none" as const; +describe("idle backup checkpoint", () => { + const database = (blocked = false) => { + const transaction = vi.fn(async (run: (tx: unknown) => Promise) => run({ execute: async () => [{ blocked }] })); + return { db: { transaction } as unknown as Db, transaction }; + }; + const backup = (work: () => Promise = async () => true) => vi.fn(async () => { + const finish = beginIdleTrackedWork(); + try { return await work(); } finally { finish(); } + }); + it("backs up after durable inspection and rechecks local work before authorizing sleep", async () => { + const { db, transaction } = database(); + const local = vi.fn(emptyLocal); + const checkpoint = backup(async () => { + expect(transaction).toHaveBeenCalledOnce(); + expect(idleWorkSnapshot().active).toBe(1); + return true; + }); + expect(await readIdleSleepSafety(db, owned, () => now, ownerId, local, undefined, checkpoint)).toEqual(none); + expect(checkpoint).toHaveBeenCalledOnce(); + expect(local).toHaveBeenCalledTimes(2); + }); + it.each(["present", "unknown"] as const)("avoids DB probes and backups for known local %s work", async state => { + const { db, transaction } = database(); + const checkpoint = backup(); + const plugins = vi.fn(); + expect(await readIdleSleepSafety(db, owned, () => now, ownerId, async () => state, plugins, checkpoint)) + .toEqual({ version: 1, backgroundWork: state }); + expect(transaction).not.toHaveBeenCalled(); + expect(plugins).not.toHaveBeenCalled(); + expect(checkpoint).not.toHaveBeenCalled(); + }); + it("does not start a backup for durable work or an unowned hold", async () => { + const checkpoint = backup(); + expect(await readIdleSleepSafety(database(true).db, owned, () => now, ownerId, emptyLocal, undefined, checkpoint)).toEqual(present); + expect(await readIdleSleepSafety(database().db, owned, () => now, "stale", emptyLocal, undefined, checkpoint)).toEqual(unknown); + expect(checkpoint).not.toHaveBeenCalled(); + }); + it.each(["failed", "threw", "untracked", "concurrent", "expired", "replaced"])("keeps the instance awake when the backup is %s", async kind => { + let clock = now; + let status = owned(); + const work = async () => { + if (kind === "threw") throw new Error("private archive path"); + if (kind === "concurrent") { const finish = beginIdleTrackedWork(); finish(); } + if (kind === "expired") clock += 60_000; + if (kind === "replaced") status = { ...status, ownerId: "replacement" }; + return kind !== "failed"; + }; + const checkpoint = kind === "untracked" ? work : backup(work); + expect(await readIdleSleepSafety(database().db, () => status, () => clock, ownerId, emptyLocal, undefined, checkpoint)).toEqual(unknown); + expect(idleWorkSnapshot().active).toBe(0); + }); + it("retains actual backup work after the owning hold expires", async () => { + let finishBackup!: () => void; + const pending = new Promise(resolve => { finishBackup = resolve; }); + const checkpoint = backup(async () => { await pending; return true; }); + let clock = now; + const scan = readIdleSleepSafety(database().db, owned, () => clock, ownerId, emptyLocal, undefined, checkpoint); + await vi.waitFor(() => expect(checkpoint).toHaveBeenCalledOnce()); + clock += 60_000; + expect(idleWorkSnapshot().active).toBe(1); + expect(await readIdleSleepSafety(database().db, owned, () => clock, ownerId, emptyLocal)).toEqual(unknown); + finishBackup(); + expect(await scan).toEqual(unknown); + expect(idleWorkSnapshot().active).toBe(0); + }); +}); + describe("idle sleep safety failure boundaries", () => { it.each([ @@ -36,7 +103,7 @@ describe("idle sleep safety failure boundaries", () => { it("fails closed when the database is unavailable", async () => { const db = { transaction: vi.fn().mockRejectedValue(new Error("private connection detail")) } as unknown as Db; - expect(await readIdleSleepSafety(db, held, () => now)).toEqual(unknown); + expect(await readIdleSleepSafety(db, held, () => now, undefined, emptyLocal)).toEqual(unknown); }); it.each([ @@ -46,14 +113,14 @@ describe("idle sleep safety failure boundaries", () => { const getStatus = vi.fn().mockReturnValueOnce(held()).mockReturnValue({ ...held(), ...change }); const execute = vi.fn().mockResolvedValue([{ blocked: true }]); const transaction = vi.fn(async (fn: (tx: unknown) => Promise) => fn({ execute })); - expect(await readIdleSleepSafety({ transaction } as unknown as Db, getStatus, () => now)).toEqual(unknown); + expect(await readIdleSleepSafety({ transaction } as unknown as Db, getStatus, () => now, undefined, emptyLocal)).toEqual(unknown); expect(transaction).toHaveBeenCalledWith(expect.any(Function), { isolationLevel: "repeatable read", accessMode: "read only" }); }); it("reports persisted work when the admission hold remains unchanged", async () => { const execute = vi.fn().mockResolvedValue([{ blocked: true }]); const transaction = vi.fn(async (fn: (tx: unknown) => Promise) => fn({ execute })); - expect(await readIdleSleepSafety({ transaction } as unknown as Db, held, () => now)).toEqual(present); + expect(await readIdleSleepSafety({ transaction } as unknown as Db, held, () => now, undefined, emptyLocal)).toEqual(present); }); }); diff --git a/server/src/__tests__/instance-settings-routes.test.ts b/server/src/__tests__/instance-settings-routes.test.ts index ffee8171aaf..5d6cf9d0345 100644 --- a/server/src/__tests__/instance-settings-routes.test.ts +++ b/server/src/__tests__/instance-settings-routes.test.ts @@ -77,7 +77,7 @@ describe("instance settings routes", () => { return { errorHandler, instanceSettingsRoutes }; }); - function createApp(actor: any, pluginWorkers?: PluginWorkerManager) { + function createApp(actor: any, pluginWorkers?: PluginWorkerManager, prepareBackup?: () => Promise) { const { errorHandler, instanceSettingsRoutes } = routeModules.value; const app = express(); app.use(express.json()); @@ -85,7 +85,7 @@ describe("instance settings routes", () => { req.actor = actor; next(); }); - app.use("/api", instanceSettingsRoutes(mockDb as any, pluginWorkers)); + app.use("/api", instanceSettingsRoutes(mockDb as any, pluginWorkers, prepareBackup)); app.use(errorHandler); return app; } @@ -973,7 +973,7 @@ describe("instance settings routes", () => { await request(createApp(adminActor)).post("/api/instance/task-drain").send({ purpose: "idle", ttlMs: 60_000 }).expect(200); expect(mockHeartbeatService.computeTaskDrain).toHaveBeenCalledWith({ purpose: "idle", ttlMs: 60_000 }); await request(createApp(adminActor)).get(`/api/instance/task-drain?idleSleepSafety=1&ownerId=${ownerId}`).expect(200); - expect(mockReadIdleSleepSafety).toHaveBeenLastCalledWith(mockDb, expect.any(Function), Date.now, ownerId, undefined, undefined); + expect(mockReadIdleSleepSafety).toHaveBeenLastCalledWith(mockDb, expect.any(Function), Date.now, ownerId, expect.any(Function), undefined, undefined); }); it.each([{}, { ttlMs: null }, { ttlMs: 1 }, { ttlMs: 300_001 }])("rejects an unbounded or invalid idle TTL: %j", async body => { @@ -1000,7 +1000,7 @@ describe("instance settings routes", () => { const res = await request(createApp(adminActor)).get("/api/instance/task-drain?idleSleepSafety=1"); expect(res.status).toBe(200); expect(res.body).toEqual({ ...idleStatus, idleSleepSafety: report }); - expect(mockReadIdleSleepSafety).toHaveBeenCalledWith(mockDb, expect.any(Function), Date.now, undefined, undefined, undefined); + expect(mockReadIdleSleepSafety).toHaveBeenCalledWith(mockDb, expect.any(Function), Date.now, undefined, expect.any(Function), undefined, undefined); expect(mockReadIdleSleepSafety.mock.calls[0][1]()).toEqual(idleStatus); }); @@ -1028,6 +1028,42 @@ describe("instance settings routes", () => { expect(release).toHaveBeenCalledOnce(); }); + it("passes the operator backup hook only through the protected safety report", async () => { + const backup = vi.fn(async () => true); + mockHeartbeatService.getTaskDrainStatus.mockReturnValue(idleStatus); + const app = createApp(adminActor, undefined, backup); + await request(app).get("/api/instance/task-drain").expect(200); + expect(mockReadIdleSleepSafety).not.toHaveBeenCalled(); + await request(app).get("/api/instance/task-drain?idleSleepSafety=1&ownerId=fixture-owner").expect(200); + expect(mockReadIdleSleepSafety).toHaveBeenCalledWith(mockDb, expect.any(Function), Date.now, + "fixture-owner", expect.any(Function), undefined, backup); + mockReadIdleSleepSafety.mockClear(); + await request(createApp(nonAdminActor, undefined, backup)).get("/api/instance/task-drain?idleSleepSafety=1").expect(403); + expect(mockReadIdleSleepSafety).not.toHaveBeenCalled(); + expect(backup).not.toHaveBeenCalled(); + }); + + it("reserves managed checkpoint creation for verified Cloud control actors", async () => { + const backup = vi.fn(async () => true); + process.env.PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN = "fixture-managed-signal"; + mockHeartbeatService.getTaskDrainStatus.mockReturnValue(idleStatus); + try { + for (const source of ["session", "cloud_tenant", "local_implicit"]) { + const tenantAdmin = createApp({ ...adminActor, source }, undefined, backup); + await request(tenantAdmin).get("/api/instance/task-drain?idleSleepSafety=1&ownerId=fixture-owner") + .set("x-paperclip-cloud-control", "unverified-header").expect(403); + } + expect(mockReadIdleSleepSafety).not.toHaveBeenCalled(); + expect(backup).not.toHaveBeenCalled(); + const control = createApp({ ...adminActor, source: "cloud_control" }, undefined, backup); + await request(control).get("/api/instance/task-drain?idleSleepSafety=1&ownerId=fixture-owner").expect(200); + expect(mockReadIdleSleepSafety).toHaveBeenCalledWith(mockDb, expect.any(Function), Date.now, + "fixture-owner", expect.any(Function), undefined, backup); + } finally { + delete process.env.PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN; + } + }); + it.each([ ["company member", nonAdminActor], ["agent", { type: "agent", agentId: "agent-1", companyId: "company-1" }], diff --git a/server/src/__tests__/server-startup-feedback-export.test.ts b/server/src/__tests__/server-startup-feedback-export.test.ts index 7b89d44dd9a..449d8d5b5dd 100644 --- a/server/src/__tests__/server-startup-feedback-export.test.ts +++ b/server/src/__tests__/server-startup-feedback-export.test.ts @@ -1,9 +1,12 @@ import { idleWorkSnapshot } from "../services/task-admission.js"; import { readIdleSleepSafety } from "../services/idle-sleep-safety.js"; +import * as idleLocalWork from "../services/idle-local-work.js"; +import { idleBackupWakeMarker, markIdleBackupWakeRequired } from "../services/idle-database-backup.js"; import { reconcileBuiltInAgentsOnStartup, reconcileCodexLocalManagedHomesOnStartup, reconcilePersistedRuntimeServicesOnStartup } from "../services/index.js"; import { runDatabaseBackup } from "@paperclipai/db"; import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; -import { chmodSync, mkdirSync, mkdtempSync, readFileSync, rmSync, statSync, symlinkSync, writeFileSync } from "node:fs"; +import { chmodSync, existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, statSync, symlinkSync, writeFileSync } from "node:fs"; +import { gzipSync } from "node:zlib"; import { tmpdir } from "node:os"; import path from "node:path"; @@ -441,6 +444,69 @@ describe("startServer feedback export wiring", () => { process.env.BETTER_AUTH_SECRET = "test-secret"; }); + it.each([false, true].flatMap(checkpointMode => [false, true].map(fails => ({ checkpointMode, fails })))) + ("finishes restart backup before idle eligibility (checkpoint=$checkpointMode, failure=$fails)", async ({ checkpointMode, fails }) => { + const directory = mkdtempSync(path.join(tmpdir(), "startup-idle-backup-")); + const previousFlag = process.env.PAPERCLIP_DB_BACKUP_IDLE_CHECKPOINT_ENABLED; + process.env.PAPERCLIP_DB_BACKUP_IDLE_CHECKPOINT_ENABLED = checkpointMode ? "1" : "0"; + loadConfigMock.mockReturnValue(buildTestConfig({ databaseBackupEnabled: true, databaseBackupDir: directory })); + const eligible = vi.spyOn(idleLocalWork, "markIdleStartupComplete"); + let retry: (() => void) | undefined; + const timers = vi.spyOn(globalThis, "setInterval").mockImplementation(((callback: () => void, ms: number) => { + if (ms === 60 * 60 * 1000) retry = callback; + return 1 as unknown as ReturnType; + }) as typeof setInterval); + let finish!: () => void; + const pending = new Promise(resolve => { finish = resolve; }); + const writeBackup = async (options: Parameters[0]) => { + const backupFile = path.join(directory, "paperclip-restart.sql.gz"); + const bytes = gzipSync("SELECT 'restart recovery';\n"); + writeFileSync(backupFile, bytes); + expect(options.retention).toEqual({ dailyDays: 7, weeklyWeeks: 4, monthlyMonths: 1 }); + expect(options.verifyBeforePrune).toEqual(expect.any(Function)); + await options.verifyBeforePrune!(backupFile); + return { backupFile, sizeBytes: bytes.length, prunedCount: 0 }; + }; + vi.mocked(runDatabaseBackup).mockImplementationOnce(async options => { + await pending; + if (fails) throw new Error("fixture restart backup failed"); + return writeBackup(options); + }); + let starting: ReturnType | undefined; + try { + await markIdleBackupWakeRequired(directory); + starting = startServer(); + await vi.waitFor(() => expect(runDatabaseBackup).toHaveBeenCalledOnce()); + expect(eligible).not.toHaveBeenCalled(); + expect(idleWorkSnapshot().active).toBeGreaterThan(0); + finish(); + await starting; + expect(eligible).toHaveBeenCalledWith({ scheduledBackups: true }); + const options = createAppMock.mock.calls[0]?.[1] as unknown as { prepareIdleDatabaseBackup?: unknown }; + expect(Boolean(options.prepareIdleDatabaseBackup)).toBe(checkpointMode); + expect(existsSync(idleBackupWakeMarker(directory))).toBe(fails); + if (fails) { + vi.mocked(runDatabaseBackup).mockImplementationOnce(writeBackup); + expect(retry).toEqual(expect.any(Function)); + const activeBeforeRetry = idleWorkSnapshot().active; + retry!(); + await vi.waitFor(() => { + expect(existsSync(idleBackupWakeMarker(directory))).toBe(false); + // Marker removal precedes directory fsync and release of the work receipt. + expect(idleWorkSnapshot().active).toBe(activeBeforeRetry); + }); + expect(runDatabaseBackup).toHaveBeenCalledTimes(2); + } + } finally { + finish(); + await starting; + timers.mockRestore(); eligible.mockRestore(); + if (previousFlag === undefined) delete process.env.PAPERCLIP_DB_BACKUP_IDLE_CHECKPOINT_ENABLED; + else process.env.PAPERCLIP_DB_BACKUP_IDLE_CHECKPOINT_ENABLED = previousFlag; + rmSync(directory, { recursive: true, force: true }); + } + }); + it.each(([ ["built-in agents", reconcileBuiltInAgentsOnStartup], ["managed homes", reconcileCodexLocalManagedHomesOnStartup], diff --git a/server/src/app.ts b/server/src/app.ts index db931861787..bda8217716a 100644 --- a/server/src/app.ts +++ b/server/src/app.ts @@ -491,6 +491,7 @@ export async function createApp( }): Promise; }; databaseBackupService?: InstanceDatabaseBackupService; + prepareIdleDatabaseBackup?: () => Promise; databaseBackupHealth?: InspectDatabaseBackupHealthOptions; deploymentMode: DeploymentMode; deploymentExposure: DeploymentExposure; @@ -836,7 +837,7 @@ export async function createApp( api.use(resourceMembershipRoutes(db)); api.use(primaryAgentRoutes(db)); api.use(inboxDismissalRoutes(db)); - api.use(instanceSettingsRoutes(db, workerManager)); + api.use(instanceSettingsRoutes(db, workerManager, opts.prepareIdleDatabaseBackup)); if (opts.databaseBackupService) { api.use(instanceDatabaseBackupRoutes(opts.databaseBackupService)); } diff --git a/server/src/index.ts b/server/src/index.ts index 0c79e723e62..ce72014df01 100644 --- a/server/src/index.ts +++ b/server/src/index.ts @@ -1,5 +1,6 @@ import { isIdleTaskDrainActive, beginIdleTrackedWork, trackIdleWork } from "./services/task-admission.js"; import { markIdleStartupComplete } from "./services/idle-local-work.js"; +import { clearIdleBackupWakeRequired, idleBackupWakeMarker, markIdleBackupWakeRequired, verifyIdleDatabaseBackup } from "./services/idle-database-backup.js"; import { cloudWarmStandbyServerOptions } from "./middleware/cloud-warm-standby.js"; import { createCloudWarmStandby } from "./services/cloud-warm-standby.js"; import { subscribeAllCompanyLiveEvents } from "./services/live-events.js"; @@ -819,6 +820,8 @@ async function startServerWithDatabaseTeardown( resolve(config.databaseBackupDir, "..", "db-backup-to-s3.failure"), ]; let databaseBackupInFlight = false; + const idleDatabaseBackupEnabled = config.databaseBackupEnabled + && process.env.PAPERCLIP_DB_BACKUP_IDLE_CHECKPOINT_ENABLED === "1"; const runServerDatabaseBackup = async ( trigger: InstanceDatabaseBackupTrigger, ): Promise => { @@ -835,19 +838,25 @@ async function startServerWithDatabaseTeardown( const finishIdleBackup = beginIdleTrackedWork(); const startedAt = new Date(); const startedAtMs = Date.now(); - const label = trigger === "scheduled" ? "Automatic" : "Manual"; + const label = trigger === "manual" ? "Manual" : trigger === "idle" ? "Pre-sleep" : "Automatic"; try { + if (trigger === "idle") await markIdleBackupWakeRequired(config.databaseBackupDir); logger.info({ backupDir: config.databaseBackupDir, trigger }, `${label} database backup starting`); // Read retention from Instance Settings (DB) so changes take effect without restart. const generalSettings = await backupSettingsSvc.getGeneral(); const retention = generalSettings.backupRetention; + const verifyCheckpoint = trigger === "idle" || existsSync(idleBackupWakeMarker(config.databaseBackupDir)); const result = await runDatabaseBackup({ connectionString: activeDatabaseConnectionString, backupDir: config.databaseBackupDir, retention, filenamePrefix: "paperclip", + verifyBeforePrune: verifyCheckpoint ? verifyIdleDatabaseBackup : undefined, }); + if (verifyCheckpoint) { + if (trigger !== "idle") await clearIdleBackupWakeRequired(config.databaseBackupDir); + } const finishedAt = new Date(); const response: InstanceDatabaseBackupRunResult = { ...result, @@ -892,6 +901,9 @@ async function startServerWithDatabaseTeardown( // self-hosted: createApp falls back to its built-in kubernetes-only default. const managedPluginAutoInstall = managedConfig?.plugins.autoInstall ?? null; const app = await createApp(db as any, { + prepareIdleDatabaseBackup: idleDatabaseBackupEnabled + ? async () => (await runServerDatabaseBackup("idle")) !== null + : undefined, cloudWarmStandby: isWarmStandby, uiMode, serverPort: listenPort, @@ -1872,6 +1884,12 @@ async function startServerWithDatabaseTeardown( }, "Automatic database backups enabled", ); + // The final pre-sleep dump remains on the volume throughout sleep. Take + // a fresh backup on restart before idle eligibility is enabled, including + // after rollback of the opt-in flag. Failure retains the marker for retry. + if (!isWarmStandby() && existsSync(idleBackupWakeMarker(config.databaseBackupDir))) { + await runServerDatabaseBackup("scheduled").catch(() => {}); + } setInterval(() => { if (isWarmStandby() || isIdleTaskDrainActive()) return; void runServerDatabaseBackup("scheduled").catch(() => { diff --git a/server/src/routes/instance-database-backups.ts b/server/src/routes/instance-database-backups.ts index f05e3361c2c..9635a1cb31b 100644 --- a/server/src/routes/instance-database-backups.ts +++ b/server/src/routes/instance-database-backups.ts @@ -4,7 +4,7 @@ import { forbidden } from "../errors.js"; import { isCloudManagedInstance } from "../services/cloud-instance.js"; import { assertInstanceAdmin } from "./authz.js"; -export type InstanceDatabaseBackupTrigger = "manual" | "scheduled"; +export type InstanceDatabaseBackupTrigger = "manual" | "scheduled" | "idle"; export type InstanceDatabaseBackupRunResult = RunDatabaseBackupResult & { trigger: InstanceDatabaseBackupTrigger; diff --git a/server/src/routes/instance-settings.ts b/server/src/routes/instance-settings.ts index ef578bb7fda..0c126b729e1 100644 --- a/server/src/routes/instance-settings.ts +++ b/server/src/routes/instance-settings.ts @@ -15,6 +15,7 @@ import { } from "../services/cloud-instance.js"; import { getHiddenSettings } from "../services/settings-visibility.js"; import { readIdleSleepSafety } from "../services/idle-sleep-safety.js"; +import { readIdleLocalWork } from "../services/idle-local-work.js"; import type { PluginWorkerManager } from "../services/plugin-worker-manager.js"; import { validate } from "../middleware/validate.js"; import { logger } from "../middleware/logger.js"; @@ -120,7 +121,7 @@ function withTaskDrainTransition(run: () => Promise): Promise { return turn; } -export function instanceSettingsRoutes(db: Db, pluginWorkers?: PluginWorkerManager) { +export function instanceSettingsRoutes(db: Db, pluginWorkers?: PluginWorkerManager, prepareIdleDatabaseBackup?: () => Promise) { const router = Router(); const svc = instanceSettingsService(db); const environments = environmentService(db); @@ -304,9 +305,18 @@ export function instanceSettingsRoutes(db: Db, pluginWorkers?: PluginWorkerManag // The report covers every company in this process. Ordinary company // members may read process counters, but not instance-wide work state. assertCanManageInstanceSettings(req); + // A checkpoint performs platform-owned backup work. Tenant owners can + // have instance-admin elevation, but that must not grant the manual + // backup authority which managed instances deliberately withhold. + if (prepareIdleDatabaseBackup && isCloudManagedInstance() && req.actor.source !== "cloud_control") { + throw forbidden("Backup checkpoints are platform-managed on cloud-managed instances", { + code: "database_backups_platform_managed", + }); + } const idleSleepSafety = await readIdleSleepSafety(db, () => heartbeat.getTaskDrainStatus(), Date.now, - typeof req.query.ownerId === "string" ? req.query.ownerId : undefined, undefined, - pluginWorkers?.inspectIdleSleep?.bind(pluginWorkers)); + typeof req.query.ownerId === "string" ? req.query.ownerId : undefined, + () => readIdleLocalWork({ backupCheckpoint: Boolean(prepareIdleDatabaseBackup) }), + pluginWorkers?.inspectIdleSleep?.bind(pluginWorkers), prepareIdleDatabaseBackup); res.json({ ...heartbeat.getTaskDrainStatus(), idleSleepSafety }); return; } diff --git a/server/src/services/idle-database-backup.ts b/server/src/services/idle-database-backup.ts new file mode 100644 index 00000000000..271750d118c --- /dev/null +++ b/server/src/services/idle-database-backup.ts @@ -0,0 +1,52 @@ +import { constants, createReadStream } from "node:fs"; +import { mkdir, open, unlink } from "node:fs/promises"; +import path from "node:path"; +import { Writable } from "node:stream"; +import { pipeline } from "node:stream/promises"; +import { createGunzip } from "node:zlib"; + +// The marker survives process replacement on persistent instance storage. +// It contains no credentials or tenant data, and never substitutes for a dump. +export const idleBackupWakeMarker = (directory: string) => path.join(directory, ".idle-backup-wake-required"); + +async function syncDirectory(directory: string): Promise { + const handle = await open(directory, "r"); + try { await handle.sync(); } finally { await handle.close(); } +} + +export async function markIdleBackupWakeRequired(directory: string): Promise { + await mkdir(directory, { recursive: true }); + const marker = await open(idleBackupWakeMarker(directory), constants.O_WRONLY | constants.O_CREAT | constants.O_NOFOLLOW, 0o600); + try { + await marker.writeFile("Take a fresh database backup after this process restarts.\n"); + await marker.sync(); + } finally { await marker.close(); } + await syncDirectory(directory); +} + +/** Read the entire archive to verify the gzip trailer, then persist its bytes + * and directory entry before allowing compute to stop. The backup runner + * retains its in-flight receipt until this function settles. */ +export async function verifyIdleDatabaseBackup(backupFile: string): Promise { + const handle = await open(backupFile, constants.O_RDONLY | constants.O_NOFOLLOW); + try { + const stat = await handle.stat(); + if (!stat.isFile() || stat.size === 0) throw new Error("Idle backup archive is empty or invalid"); + let bytes = 0; + await pipeline( + createReadStream(backupFile, { fd: handle.fd, autoClose: false }), + createGunzip(), + new Writable({ write(chunk, _encoding, done) { bytes += chunk.length; done(); } }), + ); + if (bytes === 0) throw new Error("Idle backup archive contains no SQL"); + await handle.sync(); + } finally { await handle.close(); } + await syncDirectory(path.dirname(backupFile)); +} + +export async function clearIdleBackupWakeRequired(directory: string): Promise { + await unlink(idleBackupWakeMarker(directory)).catch((error: NodeJS.ErrnoException) => { + if (error.code !== "ENOENT") throw error; + }); + await syncDirectory(directory); +} diff --git a/server/src/services/idle-local-work.ts b/server/src/services/idle-local-work.ts index 1095c298f80..2d54ad90309 100644 --- a/server/src/services/idle-local-work.ts +++ b/server/src/services/idle-local-work.ts @@ -33,9 +33,11 @@ export async function inspectIdleSpool(directory: string): Promise { +export async function readIdleLocalWork(options: { backupCheckpoint?: boolean } = {}): Promise { if (!startupComplete || !ingressTracked) return "unknown"; - if (scheduledBackups || idleWorkSnapshot().active !== 0) return "present"; + // This exemption belongs only to a scan which will finish a new backup + // under the same admission hold before it can return an empty report. + if ((scheduledBackups && !options.backupCheckpoint) || idleWorkSnapshot().active !== 0) return "present"; const directories = new Set([...spoolDirectories, idleAccountingSpoolPath(), idleOrphanSpoolPath()]); const results = await Promise.all([...directories].map(inspectIdleSpool)); if (results.includes("unknown")) return "unknown"; diff --git a/server/src/services/idle-sleep-safety.ts b/server/src/services/idle-sleep-safety.ts index ad24e3037d1..a8c7a80e13b 100644 --- a/server/src/services/idle-sleep-safety.ts +++ b/server/src/services/idle-sleep-safety.ts @@ -97,12 +97,17 @@ export async function readIdleSleepSafety( ownerId?: string, inspectLocalWork: () => Promise = readIdleLocalWork, inspectPlugins?: InspectIdlePlugins, + prepareBackup?: () => Promise, ): Promise { const unknown: IdleSleepSafety = { version: 1, backgroundWork: "unknown" }; const before = getDrainStatus(); const localBefore = idleWorkSnapshot(); if (!sameQuietHold(before, before, now())) return unknown; try { + // Reject known local blockers before querying (and potentially waking) + // the tenant database. Keep the final inspection below as a second fence. + const localFirst = await inspectLocalWork(); + if (localFirst !== "none") return { version: 1, backgroundWork: localFirst }; let quietPlugins: string[] = []; if (inspectPlugins) { if (!ownerId || ownerId !== before.ownerId || before.expiresAt === null || localBefore.active !== 0) return unknown; @@ -124,11 +129,21 @@ export async function readIdleSleepSafety( if (blocked === undefined || !sameQuietHold(before, getDrainStatus(), now())) return unknown; if (blocked) return { version: 1, backgroundWork: "present" }; if (!ownerId || ownerId !== before.ownerId || before.expiresAt === null || localBefore.active !== 0) return unknown; + let expectedGeneration = localBefore.generation; + if (prepareBackup) { + if (!sameQuietHold(before, getDrainStatus(), now()) || + idleWorkSnapshot().generation !== expectedGeneration || idleWorkSnapshot().active !== 0) return unknown; + // The runner owns exactly one tracked-work receipt through dump, + // archive verification and fsync. Its start/finish are the only work + // allowed during this scan. Any concurrent work invalidates the dump. + if (!(await prepareBackup())) return unknown; + expectedGeneration += 2; + } const local = await inspectLocalWork(); const after = getDrainStatus(); const localAfter = idleWorkSnapshot(); if (!sameQuietHold(before, after, now()) || localAfter.active !== 0 || - localBefore.generation !== localAfter.generation) return unknown; + expectedGeneration !== localAfter.generation) return unknown; return { version: 1, backgroundWork: local }; } catch { From 1c4ce44e137b22a3f27a7383caf2fc65eda22fac Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Thu, 8 Oct 2026 19:27:39 -0700 Subject: [PATCH 3/5] fix: retain bounded workspace transfer failure evidence (#15637) ## Thinking Path > - Paperclip manages AI agents and the work they produce. > - Sandbox work must return to the host when an agent run ends. > - A failed transfer must keep its source available for recovery. > - Some transfer errors lose their useful fields before restore diagnostics reach Sentry. > - This change records fixed transfer stages, failure kinds, and numeric RPC codes. > - The next failure can identify the operation that failed without exposing workspace contents. ## Linked Issues or Issue Description **What happened?** A workspace transfer can fail after model execution succeeds. Several command, archive validation, and download errors become plain errors. The existing RPC envelope then reports `unknown`, with no transfer stage or command exit code. A host RPC timeout also loses its code from restore diagnostics. **Expected behavior** Keep bounded evidence through the provider, worker RPC, restore result, and Sentry context. Keep the same exception, error message, retry rules, and source retention policy. A transfer timeout must remain separate from the model execution timeout flag. **Steps to reproduce** Return a nonzero exit code from the sandbox archive command, return a per-file download error, exceed a tar listing limit, or time out the sync-out RPC. Observe the missing fields in the saved restore diagnostic. The added tests use local fixtures for these cases. Related public work: #15479 preserves the source after restore failure. #15481 carries bounded sync-out diagnostics through RPC. This change supplies missing producer evidence and extends that same envelope. I checked the roadmap and searched open PRs for duplicate transfer diagnostic work. ## What Changed - Add optional transfer stage and failure kind fields to the existing diagnostic envelope. Capture command exits and listing deadlines at their producers. - Record only known codes from typed RPC errors at the sync-out boundary. Revalidate every field before persistence and Sentry projection. - Keep annotations private to each outbound request and restore settlement. Preserve frozen error identity and prevent evidence from leaking across concurrent or later calls. - Document the fields. Cover producer failures, quota limits, old workers, concurrent error reuse, and redaction through the real Sentry SDK. ## Verification - Focused SDK, provider, host, persistence, and real Sentry tests: 449 passed, no skips. Independent review also ran the focused contracts and all provider tests. The optional Sentry SDK is required for this check, so the contract tests cannot skip. - A compiled Daytona transfer and compiled SDK worker pass a local RPC round trip. This uses the development TypeScript loader for workspace dependency exports. The transfer and capture modules are compiled JavaScript. - `pnpm -r typecheck` and `pnpm build` passed. The excluded Daytona package also passed `tsc --noEmit`. - The initial local `pnpm test:run` stopped in the general-server group: 60 suites could not initialize embedded PostgreSQL because the isolated install skipped its Darwin library-link setup. The existing package postinstall repairs this; `initdb --version` now passes. All 60 affected suites then passed: 1,342 tests, no skips. No local full aggregate pass is claimed. - Independent source review covers privacy, concurrency, packaging, and recovery behavior. ## Risks - These diagnostics help identify future failures. They do not establish or repair the cause of a past transfer failure. - New fields are optional. Older workers remain compatible. Unknown values are omitted. - The change adds a small request-local diagnostic scope. It keeps the original thrown errors, archive confinement, cleanup order, timeout values, retry count, and source retention policy. - No schema or deployment changes are required. No live provider actions are part of validation. ## Model Used OpenAI Codex, based on GPT-6, with repository inspection, code execution, and independent agent review. The exact deployed model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/observability.md | 18 +++ .../src/workspace-restore-diagnostics.test.ts | 32 ++++- .../src/workspace-restore-diagnostics.ts | 55 +++++++-- .../daytona/src/file-sync-diagnostics.test.ts | 113 ++++++++++++++++++ .../daytona/src/file-sync.ts | 86 +++++++------ .../daytona/src/plugin.test.ts | 53 ++++---- .../sandbox-providers/daytona/src/plugin.ts | 11 +- .../daytona/src/tar-listing.test.ts | 46 ++++++- .../plugins/sdk/src/environment-sync-error.ts | 89 +++++++++++++- packages/plugins/sdk/src/index.ts | 2 +- packages/plugins/sdk/src/worker-rpc-host.ts | 9 +- .../sdk/tests/environment-sync-error.test.ts | 69 ++++++++++- .../environment-sync-out-error.test.ts | 72 ++++++++++- .../run-failure-sentry-real-sdk.test.ts | 60 ++++++++++ .../__tests__/run-failure-diagnostics.test.ts | 30 +++++ .../services/environment-sync-out-error.ts | 14 ++- .../src/services/run-failure-diagnostics.ts | 3 + 17 files changed, 660 insertions(+), 102 deletions(-) create mode 100644 packages/plugins/sandbox-providers/daytona/src/file-sync-diagnostics.test.ts diff --git a/doc/observability.md b/doc/observability.md index f71a6a2b304..e8d6cbb2151 100644 --- a/doc/observability.md +++ b/doc/observability.md @@ -661,6 +661,24 @@ The envelope excludes provider messages, response bodies, names, paths, and credentials. The existing RPC error message and code remain unchanged, as do restore classification, retries, and source retention. Older workers without the envelope still report `unknown` when no structured cause is available. +For native transfers, optional `workspaceRestoreTransferStep` identifies +`sandbox_access`, `sandbox_guard`, `file_download`, `file_finalize`, +`archive_create`, `archive_download`, `archive_validate`, or `archive_extract`. +`workspaceRestoreTransferFailureKind` distinguishes a producer-confirmed +`command_failed`, `download_failed`, `download_missing`, `unsafe_archive`, +`listing_timeout`, `listing_entry_limit`, `listing_byte_limit`, +`listing_line_limit`, or `listing_stderr_limit`. These labels are fixed values, +not classifications inferred from provider message text. Missing labels remain +absent. A listing deadline records `ETIMEDOUT`; sandbox and listing commands +retain bounded nonzero exit codes even when their error text is wrapped. +`workspaceRestoreRpcCode` retains only known numeric plugin RPC error codes from +a typed error at the host's sync-out boundary. In particular, RPC `TIMEOUT` +(`-32003`) is separate from the model execution's `timedOut` value; successful +model execution does not exclude a later transfer timeout. Transfer evidence is +isolated per outbound request and per restore settlement, then independently +revalidated before persistence and Sentry projection. These fields only accompany +`workspace_transfer` or `asset_restore` diagnostics and do not authorize cleanup +or retries. Older workers remain compatible. For `git_integration`, optional `workspaceRestoreGitCommand` identifies the fixed command family: `rev_parse`, `symbolic_ref`, `merge_base`, `merge_tree`, `commit_tree`, `update_ref`, or `log`. `workspaceRestoreGitFailureKind` is diff --git a/packages/adapter-utils/src/workspace-restore-diagnostics.test.ts b/packages/adapter-utils/src/workspace-restore-diagnostics.test.ts index 39e82e19297..936b68e302c 100644 --- a/packages/adapter-utils/src/workspace-restore-diagnostics.test.ts +++ b/packages/adapter-utils/src/workspace-restore-diagnostics.test.ts @@ -1,7 +1,7 @@ import { describe, expect, it, vi } from "vitest"; import { classifyWorkspaceRestoreFailure } from "./workspace-restore-merge.js"; import { - getWorkspaceRestoreDiagnostic, preserveWorkspaceRestoreErrorDiagnostic, recordWorkspaceRestoreDiagnostic, + withWorkspaceRestoreDiagnosticCapture, getWorkspaceRestoreDiagnostic, preserveWorkspaceRestoreErrorDiagnostic, recordWorkspaceRestoreDiagnostic, sanitizeWorkspaceRestoreDiagnostic, withWorkspaceRestoreDiagnostics, withWorkspaceRestoreStep, withWorkspaceRestoreGitCommand, type WorkspaceRestoreDiagnostic, } from "./workspace-restore-diagnostics.js"; @@ -297,3 +297,33 @@ describe("Git integration diagnostic privacy and attribution", () => { }); }); }); + + +it("isolates wrapped transfer evidence across settlements reusing a frozen error", async () => { + const shared = Object.freeze(new Error("private-message")); + let resume!: () => void; + const gate = new Promise(resolve => { resume = resolve; }); + let started!: () => void; + const ready = new Promise(resolve => { started = resolve; }); + const first = withWorkspaceRestoreDiagnosticCapture(async () => { + await expect(withWorkspaceRestoreDiagnostics("workspace", () => withWorkspaceRestoreStep("workspace_transfer", async () => { + preserveWorkspaceRestoreErrorDiagnostic(shared, {}, { transferStep: "archive_create" }); + started(); await gate; throw shared; + }))).rejects.toBe(shared); + return getWorkspaceRestoreDiagnostic(shared); + }); + await ready; + const second = await withWorkspaceRestoreDiagnosticCapture(async () => { + await expect(withWorkspaceRestoreDiagnostics("workspace", () => withWorkspaceRestoreStep("workspace_transfer", async () => { + throw preserveWorkspaceRestoreErrorDiagnostic(shared, {}, { transferStep: "archive_download" }); + }))).rejects.toBe(shared); + return getWorkspaceRestoreDiagnostic(shared); + }); + resume(); + expect((await first)?.transferStep).toBe("archive_create"); + expect(second?.transferStep).toBe("archive_download"); + await withWorkspaceRestoreDiagnosticCapture(async () => { + await expect(withWorkspaceRestoreDiagnostics("workspace", () => withWorkspaceRestoreStep("workspace_transfer", async () => { throw shared; }))).rejects.toBe(shared); + expect(getWorkspaceRestoreDiagnostic(shared)?.transferStep).toBeUndefined(); + }); +}); diff --git a/packages/adapter-utils/src/workspace-restore-diagnostics.ts b/packages/adapter-utils/src/workspace-restore-diagnostics.ts index afb8d659c96..8ac378374f8 100644 --- a/packages/adapter-utils/src/workspace-restore-diagnostics.ts +++ b/packages/adapter-utils/src/workspace-restore-diagnostics.ts @@ -15,6 +15,18 @@ const GIT_FAILURE_KINDS = new Set([ "merge_conflict", "invalid_object", "ref_conflict", "permission_denied", "unknown", ] as const); type GitFailureKind = typeof GIT_FAILURE_KINDS extends Set ? T : never; +const TRANSFER_STEPS = new Set([ + "sandbox_access", "sandbox_guard", "file_download", "file_finalize", + "archive_create", "archive_download", "archive_validate", "archive_extract", +] as const); +type TransferStep = typeof TRANSFER_STEPS extends Set ? T : never; +const TRANSFER_FAILURE_KINDS = new Set([ + "command_failed", "download_failed", "download_missing", "unsafe_archive", + "listing_timeout", "listing_entry_limit", "listing_byte_limit", "listing_line_limit", "listing_stderr_limit", +] as const); +type TransferFailureKind = typeof TRANSFER_FAILURE_KINDS extends Set ? T : never; +// Deliberately revalidate the known plugin RPC constants at the persistence boundary. +const RPC_CODES = new Set([-32000, -32001, -32002, -32003, -32004, -32005, -32006, -32099]); export interface WorkspaceRestoreDiagnostic { phase: RestorePhase; step?: WorkspaceRestoreStep; @@ -23,13 +35,16 @@ export interface WorkspaceRestoreDiagnostic { exitCode?: number; gitCommand?: WorkspaceRestoreGitCommand; gitFailureKind?: GitFailureKind; + transferStep?: TransferStep; + transferFailureKind?: TransferFailureKind; + rpcCode?: number; } const ERROR_CODES = new Set([ "ENOENT", "EACCES", "EPERM", "ENOSPC", "EIO", "EXDEV", "ENOTDIR", "EISDIR", "ECONNRESET", "ECONNREFUSED", "ETIMEDOUT", "EPIPE", "ENOTFOUND", "EAI_AGAIN", "ABORT_ERR", "UND_ERR_CONNECT_TIMEOUT", "UND_ERR_SOCKET", ]); -type ErrorDiagnostic = Pick; +type ErrorDiagnostic = Pick; interface DiagnosticScope { active: boolean; sequence: number; @@ -44,11 +59,14 @@ const activeDiagnostic = new AsyncLocalStorage(); // Never attach a raw cause to an error just to retain a numeric Git exit code. const wrappedDiagnostics = new WeakMap(); const restoreDiagnostics = new WeakMap(); -const diagnosticCapture = new AsyncLocalStorage>(); +const diagnosticCapture = new AsyncLocalStorage<{ + restores: WeakMap; + wrapped: WeakMap; +}>(); /** Keep each settlement's diagnostic receipt isolated across asynchronous work. */ export function withWorkspaceRestoreDiagnosticCapture(operation: () => Promise): Promise { - return diagnosticCapture.run(new WeakMap(), operation); + return diagnosticCapture.run({ restores: new WeakMap(), wrapped: new WeakMap() }, operation); } function readField(value: Record, key: string): unknown { @@ -108,7 +126,10 @@ function diagnostic(error: unknown): ErrorDiagnostic { // SDKs wrap transport errors in a cause. Bound traversal, including cycles. for (let depth = 0; depth < 4 && current && typeof current === "object"; depth++) { const value = current as Record; - const saved = wrappedDiagnostics.get(value); + const saved = (diagnosticCapture.getStore()?.wrapped ?? wrappedDiagnostics).get(value); + result.transferStep ??= saved?.transferStep; + result.transferFailureKind ??= saved?.transferFailureKind; + result.rpcCode ??= saved?.rpcCode; const code = saved?.errorCode !== "unknown" && saved?.errorCode !== undefined ? saved.errorCode : readField(value, "code"); if (result.errorCode === "unknown" && typeof code === "string" && ERROR_CODES.has(code)) { @@ -129,14 +150,27 @@ function diagnostic(error: unknown): ErrorDiagnostic { } /** Retain bounded fields across an existing error wrapper, without a raw cause. */ -export function preserveWorkspaceRestoreErrorDiagnostic(wrapper: T, source: unknown): T { - wrappedDiagnostics.set(wrapper, diagnostic(source)); +export function preserveWorkspaceRestoreErrorDiagnostic(wrapper: T, source: unknown, transferEvidence?: unknown): T { + (diagnosticCapture.getStore()?.wrapped ?? wrappedDiagnostics).set(wrapper, { ...diagnostic(source), ...sanitizeTransferEvidence(transferEvidence) }); const scope = activeDiagnostic.getStore(); const failure = scope?.active ? scope.failures.get(source) : undefined; if (failure) scope!.failures.set(wrapper, failure); return wrapper; } +function sanitizeTransferEvidence(value: unknown): Pick { + if (!value || typeof value !== "object") return {}; + const record = value as Record; + const step = readField(record, "transferStep"); + const kind = readField(record, "transferFailureKind"); + const rpcCode = readField(record, "rpcCode"); + return { + ...(typeof step === "string" && TRANSFER_STEPS.has(step as TransferStep) ? { transferStep: step as TransferStep } : {}), + ...(typeof kind === "string" && TRANSFER_FAILURE_KINDS.has(kind as TransferFailureKind) ? { transferFailureKind: kind as TransferFailureKind } : {}), + ...(typeof rpcCode === "number" && RPC_CODES.has(rpcCode) ? { rpcCode } : {}), + }; +} + /** Decode persisted adapter metadata; never copy arbitrary properties or text. */ export function sanitizeWorkspaceRestoreDiagnostic(value: unknown): WorkspaceRestoreDiagnostic | undefined { if (!value || typeof value !== "object") return undefined; @@ -153,6 +187,7 @@ export function sanitizeWorkspaceRestoreDiagnostic(value: unknown): WorkspaceRes && typeof gitCommand === "string" && GIT_COMMANDS.has(gitCommand as WorkspaceRestoreGitCommand); return { phase, + ...(step === "workspace_transfer" || step === "asset_restore" ? sanitizeTransferEvidence(record) : {}), ...(typeof step === "string" && RESTORE_STEPS.has(step as WorkspaceRestoreStep) ? { step: step as WorkspaceRestoreStep } : {}), errorCode: typeof code === "string" && ERROR_CODES.has(code) ? code : "unknown", @@ -168,14 +203,14 @@ export function sanitizeWorkspaceRestoreDiagnostic(value: unknown): WorkspaceRes export function getWorkspaceRestoreDiagnostic(error: unknown): WorkspaceRestoreDiagnostic | undefined { return error && typeof error === "object" - ? sanitizeWorkspaceRestoreDiagnostic((diagnosticCapture.getStore() ?? restoreDiagnostics).get(error)) : undefined; + ? sanitizeWorkspaceRestoreDiagnostic((diagnosticCapture.getStore()?.restores ?? restoreDiagnostics).get(error)) : undefined; } /** Preserve the scheduler-selected task's snapshot when errors share identity. */ export function recordWorkspaceRestoreDiagnostic(error: unknown, diagnostic: WorkspaceRestoreDiagnostic | undefined): void { const safe = sanitizeWorkspaceRestoreDiagnostic(diagnostic); if (error && typeof error === "object") { - const receipts = diagnosticCapture.getStore() ?? restoreDiagnostics; + const receipts = diagnosticCapture.getStore()?.restores ?? restoreDiagnostics; if (safe) receipts.set(error, safe); else receipts.delete(error); } @@ -229,10 +264,10 @@ export async function withWorkspaceRestoreDiagnostics( } catch (error) { const failure = scope.failures.get(error); const step = failure?.step; - const fields = { phase, ...(step ? { step } : {}), ...diagnostic(error), + const fields = sanitizeWorkspaceRestoreDiagnostic({ phase, ...(step ? { step } : {}), ...diagnostic(error), ...(phase === "workspace" && step === "git_integration" && failure?.gitCommand ? { gitCommand: failure.gitCommand, gitFailureKind: failure.gitFailureKind } : {}), - }; + })!; recordWorkspaceRestoreDiagnostic(error, fields); if (parent?.active && step) parent.failures.set(error, { ...failure, sequence: ++parent.sequence, step }); try { diff --git a/packages/plugins/sandbox-providers/daytona/src/file-sync-diagnostics.test.ts b/packages/plugins/sandbox-providers/daytona/src/file-sync-diagnostics.test.ts new file mode 100644 index 00000000000..4e5e2da1cee --- /dev/null +++ b/packages/plugins/sandbox-providers/daytona/src/file-sync-diagnostics.test.ts @@ -0,0 +1,113 @@ +import { promises as fs } from "node:fs"; +import { execFileSync } from "node:child_process"; +import os from "node:os"; +import path from "node:path"; +import { afterEach, expect, it, vi } from "vitest"; +import { environmentSyncErrorData, withEnvironmentSyncErrorCapture, readEnvironmentSyncErrorDiagnostic } from "@paperclipai/plugin-sdk"; +vi.mock("./plugin.js", async () => { + const { NOOP_PLUGIN_TRACER } = await import("@paperclipai/plugin-sdk"); + return { getPluginTracer: () => NOOP_PLUGIN_TRACER }; +}); +import { performSyncOut } from "./file-sync.js"; + +const dirs: string[] = []; +afterEach(async () => { await Promise.all(dirs.splice(0).map(dir => fs.rm(dir, { recursive: true, force: true }))); }); + +it.each([ + ["guard", "sandbox_guard", "command_failed", 42], + ["create", "archive_create", "command_failed", 2], + ["download", "archive_download", "download_failed", undefined], + ["missing", "archive_download", "download_missing", undefined], + ["file", "file_download", "download_failed", undefined], + ["file-missing", "file_download", "download_missing", undefined], +] as const)("records producer evidence for %s without copying output or paths", async (failure, step, kind, exitCode) => { + const dir = await fs.mkdtemp(path.join(os.tmpdir(), "daytona-diagnostic-")); + dirs.push(dir); + const sandbox = { + process: { executeCommand: async (command: string) => ({ + exitCode: failure === "guard" ? 42 : failure === "create" && command.includes("tar -c") ? 2 : 0, + result: "private-command-output", + }) }, + fs: { + downloadFiles: async (requests: Array<{ source: string }>) => failure.endsWith("missing") ? [] + : requests.map(request => ({ source: request.source, error: "private-download-error" })), + deleteFile: async () => undefined, + }, + }; + await withEnvironmentSyncErrorCapture(async () => { + let caught: unknown; + try { + await performSyncOut({ sandbox: sandbox as never, remoteDir: "/workspace", timeoutSeconds: 30, + operations: [{ operationId: "fixture", files: [{ + sourcePath: "/workspace/private-path", targetPath: path.join(dir, "target"), kind: failure.startsWith("file") ? "file" : "directory", + }] }], + }); + } catch (error) { caught = error; } + expect(caught).toBeInstanceOf(Error); + const diagnostic = readEnvironmentSyncErrorDiagnostic({ data: environmentSyncErrorData(caught) }); + expect(diagnostic).toEqual({ errorCode: "unknown", transferStep: step, transferFailureKind: kind, + ...(exitCode !== undefined ? { exitCode } : {}), + }); + expect(JSON.stringify(diagnostic)).not.toContain("private-"); + expect(caught).not.toHaveProperty("transferStep"); + expect(caught).not.toHaveProperty("exitCode"); + }); +}); + +it("keeps a thrown download exception identical and its nested safe code", async () => { + const original = Object.freeze(new Error("private-download-error", { cause: { code: "ECONNRESET", token: "private-token" } })); + const sandbox = { + process: { executeCommand: async () => ({ exitCode: 0 }) }, + fs: { downloadFiles: async () => { throw original; }, deleteFile: async () => undefined }, + }; + await withEnvironmentSyncErrorCapture(async () => { + await expect(performSyncOut({ sandbox: sandbox as never, remoteDir: "/workspace", timeoutSeconds: 30, + operations: [{ operationId: "fixture", files: [{ sourcePath: "/workspace/private-path", targetPath: "/unused", kind: "directory" }] }], + })).rejects.toBe(original); + expect(readEnvironmentSyncErrorDiagnostic({ data: environmentSyncErrorData(original) })).toEqual({ + errorCode: "ECONNRESET", transferStep: "archive_download", + }); + }); +}); + + +it.each(["file_finalize", "archive_validate", "archive_extract"] as const)("identifies the host-side %s failure without relaxing confinement", async (step) => { + const dir = await fs.mkdtemp(path.join(os.tmpdir(), "daytona-host-diagnostic-")); + dirs.push(dir); + const target = path.join(dir, "target"); + const archive = path.join(dir, "safe.tar"); + if (step === "archive_extract") { + await fs.writeFile(path.join(dir, "safe.txt"), "fixture"); + execFileSync("tar", ["-cf", archive, "-C", dir, "safe.txt"]); + await fs.writeFile(target, "preserve target"); + } + const sandbox = { + process: { executeCommand: async () => ({ exitCode: 0 }) }, + fs: { + downloadFiles: async (requests: Array<{ source: string; destination: string }>) => { + for (const request of requests) { + if (step === "archive_validate") await fs.writeFile(request.destination, "invalid private archive"); + if (step === "archive_extract") await fs.copyFile(archive, request.destination); + } + return requests.map(request => ({ source: request.source })); + }, + deleteFile: async () => undefined, + }, + }; + await withEnvironmentSyncErrorCapture(async () => { + let caught: unknown; + try { + await performSyncOut({ sandbox: sandbox as never, remoteDir: "/workspace", timeoutSeconds: 30, + operations: [{ operationId: "fixture", files: [{ sourcePath: "/workspace/private-path", targetPath: target, + kind: step === "file_finalize" ? "file" : "directory" }] }], + }); + } catch (error) { caught = error; } + expect(caught).toBeInstanceOf(Error); + const diagnostic = readEnvironmentSyncErrorDiagnostic({ data: environmentSyncErrorData(caught) }); + expect(diagnostic?.transferStep).toBe(step); + expect(JSON.stringify(diagnostic)).not.toContain("private"); + if (step === "file_finalize") expect(diagnostic?.errorCode).toBe("ENOENT"); + if (step === "archive_validate") expect(diagnostic?.transferFailureKind).toBe("command_failed"); + if (step === "archive_extract") expect(await fs.readFile(target, "utf8")).toBe("preserve target"); + }); +}); diff --git a/packages/plugins/sandbox-providers/daytona/src/file-sync.ts b/packages/plugins/sandbox-providers/daytona/src/file-sync.ts index ad84d90d394..89847f9ddbd 100644 --- a/packages/plugins/sandbox-providers/daytona/src/file-sync.ts +++ b/packages/plugins/sandbox-providers/daytona/src/file-sync.ts @@ -14,6 +14,7 @@ import type { PluginSyncFileMapping, PluginSyncOperation, } from "@paperclipai/plugin-sdk"; +import { recordEnvironmentSyncError, withEnvironmentSyncTransferStep } from "@paperclipai/plugin-sdk"; import { getPluginTracer } from "./plugin.js"; const execFileAsync = promisify(execFile); @@ -318,9 +319,13 @@ export async function assertTarballEntriesConfined( const validateLine = (line: Buffer) => { // Count empty lines too, so whitespace cannot evade the parsing-work quota. if (++entries > TAR_LISTING_MAX_ENTRIES) { - throw new Error("Daytona syncOut tar listing entry limit exceeded (250000)"); + throw recordEnvironmentSyncError(new Error("Daytona syncOut tar listing entry limit exceeded (250000)"), { transferFailureKind: "listing_entry_limit" }); + } + try { assertTarListingLineConfined(line.toString("utf8")); } + catch (error) { + if (error instanceof UnsafeOutboundArchiveError) recordEnvironmentSyncError(error, { transferFailureKind: "unsafe_archive" }); + throw error; } - assertTarListingLineConfined(line.toString("utf8")); }; const closed = new Promise<{ code: number | null; signal: NodeJS.Signals | null }>((resolve) => { child.once("error", (error) => { spawnError = error; }); @@ -331,11 +336,11 @@ export async function assertTarballEntriesConfined( child.kill("SIGKILL"); }; const timer = setTimeout(() => { - stop(new Error("Daytona syncOut tar listing validation timed out")); + stop(recordEnvironmentSyncError(new Error("Daytona syncOut tar listing validation timed out"), { transferFailureKind: "listing_timeout", errorCode: "ETIMEDOUT" })); }, Math.max(1, Math.min(timeoutMs, TAR_LISTING_TIMEOUT_MS))); child.stderr.on("data", (chunk: Buffer) => { if (stderr.length + chunk.length > TAR_LISTING_MAX_STDERR_BYTES) { - stop(new Error("Daytona syncOut tar listing diagnostics exceed the byte limit")); + stop(recordEnvironmentSyncError(new Error("Daytona syncOut tar listing diagnostics exceed the byte limit"), { transferFailureKind: "listing_stderr_limit" })); return; } stderr = Buffer.concat([stderr, chunk]); @@ -346,14 +351,14 @@ export async function assertTarballEntriesConfined( const bytes = chunk as Buffer; totalBytes += bytes.length; if (totalBytes > TAR_LISTING_MAX_TOTAL_BYTES) { - throw new Error("Daytona syncOut tar total listing byte limit exceeded (64 MiB)"); + throw recordEnvironmentSyncError(new Error("Daytona syncOut tar total listing byte limit exceeded (64 MiB)"), { transferFailureKind: "listing_byte_limit" }); } let start = 0; while (start < bytes.length) { const newline = bytes.indexOf(10, start); const end = newline < 0 ? bytes.length : newline; if (pending.length + end - start > TAR_LISTING_MAX_LINE_BYTES) { - throw new Error("Daytona syncOut refusing tarball with an entry listing exceeding the byte limit"); + throw recordEnvironmentSyncError(new Error("Daytona syncOut refusing tarball with an entry listing exceeding the byte limit"), { transferFailureKind: "listing_line_limit" }); } pending = Buffer.concat([pending, bytes.subarray(start, end)]); if (newline < 0) break; @@ -367,7 +372,9 @@ export async function assertTarballEntriesConfined( if (failure) throw failure; if (spawnError) throw spawnError; if (result.code !== 0) { - throw new Error(`Daytona syncOut tar listing failed (${result.signal ?? result.code}): ${stderr.toString("utf8").trim()}`); + throw recordEnvironmentSyncError(new Error(`Daytona syncOut tar listing failed (${result.signal ?? result.code}): ${stderr.toString("utf8").trim()}`), { + transferFailureKind: "command_failed", exitCode: result.code ?? undefined, + }); } } catch (error) { stop(error instanceof Error ? error : new Error(String(error))); @@ -381,11 +388,13 @@ export async function assertTarballEntriesConfined( async function extractHostTarball(input: { archivePath: string; localDir: string }): Promise { // The archive is sandbox-authored and untrusted: validate every member (and // link target) is confined before letting host-side tar write a single byte. - await assertTarballEntriesConfined(input.archivePath); - await fs.mkdir(input.localDir, { recursive: true }); - await execFileAsync("tar", ["-xf", input.archivePath, "-C", input.localDir], { - env: { ...process.env, COPYFILE_DISABLE: "1" }, - maxBuffer: 32 * 1024 * 1024, + await withEnvironmentSyncTransferStep("archive_validate", () => assertTarballEntriesConfined(input.archivePath)); + await withEnvironmentSyncTransferStep("archive_extract", async () => { + await fs.mkdir(input.localDir, { recursive: true }); + await execFileAsync("tar", ["-xf", input.archivePath, "-C", input.localDir], { + env: { ...process.env, COPYFILE_DISABLE: "1" }, + maxBuffer: 32 * 1024 * 1024, + }); }); } @@ -423,7 +432,9 @@ async function assertSandboxCommandOkWithOutput( const result = await sandbox.process.executeCommand(command, undefined, undefined, timeoutSeconds); if ((result.exitCode ?? 1) !== 0) { const detail = (result.result ?? result.artifacts?.stdout ?? "").toString().trim(); - throw new Error(`Daytona ${label} command failed (exit ${result.exitCode ?? "unknown"})${detail ? `: ${detail}` : ""}`); + throw recordEnvironmentSyncError(new Error(`Daytona ${label} command failed (exit ${result.exitCode ?? "unknown"})${detail ? `: ${detail}` : ""}`), { + transferFailureKind: "command_failed", exitCode: result.exitCode, + }); } return (result.result ?? result.artifacts?.stdout ?? "").toString(); } @@ -1143,17 +1154,14 @@ async function syncOutFileMappings(input: { const { sandbox, mappings, remoteDir, timeoutSeconds } = input; if (mappings.length === 0) return { filesTransferred: 0, bytesTransferred: 0 }; - for (const mapping of mappings) { - assertConfinedSandboxPath(remoteDir, mapping.sourcePath, "source"); - } // Close the validation→download TOCTOU: instead of re-opening each mutable // source, validate-and-snapshot it in one atomic sandbox-side step and download // the immutable snapshot. `snapshots` is index-aligned with `mappings`. - const snapshots = await snapshotOutboundFileSources({ - sandbox, - remoteDir, - sources: mappings.map((mapping) => mapping.sourcePath), - timeoutSeconds, + const snapshots = await withEnvironmentSyncTransferStep("sandbox_guard", async () => { + for (const mapping of mappings) assertConfinedSandboxPath(remoteDir, mapping.sourcePath, "source"); + return snapshotOutboundFileSources({ + sandbox, remoteDir, sources: mappings.map((mapping) => mapping.sourcePath), timeoutSeconds, + }); }); const requests: FileDownloadRequest[] = []; @@ -1179,6 +1187,7 @@ async function syncOutFileMappings(input: { await fs.mkdir(path.dirname(entry.target), { recursive: true }); } } catch (error) { + recordEnvironmentSyncError(error, { transferStep: "file_finalize" }); await cleanup(); throw error; } @@ -1193,7 +1202,7 @@ async function syncOutFileMappings(input: { try { // One batched bulk download for all file mappings, reading the snapshots. // `transfer` span: the real byte download — `sandbox.fs.downloadFiles`. - responses = await withProviderSpan({ + responses = await withEnvironmentSyncTransferStep("file_download", () => withProviderSpan({ name: "transfer", wallMsAttr: SPAN_ATTR.transferWallMs, attributes: { @@ -1201,7 +1210,7 @@ async function syncOutFileMappings(input: { [SPAN_ATTR.transferDirection]: "outbound", }, run: () => sandbox.fs.downloadFiles(requests, timeoutSeconds), - }); + })); } catch (error) { await cleanup(); throw error; @@ -1215,9 +1224,9 @@ async function syncOutFileMappings(input: { const response = bySource.get(entry.snapshot); if (!response || response.error) { await cleanup(); - throw new Error( + throw recordEnvironmentSyncError(new Error( `Daytona syncOut download failed for ${entry.source}: ${response?.error ?? "no response returned"}`, - ); + ), { transferStep: "file_download", transferFailureKind: response ? "download_failed" : "download_missing" }); } } @@ -1233,6 +1242,7 @@ async function syncOutFileMappings(input: { await fs.rename(entry.temp, entry.target); } } catch (error) { + recordEnvironmentSyncError(error, { transferStep: "file_finalize" }); await cleanup(); throw error; } @@ -1251,16 +1261,15 @@ async function syncOutDirectoryMapping(input: { onArchiveRecovery?: () => void; }): Promise<{ filesTransferred: number; bytesTransferred: number }> { const { sandbox, mapping, remoteDir, timeoutSeconds } = input; - assertConfinedSandboxPath(remoteDir, mapping.sourcePath, "source"); // Count the serial sandbox round trips before the transfer, so the transfer // span records how much of the wall time is guard cost. let guardRoundTrips = 0; - await assertSandboxPathsConfined({ - sandbox, - remoteDir, - paths: [mapping.sourcePath], - timeoutSeconds, - label: "outbound symlink-escape guard", + await withEnvironmentSyncTransferStep("sandbox_guard", async () => { + assertConfinedSandboxPath(remoteDir, mapping.sourcePath, "source"); + await assertSandboxPathsConfined({ + sandbox, remoteDir, paths: [mapping.sourcePath], timeoutSeconds, + label: "outbound symlink-escape guard", + }); }); guardRoundTrips += 1; @@ -1310,10 +1319,11 @@ async function syncOutDirectoryMapping(input: { let bytesTransferred = 0; try { for (let attempt = 0; attempt < 2; attempt += 1) { - await assertSandboxCommandOk(sandbox, `sh -c ${shellQuote(attempt === 0 ? tarScript : confinedEntriesScript)}`, timeoutSeconds, "syncOut tar"); + await withEnvironmentSyncTransferStep("archive_create", () => + assertSandboxCommandOk(sandbox, `sh -c ${shellQuote(attempt === 0 ? tarScript : confinedEntriesScript)}`, timeoutSeconds, "syncOut tar")); guardRoundTrips += 1; // `transfer` span: the real byte download — `sandbox.fs.downloadFiles`. - const responses = await withProviderSpan({ + const responses = await withEnvironmentSyncTransferStep("archive_download", () => withProviderSpan({ name: "transfer", wallMsAttr: SPAN_ATTR.transferWallMs, attributes: { @@ -1322,14 +1332,14 @@ async function syncOutDirectoryMapping(input: { }, run: () => sandbox.fs.downloadFiles([{ source: remoteTar, destination: localTar }], timeoutSeconds), - }); + })); const response = responses.find((entry) => entry.source === remoteTar) ?? responses[0]; if (!response || response.error) { - throw new Error( + throw recordEnvironmentSyncError(new Error( `Daytona syncOut directory download failed for ${mapping.sourcePath}: ${response?.error ?? "no response returned"}`, - ); + ), { transferStep: "archive_download", transferFailureKind: response ? "download_failed" : "download_missing" }); } - bytesTransferred += (await fs.stat(localTar)).size; + bytesTransferred += await withEnvironmentSyncTransferStep("archive_download", async () => (await fs.stat(localTar)).size); try { await extractHostTarball({ archivePath: localTar, localDir: mapping.targetPath }); break; diff --git a/packages/plugins/sandbox-providers/daytona/src/plugin.test.ts b/packages/plugins/sandbox-providers/daytona/src/plugin.test.ts index 6022d9b3fe7..79eb85bd0a5 100644 --- a/packages/plugins/sandbox-providers/daytona/src/plugin.test.ts +++ b/packages/plugins/sandbox-providers/daytona/src/plugin.test.ts @@ -36,7 +36,7 @@ import plugin, { __setDaytonaPluginContextForTest, } from "./plugin.js"; import type { PluginContext } from "@paperclipai/plugin-sdk"; -import { environmentCreationCleanupErrorData, readEnvironmentCreationCleanupError } from "@paperclipai/plugin-sdk"; +import { environmentSyncErrorData, readEnvironmentSyncErrorDiagnostic, withEnvironmentSyncErrorCapture, environmentCreationCleanupErrorData, readEnvironmentCreationCleanupError } from "@paperclipai/plugin-sdk"; import manifest from "./manifest.js"; import { parseTarVerboseListingLine, splitLinkEntryOnce } from "./file-sync.js"; @@ -5263,33 +5263,32 @@ describe("daytona native file-sync hooks", () => { }); }); - it("classifies a deleted sandbox during syncOut with a stable unrecoverable code", async () => { + it.each(["sandbox_access", "file_download"] as const)("keeps the deleted-sandbox policy and bounded %s evidence", async (step) => { const hostDir = await makeHostDir(); - mockGet.mockRejectedValue( - new MockDaytonaNotFoundError("provider detail must not escape"), - ); - - await expect( - plugin.definition.onEnvironmentSyncOut?.({ - driverKey: "daytona", - companyId: "company-1", - environmentId: "env-1", - config: { timeoutMs: 300000, reuseLease: true }, - lease: syncLease(), - operations: [ - { - operationId: "sync-op-missing-sandbox", - files: [ - { - sourcePath: `${REMOTE_DIR}/out/result.txt`, - targetPath: path.join(hostDir, "result.txt"), - kind: "file", - }, - ], - }, - ], - }), - ).rejects.toThrow("daytona_sandbox_not_found"); + const source = Object.freeze(Object.assign(new MockDaytonaNotFoundError("private-provider-detail"), { + status: 404, cause: { code: "EIO", token: "private-provider-token" }, + })); + if (step === "sandbox_access") mockGet.mockRejectedValue(source); + else { + const sandbox = createMockSandbox(); + sandbox.fs.downloadFiles.mockRejectedValue(source); + mockGet.mockResolvedValue(sandbox); + } + await withEnvironmentSyncErrorCapture(async () => { + const error = await plugin.definition.onEnvironmentSyncOut!({ + driverKey: "daytona", companyId: "company-1", environmentId: "env-1", + config: { timeoutMs: 300000, reuseLease: true }, lease: syncLease(), + operations: [{ operationId: "sync-op-missing-sandbox", files: [{ + sourcePath: `${REMOTE_DIR}/out/result.txt`, targetPath: path.join(hostDir, "result.txt"), kind: "file", + }] }], + }).catch(error => error); + expect(error).toBeInstanceOf(Error); + expect(error.message).toBe("daytona_sandbox_not_found"); + expect(error).not.toHaveProperty("cause"); + const diagnostic = readEnvironmentSyncErrorDiagnostic({ data: environmentSyncErrorData(error) }); + expect(diagnostic).toEqual({ errorCode: "EIO", httpStatus: 404, transferStep: step }); + expect(JSON.stringify(diagnostic)).not.toContain("private-provider-"); + }); }); it("syncOut snapshot guard re-checks the resolved source is a non-symlink regular file immediately before copying (validation→copy TOCTOU)", async () => { diff --git a/packages/plugins/sandbox-providers/daytona/src/plugin.ts b/packages/plugins/sandbox-providers/daytona/src/plugin.ts index 9c085d6a696..bbfa70a049d 100644 --- a/packages/plugins/sandbox-providers/daytona/src/plugin.ts +++ b/packages/plugins/sandbox-providers/daytona/src/plugin.ts @@ -9,7 +9,7 @@ import type { Resources, Sandbox, } from "@daytonaio/sdk"; -import { decodeChannelBytes, definePlugin, NOOP_PLUGIN_TRACER, PluginEnvironmentCreationCleanupError, readEnvironmentCreationCleanupError } from "@paperclipai/plugin-sdk"; +import { decodeChannelBytes, definePlugin, NOOP_PLUGIN_TRACER, PluginEnvironmentCreationCleanupError, readEnvironmentCreationCleanupError, withEnvironmentSyncTransferStep, preserveEnvironmentSyncErrorDiagnostic } from "@paperclipai/plugin-sdk"; import type { PluginContext, PluginEnvironmentCreationCleanup, @@ -3271,8 +3271,11 @@ const plugin = definePlugin({ }; try { return await withSandboxActivityGate(scope, async () => { - const sandbox = await getSandbox(scope, { bypassTeardownGate: true }); - await ensureSandboxStarted(sandbox, timeoutSeconds); + const sandbox = await withEnvironmentSyncTransferStep("sandbox_access", async () => { + const resolved = await getSandbox(scope, { bypassTeardownGate: true }); + await ensureSandboxStarted(resolved, timeoutSeconds); + return resolved; + }); const result = await performSyncOut({ sandbox, operations: params.operations, @@ -3288,7 +3291,7 @@ const plugin = definePlugin({ // unexported workspace bytes no longer exist. Convert the SDK class to a // stable cross-worker message; every other error remains retryable. if (error instanceof DaytonaNotFoundError) { - throw new Error("daytona_sandbox_not_found"); + throw preserveEnvironmentSyncErrorDiagnostic(new Error("daytona_sandbox_not_found"), error); } throw error; } diff --git a/packages/plugins/sandbox-providers/daytona/src/tar-listing.test.ts b/packages/plugins/sandbox-providers/daytona/src/tar-listing.test.ts index 796e93bc00b..cf0fb98a770 100644 --- a/packages/plugins/sandbox-providers/daytona/src/tar-listing.test.ts +++ b/packages/plugins/sandbox-providers/daytona/src/tar-listing.test.ts @@ -4,6 +4,7 @@ import path from "node:path"; import { afterEach, expect, it, vi } from "vitest"; vi.mock("./plugin.js", () => ({ getPluginTracer: () => undefined })); +import { environmentSyncErrorData, withEnvironmentSyncErrorCapture, withEnvironmentSyncTransferStep, readEnvironmentSyncErrorDiagnostic } from "@paperclipai/plugin-sdk"; import { assertTarballEntriesConfined } from "./file-sync.js"; const temporaryDirectories: string[] = []; @@ -100,18 +101,28 @@ it.each([false, true])("bounds aggregate listing bytes at 64 MiB (over quota: %s const line = "-rw-r--r-- 0/0 0 2026-09-27 12:00 ".padEnd(4095, "x") + "\n"; const count = 64 * 1024 * 1024 / Buffer.byteLength(line) + Number(overQuota); const checkReaped = await fakeTar(listingWriter(line, count)); - const result = assertTarballEntriesConfined("unused.tar"); - if (overQuota) await expect(result).rejects.toThrow("total listing byte limit"); - else await expect(result).resolves.toBeUndefined(); + await withEnvironmentSyncErrorCapture(async () => { + const result = assertTarballEntriesConfined("unused.tar"); + if (overQuota) { + const error = await result.catch(error => error); + expect(error.message).toContain("total listing byte limit"); + expect(readEnvironmentSyncErrorDiagnostic({ data: environmentSyncErrorData(error) })?.transferFailureKind).toBe("listing_byte_limit"); + } else await expect(result).resolves.toBeUndefined(); + }); await checkReaped(); }, 30_000); it.each([false, true])("bounds aggregate listing entries at 250,000 (over quota: %s)", async (overQuota) => { // The over-quota member has no newline: EOF must pass the same admission gate. const checkReaped = await fakeTar(listingWriter(safeLine + "\n", 250_000, overQuota ? safeLine : "")); - const result = assertTarballEntriesConfined("unused.tar"); - if (overQuota) await expect(result).rejects.toThrow("listing entry limit"); - else await expect(result).resolves.toBeUndefined(); + await withEnvironmentSyncErrorCapture(async () => { + const result = assertTarballEntriesConfined("unused.tar"); + if (overQuota) { + const error = await result.catch(error => error); + expect(error.message).toContain("listing entry limit"); + expect(readEnvironmentSyncErrorDiagnostic({ data: environmentSyncErrorData(error) })?.transferFailureKind).toBe("listing_entry_limit"); + } else await expect(result).resolves.toBeUndefined(); + }); await checkReaped(); }, 30_000); @@ -189,3 +200,26 @@ it("rejects spawn failures without hanging", async () => { vi.stubEnv("PATH", await temporaryDirectory()); await expect(assertTarballEntriesConfined("unused.tar")).rejects.toThrow("ENOENT"); }); + + +it.each([ + ["process.exitCode = 2;", "command_failed", 2, "unknown"], + ['process.stdout.write("x".repeat(65_537)); setInterval(() => {}, 1_000);', "listing_line_limit", undefined, "unknown"], + ['process.stderr.write("x".repeat(65_537)); setInterval(() => {}, 1_000);', "listing_stderr_limit", undefined, "unknown"], + ['process.stdout.write("invalid-private-listing");', "unsafe_archive", undefined, "unknown"], + ['setInterval(() => {}, 1_000);', "listing_timeout", undefined, "ETIMEDOUT"], +] as const)("records bounded listing evidence and reaps the child (%s)", async (script, kind, exitCode, errorCode) => { + const checkReaped = await fakeTar(script); + await withEnvironmentSyncErrorCapture(async () => { + let caught: unknown; + try { await withEnvironmentSyncTransferStep("archive_validate", () => assertTarballEntriesConfined("private-archive.tar", 2_000)); } + catch (error) { caught = error; } + expect(caught).toBeInstanceOf(Error); + const diagnostic = readEnvironmentSyncErrorDiagnostic({ data: environmentSyncErrorData(caught) }); + expect(diagnostic).toEqual({ errorCode, transferStep: "archive_validate", transferFailureKind: kind, + ...(exitCode !== undefined ? { exitCode } : {}), + }); + expect(JSON.stringify(diagnostic)).not.toContain("private-"); + }); + await checkReaped(); +}); diff --git a/packages/plugins/sdk/src/environment-sync-error.ts b/packages/plugins/sdk/src/environment-sync-error.ts index 0450310a07a..a630bb126e1 100644 --- a/packages/plugins/sdk/src/environment-sync-error.ts +++ b/packages/plugins/sdk/src/environment-sync-error.ts @@ -1,8 +1,67 @@ +import { AsyncLocalStorage } from "node:async_hooks"; + /** Diagnostic evidence only. Never use this envelope to authorize cleanup or retry. */ export interface PluginEnvironmentSyncErrorDiagnostic { errorCode: string; httpStatus?: number; exitCode?: number; + transferStep?: EnvironmentSyncTransferStep; + transferFailureKind?: EnvironmentSyncTransferFailureKind; +} + +const TRANSFER_STEPS = new Set([ + "sandbox_access", "sandbox_guard", "file_download", "file_finalize", + "archive_create", "archive_download", "archive_validate", "archive_extract", +] as const); +export type EnvironmentSyncTransferStep = typeof TRANSFER_STEPS extends Set ? T : never; +const TRANSFER_FAILURE_KINDS = new Set([ + "command_failed", "download_failed", "download_missing", "unsafe_archive", + "listing_timeout", "listing_entry_limit", "listing_byte_limit", "listing_line_limit", "listing_stderr_limit", +] as const); +export type EnvironmentSyncTransferFailureKind = typeof TRANSFER_FAILURE_KINDS extends Set ? T : never; +interface Capture { + active: boolean; + sequence: number; + errors: WeakMap; +} +const capture = new AsyncLocalStorage(); + +/** Scope evidence to one outbound RPC, including its error serialization. */ +export async function withEnvironmentSyncErrorCapture(operation: () => Promise): Promise { + const scope: Capture = { active: true, sequence: 0, errors: new WeakMap() }; + return capture.run(scope, async () => { + try { return await operation(); } + finally { scope.active = false; } + }); +} + +/** Producer-owned evidence only; never mutate, wrap, or retain a raw cause. */ +export function recordEnvironmentSyncError(error: T, diagnostic: Partial): T { + const scope = capture.getStore(); + const safe = sanitize(diagnostic); + if (scope?.active && error && typeof error === "object") { + if (safe) scope.errors.set(error, { sequence: ++scope.sequence, diagnostic: safe }); + else scope.errors.delete(error); + } + return error; +} + +/** A nested failing stage wins; a later attempt gets fresh attribution. */ +export async function withEnvironmentSyncTransferStep(step: EnvironmentSyncTransferStep, operation: () => Promise): Promise { + const scope = capture.getStore(); + if (!scope?.active || !TRANSFER_STEPS.has(step)) return await operation(); + const started = scope.sequence; + try { return await operation(); } + catch (error) { + if (scope.active && error && typeof error === "object") { + const saved = scope.errors.get(error); + const diagnostic = saved && saved.sequence > started ? saved.diagnostic : { errorCode: "unknown" }; + scope.errors.set(error, { + sequence: ++scope.sequence, diagnostic: { ...diagnostic, transferStep: diagnostic.transferStep ?? step }, + }); + } + throw error; + } } const SCHEMA = "paperclip/environment-sync-error/v1"; @@ -29,13 +88,23 @@ function sanitize(value: unknown): PluginEnvironmentSyncErrorDiagnostic | undefi const errorCode = typeof code === "string" && ERROR_CODES.has(code) ? code : "unknown"; const httpStatus = integer(field(value, "httpStatus"), 400, 599); const exitCode = integer(field(value, "exitCode"), 1, 255); - if (errorCode === "unknown" && httpStatus === undefined && exitCode === undefined) return undefined; - return { errorCode, ...(httpStatus !== undefined ? { httpStatus } : {}), ...(exitCode !== undefined ? { exitCode } : {}) }; + const step = field(value, "transferStep"); + const kind = field(value, "transferFailureKind"); + const transferStep = typeof step === "string" && TRANSFER_STEPS.has(step as EnvironmentSyncTransferStep) ? step as EnvironmentSyncTransferStep : undefined; + const transferFailureKind = typeof kind === "string" && TRANSFER_FAILURE_KINDS.has(kind as EnvironmentSyncTransferFailureKind) ? kind as EnvironmentSyncTransferFailureKind : undefined; + if (errorCode === "unknown" && httpStatus === undefined && exitCode === undefined && !transferStep && !transferFailureKind) return undefined; + return { errorCode, ...(httpStatus !== undefined ? { httpStatus } : {}), ...(exitCode !== undefined ? { exitCode } : {}), + ...(transferStep ? { transferStep } : {}), ...(transferFailureKind ? { transferFailureKind } : {}), + }; } /** Capture known codes and numbers, never arbitrary error data, text, or names. */ -export function environmentSyncErrorData(error: unknown): unknown { +function captureDiagnostic(error: unknown): PluginEnvironmentSyncErrorDiagnostic | undefined { const diagnostic: PluginEnvironmentSyncErrorDiagnostic = { errorCode: "unknown" }; + const scope = capture.getStore(); + if (scope?.active && error && typeof error === "object") { + Object.assign(diagnostic, scope.errors.get(error)?.diagnostic); + } let current = error; // Bound nested SDK causes, including cyclic errors and throwing getters. for (let depth = 0; depth < 4 && current && typeof current === "object"; depth++) { @@ -47,8 +116,18 @@ export function environmentSyncErrorData(error: unknown): unknown { diagnostic.exitCode ??= integer(field(current, "exitCode"), 1, 255) ?? integer(code, 1, 255); current = field(current, "cause"); } - const safe = sanitize(diagnostic); - return safe ? { schema: SCHEMA, diagnostic: safe } : undefined; + return sanitize(diagnostic); +} + +export function environmentSyncErrorData(error: unknown): unknown { + const diagnostic = captureDiagnostic(error); + return diagnostic ? { schema: SCHEMA, diagnostic } : undefined; +} + +/** Retain bounded evidence across an existing wrapper, without adding a raw cause. */ +export function preserveEnvironmentSyncErrorDiagnostic(wrapper: T, source: unknown): T { + const diagnostic = captureDiagnostic(source); + return recordEnvironmentSyncError(wrapper, diagnostic ?? { errorCode: "unknown" }); } /** Revalidate the worker envelope before adapting it into host-only diagnostics. */ diff --git a/packages/plugins/sdk/src/index.ts b/packages/plugins/sdk/src/index.ts index 7610aaa9a1d..50430912516 100644 --- a/packages/plugins/sdk/src/index.ts +++ b/packages/plugins/sdk/src/index.ts @@ -452,7 +452,7 @@ export { export { PluginEnvironmentCreationCleanupError, environmentCreationCleanupErrorData, readEnvironmentCreationCleanupError } from "./environment-creation-cleanup.js"; export type { PluginEnvironmentCreationCleanup } from "./environment-creation-cleanup.js"; -export { readEnvironmentSyncErrorDiagnostic } from "./environment-sync-error.js"; +export { preserveEnvironmentSyncErrorDiagnostic, environmentSyncErrorData, readEnvironmentSyncErrorDiagnostic, withEnvironmentSyncErrorCapture, withEnvironmentSyncTransferStep, recordEnvironmentSyncError } from "./environment-sync-error.js"; export type { PluginEnvironmentSyncErrorDiagnostic } from "./environment-sync-error.js"; export type { AiConnectionPool, AiConnectionPoolConfig, AiConnectionPoolMember, AiConnectionRouterRequest, AiConnectionRouterResult, AiConnectionRouterSelection } from "@paperclipai/shared"; diff --git a/packages/plugins/sdk/src/worker-rpc-host.ts b/packages/plugins/sdk/src/worker-rpc-host.ts index d6781c1b6db..a374e31922d 100644 --- a/packages/plugins/sdk/src/worker-rpc-host.ts +++ b/packages/plugins/sdk/src/worker-rpc-host.ts @@ -1,6 +1,6 @@ import type { AiConnectionRouterRequest, AiConnectionRouterResult } from "@paperclipai/shared"; import { environmentCreationCleanupErrorData } from "./environment-creation-cleanup.js"; -import { environmentSyncErrorData } from "./environment-sync-error.js"; +import { environmentSyncErrorData, withEnvironmentSyncErrorCapture } from "./environment-sync-error.js"; import { createPluginIdleDrain } from "./idle-drain.js"; /** * Worker-side RPC host — runs inside the child process spawned by the host. @@ -1570,6 +1570,13 @@ export function startWorkerRpcHost(options: WorkerRpcHostOptions): WorkerRpcHost * Dispatches to the correct handler based on the method name. */ async function handleHostRequest(request: JsonRpcRequest): Promise { + if (request.method === "environmentSyncOut") { + return withEnvironmentSyncErrorCapture(() => handleHostRequestInScope(request)); + } + return handleHostRequestInScope(request); + } + + async function handleHostRequestInScope(request: JsonRpcRequest): Promise { const { id, method, params } = request; let done: (() => void) | undefined; try { diff --git a/packages/plugins/sdk/tests/environment-sync-error.test.ts b/packages/plugins/sdk/tests/environment-sync-error.test.ts index 26d98054936..c2480a707ce 100644 --- a/packages/plugins/sdk/tests/environment-sync-error.test.ts +++ b/packages/plugins/sdk/tests/environment-sync-error.test.ts @@ -1,5 +1,5 @@ import { describe, expect, it } from "vitest"; -import { environmentSyncErrorData, readEnvironmentSyncErrorDiagnostic } from "../src/environment-sync-error.js"; +import { preserveEnvironmentSyncErrorDiagnostic, environmentSyncErrorData, readEnvironmentSyncErrorDiagnostic, withEnvironmentSyncErrorCapture, withEnvironmentSyncTransferStep, recordEnvironmentSyncError } from "../src/environment-sync-error.js"; const schema = "paperclip/environment-sync-error/v1"; @@ -46,3 +46,70 @@ describe("environment sync diagnostic envelope", () => { expect(readEnvironmentSyncErrorDiagnostic(Object.defineProperty({}, "data", { get() { throw new Error("private"); } }))).toBeUndefined(); }); }); + + +describe("request-local transfer evidence", () => { + it("preserves frozen error identity, nested attribution, and later-attempt attribution", async () => { + const error = Object.freeze(new Error("private-message")); + const before = Object.getOwnPropertyDescriptors(error); + await withEnvironmentSyncErrorCapture(async () => { + await expect(withEnvironmentSyncTransferStep("archive_download", () => + withEnvironmentSyncTransferStep("sandbox_access", async () => { throw error; }))).rejects.toBe(error); + expect(readEnvironmentSyncErrorDiagnostic({ data: environmentSyncErrorData(error) })?.transferStep).toBe("sandbox_access"); + await expect(withEnvironmentSyncTransferStep("archive_create", async () => { throw error; })).rejects.toBe(error); + expect(readEnvironmentSyncErrorDiagnostic({ data: environmentSyncErrorData(error) })?.transferStep).toBe("archive_create"); + }); + expect(environmentSyncErrorData(error)).toBeUndefined(); + expect(Object.getOwnPropertyDescriptors(error)).toEqual(before); + await withEnvironmentSyncErrorCapture(async () => expect(environmentSyncErrorData(error)).toBeUndefined()); + }); + + it("does not trust labels on raw errors, their causes, or unknown producer values", async () => { + const error = { transferStep: "archive_create", transferFailureKind: "command_failed", + cause: { transferStep: "archive_validate", message: "private-message" } }; + await withEnvironmentSyncErrorCapture(async () => { + expect(environmentSyncErrorData(error)).toBeUndefined(); + recordEnvironmentSyncError(error, { transferStep: "private-stage", transferFailureKind: "private-kind", exitCode: 900 } as never); + expect(environmentSyncErrorData(error)).toBeUndefined(); + }); + expect(readEnvironmentSyncErrorDiagnostic({ data: { schema, diagnostic: { + errorCode: "unknown", transferStep: "private-stage", transferFailureKind: "private-kind", rpcCode: -32003, + } } })).toBeUndefined(); + }); + + it("omits evidence from detached work after the RPC scope closes", async () => { + const error = new Error("private-message"); + let resume!: () => void; + const gate = new Promise(resolve => { resume = resolve; }); + let detached!: Promise; + await withEnvironmentSyncErrorCapture(async () => { + detached = withEnvironmentSyncTransferStep("archive_create", async () => { await gate; throw error; }) + .catch(failure => environmentSyncErrorData(failure)); + }); + resume(); + expect(await detached).toBeUndefined(); + }); +}); + + +it("copies safe evidence into an existing wrapper only within its request", async () => { + const source = Object.freeze(Object.assign(new Error("private-message"), { status: 404, cause: { code: "EIO", token: "private-token" } })); + const wrapper = Object.freeze(new Error("stable-policy-message")); + await withEnvironmentSyncErrorCapture(async () => { + await expect(withEnvironmentSyncTransferStep("file_download", async () => { throw source; })).rejects.toBe(source); + expect(preserveEnvironmentSyncErrorDiagnostic(wrapper, source)).toBe(wrapper); + expect(readEnvironmentSyncErrorDiagnostic({ data: environmentSyncErrorData(wrapper) })).toEqual({ + errorCode: "EIO", httpStatus: 404, transferStep: "file_download", + }); + expect(wrapper).not.toHaveProperty("cause"); + expect(wrapper.message).toBe("stable-policy-message"); + // Reusing a wrapper for a later untyped failure must clear the earlier evidence. + preserveEnvironmentSyncErrorDiagnostic(wrapper, new Error("private-untyped-message")); + expect(environmentSyncErrorData(wrapper)).toBeUndefined(); + }); + expect(environmentSyncErrorData(wrapper)).toBeUndefined(); + await withEnvironmentSyncErrorCapture(async () => { + preserveEnvironmentSyncErrorDiagnostic(wrapper, source); + expect(readEnvironmentSyncErrorDiagnostic({ data: environmentSyncErrorData(wrapper) })).toEqual({ errorCode: "EIO", httpStatus: 404 }); + }); +}); diff --git a/server/src/__tests__/environment-sync-out-error.test.ts b/server/src/__tests__/environment-sync-out-error.test.ts index 36407d2453d..8ca1a0b7c77 100644 --- a/server/src/__tests__/environment-sync-out-error.test.ts +++ b/server/src/__tests__/environment-sync-out-error.test.ts @@ -3,6 +3,7 @@ import { PassThrough } from "node:stream"; import { describe, expect, it } from "vitest"; import { createRequest, definePlugin, isJsonRpcResponse, JsonRpcCallError, parseMessage, + readEnvironmentSyncErrorDiagnostic, withEnvironmentSyncTransferStep, recordEnvironmentSyncError, PLUGIN_RPC_ERROR_CODES, serializeMessage, startWorkerRpcHost, type JsonRpcResponse, } from "@paperclipai/plugin-sdk"; import { @@ -53,8 +54,9 @@ describe("environment sync-out restore diagnostics", () => { await expect(withWorkspaceRestoreDiagnostics("workspace", () => withWorkspaceRestoreStep("workspace_transfer", async () => { throw preserveEnvironmentSyncOutErrorDiagnostic(error); }), async (line) => { logs.push(line); })).rejects.toBe(error); - expect(getWorkspaceRestoreDiagnostic(error)).toEqual({ phase: "workspace", step: "workspace_transfer", errorCode: "ETIMEDOUT", httpStatus: 504 }); - expect(logs).toEqual(['[paperclip] Workspace restore diagnostic: {"phase":"workspace","step":"workspace_transfer","errorCode":"ETIMEDOUT","httpStatus":504}\n']); + expect(getWorkspaceRestoreDiagnostic(error)).toEqual({ phase: "workspace", step: "workspace_transfer", errorCode: "ETIMEDOUT", httpStatus: 504, rpcCode: PLUGIN_RPC_ERROR_CODES.WORKER_ERROR }); + expect(logs).toHaveLength(1); + expect(JSON.parse(logs[0].slice(logs[0].indexOf("{")))).toEqual(getWorkspaceRestoreDiagnostic(error)); expect({ message: error.message, code: error.code, data: error.data }).toEqual(before); expect(error).not.toHaveProperty("cause"); }); @@ -76,3 +78,69 @@ describe("environment sync-out restore diagnostics", () => { expect(preserveEnvironmentSyncOutErrorDiagnostic(null)).toBeNull(); }); }); + + +it("isolates overlapping and sequential worker calls that throw the same object", async () => { + const original = Object.freeze(new Error("Transfer failed")); + const before = Object.getOwnPropertyDescriptors(original); + const stdin = new PassThrough(); + const stdout = new PassThrough(); + const lines = createInterface({ input: stdout }); + let release!: () => void; + const gate = new Promise(resolve => { release = resolve; }); + let started!: () => void; + const ready = new Promise(resolve => { started = resolve; }); + const worker = startWorkerRpcHost({ stdin, stdout, plugin: definePlugin({ + async setup() {}, + async onEnvironmentSyncOut(params) { + if (params.operations?.[0]?.operationId === "first") { + return withEnvironmentSyncTransferStep("archive_create", async () => { + recordEnvironmentSyncError(original, { exitCode: 2, transferFailureKind: "command_failed" }); + started(); await gate; throw original; + }); + } + if (params.operations?.[0]?.operationId === "second") { + return withEnvironmentSyncTransferStep("file_download", async () => { throw original; }); + } + throw original; + }, + async onEnvironmentSyncIn() { throw original; }, + }) }); + const call = (id: string, method = "environmentSyncOut") => new Promise((resolve) => { + const listener = (wire: string) => { + const message = parseMessage(wire); + if (isJsonRpcResponse(message) && message.id === id && message.error) { + lines.off("line", listener); resolve(new JsonRpcCallError(message.error)); + } + }; + lines.on("line", listener); + stdin.write(serializeMessage(createRequest(method, { operations: [{ operationId: id, files: [] }] }, id))); + }); + try { + const first = call("first"); + await ready; + const second = await call("second"); + expect(readEnvironmentSyncErrorDiagnostic(second)).toEqual({ errorCode: "unknown", transferStep: "file_download" }); + release(); + expect(readEnvironmentSyncErrorDiagnostic(await first)).toEqual({ + errorCode: "unknown", transferStep: "archive_create", transferFailureKind: "command_failed", exitCode: 2, + }); + expect((await call("third")).data).toBeUndefined(); + expect((await call("unrelated", "environmentSyncIn")).data).toBeUndefined(); + expect(Object.getOwnPropertyDescriptors(original)).toEqual(before); + } finally { release(); worker.stop(); lines.close(); stdin.destroy(); stdout.destroy(); } +}); + +it.each([ + [new JsonRpcCallError({ code: PLUGIN_RPC_ERROR_CODES.TIMEOUT, message: "private-rpc-message" }), PLUGIN_RPC_ERROR_CODES.TIMEOUT], + [new JsonRpcCallError({ code: 123, message: "private-rpc-message" }), undefined], + [Object.assign(new Error("private-rpc-message"), { code: PLUGIN_RPC_ERROR_CODES.TIMEOUT }), undefined], +])("records known typed RPC codes without interpreting arbitrary errors", async (error, rpcCode) => { + const logs: string[] = []; + await expect(withWorkspaceRestoreDiagnostics("workspace", () => withWorkspaceRestoreStep("workspace_transfer", async () => { + throw preserveEnvironmentSyncOutErrorDiagnostic(error); + }), async line => { logs.push(line); })).rejects.toBe(error); + expect(getWorkspaceRestoreDiagnostic(error)?.rpcCode).toBe(rpcCode); + expect(getWorkspaceRestoreDiagnostic(error)?.transferStep).toBeUndefined(); + expect(JSON.stringify(logs)).not.toContain("private-"); +}); diff --git a/server/src/__tests__/run-failure-sentry-real-sdk.test.ts b/server/src/__tests__/run-failure-sentry-real-sdk.test.ts index f92e0a42bc0..5f5e2c180eb 100644 --- a/server/src/__tests__/run-failure-sentry-real-sdk.test.ts +++ b/server/src/__tests__/run-failure-sentry-real-sdk.test.ts @@ -378,4 +378,64 @@ describe.skipIf(!sentryPackage)("run failure context with the real Sentry SDK", expect(unrelated).not.toHaveProperty("contexts.run_failure"); expect(unrelated).not.toHaveProperty("tags.error_code"); }); + it("sends only validated transfer and RPC evidence, isolated from unrelated captures", async () => { + const Sentry = sentryPackage!; + const events: Array> = []; + vi.stubEnv("SENTRY_DSN_BACKEND", "https://public@example.invalid/1"); + vi.doMock("../peer-version-check.js", () => ({ checkExactPeerVersions: () => ({ ok: true }) })); + vi.doMock("@sentry/node", () => ({ ...Sentry, + init: (options: Record) => Sentry.init({ ...options, + transport: () => ({ send: async () => ({}), flush: async () => true }), + beforeSend: (event: Record) => { events.push(event); return event; }, + }), + })); + vi.resetModules(); + const { sentryReady, captureRunFailure, captureException } = await import("../sentry.js"); + await sentryReady; + for (const known of [true, false]) { + const error = Object.freeze(new Error("private-transfer-message")); + // Worker/typed-host propagation has its own real RPC round-trip test. + // This narrow contract exercises the host receipt through the real Sentry + // transport without requiring plugin builds or dependency lifecycle scripts. + const evidence = { + transferStep: known ? "archive_create" : "private-transfer-path", + transferFailureKind: known ? "command_failed" : "private-transfer-command", + rpcCode: known ? -32002 : -12345, + cause: { token: "private-transfer-token" }, output: "private-transfer-output", + }; + const logs: string[] = []; + const restore = createWorkspaceRestoreTeardown({ stagedRuntime: { + restoreWorkspace: progress => withWorkspaceRestoreDiagnostics("workspace", () => + withWorkspaceRestoreStep("workspace_transfer", async () => { throw preserveWorkspaceRestoreErrorDiagnostic(error, { exitCode: 2 }, evidence); }), progress), + }, onLog: async (_stream, line) => { logs.push(line); }, startMessage: "Restoring workspace\n", failurePrefix: "Workspace restore failed" }); + const outcome = await restore(); + expect(outcome.ok).toBe(false); + if (outcome.ok) throw new Error("Expected transfer fixture failure"); + expect(outcome.diagnostic?.rpcCode).toBe(known ? -32002 : undefined); + expect(JSON.stringify({ outcome, logs })).not.toContain("private-transfer-"); + const diagnostics = sanitizeRunFailureDiagnostics(collectRunFailureDiagnostics({ resultJson: { + workspaceRestoreFailure: outcome.code, workspaceRestoreDiagnostic: { ...outcome.diagnostic, + message: "private-transfer-persisted", cause: { token: "private-transfer-token" }, + }, + } } as unknown as typeof heartbeatRuns.$inferSelect, {})); + captureRunFailure({ taskId: "11111111-1111-4111-8111-111111111111", runId: "22222222-2222-4222-8222-222222222222", + errorMessage: "Workspace restore failed. Workspace files need recovery.", errorCode: "workspace_restore_failed", + agentAdapter: "fixture-adapter", runStatus: "failed", exitCode: 1, signal: null, diagnostics, + }); + await Sentry.flush(2000); + const execution = (events.at(-1)?.contexts as Record>).run_execution; + expect(execution.workspaceRestoreTransferStep).toBe(known ? "archive_create" : undefined); + expect(execution.workspaceRestoreTransferFailureKind).toBe(known ? "command_failed" : undefined); + expect(execution.workspaceRestoreRpcCode).toBe(known ? -32002 : undefined); + expect(execution.workspaceRestoreExitCode).toBe(2); + } + captureException(new Error("Unrelated transfer fixture")); + await Sentry.flush(2000); + expect(events).toHaveLength(3); + expect(events[2]).not.toHaveProperty("contexts.run_execution"); + expect(events[2]).not.toHaveProperty("contexts.run_failure"); + expect(events[2]).not.toHaveProperty("tags.error_code"); + expect(JSON.stringify(events)).not.toContain("private-transfer-"); + }); + }); diff --git a/server/src/services/__tests__/run-failure-diagnostics.test.ts b/server/src/services/__tests__/run-failure-diagnostics.test.ts index 0bf0d2a9eb8..868e1841e1a 100644 --- a/server/src/services/__tests__/run-failure-diagnostics.test.ts +++ b/server/src/services/__tests__/run-failure-diagnostics.test.ts @@ -342,3 +342,33 @@ describe("run failure diagnostics", () => { expect(collectRunFailureDiagnostics(run({ startedAt: new Date(NaN), finishedAt: new Date() }), {}).execution).not.toHaveProperty("durationMs"); }); }); + + +it("revalidates persisted transfer evidence and excludes arbitrary nested payloads", () => { + for (const known of [true, false]) { + const result = sanitizeRunFailureDiagnostics(collectRunFailureDiagnostics(run({ resultJson: { + workspaceRestoreFailure: "restore_failed", workspaceRestoreDiagnostic: { + phase: "workspace", step: "workspace_transfer", errorCode: "unknown", + transferStep: known ? "archive_create" : "private-command", + transferFailureKind: known ? "command_failed" : "private-output", rpcCode: known ? -32003 : -12345, + command: "private-command", cause: { message: "private-nested-message" }, + }, + } }), {})); + expect(result.execution.workspaceRestoreTransferStep).toBe(known ? "archive_create" : undefined); + expect(result.execution.workspaceRestoreTransferFailureKind).toBe(known ? "command_failed" : undefined); + expect(result.execution.workspaceRestoreRpcCode).toBe(known ? -32003 : undefined); + expect(JSON.stringify(result)).not.toContain("private-"); + } +}); + +it("does not project transfer fields into an unrelated restore step", () => { + const result = collectRunFailureDiagnostics(run({ resultJson: { + workspaceRestoreFailure: "restore_failed", workspaceRestoreDiagnostic: { + phase: "workspace", step: "git_import", errorCode: "unknown", + transferStep: "archive_create", transferFailureKind: "command_failed", rpcCode: -32003, + }, + } }), {}); + expect(result.execution).not.toHaveProperty("workspaceRestoreTransferStep"); + expect(result.execution).not.toHaveProperty("workspaceRestoreTransferFailureKind"); + expect(result.execution).not.toHaveProperty("workspaceRestoreRpcCode"); +}); diff --git a/server/src/services/environment-sync-out-error.ts b/server/src/services/environment-sync-out-error.ts index f9574f696a7..64a4aa75a23 100644 --- a/server/src/services/environment-sync-out-error.ts +++ b/server/src/services/environment-sync-out-error.ts @@ -1,15 +1,17 @@ -import { readEnvironmentSyncErrorDiagnostic } from "@paperclipai/plugin-sdk"; +import { readEnvironmentSyncErrorDiagnostic, JsonRpcCallError, PLUGIN_RPC_ERROR_CODES } from "@paperclipai/plugin-sdk"; import { preserveWorkspaceRestoreErrorDiagnostic } from "@paperclipai/adapter-utils/workspace-restore-diagnostics"; /** Keep the original RPC failure and policy fields; add only private diagnostic evidence. */ export function preserveEnvironmentSyncOutErrorDiagnostic(error: unknown): unknown { const diagnostic = readEnvironmentSyncErrorDiagnostic(error); - if (diagnostic && error && typeof error === "object") { + const rpcCode = error instanceof JsonRpcCallError && (Object.values(PLUGIN_RPC_ERROR_CODES) as number[]).includes(error.code) + ? error.code : undefined; + if ((diagnostic || rpcCode !== undefined) && error && typeof error === "object") { preserveWorkspaceRestoreErrorDiagnostic(error, { - code: diagnostic.errorCode, - status: diagnostic.httpStatus, - exitCode: diagnostic.exitCode, - }); + code: diagnostic?.errorCode, + status: diagnostic?.httpStatus, + exitCode: diagnostic?.exitCode, + }, { ...diagnostic, rpcCode }); } return error; } diff --git a/server/src/services/run-failure-diagnostics.ts b/server/src/services/run-failure-diagnostics.ts index 605f0b8afef..71188371e37 100644 --- a/server/src/services/run-failure-diagnostics.ts +++ b/server/src/services/run-failure-diagnostics.ts @@ -203,6 +203,9 @@ export function collectRunFailureDiagnostics(run: Run, options: RunFailureReport if (diagnostic.step) execution.workspaceRestoreStep = diagnostic.step; if (diagnostic.httpStatus !== undefined) execution.workspaceRestoreHttpStatus = diagnostic.httpStatus; if (diagnostic.exitCode !== undefined) execution.workspaceRestoreExitCode = diagnostic.exitCode; + if (diagnostic.transferStep) execution.workspaceRestoreTransferStep = diagnostic.transferStep; + if (diagnostic.transferFailureKind) execution.workspaceRestoreTransferFailureKind = diagnostic.transferFailureKind; + if (diagnostic.rpcCode !== undefined) execution.workspaceRestoreRpcCode = diagnostic.rpcCode; if (diagnostic.gitCommand) execution.workspaceRestoreGitCommand = diagnostic.gitCommand; if (diagnostic.gitFailureKind) execution.workspaceRestoreGitFailureKind = diagnostic.gitFailureKind; } From b9750b152fccaba51939fbf2b48e237087626f33 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu, 8 Oct 2026 22:36:44 -0700 Subject: [PATCH 4/5] build(deps): bump rustls from 0.23.43 to 0.23.45 in /packages/paperclip-runner/runner (#15295) Bumps [rustls](https://github.com/rustls/rustls) from 0.23.43 to 0.23.45.
Commits
  • 2976d90 Prepare 0.23.45
  • f9d4ee9 Handshake "alignment" check covers previously-received messages
  • c4e92e8 Keep data needed for HRR processing together
  • e553a7a server: TLS1.2 is not available after a HRR
  • 5dff9da Test whether server negotiates TLS1.2 after HRR
  • 185a063 reject a second ClientHello that changes the cipher suite
  • 9fafe6d reject a second ClientHello that drops pre_shared_key
  • 1b42c5f providers: zeroize private key DER
  • 64ad386 Bump version to 0.23.44
  • 1efbf66 bogo: remove PostQuantum setup
  • Additional commits viewable in compare view

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=rustls&package-manager=cargo&previous-version=0.23.43&new-version=0.23.45)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/paperclipai/paperclip/network/alerts).
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- packages/paperclip-runner/runner/Cargo.lock | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/packages/paperclip-runner/runner/Cargo.lock b/packages/paperclip-runner/runner/Cargo.lock index 0d730210424..068c99f9fa0 100644 --- a/packages/paperclip-runner/runner/Cargo.lock +++ b/packages/paperclip-runner/runner/Cargo.lock @@ -2211,9 +2211,9 @@ dependencies = [ [[package]] name = "rustls" -version = "0.23.43" +version = "0.23.45" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0283386ce02abc0151e1761d08802dfe86c173b0b494af5cbc086574e453da06" +checksum = "0d41d731c7d2f962d1ccc364cec258de3c0e93b38c2fb3ba97ac74513048d634" dependencies = [ "aws-lc-rs", "once_cell", From f6d2a6b79edbb6c7820e5356ddb0c4debb33c57a Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 9 Oct 2026 07:59:19 -0500 Subject: [PATCH 5/5] refactor(heartbeat): extract queue admission and wakeup dispatch (#15626) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The heartbeat service admits and dispatches agent runs. > - Earlier extractions separated workspaces, preparation, state, retries, and restart recovery. > - Queue admission and wakeup dispatch still occupy more than 4,200 lines in the main service. > - Their transaction, budget, ownership, and concurrency checks must stay together. > - This pull request moves those operations into one queue module and adds boundary tests. > - Reviewers can inspect queue policy separately from adapter execution. ## Linked Issues or Issue Description **What existing behavior does this improve?** The structure and testability of heartbeat queue admission and wakeup dispatch. **Subsystem affected** server/ — orchestration services. **Current behavior** Wake coalescing, queued claims, dispatch, daily limits, and native wake intents live inside `heartbeat.ts`. **Proposed behavior** Keep their behavior in `server/src/services/heartbeat/queue.ts`. Bind lifecycle callbacks and shared ownership with `createHeartbeatQueue`. **Reason and benefit** This removes 4,243 lines from `heartbeat.ts`, leaving 14,201 lines. The queue module has 4,576 lines and keeps related policy together in one file. **Breaking changes** None. The service methods, public exports, database transactions, locks, and execution gates keep their existing behavior. **Additional context** Continues #15622 and the earlier merged heartbeat extractions. A search of open and closed PRs found no duplicate queue extraction. Related #15600 adds a new experimental scheduler; this change preserves the current scheduler policy. This does not add a roadmap feature. ## What Changed - Move wake coalescing and batching, queue claims, daily caps, concurrency and priority checks, timer admission, queue resumption, and native status wake intents into `heartbeat/queue.ts`. - Pass lifecycle callbacks and the existing process-wide execution and wakeup sets into the factory. Preserve construction order with three forwarding callbacks. - Keep adapter execution, cancellation, and status writes in the service. - Add nine tests for inert construction, scheduling suppression, shared wake tracking, policy wiring, claim release, newer ownership, and budget gates. - Document the queue module boundary in `doc/DEVELOPING.md`. ## Verification - Before extraction: 133 tests passed across nine existing suites. - After extraction: **150 tests passed across 12 files**, including all nine new module tests and real PostgreSQL coverage. ```sh pnpm exec vitest run server/src/services/heartbeat/queue.test.ts server/src/__tests__/heartbeat-queued-run-claim-isolation.test.ts server/src/__tests__/heartbeat-start-lock.test.ts server/src/__tests__/heartbeat-stale-queue-invalidation.test.ts server/src/__tests__/heartbeat-scheduling-suppression.test.ts server/src/__tests__/heartbeat-comment-wake-batching.test.ts server/src/__tests__/heartbeat-running-followup.test.ts server/src/__tests__/heartbeat-dependency-scheduling.test.ts server/src/__tests__/heartbeat-auto-checkout.test.ts server/src/__tests__/heartbeat-archived-company-guard.test.ts server/src/__tests__/heartbeat-task-drain-admission-release.test.ts server/src/__tests__/heartbeat-worktree-suppression.test.ts ``` - Additional control and recovery coverage: **390 tests passed across five files**. Combined focused coverage is **540 passing tests across 17 files**, with no skips. ```sh pnpm exec vitest run server/src/__tests__/heartbeat-process-recovery.test.ts server/src/__tests__/heartbeat-native-cleanup-admission.test.ts server/src/__tests__/heartbeat-native-status-context.test.ts server/src/__tests__/heartbeat-task-drain.test.ts server/src/services/chat-control-admission-retry.test.ts ``` - Full `pnpm -r typecheck` and `pnpm build` passed. - Full local `pnpm test:run` was started and then stopped after the full CI suite passed. It did not finish and is not counted as a full local pass. - All **54 current-head checks passed**; two optional Storybook checks were skipped. This includes the full general and serialized test matrix, typechecks, build checks, Runner verification, and canary dry run. [CI run](https://github.com/paperclipai/paperclip/actions/runs/37847175452). - Greptile completed on head `cc0001961a052a716d81c136929f89312e0b5dee` with **5/5 and no actionable findings**. There are no review threads or merge conflicts. - Structural comparison confirms 31 moved function bodies and eight declarations match the originals. Remaining service function bodies and all 140 public exports are preserved. - An initial local run skipped database tests because the host had exhausted its PostgreSQL shared-memory slots. After verified stale segments were cleared, all 150 focused tests ran and passed with no skips. ## Risks Moving closures can break shared ownership or callback construction order. The factory receives the original sets and explicit callbacks. It starts no work during construction. Transaction boundaries, compare-and-set predicates, budget gates, cleanup holds, and lock ordering stay intact. This PR has no schema or API changes. ## Model Used OpenAI GPT-6 via Codex. The exact serving model ID and context window were not exposed in this session. Used reasoning, repository inspection, shell tools, and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip --- doc/DEVELOPING.md | 13 +- server/src/services/heartbeat.ts | 4469 +----------------- server/src/services/heartbeat/queue.test.ts | 186 + server/src/services/heartbeat/queue.ts | 4576 +++++++++++++++++++ 4 files changed, 4886 insertions(+), 4358 deletions(-) create mode 100644 server/src/services/heartbeat/queue.test.ts create mode 100644 server/src/services/heartbeat/queue.ts diff --git a/doc/DEVELOPING.md b/doc/DEVELOPING.md index 84dfd41b1e5..318fe321025 100644 --- a/doc/DEVELOPING.md +++ b/doc/DEVELOPING.md @@ -1240,8 +1240,8 @@ Run preparation is in `server/src/services/heartbeat/run-preparation.ts`. It own issue and wake context, responsible-user resolution, routine environment snapshots, skill mentions, adapter environment configuration, and MCP/tool access setup. `createHeartbeatRunPreparation(db)` binds the context loaders to a service's database -without doing database work during construction. `heartbeat.ts` keeps queueing, -dispatch, cancellation, and execution order, and re-exports the existing +without doing database work during construction. `heartbeat.ts` composes queue +dispatch with cancellation and execution, and re-exports the existing public helpers and configuration-incomplete error class. Keep preparation policy changes in this module and its tests. @@ -1272,6 +1272,15 @@ and shared execution sets so separate service instances keep the same ownership and shutdown barriers. Cleanup single-flight state stays at module scope. Keep recovery policy changes separate from retry scheduling and execution changes. +Queue admission and wakeup dispatch are in `server/src/services/heartbeat/queue.ts`. +It owns wake coalescing and batching, queued-run claims, daily heartbeat caps, +concurrency and priority checks, timer admission, and native status wake intents. +`createHeartbeatQueue` binds these operations without querying or starting work. +The service supplies lifecycle effects, execution callbacks, worktree gates, and +its process-wide execution and wakeup promise sets. Forwarding callbacks preserve +construction order for retry and recovery services. Keep queue policy changes +separate from this extraction and from adapter execution changes. + ## Wake Context Delivery Built-in adapters deliver wake context through the run prompt, including structured diff --git a/server/src/services/heartbeat.ts b/server/src/services/heartbeat.ts index 56731629407..af9e1605ce4 100644 --- a/server/src/services/heartbeat.ts +++ b/server/src/services/heartbeat.ts @@ -1,3 +1,16 @@ +import { + ChatControlRecoveryUnresolvedError, + shouldDeferFollowupWakeForSameIssue, + shouldQueueFollowupForRunningIssueWake, + DEFERRED_WAKE_CONTEXT_KEY, + EXECUTION_PATH_HEARTBEAT_RUN_STATUSES, + shouldRequireIssueCommentForWake, + createHeartbeatQueue, +} from "./heartbeat/queue.js"; +export { + shouldDeferFollowupWakeForSameIssue, + shouldQueueFollowupForRunningIssueWake, +} from "./heartbeat/queue.js"; import { HEARTBEAT_RUN_TERMINAL_STATUSES, DETACHED_PROCESS_ERROR_CODE, @@ -90,9 +103,6 @@ import { createAdapterRuntimeMcpAccess, createManagedMcpRunConfig, revokeHeartbeatRunGatewayTokens, - enrichWakeContextSnapshot, - hasInteractionContinuationWakeContext, - isInteractionResolutionWakePayload, createHeartbeatRunPreparation, } from "./heartbeat/run-preparation.js"; export { @@ -232,7 +242,10 @@ import { resolvePaperclipInstanceRoot } from "../home-paths.js"; import { dotRunnerBroker } from "./dot-runner-broker.js"; import { isAiAuthenticationBlocked } from "./ai-auth-failure.js"; import { nativeRetryCancellationCommitCondition, rethrowNativeCancellationLockConflict, claimCancellationRequest, startupCancellationFence } from "./native-runtime/native-cancellation-request.js"; -import { CHAT_COMPLETION_WAKE_REASON, prepareChatCompletionTurn, chatCompletionInstruction, isCompletedOnboardingHandoffWake } from "./chat-completion-delivery.js"; +import { + prepareChatCompletionTurn, + chatCompletionInstruction, +} from "./chat-completion-delivery.js"; import { AgentDirectoryReuseInvalidatedError, isAgentDirectoryCopy } from "./agent-directory-working-copies.js"; import type { PaperclipTurnContext } from "@paperclipai/adapter-utils/server-utils"; @@ -250,23 +263,27 @@ import { reserveRunBudget } from "./budget-reservations.js"; import { accountRunCost, createCostAccountingReconciler } from "./run-cost-accounting.js"; import { createRunUsageRecorder } from "./usage-receipts.js"; import { applyWorkspaceRestoreFailure } from "@paperclipai/adapter-utils/workspace-restore-result"; -import { compareCents, hasWorkspaceRestoreFailure } from "@paperclipai/shared"; import { - nonIdleSlackIssueCondition, -} from "./slack-conversation-state.js"; + hasWorkspaceRestoreFailure, +} from "@paperclipai/shared"; + import { settleSlackConversation } from "./slack-conversation-lifecycle.js"; import { toolActionDeliveryService } from "./tool-action-delivery.js"; import { readQueuedInteractionResponse } from "./queued-interaction-response.js"; -import { isConversation, isConversationExecutionWake, isWaitingConversation, prepareConversationTurn, settleConversationTurn } from "./agent-conversations.js"; +import { + isConversation, + isWaitingConversation, + prepareConversationTurn, + settleConversationTurn, +} from "./agent-conversations.js"; import { withAdapterExecutionPhase } from "@paperclipai/adapter-utils/execution-phase"; import { getConversationConfirmationContext } from "./conversation-confirmation-context.js"; import { PROCESS_IDENTITY_RECORDED, recordNativeLocalProcessStop } from "./native-local-process-stop.js"; import { hasAcknowledgedNativeReassignmentStopIntent, hasAcknowledgedNativeStopIntent, isAcknowledgedNativeStop, acknowledgedNativeStopExecutionHasStopped } from "./acknowledged-native-stop.js"; import { legacyControllerBootId, - legacyControllerClaim, renewLegacyControllerLease, watchLegacyControllerLease, } from "./legacy-controller-lease.js"; @@ -281,7 +298,9 @@ import { admitExplicitNativeContinuation, undeliveredLegacyUserCommentIds, } from "./explicit-native-continuation.js"; -import { canRetryStoppedRun, isCancelledNativeStartup } from "./cancelled-native-startup.js"; +import { + isCancelledNativeStartup, +} from "./cancelled-native-startup.js"; import { connectionIntentService } from "./connection-intents.js"; import { prepareManagedAiRuntime, @@ -295,9 +314,9 @@ import { aiConnectionRouterService, AiConnectionPoolExhausted, applyAiConnection import { aiConnectionSessionCompatibilityInputs, managedAiSessionIdentityCompatible } from "./ai-connection-session.js"; import { executionBlockerPredicate, getExecutionBlocker } from "./execution-blocker.js"; import { CONVERSATION_CONTINUATION_POLICY, claimedAdapterType, runUsedConversationAdapter, hasConversationContinuationPolicy, isConversationAdapter } from "./conversation-continuation.js"; -import { recordExecutionWait } from "./execution-wait.js"; + import { getNativeReviewAssignment, readNativeReviewAssignmentContext } from "./native-runtime/native-review-participant.js"; -import { claimQueuedNativeReviewRun } from "./native-runtime/native-review-dispatch.js"; + import { buildNativeReviewRequest } from "./native-runtime/native-review-prompt.js"; import { legacyExecutionNeedsReconciliationWithEvidence, @@ -323,11 +342,7 @@ import { initializeRunIdentity, } from "./run-identity.js"; import { - assertDurableChatWakeupReceipt, - assertDurableChatWakeupRequest, authorizeFailedChatRunRetryWake, - FailedChatRunRetryAuthorizationError, - unadmittedChatWakeupCondition, } from "./durable-chat-wakeup.js"; import { prepareHeartbeatGitHubLaunchers } from "./heartbeat-github-launchers.js"; import { @@ -335,9 +350,11 @@ import { prepareGitHubExecutionEnvironment, startAdapterExecutionTargetPaperclipBridge, } from "@paperclipai/adapter-utils/execution-target"; -import { agentService } from "./agents.js"; + import { agentInstructionWorkingCopyService, instructionWorkingCopyGuidance } from "./agent-instruction-working-copies.js"; -import { normalizeLegacyRunnerProvider, resolveManagedOpenAiBilling } from "@paperclipai/adapter-utils"; +import { + resolveManagedOpenAiBilling, +} from "@paperclipai/adapter-utils"; import fs from "node:fs/promises"; import path from "node:path"; import { execFile as execFileCallback } from "node:child_process"; @@ -364,7 +381,6 @@ import { } from "drizzle-orm"; import type { Db } from "@paperclipai/db"; import { - AGENT_DEFAULT_MAX_CONCURRENT_RUNS, CHAT_PROVIDERS, ISSUE_CONTINUATION_SUMMARY_DOCUMENT_KEY, ISSUE_DISPOSITION_REPAIR_RETRY_REASON, @@ -389,16 +405,12 @@ import { activityLog, approvals, chatActions, - chatConversations, - chatEndpoints, companySkillTestRuns, companies, completionContracts, - costEvents, documentRevisions, environmentLeases, issueDocuments, - executionWorkspaces, heartbeatRunEvents, heartbeatRuns, issueApprovals, @@ -432,7 +444,6 @@ import { export { scrubGitCredentialText }; import { publishLiveEvent } from "./live-events.js"; import { - allocateHeartbeatRunEventSeq, appendHeartbeatRunEvent, type AppendHeartbeatRunEventInput, } from "./heartbeat-run-events.js"; @@ -518,9 +529,7 @@ import { asBoolean, asNumber, } from "../adapters/utils.js"; -import { - isExternalChatWaitAuthorizationContention, -} from "./native-runtime/chat-attachment-reuse.js"; + import { EXTERNAL_CHAT_QUESTION_RESPONSE_KEY, } from "./native-runtime/external-chat-question-response.js"; @@ -575,25 +584,13 @@ import { } from "./heartbeat-stop-metadata.js"; import { CHAT_CONTROL_RECOVERY_ADMISSION_KEY, - CHAT_CONTROL_RECOVERY_STOP_CODE, CHAT_CONTROL_RECOVERY_UNRESOLVED_CODE, - chatControlRecoveryAdmission, - readChatControlRecoveryAdmission, - readChatControlNativeParent, - readChatControlRecoveryStop, } from "./chat-control-recovery-stop.js"; import { classifyRunLiveness, type RunLivenessClassificationInput, } from "./run-liveness.js"; -import { - ISSUE_NEW_INPUT_ACTIVITY_ACTIONS, - ISSUE_PROGRESS_ACTIVITY_ACTIONS, - ISSUE_REWAKE_LOOKBACK_MS, - ISSUE_REWAKE_RUN_SAMPLE_LIMIT, - evaluateIssueRewakeThrottle, - isThrottleCandidateIssueRewake, -} from "./issue-rewake-throttle.js"; + import { logActivity, publishPluginDomainEvent, @@ -633,10 +630,8 @@ import { } from "./runner-goals.js"; import { resolveChatRunPresentationAuthorizationReason } from "./chat-run-publications.js"; - - import { visibleIssueCondition } from "./issue-visibility.js"; -import { ISSUE_BLOCKERS_RESOLVED_WAKE_REASON } from "./issue-dependency-wakeups.js"; + import { buildIssueMonitorClearedPatch, buildIssueMonitorTriggeredPatch, @@ -644,8 +639,6 @@ import { parseIssueExecutionState, } from "./issue-execution-policy.js"; import { - ISSUE_TREE_CONTROL_INTERACTION_WAKE_REASONS, - isVerifiedIssueTreeControlInteractionWake, issueTreeControlService, } from "./issue-tree-control.js"; import { @@ -681,7 +674,6 @@ import { buildExecutionWorkspaceAdapterConfig, gateProjectExecutionWorkspacePolicy, issueExecutionWorkspaceModeForPersistedWorkspace, - isUnrunnableWorktreeCombo, parseIssueExecutionWorkspaceSettings, parseProjectExecutionWorkspacePolicy, resolveEffectiveWorkspaceStrategyType, @@ -689,9 +681,6 @@ import { resolveExecutionWorkspaceMode, resolveSharedWorkspaceConcurrency, selectEnvironmentExecutionWorkspaceSettings, - WORKSPACE_WORKTREE_REQUIRES_PROJECT_CODE, - WORKSPACE_WORKTREE_REQUIRES_PROJECT_MESSAGE, - WORKSPACE_WORKTREE_REQUIRES_PROJECT_REMEDIATION, } from "./execution-workspace-policy.js"; import { instanceSettingsService, @@ -739,7 +728,6 @@ import { isNonAssigneeWorkspaceBusyRetry, extractWakeCommentIds, deriveCommentId, - allowsIssueInteractionWake, isResolvedInteractionContinuationWakeContext, } from "../modules/run-dispatch/index.js"; import { @@ -760,15 +748,13 @@ import { } from "./recovery/review-path-recovery.js"; import { resolveRequiredSuccessfulRunHandoffOnValidPath } from "./successful-run-handoff-state.js"; import { taskWatchdogService } from "./task-watchdogs.js"; -import { withAgentStartLock } from "./agent-start-lock.js"; + import { evaluateAgentInvokability, evaluateAgentInvokabilityFromDb, - shouldCancelRunsForNonInvokableAgent, DIRECT_NON_INVOKABLE_STATUSES, - type AgentOrgRow, } from "./agent-invokability.js"; -import { isHeartbeatWakeOnDemandEnabled } from "./heartbeat-policy.js"; + import { redactQuarantinedBodyForHigherTrust, sanitizeQuarantinedCommentForHigherTrust, @@ -796,7 +782,7 @@ import { import { environmentService } from "./environments.js"; import { parseExecutionPolicyBootstrapEnv } from "./execution-policy-bootstrap.js"; -import { retryChatControlAdmission } from "./chat-control-admission-retry.js"; + import { environmentRuntimeService, type ProviderResourceDisposition, @@ -849,32 +835,21 @@ export function redactSuccessfulRunHandoffEvidence( ); } -const HEARTBEAT_MAX_CONCURRENT_RUNS_DEFAULT = AGENT_DEFAULT_MAX_CONCURRENT_RUNS; -const HEARTBEAT_MAX_CONCURRENT_RUNS_MIN = 1; -const HEARTBEAT_MAX_CONCURRENT_RUNS_MAX = 50; const LIVENESS_BOOKKEEPING_ACTIVITY_ACTIONS = [ "environment.lease_acquired", "environment.lease_released", "cost.reported", ]; -const DEFERRED_WAKE_CONTEXT_KEY = "_paperclipWakeContext"; - const ACCEPTED_PLAN_CONVERSION_SKILL_KEY = "paperclipai/paperclip/paperclip-converting-plans-to-tasks"; const execFile = promisify(execFileCallback); -const EXECUTION_PATH_HEARTBEAT_RUN_STATUSES = [ - "queued", - "running", - "scheduled_retry", -] as const; - const UNSUCCESSFUL_HEARTBEAT_RUN_TERMINAL_STATUSES = [ "failed", "cancelled", "timed_out", ] as const; -const TIMER_ACTIONABLE_ISSUE_STATUSES = ["todo", "in_progress"] as const; + export { ACTIVE_RUN_OUTPUT_CONTINUE_REARM_MS, ACTIVE_RUN_OUTPUT_CRITICAL_THRESHOLD_MS, @@ -929,14 +904,6 @@ function nonRetryablePreflightFailureCode(error: unknown): string | null { const code = readNonEmptyString(parseObject(error.details).code); return code && NON_RETRYABLE_PREFLIGHT_FAILURE_CODES.has(code) ? code : null; } -class ChatControlRecoveryUnresolvedError extends Error { - constructor() { - super( - "Run admission could not acquire its database locks after bounded retries. No provider work started. Review database contention and send a fresh request; this attempt will not automatically retry.", - ); - } -} - export { MAX_TURN_CONTINUATION_RETRY_REASON }; export const MAX_TURN_CONTINUATION_WAKE_REASON = "max_turns_continuation_retry"; const MAX_TURN_CONTINUATION_DEFAULT_MAX_ATTEMPTS = 2; @@ -1070,13 +1037,6 @@ function mergeAdapterRecoveryMetadata(input: { : {}), }; } -const RUNNING_ISSUE_WAKE_REASONS_REQUIRING_FOLLOWUP = new Set([ - CHAT_COMPLETION_WAKE_REASON, - "approval_approved", - ISSUE_BLOCKERS_RESOLVED_WAKE_REASON, - "issue_recovery_action_restored", -]); - // Routes and the scheduler construct separate heartbeatService instances, but // they must agree on in-process adapter executions when reaping stale runs. const activeRunExecutions = new Set(); @@ -1241,19 +1201,6 @@ export function isConfigurationIncompleteFailedRun( ); } - - -function normalizeMaxConcurrentRuns(value: unknown) { - const parsed = Math.floor( - asNumber(value, HEARTBEAT_MAX_CONCURRENT_RUNS_DEFAULT), - ); - if (!Number.isFinite(parsed)) return HEARTBEAT_MAX_CONCURRENT_RUNS_DEFAULT; - return Math.max( - HEARTBEAT_MAX_CONCURRENT_RUNS_MIN, - Math.min(HEARTBEAT_MAX_CONCURRENT_RUNS_MAX, parsed), - ); -} - interface ParsedIssueAssigneeAdapterOverrides { adapterConfig: Record | null; useProjectWorkspace: boolean | null; @@ -1311,51 +1258,6 @@ function parseIssueAssigneeAdapterOverrides( }; } -function shouldRequireIssueCommentForWake( - contextSnapshot: Record | null | undefined, -) { - if (contextSnapshot?.skipIssueComment === true) return false; - - const wakeReason = readNonEmptyString(contextSnapshot?.wakeReason); - return ( - wakeReason === "issue_assigned" || - wakeReason === "execution_review_requested" || - wakeReason === "execution_approval_requested" || - wakeReason === "execution_changes_requested" - ); -} - -async function listUnresolvedBlockerSummaries( - dbOrTx: Pick, - companyId: string, - issueId: string, - unresolvedBlockerIssueIds: string[], -) { - const ids = [...new Set(unresolvedBlockerIssueIds.filter(Boolean))]; - if (ids.length === 0) return []; - return dbOrTx - .select({ - id: issues.id, - identifier: issues.identifier, - title: issues.title, - status: issues.status, - priority: issues.priority, - assigneeAgentId: issues.assigneeAgentId, - assigneeUserId: issues.assigneeUserId, - }) - .from(issueRelations) - .innerJoin(issues, eq(issueRelations.issueId, issues.id)) - .where( - and( - eq(issueRelations.companyId, companyId), - eq(issueRelations.type, "blocks"), - eq(issueRelations.relatedIssueId, issueId), - inArray(issues.id, ids), - ), - ) - .orderBy(asc(issues.title)); -} - export function formatRuntimeWorkspaceWarningLog(warning: string) { return { stream: "stdout" as const, @@ -1415,20 +1317,6 @@ export function isWorkspaceSyncConflictFailure( ); } -export function shouldDeferFollowupWakeForSameIssue(input: { - activeRunStatus: string | null | undefined; - isSameExecutionAgent: boolean; - wakeCommentId: string | null | undefined; - forceFreshSession: boolean; -}) { - // A comment follow-up or explicit fresh-session wake needs a new run boundary. - if (!input.isSameExecutionAgent) return false; - if (input.activeRunStatus !== "running") return false; - if (input.wakeCommentId) return true; - if (input.forceFreshSession) return true; - return false; -} - export function shouldAutoCheckoutIssueForWake(input: { contextSnapshot: Record | null | undefined; issueStatus: string | null; @@ -1468,31 +1356,6 @@ export function resolvedInteractionCheckoutExpectedStatuses() { return ["in_progress", "in_review"] as const; } -export function shouldQueueFollowupForRunningIssueWake(input: { - contextSnapshot: Record | null | undefined; - wakeCommentId: string | null; -}) { - if (input.wakeCommentId) return true; - // A structured interaction response is new input just like a comment. It - // must run after the turn that created the interaction instead of being - // merged into that still-running turn. - if ( - readNonEmptyString(input.contextSnapshot?.interactionId) && - readNonEmptyString(input.contextSnapshot?.interactionStatus) - ) { - return true; - } - const wakeReason = readNonEmptyString(input.contextSnapshot?.wakeReason); - if (wakeReason === "issue_children_completed" && ( - input.contextSnapshot?.onboardingCompletion === true || - (input.contextSnapshot?.statusDecisionSource === "native_status_decision" && - readNonEmptyString(input.contextSnapshot?.nativeChildCompletionDecisionId)) - )) return true; - return Boolean( - wakeReason && RUNNING_ISSUE_WAKE_REASONS_REQUIRING_FOLLOWUP.has(wakeReason), - ); -} - function isCheckoutConflictError(error: unknown): boolean { return ( error instanceof HttpError && @@ -2485,6 +2348,77 @@ export function heartbeatService( }, }); + const heartbeatQueue = createHeartbeatQueue(db, { + setRunStatus, + setWakeupStatus, + appendRunEvent, + releaseIssueExecutionAndPromote, + issuesSvc, + options, + publishRunLifecyclePluginEvent, + finalizeAgentStatus, + getAgent, + cancelRunInternal, + toAgentOrgRow, + getAgentInvokability, + budgets, + activeRunExecutions, + treeControlSvc, + runDispatch, + applyRunDispatchPostCommitEffects, + resolveResponsibleUserIdForRun, + getIssueExecutionContext, + isHeartbeatRunTerminalStatus, + liveRunExecutions, + getSchedulingSuppression, + resumeExecutionWaitComments, + getWorktreeExecutionCutoff, + resumeQueuedCommentInterrupt, + resumeSavedLegacyComments, + cancelActiveForAgentInternal, + listCompanyAgentOrgRows, + executeRun, + activeRunExecutionPromises, + activeWakeupPromises, + instanceSettings, + getRun, + sweepPendingCleanupLeases, + resolveExplicitResumeSessionOverride, + resolveSessionBeforeForWakeup, + hasResolvablePriorSessionWorkspaceForWake, + getRoutineEnvForExecutionIssue, + resolveResponsibleUserIdForRunSeed, + formatIssueIdentifierLink, + wakeQueue, + isSameTaskScope, + runTaskKey, + filterZombieCoalesceTarget, + }); + + const { + countRunningRunsForAgent, + releaseRunClaimedJustBeforeSuppression, + claimQueuedRun, + withChatControlRecoveryGate, + trackWakeup, + resumeQueuedRuns, + parseHeartbeatPolicy, + claimDueTimerHeartbeat, + } = heartbeatQueue; + + // Earlier factories capture these callbacks before queue construction. + function enqueueWakeup(...args: Parameters) { + return heartbeatQueue.enqueueWakeup(...args); + } + + function startNextQueuedRunForAgent(...args: Parameters) { + return heartbeatQueue.startNextQueuedRunForAgent(...args); + } + + function dispatchPendingNativeStatusWakeups(...args: Parameters) { + return heartbeatQueue.dispatchPendingNativeStatusWakeups(...args); + } + // Applies the post-commit effects a wake-queue release returns, exactly as // the original release function did before its writes moved into that // module: publish + dispatch a promoted or recovery run, log a reopened @@ -5869,252 +5803,6 @@ export function heartbeatService( return { outcome: "retry_exhausted" as const, queuedRun: null }; } - function parseHeartbeatPolicy(agent: typeof agents.$inferSelect) { - const runtimeConfig = parseObject(agent.runtimeConfig); - const heartbeat = parseObject(runtimeConfig.heartbeat); - - return { - enabled: asBoolean(heartbeat.enabled, false), - intervalSec: Math.max(0, asNumber(heartbeat.intervalSec, 0)), - wakeOnDemand: isHeartbeatWakeOnDemandEnabled(agent), - // A Dot binding has one external turn. Competing assignments must retain - // their queue position instead of claiming a second run that cannot bind. - maxConcurrentRuns: agent.adapterType === "paperclip_runner" && - parseObject(agent.adapterConfig).provider === "openai_dot" - ? 1 : normalizeMaxConcurrentRuns(heartbeat.maxConcurrentRuns), - skipTimerWhenNoActionableWork: asBoolean( - heartbeat.skipTimerWhenNoActionableWork ?? - heartbeat.requireActionableTimerWork ?? - heartbeat.issueOnlyTimer, - false, - ), - maxDailyRuns: normalizeOptionalNonNegativeInteger( - heartbeat.maxDailyRuns ?? - heartbeat.dailyRunLimit ?? - heartbeat.dailyRunCap ?? - heartbeat.maxRunsPerDay, - ), - maxDailyCostCents: normalizeOptionalNonNegativeInteger( - heartbeat.maxDailyCostCents ?? - heartbeat.dailyCostCentsLimit ?? - heartbeat.dailySpendCentsLimit ?? - heartbeat.dailyBudgetCents, - ), - }; - } - - function normalizeOptionalNonNegativeInteger(value: unknown) { - if (value === null || value === undefined || value === "") return null; - const normalized = Math.floor(asNumber(value, 0)); - return normalized >= 0 ? normalized : null; - } - - function currentUtcDayWindow(now = new Date()) { - const start = new Date( - Date.UTC( - now.getUTCFullYear(), - now.getUTCMonth(), - now.getUTCDate(), - 0, - 0, - 0, - 0, - ), - ); - const end = new Date( - Date.UTC( - now.getUTCFullYear(), - now.getUTCMonth(), - now.getUTCDate() + 1, - 0, - 0, - 0, - 0, - ), - ); - return { start, end }; - } - - async function getHeartbeatDailyCapBlock( - agent: typeof agents.$inferSelect, - policy: ReturnType, - options: { - checkRunCap?: boolean; - checkCostCap?: boolean; - excludeRunId?: string | null; - } = {}, - client: Pick = db, - ) { - const checkRunCap = options.checkRunCap ?? true; - const checkCostCap = options.checkCostCap ?? true; - const { start, end } = currentUtcDayWindow(); - if (checkRunCap && policy.maxDailyRuns !== null) { - const conditions = [ - eq(heartbeatRuns.companyId, agent.companyId), - eq(heartbeatRuns.agentId, agent.id), - gte(heartbeatRuns.startedAt, start), - lt(heartbeatRuns.startedAt, end), - notInArray(heartbeatRuns.status, ["queued", "scheduled_retry"]), - ]; - if (options.excludeRunId) { - conditions.push(sql`${heartbeatRuns.id} <> ${options.excludeRunId}`); - } - const [row] = await client - .select({ total: sql`count(*)::integer` }) - .from(heartbeatRuns) - .where(and(...conditions)); - const observed = Number(row?.total ?? 0); - if (observed >= policy.maxDailyRuns) { - return { - reason: "heartbeat.daily_run_limit", - observed, - limit: policy.maxDailyRuns, - }; - } - } - - if (checkCostCap && policy.maxDailyCostCents !== null) { - const [row] = await client - .select({ - total: sql`coalesce(sum(${costEvents.costCents}), 0)::text`, - }) - .from(costEvents) - .where( - and( - eq(costEvents.companyId, agent.companyId), - eq(costEvents.agentId, agent.id), - gte(costEvents.occurredAt, start), - lt(costEvents.occurredAt, end), - ), - ); - const observed = Number(row?.total ?? 0); - if (compareCents(String(row?.total ?? 0), policy.maxDailyCostCents) >= 0) { - return { - reason: "heartbeat.daily_cost_limit", - observed, - limit: policy.maxDailyCostCents, - }; - } - } - - return null; - } - - async function cancelQueuedRunForHeartbeatDailyCap( - run: typeof heartbeatRuns.$inferSelect, - dailyCapBlock: NonNullable< - Awaited> - >, - ) { - const now = new Date(); - const reason = - "Cancelled because the agent reached a per-day heartbeat budget cap before adapter invocation"; - const cancelled = await setRunStatus(run.id, "cancelled", { - finishedAt: now, - error: reason, - errorCode: dailyCapBlock.reason, - resultJson: { - ...parseObject(run.resultJson), - stopReason: dailyCapBlock.reason, - observed: dailyCapBlock.observed, - limit: dailyCapBlock.limit, - effectiveTimeoutSec: 0, - timeoutConfigured: false, - timeoutSource: "heartbeat_daily_cap_gate", - timeoutFired: false, - }, - }); - if (!cancelled) return null; - - await setWakeupStatus(run.wakeupRequestId, "skipped", { - finishedAt: now, - error: reason, - }); - - await appendRunEvent(cancelled, { - eventType: "lifecycle", - stream: "system", - level: "warn", - message: reason, - payload: { - reason: dailyCapBlock.reason, - observed: dailyCapBlock.observed, - limit: dailyCapBlock.limit, - }, - }); - - await releaseIssueExecutionAndPromote(cancelled, { - suppressImmediateRecovery: true, - }); - - return cancelled; - } - - async function hasActionableTimerWork(agent: typeof agents.$inferSelect) { - const row = await db - .select({ id: issues.id }) - .from(issues) - .where( - and( - eq(issues.companyId, agent.companyId), - eq(issues.assigneeAgentId, agent.id), - isNull(issues.assigneeUserId), - isNull(issues.hiddenAt), - inArray(issues.status, [...TIMER_ACTIONABLE_ISSUE_STATUSES]), - isNull(issues.conversationAgentId), - nonIdleSlackIssueCondition(), - ), - ) - .limit(1) - .then((rows) => rows[0] ?? null); - return Boolean(row); - } - - async function markTimerHeartbeatChecked( - agentId: string, - source: WakeupOptions["source"], - ) { - if (source !== "timer") return; - await db - .update(agents) - .set({ - lastHeartbeatAt: new Date(), - updatedAt: new Date(), - }) - .where(eq(agents.id, agentId)); - } - - async function claimDueTimerHeartbeat( - agent: typeof agents.$inferSelect, - now: Date, - intervalSec: number, - ) { - const dueBefore = new Date(now.getTime() - intervalSec * 1000); - const claimed = await db - .update(agents) - .set({ - lastHeartbeatAt: now, - updatedAt: now, - }) - .where( - and( - eq(agents.id, agent.id), - eq(agents.companyId, agent.companyId), - or( - lte(agents.lastHeartbeatAt, dueBefore), - and( - isNull(agents.lastHeartbeatAt), - lte(agents.createdAt, dueBefore), - ), - ), - ), - ) - .returning({ id: agents.id }) - .then((rows) => rows[0] ?? null); - if (!claimed) return null; - return { wasFirstHeartbeat: !agent.lastHeartbeatAt }; - } - function parseMaxTurnContinuationPolicy( agent: typeof agents.$inferSelect, ): MaxTurnContinuationPolicy { @@ -6144,1090 +5832,6 @@ export function heartbeatService( }; } - function issueRunPriorityRank(priority: string | null | undefined) { - switch (priority) { - case "critical": - return 0; - case "high": - return 1; - case "medium": - return 2; - case "low": - return 3; - default: - return 4; - } - } - - async function listQueuedRunDependencyReadiness( - companyId: string, - queuedRuns: Array, - ) { - const issueIds = [ - ...new Set( - queuedRuns - .map((run) => - readNonEmptyString(parseObject(run.contextSnapshot).issueId), - ) - .filter((issueId): issueId is string => Boolean(issueId)), - ), - ]; - if (issueIds.length === 0) { - return new Map< - string, - Awaited> - >(); - } - return issuesSvc.listDependencyReadiness(companyId, issueIds); - } - - async function countRunningRunsForAgent(agentId: string) { - const [{ count }] = await db - .select({ count: sql`count(*)` }) - .from(heartbeatRuns) - .where( - and( - eq(heartbeatRuns.agentId, agentId), - eq(heartbeatRuns.status, "running"), - ), - ); - return Number(count ?? 0); - } - - async function withChatControlRecoveryGate( - run: typeof heartbeatRuns.$inferSelect, - stage: "claim" | "dispatch", - onClear: (tx: Db) => Promise, - nativeRecovery = false, - ) { - const issueId = readNonEmptyString( - parseObject(run.contextSnapshot).issueId, - ); - if (!issueId || run.invocationSource !== "automation") return onClear(db); - await options.beforeChatControlRecoveryCheck?.({ - runId: run.id, - issueId, - stage, - }); - if (stage === "dispatch" && nativeRecovery) { - // Protected historical/already-admitted owners did not previously take - // this fence's locks. A fresh exact read keeps unrelated Board contention - // from turning their recovery into a new admission failure. - const [current] = await db - .select() - .from(heartbeatRuns) - .where( - and( - eq(heartbeatRuns.id, run.id), - eq(heartbeatRuns.companyId, run.companyId), - eq(heartbeatRuns.agentId, run.agentId), - eq(heartbeatRuns.status, "running"), - ), - ) - .limit(1); - if ( - current && - current.wakeupRequestId === run.wakeupRequestId && - parseObject(current.contextSnapshot).issueId === issueId - ) { - const admission = readChatControlRecoveryAdmission(current); - const expected = readChatControlRecoveryAdmission(run); - if ( - expected !== "invalid" && - (admission === "admitted" || - (admission === "historical" && expected === "historical")) - ) - return onClear(db); - } - } - let terminal: typeof heartbeatRuns.$inferSelect | null = null; - try { - const attempt = () => db.transaction(async (tx) => { - terminal = null; - // Same queue-edit lock order, then the close committer's conversation - // row. NOWAIT releases partial locks on contention. Claim defers to the - // queue; dispatch retries this transaction before considering failure. - const [issue] = await tx - .select({ id: issues.id }) - .from(issues) - .where( - and(eq(issues.id, issueId), eq(issues.companyId, run.companyId)), - ) - .for("update", { noWait: true }) - .limit(1); - if (!issue) return null; - if (run.wakeupRequestId) - await tx - .select({ id: agentWakeupRequests.id }) - .from(agentWakeupRequests) - .where( - and( - eq(agentWakeupRequests.id, run.wakeupRequestId), - eq(agentWakeupRequests.companyId, run.companyId), - eq(agentWakeupRequests.agentId, run.agentId), - ), - ) - .for("update", { noWait: true }) - .limit(1); - const [current] = await tx - .select() - .from(heartbeatRuns) - .where( - and( - eq(heartbeatRuns.id, run.id), - eq(heartbeatRuns.companyId, run.companyId), - eq(heartbeatRuns.agentId, run.agentId), - ), - ) - .for("update", { noWait: true }) - .limit(1); - if ( - !current || - current.status !== (stage === "claim" ? "queued" : "running") - ) - return null; - const admission = readChatControlRecoveryAdmission(current); - const expectedAdmission = readChatControlRecoveryAdmission(run); - const admissionLost = - expectedAdmission !== "historical" && admission === "historical"; - if ( - stage === "dispatch" && - nativeRecovery && - !admissionLost && - (admission === "historical" || admission === "admitted") - ) - return onClear(tx as unknown as Db); - // A new warm child can inherit a prior runner's PID during preparation. - // Those fields do not prove this run's turn was dispatched. - // Exact admitted/historical recovery retains its existing ownership; - // a marked, unadmitted bootstrap must earn admission even after crash. - const proof = - admission === "invalid" || admissionLost - ? { kind: "unresolved" as const } - : await readChatControlRecoveryStop( - tx as unknown as Db, - { - companyId: run.companyId, - issueId, - agentId: run.agentId, - sourceRunId: run.id, - }, - true, - ); - if (proof.kind === "clear") { - if (stage === "claim" || admission === "required") - await tx - .update(heartbeatRuns) - .set({ - runnerProfileJson: { - ...parseObject(current.runnerProfileJson), - [CHAT_CONTROL_RECOVERY_ADMISSION_KEY]: - chatControlRecoveryAdmission( - current, - stage === "dispatch" - ? "admitted" - : admission === "admitted" - ? "admitted" - : "required", - ), - }, - updatedAt: new Date(), - }) - .where(eq(heartbeatRuns.id, current.id)); - return onClear(tx as unknown as Db); - } - if (proof.kind === "unresolved" && stage === "claim") return null; - const now = new Date(); - const stopped = proof.kind === "stopped"; - const code = stopped - ? CHAT_CONTROL_RECOVERY_STOP_CODE - : CHAT_CONTROL_RECOVERY_UNRESOLVED_CODE; - const error = stopped - ? "Automatic continuation stopped by the committed chat conversation close. Send a new request in chat or on the Board to start fresh work." - : "Automatic continuation source could not be verified before provider admission. Review the task and send a fresh request; this attempt will not automatically retry."; - [terminal] = await tx - .update(heartbeatRuns) - .set({ - status: stopped ? "cancelled" : "failed", - errorCode: code, - error, - finishedAt: now, - resultJson: { - ...parseObject(current.resultJson), - automaticRecovery: { - code, - providerDispatched: false, - ...(stopped - ? { - sourceRunId: proof.sourceRunId, - conversationId: proof.conversationId, - publicationId: proof.publicationId, - } - : {}), - }, - }, - updatedAt: now, - }) - .where( - and( - eq(heartbeatRuns.id, current.id), - eq(heartbeatRuns.status, current.status), - ), - ) - .returning(); - if (!terminal) return null; - if (current.wakeupRequestId) - await tx - .update(agentWakeupRequests) - .set({ - status: stopped ? "skipped" : "failed", - error, - finishedAt: now, - updatedAt: now, - }) - .where( - and( - eq(agentWakeupRequests.id, current.wakeupRequestId), - eq(agentWakeupRequests.companyId, current.companyId), - eq(agentWakeupRequests.agentId, current.agentId), - eq(agentWakeupRequests.runId, current.id), - ne(agentWakeupRequests.status, "cancelled"), - ), - ); - await tx - .update(issues) - .set({ - executionRunId: null, - executionAgentNameKey: null, - executionLockedAt: null, - updatedAt: now, - }) - .where( - and( - eq(issues.companyId, current.companyId), - eq(issues.id, issueId), - eq(issues.executionRunId, current.id), - ), - ); - return null; - }); - const result = stage === "dispatch" - ? await retryChatControlAdmission(attempt) - : await attempt(); - if (terminal) { - const settled = terminal as typeof heartbeatRuns.$inferSelect; - publishLiveEvent({ - companyId: settled.companyId, - type: "heartbeat.run.status", - payload: { - runId: settled.id, - agentId: settled.agentId, - status: settled.status, - errorCode: settled.errorCode, - error: settled.error, - }, - }); - publishRunLifecyclePluginEvent(settled); - if (stage === "dispatch") - await finalizeAgentStatus(settled.agentId, "cancelled"); - } - return result; - } catch (error) { - if (!isExternalChatWaitAuthorizationContention(error)) throw error; - if (stage === "dispatch") { - // No effect was admitted. Let the existing setup-failure path settle - // this attempt distinctly; it must not become a successful close. - throw new ChatControlRecoveryUnresolvedError(); - } - return null; - } - } - - async function claimQueuedRun( - run: typeof heartbeatRuns.$inferSelect, - companyAgents?: AgentOrgRow[], - ) { - if (run.status !== "queued") return run; - const agent = await getAgent(run.agentId); - if (!agent) { - await cancelRunInternal( - run.id, - "Cancelled because the agent no longer exists", - ); - return null; - } - const invokability = companyAgents - ? evaluateAgentInvokability(toAgentOrgRow(agent), companyAgents) - : await getAgentInvokability(agent); - if (!invokability.invokable) { - await cancelRunInternal( - run.id, - `Cancelled because the agent is not invokable: ${invokability.reason}`, - ); - return null; - } - - const context = parseObject(run.contextSnapshot); - const budgetBlock = await budgets.getInvocationBlock( - run.companyId, - run.agentId, - { - issueId: readNonEmptyString(context.issueId), - projectId: readNonEmptyString(context.projectId), - }, - ); - if (budgetBlock) { - await cancelRunInternal(run.id, budgetBlock.reason); - return null; - } - - const dailyCapBlock = await getHeartbeatDailyCapBlock( - agent, - parseHeartbeatPolicy(agent), - { - excludeRunId: run.id, - checkRunCap: true, - checkCostCap: true, - }, - ); - if (dailyCapBlock) { - await cancelQueuedRunForHeartbeatDailyCap(run, dailyCapBlock); - return null; - } - - const issueId = readNonEmptyString(context.issueId); - if (issueId && activeRunExecutions.size > 0) { - // Native finalization publishes success before workspace synchronization, - // provider suspension, and lease release finish. A queued comment must - // not acquire a fresh sandbox while its predecessor still owns that work. - // The executor's finally block retries this agent after removing its owner. - const [settlingOwner] = await db.select({ id: heartbeatRuns.id }) - .from(heartbeatRuns) - .where(and( - eq(heartbeatRuns.companyId, run.companyId), - eq(heartbeatRuns.agentId, run.agentId), - eq(heartbeatRuns.runtimeMode, "native"), - inArray(heartbeatRuns.id, [...activeRunExecutions]), - sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issueId}`, - )) - .limit(1); - if (settlingOwner) return null; - } - if (issueId) { - const activePauseHold = await treeControlSvc.getActivePauseHoldGate( - run.companyId, - issueId, - ); - const treeHoldInteractionWake = - activePauseHold && - (await isVerifiedIssueTreeControlInteractionWake(db, { - companyId: run.companyId, - issueId, - agentId: run.agentId, - runId: run.id, - wakeupRequestId: run.wakeupRequestId, - contextSnapshot: context, - })); - if (activePauseHold && !treeHoldInteractionWake) { - await cancelRunInternal( - run.id, - "Cancelled because issue is held by an active subtree pause hold", - ); - await logActivity(db, { - companyId: run.companyId, - actorType: "system", - actorId: "system", - agentId: run.agentId, - runId: run.id, - action: "issue.tree_hold_run_interrupted", - entityType: "heartbeat_run", - entityId: run.id, - issueId: issueId, - details: { - issueId, - holdId: activePauseHold.holdId, - rootIssueId: activePauseHold.rootIssueId, - source: "heartbeat.claim_queued_run", - securityPrinciples: [ - "Complete Mediation", - "Fail Securely", - "Secure Defaults", - ], - }, - }); - return null; - } - - const dependencyReadiness = await issuesSvc.listDependencyReadiness( - run.companyId, - [issueId], - ); - const readiness = dependencyReadiness.get(issueId); - const unresolvedBlockerCount = readiness?.unresolvedBlockerCount ?? 0; - if ( - unresolvedBlockerCount > 0 && - !allowsIssueInteractionWake( - context, - ISSUE_TREE_CONTROL_INTERACTION_WAKE_REASONS, - ) - ) { - await cancelQueuedRunForBlockedDependencies( - run, - issueId, - readiness?.unresolvedBlockerIssueIds ?? [], - ); - logger.info( - { runId: run.id, issueId, unresolvedBlockerCount }, - "claimQueuedRun: cancelled blocked queued run", - ); - return null; - } - - const staleness = await runDispatch.cancelStaleQueuedRun({ - runId: run.id, - companyId: run.companyId, - expectedStatus: "queued", - }); - if (staleness.outcome === "cancelled" || staleness.outcome === "deferred") { - applyRunDispatchPostCommitEffects(staleness.postCommitEffects); - logger.info( - { - runId: run.id, - issueId, - outcome: staleness.outcome, - errorCode: staleness.outcome === "cancelled" ? staleness.errorCode : undefined, - }, - "claimQueuedRun: withheld queued run at the execution gate", - ); - return null; - } - } - - const claimedAt = new Date(); - const responsibleUserId = await resolveResponsibleUserIdForRun({ - run, - contextSnapshot: context, - issueContext: issueId - ? await getIssueExecutionContext(run.companyId, issueId) - : null, - routineEnvContext: { - routineId: null, - env: null, - responsibleUserId: null, - }, - }); - // All ordinary and comment claims use the same company-scoped issue - // lock. A batch may claim several runs before executeRun tracks any owner. - async function lockIssueExecutionClaim(tx: Db) { - const [owner] = issueId ? await tx.select({ - assigneeAgentId: issues.assigneeAgentId, - executionRunId: issues.executionRunId, - checkoutRunId: issues.checkoutRunId, - }).from(issues).where(and( - eq(issues.id, issueId), eq(issues.companyId, run.companyId), - )).for("update") : []; - const ownsIssue = owner?.assigneeAgentId === run.agentId && - context.wakeReason !== "source_scoped_recovery_action"; - if (ownsIssue && ["native_safe_replacement", "native_provider_overloaded"].includes(run.scheduledRetryReason ?? "") && - owner.checkoutRunId && owner.checkoutRunId !== run.id) { - return { ownsIssue, blocked: true }; - } - if (run.scheduledRetryReason === "native_provider_overloaded" && owner?.executionRunId && - owner.executionRunId !== run.id && owner.executionRunId !== run.retryOfRunId) { - return { ownsIssue, blocked: true }; - } - const previousRunId = run.scheduledRetryReason === "native_provider_overloaded" - ? run.retryOfRunId : ownsIssue ? owner?.executionRunId : null; - if (previousRunId && previousRunId !== run.id) { - const [previous] = await tx.select({ status: heartbeatRuns.status }) - .from(heartbeatRuns).where(and( - eq(heartbeatRuns.id, previousRunId), - eq(heartbeatRuns.companyId, run.companyId), - )); - // A terminal result can precede workspace/lease cleanup on this or - // another controller. Local absence alone is not a release receipt. - if (!isHeartbeatRunTerminalStatus(previous?.status) || - liveRunExecutions.has(previousRunId)) { - return { ownsIssue, blocked: true }; - } - const [pendingLease] = await tx.select({ id: environmentLeases.id }) - .from(environmentLeases).where(and( - eq(environmentLeases.companyId, run.companyId), - eq(environmentLeases.heartbeatRunId, previousRunId), - or(and(isNull(environmentLeases.releasedAt), - // Warm release deliberately retains the sandbox. Its successful - // receipt settles the old run without destroying the resource. - sql`not coalesce(${environmentLeases.status} = 'retained' - and ${environmentLeases.leasePolicy} = 'reuse_by_environment' - and ${environmentLeases.cleanupStatus} = 'success', false)`), - eq(environmentLeases.status, "pending_cleanup"), - eq(environmentLeases.cleanupStatus, "failed")), - )).limit(1); - const [finalization] = await tx.select({ - phase: nativeRunFinalizations.phase, leaseOwner: nativeRunFinalizations.leaseOwner, - }).from(nativeRunFinalizations).where(and( - eq(nativeRunFinalizations.companyId, run.companyId), - eq(nativeRunFinalizations.runId, previousRunId), - )); - if (pendingLease || (finalization && (finalization.leaseOwner || - !["committed", "applied", "terminal_failure"].includes(finalization.phase)))) { - return { ownsIssue, blocked: true }; - } - } - return { ownsIssue, blocked: false }; - } - async function bindClaimedIssueExecution(tx: Db, ownsIssue: boolean, claimedRun: typeof heartbeatRuns.$inferSelect | null | undefined) { - if (!claimedRun || !issueId || !ownsIssue) return; - await tx.update(issues).set({ - executionRunId: claimedRun.id, - executionAgentNameKey: normalizeAgentNameKey(agent.name), - executionLockedAt: claimedAt, - updatedAt: claimedAt, - }).where(and(eq(issues.id, issueId), eq(issues.companyId, run.companyId))); - } - const nativeReviewContext = readNativeReviewAssignmentContext(context); - const queuedCommentIds = queuedCommentIdsFromRunContext(context); - if ( - issueId && - run.invocationSource === "automation" && - queuedCommentIds.length > 0 - ) - await options.beforeChatControlRecoveryCheck?.({ - runId: run.id, - issueId, - stage: "claim", - }); - const queuedCommentClaim = - !nativeReviewContext && issueId && run.wakeupRequestId && queuedCommentIds.length > 0 - ? await db - .transaction(async (tx) => { - // Match the queue-edit lock order: issue, wake, then run. Once the - // run becomes running, a concurrent discard must observe the - // claimed wake and return an explicit conflict; if discard wins, - // this claim observes the cancelled queue and does no work. - const issueClaim = await lockIssueExecutionClaim(tx as unknown as Db); - if (issueClaim.blocked) return { kind: "stale" as const, run: null }; - const wake = await tx - .select() - .from(agentWakeupRequests) - .where( - and( - eq(agentWakeupRequests.id, run.wakeupRequestId!), - eq(agentWakeupRequests.companyId, run.companyId), - eq(agentWakeupRequests.agentId, run.agentId), - ), - ) - .for("update") - .limit(1) - .then((rows) => rows[0] ?? null); - const lockedRun = await tx - .select() - .from(heartbeatRuns) - .where( - and( - eq(heartbeatRuns.id, run.id), - eq(heartbeatRuns.companyId, run.companyId), - eq(heartbeatRuns.agentId, run.agentId), - ), - ) - .for("update") - .limit(1) - .then((rows) => rows[0] ?? null); - if ( - !wake || - wake.status !== "queued" || - wake.runId !== run.id || - !lockedRun || - lockedRun.status !== "queued" || - lockedRun.wakeupRequestId !== wake.id - ) { - return { kind: "stale" as const, run: null }; - } - - if (lockedRun.invocationSource === "automation") { - const admission = readChatControlRecoveryAdmission(lockedRun); - if (admission === "invalid") - return { kind: "stale" as const, run: null }; - const proof = await readChatControlRecoveryStop( - tx as unknown as Db, - { - companyId: lockedRun.companyId, - issueId, - agentId: lockedRun.agentId, - sourceRunId: lockedRun.id, - }, - true, - ); - if (proof.kind === "unresolved") - return { kind: "stale" as const, run: null }; - if (proof.kind === "stopped") { - const error = - "Automatic continuation stopped by the committed chat conversation close. Send a fresh request in chat or on the Board."; - const [cancelled] = await tx - .update(heartbeatRuns) - .set({ - status: "cancelled", - errorCode: CHAT_CONTROL_RECOVERY_STOP_CODE, - error, - finishedAt: claimedAt, - resultJson: { - ...parseObject(lockedRun.resultJson), - automaticRecovery: { - code: CHAT_CONTROL_RECOVERY_STOP_CODE, - providerDispatched: false, - sourceRunId: proof.sourceRunId, - conversationId: proof.conversationId, - publicationId: proof.publicationId, - }, - }, - updatedAt: claimedAt, - }) - .where( - and( - eq(heartbeatRuns.id, lockedRun.id), - eq(heartbeatRuns.status, "queued"), - ), - ) - .returning(); - if (!cancelled) return { kind: "stale" as const, run: null }; - await tx - .update(agentWakeupRequests) - .set({ - status: "skipped", - error, - finishedAt: claimedAt, - updatedAt: claimedAt, - }) - .where(eq(agentWakeupRequests.id, wake.id)); - await tx - .update(issues) - .set({ - executionRunId: null, - executionAgentNameKey: null, - executionLockedAt: null, - updatedAt: claimedAt, - }) - .where( - and( - eq(issues.id, issueId), - eq(issues.companyId, run.companyId), - eq(issues.executionRunId, run.id), - ), - ); - return { kind: "cancelled" as const, run: cancelled }; - } - await tx - .update(heartbeatRuns) - .set({ - runnerProfileJson: { - ...parseObject(lockedRun.runnerProfileJson), - [CHAT_CONTROL_RECOVERY_ADMISSION_KEY]: - chatControlRecoveryAdmission( - lockedRun, - admission === "admitted" ? "admitted" : "required", - ), - }, - updatedAt: claimedAt, - }) - .where(eq(heartbeatRuns.id, lockedRun.id)); - } - const authoritativeIds = queuedCommentIdsFromWakePayload( - wake.payload, - ); - if (authoritativeIds.length === 0) { - // Legacy/direct comment wakes carry comment ids in their ordinary - // payload and context, not in the authoritative queued-message - // envelope. Preserve their established claim path; only an - // explicitly bound queued-message envelope is subject to the - // live-comment discard gate below. - const [claimedRun] = await tx - .update(heartbeatRuns) - .set({ - status: "running", - runnerProfileJson: sql`(case when jsonb_typeof(${heartbeatRuns.runnerProfileJson}) = 'object' then ${heartbeatRuns.runnerProfileJson} else '{}'::jsonb end) || ${JSON.stringify({ adapterDispatch: { adapterType: agent.adapterType } })}::jsonb`, - ...legacyControllerClaim(run.runtimeMode), - responsibleUserId, - startedAt: lockedRun.startedAt ?? claimedAt, - updatedAt: claimedAt, - }) - .where( - and( - eq(heartbeatRuns.id, lockedRun.id), - eq(heartbeatRuns.status, "queued"), - ), - ) - .returning(); - await bindClaimedIssueExecution(tx as unknown as Db, issueClaim.ownsIssue, claimedRun); - return claimedRun - ? { kind: "claimed" as const, run: claimedRun } - : { kind: "stale" as const, run: null }; - } - const commentRows = await tx - .select({ - id: issueComments.id, - deletedAt: issueComments.deletedAt, - }) - .from(issueComments) - .where( - and( - eq(issueComments.companyId, run.companyId), - eq(issueComments.issueId, issueId), - inArray(issueComments.id, authoritativeIds), - ), - ); - const liveIds = authoritativeIds.filter((commentId) => { - const comment = commentRows.find((row) => row.id === commentId); - return Boolean(comment && !comment.deletedAt); - }); - if (liveIds.length === 0) { - const reason = "Queued messages were discarded before dispatch"; - const [cancelled] = await tx - .update(heartbeatRuns) - .set({ - status: "cancelled", - finishedAt: claimedAt, - error: reason, - errorCode: "queued_comment_discarded", - updatedAt: claimedAt, - }) - .where( - and( - eq(heartbeatRuns.id, lockedRun.id), - eq(heartbeatRuns.status, "queued"), - ), - ) - .returning(); - await tx - .update(agentWakeupRequests) - .set({ - status: "cancelled", - finishedAt: claimedAt, - error: reason, - updatedAt: claimedAt, - }) - .where(eq(agentWakeupRequests.id, wake.id)); - await tx - .update(issues) - .set({ - executionRunId: null, - executionAgentNameKey: null, - executionLockedAt: null, - updatedAt: claimedAt, - }) - .where( - and( - eq(issues.id, issueId), - eq(issues.companyId, run.companyId), - eq(issues.executionRunId, run.id), - ), - ); - return { - kind: "cancelled" as const, - run: cancelled ?? lockedRun, - }; - } - - await tx - .update(agentWakeupRequests) - .set({ - status: "claimed", - claimedAt, - payload: withQueuedCommentIdsInWakePayload( - wake.payload, - liveIds, - ), - updatedAt: claimedAt, - }) - .where(eq(agentWakeupRequests.id, wake.id)); - const [claimedRun] = await tx - .update(heartbeatRuns) - .set({ - status: "running", - runnerProfileJson: sql`(case when jsonb_typeof(${heartbeatRuns.runnerProfileJson}) = 'object' then ${heartbeatRuns.runnerProfileJson} else '{}'::jsonb end) || ${JSON.stringify({ adapterDispatch: { adapterType: agent.adapterType } })}::jsonb`, - ...legacyControllerClaim(run.runtimeMode), - responsibleUserId, - startedAt: lockedRun.startedAt ?? claimedAt, - contextSnapshot: withQueuedCommentIdsInRunContext( - lockedRun.contextSnapshot, - liveIds, - ), - updatedAt: claimedAt, - }) - .where( - and( - eq(heartbeatRuns.id, lockedRun.id), - eq(heartbeatRuns.status, "queued"), - ), - ) - .returning(); - await bindClaimedIssueExecution(tx as unknown as Db, issueClaim.ownsIssue, claimedRun); - return claimedRun - ? { kind: "claimed" as const, run: claimedRun } - : { kind: "stale" as const, run: null }; - }) - .catch((error) => { - if (isExternalChatWaitAuthorizationContention(error)) - return { kind: "stale" as const, run: null }; - throw error; - }) - : null; - if (queuedCommentClaim?.kind === "cancelled") { - await appendRunEvent(queuedCommentClaim.run, { - eventType: "lifecycle", - stream: "system", - level: "warn", - message: - queuedCommentClaim.run.error ?? - "Queued messages were discarded before dispatch", - }); - publishLiveEvent({ - companyId: queuedCommentClaim.run.companyId, - type: "heartbeat.run.status", - payload: { - runId: queuedCommentClaim.run.id, - agentId: queuedCommentClaim.run.agentId, - status: queuedCommentClaim.run.status, - invocationSource: queuedCommentClaim.run.invocationSource, - triggerDetail: queuedCommentClaim.run.triggerDetail, - error: queuedCommentClaim.run.error ?? null, - errorCode: queuedCommentClaim.run.errorCode ?? null, - startedAt: queuedCommentClaim.run.startedAt - ? new Date(queuedCommentClaim.run.startedAt).toISOString() - : null, - finishedAt: queuedCommentClaim.run.finishedAt - ? new Date(queuedCommentClaim.run.finishedAt).toISOString() - : null, - }, - }); - publishRunLifecyclePluginEvent(queuedCommentClaim.run); - // Fire-and-forget: nothing else in this path depends on the emission, - // so it must not delay the return. - void emitAgentTaskRun(db, queuedCommentClaim.run); - return null; - } - const claimed = queuedCommentClaim - ? queuedCommentClaim.run - : await withChatControlRecoveryGate(run, "claim", async (tx) => { - const claimValues = { - status: "running", - runnerProfileJson: sql`(case when jsonb_typeof(${heartbeatRuns.runnerProfileJson}) = 'object' then ${heartbeatRuns.runnerProfileJson} else '{}'::jsonb end) || ${JSON.stringify({ adapterDispatch: { adapterType: agent.adapterType } })}::jsonb`, - ...legacyControllerClaim(run.runtimeMode), - responsibleUserId, - startedAt: run.startedAt ?? claimedAt, - updatedAt: claimedAt, - }; - if (nativeReviewContext) { - if (run.scheduledRetryReason === "native_provider_overloaded") { - const predecessor = await lockIssueExecutionClaim(tx); - if (predecessor.blocked) return null; - } - return claimQueuedNativeReviewRun(tx, { - run, claimedAt, claimValues, - agentNameKey: normalizeAgentNameKey(agent.name), - }); - } - return tx.transaction(async (claimTx) => { - const issueClaim = await lockIssueExecutionClaim(claimTx as unknown as Db); - if (issueClaim.blocked) return null; - const claimedRun = await claimTx.update(heartbeatRuns).set(claimValues).where(and( - eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.status, "queued"), - )).returning().then((rows) => rows[0] ?? null); - await bindClaimedIssueExecution(claimTx as unknown as Db, issueClaim.ownsIssue, claimedRun); - return claimedRun; - }); - }); - if (!claimed) return null; - - publishLiveEvent({ - companyId: claimed.companyId, - type: "heartbeat.run.status", - payload: { - runId: claimed.id, - agentId: claimed.agentId, - status: claimed.status, - invocationSource: claimed.invocationSource, - triggerDetail: claimed.triggerDetail, - error: claimed.error ?? null, - errorCode: claimed.errorCode ?? null, - startedAt: claimed.startedAt - ? new Date(claimed.startedAt).toISOString() - : null, - finishedAt: claimed.finishedAt - ? new Date(claimed.finishedAt).toISOString() - : null, - }, - }); - publishRunLifecyclePluginEvent(claimed); - - if (!nativeReviewContext) { - await setWakeupStatus(claimed.wakeupRequestId, "claimed", { claimedAt }); - } - - // Fix A (lazy locking): stamp executionRunId now that the run is actually running, - // not at queue time. Guard is idempotent — safe if called more than once. - const claimedContext = parseObject(claimed.contextSnapshot); - const claimedIssueId = readNonEmptyString(claimedContext.issueId); - const claimedWakeReason = readNonEmptyString(claimedContext.wakeReason); - if ( - !nativeReviewContext && claimedIssueId && - claimedWakeReason !== "source_scoped_recovery_action" - ) { - const claimedAgent = await getAgent(claimed.agentId); - await db - .update(issues) - .set({ - executionRunId: claimed.id, - executionAgentNameKey: normalizeAgentNameKey(claimedAgent?.name), - executionLockedAt: claimedAt, - updatedAt: claimedAt, - }) - .where( - and( - eq(issues.id, claimedIssueId), - eq(issues.companyId, claimed.companyId), - // Mention/context runs can touch an issue, but only the current assignee - // owns the issue execution lock shown as the active run. - eq(issues.assigneeAgentId, claimed.agentId), - ["native_safe_replacement", "native_provider_overloaded"].includes(claimed.scheduledRetryReason ?? "") - ? or( - isNull(issues.checkoutRunId), - eq(issues.checkoutRunId, claimed.id), - ) - : undefined, - or( - isNull(issues.executionRunId), - eq(issues.executionRunId, claimed.id), - ), - ), - ); - } - - return claimed; - } - - // startNextQueuedRunForAgent checks admission suppression once, then claims - // runs (sets status "running"), then dispatches each to executeRun, which - // checks suppression again before it does any work. Suppression (task - // drain, worktree mode, a database restore) can start in the gap between - // those two checks. When executeRun's check catches that, the run is - // already claimed — release it back to "queued" so it does not keep a - // running execution lock that nothing will ever process. This runs inside - // the same promise startNextQueuedRunForAgent already tracks in - // activeRunExecutionPromises, so getTaskDrainStatus() keeps reporting the - // run as active until the release finishes. - // - // The run row, the wakeup request, and the issue execution lock all guard - // the same claim, so one transaction commits all three writes together. A - // partial write (for example the run flips to "queued" but the wakeup or - // issue update then fails) would let the execution promise clear from - // activeRunExecutionPromises while the wakeup stayed "claimed" or the - // issue stayed locked to a queued run — task-drain status would then read - // quiescent while the database still held part of the old claim. - async function releaseRunClaimedJustBeforeSuppression(runId: string) { - const now = new Date(); - await db.transaction(async (tx) => { - const released = await tx - .update(heartbeatRuns) - .set({ - status: "queued", - startedAt: null, - responsibleUserId: null, - updatedAt: now, - }) - .where( - and(eq(heartbeatRuns.id, runId), eq(heartbeatRuns.status, "running")), - ) - .returning() - .then((rows) => rows[0] ?? null); - if (!released) return; - - if (released.wakeupRequestId) { - await tx - .update(agentWakeupRequests) - .set({ status: "queued", claimedAt: null, updatedAt: now }) - .where(and(eq(agentWakeupRequests.id, released.wakeupRequestId), ne(agentWakeupRequests.status, "cancelled"))); - } - - const context = parseObject(released.contextSnapshot); - const issueId = readNonEmptyString(context.issueId); - if (issueId) { - await tx - .update(issues) - .set({ - executionRunId: null, - executionAgentNameKey: null, - executionLockedAt: null, - updatedAt: now, - }) - .where( - and( - eq(issues.id, issueId), - eq(issues.companyId, released.companyId), - eq(issues.executionRunId, released.id), - ), - ); - } - }); - } - - async function cancelQueuedRunForBlockedDependencies( - run: typeof heartbeatRuns.$inferSelect, - issueId: string, - unresolvedBlockerIssueIds: string[], - ) { - const now = new Date(); - const reason = - "Cancelled because issue dependencies are still blocked; Paperclip will wake the assignee when blockers resolve"; - const cancelled = await setRunStatus(run.id, "cancelled", { - finishedAt: now, - error: reason, - errorCode: "issue_dependencies_blocked", - resultJson: { - ...parseObject(run.resultJson), - stopReason: "issue_dependencies_blocked", - effectiveTimeoutSec: 0, - timeoutConfigured: false, - timeoutSource: "dependency_gate", - timeoutFired: false, - }, - }); - if (!cancelled) return null; - - await setWakeupStatus(run.wakeupRequestId, "skipped", { - finishedAt: now, - error: reason, - }); - - await db - .update(issues) - .set({ - executionRunId: null, - executionAgentNameKey: null, - executionLockedAt: null, - updatedAt: now, - }) - .where( - and( - eq(issues.companyId, run.companyId), - eq(issues.id, issueId), - eq(issues.executionRunId, run.id), - ), - ); - - await appendRunEvent(cancelled, { - eventType: "lifecycle", - stream: "system", - level: "warn", - message: reason, - payload: { - issueId, - unresolvedBlockerIssueIds, - }, - }); - - return cancelled; - } - function truncateAgentErrorReason( reason: string | null | undefined, ): string | null { @@ -7567,110 +6171,6 @@ export function heartbeatService( .then((rows) => rows[0] ?? null); } - async function resumeQueuedRuns() { - if ((await getSchedulingSuppression()).suppressed) return; - await resumeExecutionWaitComments(); - const cutoff = await getWorktreeExecutionCutoff(); - const pendingInterrupts = await db.select({ id: agentWakeupRequests.id, companyId: agentWakeupRequests.companyId }) - .from(agentWakeupRequests).innerJoin(companies, eq(companies.id, agentWakeupRequests.companyId)) - .where(and(eq(agentWakeupRequests.status, "deferred_issue_execution"), - eq(companies.status, "active"), - sql`${agentWakeupRequests.payload}->'queuedCommentInterrupt' is not null`, - lte(agentWakeupRequests.updatedAt, new Date(Date.now() - 30_000)), - cutoff ? gte(agentWakeupRequests.requestedAt, cutoff) : undefined)) - .orderBy(asc(agentWakeupRequests.updatedAt)).limit(50); - for (const wake of pendingInterrupts) { - await db.update(agentWakeupRequests).set({ updatedAt: new Date() }).where(and( - eq(agentWakeupRequests.id, wake.id), eq(agentWakeupRequests.status, "deferred_issue_execution"), - )); - await resumeQueuedCommentInterrupt(wake.companyId, wake.id).catch(err => { - logger.warn({ err, queueId: wake.id }, "failed to resume interrupted comment queue"); - }); - } - // A server restart or a message/cleanup race can leave a deferred wake - // after its owner has released the issue lock. Revisit it through the same - // release admission, so recovery holds and operator Stops still apply. - const strandedQueues = await db.select({ wake: agentWakeupRequests }) - .from(agentWakeupRequests) - .innerJoin(issues, and(eq(issues.companyId, agentWakeupRequests.companyId), - sql`${issues.id}::text = ${agentWakeupRequests.payload}->>'issueId'`, - eq(issues.assigneeAgentId, agentWakeupRequests.agentId))) - .innerJoin(companies, and(eq(companies.id, issues.companyId), eq(companies.status, "active"))) - .where(and(eq(agentWakeupRequests.status, "deferred_issue_execution"), - isNull(issues.executionRunId), - or(and( - sql`jsonb_typeof(${agentWakeupRequests.payload} #> '{_paperclipWakeContext,wakeCommentIds}') = 'array'`, - sql`${agentWakeupRequests.payload} #> '{_paperclipWakeContext,wakeCommentIds}' <> '[]'::jsonb`, - ), sql`${agentWakeupRequests.payload}->>'mutation' = 'interaction'`), - sql`${agentWakeupRequests.payload}->'queuedCommentInterrupt' is null`, - cutoff ? gte(agentWakeupRequests.requestedAt, cutoff) : undefined)) - .orderBy(asc(agentWakeupRequests.updatedAt)).limit(50); - for (const { wake } of strandedQueues) { - if (!queuedCommentIdsFromWakePayload(wake.payload).length && - !await readQueuedInteractionResponse(db, wake.companyId, String(wake.payload?.issueId), wake.payload)) continue; - const [latest] = await db.select().from(heartbeatRuns).where(and( - eq(heartbeatRuns.companyId, wake.companyId), eq(heartbeatRuns.agentId, wake.agentId), - sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${String(wake.payload?.issueId)}`, - )).orderBy(desc(heartbeatRuns.createdAt), desc(heartbeatRuns.id)).limit(1); - await db.update(agentWakeupRequests).set({ updatedAt: new Date() }).where(and( - eq(agentWakeupRequests.id, wake.id), eq(agentWakeupRequests.status, "deferred_issue_execution"), - )); - if (!latest || latest.runtimeMode !== "legacy" || !isHeartbeatRunTerminalStatus(latest.status)) continue; - const cancelledAdmission = latest.status === "cancelled" && !latest.startedAt && - latest.errorCode === "execution_reconciliation_required"; - if ((latest.status !== "cancelled" || cancelledAdmission) && await getExecutionBlocker(db, wake.companyId, String(wake.payload?.issueId))) { - await resumeSavedLegacyComments(wake.companyId, wake.id).catch(err => { - logger.warn({ err, queueId: wake.id }, "failed to deliver saved legacy comment after recovery stopped"); - }); - } - await releaseIssueExecutionAndPromote(latest, { suppressImmediateRecovery: true }).catch(err => { - logger.warn({ err, queueId: wake.id }, "failed to promote stranded legacy comments"); - }); - } - - // The cancellation marker is durable intent. Retry while its exact queue - // is still deferred, including after a failed cleanup promotion or restart. - // Normal admission still checks process ownership, leases, pauses, and scope. - const interruptedQueues = await db - .select({ id: heartbeatRuns.id, companyId: heartbeatRuns.companyId }) - .from(agentWakeupRequests) - .innerJoin(heartbeatRuns, and( - sql`${heartbeatRuns.resultJson}->>'queuedCommentInterruptQueueId' = ${agentWakeupRequests.id}::text`, - eq(heartbeatRuns.companyId, agentWakeupRequests.companyId), - eq(heartbeatRuns.agentId, agentWakeupRequests.agentId), - )) - .innerJoin(companies, eq(companies.id, heartbeatRuns.companyId)) - .where(and( - eq(agentWakeupRequests.status, "deferred_issue_execution"), - eq(heartbeatRuns.status, "cancelled"), - eq(heartbeatRuns.runtimeMode, "legacy"), - eq(companies.status, "active"), - cutoff ? gte(heartbeatRuns.createdAt, cutoff) : undefined, - )); - for (const run of interruptedQueues) { - await releaseIssueExecutionAndPromote(run, { suppressImmediateRecovery: true }).catch((err) => { - logger.error({ err, runId: run.id }, "failed to retry interrupted comment queue"); - }); - } - - const queuedRuns = await db - .select({ agentId: heartbeatRuns.agentId }) - .from(heartbeatRuns) - .innerJoin(companies, eq(companies.id, heartbeatRuns.companyId)) - .where( - and( - eq(heartbeatRuns.status, "queued"), - eq(companies.status, "active"), - cutoff ? gte(heartbeatRuns.createdAt, cutoff) : undefined, - ), - ); - - const agentIds = [...new Set(queuedRuns.map((r) => r.agentId))]; - for (const agentId of agentIds) { - await startNextQueuedRunForAgent(agentId); - } - } - async function recoverActiveSessionGoals() { if ((await getSchedulingSuppression()).suppressed) { return { scanned: 0, enqueued: 0 }; @@ -7927,203 +6427,6 @@ export function heartbeatService( } - // A 403 from claimQueuedRun comes from the run's own persisted identity - // (an unverifiable interrupt receipt, a manual wake with no user). Those rows - // do not change, so the claim fails the same way on every pass and restart. - function isPermanentClaimRejection(err: unknown): err is HttpError { - return err instanceof HttpError && err.status === 403; - } - - // Other 4xx rejections can clear later (a responsible user gets assigned, a - // conflicting claim finishes). Keep the run queued, but do not let it stop - // the rest of the queue or startup recovery. - function isDeferrableClaimRejection(err: unknown): err is HttpError { - return err instanceof HttpError && err.status >= 400 && err.status < 500; - } - - // Settle runs that can never be claimed. Letting the error escape stalls the - // agent's queue and, during startup recovery, stops the server from booting. - async function cancelRejectedQueuedRuns( - rejected: Array<{ run: typeof heartbeatRuns.$inferSelect; err: HttpError }>, - ) { - for (const { run, err } of rejected) { - logger.warn( - { err, runId: run.id, agentId: run.agentId, companyId: run.companyId }, - "cancelling queued heartbeat run whose claim was rejected", - ); - try { - await cancelRunInternal( - run.id, - `Cancelled because the queued run cannot be claimed: ${err.message}`, - { errorCode: "queued_run_claim_rejected" }, - ); - } catch (cancelErr) { - logger.error( - { err: cancelErr, runId: run.id }, - "failed to cancel queued heartbeat run whose claim was rejected; it stays queued for the next recovery pass", - ); - } - } - } - - async function startNextQueuedRunForAgent(agentId: string) { - if ((await getSchedulingSuppression()).suppressed) return []; - const cutoff = await getWorktreeExecutionCutoff(); - // Cancelled after the start lock is released: cancelRunInternal promotes the - // agent's next queued run, which takes this same lock. - const rejectedClaims: Array<{ run: typeof heartbeatRuns.$inferSelect; err: HttpError }> = []; - - return withAgentStartLock(agentId, async () => { - const agent = await getAgent(agentId); - if (!agent) return []; - const invokability = await getAgentInvokability(agent); - if (!invokability.invokable) { - if (shouldCancelRunsForNonInvokableAgent(invokability)) { - await cancelActiveForAgentInternal( - agentId, - `Cancelled because the agent is not invokable: ${invokability.reason}`, - ); - } - return []; - } - const policy = parseHeartbeatPolicy(agent); - const runningCount = await countRunningRunsForAgent(agentId); - const availableSlots = Math.max( - 0, - policy.maxConcurrentRuns - runningCount, - ); - if (availableSlots <= 0) return []; - - const queuedRuns = await db - .select() - .from(heartbeatRuns) - .where( - and( - eq(heartbeatRuns.agentId, agentId), - eq(heartbeatRuns.status, "queued"), - cutoff ? gte(heartbeatRuns.createdAt, cutoff) : undefined, - ), - ) - .orderBy(asc(heartbeatRuns.createdAt)); - if (queuedRuns.length === 0) return []; - - const dependencyReadiness = await listQueuedRunDependencyReadiness( - agent.companyId, - queuedRuns, - ); - const queuedIssueIds = [ - ...new Set( - queuedRuns - .map((run) => - readNonEmptyString(parseObject(run.contextSnapshot).issueId), - ) - .filter((issueId): issueId is string => Boolean(issueId)), - ), - ]; - const issueRows = await db - .select({ - id: issues.id, - status: issues.status, - priority: issues.priority, - }) - .from(issues) - .where( - queuedIssueIds.length > 0 - ? and( - eq(issues.companyId, agent.companyId), - inArray(issues.id, queuedIssueIds), - ) - : sql`false`, - ); - const issueById = new Map(issueRows.map((row) => [row.id, row])); - const companyAgents = await listCompanyAgentOrgRows(agent.companyId); - const prioritizedRuns = [...queuedRuns].sort((left, right) => { - const leftIssueId = readNonEmptyString( - parseObject(left.contextSnapshot).issueId, - ); - const rightIssueId = readNonEmptyString( - parseObject(right.contextSnapshot).issueId, - ); - const leftReadiness = leftIssueId - ? dependencyReadiness.get(leftIssueId) - : null; - const rightReadiness = rightIssueId - ? dependencyReadiness.get(rightIssueId) - : null; - const leftReady = leftIssueId - ? (leftReadiness?.isDependencyReady ?? true) - : true; - const rightReady = rightIssueId - ? (rightReadiness?.isDependencyReady ?? true) - : true; - const leftIssue = leftIssueId ? issueById.get(leftIssueId) : null; - const rightIssue = rightIssueId ? issueById.get(rightIssueId) : null; - const leftRank = leftIssueId - ? leftReady - ? leftIssue?.status === "in_progress" - ? 0 - : 1 - : 3 - : 2; - const rightRank = rightIssueId - ? rightReady - ? rightIssue?.status === "in_progress" - ? 0 - : 1 - : 3 - : 2; - if (leftRank !== rightRank) return leftRank - rightRank; - const leftPriorityRank = issueRunPriorityRank(leftIssue?.priority); - const rightPriorityRank = issueRunPriorityRank(rightIssue?.priority); - if (leftPriorityRank !== rightPriorityRank) - return leftPriorityRank - rightPriorityRank; - return left.createdAt.getTime() - right.createdAt.getTime(); - }); - - const claimedRuns: Array = []; - for (const queuedRun of prioritizedRuns) { - if (claimedRuns.length >= availableSlots) break; - let claimed: typeof heartbeatRuns.$inferSelect | null; - try { - claimed = await claimQueuedRun(queuedRun, companyAgents); - } catch (err) { - if (isPermanentClaimRejection(err)) { - rejectedClaims.push({ run: queuedRun, err }); - continue; - } - if (!isDeferrableClaimRejection(err)) throw err; - logger.warn( - { err, runId: queuedRun.id, agentId: queuedRun.agentId, companyId: queuedRun.companyId }, - "queued heartbeat run claim was rejected; leaving it queued for the next recovery pass", - ); - continue; - } - if (claimed) claimedRuns.push(claimed); - } - if (claimedRuns.length === 0) return []; - - for (const claimedRun of claimedRuns) { - const execution = executeRun(claimedRun.id).catch((err) => { - logger.error( - { err, runId: claimedRun.id }, - "queued heartbeat execution failed", - ); - }); - // Register the in-flight execution so drainActiveRunExecutions() can await - // it. executeRun resolves only after its finally block finishes flushing - // run rows/events, so awaiting this promise guarantees the run's writes - // have landed before a caller (e.g. a test's afterEach) mutates the DB. - activeRunExecutionPromises.add(execution); - void execution.finally(() => { - // drainActiveRunExecutions loops on activeRunExecutionPromises.size, - // so an entry that never clears here would hang it forever. - activeRunExecutionPromises.delete(execution); - }); - } - return claimedRuns; - }).finally(() => cancelRejectedQueuedRuns(rejectedClaims)); - } - // Await every background heartbeat execution that is currently in flight. A // draining run can, in its finally block, promote and dispatch the next queued // run for the same agent — that follow-up execution is registered in the set @@ -8194,25 +6497,6 @@ export function heartbeatService( nativeSessionResumeDispatchTimers.set(runId, timer); } - // Public wakeup entry point. Callers dispatch it fire-and-forget, so register - // the promise in activeWakeupPromises before it starts its asynchronous - // prologue. drainActiveRunExecutions can then await a wake that is still before - // run registration. Internal callers reference enqueueWakeup directly and - // already await it, so they do not need this registration. - function trackWakeup( - agentId: string, - opts: WakeupOptions = {}, - ): ReturnType { - const promise = enqueueWakeup(agentId, opts); - activeWakeupPromises.add(promise); - void promise - .catch(() => {}) - .finally(() => { - activeWakeupPromises.delete(promise); - }); - return promise; - } - async function executeRun( runId: string, runOptions: { @@ -15013,2533 +13297,6 @@ export function heartbeatService( } } - async function enqueueWakeup(agentId: string, opts: WakeupOptions = {}, executionWaitRequestId?: string) { - const source = opts.source ?? "on_demand"; - const triggerDetail = opts.triggerDetail ?? null; - const contextSnapshot: Record = { - ...(opts.contextSnapshot ?? {}), - }; - const reason = opts.reason ?? null; - let payload = opts.payload ? { ...opts.payload } : null; - // Only the board queue route can record interruption authority on an - // existing receipt. Never accept this internal marker from a wake caller. - if (payload) { - delete payload.queuedCommentInterrupt; - delete payload.manualUserWake; - } - if (opts.manualUserWake) { - if (opts.requestedByActorType !== "user" || !opts.requestedByActorId || opts.failedRunId) { - throw new HttpError(403, "Manual wake requires an authenticated user"); - } - payload = { ...payload, manualUserWake: true }; - } - const executionReconciliationWake = - contextSnapshot.source === "execution.reconciled" || - opts.idempotencyKey?.startsWith("execution-reconciliation:") === true; - const { - contextSnapshot: enrichedContextSnapshot, - issueIdFromPayload, - taskKey, - wakeCommentId, - } = enrichWakeContextSnapshot({ - contextSnapshot, - reason, - source, - triggerDetail, - payload, - }); - // Keep each request's own server-derived origin, including coalesced wakes. - // A run's merged context cannot establish which caller authored one receipt. - // Overwrite caller-supplied nested context rather than trusting it. - payload = { ...payload, [DEFERRED_WAKE_CONTEXT_KEY]: { ...enrichedContextSnapshot } }; - let issueId = - readNonEmptyString(enrichedContextSnapshot.issueId) ?? issueIdFromPayload; - if (executionReconciliationWake && !issueId) return null; - - let agent = await getAgent(agentId); - if (!agent) throw notFound("Agent not found"); - // Mentions only annotate comments. Ignore legacy callers before creating - // a run or deferred request; assignment and review have their own wakes. - if (reason === "issue_comment_mentioned" || enrichedContextSnapshot.wakeReason === "issue_comment_mentioned") return null; - if (issueId) { - const conversation = await getIssueExecutionContext(agent.companyId, issueId); - if (reason === "issue_children_completed" && conversation?.originKind === "onboarding_first_task") enrichedContextSnapshot.onboardingCompletion = true; - if (isConversation(conversation)) { - if (opts.manualUserWake && conversation!.conversationUserId !== opts.requestedByActorId) { - throw new HttpError(403, "Only the conversation owner can start a chat run"); - } - if (isConversationExecutionWake(conversation, reason ?? readNonEmptyString(enrichedContextSnapshot.wakeReason))) return null; - if (agent.id !== conversation!.conversationAgentId) return null; - if (!(await instanceSettings.getExperimental()).enableAgentChat) return null; - if (!wakeCommentId && isWaitingConversation(conversation) && !hasInteractionContinuationWakeContext(enrichedContextSnapshot) && reason !== CHAT_COMPLETION_WAKE_REASON) return null; - } - } - if (agent.adapterType === "paperclip_runner") { - const oldConfig = parseObject(agent.adapterConfig); - const nextConfig = normalizeLegacyRunnerProvider(oldConfig); - if (nextConfig !== oldConfig) { - await agentService(db).update( - agent.id, - { adapterConfig: nextConfig }, - { - recordRevision: { - source: "normalize_runner_provider", - createdByAgentId: null, - createdByUserId: null, - }, - }, - ); - await logActivity(db, { - companyId: agent.companyId, - actorType: "system", - actorId: "heartbeat", - action: "agent.updated", - entityType: "agent", - entityId: agent.id, - details: { provider: "codex", reason: "native_codex_provider" }, - }); - agent = (await getAgent(agentId))!; - } - } - - if (opts.failedRunId) { - const failed = await getRun(opts.failedRunId, { includeExecutionEvidence: true }); - if (opts.requestedByActorType !== "user" || !opts.requestedByActorId || - reason !== "retry_failed_run" || source !== "on_demand" || triggerDetail !== "manual" || - !failed || failed.companyId !== agent.companyId || failed.agentId !== agentId || - !(await canRetryStoppedRun(db, failed)) || - (failed.nativeIssueId ?? readNonEmptyString(failed.contextSnapshot?.issueId)) !== issueId) { - throw conflict("The selected failed run cannot be retried for this task."); - } - if (!activeRunExecutions.has(failed.id) && !adapterExecutionControls.has(failed.id)) { - await sweepPendingCleanupLeases({ explicitRetry: { - companyId: failed.companyId, runId: failed.id, actorId: opts.requestedByActorId, - } }); - } - if (isConversationAdapter(agent.adapterType) || agent.adapterType === "paperclip_runner") { - enrichedContextSnapshot.previousRunId = failed.id; - enrichedContextSnapshot.forceFreshSession = true; - } - } - - const durableRequest = opts.durableChatRequest; - if (durableRequest) { - assertDurableChatWakeupRequest(durableRequest, { - agentId, - companyId: agent.companyId, - issueId, - commentId: wakeCommentId ?? null, - requestedByActorType: opts.requestedByActorType, - requestedByActorId: opts.requestedByActorId, - }); - opts = { ...opts, idempotencyKey: durableRequest.idempotencyKey }; - } - if ( - Object.hasOwn(enrichedContextSnapshot, "chatFailedRunRetry") && - !durableRequest?.failedRunRetry - ) { - throw new FailedChatRunRetryAuthorizationError(); - } - if (durableRequest?.failedRunRetry) { - opts = { ...opts, allowRunCoalescing: false }; - } - const dotRequest = opts.durableDotRequest; - if (dotRequest && (durableRequest || dotRequest.agentId !== agentId || - dotRequest.companyId !== agent.companyId || dotRequest.issueId !== issueId || - dotRequest.requestId !== payload?.dotRequestId || source !== "assignment" || - opts.requestedByActorType !== "agent" || opts.requestedByActorId !== agentId || - agent.adapterType !== "paperclip_runner" || parseObject(agent.adapterConfig).provider !== "openai_dot" || - opts.idempotencyKey !== dotRequest.idempotencyKey)) { - throw conflict("Dot work request does not match its admission authority."); - } - const receiptRequest = durableRequest ?? dotRequest; - const durableReceiptFields = receiptRequest - ? { id: receiptRequest.id, requestedAt: receiptRequest.requestedAt } - : {}; - const existingDurableReceipt = async (queryDb: Db) => { - if (!receiptRequest) return null; - const receipt = await queryDb - .select() - .from(agentWakeupRequests) - .where(eq(agentWakeupRequests.id, receiptRequest.id)) - .limit(1) - .then((rows) => rows[0] ?? null); - if (receipt && durableRequest) assertDurableChatWakeupReceipt(durableRequest, receipt); - if (receipt && dotRequest && (receipt.companyId !== dotRequest.companyId || - receipt.agentId !== dotRequest.agentId || receipt.source !== "assignment" || - receipt.requestedByActorType !== "agent" || receipt.requestedByActorId !== dotRequest.agentId || - receipt.idempotencyKey !== dotRequest.idempotencyKey || receipt.payload?.issueId !== dotRequest.issueId || - receipt.payload?.dotRequestId !== dotRequest.requestId)) { - throw conflict("requestId was reused for another task."); - } - return receipt; - }; - const priorReceipt = await existingDurableReceipt(db); - if (priorReceipt) { - // Replaying an admission receipt is not fresh authority to dispatch it. - // The normal queue owns dispatch and its current execution-policy checks. - return priorReceipt.runId ? getRun(priorReceipt.runId) : null; - } - - const agentDebug = parseObject(parseObject(agent.runtimeConfig).debug); - const runDebug = parseObject(enrichedContextSnapshot.debug); - if ( - agentDebug.providerTrace === "raw" && - runDebug.providerTrace !== "raw" - ) { - enrichedContextSnapshot.debug = { - ...runDebug, - providerTrace: "raw", - }; - enrichedContextSnapshot.providerTraceRequestedBy = `agent:${agent.id}:debug-setting`; - enrichedContextSnapshot.providerTraceRequestSource = - "agent_debug_setting"; - } - - // Automatic signals are replaceable; user input and interaction delivery - // retain distinct durable receipts even when the same gate blocks them. - const coalesceExecutionWait = - opts.requestedByActorType === "system" && - !receiptRequest && - !wakeCommentId && - queuedCommentIdsFromRunContext(enrichedContextSnapshot).length === 0 && - !isInteractionResolutionWakePayload(payload ?? {}) && - !hasInteractionContinuationWakeContext(enrichedContextSnapshot); - const writeSkippedRequest = async ( - skipReason: string, - patch: Partial = {}, - waitCondition?: Record, - ) => { - if (executionWaitRequestId) { - await db.update(agentWakeupRequests).set({ - payload: sql`jsonb_set(coalesce(${agentWakeupRequests.payload}, '{}'::jsonb), '{executionWait}', - coalesce(${agentWakeupRequests.payload}->'executionWait', '{}'::jsonb) || ${JSON.stringify({ - reason: skipReason, message: patch.error ?? (skipReason === "issue_tree_hold_active" - ? "This task is paused. Resume it to send your saved message." - : "Waiting for task execution to be enabled. Your message is saved."), - })}::jsonb)`, - updatedAt: new Date(), - }).where(and(eq(agentWakeupRequests.id, executionWaitRequestId), - eq(agentWakeupRequests.companyId, agent.companyId), eq(agentWakeupRequests.agentId, agentId), - eq(agentWakeupRequests.status, "deferred_issue_execution"))); - return { created: false }; - } - const request = { - ...durableReceiptFields, - companyId: agent.companyId, - agentId, - source, - triggerDetail, - reason: skipReason, - payload, - status: "skipped", - requestedByActorType: opts.requestedByActorType ?? null, - requestedByActorId: opts.requestedByActorId ?? null, - idempotencyKey: opts.idempotencyKey ?? null, - finishedAt: new Date(), - ...patch, - }; - if (waitCondition && issueId && isUuidLike(issueId)) { - const waitIssueId = issueId; - return db.transaction(async (tx) => { - await tx.execute(sql`select id from issues where id = ${waitIssueId} and company_id = ${agent.companyId} for update`); - return recordExecutionWait(tx as unknown as Db, { - issueId: waitIssueId, request, condition: waitCondition, coalesce: coalesceExecutionWait, - }); - }); - } - await db.insert(agentWakeupRequests).values(request); - return { created: true }; - }; - const writeSkippedHeartbeatRequest = async ( - skipReason: string, - details: Record, - ) => { - await writeSkippedRequest(skipReason, { - payload: { - ...(payload ?? {}), - heartbeatSkip: details, - }, - }); - }; - - const schedulingSuppression = await getSchedulingSuppression(); - // A task drain holds ADMISSION, not the request. The drain is a - // process-local pre-restart hold, so a wake that arrives while it is - // active still names real work that must run once the process comes - // back: leave it in the durable queue and let the dispatch-side checks - // (startNextQueuedRunForAgent / executeRun) keep it from starting until - // the drain lifts or the restart clears it. Writing it as `skipped` - // here dropped the wake permanently — an accepted plan whose - // continuation wake landed mid-drain left its issue in `todo` with no - // run and no path until a person noticed. - if ( - schedulingSuppression.suppressed && - schedulingSuppression.reason !== "task_drain" - ) { - await writeSkippedHeartbeatRequest("heartbeat.scheduling_suppressed", { - reason: schedulingSuppression.reason, - }); - return null; - } - - const worktreeExecutionCutoff = - opts.requestedByActorType === "user" - ? null - : await getWorktreeExecutionCutoff(); - - const company = await db - .select({ status: companies.status }) - .from(companies) - .where(eq(companies.id, agent.companyId)) - .then((rows) => rows[0] ?? null); - - if (!company || company.status !== "active") { - const companyStatus = company?.status ?? "missing"; - if (opts.requestedByActorType === "user") { - throw conflict("Company is not active", { status: companyStatus }); - } - await writeSkippedRequest("company.inactive", { - error: `Wake suppressed because company status is ${companyStatus}`, - }, { companyStatus }); - return null; - } - - const explicitResumeSession = await resolveExplicitResumeSessionOverride( - agent, - payload, - taskKey, - ); - if (explicitResumeSession) { - enrichedContextSnapshot.resumeFromRunId = - explicitResumeSession.resumeFromRunId; - enrichedContextSnapshot.resumeSessionDisplayId = - explicitResumeSession.sessionDisplayId; - enrichedContextSnapshot.resumeSessionParams = - explicitResumeSession.sessionParams; - if ( - !readNonEmptyString(enrichedContextSnapshot.issueId) && - explicitResumeSession.issueId - ) { - enrichedContextSnapshot.issueId = explicitResumeSession.issueId; - } - if ( - !readNonEmptyString(enrichedContextSnapshot.taskId) && - explicitResumeSession.taskId - ) { - enrichedContextSnapshot.taskId = explicitResumeSession.taskId; - } - if ( - !readNonEmptyString(enrichedContextSnapshot.taskKey) && - explicitResumeSession.taskKey - ) { - enrichedContextSnapshot.taskKey = explicitResumeSession.taskKey; - } - issueId = readNonEmptyString(enrichedContextSnapshot.issueId) ?? issueId; - } - const effectiveTaskKey = - readNonEmptyString(enrichedContextSnapshot.taskKey) ?? taskKey; - const sessionBefore = - explicitResumeSession?.sessionDisplayId ?? - (await resolveSessionBeforeForWakeup(agent, effectiveTaskKey)); - let hasResolvablePriorSessionWorkspace: boolean | null = null; - const resolveHasResolvablePriorSessionWorkspace = async () => { - if (hasResolvablePriorSessionWorkspace !== null) - return hasResolvablePriorSessionWorkspace; - hasResolvablePriorSessionWorkspace = issueId - ? await hasResolvablePriorSessionWorkspaceForWake({ - agent, - contextSnapshot: enrichedContextSnapshot, - taskKey: effectiveTaskKey, - explicitResumeSession, - }) - : false; - return hasResolvablePriorSessionWorkspace; - }; - const continuationAttempt = readContinuationAttempt( - enrichedContextSnapshot.livenessContinuationAttempt, - ); - - let projectId = readNonEmptyString(enrichedContextSnapshot.projectId); - if (!projectId && issueId) { - // Look up by either UUID or identifier (e.g. "ENV-13"), but always scope - // by companyId so a row from another tenant can never be returned even - // when identifiers collide across companies. Guard the UUID arm because - // issues.id is a Postgres uuid column — passing "ENV-13" into eq(issues.id, …) - // would fail with an invalid-input-syntax cast error before the OR is - // evaluated. - const lookupIsUuid = isUuidLike(issueId); - const idMatch = lookupIsUuid - ? or( - eq(issues.id, issueId), - eq(issues.identifier, issueId.toUpperCase()), - ) - : eq(issues.identifier, issueId.toUpperCase()); - const resolvedIssue = await db - .select({ - id: issues.id, - projectId: issues.projectId, - createdAt: issues.createdAt, - }) - .from(issues) - .where(and(eq(issues.companyId, agent.companyId), idMatch)) - .then((rows) => rows[0] ?? null); - if (resolvedIssue) { - if ( - worktreeExecutionCutoff && - resolvedIssue.createdAt < worktreeExecutionCutoff - ) { - await writeSkippedHeartbeatRequest( - "heartbeat.worktree_execution_cutoff", - { - reason: "worktree_execution_cutoff", - cutoff: worktreeExecutionCutoff.toISOString(), - issueId: resolvedIssue.id, - }, - ); - return null; - } - projectId = resolvedIssue.projectId ?? null; - // Canonicalize context to the UUID so downstream lookups always use UUID - if (resolvedIssue.id !== issueId) { - issueId = resolvedIssue.id; - enrichedContextSnapshot.issueId = issueId; - if (readNonEmptyString(enrichedContextSnapshot.taskId)) { - enrichedContextSnapshot.taskId = issueId; - } - } - } - } - // Propagate projectId into context so resolveWorkspaceForRun can bind the - // project workspace even when context.projectId wasn't set by the caller. - if (projectId && !readNonEmptyString(enrichedContextSnapshot.projectId)) { - enrichedContextSnapshot.projectId = projectId; - } - const isolatedWorkspacesEnabled = issueId - ? (await instanceSettings.getExperimental()).enableIsolatedWorkspaces - : false; - let operatorResponsibleUserId: string | null = opts.manualUserWake ? opts.requestedByActorId! : null; - let queuedResponsibleUserIdPromise: Promise | null = null; - const resolveQueuedResponsibleUserId = () => { - if (operatorResponsibleUserId) return Promise.resolve(operatorResponsibleUserId); - queuedResponsibleUserIdPromise ??= (async () => { - const queuedIssueContext = issueId - ? await getIssueExecutionContext(agent.companyId, issueId) - : null; - const queuedRoutineEnvContext = await getRoutineEnvForExecutionIssue( - agent.companyId, - queuedIssueContext, - ); - const queuedResponsibleUserId = - await resolveResponsibleUserIdForRunSeed({ - companyId: agent.companyId, - contextSnapshot: enrichedContextSnapshot, - issueContext: queuedIssueContext, - routineEnvContext: queuedRoutineEnvContext, - requestedByActorType: opts.requestedByActorType ?? null, - requestedByActorId: opts.requestedByActorId ?? null, - source, - triggerDetail, - }); - if (!queuedResponsibleUserId) { - throw new HttpError( - 422, - "Unable to resolve responsible user for heartbeat run dispatch", - { - code: "responsible_user_unresolved", - agentId, - companyId: agent.companyId, - issueId: issueId ?? null, - source, - triggerDetail, - wakeReason: readNonEmptyString( - enrichedContextSnapshot.wakeReason, - ), - }, - ); - } - return queuedResponsibleUserId; - })(); - return queuedResponsibleUserIdPromise; - }; - - const budgetBlock = await budgets.getInvocationBlock( - agent.companyId, - agentId, - { - issueId, - projectId, - }, - ); - if (budgetBlock) { - await writeSkippedRequest("budget.blocked", { error: budgetBlock.reason }, { - scopeType: budgetBlock.scopeType, scopeId: budgetBlock.scopeId, - }); - throw conflict(budgetBlock.reason, { - scopeType: budgetBlock.scopeType, - scopeId: budgetBlock.scopeId, - }); - } - - const invokability = await getAgentInvokability(agent); - if (!invokability.invokable) { - if (opts.requestedByActorType !== "user" || executionWaitRequestId) { - await writeSkippedRequest("agent.not_invokable", { - error: invokability.message, - }, { status: agent.status, reason: invokability.reason }); - } - throw conflict(invokability.message, { - status: agent.status, - reason: invokability.reason, - invalidOrgChain: invokability.invalidOrgChain, - ...invokability.details, - }); - } - - const policy = parseHeartbeatPolicy(agent); - - if (source === "timer" && !policy.enabled) { - await writeSkippedRequest("heartbeat.disabled", {}, { enabled: false }); - return null; - } - if (source !== "timer" && !policy.wakeOnDemand) { - await writeSkippedRequest("heartbeat.wakeOnDemand.disabled", {}, { wakeOnDemand: false }); - return null; - } - - const genericTimerWake = - source === "timer" && - !issueId && - !wakeCommentId && - !readNonEmptyString(enrichedContextSnapshot.taskId) && - !readNonEmptyString(enrichedContextSnapshot.taskKey); - if ( - policy.skipTimerWhenNoActionableWork && - genericTimerWake && - !(await hasActionableTimerWork(agent)) - ) { - await writeSkippedHeartbeatRequest("heartbeat.timer.no_actionable_work", { - reason: - "No assigned todo or in_progress issue requires this agent before timer adapter invocation.", - }); - await markTimerHeartbeatChecked(agentId, source); - return null; - } - - if (issueId) { - const activePauseHold = await treeControlSvc.getActivePauseHoldGate( - agent.companyId, - issueId, - ); - if (activePauseHold) { - const treeHoldInteractionWake = - await isVerifiedIssueTreeControlInteractionWake(db, { - companyId: agent.companyId, - issueId, - agentId, - contextSnapshot: enrichedContextSnapshot, - requestedByActorType: opts.requestedByActorType, - requestedByActorId: opts.requestedByActorId, - }); - - if (!treeHoldInteractionWake) { - const wait = await writeSkippedRequest("issue_tree_hold_active", {}, { - holdId: activePauseHold.holdId, - }); - if (wait.created) await logActivity(db, { - companyId: agent.companyId, - actorType: "system", - actorId: "system", - agentId, - runId: null, - action: "issue.tree_hold_wakeup_deferred", - entityType: "issue", - entityId: issueId, - details: { - holdId: activePauseHold.holdId, - rootIssueId: activePauseHold.rootIssueId, - requestedReason: reason, - source, - triggerDetail, - securityPrinciples: [ - "Complete Mediation", - "Fail Securely", - "Secure Defaults", - ], - }, - }); - return null; - } - - enrichedContextSnapshot.treeHoldInteraction = true; - enrichedContextSnapshot.activeTreeHold = { - holdId: activePauseHold.holdId, - rootIssueId: activePauseHold.rootIssueId, - mode: activePauseHold.mode, - reason: activePauseHold.reason, - releasePolicy: activePauseHold.releasePolicy, - interaction: true, - }; - } - } - - if (issueId) { - // Mention-triggered wakes can request input from another agent, but they must - // still respect the issue execution lock so a second agent cannot start on the - // same issue workspace while the assignee already has a live run. - const agentNameKey = normalizeAgentNameKey(agent.name); - - const cancelledRunsToEmit: (typeof heartbeatRuns.$inferSelect)[] = []; - - const outcome = await db - .transaction(async (tx) => { - await tx.execute( - sql`select id from issues where id = ${issueId} and company_id = ${agent.companyId} for update`, - ); - - if (executionWaitRequestId) { - const [pending] = await tx.select().from(agentWakeupRequests).where(and( - eq(agentWakeupRequests.id, executionWaitRequestId), eq(agentWakeupRequests.companyId, agent.companyId), - eq(agentWakeupRequests.agentId, agentId), eq(agentWakeupRequests.status, "deferred_issue_execution"), - // A user message can join a queue originally created by a - // system wake. Admission validates the saved user comment or board click. - (opts.queuedCommentInterruptId ?? opts.queuedCommentRequestId) === executionWaitRequestId - ? undefined : eq(agentWakeupRequests.requestedByActorType, "user"), - opts.queuedCommentInterruptId === executionWaitRequestId - ? sql`${agentWakeupRequests.payload}->'queuedCommentInterrupt'->>'actorId' = ${opts.requestedByActorId ?? ""}` - : opts.queuedCommentRequestId === executionWaitRequestId ? undefined - : eq(agentWakeupRequests.requestedByActorId, opts.requestedByActorId ?? ""), - sql`${agentWakeupRequests.payload}->>'issueId' = ${issueId}`, - )); - // The issue lock serializes cleanup callbacks and periodic workers. - // An adopted, discarded, or edited receipt is no longer authority. - if (!pending || (!(wakeCommentId && queuedCommentIdsFromWakePayload(pending.payload).includes(wakeCommentId)) && - !(opts.queuedCommentInterruptId && await readQueuedInteractionResponse(tx as unknown as Db, - agent.companyId, issueId, pending.payload)))) { - return { kind: "deferred" as const }; - } - if (opts.queuedCommentRequestId) { - const ids = await undeliveredLegacyUserCommentIds(tx as unknown as Db, - agent.companyId, issueId, agentId, queuedCommentIdsFromWakePayload(pending.payload)); - if (!wakeCommentId || !ids.includes(wakeCommentId)) return { kind: "deferred" as const }; - pending.payload = withQueuedCommentIdsInWakePayload(parseObject(pending.payload), ids); - await tx.update(agentWakeupRequests).set({ payload: pending.payload }).where(and( - eq(agentWakeupRequests.id, pending.id), eq(agentWakeupRequests.companyId, agent.companyId), - )); - } - if (!opts.queuedCommentInterruptId && !opts.queuedCommentRequestId && pending.payload?.manualUserWake === true) { - // A persisted manual wake keeps its actor when an execution wait - // resumes. The locked receipt above has revalidated that actor. - payload = { ...payload, manualUserWake: true }; - operatorResponsibleUserId = opts.requestedByActorId!; - } - if (opts.queuedCommentInterruptId) { - // The locked board receipt supplies execution authority even when - // another user authored the messages. Dispatch revalidates the receipt. - operatorResponsibleUserId = opts.requestedByActorId!; - } - if (opts.queuedCommentInterruptId || opts.queuedCommentRequestId) { - // Edits/discards between the click and dispatch remain authoritative. - Object.assign(enrichedContextSnapshot, withQueuedCommentIdsInRunContext( - enrichedContextSnapshot, queuedCommentIdsFromWakePayload(pending.payload), - )); - } - } - let automaticParentRunId: string | null = null; - if ( - source === "automation" && - opts.requestedByActorType === "system" - ) { - const nativeParent = await readChatControlNativeParent( - tx as unknown as Db, - { companyId: agent.companyId, issueId, agentId }, - opts.requestedByActorId ?? null, - ); - const genericParent = - !opts.requestedByActorId && reason === "issue_continuation_needed" - ? readNonEmptyString(enrichedContextSnapshot.retryOfRunId) - : null; - automaticParentRunId = - nativeParent.kind === "parent" - ? nativeParent.runId - : genericParent; - if (automaticParentRunId) { - const [parent] = await tx - .select({ - agentId: heartbeatRuns.agentId, - context: heartbeatRuns.contextSnapshot, - }) - .from(heartbeatRuns) - .where( - and( - eq(heartbeatRuns.companyId, agent.companyId), - eq(heartbeatRuns.id, automaticParentRunId), - ), - ) - .limit(1); - if ( - parent && - (parseObject(parent.context).issueId ?? - parseObject(parent.context).taskId) === issueId && - parent.agentId !== agentId - ) - automaticParentRunId = null; - } - const proof = - nativeParent.kind === "unresolved" - ? { kind: "unresolved" as const } - : automaticParentRunId - ? await readChatControlRecoveryStop( - tx as unknown as Db, - { - companyId: agent.companyId, - issueId, - agentId, - sourceRunId: automaticParentRunId, - }, - true, - ) - : { kind: "clear" as const }; - if (proof.kind !== "clear") { - await tx - .insert(agentWakeupRequests) - .values({ - companyId: agent.companyId, - agentId, - source, - triggerDetail, - reason: - proof.kind === "stopped" - ? CHAT_CONTROL_RECOVERY_STOP_CODE - : CHAT_CONTROL_RECOVERY_UNRESOLVED_CODE, - payload: { issueId }, - status: "skipped", - requestedByActorType: "system", - requestedByActorId: opts.requestedByActorId ?? null, - idempotencyKey: opts.idempotencyKey ?? null, - finishedAt: new Date(), - }); - return { kind: "skipped" as const }; - } - } - - const durableReceipt = await existingDurableReceipt( - tx as unknown as Db, - ); - if (durableReceipt) - return { kind: "durable" as const, receipt: durableReceipt }; - const failedChatRetry = await authorizeFailedChatRunRetryWake( - db, - tx as unknown as Db, - { - phase: "admission", - wakeupRequestId: durableRequest?.id ?? null, - companyId: agent.companyId, - agentId, - issueId, - contextSnapshot: enrichedContextSnapshot, - }, - ); - if ( - failedChatRetry !== Boolean(durableRequest?.failedRunRetry) || - (failedChatRetry && - durableRequest?.failedRunRetry?.failedRunId !== - readNonEmptyString(enrichedContextSnapshot.retryOfRunId)) - ) { - throw new FailedChatRunRetryAuthorizationError(); - } - if (durableRequest) { - if (issueId !== durableRequest.issueId) - throw new Error("chat_inbound_wakeup_binding_denied"); - await durableRequest.authorize(tx as unknown as Db); - } else { - if ( - source === "on_demand" && - triggerDetail === "manual" && - reason === "retry_failed_run" - ) { - // Generic Board retry does not carry server-authorized failed-run - // lineage. Caller-supplied source/comment/retry markers cannot - // restore it, including for a retired conversation generation. - const [chatBinding] = await tx - .select({ id: chatConversations.id }) - .from(chatConversations) - .innerJoin(chatEndpoints, eq(chatEndpoints.id, chatConversations.endpointId)) - .where( - and( - eq(chatEndpoints.externalExecutionPolicy, "restricted"), - eq(chatConversations.companyId, agent.companyId), - eq(chatConversations.issueId, issueId), - ), - ) - .limit(1); - if (chatBinding) { - throw conflict( - "Retry needs the exact failed chat request and current access. Send the request again in the current connected conversation.", - { code: "chat_failed_run_retry_requires_authorized_context" }, - ); - } - } - const [held] = await tx - .select({ id: issues.id }) - .from(issues) - .where( - and( - eq(issues.id, issueId), - eq(issues.companyId, agent.companyId), - unadmittedChatWakeupCondition(issues.id, issues.companyId), - ), - ) - .limit(1); - if (held) { - throw conflict( - "This task has external chat input that has not been admitted. Check the connection's Activity; let pending input finish, or restore access and send a new authorized message before starting the task.", - { code: "chat_inbound_wakeup_unadmitted", issueId }, - ); - } - } - - const issue = await tx - .select({ - id: issues.id, - companyId: issues.companyId, - identifier: issues.identifier, - conversationAgentId: issues.conversationAgentId, - conversationUserId: issues.conversationUserId, - conversationState: issues.conversationState, - status: issues.status, - statusVersion: issues.statusVersion, - projectId: issues.projectId, - projectWorkspaceId: issues.projectWorkspaceId, - executionWorkspaceId: issues.executionWorkspaceId, - executionWorkspacePreference: issues.executionWorkspacePreference, - executionWorkspaceSettings: issues.executionWorkspaceSettings, - assigneeAgentId: issues.assigneeAgentId, - assigneeUserId: issues.assigneeUserId, - monitorNextCheckAt: issues.monitorNextCheckAt, - monitorWakeRequestedAt: issues.monitorWakeRequestedAt, - executionRunId: issues.executionRunId, - executionAgentNameKey: issues.executionAgentNameKey, - createdAt: issues.createdAt, - }) - .from(issues) - .where( - and( - eq(issues.id, issueId), - eq(issues.companyId, agent.companyId), - ), - ) - .then((rows) => rows[0] ?? null); - - if (!issue) { - await tx.insert(agentWakeupRequests).values({ - ...durableReceiptFields, - companyId: agent.companyId, - agentId, - source, - triggerDetail, - reason: "issue_execution_issue_not_found", - payload, - status: "skipped", - requestedByActorType: opts.requestedByActorType ?? null, - requestedByActorId: opts.requestedByActorId ?? null, - idempotencyKey: opts.idempotencyKey ?? null, - finishedAt: new Date(), - }); - return { kind: "skipped" as const }; - } - - const issueStateGuard = opts.issueStateGuard; - const activeMonitorRun = issueStateGuard?.monitorNextCheckAt === undefined ? null - : await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and( - eq(heartbeatRuns.companyId, issue.companyId), eq(heartbeatRuns.nativeIssueId, issue.id), - eq(heartbeatRuns.runtimeMode, "native"), inArray(heartbeatRuns.status, ["queued", "running", "scheduled_retry"]), - )).limit(1).then(rows => rows[0] ?? null); - if ( - issueStateGuard && - (!issueStateGuard.statuses.includes(issue.status) || - issue.assigneeAgentId !== issueStateGuard.assigneeAgentId || - (issueStateGuard.statusVersion !== undefined && issue.statusVersion !== issueStateGuard.statusVersion) || - (issueStateGuard.monitorNextCheckAt !== undefined && ( - activeMonitorRun !== null || issue.assigneeUserId !== null || - issue.monitorNextCheckAt?.toISOString() !== issueStateGuard.monitorNextCheckAt || - issue.monitorWakeRequestedAt?.toISOString() !== issueStateGuard.monitorWakeRequestedAt - ))) - ) { - // A deferred monitor retains its schedule; do not create a receipt - // that could suppress its next admission attempt. - if (issueStateGuard.monitorNextCheckAt !== undefined) return { kind: "skipped" as const }; - await tx.insert(agentWakeupRequests).values({ - ...durableReceiptFields, - companyId: agent.companyId, - agentId, - source, - triggerDetail, - reason: "issue_state_guard_mismatch", - payload: { - ...(payload ?? {}), - heartbeatSkip: { - reason: - "Issue status or assignee changed before the wake could be queued.", - issueId: issue.id, - expectedStatuses: issueStateGuard.statuses, - actualStatus: issue.status, - expectedAssigneeAgentId: issueStateGuard.assigneeAgentId, - actualAssigneeAgentId: issue.assigneeAgentId, - }, - }, - status: "skipped", - requestedByActorType: opts.requestedByActorType ?? null, - requestedByActorId: opts.requestedByActorId ?? null, - idempotencyKey: opts.idempotencyKey ?? null, - finishedAt: new Date(), - }); - return { kind: "skipped" as const }; - } - - if (opts.failedRunId) { - // The issue lock makes double-clicks and network retries adopt the - // same successor, including after it has already finished. - const [previousRetry] = await tx.select().from(heartbeatRuns).where(and( - eq(heartbeatRuns.companyId, issue.companyId), eq(heartbeatRuns.agentId, agentId), - eq(heartbeatRuns.retryOfRunId, opts.failedRunId), - sql`${heartbeatRuns.contextSnapshot}->>'wakeReason' = 'retry_failed_run'`, - )).orderBy(desc(heartbeatRuns.createdAt)).limit(1); - if (previousRetry) return { kind: "replayed" as const, run: previousRetry }; - } - - let reconciledSourceRunId: string | null = null; - let reconciledRestoreRetryCount: number | null = null; - if (executionReconciliationWake) { - const actionId = readNonEmptyString( - enrichedContextSnapshot.recoveryActionId, - ); - if ( - !actionId || - !isUuidLike(actionId) || - source !== "automation" || - triggerDetail !== "system" || - reason !== "issue_recovery_action_restored" || - opts.requestedByActorType !== "system" || - opts.requestedByActorId !== "execution-recovery" || - opts.idempotencyKey !== `execution-reconciliation:${actionId}` || - enrichedContextSnapshot.source !== "execution.reconciled" || - enrichedContextSnapshot.forceFreshSession !== true || - payload?.issueId !== issue.id || - payload?.recoveryActionId !== actionId || - issue.assigneeAgentId !== agentId || - ["done", "cancelled"].includes(issue.status) - ) - return { kind: "skipped" as const }; - - // The issue lock serializes all admissions for this source. Validate - // the durable operator decision, then reconcile a prior queue commit - // before considering a new wake (including a now-terminal successor). - const [action] = await tx - .select() - .from(issueRecoveryActions) - .where( - and( - eq(issueRecoveryActions.companyId, issue.companyId), - eq(issueRecoveryActions.sourceIssueId, issue.id), - eq(issueRecoveryActions.id, actionId), - ), - ) - .for("update"); - const decision = parseObject( - action?.evidence.executionReconciliation, - ); - const sourceRunId = readNonEmptyString(decision.runId); - if ( - !action || - action.status !== "resolved" || - action.kind !== "active_run_watchdog" || - action.returnOwnerAgentId !== agentId || - !sourceRunId || - !isUuidLike(sourceRunId) || - decision.providerStopped !== true || - !["completed", "not_performed", "mixed"].includes( - String(decision.actionOutcome), - ) || - !readNonEmptyString(decision.outcomeEvidence) || - enrichedContextSnapshot.previousRunId !== sourceRunId || - enrichedContextSnapshot.retryOfRunId !== sourceRunId || - !["pending", "delivered"].includes( - String(action.evidence.continuationDelivery), - ) - ) - return { kind: "skipped" as const }; - - const [existingWake] = await tx - .select() - .from(agentWakeupRequests) - .where( - and( - eq(agentWakeupRequests.companyId, issue.companyId), - eq(agentWakeupRequests.agentId, agentId), - eq(agentWakeupRequests.idempotencyKey, opts.idempotencyKey), - ne(agentWakeupRequests.status, "skipped"), - ), - ) - .orderBy(asc(agentWakeupRequests.requestedAt)) - .limit(1); - if (existingWake) { - if ( - existingWake.payload?.issueId !== issue.id || - existingWake.payload?.recoveryActionId !== action.id || - existingWake.requestedByActorType !== "system" || - existingWake.requestedByActorId !== "execution-recovery" || - !existingWake.runId - ) - return { kind: "deferred" as const }; - const [existingRun] = await tx - .select() - .from(heartbeatRuns) - .where( - and( - eq(heartbeatRuns.companyId, issue.companyId), - eq(heartbeatRuns.agentId, agentId), - eq(heartbeatRuns.id, existingWake.runId), - ), - ); - if ( - !existingRun || - existingRun.contextSnapshot?.issueId !== issue.id || - existingRun.contextSnapshot?.recoveryActionId !== action.id || - existingRun.contextSnapshot?.previousRunId !== sourceRunId - ) - return { kind: "deferred" as const }; - return { kind: "replayed" as const, run: existingRun }; - } - if (action.evidence.continuationDelivery !== "pending") - return { kind: "skipped" as const }; - const [reconciledRun] = await tx.select().from(heartbeatRuns).where(and( - eq(heartbeatRuns.companyId, issue.companyId), eq(heartbeatRuns.id, sourceRunId), - )); - if (hasWorkspaceRestoreFailure(reconciledRun?.resultJson)) { - if ((readNonEmptyString(decision.workspaceRepairEvidence)?.length ?? 0) < 20) - return { kind: "skipped" as const }; - // Repair does not reset the remaining automatic retry budget. - reconciledRestoreRetryCount = executionFailureRetryCount(reconciledRun!); - } - reconciledSourceRunId = sourceRunId; - } - - let continuationWait = { reason: "execution_recovery", message: "Waiting for execution recovery. Your message is saved." }; - const deferBlockedExecution = async ( - executionBlocker: NonNullable>>, - ) => { - const condition = { recoveryActionId: executionBlocker.recoveryActionId, ...continuationWait }; - if (executionWaitRequestId) { - await tx.update(agentWakeupRequests).set({ - payload: sql`jsonb_set(coalesce(${agentWakeupRequests.payload}, '{}'::jsonb), '{executionWait}', ${JSON.stringify(condition)}::jsonb)`, - updatedAt: new Date(), - }).where(eq(agentWakeupRequests.id, executionWaitRequestId)); - return { kind: "deferred" as const }; - } - if (durableRequest || wakeCommentId || - hasInteractionContinuationWakeContext(enrichedContextSnapshot) || - readNonEmptyString(enrichedContextSnapshot.nativeStatusWakeIntentId)) { - await tx.insert(agentWakeupRequests).values({ - ...durableReceiptFields, - companyId: agent.companyId, agentId, source, triggerDetail, reason, - payload: withQueuedCommentIdsInWakePayload({ - ...payload, - issueId: issue.id, - [DEFERRED_WAKE_CONTEXT_KEY]: enrichedContextSnapshot, - executionWait: condition, - }, [...new Set([ - ...queuedCommentIdsFromRunContext(enrichedContextSnapshot), - ...(wakeCommentId ? [wakeCommentId] : []), - ])]), - status: "deferred_issue_execution", - requestedByActorType: opts.requestedByActorType ?? null, - requestedByActorId: opts.requestedByActorId ?? null, - idempotencyKey: opts.idempotencyKey ?? null, - }); - } else { - await recordExecutionWait(tx as unknown as Db, { - issueId: issue.id, condition, coalesce: coalesceExecutionWait, - request: { - ...durableReceiptFields, - companyId: agent.companyId, agentId, source, triggerDetail, - reason: "execution_reconciliation_required", - error: executionBlocker.nextAction, - payload, - requestedByActorType: opts.requestedByActorType ?? null, - requestedByActorId: opts.requestedByActorId ?? null, - idempotencyKey: opts.idempotencyKey ?? null, - }, - }); - } - return { kind: "deferred" as const }; - }; - const explicitContinuationRunId = randomUUID(); - const executionBlocker = await getExecutionBlocker( - tx as unknown as Db, issue.companyId, issue.id, - { conversationResetCommentId: opts.requestedByActorType === "user" ? wakeCommentId : null }, - ); - // Prove eligibility without retiring the hold. Later gates can still - // decline this wake; hold retirement and successor creation stay atomic. - // A bound chat request has already rechecked its current principal - // above. Treat its new user message like a board comment, but keep - // failed-run retry actions on their separate exact-request path. - if (executionBlocker && !(await admitExplicitNativeContinuation({ - db: tx as unknown as Db, companyId: issue.companyId, issueId: issue.id, - agentId, actorType: opts.requestedByActorType, actorId: opts.requestedByActorId, - reason: durableRequest && !failedChatRetry ? "issue_commented" : reason, - commentId: wakeCommentId ?? null, failedRunId: opts.failedRunId, successorRunId: explicitContinuationRunId, - queuedCommentInterruptId: opts.queuedCommentInterruptId, - queuedCommentRequestId: opts.queuedCommentRequestId, - dryRun: true, - onBlocked: (reason, message) => { continuationWait = { reason, message }; }, - }))) return deferBlockedExecution(executionBlocker); - - - if ( - worktreeExecutionCutoff && - issue.createdAt < worktreeExecutionCutoff - ) { - await tx.insert(agentWakeupRequests).values({ - ...durableReceiptFields, - companyId: agent.companyId, - agentId, - source, - triggerDetail, - reason: "heartbeat.worktree_execution_cutoff", - payload: { - ...(payload ?? {}), - heartbeatSkip: { - reason: "worktree_execution_cutoff", - cutoff: worktreeExecutionCutoff.toISOString(), - issueId: issue.id, - }, - }, - status: "skipped", - requestedByActorType: opts.requestedByActorType ?? null, - requestedByActorId: opts.requestedByActorId ?? null, - idempotencyKey: opts.idempotencyKey ?? null, - finishedAt: new Date(), - }); - return { kind: "skipped" as const }; - } - - const cancelStaleScheduledRetry = async ( - scheduledRun: typeof heartbeatRuns.$inferSelect, - ) => { - const issueCancelled = issue.status === "cancelled"; - if ( - scheduledRun.status !== "scheduled_retry" || - (scheduledRun.agentId === issue.assigneeAgentId && - !issueCancelled) - ) { - return false; - } - - const now = new Date(); - const reason = issueCancelled - ? "Cancelled because the issue was cancelled before the scheduled retry became due" - : "Cancelled because the issue was reassigned before the scheduled retry became due"; - const cancelled = await tx - .update(heartbeatRuns) - .set({ - status: "cancelled", - finishedAt: now, - error: reason, - errorCode: issueCancelled - ? "issue_cancelled" - : "issue_reassigned", - updatedAt: now, - }) - .where( - and( - eq(heartbeatRuns.id, scheduledRun.id), - eq(heartbeatRuns.status, "scheduled_retry"), - ), - ) - .returning() - .then((rows) => rows[0] ?? null); - - if (!cancelled) return false; - - if (scheduledRun.wakeupRequestId) { - await tx - .update(agentWakeupRequests) - .set({ - status: "cancelled", - finishedAt: now, - error: reason, - updatedAt: now, - }) - .where( - eq(agentWakeupRequests.id, scheduledRun.wakeupRequestId), - ); - } - - if (issue.executionRunId === scheduledRun.id) { - await tx - .update(issues) - .set({ - executionRunId: null, - executionAgentNameKey: null, - executionLockedAt: null, - updatedAt: now, - }) - .where( - and( - eq(issues.id, issue.id), - eq(issues.executionRunId, scheduledRun.id), - ), - ); - } - - const eventSeq = await allocateHeartbeatRunEventSeq( - tx as unknown as Db, - cancelled.id, - ); - - await tx.insert(heartbeatRunEvents).values({ - companyId: cancelled.companyId, - runId: cancelled.id, - agentId: cancelled.agentId, - seq: eventSeq, - eventType: "lifecycle", - stream: "system", - level: "warn", - message: issueCancelled - ? "Scheduled retry cancelled because issue was cancelled before it became due" - : "Scheduled retry cancelled because issue ownership changed before it became due", - payload: { - issueId: issue.id, - issueStatus: issue.status, - scheduledRetryAttempt: cancelled.scheduledRetryAttempt, - scheduledRetryAt: cancelled.scheduledRetryAt - ? new Date(cancelled.scheduledRetryAt).toISOString() - : null, - scheduledRetryReason: cancelled.scheduledRetryReason, - previousRetryAgentId: cancelled.agentId, - currentAssigneeAgentId: issue.assigneeAgentId, - }, - }); - await tx - .update(heartbeatRuns) - .set({ nextEventSeq: eventSeq + 1, updatedAt: now }) - .where(eq(heartbeatRuns.id, cancelled.id)); - - cancelledRunsToEmit.push(cancelled); - - return true; - }; - - let activeExecutionRun = issue.executionRunId - ? await tx - .select() - .from(heartbeatRuns) - .where(eq(heartbeatRuns.id, issue.executionRunId)) - .then((rows) => rows[0] ?? null) - : null; - - if ( - activeExecutionRun && - !EXECUTION_PATH_HEARTBEAT_RUN_STATUSES.includes( - activeExecutionRun.status as (typeof EXECUTION_PATH_HEARTBEAT_RUN_STATUSES)[number], - ) - ) { - activeExecutionRun = null; - } - - if ( - activeExecutionRun && - (await cancelStaleScheduledRetry(activeExecutionRun)) - ) { - activeExecutionRun = null; - } - - // A queued/scheduled run holding the lock for an agent that is - // no longer the issue's assignee is stale by design — the issue - // has been re-routed (e.g. blocked → in_review with a different - // assignee). Cancel it and release the lock; otherwise the new - // assignee's wake gets parked in `deferred_issue_execution` - // forever, because the original queued holder will never run - // (the issue's status / target now belongs to someone else). - // - // Race guard: pin the cancel UPDATE to the exact non-running - // status we read above. A worker could transition the holder - // from `queued` → `running` between the SELECT and this UPDATE; - // the status predicate ensures we never clobber a freshly- - // claimed running run. If zero rows matched, leave - // `activeExecutionRun` populated so the defer path runs - // normally against the now-running holder. - if ( - activeExecutionRun && - activeExecutionRun.status !== "running" && - issue.assigneeAgentId && - activeExecutionRun.agentId !== issue.assigneeAgentId - ) { - const cancelled = await tx - .update(heartbeatRuns) - .set({ - status: "cancelled", - finishedAt: new Date(), - error: - "Execution lock released after issue reassigned to a different agent", - errorCode: "lock_released_on_reassignment", - updatedAt: new Date(), - }) - .where( - and( - eq(heartbeatRuns.id, activeExecutionRun.id), - eq(heartbeatRuns.status, activeExecutionRun.status), - ), - ) - .returning(); - if (cancelled.length > 0) { - cancelledRunsToEmit.push(cancelled[0]); - if (activeExecutionRun.wakeupRequestId) { - await tx - .update(agentWakeupRequests) - .set({ - status: "cancelled", - finishedAt: new Date(), - error: - "Execution lock released after issue reassigned to a different agent", - updatedAt: new Date(), - }) - .where( - eq( - agentWakeupRequests.id, - activeExecutionRun.wakeupRequestId, - ), - ); - } - activeExecutionRun = null; - } - } - - if (!activeExecutionRun && issue.executionRunId) { - await tx - .update(issues) - .set({ - executionRunId: null, - executionAgentNameKey: null, - executionLockedAt: null, - updatedAt: new Date(), - }) - .where(eq(issues.id, issue.id)); - } - - if (!activeExecutionRun) { - const legacyRun = await tx - .select() - .from(heartbeatRuns) - .where( - and( - eq(heartbeatRuns.companyId, issue.companyId), - inArray(heartbeatRuns.status, [ - ...EXECUTION_PATH_HEARTBEAT_RUN_STATUSES, - ]), - sql`${heartbeatRuns.contextSnapshot} ->> 'issueId' = ${issue.id}`, - ), - ) - .orderBy( - sql`case when ${heartbeatRuns.status} = 'running' then 0 else 1 end`, - asc(heartbeatRuns.createdAt), - ) - .limit(1) - .then((rows) => rows[0] ?? null); - - if (legacyRun) { - if (await cancelStaleScheduledRetry(legacyRun)) { - activeExecutionRun = null; - } else { - activeExecutionRun = legacyRun; - const legacyAgent = await tx - .select({ name: agents.name }) - .from(agents) - .where(eq(agents.id, legacyRun.agentId)) - .then((rows) => rows[0] ?? null); - await tx - .update(issues) - .set({ - executionRunId: legacyRun.id, - executionAgentNameKey: normalizeAgentNameKey( - legacyAgent?.name, - ), - executionLockedAt: new Date(), - updatedAt: new Date(), - }) - .where(eq(issues.id, issue.id)); - } - } - } - - const dependencyReadiness = await issuesSvc - .listDependencyReadiness(issue.companyId, [issue.id], tx) - .then((rows) => rows.get(issue.id) ?? null); - - // Blocked descendants should stay idle until the final blocker resolves. - // Human comment/mention wakes are the exception: they may run in a - // bounded interaction mode so the assignee can answer or triage. - const blockedInteractionWake = - dependencyReadiness && - !dependencyReadiness.isDependencyReady && - allowsIssueInteractionWake( - enrichedContextSnapshot, - ISSUE_TREE_CONTROL_INTERACTION_WAKE_REASONS, - ); - - if (blockedInteractionWake) { - enrichedContextSnapshot.dependencyBlockedInteraction = true; - enrichedContextSnapshot.unresolvedBlockerIssueIds = - dependencyReadiness.unresolvedBlockerIssueIds; - enrichedContextSnapshot.unresolvedBlockerCount = - dependencyReadiness.unresolvedBlockerCount; - enrichedContextSnapshot.unresolvedBlockerSummaries = - await listUnresolvedBlockerSummaries( - tx, - issue.companyId, - issue.id, - dependencyReadiness.unresolvedBlockerIssueIds, - ); - } - - if ( - !activeExecutionRun && - dependencyReadiness && - !dependencyReadiness.isDependencyReady && - !blockedInteractionWake - ) { - await recordExecutionWait(tx as unknown as Db, { - issueId: issue.id, - coalesce: coalesceExecutionWait, - condition: { unresolvedBlockerIssueIds: [...dependencyReadiness.unresolvedBlockerIssueIds].sort() }, - request: { - ...durableReceiptFields, - companyId: agent.companyId, - agentId, - source, - triggerDetail, - reason: "issue_dependencies_blocked", - payload: { - ...(payload ?? {}), - issueId, - unresolvedBlockerIssueIds: - dependencyReadiness.unresolvedBlockerIssueIds, - }, - status: "skipped", - requestedByActorType: opts.requestedByActorType ?? null, - requestedByActorId: opts.requestedByActorId ?? null, - idempotencyKey: opts.idempotencyKey ?? null, - finishedAt: new Date(), - }, - }); - return { kind: "skipped" as const }; - } - - if ( - isolatedWorkspacesEnabled && - !activeExecutionRun && - issue.status !== "done" && - issue.status !== "cancelled" - ) { - const issueSettings = parseIssueExecutionWorkspaceSettings( - issue.executionWorkspaceSettings, - ); - const resolvedMode = resolveExecutionWorkspaceMode({ - projectPolicy: null, - issueSettings, - legacyUseProjectWorkspace: null, - }); - const workspaceManagedConfig = buildExecutionWorkspaceAdapterConfig( - { - agentConfig: parseObject(agent.adapterConfig), - projectPolicy: null, - issueSettings, - mode: resolvedMode, - legacyUseProjectWorkspace: null, - }, - ); - const resolvedStrategy = resolveEffectiveWorkspaceStrategyType( - resolvedMode, - workspaceManagedConfig, - ); - const existingExecutionWorkspaceStatus = issue.executionWorkspaceId - ? await tx - .select({ status: executionWorkspaces.status }) - .from(executionWorkspaces) - .where( - and( - eq(executionWorkspaces.id, issue.executionWorkspaceId), - eq(executionWorkspaces.companyId, issue.companyId), - ), - ) - .then((rows) => rows[0]?.status ?? null) - : null; - const reuseRequest = resolveExecutionWorkspaceReuseRequestForIssue({ - issueExecutionWorkspaceId: issue.executionWorkspaceId, - issueExecutionWorkspacePreference: - issue.executionWorkspacePreference, - existingExecutionWorkspaceStatus, - }); - const hasResolvablePriorSessionWorkspace = - await resolveHasResolvablePriorSessionWorkspace(); - - if ( - isUnrunnableWorktreeCombo({ - issue: { - projectId: issue.projectId ?? projectId ?? null, - projectWorkspaceId: issue.projectWorkspaceId, - executionWorkspaceId: issue.executionWorkspaceId, - executionWorkspacePreference: - issue.executionWorkspacePreference, - }, - resolvedMode, - resolvedStrategy, - reusableExecutionWorkspaceAvailable: - reuseRequest.existingExecutionWorkspaceAvailable, - hasResolvablePriorSessionWorkspace, - }) - ) { - const now = new Date(); - const issueLabel = formatIssueIdentifierLink( - issue.identifier, - issue.id, - ); - const blockedComment = [ - `Paperclip blocked ${issueLabel} before dispatch because its workspace settings are not runnable.`, - "", - `- Code: \`${WORKSPACE_WORKTREE_REQUIRES_PROJECT_CODE}\``, - `- Reason: ${WORKSPACE_WORKTREE_REQUIRES_PROJECT_MESSAGE}`, - `- Next action: ${WORKSPACE_WORKTREE_REQUIRES_PROJECT_REMEDIATION}`, - ].join("\n"); - await tx - .update(issues) - .set({ - status: "blocked", - checkoutRunId: null, - executionRunId: null, - executionAgentNameKey: null, - executionLockedAt: null, - updatedAt: now, - }) - .where(eq(issues.id, issue.id)); - await tx.insert(issueComments).values({ - companyId: issue.companyId, - issueId: issue.id, - body: blockedComment, - createdAt: now, - updatedAt: now, - }); - await tx.insert(agentWakeupRequests).values({ - ...durableReceiptFields, - companyId: agent.companyId, - agentId, - source, - triggerDetail, - reason: WORKSPACE_WORKTREE_REQUIRES_PROJECT_CODE, - payload: { - ...(payload ?? {}), - issueId, - heartbeatSkip: { - code: WORKSPACE_WORKTREE_REQUIRES_PROJECT_CODE, - reason: WORKSPACE_WORKTREE_REQUIRES_PROJECT_MESSAGE, - remediation: - WORKSPACE_WORKTREE_REQUIRES_PROJECT_REMEDIATION, - }, - }, - status: "skipped", - requestedByActorType: opts.requestedByActorType ?? null, - requestedByActorId: opts.requestedByActorId ?? null, - idempotencyKey: opts.idempotencyKey ?? null, - finishedAt: now, - }); - await logActivity(tx as unknown as Db, { - companyId: issue.companyId, - actorType: "system", - actorId: "system", - agentId, - runId: null, - action: "issue.workspace_preflight_blocked", - entityType: "issue", - entityId: issue.id, - details: { - code: WORKSPACE_WORKTREE_REQUIRES_PROJECT_CODE, - reason: WORKSPACE_WORKTREE_REQUIRES_PROJECT_MESSAGE, - remediation: WORKSPACE_WORKTREE_REQUIRES_PROJECT_REMEDIATION, - requestedReason: reason, - source, - triggerDetail, - resolvedMode, - resolvedStrategy, - hasResolvablePriorSessionWorkspace, - }, - }); - return { kind: "skipped" as const }; - } - } - - if (activeExecutionRun) { - // The resolved action is already a durable retry outbox. Do not merge - // its fresh-session contract into unrelated work or create a second - // deferred wake that could later replay the same reconciliation. - if (reconciledSourceRunId) return { kind: "deferred" as const }; - - const admissionScope = wakeQueue.createAdmissionTransactionScope( - agent.companyId, - tx as unknown as Db, - ); - const admission = await wakeQueue.admitWakeBehindIssueExecution( - admissionScope, - { - companyId: agent.companyId, - issueId: issue.id, - agentId, - agentNameKey, - issueExecutionAgentNameKey: issue.executionAgentNameKey, - activeExecutionRun: { - id: activeExecutionRun.id, - agentId: activeExecutionRun.agentId, - status: activeExecutionRun.status, - contextSnapshot: activeExecutionRun.contextSnapshot, - wakeupRequestId: activeExecutionRun.wakeupRequestId, - }, - allowRunCoalescing: isConversation(issue) ? false : opts.allowRunCoalescing, - durableReceipt: receiptRequest - ? { - id: receiptRequest.id, - requestedAt: receiptRequest.requestedAt, - } - : undefined, - reason, - liveRunExecutions, - wakeCommentId, - forceFreshSession: - enrichedContextSnapshot.forceFreshSession === true, - contextSnapshot: enrichedContextSnapshot, - source, - triggerDetail, - payload, - requestedByActorType: opts.requestedByActorType ?? null, - requestedByActorId: opts.requestedByActorId ?? null, - idempotencyKey: opts.idempotencyKey ?? null, - }, - ); - - if (admission.kind === "coalesced") { - return { - kind: "coalesced" as const, - run: admission.run as typeof heartbeatRuns.$inferSelect, - }; - } - if (admission.kind === "deferred") { - return { kind: "deferred" as const }; - } - // admission.kind === "proceed": no active run absorbed this wake, - // so fall through to the ordinary queue path below. - } - - // PAP-13775: no live run holds the lock, so this wake would start a - // fresh adapter session. If this agent's recent runs on this issue - // keep succeeding without any issue-visible progress and the wake - // carries no new information, hold it back for an escalating cooldown - // so external pollers/reconcilers can't storm full-price sessions. - // Server-side recovery retries insert runs directly and never reach - // this gate. - if ( - isThrottleCandidateIssueRewake({ - reason, - wakeCommentId: wakeCommentId ?? null, - requestedByActorType: opts.requestedByActorType ?? null, - forceFreshSession: - enrichedContextSnapshot.forceFreshSession === true, - hasExplicitResume: Boolean(explicitResumeSession), - }) - ) { - const throttleNow = new Date(); - const recentTerminalRuns = await tx - .select({ - id: heartbeatRuns.id, - status: heartbeatRuns.status, - finishedAt: heartbeatRuns.finishedAt, - }) - .from(heartbeatRuns) - .where( - and( - eq(heartbeatRuns.companyId, agent.companyId), - eq(heartbeatRuns.agentId, agentId), - sql`${heartbeatRuns.finishedAt} is not null`, - gte( - heartbeatRuns.finishedAt, - new Date(throttleNow.getTime() - ISSUE_REWAKE_LOOKBACK_MS), - ), - sql`${heartbeatRuns.contextSnapshot} ->> 'issueId' = ${issue.id}`, - ), - ) - .orderBy(desc(heartbeatRuns.finishedAt)) - .limit(ISSUE_REWAKE_RUN_SAMPLE_LIMIT); - - if (recentTerminalRuns.length > 0) { - const sampleRunIds = recentTerminalRuns.map( - (sampleRun) => sampleRun.id, - ); - const progressRows = await tx - .select({ runId: activityLog.runId }) - .from(activityLog) - .where( - and( - eq(activityLog.companyId, agent.companyId), - eq(activityLog.entityType, "issue"), - eq(activityLog.entityId, issue.id), - inArray(activityLog.runId, sampleRunIds), - inArray( - activityLog.action, - ISSUE_PROGRESS_ACTIVITY_ACTIONS, - ), - ), - ); - const lastRunFinishedAt = - recentTerminalRuns[0]?.finishedAt ?? null; - const newInputRows = lastRunFinishedAt - ? await tx - .select({ id: activityLog.id }) - .from(activityLog) - .where( - and( - eq(activityLog.companyId, agent.companyId), - eq(activityLog.entityType, "issue"), - eq(activityLog.entityId, issue.id), - gt(activityLog.createdAt, lastRunFinishedAt), - inArray( - activityLog.action, - ISSUE_NEW_INPUT_ACTIVITY_ACTIONS, - ), - wakeCommentId && opts.requestedByActorType === "agent" - ? ne(activityLog.actorType, "agent") - : undefined, - ), - ) - .limit(1) - : []; - - const throttleDecision = evaluateIssueRewakeThrottle({ - now: throttleNow, - recentTerminalRuns, - runIdsWithIssueProgress: new Set( - progressRows - .map((row) => row.runId) - .filter((runId): runId is string => Boolean(runId)), - ), - // For an agent comment wake, the query excludes agent-authored - // activity while preserving genuinely new user/system input. - // Presentation/author metadata therefore cannot smuggle human - // wake privilege, nor can it mask an actual human response. - hasNewIssueInputSinceLastRun: newInputRows.length > 0, - }); - - if (throttleDecision.blocked) { - await tx.insert(agentWakeupRequests).values({ - ...durableReceiptFields, - companyId: agent.companyId, - agentId, - source, - triggerDetail, - reason: "issue_rewake_throttled", - payload: { - ...(payload ?? {}), - issueId, - heartbeatSkip: { - reason: "issue_rewake_throttled", - requestedReason: reason, - noProgressStreak: throttleDecision.noProgressStreak, - cooldownMs: throttleDecision.cooldownMs, - lastRunFinishedAt: - throttleDecision.lastRunFinishedAt.toISOString(), - nextAllowedAt: - throttleDecision.nextAllowedAt.toISOString(), - }, - }, - status: "skipped", - requestedByActorType: opts.requestedByActorType ?? null, - requestedByActorId: opts.requestedByActorId ?? null, - idempotencyKey: opts.idempotencyKey ?? null, - finishedAt: throttleNow, - }); - return { kind: "skipped" as const }; - } - } - } - - const dailyCapBlock = await getHeartbeatDailyCapBlock( - agent, - policy, - {}, - tx, - ); - if (dailyCapBlock) { - if (executionWaitRequestId && executionBlocker) { - continuationWait = { reason: dailyCapBlock.reason, - message: "The agent has reached its daily limit. Your message is saved until work can resume." }; - return deferBlockedExecution(executionBlocker); - } - const now = new Date(); - await tx.insert(agentWakeupRequests).values({ - ...durableReceiptFields, - companyId: agent.companyId, - agentId, - source, - triggerDetail, - reason: dailyCapBlock.reason, - payload: { - ...(payload ?? {}), - heartbeatSkip: { - reason: - "Per-agent heartbeat daily cap reached before adapter invocation.", - observed: dailyCapBlock.observed, - limit: dailyCapBlock.limit, - }, - }, - status: "skipped", - requestedByActorType: opts.requestedByActorType ?? null, - requestedByActorId: opts.requestedByActorId ?? null, - idempotencyKey: opts.idempotencyKey ?? null, - finishedAt: now, - }); - if (source === "timer") { - await tx - .update(agents) - .set({ - lastHeartbeatAt: now, - updatedAt: now, - }) - .where(eq(agents.id, agentId)); - } - return { kind: "skipped" as const }; - } - - const explicitContinuation = await admitExplicitNativeContinuation({ - db: tx as unknown as Db, companyId: issue.companyId, issueId: issue.id, - agentId, actorType: opts.requestedByActorType, actorId: opts.requestedByActorId, - reason: durableRequest && !failedChatRetry ? "issue_commented" : reason, - commentId: wakeCommentId ?? null, failedRunId: opts.failedRunId, successorRunId: explicitContinuationRunId, - queuedCommentInterruptId: opts.queuedCommentInterruptId, - queuedCommentRequestId: opts.queuedCommentRequestId, - }); - if (!explicitContinuation && executionBlocker) return deferBlockedExecution(executionBlocker); - if (explicitContinuation) { - enrichedContextSnapshot.forceFreshSession = true; - enrichedContextSnapshot.previousRunId = explicitContinuation.previousRunId; - enrichedContextSnapshot.explicitUserContinuation = explicitContinuation; - } - - const wakeupRequest = await tx - .insert(agentWakeupRequests) - .values({ - ...durableReceiptFields, - companyId: agent.companyId, - agentId, - source, - triggerDetail, - reason, - payload, - status: "queued", - requestedByActorType: opts.requestedByActorType ?? null, - requestedByActorId: opts.requestedByActorId ?? null, - idempotencyKey: opts.idempotencyKey ?? null, - }) - .returning() - .then((rows) => rows[0]); - - // A handoff changes the executor, not the owner of saved user input. - // Validate its exact stopped source while the issue row is locked; - // unrelated agents and dedicated continuations keep their own wakes. - const interruptedRunId = readNonEmptyString(enrichedContextSnapshot.interruptedRunId); - const handoffSource = source === "assignment" && reason === "issue_assigned" && - issue.assigneeAgentId === agentId && interruptedRunId && isUuidLike(interruptedRunId) - ? await tx.select({ agentId: heartbeatRuns.agentId }).from(heartbeatRuns).where(and( - eq(heartbeatRuns.id, interruptedRunId), eq(heartbeatRuns.companyId, issue.companyId), - eq(heartbeatRuns.status, "cancelled"), eq(heartbeatRuns.errorCode, "issue_reassigned"), - ne(heartbeatRuns.agentId, agentId), - sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issue.id}`, - or(isNull(heartbeatRuns.nativeIssueId), eq(heartbeatRuns.nativeIssueId, issue.id)), - )).then(rows => rows[0] ?? null) - : null; - const canCoalesceComments = !isConversation(issue) && opts.allowRunCoalescing !== false; - // A resumed receipt already passed admission under this issue lock. - // Consume it with its successor even when chat keeps other messages - // in separate turns, or finalization will deliver it a second time. - const pendingComments = - (executionWaitRequestId || canCoalesceComments) && - !(await getExecutionBlocker(tx as unknown as Db, issue.companyId, issue.id)) - ? await tx - .select() - .from(agentWakeupRequests) - .where( - and( - eq(agentWakeupRequests.companyId, issue.companyId), - inArray(agentWakeupRequests.agentId, handoffSource ? [agentId, handoffSource.agentId] : [agentId]), - eq(agentWakeupRequests.status, "deferred_issue_execution"), - sql`${agentWakeupRequests.payload}->>'issueId' = ${issue.id}`, - canCoalesceComments ? undefined : eq(agentWakeupRequests.id, executionWaitRequestId!), - ), - ) - .orderBy(asc(agentWakeupRequests.requestedAt)) - : []; - const adoptedComments = pendingComments.filter((wake) => { - if (wake.id === executionWaitRequestId) return true; - if (wake.id === opts.queuedCommentInterruptId || wake.id === opts.queuedCommentRequestId) return true; - const deferredPayload = parseObject(wake.payload); - const deferredContext = parseObject( - deferredPayload[DEFERRED_WAKE_CONTEXT_KEY], - ); - // Dedicated interaction wakes carry their own source and session - // contract. ID-only adoption must not erase that continuation. - return ( - // Durable chat work must keep its receipt, actor, source, and - // session contract through normal promotion and authorization. - !wake.idempotencyKey?.startsWith("chat-inbound:") && - !isInteractionResolutionWakePayload(deferredPayload) && - !hasInteractionContinuationWakeContext(deferredContext) && - ["issue_commented", "issue_reopened_via_comment"].includes(String(deferredContext.wakeReason ?? wake.reason)) && - queuedCommentIdsFromWakePayload(wake.payload).length > 0 - ); - }); - let adoptedCommentIds = [ - ...new Set([ - ...adoptedComments.flatMap((wake) => - queuedCommentIdsFromWakePayload(wake.payload), - ), - ...queuedCommentIdsFromRunContext(enrichedContextSnapshot), - ]), - ]; - if (opts.queuedCommentRequestId || handoffSource) { - adoptedCommentIds = await undeliveredLegacyUserCommentIds(tx as unknown as Db, - agent.companyId, issueId, agentId, adoptedCommentIds); - } - const newRun = await tx - .insert(heartbeatRuns) - .values({ - ...(explicitContinuation ? { id: explicitContinuationRunId } : {}), - companyId: agent.companyId, - agentId, - scopeKind: readNonEmptyString(enrichedContextSnapshot.issueId) ? "issue" : "company", - issueId: readNonEmptyString(enrichedContextSnapshot.issueId), - invocationSource: source, - triggerDetail, - status: "queued", - responsibleUserId: await resolveQueuedResponsibleUserId(), - wakeupRequestId: wakeupRequest.id, - retryOfRunId: failedChatRetry - ? durableRequest!.failedRunRetry!.failedRunId - : opts.failedRunId ?? automaticParentRunId, - contextSnapshot: adoptedComments.length - ? withQueuedCommentIdsInRunContext( - enrichedContextSnapshot, - adoptedCommentIds, - ) - : enrichedContextSnapshot, - sessionIdBefore: explicitContinuation ? null : sessionBefore, - continuationAttempt, - ...(reconciledSourceRunId - ? { retryOfRunId: reconciledSourceRunId } - : {}), - ...(reconciledRestoreRetryCount !== null ? { - scheduledRetryAttempt: reconciledRestoreRetryCount, - scheduledRetryReason: "transient_failure", - } : {}), - }) - .returning() - .then((rows) => rows[0]); - - await tx - .update(agentWakeupRequests) - .set({ - runId: newRun.id, - updatedAt: new Date(), - }) - .where(eq(agentWakeupRequests.id, wakeupRequest.id)); - - if (adoptedComments.length) { - await tx - .update(agentWakeupRequests) - .set({ - status: "coalesced", - runId: newRun.id, - finishedAt: new Date(), - updatedAt: new Date(), - }) - .where( - inArray( - agentWakeupRequests.id, - adoptedComments.map((wake) => wake.id), - ), - ); - await tx - .update(agentWakeupRequests) - .set({ - payload: withQueuedCommentIdsInWakePayload(payload, adoptedCommentIds), - }) - .where(eq(agentWakeupRequests.id, wakeupRequest.id)); - } - - // executionRunId is NOT stamped here (enqueueWakeup queues the run but - // doesn't start it). It will be stamped in claimQueuedRun() once the run - // transitions to "running" — Fix A (lazy locking). - - return { kind: "queued" as const, run: newRun }; - }) - .catch((error) => { - if (isExternalChatWaitAuthorizationContention(error)) - return { kind: "deferred" as const }; - throw error; - }); - - // Telemetry for the cancelled runs is best-effort background work. - // Fire it here and never await it: none of the lifecycle work below, - // nor this function's return, depends on it, so a slow telemetry - // lookup must not delay them. - for (const cancelledRun of cancelledRunsToEmit) { - void emitAgentTaskRun(db, cancelledRun); - } - - if (outcome.kind === "durable") { - return outcome.receipt.runId ? getRun(outcome.receipt.runId) : null; - } - if (outcome.kind === "deferred" || outcome.kind === "skipped") { - return null; - } - if (outcome.kind === "coalesced") { - await startNextQueuedRunForAgent(agent.id); - return outcome.run; - } - if (outcome.kind === "replayed") { - if (outcome.run.status === "queued") - await startNextQueuedRunForAgent(agent.id); - return outcome.run; - } - - const newRun = outcome.run; - publishLiveEvent({ - companyId: newRun.companyId, - type: "heartbeat.run.queued", - payload: { - runId: newRun.id, - agentId: newRun.agentId, - invocationSource: newRun.invocationSource, - triggerDetail: newRun.triggerDetail, - wakeupRequestId: newRun.wakeupRequestId, - }, - }); - - await startNextQueuedRunForAgent(agent.id); - return newRun; - } - - if (durableRequest) throw new Error("chat_inbound_wakeup_binding_denied"); - - const activeRuns = await db - .select() - .from(heartbeatRuns) - .where( - and( - eq(heartbeatRuns.agentId, agentId), - inArray(heartbeatRuns.status, [ - ...EXECUTION_PATH_HEARTBEAT_RUN_STATUSES, - ]), - ), - ) - .orderBy(desc(heartbeatRuns.createdAt)); - - const sameScopeQueuedRun = activeRuns.find( - (candidate) => - candidate.status === "queued" && - isSameTaskScope(runTaskKey(candidate), taskKey), - ); - const sameScopeScheduledRetryRun = activeRuns.find( - (candidate) => - candidate.status === "scheduled_retry" && - isSameTaskScope(runTaskKey(candidate), taskKey), - ); - const sameScopeRunningRun = activeRuns.find( - (candidate) => - candidate.status === "running" && - isSameTaskScope(runTaskKey(candidate), taskKey), - ); - const shouldQueueFollowupForRunningWake = - Boolean(sameScopeRunningRun) && - !sameScopeQueuedRun && - shouldQueueFollowupForRunningIssueWake({ - contextSnapshot: enrichedContextSnapshot, - wakeCommentId, - }); - // Unscoped manual wakes need their own receipt and execution identity too. - const rawCoalescedTarget = - opts.allowRunCoalescing === false || opts.manualUserWake - ? null - : (sameScopeQueuedRun ?? - sameScopeScheduledRetryRun ?? - (shouldQueueFollowupForRunningWake - ? null - : (sameScopeRunningRun ?? null))); - - const coalescedTargetRun = filterZombieCoalesceTarget( - rawCoalescedTarget, - liveRunExecutions, - ); - - if (coalescedTargetRun) { - const mergedContextSnapshot = mergeCoalescedContextSnapshot( - coalescedTargetRun.contextSnapshot, - enrichedContextSnapshot, - { - preserveExistingInteractionContinuation: - coalescedTargetRun.status === "queued" || - coalescedTargetRun.status === "scheduled_retry", - }, - ); - const mergedRun = await db - .update(heartbeatRuns) - .set({ - contextSnapshot: mergedContextSnapshot, - updatedAt: new Date(), - }) - .where(eq(heartbeatRuns.id, coalescedTargetRun.id)) - .returning() - .then((rows) => rows[0] ?? coalescedTargetRun); - - await db.insert(agentWakeupRequests).values({ - ...durableReceiptFields, - companyId: agent.companyId, - agentId, - source, - triggerDetail, - reason, - payload, - status: "coalesced", - coalescedCount: 1, - requestedByActorType: opts.requestedByActorType ?? null, - requestedByActorId: opts.requestedByActorId ?? null, - idempotencyKey: opts.idempotencyKey ?? null, - runId: mergedRun.id, - finishedAt: new Date(), - }); - return mergedRun; - } - - const queueOutcome = await db.transaction(async (tx) => { - await tx.execute( - sql`select id from agents where id = ${agentId} and company_id = ${agent.companyId} for update`, - ); - - const dailyCapBlock = await getHeartbeatDailyCapBlock( - agent, - policy, - {}, - tx, - ); - if (dailyCapBlock) { - const now = new Date(); - await tx.insert(agentWakeupRequests).values({ - ...durableReceiptFields, - companyId: agent.companyId, - agentId, - source, - triggerDetail, - reason: dailyCapBlock.reason, - payload: { - ...(payload ?? {}), - heartbeatSkip: { - reason: - "Per-agent heartbeat daily cap reached before adapter invocation.", - observed: dailyCapBlock.observed, - limit: dailyCapBlock.limit, - }, - }, - status: "skipped", - requestedByActorType: opts.requestedByActorType ?? null, - requestedByActorId: opts.requestedByActorId ?? null, - idempotencyKey: opts.idempotencyKey ?? null, - finishedAt: now, - }); - if (source === "timer") { - await tx - .update(agents) - .set({ - lastHeartbeatAt: now, - updatedAt: now, - }) - .where(eq(agents.id, agentId)); - } - return { kind: "skipped" as const }; - } - - const wakeupRequest = await tx - .insert(agentWakeupRequests) - .values({ - ...durableReceiptFields, - companyId: agent.companyId, - agentId, - source, - triggerDetail, - reason, - payload, - status: "queued", - requestedByActorType: opts.requestedByActorType ?? null, - requestedByActorId: opts.requestedByActorId ?? null, - idempotencyKey: opts.idempotencyKey ?? null, - }) - .returning() - .then((rows) => rows[0]); - - const newRun = await tx - .insert(heartbeatRuns) - .values({ - companyId: agent.companyId, - agentId, - scopeKind: readNonEmptyString(enrichedContextSnapshot.issueId) ? "issue" : "company", - issueId: readNonEmptyString(enrichedContextSnapshot.issueId), - invocationSource: source, - triggerDetail, - status: "queued", - responsibleUserId: await resolveQueuedResponsibleUserId(), - wakeupRequestId: wakeupRequest.id, - contextSnapshot: enrichedContextSnapshot, - sessionIdBefore: sessionBefore, - continuationAttempt, - }) - .returning() - .then((rows) => rows[0]); - - await tx - .update(agentWakeupRequests) - .set({ - runId: newRun.id, - updatedAt: new Date(), - }) - .where(eq(agentWakeupRequests.id, wakeupRequest.id)); - - return { kind: "queued" as const, run: newRun }; - }); - - if (queueOutcome.kind === "skipped") return null; - const newRun = queueOutcome.run; - - publishLiveEvent({ - companyId: newRun.companyId, - type: "heartbeat.run.queued", - payload: { - runId: newRun.id, - agentId: newRun.agentId, - invocationSource: newRun.invocationSource, - triggerDetail: newRun.triggerDetail, - wakeupRequestId: newRun.wakeupRequestId, - }, - }); - - await startNextQueuedRunForAgent(agent.id); - - return newRun; - } - - /** - * Native status commitment deliberately persists dependency/parent wake - * intents in the same transaction as the authoritative status projection. - * Those rows are not runnable until the heartbeat scheduler has applied its - * normal policy, workspace, concurrency, and responsible-user checks. Bridge - * the durable intent into that scheduler here instead of treating a bare - * `agent_wakeup_requests` row as if it were already a queued heartbeat run. - * - * The intent is claimed before dispatch. A deterministic dispatcher actor id - * lets a later sweep recover the narrow process-crash window after the real - * wake was inserted but before the intent was linked to it. Dispatch failure - * only requeues the intent; it never changes the provider run outcome. - */ - async function dispatchPendingNativeStatusWakeups( - input: { - companyId?: string; - limit?: number; - staleClaimMs?: number; - } = {}, - ) { - const now = new Date(); - const staleClaimMs = Math.max(1_000, input.staleClaimMs ?? 60_000); - const candidates = await db - .select() - .from(agentWakeupRequests) - .where( - and( - input.companyId - ? eq(agentWakeupRequests.companyId, input.companyId) - : undefined, - eq(agentWakeupRequests.requestedByActorType, "system"), - eq(agentWakeupRequests.requestedByActorId, "native-status-committer"), - inArray(agentWakeupRequests.status, ["queued", "claimed"]), - isNull(agentWakeupRequests.runId), - ), - ) - .orderBy(asc(agentWakeupRequests.requestedAt)) - .limit(Math.max(1, Math.min(input.limit ?? 100, 500))); - - let dispatched = 0; - let recovered = 0; - let deferred = 0; - const deliveredByIssueScope = new Map< - string, - { runId: string | null; status: string } - >(); - - for (const candidate of candidates) { - const dispatchActorId = `native-status-wake-dispatch:${candidate.id}`; - const existingDispatch = await db - .select() - .from(agentWakeupRequests) - .where( - and( - eq(agentWakeupRequests.companyId, candidate.companyId), - eq(agentWakeupRequests.requestedByActorType, "system"), - eq(agentWakeupRequests.requestedByActorId, dispatchActorId), - ), - ) - .orderBy(desc(agentWakeupRequests.requestedAt)) - .limit(1) - .then((rows) => rows[0] ?? null); - - if (existingDispatch) { - const recoveredStatus = existingDispatch.runId - ? "coalesced" - : existingDispatch.status === "deferred_issue_execution" - ? "coalesced" - : existingDispatch.status; - await db - .update(agentWakeupRequests) - .set({ - status: recoveredStatus, - runId: existingDispatch.runId, - finishedAt: - (existingDispatch.runId ?? existingDispatch.finishedAt) - ? (existingDispatch.finishedAt ?? now) - : null, - error: existingDispatch.error, - updatedAt: now, - }) - .where( - and( - eq(agentWakeupRequests.id, candidate.id), - isNull(agentWakeupRequests.runId), - ), - ); - recovered += 1; - continue; - } - - if ( - candidate.status === "claimed" && - candidate.claimedAt && - now.getTime() - candidate.claimedAt.getTime() < staleClaimMs - ) { - deferred += 1; - continue; - } - - const claimed = await db - .update(agentWakeupRequests) - .set({ - status: "claimed", - claimedAt: now, - error: null, - updatedAt: now, - }) - .where( - and( - eq(agentWakeupRequests.id, candidate.id), - isNull(agentWakeupRequests.runId), - candidate.status === "claimed" - ? eq(agentWakeupRequests.status, "claimed") - : eq(agentWakeupRequests.status, "queued"), - ), - ) - .returning({ id: agentWakeupRequests.id }); - if (claimed.length === 0) continue; - - const payload = parseObject(candidate.payload); - const wakeContext = parseObject(payload._paperclipWakeContext); - const issueId = - readNonEmptyString(payload.issueId) ?? - readNonEmptyString(payload.taskId) ?? - readNonEmptyString(wakeContext.issueId) ?? - null; - const scopeKey = issueId - ? `${candidate.companyId}:${candidate.agentId}:${issueId}:${readNativeReviewAssignmentContext(wakeContext)?.nativeReviewInteractionId ?? ""}` - : null; - const priorDelivery = scopeKey - ? deliveredByIssueScope.get(scopeKey) - : null; - if (priorDelivery) { - await db - .update(agentWakeupRequests) - .set({ - status: "coalesced", - runId: priorDelivery.runId, - finishedAt: new Date(), - error: null, - updatedAt: new Date(), - }) - .where(eq(agentWakeupRequests.id, candidate.id)); - recovered += 1; - continue; - } - - let completedOnboardingGuard: WakeupOptions["issueStateGuard"]; - if (issueId) { - const targetIssue = await db - .select({ - status: issues.status, - statusVersion: issues.statusVersion, - assigneeAgentId: issues.assigneeAgentId, - }) - .from(issues) - .where( - and( - eq(issues.id, issueId), - eq(issues.companyId, candidate.companyId), - ), - ) - .limit(1) - .then((rows) => rows[0] ?? null); - const nativeReview = candidate.reason === "native_completion_review" - ? await getNativeReviewAssignment(db, { - companyId: candidate.companyId, issueId, agentId: candidate.agentId, - contextSnapshot: wakeContext, - }) - : null; - const onboardingResultReport = targetIssue?.status === "done" && await isCompletedOnboardingHandoffWake(db, { - companyId: candidate.companyId, issueId, agentId: candidate.agentId, - reason: candidate.reason, contextSnapshot: wakeContext, - }); - if (onboardingResultReport) completedOnboardingGuard = { assigneeAgentId: candidate.agentId, statuses: ["done"], statusVersion: targetIssue!.statusVersion }; - if ( - !targetIssue || - (["done", "cancelled"].includes(targetIssue.status) && !onboardingResultReport) || - (targetIssue.assigneeAgentId !== candidate.agentId && !nativeReview) || - (candidate.reason === "native_completion_review" && !nativeReview) - ) { - await db - .update(agentWakeupRequests) - .set({ - status: "skipped", - finishedAt: new Date(), - error: !targetIssue - ? "Native status wake target no longer exists" - : ["done", "cancelled"].includes(targetIssue.status) - ? `Native status wake target is already ${targetIssue.status}` - : "Native status wake target has a different assignee", - updatedAt: new Date(), - }) - .where(eq(agentWakeupRequests.id, candidate.id)); - recovered += 1; - continue; - } - } - - try { - const wakeRun = await enqueueWakeup(candidate.agentId, { - source: candidate.source as WakeupOptions["source"], - triggerDetail: (candidate.triggerDetail ?? - "system") as WakeupOptions["triggerDetail"], - reason: candidate.reason, - payload, - idempotencyKey: candidate.idempotencyKey, - requestedByActorType: "system", - requestedByActorId: dispatchActorId, - ...(completedOnboardingGuard ? { issueStateGuard: completedOnboardingGuard } : {}), - contextSnapshot: { - ...wakeContext, - ...(issueId ? { issueId, taskId: issueId } : {}), - wakeReason: candidate.reason, - source: "native_status_decision", - statusDecisionSource: "native_status_decision", - nativeStatusWakeIntentId: candidate.id, - }, - }); - - const delivered = await db - .select() - .from(agentWakeupRequests) - .where( - and( - eq(agentWakeupRequests.companyId, candidate.companyId), - eq(agentWakeupRequests.requestedByActorType, "system"), - eq(agentWakeupRequests.requestedByActorId, dispatchActorId), - ), - ) - .orderBy(desc(agentWakeupRequests.requestedAt)) - .limit(1) - .then((rows) => rows[0] ?? null); - - // The committer intent is the durable outbox entry. When admission is - // blocked, enqueueWakeup creates a separate deferred dispatch receipt; - // leave the original intent coalesced so the release drain cannot - // promote both rows (the original has no nativeStatusWakeIntentId - // provenance and would otherwise run once before the dispatch receipt). - const deliveredStatus = delivered?.status ?? "queued"; - await db - .update(agentWakeupRequests) - .set({ - status: wakeRun || deliveredStatus === "deferred_issue_execution" - ? "coalesced" - : deliveredStatus, - runId: wakeRun?.id ?? delivered?.runId ?? null, - finishedAt: - wakeRun || delivered?.finishedAt - ? (delivered?.finishedAt ?? new Date()) - : null, - error: delivered?.error ?? null, - updatedAt: new Date(), - }) - .where(eq(agentWakeupRequests.id, candidate.id)); - - if (scopeKey) { - deliveredByIssueScope.set(scopeKey, { - runId: wakeRun?.id ?? delivered?.runId ?? null, - status: wakeRun ? "coalesced" : (delivered?.status ?? "queued"), - }); - } - - if (wakeRun) dispatched += 1; - else deferred += 1; - } catch (error) { - await db - .update(agentWakeupRequests) - .set({ - status: "queued", - claimedAt: null, - error: - error instanceof Error - ? error.message.slice(0, 1_000) - : String(error).slice(0, 1_000), - updatedAt: new Date(), - }) - .where( - and( - eq(agentWakeupRequests.id, candidate.id), - eq(agentWakeupRequests.status, "claimed"), - isNull(agentWakeupRequests.runId), - ), - ); - logger.warn( - { - err: error, - wakeupRequestId: candidate.id, - agentId: candidate.agentId, - }, - "failed to dispatch persisted native status wake intent", - ); - deferred += 1; - } - } - - return { scanned: candidates.length, dispatched, recovered, deferred }; - } - async function listProjectScopedRunIds(companyId: string, projectId: string, createdBefore?: Date) { const runIssueId = sql< string | null diff --git a/server/src/services/heartbeat/queue.test.ts b/server/src/services/heartbeat/queue.test.ts new file mode 100644 index 00000000000..c3139de1f8b --- /dev/null +++ b/server/src/services/heartbeat/queue.test.ts @@ -0,0 +1,186 @@ +import type { Db } from "@paperclipai/db"; +import { agents, agentWakeupRequests, companies, createDb, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, heartbeatRuns, issues } from "@paperclipai/db"; +import { eq, sql } from "drizzle-orm"; +import { afterAll, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js"; +import { createRunDispatch } from "../../modules/run-dispatch/index.js"; +import type { createWakeQueue } from "../../modules/wake-queue/index.js"; +import { createHeartbeatRunState } from "./run-state.js"; +import { createHeartbeatRunPreparation } from "./run-preparation.js"; +import { createHeartbeatQueue, type HeartbeatQueueDependencies } from "./queue.js"; + +type Run = typeof heartbeatRuns.$inferSelect; +function callbacks(db: Db) { + const state = createHeartbeatRunState(db); + const preparation = createHeartbeatRunPreparation(db); + return { + ...preparation, + ...state, + getAgent: vi.fn(async () => { throw new Error("Unexpected agent lookup"); }), + options: {}, + setRunStatus: vi.fn(async (id, status, patch) => + db.update(heartbeatRuns).set({ ...patch, status }).where(eq(heartbeatRuns.id, id)).returning().then(rows => rows[0] ?? null)), + setWakeupStatus: vi.fn(async () => undefined), + appendRunEvent: vi.fn(async () => undefined), + finalizeAgentStatus: vi.fn(async () => undefined), + getAgentInvokability: vi.fn(async () => ({ invokable: true })), + getWorktreeExecutionCutoff: vi.fn(async () => null), + applyRunDispatchPostCommitEffects: vi.fn(), + issuesSvc: { + getDependencyReadiness: vi.fn(), + listDependencyReadiness: vi.fn(async () => new Map()), + }, + treeControlSvc: { getActivePauseHoldGate: vi.fn(async () => null) }, + budgets: { getInvocationBlock: vi.fn(async () => null) }, + instanceSettings: { getExperimental: vi.fn() }, + runDispatch: createRunDispatch(db), + wakeQueue: { + createAdmissionTransactionScope: vi.fn["createAdmissionTransactionScope"]>(), + admitWakeBehindIssueExecution: vi.fn["admitWakeBehindIssueExecution"]>(), + }, + activeRunExecutions: new Set(), + activeRunExecutionPromises: new Set>(), + activeWakeupPromises: new Set>(), + liveRunExecutions: { has: vi.fn(() => false) }, + isHeartbeatRunTerminalStatus: (status: string | null | undefined) => ["succeeded", "failed", "cancelled", "timed_out"].includes(status ?? ""), + isSameTaskScope: (left: string | null, right: string | null) => left === right, + runTaskKey: (run: Run) => typeof run.contextSnapshot?.issueId === "string" ? run.contextSnapshot.issueId : null, + filterZombieCoalesceTarget: (run: T | null) => run, + getSchedulingSuppression: vi.fn(async () => ({ suppressed: false, reason: null } as Awaited>)), + executeRun: vi.fn(async () => undefined), + releaseIssueExecutionAndPromote: vi.fn(async () => undefined), + publishRunLifecyclePluginEvent: vi.fn(), + cancelRunInternal: vi.fn(async () => undefined), + cancelActiveForAgentInternal: vi.fn(async () => undefined), + resumeExecutionWaitComments: vi.fn(async () => undefined), + resumeQueuedCommentInterrupt: vi.fn(async () => undefined), + resumeSavedLegacyComments: vi.fn(async () => undefined), + formatIssueIdentifierLink: (_identifier: string | null, fallback: string) => fallback, + sweepPendingCleanupLeases: vi.fn(), + } satisfies HeartbeatQueueDependencies; +} +function guardedDatabase() { + const access = vi.fn(() => { throw new Error("Unexpected queue database access"); }); + return { db: new Proxy({}, { get: access }) as Db, access }; +} + +describe("heartbeat queue module boundary", () => { + it("constructs without querying, dispatching, or registering background work", () => { + const database = guardedDatabase(); + const deps = callbacks(database.db); + createHeartbeatQueue(database.db, deps); + expect(database.access).not.toHaveBeenCalled(); + for (const callback of Object.values(deps)) if (vi.isMockFunction(callback)) expect(callback).not.toHaveBeenCalled(); + expect(deps.activeRunExecutionPromises.size).toBe(0); + expect(deps.activeWakeupPromises.size).toBe(0); + }); + + it.each(["startNextQueuedRunForAgent", "resumeQueuedRuns"] as const)("honors suppression before %s reads or resumes work", async (operation) => { + const database = guardedDatabase(); + const deps = callbacks(database.db); + deps.getSchedulingSuppression.mockResolvedValue({ suppressed: true, reason: "task_drain" }); + const queue = createHeartbeatQueue(database.db, deps); + await queue[operation]("agent"); + expect(database.access).not.toHaveBeenCalled(); + expect(deps.getAgent).not.toHaveBeenCalled(); + expect(deps.resumeExecutionWaitComments).not.toHaveBeenCalled(); + expect(deps.executeRun).not.toHaveBeenCalled(); + }); + + it("tracks pending wakes across instances and clears each rejected wake", async () => { + const database = guardedDatabase(); + const deps = callbacks(database.db); + let rejectFirst!: (reason: Error) => void; + let rejectSecond!: (reason: Error) => void; + deps.getAgent.mockImplementationOnce(() => new Promise((_resolve, reject) => { rejectFirst = reject; })); + deps.getAgent.mockImplementationOnce(() => new Promise((_resolve, reject) => { rejectSecond = reject; })); + const first = createHeartbeatQueue(database.db, deps).trackWakeup("first"); + const second = createHeartbeatQueue(database.db, deps).trackWakeup("second"); + const firstCheck = expect(first).rejects.toThrow("first lookup failed"); + const secondCheck = expect(second).rejects.toThrow("second lookup failed"); + expect(deps.activeWakeupPromises).toEqual(new Set([first, second])); + rejectFirst(new Error("first lookup failed")); + await firstCheck; + await Promise.resolve(); + expect(deps.activeWakeupPromises).toEqual(new Set([second])); + rejectSecond(new Error("second lookup failed")); + await secondCheck; + await Promise.resolve(); + expect(deps.activeWakeupPromises.size).toBe(0); + expect(deps.activeRunExecutionPromises.size).toBe(0); + expect(database.access).not.toHaveBeenCalled(); + }); + + it("uses the same policy limits for dispatch and execution", () => { + const database = guardedDatabase(); + const queue = createHeartbeatQueue(database.db, callbacks(database.db)); + const agent = { adapterType: "paperclip_runner", adapterConfig: { provider: "openai_dot" }, runtimeConfig: { heartbeat: { maxConcurrentRuns: 20, maxDailyRuns: 3.8 } } } as unknown as typeof agents.$inferSelect; + expect(queue.parseHeartbeatPolicy(agent)).toMatchObject({ maxConcurrentRuns: 1, maxDailyRuns: 3 }); + expect(queue.parseHeartbeatPolicy({ ...agent, adapterConfig: { provider: "codex" } })).toMatchObject({ maxConcurrentRuns: 20, maxDailyRuns: 3 }); + expect(database.access).not.toHaveBeenCalled(); + }); +}); + +const support = await getEmbeddedPostgresTestSupport(); +if (!support.supported) console.warn(`Skipping queue module database tests: ${support.reason}`); +describe.skipIf(!support.supported)("heartbeat queue database wiring", () => { + let database: Awaited>; + let db: Db; + beforeAll(async () => { + database = await startEmbeddedPostgresTestDatabase("paperclip-heartbeat-queue-module"); + db = createDb(database.connectionString); + }, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS); + afterAll(async () => { + await db?.$client.end(); + await database?.cleanup(); + }); + beforeEach(async () => { + await db.execute(sql`truncate table companies restart identity cascade`); + }); + + async function fixture(status = "running") { + const [company] = await db.insert(companies).values({ name: "Queue module", issuePrefix: "QM" }).returning(); + const [agent] = await db.insert(agents).values({ companyId: company.id, name: "Queue agent", role: "engineer", status: "idle", adapterType: "process", adapterConfig: {} }).returning(); + const [issue] = await db.insert(issues).values({ companyId: company.id, title: "Queue task", status: "todo", assigneeAgentId: agent.id }).returning(); + const [wake] = await db.insert(agentWakeupRequests).values({ companyId: company.id, agentId: agent.id, source: "assignment", status: "claimed", claimedAt: new Date() }).returning(); + const [run] = await db.insert(heartbeatRuns).values({ companyId: company.id, agentId: agent.id, status, invocationSource: "assignment", wakeupRequestId: wake.id, startedAt: status === "running" ? new Date() : null, contextSnapshot: { issueId: issue.id } }).returning(); + await db.update(issues).set({ executionRunId: run.id, executionAgentNameKey: agent.name, executionLockedAt: new Date() }).where(eq(issues.id, issue.id)); + return { company, agent, issue, wake, run }; + } + + it.each(["claimed", "cancelled"] as const)("releases a running claim while preserving a %s wake", async (wakeStatus) => { + const { run, issue, wake } = await fixture(); + await db.update(agentWakeupRequests).set({ status: wakeStatus }).where(eq(agentWakeupRequests.id, wake.id)); + await createHeartbeatQueue(db, callbacks(db)).releaseRunClaimedJustBeforeSuppression(run.id); + const [released] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)); + const [releasedIssue] = await db.select().from(issues).where(eq(issues.id, issue.id)); + const [releasedWake] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, wake.id)); + expect(released).toMatchObject({ status: "queued", startedAt: null, responsibleUserId: null }); + expect(releasedIssue).toMatchObject({ executionRunId: null, executionAgentNameKey: null, executionLockedAt: null }); + expect(releasedWake.status).toBe(wakeStatus === "cancelled" ? "cancelled" : "queued"); + if (wakeStatus === "claimed") expect(releasedWake.claimedAt).toBeNull(); + }); + + it("preserves a newer execution owner when releasing an old claim", async () => { + const { run, issue, agent, company } = await fixture(); + const [newOwner] = await db.insert(heartbeatRuns).values({ companyId: company.id, agentId: agent.id, status: "running", invocationSource: "assignment" }).returning(); + await db.update(issues).set({ executionRunId: newOwner.id }).where(eq(issues.id, issue.id)); + await createHeartbeatQueue(db, callbacks(db)).releaseRunClaimedJustBeforeSuppression(run.id); + const [current] = await db.select().from(issues).where(eq(issues.id, issue.id)); + expect(current.executionRunId).toBe(newOwner.id); + }); + + it("cancels at the daily run cap before claiming or dispatching", async () => { + const { run, agent } = await fixture("queued"); + const cappedAgent = { ...agent, runtimeConfig: { heartbeat: { maxDailyRuns: 0 } } }; + const deps = callbacks(db); + deps.getAgent.mockResolvedValue(cappedAgent); + expect(await createHeartbeatQueue(db, deps).claimQueuedRun(run)).toBeNull(); + const [cancelled] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)); + expect(cancelled).toMatchObject({ status: "cancelled", errorCode: "heartbeat.daily_run_limit" }); + expect(deps.setWakeupStatus).toHaveBeenCalledWith(run.wakeupRequestId, "skipped", expect.any(Object)); + expect(deps.releaseIssueExecutionAndPromote).toHaveBeenCalledWith(cancelled, { suppressImmediateRecovery: true }); + expect(deps.treeControlSvc.getActivePauseHoldGate).not.toHaveBeenCalled(); + expect(deps.executeRun).not.toHaveBeenCalled(); + }); +}); diff --git a/server/src/services/heartbeat/queue.ts b/server/src/services/heartbeat/queue.ts new file mode 100644 index 00000000000..04e51012bfe --- /dev/null +++ b/server/src/services/heartbeat/queue.ts @@ -0,0 +1,4576 @@ +import { normalizeAgentNameKey } from "./retries.js"; +import { + type WakeupOptions, + mergeCoalescedContextSnapshot, + enrichWakeContextSnapshot, + hasInteractionContinuationWakeContext, + isInteractionResolutionWakePayload, +} from "./run-preparation.js"; +import { resolveExecutionWorkspaceReuseRequestForIssue } from "./workspaces.js"; +import { + CHAT_COMPLETION_WAKE_REASON, + isCompletedOnboardingHandoffWake, +} from "../chat-completion-delivery.js"; +import { + compareCents, + hasWorkspaceRestoreFailure, +} from "@paperclipai/shared"; +import { nonIdleSlackIssueCondition } from "../slack-conversation-state.js"; +import { readQueuedInteractionResponse } from "../queued-interaction-response.js"; +import { + isConversation, + isConversationExecutionWake, + isWaitingConversation, +} from "../agent-conversations.js"; +import { legacyControllerClaim } from "../legacy-controller-lease.js"; +import { + admitExplicitNativeContinuation, + undeliveredLegacyUserCommentIds, +} from "../explicit-native-continuation.js"; +import { canRetryStoppedRun } from "../cancelled-native-startup.js"; +import { getExecutionBlocker } from "../execution-blocker.js"; +import { isConversationAdapter } from "../conversation-continuation.js"; +import { recordExecutionWait } from "../execution-wait.js"; +import { + getNativeReviewAssignment, + readNativeReviewAssignmentContext, +} from "../native-runtime/native-review-participant.js"; +import { claimQueuedNativeReviewRun } from "../native-runtime/native-review-dispatch.js"; +import { adapterExecutionControls } from "../adapter-execution-control.js"; +import { executionFailureRetryCount } from "../execution-recovery-attempt.js"; +import { + assertDurableChatWakeupReceipt, + assertDurableChatWakeupRequest, + authorizeFailedChatRunRetryWake, + FailedChatRunRetryAuthorizationError, + unadmittedChatWakeupCondition, +} from "../durable-chat-wakeup.js"; +import { agentService } from "../agents.js"; +import { normalizeLegacyRunnerProvider } from "@paperclipai/adapter-utils"; +import { randomUUID } from "node:crypto"; +import { + and, + asc, + desc, + eq, + gt, + gte, + inArray, + isNull, + lt, + lte, + ne, + notInArray, + or, + sql, +} from "drizzle-orm"; +import type { Db } from "@paperclipai/db"; +import { AGENT_DEFAULT_MAX_CONCURRENT_RUNS } from "@paperclipai/shared"; +import { + agents, + agentWakeupRequests, + activityLog, + chatConversations, + chatEndpoints, + companies, + costEvents, + environmentLeases, + executionWorkspaces, + heartbeatRunEvents, + heartbeatRuns, + issueComments, + issueRecoveryActions, + issueRelations, + issues, + nativeRunFinalizations, +} from "@paperclipai/db"; +import { + conflict, + HttpError, + notFound, +} from "../../errors.js"; +import { logger } from "../../middleware/logger.js"; +import { publishLiveEvent } from "../live-events.js"; +import { allocateHeartbeatRunEventSeq } from "../heartbeat-run-events.js"; +import { + queuedCommentIdsFromWakePayload, + queuedCommentIdsFromRunContext, + withQueuedCommentIdsInWakePayload, + withQueuedCommentIdsInRunContext, +} from "../issue-queued-comment-queue.js"; +import { + parseObject, + asBoolean, + asNumber, +} from "../../adapters/utils.js"; +import { isExternalChatWaitAuthorizationContention } from "../native-runtime/chat-attachment-reuse.js"; +import { emitAgentTaskRun } from "../agent-task-run-telemetry.js"; +import { + CHAT_CONTROL_RECOVERY_ADMISSION_KEY, + CHAT_CONTROL_RECOVERY_STOP_CODE, + CHAT_CONTROL_RECOVERY_UNRESOLVED_CODE, + chatControlRecoveryAdmission, + readChatControlRecoveryAdmission, + readChatControlNativeParent, + readChatControlRecoveryStop, +} from "../chat-control-recovery-stop.js"; +import { + ISSUE_NEW_INPUT_ACTIVITY_ACTIONS, + ISSUE_PROGRESS_ACTIVITY_ACTIONS, + ISSUE_REWAKE_LOOKBACK_MS, + ISSUE_REWAKE_RUN_SAMPLE_LIMIT, + evaluateIssueRewakeThrottle, + isThrottleCandidateIssueRewake, +} from "../issue-rewake-throttle.js"; +import { logActivity } from "../activity-log.js"; +import { ISSUE_BLOCKERS_RESOLVED_WAKE_REASON } from "../issue-dependency-wakeups.js"; +import { + ISSUE_TREE_CONTROL_INTERACTION_WAKE_REASONS, + isVerifiedIssueTreeControlInteractionWake, +} from "../issue-tree-control.js"; +import { + buildExecutionWorkspaceAdapterConfig, + isUnrunnableWorktreeCombo, + parseIssueExecutionWorkspaceSettings, + resolveEffectiveWorkspaceStrategyType, + resolveExecutionWorkspaceMode, + WORKSPACE_WORKTREE_REQUIRES_PROJECT_CODE, + WORKSPACE_WORKTREE_REQUIRES_PROJECT_MESSAGE, + WORKSPACE_WORKTREE_REQUIRES_PROJECT_REMEDIATION, +} from "../execution-workspace-policy.js"; +import { readContinuationAttempt } from "../recovery/index.js"; +import { allowsIssueInteractionWake } from "../../modules/run-dispatch/index.js"; +import { withAgentStartLock } from "../agent-start-lock.js"; +import { + evaluateAgentInvokability, + shouldCancelRunsForNonInvokableAgent, + type AgentOrgRow, +} from "../agent-invokability.js"; +import { isHeartbeatWakeOnDemandEnabled } from "../heartbeat-policy.js"; +import { isUuidLike } from "@paperclipai/shared"; +import { retryChatControlAdmission } from "../chat-control-admission-retry.js"; + +import type { HeartbeatServiceOptions } from "../heartbeat.js"; +import type { createHeartbeatRunState } from "./run-state.js"; +import type { createHeartbeatRunPreparation } from "./run-preparation.js"; +import type { HeartbeatRetryDependencies } from "./retries.js"; +import type { HeartbeatRecoveryDependencies, createHeartbeatRecovery } from "./recovery.js"; +import type { issueService } from "../issues.js"; +import type { issueTreeControlService } from "../issue-tree-control.js"; +import type { budgetService } from "../budgets.js"; +import type { instanceSettingsService } from "../instance-settings.js"; +import type { createRunDispatch } from "../../modules/run-dispatch/index.js"; +import type { createWakeQueue } from "../../modules/wake-queue/index.js"; + +type HeartbeatRun = typeof heartbeatRuns.$inferSelect; +type HeartbeatRunState = ReturnType; +type RunPreparation = ReturnType; + +/** Queue policy uses the service's lifecycle effects and process-wide ownership sets. */ +export interface HeartbeatQueueDependencies extends Pick, + Pick, + Pick, + Pick { + sweepPendingCleanupLeases: ReturnType["sweepPendingCleanupLeases"]; + options: Pick; + issuesSvc: Pick, "getDependencyReadiness" | "listDependencyReadiness">; + treeControlSvc: Pick, "getActivePauseHoldGate">; + budgets: Pick, "getInvocationBlock">; + instanceSettings: Pick, "getExperimental">; + runDispatch: Pick, "cancelStaleQueuedRun">; + wakeQueue: Pick, "createAdmissionTransactionScope" | "admitWakeBehindIssueExecution">; + activeRunExecutions: Set; + activeRunExecutionPromises: Set>; + activeWakeupPromises: Set>; + liveRunExecutions: { has(id: string): boolean }; + isHeartbeatRunTerminalStatus: (status: string | null | undefined) => boolean; + isSameTaskScope: (left: string | null, right: string | null) => boolean; + runTaskKey: (run: HeartbeatRun) => string | null; + filterZombieCoalesceTarget: (run: T | null, live: { has(id: string): boolean }) => T | null; + getSchedulingSuppression: () => Promise<{ suppressed: boolean; reason: "worktree_instance" | "database_restore_in_progress" | "task_drain" | null }>; + executeRun: (runId: string) => Promise; + releaseIssueExecutionAndPromote: (run: Pick, options?: { suppressImmediateRecovery?: boolean }) => Promise; + publishRunLifecyclePluginEvent: (run: HeartbeatRun) => void; + cancelRunInternal: (runId: string, reason?: string, options?: { errorCode?: string }) => Promise; + cancelActiveForAgentInternal: (agentId: string, reason: string) => Promise; + resumeExecutionWaitComments: () => Promise; + resumeQueuedCommentInterrupt: (companyId: string, queueId: string) => Promise; + resumeSavedLegacyComments: (companyId: string, queueId: string) => Promise; + formatIssueIdentifierLink: (identifier: string | null, fallback: string) => string; +} + +const HEARTBEAT_MAX_CONCURRENT_RUNS_DEFAULT = AGENT_DEFAULT_MAX_CONCURRENT_RUNS; + +const HEARTBEAT_MAX_CONCURRENT_RUNS_MIN = 1; + +const HEARTBEAT_MAX_CONCURRENT_RUNS_MAX = 50; + +export const DEFERRED_WAKE_CONTEXT_KEY = "_paperclipWakeContext"; + +export const EXECUTION_PATH_HEARTBEAT_RUN_STATUSES = [ + "queued", + "running", + "scheduled_retry", +] as const; + +const TIMER_ACTIONABLE_ISSUE_STATUSES = ["todo", "in_progress"] as const; + +export class ChatControlRecoveryUnresolvedError extends Error { + constructor() { + super( + "Run admission could not acquire its database locks after bounded retries. No provider work started. Review database contention and send a fresh request; this attempt will not automatically retry.", + ); + } +} + +const RUNNING_ISSUE_WAKE_REASONS_REQUIRING_FOLLOWUP = new Set([ + CHAT_COMPLETION_WAKE_REASON, + "approval_approved", + ISSUE_BLOCKERS_RESOLVED_WAKE_REASON, + "issue_recovery_action_restored", +]); + +function normalizeMaxConcurrentRuns(value: unknown) { + const parsed = Math.floor( + asNumber(value, HEARTBEAT_MAX_CONCURRENT_RUNS_DEFAULT), + ); + if (!Number.isFinite(parsed)) return HEARTBEAT_MAX_CONCURRENT_RUNS_DEFAULT; + return Math.max( + HEARTBEAT_MAX_CONCURRENT_RUNS_MIN, + Math.min(HEARTBEAT_MAX_CONCURRENT_RUNS_MAX, parsed), + ); +} + +export function shouldRequireIssueCommentForWake( + contextSnapshot: Record | null | undefined, +) { + if (contextSnapshot?.skipIssueComment === true) return false; + + const wakeReason = readNonEmptyString(contextSnapshot?.wakeReason); + return ( + wakeReason === "issue_assigned" || + wakeReason === "execution_review_requested" || + wakeReason === "execution_approval_requested" || + wakeReason === "execution_changes_requested" + ); +} + +async function listUnresolvedBlockerSummaries( + dbOrTx: Pick, + companyId: string, + issueId: string, + unresolvedBlockerIssueIds: string[], +) { + const ids = [...new Set(unresolvedBlockerIssueIds.filter(Boolean))]; + if (ids.length === 0) return []; + return dbOrTx + .select({ + id: issues.id, + identifier: issues.identifier, + title: issues.title, + status: issues.status, + priority: issues.priority, + assigneeAgentId: issues.assigneeAgentId, + assigneeUserId: issues.assigneeUserId, + }) + .from(issueRelations) + .innerJoin(issues, eq(issueRelations.issueId, issues.id)) + .where( + and( + eq(issueRelations.companyId, companyId), + eq(issueRelations.type, "blocks"), + eq(issueRelations.relatedIssueId, issueId), + inArray(issues.id, ids), + ), + ) + .orderBy(asc(issues.title)); +} + +export function shouldDeferFollowupWakeForSameIssue(input: { + activeRunStatus: string | null | undefined; + isSameExecutionAgent: boolean; + wakeCommentId: string | null | undefined; + forceFreshSession: boolean; +}) { + // A comment follow-up or explicit fresh-session wake needs a new run boundary. + if (!input.isSameExecutionAgent) return false; + if (input.activeRunStatus !== "running") return false; + if (input.wakeCommentId) return true; + if (input.forceFreshSession) return true; + return false; +} + +export function shouldQueueFollowupForRunningIssueWake(input: { + contextSnapshot: Record | null | undefined; + wakeCommentId: string | null; +}) { + if (input.wakeCommentId) return true; + // A structured interaction response is new input just like a comment. It + // must run after the turn that created the interaction instead of being + // merged into that still-running turn. + if ( + readNonEmptyString(input.contextSnapshot?.interactionId) && + readNonEmptyString(input.contextSnapshot?.interactionStatus) + ) { + return true; + } + const wakeReason = readNonEmptyString(input.contextSnapshot?.wakeReason); + if (wakeReason === "issue_children_completed" && ( + input.contextSnapshot?.onboardingCompletion === true || + (input.contextSnapshot?.statusDecisionSource === "native_status_decision" && + readNonEmptyString(input.contextSnapshot?.nativeChildCompletionDecisionId)) + )) return true; + return Boolean( + wakeReason && RUNNING_ISSUE_WAKE_REASONS_REQUIRING_FOLLOWUP.has(wakeReason), + ); +} + +function readNonEmptyString(value: unknown): string | null { + return typeof value === "string" && value.trim().length > 0 ? value : null; +} + +/** Queue admission and dispatch share the service lifecycle callbacks and execution ownership. */ +export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDependencies) { + const { + setRunStatus, + setWakeupStatus, + appendRunEvent, + releaseIssueExecutionAndPromote, + issuesSvc, + options, + publishRunLifecyclePluginEvent, + finalizeAgentStatus, + getAgent, + cancelRunInternal, + toAgentOrgRow, + getAgentInvokability, + budgets, + activeRunExecutions, + treeControlSvc, + runDispatch, + applyRunDispatchPostCommitEffects, + resolveResponsibleUserIdForRun, + getIssueExecutionContext, + isHeartbeatRunTerminalStatus, + liveRunExecutions, + getSchedulingSuppression, + resumeExecutionWaitComments, + getWorktreeExecutionCutoff, + resumeQueuedCommentInterrupt, + resumeSavedLegacyComments, + cancelActiveForAgentInternal, + listCompanyAgentOrgRows, + executeRun, + activeRunExecutionPromises, + activeWakeupPromises, + instanceSettings, + getRun, + sweepPendingCleanupLeases, + resolveExplicitResumeSessionOverride, + resolveSessionBeforeForWakeup, + hasResolvablePriorSessionWorkspaceForWake, + getRoutineEnvForExecutionIssue, + resolveResponsibleUserIdForRunSeed, + formatIssueIdentifierLink, + wakeQueue, + isSameTaskScope, + runTaskKey, + filterZombieCoalesceTarget, + } = dependencies; + + function parseHeartbeatPolicy(agent: typeof agents.$inferSelect) { + const runtimeConfig = parseObject(agent.runtimeConfig); + const heartbeat = parseObject(runtimeConfig.heartbeat); + + return { + enabled: asBoolean(heartbeat.enabled, false), + intervalSec: Math.max(0, asNumber(heartbeat.intervalSec, 0)), + wakeOnDemand: isHeartbeatWakeOnDemandEnabled(agent), + // A Dot binding has one external turn. Competing assignments must retain + // their queue position instead of claiming a second run that cannot bind. + maxConcurrentRuns: agent.adapterType === "paperclip_runner" && + parseObject(agent.adapterConfig).provider === "openai_dot" + ? 1 : normalizeMaxConcurrentRuns(heartbeat.maxConcurrentRuns), + skipTimerWhenNoActionableWork: asBoolean( + heartbeat.skipTimerWhenNoActionableWork ?? + heartbeat.requireActionableTimerWork ?? + heartbeat.issueOnlyTimer, + false, + ), + maxDailyRuns: normalizeOptionalNonNegativeInteger( + heartbeat.maxDailyRuns ?? + heartbeat.dailyRunLimit ?? + heartbeat.dailyRunCap ?? + heartbeat.maxRunsPerDay, + ), + maxDailyCostCents: normalizeOptionalNonNegativeInteger( + heartbeat.maxDailyCostCents ?? + heartbeat.dailyCostCentsLimit ?? + heartbeat.dailySpendCentsLimit ?? + heartbeat.dailyBudgetCents, + ), + }; + } + + function normalizeOptionalNonNegativeInteger(value: unknown) { + if (value === null || value === undefined || value === "") return null; + const normalized = Math.floor(asNumber(value, 0)); + return normalized >= 0 ? normalized : null; + } + + function currentUtcDayWindow(now = new Date()) { + const start = new Date( + Date.UTC( + now.getUTCFullYear(), + now.getUTCMonth(), + now.getUTCDate(), + 0, + 0, + 0, + 0, + ), + ); + const end = new Date( + Date.UTC( + now.getUTCFullYear(), + now.getUTCMonth(), + now.getUTCDate() + 1, + 0, + 0, + 0, + 0, + ), + ); + return { start, end }; + } + + async function getHeartbeatDailyCapBlock( + agent: typeof agents.$inferSelect, + policy: ReturnType, + options: { + checkRunCap?: boolean; + checkCostCap?: boolean; + excludeRunId?: string | null; + } = {}, + client: Pick = db, + ) { + const checkRunCap = options.checkRunCap ?? true; + const checkCostCap = options.checkCostCap ?? true; + const { start, end } = currentUtcDayWindow(); + if (checkRunCap && policy.maxDailyRuns !== null) { + const conditions = [ + eq(heartbeatRuns.companyId, agent.companyId), + eq(heartbeatRuns.agentId, agent.id), + gte(heartbeatRuns.startedAt, start), + lt(heartbeatRuns.startedAt, end), + notInArray(heartbeatRuns.status, ["queued", "scheduled_retry"]), + ]; + if (options.excludeRunId) { + conditions.push(sql`${heartbeatRuns.id} <> ${options.excludeRunId}`); + } + const [row] = await client + .select({ total: sql`count(*)::integer` }) + .from(heartbeatRuns) + .where(and(...conditions)); + const observed = Number(row?.total ?? 0); + if (observed >= policy.maxDailyRuns) { + return { + reason: "heartbeat.daily_run_limit", + observed, + limit: policy.maxDailyRuns, + }; + } + } + + if (checkCostCap && policy.maxDailyCostCents !== null) { + const [row] = await client + .select({ + total: sql`coalesce(sum(${costEvents.costCents}), 0)::text`, + }) + .from(costEvents) + .where( + and( + eq(costEvents.companyId, agent.companyId), + eq(costEvents.agentId, agent.id), + gte(costEvents.occurredAt, start), + lt(costEvents.occurredAt, end), + ), + ); + const observed = Number(row?.total ?? 0); + if (compareCents(String(row?.total ?? 0), policy.maxDailyCostCents) >= 0) { + return { + reason: "heartbeat.daily_cost_limit", + observed, + limit: policy.maxDailyCostCents, + }; + } + } + + return null; + } + + async function cancelQueuedRunForHeartbeatDailyCap( + run: typeof heartbeatRuns.$inferSelect, + dailyCapBlock: NonNullable< + Awaited> + >, + ) { + const now = new Date(); + const reason = + "Cancelled because the agent reached a per-day heartbeat budget cap before adapter invocation"; + const cancelled = await setRunStatus(run.id, "cancelled", { + finishedAt: now, + error: reason, + errorCode: dailyCapBlock.reason, + resultJson: { + ...parseObject(run.resultJson), + stopReason: dailyCapBlock.reason, + observed: dailyCapBlock.observed, + limit: dailyCapBlock.limit, + effectiveTimeoutSec: 0, + timeoutConfigured: false, + timeoutSource: "heartbeat_daily_cap_gate", + timeoutFired: false, + }, + }); + if (!cancelled) return null; + + await setWakeupStatus(run.wakeupRequestId, "skipped", { + finishedAt: now, + error: reason, + }); + + await appendRunEvent(cancelled, { + eventType: "lifecycle", + stream: "system", + level: "warn", + message: reason, + payload: { + reason: dailyCapBlock.reason, + observed: dailyCapBlock.observed, + limit: dailyCapBlock.limit, + }, + }); + + await releaseIssueExecutionAndPromote(cancelled, { + suppressImmediateRecovery: true, + }); + + return cancelled; + } + + async function hasActionableTimerWork(agent: typeof agents.$inferSelect) { + const row = await db + .select({ id: issues.id }) + .from(issues) + .where( + and( + eq(issues.companyId, agent.companyId), + eq(issues.assigneeAgentId, agent.id), + isNull(issues.assigneeUserId), + isNull(issues.hiddenAt), + inArray(issues.status, [...TIMER_ACTIONABLE_ISSUE_STATUSES]), + isNull(issues.conversationAgentId), + nonIdleSlackIssueCondition(), + ), + ) + .limit(1) + .then((rows) => rows[0] ?? null); + return Boolean(row); + } + + async function markTimerHeartbeatChecked( + agentId: string, + source: WakeupOptions["source"], + ) { + if (source !== "timer") return; + await db + .update(agents) + .set({ + lastHeartbeatAt: new Date(), + updatedAt: new Date(), + }) + .where(eq(agents.id, agentId)); + } + + async function claimDueTimerHeartbeat( + agent: typeof agents.$inferSelect, + now: Date, + intervalSec: number, + ) { + const dueBefore = new Date(now.getTime() - intervalSec * 1000); + const claimed = await db + .update(agents) + .set({ + lastHeartbeatAt: now, + updatedAt: now, + }) + .where( + and( + eq(agents.id, agent.id), + eq(agents.companyId, agent.companyId), + or( + lte(agents.lastHeartbeatAt, dueBefore), + and( + isNull(agents.lastHeartbeatAt), + lte(agents.createdAt, dueBefore), + ), + ), + ), + ) + .returning({ id: agents.id }) + .then((rows) => rows[0] ?? null); + if (!claimed) return null; + return { wasFirstHeartbeat: !agent.lastHeartbeatAt }; + } + + function issueRunPriorityRank(priority: string | null | undefined) { + switch (priority) { + case "critical": + return 0; + case "high": + return 1; + case "medium": + return 2; + case "low": + return 3; + default: + return 4; + } + } + + async function listQueuedRunDependencyReadiness( + companyId: string, + queuedRuns: Array, + ) { + const issueIds = [ + ...new Set( + queuedRuns + .map((run) => + readNonEmptyString(parseObject(run.contextSnapshot).issueId), + ) + .filter((issueId): issueId is string => Boolean(issueId)), + ), + ]; + if (issueIds.length === 0) { + return new Map< + string, + Awaited> + >(); + } + return issuesSvc.listDependencyReadiness(companyId, issueIds); + } + + async function countRunningRunsForAgent(agentId: string) { + const [{ count }] = await db + .select({ count: sql`count(*)` }) + .from(heartbeatRuns) + .where( + and( + eq(heartbeatRuns.agentId, agentId), + eq(heartbeatRuns.status, "running"), + ), + ); + return Number(count ?? 0); + } + + async function withChatControlRecoveryGate( + run: typeof heartbeatRuns.$inferSelect, + stage: "claim" | "dispatch", + onClear: (tx: Db) => Promise, + nativeRecovery = false, + ) { + const issueId = readNonEmptyString( + parseObject(run.contextSnapshot).issueId, + ); + if (!issueId || run.invocationSource !== "automation") return onClear(db); + await options.beforeChatControlRecoveryCheck?.({ + runId: run.id, + issueId, + stage, + }); + if (stage === "dispatch" && nativeRecovery) { + // Protected historical/already-admitted owners did not previously take + // this fence's locks. A fresh exact read keeps unrelated Board contention + // from turning their recovery into a new admission failure. + const [current] = await db + .select() + .from(heartbeatRuns) + .where( + and( + eq(heartbeatRuns.id, run.id), + eq(heartbeatRuns.companyId, run.companyId), + eq(heartbeatRuns.agentId, run.agentId), + eq(heartbeatRuns.status, "running"), + ), + ) + .limit(1); + if ( + current && + current.wakeupRequestId === run.wakeupRequestId && + parseObject(current.contextSnapshot).issueId === issueId + ) { + const admission = readChatControlRecoveryAdmission(current); + const expected = readChatControlRecoveryAdmission(run); + if ( + expected !== "invalid" && + (admission === "admitted" || + (admission === "historical" && expected === "historical")) + ) + return onClear(db); + } + } + let terminal: typeof heartbeatRuns.$inferSelect | null = null; + try { + const attempt = () => db.transaction(async (tx) => { + terminal = null; + // Same queue-edit lock order, then the close committer's conversation + // row. NOWAIT releases partial locks on contention. Claim defers to the + // queue; dispatch retries this transaction before considering failure. + const [issue] = await tx + .select({ id: issues.id }) + .from(issues) + .where( + and(eq(issues.id, issueId), eq(issues.companyId, run.companyId)), + ) + .for("update", { noWait: true }) + .limit(1); + if (!issue) return null; + if (run.wakeupRequestId) + await tx + .select({ id: agentWakeupRequests.id }) + .from(agentWakeupRequests) + .where( + and( + eq(agentWakeupRequests.id, run.wakeupRequestId), + eq(agentWakeupRequests.companyId, run.companyId), + eq(agentWakeupRequests.agentId, run.agentId), + ), + ) + .for("update", { noWait: true }) + .limit(1); + const [current] = await tx + .select() + .from(heartbeatRuns) + .where( + and( + eq(heartbeatRuns.id, run.id), + eq(heartbeatRuns.companyId, run.companyId), + eq(heartbeatRuns.agentId, run.agentId), + ), + ) + .for("update", { noWait: true }) + .limit(1); + if ( + !current || + current.status !== (stage === "claim" ? "queued" : "running") + ) + return null; + const admission = readChatControlRecoveryAdmission(current); + const expectedAdmission = readChatControlRecoveryAdmission(run); + const admissionLost = + expectedAdmission !== "historical" && admission === "historical"; + if ( + stage === "dispatch" && + nativeRecovery && + !admissionLost && + (admission === "historical" || admission === "admitted") + ) + return onClear(tx as unknown as Db); + // A new warm child can inherit a prior runner's PID during preparation. + // Those fields do not prove this run's turn was dispatched. + // Exact admitted/historical recovery retains its existing ownership; + // a marked, unadmitted bootstrap must earn admission even after crash. + const proof = + admission === "invalid" || admissionLost + ? { kind: "unresolved" as const } + : await readChatControlRecoveryStop( + tx as unknown as Db, + { + companyId: run.companyId, + issueId, + agentId: run.agentId, + sourceRunId: run.id, + }, + true, + ); + if (proof.kind === "clear") { + if (stage === "claim" || admission === "required") + await tx + .update(heartbeatRuns) + .set({ + runnerProfileJson: { + ...parseObject(current.runnerProfileJson), + [CHAT_CONTROL_RECOVERY_ADMISSION_KEY]: + chatControlRecoveryAdmission( + current, + stage === "dispatch" + ? "admitted" + : admission === "admitted" + ? "admitted" + : "required", + ), + }, + updatedAt: new Date(), + }) + .where(eq(heartbeatRuns.id, current.id)); + return onClear(tx as unknown as Db); + } + if (proof.kind === "unresolved" && stage === "claim") return null; + const now = new Date(); + const stopped = proof.kind === "stopped"; + const code = stopped + ? CHAT_CONTROL_RECOVERY_STOP_CODE + : CHAT_CONTROL_RECOVERY_UNRESOLVED_CODE; + const error = stopped + ? "Automatic continuation stopped by the committed chat conversation close. Send a new request in chat or on the Board to start fresh work." + : "Automatic continuation source could not be verified before provider admission. Review the task and send a fresh request; this attempt will not automatically retry."; + [terminal] = await tx + .update(heartbeatRuns) + .set({ + status: stopped ? "cancelled" : "failed", + errorCode: code, + error, + finishedAt: now, + resultJson: { + ...parseObject(current.resultJson), + automaticRecovery: { + code, + providerDispatched: false, + ...(stopped + ? { + sourceRunId: proof.sourceRunId, + conversationId: proof.conversationId, + publicationId: proof.publicationId, + } + : {}), + }, + }, + updatedAt: now, + }) + .where( + and( + eq(heartbeatRuns.id, current.id), + eq(heartbeatRuns.status, current.status), + ), + ) + .returning(); + if (!terminal) return null; + if (current.wakeupRequestId) + await tx + .update(agentWakeupRequests) + .set({ + status: stopped ? "skipped" : "failed", + error, + finishedAt: now, + updatedAt: now, + }) + .where( + and( + eq(agentWakeupRequests.id, current.wakeupRequestId), + eq(agentWakeupRequests.companyId, current.companyId), + eq(agentWakeupRequests.agentId, current.agentId), + eq(agentWakeupRequests.runId, current.id), + ne(agentWakeupRequests.status, "cancelled"), + ), + ); + await tx + .update(issues) + .set({ + executionRunId: null, + executionAgentNameKey: null, + executionLockedAt: null, + updatedAt: now, + }) + .where( + and( + eq(issues.companyId, current.companyId), + eq(issues.id, issueId), + eq(issues.executionRunId, current.id), + ), + ); + return null; + }); + const result = stage === "dispatch" + ? await retryChatControlAdmission(attempt) + : await attempt(); + if (terminal) { + const settled = terminal as typeof heartbeatRuns.$inferSelect; + publishLiveEvent({ + companyId: settled.companyId, + type: "heartbeat.run.status", + payload: { + runId: settled.id, + agentId: settled.agentId, + status: settled.status, + errorCode: settled.errorCode, + error: settled.error, + }, + }); + publishRunLifecyclePluginEvent(settled); + if (stage === "dispatch") + await finalizeAgentStatus(settled.agentId, "cancelled"); + } + return result; + } catch (error) { + if (!isExternalChatWaitAuthorizationContention(error)) throw error; + if (stage === "dispatch") { + // No effect was admitted. Let the existing setup-failure path settle + // this attempt distinctly; it must not become a successful close. + throw new ChatControlRecoveryUnresolvedError(); + } + return null; + } + } + + async function claimQueuedRun( + run: typeof heartbeatRuns.$inferSelect, + companyAgents?: AgentOrgRow[], + ) { + if (run.status !== "queued") return run; + const agent = await getAgent(run.agentId); + if (!agent) { + await cancelRunInternal( + run.id, + "Cancelled because the agent no longer exists", + ); + return null; + } + const invokability = companyAgents + ? evaluateAgentInvokability(toAgentOrgRow(agent), companyAgents) + : await getAgentInvokability(agent); + if (!invokability.invokable) { + await cancelRunInternal( + run.id, + `Cancelled because the agent is not invokable: ${invokability.reason}`, + ); + return null; + } + + const context = parseObject(run.contextSnapshot); + const budgetBlock = await budgets.getInvocationBlock( + run.companyId, + run.agentId, + { + issueId: readNonEmptyString(context.issueId), + projectId: readNonEmptyString(context.projectId), + }, + ); + if (budgetBlock) { + await cancelRunInternal(run.id, budgetBlock.reason); + return null; + } + + const dailyCapBlock = await getHeartbeatDailyCapBlock( + agent, + parseHeartbeatPolicy(agent), + { + excludeRunId: run.id, + checkRunCap: true, + checkCostCap: true, + }, + ); + if (dailyCapBlock) { + await cancelQueuedRunForHeartbeatDailyCap(run, dailyCapBlock); + return null; + } + + const issueId = readNonEmptyString(context.issueId); + if (issueId && activeRunExecutions.size > 0) { + // Native finalization publishes success before workspace synchronization, + // provider suspension, and lease release finish. A queued comment must + // not acquire a fresh sandbox while its predecessor still owns that work. + // The executor's finally block retries this agent after removing its owner. + const [settlingOwner] = await db.select({ id: heartbeatRuns.id }) + .from(heartbeatRuns) + .where(and( + eq(heartbeatRuns.companyId, run.companyId), + eq(heartbeatRuns.agentId, run.agentId), + eq(heartbeatRuns.runtimeMode, "native"), + inArray(heartbeatRuns.id, [...activeRunExecutions]), + sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issueId}`, + )) + .limit(1); + if (settlingOwner) return null; + } + if (issueId) { + const activePauseHold = await treeControlSvc.getActivePauseHoldGate( + run.companyId, + issueId, + ); + const treeHoldInteractionWake = + activePauseHold && + (await isVerifiedIssueTreeControlInteractionWake(db, { + companyId: run.companyId, + issueId, + agentId: run.agentId, + runId: run.id, + wakeupRequestId: run.wakeupRequestId, + contextSnapshot: context, + })); + if (activePauseHold && !treeHoldInteractionWake) { + await cancelRunInternal( + run.id, + "Cancelled because issue is held by an active subtree pause hold", + ); + await logActivity(db, { + companyId: run.companyId, + actorType: "system", + actorId: "system", + agentId: run.agentId, + runId: run.id, + action: "issue.tree_hold_run_interrupted", + entityType: "heartbeat_run", + entityId: run.id, + issueId: issueId, + details: { + issueId, + holdId: activePauseHold.holdId, + rootIssueId: activePauseHold.rootIssueId, + source: "heartbeat.claim_queued_run", + securityPrinciples: [ + "Complete Mediation", + "Fail Securely", + "Secure Defaults", + ], + }, + }); + return null; + } + + const dependencyReadiness = await issuesSvc.listDependencyReadiness( + run.companyId, + [issueId], + ); + const readiness = dependencyReadiness.get(issueId); + const unresolvedBlockerCount = readiness?.unresolvedBlockerCount ?? 0; + if ( + unresolvedBlockerCount > 0 && + !allowsIssueInteractionWake( + context, + ISSUE_TREE_CONTROL_INTERACTION_WAKE_REASONS, + ) + ) { + await cancelQueuedRunForBlockedDependencies( + run, + issueId, + readiness?.unresolvedBlockerIssueIds ?? [], + ); + logger.info( + { runId: run.id, issueId, unresolvedBlockerCount }, + "claimQueuedRun: cancelled blocked queued run", + ); + return null; + } + + const staleness = await runDispatch.cancelStaleQueuedRun({ + runId: run.id, + companyId: run.companyId, + expectedStatus: "queued", + }); + if (staleness.outcome === "cancelled" || staleness.outcome === "deferred") { + applyRunDispatchPostCommitEffects(staleness.postCommitEffects); + logger.info( + { + runId: run.id, + issueId, + outcome: staleness.outcome, + errorCode: staleness.outcome === "cancelled" ? staleness.errorCode : undefined, + }, + "claimQueuedRun: withheld queued run at the execution gate", + ); + return null; + } + } + + const claimedAt = new Date(); + const responsibleUserId = await resolveResponsibleUserIdForRun({ + run, + contextSnapshot: context, + issueContext: issueId + ? await getIssueExecutionContext(run.companyId, issueId) + : null, + routineEnvContext: { + routineId: null, + env: null, + responsibleUserId: null, + }, + }); + // All ordinary and comment claims use the same company-scoped issue + // lock. A batch may claim several runs before executeRun tracks any owner. + async function lockIssueExecutionClaim(tx: Db) { + const [owner] = issueId ? await tx.select({ + assigneeAgentId: issues.assigneeAgentId, + executionRunId: issues.executionRunId, + checkoutRunId: issues.checkoutRunId, + }).from(issues).where(and( + eq(issues.id, issueId), eq(issues.companyId, run.companyId), + )).for("update") : []; + const ownsIssue = owner?.assigneeAgentId === run.agentId && + context.wakeReason !== "source_scoped_recovery_action"; + if (ownsIssue && ["native_safe_replacement", "native_provider_overloaded"].includes(run.scheduledRetryReason ?? "") && + owner.checkoutRunId && owner.checkoutRunId !== run.id) { + return { ownsIssue, blocked: true }; + } + if (run.scheduledRetryReason === "native_provider_overloaded" && owner?.executionRunId && + owner.executionRunId !== run.id && owner.executionRunId !== run.retryOfRunId) { + return { ownsIssue, blocked: true }; + } + const previousRunId = run.scheduledRetryReason === "native_provider_overloaded" + ? run.retryOfRunId : ownsIssue ? owner?.executionRunId : null; + if (previousRunId && previousRunId !== run.id) { + const [previous] = await tx.select({ status: heartbeatRuns.status }) + .from(heartbeatRuns).where(and( + eq(heartbeatRuns.id, previousRunId), + eq(heartbeatRuns.companyId, run.companyId), + )); + // A terminal result can precede workspace/lease cleanup on this or + // another controller. Local absence alone is not a release receipt. + if (!isHeartbeatRunTerminalStatus(previous?.status) || + liveRunExecutions.has(previousRunId)) { + return { ownsIssue, blocked: true }; + } + const [pendingLease] = await tx.select({ id: environmentLeases.id }) + .from(environmentLeases).where(and( + eq(environmentLeases.companyId, run.companyId), + eq(environmentLeases.heartbeatRunId, previousRunId), + or(and(isNull(environmentLeases.releasedAt), + // Warm release deliberately retains the sandbox. Its successful + // receipt settles the old run without destroying the resource. + sql`not coalesce(${environmentLeases.status} = 'retained' + and ${environmentLeases.leasePolicy} = 'reuse_by_environment' + and ${environmentLeases.cleanupStatus} = 'success', false)`), + eq(environmentLeases.status, "pending_cleanup"), + eq(environmentLeases.cleanupStatus, "failed")), + )).limit(1); + const [finalization] = await tx.select({ + phase: nativeRunFinalizations.phase, leaseOwner: nativeRunFinalizations.leaseOwner, + }).from(nativeRunFinalizations).where(and( + eq(nativeRunFinalizations.companyId, run.companyId), + eq(nativeRunFinalizations.runId, previousRunId), + )); + if (pendingLease || (finalization && (finalization.leaseOwner || + !["committed", "applied", "terminal_failure"].includes(finalization.phase)))) { + return { ownsIssue, blocked: true }; + } + } + return { ownsIssue, blocked: false }; + } + async function bindClaimedIssueExecution(tx: Db, ownsIssue: boolean, claimedRun: typeof heartbeatRuns.$inferSelect | null | undefined) { + if (!claimedRun || !issueId || !ownsIssue) return; + await tx.update(issues).set({ + executionRunId: claimedRun.id, + executionAgentNameKey: normalizeAgentNameKey(agent.name), + executionLockedAt: claimedAt, + updatedAt: claimedAt, + }).where(and(eq(issues.id, issueId), eq(issues.companyId, run.companyId))); + } + const nativeReviewContext = readNativeReviewAssignmentContext(context); + const queuedCommentIds = queuedCommentIdsFromRunContext(context); + if ( + issueId && + run.invocationSource === "automation" && + queuedCommentIds.length > 0 + ) + await options.beforeChatControlRecoveryCheck?.({ + runId: run.id, + issueId, + stage: "claim", + }); + const queuedCommentClaim = + !nativeReviewContext && issueId && run.wakeupRequestId && queuedCommentIds.length > 0 + ? await db + .transaction(async (tx) => { + // Match the queue-edit lock order: issue, wake, then run. Once the + // run becomes running, a concurrent discard must observe the + // claimed wake and return an explicit conflict; if discard wins, + // this claim observes the cancelled queue and does no work. + const issueClaim = await lockIssueExecutionClaim(tx as unknown as Db); + if (issueClaim.blocked) return { kind: "stale" as const, run: null }; + const wake = await tx + .select() + .from(agentWakeupRequests) + .where( + and( + eq(agentWakeupRequests.id, run.wakeupRequestId!), + eq(agentWakeupRequests.companyId, run.companyId), + eq(agentWakeupRequests.agentId, run.agentId), + ), + ) + .for("update") + .limit(1) + .then((rows) => rows[0] ?? null); + const lockedRun = await tx + .select() + .from(heartbeatRuns) + .where( + and( + eq(heartbeatRuns.id, run.id), + eq(heartbeatRuns.companyId, run.companyId), + eq(heartbeatRuns.agentId, run.agentId), + ), + ) + .for("update") + .limit(1) + .then((rows) => rows[0] ?? null); + if ( + !wake || + wake.status !== "queued" || + wake.runId !== run.id || + !lockedRun || + lockedRun.status !== "queued" || + lockedRun.wakeupRequestId !== wake.id + ) { + return { kind: "stale" as const, run: null }; + } + + if (lockedRun.invocationSource === "automation") { + const admission = readChatControlRecoveryAdmission(lockedRun); + if (admission === "invalid") + return { kind: "stale" as const, run: null }; + const proof = await readChatControlRecoveryStop( + tx as unknown as Db, + { + companyId: lockedRun.companyId, + issueId, + agentId: lockedRun.agentId, + sourceRunId: lockedRun.id, + }, + true, + ); + if (proof.kind === "unresolved") + return { kind: "stale" as const, run: null }; + if (proof.kind === "stopped") { + const error = + "Automatic continuation stopped by the committed chat conversation close. Send a fresh request in chat or on the Board."; + const [cancelled] = await tx + .update(heartbeatRuns) + .set({ + status: "cancelled", + errorCode: CHAT_CONTROL_RECOVERY_STOP_CODE, + error, + finishedAt: claimedAt, + resultJson: { + ...parseObject(lockedRun.resultJson), + automaticRecovery: { + code: CHAT_CONTROL_RECOVERY_STOP_CODE, + providerDispatched: false, + sourceRunId: proof.sourceRunId, + conversationId: proof.conversationId, + publicationId: proof.publicationId, + }, + }, + updatedAt: claimedAt, + }) + .where( + and( + eq(heartbeatRuns.id, lockedRun.id), + eq(heartbeatRuns.status, "queued"), + ), + ) + .returning(); + if (!cancelled) return { kind: "stale" as const, run: null }; + await tx + .update(agentWakeupRequests) + .set({ + status: "skipped", + error, + finishedAt: claimedAt, + updatedAt: claimedAt, + }) + .where(eq(agentWakeupRequests.id, wake.id)); + await tx + .update(issues) + .set({ + executionRunId: null, + executionAgentNameKey: null, + executionLockedAt: null, + updatedAt: claimedAt, + }) + .where( + and( + eq(issues.id, issueId), + eq(issues.companyId, run.companyId), + eq(issues.executionRunId, run.id), + ), + ); + return { kind: "cancelled" as const, run: cancelled }; + } + await tx + .update(heartbeatRuns) + .set({ + runnerProfileJson: { + ...parseObject(lockedRun.runnerProfileJson), + [CHAT_CONTROL_RECOVERY_ADMISSION_KEY]: + chatControlRecoveryAdmission( + lockedRun, + admission === "admitted" ? "admitted" : "required", + ), + }, + updatedAt: claimedAt, + }) + .where(eq(heartbeatRuns.id, lockedRun.id)); + } + const authoritativeIds = queuedCommentIdsFromWakePayload( + wake.payload, + ); + if (authoritativeIds.length === 0) { + // Legacy/direct comment wakes carry comment ids in their ordinary + // payload and context, not in the authoritative queued-message + // envelope. Preserve their established claim path; only an + // explicitly bound queued-message envelope is subject to the + // live-comment discard gate below. + const [claimedRun] = await tx + .update(heartbeatRuns) + .set({ + status: "running", + runnerProfileJson: sql`(case when jsonb_typeof(${heartbeatRuns.runnerProfileJson}) = 'object' then ${heartbeatRuns.runnerProfileJson} else '{}'::jsonb end) || ${JSON.stringify({ adapterDispatch: { adapterType: agent.adapterType } })}::jsonb`, + ...legacyControllerClaim(run.runtimeMode), + responsibleUserId, + startedAt: lockedRun.startedAt ?? claimedAt, + updatedAt: claimedAt, + }) + .where( + and( + eq(heartbeatRuns.id, lockedRun.id), + eq(heartbeatRuns.status, "queued"), + ), + ) + .returning(); + await bindClaimedIssueExecution(tx as unknown as Db, issueClaim.ownsIssue, claimedRun); + return claimedRun + ? { kind: "claimed" as const, run: claimedRun } + : { kind: "stale" as const, run: null }; + } + const commentRows = await tx + .select({ + id: issueComments.id, + deletedAt: issueComments.deletedAt, + }) + .from(issueComments) + .where( + and( + eq(issueComments.companyId, run.companyId), + eq(issueComments.issueId, issueId), + inArray(issueComments.id, authoritativeIds), + ), + ); + const liveIds = authoritativeIds.filter((commentId) => { + const comment = commentRows.find((row) => row.id === commentId); + return Boolean(comment && !comment.deletedAt); + }); + if (liveIds.length === 0) { + const reason = "Queued messages were discarded before dispatch"; + const [cancelled] = await tx + .update(heartbeatRuns) + .set({ + status: "cancelled", + finishedAt: claimedAt, + error: reason, + errorCode: "queued_comment_discarded", + updatedAt: claimedAt, + }) + .where( + and( + eq(heartbeatRuns.id, lockedRun.id), + eq(heartbeatRuns.status, "queued"), + ), + ) + .returning(); + await tx + .update(agentWakeupRequests) + .set({ + status: "cancelled", + finishedAt: claimedAt, + error: reason, + updatedAt: claimedAt, + }) + .where(eq(agentWakeupRequests.id, wake.id)); + await tx + .update(issues) + .set({ + executionRunId: null, + executionAgentNameKey: null, + executionLockedAt: null, + updatedAt: claimedAt, + }) + .where( + and( + eq(issues.id, issueId), + eq(issues.companyId, run.companyId), + eq(issues.executionRunId, run.id), + ), + ); + return { + kind: "cancelled" as const, + run: cancelled ?? lockedRun, + }; + } + + await tx + .update(agentWakeupRequests) + .set({ + status: "claimed", + claimedAt, + payload: withQueuedCommentIdsInWakePayload( + wake.payload, + liveIds, + ), + updatedAt: claimedAt, + }) + .where(eq(agentWakeupRequests.id, wake.id)); + const [claimedRun] = await tx + .update(heartbeatRuns) + .set({ + status: "running", + runnerProfileJson: sql`(case when jsonb_typeof(${heartbeatRuns.runnerProfileJson}) = 'object' then ${heartbeatRuns.runnerProfileJson} else '{}'::jsonb end) || ${JSON.stringify({ adapterDispatch: { adapterType: agent.adapterType } })}::jsonb`, + ...legacyControllerClaim(run.runtimeMode), + responsibleUserId, + startedAt: lockedRun.startedAt ?? claimedAt, + contextSnapshot: withQueuedCommentIdsInRunContext( + lockedRun.contextSnapshot, + liveIds, + ), + updatedAt: claimedAt, + }) + .where( + and( + eq(heartbeatRuns.id, lockedRun.id), + eq(heartbeatRuns.status, "queued"), + ), + ) + .returning(); + await bindClaimedIssueExecution(tx as unknown as Db, issueClaim.ownsIssue, claimedRun); + return claimedRun + ? { kind: "claimed" as const, run: claimedRun } + : { kind: "stale" as const, run: null }; + }) + .catch((error) => { + if (isExternalChatWaitAuthorizationContention(error)) + return { kind: "stale" as const, run: null }; + throw error; + }) + : null; + if (queuedCommentClaim?.kind === "cancelled") { + await appendRunEvent(queuedCommentClaim.run, { + eventType: "lifecycle", + stream: "system", + level: "warn", + message: + queuedCommentClaim.run.error ?? + "Queued messages were discarded before dispatch", + }); + publishLiveEvent({ + companyId: queuedCommentClaim.run.companyId, + type: "heartbeat.run.status", + payload: { + runId: queuedCommentClaim.run.id, + agentId: queuedCommentClaim.run.agentId, + status: queuedCommentClaim.run.status, + invocationSource: queuedCommentClaim.run.invocationSource, + triggerDetail: queuedCommentClaim.run.triggerDetail, + error: queuedCommentClaim.run.error ?? null, + errorCode: queuedCommentClaim.run.errorCode ?? null, + startedAt: queuedCommentClaim.run.startedAt + ? new Date(queuedCommentClaim.run.startedAt).toISOString() + : null, + finishedAt: queuedCommentClaim.run.finishedAt + ? new Date(queuedCommentClaim.run.finishedAt).toISOString() + : null, + }, + }); + publishRunLifecyclePluginEvent(queuedCommentClaim.run); + // Fire-and-forget: nothing else in this path depends on the emission, + // so it must not delay the return. + void emitAgentTaskRun(db, queuedCommentClaim.run); + return null; + } + const claimed = queuedCommentClaim + ? queuedCommentClaim.run + : await withChatControlRecoveryGate(run, "claim", async (tx) => { + const claimValues = { + status: "running", + runnerProfileJson: sql`(case when jsonb_typeof(${heartbeatRuns.runnerProfileJson}) = 'object' then ${heartbeatRuns.runnerProfileJson} else '{}'::jsonb end) || ${JSON.stringify({ adapterDispatch: { adapterType: agent.adapterType } })}::jsonb`, + ...legacyControllerClaim(run.runtimeMode), + responsibleUserId, + startedAt: run.startedAt ?? claimedAt, + updatedAt: claimedAt, + }; + if (nativeReviewContext) { + if (run.scheduledRetryReason === "native_provider_overloaded") { + const predecessor = await lockIssueExecutionClaim(tx); + if (predecessor.blocked) return null; + } + return claimQueuedNativeReviewRun(tx, { + run, claimedAt, claimValues, + agentNameKey: normalizeAgentNameKey(agent.name), + }); + } + return tx.transaction(async (claimTx) => { + const issueClaim = await lockIssueExecutionClaim(claimTx as unknown as Db); + if (issueClaim.blocked) return null; + const claimedRun = await claimTx.update(heartbeatRuns).set(claimValues).where(and( + eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.status, "queued"), + )).returning().then((rows) => rows[0] ?? null); + await bindClaimedIssueExecution(claimTx as unknown as Db, issueClaim.ownsIssue, claimedRun); + return claimedRun; + }); + }); + if (!claimed) return null; + + publishLiveEvent({ + companyId: claimed.companyId, + type: "heartbeat.run.status", + payload: { + runId: claimed.id, + agentId: claimed.agentId, + status: claimed.status, + invocationSource: claimed.invocationSource, + triggerDetail: claimed.triggerDetail, + error: claimed.error ?? null, + errorCode: claimed.errorCode ?? null, + startedAt: claimed.startedAt + ? new Date(claimed.startedAt).toISOString() + : null, + finishedAt: claimed.finishedAt + ? new Date(claimed.finishedAt).toISOString() + : null, + }, + }); + publishRunLifecyclePluginEvent(claimed); + + if (!nativeReviewContext) { + await setWakeupStatus(claimed.wakeupRequestId, "claimed", { claimedAt }); + } + + // Fix A (lazy locking): stamp executionRunId now that the run is actually running, + // not at queue time. Guard is idempotent — safe if called more than once. + const claimedContext = parseObject(claimed.contextSnapshot); + const claimedIssueId = readNonEmptyString(claimedContext.issueId); + const claimedWakeReason = readNonEmptyString(claimedContext.wakeReason); + if ( + !nativeReviewContext && claimedIssueId && + claimedWakeReason !== "source_scoped_recovery_action" + ) { + const claimedAgent = await getAgent(claimed.agentId); + await db + .update(issues) + .set({ + executionRunId: claimed.id, + executionAgentNameKey: normalizeAgentNameKey(claimedAgent?.name), + executionLockedAt: claimedAt, + updatedAt: claimedAt, + }) + .where( + and( + eq(issues.id, claimedIssueId), + eq(issues.companyId, claimed.companyId), + // Mention/context runs can touch an issue, but only the current assignee + // owns the issue execution lock shown as the active run. + eq(issues.assigneeAgentId, claimed.agentId), + ["native_safe_replacement", "native_provider_overloaded"].includes(claimed.scheduledRetryReason ?? "") + ? or( + isNull(issues.checkoutRunId), + eq(issues.checkoutRunId, claimed.id), + ) + : undefined, + or( + isNull(issues.executionRunId), + eq(issues.executionRunId, claimed.id), + ), + ), + ); + } + + return claimed; + } + + // startNextQueuedRunForAgent checks admission suppression once, then claims + // runs (sets status "running"), then dispatches each to executeRun, which + // checks suppression again before it does any work. Suppression (task + // drain, worktree mode, a database restore) can start in the gap between + // those two checks. When executeRun's check catches that, the run is + // already claimed — release it back to "queued" so it does not keep a + // running execution lock that nothing will ever process. This runs inside + // the same promise startNextQueuedRunForAgent already tracks in + // activeRunExecutionPromises, so getTaskDrainStatus() keeps reporting the + // run as active until the release finishes. + // + // The run row, the wakeup request, and the issue execution lock all guard + // the same claim, so one transaction commits all three writes together. A + // partial write (for example the run flips to "queued" but the wakeup or + // issue update then fails) would let the execution promise clear from + // activeRunExecutionPromises while the wakeup stayed "claimed" or the + // issue stayed locked to a queued run — task-drain status would then read + // quiescent while the database still held part of the old claim. + async function releaseRunClaimedJustBeforeSuppression(runId: string) { + const now = new Date(); + await db.transaction(async (tx) => { + const released = await tx + .update(heartbeatRuns) + .set({ + status: "queued", + startedAt: null, + responsibleUserId: null, + updatedAt: now, + }) + .where( + and(eq(heartbeatRuns.id, runId), eq(heartbeatRuns.status, "running")), + ) + .returning() + .then((rows) => rows[0] ?? null); + if (!released) return; + + if (released.wakeupRequestId) { + await tx + .update(agentWakeupRequests) + .set({ status: "queued", claimedAt: null, updatedAt: now }) + .where(and(eq(agentWakeupRequests.id, released.wakeupRequestId), ne(agentWakeupRequests.status, "cancelled"))); + } + + const context = parseObject(released.contextSnapshot); + const issueId = readNonEmptyString(context.issueId); + if (issueId) { + await tx + .update(issues) + .set({ + executionRunId: null, + executionAgentNameKey: null, + executionLockedAt: null, + updatedAt: now, + }) + .where( + and( + eq(issues.id, issueId), + eq(issues.companyId, released.companyId), + eq(issues.executionRunId, released.id), + ), + ); + } + }); + } + + async function cancelQueuedRunForBlockedDependencies( + run: typeof heartbeatRuns.$inferSelect, + issueId: string, + unresolvedBlockerIssueIds: string[], + ) { + const now = new Date(); + const reason = + "Cancelled because issue dependencies are still blocked; Paperclip will wake the assignee when blockers resolve"; + const cancelled = await setRunStatus(run.id, "cancelled", { + finishedAt: now, + error: reason, + errorCode: "issue_dependencies_blocked", + resultJson: { + ...parseObject(run.resultJson), + stopReason: "issue_dependencies_blocked", + effectiveTimeoutSec: 0, + timeoutConfigured: false, + timeoutSource: "dependency_gate", + timeoutFired: false, + }, + }); + if (!cancelled) return null; + + await setWakeupStatus(run.wakeupRequestId, "skipped", { + finishedAt: now, + error: reason, + }); + + await db + .update(issues) + .set({ + executionRunId: null, + executionAgentNameKey: null, + executionLockedAt: null, + updatedAt: now, + }) + .where( + and( + eq(issues.companyId, run.companyId), + eq(issues.id, issueId), + eq(issues.executionRunId, run.id), + ), + ); + + await appendRunEvent(cancelled, { + eventType: "lifecycle", + stream: "system", + level: "warn", + message: reason, + payload: { + issueId, + unresolvedBlockerIssueIds, + }, + }); + + return cancelled; + } + + async function resumeQueuedRuns() { + if ((await getSchedulingSuppression()).suppressed) return; + await resumeExecutionWaitComments(); + const cutoff = await getWorktreeExecutionCutoff(); + const pendingInterrupts = await db.select({ id: agentWakeupRequests.id, companyId: agentWakeupRequests.companyId }) + .from(agentWakeupRequests).innerJoin(companies, eq(companies.id, agentWakeupRequests.companyId)) + .where(and(eq(agentWakeupRequests.status, "deferred_issue_execution"), + eq(companies.status, "active"), + sql`${agentWakeupRequests.payload}->'queuedCommentInterrupt' is not null`, + lte(agentWakeupRequests.updatedAt, new Date(Date.now() - 30_000)), + cutoff ? gte(agentWakeupRequests.requestedAt, cutoff) : undefined)) + .orderBy(asc(agentWakeupRequests.updatedAt)).limit(50); + for (const wake of pendingInterrupts) { + await db.update(agentWakeupRequests).set({ updatedAt: new Date() }).where(and( + eq(agentWakeupRequests.id, wake.id), eq(agentWakeupRequests.status, "deferred_issue_execution"), + )); + await resumeQueuedCommentInterrupt(wake.companyId, wake.id).catch(err => { + logger.warn({ err, queueId: wake.id }, "failed to resume interrupted comment queue"); + }); + } + // A server restart or a message/cleanup race can leave a deferred wake + // after its owner has released the issue lock. Revisit it through the same + // release admission, so recovery holds and operator Stops still apply. + const strandedQueues = await db.select({ wake: agentWakeupRequests }) + .from(agentWakeupRequests) + .innerJoin(issues, and(eq(issues.companyId, agentWakeupRequests.companyId), + sql`${issues.id}::text = ${agentWakeupRequests.payload}->>'issueId'`, + eq(issues.assigneeAgentId, agentWakeupRequests.agentId))) + .innerJoin(companies, and(eq(companies.id, issues.companyId), eq(companies.status, "active"))) + .where(and(eq(agentWakeupRequests.status, "deferred_issue_execution"), + isNull(issues.executionRunId), + or(and( + sql`jsonb_typeof(${agentWakeupRequests.payload} #> '{_paperclipWakeContext,wakeCommentIds}') = 'array'`, + sql`${agentWakeupRequests.payload} #> '{_paperclipWakeContext,wakeCommentIds}' <> '[]'::jsonb`, + ), sql`${agentWakeupRequests.payload}->>'mutation' = 'interaction'`), + sql`${agentWakeupRequests.payload}->'queuedCommentInterrupt' is null`, + cutoff ? gte(agentWakeupRequests.requestedAt, cutoff) : undefined)) + .orderBy(asc(agentWakeupRequests.updatedAt)).limit(50); + for (const { wake } of strandedQueues) { + if (!queuedCommentIdsFromWakePayload(wake.payload).length && + !await readQueuedInteractionResponse(db, wake.companyId, String(wake.payload?.issueId), wake.payload)) continue; + const [latest] = await db.select().from(heartbeatRuns).where(and( + eq(heartbeatRuns.companyId, wake.companyId), eq(heartbeatRuns.agentId, wake.agentId), + sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${String(wake.payload?.issueId)}`, + )).orderBy(desc(heartbeatRuns.createdAt), desc(heartbeatRuns.id)).limit(1); + await db.update(agentWakeupRequests).set({ updatedAt: new Date() }).where(and( + eq(agentWakeupRequests.id, wake.id), eq(agentWakeupRequests.status, "deferred_issue_execution"), + )); + if (!latest || latest.runtimeMode !== "legacy" || !isHeartbeatRunTerminalStatus(latest.status)) continue; + const cancelledAdmission = latest.status === "cancelled" && !latest.startedAt && + latest.errorCode === "execution_reconciliation_required"; + if ((latest.status !== "cancelled" || cancelledAdmission) && await getExecutionBlocker(db, wake.companyId, String(wake.payload?.issueId))) { + await resumeSavedLegacyComments(wake.companyId, wake.id).catch(err => { + logger.warn({ err, queueId: wake.id }, "failed to deliver saved legacy comment after recovery stopped"); + }); + } + await releaseIssueExecutionAndPromote(latest, { suppressImmediateRecovery: true }).catch(err => { + logger.warn({ err, queueId: wake.id }, "failed to promote stranded legacy comments"); + }); + } + + // The cancellation marker is durable intent. Retry while its exact queue + // is still deferred, including after a failed cleanup promotion or restart. + // Normal admission still checks process ownership, leases, pauses, and scope. + const interruptedQueues = await db + .select({ id: heartbeatRuns.id, companyId: heartbeatRuns.companyId }) + .from(agentWakeupRequests) + .innerJoin(heartbeatRuns, and( + sql`${heartbeatRuns.resultJson}->>'queuedCommentInterruptQueueId' = ${agentWakeupRequests.id}::text`, + eq(heartbeatRuns.companyId, agentWakeupRequests.companyId), + eq(heartbeatRuns.agentId, agentWakeupRequests.agentId), + )) + .innerJoin(companies, eq(companies.id, heartbeatRuns.companyId)) + .where(and( + eq(agentWakeupRequests.status, "deferred_issue_execution"), + eq(heartbeatRuns.status, "cancelled"), + eq(heartbeatRuns.runtimeMode, "legacy"), + eq(companies.status, "active"), + cutoff ? gte(heartbeatRuns.createdAt, cutoff) : undefined, + )); + for (const run of interruptedQueues) { + await releaseIssueExecutionAndPromote(run, { suppressImmediateRecovery: true }).catch((err) => { + logger.error({ err, runId: run.id }, "failed to retry interrupted comment queue"); + }); + } + + const queuedRuns = await db + .select({ agentId: heartbeatRuns.agentId }) + .from(heartbeatRuns) + .innerJoin(companies, eq(companies.id, heartbeatRuns.companyId)) + .where( + and( + eq(heartbeatRuns.status, "queued"), + eq(companies.status, "active"), + cutoff ? gte(heartbeatRuns.createdAt, cutoff) : undefined, + ), + ); + + const agentIds = [...new Set(queuedRuns.map((r) => r.agentId))]; + for (const agentId of agentIds) { + await startNextQueuedRunForAgent(agentId); + } + } + + // A 403 from claimQueuedRun comes from the run's own persisted identity + // (an unverifiable interrupt receipt, a manual wake with no user). Those rows + // do not change, so the claim fails the same way on every pass and restart. + function isPermanentClaimRejection(err: unknown): err is HttpError { + return err instanceof HttpError && err.status === 403; + } + + // Other 4xx rejections can clear later (a responsible user gets assigned, a + // conflicting claim finishes). Keep the run queued, but do not let it stop + // the rest of the queue or startup recovery. + function isDeferrableClaimRejection(err: unknown): err is HttpError { + return err instanceof HttpError && err.status >= 400 && err.status < 500; + } + + // Settle runs that can never be claimed. Letting the error escape stalls the + // agent's queue and, during startup recovery, stops the server from booting. + async function cancelRejectedQueuedRuns( + rejected: Array<{ run: typeof heartbeatRuns.$inferSelect; err: HttpError }>, + ) { + for (const { run, err } of rejected) { + logger.warn( + { err, runId: run.id, agentId: run.agentId, companyId: run.companyId }, + "cancelling queued heartbeat run whose claim was rejected", + ); + try { + await cancelRunInternal( + run.id, + `Cancelled because the queued run cannot be claimed: ${err.message}`, + { errorCode: "queued_run_claim_rejected" }, + ); + } catch (cancelErr) { + logger.error( + { err: cancelErr, runId: run.id }, + "failed to cancel queued heartbeat run whose claim was rejected; it stays queued for the next recovery pass", + ); + } + } + } + + async function startNextQueuedRunForAgent(agentId: string) { + if ((await getSchedulingSuppression()).suppressed) return []; + const cutoff = await getWorktreeExecutionCutoff(); + // Cancelled after the start lock is released: cancelRunInternal promotes the + // agent's next queued run, which takes this same lock. + const rejectedClaims: Array<{ run: typeof heartbeatRuns.$inferSelect; err: HttpError }> = []; + + return withAgentStartLock(agentId, async () => { + const agent = await getAgent(agentId); + if (!agent) return []; + const invokability = await getAgentInvokability(agent); + if (!invokability.invokable) { + if (shouldCancelRunsForNonInvokableAgent(invokability)) { + await cancelActiveForAgentInternal( + agentId, + `Cancelled because the agent is not invokable: ${invokability.reason}`, + ); + } + return []; + } + const policy = parseHeartbeatPolicy(agent); + const runningCount = await countRunningRunsForAgent(agentId); + const availableSlots = Math.max( + 0, + policy.maxConcurrentRuns - runningCount, + ); + if (availableSlots <= 0) return []; + + const queuedRuns = await db + .select() + .from(heartbeatRuns) + .where( + and( + eq(heartbeatRuns.agentId, agentId), + eq(heartbeatRuns.status, "queued"), + cutoff ? gte(heartbeatRuns.createdAt, cutoff) : undefined, + ), + ) + .orderBy(asc(heartbeatRuns.createdAt)); + if (queuedRuns.length === 0) return []; + + const dependencyReadiness = await listQueuedRunDependencyReadiness( + agent.companyId, + queuedRuns, + ); + const queuedIssueIds = [ + ...new Set( + queuedRuns + .map((run) => + readNonEmptyString(parseObject(run.contextSnapshot).issueId), + ) + .filter((issueId): issueId is string => Boolean(issueId)), + ), + ]; + const issueRows = await db + .select({ + id: issues.id, + status: issues.status, + priority: issues.priority, + }) + .from(issues) + .where( + queuedIssueIds.length > 0 + ? and( + eq(issues.companyId, agent.companyId), + inArray(issues.id, queuedIssueIds), + ) + : sql`false`, + ); + const issueById = new Map(issueRows.map((row) => [row.id, row])); + const companyAgents = await listCompanyAgentOrgRows(agent.companyId); + const prioritizedRuns = [...queuedRuns].sort((left, right) => { + const leftIssueId = readNonEmptyString( + parseObject(left.contextSnapshot).issueId, + ); + const rightIssueId = readNonEmptyString( + parseObject(right.contextSnapshot).issueId, + ); + const leftReadiness = leftIssueId + ? dependencyReadiness.get(leftIssueId) + : null; + const rightReadiness = rightIssueId + ? dependencyReadiness.get(rightIssueId) + : null; + const leftReady = leftIssueId + ? (leftReadiness?.isDependencyReady ?? true) + : true; + const rightReady = rightIssueId + ? (rightReadiness?.isDependencyReady ?? true) + : true; + const leftIssue = leftIssueId ? issueById.get(leftIssueId) : null; + const rightIssue = rightIssueId ? issueById.get(rightIssueId) : null; + const leftRank = leftIssueId + ? leftReady + ? leftIssue?.status === "in_progress" + ? 0 + : 1 + : 3 + : 2; + const rightRank = rightIssueId + ? rightReady + ? rightIssue?.status === "in_progress" + ? 0 + : 1 + : 3 + : 2; + if (leftRank !== rightRank) return leftRank - rightRank; + const leftPriorityRank = issueRunPriorityRank(leftIssue?.priority); + const rightPriorityRank = issueRunPriorityRank(rightIssue?.priority); + if (leftPriorityRank !== rightPriorityRank) + return leftPriorityRank - rightPriorityRank; + return left.createdAt.getTime() - right.createdAt.getTime(); + }); + + const claimedRuns: Array = []; + for (const queuedRun of prioritizedRuns) { + if (claimedRuns.length >= availableSlots) break; + let claimed: typeof heartbeatRuns.$inferSelect | null; + try { + claimed = await claimQueuedRun(queuedRun, companyAgents); + } catch (err) { + if (isPermanentClaimRejection(err)) { + rejectedClaims.push({ run: queuedRun, err }); + continue; + } + if (!isDeferrableClaimRejection(err)) throw err; + logger.warn( + { err, runId: queuedRun.id, agentId: queuedRun.agentId, companyId: queuedRun.companyId }, + "queued heartbeat run claim was rejected; leaving it queued for the next recovery pass", + ); + continue; + } + if (claimed) claimedRuns.push(claimed); + } + if (claimedRuns.length === 0) return []; + + for (const claimedRun of claimedRuns) { + const execution = executeRun(claimedRun.id).catch((err) => { + logger.error( + { err, runId: claimedRun.id }, + "queued heartbeat execution failed", + ); + }); + // Register the in-flight execution so drainActiveRunExecutions() can await + // it. executeRun resolves only after its finally block finishes flushing + // run rows/events, so awaiting this promise guarantees the run's writes + // have landed before a caller (e.g. a test's afterEach) mutates the DB. + activeRunExecutionPromises.add(execution); + void execution.finally(() => { + // drainActiveRunExecutions loops on activeRunExecutionPromises.size, + // so an entry that never clears here would hang it forever. + activeRunExecutionPromises.delete(execution); + }); + } + return claimedRuns; + }).finally(() => cancelRejectedQueuedRuns(rejectedClaims)); + } + + // Public wakeup entry point. Callers dispatch it fire-and-forget, so register + // the promise in activeWakeupPromises before it starts its asynchronous + // prologue. drainActiveRunExecutions can then await a wake that is still before + // run registration. Internal callers reference enqueueWakeup directly and + // already await it, so they do not need this registration. + function trackWakeup( + agentId: string, + opts: WakeupOptions = {}, + ): ReturnType { + const promise = enqueueWakeup(agentId, opts); + activeWakeupPromises.add(promise); + void promise + .catch(() => {}) + .finally(() => { + activeWakeupPromises.delete(promise); + }); + return promise; + } + + async function enqueueWakeup(agentId: string, opts: WakeupOptions = {}, executionWaitRequestId?: string) { + const source = opts.source ?? "on_demand"; + const triggerDetail = opts.triggerDetail ?? null; + const contextSnapshot: Record = { + ...(opts.contextSnapshot ?? {}), + }; + const reason = opts.reason ?? null; + let payload = opts.payload ? { ...opts.payload } : null; + // Only the board queue route can record interruption authority on an + // existing receipt. Never accept this internal marker from a wake caller. + if (payload) { + delete payload.queuedCommentInterrupt; + delete payload.manualUserWake; + } + if (opts.manualUserWake) { + if (opts.requestedByActorType !== "user" || !opts.requestedByActorId || opts.failedRunId) { + throw new HttpError(403, "Manual wake requires an authenticated user"); + } + payload = { ...payload, manualUserWake: true }; + } + const executionReconciliationWake = + contextSnapshot.source === "execution.reconciled" || + opts.idempotencyKey?.startsWith("execution-reconciliation:") === true; + const { + contextSnapshot: enrichedContextSnapshot, + issueIdFromPayload, + taskKey, + wakeCommentId, + } = enrichWakeContextSnapshot({ + contextSnapshot, + reason, + source, + triggerDetail, + payload, + }); + // Keep each request's own server-derived origin, including coalesced wakes. + // A run's merged context cannot establish which caller authored one receipt. + // Overwrite caller-supplied nested context rather than trusting it. + payload = { ...payload, [DEFERRED_WAKE_CONTEXT_KEY]: { ...enrichedContextSnapshot } }; + let issueId = + readNonEmptyString(enrichedContextSnapshot.issueId) ?? issueIdFromPayload; + if (executionReconciliationWake && !issueId) return null; + + let agent = await getAgent(agentId); + if (!agent) throw notFound("Agent not found"); + // Mentions only annotate comments. Ignore legacy callers before creating + // a run or deferred request; assignment and review have their own wakes. + if (reason === "issue_comment_mentioned" || enrichedContextSnapshot.wakeReason === "issue_comment_mentioned") return null; + if (issueId) { + const conversation = await getIssueExecutionContext(agent.companyId, issueId); + if (reason === "issue_children_completed" && conversation?.originKind === "onboarding_first_task") enrichedContextSnapshot.onboardingCompletion = true; + if (isConversation(conversation)) { + if (opts.manualUserWake && conversation!.conversationUserId !== opts.requestedByActorId) { + throw new HttpError(403, "Only the conversation owner can start a chat run"); + } + if (isConversationExecutionWake(conversation, reason ?? readNonEmptyString(enrichedContextSnapshot.wakeReason))) return null; + if (agent.id !== conversation!.conversationAgentId) return null; + if (!(await instanceSettings.getExperimental()).enableAgentChat) return null; + if (!wakeCommentId && isWaitingConversation(conversation) && !hasInteractionContinuationWakeContext(enrichedContextSnapshot) && reason !== CHAT_COMPLETION_WAKE_REASON) return null; + } + } + if (agent.adapterType === "paperclip_runner") { + const oldConfig = parseObject(agent.adapterConfig); + const nextConfig = normalizeLegacyRunnerProvider(oldConfig); + if (nextConfig !== oldConfig) { + await agentService(db).update( + agent.id, + { adapterConfig: nextConfig }, + { + recordRevision: { + source: "normalize_runner_provider", + createdByAgentId: null, + createdByUserId: null, + }, + }, + ); + await logActivity(db, { + companyId: agent.companyId, + actorType: "system", + actorId: "heartbeat", + action: "agent.updated", + entityType: "agent", + entityId: agent.id, + details: { provider: "codex", reason: "native_codex_provider" }, + }); + agent = (await getAgent(agentId))!; + } + } + + if (opts.failedRunId) { + const failed = await getRun(opts.failedRunId, { includeExecutionEvidence: true }); + if (opts.requestedByActorType !== "user" || !opts.requestedByActorId || + reason !== "retry_failed_run" || source !== "on_demand" || triggerDetail !== "manual" || + !failed || failed.companyId !== agent.companyId || failed.agentId !== agentId || + !(await canRetryStoppedRun(db, failed)) || + (failed.nativeIssueId ?? readNonEmptyString(failed.contextSnapshot?.issueId)) !== issueId) { + throw conflict("The selected failed run cannot be retried for this task."); + } + if (!activeRunExecutions.has(failed.id) && !adapterExecutionControls.has(failed.id)) { + await sweepPendingCleanupLeases({ explicitRetry: { + companyId: failed.companyId, runId: failed.id, actorId: opts.requestedByActorId, + } }); + } + if (isConversationAdapter(agent.adapterType) || agent.adapterType === "paperclip_runner") { + enrichedContextSnapshot.previousRunId = failed.id; + enrichedContextSnapshot.forceFreshSession = true; + } + } + + const durableRequest = opts.durableChatRequest; + if (durableRequest) { + assertDurableChatWakeupRequest(durableRequest, { + agentId, + companyId: agent.companyId, + issueId, + commentId: wakeCommentId ?? null, + requestedByActorType: opts.requestedByActorType, + requestedByActorId: opts.requestedByActorId, + }); + opts = { ...opts, idempotencyKey: durableRequest.idempotencyKey }; + } + if ( + Object.hasOwn(enrichedContextSnapshot, "chatFailedRunRetry") && + !durableRequest?.failedRunRetry + ) { + throw new FailedChatRunRetryAuthorizationError(); + } + if (durableRequest?.failedRunRetry) { + opts = { ...opts, allowRunCoalescing: false }; + } + const dotRequest = opts.durableDotRequest; + if (dotRequest && (durableRequest || dotRequest.agentId !== agentId || + dotRequest.companyId !== agent.companyId || dotRequest.issueId !== issueId || + dotRequest.requestId !== payload?.dotRequestId || source !== "assignment" || + opts.requestedByActorType !== "agent" || opts.requestedByActorId !== agentId || + agent.adapterType !== "paperclip_runner" || parseObject(agent.adapterConfig).provider !== "openai_dot" || + opts.idempotencyKey !== dotRequest.idempotencyKey)) { + throw conflict("Dot work request does not match its admission authority."); + } + const receiptRequest = durableRequest ?? dotRequest; + const durableReceiptFields = receiptRequest + ? { id: receiptRequest.id, requestedAt: receiptRequest.requestedAt } + : {}; + const existingDurableReceipt = async (queryDb: Db) => { + if (!receiptRequest) return null; + const receipt = await queryDb + .select() + .from(agentWakeupRequests) + .where(eq(agentWakeupRequests.id, receiptRequest.id)) + .limit(1) + .then((rows) => rows[0] ?? null); + if (receipt && durableRequest) assertDurableChatWakeupReceipt(durableRequest, receipt); + if (receipt && dotRequest && (receipt.companyId !== dotRequest.companyId || + receipt.agentId !== dotRequest.agentId || receipt.source !== "assignment" || + receipt.requestedByActorType !== "agent" || receipt.requestedByActorId !== dotRequest.agentId || + receipt.idempotencyKey !== dotRequest.idempotencyKey || receipt.payload?.issueId !== dotRequest.issueId || + receipt.payload?.dotRequestId !== dotRequest.requestId)) { + throw conflict("requestId was reused for another task."); + } + return receipt; + }; + const priorReceipt = await existingDurableReceipt(db); + if (priorReceipt) { + // Replaying an admission receipt is not fresh authority to dispatch it. + // The normal queue owns dispatch and its current execution-policy checks. + return priorReceipt.runId ? getRun(priorReceipt.runId) : null; + } + + const agentDebug = parseObject(parseObject(agent.runtimeConfig).debug); + const runDebug = parseObject(enrichedContextSnapshot.debug); + if ( + agentDebug.providerTrace === "raw" && + runDebug.providerTrace !== "raw" + ) { + enrichedContextSnapshot.debug = { + ...runDebug, + providerTrace: "raw", + }; + enrichedContextSnapshot.providerTraceRequestedBy = `agent:${agent.id}:debug-setting`; + enrichedContextSnapshot.providerTraceRequestSource = + "agent_debug_setting"; + } + + // Automatic signals are replaceable; user input and interaction delivery + // retain distinct durable receipts even when the same gate blocks them. + const coalesceExecutionWait = + opts.requestedByActorType === "system" && + !receiptRequest && + !wakeCommentId && + queuedCommentIdsFromRunContext(enrichedContextSnapshot).length === 0 && + !isInteractionResolutionWakePayload(payload ?? {}) && + !hasInteractionContinuationWakeContext(enrichedContextSnapshot); + const writeSkippedRequest = async ( + skipReason: string, + patch: Partial = {}, + waitCondition?: Record, + ) => { + if (executionWaitRequestId) { + await db.update(agentWakeupRequests).set({ + payload: sql`jsonb_set(coalesce(${agentWakeupRequests.payload}, '{}'::jsonb), '{executionWait}', + coalesce(${agentWakeupRequests.payload}->'executionWait', '{}'::jsonb) || ${JSON.stringify({ + reason: skipReason, message: patch.error ?? (skipReason === "issue_tree_hold_active" + ? "This task is paused. Resume it to send your saved message." + : "Waiting for task execution to be enabled. Your message is saved."), + })}::jsonb)`, + updatedAt: new Date(), + }).where(and(eq(agentWakeupRequests.id, executionWaitRequestId), + eq(agentWakeupRequests.companyId, agent.companyId), eq(agentWakeupRequests.agentId, agentId), + eq(agentWakeupRequests.status, "deferred_issue_execution"))); + return { created: false }; + } + const request = { + ...durableReceiptFields, + companyId: agent.companyId, + agentId, + source, + triggerDetail, + reason: skipReason, + payload, + status: "skipped", + requestedByActorType: opts.requestedByActorType ?? null, + requestedByActorId: opts.requestedByActorId ?? null, + idempotencyKey: opts.idempotencyKey ?? null, + finishedAt: new Date(), + ...patch, + }; + if (waitCondition && issueId && isUuidLike(issueId)) { + const waitIssueId = issueId; + return db.transaction(async (tx) => { + await tx.execute(sql`select id from issues where id = ${waitIssueId} and company_id = ${agent.companyId} for update`); + return recordExecutionWait(tx as unknown as Db, { + issueId: waitIssueId, request, condition: waitCondition, coalesce: coalesceExecutionWait, + }); + }); + } + await db.insert(agentWakeupRequests).values(request); + return { created: true }; + }; + const writeSkippedHeartbeatRequest = async ( + skipReason: string, + details: Record, + ) => { + await writeSkippedRequest(skipReason, { + payload: { + ...(payload ?? {}), + heartbeatSkip: details, + }, + }); + }; + + const schedulingSuppression = await getSchedulingSuppression(); + // A task drain holds ADMISSION, not the request. The drain is a + // process-local pre-restart hold, so a wake that arrives while it is + // active still names real work that must run once the process comes + // back: leave it in the durable queue and let the dispatch-side checks + // (startNextQueuedRunForAgent / executeRun) keep it from starting until + // the drain lifts or the restart clears it. Writing it as `skipped` + // here dropped the wake permanently — an accepted plan whose + // continuation wake landed mid-drain left its issue in `todo` with no + // run and no path until a person noticed. + if ( + schedulingSuppression.suppressed && + schedulingSuppression.reason !== "task_drain" + ) { + await writeSkippedHeartbeatRequest("heartbeat.scheduling_suppressed", { + reason: schedulingSuppression.reason, + }); + return null; + } + + const worktreeExecutionCutoff = + opts.requestedByActorType === "user" + ? null + : await getWorktreeExecutionCutoff(); + + const company = await db + .select({ status: companies.status }) + .from(companies) + .where(eq(companies.id, agent.companyId)) + .then((rows) => rows[0] ?? null); + + if (!company || company.status !== "active") { + const companyStatus = company?.status ?? "missing"; + if (opts.requestedByActorType === "user") { + throw conflict("Company is not active", { status: companyStatus }); + } + await writeSkippedRequest("company.inactive", { + error: `Wake suppressed because company status is ${companyStatus}`, + }, { companyStatus }); + return null; + } + + const explicitResumeSession = await resolveExplicitResumeSessionOverride( + agent, + payload, + taskKey, + ); + if (explicitResumeSession) { + enrichedContextSnapshot.resumeFromRunId = + explicitResumeSession.resumeFromRunId; + enrichedContextSnapshot.resumeSessionDisplayId = + explicitResumeSession.sessionDisplayId; + enrichedContextSnapshot.resumeSessionParams = + explicitResumeSession.sessionParams; + if ( + !readNonEmptyString(enrichedContextSnapshot.issueId) && + explicitResumeSession.issueId + ) { + enrichedContextSnapshot.issueId = explicitResumeSession.issueId; + } + if ( + !readNonEmptyString(enrichedContextSnapshot.taskId) && + explicitResumeSession.taskId + ) { + enrichedContextSnapshot.taskId = explicitResumeSession.taskId; + } + if ( + !readNonEmptyString(enrichedContextSnapshot.taskKey) && + explicitResumeSession.taskKey + ) { + enrichedContextSnapshot.taskKey = explicitResumeSession.taskKey; + } + issueId = readNonEmptyString(enrichedContextSnapshot.issueId) ?? issueId; + } + const effectiveTaskKey = + readNonEmptyString(enrichedContextSnapshot.taskKey) ?? taskKey; + const sessionBefore = + explicitResumeSession?.sessionDisplayId ?? + (await resolveSessionBeforeForWakeup(agent, effectiveTaskKey)); + let hasResolvablePriorSessionWorkspace: boolean | null = null; + const resolveHasResolvablePriorSessionWorkspace = async () => { + if (hasResolvablePriorSessionWorkspace !== null) + return hasResolvablePriorSessionWorkspace; + hasResolvablePriorSessionWorkspace = issueId + ? await hasResolvablePriorSessionWorkspaceForWake({ + agent, + contextSnapshot: enrichedContextSnapshot, + taskKey: effectiveTaskKey, + explicitResumeSession, + }) + : false; + return hasResolvablePriorSessionWorkspace; + }; + const continuationAttempt = readContinuationAttempt( + enrichedContextSnapshot.livenessContinuationAttempt, + ); + + let projectId = readNonEmptyString(enrichedContextSnapshot.projectId); + if (!projectId && issueId) { + // Look up by either UUID or identifier (e.g. "ENV-13"), but always scope + // by companyId so a row from another tenant can never be returned even + // when identifiers collide across companies. Guard the UUID arm because + // issues.id is a Postgres uuid column — passing "ENV-13" into eq(issues.id, …) + // would fail with an invalid-input-syntax cast error before the OR is + // evaluated. + const lookupIsUuid = isUuidLike(issueId); + const idMatch = lookupIsUuid + ? or( + eq(issues.id, issueId), + eq(issues.identifier, issueId.toUpperCase()), + ) + : eq(issues.identifier, issueId.toUpperCase()); + const resolvedIssue = await db + .select({ + id: issues.id, + projectId: issues.projectId, + createdAt: issues.createdAt, + }) + .from(issues) + .where(and(eq(issues.companyId, agent.companyId), idMatch)) + .then((rows) => rows[0] ?? null); + if (resolvedIssue) { + if ( + worktreeExecutionCutoff && + resolvedIssue.createdAt < worktreeExecutionCutoff + ) { + await writeSkippedHeartbeatRequest( + "heartbeat.worktree_execution_cutoff", + { + reason: "worktree_execution_cutoff", + cutoff: worktreeExecutionCutoff.toISOString(), + issueId: resolvedIssue.id, + }, + ); + return null; + } + projectId = resolvedIssue.projectId ?? null; + // Canonicalize context to the UUID so downstream lookups always use UUID + if (resolvedIssue.id !== issueId) { + issueId = resolvedIssue.id; + enrichedContextSnapshot.issueId = issueId; + if (readNonEmptyString(enrichedContextSnapshot.taskId)) { + enrichedContextSnapshot.taskId = issueId; + } + } + } + } + // Propagate projectId into context so resolveWorkspaceForRun can bind the + // project workspace even when context.projectId wasn't set by the caller. + if (projectId && !readNonEmptyString(enrichedContextSnapshot.projectId)) { + enrichedContextSnapshot.projectId = projectId; + } + const isolatedWorkspacesEnabled = issueId + ? (await instanceSettings.getExperimental()).enableIsolatedWorkspaces + : false; + let operatorResponsibleUserId: string | null = opts.manualUserWake ? opts.requestedByActorId! : null; + let queuedResponsibleUserIdPromise: Promise | null = null; + const resolveQueuedResponsibleUserId = () => { + if (operatorResponsibleUserId) return Promise.resolve(operatorResponsibleUserId); + queuedResponsibleUserIdPromise ??= (async () => { + const queuedIssueContext = issueId + ? await getIssueExecutionContext(agent.companyId, issueId) + : null; + const queuedRoutineEnvContext = await getRoutineEnvForExecutionIssue( + agent.companyId, + queuedIssueContext, + ); + const queuedResponsibleUserId = + await resolveResponsibleUserIdForRunSeed({ + companyId: agent.companyId, + contextSnapshot: enrichedContextSnapshot, + issueContext: queuedIssueContext, + routineEnvContext: queuedRoutineEnvContext, + requestedByActorType: opts.requestedByActorType ?? null, + requestedByActorId: opts.requestedByActorId ?? null, + source, + triggerDetail, + }); + if (!queuedResponsibleUserId) { + throw new HttpError( + 422, + "Unable to resolve responsible user for heartbeat run dispatch", + { + code: "responsible_user_unresolved", + agentId, + companyId: agent.companyId, + issueId: issueId ?? null, + source, + triggerDetail, + wakeReason: readNonEmptyString( + enrichedContextSnapshot.wakeReason, + ), + }, + ); + } + return queuedResponsibleUserId; + })(); + return queuedResponsibleUserIdPromise; + }; + + const budgetBlock = await budgets.getInvocationBlock( + agent.companyId, + agentId, + { + issueId, + projectId, + }, + ); + if (budgetBlock) { + await writeSkippedRequest("budget.blocked", { error: budgetBlock.reason }, { + scopeType: budgetBlock.scopeType, scopeId: budgetBlock.scopeId, + }); + throw conflict(budgetBlock.reason, { + scopeType: budgetBlock.scopeType, + scopeId: budgetBlock.scopeId, + }); + } + + const invokability = await getAgentInvokability(agent); + if (!invokability.invokable) { + if (opts.requestedByActorType !== "user" || executionWaitRequestId) { + await writeSkippedRequest("agent.not_invokable", { + error: invokability.message, + }, { status: agent.status, reason: invokability.reason }); + } + throw conflict(invokability.message, { + status: agent.status, + reason: invokability.reason, + invalidOrgChain: invokability.invalidOrgChain, + ...invokability.details, + }); + } + + const policy = parseHeartbeatPolicy(agent); + + if (source === "timer" && !policy.enabled) { + await writeSkippedRequest("heartbeat.disabled", {}, { enabled: false }); + return null; + } + if (source !== "timer" && !policy.wakeOnDemand) { + await writeSkippedRequest("heartbeat.wakeOnDemand.disabled", {}, { wakeOnDemand: false }); + return null; + } + + const genericTimerWake = + source === "timer" && + !issueId && + !wakeCommentId && + !readNonEmptyString(enrichedContextSnapshot.taskId) && + !readNonEmptyString(enrichedContextSnapshot.taskKey); + if ( + policy.skipTimerWhenNoActionableWork && + genericTimerWake && + !(await hasActionableTimerWork(agent)) + ) { + await writeSkippedHeartbeatRequest("heartbeat.timer.no_actionable_work", { + reason: + "No assigned todo or in_progress issue requires this agent before timer adapter invocation.", + }); + await markTimerHeartbeatChecked(agentId, source); + return null; + } + + if (issueId) { + const activePauseHold = await treeControlSvc.getActivePauseHoldGate( + agent.companyId, + issueId, + ); + if (activePauseHold) { + const treeHoldInteractionWake = + await isVerifiedIssueTreeControlInteractionWake(db, { + companyId: agent.companyId, + issueId, + agentId, + contextSnapshot: enrichedContextSnapshot, + requestedByActorType: opts.requestedByActorType, + requestedByActorId: opts.requestedByActorId, + }); + + if (!treeHoldInteractionWake) { + const wait = await writeSkippedRequest("issue_tree_hold_active", {}, { + holdId: activePauseHold.holdId, + }); + if (wait.created) await logActivity(db, { + companyId: agent.companyId, + actorType: "system", + actorId: "system", + agentId, + runId: null, + action: "issue.tree_hold_wakeup_deferred", + entityType: "issue", + entityId: issueId, + details: { + holdId: activePauseHold.holdId, + rootIssueId: activePauseHold.rootIssueId, + requestedReason: reason, + source, + triggerDetail, + securityPrinciples: [ + "Complete Mediation", + "Fail Securely", + "Secure Defaults", + ], + }, + }); + return null; + } + + enrichedContextSnapshot.treeHoldInteraction = true; + enrichedContextSnapshot.activeTreeHold = { + holdId: activePauseHold.holdId, + rootIssueId: activePauseHold.rootIssueId, + mode: activePauseHold.mode, + reason: activePauseHold.reason, + releasePolicy: activePauseHold.releasePolicy, + interaction: true, + }; + } + } + + if (issueId) { + // Mention-triggered wakes can request input from another agent, but they must + // still respect the issue execution lock so a second agent cannot start on the + // same issue workspace while the assignee already has a live run. + const agentNameKey = normalizeAgentNameKey(agent.name); + + const cancelledRunsToEmit: (typeof heartbeatRuns.$inferSelect)[] = []; + + const outcome = await db + .transaction(async (tx) => { + await tx.execute( + sql`select id from issues where id = ${issueId} and company_id = ${agent.companyId} for update`, + ); + + if (executionWaitRequestId) { + const [pending] = await tx.select().from(agentWakeupRequests).where(and( + eq(agentWakeupRequests.id, executionWaitRequestId), eq(agentWakeupRequests.companyId, agent.companyId), + eq(agentWakeupRequests.agentId, agentId), eq(agentWakeupRequests.status, "deferred_issue_execution"), + // A user message can join a queue originally created by a + // system wake. Admission validates the saved user comment or board click. + (opts.queuedCommentInterruptId ?? opts.queuedCommentRequestId) === executionWaitRequestId + ? undefined : eq(agentWakeupRequests.requestedByActorType, "user"), + opts.queuedCommentInterruptId === executionWaitRequestId + ? sql`${agentWakeupRequests.payload}->'queuedCommentInterrupt'->>'actorId' = ${opts.requestedByActorId ?? ""}` + : opts.queuedCommentRequestId === executionWaitRequestId ? undefined + : eq(agentWakeupRequests.requestedByActorId, opts.requestedByActorId ?? ""), + sql`${agentWakeupRequests.payload}->>'issueId' = ${issueId}`, + )); + // The issue lock serializes cleanup callbacks and periodic workers. + // An adopted, discarded, or edited receipt is no longer authority. + if (!pending || (!(wakeCommentId && queuedCommentIdsFromWakePayload(pending.payload).includes(wakeCommentId)) && + !(opts.queuedCommentInterruptId && await readQueuedInteractionResponse(tx as unknown as Db, + agent.companyId, issueId, pending.payload)))) { + return { kind: "deferred" as const }; + } + if (opts.queuedCommentRequestId) { + const ids = await undeliveredLegacyUserCommentIds(tx as unknown as Db, + agent.companyId, issueId, agentId, queuedCommentIdsFromWakePayload(pending.payload)); + if (!wakeCommentId || !ids.includes(wakeCommentId)) return { kind: "deferred" as const }; + pending.payload = withQueuedCommentIdsInWakePayload(parseObject(pending.payload), ids); + await tx.update(agentWakeupRequests).set({ payload: pending.payload }).where(and( + eq(agentWakeupRequests.id, pending.id), eq(agentWakeupRequests.companyId, agent.companyId), + )); + } + if (!opts.queuedCommentInterruptId && !opts.queuedCommentRequestId && pending.payload?.manualUserWake === true) { + // A persisted manual wake keeps its actor when an execution wait + // resumes. The locked receipt above has revalidated that actor. + payload = { ...payload, manualUserWake: true }; + operatorResponsibleUserId = opts.requestedByActorId!; + } + if (opts.queuedCommentInterruptId) { + // The locked board receipt supplies execution authority even when + // another user authored the messages. Dispatch revalidates the receipt. + operatorResponsibleUserId = opts.requestedByActorId!; + } + if (opts.queuedCommentInterruptId || opts.queuedCommentRequestId) { + // Edits/discards between the click and dispatch remain authoritative. + Object.assign(enrichedContextSnapshot, withQueuedCommentIdsInRunContext( + enrichedContextSnapshot, queuedCommentIdsFromWakePayload(pending.payload), + )); + } + } + let automaticParentRunId: string | null = null; + if ( + source === "automation" && + opts.requestedByActorType === "system" + ) { + const nativeParent = await readChatControlNativeParent( + tx as unknown as Db, + { companyId: agent.companyId, issueId, agentId }, + opts.requestedByActorId ?? null, + ); + const genericParent = + !opts.requestedByActorId && reason === "issue_continuation_needed" + ? readNonEmptyString(enrichedContextSnapshot.retryOfRunId) + : null; + automaticParentRunId = + nativeParent.kind === "parent" + ? nativeParent.runId + : genericParent; + if (automaticParentRunId) { + const [parent] = await tx + .select({ + agentId: heartbeatRuns.agentId, + context: heartbeatRuns.contextSnapshot, + }) + .from(heartbeatRuns) + .where( + and( + eq(heartbeatRuns.companyId, agent.companyId), + eq(heartbeatRuns.id, automaticParentRunId), + ), + ) + .limit(1); + if ( + parent && + (parseObject(parent.context).issueId ?? + parseObject(parent.context).taskId) === issueId && + parent.agentId !== agentId + ) + automaticParentRunId = null; + } + const proof = + nativeParent.kind === "unresolved" + ? { kind: "unresolved" as const } + : automaticParentRunId + ? await readChatControlRecoveryStop( + tx as unknown as Db, + { + companyId: agent.companyId, + issueId, + agentId, + sourceRunId: automaticParentRunId, + }, + true, + ) + : { kind: "clear" as const }; + if (proof.kind !== "clear") { + await tx + .insert(agentWakeupRequests) + .values({ + companyId: agent.companyId, + agentId, + source, + triggerDetail, + reason: + proof.kind === "stopped" + ? CHAT_CONTROL_RECOVERY_STOP_CODE + : CHAT_CONTROL_RECOVERY_UNRESOLVED_CODE, + payload: { issueId }, + status: "skipped", + requestedByActorType: "system", + requestedByActorId: opts.requestedByActorId ?? null, + idempotencyKey: opts.idempotencyKey ?? null, + finishedAt: new Date(), + }); + return { kind: "skipped" as const }; + } + } + + const durableReceipt = await existingDurableReceipt( + tx as unknown as Db, + ); + if (durableReceipt) + return { kind: "durable" as const, receipt: durableReceipt }; + const failedChatRetry = await authorizeFailedChatRunRetryWake( + db, + tx as unknown as Db, + { + phase: "admission", + wakeupRequestId: durableRequest?.id ?? null, + companyId: agent.companyId, + agentId, + issueId, + contextSnapshot: enrichedContextSnapshot, + }, + ); + if ( + failedChatRetry !== Boolean(durableRequest?.failedRunRetry) || + (failedChatRetry && + durableRequest?.failedRunRetry?.failedRunId !== + readNonEmptyString(enrichedContextSnapshot.retryOfRunId)) + ) { + throw new FailedChatRunRetryAuthorizationError(); + } + if (durableRequest) { + if (issueId !== durableRequest.issueId) + throw new Error("chat_inbound_wakeup_binding_denied"); + await durableRequest.authorize(tx as unknown as Db); + } else { + if ( + source === "on_demand" && + triggerDetail === "manual" && + reason === "retry_failed_run" + ) { + // Generic Board retry does not carry server-authorized failed-run + // lineage. Caller-supplied source/comment/retry markers cannot + // restore it, including for a retired conversation generation. + const [chatBinding] = await tx + .select({ id: chatConversations.id }) + .from(chatConversations) + .innerJoin(chatEndpoints, eq(chatEndpoints.id, chatConversations.endpointId)) + .where( + and( + eq(chatEndpoints.externalExecutionPolicy, "restricted"), + eq(chatConversations.companyId, agent.companyId), + eq(chatConversations.issueId, issueId), + ), + ) + .limit(1); + if (chatBinding) { + throw conflict( + "Retry needs the exact failed chat request and current access. Send the request again in the current connected conversation.", + { code: "chat_failed_run_retry_requires_authorized_context" }, + ); + } + } + const [held] = await tx + .select({ id: issues.id }) + .from(issues) + .where( + and( + eq(issues.id, issueId), + eq(issues.companyId, agent.companyId), + unadmittedChatWakeupCondition(issues.id, issues.companyId), + ), + ) + .limit(1); + if (held) { + throw conflict( + "This task has external chat input that has not been admitted. Check the connection's Activity; let pending input finish, or restore access and send a new authorized message before starting the task.", + { code: "chat_inbound_wakeup_unadmitted", issueId }, + ); + } + } + + const issue = await tx + .select({ + id: issues.id, + companyId: issues.companyId, + identifier: issues.identifier, + conversationAgentId: issues.conversationAgentId, + conversationUserId: issues.conversationUserId, + conversationState: issues.conversationState, + status: issues.status, + statusVersion: issues.statusVersion, + projectId: issues.projectId, + projectWorkspaceId: issues.projectWorkspaceId, + executionWorkspaceId: issues.executionWorkspaceId, + executionWorkspacePreference: issues.executionWorkspacePreference, + executionWorkspaceSettings: issues.executionWorkspaceSettings, + assigneeAgentId: issues.assigneeAgentId, + assigneeUserId: issues.assigneeUserId, + monitorNextCheckAt: issues.monitorNextCheckAt, + monitorWakeRequestedAt: issues.monitorWakeRequestedAt, + executionRunId: issues.executionRunId, + executionAgentNameKey: issues.executionAgentNameKey, + createdAt: issues.createdAt, + }) + .from(issues) + .where( + and( + eq(issues.id, issueId), + eq(issues.companyId, agent.companyId), + ), + ) + .then((rows) => rows[0] ?? null); + + if (!issue) { + await tx.insert(agentWakeupRequests).values({ + ...durableReceiptFields, + companyId: agent.companyId, + agentId, + source, + triggerDetail, + reason: "issue_execution_issue_not_found", + payload, + status: "skipped", + requestedByActorType: opts.requestedByActorType ?? null, + requestedByActorId: opts.requestedByActorId ?? null, + idempotencyKey: opts.idempotencyKey ?? null, + finishedAt: new Date(), + }); + return { kind: "skipped" as const }; + } + + const issueStateGuard = opts.issueStateGuard; + const activeMonitorRun = issueStateGuard?.monitorNextCheckAt === undefined ? null + : await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and( + eq(heartbeatRuns.companyId, issue.companyId), eq(heartbeatRuns.nativeIssueId, issue.id), + eq(heartbeatRuns.runtimeMode, "native"), inArray(heartbeatRuns.status, ["queued", "running", "scheduled_retry"]), + )).limit(1).then(rows => rows[0] ?? null); + if ( + issueStateGuard && + (!issueStateGuard.statuses.includes(issue.status) || + issue.assigneeAgentId !== issueStateGuard.assigneeAgentId || + (issueStateGuard.statusVersion !== undefined && issue.statusVersion !== issueStateGuard.statusVersion) || + (issueStateGuard.monitorNextCheckAt !== undefined && ( + activeMonitorRun !== null || issue.assigneeUserId !== null || + issue.monitorNextCheckAt?.toISOString() !== issueStateGuard.monitorNextCheckAt || + issue.monitorWakeRequestedAt?.toISOString() !== issueStateGuard.monitorWakeRequestedAt + ))) + ) { + // A deferred monitor retains its schedule; do not create a receipt + // that could suppress its next admission attempt. + if (issueStateGuard.monitorNextCheckAt !== undefined) return { kind: "skipped" as const }; + await tx.insert(agentWakeupRequests).values({ + ...durableReceiptFields, + companyId: agent.companyId, + agentId, + source, + triggerDetail, + reason: "issue_state_guard_mismatch", + payload: { + ...(payload ?? {}), + heartbeatSkip: { + reason: + "Issue status or assignee changed before the wake could be queued.", + issueId: issue.id, + expectedStatuses: issueStateGuard.statuses, + actualStatus: issue.status, + expectedAssigneeAgentId: issueStateGuard.assigneeAgentId, + actualAssigneeAgentId: issue.assigneeAgentId, + }, + }, + status: "skipped", + requestedByActorType: opts.requestedByActorType ?? null, + requestedByActorId: opts.requestedByActorId ?? null, + idempotencyKey: opts.idempotencyKey ?? null, + finishedAt: new Date(), + }); + return { kind: "skipped" as const }; + } + + if (opts.failedRunId) { + // The issue lock makes double-clicks and network retries adopt the + // same successor, including after it has already finished. + const [previousRetry] = await tx.select().from(heartbeatRuns).where(and( + eq(heartbeatRuns.companyId, issue.companyId), eq(heartbeatRuns.agentId, agentId), + eq(heartbeatRuns.retryOfRunId, opts.failedRunId), + sql`${heartbeatRuns.contextSnapshot}->>'wakeReason' = 'retry_failed_run'`, + )).orderBy(desc(heartbeatRuns.createdAt)).limit(1); + if (previousRetry) return { kind: "replayed" as const, run: previousRetry }; + } + + let reconciledSourceRunId: string | null = null; + let reconciledRestoreRetryCount: number | null = null; + if (executionReconciliationWake) { + const actionId = readNonEmptyString( + enrichedContextSnapshot.recoveryActionId, + ); + if ( + !actionId || + !isUuidLike(actionId) || + source !== "automation" || + triggerDetail !== "system" || + reason !== "issue_recovery_action_restored" || + opts.requestedByActorType !== "system" || + opts.requestedByActorId !== "execution-recovery" || + opts.idempotencyKey !== `execution-reconciliation:${actionId}` || + enrichedContextSnapshot.source !== "execution.reconciled" || + enrichedContextSnapshot.forceFreshSession !== true || + payload?.issueId !== issue.id || + payload?.recoveryActionId !== actionId || + issue.assigneeAgentId !== agentId || + ["done", "cancelled"].includes(issue.status) + ) + return { kind: "skipped" as const }; + + // The issue lock serializes all admissions for this source. Validate + // the durable operator decision, then reconcile a prior queue commit + // before considering a new wake (including a now-terminal successor). + const [action] = await tx + .select() + .from(issueRecoveryActions) + .where( + and( + eq(issueRecoveryActions.companyId, issue.companyId), + eq(issueRecoveryActions.sourceIssueId, issue.id), + eq(issueRecoveryActions.id, actionId), + ), + ) + .for("update"); + const decision = parseObject( + action?.evidence.executionReconciliation, + ); + const sourceRunId = readNonEmptyString(decision.runId); + if ( + !action || + action.status !== "resolved" || + action.kind !== "active_run_watchdog" || + action.returnOwnerAgentId !== agentId || + !sourceRunId || + !isUuidLike(sourceRunId) || + decision.providerStopped !== true || + !["completed", "not_performed", "mixed"].includes( + String(decision.actionOutcome), + ) || + !readNonEmptyString(decision.outcomeEvidence) || + enrichedContextSnapshot.previousRunId !== sourceRunId || + enrichedContextSnapshot.retryOfRunId !== sourceRunId || + !["pending", "delivered"].includes( + String(action.evidence.continuationDelivery), + ) + ) + return { kind: "skipped" as const }; + + const [existingWake] = await tx + .select() + .from(agentWakeupRequests) + .where( + and( + eq(agentWakeupRequests.companyId, issue.companyId), + eq(agentWakeupRequests.agentId, agentId), + eq(agentWakeupRequests.idempotencyKey, opts.idempotencyKey), + ne(agentWakeupRequests.status, "skipped"), + ), + ) + .orderBy(asc(agentWakeupRequests.requestedAt)) + .limit(1); + if (existingWake) { + if ( + existingWake.payload?.issueId !== issue.id || + existingWake.payload?.recoveryActionId !== action.id || + existingWake.requestedByActorType !== "system" || + existingWake.requestedByActorId !== "execution-recovery" || + !existingWake.runId + ) + return { kind: "deferred" as const }; + const [existingRun] = await tx + .select() + .from(heartbeatRuns) + .where( + and( + eq(heartbeatRuns.companyId, issue.companyId), + eq(heartbeatRuns.agentId, agentId), + eq(heartbeatRuns.id, existingWake.runId), + ), + ); + if ( + !existingRun || + existingRun.contextSnapshot?.issueId !== issue.id || + existingRun.contextSnapshot?.recoveryActionId !== action.id || + existingRun.contextSnapshot?.previousRunId !== sourceRunId + ) + return { kind: "deferred" as const }; + return { kind: "replayed" as const, run: existingRun }; + } + if (action.evidence.continuationDelivery !== "pending") + return { kind: "skipped" as const }; + const [reconciledRun] = await tx.select().from(heartbeatRuns).where(and( + eq(heartbeatRuns.companyId, issue.companyId), eq(heartbeatRuns.id, sourceRunId), + )); + if (hasWorkspaceRestoreFailure(reconciledRun?.resultJson)) { + if ((readNonEmptyString(decision.workspaceRepairEvidence)?.length ?? 0) < 20) + return { kind: "skipped" as const }; + // Repair does not reset the remaining automatic retry budget. + reconciledRestoreRetryCount = executionFailureRetryCount(reconciledRun!); + } + reconciledSourceRunId = sourceRunId; + } + + let continuationWait = { reason: "execution_recovery", message: "Waiting for execution recovery. Your message is saved." }; + const deferBlockedExecution = async ( + executionBlocker: NonNullable>>, + ) => { + const condition = { recoveryActionId: executionBlocker.recoveryActionId, ...continuationWait }; + if (executionWaitRequestId) { + await tx.update(agentWakeupRequests).set({ + payload: sql`jsonb_set(coalesce(${agentWakeupRequests.payload}, '{}'::jsonb), '{executionWait}', ${JSON.stringify(condition)}::jsonb)`, + updatedAt: new Date(), + }).where(eq(agentWakeupRequests.id, executionWaitRequestId)); + return { kind: "deferred" as const }; + } + if (durableRequest || wakeCommentId || + hasInteractionContinuationWakeContext(enrichedContextSnapshot) || + readNonEmptyString(enrichedContextSnapshot.nativeStatusWakeIntentId)) { + await tx.insert(agentWakeupRequests).values({ + ...durableReceiptFields, + companyId: agent.companyId, agentId, source, triggerDetail, reason, + payload: withQueuedCommentIdsInWakePayload({ + ...payload, + issueId: issue.id, + [DEFERRED_WAKE_CONTEXT_KEY]: enrichedContextSnapshot, + executionWait: condition, + }, [...new Set([ + ...queuedCommentIdsFromRunContext(enrichedContextSnapshot), + ...(wakeCommentId ? [wakeCommentId] : []), + ])]), + status: "deferred_issue_execution", + requestedByActorType: opts.requestedByActorType ?? null, + requestedByActorId: opts.requestedByActorId ?? null, + idempotencyKey: opts.idempotencyKey ?? null, + }); + } else { + await recordExecutionWait(tx as unknown as Db, { + issueId: issue.id, condition, coalesce: coalesceExecutionWait, + request: { + ...durableReceiptFields, + companyId: agent.companyId, agentId, source, triggerDetail, + reason: "execution_reconciliation_required", + error: executionBlocker.nextAction, + payload, + requestedByActorType: opts.requestedByActorType ?? null, + requestedByActorId: opts.requestedByActorId ?? null, + idempotencyKey: opts.idempotencyKey ?? null, + }, + }); + } + return { kind: "deferred" as const }; + }; + const explicitContinuationRunId = randomUUID(); + const executionBlocker = await getExecutionBlocker( + tx as unknown as Db, issue.companyId, issue.id, + { conversationResetCommentId: opts.requestedByActorType === "user" ? wakeCommentId : null }, + ); + // Prove eligibility without retiring the hold. Later gates can still + // decline this wake; hold retirement and successor creation stay atomic. + // A bound chat request has already rechecked its current principal + // above. Treat its new user message like a board comment, but keep + // failed-run retry actions on their separate exact-request path. + if (executionBlocker && !(await admitExplicitNativeContinuation({ + db: tx as unknown as Db, companyId: issue.companyId, issueId: issue.id, + agentId, actorType: opts.requestedByActorType, actorId: opts.requestedByActorId, + reason: durableRequest && !failedChatRetry ? "issue_commented" : reason, + commentId: wakeCommentId ?? null, failedRunId: opts.failedRunId, successorRunId: explicitContinuationRunId, + queuedCommentInterruptId: opts.queuedCommentInterruptId, + queuedCommentRequestId: opts.queuedCommentRequestId, + dryRun: true, + onBlocked: (reason, message) => { continuationWait = { reason, message }; }, + }))) return deferBlockedExecution(executionBlocker); + + + if ( + worktreeExecutionCutoff && + issue.createdAt < worktreeExecutionCutoff + ) { + await tx.insert(agentWakeupRequests).values({ + ...durableReceiptFields, + companyId: agent.companyId, + agentId, + source, + triggerDetail, + reason: "heartbeat.worktree_execution_cutoff", + payload: { + ...(payload ?? {}), + heartbeatSkip: { + reason: "worktree_execution_cutoff", + cutoff: worktreeExecutionCutoff.toISOString(), + issueId: issue.id, + }, + }, + status: "skipped", + requestedByActorType: opts.requestedByActorType ?? null, + requestedByActorId: opts.requestedByActorId ?? null, + idempotencyKey: opts.idempotencyKey ?? null, + finishedAt: new Date(), + }); + return { kind: "skipped" as const }; + } + + const cancelStaleScheduledRetry = async ( + scheduledRun: typeof heartbeatRuns.$inferSelect, + ) => { + const issueCancelled = issue.status === "cancelled"; + if ( + scheduledRun.status !== "scheduled_retry" || + (scheduledRun.agentId === issue.assigneeAgentId && + !issueCancelled) + ) { + return false; + } + + const now = new Date(); + const reason = issueCancelled + ? "Cancelled because the issue was cancelled before the scheduled retry became due" + : "Cancelled because the issue was reassigned before the scheduled retry became due"; + const cancelled = await tx + .update(heartbeatRuns) + .set({ + status: "cancelled", + finishedAt: now, + error: reason, + errorCode: issueCancelled + ? "issue_cancelled" + : "issue_reassigned", + updatedAt: now, + }) + .where( + and( + eq(heartbeatRuns.id, scheduledRun.id), + eq(heartbeatRuns.status, "scheduled_retry"), + ), + ) + .returning() + .then((rows) => rows[0] ?? null); + + if (!cancelled) return false; + + if (scheduledRun.wakeupRequestId) { + await tx + .update(agentWakeupRequests) + .set({ + status: "cancelled", + finishedAt: now, + error: reason, + updatedAt: now, + }) + .where( + eq(agentWakeupRequests.id, scheduledRun.wakeupRequestId), + ); + } + + if (issue.executionRunId === scheduledRun.id) { + await tx + .update(issues) + .set({ + executionRunId: null, + executionAgentNameKey: null, + executionLockedAt: null, + updatedAt: now, + }) + .where( + and( + eq(issues.id, issue.id), + eq(issues.executionRunId, scheduledRun.id), + ), + ); + } + + const eventSeq = await allocateHeartbeatRunEventSeq( + tx as unknown as Db, + cancelled.id, + ); + + await tx.insert(heartbeatRunEvents).values({ + companyId: cancelled.companyId, + runId: cancelled.id, + agentId: cancelled.agentId, + seq: eventSeq, + eventType: "lifecycle", + stream: "system", + level: "warn", + message: issueCancelled + ? "Scheduled retry cancelled because issue was cancelled before it became due" + : "Scheduled retry cancelled because issue ownership changed before it became due", + payload: { + issueId: issue.id, + issueStatus: issue.status, + scheduledRetryAttempt: cancelled.scheduledRetryAttempt, + scheduledRetryAt: cancelled.scheduledRetryAt + ? new Date(cancelled.scheduledRetryAt).toISOString() + : null, + scheduledRetryReason: cancelled.scheduledRetryReason, + previousRetryAgentId: cancelled.agentId, + currentAssigneeAgentId: issue.assigneeAgentId, + }, + }); + await tx + .update(heartbeatRuns) + .set({ nextEventSeq: eventSeq + 1, updatedAt: now }) + .where(eq(heartbeatRuns.id, cancelled.id)); + + cancelledRunsToEmit.push(cancelled); + + return true; + }; + + let activeExecutionRun = issue.executionRunId + ? await tx + .select() + .from(heartbeatRuns) + .where(eq(heartbeatRuns.id, issue.executionRunId)) + .then((rows) => rows[0] ?? null) + : null; + + if ( + activeExecutionRun && + !EXECUTION_PATH_HEARTBEAT_RUN_STATUSES.includes( + activeExecutionRun.status as (typeof EXECUTION_PATH_HEARTBEAT_RUN_STATUSES)[number], + ) + ) { + activeExecutionRun = null; + } + + if ( + activeExecutionRun && + (await cancelStaleScheduledRetry(activeExecutionRun)) + ) { + activeExecutionRun = null; + } + + // A queued/scheduled run holding the lock for an agent that is + // no longer the issue's assignee is stale by design — the issue + // has been re-routed (e.g. blocked → in_review with a different + // assignee). Cancel it and release the lock; otherwise the new + // assignee's wake gets parked in `deferred_issue_execution` + // forever, because the original queued holder will never run + // (the issue's status / target now belongs to someone else). + // + // Race guard: pin the cancel UPDATE to the exact non-running + // status we read above. A worker could transition the holder + // from `queued` → `running` between the SELECT and this UPDATE; + // the status predicate ensures we never clobber a freshly- + // claimed running run. If zero rows matched, leave + // `activeExecutionRun` populated so the defer path runs + // normally against the now-running holder. + if ( + activeExecutionRun && + activeExecutionRun.status !== "running" && + issue.assigneeAgentId && + activeExecutionRun.agentId !== issue.assigneeAgentId + ) { + const cancelled = await tx + .update(heartbeatRuns) + .set({ + status: "cancelled", + finishedAt: new Date(), + error: + "Execution lock released after issue reassigned to a different agent", + errorCode: "lock_released_on_reassignment", + updatedAt: new Date(), + }) + .where( + and( + eq(heartbeatRuns.id, activeExecutionRun.id), + eq(heartbeatRuns.status, activeExecutionRun.status), + ), + ) + .returning(); + if (cancelled.length > 0) { + cancelledRunsToEmit.push(cancelled[0]); + if (activeExecutionRun.wakeupRequestId) { + await tx + .update(agentWakeupRequests) + .set({ + status: "cancelled", + finishedAt: new Date(), + error: + "Execution lock released after issue reassigned to a different agent", + updatedAt: new Date(), + }) + .where( + eq( + agentWakeupRequests.id, + activeExecutionRun.wakeupRequestId, + ), + ); + } + activeExecutionRun = null; + } + } + + if (!activeExecutionRun && issue.executionRunId) { + await tx + .update(issues) + .set({ + executionRunId: null, + executionAgentNameKey: null, + executionLockedAt: null, + updatedAt: new Date(), + }) + .where(eq(issues.id, issue.id)); + } + + if (!activeExecutionRun) { + const legacyRun = await tx + .select() + .from(heartbeatRuns) + .where( + and( + eq(heartbeatRuns.companyId, issue.companyId), + inArray(heartbeatRuns.status, [ + ...EXECUTION_PATH_HEARTBEAT_RUN_STATUSES, + ]), + sql`${heartbeatRuns.contextSnapshot} ->> 'issueId' = ${issue.id}`, + ), + ) + .orderBy( + sql`case when ${heartbeatRuns.status} = 'running' then 0 else 1 end`, + asc(heartbeatRuns.createdAt), + ) + .limit(1) + .then((rows) => rows[0] ?? null); + + if (legacyRun) { + if (await cancelStaleScheduledRetry(legacyRun)) { + activeExecutionRun = null; + } else { + activeExecutionRun = legacyRun; + const legacyAgent = await tx + .select({ name: agents.name }) + .from(agents) + .where(eq(agents.id, legacyRun.agentId)) + .then((rows) => rows[0] ?? null); + await tx + .update(issues) + .set({ + executionRunId: legacyRun.id, + executionAgentNameKey: normalizeAgentNameKey( + legacyAgent?.name, + ), + executionLockedAt: new Date(), + updatedAt: new Date(), + }) + .where(eq(issues.id, issue.id)); + } + } + } + + const dependencyReadiness = await issuesSvc + .listDependencyReadiness(issue.companyId, [issue.id], tx) + .then((rows) => rows.get(issue.id) ?? null); + + // Blocked descendants should stay idle until the final blocker resolves. + // Human comment/mention wakes are the exception: they may run in a + // bounded interaction mode so the assignee can answer or triage. + const blockedInteractionWake = + dependencyReadiness && + !dependencyReadiness.isDependencyReady && + allowsIssueInteractionWake( + enrichedContextSnapshot, + ISSUE_TREE_CONTROL_INTERACTION_WAKE_REASONS, + ); + + if (blockedInteractionWake) { + enrichedContextSnapshot.dependencyBlockedInteraction = true; + enrichedContextSnapshot.unresolvedBlockerIssueIds = + dependencyReadiness.unresolvedBlockerIssueIds; + enrichedContextSnapshot.unresolvedBlockerCount = + dependencyReadiness.unresolvedBlockerCount; + enrichedContextSnapshot.unresolvedBlockerSummaries = + await listUnresolvedBlockerSummaries( + tx, + issue.companyId, + issue.id, + dependencyReadiness.unresolvedBlockerIssueIds, + ); + } + + if ( + !activeExecutionRun && + dependencyReadiness && + !dependencyReadiness.isDependencyReady && + !blockedInteractionWake + ) { + await recordExecutionWait(tx as unknown as Db, { + issueId: issue.id, + coalesce: coalesceExecutionWait, + condition: { unresolvedBlockerIssueIds: [...dependencyReadiness.unresolvedBlockerIssueIds].sort() }, + request: { + ...durableReceiptFields, + companyId: agent.companyId, + agentId, + source, + triggerDetail, + reason: "issue_dependencies_blocked", + payload: { + ...(payload ?? {}), + issueId, + unresolvedBlockerIssueIds: + dependencyReadiness.unresolvedBlockerIssueIds, + }, + status: "skipped", + requestedByActorType: opts.requestedByActorType ?? null, + requestedByActorId: opts.requestedByActorId ?? null, + idempotencyKey: opts.idempotencyKey ?? null, + finishedAt: new Date(), + }, + }); + return { kind: "skipped" as const }; + } + + if ( + isolatedWorkspacesEnabled && + !activeExecutionRun && + issue.status !== "done" && + issue.status !== "cancelled" + ) { + const issueSettings = parseIssueExecutionWorkspaceSettings( + issue.executionWorkspaceSettings, + ); + const resolvedMode = resolveExecutionWorkspaceMode({ + projectPolicy: null, + issueSettings, + legacyUseProjectWorkspace: null, + }); + const workspaceManagedConfig = buildExecutionWorkspaceAdapterConfig( + { + agentConfig: parseObject(agent.adapterConfig), + projectPolicy: null, + issueSettings, + mode: resolvedMode, + legacyUseProjectWorkspace: null, + }, + ); + const resolvedStrategy = resolveEffectiveWorkspaceStrategyType( + resolvedMode, + workspaceManagedConfig, + ); + const existingExecutionWorkspaceStatus = issue.executionWorkspaceId + ? await tx + .select({ status: executionWorkspaces.status }) + .from(executionWorkspaces) + .where( + and( + eq(executionWorkspaces.id, issue.executionWorkspaceId), + eq(executionWorkspaces.companyId, issue.companyId), + ), + ) + .then((rows) => rows[0]?.status ?? null) + : null; + const reuseRequest = resolveExecutionWorkspaceReuseRequestForIssue({ + issueExecutionWorkspaceId: issue.executionWorkspaceId, + issueExecutionWorkspacePreference: + issue.executionWorkspacePreference, + existingExecutionWorkspaceStatus, + }); + const hasResolvablePriorSessionWorkspace = + await resolveHasResolvablePriorSessionWorkspace(); + + if ( + isUnrunnableWorktreeCombo({ + issue: { + projectId: issue.projectId ?? projectId ?? null, + projectWorkspaceId: issue.projectWorkspaceId, + executionWorkspaceId: issue.executionWorkspaceId, + executionWorkspacePreference: + issue.executionWorkspacePreference, + }, + resolvedMode, + resolvedStrategy, + reusableExecutionWorkspaceAvailable: + reuseRequest.existingExecutionWorkspaceAvailable, + hasResolvablePriorSessionWorkspace, + }) + ) { + const now = new Date(); + const issueLabel = formatIssueIdentifierLink( + issue.identifier, + issue.id, + ); + const blockedComment = [ + `Paperclip blocked ${issueLabel} before dispatch because its workspace settings are not runnable.`, + "", + `- Code: \`${WORKSPACE_WORKTREE_REQUIRES_PROJECT_CODE}\``, + `- Reason: ${WORKSPACE_WORKTREE_REQUIRES_PROJECT_MESSAGE}`, + `- Next action: ${WORKSPACE_WORKTREE_REQUIRES_PROJECT_REMEDIATION}`, + ].join("\n"); + await tx + .update(issues) + .set({ + status: "blocked", + checkoutRunId: null, + executionRunId: null, + executionAgentNameKey: null, + executionLockedAt: null, + updatedAt: now, + }) + .where(eq(issues.id, issue.id)); + await tx.insert(issueComments).values({ + companyId: issue.companyId, + issueId: issue.id, + body: blockedComment, + createdAt: now, + updatedAt: now, + }); + await tx.insert(agentWakeupRequests).values({ + ...durableReceiptFields, + companyId: agent.companyId, + agentId, + source, + triggerDetail, + reason: WORKSPACE_WORKTREE_REQUIRES_PROJECT_CODE, + payload: { + ...(payload ?? {}), + issueId, + heartbeatSkip: { + code: WORKSPACE_WORKTREE_REQUIRES_PROJECT_CODE, + reason: WORKSPACE_WORKTREE_REQUIRES_PROJECT_MESSAGE, + remediation: + WORKSPACE_WORKTREE_REQUIRES_PROJECT_REMEDIATION, + }, + }, + status: "skipped", + requestedByActorType: opts.requestedByActorType ?? null, + requestedByActorId: opts.requestedByActorId ?? null, + idempotencyKey: opts.idempotencyKey ?? null, + finishedAt: now, + }); + await logActivity(tx as unknown as Db, { + companyId: issue.companyId, + actorType: "system", + actorId: "system", + agentId, + runId: null, + action: "issue.workspace_preflight_blocked", + entityType: "issue", + entityId: issue.id, + details: { + code: WORKSPACE_WORKTREE_REQUIRES_PROJECT_CODE, + reason: WORKSPACE_WORKTREE_REQUIRES_PROJECT_MESSAGE, + remediation: WORKSPACE_WORKTREE_REQUIRES_PROJECT_REMEDIATION, + requestedReason: reason, + source, + triggerDetail, + resolvedMode, + resolvedStrategy, + hasResolvablePriorSessionWorkspace, + }, + }); + return { kind: "skipped" as const }; + } + } + + if (activeExecutionRun) { + // The resolved action is already a durable retry outbox. Do not merge + // its fresh-session contract into unrelated work or create a second + // deferred wake that could later replay the same reconciliation. + if (reconciledSourceRunId) return { kind: "deferred" as const }; + + const admissionScope = wakeQueue.createAdmissionTransactionScope( + agent.companyId, + tx as unknown as Db, + ); + const admission = await wakeQueue.admitWakeBehindIssueExecution( + admissionScope, + { + companyId: agent.companyId, + issueId: issue.id, + agentId, + agentNameKey, + issueExecutionAgentNameKey: issue.executionAgentNameKey, + activeExecutionRun: { + id: activeExecutionRun.id, + agentId: activeExecutionRun.agentId, + status: activeExecutionRun.status, + contextSnapshot: activeExecutionRun.contextSnapshot, + wakeupRequestId: activeExecutionRun.wakeupRequestId, + }, + allowRunCoalescing: isConversation(issue) ? false : opts.allowRunCoalescing, + durableReceipt: receiptRequest + ? { + id: receiptRequest.id, + requestedAt: receiptRequest.requestedAt, + } + : undefined, + reason, + liveRunExecutions, + wakeCommentId, + forceFreshSession: + enrichedContextSnapshot.forceFreshSession === true, + contextSnapshot: enrichedContextSnapshot, + source, + triggerDetail, + payload, + requestedByActorType: opts.requestedByActorType ?? null, + requestedByActorId: opts.requestedByActorId ?? null, + idempotencyKey: opts.idempotencyKey ?? null, + }, + ); + + if (admission.kind === "coalesced") { + return { + kind: "coalesced" as const, + run: admission.run as typeof heartbeatRuns.$inferSelect, + }; + } + if (admission.kind === "deferred") { + return { kind: "deferred" as const }; + } + // admission.kind === "proceed": no active run absorbed this wake, + // so fall through to the ordinary queue path below. + } + + // PAP-13775: no live run holds the lock, so this wake would start a + // fresh adapter session. If this agent's recent runs on this issue + // keep succeeding without any issue-visible progress and the wake + // carries no new information, hold it back for an escalating cooldown + // so external pollers/reconcilers can't storm full-price sessions. + // Server-side recovery retries insert runs directly and never reach + // this gate. + if ( + isThrottleCandidateIssueRewake({ + reason, + wakeCommentId: wakeCommentId ?? null, + requestedByActorType: opts.requestedByActorType ?? null, + forceFreshSession: + enrichedContextSnapshot.forceFreshSession === true, + hasExplicitResume: Boolean(explicitResumeSession), + }) + ) { + const throttleNow = new Date(); + const recentTerminalRuns = await tx + .select({ + id: heartbeatRuns.id, + status: heartbeatRuns.status, + finishedAt: heartbeatRuns.finishedAt, + }) + .from(heartbeatRuns) + .where( + and( + eq(heartbeatRuns.companyId, agent.companyId), + eq(heartbeatRuns.agentId, agentId), + sql`${heartbeatRuns.finishedAt} is not null`, + gte( + heartbeatRuns.finishedAt, + new Date(throttleNow.getTime() - ISSUE_REWAKE_LOOKBACK_MS), + ), + sql`${heartbeatRuns.contextSnapshot} ->> 'issueId' = ${issue.id}`, + ), + ) + .orderBy(desc(heartbeatRuns.finishedAt)) + .limit(ISSUE_REWAKE_RUN_SAMPLE_LIMIT); + + if (recentTerminalRuns.length > 0) { + const sampleRunIds = recentTerminalRuns.map( + (sampleRun) => sampleRun.id, + ); + const progressRows = await tx + .select({ runId: activityLog.runId }) + .from(activityLog) + .where( + and( + eq(activityLog.companyId, agent.companyId), + eq(activityLog.entityType, "issue"), + eq(activityLog.entityId, issue.id), + inArray(activityLog.runId, sampleRunIds), + inArray( + activityLog.action, + ISSUE_PROGRESS_ACTIVITY_ACTIONS, + ), + ), + ); + const lastRunFinishedAt = + recentTerminalRuns[0]?.finishedAt ?? null; + const newInputRows = lastRunFinishedAt + ? await tx + .select({ id: activityLog.id }) + .from(activityLog) + .where( + and( + eq(activityLog.companyId, agent.companyId), + eq(activityLog.entityType, "issue"), + eq(activityLog.entityId, issue.id), + gt(activityLog.createdAt, lastRunFinishedAt), + inArray( + activityLog.action, + ISSUE_NEW_INPUT_ACTIVITY_ACTIONS, + ), + wakeCommentId && opts.requestedByActorType === "agent" + ? ne(activityLog.actorType, "agent") + : undefined, + ), + ) + .limit(1) + : []; + + const throttleDecision = evaluateIssueRewakeThrottle({ + now: throttleNow, + recentTerminalRuns, + runIdsWithIssueProgress: new Set( + progressRows + .map((row) => row.runId) + .filter((runId): runId is string => Boolean(runId)), + ), + // For an agent comment wake, the query excludes agent-authored + // activity while preserving genuinely new user/system input. + // Presentation/author metadata therefore cannot smuggle human + // wake privilege, nor can it mask an actual human response. + hasNewIssueInputSinceLastRun: newInputRows.length > 0, + }); + + if (throttleDecision.blocked) { + await tx.insert(agentWakeupRequests).values({ + ...durableReceiptFields, + companyId: agent.companyId, + agentId, + source, + triggerDetail, + reason: "issue_rewake_throttled", + payload: { + ...(payload ?? {}), + issueId, + heartbeatSkip: { + reason: "issue_rewake_throttled", + requestedReason: reason, + noProgressStreak: throttleDecision.noProgressStreak, + cooldownMs: throttleDecision.cooldownMs, + lastRunFinishedAt: + throttleDecision.lastRunFinishedAt.toISOString(), + nextAllowedAt: + throttleDecision.nextAllowedAt.toISOString(), + }, + }, + status: "skipped", + requestedByActorType: opts.requestedByActorType ?? null, + requestedByActorId: opts.requestedByActorId ?? null, + idempotencyKey: opts.idempotencyKey ?? null, + finishedAt: throttleNow, + }); + return { kind: "skipped" as const }; + } + } + } + + const dailyCapBlock = await getHeartbeatDailyCapBlock( + agent, + policy, + {}, + tx, + ); + if (dailyCapBlock) { + if (executionWaitRequestId && executionBlocker) { + continuationWait = { reason: dailyCapBlock.reason, + message: "The agent has reached its daily limit. Your message is saved until work can resume." }; + return deferBlockedExecution(executionBlocker); + } + const now = new Date(); + await tx.insert(agentWakeupRequests).values({ + ...durableReceiptFields, + companyId: agent.companyId, + agentId, + source, + triggerDetail, + reason: dailyCapBlock.reason, + payload: { + ...(payload ?? {}), + heartbeatSkip: { + reason: + "Per-agent heartbeat daily cap reached before adapter invocation.", + observed: dailyCapBlock.observed, + limit: dailyCapBlock.limit, + }, + }, + status: "skipped", + requestedByActorType: opts.requestedByActorType ?? null, + requestedByActorId: opts.requestedByActorId ?? null, + idempotencyKey: opts.idempotencyKey ?? null, + finishedAt: now, + }); + if (source === "timer") { + await tx + .update(agents) + .set({ + lastHeartbeatAt: now, + updatedAt: now, + }) + .where(eq(agents.id, agentId)); + } + return { kind: "skipped" as const }; + } + + const explicitContinuation = await admitExplicitNativeContinuation({ + db: tx as unknown as Db, companyId: issue.companyId, issueId: issue.id, + agentId, actorType: opts.requestedByActorType, actorId: opts.requestedByActorId, + reason: durableRequest && !failedChatRetry ? "issue_commented" : reason, + commentId: wakeCommentId ?? null, failedRunId: opts.failedRunId, successorRunId: explicitContinuationRunId, + queuedCommentInterruptId: opts.queuedCommentInterruptId, + queuedCommentRequestId: opts.queuedCommentRequestId, + }); + if (!explicitContinuation && executionBlocker) return deferBlockedExecution(executionBlocker); + if (explicitContinuation) { + enrichedContextSnapshot.forceFreshSession = true; + enrichedContextSnapshot.previousRunId = explicitContinuation.previousRunId; + enrichedContextSnapshot.explicitUserContinuation = explicitContinuation; + } + + const wakeupRequest = await tx + .insert(agentWakeupRequests) + .values({ + ...durableReceiptFields, + companyId: agent.companyId, + agentId, + source, + triggerDetail, + reason, + payload, + status: "queued", + requestedByActorType: opts.requestedByActorType ?? null, + requestedByActorId: opts.requestedByActorId ?? null, + idempotencyKey: opts.idempotencyKey ?? null, + }) + .returning() + .then((rows) => rows[0]); + + // A handoff changes the executor, not the owner of saved user input. + // Validate its exact stopped source while the issue row is locked; + // unrelated agents and dedicated continuations keep their own wakes. + const interruptedRunId = readNonEmptyString(enrichedContextSnapshot.interruptedRunId); + const handoffSource = source === "assignment" && reason === "issue_assigned" && + issue.assigneeAgentId === agentId && interruptedRunId && isUuidLike(interruptedRunId) + ? await tx.select({ agentId: heartbeatRuns.agentId }).from(heartbeatRuns).where(and( + eq(heartbeatRuns.id, interruptedRunId), eq(heartbeatRuns.companyId, issue.companyId), + eq(heartbeatRuns.status, "cancelled"), eq(heartbeatRuns.errorCode, "issue_reassigned"), + ne(heartbeatRuns.agentId, agentId), + sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issue.id}`, + or(isNull(heartbeatRuns.nativeIssueId), eq(heartbeatRuns.nativeIssueId, issue.id)), + )).then(rows => rows[0] ?? null) + : null; + const canCoalesceComments = !isConversation(issue) && opts.allowRunCoalescing !== false; + // A resumed receipt already passed admission under this issue lock. + // Consume it with its successor even when chat keeps other messages + // in separate turns, or finalization will deliver it a second time. + const pendingComments = + (executionWaitRequestId || canCoalesceComments) && + !(await getExecutionBlocker(tx as unknown as Db, issue.companyId, issue.id)) + ? await tx + .select() + .from(agentWakeupRequests) + .where( + and( + eq(agentWakeupRequests.companyId, issue.companyId), + inArray(agentWakeupRequests.agentId, handoffSource ? [agentId, handoffSource.agentId] : [agentId]), + eq(agentWakeupRequests.status, "deferred_issue_execution"), + sql`${agentWakeupRequests.payload}->>'issueId' = ${issue.id}`, + canCoalesceComments ? undefined : eq(agentWakeupRequests.id, executionWaitRequestId!), + ), + ) + .orderBy(asc(agentWakeupRequests.requestedAt)) + : []; + const adoptedComments = pendingComments.filter((wake) => { + if (wake.id === executionWaitRequestId) return true; + if (wake.id === opts.queuedCommentInterruptId || wake.id === opts.queuedCommentRequestId) return true; + const deferredPayload = parseObject(wake.payload); + const deferredContext = parseObject( + deferredPayload[DEFERRED_WAKE_CONTEXT_KEY], + ); + // Dedicated interaction wakes carry their own source and session + // contract. ID-only adoption must not erase that continuation. + return ( + // Durable chat work must keep its receipt, actor, source, and + // session contract through normal promotion and authorization. + !wake.idempotencyKey?.startsWith("chat-inbound:") && + !isInteractionResolutionWakePayload(deferredPayload) && + !hasInteractionContinuationWakeContext(deferredContext) && + ["issue_commented", "issue_reopened_via_comment"].includes(String(deferredContext.wakeReason ?? wake.reason)) && + queuedCommentIdsFromWakePayload(wake.payload).length > 0 + ); + }); + let adoptedCommentIds = [ + ...new Set([ + ...adoptedComments.flatMap((wake) => + queuedCommentIdsFromWakePayload(wake.payload), + ), + ...queuedCommentIdsFromRunContext(enrichedContextSnapshot), + ]), + ]; + if (opts.queuedCommentRequestId || handoffSource) { + adoptedCommentIds = await undeliveredLegacyUserCommentIds(tx as unknown as Db, + agent.companyId, issueId, agentId, adoptedCommentIds); + } + const newRun = await tx + .insert(heartbeatRuns) + .values({ + ...(explicitContinuation ? { id: explicitContinuationRunId } : {}), + companyId: agent.companyId, + agentId, + scopeKind: readNonEmptyString(enrichedContextSnapshot.issueId) ? "issue" : "company", + issueId: readNonEmptyString(enrichedContextSnapshot.issueId), + invocationSource: source, + triggerDetail, + status: "queued", + responsibleUserId: await resolveQueuedResponsibleUserId(), + wakeupRequestId: wakeupRequest.id, + retryOfRunId: failedChatRetry + ? durableRequest!.failedRunRetry!.failedRunId + : opts.failedRunId ?? automaticParentRunId, + contextSnapshot: adoptedComments.length + ? withQueuedCommentIdsInRunContext( + enrichedContextSnapshot, + adoptedCommentIds, + ) + : enrichedContextSnapshot, + sessionIdBefore: explicitContinuation ? null : sessionBefore, + continuationAttempt, + ...(reconciledSourceRunId + ? { retryOfRunId: reconciledSourceRunId } + : {}), + ...(reconciledRestoreRetryCount !== null ? { + scheduledRetryAttempt: reconciledRestoreRetryCount, + scheduledRetryReason: "transient_failure", + } : {}), + }) + .returning() + .then((rows) => rows[0]); + + await tx + .update(agentWakeupRequests) + .set({ + runId: newRun.id, + updatedAt: new Date(), + }) + .where(eq(agentWakeupRequests.id, wakeupRequest.id)); + + if (adoptedComments.length) { + await tx + .update(agentWakeupRequests) + .set({ + status: "coalesced", + runId: newRun.id, + finishedAt: new Date(), + updatedAt: new Date(), + }) + .where( + inArray( + agentWakeupRequests.id, + adoptedComments.map((wake) => wake.id), + ), + ); + await tx + .update(agentWakeupRequests) + .set({ + payload: withQueuedCommentIdsInWakePayload(payload, adoptedCommentIds), + }) + .where(eq(agentWakeupRequests.id, wakeupRequest.id)); + } + + // executionRunId is NOT stamped here (enqueueWakeup queues the run but + // doesn't start it). It will be stamped in claimQueuedRun() once the run + // transitions to "running" — Fix A (lazy locking). + + return { kind: "queued" as const, run: newRun }; + }) + .catch((error) => { + if (isExternalChatWaitAuthorizationContention(error)) + return { kind: "deferred" as const }; + throw error; + }); + + // Telemetry for the cancelled runs is best-effort background work. + // Fire it here and never await it: none of the lifecycle work below, + // nor this function's return, depends on it, so a slow telemetry + // lookup must not delay them. + for (const cancelledRun of cancelledRunsToEmit) { + void emitAgentTaskRun(db, cancelledRun); + } + + if (outcome.kind === "durable") { + return outcome.receipt.runId ? getRun(outcome.receipt.runId) : null; + } + if (outcome.kind === "deferred" || outcome.kind === "skipped") { + return null; + } + if (outcome.kind === "coalesced") { + await startNextQueuedRunForAgent(agent.id); + return outcome.run; + } + if (outcome.kind === "replayed") { + if (outcome.run.status === "queued") + await startNextQueuedRunForAgent(agent.id); + return outcome.run; + } + + const newRun = outcome.run; + publishLiveEvent({ + companyId: newRun.companyId, + type: "heartbeat.run.queued", + payload: { + runId: newRun.id, + agentId: newRun.agentId, + invocationSource: newRun.invocationSource, + triggerDetail: newRun.triggerDetail, + wakeupRequestId: newRun.wakeupRequestId, + }, + }); + + await startNextQueuedRunForAgent(agent.id); + return newRun; + } + + if (durableRequest) throw new Error("chat_inbound_wakeup_binding_denied"); + + const activeRuns = await db + .select() + .from(heartbeatRuns) + .where( + and( + eq(heartbeatRuns.agentId, agentId), + inArray(heartbeatRuns.status, [ + ...EXECUTION_PATH_HEARTBEAT_RUN_STATUSES, + ]), + ), + ) + .orderBy(desc(heartbeatRuns.createdAt)); + + const sameScopeQueuedRun = activeRuns.find( + (candidate) => + candidate.status === "queued" && + isSameTaskScope(runTaskKey(candidate), taskKey), + ); + const sameScopeScheduledRetryRun = activeRuns.find( + (candidate) => + candidate.status === "scheduled_retry" && + isSameTaskScope(runTaskKey(candidate), taskKey), + ); + const sameScopeRunningRun = activeRuns.find( + (candidate) => + candidate.status === "running" && + isSameTaskScope(runTaskKey(candidate), taskKey), + ); + const shouldQueueFollowupForRunningWake = + Boolean(sameScopeRunningRun) && + !sameScopeQueuedRun && + shouldQueueFollowupForRunningIssueWake({ + contextSnapshot: enrichedContextSnapshot, + wakeCommentId, + }); + // Unscoped manual wakes need their own receipt and execution identity too. + const rawCoalescedTarget = + opts.allowRunCoalescing === false || opts.manualUserWake + ? null + : (sameScopeQueuedRun ?? + sameScopeScheduledRetryRun ?? + (shouldQueueFollowupForRunningWake + ? null + : (sameScopeRunningRun ?? null))); + + const coalescedTargetRun = filterZombieCoalesceTarget( + rawCoalescedTarget, + liveRunExecutions, + ); + + if (coalescedTargetRun) { + const mergedContextSnapshot = mergeCoalescedContextSnapshot( + coalescedTargetRun.contextSnapshot, + enrichedContextSnapshot, + { + preserveExistingInteractionContinuation: + coalescedTargetRun.status === "queued" || + coalescedTargetRun.status === "scheduled_retry", + }, + ); + const mergedRun = await db + .update(heartbeatRuns) + .set({ + contextSnapshot: mergedContextSnapshot, + updatedAt: new Date(), + }) + .where(eq(heartbeatRuns.id, coalescedTargetRun.id)) + .returning() + .then((rows) => rows[0] ?? coalescedTargetRun); + + await db.insert(agentWakeupRequests).values({ + ...durableReceiptFields, + companyId: agent.companyId, + agentId, + source, + triggerDetail, + reason, + payload, + status: "coalesced", + coalescedCount: 1, + requestedByActorType: opts.requestedByActorType ?? null, + requestedByActorId: opts.requestedByActorId ?? null, + idempotencyKey: opts.idempotencyKey ?? null, + runId: mergedRun.id, + finishedAt: new Date(), + }); + return mergedRun; + } + + const queueOutcome = await db.transaction(async (tx) => { + await tx.execute( + sql`select id from agents where id = ${agentId} and company_id = ${agent.companyId} for update`, + ); + + const dailyCapBlock = await getHeartbeatDailyCapBlock( + agent, + policy, + {}, + tx, + ); + if (dailyCapBlock) { + const now = new Date(); + await tx.insert(agentWakeupRequests).values({ + ...durableReceiptFields, + companyId: agent.companyId, + agentId, + source, + triggerDetail, + reason: dailyCapBlock.reason, + payload: { + ...(payload ?? {}), + heartbeatSkip: { + reason: + "Per-agent heartbeat daily cap reached before adapter invocation.", + observed: dailyCapBlock.observed, + limit: dailyCapBlock.limit, + }, + }, + status: "skipped", + requestedByActorType: opts.requestedByActorType ?? null, + requestedByActorId: opts.requestedByActorId ?? null, + idempotencyKey: opts.idempotencyKey ?? null, + finishedAt: now, + }); + if (source === "timer") { + await tx + .update(agents) + .set({ + lastHeartbeatAt: now, + updatedAt: now, + }) + .where(eq(agents.id, agentId)); + } + return { kind: "skipped" as const }; + } + + const wakeupRequest = await tx + .insert(agentWakeupRequests) + .values({ + ...durableReceiptFields, + companyId: agent.companyId, + agentId, + source, + triggerDetail, + reason, + payload, + status: "queued", + requestedByActorType: opts.requestedByActorType ?? null, + requestedByActorId: opts.requestedByActorId ?? null, + idempotencyKey: opts.idempotencyKey ?? null, + }) + .returning() + .then((rows) => rows[0]); + + const newRun = await tx + .insert(heartbeatRuns) + .values({ + companyId: agent.companyId, + agentId, + scopeKind: readNonEmptyString(enrichedContextSnapshot.issueId) ? "issue" : "company", + issueId: readNonEmptyString(enrichedContextSnapshot.issueId), + invocationSource: source, + triggerDetail, + status: "queued", + responsibleUserId: await resolveQueuedResponsibleUserId(), + wakeupRequestId: wakeupRequest.id, + contextSnapshot: enrichedContextSnapshot, + sessionIdBefore: sessionBefore, + continuationAttempt, + }) + .returning() + .then((rows) => rows[0]); + + await tx + .update(agentWakeupRequests) + .set({ + runId: newRun.id, + updatedAt: new Date(), + }) + .where(eq(agentWakeupRequests.id, wakeupRequest.id)); + + return { kind: "queued" as const, run: newRun }; + }); + + if (queueOutcome.kind === "skipped") return null; + const newRun = queueOutcome.run; + + publishLiveEvent({ + companyId: newRun.companyId, + type: "heartbeat.run.queued", + payload: { + runId: newRun.id, + agentId: newRun.agentId, + invocationSource: newRun.invocationSource, + triggerDetail: newRun.triggerDetail, + wakeupRequestId: newRun.wakeupRequestId, + }, + }); + + await startNextQueuedRunForAgent(agent.id); + + return newRun; + } + + /** + * Native status commitment deliberately persists dependency/parent wake + * intents in the same transaction as the authoritative status projection. + * Those rows are not runnable until the heartbeat scheduler has applied its + * normal policy, workspace, concurrency, and responsible-user checks. Bridge + * the durable intent into that scheduler here instead of treating a bare + * `agent_wakeup_requests` row as if it were already a queued heartbeat run. + * + * The intent is claimed before dispatch. A deterministic dispatcher actor id + * lets a later sweep recover the narrow process-crash window after the real + * wake was inserted but before the intent was linked to it. Dispatch failure + * only requeues the intent; it never changes the provider run outcome. + */ + async function dispatchPendingNativeStatusWakeups( + input: { + companyId?: string; + limit?: number; + staleClaimMs?: number; + } = {}, + ) { + const now = new Date(); + const staleClaimMs = Math.max(1_000, input.staleClaimMs ?? 60_000); + const candidates = await db + .select() + .from(agentWakeupRequests) + .where( + and( + input.companyId + ? eq(agentWakeupRequests.companyId, input.companyId) + : undefined, + eq(agentWakeupRequests.requestedByActorType, "system"), + eq(agentWakeupRequests.requestedByActorId, "native-status-committer"), + inArray(agentWakeupRequests.status, ["queued", "claimed"]), + isNull(agentWakeupRequests.runId), + ), + ) + .orderBy(asc(agentWakeupRequests.requestedAt)) + .limit(Math.max(1, Math.min(input.limit ?? 100, 500))); + + let dispatched = 0; + let recovered = 0; + let deferred = 0; + const deliveredByIssueScope = new Map< + string, + { runId: string | null; status: string } + >(); + + for (const candidate of candidates) { + const dispatchActorId = `native-status-wake-dispatch:${candidate.id}`; + const existingDispatch = await db + .select() + .from(agentWakeupRequests) + .where( + and( + eq(agentWakeupRequests.companyId, candidate.companyId), + eq(agentWakeupRequests.requestedByActorType, "system"), + eq(agentWakeupRequests.requestedByActorId, dispatchActorId), + ), + ) + .orderBy(desc(agentWakeupRequests.requestedAt)) + .limit(1) + .then((rows) => rows[0] ?? null); + + if (existingDispatch) { + const recoveredStatus = existingDispatch.runId + ? "coalesced" + : existingDispatch.status === "deferred_issue_execution" + ? "coalesced" + : existingDispatch.status; + await db + .update(agentWakeupRequests) + .set({ + status: recoveredStatus, + runId: existingDispatch.runId, + finishedAt: + (existingDispatch.runId ?? existingDispatch.finishedAt) + ? (existingDispatch.finishedAt ?? now) + : null, + error: existingDispatch.error, + updatedAt: now, + }) + .where( + and( + eq(agentWakeupRequests.id, candidate.id), + isNull(agentWakeupRequests.runId), + ), + ); + recovered += 1; + continue; + } + + if ( + candidate.status === "claimed" && + candidate.claimedAt && + now.getTime() - candidate.claimedAt.getTime() < staleClaimMs + ) { + deferred += 1; + continue; + } + + const claimed = await db + .update(agentWakeupRequests) + .set({ + status: "claimed", + claimedAt: now, + error: null, + updatedAt: now, + }) + .where( + and( + eq(agentWakeupRequests.id, candidate.id), + isNull(agentWakeupRequests.runId), + candidate.status === "claimed" + ? eq(agentWakeupRequests.status, "claimed") + : eq(agentWakeupRequests.status, "queued"), + ), + ) + .returning({ id: agentWakeupRequests.id }); + if (claimed.length === 0) continue; + + const payload = parseObject(candidate.payload); + const wakeContext = parseObject(payload._paperclipWakeContext); + const issueId = + readNonEmptyString(payload.issueId) ?? + readNonEmptyString(payload.taskId) ?? + readNonEmptyString(wakeContext.issueId) ?? + null; + const scopeKey = issueId + ? `${candidate.companyId}:${candidate.agentId}:${issueId}:${readNativeReviewAssignmentContext(wakeContext)?.nativeReviewInteractionId ?? ""}` + : null; + const priorDelivery = scopeKey + ? deliveredByIssueScope.get(scopeKey) + : null; + if (priorDelivery) { + await db + .update(agentWakeupRequests) + .set({ + status: "coalesced", + runId: priorDelivery.runId, + finishedAt: new Date(), + error: null, + updatedAt: new Date(), + }) + .where(eq(agentWakeupRequests.id, candidate.id)); + recovered += 1; + continue; + } + + let completedOnboardingGuard: WakeupOptions["issueStateGuard"]; + if (issueId) { + const targetIssue = await db + .select({ + status: issues.status, + statusVersion: issues.statusVersion, + assigneeAgentId: issues.assigneeAgentId, + }) + .from(issues) + .where( + and( + eq(issues.id, issueId), + eq(issues.companyId, candidate.companyId), + ), + ) + .limit(1) + .then((rows) => rows[0] ?? null); + const nativeReview = candidate.reason === "native_completion_review" + ? await getNativeReviewAssignment(db, { + companyId: candidate.companyId, issueId, agentId: candidate.agentId, + contextSnapshot: wakeContext, + }) + : null; + const onboardingResultReport = targetIssue?.status === "done" && await isCompletedOnboardingHandoffWake(db, { + companyId: candidate.companyId, issueId, agentId: candidate.agentId, + reason: candidate.reason, contextSnapshot: wakeContext, + }); + if (onboardingResultReport) completedOnboardingGuard = { assigneeAgentId: candidate.agentId, statuses: ["done"], statusVersion: targetIssue!.statusVersion }; + if ( + !targetIssue || + (["done", "cancelled"].includes(targetIssue.status) && !onboardingResultReport) || + (targetIssue.assigneeAgentId !== candidate.agentId && !nativeReview) || + (candidate.reason === "native_completion_review" && !nativeReview) + ) { + await db + .update(agentWakeupRequests) + .set({ + status: "skipped", + finishedAt: new Date(), + error: !targetIssue + ? "Native status wake target no longer exists" + : ["done", "cancelled"].includes(targetIssue.status) + ? `Native status wake target is already ${targetIssue.status}` + : "Native status wake target has a different assignee", + updatedAt: new Date(), + }) + .where(eq(agentWakeupRequests.id, candidate.id)); + recovered += 1; + continue; + } + } + + try { + const wakeRun = await enqueueWakeup(candidate.agentId, { + source: candidate.source as WakeupOptions["source"], + triggerDetail: (candidate.triggerDetail ?? + "system") as WakeupOptions["triggerDetail"], + reason: candidate.reason, + payload, + idempotencyKey: candidate.idempotencyKey, + requestedByActorType: "system", + requestedByActorId: dispatchActorId, + ...(completedOnboardingGuard ? { issueStateGuard: completedOnboardingGuard } : {}), + contextSnapshot: { + ...wakeContext, + ...(issueId ? { issueId, taskId: issueId } : {}), + wakeReason: candidate.reason, + source: "native_status_decision", + statusDecisionSource: "native_status_decision", + nativeStatusWakeIntentId: candidate.id, + }, + }); + + const delivered = await db + .select() + .from(agentWakeupRequests) + .where( + and( + eq(agentWakeupRequests.companyId, candidate.companyId), + eq(agentWakeupRequests.requestedByActorType, "system"), + eq(agentWakeupRequests.requestedByActorId, dispatchActorId), + ), + ) + .orderBy(desc(agentWakeupRequests.requestedAt)) + .limit(1) + .then((rows) => rows[0] ?? null); + + // The committer intent is the durable outbox entry. When admission is + // blocked, enqueueWakeup creates a separate deferred dispatch receipt; + // leave the original intent coalesced so the release drain cannot + // promote both rows (the original has no nativeStatusWakeIntentId + // provenance and would otherwise run once before the dispatch receipt). + const deliveredStatus = delivered?.status ?? "queued"; + await db + .update(agentWakeupRequests) + .set({ + status: wakeRun || deliveredStatus === "deferred_issue_execution" + ? "coalesced" + : deliveredStatus, + runId: wakeRun?.id ?? delivered?.runId ?? null, + finishedAt: + wakeRun || delivered?.finishedAt + ? (delivered?.finishedAt ?? new Date()) + : null, + error: delivered?.error ?? null, + updatedAt: new Date(), + }) + .where(eq(agentWakeupRequests.id, candidate.id)); + + if (scopeKey) { + deliveredByIssueScope.set(scopeKey, { + runId: wakeRun?.id ?? delivered?.runId ?? null, + status: wakeRun ? "coalesced" : (delivered?.status ?? "queued"), + }); + } + + if (wakeRun) dispatched += 1; + else deferred += 1; + } catch (error) { + await db + .update(agentWakeupRequests) + .set({ + status: "queued", + claimedAt: null, + error: + error instanceof Error + ? error.message.slice(0, 1_000) + : String(error).slice(0, 1_000), + updatedAt: new Date(), + }) + .where( + and( + eq(agentWakeupRequests.id, candidate.id), + eq(agentWakeupRequests.status, "claimed"), + isNull(agentWakeupRequests.runId), + ), + ); + logger.warn( + { + err: error, + wakeupRequestId: candidate.id, + agentId: candidate.agentId, + }, + "failed to dispatch persisted native status wake intent", + ); + deferred += 1; + } + } + + return { scanned: candidates.length, dispatched, recovered, deferred }; + } + + return { + enqueueWakeup, + dispatchPendingNativeStatusWakeups, + startNextQueuedRunForAgent, + countRunningRunsForAgent, + releaseRunClaimedJustBeforeSuppression, + claimQueuedRun, + withChatControlRecoveryGate, + trackWakeup, + resumeQueuedRuns, + parseHeartbeatPolicy, + claimDueTimerHeartbeat, + }; +}