From 40701adecf76c76f49eb675191ba39f44e3db5d1 Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Sat, 10 Oct 2026 13:33:34 -0700 Subject: [PATCH 1/7] fix(release): preserve pinned master identity for canary publishing (#15770) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A master push publishes a canary from its exact source commit. > - The publisher now checks out that commit by SHA. Git leaves HEAD detached. > - The release script requires the local branch name to be master. > - This pull request verifies the trusted push and source before attaching local master at the same commit. > - The canary can pass the existing branch guard without following a newer master tip. ## Linked Issues or Issue Description Related: #15712 pinned the publisher checkout and bound the smoke lockfile to its source. I searched open PRs and issues for canary, detached HEAD, and checkout fixes. No overlapping fix was found. #15768 addresses a later tarball visibility error and does not change this branch guard. **What happened?** The canary publisher checks out the exact push SHA. The checkout is detached. `release.sh canary --skip-verify` then fails because the local branch is not master. Publication does not start. The [failed master run](https://github.com/paperclipai/paperclip/actions/runs/38008956836) shows this failure. **Expected behavior** The trusted master publisher must keep the exact verified push commit and pass the existing local master guard. It must not follow a newer branch tip. **Steps to reproduce** 1. Check out a master push commit by its full SHA. 2. Confirm that HEAD is detached. 3. Run the real `require_on_master_branch` helper from `scripts/release-lib.sh`. 4. Observe the detached-HEAD rejection. The tests reproduce this without calling a publisher. **Paperclip version or commit** `baa94bbea58919bcc6cf247a708f04e6e1002679`. **Deployment mode** The GitHub Actions master canary publishing lane. ## What Changed - Add one step immediately after the pinned publisher checkout. - Require the canonical repository, a push event, and the master ref before changing any Git ref. - Validate the full source SHA and require the checked-out HEAD to match it. - Attach local master to that exact SHA and recheck HEAD. Do not fetch or select a floating branch. - Preserve release scripts, local branch guards, package authentication, other publishing lanes, and source-named lockfile artifacts. - Add disposable real-Git tests for a pinned source behind local and remote master, rejected trust and SHA inputs, unchanged refs on rejection, and the original branch guard. Document the branch handoff. ## Verification - `node --test scripts/__tests__/release-verify-workflow.test.mjs scripts/cloud-source-verification.test.mjs scripts/standard-image-contract.test.mjs scripts/release-lib.test.mjs` passed all 82 tests on Node 25.6.1. Twelve new cases include eleven real-Git fixtures and one workflow invariant check. The tests run the actual workflow shell and the unchanged release guard. - `actionlint -shellcheck= .github/workflows/release.yml` passed. ShellCheck passed for the new shell. Full local actionlint reports four pre-existing advisories in other steps; their text matches the base exactly. This change adds no advisory. - Workspace `pnpm -r typecheck` and `pnpm build` passed on Node 25.6.1. The local tests cover the changed release flow; the full application test suite runs in hosted CI. - [Hosted full CI](https://github.com/paperclipai/paperclip/actions/runs/38011530698) passed all 47 jobs at commit `ffff774a2629663fa71e2fad5143da0696afc5e6`. This includes policy, typecheck, release registry, build, runner tests, application tests, browser tests, and the canary dry run. Hosted policy ran all 12 new guard tests and passed all 67 source-policy tests. - Greptile scored the same commit 5/5 with no actionable issues. There are zero review threads. Separate source review approved all three changed files and 32 dependencies. It passed 82 tests and caught four broken variants. - The supply chain scanner returned neutral with zero annotations. It cannot verify this inline workflow change and reports limited coverage for existing action references. Manual review confirmed that all action references and pins are unchanged. The security, Socket, and Snyk checks passed. - No live canary install, registry publication, Docker image operation, workflow rerun, or deployment was performed. Tests use disposable Git repositories and local fixtures. ## Risks Low risk. The new step changes only local Git state in the trusted canary publisher. A wrong event or source fails before ref changes. The original release and authentication guards remain in place. The exact checkout and artifact source stay unchanged. Revert the new step to restore the previous behavior. No runtime or database change is required. ## Model Used OpenAI GPT-6 through Codex. Used repository inspection, code execution, Git and Bash fixtures, and independent agent review. The runtime does not expose a more specific model snapshot or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip --- .github/workflows/release.yml | 19 +++ doc/RELEASING.md | 4 + .../release-verify-workflow.test.mjs | 125 ++++++++++++++++++ 3 files changed, 148 insertions(+) diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 4c2dfb48818..a2c51dbc747 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -361,6 +361,25 @@ jobs: ref: ${{ github.sha }} fetch-depth: 0 + - name: Attach verified canary source to local master + env: + SOURCE_SHA: ${{ github.sha }} + run: | + set -euo pipefail + if [ "$GITHUB_REPOSITORY" != "paperclipai/paperclip" ] || + [ "$GITHUB_EVENT_NAME" != "push" ] || + [ "$GITHUB_REF" != "refs/heads/master" ]; then + echo "Error: canary branch attachment requires a trusted master push." >&2 + exit 1 + fi + if ! [[ "$SOURCE_SHA" =~ ^[0-9a-f]{40}$ ]] || + [ "$(git rev-parse HEAD)" != "$SOURCE_SHA" ]; then + echo "Error: canary checkout does not match the pinned source SHA." >&2 + exit 1 + fi + git checkout -B master "$SOURCE_SHA" + test "$(git rev-parse HEAD)" = "$SOURCE_SHA" + - name: Setup pnpm uses: pnpm/action-setup@v6 with: diff --git a/doc/RELEASING.md b/doc/RELEASING.md index cc92f172121..771d3acb5bc 100644 --- a/doc/RELEASING.md +++ b/doc/RELEASING.md @@ -64,6 +64,10 @@ internal traceability tag. ## Core Invariants - canaries publish from `master` +- the canary publisher checks out the exact master push SHA. It verifies the + repository, event, branch, and checked-out commit before attaching local + `master` at that same SHA. It never switches to a newer branch tip. The local + release script still rejects detached or non-master checkouts. - nightlies republish a commit that already shipped a canary (the commit must carry a `canary/v*` tag), and only after the release smoke suite passes against that exact published canary diff --git a/scripts/__tests__/release-verify-workflow.test.mjs b/scripts/__tests__/release-verify-workflow.test.mjs index 10b3dddb7f9..9e5364c869a 100644 --- a/scripts/__tests__/release-verify-workflow.test.mjs +++ b/scripts/__tests__/release-verify-workflow.test.mjs @@ -102,6 +102,131 @@ test("canary reuses exact-source proof while stable keeps full verification", () ); }); +function canaryAttachmentScript() { + const publish = readWorkflow("release.yml").split(" publish_canary:\n")[1].split(" smoke_canary_onboarding:\n")[0]; + const step = publish.split(" - name: Attach verified canary source to local master\n")[1]?.split("\n - ")[0]; + assert.ok(step, "the trusted publisher must attach its pinned checkout before releasing"); + assert.match(step, /SOURCE_SHA: \$\{\{ github\.sha \}\}/); + const script = step.match(/run: \|\n((?: {10}[^\n]*\n)+)/)?.[1]; + assert.ok(script, "missing inline canary branch attachment shell"); + return script.replace(/^ {10}/gm, ""); +} + +function withCanaryGitFixture(check) { + const root = mkdtempSync(path.join(os.tmpdir(), "canary-publisher-git-")); + const cwd = path.join(root, "repository with spaces"); + mkdirSync(cwd); + const git = (...args) => { + const result = spawnSync("git", args, { cwd, encoding: "utf8" }); + assert.equal(result.status, 0, result.stderr); + return result.stdout.trim(); + }; + try { + git("init", "--initial-branch=master"); + git("config", "user.name", "Release fixture"); + git("config", "user.email", "release-fixture@example.invalid"); + git("config", "commit.gpgsign", "false"); + git("config", "core.hooksPath", "/dev/null"); + writeFileSync(path.join(cwd, "source.txt"), "pinned source\n"); + git("add", "source.txt"); + git("commit", "-m", "Create pinned source"); + const pinned = git("rev-parse", "HEAD"); + writeFileSync(path.join(cwd, "source.txt"), "newer master source\n"); + git("commit", "-am", "Advance master"); + const newer = git("rev-parse", "HEAD"); + git("update-ref", "refs/remotes/origin/master", newer); + git("checkout", "--detach", pinned); + const env = { + ...process.env, REPO_ROOT: cwd, SOURCE_SHA: pinned, + GITHUB_ACTIONS: "true", GITHUB_REPOSITORY: "paperclipai/paperclip", + GITHUB_EVENT_NAME: "push", GITHUB_REF: "refs/heads/master", + }; + const attach = (overrides = {}) => spawnSync("bash", ["--noprofile", "--norc", "-c", canaryAttachmentScript()], { + cwd, env: { ...env, ...overrides }, encoding: "utf8", timeout: 5_000, + }); + // Run the original release guard without running release.sh, npm, or a publisher. + const guard = () => spawnSync("bash", ["-c", 'source "$1"; require_on_master_branch', "guard", path.join(repoRoot, "scripts/release-lib.sh")], { + cwd, env, encoding: "utf8", timeout: 5_000, + }); + check({ cwd, git, pinned, newer, attach, guard }); + } finally { + rmSync(root, { recursive: true, force: true }); + } +} + +test("canary publisher attaches the exact source while local and remote master are newer", () => { + withCanaryGitFixture(({ cwd, git, pinned, newer, attach, guard }) => { + assert.equal(guard().status, 1, "the unchanged release guard must reject detached HEAD even in Actions"); + assert.equal(git("rev-parse", "master"), newer); + const result = attach(); + assert.equal(result.status, 0, result.stderr); + assert.equal(git("symbolic-ref", "--short", "HEAD"), "master"); + assert.equal(git("rev-parse", "HEAD"), pinned); + assert.equal(git("rev-parse", "master"), pinned); + assert.equal(git("rev-parse", "origin/master"), newer, "do not fetch or follow floating remote master"); + assert.equal(readFileSync(path.join(cwd, "source.txt"), "utf8"), "pinned source\n"); + assert.equal(git("status", "--porcelain"), ""); + assert.equal(guard().status, 0, "the real release guard must accept the pinned local master"); + }); +}); + +for (const [name, overrides] of [ + ["fork repository", { GITHUB_REPOSITORY: "example/paperclip" }], + ["manual dispatch", { GITHUB_EVENT_NAME: "workflow_dispatch" }], + ["scheduled release", { GITHUB_EVENT_NAME: "schedule" }], + ["non-master push", { GITHUB_REF: "refs/heads/topic" }], + ["tag push", { GITHUB_REF: "refs/tags/canary/v1" }], + ["missing source SHA", { SOURCE_SHA: "" }], + ["abbreviated source SHA", { SOURCE_SHA: "a".repeat(12) }], + ["malformed source SHA", { SOURCE_SHA: "z".repeat(40) }], +]) { + test(`canary publisher rejects ${name} before changing any Git reference`, () => { + withCanaryGitFixture(({ git, pinned, attach, guard }) => { + const refs = git("show-ref"); + const result = attach(overrides); + assert.equal(result.status, 1, result.stderr); + assert.equal(git("show-ref"), refs); + assert.equal(git("rev-parse", "HEAD"), pinned); + assert.equal(guard().status, 1, "the rejected checkout must remain detached"); + }); + }); +} + +test("canary publisher rejects a different valid source SHA before changing Git references", () => { + withCanaryGitFixture(({ git, pinned, newer, attach, guard }) => { + const refs = git("show-ref"); + const result = attach({ SOURCE_SHA: newer }); + assert.equal(result.status, 1, result.stderr); + assert.equal(git("show-ref"), refs); + assert.equal(git("rev-parse", "HEAD"), pinned); + assert.equal(guard().status, 1); + }); +}); + +test("the original canary release guard still rejects a non-master local branch in Actions", () => { + withCanaryGitFixture(({ git, pinned, guard }) => { + git("checkout", "-b", "topic", pinned); + const refs = git("show-ref"); + assert.equal(guard().status, 1); + assert.equal(git("symbolic-ref", "--short", "HEAD"), "topic"); + assert.equal(git("show-ref"), refs); + }); +}); + +test("canary branch attachment keeps exact-source checkout and artifact order in its publishing lane", () => { + const workflow = readWorkflow("release.yml"); + const publish = workflow.split(" publish_canary:\n")[1].split(" smoke_canary_onboarding:\n")[0]; + assert.equal(workflow.split("name: Attach verified canary source to local master").length, 2); + assert.match(publish, /ref: \$\{\{ github\.sha \}\}/); + assert.match(publish, /name: canary-smoke-lockfile-\$\{\{ github\.sha \}\}/); + const steps = ["Checkout repository", "Attach verified canary source to local master", "Install dependencies", "Save canary smoke lockfile", "Restore tracked install-time changes", "Publish canary"]; + for (let i = 1; i < steps.length; i++) { + assert.ok(publish.indexOf(`name: ${steps[i - 1]}`) < publish.indexOf(`name: ${steps[i]}`)); + } + assert.match(publish, /run: \.\/scripts\/release\.sh canary --skip-verify/); + assert.doesNotMatch(canaryAttachmentScript(), /git fetch|git pull|checkout (?:master|origin\/master)(?:\s|$)/); +}); + test("source proof requires every source check and does not wait on image publication", () => { const readiness = readWorkflow("cloud-readiness.yml"); const proof = readiness.split(" source_verified:\n")[1]; From 9ba5569030cc4f37e53906d19e13778c81c99ecc Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Sat, 10 Oct 2026 13:52:05 -0700 Subject: [PATCH 2/7] ci: refresh the general-server shard durations (#15826) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Pull requests to this repository run a large CI matrix. The slowest check sets the time a contributor waits for a green run. > - The general-server vitest lane runs with one worker per job, so it can only go faster across runners. Twelve shards share its suites, and a recorded duration manifest decides which suite goes to which shard. > - That manifest was recorded on 2026-09-15 from 646 suites. The lane now has 871 suites and about 5031s of serial vitest time. 230 suites have no record and get the median weight, so the shards drifted apart. In run 38076972444 (2026-10-10) they ran 313s to 554s of tests, and `General tests (server (2/12))` at 618s was the slowest check of the whole run. > - An unbalanced lane wastes the runner time it already spends. The total work does not change, but one shard holds the run. > - This pull request records the current duration of every suite from two recent PR runs and keeps the same twelve shards. > - The benefit is that the twelve shards level to about 415s of tests each. The slowest server shard should drop from about 618s to about 480s, and this lane leaves the critical path. ## Linked Issues or Issue Description **Describe the enhancement** The general-server PR test lane is out of balance. Its shard manifest, `scripts/general-server-shard-durations.json`, is three and a half weeks old and misses 230 of the 871 current suites. The heaviest shard runs 554s of tests while the lightest runs 313s. **Why is this needed** The slowest check decides the wall-clock time of every PR run. In the last fully green runs the slowest check was a general-server shard (618s, 562s, 585s, 646s in runs 38076972444, 38076844891, 38076844368 and 38073555316). A balanced lane removes about 135s from that shard at no extra runner cost. **Proposed solution** Re-record the per-suite durations from two recent PR runs (38076844891 and 38076972444) with the method that the manifest's `$comment` documents: diff consecutive per-suite completion timestamps in each shard log, and measure the first suite from the Vitest `RUN` banner. Average the two runs. Keep the chat and native-runner entries from the old manifest, because those suites run in their own lanes and do not appear in the shard logs. Drop the two suites that no longer exist. **Alternatives considered** More shards would also shorten the lane (fourteen shards level at about 355s, sixteen at about 311s). They are not needed now: `Build` (489-592s) and `General tests (workspaces-b)` (513-618s) already sit above the balanced twelve-shard level, so extra runners would not shorten the run. Those two checks are the next targets. ## What Changed - `scripts/general-server-shard-durations.json`: 873 recorded suites (was 646). 870 come from the two 2026-10-10 runs, 3 carry over (chat and native-runner suites), 2 obsolete entries are gone. The `$comment` names the source runs, the fleet, and the carry-over rule. - `.github/workflows/pr-trusted.yml`: a comment on the general tests matrix records the 2026-10-10 measurements and why the shard count stays at twelve. No workflow behavior changes. ## Verification - `node --test scripts/__tests__/run-vitest-stable-shard.test.mjs`: 29 pass. This covers the manifest coverage check, the duration-balance check (spread must stay under the heaviest suite), and the complete non-overlapping partition. - `node --test .github/scripts/tests/pr-dependency-cache.test.mjs .github/scripts/tests/pr-runner-rust-cache.test.mjs .github/scripts/tests/lockfile-refresh-workflows.test.mjs`: 17 pass. - Measurement check: the per-shard sum of the recorded durations matches Vitest's own `Duration` line within 1s on all 24 shard logs. - Simulation with the real planner (`scripts/run-vitest-stable.mjs --dry-run`, `GITHUB_WORKFLOW=PR`) on the current suite list: with the old manifest the twelve shards carry 321s to 510s of recorded tests; with this manifest they carry 413s to 415s. - Expected result on the first PR run after merge: every `General tests (server (n/12))` job finishes in about 480s (415s of tests plus about 65s of setup) instead of 374s to 618s. ## Risks - Low risk. The file only sets weights for the shard planner. A wrong weight moves a suite to a different shard; it cannot drop or duplicate a suite, because the guard tests assert the partition is complete and non-overlapping. - The durations come from the `runs-on` public PR fleet. GitHub-hosted runners are faster in absolute terms, but the relative weights, which are all the planner uses, are the same. - The manifest will go stale again as suites grow. The guard test fails when coverage drops under 50%, and the matrix comment records how to refresh it. ## Model Used - Claude (Anthropic), model ID `claude-fable-5-1`, running as a Paperclip agent through the Claude local adapter with tool use (shell, file edits, GitHub CLI). The agent collected the CI data, built the manifest, ran the tests, and wrote this description. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [ ] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com> Co-authored-by: Claude Fable 5.1 --- .github/workflows/pr-trusted.yml | 9 + scripts/general-server-shard-durations.json | 1516 +++++++++++-------- 2 files changed, 881 insertions(+), 644 deletions(-) diff --git a/.github/workflows/pr-trusted.yml b/.github/workflows/pr-trusted.yml index 5f02a99c5f4..56b6f594c12 100644 --- a/.github/workflows/pr-trusted.yml +++ b/.github/workflows/pr-trusted.yml @@ -502,6 +502,15 @@ jobs: # across three dedicated runners (~143s each), and the remaining # ~2755s levels across twelve duration-balanced shards at ~230s # each, in line with the other ~200-290s lanes. + # + # The balance only holds while the manifest is fresh. By 2026-10-10 + # the without-chat group had grown to 871 suites and ~5031s of serial + # vitest time (mean of runs 38076844891 and 38076972444), 230 of + # them unrecorded and weighted at the median, so the twelve shards + # ran 313-554s of tests (server 2/12 at 618s was the slowest check + # in the run). Re-recorded durations level them to ~415s each; the + # shard count stays at twelve because Build and workspaces-b sit + # above that already. - group: general-server-without-chat group_label: server (1/12) shard_index: 0 diff --git a/scripts/general-server-shard-durations.json b/scripts/general-server-shard-durations.json index 7b910562676..7b08502878a 100644 --- a/scripts/general-server-shard-durations.json +++ b/scripts/general-server-shard-durations.json @@ -1,652 +1,880 @@ { - "$comment": "Per-suite wall-clock durations (ms) for the general-server vitest lane, used by scripts/general-server-shard.mjs to balance suites across the PR shard matrix. Averaged from two real PR runs of .github/workflows/pr.yml (actions runs 35036001734 and 35024948947, 2026-09-15) by diffing consecutive per-suite completion timestamps in the 'Run grouped general test suites' logs across every shard — that captures each suite's true serial cost (import, setup, and run) under maxWorkers=1. The first suite of each shard is measured from Vitest's RUN banner, so per-shard startup lands on a measured suite instead of vanishing. Suites missing here get the median weight, so the manifest only needs occasional refreshes; regenerate the same way when the lane's shard steps drift apart by more than the heaviest suite.", + "$comment": "Per-suite wall-clock durations (ms) for the general-server vitest lane, used by scripts/general-server-shard.mjs to balance suites across the PR shard matrix. Averaged from two real PR runs of .github/workflows/pr.yml (actions runs 38076844891 and 38076972444, 2026-10-10, on the runs-on paperclip-public-pr-x64 fleet) by diffing consecutive per-suite completion timestamps in the 'Run grouped general test suites' logs across every shard — that captures each suite's true serial cost (import, setup, and run) under maxWorkers=1. The first suite of each shard is measured from Vitest's RUN banner, so per-shard startup lands on a measured suite instead of vanishing. The chat and native-runner suites run in their own PR lanes, so their entries carry over from the 2026-09-15 measurement. Suites missing here get the median weight, so the manifest only needs occasional refreshes; regenerate the same way when the lane's shard steps drift apart by more than the heaviest suite.", "unit": "ms", "durations": { - "server/src/__tests__/access-service.test.ts": 6334, - "server/src/__tests__/access-validators.test.ts": 643, - "server/src/__tests__/activity-log-responsible-user.test.ts": 5558, - "server/src/__tests__/activity-service.test.ts": 6025, - "server/src/__tests__/adapter-models.test.ts": 1186, - "server/src/__tests__/adapter-registry.test.ts": 1215, - "server/src/__tests__/adapter-session-codecs.test.ts": 788, - "server/src/__tests__/agent-auth-jwt.test.ts": 709, - "server/src/__tests__/agent-auth-middleware.test.ts": 1106, - "server/src/__tests__/agent-conversations.test.ts": 10707, - "server/src/__tests__/agent-hire-ai-connections.test.ts": 9836, - "server/src/__tests__/agent-instructions-service.test.ts": 206, - "server/src/__tests__/agent-invokability.test.ts": 1093, - "server/src/__tests__/agent-permissions-service.test.ts": 721, - "server/src/__tests__/agent-shortname-collision.test.ts": 2217, - "server/src/__tests__/agent-skill-contract.test.ts": 224, - "server/src/__tests__/agent-task-run-telemetry.test.ts": 6355, - "server/src/__tests__/agentmail-api.test.ts": 1037, - "server/src/__tests__/agents-adapter-config-user-secret.test.ts": 6749, - "server/src/__tests__/agents-claude-oauth-binding.test.ts": 8607, - "server/src/__tests__/agents-paperclip-runner-skills.test.ts": 663, - "server/src/__tests__/agents-pending-approval-config.test.ts": 6398, - "server/src/__tests__/agents-service-clear-error.test.ts": 6976, - "server/src/__tests__/agents-service-secret-bindings.test.ts": 7149, - "server/src/__tests__/ai-connections.test.ts": 9795, - "server/src/__tests__/ai-legacy-compatibility.test.ts": 6594, - "server/src/__tests__/announcement-animation.test.ts": 1109, - "server/src/__tests__/announcement-feed.test.ts": 1274, - "server/src/__tests__/announcement-publisher.test.ts": 1232, - "server/src/__tests__/api-compression.test.ts": 307, - "server/src/__tests__/app-hmr-port.test.ts": 5467, - "server/src/__tests__/app-private-hostname-gate.test.ts": 5273, - "server/src/__tests__/app-vite-dev-routing.test.ts": 3593, - "server/src/__tests__/approvals-service.test.ts": 995, - "server/src/__tests__/artifact-review-documents-service.test.ts": 5832, - "server/src/__tests__/attachment-types.test.ts": 166, - "server/src/__tests__/attention-service.test.ts": 8689, - "server/src/__tests__/authorization-service.test.ts": 7152, - "server/src/__tests__/aws-secrets-manager-provider.test.ts": 291, - "server/src/__tests__/better-auth-credential-signup.integration.test.ts": 6546, - "server/src/__tests__/better-auth.test.ts": 1039, - "server/src/__tests__/board-claim.test.ts": 5641, - "server/src/__tests__/board-mutation-guard.test.ts": 233, - "server/src/__tests__/board-native-attachments.integration.test.ts": 6974, - "server/src/__tests__/body-limits.test.ts": 188, - "server/src/__tests__/budgets-service.test.ts": 5588, - "server/src/__tests__/build-commit.test.ts": 156, - "server/src/__tests__/build-version.test.ts": 186, - "server/src/__tests__/built-in-agents.test.ts": 14258, - "server/src/__tests__/bundled-plugins.test.ts": 221, - "server/src/__tests__/change-consent-gate.test.ts": 5793, + "server/scripts/verify-runner-vendor-dependencies.test.mjs": 186, + "server/src/__tests__/access-service.test.ts": 8447, + "server/src/__tests__/access-validators.test.ts": 744, + "server/src/__tests__/activity-credential-access.test.ts": 7944, + "server/src/__tests__/activity-log-responsible-user.test.ts": 8073, + "server/src/__tests__/activity-service.test.ts": 10246, + "server/src/__tests__/adapter-cost-failure-paths.test.ts": 5615, + "server/src/__tests__/adapter-models.test.ts": 1434, + "server/src/__tests__/adapter-registry.test.ts": 1448, + "server/src/__tests__/adapter-session-codecs.test.ts": 858, + "server/src/__tests__/agent-auth-jwt.test.ts": 737, + "server/src/__tests__/agent-auth-middleware.test.ts": 1290, + "server/src/__tests__/agent-avatars.test.ts": 8287, + "server/src/__tests__/agent-commentary-daytona-smoke.test.ts": 1254, + "server/src/__tests__/agent-commentary-helper.test.ts": 10976, + "server/src/__tests__/agent-commentary-skills.test.ts": 930, + "server/src/__tests__/agent-commentary.integration.test.ts": 15224, + "server/src/__tests__/agent-conversations.test.ts": 16683, + "server/src/__tests__/agent-default-configure-grants.test.ts": 9156, + "server/src/__tests__/agent-directory-probe.test.ts": 1493, + "server/src/__tests__/agent-directory-working-copies.test.ts": 45751, + "server/src/__tests__/agent-file-checkpoints.test.ts": 1528, + "server/src/__tests__/agent-harness-verification.test.ts": 2356, + "server/src/__tests__/agent-hire-ai-connections.test.ts": 17137, + "server/src/__tests__/agent-identity-master-key.test.ts": 519, + "server/src/__tests__/agent-identity.test.ts": 21467, + "server/src/__tests__/agent-instruction-git-exclusion.test.ts": 430, + "server/src/__tests__/agent-instruction-revisions.test.ts": 9689, + "server/src/__tests__/agent-instruction-working-copies.test.ts": 11715, + "server/src/__tests__/agent-instructions-service.test.ts": 1367, + "server/src/__tests__/agent-invokability.test.ts": 1116, + "server/src/__tests__/agent-permissions-service.test.ts": 680, + "server/src/__tests__/agent-persona-persistence.test.ts": 8731, + "server/src/__tests__/agent-profile-avatar.test.ts": 10471, + "server/src/__tests__/agent-shortname-collision.test.ts": 2260, + "server/src/__tests__/agent-skill-contract.test.ts": 201, + "server/src/__tests__/agent-task-run-telemetry.test.ts": 8482, + "server/src/__tests__/agentmail-api.test.ts": 1381, + "server/src/__tests__/agents-adapter-config-user-secret.test.ts": 10074, + "server/src/__tests__/agents-claude-oauth-binding.test.ts": 11793, + "server/src/__tests__/agents-paperclip-runner-skills.test.ts": 780, + "server/src/__tests__/agents-pending-approval-config.test.ts": 8779, + "server/src/__tests__/agents-service-clear-error.test.ts": 9643, + "server/src/__tests__/agents-service-secret-bindings.test.ts": 9581, + "server/src/__tests__/aggregator-app-discovery.test.ts": 250, + "server/src/__tests__/aggregator-app-sync.test.ts": 10208, + "server/src/__tests__/ai-auth-failure.test.ts": 777, + "server/src/__tests__/ai-connection-session.test.ts": 2974, + "server/src/__tests__/ai-connections.test.ts": 33549, + "server/src/__tests__/ai-legacy-compatibility.test.ts": 9149, + "server/src/__tests__/ai-provider-routing.test.ts": 749, + "server/src/__tests__/announcement-animation.test.ts": 1315, + "server/src/__tests__/announcement-feed.test.ts": 1395, + "server/src/__tests__/announcement-publisher.test.ts": 1360, + "server/src/__tests__/api-compression.test.ts": 266, + "server/src/__tests__/app-hmr-port.test.ts": 6653, + "server/src/__tests__/app-private-hostname-gate.test.ts": 7728, + "server/src/__tests__/app-vite-dev-routing.test.ts": 5394, + "server/src/__tests__/approvals-service.test.ts": 2191, + "server/src/__tests__/artifact-review-documents-service.test.ts": 8702, + "server/src/__tests__/attachment-types.test.ts": 164, + "server/src/__tests__/attention-service.test.ts": 12212, + "server/src/__tests__/authorization-service.test.ts": 10426, + "server/src/__tests__/aws-secrets-manager-provider.test.ts": 263, + "server/src/__tests__/better-auth-credential-signup.integration.test.ts": 8331, + "server/src/__tests__/better-auth.test.ts": 955, + "server/src/__tests__/board-claim.test.ts": 7534, + "server/src/__tests__/board-mutation-guard.test.ts": 235, + "server/src/__tests__/board-native-attachments.integration.test.ts": 9772, + "server/src/__tests__/body-limits.test.ts": 163, + "server/src/__tests__/browser-use-connection.test.ts": 18179, + "server/src/__tests__/browser-use-viewport.test.ts": 800, + "server/src/__tests__/browser-use-work.test.ts": 591, + "server/src/__tests__/browser-use.test.ts": 746, + "server/src/__tests__/budgets-service.test.ts": 9254, + "server/src/__tests__/build-commit.test.ts": 163, + "server/src/__tests__/build-version.test.ts": 152, + "server/src/__tests__/built-in-agents.test.ts": 25931, + "server/src/__tests__/bundled-plugins.test.ts": 282, + "server/src/__tests__/change-consent-gate.test.ts": 7435, + "server/src/__tests__/chat-action-work.test.ts": 599, "server/src/__tests__/chat-channels.integration.test.ts": 428744, - "server/src/__tests__/chat-interaction-publications.test.ts": 7076, - "server/src/__tests__/chat-project-tools.test.ts": 10896, - "server/src/__tests__/chat-webhook-diagnostics.test.ts": 7018, - "server/src/__tests__/chat-webhook-public-url.test.ts": 1193, - "server/src/__tests__/claude-local-adapter-environment.test.ts": 1176, - "server/src/__tests__/claude-local-adapter.test.ts": 758, - "server/src/__tests__/claude-local-execute.test.ts": 3581, - "server/src/__tests__/claude-local-skill-sync.test.ts": 639, - "server/src/__tests__/cleanup-removal-service.test.ts": 7531, - "server/src/__tests__/cli-invocation-safety.test.ts": 1940, - "server/src/__tests__/cloud-control-task-drain.test.ts": 833, - "server/src/__tests__/cloud-image-bundled-plugins.test.ts": 162, - "server/src/__tests__/cloud-image-sentry.test.ts": 257, - "server/src/__tests__/cloud-instance.test.ts": 185, - "server/src/__tests__/cloud-runtime-identity.test.ts": 8181, - "server/src/__tests__/cloud-tenant-company-provisioning.test.ts": 5723, - "server/src/__tests__/cloud-tenant-transient-db-retry.test.ts": 1487, - "server/src/__tests__/cloud-ui-snippet.test.ts": 194, - "server/src/__tests__/codex-auth-reconciliation.test.ts": 1248, - "server/src/__tests__/codex-local-adapter-environment.test.ts": 905, - "server/src/__tests__/codex-local-adapter.test.ts": 786, - "server/src/__tests__/codex-local-execute.test.ts": 4536, - "server/src/__tests__/codex-local-skill-injection.test.ts": 786, - "server/src/__tests__/codex-local-skill-sync.test.ts": 632, - "server/src/__tests__/companies-service.test.ts": 7789, - "server/src/__tests__/company-artifacts-service.test.ts": 7582, - "server/src/__tests__/company-cloud-floor.test.ts": 1266, - "server/src/__tests__/company-import-cloud-floor.test.ts": 1421, - "server/src/__tests__/company-portability-import-batching.test.ts": 8021, - "server/src/__tests__/company-search-extract-service.test.ts": 6125, - "server/src/__tests__/company-search-service.test.ts": 6835, - "server/src/__tests__/company-skill-import-boundary.test.ts": 6629, - "server/src/__tests__/company-skill-policy-service.test.ts": 5806, - "server/src/__tests__/company-skill-test-runs-service.test.ts": 10485, - "server/src/__tests__/company-skills-catalog-service.test.ts": 10675, - "server/src/__tests__/company-skills-detail.test.ts": 6905, - "server/src/__tests__/company-skills-service.test.ts": 22837, - "server/src/__tests__/company-skills.test.ts": 2063, - "server/src/__tests__/company-transfer-runs.test.ts": 5541, - "server/src/__tests__/config-file.test.ts": 646, - "server/src/__tests__/connection-grant-resolution.test.ts": 1732, - "server/src/__tests__/connection-intents-service.test.ts": 8169, - "server/src/__tests__/container-init-reaping.test.ts": 189, - "server/src/__tests__/cross-issue-influence-limit-postgres.test.ts": 5522, - "server/src/__tests__/cross-issue-influence-limit.test.ts": 990, - "server/src/__tests__/cursor-local-adapter-environment.test.ts": 3049, - "server/src/__tests__/cursor-local-adapter.test.ts": 797, - "server/src/__tests__/cursor-local-execute.test.ts": 1840, - "server/src/__tests__/cursor-local-skill-injection.test.ts": 712, - "server/src/__tests__/cursor-local-skill-sync.test.ts": 798, - "server/src/__tests__/dashboard-service.test.ts": 5754, - "server/src/__tests__/db-errors.test.ts": 190, - "server/src/__tests__/decision-retention-service.test.ts": 7932, - "server/src/__tests__/decision-training.test.ts": 8829, - "server/src/__tests__/decision-wakeup.test.ts": 161, - "server/src/__tests__/decisions-service.test.ts": 9462, - "server/src/__tests__/dev-runner-native-binary.test.ts": 213, - "server/src/__tests__/dev-runner-options.test.ts": 168, - "server/src/__tests__/dev-runner-output.test.ts": 208, - "server/src/__tests__/dev-runner-paths.test.ts": 184, - "server/src/__tests__/dev-runner-snapshot.test.ts": 164, - "server/src/__tests__/dev-runner-worktree.test.ts": 193, - "server/src/__tests__/dev-server-status.test.ts": 305, - "server/src/__tests__/dev-watch-ignore.test.ts": 192, - "server/src/__tests__/device-login-credential-read.test.ts": 633, - "server/src/__tests__/device-login-reaper.test.ts": 1076, - "server/src/__tests__/device-login-service.test.ts": 4771, - "server/src/__tests__/disposition-repair.test.ts": 2246, - "server/src/__tests__/docker-build-stamp.test.ts": 196, - "server/src/__tests__/docker-entrypoint.test.ts": 273, - "server/src/__tests__/document-annotations-service.test.ts": 8516, - "server/src/__tests__/documents-service.test.ts": 6625, - "server/src/__tests__/documents.test.ts": 961, - "server/src/__tests__/duplex-observability-recorder.test.ts": 205, - "server/src/__tests__/durable-chat-wakeup.test.ts": 12293, - "server/src/__tests__/effective-run-config-fingerprints.test.ts": 172, - "server/src/__tests__/email-channels.integration.test.ts": 14127, - "server/src/__tests__/embedded-postgres-supervisor.test.ts": 158, - "server/src/__tests__/env-file-policy.test.ts": 202, - "server/src/__tests__/environment-capability-contract.test.ts": 1662, - "server/src/__tests__/environment-config.test.ts": 1450, - "server/src/__tests__/environment-custom-image-terminal-ws.test.ts": 2815, - "server/src/__tests__/environment-custom-images-service.test.ts": 10412, - "server/src/__tests__/environment-driver-traits.test.ts": 161, - "server/src/__tests__/environment-execution-target-capabilities.test.ts": 813, - "server/src/__tests__/environment-execution-target-duplex-kill-switch.test.ts": 841, - "server/src/__tests__/environment-execution-target-duplex.test.ts": 1134, - "server/src/__tests__/environment-execution-target.test.ts": 796, - "server/src/__tests__/environment-live-ssh.test.ts": 1595, - "server/src/__tests__/environment-probe.test.ts": 1483, - "server/src/__tests__/environment-run-orchestrator.test.ts": 1629, - "server/src/__tests__/environment-runtime-driver-contract.test.ts": 5548, - "server/src/__tests__/environment-runtime-duplex-bridge-input.test.ts": 4620, - "server/src/__tests__/environment-runtime.test.ts": 9242, - "server/src/__tests__/environment-service.test.ts": 6048, - "server/src/__tests__/environment-test-harness.test.ts": 675, - "server/src/__tests__/error-handler.test.ts": 217, - "server/src/__tests__/execution-lock-orphan-cleanup.test.ts": 7076, - "server/src/__tests__/execution-workspace-policy.test.ts": 608, - "server/src/__tests__/execution-workspace-reopen.test.ts": 6968, - "server/src/__tests__/execution-workspace-runtime-control-conflict.test.ts": 2223, - "server/src/__tests__/execution-workspaces-service.test.ts": 20667, - "server/src/__tests__/export-fidelity.test.ts": 5683, - "server/src/__tests__/external-objects-service.test.ts": 6727, - "server/src/__tests__/feedback-flush-controller.test.ts": 3610, - "server/src/__tests__/feedback-service.test.ts": 5887, - "server/src/__tests__/feedback-share-client.test.ts": 193, - "server/src/__tests__/file-delivery-bridges.test.ts": 14084, - "server/src/__tests__/file-resources-git-scan-load.test.ts": 2835, - "server/src/__tests__/file-resources.test.ts": 13553, - "server/src/__tests__/first-admin-claim.test.ts": 5526, - "server/src/__tests__/folders-service.test.ts": 6914, - "server/src/__tests__/forbidden-tokens.test.ts": 192, - "server/src/__tests__/gemini-local-adapter-environment.test.ts": 680, - "server/src/__tests__/gemini-local-adapter.test.ts": 741, - "server/src/__tests__/gemini-local-execute.test.ts": 862, - "server/src/__tests__/gemini-local-skill-sync.test.ts": 713, - "server/src/__tests__/general-capability-classifier.test.ts": 1457, - "server/src/__tests__/generic-mcp-connection.test.ts": 11804, - "server/src/__tests__/git-credentials.test.ts": 2163, - "server/src/__tests__/github-commit-details.test.ts": 2068, - "server/src/__tests__/github-connection-events.test.ts": 7465, - "server/src/__tests__/github-grant-metadata.test.ts": 1773, - "server/src/__tests__/github-operation-credentials.test.ts": 7527, - "server/src/__tests__/google-sheets-gallery.test.ts": 2025, - "server/src/__tests__/grok-local-skill-sync.test.ts": 802, - "server/src/__tests__/health-workspace-readiness.test.ts": 1392, - "server/src/__tests__/heartbeat-accepted-plan-workspace-refresh.test.ts": 10247, - "server/src/__tests__/heartbeat-active-run-output-watchdog.test.ts": 10196, - "server/src/__tests__/heartbeat-agent-session-message.test.ts": 3713, - "server/src/__tests__/heartbeat-archived-company-guard.test.ts": 7346, - "server/src/__tests__/heartbeat-auto-checkout.test.ts": 2631, - "server/src/__tests__/heartbeat-comment-wake-batching.test.ts": 31799, - "server/src/__tests__/heartbeat-context-summary.test.ts": 2323, - "server/src/__tests__/heartbeat-cost-accounting.test.ts": 2650, - "server/src/__tests__/heartbeat-direct-adapter-native-isolation.test.ts": 9556, - "server/src/__tests__/heartbeat-issue-rewake-throttle.test.ts": 10312, - "server/src/__tests__/heartbeat-ledger-billing-code.test.ts": 2640, - "server/src/__tests__/heartbeat-list.test.ts": 7073, - "server/src/__tests__/heartbeat-local-environment.test.ts": 9429, - "server/src/__tests__/heartbeat-lock-release-on-reassignment.test.ts": 9124, - "server/src/__tests__/heartbeat-managed-clone-credentials.test.ts": 3610, - "server/src/__tests__/heartbeat-native-runner-cancellation.test.ts": 2985, - "server/src/__tests__/heartbeat-native-runner-selection.test.ts": 189, - "server/src/__tests__/heartbeat-native-status-context.test.ts": 2617, - "server/src/__tests__/heartbeat-pending-cleanup-sweep.test.ts": 9191, - "server/src/__tests__/heartbeat-plugin-environment.test.ts": 7202, - "server/src/__tests__/heartbeat-project-env.test.ts": 2879, - "server/src/__tests__/heartbeat-project-repositories.test.ts": 12549, - "server/src/__tests__/heartbeat-referenced-projects.test.ts": 7337, - "server/src/__tests__/heartbeat-remote-referenced-projects.test.ts": 2667, - "server/src/__tests__/heartbeat-responsible-user-invariant.test.ts": 12037, - "server/src/__tests__/heartbeat-retry-scheduling.test.ts": 13920, - "server/src/__tests__/heartbeat-reviewed-chat-binding.integration.test.ts": 7952, - "server/src/__tests__/heartbeat-run-event-sequencing.test.ts": 12508, - "server/src/__tests__/heartbeat-run-lease-release-terminalization.test.ts": 7583, - "server/src/__tests__/heartbeat-run-log.test.ts": 2657, - "server/src/__tests__/heartbeat-run-status-payload.test.ts": 2921, - "server/src/__tests__/heartbeat-run-summary.test.ts": 214, - "server/src/__tests__/heartbeat-run-terminalize-before-release.test.ts": 7102, - "server/src/__tests__/heartbeat-running-followup.test.ts": 2638, - "server/src/__tests__/heartbeat-runtime-mcp-servers.test.ts": 9360, - "server/src/__tests__/heartbeat-runtime-skills.test.ts": 10111, - "server/src/__tests__/heartbeat-runtime-state.test.ts": 7809, - "server/src/__tests__/heartbeat-scheduling-suppression.test.ts": 2650, - "server/src/__tests__/heartbeat-stale-queue-invalidation.test.ts": 27050, - "server/src/__tests__/heartbeat-start-lock.test.ts": 234, - "server/src/__tests__/heartbeat-status-preserving-telemetry.test.ts": 7639, - "server/src/__tests__/heartbeat-task-drain-admission-release.test.ts": 8838, - "server/src/__tests__/heartbeat-task-drain.test.ts": 2282, - "server/src/__tests__/heartbeat-task-session-reset.test.ts": 7264, - "server/src/__tests__/heartbeat-timer-wake-session-reset-pf4.test.ts": 2642, - "server/src/__tests__/heartbeat-workspace-branch-containment.test.ts": 11586, - "server/src/__tests__/heartbeat-workspace-busy.test.ts": 14605, - "server/src/__tests__/heartbeat-workspace-finalize-branch.test.ts": 10413, - "server/src/__tests__/heartbeat-workspace-ready-comment.test.ts": 2933, - "server/src/__tests__/heartbeat-workspace-session.test.ts": 2611, - "server/src/__tests__/heartbeat-worktree-suppression.test.ts": 9941, - "server/src/__tests__/heartbeat-zombie-guard.test.ts": 2828, - "server/src/__tests__/hire-hook.test.ts": 634, - "server/src/__tests__/hiring-operational-examples.test.ts": 667, - "server/src/__tests__/http-adapter-remote-fetch.test.ts": 230, - "server/src/__tests__/http-log-policy.test.ts": 207, - "server/src/__tests__/http-log-redaction.test.ts": 1360, - "server/src/__tests__/inbox-dismissals.test.ts": 7106, - "server/src/__tests__/instance-settings-cloud-defaults.test.ts": 1010, - "server/src/__tests__/instance-settings-managed-overlay.test.ts": 1165, - "server/src/__tests__/instance-settings-operator-defaults.test.ts": 1130, - "server/src/__tests__/instance-settings-service.test.ts": 1163, - "server/src/__tests__/instrumentation.test.ts": 245, - "server/src/__tests__/interaction-resolution-cross-issue-cap-postgres.test.ts": 11048, - "server/src/__tests__/invite-join-grants.test.ts": 635, - "server/src/__tests__/invite-rate-limit.test.ts": 183, - "server/src/__tests__/invite-token-entropy.test.ts": 2816, - "server/src/__tests__/issue-blocker-attention.test.ts": 7828, - "server/src/__tests__/issue-comment-redaction.test.ts": 8688, - "server/src/__tests__/issue-continuation-summary.test.ts": 1104, - "server/src/__tests__/issue-execution-policy.test.ts": 752, - "server/src/__tests__/issue-existing-branch-validation-status.test.ts": 1328, - "server/src/__tests__/issue-goal-fallback.test.ts": 188, - "server/src/__tests__/issue-interactions-read-only-contract.test.ts": 192, - "server/src/__tests__/issue-liveness.test.ts": 557, - "server/src/__tests__/issue-monitor-scheduler.test.ts": 16200, - "server/src/__tests__/issue-recovery-actions.test.ts": 12381, - "server/src/__tests__/issue-references-service.test.ts": 6056, - "server/src/__tests__/issue-remove-cascade.test.ts": 7455, - "server/src/__tests__/issue-review-attention.test.ts": 7298, - "server/src/__tests__/issue-review-policy.test.ts": 5950, - "server/src/__tests__/issue-rewake-throttle.test.ts": 165, - "server/src/__tests__/issue-runtime-workspace-binding.test.ts": 6869, - "server/src/__tests__/issue-thread-interactions-service.test.ts": 8954, - "server/src/__tests__/issue-thread-interactions-telemetry.test.ts": 7388, - "server/src/__tests__/issue-tree-control-service-unit.test.ts": 1131, - "server/src/__tests__/issue-tree-control-service.test.ts": 8166, - "server/src/__tests__/issues-list-query-parsing.test.ts": 7945, - "server/src/__tests__/issues-parent-id-alias.test.ts": 7482, - "server/src/__tests__/issues-user-context.test.ts": 1984, - "server/src/__tests__/join-request-dedupe.test.ts": 183, - "server/src/__tests__/json-schema-secret-refs.test.ts": 159, - "server/src/__tests__/kimi-local-skill-sync.test.ts": 605, - "server/src/__tests__/legacy-finalization-regression.test.ts": 8295, - "server/src/__tests__/live-events-ws.test.ts": 771, - "server/src/__tests__/local-ai-credential-file.test.ts": 164, - "server/src/__tests__/local-ai-credentials.test.ts": 720, - "server/src/__tests__/local-ai-login-policy.test.ts": 192, - "server/src/__tests__/local-service-supervisor.test.ts": 4146, - "server/src/__tests__/log-redaction.test.ts": 188, - "server/src/__tests__/logger-tz.test.ts": 215, - "server/src/__tests__/login-session-runtime.test.ts": 1329, - "server/src/__tests__/managed-config.test.ts": 679, - "server/src/__tests__/managed-loopback-auth.test.ts": 6465, - "server/src/__tests__/managed-runtime-public-url.test.ts": 560, - "server/src/__tests__/mcp-http.test.ts": 221, - "server/src/__tests__/merged-pr-confirmation-sweep.test.ts": 6865, - "server/src/__tests__/monthly-spend-service.test.ts": 2355, - "server/src/__tests__/native-event-boundary-golden.test.ts": 5431, - "server/src/__tests__/native-finalization-migration.test.ts": 5232, - "server/src/__tests__/native-finalization-recovery.test.ts": 11707, - "server/src/__tests__/native-interaction-bridge.test.ts": 8463, - "server/src/__tests__/native-run-finalizer.test.ts": 13674, - "server/src/__tests__/native-runner-input-boundary.test.ts": 211, - "server/src/__tests__/native-runner-phase6.integration.test.ts": 14534, - "server/src/__tests__/native-sandbox-lifecycle.test.ts": 2285, - "server/src/__tests__/native-session-resumption.test.ts": 22979, - "server/src/__tests__/native-status-arbiter-corpus.test.ts": 18510, - "server/src/__tests__/native-workspace-sync.test.ts": 1391, - "server/src/__tests__/oauth-client-metadata.test.ts": 1752, - "server/src/__tests__/openclaw-gateway-adapter.test.ts": 734, - "server/src/__tests__/opencode-local-adapter.test.ts": 777, - "server/src/__tests__/opencode-local-skill-sync.test.ts": 730, - "server/src/__tests__/paperclip-env.test.ts": 595, - "server/src/__tests__/paperclip-issue-update-helper.test.ts": 4367, - "server/src/__tests__/paperclip-semantic-conformance.test.ts": 7858, - "server/src/__tests__/paperclip-skill-utils.test.ts": 8266, - "server/src/__tests__/parse-status-filter.test.ts": 1753, - "server/src/__tests__/photon/channel.integration.test.ts": 17734, - "server/src/__tests__/photon/photon.test.ts": 1049, - "server/src/__tests__/pi-local-adapter-environment.test.ts": 819, - "server/src/__tests__/pi-local-execute.test.ts": 963, - "server/src/__tests__/pi-local-skill-sync.test.ts": 793, - "server/src/__tests__/pipelines-service.test.ts": 10897, - "server/src/__tests__/plugin-access-authorization-host-services.test.ts": 7771, - "server/src/__tests__/plugin-agent-sessions.test.ts": 1857, - "server/src/__tests__/plugin-config-startup-delivery.test.ts": 5604, - "server/src/__tests__/plugin-database.test.ts": 6829, - "server/src/__tests__/plugin-dev-watcher.test.ts": 243, - "server/src/__tests__/plugin-environment-driver-ready-recovery.test.ts": 5294, - "server/src/__tests__/plugin-environment-driver-sandbox-capabilities.test.ts": 191, - "server/src/__tests__/plugin-environment-driver-seam.test.ts": 764, - "server/src/__tests__/plugin-execution-workspace-bridge.test.ts": 160, - "server/src/__tests__/plugin-host-services-span.test.ts": 3086, - "server/src/__tests__/plugin-install-autobuild.test.ts": 7981, - "server/src/__tests__/plugin-install-guard.test.ts": 1254, - "server/src/__tests__/plugin-lifecycle-restart.test.ts": 1339, - "server/src/__tests__/plugin-loader-error-retry.test.ts": 1234, - "server/src/__tests__/plugin-local-folders.test.ts": 251, - "server/src/__tests__/plugin-managed-agents.test.ts": 7785, - "server/src/__tests__/plugin-managed-routines.test.ts": 6739, - "server/src/__tests__/plugin-managed-skills.test.ts": 8331, - "server/src/__tests__/plugin-orchestration-apis.test.ts": 10461, - "server/src/__tests__/plugin-sandbox-provider-config-validation.test.ts": 166, - "server/src/__tests__/plugin-sdk-orchestration-contract.test.ts": 696, - "server/src/__tests__/plugin-sdk-testing.test.ts": 719, - "server/src/__tests__/plugin-secrets-handler.test.ts": 5977, - "server/src/__tests__/plugin-telemetry-bridge.test.ts": 2401, - "server/src/__tests__/plugin-tenant-isolation.test.ts": 6827, - "server/src/__tests__/plugin-tool-dispatcher-pluginDbId.test.ts": 787, - "server/src/__tests__/plugin-ui-static.test.ts": 1076, - "server/src/__tests__/plugin-worker-manager-duplex.test.ts": 2338, - "server/src/__tests__/plugin-worker-manager.test.ts": 4215, - "server/src/__tests__/private-hostname-guard.test.ts": 254, - "server/src/__tests__/private-json-etag.test.ts": 246, - "server/src/__tests__/project-goal-validation.test.ts": 5592, - "server/src/__tests__/project-icon-persistence.test.ts": 5688, - "server/src/__tests__/project-list-metrics.test.ts": 1156, - "server/src/__tests__/project-repositories-persistence.test.ts": 6390, - "server/src/__tests__/project-repositories.test.ts": 2019, - "server/src/__tests__/project-shortname-resolution.test.ts": 1301, - "server/src/__tests__/provider-trace-store.test.ts": 764, - "server/src/__tests__/qa-routine-secrets-e2e.test.ts": 8308, - "server/src/__tests__/question-response-delivery.test.ts": 8581, - "server/src/__tests__/quota-windows-service.test.ts": 190, - "server/src/__tests__/quota-windows.test.ts": 811, - "server/src/__tests__/recovery-classifiers.test.ts": 1779, - "server/src/__tests__/recovery-observability.test.ts": 6511, - "server/src/__tests__/recovery-stale-issue-lock-sweep.test.ts": 7764, - "server/src/__tests__/redact-sensitive.test.ts": 209, - "server/src/__tests__/remote-http-endpoint-guard.test.ts": 218, - "server/src/__tests__/remote-http-rebinding.test.ts": 864, - "server/src/__tests__/rerun-survival.integration.test.ts": 739, - "server/src/__tests__/reusable-leases-default.test.ts": 548, - "server/src/__tests__/routable-blocked.test.ts": 185, - "server/src/__tests__/routine-run-telemetry.test.ts": 8288, - "server/src/__tests__/routines-service.test.ts": 13600, - "server/src/__tests__/run-continuations.test.ts": 627, - "server/src/__tests__/run-failure-sentry.test.ts": 231, - "server/src/__tests__/run-identity.test.ts": 6630, - "server/src/__tests__/run-liveness.test.ts": 204, - "server/src/__tests__/run-secret-redaction.test.ts": 653, - "server/src/__tests__/run-trust-preset.test.ts": 5707, - "server/src/__tests__/runner-connection-eval-fixtures.test.ts": 9969, - "server/src/__tests__/runtime-api.test.ts": 209, - "server/src/__tests__/runtime-skill-cache.test.ts": 3370, - "server/src/__tests__/sandbox-orphan-cleanup-spool.test.ts": 250, - "server/src/__tests__/sandbox-provider-runtime.test.ts": 197, - "server/src/__tests__/secret-provider-registry.test.ts": 206, - "server/src/__tests__/secrets-service-dangling-user-secret.test.ts": 6065, - "server/src/__tests__/secrets-service-user-secret-owner-scoped.test.ts": 6087, - "server/src/__tests__/secrets-service.test.ts": 8564, - "server/src/__tests__/sentry-dsn.test.ts": 205, - "server/src/__tests__/sentry.test.ts": 1139, - "server/src/__tests__/server-info.test.ts": 252, - "server/src/__tests__/server-package-build-script.test.ts": 184, - "server/src/__tests__/server-package-otel-peer-metadata.test.ts": 188, - "server/src/__tests__/server-package-sentry-peer-metadata.test.ts": 182, - "server/src/__tests__/server-startup-feedback-export.test.ts": 1425, - "server/src/__tests__/service-version.test.ts": 202, - "server/src/__tests__/shared-telemetry-events.test.ts": 203, - "server/src/__tests__/skills-catalog-service.test.ts": 228, - "server/src/__tests__/smoke-lab.test.ts": 7255, - "server/src/__tests__/source-trust.test.ts": 5684, - "server/src/__tests__/startup-refusals.test.ts": 204, - "server/src/__tests__/static-index-html.test.ts": 284, - "server/src/__tests__/static-ui-cache.test.ts": 157, - "server/src/__tests__/status-card-update-engine.test.ts": 651, - "server/src/__tests__/status-cards.test.ts": 9392, - "server/src/__tests__/storage-local-provider.test.ts": 203, - "server/src/__tests__/summary-slots.test.ts": 8619, - "server/src/__tests__/task-search-quality.test.ts": 7470, - "server/src/__tests__/task-search-query.test.ts": 1158, - "server/src/__tests__/task-watchdogs-classifier.test.ts": 2008, - "server/src/__tests__/task-watchdogs-incident-fixture.test.ts": 158, - "server/src/__tests__/task-watchdogs-scheduler.test.ts": 8370, - "server/src/__tests__/teams-catalog-install-no-overrides.test.ts": 11259, - "server/src/__tests__/teams-catalog-service.test.ts": 720, - "server/src/__tests__/telemetry-client-flush.test.ts": 171, - "server/src/__tests__/tool-access-policy-service.test.ts": 8350, - "server/src/__tests__/tool-access-service.test.ts": 36320, - "server/src/__tests__/tool-connection-removal.test.ts": 8967, - "server/src/__tests__/tool-connection-visibility.test.ts": 2412, - "server/src/__tests__/tool-content-guards.test.ts": 222, - "server/src/__tests__/tool-gateway-service.test.ts": 13810, - "server/src/__tests__/tool-gateway.test.ts": 18820, - "server/src/__tests__/tool-oauth-legacy-backfill.test.ts": 6532, - "server/src/__tests__/tool-review-queue-unsigned-request.test.ts": 6766, - "server/src/__tests__/trust-preset-resolver.test.ts": 672, - "server/src/__tests__/trust-proxy.test.ts": 248, - "server/src/__tests__/ui-branding.test.ts": 186, - "server/src/__tests__/version.test.ts": 179, - "server/src/__tests__/vite-html-renderer.test.ts": 165, - "server/src/__tests__/vitest-chat-shards.test.ts": 7687, - "server/src/__tests__/work-product-runtime-reconciliation.test.ts": 2166, - "server/src/__tests__/work-products.test.ts": 1995, - "server/src/__tests__/work-timeline-service.test.ts": 7827, - "server/src/__tests__/workspace-git-inspection-cache.test.ts": 191, - "server/src/__tests__/workspace-instance-cleanup.test.ts": 192, - "server/src/__tests__/workspace-login-handoff-endpoint.test.ts": 1180, - "server/src/__tests__/workspace-login-handoff-issuer.test.ts": 654, - "server/src/__tests__/workspace-login-handoff.test.ts": 208, - "server/src/__tests__/workspace-operations-reconciliation.test.ts": 6083, - "server/src/__tests__/workspace-readiness.test.ts": 785, - "server/src/__tests__/workspace-reaper-cooldown-config.test.ts": 746, - "server/src/__tests__/workspace-runtime-control-recovery.test.ts": 6656, - "server/src/__tests__/workspace-runtime-exposure-reservation.test.ts": 11616, - "server/src/__tests__/workspace-runtime-leases.test.ts": 6220, - "server/src/__tests__/workspace-runtime-start-terminality.test.ts": 5581, - "server/src/__tests__/workspace-runtime.test.ts": 129167, - "server/src/__tests__/worktree-config.test.ts": 1201, - "server/src/__tests__/worktree-seed-server-spawn.test.ts": 2118, - "server/src/__tests__/write-build-stamp.test.ts": 208, - "server/src/__tests__/write-path-membership.test.ts": 1347, - "server/src/adapters/claude-agent-id-header.test.ts": 188, - "server/src/adapters/http/execute.test.ts": 725, - "server/src/adapters/plugin-loader.test.ts": 183, - "server/src/adapters/registry.test.ts": 1165, - "server/src/embedded-postgres-owner.test.ts": 204, - "server/src/instrumentation.test.ts": 203, - "server/src/lib/request-promise-memo.test.ts": 195, - "server/src/middleware/cloud-tenant-actor.test.ts": 1166, - "server/src/modules/active-run-watchdog/adapters/process.test.ts": 595, - "server/src/modules/active-run-watchdog/application/use-cases.test.ts": 196, - "server/src/modules/active-run-watchdog/domain/policy.test.ts": 193, - "server/src/modules/run-dispatch/adapters/postgres.test.ts": 8784, - "server/src/modules/run-dispatch/application/use-cases.test.ts": 164, - "server/src/modules/run-dispatch/domain/policy.test.ts": 204, - "server/src/modules/run-dispatch/domain/wake-context.test.ts": 184, - "server/src/modules/wake-queue/adapters/postgres.test.ts": 9247, - "server/src/modules/wake-queue/adapters/queued-comment-postgres.test.ts": 5681, - "server/src/modules/wake-queue/application/queued-comment-use-cases.test.ts": 201, - "server/src/modules/wake-queue/application/use-cases.test.ts": 2061, - "server/src/modules/wake-queue/domain/context.test.ts": 2149, - "server/src/modules/wake-queue/domain/policy.test.ts": 175, - "server/src/realtime/runner-prp-outbound.test.ts": 203, - "server/src/realtime/runner-prp-ws.test.ts": 1096, - "server/src/routes/adapters.test.ts": 1621, - "server/src/routes/chat-channels.identity.test.ts": 4004, - "server/src/routes/chat-channels.webhook.test.ts": 3139, - "server/src/routes/connection-intents.test.ts": 2045, - "server/src/routes/tool-access-connection-intent.test.ts": 2817, - "server/src/runtime-tools-token.test.ts": 208, - "server/src/services/__tests__/run-failure-report.test.ts": 5457, - "server/src/services/acknowledged-native-stop.test.ts": 709, - "server/src/services/adapter-execution-control.test.ts": 188, - "server/src/services/adapter-login-lease.test.ts": 162, - "server/src/services/adapter-models-env.test.ts": 202, - "server/src/services/adapter-registry-bootstrap.reconcile.test.ts": 1260, - "server/src/services/adapter-registry-bootstrap.test.ts": 1138, - "server/src/services/approved-execution-wait.test.ts": 196, - "server/src/services/attention-resolver-audience.test.ts": 2010, - "server/src/services/board-auth.test.ts": 775, - "server/src/services/chat-control-admission-retry.test.ts": 1736, - "server/src/services/chat-discord-adapter-patch.test.ts": 3921, - "server/src/services/chat-discord-command-registration-store.test.ts": 6418, - "server/src/services/chat-discord-command-registration.test.ts": 349, - "server/src/services/chat-discord-modal-wire.test.ts": 1642, - "server/src/services/chat-discord-native-command.test.ts": 2105, - "server/src/services/chat-discord-native-commands.test.ts": 600, - "server/src/services/chat-discord-native-interaction.test.ts": 2629, - "server/src/services/chat-discord-native-modal.test.ts": 1649, - "server/src/services/chat-discord-question-forms.test.ts": 282, - "server/src/services/chat-discord.test.ts": 568, - "server/src/services/chat-github-attachments.test.ts": 2429, - "server/src/services/chat-github-provider-stress.test.ts": 1653, - "server/src/services/chat-github-receipt-reactions.test.ts": 446, - "server/src/services/chat-github-webhook-config.test.ts": 274, - "server/src/services/chat-inbound-wakeup-publications.test.ts": 769, - "server/src/services/chat-interaction-arbitration.test.ts": 4500, - "server/src/services/chat-interaction-publications.test.ts": 1271, - "server/src/services/chat-outbound-attachment-hydration.test.ts": 3704, - "server/src/services/chat-provider-inventory.test.ts": 244, - "server/src/services/chat-provider-lifecycle.test.ts": 183, - "server/src/services/chat-provider-links.test.ts": 189, - "server/src/services/chat-publication-batches.test.ts": 659, - "server/src/services/chat-publication-errors.test.ts": 228, - "server/src/services/chat-publication-projection.test.ts": 188, - "server/src/services/chat-publication-ready-stream.test.ts": 2342, - "server/src/services/chat-publication-reconciliation.test.ts": 203, - "server/src/services/chat-publication-stream.test.ts": 1167, - "server/src/services/chat-publication-text-parts.test.ts": 12813, - "server/src/services/chat-question-forms.test.ts": 1048, - "server/src/services/chat-run-publications.test.ts": 2040, - "server/src/services/chat-sdk-runtime.adapters.test.ts": 2659, - "server/src/services/chat-sdk-runtime.test.ts": 1841, - "server/src/services/chat-sdk-state.test.ts": 192, - "server/src/services/chat-slack-native-modal.test.ts": 2557, - "server/src/services/chat-slack-receipts.test.ts": 4349, - "server/src/services/chat-slack-sessions.test.ts": 176, - "server/src/services/chat-task-url.test.ts": 1997, - "server/src/services/chat-teams-credentials.test.ts": 661, - "server/src/services/chat-teams-file-consent-runtime.test.ts": 2178, - "server/src/services/chat-teams-file-consent.contract.test.ts": 718, - "server/src/services/chat-teams-file-consent.test.ts": 404, - "server/src/services/chat-teams-file-publication.test.ts": 6583, - "server/src/services/chat-teams-file-transfers.test.ts": 9209, - "server/src/services/chat-teams-inline-image-intake.test.ts": 1694, - "server/src/services/chat-teams-inline-images.test.ts": 2106, - "server/src/services/chat-teams-native-modal.test.ts": 2528, - "server/src/services/chat-teams-personal-recipient.test.ts": 835, - "server/src/services/chat-telegram-draft-stop.test.ts": 299, - "server/src/services/chat-telegram-ephemeral-runtime.test.ts": 1982, - "server/src/services/chat-telegram-ephemeral.test.ts": 243, - "server/src/services/chat-telegram-media-intake.test.ts": 2067, - "server/src/services/chat-telegram-photo.test.ts": 255, - "server/src/services/chat-telegram-rich-intake.test.ts": 1987, - "server/src/services/chat-telegram-stop-subscription.test.ts": 192, - "server/src/services/chat-telegram-video-note.test.ts": 2024, - "server/src/services/cloud-runtime-identity.test.ts": 619, - "server/src/services/composio.test.ts": 235, - "server/src/services/environment-custom-image-terminal-sessions.test.ts": 218, - "server/src/services/execution-allowlist.test.ts": 192, - "server/src/services/execution-continuation.test.ts": 6051, - "server/src/services/execution-control-reconciliation.test.ts": 6450, - "server/src/services/execution-policy-bootstrap.test.ts": 978, - "server/src/services/execution-projection.test.ts": 620, - "server/src/services/execution-recovery-attempt.test.ts": 189, - "server/src/services/explicit-native-continuation.test.ts": 14421, - "server/src/services/gmail-tool-governance.test.ts": 2225, - "server/src/services/heartbeat-run-runtime-status.test.ts": 205, - "server/src/services/heartbeat-runner-provider-config.test.ts": 175, - "server/src/services/heartbeat-stop-metadata.test.ts": 204, - "server/src/services/hot-restart.test.ts": 266, - "server/src/services/issue-comment-wakeup.test.ts": 180, - "server/src/services/issue-dependency-wakeups.test.ts": 629, - "server/src/services/issue-prefix.test.ts": 940, - "server/src/services/issue-queued-comment-queue.test.ts": 202, - "server/src/services/issue-thread-interaction-resolution.test.ts": 669, - "server/src/services/issue-thread-interactions.test.ts": 5185, - "server/src/services/legacy-controller-lease.test.ts": 7176, - "server/src/services/legacy-execution-recovery.test.ts": 1142, - "server/src/services/login-command.test.ts": 973, - "server/src/services/managed-environments.test.ts": 1786, - "server/src/services/managed-resource-drift.test.ts": 166, - "server/src/services/native-runtime/chat-attachment-read.test.ts": 8659, - "server/src/services/native-runtime/chat-attachment-reuse.test.ts": 7638, - "server/src/services/native-runtime/completion-contracts.test.ts": 618, - "server/src/services/native-runtime/continuation-contract.test.ts": 5616, - "server/src/services/native-runtime/current-wake-comments.test.ts": 7632, - "server/src/services/native-runtime/evidence-classifier.test.ts": 1067, - "server/src/services/native-runtime/external-chat-wait.integration.test.ts": 44215, - "server/src/services/native-runtime/native-chat-workspace.test.ts": 6604, + "server/src/__tests__/chat-completion-delivery.test.ts": 13194, + "server/src/__tests__/chat-delivery-work.test.ts": 638, + "server/src/__tests__/chat-interaction-publications.test.ts": 9294, + "server/src/__tests__/chat-project-tools.test.ts": 18910, + "server/src/__tests__/chat-webhook-diagnostics.test.ts": 3738, + "server/src/__tests__/chat-webhook-public-url.test.ts": 1448, + "server/src/__tests__/claude-local-adapter-environment.test.ts": 1405, + "server/src/__tests__/claude-local-adapter.test.ts": 901, + "server/src/__tests__/claude-local-execute.test.ts": 6842, + "server/src/__tests__/claude-local-skill-sync.test.ts": 857, + "server/src/__tests__/cleanup-removal-service.test.ts": 10580, + "server/src/__tests__/cli-invocation-safety.test.ts": 1157, + "server/src/__tests__/cloud-control-task-drain.test.ts": 747, + "server/src/__tests__/cloud-image-bundled-plugins.test.ts": 236, + "server/src/__tests__/cloud-image-sentry.test.ts": 227, + "server/src/__tests__/cloud-instance.test.ts": 165, + "server/src/__tests__/cloud-lifecycle-sync.test.ts": 194, + "server/src/__tests__/cloud-portfolio-error.test.ts": 168, + "server/src/__tests__/cloud-runtime-identity.test.ts": 10931, + "server/src/__tests__/cloud-tenant-company-provisioning.test.ts": 8210, + "server/src/__tests__/cloud-tenant-transient-db-retry.test.ts": 1774, + "server/src/__tests__/cloud-warm-standby.test.ts": 2838, + "server/src/__tests__/codex-auth-reconciliation.test.ts": 1286, + "server/src/__tests__/codex-local-adapter-environment.test.ts": 1111, + "server/src/__tests__/codex-local-adapter.test.ts": 878, + "server/src/__tests__/codex-local-execute.test.ts": 5333, + "server/src/__tests__/codex-local-skill-injection.test.ts": 881, + "server/src/__tests__/codex-local-skill-sync.test.ts": 814, + "server/src/__tests__/companies-service.test.ts": 11742, + "server/src/__tests__/company-artifacts-service.test.ts": 11341, + "server/src/__tests__/company-cloud-floor.test.ts": 1295, + "server/src/__tests__/company-import-cloud-floor.test.ts": 1454, + "server/src/__tests__/company-portability-import-batching.test.ts": 9596, + "server/src/__tests__/company-quota-windows.test.ts": 712, + "server/src/__tests__/company-search-extract-service.test.ts": 8217, + "server/src/__tests__/company-search-service.test.ts": 10949, + "server/src/__tests__/company-skill-import-boundary.test.ts": 9611, + "server/src/__tests__/company-skill-policy-service.test.ts": 8324, + "server/src/__tests__/company-skill-test-runs-service.test.ts": 14357, + "server/src/__tests__/company-skills-catalog-service.test.ts": 13056, + "server/src/__tests__/company-skills-detail.test.ts": 9399, + "server/src/__tests__/company-skills-service.test.ts": 29990, + "server/src/__tests__/company-skills.test.ts": 2402, + "server/src/__tests__/company-transfer-runs.test.ts": 7509, + "server/src/__tests__/composio-app-setup.test.ts": 11457, + "server/src/__tests__/config-file.test.ts": 823, + "server/src/__tests__/confirmation-comment-resolution.test.ts": 11157, + "server/src/__tests__/connection-aggregator-fallback.test.ts": 12200, + "server/src/__tests__/connection-grant-resolution.test.ts": 2283, + "server/src/__tests__/connection-instructions.test.ts": 9542, + "server/src/__tests__/connection-intents-service.test.ts": 11298, + "server/src/__tests__/container-init-reaping.test.ts": 160, + "server/src/__tests__/cross-issue-influence-limit-postgres.test.ts": 8263, + "server/src/__tests__/cross-issue-influence-limit.test.ts": 1202, + "server/src/__tests__/cursor-local-adapter-environment.test.ts": 3187, + "server/src/__tests__/cursor-local-adapter.test.ts": 878, + "server/src/__tests__/cursor-local-execute.test.ts": 4835, + "server/src/__tests__/cursor-local-skill-injection.test.ts": 889, + "server/src/__tests__/cursor-local-skill-sync.test.ts": 810, + "server/src/__tests__/customer-success-cloud.test.ts": 1410, + "server/src/__tests__/customer-success.test.ts": 8115, + "server/src/__tests__/dashboard-disconnect-retry.test.ts": 2267, + "server/src/__tests__/dashboard-service.test.ts": 8100, + "server/src/__tests__/db-errors.test.ts": 158, + "server/src/__tests__/decision-model-provider.test.ts": 842, + "server/src/__tests__/decision-models.test.ts": 12580, + "server/src/__tests__/decision-retention-service.test.ts": 9478, + "server/src/__tests__/decision-training.test.ts": 11230, + "server/src/__tests__/decision-wakeup.test.ts": 159, + "server/src/__tests__/decisions-service.test.ts": 12826, + "server/src/__tests__/delivery-work-coordinator.test.ts": 648, + "server/src/__tests__/dev-runner-native-binary.test.ts": 165, + "server/src/__tests__/dev-runner-options.test.ts": 169, + "server/src/__tests__/dev-runner-output.test.ts": 183, + "server/src/__tests__/dev-runner-paths.test.ts": 164, + "server/src/__tests__/dev-runner-snapshot.test.ts": 173, + "server/src/__tests__/dev-runner-worktree.test.ts": 163, + "server/src/__tests__/dev-server-status.test.ts": 293, + "server/src/__tests__/dev-watch-ignore.test.ts": 165, + "server/src/__tests__/device-login-credential-read.test.ts": 516, + "server/src/__tests__/device-login-reaper.test.ts": 1179, + "server/src/__tests__/device-login-service.test.ts": 8386, + "server/src/__tests__/disposition-repair.test.ts": 2374, + "server/src/__tests__/distribution-plugin-catalog.test.ts": 295, + "server/src/__tests__/docker-build-stamp.test.ts": 176, + "server/src/__tests__/docker-entrypoint.test.ts": 221, + "server/src/__tests__/document-annotations-service.test.ts": 10788, + "server/src/__tests__/documents-service.test.ts": 8463, + "server/src/__tests__/documents.test.ts": 1114, + "server/src/__tests__/dot-invitations.test.ts": 9385, + "server/src/__tests__/dot-oauth-onboarding.test.ts": 10191, + "server/src/__tests__/duplex-observability-recorder.test.ts": 164, + "server/src/__tests__/durable-chat-wakeup.test.ts": 20437, + "server/src/__tests__/durable-directory.test.ts": 1224, + "server/src/__tests__/effective-run-config-fingerprints.test.ts": 584, + "server/src/__tests__/email-address-check.test.ts": 1391, + "server/src/__tests__/email-channels.integration.test.ts": 39659, + "server/src/__tests__/email-idle-work.test.ts": 2288, + "server/src/__tests__/embedded-postgres-supervisor.test.ts": 164, + "server/src/__tests__/env-file-policy.test.ts": 173, + "server/src/__tests__/environment-acquisition-diagnostics.test.ts": 2105, + "server/src/__tests__/environment-capability-contract.test.ts": 1977, + "server/src/__tests__/environment-config.test.ts": 1983, + "server/src/__tests__/environment-custom-image-terminal-ws.test.ts": 2776, + "server/src/__tests__/environment-custom-images-service.test.ts": 13530, + "server/src/__tests__/environment-driver-traits.test.ts": 150, + "server/src/__tests__/environment-execution-target-capabilities.test.ts": 799, + "server/src/__tests__/environment-execution-target-duplex-kill-switch.test.ts": 839, + "server/src/__tests__/environment-execution-target-duplex.test.ts": 1637, + "server/src/__tests__/environment-execution-target.test.ts": 873, + "server/src/__tests__/environment-live-ssh.test.ts": 1679, + "server/src/__tests__/environment-probe.test.ts": 2062, + "server/src/__tests__/environment-run-orchestrator.test.ts": 2169, + "server/src/__tests__/environment-runtime-driver-contract.test.ts": 6427, + "server/src/__tests__/environment-runtime-duplex-bridge-input.test.ts": 5642, + "server/src/__tests__/environment-runtime.test.ts": 26265, + "server/src/__tests__/environment-service.test.ts": 7768, + "server/src/__tests__/environment-sync-out-error.test.ts": 709, + "server/src/__tests__/environment-test-harness.test.ts": 735, + "server/src/__tests__/error-handler.test.ts": 1155, + "server/src/__tests__/execution-lock-orphan-cleanup.test.ts": 9929, + "server/src/__tests__/execution-workspace-policy.test.ts": 766, + "server/src/__tests__/execution-workspace-reopen.test.ts": 10462, + "server/src/__tests__/execution-workspace-runtime-control-conflict.test.ts": 2686, + "server/src/__tests__/execution-workspaces-service.test.ts": 33572, + "server/src/__tests__/export-fidelity.test.ts": 7293, + "server/src/__tests__/external-objects-service.test.ts": 9447, + "server/src/__tests__/fast-response-models.test.ts": 167, + "server/src/__tests__/fast-response-provider.test.ts": 972, + "server/src/__tests__/fast-responses.test.ts": 10807, + "server/src/__tests__/feedback-flush-controller.test.ts": 4249, + "server/src/__tests__/feedback-service.test.ts": 8478, + "server/src/__tests__/feedback-share-client.test.ts": 172, + "server/src/__tests__/file-delivery-bridges.test.ts": 16263, + "server/src/__tests__/file-resources-git-scan-load.test.ts": 4087, + "server/src/__tests__/file-resources.test.ts": 19949, + "server/src/__tests__/first-admin-claim.test.ts": 8121, + "server/src/__tests__/folders-service.test.ts": 8899, + "server/src/__tests__/forbidden-tokens.test.ts": 170, + "server/src/__tests__/gemini-local-adapter-environment.test.ts": 915, + "server/src/__tests__/gemini-local-adapter.test.ts": 913, + "server/src/__tests__/gemini-local-execute.test.ts": 1108, + "server/src/__tests__/gemini-local-skill-sync.test.ts": 824, + "server/src/__tests__/general-capability-classifier.test.ts": 2001, + "server/src/__tests__/generic-mcp-connection.test.ts": 23336, + "server/src/__tests__/git-credentials.test.ts": 2266, + "server/src/__tests__/github-commit-details.test.ts": 2278, + "server/src/__tests__/github-connection-events.test.ts": 9238, + "server/src/__tests__/github-grant-metadata.test.ts": 2300, + "server/src/__tests__/github-operation-credentials.test.ts": 11187, + "server/src/__tests__/github-skill-source.test.ts": 2913, + "server/src/__tests__/google-sheets-gallery.test.ts": 2307, + "server/src/__tests__/grok-local-skill-sync.test.ts": 783, + "server/src/__tests__/health-workspace-readiness.test.ts": 2039, + "server/src/__tests__/heartbeat-accepted-plan-workspace-refresh.test.ts": 14233, + "server/src/__tests__/heartbeat-active-run-output-watchdog.test.ts": 12611, + "server/src/__tests__/heartbeat-agent-session-message.test.ts": 3483, + "server/src/__tests__/heartbeat-archived-company-guard.test.ts": 10431, + "server/src/__tests__/heartbeat-auto-checkout.test.ts": 3274, + "server/src/__tests__/heartbeat-chat-task-link.test.ts": 3135, + "server/src/__tests__/heartbeat-comment-wake-batching.test.ts": 48398, + "server/src/__tests__/heartbeat-context-summary.test.ts": 3034, + "server/src/__tests__/heartbeat-cost-accounting.test.ts": 2824, + "server/src/__tests__/heartbeat-direct-adapter-native-isolation.test.ts": 12836, + "server/src/__tests__/heartbeat-issue-rewake-throttle.test.ts": 14666, + "server/src/__tests__/heartbeat-ledger-billing-code.test.ts": 2939, + "server/src/__tests__/heartbeat-list.test.ts": 9777, + "server/src/__tests__/heartbeat-local-environment.test.ts": 11581, + "server/src/__tests__/heartbeat-lock-release-on-reassignment.test.ts": 12529, + "server/src/__tests__/heartbeat-managed-clone-credentials.test.ts": 4062, + "server/src/__tests__/heartbeat-model-fallback-warning.test.ts": 2941, + "server/src/__tests__/heartbeat-native-cleanup-admission.test.ts": 11377, + "server/src/__tests__/heartbeat-native-runner-cancellation.test.ts": 2802, + "server/src/__tests__/heartbeat-native-runner-selection.test.ts": 1173, + "server/src/__tests__/heartbeat-native-status-context.test.ts": 2793, + "server/src/__tests__/heartbeat-orphaned-active-lease-sweep.test.ts": 11197, + "server/src/__tests__/heartbeat-pending-cleanup-sweep.test.ts": 11748, + "server/src/__tests__/heartbeat-plugin-environment.test.ts": 9857, + "server/src/__tests__/heartbeat-project-env.test.ts": 3014, + "server/src/__tests__/heartbeat-project-repositories.test.ts": 21194, + "server/src/__tests__/heartbeat-queued-run-claim-isolation.test.ts": 12879, + "server/src/__tests__/heartbeat-referenced-projects.test.ts": 9793, + "server/src/__tests__/heartbeat-remote-referenced-projects.test.ts": 2938, + "server/src/__tests__/heartbeat-responsible-user-invariant.test.ts": 24742, + "server/src/__tests__/heartbeat-retry-scheduling.test.ts": 19896, + "server/src/__tests__/heartbeat-reviewed-chat-binding.integration.test.ts": 16691, + "server/src/__tests__/heartbeat-run-event-sequencing.test.ts": 19692, + "server/src/__tests__/heartbeat-run-lease-release-terminalization.test.ts": 10782, + "server/src/__tests__/heartbeat-run-log.test.ts": 3106, + "server/src/__tests__/heartbeat-run-status-payload.test.ts": 3009, + "server/src/__tests__/heartbeat-run-summary.test.ts": 186, + "server/src/__tests__/heartbeat-run-terminalize-before-release.test.ts": 10597, + "server/src/__tests__/heartbeat-running-followup.test.ts": 3054, + "server/src/__tests__/heartbeat-runtime-mcp-servers.test.ts": 11951, + "server/src/__tests__/heartbeat-runtime-skills.test.ts": 13151, + "server/src/__tests__/heartbeat-runtime-state.test.ts": 10413, + "server/src/__tests__/heartbeat-scheduling-suppression.test.ts": 2983, + "server/src/__tests__/heartbeat-slack-invitation.test.ts": 12647, + "server/src/__tests__/heartbeat-stale-queue-invalidation.test.ts": 62987, + "server/src/__tests__/heartbeat-start-lock.test.ts": 174, + "server/src/__tests__/heartbeat-status-preserving-telemetry.test.ts": 9550, + "server/src/__tests__/heartbeat-task-drain-admission-release.test.ts": 13413, + "server/src/__tests__/heartbeat-task-drain.test.ts": 2891, + "server/src/__tests__/heartbeat-task-session-reset.test.ts": 10038, + "server/src/__tests__/heartbeat-timer-wake-session-reset-pf4.test.ts": 3049, + "server/src/__tests__/heartbeat-workspace-branch-containment.test.ts": 18234, + "server/src/__tests__/heartbeat-workspace-busy.test.ts": 28002, + "server/src/__tests__/heartbeat-workspace-finalize-branch.test.ts": 13263, + "server/src/__tests__/heartbeat-workspace-ready-comment.test.ts": 2827, + "server/src/__tests__/heartbeat-workspace-session.test.ts": 3933, + "server/src/__tests__/heartbeat-worktree-suppression.test.ts": 12281, + "server/src/__tests__/heartbeat-zombie-guard.test.ts": 2823, + "server/src/__tests__/helpers/truncate-with-deadlock-retry.test.ts": 345, + "server/src/__tests__/hire-hook.test.ts": 630, + "server/src/__tests__/hiring-operational-examples.test.ts": 700, + "server/src/__tests__/http-adapter-remote-fetch.test.ts": 181, + "server/src/__tests__/http-log-policy.test.ts": 175, + "server/src/__tests__/http-log-redaction.test.ts": 1483, + "server/src/__tests__/human-directed-work.test.ts": 10053, + "server/src/__tests__/idle-accounting-work.test.ts": 8526, + "server/src/__tests__/idle-admission.test.ts": 377, + "server/src/__tests__/idle-database-backup.test.ts": 12993, + "server/src/__tests__/idle-local-work.test.ts": 178, + "server/src/__tests__/idle-sleep-safety.test.ts": 24972, + "server/src/__tests__/inbox-dismissals.test.ts": 11123, + "server/src/__tests__/instance-settings-cloud-defaults.test.ts": 1179, + "server/src/__tests__/instance-settings-managed-overlay.test.ts": 1161, + "server/src/__tests__/instance-settings-operator-defaults.test.ts": 1156, + "server/src/__tests__/instance-settings-service.test.ts": 1098, + "server/src/__tests__/instrumentation.test.ts": 230, + "server/src/__tests__/interaction-resolution-cross-issue-cap-postgres.test.ts": 11676, + "server/src/__tests__/invite-join-grants.test.ts": 706, + "server/src/__tests__/invite-rate-limit.test.ts": 153, + "server/src/__tests__/invite-token-entropy.test.ts": 3531, + "server/src/__tests__/issue-blocker-attention.test.ts": 11127, + "server/src/__tests__/issue-comment-redaction.test.ts": 10738, + "server/src/__tests__/issue-continuation-summary.test.ts": 1166, + "server/src/__tests__/issue-execution-policy-persistence.test.ts": 9062, + "server/src/__tests__/issue-execution-policy.test.ts": 873, + "server/src/__tests__/issue-existing-branch-validation-status.test.ts": 1396, + "server/src/__tests__/issue-goal-fallback.test.ts": 164, + "server/src/__tests__/issue-interactions-read-only-contract.test.ts": 160, + "server/src/__tests__/issue-liveness.test.ts": 900, + "server/src/__tests__/issue-monitor-scheduler.test.ts": 29650, + "server/src/__tests__/issue-recovery-actions.test.ts": 18737, + "server/src/__tests__/issue-references-service.test.ts": 8595, + "server/src/__tests__/issue-remove-cascade.test.ts": 9294, + "server/src/__tests__/issue-review-attention.test.ts": 9710, + "server/src/__tests__/issue-review-policy.test.ts": 7578, + "server/src/__tests__/issue-rewake-throttle.test.ts": 158, + "server/src/__tests__/issue-runtime-workspace-binding.test.ts": 9585, + "server/src/__tests__/issue-thread-interactions-service.test.ts": 15275, + "server/src/__tests__/issue-thread-interactions-telemetry.test.ts": 9603, + "server/src/__tests__/issue-tree-control-service-unit.test.ts": 1138, + "server/src/__tests__/issue-tree-control-service.test.ts": 9622, + "server/src/__tests__/issues-list-query-parsing.test.ts": 10346, + "server/src/__tests__/issues-parent-id-alias.test.ts": 10268, + "server/src/__tests__/issues-user-context.test.ts": 2295, + "server/src/__tests__/join-request-dedupe.test.ts": 171, + "server/src/__tests__/json-parse-error-handler.test.ts": 1367, + "server/src/__tests__/json-schema-secret-refs.test.ts": 161, + "server/src/__tests__/kimi-local-skill-sync.test.ts": 831, + "server/src/__tests__/legacy-continuation-authority.test.ts": 13939, + "server/src/__tests__/legacy-finalization-regression.test.ts": 10146, + "server/src/__tests__/legacy-runtime-tool-refresh.test.ts": 918, + "server/src/__tests__/legacy-workspace-restore-recovery.test.ts": 10444, + "server/src/__tests__/live-events-toast.test.ts": 712, + "server/src/__tests__/live-events-ws.test.ts": 1357, + "server/src/__tests__/local-ai-credential-file.test.ts": 171, + "server/src/__tests__/local-ai-credentials.test.ts": 823, + "server/src/__tests__/local-ai-login-policy.test.ts": 162, + "server/src/__tests__/local-service-supervisor.test.ts": 4591, + "server/src/__tests__/log-redaction.test.ts": 165, + "server/src/__tests__/logger-tz.test.ts": 193, + "server/src/__tests__/login-session-runtime.test.ts": 1220, + "server/src/__tests__/managed-config.test.ts": 710, + "server/src/__tests__/managed-loopback-auth.test.ts": 8012, + "server/src/__tests__/managed-runtime-public-url.test.ts": 733, + "server/src/__tests__/managed-worktree-inspection.test.ts": 2519, + "server/src/__tests__/mcp-http.test.ts": 189, + "server/src/__tests__/memory-connectors.test.ts": 2277, + "server/src/__tests__/merged-pr-confirmation-sweep.test.ts": 9905, + "server/src/__tests__/monthly-spend-service.test.ts": 3181, + "server/src/__tests__/native-cancellation-request.integration.test.ts": 9488, + "server/src/__tests__/native-connection-baseline-fixture.test.ts": 14182, + "server/src/__tests__/native-event-boundary-golden.test.ts": 7599, + "server/src/__tests__/native-finalization-migration.test.ts": 7280, + "server/src/__tests__/native-finalization-recovery.test.ts": 17018, + "server/src/__tests__/native-interaction-bridge.test.ts": 9388, + "server/src/__tests__/native-procedure-measurement.test.ts": 2848, + "server/src/__tests__/native-run-finalizer.test.ts": 19882, + "server/src/__tests__/native-runner-input-boundary.test.ts": 204, + "server/src/__tests__/native-runner-phase6.integration.test.ts": 19600, + "server/src/__tests__/native-sandbox-lifecycle.test.ts": 2850, + "server/src/__tests__/native-session-resumption.test.ts": 32645, + "server/src/__tests__/native-status-arbiter-corpus.test.ts": 27304, + "server/src/__tests__/native-workspace-sync-history.test.ts": 5477, + "server/src/__tests__/native-workspace-sync.test.ts": 1275, + "server/src/__tests__/oauth-client-metadata.test.ts": 2323, + "server/src/__tests__/openclaw-gateway-adapter.test.ts": 843, + "server/src/__tests__/opencode-local-adapter.test.ts": 801, + "server/src/__tests__/opencode-local-skill-sync.test.ts": 813, + "server/src/__tests__/owner-chat-instruction-authority.test.ts": 11518, + "server/src/__tests__/paperclip-env.test.ts": 778, + "server/src/__tests__/paperclip-issue-update-helper.test.ts": 4937, + "server/src/__tests__/paperclip-semantic-conformance.test.ts": 11300, + "server/src/__tests__/paperclip-skill-utils.test.ts": 8496, + "server/src/__tests__/parse-status-filter.test.ts": 2204, + "server/src/__tests__/persisted-workspace-source.test.ts": 4374, + "server/src/__tests__/photon/channel.integration.test.ts": 27031, + "server/src/__tests__/photon/photon.test.ts": 1180, + "server/src/__tests__/pi-local-adapter-environment.test.ts": 912, + "server/src/__tests__/pi-local-execute.test.ts": 1068, + "server/src/__tests__/pi-local-skill-sync.test.ts": 806, + "server/src/__tests__/pipelines-service.test.ts": 13114, + "server/src/__tests__/plan-task-fixture.test.ts": 12967, + "server/src/__tests__/plugin-access-authorization-host-services.test.ts": 10822, + "server/src/__tests__/plugin-agent-sessions.test.ts": 2409, + "server/src/__tests__/plugin-config-startup-delivery.test.ts": 8163, + "server/src/__tests__/plugin-database.test.ts": 8634, + "server/src/__tests__/plugin-dev-watcher.test.ts": 255, + "server/src/__tests__/plugin-environment-driver-ready-recovery.test.ts": 6732, + "server/src/__tests__/plugin-environment-driver-sandbox-capabilities.test.ts": 167, + "server/src/__tests__/plugin-environment-driver-seam.test.ts": 854, + "server/src/__tests__/plugin-execution-workspace-bridge.test.ts": 185, + "server/src/__tests__/plugin-host-services-span.test.ts": 3026, + "server/src/__tests__/plugin-idle-manager.test.ts": 3863, + "server/src/__tests__/plugin-idle-rpc.test.ts": 282, + "server/src/__tests__/plugin-install-autobuild.test.ts": 10710, + "server/src/__tests__/plugin-install-guard.test.ts": 1356, + "server/src/__tests__/plugin-lifecycle-restart.test.ts": 1168, + "server/src/__tests__/plugin-loader-error-retry.test.ts": 1419, + "server/src/__tests__/plugin-local-folders.test.ts": 212, + "server/src/__tests__/plugin-managed-agents.test.ts": 11887, + "server/src/__tests__/plugin-managed-routines.test.ts": 9690, + "server/src/__tests__/plugin-managed-skills.test.ts": 11378, + "server/src/__tests__/plugin-orchestration-apis.test.ts": 14904, + "server/src/__tests__/plugin-organization-switcher.test.ts": 736, + "server/src/__tests__/plugin-sandbox-provider-config-validation.test.ts": 177, + "server/src/__tests__/plugin-sdk-orchestration-contract.test.ts": 858, + "server/src/__tests__/plugin-sdk-testing.test.ts": 737, + "server/src/__tests__/plugin-secrets-handler.test.ts": 9449, + "server/src/__tests__/plugin-telemetry-bridge.test.ts": 2951, + "server/src/__tests__/plugin-tenant-isolation.test.ts": 10381, + "server/src/__tests__/plugin-tool-dispatcher-pluginDbId.test.ts": 665, + "server/src/__tests__/plugin-ui-static.test.ts": 1332, + "server/src/__tests__/plugin-worker-manager-duplex.test.ts": 2651, + "server/src/__tests__/plugin-worker-manager.test.ts": 4242, + "server/src/__tests__/primary-agent.test.ts": 11142, + "server/src/__tests__/privacy-production-review.test.ts": 12043, + "server/src/__tests__/private-hostname-guard.test.ts": 221, + "server/src/__tests__/private-json-etag.test.ts": 217, + "server/src/__tests__/project-goal-validation.test.ts": 8385, + "server/src/__tests__/project-icon-persistence.test.ts": 8365, + "server/src/__tests__/project-list-metrics.test.ts": 1149, + "server/src/__tests__/project-repositories-persistence.test.ts": 9637, + "server/src/__tests__/project-repositories.test.ts": 2275, + "server/src/__tests__/project-shortname-resolution.test.ts": 1181, + "server/src/__tests__/project-tools-pagination.test.ts": 1051, + "server/src/__tests__/provider-trace-store.test.ts": 653, + "server/src/__tests__/public-mcp-client-metadata.test.ts": 748, + "server/src/__tests__/public-mcp.test.ts": 25756, + "server/src/__tests__/qa-routine-secrets-e2e.test.ts": 11519, + "server/src/__tests__/question-response-delivery.test.ts": 13572, + "server/src/__tests__/quota-windows-service.test.ts": 1324, + "server/src/__tests__/quota-windows.test.ts": 859, + "server/src/__tests__/railway-connection.test.ts": 11007, + "server/src/__tests__/railway-ssh.test.ts": 1249, + "server/src/__tests__/railway.test.ts": 272, + "server/src/__tests__/recovery-classifiers.test.ts": 2321, + "server/src/__tests__/recovery-observability.test.ts": 8196, + "server/src/__tests__/recovery-stale-issue-lock-sweep.test.ts": 11076, + "server/src/__tests__/redact-sensitive.test.ts": 177, + "server/src/__tests__/remote-http-endpoint-guard.test.ts": 175, + "server/src/__tests__/remote-http-rebinding.test.ts": 1077, + "server/src/__tests__/remote-mcp-connectors.test.ts": 11408, + "server/src/__tests__/remote-mcp-pending.test.ts": 2325, + "server/src/__tests__/remote-mcp-protocol.test.ts": 177, + "server/src/__tests__/remote-mcp-redaction.test.ts": 169, + "server/src/__tests__/remote-pi-companion.test.ts": 1019, + "server/src/__tests__/rerun-survival.integration.test.ts": 1255, + "server/src/__tests__/resource-lifecycle-events.test.ts": 12158, + "server/src/__tests__/reusable-leases-default.test.ts": 721, + "server/src/__tests__/routable-blocked.test.ts": 167, + "server/src/__tests__/routine-run-telemetry.test.ts": 9407, + "server/src/__tests__/routines-service.test.ts": 17843, + "server/src/__tests__/run-continuations.test.ts": 648, + "server/src/__tests__/run-failure-sentry-real-sdk.test.ts": 792, + "server/src/__tests__/run-failure-sentry.test.ts": 7601, + "server/src/__tests__/run-identity.test.ts": 8747, + "server/src/__tests__/run-liveness.test.ts": 712, + "server/src/__tests__/run-secret-redaction.test.ts": 604, + "server/src/__tests__/run-trust-preset.test.ts": 8139, + "server/src/__tests__/runner-connection-eval-fixtures.test.ts": 11889, + "server/src/__tests__/runner-project-icon-contract.test.ts": 1030, + "server/src/__tests__/runtime-api.test.ts": 171, + "server/src/__tests__/runtime-skill-cache.test.ts": 684, + "server/src/__tests__/sandbox-orphan-cleanup-spool.test.ts": 213, + "server/src/__tests__/sandbox-provider-runtime.test.ts": 158, + "server/src/__tests__/secret-provider-registry.test.ts": 224, + "server/src/__tests__/secrets-service-dangling-user-secret.test.ts": 8869, + "server/src/__tests__/secrets-service-user-secret-owner-scoped.test.ts": 8535, + "server/src/__tests__/secrets-service.test.ts": 12971, + "server/src/__tests__/sentry-dsn.test.ts": 164, + "server/src/__tests__/sentry.test.ts": 8683, + "server/src/__tests__/server-info.test.ts": 258, + "server/src/__tests__/server-package-build-script.test.ts": 166, + "server/src/__tests__/server-package-otel-peer-metadata.test.ts": 166, + "server/src/__tests__/server-package-sentry-peer-metadata.test.ts": 144, + "server/src/__tests__/server-startup-feedback-export.test.ts": 2652, + "server/src/__tests__/service-read-disconnect-retry.test.ts": 4602, + "server/src/__tests__/service-version.test.ts": 169, + "server/src/__tests__/shared-telemetry-events.test.ts": 171, + "server/src/__tests__/skill-source-git-snapshot.test.ts": 3138, + "server/src/__tests__/skill-source-github-access.test.ts": 2463, + "server/src/__tests__/skill-source-scan-limit.test.ts": 176, + "server/src/__tests__/skill-sources.test.ts": 11245, + "server/src/__tests__/skills-catalog-service.test.ts": 220, + "server/src/__tests__/slack-conversation-lifecycle.test.ts": 12152, + "server/src/__tests__/smoke-lab.test.ts": 10616, + "server/src/__tests__/source-trust.test.ts": 8122, + "server/src/__tests__/speko-call-report.test.ts": 709, + "server/src/__tests__/speko-delivery-diagnostics.test.ts": 742, + "server/src/__tests__/speko-protocol.test.ts": 757, + "server/src/__tests__/speko-provider.test.ts": 815, + "server/src/__tests__/startup-refusals.test.ts": 190, + "server/src/__tests__/static-index-html.test.ts": 228, + "server/src/__tests__/static-ui-cache.test.ts": 158, + "server/src/__tests__/static-ui-compression.test.ts": 255, + "server/src/__tests__/status-card-update-engine.test.ts": 772, + "server/src/__tests__/status-cards.test.ts": 13788, + "server/src/__tests__/storage-local-provider.test.ts": 196, + "server/src/__tests__/storage-s3-multipart.test.ts": 245, + "server/src/__tests__/subscription-refresh.test.ts": 940, + "server/src/__tests__/subscriptions.test.ts": 10904, + "server/src/__tests__/summary-slots.test.ts": 11719, + "server/src/__tests__/task-search-quality.test.ts": 10050, + "server/src/__tests__/task-search-query.test.ts": 1159, + "server/src/__tests__/task-watchdogs-classifier.test.ts": 2281, + "server/src/__tests__/task-watchdogs-incident-fixture.test.ts": 169, + "server/src/__tests__/task-watchdogs-scheduler.test.ts": 11102, + "server/src/__tests__/teams-catalog-install-no-overrides.test.ts": 14095, + "server/src/__tests__/teams-catalog-service.test.ts": 782, + "server/src/__tests__/telemetry-client-flush.test.ts": 178, + "server/src/__tests__/tool-access-connector-telemetry.test.ts": 9714, + "server/src/__tests__/tool-access-policy-service.test.ts": 9912, + "server/src/__tests__/tool-access-service.test.ts": 51001, + "server/src/__tests__/tool-connection-removal.test.ts": 11665, + "server/src/__tests__/tool-connection-visibility.test.ts": 3084, + "server/src/__tests__/tool-content-guards.test.ts": 178, + "server/src/__tests__/tool-gateway-connector-telemetry.test.ts": 12506, + "server/src/__tests__/tool-gateway-discovery-http.test.ts": 10729, + "server/src/__tests__/tool-gateway-listing-memory.test.ts": 10922, + "server/src/__tests__/tool-gateway-service.test.ts": 16209, + "server/src/__tests__/tool-gateway.test.ts": 40912, + "server/src/__tests__/tool-grant-scope.test.ts": 10918, + "server/src/__tests__/tool-oauth-legacy-backfill.test.ts": 8636, + "server/src/__tests__/tool-review-queue-unsigned-request.test.ts": 9588, + "server/src/__tests__/trust-preset-resolver.test.ts": 745, + "server/src/__tests__/trust-proxy.test.ts": 214, + "server/src/__tests__/ui-branding.test.ts": 162, + "server/src/__tests__/version.test.ts": 186, + "server/src/__tests__/vite-html-renderer.test.ts": 157, + "server/src/__tests__/vitest-chat-shards.test.ts": 6218, + "server/src/__tests__/voice-session-store.integration.test.ts": 23337, + "server/src/__tests__/work-product-runtime-reconciliation.test.ts": 2358, + "server/src/__tests__/work-product-workspace-validation.test.ts": 9523, + "server/src/__tests__/work-products.test.ts": 2273, + "server/src/__tests__/work-scheduler.test.ts": 170, + "server/src/__tests__/work-timeline-service.test.ts": 11755, + "server/src/__tests__/workspace-git-inspection-cache.test.ts": 172, + "server/src/__tests__/workspace-instance-cleanup.test.ts": 182, + "server/src/__tests__/workspace-login-handoff-endpoint.test.ts": 1121, + "server/src/__tests__/workspace-login-handoff-issuer.test.ts": 663, + "server/src/__tests__/workspace-login-handoff.test.ts": 176, + "server/src/__tests__/workspace-operations-reconciliation.test.ts": 8533, + "server/src/__tests__/workspace-readiness.test.ts": 714, + "server/src/__tests__/workspace-reaper-cooldown-config.test.ts": 759, + "server/src/__tests__/workspace-runtime-control-recovery.test.ts": 8883, + "server/src/__tests__/workspace-runtime-exposure-reservation.test.ts": 11865, + "server/src/__tests__/workspace-runtime-https-live-exercise.test.ts": 2771, + "server/src/__tests__/workspace-runtime-leases.test.ts": 8734, + "server/src/__tests__/workspace-runtime-start-terminality.test.ts": 5643, + "server/src/__tests__/workspace-runtime.test.ts": 136113, + "server/src/__tests__/worktree-config.test.ts": 1283, + "server/src/__tests__/worktree-seed-server-spawn.test.ts": 3931, + "server/src/__tests__/write-build-stamp.test.ts": 227, + "server/src/__tests__/write-path-membership.test.ts": 1348, + "server/src/adapters/claude-agent-id-header.test.ts": 168, + "server/src/adapters/http/execute.test.ts": 740, + "server/src/adapters/plugin-loader.test.ts": 189, + "server/src/adapters/registry.test.ts": 1429, + "server/src/embedded-postgres-owner.test.ts": 162, + "server/src/http/server.test.ts": 684, + "server/src/instrumentation.test.ts": 176, + "server/src/lib/request-promise-memo.test.ts": 165, + "server/src/middleware/cloud-tenant-actor.test.ts": 1326, + "server/src/modules/active-run-watchdog/adapters/process.test.ts": 770, + "server/src/modules/active-run-watchdog/application/use-cases.test.ts": 177, + "server/src/modules/active-run-watchdog/domain/policy.test.ts": 178, + "server/src/modules/agent-lifecycle/__tests__/diagnostics.test.ts": 2361, + "server/src/modules/agent-lifecycle/__tests__/lifecycle.test.ts": 12810, + "server/src/modules/agent-lifecycle/__tests__/policy-sweep.test.ts": 9782, + "server/src/modules/run-dispatch/adapters/postgres.test.ts": 12179, + "server/src/modules/run-dispatch/application/use-cases.test.ts": 165, + "server/src/modules/run-dispatch/domain/policy.test.ts": 166, + "server/src/modules/run-dispatch/domain/wake-context.test.ts": 160, + "server/src/modules/wake-queue/adapters/postgres.test.ts": 11781, + "server/src/modules/wake-queue/adapters/queued-comment-postgres.test.ts": 7983, + "server/src/modules/wake-queue/application/queued-comment-use-cases.test.ts": 177, + "server/src/modules/wake-queue/application/use-cases.test.ts": 2659, + "server/src/modules/wake-queue/domain/context.test.ts": 2334, + "server/src/modules/wake-queue/domain/policy.test.ts": 197, + "server/src/realtime/runner-prp-outbound.test.ts": 184, + "server/src/realtime/runner-prp-ws.test.ts": 1006, + "server/src/routes/adapters.test.ts": 2025, + "server/src/routes/chat-channels.identity.test.ts": 4371, + "server/src/routes/chat-channels.webhook.test.ts": 3732, + "server/src/routes/connection-intents.test.ts": 2823, + "server/src/routes/setup-token-route.test.ts": 24121, + "server/src/routes/tool-access-connection-intent.test.ts": 3456, + "server/src/runtime-tools-token.test.ts": 158, + "server/src/services/__tests__/git-connection-failure.test.ts": 173, + "server/src/services/__tests__/mcp-connection-failure.test.ts": 177, + "server/src/services/__tests__/process-loss-diagnostics.test.ts": 162, + "server/src/services/__tests__/run-failure-diagnostics.test.ts": 845, + "server/src/services/__tests__/run-failure-report.test.ts": 10795, + "server/src/services/acknowledged-native-stop.test.ts": 624, + "server/src/services/adapter-execution-control.test.ts": 721, + "server/src/services/adapter-execution-phase.test.ts": 162, + "server/src/services/adapter-login-lease.test.ts": 166, + "server/src/services/adapter-models-env.test.ts": 162, + "server/src/services/adapter-registry-bootstrap.reconcile.test.ts": 1517, + "server/src/services/adapter-registry-bootstrap.test.ts": 1482, + "server/src/services/ai-auth-failure.test.ts": 765, + "server/src/services/ai-connection-configuration-failure.test.ts": 161, + "server/src/services/ai-connection-usage.test.ts": 841, + "server/src/services/anthropic-pricing.test.ts": 708, + "server/src/services/app-webhook.test.ts": 166, + "server/src/services/approved-execution-wait.test.ts": 156, + "server/src/services/attention-resolver-audience.test.ts": 2253, + "server/src/services/board-auth.test.ts": 622, + "server/src/services/chat-communication-guidance.test.ts": 728, + "server/src/services/chat-control-admission-retry.test.ts": 2203, + "server/src/services/chat-discord-adapter-patch.test.ts": 3907, + "server/src/services/chat-discord-command-registration-store.test.ts": 8644, + "server/src/services/chat-discord-command-registration.test.ts": 297, + "server/src/services/chat-discord-modal-wire.test.ts": 1892, + "server/src/services/chat-discord-native-command.test.ts": 1878, + "server/src/services/chat-discord-native-commands.test.ts": 484, + "server/src/services/chat-discord-native-interaction.test.ts": 2339, + "server/src/services/chat-discord-native-modal.test.ts": 1876, + "server/src/services/chat-discord-question-forms.test.ts": 246, + "server/src/services/chat-discord.test.ts": 491, + "server/src/services/chat-github-attachments.test.ts": 2802, + "server/src/services/chat-github-events.test.ts": 1170, + "server/src/services/chat-github-provider-stress.test.ts": 1679, + "server/src/services/chat-github-receipt-reactions.test.ts": 375, + "server/src/services/chat-github-review-policy.test.ts": 733, + "server/src/services/chat-github-webhook-config.test.ts": 280, + "server/src/services/chat-github-wizard-origin.test.ts": 2280, + "server/src/services/chat-inbound-wakeup-publications.test.ts": 617, + "server/src/services/chat-interaction-arbitration.test.ts": 6762, + "server/src/services/chat-interaction-publications.test.ts": 1265, + "server/src/services/chat-outbound-attachment-hydration.test.ts": 3916, + "server/src/services/chat-provider-inventory.test.ts": 190, + "server/src/services/chat-provider-lifecycle.test.ts": 186, + "server/src/services/chat-provider-links.test.ts": 162, + "server/src/services/chat-publication-batches.test.ts": 737, + "server/src/services/chat-publication-errors.test.ts": 189, + "server/src/services/chat-publication-projection.test.ts": 196, + "server/src/services/chat-publication-ready-stream.test.ts": 1939, + "server/src/services/chat-publication-reconciliation.test.ts": 180, + "server/src/services/chat-publication-stream.test.ts": 1022, + "server/src/services/chat-publication-text-parts.test.ts": 13871, + "server/src/services/chat-question-forms.test.ts": 867, + "server/src/services/chat-run-publications.test.ts": 2246, + "server/src/services/chat-sdk-runtime.adapters.test.ts": 2430, + "server/src/services/chat-sdk-runtime.test.ts": 2005, + "server/src/services/chat-sdk-state.test.ts": 177, + "server/src/services/chat-slack-native-modal.test.ts": 2161, + "server/src/services/chat-slack-receipts.test.ts": 4303, + "server/src/services/chat-slack-sessions.test.ts": 200, + "server/src/services/chat-task-url.test.ts": 2162, + "server/src/services/chat-teams-credentials.test.ts": 768, + "server/src/services/chat-teams-file-consent-runtime.test.ts": 2023, + "server/src/services/chat-teams-file-consent.contract.test.ts": 716, + "server/src/services/chat-teams-file-consent.test.ts": 443, + "server/src/services/chat-teams-file-publication.test.ts": 8569, + "server/src/services/chat-teams-file-transfers.test.ts": 11310, + "server/src/services/chat-teams-inline-image-intake.test.ts": 1811, + "server/src/services/chat-teams-inline-images.test.ts": 1953, + "server/src/services/chat-teams-native-modal.test.ts": 2384, + "server/src/services/chat-teams-personal-recipient.test.ts": 644, + "server/src/services/chat-telegram-draft-stop.test.ts": 301, + "server/src/services/chat-telegram-ephemeral-runtime.test.ts": 2037, + "server/src/services/chat-telegram-ephemeral.test.ts": 221, + "server/src/services/chat-telegram-media-intake.test.ts": 1853, + "server/src/services/chat-telegram-photo.test.ts": 262, + "server/src/services/chat-telegram-rich-intake.test.ts": 1731, + "server/src/services/chat-telegram-stop-subscription.test.ts": 161, + "server/src/services/chat-telegram-video-note.test.ts": 1730, + "server/src/services/cloud-runtime-identity.test.ts": 635, + "server/src/services/cloud-warm-standby.test.ts": 605, + "server/src/services/connector-telemetry.test.ts": 1207, + "server/src/services/connectors/slack-access.test.ts": 1147, + "server/src/services/connectors/slack-native-search.test.ts": 795, + "server/src/services/environment-custom-image-terminal-sessions.test.ts": 172, + "server/src/services/execution-allowlist.test.ts": 171, + "server/src/services/execution-continuation-provenance.test.ts": 8783, + "server/src/services/execution-continuation.test.ts": 8239, + "server/src/services/execution-control-reconciliation.test.ts": 14764, + "server/src/services/execution-policy-bootstrap.test.ts": 1492, + "server/src/services/execution-projection-read-retry.test.ts": 2712, + "server/src/services/execution-projection.test.ts": 739, + "server/src/services/execution-recovery-attempt.test.ts": 150, + "server/src/services/explicit-native-continuation.test.ts": 39692, + "server/src/services/fireflies-webhook.test.ts": 171, + "server/src/services/gmail-tool-governance.test.ts": 2503, + "server/src/services/google-chat-tool-policy.test.ts": 172, + "server/src/services/heartbeat-github-launchers.test.ts": 1082, + "server/src/services/heartbeat-run-runtime-status.test.ts": 184, + "server/src/services/heartbeat-runner-provider-config.test.ts": 1038, + "server/src/services/heartbeat-stop-metadata.test.ts": 785, + "server/src/services/heartbeat/queue.test.ts": 10333, + "server/src/services/heartbeat/recovery.test.ts": 10484, + "server/src/services/heartbeat/retries.test.ts": 10211, + "server/src/services/heartbeat/run-completion.test.ts": 14764, + "server/src/services/heartbeat/run-control.test.ts": 11572, + "server/src/services/heartbeat/run-lifecycle.test.ts": 12026, + "server/src/services/heartbeat/run-log.test.ts": 782, + "server/src/services/heartbeat/run-preparation.test.ts": 2755, + "server/src/services/heartbeat/run-state.test.ts": 10457, + "server/src/services/heartbeat/runtime-execution.test.ts": 13791, + "server/src/services/heartbeat/runtime-selection.test.ts": 13055, + "server/src/services/heartbeat/scheduling.test.ts": 12915, + "server/src/services/heartbeat/task-markdown.test.ts": 1253, + "server/src/services/heartbeat/workspace-preparation.test.ts": 14044, + "server/src/services/heartbeat/workspaces.test.ts": 3288, + "server/src/services/hot-restart.test.ts": 630, + "server/src/services/isolated-task-directory.test.ts": 987, + "server/src/services/issue-comment-wakeup.test.ts": 154, + "server/src/services/issue-dependency-wakeups.test.ts": 602, + "server/src/services/issue-prefix.test.ts": 851, + "server/src/services/issue-question-context.test.ts": 7590, + "server/src/services/issue-queued-comment-queue.test.ts": 158, + "server/src/services/issue-read-timing.test.ts": 180, + "server/src/services/issue-thread-interaction-resolution.test.ts": 741, + "server/src/services/issue-thread-interactions.test.ts": 6562, + "server/src/services/legacy-controller-lease.test.ts": 10206, + "server/src/services/legacy-execution-recovery.test.ts": 2262, + "server/src/services/local-ai-browser-login.test.ts": 1873, + "server/src/services/login-command.test.ts": 1515, + "server/src/services/managed-environments.test.ts": 2054, + "server/src/services/managed-resource-drift.test.ts": 165, + "server/src/services/mcp-tool-names.test.ts": 154, + "server/src/services/native-runtime/acpx-qualification.test.ts": 1152, + "server/src/services/native-runtime/agent-instruction-tools.integration.test.ts": 10966, + "server/src/services/native-runtime/agent-instruction-tools.test.ts": 2619, + "server/src/services/native-runtime/assigned-mcp-tools.test.ts": 2476, + "server/src/services/native-runtime/chat-attachment-read.test.ts": 10858, + "server/src/services/native-runtime/chat-attachment-reuse.test.ts": 9459, + "server/src/services/native-runtime/codex-model-fallback.test.ts": 784, + "server/src/services/native-runtime/codex-runtime-compatibility.test.ts": 154, + "server/src/services/native-runtime/completion-contracts.test.ts": 636, + "server/src/services/native-runtime/confirmation-comment-api.integration.test.ts": 13210, + "server/src/services/native-runtime/continuation-contract.test.ts": 8112, + "server/src/services/native-runtime/create-skill.integration.test.ts": 14421, + "server/src/services/native-runtime/current-wake-comments.test.ts": 9866, + "server/src/services/native-runtime/evidence-classifier.test.ts": 978, + "server/src/services/native-runtime/external-chat-wait.integration.test.ts": 58620, + "server/src/services/native-runtime/grok-runner-credentials.test.ts": 182, + "server/src/services/native-runtime/handoff-plan-context.test.ts": 8275, + "server/src/services/native-runtime/historical-task-questions.test.ts": 10211, + "server/src/services/native-runtime/managed-native-credentials.test.ts": 158, + "server/src/services/native-runtime/native-agent-runtime-inheritance.test.ts": 1083, + "server/src/services/native-runtime/native-cancellation-request.test.ts": 666, + "server/src/services/native-runtime/native-chat-workspace.test.ts": 10149, "server/src/services/native-runtime/native-codex-runner.integration.test.ts": 224015, - "server/src/services/native-runtime/native-codex-runner.test.ts": 2284, - "server/src/services/native-runtime/native-deliverable-feedback.test.ts": 762, - "server/src/services/native-runtime/native-execution-input.test.ts": 1287, - "server/src/services/native-runtime/native-maintenance-no-launch.test.ts": 546, - "server/src/services/native-runtime/native-provider-capacity.integration.test.ts": 6882, - "server/src/services/native-runtime/native-question-bridge.test.ts": 13420, - "server/src/services/native-runtime/native-replacement-evidence.test.ts": 163, - "server/src/services/native-runtime/native-restart-recovery.test.ts": 756, - "server/src/services/native-runtime/native-run-finalizer-telemetry.test.ts": 7647, - "server/src/services/native-runtime/native-run-trace.test.ts": 231, - "server/src/services/native-runtime/native-runner-attachment-staging.test.ts": 1732, - "server/src/services/native-runtime/native-runner-file-handoff.test.ts": 7340, - "server/src/services/native-runtime/native-runner-ownership.test.ts": 716, + "server/src/services/native-runtime/native-codex-runner.test.ts": 2297, + "server/src/services/native-runtime/native-completion-feedback.test.ts": 10473, + "server/src/services/native-runtime/native-completion-format-feedback.test.ts": 1987, + "server/src/services/native-runtime/native-continuation.test.ts": 776, + "server/src/services/native-runtime/native-deliverable-feedback.test.ts": 598, + "server/src/services/native-runtime/native-execution-input.test.ts": 1659, + "server/src/services/native-runtime/native-github-access.test.ts": 2623, + "server/src/services/native-runtime/native-harness-backup-cleanup.test.ts": 167, + "server/src/services/native-runtime/native-maintenance-no-launch.test.ts": 599, + "server/src/services/native-runtime/native-permission-decline.test.ts": 986, + "server/src/services/native-runtime/native-permission-projection.test.ts": 2223, + "server/src/services/native-runtime/native-plan-wait-receipts.test.ts": 161, + "server/src/services/native-runtime/native-plan-wait.test.ts": 2710, + "server/src/services/native-runtime/native-provider-capacity.integration.test.ts": 8906, + "server/src/services/native-runtime/native-provider-failure.integration.test.ts": 10743, + "server/src/services/native-runtime/native-provider-failure.test.ts": 177, + "server/src/services/native-runtime/native-provider-overload.integration.test.ts": 10700, + "server/src/services/native-runtime/native-question-bridge.test.ts": 30037, + "server/src/services/native-runtime/native-replacement-evidence.test.ts": 169, + "server/src/services/native-runtime/native-restart-recovery.test.ts": 1541, + "server/src/services/native-runtime/native-review-participant.test.ts": 1139, + "server/src/services/native-runtime/native-review-prompt.test.ts": 152, + "server/src/services/native-runtime/native-run-finalizer-telemetry.test.ts": 12067, + "server/src/services/native-runtime/native-run-trace.test.ts": 187, + "server/src/services/native-runtime/native-runner-attachment-staging.test.ts": 2326, + "server/src/services/native-runtime/native-runner-file-handoff.test.ts": 11450, + "server/src/services/native-runtime/native-runner-ownership.test.ts": 607, "server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts": 7940, - "server/src/services/native-runtime/native-safe-replacement.test.ts": 23437, - "server/src/services/native-runtime/native-session-executor.test.ts": 14430, - "server/src/services/native-runtime/native-session-resume.test.ts": 7411, - "server/src/services/native-runtime/native-ssh-command-runner.test.ts": 1019, - "server/src/services/native-runtime/native-workspace-finalizer.test.ts": 6191, - "server/src/services/native-runtime/paperclip-control-plane-port.test.ts": 13807, - "server/src/services/native-runtime/paperclip-runner-tool-authority.test.ts": 6501, - "server/src/services/native-runtime/remote-deliverable-file.test.ts": 753, - "server/src/services/native-runtime/remote-provider-pack.test.ts": 191, - "server/src/services/native-runtime/runner-api-files.test.ts": 211, - "server/src/services/native-runtime/runner-api-rollout.test.ts": 160, - "server/src/services/native-runtime/runner-api.integration.test.ts": 12758, - "server/src/services/native-runtime/runner-api.test.ts": 1456, - "server/src/services/native-runtime/runner-prp-coordinator.test.ts": 8049, - "server/src/services/native-runtime/runtime-context.test.ts": 1845, - "server/src/services/native-runtime/runtime-mode.test.ts": 211, - "server/src/services/native-runtime/runtime-request-resolution-authority.test.ts": 713, - "server/src/services/native-runtime/status-arbiter.test.ts": 204, - "server/src/services/native-runtime/stopped-codex-turn.test.ts": 173, - "server/src/services/onboarding-first-task-assets.test.ts": 650, - "server/src/services/onboarding-greeting.test.ts": 636, - "server/src/services/paperclip-cloud-connector-enrollment.test.ts": 1233, - "server/src/services/paperclip-cloud-connector.test.ts": 1277, - "server/src/services/provider-trace-workspace-diff-reprojection.test.ts": 874, - "server/src/services/recovery/native-runner-routing.test.ts": 2032, - "server/src/services/recovery/provider-failure-classification.test.ts": 2033, - "server/src/services/recovery/review-path-recovery.test.ts": 185, - "server/src/services/recovery/service.pause-durability.test.ts": 2026, - "server/src/services/recovery/status-only-context.test.ts": 186, - "server/src/services/recovery/stranded-notice.test.ts": 1140, - "server/src/services/recovery/successful-run-handoff.test.ts": 1153, - "server/src/services/remote-agent-profiles.test.ts": 768, - "server/src/services/remote-url-credentials.test.ts": 159, - "server/src/services/responsible-user-denial-run-outcomes.test.ts": 1132, - "server/src/services/routines-formatter-cache.test.ts": 3592, - "server/src/services/run-log-store.test.ts": 558, - "server/src/services/run-scratch.test.ts": 216, - "server/src/services/runner-goal-control-broker.test.ts": 186, - "server/src/services/runner-goals.test.ts": 6061, - "server/src/services/runtime-exposure/broker-client.test.ts": 353, - "server/src/services/runtime-exposure/exposure-manager.test.ts": 656, - "server/src/services/runtime-exposure/loopback-listener.test.ts": 232, - "server/src/services/runtime-exposure/port-pair.test.ts": 649, - "server/src/services/runtime-exposure/port-reservation.test.ts": 671, - "server/src/services/runtime-exposure/tailscale-hostname.test.ts": 188, - "server/src/services/safe-native-chat-progress.test.ts": 217, - "server/src/services/session-workspace-cwd.test.ts": 204, - "server/src/services/setup-token-reaper.test.ts": 1656, - "server/src/services/setup-token-session.test.ts": 7291, - "server/src/services/setup-token-transport-binding.test.ts": 2042, - "server/src/services/tool-profile-binding-precedence.test.ts": 160, - "server/src/services/vercel-connect.test.ts": 266, - "server/src/services/workspace-git-operation-scheduler.test.ts": 1567, - "server/src/services/workspace-runtime-exposure-backfill.test.ts": 2194, - "server/src/services/workspace-runtime-exposure.test.ts": 14621, - "server/src/services/workspace-runtime-read-model.test.ts": 958, - "server/src/services/workspace-runtime-ready-comment.test.ts": 2176, - "server/src/shutdown.test.ts": 331, - "server/src/url-utils.test.ts": 183 + "server/src/services/native-runtime/native-safe-replacement.test.ts": 28876, + "server/src/services/native-runtime/native-session-executor.test.ts": 127716, + "server/src/services/native-runtime/native-session-handoff.test.ts": 8274, + "server/src/services/native-runtime/native-session-resume.test.ts": 9887, + "server/src/services/native-runtime/native-ssh-command-runner.test.ts": 1142, + "server/src/services/native-runtime/native-tool-trace.test.ts": 581, + "server/src/services/native-runtime/native-warm-transition-restart.integration.test.ts": 2492, + "server/src/services/native-runtime/native-workspace-best-effort.test.ts": 1161, + "server/src/services/native-runtime/native-workspace-export-resume.live.test.ts": 2865, + "server/src/services/native-runtime/native-workspace-export-retry.test.ts": 12329, + "server/src/services/native-runtime/native-workspace-finalizer-failure.test.ts": 2318, + "server/src/services/native-runtime/native-workspace-finalizer.test.ts": 9543, + "server/src/services/native-runtime/paperclip-control-plane-port.test.ts": 20670, + "server/src/services/native-runtime/paperclip-runner-real-server.integration.test.ts": 2492, + "server/src/services/native-runtime/paperclip-runner-tool-authority.test.ts": 10054, + "server/src/services/native-runtime/remote-deliverable-file.test.ts": 735, + "server/src/services/native-runtime/remote-provider-pack.test.ts": 158, + "server/src/services/native-runtime/runner-api-files.test.ts": 168, + "server/src/services/native-runtime/runner-api-response.test.ts": 411, + "server/src/services/native-runtime/runner-api-rollout.test.ts": 151, + "server/src/services/native-runtime/runner-api.integration.test.ts": 27868, + "server/src/services/native-runtime/runner-api.test.ts": 1949, + "server/src/services/native-runtime/runner-bridge-tools.test.ts": 2461, + "server/src/services/native-runtime/runner-prp-coordinator.test.ts": 9624, + "server/src/services/native-runtime/runner-task-attachments.test.ts": 10554, + "server/src/services/native-runtime/runtime-context.test.ts": 2186, + "server/src/services/native-runtime/runtime-mode.test.ts": 1075, + "server/src/services/native-runtime/runtime-request-resolution-authority.test.ts": 629, + "server/src/services/native-runtime/status-arbiter.test.ts": 177, + "server/src/services/native-runtime/stopped-codex-turn.test.ts": 178, + "server/src/services/native-runtime/task-creation-backlog.test.ts": 10405, + "server/src/services/native-runtime/task-monitor.integration.test.ts": 10207, + "server/src/services/native-runtime/task-reassignment.test.ts": 10500, + "server/src/services/native-runtime/update-skill.integration.test.ts": 12293, + "server/src/services/onboarding-first-task-assets.test.ts": 741, + "server/src/services/onboarding-greeting.test.ts": 741, + "server/src/services/openrouter-models.test.ts": 165, + "server/src/services/paperclip-cloud-connector-enrollment.test.ts": 1301, + "server/src/services/paperclip-cloud-connector.test.ts": 1220, + "server/src/services/provider-trace-workspace-diff-reprojection.test.ts": 1049, + "server/src/services/question-interaction-answers.test.ts": 2315, + "server/src/services/question-pattern-validation.test.ts": 2294, + "server/src/services/recovery/legacy-continuation.test.ts": 165, + "server/src/services/recovery/native-runner-routing.test.ts": 2198, + "server/src/services/recovery/provider-failure-classification.test.ts": 2382, + "server/src/services/recovery/review-path-recovery.test.ts": 2334, + "server/src/services/recovery/service.pause-durability.test.ts": 2435, + "server/src/services/recovery/status-only-context.test.ts": 181, + "server/src/services/recovery/stranded-notice.test.ts": 1190, + "server/src/services/recovery/successful-run-handoff.test.ts": 1122, + "server/src/services/remote-agent-profiles.test.ts": 608, + "server/src/services/remote-url-credentials.test.ts": 167, + "server/src/services/responsible-user-denial-run-outcomes.test.ts": 1228, + "server/src/services/routines-formatter-cache.test.ts": 3628, + "server/src/services/run-cancellation.test.ts": 158, + "server/src/services/run-log-store-cancellation.test.ts": 456, + "server/src/services/run-log-store.test.ts": 867, + "server/src/services/run-scratch.test.ts": 174, + "server/src/services/runner-goal-control-broker.test.ts": 157, + "server/src/services/runner-goals.test.ts": 8268, + "server/src/services/runtime-exposure/broker-client.test.ts": 316, + "server/src/services/runtime-exposure/exposure-manager.test.ts": 699, + "server/src/services/runtime-exposure/loopback-listener.test.ts": 181, + "server/src/services/runtime-exposure/port-pair.test.ts": 727, + "server/src/services/runtime-exposure/port-reservation.test.ts": 688, + "server/src/services/runtime-exposure/tailscale-hostname.test.ts": 153, + "server/src/services/safe-native-chat-progress.test.ts": 166, + "server/src/services/session-workspace-cwd.test.ts": 160, + "server/src/services/setup-token-reaper.test.ts": 2025, + "server/src/services/setup-token-session.test.ts": 8660, + "server/src/services/setup-token-transport-binding.test.ts": 2151, + "server/src/services/skill-tools.test.ts": 760, + "server/src/services/slack-mcp-error.test.ts": 191, + "server/src/services/tool-discovery-scheduler.test.ts": 164, + "server/src/services/tool-profile-binding-precedence.test.ts": 167, + "server/src/services/vercel-connect.test.ts": 211, + "server/src/services/workspace-git-operation-scheduler.test.ts": 1686, + "server/src/services/workspace-git-snapshot-streaming.test.ts": 116534, + "server/src/services/workspace-restore-reconciliation.test.ts": 8823, + "server/src/services/workspace-runtime-exposure-backfill.test.ts": 2178, + "server/src/services/workspace-runtime-exposure.test.ts": 14607, + "server/src/services/workspace-runtime-read-model.test.ts": 1136, + "server/src/services/workspace-runtime-ready-comment.test.ts": 2250, + "server/src/shutdown.test.ts": 311, + "server/src/url-utils.test.ts": 166 } } From 4f680bc85c824940f4a71ea37074f3de8efba24e Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Sat, 10 Oct 2026 16:49:15 -0700 Subject: [PATCH 3/7] fix(release): retry verified canary tarball 404s (#15768) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Canary releases test the published npm packages through a clean onboarding install. > - A package can be announced in registry metadata while a fresh install gets a tarball 404. > - The existing ETARGET retry does not cover that distinct error. > - This pull request retries only a verified public canary tarball acquisition failure. > - Permanent errors still fail, and onboarding runs once after acquisition succeeds. ## Linked Issues or Issue Description Refs: #15743 **What happened?** The [Release run](https://github.com/paperclipai/paperclip/actions/runs/37998601578) published canary `2026.1009.0-canary.17`. Its registry verification and publisher install passed. The separate fresh onboarding install then failed with a final npm E404 for that exact `@paperclipai/server` tarball. The tarball was available on a later read. This is a separate failure from ETARGET; it does not prove the cause of the registry visibility difference. **Expected behavior** Allow a bounded acquisition retry when the effective registry is public npm and fresh primary metadata announces the exact released canary tarball. Keep other E404, authentication, configuration, and onboarding errors fatal. **Steps to reproduce** 1. Run the offline helper fixture with a first-attempt E404 for an allowlisted package at the requested canary version. 2. Return canonical effective npm registry configuration and matching primary metadata. 3. Make the second install succeed. Confirm two installs and one onboarding invocation. 4. Keep the tarball unavailable. Confirm three installs, a final failure, and no onboarding invocation. **Paperclip version or commit** The observed run used commit `ce024457f3d747657501fbcfe8eac71f6772150f`. This fix builds on the merged acquisition helper and current master. **Deployment mode** Other: the published npm canary onboarding check in release CI. ## What Changed - Add a canary-only classifier for one unambiguous final E404 at the canonical tarball URL of a CI-published package from the existing release manifest. - Query effective npm registry configuration in the failed install's prefix, environment, and working directory. Keep bounded configuration output in memory without logging it. Respect scoped registry precedence. - Require fresh primary metadata to match the package, version, tarball URL, and valid SHA512 integrity syntax. Reject redirects, conflicting errors, truncated logs, other artifacts, and unknown or custom effective registries. - Share one five-second configuration and metadata deadline within the existing two-minute acquisition budget. Keep three install attempts, refreshed retry metadata, process cleanup, and one onboarding invocation. - Bound metadata header and body waits even if transport abort propagation stalls. Cancel the reader and any late response body without waiting for cancellation acknowledgements. - Preserve the existing ETARGET classifier and its character-based stderr tail. Add offline subprocess, actual npm configuration, body cancellation, budget, identity, and failure tests. Document the scope and limits. ## Verification - `node --test scripts/__tests__/release-verify-workflow.test.mjs scripts/cloud-source-verification.test.mjs scripts/standard-image-contract.test.mjs`: 73 passed, no skips, under Node 22 on the PR head. - `pnpm test:release-registry`: 153 passed, no skips, on the PR head. This includes the existing real Playwright cleanup and startup timeout tests. - [Hosted CI](https://github.com/paperclipai/paperclip/actions/runs/38011028296) passed on this head. The rollup has 53 successful checks and two expected Storybook skips. This includes typecheck, build, the full general and serialized test suites, E2E, and Canary Dry Run. - The new tests use fixture-owned npm configuration and local HTTP servers. No published canary tarballs were downloaded or installed for this change. No release or deployment was triggered. - `pnpm test:run` was not run locally. The full split test suites passed in hosted CI. Independent source checks on this head passed 46 committed tests, 22 boundary checks, and six negative controls. Greptile scored this head 5/5 with no unresolved threads. - The source diff and PR text were checked for secrets, private identifiers, and unintended artifacts. ## Risks - The new retry is conservative. A configuration capture over 4 KiB, metadata over 64 KiB, or a check that exceeds five seconds denies retry. A persistent tarball failure still fails after three installs. - Matching primary metadata proves the announced artifact identity and integrity syntax. It does not independently verify tarball bytes. The subsequent npm install retains its integrity check. - Registry checks use npm's resolved global and applicable scope values. They do not add credentials or change npm configuration. Existing ETARGET behavior, onboarding assertions, and overall startup deadlines remain unchanged. ## Model Used OpenAI GPT-6 (Codex). The exact model variant and context window are not exposed in this session. Used reasoning, code execution, offline fixtures, and GitHub API tools. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip --- doc/RELEASING.md | 19 +- scripts/__tests__/canary-startup-fixture.mjs | 41 ++- .../release-verify-workflow.test.mjs | 326 ++++++++++++++++++ .../start-published-canary.mjs | 149 +++++++- 4 files changed, 523 insertions(+), 12 deletions(-) diff --git a/doc/RELEASING.md b/doc/RELEASING.md index 771d3acb5bc..dbf40956a50 100644 --- a/doc/RELEASING.md +++ b/doc/RELEASING.md @@ -104,7 +104,7 @@ It: - publishes the user-facing `paperclipai` package last, so `paperclipai@canary` does not advance before the full package set exists - verifies that `canary` resolves to the just-published version and that published internal dependencies exist on npm - installs `paperclipai@canary` into a clean temporary prefix as the final npm gate -- starts the exact published canary through a separate fresh npm install for the onboarding smoke; only npm `ETARGET` install failures retry, up to three attempts within two minutes, before onboarding runs once under the existing five-minute startup deadline +- starts the exact published canary through a separate fresh npm install for the onboarding smoke; npm `ETARGET` and verified canary tarball availability failures retry, up to three attempts within two minutes, before onboarding runs once under the existing five-minute startup deadline - fails by default if npm leaves `latest` pointing at a canary; use `--allow-canary-latest` only when that state is intentional - creates a git tag `canary/vYYYY.MDD.P-canary.N` @@ -116,6 +116,23 @@ current patches and manifests while the separate lockfile-refresh PR is still pending, without resolving new dependencies in the smoke job. The smoke still installs and tests the exact published canary version, not a workspace build. +The tarball retry applies only to an unambiguous npm `E404` for a canonical +public-registry tarball of a CI-published Paperclip package at that exact canary +version. The helper queries npm's effective global and applicable scoped registry +in the failed install's prefix, working directory, and environment. A custom or +unknown effective registry fails. Configuration output stays private in memory. +Fresh primary-registry metadata must match the package, version, and tarball URL +and announce a valid SHA512 integrity value. This establishes the announced +artifact identity; it does not download the tarball or independently verify its +contents. The actual npm install still checks tarball integrity. + +Configuration and metadata checks share a five-second deadline, including the +response body, within the same two-minute acquisition budget. Metadata is limited +to 64 KiB. Redirects, authentication errors, missing metadata, conflicting npm +errors, and other artifacts fail. A persistent promised-tarball failure also +fails after three installs. Onboarding errors never restart acquisition or +repeat onboarding. + The lockfile artifact is retained for 14 days. A publisher-job rerun replaces its source-named artifact; a smoke-only rerun uses the existing artifact. A missing artifact fails the job rather than falling back to a different dependency set. diff --git a/scripts/__tests__/canary-startup-fixture.mjs b/scripts/__tests__/canary-startup-fixture.mjs index d75d1fa944c..4a51ce621ba 100644 --- a/scripts/__tests__/canary-startup-fixture.mjs +++ b/scripts/__tests__/canary-startup-fixture.mjs @@ -10,7 +10,8 @@ const repoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), ".." // Exercise the real startup helper and child-process boundary without registry // traffic. The npm fixture installs a real executable into its isolated prefix. -export async function canaryStartup({ mode, onboard = "success", budget = 5000, delay = 5, cancelAtKind, playwright = false }) { +export async function canaryStartup({ mode, onboard = "success", budget = 5000, delay = 5, cancelAtKind, playwright = false, + config = "public", metadata = "valid", version = "2026.1009.0-canary.1" }) { const root = mkdtempSync(path.join(os.tmpdir(), "canary-startup-test-")); const calls = path.join(root, "calls.jsonl"); const fixtureBin = path.join(root, "bin"); @@ -39,15 +40,35 @@ if (process.env.FIXTURE_ONBOARD === "serve") { const fs = require("node:fs"); const path = require("node:path"); const args = process.argv.slice(2); +if (args[0] === "config") { + fs.appendFileSync(process.env.FIXTURE_CALLS, JSON.stringify({kind:"config", pid:process.pid, args}) + "\\n"); + if (process.env.FIXTURE_CONFIG === "hang") { + const { spawn } = require("node:child_process"); + const child = spawn(process.execPath, ["-e", "process.on('SIGTERM',()=>{}); setTimeout(()=>process.exit(0),10000)"], {stdio:"ignore"}); + fs.appendFileSync(process.env.FIXTURE_CALLS, JSON.stringify({kind:"config-descendant", pid:child.pid}) + "\\n"); + process.on("SIGTERM", () => {}); + setTimeout(() => process.exit(0), 10000); + } else { + if (process.env.FIXTURE_CONFIG === "large") console.log("x".repeat(8192)); + console.log("registry=" + (process.env.FIXTURE_CONFIG === "private" ? "https://fixture-user:fixture-password@registry.example.invalid/" : "https://registry.npmjs.org/") + "\\n@paperclipai:registry=undefined"); + console.error("fixture config diagnostic must stay private"); + } +} else { const entries = fs.existsSync(process.env.FIXTURE_CALLS) ? fs.readFileSync(process.env.FIXTURE_CALLS, "utf8").trim().split("\\n").filter(Boolean).map(JSON.parse) : []; const attempt = entries.filter(e => e.kind === "npm").length + 1; fs.appendFileSync(process.env.FIXTURE_CALLS, JSON.stringify({kind:"npm", pid:process.pid, args, cache:process.env.npm_config_cache}) + "\\n"); const mode = process.env.FIXTURE_MODE; if (mode === "hang") { setTimeout(() => {}, 10000); } else if (mode === "permanent" || (mode === "recover" && attempt === 1)) { console.error("npm error code ETARGET"); process.exit(1); } +else if (mode === "multibyte-recover" && attempt === 1) { console.error("npm error code ETARGET\\n" + "界".repeat(30 * 1024)); process.exit(1); } else if (mode === "auth") { console.error("npm error code E401"); process.exit(1); } else if (mode === "mixed") { console.error("npm error code ETARGET\\nnpm error code E401"); process.exit(1); } else if (mode === "network") { console.error("npm error code ECONNRESET"); process.exit(1); } +else if (mode === "tarball-permanent" || mode === "tarball-large" || (mode === "tarball-recover" && attempt === 1)) { + const url = "https://registry.npmjs.org/@paperclipai/server/-/server-" + process.env.PAPERCLIPAI_VERSION + ".tgz"; + if (mode === "tarball-large") console.error("npm error code E401\\n" + "x".repeat(70 * 1024)); + console.error("npm error code E404\\nnpm error 404 Not Found - GET " + url + " - Not found"); process.exit(1); +} else { const prefix = args[args.indexOf("--prefix") + 1]; const bin = path.join(prefix, "node_modules", ".bin"); @@ -55,12 +76,24 @@ else { fs.writeFileSync(path.join(bin, "paperclipai"), ${JSON.stringify(cli).replaceAll('$','\$')}); fs.chmodSync(path.join(bin, "paperclipai"), 0o755); } +} `); chmodSync(npmFixture, 0o755); const helper = new URL("../../tests/canary-onboarding/start-published-canary.mjs", import.meta.url).href; const driver = path.join(root, "driver.mjs"); writeFileSync(driver, `import { startPublishedCanary } from ${JSON.stringify(helper)}; -try { await startPublishedCanary({version:"2026.1009.0-canary.1", workspace:${JSON.stringify(root)}, dataDir:${JSON.stringify(path.join(root,"data"))}, installBudgetMs:${budget}, retryDelayMs:${delay}}); } +import { appendFileSync } from "node:fs"; +globalThis.fetch = async (url, options) => { + appendFileSync(process.env.FIXTURE_CALLS, JSON.stringify({kind:"metadata", url, redirect:options.redirect, credentials:options.credentials, headers:options.headers}) + "\\n"); + const manifest = {name:"@paperclipai/server",version:process.env.PAPERCLIPAI_VERSION,dist:{tarball:"https://registry.npmjs.org/@paperclipai/server/-/server-" + process.env.PAPERCLIPAI_VERSION + ".tgz",integrity:"sha512-" + Buffer.alloc(64).toString("base64")}}; + const body = process.env.FIXTURE_METADATA === "hang" ? new ReadableStream({ + start(controller) { options.signal.addEventListener("abort", () => controller.error(options.signal.reason), {once:true}); }, + }) : JSON.stringify(manifest); + const response = new Response(body, {status:process.env.FIXTURE_METADATA === "missing" ? 404 : 200}); + Object.defineProperty(response,"url",{value:url}); + return response; +}; +try { await startPublishedCanary({version:${JSON.stringify(version)}, workspace:${JSON.stringify(root)}, dataDir:${JSON.stringify(path.join(root,"data"))}, installBudgetMs:${budget}, retryDelayMs:${delay}}); } catch (error) { console.error(error.message); process.exitCode = 1; } `); const playwrightConfig = path.join(root, "playwright.config.mjs"); @@ -80,8 +113,8 @@ export default {...original, testDir:${JSON.stringify(root)}, testMatch:"fixture ? [fileURLToPath(helper), "2026.1009.0-canary.1", root, path.join(root, "data")] : [driver], { cwd: repoRoot, env: { ...process.env, PATH: fixtureBin + path.delimiter + process.env.PATH, - FIXTURE_MODE: mode, FIXTURE_ONBOARD: onboard, FIXTURE_CALLS: calls, FIXTURE_PORT: String(port), - PAPERCLIPAI_VERSION: "2026.1009.0-canary.1", PAPERCLIP_CANARY_SMOKE_BASE_URL: `http://127.0.0.1:${port}`, PAPERCLIP_CANARY_SMOKE_SERVER_LOG: path.join(root,"server.log"), + FIXTURE_MODE: mode, FIXTURE_ONBOARD: onboard, FIXTURE_CALLS: calls, FIXTURE_PORT: String(port), FIXTURE_CONFIG: config, FIXTURE_METADATA: metadata, + PAPERCLIPAI_VERSION: version, PAPERCLIP_CANARY_SMOKE_BASE_URL: `http://127.0.0.1:${port}`, PAPERCLIP_CANARY_SMOKE_SERVER_LOG: path.join(root,"server.log"), npm_config_cache: path.join(root, "fresh-cache"), }, stdio: ["ignore", "pipe", "pipe"] }); let output = ""; diff --git a/scripts/__tests__/release-verify-workflow.test.mjs b/scripts/__tests__/release-verify-workflow.test.mjs index 9e5364c869a..043636edb91 100644 --- a/scripts/__tests__/release-verify-workflow.test.mjs +++ b/scripts/__tests__/release-verify-workflow.test.mjs @@ -5,7 +5,9 @@ import os from "node:os"; import path from "node:path"; import { fileURLToPath } from "node:url"; import test from "node:test"; +import { createServer } from "node:http"; import { canaryStartup } from "./canary-startup-fixture.mjs"; +import { canRetryPublishedTarball404, TARBALL_METADATA_TIMEOUT_MS } from "../../tests/canary-onboarding/start-published-canary.mjs"; const repoRoot = path.resolve( path.dirname(fileURLToPath(import.meta.url)), @@ -830,3 +832,327 @@ test("nested ETARGET output does not retry a final authentication or network fai assert.equal(result.calls.length, 1); } }); + +const exactCanary = "2026.1009.0-canary.1"; +const canaryTarball = `https://registry.npmjs.org/@paperclipai/server/-/server-${exactCanary}.tgz`; +const tarballFailure = (url = canaryTarball, prefix = "npm error") => `${prefix} code E404\n${prefix} 404 Not Found - GET ${url} - Not found\n${prefix} 404\n${prefix} 404 The requested resource '@paperclipai/server@${url}' could not be found or you do not have permission to access it.\n`; + +function tarballProof({ config = {}, manifest = {}, status = 200, redirected = false, responseUrl, rawBody } = {}) { + const calls = []; + const dist = { tarball: canaryTarball, integrity: `sha512-${Buffer.alloc(64).toString("base64")}`, ...manifest.dist }; + const data = { name: "@paperclipai/server", version: exactCanary, ...manifest, dist }; + return { calls, options: { + signal: new AbortController().signal, prefix: "/fixture/install", env: {}, + runConfig: async (command, args, options) => { + calls.push({ kind: "config", command, args, options }); + return { code: 0, stdout: "registry=https://registry.npmjs.org/\n@paperclipai:registry=undefined\n", ...config }; + }, + fetchMetadata: async (url, options) => { + calls.push({ kind: "metadata", url, options }); + return { status, redirected, url: responseUrl ?? url, body: new Response(rawBody ?? JSON.stringify(data)).body }; + }, + } }; +} + +test("exact published tarball proof uses quiet effective config and fresh public metadata without credentials", async () => { + for (const prefix of ["npm error", "npm ERR!"]) { + const fixture = tarballProof(); + assert.equal(await canRetryPublishedTarball404(tarballFailure(canaryTarball, prefix), exactCanary, fixture.options), true); + const [config, metadata] = fixture.calls; + assert.deepEqual(config.args, ["config", "get", "registry", "@paperclipai:registry", "--prefix", "/fixture/install"]); + assert.equal(config.options.env, fixture.options.env); + assert.equal(config.options.quiet, true); + assert.equal(config.options.captureStdout, true); + assert.equal(config.options.captureStderr, true); + assert.equal(config.options.outputLimit, 4096); + assert.equal(metadata.url, `https://registry.npmjs.org/%40paperclipai%2Fserver/${exactCanary}`); + assert.equal(metadata.options.signal, config.options.signal); + assert.equal(metadata.options.redirect, "error"); + assert.equal(metadata.options.credentials, "omit"); + assert.deepEqual(metadata.options.headers, { accept: "application/json", "cache-control": "no-cache" }); + } +}); + +test("mixed npm errors and noncanonical or unreleased tarballs never read config or metadata", async () => { + const errors = [ + "npm error code ETARGET\n", "npm error code E401\n", "npm error code ECONNRESET\n", + `${tarballFailure()}npm error code E401\n`, `${tarballFailure()}npm error code 1\n`, + `npm error code ENEEDAUTH\n${tarballFailure()}`, `${tarballFailure()}${tarballFailure()}`, + `${tarballFailure()}npm error 404 Not Found - GET https://registry.example.invalid/a.tgz - Not found\n`, + `${tarballFailure()}npm error 404 The requested resource 'other@${canaryTarball}' could not be found\n`, + tarballFailure().replace(" - GET ", " - PUT "), + ...[ + canaryTarball + "?token=fixture", canaryTarball + "#fragment", canaryTarball.replace("npmjs.org/", "npmjs.org:443/"), + canaryTarball.replace("npmjs.org/", "npmjs.org.example.invalid/"), canaryTarball.replace("npmjs.org/", "fixture@npmjs.org/"), + canaryTarball.replace("https:", "http:"), canaryTarball.replace("@paperclipai/server", "%40paperclipai%2Fserver"), + canaryTarball.replace("@paperclipai/server", "@paperclipai/../server"), canaryTarball.replace(exactCanary, "2026.1009.0-canary.2"), + "https://registry.npmjs.org/@paperclipai/not-released/-/not-released-2026.1009.0-canary.1.tgz", + "https://registry.npmjs.org/@paperclipai/plugin-createos/-/plugin-createos-2026.1009.0-canary.1.tgz", + "https://registry.npmjs.org/%40paperclipai%2Fserver/2026.1009.0-canary.1", + ].map((url) => tarballFailure(url)), + ]; + for (const error of errors) { + const fixture = tarballProof(); + assert.equal(await canRetryPublishedTarball404(error, exactCanary, fixture.options), false, error); + assert.deepEqual(fixture.calls, [], error); + } + for (const version of ["2026.1009.0", "2026.1009.0-nightly.1", "2026.1009.0-beta.1", "2026.1009.0-canary.1+build"]) { + const fixture = tarballProof(); + assert.equal(await canRetryPublishedTarball404(tarballFailure(canaryTarball.replace(exactCanary, version)), version, fixture.options), false); + assert.deepEqual(fixture.calls, []); + } +}); + +test("effective registry authority respects scope precedence and rejects unknown or private config", async () => { + const configs = [ + [{ stdout: "registry=https://registry.example.invalid/\n@paperclipai:registry=https://registry.npmjs.org/\n" }, true], + [{ stdout: "registry=undefined\n@paperclipai:registry=https://registry.npmjs.org/\n" }, true], + [{ stdout: "registry=https://registry.npmjs.org/\n@paperclipai:registry=https://registry.example.invalid/\n" }, false], + [{ stdout: "registry=https://registry.example.invalid/\n@paperclipai:registry=undefined\n" }, false], + [{ stdout: "registry=undefined\n@paperclipai:registry=undefined\n" }, false], + [{ stdout: "registry=https://registry.npmjs.org/\n@paperclipai:registry=\n" }, false], + [{ stdout: "registry=https://fixture:secret@registry.npmjs.org/\n@paperclipai:registry=undefined\n" }, false], + [{ stdout: "unexpected output\n" }, false], [{ code: 1 }, false], [{ signal: "SIGTERM" }, false], [{ outputTruncated: true }, false], + ]; + for (const [config, expected] of configs) { + const fixture = tarballProof({ config }); + assert.equal(await canRetryPublishedTarball404(tarballFailure(), exactCanary, fixture.options), expected); + assert.equal(fixture.calls.filter((call) => call.kind === "metadata").length, expected ? 1 : 0); + } +}); + +test("metadata must identify the exact published artifact with valid SHA512 syntax, not merely exist", async () => { + const fixtures = [ + { status: 404 }, { status: 401 }, { status: 403 }, { status: 302 }, { redirected: true }, + { responseUrl: "https://registry.example.invalid/metadata" }, { rawBody: "not json" }, + { manifest: { name: "@paperclipai/shared" } }, { manifest: { version: "2026.1009.0-canary.2" } }, + { manifest: { dist: { tarball: canaryTarball + "?query=1" } } }, + ...[undefined, "", "sha512-A===", `sha256-${Buffer.alloc(32).toString("base64")}`, `sha512-${Buffer.alloc(64).toString("base64")} extra`] + .map((integrity) => ({ manifest: { dist: { integrity } } })), + { rawBody: JSON.stringify({ padding: "x".repeat(65 * 1024) }) }, + ]; + for (const input of fixtures) { + const fixture = tarballProof(input); + assert.equal(await canRetryPublishedTarball404(tarballFailure(), exactCanary, fixture.options), false, JSON.stringify(input).slice(0, 200)); + } +}); + +test("rejected and oversized native response bodies cancel without waiting for slow or rejected cleanup", async () => { + for (const [status, pending] of [[200, true], [200, false], [404, true], [404, false]]) { + let cancelled = false; + const fixture = tarballProof(); + fixture.options.fetchMetadata = async (url) => ({ status, redirected: false, url, body: new Response(new ReadableStream({ + start(controller) { controller.enqueue(new Uint8Array(65 * 1024)); }, + cancel() { cancelled = true; return pending ? new Promise(() => {}) : Promise.reject(new Error("fixture rejected cancellation")); }, + })).body }); + const started = Date.now(); + assert.equal(await canRetryPublishedTarball404(tarballFailure(), exactCanary, fixture.options), false); + assert.equal(cancelled, true); + assert.ok(Date.now() - started < 1000, "cancellation acknowledgement must not extend a rejection"); + } +}); + +test("aborting a stalled metadata read cancels its reader without waiting for acknowledgement", { timeout: 2000 }, async () => { + let cancelled = false; + const fixture = tarballProof(); + fixture.options.fetchMetadata = async (url) => ({ status: 200, redirected: false, url, + body: new Response(new ReadableStream({ + cancel() { cancelled = true; return new Promise(() => {}); }, + })).body }); + const budget = new AbortController(); + const timer = setTimeout(() => budget.abort(new Error("fixture body budget expired")), 100); + const started = Date.now(); + try { + await assert.rejects(canRetryPublishedTarball404(tarballFailure(), exactCanary, { + ...fixture.options, signal: budget.signal, + }), /fixture body budget expired/); + assert.equal(cancelled, true); + assert.ok(Date.now() - started < 1000); + } finally { clearTimeout(timer); } +}); + +test("metadata headers cannot outlive the budget and late response bodies are cancelled", { timeout: 2000 }, async () => { + let headersArrive; + let cancelled = false; + const fixture = tarballProof(); + fixture.options.fetchMetadata = (url) => new Promise((resolve) => { + headersArrive = () => resolve({ status: 200, redirected: false, url, + body: new Response(new ReadableStream({ cancel() { cancelled = true; } })).body }); + }); + const budget = new AbortController(); + const timer = setTimeout(() => budget.abort(new Error("fixture headers budget expired")), 100); + const started = Date.now(); + try { + await assert.rejects(canRetryPublishedTarball404(tarballFailure(), exactCanary, { + ...fixture.options, signal: budget.signal, + }), /fixture headers budget expired/); + assert.ok(Date.now() - started < 1000); + headersArrive(); + await new Promise((resolve) => setImmediate(resolve)); + assert.equal(cancelled, true); + } finally { clearTimeout(timer); } +}); + +test("real npm effective registry reads isolated prefix, user, and environment configuration", async () => { + const root = mkdtempSync(path.join(os.tmpdir(), "canary-config-proof-")); + mkdirSync(path.join(root, "prefix")); + writeFileSync(path.join(root, "global.npmrc"), ""); + const env = Object.fromEntries(Object.entries(process.env).filter(([key]) => !key.toLowerCase().startsWith("npm_config_"))); + env.npm_config_userconfig = path.join(root, "user.npmrc"); + env.npm_config_globalconfig = path.join(root, "global.npmrc"); + try { + for (const [user, project, override, expected] of [ + ["registry=https://registry.npmjs.org/\n", "", undefined, true], + ["registry=https://registry.example.invalid/\n", "@paperclipai:registry=https://registry.npmjs.org/\n", undefined, true], + ["registry=https://registry.npmjs.org/\n", "@paperclipai:registry=https://registry.example.invalid/\n", undefined, false], + ["registry=https://registry.npmjs.org/\n", "", "https://registry.example.invalid/", false], + ]) { + writeFileSync(path.join(root, "user.npmrc"), user); + writeFileSync(path.join(root, "prefix", ".npmrc"), project); + const fixture = tarballProof(); + delete fixture.options.runConfig; + assert.equal(await canRetryPublishedTarball404(tarballFailure(), exactCanary, { + ...fixture.options, prefix: path.join(root, "prefix"), env: { ...env, ...(override ? { npm_config_registry: override } : {}) }, + }), expected); + assert.equal(fixture.calls.length, expected ? 1 : 0); + } + } finally { rmSync(root, { recursive: true, force: true }); } +}); + +async function stalledMetadata(run) { + let closed = false; + let closing = false; + let observeClose; + const connectionClosed = new Promise((resolve) => { observeClose = resolve; }); + const server = createServer((_request, response) => { + response.on("close", () => { closed = true; observeClose(); }); + response.writeHead(200, { "content-type": "application/json" }); + response.write('{"name":'); + }); + // An HTTP client can replace an aborted pooled connection while shutdown + // starts. Own late connections too, so server.close cannot wait on them. + server.on("connection", (socket) => { if (closing) socket.destroy(); }); + await new Promise((resolve) => server.listen(0, "127.0.0.1", resolve)); + const fixture = tarballProof(); + fixture.options.fetchMetadata = async (url, options) => { + const response = await fetch(`http://127.0.0.1:${server.address().port}/metadata`, options); + Object.defineProperty(response, "url", { value: url }); + return response; + }; + let closeDeadline; + try { + await run(fixture.options); + await Promise.race([connectionClosed, new Promise((_, reject) => { + closeDeadline = setTimeout(() => reject(new Error("metadata connection outlived cancellation")), 1000); + })]); + assert.equal(closed, true, "metadata connection must close before fixture-forced cleanup"); + } + finally { + clearTimeout(closeDeadline); + closing = true; + await new Promise((resolve) => { + server.close(resolve); + server.closeAllConnections(); + }); + } +} + +test("shared acquisition cancellation aborts the metadata body before the lookup deadline", async () => { + await stalledMetadata(async (options) => { + const budget = new AbortController(); + const timer = setTimeout(() => budget.abort(new Error("fixture shared budget expired")), 250); + try { + await assert.rejects(canRetryPublishedTarball404(tarballFailure(), exactCanary, { ...options, signal: budget.signal }), /fixture shared budget expired/); + } finally { clearTimeout(timer); } + }); +}); + +test("the short config and metadata deadline also covers a stalled real HTTP body", { timeout: 10_000 }, async () => { + assert.equal(TARBALL_METADATA_TIMEOUT_MS, 5000); + const started = Date.now(); + await stalledMetadata(async (options) => { + assert.equal(await canRetryPublishedTarball404(tarballFailure(), exactCanary, options), false); + }); + assert.ok(Date.now() - started < 7000, "metadata must not consume the 120s acquisition budget"); +}); + +test("a verified canary tarball404 retries acquisition and runs onboarding once", async () => { + const result = await canaryStartup({ mode: "tarball-recover" }); + assert.equal(result.code, 0, result.output); + assert.equal(result.calls.filter((call) => call.kind === "npm").length, 2); + assert.equal(result.calls.filter((call) => call.kind === "config").length, 1); + assert.equal(result.calls.filter((call) => call.kind === "metadata").length, 1); + assert.equal(result.calls.filter((call) => call.kind === "onboard").length, 1); + assert.doesNotMatch(result.output, /fixture config diagnostic/); +}); + +test("persistent promised tarball404s stop at three installs without a final lookup or onboarding", async () => { + const result = await canaryStartup({ mode: "tarball-permanent" }); + assert.equal(result.code, 1); + assert.equal(result.calls.filter((call) => call.kind === "npm").length, 3); + assert.equal(result.calls.filter((call) => call.kind === "config").length, 2); + assert.equal(result.calls.filter((call) => call.kind === "metadata").length, 2); + assert.equal(result.calls.filter((call) => call.kind === "onboard").length, 0); + assert.match(result.output, /installation failed.*onboarding was not started/); +}); + +test("private or oversized config and unpublished metadata404 fail without retry or leaking config", async () => { + for (const input of [{ config: "private" }, { config: "large" }, { metadata: "missing" }]) { + const result = await canaryStartup({ mode: "tarball-permanent", ...input }); + assert.equal(result.code, 1); + assert.equal(result.calls.filter((call) => call.kind === "npm").length, 1); + assert.equal(result.calls.filter((call) => call.kind === "onboard").length, 0); + assert.doesNotMatch(result.output, /fixture-user|fixture-password|fixture config diagnostic/); + } +}); + +test("truncated npm output cannot hide an earlier auth failure and qualify a tarball retry", async () => { + const result = await canaryStartup({ mode: "tarball-large" }); + assert.equal(result.code, 1); + assert.equal(result.calls.filter((call) => call.kind === "npm").length, 1); + assert.equal(result.calls.filter((call) => call.kind === "config").length, 0); + assert.equal(result.calls.filter((call) => call.kind === "onboard").length, 0); +}); + +test("tarball recovery does not replay onboarding failures or change ETARGET for other exact versions", async () => { + const failed = await canaryStartup({ mode: "tarball-recover", onboard: "fail" }); + assert.equal(failed.code, 1); + assert.equal(failed.calls.filter((call) => call.kind === "npm").length, 2); + assert.equal(failed.calls.filter((call) => call.kind === "onboard").length, 1); + assert.match(failed.output, /onboarding failed \(exit 17\)/); + const other = await canaryStartup({ mode: "recover", version: "2026.1009.0-nightly.1" }); + assert.equal(other.code, 0, other.output); + assert.equal(other.calls.filter((call) => call.kind === "npm").length, 2); + assert.equal(other.calls.filter((call) => call.kind === "onboard").length, 1); + assert.equal(other.calls.filter((call) => call.kind === "config").length, 0); +}); + +test("existing ETARGET character-tail behavior survives long multibyte npm diagnostics", async () => { + const result = await canaryStartup({ mode: "multibyte-recover" }); + assert.equal(result.code, 0, result.output); + assert.equal(result.calls.filter((call) => call.kind === "npm").length, 2); + assert.equal(result.calls.filter((call) => call.kind === "onboard").length, 1); + assert.equal(result.calls.filter((call) => call.kind === "config").length, 0); +}); + +test("metadata lookup and backoff both consume the unchanged shared acquisition budget", async () => { + for (const input of [{ metadata: "hang" }, { delay: 5000 }]) { + const result = await canaryStartup({ mode: "tarball-permanent", budget: 1000, ...input }); + assert.equal(result.code, 1); + assert.equal(result.calls.filter((call) => call.kind === "npm").length, 1); + assert.equal(result.calls.filter((call) => call.kind === "onboard").length, 0); + assert.match(result.output, /exceeded its startup budget/); + assert.ok(result.elapsed < 4000, `shared-budget cancellation took ${result.elapsed}ms`); + } +}); + +test("cancellation reaps a config process and its descendant that ignore SIGTERM", async () => { + const result = await canaryStartup({ mode: "tarball-permanent", config: "hang", cancelAtKind: "config-descendant" }); + assert.equal(result.code, 1); + assert.equal(result.calls.filter((call) => call.kind === "npm").length, 1); + assert.equal(result.calls.filter((call) => call.kind === "onboard").length, 0); + for (const child of result.calls.filter((call) => ["config", "config-descendant"].includes(call.kind))) { + assert.throws(() => process.kill(child.pid, 0), { code: "ESRCH" }); + } + assert.match(result.output, /stopped by SIGTERM/); +}); diff --git a/tests/canary-onboarding/start-published-canary.mjs b/tests/canary-onboarding/start-published-canary.mjs index d52d0f87dae..f294d8a9f1d 100644 --- a/tests/canary-onboarding/start-published-canary.mjs +++ b/tests/canary-onboarding/start-published-canary.mjs @@ -2,21 +2,148 @@ import { spawn } from "node:child_process"; import { mkdtemp, rm } from "node:fs/promises"; import path from "node:path"; import { fileURLToPath } from "node:url"; +import { loadReleaseManifest } from "../../scripts/release-package-map.mjs"; // Leave three minutes for onboarding inside Playwright's existing five-minute // startup deadline. This budget includes every install attempt and backoff. export const INSTALL_BUDGET_MS = 120_000; const MAX_INSTALL_ATTEMPTS = 3; const STDERR_TAIL_CHARS = 64 * 1024; +export const TARBALL_METADATA_TIMEOUT_MS = 5_000; +const METADATA_LIMIT_BYTES = 64 * 1024; +const PRIMARY_REGISTRY = "https://registry.npmjs.org/"; +const RELEASE_PACKAGE_NAMES = new Set(loadReleaseManifest() + .filter((entry) => entry.publishFromCi === true).map((entry) => entry.name)); -function run(command, args, { env, signal, captureStderr = false }) { +function withAbort(pending, signal) { + return new Promise((resolve, reject) => { + const abort = () => { + signal.removeEventListener("abort", abort); + reject(signal.reason); + }; + const finish = (settle) => (value) => { + signal.removeEventListener("abort", abort); + settle(value); + }; + // Keep handlers on the original operation even after the bounded wait + // ends. A transport need not settle immediately when its signal aborts. + Promise.resolve(pending).then(finish(resolve), finish(reject)); + signal.addEventListener("abort", abort, { once: true }); + if (signal.aborted) abort(); + }); +} + +export async function canRetryPublishedTarball404(stderr, version, { + signal, prefix, env = process.env, fetchMetadata = globalThis.fetch, runConfig = run, +}) { + // A package metadata 404, a third-party artifact, or another release must not + // be mistaken for availability of this canary's announced public tarball. + if (!/^\d+\.\d+\.\d+-canary\.\d+$/.test(version)) return false; + const codes = [...stderr.matchAll(/^npm (?:error|ERR!) code ([A-Z0-9_]+)\r?$/gm)]; + const final = codes.at(-1); + if (codes.length !== 1 || final?.[1] !== "E404") return false; + const block = stderr.slice(final.index); + const requests = [...block.matchAll(/^npm (?:error|ERR!) 404 Not Found - GET (\S+) - Not found\r?$/gm)]; + const httpLines = stderr.match(/^npm (?:error|ERR!).* - (?:GET|HEAD|POST|PUT|DELETE|PATCH) /gm) ?? []; + if (requests.length !== 1 || httpLines.length !== 1) return false; + const tarball = requests[0][1]; + const packageName = [...RELEASE_PACKAGE_NAMES].find((name) => { + const basename = name.split("/").at(-1); + return tarball === `${PRIMARY_REGISTRY}${name}/-/${basename}-${version}.tgz`; + }); + if (!packageName) return false; + const resources = [...block.matchAll(/^npm (?:error|ERR!) 404\s+(?:The requested resource )?'([^']+)'(?: could not be found| is not in this registry)/gm)]; + if (resources.some((resource) => resource[1] !== `${packageName}@${tarball}`)) return false; + signal.throwIfAborted(); + const timeout = new AbortController(); + const timer = setTimeout(() => timeout.abort(new Error("Canary tarball metadata lookup timed out")), TARBALL_METADATA_TIMEOUT_MS); + const lookupSignal = AbortSignal.any([signal, timeout.signal]); + const metadataUrl = `${PRIMARY_REGISTRY}${encodeURIComponent(packageName)}/${version}`; + let response; + try { + // Query only registry values in the failed install's effective prefix, + // inherited cwd and environment. Keep even warnings out of the logs: + // configuration values can contain credentials. Config and HTTP share + // this short deadline, inside the unchanged installation budget. + const config = await runConfig(process.platform === "win32" ? "npm.cmd" : "npm", [ + "config", "get", "registry", "@paperclipai:registry", "--prefix", prefix, + ], { env, signal: lookupSignal, captureStdout: true, captureStderr: true, quiet: true, outputLimit: 4096 }); + if (config.code !== 0 || config.signal || config.outputTruncated) return false; + const lines = config.stdout.trim().split(/\r?\n/); + if (lines.length !== 2 || !lines[0].startsWith("registry=") || !lines[1].startsWith("@paperclipai:registry=")) return false; + const globalRegistry = lines[0].slice("registry=".length); + const scoped = lines[1].slice("@paperclipai:registry=".length); + const effectiveRegistry = packageName.startsWith("@") && scoped !== "undefined" ? scoped : globalRegistry; + if (![PRIMARY_REGISTRY, PRIMARY_REGISTRY.slice(0, -1)].includes(effectiveRegistry)) return false; + response = await withAbort(Promise.resolve(fetchMetadata(metadataUrl, { + signal: lookupSignal, redirect: "error", credentials: "omit", + headers: { accept: "application/json", "cache-control": "no-cache" }, + })).then((arrived) => { + if (lookupSignal.aborted) void arrived.body?.cancel().catch(() => {}); + return arrived; + }), lookupSignal); + if (response.status !== 200 || response.redirected || response.url !== metadataUrl || !response.body) return false; + const reader = response.body.getReader(); + const cancelReader = () => { void reader.cancel().catch(() => {}); }; + lookupSignal.addEventListener("abort", cancelReader, { once: true }); + const chunks = []; + let bytes = 0; + try { + while (true) { + lookupSignal.throwIfAborted(); + const { done, value } = await withAbort(reader.read(), lookupSignal); + if (done) break; + bytes += value.byteLength; + if (bytes > METADATA_LIMIT_BYTES) return false; + chunks.push(value); + } + } finally { + // Initiate cleanup without extending either deadline for a transport's + // pending/rejected cancellation acknowledgement. + lookupSignal.removeEventListener("abort", cancelReader); + cancelReader(); + reader.releaseLock(); + } + lookupSignal.throwIfAborted(); + const manifest = JSON.parse(Buffer.concat(chunks).toString("utf8")); + const integrity = manifest.dist?.integrity; + // This validates the integrity announced by primary metadata. No tarball + // is downloaded here, and this is not an independent content digest proof. + return manifest.name === packageName && manifest.version === version + && manifest.dist?.tarball === tarball && typeof integrity === "string" + && /^sha512-[A-Za-z0-9+/]{86}==$/.test(integrity) + && Buffer.from(integrity.slice(7), "base64").toString("base64") === integrity.slice(7); + } catch { + signal.throwIfAborted(); + return false; + } finally { + void response?.body?.cancel().catch(() => {}); + clearTimeout(timer); + } +} + +function run(command, args, { env, signal, captureStderr = false, captureStdout = false, quiet = false, outputLimit = STDERR_TAIL_CHARS }) { return new Promise((resolve, reject) => { signal.throwIfAborted(); const grouped = process.platform !== "win32"; const child = spawn(command, args, { - env, detached: grouped, stdio: ["ignore", "inherit", captureStderr ? "pipe" : "inherit"], + env, detached: grouped, stdio: ["ignore", captureStdout ? "pipe" : "inherit", captureStderr ? "pipe" : "inherit"], }); - let stderr = ""; + let stderr = quiet ? Buffer.alloc(0) : ""; + let stdout = quiet ? Buffer.alloc(0) : ""; + let outputTruncated = false; + const retain = (previous, chunk) => { + if (quiet) { + outputTruncated ||= previous.length + chunk.byteLength > outputLimit; + return Buffer.concat([previous, chunk]).subarray(-outputLimit); + } + // Preserve the existing install stderr character tail and ETARGET + // behavior, including multibyte diagnostics. Only private config capture + // uses a byte bound; any truncated E404 evidence is rejected. + const next = previous + chunk.toString(); + outputTruncated ||= next.length > outputLimit; + return next.slice(-outputLimit); + }; let stopping; const kill = (kind) => { try { @@ -34,15 +161,20 @@ function run(command, args, { env, signal, captureStderr = false }) { }; signal.addEventListener("abort", stop, { once: true }); child.stderr?.on("data", (chunk) => { - process.stderr.write(chunk); - stderr = (stderr + chunk.toString()).slice(-STDERR_TAIL_CHARS); + if (!quiet) process.stderr.write(chunk); + stderr = retain(stderr, chunk); + }); + child.stdout?.on("data", (chunk) => { + if (!quiet) process.stdout.write(chunk); + stdout = retain(stdout, chunk); }); child.on("error", reject); child.on("close", async (code, exitSignal) => { signal.removeEventListener("abort", stop); await stopping; if (signal.aborted) reject(signal.reason); - else resolve({ code, signal: exitSignal, stderr }); + else resolve({ code, signal: exitSignal, + stderr: quiet ? stderr.toString("utf8") : stderr, stdout: quiet ? stdout.toString("utf8") : stdout, outputTruncated }); }); }); } @@ -85,7 +217,10 @@ export async function startPublishedCanary({ if (result.code === 0) break; const codes = [...result.stderr.matchAll(/^npm (?:error|ERR!) code ([A-Z0-9_]+)\r?$/gm)]; const missingVersion = codes.at(-1)?.[1] === "ETARGET"; - if (!missingVersion || result.signal || attempt === MAX_INSTALL_ATTEMPTS) { + const retryable = !result.signal && attempt < MAX_INSTALL_ATTEMPTS + && (missingVersion || (!result.outputTruncated + && await canRetryPublishedTarball404(result.stderr, version, { signal: installSignal, prefix, env }))); + if (!retryable) { throw new Error(`Canary npm installation failed (exit ${result.code ?? result.signal}); onboarding was not started`); } await rm(prefix, { recursive: true, force: true }); From 07f3536e37687a42def210ecffb598f3e58813b4 Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Sat, 10 Oct 2026 17:10:23 -0700 Subject: [PATCH 4/7] fix(connections): discover Sentry OAuth through its MCP resource (#15837) Correct Sentry OAuth discovery and cover new and saved client registration. Co-Authored-By: Paperclip --- doc/connections/CONNECTOR-PLAYBOOK.md | 34 ++++---- doc/connections/README.md | 2 +- doc/connections/SENTRY.md | 44 ++++++++++ packages/shared/src/app-definitions.test.ts | 7 ++ .../shared/src/app-definitions/sentry.json | 2 +- scripts/ingest-app-definitions.mjs | 2 +- .../src/__tests__/tool-access-service.test.ts | 85 +++++++++++++++++++ 7 files changed, 158 insertions(+), 18 deletions(-) create mode 100644 doc/connections/SENTRY.md diff --git a/doc/connections/CONNECTOR-PLAYBOOK.md b/doc/connections/CONNECTOR-PLAYBOOK.md index c04bed4648c..04c650f7f11 100644 --- a/doc/connections/CONNECTOR-PLAYBOOK.md +++ b/doc/connections/CONNECTOR-PLAYBOOK.md @@ -1712,27 +1712,31 @@ The broker implements this in `discoverOAuthEndpoints` unconditional. `oauthEndpointsForConnection` resolves endpoints in this order: -1. If the manifest's method `defaults` ship a **complete** pair - (`authorizationEndpoint` **and** `tokenEndpoint`), those are used - unconditionally. `discoverOAuthEndpoints` never runs in this case, so - endpoints stored on the connection's own OAuth config and 401 challenge - hints are **not consulted at all**. -2. Otherwise, for `mcp_remote` connections, the broker calls +1. A method's `defaults.discoveryUrl` is authoritative. The broker resolves + that document before cached endpoints, including when a saved connection + resumes or reconnects. It must describe the MCP resource or its authorization + server, not a separate web/API OAuth service. Failure stops setup. +2. A complete manifest pair (`authorizationEndpoint` and `tokenEndpoint`) + replaces discovery when the method does not support dynamic registration, + or when the connection uses an operator-entered client. A method that can + register automatically still discovers its MCP authorization server. +3. Otherwise, for `mcp_remote` connections, the broker calls `discoverOAuthEndpoints`, which first checks endpoints already stored on the connection's own OAuth config (falling back field-by-field to the 401 challenge hints); a complete stored/hinted pair is used as-is — no `.well-known` fetch. -3. Only when neither of the above yields a complete pair does the broker run +4. Only when none of the above yields a complete pair does the broker run the RFC 9728 → RFC 8414 discovery chain above. -Consequence: complete manifest endpoint hints are **authoritative, not -hints** — they override even endpoints that an earlier discovery persisted -on the connection, and if they go stale the broker keeps using them. For -discovery-capable vendors, ship only `serverUrl` in `defaults` (as -`notion.json` does) so the broker discovers fresh endpoints at connect -time; add explicit `authorizationEndpoint`/`tokenEndpoint` only for vendors -that do not publish RFC 9728/8414 metadata, and then own keeping them -current. +For discovery-capable vendors, prefer `serverUrl` alone (as `notion.json` +does), or pin the reviewed protected-resource document when the provider +requires an exact path. Keep explicit metadata and endpoint URLs current. +When changing discovery precedence, test each affected catalog method against +its actual metadata shape. Cover both new setup and a saved registration. +A metadata preflight that finds registration somewhere in its search is not +proof that sign-in selects that issuer. Sentry's web/API and MCP hosts both +serve valid metadata, but only its MCP issuer supports automatic registration; +see [Sentry](./SENTRY.md). ### Dynamic client registration (RFC 7591) diff --git a/doc/connections/README.md b/doc/connections/README.md index 829ff427b52..df86a57e61a 100644 --- a/doc/connections/README.md +++ b/doc/connections/README.md @@ -13,7 +13,7 @@ Runtime authentication: [AI Connections](./AI-CONNECTIONS.md). Long-term memory: [Experimental memory connectors](./MEMORY.md). Provider notes: [Google Workspace](./GOOGLE-WORKSPACE.md), -[Gmail](./GMAIL.md), [Asana](./ASANA.md), [PostHog](./POSTHOG.md), [Neon](./NEON.md), [Superagent](./SUPERAGENT.md), [Gauge](./GAUGE.md), +[Gmail](./GMAIL.md), [Asana](./ASANA.md), [Sentry](./SENTRY.md), [PostHog](./POSTHOG.md), [Neon](./NEON.md), [Superagent](./SUPERAGENT.md), [Gauge](./GAUGE.md), [AgentMail](./AGENTMAIL.md), [Telem.AI](./TELEM.md), [iMessage Photon](./IMESSAGE-PHOTON.md), and [Enterpret](./ENTERPRET.md). Optional credential custody: [Vercel Connect](./VERCEL-CONNECT.md). diff --git a/doc/connections/SENTRY.md b/doc/connections/SENTRY.md new file mode 100644 index 00000000000..8a05839ee3d --- /dev/null +++ b/doc/connections/SENTRY.md @@ -0,0 +1,44 @@ +# Sentry + +Paperclip connects to Sentry's [official MCP server](https://mcp.sentry.dev/) +at `https://mcp.sentry.dev/mcp`. Browser sign-in uses the MCP authorization +server. A normal connection does not require a customer-created OAuth app. + +## Discovery + +The method pins the [MCP protected-resource metadata](https://mcp.sentry.dev/.well-known/oauth-protected-resource/mcp). +That document names `https://mcp.sentry.dev` as its authorization server. +Its [authorization-server metadata](https://mcp.sentry.dev/.well-known/oauth-authorization-server) +advertises automatic client registration, PKCE S256, refresh tokens, and +client ID metadata documents. Paperclip retains its normal client selection +and binding checks. + +Do not use `https://sentry.io/.well-known/oauth-authorization-server` for this +method. That server describes the web/API OAuth service. It does not advertise +MCP client registration. Both metadata URLs return valid JSON, so checking +only HTTP status or the presence of authorization/token endpoints misses this +failure. + +## Repair an existing connection + +After deploying the corrected catalog, resume or reconnect the existing Sentry +connection. Keep its connection identity and access policies. Complete browser +consent if required. Confirm the intended organization and projects through a +read-only tool call from the assigned agent. + +The catalog repair fixes setup and reconnect discovery. It cannot restore an +expired or revoked grant without the owner's consent. If the saved connection +changes to **Finish setup**, inspect its health error, refresh outcome, and +transition timestamp separately. A setup error alone does not prove why the +previous grant stopped working. + +## Verification + +The service regression uses separate, realistic metadata for both Sentry +hosts. It checks new registration and reuse of an existing MCP registration, +the authorization URL, resource, requested scopes, and PKCE. These fixtures +prove Paperclip's endpoint selection; they do not prove live account consent. + +Before claiming restoration, verify browser consent, catalog discovery, and a +real agent read. Record token refresh separately. Do not publish credentials +or unredacted authorization URLs in evidence. diff --git a/packages/shared/src/app-definitions.test.ts b/packages/shared/src/app-definitions.test.ts index f92d1f1e6d8..ef162f11466 100644 --- a/packages/shared/src/app-definitions.test.ts +++ b/packages/shared/src/app-definitions.test.ts @@ -689,6 +689,13 @@ describe("AppDefinition catalog", () => { consoleLinks: { register: "https://app.asana.com/0/my-apps" }, }); expect(method("linear")?.ownershipModes).toEqual(["dcr", "customer"]); + expect(method("sentry")).toMatchObject({ + key: "mcp-oauth", + defaults: { + serverUrl: "https://mcp.sentry.dev/mcp", + discoveryUrl: "https://mcp.sentry.dev/.well-known/oauth-protected-resource/mcp", + }, + }); expect(method("zapier")).toMatchObject({ key: "generated-url", auth: "none", diff --git a/packages/shared/src/app-definitions/sentry.json b/packages/shared/src/app-definitions/sentry.json index b626e6e870d..5a85dd024f1 100644 --- a/packages/shared/src/app-definitions/sentry.json +++ b/packages/shared/src/app-definitions/sentry.json @@ -26,7 +26,7 @@ "whenToUse": "Use the provider-hosted connection for the quickest setup.", "defaults": { "serverUrl": "https://mcp.sentry.dev/mcp", - "discoveryUrl": "https://sentry.io/.well-known/oauth-authorization-server", + "discoveryUrl": "https://mcp.sentry.dev/.well-known/oauth-protected-resource/mcp", "scopesHint": [ "org:read", "project:write", diff --git a/scripts/ingest-app-definitions.mjs b/scripts/ingest-app-definitions.mjs index 73151fa9125..e8be36185f1 100644 --- a/scripts/ingest-app-definitions.mjs +++ b/scripts/ingest-app-definitions.mjs @@ -783,7 +783,7 @@ const apps = [ { serverUrl: "https://mcp.sentry.dev/mcp", discoveryUrl: - "https://sentry.io/.well-known/oauth-authorization-server", + "https://mcp.sentry.dev/.well-known/oauth-protected-resource/mcp", }, "S2", "Connect the Sentry organization and projects agents need for incident work.", diff --git a/server/src/__tests__/tool-access-service.test.ts b/server/src/__tests__/tool-access-service.test.ts index 97a9be58ec2..84b3152e3bf 100644 --- a/server/src/__tests__/tool-access-service.test.ts +++ b/server/src/__tests__/tool-access-service.test.ts @@ -12952,6 +12952,91 @@ describeEmbeddedPostgres("tool access service", () => { ); }); + it.each(["new", "saved"])("starts Sentry MCP sign-in for a %s connection without a manual OAuth app", async (state) => { + const company = await createCompany(db); + const service = createTestToolAccessService(db); + const connected = await service.connectGalleryApp(company.id, { + galleryKey: "sentry", connectionMethodKey: "mcp-oauth", name: "Sentry", + }); + const redirectUri = "http://localhost:3200/api/tools/oauth/callback"; + const issuer = "https://mcp.sentry.dev"; + const resource = `${issuer}/mcp`; + const scopes = ["org:read", "project:write", "team:write", "event:write", "alerts:write"]; + if (state === "saved") { + const [connection] = await db.select().from(toolConnections) + .where(eq(toolConnections.id, connected.connectionId)); + // An existing MCP registration must survive catalog changes. This is the + // state before a reconnect, with endpoints cached by successful discovery. + await db.update(toolConnections).set({ config: { + ...connection.config, + oauth: { + ...connection.config.oauth as Record, + provider: "sentry", issuer, resource, + authorizationUrl: `${issuer}/oauth/authorize`, tokenUrl: `${issuer}/oauth/token`, + metadataUrl: `${issuer}/.well-known/oauth-authorization-server`, + clientId: "saved-sentry-client", clientRegistrationSource: "dcr", + clientIssuer: issuer, clientResource: resource, + clientCompanyId: company.id, clientRedirectUri: redirectUri, + clientTokenEndpointAuthMethod: "none", + }, + } }).where(eq(toolConnections.id, connected.connectionId)); + } + const calls: string[] = []; + vi.spyOn(globalThis, "fetch").mockImplementation(async (url, init) => { + const href = String(url); + calls.push(href); + // Both hosts publish valid OAuth metadata. The web/API issuer cannot + // register an MCP client; discovery must select the resource's issuer. + if (href === "https://sentry.io/.well-known/oauth-authorization-server") { + return mcpHttpResponse({ + issuer: "https://sentry.io", authorization_endpoint: "https://sentry.io/oauth/authorize/", + token_endpoint: "https://sentry.io/oauth/token/", code_challenge_methods_supported: ["S256"], + }); + } + if (href === `${issuer}/.well-known/oauth-protected-resource/mcp`) { + return mcpHttpResponse({ resource, authorization_servers: [issuer], scopes_supported: scopes }); + } + if (href === `${issuer}/.well-known/oauth-authorization-server`) { + return mcpHttpResponse({ + issuer, authorization_endpoint: `${issuer}/oauth/authorize`, token_endpoint: `${issuer}/oauth/token`, + registration_endpoint: `${issuer}/oauth/register`, scopes_supported: scopes, + grant_types_supported: ["authorization_code", "refresh_token"], + code_challenge_methods_supported: ["S256"], + token_endpoint_auth_methods_supported: ["client_secret_basic", "client_secret_post", "none"], + client_id_metadata_document_supported: true, + }); + } + if (href === `${issuer}/oauth/register`) { + expect(JSON.parse(String(init?.body))).toMatchObject({ + redirect_uris: [redirectUri], token_endpoint_auth_method: "none", + }); + return mcpHttpResponse({ + client_id: "new-sentry-client", redirect_uris: [redirectUri], + grant_types: ["authorization_code", "refresh_token"], response_types: ["code"], + token_endpoint_auth_method: "none", + }); + } + throw new Error(`unexpected fetch ${href}`); + }); + const started = await service.startOAuth(company.id, connected.connectionId, { + redirectUri, actor: { actorType: "user", actorId: "board" }, + }); + const authorization = new URL(started.authorizationUrl); + expect(authorization.origin + authorization.pathname).toBe(`${issuer}/oauth/authorize`); + expect(authorization.searchParams.get("resource")).toBe(resource); + expect(authorization.searchParams.get("scope")?.split(" ")).toEqual(scopes); + expect(authorization.searchParams.get("code_challenge_method")).toBe("S256"); + expect(authorization.searchParams.get("client_id")).toBe(`${state}-sentry-client`); + expect(calls).not.toContain("https://sentry.io/.well-known/oauth-authorization-server"); + expect(calls.filter((url) => url === `${issuer}/oauth/register`)).toHaveLength(state === "new" ? 1 : 0); + const [saved] = await db.select().from(toolConnections) + .where(eq(toolConnections.id, connected.connectionId)); + expect(saved.config.oauth).toMatchObject({ + clientId: `${state}-sentry-client`, clientIssuer: issuer, clientResource: resource, + clientCompanyId: company.id, clientRedirectUri: redirectUri, + }); + }); + it("uses Asana v2 metadata instead of stale v1 endpoints for a saved custom app", async () => { const company = await createCompany(db); const service = createTestToolAccessService(db); From 8dd381f5ddb3501c55c715ab733f0717a12346e5 Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Sat, 10 Oct 2026 20:25:44 -0700 Subject: [PATCH 5/7] perf(ci): run the db project as its own workspaces-c lane (#15831) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Pull requests to this repository run a large CI matrix. The slowest check sets the time a contributor waits for a green run. > - The `General tests (workspaces-b)` lane runs every non-server Vitest project except `ui` and `cli` in one job, one project after the other. After #15826 leveled the server shards, this lane became the slowest check: 593s in run 38084171761, 523s in 38083084276 and 499s in 38083020834 (all 2026-10-10). > - The `@paperclipai/db` project alone is 168s to 250s of that lane's 436s to 513s of vitest time. Its `client.test.ts` suite is 117s to 189s and `backup-lib.test.ts` is 70s to 107s. The other seventeen projects together are only 263s to 276s. > - Vitest's native `--shard` does not level this lane. The db project is already file-parallel, and a hash-ordered slice can put both slow suites on one runner. > - This pull request moves the db project into its own `workspaces-c` lane in both `pr-trusted.yml` and `release-verify.yml`. The runner script gains a `general-workspaces-c` group, and `general-workspaces-b` drops the db project. > - The benefit is that both lanes should finish in about 230s to 340s instead of 499s to 593s, and `workspaces-b` leaves the critical path. The cost is one more job per run, about 60s of setup. ## Linked Issues or Issue Description **Describe the enhancement** The `General tests (workspaces-b)` PR check is the slowest check in the run. It runs seventeen Vitest projects in sequence on one runner. The `@paperclipai/db` project is one third to one half of its vitest time. **Why is this needed** The slowest check decides the wall-clock time of every PR run. In the last three fully green runs on the current server-shard manifest, `workspaces-b` took 499s, 523s and 593s (runs 38083020834, 38083084276 and 38084171761, 2026-10-10). The next slowest lanes, `Build` and the server shards, sit at about 480s to 570s. A split of `workspaces-b` removes 170s to 250s from this lane. **Proposed solution** Run the `@paperclipai/db` project as its own lane, `general-workspaces-c`. Keep the other seventeen projects in `general-workspaces-b`. Add the lane to both the PR and the release verification matrix. Extend the guard tests so the three workspace lanes still cover the root Vitest project list exactly once, so the new group rejects shard flags, and so both workflows carry the lane unsharded. **Alternatives considered** - Vitest native `--shard` on `workspaces-b`, as `workspaces-a` does. The db project is already file-parallel and its wall time is set by two long suites, so a hash-ordered slice does not give a predictable split. - Move `adapter-utils` (115s to 117s) into the new lane as well. That makes the db lane 285s to 366s of tests and the rest 147s to 160s, which is less level than db alone (168s to 250s) against the rest (263s to 276s). - Three lanes. Not needed now: after this split both lanes sit under the server shards and `Build`, so a third runner would not shorten the run. ## What Changed - `scripts/run-vitest-stable.mjs`: new `general-workspaces-c` group that runs `@paperclipai/db` alone. `general-workspaces-b` now excludes it. Local `--mode all` runs all four general groups, so every project still runs once. The dry-run output reports the new group's projects. - `.github/workflows/pr-trusted.yml`: a `workspaces-c` entry in the general tests matrix, with a comment that records the 2026-10-10 measurements. - `.github/workflows/release-verify.yml`: the same `workspaces-c` entry, to keep parity with the PR workflow. - `scripts/__tests__/run-vitest-stable-shard.test.mjs`: the coverage test includes the new group; shard flags are rejected for both `workspaces-b` and `workspaces-c`; a new test asserts that `workspaces-c` runs only the db project, that `workspaces-b` no longer carries it, and that both workflows list both lanes without shard arguments. The workflow check parses each `- group:` matrix entry with comment lines removed, so a comment between keys cannot hide a shard setting, and it first confirms the parser sees the shard keys on the `workspaces-a` entries. A small parser test covers the comment-between-keys case (Greptile P2). - `scripts/__tests__/release-verify-workflow.test.mjs`: the release workflow must carry `general-workspaces-c`. ## Verification - `GITHUB_WORKFLOW= node --test scripts/__tests__/run-vitest-stable-shard.test.mjs`: 32 pass (was 29). - `node --test scripts/__tests__/release-verify-workflow.test.mjs`: the workflow parity test passes. Two unrelated tests in that file (`a publisher lockfile lets a fresh smoke install retain validation after ... drift`) fail on my machine on pristine `master` too, because `pnpm` is not on its PATH. They pass in CI. - `node scripts/run-vitest-stable.mjs --mode general --group general-workspaces-c --dry-run` reports `["@paperclipai/db"]`; `--group general-workspaces-b` reports the other seventeen projects. - Both workflow files parse as YAML; the general tests matrix has 19 entries in `pr-trusted.yml` and 17 in `release-verify.yml`. - This PR's own run exercises the script change but not the workflow change: `pr.yml` calls `pr-trusted.yml@master`, so the matrix comes from `master` and the `workspaces-c` job appears only on PR runs created after the merge. Because the script comes from this branch, the `workspaces-b` job in this PR's run (38090713637) ran the seventeen remaining projects and finished in 340s, inside the expected 325s to 340s band. No job in that run ran `@paperclipai/db`. - To cover the gap, I ran the new lane locally with the same command CI uses, `pnpm test:run:general -- --group general-workspaces-c`: 49 test files, 180 tests, all passed, 111s of vitest time (c324d0080, Node 25, PGlite). Expected result on the first PR run after merge, from the three measured runs: | Check | Before | After (expected) | | --- | --- | --- | | `General tests (workspaces-b)` | 499-593s | ~325-340s (263-276s of vitest time plus ~60s of setup) | | `General tests (workspaces-c)` | (new) | ~230-310s (168-250s of vitest time plus ~60s of setup) | | Slowest check in the run | `workspaces-b` | `Build` (497-570s) or a server shard (~480-520s) | ## Risks - Low risk. The group split only changes which job runs the db project. The guard test asserts that the three workspace lanes cover the root Vitest project list exactly once, so a project cannot be dropped or run twice. - One more job per PR and release verification run, about 60s of setup time on the runner fleet. - The lane names are matrix labels only. The required checks are `ci / verify` and `ci / e2e`, which aggregate the matrix, so no branch ruleset needs an update. ## Model Used - Claude (Anthropic), model ID `claude-fable-5-1`, running as a Paperclip agent through the Claude local adapter with tool use (shell, file edits, GitHub CLI). The agent collected the CI data, made the change, ran the tests, and wrote this description. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [ ] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Bender (Fable) Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com> --- .github/workflows/pr-trusted.yml | 8 ++ .github/workflows/release-verify.yml | 3 + .../release-verify-workflow.test.mjs | 2 +- .../run-vitest-stable-shard.test.mjs | 82 +++++++++++++++++-- scripts/run-vitest-stable.mjs | 31 ++++++- 5 files changed, 117 insertions(+), 9 deletions(-) diff --git a/.github/workflows/pr-trusted.yml b/.github/workflows/pr-trusted.yml index 56b6f594c12..228ac3bd4b7 100644 --- a/.github/workflows/pr-trusted.yml +++ b/.github/workflows/pr-trusted.yml @@ -585,8 +585,16 @@ jobs: group_label: workspaces-a (2/2) shard_index: 1 shard_count: 2 + # workspaces-b became the slowest check (499-593s) once the server + # shards were rebalanced: the db project alone was 168-250s of its + # 436-513s of vitest time (PR runs 38084171761, 38083084276 and + # 38083020834, 2026-10-10). The db project now runs as its own + # workspaces-c lane; the rest of workspaces-b is ~265-275s of + # vitest time, so both jobs land near 300-350s with setup. - group: general-workspaces-b group_label: workspaces-b + - group: general-workspaces-c + group_label: workspaces-c steps: - name: Checkout repository diff --git a/.github/workflows/release-verify.yml b/.github/workflows/release-verify.yml index 492b7e62b47..3fd321edb68 100644 --- a/.github/workflows/release-verify.yml +++ b/.github/workflows/release-verify.yml @@ -174,6 +174,9 @@ jobs: shard_count: 2 - group: general-workspaces-b group_label: workspaces-b + # Keep parity with pr.yml: the db project runs as its own lane. + - group: general-workspaces-c + group_label: workspaces-c steps: - name: Checkout repository diff --git a/scripts/__tests__/release-verify-workflow.test.mjs b/scripts/__tests__/release-verify-workflow.test.mjs index 043636edb91..9d00a15b79e 100644 --- a/scripts/__tests__/release-verify-workflow.test.mjs +++ b/scripts/__tests__/release-verify-workflow.test.mjs @@ -505,7 +505,7 @@ test("release verify workflow covers the same split test surface as stable PR ve assert.match(buildJob, /persist-credentials: false/); assert.doesNotMatch(buildJob, /cache: pnpm/); - for (const group of ["general-server-without-chat-or-native-runner", "general-chat", "general-workspaces-a", "general-workspaces-b"]) { + for (const group of ["general-server-without-chat-or-native-runner", "general-chat", "general-workspaces-a", "general-workspaces-b", "general-workspaces-c"]) { assert.match(verifyWorkflow, new RegExp(`group: ${group}`)); } for (const [group, count] of [["general-server-without-chat-or-native-runner", 10], ["general-chat", 3]]) { diff --git a/scripts/__tests__/run-vitest-stable-shard.test.mjs b/scripts/__tests__/run-vitest-stable-shard.test.mjs index a044c67c151..3dd3e9680e0 100644 --- a/scripts/__tests__/run-vitest-stable-shard.test.mjs +++ b/scripts/__tests__/run-vitest-stable-shard.test.mjs @@ -107,10 +107,12 @@ test("general shards retain scripts and route-named suites outside the serialize ]) assert.ok(general.selectedGeneralServerSuites.includes(file), `missing configured server suite: ${file}`); }); -test("shard flags are rejected for the workspaces-b group", () => { - const result = dryRun(["--mode", "general", "--group", "general-workspaces-b", "--shard-index", "0", "--shard-count", "3"]); - assert.notEqual(result.status, 0, "workspaces-b must not accept shard flags"); -}); +for (const group of ["general-workspaces-b", "general-workspaces-c"]) { + test(`shard flags are rejected for the ${group} group`, () => { + const result = dryRun(["--mode", "general", "--group", group, "--shard-index", "0", "--shard-count", "3"]); + assert.notEqual(result.status, 0, `${group} must not accept shard flags`); + }); +} test("workspace lanes cover every non-server project in the root Vitest configuration", () => { const config = readFileSync(path.join(repoRoot, "vitest.config.ts"), "utf8"); @@ -118,11 +120,81 @@ test("workspace lanes cover every non-server project in the root Vitest configur assert.ok(roots.includes("server"), "expected the explicit root Vitest project list"); const expected = roots.filter(root => root !== "server") .map(root => JSON.parse(readFileSync(path.join(repoRoot, root, "package.json"), "utf8")).name).sort(); - const actual = ["general-workspaces-a", "general-workspaces-b"] + const actual = ["general-workspaces-a", "general-workspaces-b", "general-workspaces-c"] .flatMap(group => dryRunJson(["--mode", "general", "--group", group]).workspaceProjects).sort(); assert.deepEqual(actual, expected, "no configured project may be silently omitted or run twice"); }); +// Collects every `- group:` matrix entry in a workflow with the keys it sets. +// Comment and blank lines are dropped first, and an entry ends only when the +// indentation returns to the item level, so a comment between two keys cannot +// hide a shard setting from the assertions below. +function matrixEntries(text) { + const entries = []; + let current = null; + for (const line of text.split("\n")) { + if (/^\s*(#|$)/.test(line)) continue; + const indent = line.match(/^ */)[0].length; + const item = line.match(/^ *- group: (\S+)/); + if (item) { + current = { group: item[1], indent, keys: ["group"] }; + entries.push(current); + continue; + } + if (!current) continue; + if (indent <= current.indent) { current = null; continue; } + const key = line.match(/^ *([A-Za-z_]+):/); + if (key) current.keys.push(key[1]); + } + return entries; +} + +test("matrixEntries sees shard keys that sit behind comments or later in the entry", () => { + const entries = matrixEntries([ + " include:", + " - group: general-workspaces-b", + " group_label: workspaces-b", + " # a comment between keys must not hide the shard setting", + " shard_index: 0", + "", + " shard_count: 3", + " - group: general-workspaces-c", + " group_label: workspaces-c", + " steps:", + " - name: unrelated", + " shard_index: 9", + ].join("\n")); + assert.deepEqual(entries.map(entry => entry.group), ["general-workspaces-b", "general-workspaces-c"]); + assert.deepEqual(entries[0].keys, ["group", "group_label", "shard_index", "shard_count"]); + assert.deepEqual(entries[1].keys, ["group", "group_label"], "keys after the matrix must not leak into the last entry"); +}); + +test("workspaces-c runs the db project alone and workspaces-b no longer carries it", () => { + // The db project was 168-250s of the 436-513s workspaces-b vitest time + // (PR runs 38084171761, 38083084276 and 38083020834, 2026-10-10), so it + // runs in its own lane. The PR and release matrices both carry that lane. + const workspacesC = dryRunJson(["--mode", "general", "--group", "general-workspaces-c"]); + assert.deepEqual(workspacesC.workspaceProjects, ["@paperclipai/db"]); + assert.equal(workspacesC.workspacesVitestShard, null); + const workspacesB = dryRunJson(["--mode", "general", "--group", "general-workspaces-b"]); + assert.ok(!workspacesB.workspaceProjects.includes("@paperclipai/db"), "db must not run twice"); + assert.ok(workspacesB.workspaceProjects.length > 0, "workspaces-b must still run its other projects"); + for (const workflow of ["pr-trusted.yml", "release-verify.yml"]) { + const entries = matrixEntries(readFileSync(path.join(repoRoot, ".github", "workflows", workflow), "utf8")); + // The parser must demonstrably see shard keys in this workflow, otherwise + // the unsharded assertions below could pass for the wrong reason. + const sharded = entries.filter(entry => entry.group === "general-workspaces-a"); + assert.ok(sharded.length > 0 && sharded.every(entry => entry.keys.includes("shard_index") && entry.keys.includes("shard_count")), + `${workflow} must carry the sharded workspaces-a entries`); + for (const group of ["general-workspaces-b", "general-workspaces-c"]) { + const matching = entries.filter(entry => entry.group === group); + assert.equal(matching.length, 1, `${workflow} must run ${group} exactly once`); + assert.ok(matching[0].keys.includes("group_label"), `${workflow} must label the ${group} job`); + assert.deepEqual(matching[0].keys.filter(key => key.startsWith("shard_")), [], `${workflow} must run ${group} unsharded`); + } + } +}); + test("workspaces-a shards map to Vitest native --shard slices over a stable project list", () => { const shards = [0, 1].map((index) => dryRunJson([ diff --git a/scripts/run-vitest-stable.mjs b/scripts/run-vitest-stable.mjs index dd00f145cda..a62f90094e1 100644 --- a/scripts/run-vitest-stable.mjs +++ b/scripts/run-vitest-stable.mjs @@ -117,9 +117,27 @@ function excludedServerSuites(groupName) { } const generalWorkspacesAGroupName = "general-workspaces-a"; const generalWorkspacesBGroupName = "general-workspaces-b"; +const generalWorkspacesCGroupName = "general-workspaces-c"; const generalWorkspacesAProjects = ["@paperclipai/ui", "paperclipai"]; -const generalWorkspacesBProjects = nonServerProjects.filter((project) => !generalWorkspacesAProjects.includes(project)); -const generalGroupNames = [generalServerGroupName, generalWorkspacesAGroupName, generalWorkspacesBGroupName]; +// The db project runs in its own lane. Its vitest invocation took 168-250s +// of a 436-513s workspaces-b job (PR runs 38084171761, 38083084276 and +// 38083020834, 2026-10-10), which made workspaces-b the slowest check once +// the server shards were rebalanced. Vitest's native --shard would not help +// here: the project is already file-parallel and client.test.ts alone is +// 117-189s of its wall time, so a hash-ordered slice can land both of its +// slow suites on one runner. Splitting by project is deterministic and +// leaves the remaining projects at ~265-275s of vitest time, roughly level +// with this lane. +const generalWorkspacesCProjects = ["@paperclipai/db"]; +const generalWorkspacesBProjects = nonServerProjects.filter( + (project) => !generalWorkspacesAProjects.includes(project) && !generalWorkspacesCProjects.includes(project), +); +const generalGroupNames = [ + generalServerGroupName, + generalWorkspacesAGroupName, + generalWorkspacesBGroupName, + generalWorkspacesCGroupName, +]; const allowedGeneralGroupNames = [ ...generalGroupNames, generalServerWithoutChatGroupName, @@ -520,6 +538,11 @@ function runGeneralGroup(routeTests, groupName, shardIndex = null, shardCount = return; } + if (groupName === generalWorkspacesCGroupName) { + runProjectGroup(generalWorkspacesCProjects, groupName); + return; + } + fail(`Unknown group "${groupName}".`); } @@ -604,7 +627,9 @@ if (options.dryRun) { ? generalWorkspacesAProjects : options.group === generalWorkspacesBGroupName ? generalWorkspacesBProjects - : null, + : options.group === generalWorkspacesCGroupName + ? generalWorkspacesCProjects + : null, workspacesVitestShard: options.group === generalWorkspacesAGroupName && options.shardCount !== null && From 42ecc96556e70e9ae2e78b5ef2a1ace4f3683a3a Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Sat, 10 Oct 2026 21:10:13 -0700 Subject: [PATCH 6/7] perf(ci): prefetch the consumer image and Grok binary during package staging (#15849) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Pull requests to this repository run a large CI matrix. The slowest check sets the time a contributor waits for a green run. > - After #15826 leveled the server shards and #15831 split `workspaces-b`, the `Build` lane is the slowest check: 634s in run 38107578322 (2026-10-11 03:11Z), ahead of the next server shard at 574s. It was 533s in run 38107430861 and 543s to 568s in earlier runs that night. > - `Build` is about 100s of setup, 148s to 195s of `pnpm build`, and 278s to 320s of `scripts/verify-grok-npm-install.mjs`. The verifier stages and packs the public packages first (84s to 100s of CPU work), and only then pulls the pinned `node:24-trixie` consumer image (8 layers, 440 MB compressed) and downloads the Grok binary (65 MB gzip, 166 MB verified). Both network waits sit after the staging work on the critical path. > - Neither download depends on the packed packages. The image is pinned by digest and the Grok binary is checked against its pinned sha256 by the provisioning script, so starting them early changes no verified property. > - This pull request starts both downloads as background children before staging and joins them where they are needed: the image before the first sandbox, the binary after the negative probe. It also prints one timing line per phase so the next slow part is visible in the CI log. > - The benefit is that the whole pull and the whole Grok download hide behind the staging phase. The verifier should shed the full pull time on a fresh runner, and the next cycle can read the phase split instead of inferring it from log gaps. ## Linked Issues or Issue Description **Describe the enhancement** The `Build` PR check is now the slowest check in the run. Its last step, the public npm install verifier, pulls the consumer Docker image and downloads the Grok binary only after it has finished staging and packing the eighteen public packages. Both downloads could run while the packages are staged. **Why is this needed** The slowest check decides the wall-clock time of every PR run. In the most recent fully green run (38107578322), `Build` took 634s: `pnpm build` 195s and the verifier 320s. The verifier's log shows the staging phase ends at +100s and the Grok result arrives at +267s. The image pull and the Grok download are inside that second window, in series with the consumer install and the probes. **Proposed solution** Start `docker pull --platform linux/amd64 ` and `provision-grok.mjs` as background children at the top of the verifier, before the staging loop. Await the image right before the first sandbox call. Provision Grok into a prefetch directory and rename it into `native/grok` only after the "npm must not provision Grok" assertion and the negative probe, so that assertion keeps its meaning and only the positive probe sees the binary. Print a `paperclip.public-npm-install.timing.v1` line with wall time per phase and the time the main flow waited on each prefetch. **Alternatives considered** - Run the verifier as its own job in parallel with `Build`. It needs the full `pnpm build` output, so that job would be as long as `Build` is now. - Warm the consumer npm cache from an Actions cache. That changes the "clean public npm install" property the verifier reports. Not done here; the new timing line will show whether the consumer install is worth that discussion. - Pack the eighteen packages concurrently. The three bundled packages run their own npm installs in series inside `prepareBundledPackage`, which is synchronous. A larger restructure; left for a later cycle with timing data. - Overlap the Pi runtime setup with the Grok probes. Both use the same consumer directory from separate containers, and the probes after provisioning are short. Not worth the concurrency on shared state. ## What Changed - `scripts/prefetch-child.mjs` (new): `startPrefetch({ name, cmd, args, cwd, env, logPath })` spawns a background child with no inherited terminal and its stdout and stderr in a log file, and returns `{ name, child, done, durationMs }`. `done` resolves on exit 0 and otherwise rejects with the tail of the log; the rejection is observed inside the helper, so an unawaited failure never becomes an unhandled rejection. `durationMs` ends at the child's last write to its log (the exit event is observed late when the caller is blocked in synchronous work; both downloads print their final status line as they complete). `stopPrefetches(prefetches, { graceMs })` sends `SIGTERM` to every child still running, escalates to `SIGKILL` after the grace period, and resolves only once all of them have exited. - `scripts/verify-grok-npm-install.mjs`: the image pull and the Grok provisioning start through the helper inside the `try` block, before the staging loop. The main flow awaits the image before the first `npm install` sandbox and awaits Grok after the negative probe, then renames the staged binary into `native/grok` before the positive probe. The `finally` block awaits `stopPrefetches` and only then removes the scratch root, so cleanup never races a child's writes. The script prints one timing JSON line after the Pi probe, with the per-phase wall times, the two `*_wait_ms` values, and the corrected `*_prefetch_ms` values. - `scripts/__tests__/prefetch-child.test.mjs` (new): seven tests run small local Node children, with no Docker or network: a successful prefetch resolves and keeps its output; the duration ends at the download even when the parent is blocked for 1.5s; a nonzero exit rejects with the log tail; a command that cannot start rejects instead of hanging the join or the stop; stopping a running child waits for its exit; a child that ignores `SIGTERM` is killed after the grace period; an unawaited failure does not produce an unhandled rejection in a fresh process. - `scripts/__tests__/grok-public-install-sandbox.test.mjs`: the source test asserts that exactly the pinned image is pulled for `linux/amd64`, that Grok is provisioned into the prefetch directory and never straight into `native/grok`, that both prefetches start before staging, that the image is joined before the first sandbox, that the rename happens after the npm assertion and the negative probe and before the positive probe, that the verifier imports the helper, and that `stopPrefetches` is awaited before the single `rmSync(root, ...)`. - `package.json`: `test:release-registry` runs the new test file. ## Verification - `node --test scripts/__tests__/prefetch-child.test.mjs scripts/__tests__/grok-public-install-sandbox.test.mjs scripts/__tests__/cursor-public-install-sandbox.test.mjs scripts/__tests__/retain-runner-qualification-packages.test.mjs`: 22 pass (7 new in `prefetch-child.test.mjs`, 12 in the sandbox file). The new file passed on three consecutive runs. - `node --check scripts/verify-grok-npm-install.mjs` passes. - `node packages/paperclip-runner/scripts/provision-grok.mjs /prefetch/grok` on this host: `Verified Grok 1.0.13 (linux-x64)` in 1.0s, `native/grok` absent before the move, 166079904 bytes in place after it. - Docker is not installed on the host that produced this PR, so the verifier was not run end to end locally. This PR's own `Build` job runs `node scripts/verify-grok-npm-install.mjs` from the branch checkout, so it exercises the change in full. **Measured on this PR's first CI run** ([38108964834](https://github.com/paperclipai/paperclip/actions/runs/38108964834), created 03:37Z), against two `master` runs created in the same twelve minutes: | | 38108878166 (`master`, 03:35Z) | 38108995800 (`master`, 03:37Z) | 38108964834 (this PR, 03:37Z) | | --- | --- | --- | --- | | Verifier step | 307s | 248s | 219s | | `Build` job | 555s | 484s | 459s | | Slowest check in the run | `Build` 555s | server (8/12) 535s | server (12/12) 547s | The PR run's timing line: `stage_packages_ms 84464`, `image_wait_ms 1`, `consumer_install_ms 97305`, `lifecycle_rebuild_ms 3545`, `codex_probe_ms 1211`, `grok_wait_ms 0`, `grok_probes_ms 626`, `pi_setup_ms 22194`, `pi_probe_ms 7753`. Both waits are at most 1ms, so the staging phase hid both downloads in full. The window from the end of staging to the Grok result went from 161s to 167s on the two runs that motivated this PR to 102s here, so the hidden downloads were about 60s on a fresh `runs-on` spot instance. Against the two same-window `master` runs, the verifier step is 29s to 88s shorter and `Build` is 25s to 96s shorter; the spread is runner variance in `pnpm build` and the consumer install. The `image_prefetch_ms` and `grok_prefetch_ms` values on that first run (84550 and 186696) were the parent's late observation of each exit, not download times; that is the first Greptile finding, fixed by the second commit, which measures from the download's last output instead. ## Risks - Low risk. The verifier proves the same properties in the same order. The only ordering change is that the downloads start earlier; every assertion, probe, and sandbox argument is unchanged. - The Grok binary is written under the scratch root in a `prefetch` directory during staging. That directory is never mounted into a container, and the `native/grok` assertion still runs against an absent path. - A prefetch that fails early now fails the step at its join point, where the old code would have failed at the same download a few seconds later. Same outcome, same message source. - `*_prefetch_ms` is a last-output time. A child that printed early and then kept working silently would be under-reported; neither `docker pull` nor `provision-grok.mjs` behaves that way (both print their final status as they finish), and a silent child falls back to the observed exit time. - Cleanup now waits up to 5s for a child that ignores `SIGTERM` before killing it, so a failing run can end up to 5s later than before. Both children exit promptly on `SIGTERM` in practice. - The timing line is one more stdout line. Nothing in the repository parses the verifier's stdout; the two workflows that run it write the output to the job log or to a file. ## Model Used - Claude (Anthropic), model ID `claude-fable-5-1`, running as a Paperclip agent through the Claude local adapter with tool use (shell, file edits, GitHub CLI). The agent collected the CI data, made the change, ran the tests, and wrote this description. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [ ] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com> Co-authored-by: Claude Fable 5.1 --- package.json | 2 +- .../grok-public-install-sandbox.test.mjs | 29 ++++++ scripts/__tests__/prefetch-child.test.mjs | 93 +++++++++++++++++++ scripts/prefetch-child.mjs | 60 ++++++++++++ scripts/verify-grok-npm-install.mjs | 53 ++++++++++- 5 files changed, 232 insertions(+), 5 deletions(-) create mode 100644 scripts/__tests__/prefetch-child.test.mjs create mode 100644 scripts/prefetch-child.mjs diff --git a/package.json b/package.json index 3e026aa985a..5a5000234b9 100644 --- a/package.json +++ b/package.json @@ -61,7 +61,7 @@ "smoke:posthog-live": "node scripts/smoke/posthog-live.mjs", "smoke:pipelines-tutorial": "./scripts/smoke/pipelines-tutorial-smoke.sh", "smoke:terminal-bench-loop-skill": "node scripts/smoke/terminal-bench-loop-skill-smoke.mjs", - "test:release-registry": "node --test scripts/__tests__/canary-startup-playwright.test.mjs scripts/__tests__/cursor-public-install-sandbox.test.mjs scripts/__tests__/grok-public-install-sandbox.test.mjs scripts/__tests__/retain-runner-qualification-packages.test.mjs scripts/verify-release-registry-state.test.mjs scripts/release-package-map.test.mjs scripts/check-release-package-bootstrap.test.mjs scripts/check-no-git-push.test.mjs scripts/release-lib.test.mjs scripts/release-registry-versions.test.mjs scripts/link-plugin-dev-sdk.test.js scripts/acpx-patch-packaging.test.mjs scripts/service-onboard-smoke.test.mjs scripts/docker-onboard-smoke.test.mjs scripts/preview-artifacts.test.mjs scripts/cloud-migrator-artifacts.test.mjs", + "test:release-registry": "node --test scripts/__tests__/canary-startup-playwright.test.mjs scripts/__tests__/cursor-public-install-sandbox.test.mjs scripts/__tests__/grok-public-install-sandbox.test.mjs scripts/__tests__/prefetch-child.test.mjs scripts/__tests__/retain-runner-qualification-packages.test.mjs scripts/verify-release-registry-state.test.mjs scripts/release-package-map.test.mjs scripts/check-release-package-bootstrap.test.mjs scripts/check-no-git-push.test.mjs scripts/release-lib.test.mjs scripts/release-registry-versions.test.mjs scripts/link-plugin-dev-sdk.test.js scripts/acpx-patch-packaging.test.mjs scripts/service-onboard-smoke.test.mjs scripts/docker-onboard-smoke.test.mjs scripts/preview-artifacts.test.mjs scripts/cloud-migrator-artifacts.test.mjs", "storybook-visual:baseline": "node scripts/storybook-visual-baseline.mjs", "test:storybook-visual": "node scripts/storybook-visual-baseline.mjs download && node scripts/storybook-visual-baseline.mjs verify && pnpm build-storybook && npx playwright test --config tests/storybook-visual/playwright.config.ts", "test:storybook-visual:update": "node scripts/storybook-visual-baseline.mjs download && pnpm build-storybook && npx playwright test --config tests/storybook-visual/playwright.config.ts --update-snapshots && node scripts/storybook-visual-baseline.mjs pack", diff --git a/scripts/__tests__/grok-public-install-sandbox.test.mjs b/scripts/__tests__/grok-public-install-sandbox.test.mjs index d084ee9be81..5a3dc487e38 100644 --- a/scripts/__tests__/grok-public-install-sandbox.test.mjs +++ b/scripts/__tests__/grok-public-install-sandbox.test.mjs @@ -81,6 +81,35 @@ test('verification never elevates PR-controlled provisioning or cleanup on the h assert.ok(source.includes("const prerequisite = join(root, 'native/grok')")); }); +test('the image and Grok downloads start before staging and join without weakening the npm checks', () => { + const source = readFileSync(new URL('../verify-grok-npm-install.mjs', import.meta.url), 'utf8'); + const position = fragment => { const index = source.indexOf(fragment); assert.notEqual(index, -1, `verifier must contain ${fragment}`); return index; }; + // Exactly the pinned consumer image is pulled, for the platform the probes run on. + const imagePull = position("prefetch('image', 'docker', ['pull', '--platform', 'linux/amd64', GROK_PUBLIC_INSTALL_IMAGE])"); + // Grok is provisioned into the prefetch directory, never straight into `native/grok`. + const grokProvision = position("prefetch('grok', process.execPath, [join(repo, 'packages/paperclip-runner/scripts/provision-grok.mjs'), stagedGrok])"); + assert.ok(source.includes("const stagedGrok = join(prefetched, 'grok')")); + assert.doesNotMatch(source, /provision-grok\.mjs'\), prerequisite\]/); + // Both downloads start before any package is staged and before the first sandbox runs. + const staging = position("visit('@paperclipai/server')"); + const firstSandbox = position("isolated(['npm', 'install', '--ignore-scripts', '--omit=dev'"); + assert.ok(imagePull < staging && grokProvision < staging); + assert.ok(position("await awaited('image', imageReady)") < firstSandbox); + // The binary moves to the canonical host path only after npm is proven not to + // have provisioned it, after the negative probe, and before the positive probe. + const npmCheck = position("'npm must not provision Grok'"); + const negativeProbe = position("isolated(['node', '/packages/probe.mjs', 'missing'])"); + const grokJoin = position("await awaited('grok', grokReady)"); + const move = position('renameSync(stagedGrok, prerequisite)'); + const positiveProbe = position("isolated(['node', '/packages/probe.mjs', 'present'], { prerequisite })"); + assert.ok(npmCheck < negativeProbe && negativeProbe < grokJoin && grokJoin < move && move < positiveProbe); + // The children run through the helper covered by prefetch-child.test.mjs, and + // cleanup waits for them before removing the directory they write into. + assert.ok(source.includes("import { startPrefetch, stopPrefetches } from './prefetch-child.mjs'")); + assert.ok(position('await stopPrefetches(prefetches)') < position('rmSync(root, { recursive: true, force: true })')); + assert.equal(source.match(/rmSync\(root/g).length, 1); +}); + test('the installed Codex probe exercises the public export and rejects incomplete or mismatched packages', () => { const root = realpathSync(mkdtempSync(join(tmpdir(), 'installed-codex-probe-'))); try { diff --git a/scripts/__tests__/prefetch-child.test.mjs b/scripts/__tests__/prefetch-child.test.mjs new file mode 100644 index 00000000000..bd3d9864bef --- /dev/null +++ b/scripts/__tests__/prefetch-child.test.mjs @@ -0,0 +1,93 @@ +import assert from 'node:assert/strict'; +import { spawnSync } from 'node:child_process'; +import { mkdtempSync, readFileSync, rmSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import test from 'node:test'; +import { startPrefetch, stopPrefetches } from '../prefetch-child.mjs'; + +// Small local Node children stand in for the image pull and the Grok download; +// none of these tests needs Docker or a network. +const withScratch = async body => { + const root = mkdtempSync(join(tmpdir(), 'prefetch-child-')); + try { await body(root); } finally { rmSync(root, { recursive: true, force: true }); } +}; +const start = (root, name, script, cmd = process.execPath, args = ['-e', script]) => + startPrefetch({ name, cmd, args, cwd: root, env: process.env, logPath: join(root, `${name}.log`) }); +const blockFor = ms => Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, ms); +const untilLogHas = async (path, text) => { + for (let attempt = 0; attempt < 400; attempt++) { + try { if (readFileSync(path, 'utf8').includes(text)) return; } catch {} + await new Promise(resolve => setTimeout(resolve, 25)); + } + throw new Error(`log never contained ${text}`); +}; + +test('a successful prefetch resolves, keeps its output, and is left alone by stop', async () => withScratch(async root => { + const handle = start(root, 'ok', "setTimeout(() => process.stdout.write('fetched\\n'), 150)"); + await handle.done; + assert.equal(handle.child.exitCode, 0); + assert.equal(readFileSync(join(root, 'ok.log'), 'utf8'), 'fetched\n'); + assert.ok(handle.durationMs >= 100 && handle.durationMs < 5000, `durationMs ${handle.durationMs}`); + await stopPrefetches([handle]); + assert.equal(handle.child.signalCode, null); +})); + +test('the duration ends at the download, not when a blocked parent notices the exit', async () => withScratch(async root => { + const handle = start(root, 'early', "setTimeout(() => process.stdout.write('Status: Downloaded\\n'), 100)"); + // Synchronous work in the parent, as package staging is in the verifier. + blockFor(1500); + const noticed = Date.now(); + await handle.done; + assert.ok(Date.now() - noticed < 500, 'the exit was pending by the time the parent yielded'); + assert.ok(handle.durationMs < 900, `durationMs ${handle.durationMs} must not include the 1500ms the parent was blocked`); +})); + +test('a nonzero exit rejects with the log tail where the result is awaited', async () => withScratch(async root => { + const handle = start(root, 'bad', "process.stderr.write('registry unreachable\\n'); process.exit(3)"); + await assert.rejects(handle.done, /^Error: bad prefetch failed \(exit 3\): registry unreachable$/); + await stopPrefetches([handle]); +})); + +test('a command that cannot start rejects instead of hanging the join or the stop', async () => withScratch(async root => { + const handle = start(root, 'missing', '', join(root, 'no-such-command'), []); + await assert.rejects(handle.done, /missing prefetch failed \(.*ENOENT/); + await stopPrefetches([handle]); +})); + +test('stopping a running prefetch waits for it to exit so its directory can be removed safely', async () => withScratch(async root => { + const handle = start(root, 'slow', "process.stdout.write('started\\n'); setInterval(() => {}, 1000)"); + await untilLogHas(join(root, 'slow.log'), 'started'); + const stopping = Date.now(); + await stopPrefetches([handle]); + assert.ok(Date.now() - stopping < 4000); + assert.equal(handle.child.signalCode, 'SIGTERM'); + assert.equal(handle.child.exitCode, null); + await assert.rejects(handle.done, /slow prefetch failed \(SIGTERM\): started/); +})); + +test('a prefetch that ignores SIGTERM is killed after the grace period', async () => withScratch(async root => { + const handle = start(root, 'stubborn', "process.on('SIGTERM', () => {}); process.stdout.write('ignoring\\n'); setInterval(() => {}, 1000)"); + await untilLogHas(join(root, 'stubborn.log'), 'ignoring'); + const stopping = Date.now(); + await stopPrefetches([handle], { graceMs: 200 }); + assert.ok(Date.now() - stopping >= 200 && Date.now() - stopping < 4000); + assert.equal(handle.child.signalCode, 'SIGKILL'); +})); + +test('an unawaited failure never becomes an unhandled rejection', () => { + const script = ` + import { startPrefetch } from ${JSON.stringify(new URL('../prefetch-child.mjs', import.meta.url).href)}; + import { mkdtempSync, rmSync } from 'node:fs'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; + const root = mkdtempSync(join(tmpdir(), 'prefetch-unawaited-')); + try { + startPrefetch({ name: 'ignored', cmd: process.execPath, args: ['-e', 'process.exit(2)'], cwd: root, env: process.env, logPath: join(root, 'ignored.log') }); + await new Promise(resolve => setTimeout(resolve, 500)); + console.log('still running'); + } finally { rmSync(root, { recursive: true, force: true }); } + `; + const result = spawnSync(process.execPath, ['--input-type=module', '-e', script], { encoding: 'utf8', timeout: 10_000 }); + assert.equal(result.status, 0, result.stderr); + assert.match(result.stdout, /still running/); + assert.doesNotMatch(result.stderr, /unhandled/i); +}); diff --git a/scripts/prefetch-child.mjs b/scripts/prefetch-child.mjs new file mode 100644 index 00000000000..bdaf3d25a84 --- /dev/null +++ b/scripts/prefetch-child.mjs @@ -0,0 +1,60 @@ +// Background downloads for verification scripts: start a child early with its +// output in a log file, join it where the result is needed, and stop it before +// the scratch directory it writes into is removed. No shell, no inherited +// terminal, and a failure is reported where the caller awaits it. +import { spawn } from 'node:child_process'; +import { closeSync, openSync, readFileSync, statSync } from 'node:fs'; + +const running = child => child.pid !== undefined && child.exitCode === null && child.signalCode === null; + +// Returns { name, child, done, durationMs }. `done` resolves on exit code 0 +// and otherwise rejects with the tail of the log; the rejection is observed +// here so an unawaited failure never surfaces as an unhandled rejection. +export function startPrefetch({ name, cmd, args, cwd, env, logPath }) { + const startedAt = Date.now(); + const log = openSync(logPath, 'w'); + let child; + try { + child = spawn(cmd, args, { cwd, env, stdio: ['ignore', log, log] }); + } finally { + closeSync(log); // the child holds its own descriptor + } + const handle = { name, child, done: null, durationMs: null }; + let settled = false; + handle.done = new Promise((settle, reject) => { + const finish = failure => { + if (settled) return; + settled = true; + // The caller may be blocked in synchronous work when the child exits, so + // this event can arrive long after the fact. When the child wrote + // anything, its last write is the end time; both downloads this serves + // print their final status line as they complete. A silent child falls + // back to the time the exit was observed. + let finishedAt = Date.now(); + try { + const { mtimeMs, size } = statSync(logPath); + if (size > 0) finishedAt = Math.min(finishedAt, mtimeMs); + } catch {} + handle.durationMs = Math.max(0, Math.round(finishedAt - startedAt)); + if (failure === null) return settle(); + let output = ''; + try { output = readFileSync(logPath, 'utf8').trim().slice(-2000); } catch {} + reject(new Error(`${name} prefetch failed (${failure}): ${output}`)); + }; + child.once('error', error => finish(error.message)); + child.once('exit', (code, signal) => finish(code === 0 ? null : (signal ?? `exit ${code}`))); + }); + handle.done.catch(() => {}); + return handle; +} + +// Stops every prefetch that is still running and resolves only once all of +// them have exited, so the caller can remove their working directory without +// racing a write. A child that ignores SIGTERM is killed after `graceMs`. +export async function stopPrefetches(prefetches, { graceMs = 5000 } = {}) { + await Promise.all(prefetches.map(async ({ child, done }) => { + if (running(child)) child.kill('SIGTERM'); + const escalate = setTimeout(() => { if (running(child)) child.kill('SIGKILL'); }, graceMs); + try { await done; } catch {} finally { clearTimeout(escalate); } + })); +} diff --git a/scripts/verify-grok-npm-install.mjs b/scripts/verify-grok-npm-install.mjs index b3c2db21d23..d35c40e8183 100644 --- a/scripts/verify-grok-npm-install.mjs +++ b/scripts/verify-grok-npm-install.mjs @@ -4,12 +4,13 @@ // explicit and separate from npm installation, and is removed in finally. import assert from 'node:assert/strict'; import { execFileSync } from 'node:child_process'; -import { chmodSync, cpSync, existsSync, mkdirSync, mkdtempSync, readFileSync, readdirSync, rmSync, writeFileSync } from 'node:fs'; +import { chmodSync, cpSync, existsSync, mkdirSync, mkdtempSync, readFileSync, readdirSync, renameSync, rmSync, writeFileSync } from 'node:fs'; import { tmpdir } from 'node:os'; import { basename, dirname, join, resolve } from 'node:path'; import { fileURLToPath } from 'node:url'; import { materializePublishManifest, prepareBundledPackage } from './prepare-bundled-package.mjs'; import { GROK_PUBLIC_INSTALL_IMAGE, GROK_PUBLIC_INSTALL_LIFECYCLE, assertNoBundledCodexPayloads, grokConsumerDockerArgs, installedCodexProbeSource } from './grok-public-install-sandbox.mjs'; +import { startPrefetch, stopPrefetches } from './prefetch-child.mjs'; import { retainRunnerQualificationPackages } from './retain-runner-qualification-packages.mjs'; const repo = resolve(dirname(fileURLToPath(import.meta.url)), '..'); assert.equal(process.platform, 'linux', 'Run this verification on disposable EC2 Linux, not a developer host'); @@ -19,7 +20,31 @@ const env = { ...process.env, NODE_PATH: '', PAPERCLIP_RELEASE_REUSE_UI_DIST: '1 const run = (cmd, args, cwd = root) => execFileSync(cmd, args, { cwd, env, stdio: 'pipe', maxBuffer: 32 * 1024 * 1024 }); const sourceRevision = run('git', ['rev-parse', 'HEAD'], repo).toString().trim(); const releaseVersion = `0.0.0-grok-verify.${sourceRevision.slice(0, 12)}`; +// The two network downloads the probes need do not depend on anything packed +// below, so they overlap the CPU-bound staging instead of running after it. +// The consumer image is pinned by digest and the Grok binary is checked against +// its pinned sha256 by the provisioning script, so an early start changes no +// verified property. Neither touches the consumer, the assets, nor the +// `native/grok` path that the post-install assertion inspects. +const prefetched = join(root, 'prefetch'); mkdirSync(prefetched); +const stagedGrok = join(prefetched, 'grok'); +const prefetches = []; +const timing = {}; +// A prefetch failure surfaces where its result is awaited, never as an +// unhandled rejection that would mask an earlier staging error. +const prefetch = (name, cmd, args) => { + const handle = startPrefetch({ name, cmd, args, cwd: root, env, logPath: join(prefetched, `${name}.log`) }); + prefetches.push(handle); + return handle.done; +}; +let phaseStarted = performance.now(); +const phase = name => { const now = performance.now(); timing[`${name}_ms`] = Math.round(now - phaseStarted); phaseStarted = now; }; +const awaited = async (name, promise) => { const started = performance.now(); await promise; timing[`${name}_wait_ms`] = Math.round(performance.now() - started); }; try { + const imageReady = prefetch('image', 'docker', ['pull', '--platform', 'linux/amd64', GROK_PUBLIC_INSTALL_IMAGE]); + // Provision as the unprivileged verification user, never into the host's /opt. + // Only the positive probe sees this file at the canonical sandbox path. + const grokReady = prefetch('grok', process.execPath, [join(repo, 'packages/paperclip-runner/scripts/provision-grok.mjs'), stagedGrok]); const listing = run(process.execPath, [join(repo, 'scripts/release-package-map.mjs'), 'list'], repo).toString().trim().split('\n').map(line => line.split('\t')); const packages = new Map(listing.map(([dir, name]) => [name, { dir, manifest: JSON.parse(readFileSync(join(repo, dir, 'package.json'), 'utf8')) }])); const needed = new Set(); @@ -81,9 +106,12 @@ try { const sentinel = join(consumer, 'node_modules/paperclip-verification-lifecycle-sentinel/lifecycle-ran'); const consumerUid = process.getuid(); const isolated = (command, options = {}) => run('docker', grokConsumerDockerArgs({ assets, consumer, cache, uid: consumerUid, gid: process.getgid(), command, ...options })); + phase('stage_packages'); + await awaited('image', imageReady); // npm resolution is intentionally the public consumer graph, not the pnpm // workspace graph. Freeze that result before any lifecycle script can run. isolated(['npm', 'install', '--ignore-scripts', '--omit=dev', ...readdirSync(assets).filter(file => file.endsWith('.tgz')).map(file => `/packages/${file}`)], { download: true }); + phase('consumer_install'); assert.equal(existsSync(sentinel), false, 'Dependency download must not run lifecycle scripts'); const consumerLock = readFileSync(join(consumer, 'package-lock.json'), 'utf8'); // npm ci rejects bundled optional platform dependencies absent from its own @@ -92,6 +120,7 @@ try { isolated(GROK_PUBLIC_INSTALL_LIFECYCLE); assert.equal(readFileSync(sentinel, 'utf8'), 'ok', 'Offline lifecycle scripts must actually execute'); assert.equal(readFileSync(join(consumer, 'package-lock.json'), 'utf8'), consumerLock, 'Lifecycle execution must preserve the resolved consumer lock'); + phase('lifecycle_rebuild'); for (const name of needed) { const installedManifest = JSON.parse(readFileSync(join(consumer, 'node_modules', name, 'package.json'), 'utf8')); assert.equal(installedManifest.version, releaseVersion, `Installed release version for ${name}`); @@ -104,6 +133,7 @@ try { writeFileSync(join(assets, 'codex-probe.mjs'), installedCodexProbeSource( '/consumer/node_modules/@paperclipai/server/dist/vendor/paperclip-runner/index.js', '/consumer', codexVersion), { mode: 0o644 }); const codex = JSON.parse(isolated(['node', '/packages/codex-probe.mjs']).toString().trim()); + phase('codex_probe'); assert.equal(codex.codexConsumerHostOnly, true); assert.equal(codex.codexPlatformPackageSource, 'official npm'); assert.ok(existsSync(join(installed, 'providers/grok/launcher.cjs'))); @@ -122,10 +152,14 @@ try { `; writeFileSync(join(assets, 'probe.mjs'), probe, { mode: 0o644 }); isolated(['node', '/packages/probe.mjs', 'missing']); - // Provision as the unprivileged verification user, never into the host's /opt. - // Only the positive probe sees this file at the canonical sandbox path. - run(process.execPath, [join(repo, 'packages/paperclip-runner/scripts/provision-grok.mjs'), prerequisite]); + // The binary was provisioned outside `native/grok` while the packages were + // staged, so the assertion above still proves npm put nothing there. It moves + // into place only now, so only the positive probe sees it. + await awaited('grok', grokReady); + mkdirSync(dirname(prerequisite), { recursive: true }); + renameSync(stagedGrok, prerequisite); isolated(['node', '/packages/probe.mjs', 'present'], { prerequisite }); + phase('grok_probes'); console.log(JSON.stringify({ schema: 'paperclip.grok.public-npm-install.v1', sourceRevision, releaseVersion, lifecycleScriptsEnabled: true, lifecycleSentinelVerified: true, lifecycleNetwork: 'none', consumerImage: GROK_PUBLIC_INSTALL_IMAGE, consumerUid, consumerLockPreserved: true, cleanNpmInstall: true, packageCount: needed.size, codexPackedPayloadsOmitted: true, codexPackedPackagesChecked: tarballs.length, ...codex, builtinLauncherPresent: true, separateGrokPackage: false, npmProvisionedBinary: false, missingPrerequisiteRejected: true, provisionedBinaryVerified: true, commandLeaseVerified: true, providerCalls: 0 })); // Exercise Pi's public CLI and installed server, never a private workspace // package or binary override. Public dependency downloads are explicit and @@ -137,6 +171,7 @@ try { // temporary capacity than the smaller offline lifecycle probes. const setup = isolated(['node', '/consumer/node_modules/paperclipai/dist/index.js', 'runtime', 'setup', 'pi'], { download: true, temporarySizeMiB: 2048 }).toString(); const receipt = JSON.parse(setup.trim()); + phase('pi_setup'); assert.equal(receipt.status, 'installed_verified'); assert.equal(receipt.target, 'linux-x64'); assert.equal(readFileSync(join(consumer, 'package-lock.json'), 'utf8'), consumerLock, 'Pi setup must preserve the consumer dependency graph'); @@ -144,6 +179,13 @@ try { // Docker defaults tmpfs to noexec. The offline admission probe must execute // its verified private snapshot while keeping lifecycle/download scratch noexec. console.log(isolated(['node', '/packages/pi-public-install-probe.mjs', '/consumer/node_modules/@paperclipai/server'], { temporarySizeMiB: 2048, temporaryExecutable: true }).toString().trim()); + phase('pi_probe'); + // Wall time per phase, so the slow part of this step is visible in the CI log + // without re-running it. `*_prefetch_ms` runs from a download's start to its + // last output, and `*_wait_ms` is how long the main flow waited for a + // prefetch that had not finished yet; zero means the overlap hid it entirely. + const downloads = Object.fromEntries(prefetches.map(({ name, durationMs }) => [`${name}_prefetch_ms`, durationMs])); + console.log(JSON.stringify({ schema: 'paperclip.public-npm-install.timing.v1', ...timing, ...downloads })); if (process.env.PAPERCLIP_RUNNER_QUALIFICATION_PACKAGES_DIR) { const retained = retainRunnerQualificationPackages({ repo, output: process.env.PAPERCLIP_RUNNER_QUALIFICATION_PACKAGES_DIR, sourceRevision, releaseVersion, env, publicArchives: [...needed].map((name, index) => ({ name, file: tarballs[index] })), @@ -151,5 +193,8 @@ try { console.log(JSON.stringify(retained)); } } finally { + // An earlier failure must not leave a prefetch running, and scratch is + // removed only once both children have stopped writing into it. + await stopPrefetches(prefetches); rmSync(root, { recursive: true, force: true }); } From b36960b75871e8d335d77115b1cb4f7549cd8f70 Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Sat, 10 Oct 2026 21:47:55 -0700 Subject: [PATCH 7/7] fix(server): permit verified question-delivery history to sleep (#15851) Treat completed question-response deliveries as history only after their exact same-company handoff finishes. Preserve missing evidence, retries, accounting debt, and idle-sleep fences. Add a concurrent all-state question history index with interrupted-build repair. Keep migrations serialized without waiting snapshots that deadlock concurrent index builds. Validated by 118 focused sleep tests, six migration/snapshot tests, full repository typecheck, all CI gates, and Apex 5/5. Co-Authored-By: Paperclip --- doc/DATABASE.md | 8 + doc/idle-sleep-safety.md | 10 + packages/db/src/client.ts | 20 +- ...rtbeat-run-issue-privacy-migration.test.ts | 3 +- .../migrations/0332_little_night_nurse.sql | 1 + ...{0321_snapshot.json => 0332_snapshot.json} | 2292 ++++++++++++++++- packages/db/src/migrations/meta/_journal.json | 7 + ...uestion-response-history-migration.test.ts | 53 + .../db/src/schema/agent_wakeup_requests.ts | 6 + .../src/__tests__/idle-sleep-safety.test.ts | 149 +- server/src/services/idle-sleep-safety.ts | 27 +- 11 files changed, 2558 insertions(+), 18 deletions(-) create mode 100644 packages/db/src/migrations/0332_little_night_nurse.sql rename packages/db/src/migrations/meta/{0321_snapshot.json => 0332_snapshot.json} (96%) create mode 100644 packages/db/src/question-response-history-migration.test.ts diff --git a/doc/DATABASE.md b/doc/DATABASE.md index c6e9c5474d3..2177689e1b9 100644 --- a/doc/DATABASE.md +++ b/doc/DATABASE.md @@ -433,6 +433,14 @@ run/turn ids, payload digest, attempt/acknowledgement state, and one of `steered `issue_thread_interactions.result`. Deleting the interaction cascades its receipt, while deleting a referenced run clears that run pointer without deleting history. +The question-history index covers `(company_id, idempotency_key)` for every +`question-response:` wake state, including failed, cancelled, and skipped +receipts excluded from the delivery-idempotency fence. Migration `0332` creates +this nonunique index concurrently through the standard migration executor. +Apply it with `pnpm db:migrate` on a direct connection before enabling the new +sleep guard. The executor retries interrupted invalid builds and records history +only after the index is valid. It does not delete delivery or wake records. + ## Plugin database namespaces The plugin runtime tracks plugin-owned database namespaces and migrations in `plugin_database_namespaces` and `plugin_migrations`. Hosted deployments that separate runtime and migration connections should set `DATABASE_MIGRATION_URL`; plugin namespace migration work uses the migration connection when present. diff --git a/doc/idle-sleep-safety.md b/doc/idle-sleep-safety.md index 30622581e24..b4bf960ce59 100644 --- a/doc/idle-sleep-safety.md +++ b/doc/idle-sleep-safety.md @@ -129,6 +129,16 @@ company/agent's linked run has finished without retry or accounting debt. Image-setup history still blocks: promotion can succeed before teardown is confirmed, so a terminal label does not prove that its provider resource is gone. +Completed question-answer deliveries permit sleep only after acknowledgement +and completion of their exact target run in the same company, without retry or +accounting debt. A fallback without a target run requires its exact question's +finished durable wake receipt in the same company. A coalesced fallback also +requires its exact company/agent's linked run to finish without retry or +accounting debt. The question-history index covers every wake status. +Pending deliveries, missing +completion evidence, and unfinished handoffs still block. Failed deliveries +without a pending target are history; other work checks still apply. + Every enabled plugin blocks sleep. A version label does not prove that arbitrary worker code has no background activity. No plugin approval or configuration bypass is supported. Scheduled and externally triggered work needs a durable diff --git a/packages/db/src/client.ts b/packages/db/src/client.ts index 3f6061deaae..199a28ebe43 100644 --- a/packages/db/src/client.ts +++ b/packages/db/src/client.ts @@ -467,12 +467,12 @@ async function recordMigrationHistoryEntry( ); } -// These idempotent privacy migrations commit keyset batches inside their DO -// blocks. Execute each statement at top level so DDL locks and completed batches -// are released before the next batch. History is recorded only after completion. -const MIGRATIONS_WITH_BATCH_COMMITS = new Set([ +// These idempotent migrations require top-level statements for batch commits +// or concurrent index builds. History is recorded only after completion. +const MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS = new Set([ "0313_private_task_access.sql", "0314_private_task_draft_assets.sql", + "0332_little_night_nurse.sql", ]); async function applyPendingMigrationsManually( @@ -493,7 +493,13 @@ async function applyPendingMigrationsManually( try { // A session lock survives each batch COMMIT. Reserve the connection so // pool rotation cannot release it while another migrator is waiting. - await sql`SELECT pg_advisory_lock(hashtextextended('paperclip:migrations', 0))`; + // A blocking lock SELECT retains its statement snapshot while waiting. + // A partial concurrent index build can wait for that snapshot, forming a + // deadlock with this lock. Each failed try ends its snapshot before sleep. + while (!(await sql<{ locked: boolean }[]>` + SELECT pg_try_advisory_lock(hashtextextended('paperclip:migrations', 0)) AS locked`)[0]?.locked) { + await new Promise((resolve) => setTimeout(resolve, 100)); + } const { migrationTableSchema, columnNames } = await ensureMigrationJournalTable(sql); const qualifiedTable = `${quoteIdentifier(migrationTableSchema)}.${quoteIdentifier(DRIZZLE_MIGRATIONS_TABLE)}`; @@ -511,7 +517,7 @@ async function applyPendingMigrationsManually( const applyMigration = async () => { for (const statement of splitMigrationStatements(migrationContent)) { - if (MIGRATIONS_WITH_BATCH_COMMITS.has(migrationFile)) { + if (MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS.has(migrationFile)) { // A cancelled concurrent build leaves an invalid index. IF NOT // EXISTS alone would skip it on retry and journal an incomplete index. const index = statement.replace(/^\s*--.*$/gm, "").trim() @@ -545,7 +551,7 @@ async function applyPendingMigrationsManually( folderMillisByFileName.get(migrationFile) ?? Date.now(), ); }; - if (MIGRATIONS_WITH_BATCH_COMMITS.has(migrationFile)) { + if (MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS.has(migrationFile)) { await applyMigration(); } else { await runInTransaction(sql, applyMigration); diff --git a/packages/db/src/heartbeat-run-issue-privacy-migration.test.ts b/packages/db/src/heartbeat-run-issue-privacy-migration.test.ts index bb01e215a22..6883c0c799f 100644 --- a/packages/db/src/heartbeat-run-issue-privacy-migration.test.ts +++ b/packages/db/src/heartbeat-run-issue-privacy-migration.test.ts @@ -229,7 +229,8 @@ describeEmbeddedPostgres("heartbeat run issue privacy migration", () => { contender = applyPendingMigrations(database.connectionString).then(() => null, error => error); await expect.poll(async () => { const [row] = await sql`SELECT count(*)::int AS n FROM pg_stat_activity - WHERE datname = current_database() AND wait_event = 'advisory' + WHERE datname = current_database() AND query LIKE '%SELECT pg_try_advisory_lock%' + AND query NOT LIKE '%FROM pg_stat_activity%' AND strpos(query, 'paperclip:migrations') > 0`; return row.n; }, { timeout: 10_000 }).toBe(1); diff --git a/packages/db/src/migrations/0332_little_night_nurse.sql b/packages/db/src/migrations/0332_little_night_nurse.sql new file mode 100644 index 00000000000..dd23f331cbb --- /dev/null +++ b/packages/db/src/migrations/0332_little_night_nurse.sql @@ -0,0 +1 @@ +CREATE INDEX CONCURRENTLY IF NOT EXISTS "agent_wakeup_requests_question_response_history_idx" ON "agent_wakeup_requests" USING btree ("company_id","idempotency_key") WHERE "agent_wakeup_requests"."idempotency_key" LIKE 'question-response:%'; \ No newline at end of file diff --git a/packages/db/src/migrations/meta/0321_snapshot.json b/packages/db/src/migrations/meta/0332_snapshot.json similarity index 96% rename from packages/db/src/migrations/meta/0321_snapshot.json rename to packages/db/src/migrations/meta/0332_snapshot.json index f6a27b20eda..80f174738da 100644 --- a/packages/db/src/migrations/meta/0321_snapshot.json +++ b/packages/db/src/migrations/meta/0332_snapshot.json @@ -1,6 +1,6 @@ { - "id": "3a76c99a-51c4-41d7-b7e2-12b55d8c7d08", - "prevId": "d6277736-9c05-4304-8d16-52dc0d2fbaef", + "id": "8dadd21e-8c02-4ada-b99a-55ba133e7f70", + "prevId": "5c09a14d-e323-49b1-b855-dcd12b974bcc", "version": "7", "dialect": "postgresql", "tables": { @@ -416,6 +416,12 @@ "primaryKey": false, "notNull": false }, + "fast_response_request_id": { + "name": "fast_response_request_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, "agent_id": { "name": "agent_id", "type": "uuid", @@ -462,6 +468,27 @@ } }, "indexes": { + "budget_reservations_fast_response_idx": { + "name": "budget_reservations_fast_response_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "fast_response_request_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + }, "budget_reservations_decision_idx": { "name": "budget_reservations_decision_idx", "columns": [ @@ -580,7 +607,7 @@ "checkConstraints": { "budget_reservations_source_check": { "name": "budget_reservations_source_check", - "value": "(\"budget_reservations\".\"run_id\" is not null)::int + (\"budget_reservations\".\"decision_invocation_id\" is not null)::int = 1" + "value": "(\"budget_reservations\".\"run_id\" is not null)::int + (\"budget_reservations\".\"decision_invocation_id\" is not null)::int + (\"budget_reservations\".\"fast_response_request_id\" is not null)::int = 1" }, "budget_reservations_state_check": { "name": "budget_reservations_state_check", @@ -3737,6 +3764,28 @@ "method": "btree", "with": {} }, + "agent_wakeup_requests_question_response_history_idx": { + "name": "agent_wakeup_requests_question_response_history_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "idempotency_key", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"agent_wakeup_requests\".\"idempotency_key\" LIKE 'question-response:%'", + "concurrently": true, + "method": "btree", + "with": {} + }, "agent_wakeup_requests_connection_intent_delivery_idempotency_uq": { "name": "agent_wakeup_requests_connection_intent_delivery_idempotency_uq", "columns": [ @@ -4810,6 +4859,516 @@ }, "isRLSEnabled": false }, + "public.ai_subscription_connections": { + "name": "ai_subscription_connections", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "uuid", + "primaryKey": true, + "notNull": true, + "default": "gen_random_uuid()" + }, + "company_id": { + "name": "company_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "connection_id": { + "name": "connection_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "grant_id": { + "name": "grant_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "credential_key": { + "name": "credential_key", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "subscription_id": { + "name": "subscription_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + } + }, + "indexes": { + "ai_subscription_connections_grant_idx": { + "name": "ai_subscription_connections_grant_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "grant_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + }, + "ai_subscription_connections_credential_idx": { + "name": "ai_subscription_connections_credential_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "credential_key", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "ai_subscription_connections_account_idx": { + "name": "ai_subscription_connections_account_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "subscription_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "ai_subscription_connections_company_id_companies_id_fk": { + "name": "ai_subscription_connections_company_id_companies_id_fk", + "tableFrom": "ai_subscription_connections", + "tableTo": "companies", + "columnsFrom": [ + "company_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "ai_subscription_connections_connection_id_tool_connections_id_fk": { + "name": "ai_subscription_connections_connection_id_tool_connections_id_fk", + "tableFrom": "ai_subscription_connections", + "tableTo": "tool_connections", + "columnsFrom": [ + "connection_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "ai_subscription_connections_grant_id_connection_grants_id_fk": { + "name": "ai_subscription_connections_grant_id_connection_grants_id_fk", + "tableFrom": "ai_subscription_connections", + "tableTo": "connection_grants", + "columnsFrom": [ + "grant_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "ai_subscription_connections_subscription_id_ai_subscriptions_id_fk": { + "name": "ai_subscription_connections_subscription_id_ai_subscriptions_id_fk", + "tableFrom": "ai_subscription_connections", + "tableTo": "ai_subscriptions", + "columnsFrom": [ + "subscription_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.ai_subscription_prices": { + "name": "ai_subscription_prices", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "uuid", + "primaryKey": true, + "notNull": true, + "default": "gen_random_uuid()" + }, + "company_id": { + "name": "company_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "subscription_id": { + "name": "subscription_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "revision": { + "name": "revision", + "type": "integer", + "primaryKey": false, + "notNull": true + }, + "plan": { + "name": "plan", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "amount_cents": { + "name": "amount_cents", + "type": "numeric(24, 7)", + "primaryKey": false, + "notNull": false + }, + "currency": { + "name": "currency", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'USD'" + }, + "cadence": { + "name": "cadence", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'month'" + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'active'" + }, + "source": { + "name": "source", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "source_url": { + "name": "source_url", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "effective_at": { + "name": "effective_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "ai_subscription_prices_revision_idx": { + "name": "ai_subscription_prices_revision_idx", + "columns": [ + { + "expression": "subscription_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "revision", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + }, + "ai_subscription_prices_company_idx": { + "name": "ai_subscription_prices_company_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "ai_subscription_prices_company_id_companies_id_fk": { + "name": "ai_subscription_prices_company_id_companies_id_fk", + "tableFrom": "ai_subscription_prices", + "tableTo": "companies", + "columnsFrom": [ + "company_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "ai_subscription_prices_subscription_id_ai_subscriptions_id_fk": { + "name": "ai_subscription_prices_subscription_id_ai_subscriptions_id_fk", + "tableFrom": "ai_subscription_prices", + "tableTo": "ai_subscriptions", + "columnsFrom": [ + "subscription_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": { + "ai_subscription_prices_valid": { + "name": "ai_subscription_prices_valid", + "value": "(\"ai_subscription_prices\".\"amount_cents\" is null or \"ai_subscription_prices\".\"amount_cents\" >= 0) and \"ai_subscription_prices\".\"cadence\" in ('month','year') and \"ai_subscription_prices\".\"status\" in ('active','ended','excluded') and \"ai_subscription_prices\".\"source\" in ('catalog','user','unknown')" + } + }, + "isRLSEnabled": false + }, + "public.ai_subscriptions": { + "name": "ai_subscriptions", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "uuid", + "primaryKey": true, + "notNull": true, + "default": "gen_random_uuid()" + }, + "company_id": { + "name": "company_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "provider": { + "name": "provider", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "account_key": { + "name": "account_key", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "identity_verified": { + "name": "identity_verified", + "type": "boolean", + "primaryKey": false, + "notNull": true, + "default": false + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "owner_user_ids": { + "name": "owner_user_ids", + "type": "text[]", + "primaryKey": false, + "notNull": true, + "default": "'{}'::text[]" + }, + "shared": { + "name": "shared", + "type": "boolean", + "primaryKey": false, + "notNull": true, + "default": false + }, + "detected_plan": { + "name": "detected_plan", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "observed_at": { + "name": "observed_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "last_checked_at": { + "name": "last_checked_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "refresh_status": { + "name": "refresh_status", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "merged_into_id": { + "name": "merged_into_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "revision": { + "name": "revision", + "type": "integer", + "primaryKey": false, + "notNull": true, + "default": 0 + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "ai_subscriptions_identity_idx": { + "name": "ai_subscriptions_identity_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "provider", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "account_key", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + }, + "ai_subscriptions_company_idx": { + "name": "ai_subscriptions_company_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "ai_subscriptions_company_id_companies_id_fk": { + "name": "ai_subscriptions_company_id_companies_id_fk", + "tableFrom": "ai_subscriptions", + "tableTo": "companies", + "columnsFrom": [ + "company_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "ai_subscriptions_merged_into_id_ai_subscriptions_id_fk": { + "name": "ai_subscriptions_merged_into_id_ai_subscriptions_id_fk", + "tableFrom": "ai_subscriptions", + "tableTo": "ai_subscriptions", + "columnsFrom": [ + "merged_into_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": { + "ai_subscriptions_no_self_merge": { + "name": "ai_subscriptions_no_self_merge", + "value": "\"ai_subscriptions\".\"merged_into_id\" is null or \"ai_subscriptions\".\"merged_into_id\" <> \"ai_subscriptions\".\"id\"" + } + }, + "isRLSEnabled": false + }, "public.announcement_dismissals": { "name": "announcement_dismissals", "schema": "", @@ -9945,7 +10504,7 @@ }, "chat_endpoints_provider_check": { "name": "chat_endpoints_provider_check", - "value": "\"chat_endpoints\".\"provider\" in ('slack', 'github', 'discord', 'microsoft-teams', 'telegram', 'agentmail', 'imessage-photon')" + "value": "\"chat_endpoints\".\"provider\" in ('slack', 'github', 'discord', 'microsoft-teams', 'telegram', 'speko', 'agentmail', 'imessage-photon')" }, "chat_endpoints_status_check": { "name": "chat_endpoints_status_check", @@ -10130,7 +10689,7 @@ "checkConstraints": { "chat_external_principals_provider_check": { "name": "chat_external_principals_provider_check", - "value": "\"chat_external_principals\".\"provider\" in ('slack', 'github', 'discord', 'microsoft-teams', 'telegram', 'agentmail', 'imessage-photon')" + "value": "\"chat_external_principals\".\"provider\" in ('slack', 'github', 'discord', 'microsoft-teams', 'telegram', 'speko', 'agentmail', 'imessage-photon')" }, "chat_external_principals_kind_check": { "name": "chat_external_principals_kind_check", @@ -11456,6 +12015,31 @@ "primaryKey": false, "notNull": true }, + "owner_type": { + "name": "owner_type", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'personal'" + }, + "owner_login": { + "name": "owner_login", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "app_name": { + "name": "app_name", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "handoff": { + "name": "handoff", + "type": "jsonb", + "primaryKey": false, + "notNull": false + }, "status": { "name": "status", "type": "text", @@ -12396,6 +12980,1267 @@ }, "isRLSEnabled": false }, + "public.chat_voice_callbacks": { + "name": "chat_voice_callbacks", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "uuid", + "primaryKey": true, + "notNull": true, + "default": "gen_random_uuid()" + }, + "company_id": { + "name": "company_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "endpoint_id": { + "name": "endpoint_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "user_id": { + "name": "user_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "phone_number": { + "name": "phone_number", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "enabled": { + "name": "enabled", + "type": "boolean", + "primaryKey": false, + "notNull": true, + "default": false + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "chat_voice_callbacks_owner_uq": { + "name": "chat_voice_callbacks_owner_uq", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "endpoint_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "user_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "chat_voice_callbacks_endpoint_fk": { + "name": "chat_voice_callbacks_endpoint_fk", + "tableFrom": "chat_voice_callbacks", + "tableTo": "chat_endpoints", + "columnsFrom": [ + "company_id", + "endpoint_id" + ], + "columnsTo": [ + "company_id", + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": { + "chat_voice_callbacks_phone_check": { + "name": "chat_voice_callbacks_phone_check", + "value": "\"chat_voice_callbacks\".\"phone_number\" ~ '^\\+[1-9][0-9]{6,14}$'" + } + }, + "isRLSEnabled": false + }, + "public.chat_voice_inbound_calls": { + "name": "chat_voice_inbound_calls", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "uuid", + "primaryKey": true, + "notNull": true, + "default": "gen_random_uuid()" + }, + "company_id": { + "name": "company_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "endpoint_id": { + "name": "endpoint_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "provider_session_id": { + "name": "provider_session_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "state": { + "name": "state", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'awaiting_approval'" + }, + "approval_code": { + "name": "approval_code", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "generation": { + "name": "generation", + "type": "integer", + "primaryKey": false, + "notNull": true + }, + "credential_fingerprint": { + "name": "credential_fingerprint", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "tool_token_hash": { + "name": "tool_token_hash", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "request_fingerprint": { + "name": "request_fingerprint", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "approved_by_user_id": { + "name": "approved_by_user_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "caller_authority": { + "name": "caller_authority", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "intake_issue_id": { + "name": "intake_issue_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "requested_issue_id": { + "name": "requested_issue_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "session_id": { + "name": "session_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "expires_at": { + "name": "expires_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "chat_voice_inbound_expiry_idx": { + "name": "chat_voice_inbound_expiry_idx", + "columns": [ + { + "expression": "state", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "expires_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "chat_voice_inbound_endpoint_fk": { + "name": "chat_voice_inbound_endpoint_fk", + "tableFrom": "chat_voice_inbound_calls", + "tableTo": "chat_endpoints", + "columnsFrom": [ + "company_id", + "endpoint_id" + ], + "columnsTo": [ + "company_id", + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "chat_voice_inbound_session_fk": { + "name": "chat_voice_inbound_session_fk", + "tableFrom": "chat_voice_inbound_calls", + "tableTo": "chat_voice_sessions", + "columnsFrom": [ + "company_id", + "session_id" + ], + "columnsTo": [ + "company_id", + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + }, + "chat_voice_inbound_intake_fk": { + "name": "chat_voice_inbound_intake_fk", + "tableFrom": "chat_voice_inbound_calls", + "tableTo": "issues", + "columnsFrom": [ + "company_id", + "intake_issue_id" + ], + "columnsTo": [ + "company_id", + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + }, + "chat_voice_inbound_requested_issue_fk": { + "name": "chat_voice_inbound_requested_issue_fk", + "tableFrom": "chat_voice_inbound_calls", + "tableTo": "issues", + "columnsFrom": [ + "company_id", + "requested_issue_id" + ], + "columnsTo": [ + "company_id", + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": { + "chat_voice_inbound_provider_uq": { + "name": "chat_voice_inbound_provider_uq", + "nullsNotDistinct": false, + "columns": [ + "provider_session_id" + ] + } + }, + "policies": {}, + "checkConstraints": { + "chat_voice_inbound_state_check": { + "name": "chat_voice_inbound_state_check", + "value": "\"chat_voice_inbound_calls\".\"state\" in ('guest_intake', 'awaiting_approval', 'approving', 'approved', 'denied', 'expired', 'ended')" + }, + "chat_voice_inbound_token_check": { + "name": "chat_voice_inbound_token_check", + "value": "\"chat_voice_inbound_calls\".\"tool_token_hash\" ~ '^[a-f0-9]{64}$'" + }, + "chat_voice_inbound_code_check": { + "name": "chat_voice_inbound_code_check", + "value": "\"chat_voice_inbound_calls\".\"approval_code\" ~ '^[0-9]{6}$'" + }, + "chat_voice_inbound_generation_check": { + "name": "chat_voice_inbound_generation_check", + "value": "\"chat_voice_inbound_calls\".\"generation\" > 0" + } + }, + "isRLSEnabled": false + }, + "public.chat_voice_phone_lines": { + "name": "chat_voice_phone_lines", + "schema": "", + "columns": { + "endpoint_id": { + "name": "endpoint_id", + "type": "uuid", + "primaryKey": true, + "notNull": true + }, + "company_id": { + "name": "company_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "provider_number_id": { + "name": "provider_number_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "phone_number": { + "name": "phone_number", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "enabled": { + "name": "enabled", + "type": "boolean", + "primaryKey": false, + "notNull": true, + "default": false + }, + "guest_intake": { + "name": "guest_intake", + "type": "boolean", + "primaryKey": false, + "notNull": true, + "default": false + }, + "low_trust_environment_id": { + "name": "low_trust_environment_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": {}, + "foreignKeys": { + "chat_voice_phone_lines_low_trust_environment_id_environments_id_fk": { + "name": "chat_voice_phone_lines_low_trust_environment_id_environments_id_fk", + "tableFrom": "chat_voice_phone_lines", + "tableTo": "environments", + "columnsFrom": [ + "low_trust_environment_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + }, + "chat_voice_phone_lines_endpoint_fk": { + "name": "chat_voice_phone_lines_endpoint_fk", + "tableFrom": "chat_voice_phone_lines", + "tableTo": "chat_endpoints", + "columnsFrom": [ + "company_id", + "endpoint_id" + ], + "columnsTo": [ + "company_id", + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": { + "chat_voice_phone_lines_number_uq": { + "name": "chat_voice_phone_lines_number_uq", + "nullsNotDistinct": false, + "columns": [ + "provider_number_id" + ] + } + }, + "policies": {}, + "checkConstraints": { + "chat_voice_phone_lines_number_check": { + "name": "chat_voice_phone_lines_number_check", + "value": "\"chat_voice_phone_lines\".\"phone_number\" ~ '^\\+[1-9][0-9]{6,14}$'" + } + }, + "isRLSEnabled": false + }, + "public.chat_voice_reports": { + "name": "chat_voice_reports", + "schema": "", + "columns": { + "session_id": { + "name": "session_id", + "type": "uuid", + "primaryKey": true, + "notNull": true + }, + "company_id": { + "name": "company_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'pending'" + }, + "transcript": { + "name": "transcript", + "type": "jsonb", + "primaryKey": false, + "notNull": true, + "default": "'[]'::jsonb" + }, + "cost_micro_usd": { + "name": "cost_micro_usd", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "duration_seconds": { + "name": "duration_seconds", + "type": "integer", + "primaryKey": false, + "notNull": false + }, + "provider_updated_at": { + "name": "provider_updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "next_check_at": { + "name": "next_check_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "attempts": { + "name": "attempts", + "type": "integer", + "primaryKey": false, + "notNull": true, + "default": 0 + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "chat_voice_reports_pending_idx": { + "name": "chat_voice_reports_pending_idx", + "columns": [ + { + "expression": "status", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "next_check_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "chat_voice_reports_session_fk": { + "name": "chat_voice_reports_session_fk", + "tableFrom": "chat_voice_reports", + "tableTo": "chat_voice_sessions", + "columnsFrom": [ + "company_id", + "session_id" + ], + "columnsTo": [ + "company_id", + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": { + "chat_voice_reports_status_check": { + "name": "chat_voice_reports_status_check", + "value": "\"chat_voice_reports\".\"status\" in ('pending', 'available', 'unavailable')" + }, + "chat_voice_reports_cost_check": { + "name": "chat_voice_reports_cost_check", + "value": "\"chat_voice_reports\".\"cost_micro_usd\" is null or \"chat_voice_reports\".\"cost_micro_usd\" ~ '^[0-9]{1,30}$'" + }, + "chat_voice_reports_duration_check": { + "name": "chat_voice_reports_duration_check", + "value": "\"chat_voice_reports\".\"duration_seconds\" is null or \"chat_voice_reports\".\"duration_seconds\" >= 0" + }, + "chat_voice_reports_attempts_check": { + "name": "chat_voice_reports_attempts_check", + "value": "\"chat_voice_reports\".\"attempts\" >= 0" + } + }, + "isRLSEnabled": false + }, + "public.chat_voice_replies": { + "name": "chat_voice_replies", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "uuid", + "primaryKey": true, + "notNull": true, + "default": "gen_random_uuid()" + }, + "company_id": { + "name": "company_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "session_id": { + "name": "session_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "publication_id": { + "name": "publication_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "cursor": { + "name": "cursor", + "type": "integer", + "primaryKey": false, + "notNull": true + }, + "delivered_at": { + "name": "delivered_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "spoken_at": { + "name": "spoken_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "chat_voice_replies_publication_uq": { + "name": "chat_voice_replies_publication_uq", + "columns": [ + { + "expression": "session_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "publication_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + }, + "chat_voice_replies_cursor_uq": { + "name": "chat_voice_replies_cursor_uq", + "columns": [ + { + "expression": "session_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "cursor", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "chat_voice_replies_company_id_companies_id_fk": { + "name": "chat_voice_replies_company_id_companies_id_fk", + "tableFrom": "chat_voice_replies", + "tableTo": "companies", + "columnsFrom": [ + "company_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "chat_voice_replies_session_fk": { + "name": "chat_voice_replies_session_fk", + "tableFrom": "chat_voice_replies", + "tableTo": "chat_voice_sessions", + "columnsFrom": [ + "company_id", + "session_id" + ], + "columnsTo": [ + "company_id", + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + }, + "chat_voice_replies_publication_fk": { + "name": "chat_voice_replies_publication_fk", + "tableFrom": "chat_voice_replies", + "tableTo": "chat_publications", + "columnsFrom": [ + "company_id", + "publication_id" + ], + "columnsTo": [ + "company_id", + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": { + "chat_voice_replies_cursor_check": { + "name": "chat_voice_replies_cursor_check", + "value": "\"chat_voice_replies\".\"cursor\" > 0" + }, + "chat_voice_replies_spoken_check": { + "name": "chat_voice_replies_spoken_check", + "value": "\"chat_voice_replies\".\"spoken_at\" is null or \"chat_voice_replies\".\"delivered_at\" is not null" + } + }, + "isRLSEnabled": false + }, + "public.chat_voice_sessions": { + "name": "chat_voice_sessions", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "uuid", + "primaryKey": true, + "notNull": true, + "default": "gen_random_uuid()" + }, + "company_id": { + "name": "company_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "endpoint_id": { + "name": "endpoint_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "conversation_id": { + "name": "conversation_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "issue_id": { + "name": "issue_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "assigned_agent_id": { + "name": "assigned_agent_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "caller_id": { + "name": "caller_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "caller_authority": { + "name": "caller_authority", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "approved_by_user_id": { + "name": "approved_by_user_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "mode": { + "name": "mode", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "state": { + "name": "state", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'reserved'" + }, + "generation": { + "name": "generation", + "type": "integer", + "primaryKey": false, + "notNull": true + }, + "credential_fingerprint": { + "name": "credential_fingerprint", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "provider_session_id": { + "name": "provider_session_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "tool_token_hash": { + "name": "tool_token_hash", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "idempotency_key": { + "name": "idempotency_key", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "request_fingerprint": { + "name": "request_fingerprint", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "reply_cursor": { + "name": "reply_cursor", + "type": "integer", + "primaryKey": false, + "notNull": true, + "default": 0 + }, + "error_code": { + "name": "error_code", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "expires_at": { + "name": "expires_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true + }, + "ended_at": { + "name": "ended_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "chat_voice_sessions_provider_uq": { + "name": "chat_voice_sessions_provider_uq", + "columns": [ + { + "expression": "provider_session_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + }, + "chat_voice_sessions_request_uq": { + "name": "chat_voice_sessions_request_uq", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "caller_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "idempotency_key", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + }, + "chat_voice_sessions_live_caller_task_uq": { + "name": "chat_voice_sessions_live_caller_task_uq", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "caller_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "issue_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "where": "\"chat_voice_sessions\".\"state\" not in ('ended', 'failed', 'expired')", + "concurrently": false, + "method": "btree", + "with": {} + }, + "chat_voice_sessions_expiry_idx": { + "name": "chat_voice_sessions_expiry_idx", + "columns": [ + { + "expression": "state", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "expires_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "chat_voice_sessions_company_id_companies_id_fk": { + "name": "chat_voice_sessions_company_id_companies_id_fk", + "tableFrom": "chat_voice_sessions", + "tableTo": "companies", + "columnsFrom": [ + "company_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "chat_voice_sessions_endpoint_fk": { + "name": "chat_voice_sessions_endpoint_fk", + "tableFrom": "chat_voice_sessions", + "tableTo": "chat_endpoints", + "columnsFrom": [ + "company_id", + "endpoint_id" + ], + "columnsTo": [ + "company_id", + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + }, + "chat_voice_sessions_conversation_fk": { + "name": "chat_voice_sessions_conversation_fk", + "tableFrom": "chat_voice_sessions", + "tableTo": "chat_conversations", + "columnsFrom": [ + "company_id", + "conversation_id" + ], + "columnsTo": [ + "company_id", + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + }, + "chat_voice_sessions_issue_fk": { + "name": "chat_voice_sessions_issue_fk", + "tableFrom": "chat_voice_sessions", + "tableTo": "issues", + "columnsFrom": [ + "company_id", + "issue_id" + ], + "columnsTo": [ + "company_id", + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + }, + "chat_voice_sessions_agent_fk": { + "name": "chat_voice_sessions_agent_fk", + "tableFrom": "chat_voice_sessions", + "tableTo": "agents", + "columnsFrom": [ + "company_id", + "assigned_agent_id" + ], + "columnsTo": [ + "company_id", + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": { + "chat_voice_sessions_company_id_uq": { + "name": "chat_voice_sessions_company_id_uq", + "nullsNotDistinct": false, + "columns": [ + "company_id", + "id" + ] + } + }, + "policies": {}, + "checkConstraints": { + "chat_voice_sessions_state_check": { + "name": "chat_voice_sessions_state_check", + "value": "\"chat_voice_sessions\".\"state\" in ('reserved', 'creating', 'creation_unknown', 'connecting', 'active', 'awaiting_approval', 'ending', 'ended', 'failed', 'expired')" + }, + "chat_voice_sessions_mode_check": { + "name": "chat_voice_sessions_mode_check", + "value": "\"chat_voice_sessions\".\"mode\" in ('browser', 'inbound_phone', 'outbound_phone')" + }, + "chat_voice_sessions_authority_check": { + "name": "chat_voice_sessions_authority_check", + "value": "\"chat_voice_sessions\".\"caller_authority\" in ('member', 'instance_admin', 'local_board', 'guest_intake', 'pending_approval')" + }, + "chat_voice_sessions_counters_check": { + "name": "chat_voice_sessions_counters_check", + "value": "\"chat_voice_sessions\".\"generation\" >= 1 and \"chat_voice_sessions\".\"reply_cursor\" >= 0" + }, + "chat_voice_sessions_token_check": { + "name": "chat_voice_sessions_token_check", + "value": "\"chat_voice_sessions\".\"tool_token_hash\" ~ '^[a-f0-9]{64}$'" + }, + "chat_voice_sessions_expiry_check": { + "name": "chat_voice_sessions_expiry_check", + "value": "\"chat_voice_sessions\".\"expires_at\" > \"chat_voice_sessions\".\"created_at\"" + } + }, + "isRLSEnabled": false + }, + "public.chat_voice_tool_calls": { + "name": "chat_voice_tool_calls", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "uuid", + "primaryKey": true, + "notNull": true, + "default": "gen_random_uuid()" + }, + "company_id": { + "name": "company_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "session_id": { + "name": "session_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "provider_tool_call_id": { + "name": "provider_tool_call_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "webhook_id": { + "name": "webhook_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "fingerprint": { + "name": "fingerprint", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "tool": { + "name": "tool", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "delivery_id": { + "name": "delivery_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "response": { + "name": "response", + "type": "jsonb", + "primaryKey": false, + "notNull": true + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "chat_voice_tool_calls_tool_uq": { + "name": "chat_voice_tool_calls_tool_uq", + "columns": [ + { + "expression": "session_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "provider_tool_call_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + }, + "chat_voice_tool_calls_webhook_uq": { + "name": "chat_voice_tool_calls_webhook_uq", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "webhook_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "chat_voice_tool_calls_company_id_companies_id_fk": { + "name": "chat_voice_tool_calls_company_id_companies_id_fk", + "tableFrom": "chat_voice_tool_calls", + "tableTo": "companies", + "columnsFrom": [ + "company_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "chat_voice_tool_calls_session_fk": { + "name": "chat_voice_tool_calls_session_fk", + "tableFrom": "chat_voice_tool_calls", + "tableTo": "chat_voice_sessions", + "columnsFrom": [ + "company_id", + "session_id" + ], + "columnsTo": [ + "company_id", + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + }, + "chat_voice_tool_calls_delivery_fk": { + "name": "chat_voice_tool_calls_delivery_fk", + "tableFrom": "chat_voice_tool_calls", + "tableTo": "chat_deliveries", + "columnsFrom": [ + "company_id", + "delivery_id" + ], + "columnsTo": [ + "company_id", + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": { + "chat_voice_tool_calls_tool_check": { + "name": "chat_voice_tool_calls_tool_check", + "value": "\"chat_voice_tool_calls\".\"tool\" in ('submit_request', 'get_updates', 'answer_question')" + } + }, + "isRLSEnabled": false + }, "public.cli_auth_challenges": { "name": "cli_auth_challenges", "schema": "", @@ -17560,6 +19405,12 @@ "primaryKey": false, "notNull": false }, + "subscription_id": { + "name": "subscription_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, "billing_code": { "name": "billing_code", "type": "text", @@ -17795,6 +19646,33 @@ "method": "btree", "with": {} }, + "cost_events_company_subscription_occurred_idx": { + "name": "cost_events_company_subscription_occurred_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "subscription_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "occurred_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, "cost_events_company_agent_occurred_idx": { "name": "cost_events_company_agent_occurred_idx", "columns": [ @@ -17976,6 +19854,19 @@ ], "onDelete": "no action", "onUpdate": "no action" + }, + "cost_events_subscription_id_ai_subscriptions_id_fk": { + "name": "cost_events_subscription_id_ai_subscriptions_id_fk", + "tableFrom": "cost_events", + "tableTo": "ai_subscriptions", + "columnsFrom": [ + "subscription_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" } }, "compositePrimaryKeys": {}, @@ -17984,7 +19875,7 @@ "checkConstraints": { "cost_events_usage_kind_check": { "name": "cost_events_usage_kind_check", - "value": "\"cost_events\".\"usage_kind\" = 'decision' or (\"cost_events\".\"usage_kind\" = 'agent' and \"cost_events\".\"agent_id\" is not null)" + "value": "\"cost_events\".\"usage_kind\" in ('decision', 'fast_response') or (\"cost_events\".\"usage_kind\" = 'agent' and \"cost_events\".\"agent_id\" is not null)" }, "cost_events_nonnegative_amounts": { "name": "cost_events_nonnegative_amounts", @@ -25649,6 +27540,378 @@ "checkConstraints": {}, "isRLSEnabled": false }, + "public.company_fast_responses": { + "name": "company_fast_responses", + "schema": "", + "columns": { + "company_id": { + "name": "company_id", + "type": "uuid", + "primaryKey": true, + "notNull": true + }, + "connection_id": { + "name": "connection_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "grant_id": { + "name": "grant_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "model": { + "name": "model", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "provider": { + "name": "provider", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "enabled": { + "name": "enabled", + "type": "boolean", + "primaryKey": false, + "notNull": true, + "default": false + }, + "allow_sponsored": { + "name": "allow_sponsored", + "type": "boolean", + "primaryKey": false, + "notNull": true, + "default": true + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": {}, + "foreignKeys": { + "company_fast_responses_company_id_companies_id_fk": { + "name": "company_fast_responses_company_id_companies_id_fk", + "tableFrom": "company_fast_responses", + "tableTo": "companies", + "columnsFrom": [ + "company_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": { + "company_fast_responses_configured": { + "name": "company_fast_responses_configured", + "value": "(\"company_fast_responses\".\"connection_id\" is null) = (\"company_fast_responses\".\"grant_id\" is null) and (not \"company_fast_responses\".\"enabled\" or (\"company_fast_responses\".\"connection_id\" is not null and \"company_fast_responses\".\"model\" is not null))" + } + }, + "isRLSEnabled": false + }, + "public.fast_response_requests": { + "name": "fast_response_requests", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "uuid", + "primaryKey": true, + "notNull": true, + "default": "gen_random_uuid()" + }, + "company_id": { + "name": "company_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "source_key": { + "name": "source_key", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "source_comment_id": { + "name": "source_comment_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "issue_id": { + "name": "issue_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "project_id": { + "name": "project_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "agent_id": { + "name": "agent_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "responsible_user_id": { + "name": "responsible_user_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "sponsored": { + "name": "sponsored", + "type": "boolean", + "primaryKey": false, + "notNull": true, + "default": false + }, + "endpoint_id": { + "name": "endpoint_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "conversation_id": { + "name": "conversation_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "delivery_id": { + "name": "delivery_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "session_generation": { + "name": "session_generation", + "type": "integer", + "primaryKey": false, + "notNull": false + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'pending'" + }, + "publication_status": { + "name": "publication_status", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'pending'" + }, + "connection_id": { + "name": "connection_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "grant_id": { + "name": "grant_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "provider": { + "name": "provider", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "model": { + "name": "model", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "provider_request_id": { + "name": "provider_request_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "cost_event_id": { + "name": "cost_event_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "error_code": { + "name": "error_code", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "comment_id": { + "name": "comment_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "accepted_at": { + "name": "accepted_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "expires_at": { + "name": "expires_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true + }, + "started_at": { + "name": "started_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "finished_at": { + "name": "finished_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "duration_ms": { + "name": "duration_ms", + "type": "integer", + "primaryKey": false, + "notNull": false + }, + "input_tokens": { + "name": "input_tokens", + "type": "integer", + "primaryKey": false, + "notNull": false + }, + "output_tokens": { + "name": "output_tokens", + "type": "integer", + "primaryKey": false, + "notNull": false + } + }, + "indexes": { + "fast_response_requests_source_idx": { + "name": "fast_response_requests_source_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "source_key", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + }, + "fast_response_requests_history_idx": { + "name": "fast_response_requests_history_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "accepted_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "fast_response_requests_work_idx": { + "name": "fast_response_requests_work_idx", + "columns": [ + { + "expression": "expires_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"fast_response_requests\".\"status\" in ('pending', 'running')", + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "fast_response_requests_company_id_companies_id_fk": { + "name": "fast_response_requests_company_id_companies_id_fk", + "tableFrom": "fast_response_requests", + "tableTo": "companies", + "columnsFrom": [ + "company_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "fast_response_requests_cost_event_id_cost_events_id_fk": { + "name": "fast_response_requests_cost_event_id_cost_events_id_fk", + "tableFrom": "fast_response_requests", + "tableTo": "cost_events", + "columnsFrom": [ + "cost_event_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": { + "fast_response_requests_status_check": { + "name": "fast_response_requests_status_check", + "value": "\"fast_response_requests\".\"status\" in ('pending', 'running', 'succeeded', 'failed', 'unknown', 'skipped')" + } + }, + "isRLSEnabled": false + }, "public.feedback_exports": { "name": "feedback_exports", "schema": "", @@ -30550,6 +32813,19 @@ "primaryKey": false, "notNull": false }, + "origin": { + "name": "origin", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'comment'" + }, + "fast_response_request_id": { + "name": "fast_response_request_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, "body": { "name": "body", "type": "text", @@ -49371,7 +51647,7 @@ }, "tool_connections_transport_check": { "name": "tool_connections_transport_check", - "value": "\"tool_connections\".\"transport\" in ('mcp_remote', 'rest_api', 'local_stdio', 'chat_sdk', 'runtime_auth')" + "value": "\"tool_connections\".\"transport\" in ('mcp_remote', 'rest_api', 'local_stdio', 'chat_sdk', 'voice', 'runtime_auth')" }, "tool_connections_purpose_check": { "name": "tool_connections_purpose_check", @@ -49379,7 +51655,7 @@ }, "tool_connections_channel_transport_check": { "name": "tool_connections_channel_transport_check", - "value": "(\n (\"tool_connections\".\"connection_purpose\" = 'tool' and \"tool_connections\".\"transport\" not in ('chat_sdk', 'runtime_auth'))\n or\n (\"tool_connections\".\"connection_purpose\" = 'channel' and (\"tool_connections\".\"transport\" = 'chat_sdk' or (\"tool_connections\".\"transport\" = 'rest_api' and \"tool_connections\".\"config\"->>'provider' = 'agentmail')))\n or\n (\"tool_connections\".\"connection_purpose\" = 'ai' and \"tool_connections\".\"transport\" = 'runtime_auth')\n )" + "value": "(\n (\"tool_connections\".\"connection_purpose\" = 'tool' and \"tool_connections\".\"transport\" not in ('chat_sdk', 'voice', 'runtime_auth'))\n or\n (\"tool_connections\".\"connection_purpose\" = 'channel' and (\"tool_connections\".\"transport\" in ('chat_sdk', 'voice') or (\"tool_connections\".\"transport\" = 'rest_api' and \"tool_connections\".\"config\"->>'provider' = 'agentmail')))\n or\n (\"tool_connections\".\"connection_purpose\" = 'ai' and \"tool_connections\".\"transport\" = 'runtime_auth')\n )" }, "tool_connections_auth_kind_check": { "name": "tool_connections_auth_kind_check", diff --git a/packages/db/src/migrations/meta/_journal.json b/packages/db/src/migrations/meta/_journal.json index fa45fd59eb2..412ba94d173 100644 --- a/packages/db/src/migrations/meta/_journal.json +++ b/packages/db/src/migrations/meta/_journal.json @@ -2311,6 +2311,13 @@ "when": 1791642049733, "tag": "0331_panoramic_spyke", "breakpoints": true + }, + { + "idx": 332, + "version": "7", + "when": 1791693159075, + "tag": "0332_little_night_nurse", + "breakpoints": true } ] } \ No newline at end of file diff --git a/packages/db/src/question-response-history-migration.test.ts b/packages/db/src/question-response-history-migration.test.ts new file mode 100644 index 00000000000..f09fbe03ac4 --- /dev/null +++ b/packages/db/src/question-response-history-migration.test.ts @@ -0,0 +1,53 @@ +import { createHash, randomUUID } from "node:crypto"; +import { readFile } from "node:fs/promises"; +import postgres from "postgres"; +import { describe, expect, it } from "vitest"; +import { applyPendingMigrations } from "./client.js"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./test-embedded-postgres.js"; + +const support = await getEmbeddedPostgresTestSupport(); +const indexName = "agent_wakeup_requests_question_response_history_idx"; +const migrationFile = "0332_little_night_nurse.sql"; + +(support.supported ? describe : describe.skip)("question-response history migration", () => { + it("repairs an interrupted concurrent index build and serializes replay without deleting wakes", async () => { + const database = await startEmbeddedPostgresTestDatabase("question-history-migration-"); + const sql = postgres(database.connectionString, { max: 1, onnotice: () => {} }); + try { + const companyId = randomUUID(); + const agentId = randomUUID(); + await sql`INSERT INTO companies (id, name, issue_prefix) VALUES (${companyId}, 'Migration fixture', 'QHM')`; + await sql`INSERT INTO agents (id, company_id, name, role, adapter_type) + VALUES (${agentId}, ${companyId}, 'Fixture', 'engineer', 'process')`; + await sql`INSERT INTO agent_wakeup_requests (company_id, agent_id, source, idempotency_key, status) + SELECT ${companyId}::uuid, ${agentId}::uuid, 'automation', 'question-response:' || n::text, 'failed' + FROM generate_series(1, 3) n`; + const content = await readFile(new URL(`./migrations/${migrationFile}`, import.meta.url), "utf8"); + const hash = createHash("sha256").update(content).digest("hex"); + await sql`DELETE FROM drizzle.__drizzle_migrations WHERE hash = ${hash}`; + await sql.unsafe(`DROP INDEX ${indexName}`); + // A failed concurrent UNIQUE build leaves the same invalid-index state + // as an interrupted nonunique build, without relying on timing a cancel. + await expect(sql.unsafe(`CREATE UNIQUE INDEX CONCURRENTLY ${indexName} ON agent_wakeup_requests (company_id)`)) + .rejects.toThrow(); + const [invalid] = await sql`SELECT i.indisvalid FROM pg_index i JOIN pg_class c ON c.oid = i.indexrelid + WHERE c.relname = ${indexName}`; + expect(invalid?.indisvalid).toBe(false); + await Promise.all([applyPendingMigrations(database.connectionString), applyPendingMigrations(database.connectionString)]); + const [rebuilt] = await sql`SELECT i.indisvalid, i.indisunique, pg_get_indexdef(i.indexrelid) AS definition + FROM pg_index i JOIN pg_class c ON c.oid = i.indexrelid WHERE c.relname = ${indexName}`; + expect(rebuilt?.indisvalid).toBe(true); + expect(rebuilt?.indisunique).toBe(false); + expect(rebuilt?.definition).toContain("company_id, idempotency_key"); + expect(rebuilt?.definition).toContain("question-response:%"); + const [wakes] = await sql`SELECT count(*)::int AS n FROM agent_wakeup_requests WHERE company_id = ${companyId}`; + expect(wakes?.n).toBe(3); + const [history] = await sql`SELECT count(*)::int AS n FROM drizzle.__drizzle_migrations WHERE hash = ${hash}`; + expect(history?.n).toBe(1); + await applyPendingMigrations(database.connectionString); + } finally { + await sql.end(); + await database.cleanup(); + } + }, 60_000); +}); diff --git a/packages/db/src/schema/agent_wakeup_requests.ts b/packages/db/src/schema/agent_wakeup_requests.ts index 0537bf6ee24..ee84a395203 100644 --- a/packages/db/src/schema/agent_wakeup_requests.ts +++ b/packages/db/src/schema/agent_wakeup_requests.ts @@ -75,6 +75,12 @@ export const agentWakeupRequests = pgTable( ) .on(table.companyId, table.idempotencyKey) .where(sql`(${table.idempotencyKey} LIKE 'question-response:%' OR ${table.idempotencyKey} LIKE 'interaction:%') AND ${table.status} NOT IN ('skipped', 'failed', 'cancelled')`), + // Sleep checks also need failed/cancelled/skipped receipts, which the + // delivery idempotency fence deliberately excludes. + questionResponseHistoryIdx: index("agent_wakeup_requests_question_response_history_idx") + .on(table.companyId, table.idempotencyKey) + .concurrently() + .where(sql`${table.idempotencyKey} LIKE 'question-response:%'`), connectionIntentDeliveryIdempotencyUq: uniqueIndex("agent_wakeup_requests_connection_intent_delivery_idempotency_uq") .on(table.companyId, table.idempotencyKey) .where(sql`${table.idempotencyKey} LIKE 'connection-intent:%' AND ${table.status} NOT IN ('skipped', 'failed', 'cancelled')`), diff --git a/server/src/__tests__/idle-sleep-safety.test.ts b/server/src/__tests__/idle-sleep-safety.test.ts index 718d33ce534..f8e15241074 100644 --- a/server/src/__tests__/idle-sleep-safety.test.ts +++ b/server/src/__tests__/idle-sleep-safety.test.ts @@ -5,7 +5,8 @@ import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest import { agentInstructionWorkingCopies, agents, agentApiKeys, agentWakeupRequests, companies, companySecretProposals, createDb, adapterAuthSessions, environments, environmentLeases, executionWorkspaces, - heartbeatRuns, issues, issueWatchdogs, projects, routines, type Db, + heartbeatRuns, issues, issueWatchdogs, issueThreadInteractions, issueQuestionResponseDeliveries, + projects, routines, type Db, } from "@paperclipai/db"; import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; import { readIdleSleepSafety, type IdleSleepDrainStatus } from "../services/idle-sleep-safety.js"; @@ -184,6 +185,152 @@ if (!support.supported) console.warn(`Skipping idle sleep Postgres tests: ${supp return { companyId, agentId, runId }; } + async function seedQuestionDelivery(change: Partial = {}) { + const seeded = await seed(); + await db.update(heartbeatRuns).set({ finishedAt: new Date(now) }).where(eq(heartbeatRuns.id, seeded.runId)); + const issueId = randomUUID(); + const interactionId = randomUUID(); + await db.insert(issues).values({ id: issueId, companyId: seeded.companyId, title: "Finished question", status: "done" }); + await db.insert(issueThreadInteractions).values({ + id: interactionId, companyId: seeded.companyId, issueId, + kind: "ask_user_questions", status: "answered", payload: {}, + }); + await db.insert(issueQuestionResponseDeliveries).values({ + companyId: seeded.companyId, issueId, interactionId, sourceRunId: seeded.runId, + targetRunId: seeded.runId, correlationId: randomUUID(), payloadSha256: "fixture", + status: "fallback_queued", deliveryMode: "wake_fallback", acknowledgedAt: new Date(now), ...change, + }); + return { ...seeded, issueId, interactionId }; + } + + async function seedQuestionWake( + delivery: Awaited>, + change: Partial = {}, + ) { + await db.insert(agentWakeupRequests).values({ + companyId: delivery.companyId, agentId: delivery.agentId, source: "automation", + idempotencyKey: `question-response:${delivery.interactionId}`, + status: "completed", finishedAt: new Date(now), ...change, + }); + } + + it.each(["pending", "delivering"])("keeps a %s question delivery awake", async status => { + await seedQuestionDelivery({ status, acknowledgedAt: null }); + expect(await read()).toEqual(present); + }); + + it.each(["delivered", "fallback_queued"])("permits acknowledged %s history after its target run finishes", async status => { + await seedQuestionDelivery({ status }); + expect(await read()).toEqual(none); + }); + + it.each(["delivered", "fallback_queued"])("keeps an unacknowledged %s question delivery awake", async status => { + await seedQuestionDelivery({ status, acknowledgedAt: null }); + expect(await read()).toEqual(present); + }); + + it.each([ + ["queued", { status: "queued", finishedAt: null }], + ["running", { status: "running", finishedAt: null }], + ["unfinished", { finishedAt: null }], + ["retry", { scheduledRetryAt: new Date(now + 1000) }], + ["accounting", { costAccountingPending: true }], + ] as const)("keeps an acknowledged question delivery awake with %s target evidence", async (_name, change) => { + const { runId } = await seedQuestionDelivery(); + await db.update(heartbeatRuns).set(change).where(eq(heartbeatRuns.id, runId)); + expect(await read()).toEqual(present); + }); + + it("does not accept a foreign company's finished question target", async () => { + const foreign = await seedQuestionDelivery({ status: "failed", targetRunId: null }); + await seedQuestionDelivery({ targetRunId: foreign.runId }); + expect(await read()).toEqual(present); + }); + + it("keeps delivered history with a missing target awake", async () => { + await seedQuestionDelivery({ status: "delivered", targetRunId: null }); + expect(await read()).toEqual(present); + }); + + it("keeps fallback history without a durable wake receipt awake", async () => { + await seedQuestionDelivery({ targetRunId: null }); + expect(await read()).toEqual(present); + }); + + it.each(["completed", "failed", "cancelled", "skipped", "timed_out"])( + "permits fallback history after its exact durable wake is %s and finished", async status => { + const delivery = await seedQuestionDelivery({ targetRunId: null }); + await seedQuestionWake(delivery, { status }); + expect(await read()).toEqual(none); + }, + ); + + it.each(["queued", "deferred_issue_execution", "coalesced"])("keeps an unresolved %s question wake awake", async status => { + const delivery = await seedQuestionDelivery({ targetRunId: null }); + await seedQuestionWake(delivery, { status, finishedAt: null }); + expect(await read()).toEqual(present); + }); + + it("keeps a terminal question wake without completion evidence awake", async () => { + const delivery = await seedQuestionDelivery({ targetRunId: null }); + await seedQuestionWake(delivery, { finishedAt: null }); + expect(await read()).toEqual(present); + }); + + it("permits a deferred question fallback after its coalesced continuation finishes", async () => { + const delivery = await seedQuestionDelivery({ targetRunId: null }); + await seedQuestionWake(delivery, { status: "deferred_issue_execution", finishedAt: null }); + expect(await read()).toEqual(present); + await db.update(agentWakeupRequests).set({ + status: "coalesced", runId: delivery.runId, finishedAt: new Date(now), + }).where(eq(agentWakeupRequests.companyId, delivery.companyId)); + expect(await read()).toEqual(none); + }); + + it.each(["missing_run", "wrong_company", "wrong_agent", "unfinished", "retry", "accounting"])( + "keeps a coalesced question fallback awake with %s linked-run evidence", async kind => { + const delivery = await seedQuestionDelivery({ targetRunId: null }); + let runId = delivery.runId; + if (kind === "missing_run") runId = randomUUID(); + if (kind === "wrong_company") runId = (await seedQuestionDelivery({ status: "failed", targetRunId: null })).runId; + if (kind === "wrong_agent") { + const otherAgent = randomUUID(); + await db.insert(agents).values({ id: otherAgent, companyId: delivery.companyId, name: "Other", role: "engineer", status: "idle", adapterType: "process" }); + await db.update(heartbeatRuns).set({ agentId: otherAgent }).where(eq(heartbeatRuns.id, runId)); + } + if (kind === "unfinished") await db.update(heartbeatRuns).set({ finishedAt: null }).where(eq(heartbeatRuns.id, runId)); + if (kind === "retry") await db.update(heartbeatRuns).set({ scheduledRetryAt: new Date(now + 1000) }).where(eq(heartbeatRuns.id, runId)); + if (kind === "accounting") await db.update(heartbeatRuns).set({ costAccountingPending: true }).where(eq(heartbeatRuns.id, runId)); + await seedQuestionWake(delivery, { status: "coalesced", runId }); + expect(await read()).toEqual(present); + }, + ); + + it("does not accept another interaction's or company's question wake receipt", async () => { + const delivery = await seedQuestionDelivery({ targetRunId: null }); + await seedQuestionWake(delivery, { idempotencyKey: `question-response:${randomUUID()}` }); + const foreign = await seedQuestionDelivery({ status: "failed", targetRunId: null }); + await seedQuestionWake(delivery, { companyId: foreign.companyId, agentId: foreign.agentId }); + expect(await read()).toEqual(present); + }); + + it("permits failed question-delivery history without a pending target", async () => { + await seedQuestionDelivery({ status: "failed", targetRunId: null, acknowledgedAt: null }); + expect(await read()).toEqual(none); + }); + + it("still blocks other work after a question handoff finishes", async () => { + const { issueId } = await seedQuestionDelivery(); + await db.update(issues).set({ status: "todo" }).where(eq(issues.id, issueId)); + expect(await read()).toEqual(present); + }); + + it("requires a current quiet owner even with completed question history", async () => { + await seedQuestionDelivery(); + expect(await readIdleSleepSafety(db, owned, () => now, "stale-owner", emptyLocal)).toEqual(unknown); + expect(await readIdleSleepSafety(db, owned, () => now + 60_001, ownerId, emptyLocal)).toEqual(unknown); + }); + it("keeps saved agent files awake until deferred cleanup completes", async () => { const { companyId, agentId, runId } = await seed(); await db.insert(agentInstructionWorkingCopies).values({ diff --git a/server/src/services/idle-sleep-safety.ts b/server/src/services/idle-sleep-safety.ts index 4f2059f3bb4..3e637a8c0aa 100644 --- a/server/src/services/idle-sleep-safety.ts +++ b/server/src/services/idle-sleep-safety.ts @@ -93,13 +93,38 @@ const WORK_CHECKS = [ THEN s.provider_lease_id::uuid END) AND (l.status IN ('released', 'expired') AND l.cleanup_status = 'success' AND l.released_at IS NOT NULL) IS NOT TRUE))`, + // Answer delivery history is not work once its acknowledged handoff has + // finished. Keep pending deliveries, unfinished target runs, and missing or + // unfinished fallback wake receipts awake. Other work checks still apply. + `SELECT 1 FROM issue_question_response_deliveries d WHERE + d.status NOT IN ('delivered', 'fallback_queued', 'failed') + OR (d.status IN ('delivered', 'fallback_queued') AND d.acknowledged_at IS NULL) + OR (d.status = 'delivered' AND d.target_run_id IS NULL) + OR (d.target_run_id IS NOT NULL AND NOT EXISTS ( + SELECT 1 FROM heartbeat_runs r WHERE r.id = d.target_run_id + AND r.company_id = d.company_id + AND r.status IN ('succeeded', 'failed', 'cancelled', 'timed_out', 'interrupted') + AND r.finished_at IS NOT NULL AND r.scheduled_retry_at IS NULL + AND r.cost_accounting_pending = false)) + OR (d.status = 'fallback_queued' AND d.target_run_id IS NULL AND NOT EXISTS ( + SELECT 1 FROM agent_wakeup_requests w WHERE w.company_id = d.company_id + AND w.idempotency_key = 'question-response:' || d.interaction_id::text + AND w.idempotency_key LIKE 'question-response:%' + AND w.finished_at IS NOT NULL + AND (w.status IN ('completed', 'failed', 'cancelled', 'skipped', 'timed_out') + OR (w.status = 'coalesced' AND EXISTS ( + SELECT 1 FROM heartbeat_runs r WHERE r.id = w.run_id + AND r.company_id = w.company_id AND r.agent_id = w.agent_id + AND r.status IN ('succeeded', 'failed', 'cancelled', 'timed_out', 'interrupted') + AND r.finished_at IS NOT NULL AND r.scheduled_retry_at IS NULL + AND r.cost_accounting_pending = false)))))`, // These less common work sources fail closed on any retained state. Their // terminal-state exceptions can be added with tests for the owning service. ...[ "chat_endpoints", "chat_deliveries", "chat_publications", "chat_actions", "chat_completion_deliveries", "connection_event_deliveries", "connection_intent_deliveries", "tool_action_requests", "tool_action_deliveries", - "issue_question_response_deliveries", "workspace_runtime_services", + "workspace_runtime_services", "execution_workspace_runtime_leases", "pipeline_automation_executions", "native_run_finalizations", "company_transfer_runs", "decisions", "decision_effect_executions", "decision_archive_notification_outbox", "browser_use_sessions", "browser_use_runs",