ci: cut PR wall clock from ~16 to ~6 minutes (#13521)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Every pull request runs the Trusted PR CI workflow before merge
> - The test suites roughly tripled in six weeks, and shard balance did
not keep up, so PR runs crept from ~4 to ~17 minutes
> - Slow CI delays every merge and every contributor
> - This pull request rebalances the shards from fresh measurements,
splits the largest test files, reuses the Rust build cache in three more
jobs, and takes the policy job off the critical path
> - The benefit is a PR wall clock near 6 minutes with the same coverage

## Linked Issues or Issue Description

**What existing behavior does this improve?**

PR CI wall clock. A typical green run took 16-17 minutes. Two months ago
it took about 4 minutes.

**Subsystem affected**

The Trusted PR CI workflow (`.github/workflows/pr-trusted.yml`), the
shard-duration manifests, the vitest shard runner scripts, the
`paperclip-runner` package scripts, and the dry-run branch of
`release.sh`.

**Current behavior**

The shard-duration manifests were stale. The general-server manifest had
durations for ~400 of 649 suites. The e2e manifest was missing 14 of 29
specs. Stale median weights made shard steps range 417s-806s (server)
and 277s-745s (e2e). Three jobs each paid a ~3m40s cold cargo release
build. Every test lane waited ~60s for the policy job before it could
start.

**Proposed behavior**

All lanes finish in a narrow ~200-290s band. The manifests carry fresh
measured durations for every suite. The three largest test files are
split so no single file caps a shard. The Rust cache restore runs in
every job that builds the Runner binary. Test lanes start as soon as the
gate resolves.

**Reason and benefit**

Merges stop waiting on CI. The projected wall clock is ~6 minutes for
the same test coverage.

## What Changed

- Rebuild `scripts/general-server-shard-durations.json` (646 suites) and
`scripts/e2e-shard-durations.json` (all specs) from per-suite completion
timestamps in runs 35036001734 and 35024948947.
- Move the PR server lane to the release-verify shape:
`general-server-without-chat` across twelve duration-balanced shards,
plus the chat integration suite split by collected test location across
three dedicated lanes.
- Split `tests/e2e/chat-adapters-ui.spec.ts` into `-providers` and
`-messaging` specs, and `tests/e2e/agent-chat.spec.ts` into `-sessions`
and `-projects` specs. Each pair shares fixtures through a `.shared.ts`
module. Playwright collects the same test sets (39 and 20 tests).
- Raise e2e shards to eight and serialized shards to nine.
- Run the runner package's `check:all` as four matrix lanes:
`check:static`, `check:runner`, and two native vitest `--shard` halves.
The union is exactly `check:all`.
- Add the read-only Rust cache restore (toolchain pin, `save-if: false`)
to the Canary Dry Run, Build, and Typecheck jobs.
- Make release.sh preview publish payloads concurrently in batches of
eight during `--dry-run`. The real publish path stays strictly serial.
- Drop the policy-job lockfile artifact chain. Each lane installs with
`--frozen-lockfile` and falls back to an inline `--resolution-only`
regeneration. The policy job stays a required check through the `verify`
and `e2e` aggregates.
- Update the shard-count mirrors and workflow assertions in the
partition and gate tests.

## Verification

- `node --test scripts/__tests__/run-vitest-stable-shard.test.mjs
scripts/__tests__/e2e-shard.test.mjs` — 30 pass.
- `node --test '.github/scripts/tests/'*.test.mjs` — 410 pass.
- `node --test scripts/__tests__/release-verify-workflow.test.mjs
scripts/cloud-source-verification.test.mjs
scripts/__tests__/release-dry-run-notes.test.mjs` — 42 pass.
- `playwright test --list` collects 39 tests across the chat-adapters
split and 20 across the agent-chat split, equal to the original files.
- A local vitest collection of the chat suite partitions 995 tests into
498/497 line shards.
- Projected shard weights: server 230s x12, chat ~143s x3, e2e 207-242s
x8, serialized ~216s x9.

## Risks

- The split spec files reorder tests relative to the original files.
Every describe seeds its own company, so the specs stay independent; a
hidden cross-describe dependency would surface as a deterministic
failure in one shard.
- The inline lockfile fallback changes install behavior for
manifest-changing and stacked PRs. The policy job still validates
resolution as a required check.
- `release.sh` changes are confined to the `--dry-run` preview branch.
The publish loop is untouched. `bash -n` passes and the release dry-run
tests pass.
- One PR now schedules ~44 fleet runners. If the RunsOn fleet caps
concurrency, queueing may absorb part of the gain; watch the first runs.

## Model Used

- Claude Fable 5 (`claude-fable-5`, Anthropic), extended thinking, with
tool use (shell, file edits) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
This commit is contained in:
Devin Foley authored and GitHub committed 2026-09-16 11:45:14 -07:00
1 parent c49336acdf
commit d08abcba15
17 files changed
+5330 -4607

No files matched your search

@@ -4,7 +4,7 @@ import { readFileSync } from "node:fs";
const workflow = readFileSync(new URL("../../workflows/pr-trusted.yml", import.meta.url), "utf8");
const jobs = [...workflow.matchAll(/^ ([a-z_][a-z_0-9]*):\n([\s\S]*?)(?=^ [a-z_][a-z_0-9]*:\n|$(?![\s\S]))/gm)];
const installers = jobs.filter(([, , body]) => body.includes("run: pnpm install --frozen-lockfile"));
const installers = jobs.filter(([, , body]) => body.includes("pnpm install --frozen-lockfile"));
test("PR workflows restore dependency stores without creating branch copies", () => {
assert.equal(installers.length, 7);
@@ -20,20 +20,22 @@ test("PR workflows restore dependency stores without creating branch copies", ()
});
for (const [, job, body] of installers) {
test(`${job}: reuse master keys before restoring the resolved PR lockfile`, () => {
test(`${job}: reuse master keys before installing with an inline stale-lockfile fallback`, () => {
const locate = body.indexOf(" - name: Locate pnpm store");
const restore = body.indexOf(" - name: Restore pnpm store (read only)");
const artifact = body.indexOf(" - name: Restore regenerated PR lockfile");
const install = body.indexOf("run: pnpm install --frozen-lockfile");
assert.ok(locate >= 0 && locate < restore && restore < artifact && artifact < install);
const cache = body.slice(restore, artifact);
const install = body.indexOf(" - name: Install dependencies");
assert.ok(locate >= 0 && locate < restore && restore < install);
const cache = body.slice(restore, install);
assert.match(body.slice(locate, restore), /pnpm store path --silent/);
assert.match(body.slice(locate, restore), /node -p 'process.arch'/);
assert.match(cache, /uses: actions\/cache\/restore@[a-f0-9]{40}/);
assert.ok(cache.includes("key: node-cache-${{ runner.os }}-${{ steps.pnpm_store.outputs.arch }}-pnpm-${{ hashFiles('pnpm-lock.yaml') }}"));
assert.ok(cache.includes("restore-keys: node-cache-${{ runner.os }}-${{ steps.pnpm_store.outputs.arch }}-pnpm-"));
assert.match(body.slice(artifact, install), /if: needs.policy.outputs.lockfile_regenerated == '1'/);
assert.match(body.slice(artifact, install), /name: pr-lockfile/);
assert.doesNotMatch(body.slice(artifact, install), /continue-on-error/);
// Lanes must not wait on the policy job for a regenerated lockfile; each
// install resolves a stale one inline and then re-validates frozen.
const installStep = body.slice(install).split(" - name:")[1] ?? body.slice(install);
assert.match(installStep, /if ! pnpm install --frozen-lockfile; then/);
assert.match(installStep, /pnpm install --resolution-only --ignore-scripts --no-frozen-lockfile/);
assert.doesNotMatch(installStep, /needs\.policy/);
});
}
+399 -149
View File
@@ -257,8 +257,6 @@ jobs:
needs: [gate]
runs-on: ${{ needs.gate.outputs.runner }}
timeout-minutes: 10
outputs:
lockfile_regenerated: ${{ steps.regen_lockfile.outputs.regenerated }}
steps:
- name: Checkout repository
@@ -344,32 +342,14 @@ jobs:
PAPERCLIP_RELEASE_BOOTSTRAP_BASE_SHA="${{ github.event.pull_request.base.sha }}" \
node ./scripts/check-release-package-bootstrap.mjs "${changed_paths[@]}"
- name: Validate dependency resolution and regenerate stale lockfile
id: regen_lockfile
run: |
cp pnpm-lock.yaml "$RUNNER_TEMP/pnpm-lock.before.yaml"
pnpm install --resolution-only --ignore-scripts --no-frozen-lockfile
if cmp -s "$RUNNER_TEMP/pnpm-lock.before.yaml" pnpm-lock.yaml; then
echo "regenerated=0" >> "$GITHUB_OUTPUT"
else
echo "regenerated=1" >> "$GITHUB_OUTPUT"
fi
# Manifest-only and stacked PRs keep pnpm-lock.yaml at the default branch.
# Upload a regenerated copy whenever the checked-out merge tree needs one.
# Every downstream job then consumes the same hash without recomputing.
- name: Upload regenerated lockfile for downstream jobs
if: steps.regen_lockfile.outputs.regenerated == '1'
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: pr-lockfile
path: pnpm-lock.yaml
retention-days: 1
if-no-files-found: error
# Each lane resolves a stale lockfile inline; this early check still
# fails fast when the merge tree cannot resolve at all.
- name: Validate dependency resolution
run: pnpm install --resolution-only --ignore-scripts --no-frozen-lockfile
typecheck_release_registry:
name: Typecheck + Release Registry
needs: [gate, policy]
needs: [gate]
if: ${{ needs.gate.outputs.full_ci == 'true' }}
runs-on: ${{ needs.gate.outputs.runner }}
timeout-minutes: 20
@@ -410,15 +390,68 @@ jobs:
key: node-cache-${{ runner.os }}-${{ steps.pnpm_store.outputs.arch }}-pnpm-${{ hashFiles('pnpm-lock.yaml') }}
restore-keys: node-cache-${{ runner.os }}-${{ steps.pnpm_store.outputs.arch }}-pnpm-
- name: Restore regenerated PR lockfile (if policy uploaded one)
if: needs.policy.outputs.lockfile_regenerated == '1'
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
with:
name: pr-lockfile
path: .
- name: Install dependencies
run: pnpm install --frozen-lockfile
# Manifest-changing and stacked PRs can hold a pnpm-lock.yaml that is
# stale for the merge tree. Resolve it inline instead of waiting on a
# policy-job artifact: that dependency put the policy job's queue and
# runtime (~60s) on every lane's critical path, and policy still
# validates resolution as a required check in parallel.
run: |
if ! pnpm install --frozen-lockfile; then
echo '::notice title=Lockfile::checked-in lockfile is stale for this merge tree; resolving it inline'
pnpm install --resolution-only --ignore-scripts --no-frozen-lockfile
pnpm install --frozen-lockfile
fi
# typecheck:build-gaps builds @paperclipai/server, whose build script
# rebuilds the Runner release binary; without the shared Rust cache that
# is a ~3m40s cold compile of all third-party crates (run 35036001734,
# 2026-09-15). Same restore-only contract as Verify Paperclip Runner.
- name: Select the pinned Runner Rust toolchain
working-directory: packages/paperclip-runner
run: |
set -uo pipefail
# release-verify.yml runs on a single post-merge fleet image; the
# gate here can route to ubuntu-latest or the public PR fleet, so
# this tolerates an image without rustup instead of failing every
# pull request. Without the pin the cache key simply will not match.
if ! command -v rustup >/dev/null 2>&1; then
echo '::notice title=Runner Rust cache::rustup is unavailable; building with the image default toolchain'
exit 0
fi
rustup show
toolchain="$(rustup show active-toolchain | awk '{print $1}')"
echo "RUSTUP_TOOLCHAIN=$toolchain" >> "$GITHUB_ENV"
# rust-cache hashes every installed toolchain into the cache key, not
# only the active one. Each runner image also ships its own stable
# Rust, and those disagree across images: 1.98.0 on the RunsOn fleets
# against 1.98.1 on ubuntu-latest. Two runners that agreed on the pin
# therefore still computed different keys, and every GitHub-hosted
# pull request missed this cache while the fleet hit it. Remove every
# toolchain except the pin, so the key depends on the pinned compiler
# and the lockfile alone rather than on what the image happens to
# carry. Keep this block identical in both workflows: the reader and
# the writer must agree or the key matches nothing.
extra_toolchains="$(rustup toolchain list | awk '{print $1}' | grep -vx "$toolchain" || true)"
if [ -n "$extra_toolchains" ]; then
echo "$extra_toolchains" | xargs -n1 rustup toolchain uninstall \
|| echo '::notice title=Runner Rust cache::could not remove an extra toolchain; the cache key may not match'
fi
rustup toolchain list
- name: Restore Runner Rust dependencies (read only)
uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2
with:
workspaces: packages/paperclip-runner/runner -> target
shared-key: release-runner-v1
# Mirror the master writer: these also feed the cache key.
cache-workspace-crates: false
cache-bin: false
# Restore only. Never let a pull request evict master's entry.
save-if: false
- name: Typecheck workspaces whose build scripts skip TypeScript
run: pnpm run typecheck:build-gaps
@@ -428,7 +461,7 @@ jobs:
general_tests:
name: General tests (${{ matrix.group_label }})
needs: [gate, policy]
needs: [gate]
if: ${{ needs.gate.outputs.full_ci == 'true' }}
runs-on: ${{ needs.gate.outputs.runner }}
timeout-minutes: 20
@@ -438,31 +471,75 @@ jobs:
include:
# The server suite is pinned to maxWorkers=1 (server/vitest.config.ts),
# so it can only be parallelized across runners. Shard it to keep this
# lane off the PR critical path. Five shards because the suite has
# grown to ~946s of serial vitest wall time (run 30930345729,
# 2026-08-04): at four shards the worst shard ran 311s and was the
# slowest check in the whole PR run; five brings each shard to ~196s
# of suite time (~240s job), level with the other ~250-300s lanes.
- group: general-server
group_label: server (1/5)
# lane off the PR critical path. The suite has grown to ~3184s of
# serial vitest wall time (mean of runs 35036001734 and 35024948947,
# 2026-09-15): at five plain general-server shards with stale
# recorded durations the worst shard ran 806s of tests while the
# best ran 417s. This now uses the same shape as release-verify.yml:
# the ~429s chat integration suite splits by collected test location
# across three dedicated runners (~143s each), and the remaining
# ~2755s levels across twelve duration-balanced shards at ~230s
# each, in line with the other ~200-290s lanes.
- group: general-server-without-chat
group_label: server (1/12)
shard_index: 0
shard_count: 5
- group: general-server
group_label: server (2/5)
shard_count: 12
- group: general-server-without-chat
group_label: server (2/12)
shard_index: 1
shard_count: 5
- group: general-server
group_label: server (3/5)
shard_count: 12
- group: general-server-without-chat
group_label: server (3/12)
shard_index: 2
shard_count: 5
- group: general-server
group_label: server (4/5)
shard_count: 12
- group: general-server-without-chat
group_label: server (4/12)
shard_index: 3
shard_count: 5
- group: general-server
group_label: server (5/5)
shard_count: 12
- group: general-server-without-chat
group_label: server (5/12)
shard_index: 4
shard_count: 5
shard_count: 12
- group: general-server-without-chat
group_label: server (6/12)
shard_index: 5
shard_count: 12
- group: general-server-without-chat
group_label: server (7/12)
shard_index: 6
shard_count: 12
- group: general-server-without-chat
group_label: server (8/12)
shard_index: 7
shard_count: 12
- group: general-server-without-chat
group_label: server (9/12)
shard_index: 8
shard_count: 12
- group: general-server-without-chat
group_label: server (10/12)
shard_index: 9
shard_count: 12
- group: general-server-without-chat
group_label: server (11/12)
shard_index: 10
shard_count: 12
- group: general-server-without-chat
group_label: server (12/12)
shard_index: 11
shard_count: 12
- group: general-chat
group_label: chat (1/3)
shard_index: 0
shard_count: 3
- group: general-chat
group_label: chat (2/3)
shard_index: 1
shard_count: 3
- group: general-chat
group_label: chat (3/3)
shard_index: 2
shard_count: 3
# workspaces-a was the slowest check in the fully-green PR run
# 31371439296 (2026-08-10) at 319s, with the ui project's single
# vitest invocation accounting for ~224s and the paperclipai CLI
@@ -516,15 +593,18 @@ jobs:
key: node-cache-${{ runner.os }}-${{ steps.pnpm_store.outputs.arch }}-pnpm-${{ hashFiles('pnpm-lock.yaml') }}
restore-keys: node-cache-${{ runner.os }}-${{ steps.pnpm_store.outputs.arch }}-pnpm-
- name: Restore regenerated PR lockfile (if policy uploaded one)
if: needs.policy.outputs.lockfile_regenerated == '1'
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
with:
name: pr-lockfile
path: .
- name: Install dependencies
run: pnpm install --frozen-lockfile
# Manifest-changing and stacked PRs can hold a pnpm-lock.yaml that is
# stale for the merge tree. Resolve it inline instead of waiting on a
# policy-job artifact: that dependency put the policy job's queue and
# runtime (~60s) on every lane's critical path, and policy still
# validates resolution as a required check in parallel.
run: |
if ! pnpm install --frozen-lockfile; then
echo '::notice title=Lockfile::checked-in lockfile is stale for this merge tree; resolving it inline'
pnpm install --resolution-only --ignore-scripts --no-frozen-lockfile
pnpm install --frozen-lockfile
fi
- name: Run grouped general test suites
run: |
@@ -606,11 +686,34 @@ jobs:
esac
verify_paperclip_runner:
name: Verify Paperclip Runner
needs: [gate, policy]
name: Verify Paperclip Runner (${{ matrix.lane_label }})
needs: [gate]
if: ${{ needs.gate.outputs.full_ci == 'true' }}
runs-on: ${{ needs.gate.outputs.runner }}
timeout-minutes: 20
strategy:
fail-fast: false
matrix:
include:
# check:all spent ~380s of its ~600s cache-warm step inside the
# runner package's vitest suite (run 35036001734, 2026-09-15); the
# Rust tests took ~160s and every remaining check ~60s combined,
# which made this single job the PR critical path once the test
# lanes were rebalanced. Split the vitest suite across two runners
# with Vitest's native --shard, the ~160s Rust tests into their own
# lane, and the remaining static checks into a fourth. The four
# lanes union to exactly check:all: check:static covers
# check:eval-kernel, check:protocol-without-vitest, and
# check:api-authority; check:runner covers the Rust half; and the
# two vitest shards cover the package vitest file list.
- lane_label: static checks
command: check:static
- lane_label: rust
command: check:runner
- lane_label: vitest 1/2
command: test:typescript:vitest --shard=1/2
- lane_label: vitest 2/2
command: test:typescript:vitest --shard=2/2
steps:
- name: Checkout repository
@@ -648,15 +751,18 @@ jobs:
key: node-cache-${{ runner.os }}-${{ steps.pnpm_store.outputs.arch }}-pnpm-${{ hashFiles('pnpm-lock.yaml') }}
restore-keys: node-cache-${{ runner.os }}-${{ steps.pnpm_store.outputs.arch }}-pnpm-
- name: Restore regenerated PR lockfile (if policy uploaded one)
if: needs.policy.outputs.lockfile_regenerated == '1'
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
with:
name: pr-lockfile
path: .
- name: Install dependencies
run: pnpm install --frozen-lockfile
# Manifest-changing and stacked PRs can hold a pnpm-lock.yaml that is
# stale for the merge tree. Resolve it inline instead of waiting on a
# policy-job artifact: that dependency put the policy job's queue and
# runtime (~60s) on every lane's critical path, and policy still
# validates resolution as a required check in parallel.
run: |
if ! pnpm install --frozen-lockfile; then
echo '::notice title=Lockfile::checked-in lockfile is stale for this merge tree; resolving it inline'
pnpm install --resolution-only --ignore-scripts --no-frozen-lockfile
pnpm install --frozen-lockfile
fi
# Same restore-only contract as the pnpm store above, for the Rust
# dependency tree: master's post-merge verification is the sole writer
@@ -712,11 +818,15 @@ jobs:
save-if: false
- name: Verify Paperclip Runner
run: pnpm --filter @paperclipai/paperclip-runner check:all
# pnpm appends trailing args to the end of the script's shell chain,
# so the vitest lanes' --shard lands on `vitest run`. Do not add a
# `--` separator: pnpm forwards it literally and vitest would then
# read the shard flag as a test filter.
run: pnpm --filter @paperclipai/paperclip-runner ${{ matrix.command }}
build:
name: Build
needs: [gate, policy]
needs: [gate]
if: ${{ needs.gate.outputs.full_ci == 'true' }}
runs-on: ${{ needs.gate.outputs.runner }}
timeout-minutes: 20
@@ -757,15 +867,68 @@ jobs:
key: node-cache-${{ runner.os }}-${{ steps.pnpm_store.outputs.arch }}-pnpm-${{ hashFiles('pnpm-lock.yaml') }}
restore-keys: node-cache-${{ runner.os }}-${{ steps.pnpm_store.outputs.arch }}-pnpm-
- name: Restore regenerated PR lockfile (if policy uploaded one)
if: needs.policy.outputs.lockfile_regenerated == '1'
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
with:
name: pr-lockfile
path: .
- name: Install dependencies
run: pnpm install --frozen-lockfile
# Manifest-changing and stacked PRs can hold a pnpm-lock.yaml that is
# stale for the merge tree. Resolve it inline instead of waiting on a
# policy-job artifact: that dependency put the policy job's queue and
# runtime (~60s) on every lane's critical path, and policy still
# validates resolution as a required check in parallel.
run: |
if ! pnpm install --frozen-lockfile; then
echo '::notice title=Lockfile::checked-in lockfile is stale for this merge tree; resolving it inline'
pnpm install --resolution-only --ignore-scripts --no-frozen-lockfile
pnpm install --frozen-lockfile
fi
# pnpm build reaches the Runner package's build:binary step; without the
# shared Rust cache that is a ~3m40s cold compile of all third-party
# crates (run 35036001734, 2026-09-15). Same restore-only contract as
# Verify Paperclip Runner.
- name: Select the pinned Runner Rust toolchain
working-directory: packages/paperclip-runner
run: |
set -uo pipefail
# release-verify.yml runs on a single post-merge fleet image; the
# gate here can route to ubuntu-latest or the public PR fleet, so
# this tolerates an image without rustup instead of failing every
# pull request. Without the pin the cache key simply will not match.
if ! command -v rustup >/dev/null 2>&1; then
echo '::notice title=Runner Rust cache::rustup is unavailable; building with the image default toolchain'
exit 0
fi
rustup show
toolchain="$(rustup show active-toolchain | awk '{print $1}')"
echo "RUSTUP_TOOLCHAIN=$toolchain" >> "$GITHUB_ENV"
# rust-cache hashes every installed toolchain into the cache key, not
# only the active one. Each runner image also ships its own stable
# Rust, and those disagree across images: 1.98.0 on the RunsOn fleets
# against 1.98.1 on ubuntu-latest. Two runners that agreed on the pin
# therefore still computed different keys, and every GitHub-hosted
# pull request missed this cache while the fleet hit it. Remove every
# toolchain except the pin, so the key depends on the pinned compiler
# and the lockfile alone rather than on what the image happens to
# carry. Keep this block identical in both workflows: the reader and
# the writer must agree or the key matches nothing.
extra_toolchains="$(rustup toolchain list | awk '{print $1}' | grep -vx "$toolchain" || true)"
if [ -n "$extra_toolchains" ]; then
echo "$extra_toolchains" | xargs -n1 rustup toolchain uninstall \
|| echo '::notice title=Runner Rust cache::could not remove an extra toolchain; the cache key may not match'
fi
rustup toolchain list
- name: Restore Runner Rust dependencies (read only)
uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2
with:
workspaces: packages/paperclip-runner/runner -> target
shared-key: release-runner-v1
# Mirror the master writer: these also feed the cache key.
cache-workspace-crates: false
cache-bin: false
# Restore only. Never let a pull request evict master's entry.
save-if: false
- name: Build Runner Evalbook viewer
run: pnpm --filter @paperclipai/paperclip-runner build:issue-thread
@@ -775,7 +938,7 @@ jobs:
verify_serialized_server:
name: Verify serialized server suites (${{ matrix.shard_label }})
needs: [gate, policy]
needs: [gate]
if: ${{ needs.gate.outputs.full_ci == 'true' }}
runs-on: ${{ needs.gate.outputs.runner }}
timeout-minutes: 20
@@ -783,27 +946,41 @@ jobs:
fail-fast: false
matrix:
include:
# A successful PR run on 2026-08-17 (32012408876) spent 291s in
# serialized shard 1/5 while its siblings ran 170-201s: round-robin
# clustered the heavy suites on one runner. Shards are now balanced
# by recorded duration (scripts/serialized-shard-durations.json),
# which levels the measured 968s suite total to about 194s per
# runner before setup overhead.
# Shards are balanced by recorded duration
# (scripts/serialized-shard-durations.json, refreshed 2026-09-12);
# round-robin used to cluster the heavy suites on one runner (291s
# vs 170-201s in run 32012408876, 2026-08-17). The suite total has
# since grown to ~1926s recorded: five shards ran ~390s of tests
# each while the rebalanced general/e2e lanes run ~210-260s, so
# nine shards level this lane to ~214s and keep it off the
# critical path.
- shard_index: 0
shard_count: 5
shard_label: 1/5
shard_count: 9
shard_label: 1/9
- shard_index: 1
shard_count: 5
shard_label: 2/5
shard_count: 9
shard_label: 2/9
- shard_index: 2
shard_count: 5
shard_label: 3/5
shard_count: 9
shard_label: 3/9
- shard_index: 3
shard_count: 5
shard_label: 4/5
shard_count: 9
shard_label: 4/9
- shard_index: 4
shard_count: 5
shard_label: 5/5
shard_count: 9
shard_label: 5/9
- shard_index: 5
shard_count: 9
shard_label: 6/9
- shard_index: 6
shard_count: 9
shard_label: 7/9
- shard_index: 7
shard_count: 9
shard_label: 8/9
- shard_index: 8
shard_count: 9
shard_label: 9/9
steps:
- name: Checkout repository
@@ -841,22 +1018,25 @@ jobs:
key: node-cache-${{ runner.os }}-${{ steps.pnpm_store.outputs.arch }}-pnpm-${{ hashFiles('pnpm-lock.yaml') }}
restore-keys: node-cache-${{ runner.os }}-${{ steps.pnpm_store.outputs.arch }}-pnpm-
- name: Restore regenerated PR lockfile (if policy uploaded one)
if: needs.policy.outputs.lockfile_regenerated == '1'
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
with:
name: pr-lockfile
path: .
- name: Install dependencies
run: pnpm install --frozen-lockfile
# Manifest-changing and stacked PRs can hold a pnpm-lock.yaml that is
# stale for the merge tree. Resolve it inline instead of waiting on a
# policy-job artifact: that dependency put the policy job's queue and
# runtime (~60s) on every lane's critical path, and policy still
# validates resolution as a required check in parallel.
run: |
if ! pnpm install --frozen-lockfile; then
echo '::notice title=Lockfile::checked-in lockfile is stale for this merge tree; resolving it inline'
pnpm install --resolution-only --ignore-scripts --no-frozen-lockfile
pnpm install --frozen-lockfile
fi
- name: Run serialized server test shard
run: pnpm test:run:serialized -- --shard-index ${{ matrix.shard_index }} --shard-count ${{ matrix.shard_count }}
canary_dry_run:
name: Canary Dry Run
needs: [gate, policy]
needs: [gate]
if: ${{ needs.gate.outputs.full_ci == 'true' }}
runs-on: ${{ needs.gate.outputs.runner }}
timeout-minutes: 20
@@ -897,43 +1077,91 @@ jobs:
key: node-cache-${{ runner.os }}-${{ steps.pnpm_store.outputs.arch }}-pnpm-${{ hashFiles('pnpm-lock.yaml') }}
restore-keys: node-cache-${{ runner.os }}-${{ steps.pnpm_store.outputs.arch }}-pnpm-
- name: Restore regenerated PR lockfile (if policy uploaded one)
if: needs.policy.outputs.lockfile_regenerated == '1'
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
with:
name: pr-lockfile
path: .
- name: Install dependencies
run: pnpm install --frozen-lockfile
# Manifest-changing and stacked PRs can hold a pnpm-lock.yaml that is
# stale for the merge tree. Resolve it inline instead of waiting on a
# policy-job artifact: that dependency put the policy job's queue and
# runtime (~60s) on every lane's critical path, and policy still
# validates resolution as a required check in parallel.
run: |
if ! pnpm install --frozen-lockfile; then
echo '::notice title=Lockfile::checked-in lockfile is stale for this merge tree; resolving it inline'
pnpm install --resolution-only --ignore-scripts --no-frozen-lockfile
pnpm install --frozen-lockfile
fi
# release.sh's Step 2/7 workspace build reaches the Runner package's
# build:binary step; without the shared Rust cache that is a ~3m40s cold
# compile of all third-party crates (run 35036001734, 2026-09-15). Same
# restore-only contract as Verify Paperclip Runner.
- name: Select the pinned Runner Rust toolchain
working-directory: packages/paperclip-runner
run: |
set -uo pipefail
# release-verify.yml runs on a single post-merge fleet image; the
# gate here can route to ubuntu-latest or the public PR fleet, so
# this tolerates an image without rustup instead of failing every
# pull request. Without the pin the cache key simply will not match.
if ! command -v rustup >/dev/null 2>&1; then
echo '::notice title=Runner Rust cache::rustup is unavailable; building with the image default toolchain'
exit 0
fi
rustup show
toolchain="$(rustup show active-toolchain | awk '{print $1}')"
echo "RUSTUP_TOOLCHAIN=$toolchain" >> "$GITHUB_ENV"
# rust-cache hashes every installed toolchain into the cache key, not
# only the active one. Each runner image also ships its own stable
# Rust, and those disagree across images: 1.98.0 on the RunsOn fleets
# against 1.98.1 on ubuntu-latest. Two runners that agreed on the pin
# therefore still computed different keys, and every GitHub-hosted
# pull request missed this cache while the fleet hit it. Remove every
# toolchain except the pin, so the key depends on the pinned compiler
# and the lockfile alone rather than on what the image happens to
# carry. Keep this block identical in both workflows: the reader and
# the writer must agree or the key matches nothing.
extra_toolchains="$(rustup toolchain list | awk '{print $1}' | grep -vx "$toolchain" || true)"
if [ -n "$extra_toolchains" ]; then
echo "$extra_toolchains" | xargs -n1 rustup toolchain uninstall \
|| echo '::notice title=Runner Rust cache::could not remove an extra toolchain; the cache key may not match'
fi
rustup toolchain list
- name: Restore Runner Rust dependencies (read only)
uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2
with:
workspaces: packages/paperclip-runner/runner -> target
shared-key: release-runner-v1
# Mirror the master writer: these also feed the cache key.
cache-workspace-crates: false
cache-bin: false
# Restore only. Never let a pull request evict master's entry.
save-if: false
# `release.sh` always executes its Step 2/7 workspace build, even when
# `--skip-verify` bypasses the initial verification gate. release.sh
# also requires a clean working tree, so any in-place lockfile churn
# from `pnpm install --frozen-lockfile` must be reverted first — unless
# the policy job uploaded a regenerated lockfile (manifest-changing
# PRs), in which case we stage the artifact-restored copy into an
# ephemeral local commit so release.sh sees a clean tree and its
# workspace build sees a lockfile that matches the manifest.
# also requires a clean working tree, and the install step may have
# resolved a stale lockfile in place (manifest-changing or stacked
# PRs), so stage any changed lockfile into an ephemeral local commit:
# release.sh then sees a clean tree and its workspace build sees a
# lockfile that matches the manifests.
- name: Release canary dry run via release.sh internal build
env:
USED_ARTIFACT_LOCKFILE: ${{ needs.policy.outputs.lockfile_regenerated || '0' }}
run: |
git checkout -B master HEAD
if [ "$USED_ARTIFACT_LOCKFILE" = "1" ]; then
git add pnpm-lock.yaml
if ! git diff --cached --quiet; then
git -c user.email=ci@paperclip.local -c user.name=CI \
commit --no-verify -m "ci(canary): stage regenerated lockfile"
fi
else
if git diff --quiet pnpm-lock.yaml; then
git checkout -- pnpm-lock.yaml
else
git add pnpm-lock.yaml
git -c user.email=ci@paperclip.local -c user.name=CI \
commit --no-verify -m "ci(canary): stage regenerated lockfile"
fi
./scripts/release.sh canary --skip-verify --dry-run
e2e_shards:
name: e2e shard (${{ matrix.shard_label }})
needs: [gate, policy]
needs: [gate]
if: ${{ needs.gate.outputs.full_ci == 'true' }}
runs-on: ${{ needs.gate.outputs.runner }}
timeout-minutes: 30
@@ -945,18 +1173,37 @@ jobs:
# because every spec shares one throwaway server and some toggle
# instance-level flags, so it can only be parallelized across runners.
# Each shard boots its own server, which keeps that isolation intact.
# Three shards let the ~3min smoke-lab spec ride alone while the rest
# of the catalog splits evenly, pulling this lane off the PR critical
# path (it was the slowest check at ~8min20s with two shards).
# The catalog has grown to ~1694s of serial spec time (mean of runs
# 35036001734 and 35024948947, 2026-09-15): with three shards and
# stale durations the worst shard ran 745s of specs while the best
# ran 277s. The former floors — chat-adapters-ui at ~442s and
# agent-chat at ~310s — are each split into two specs, so eight
# shards with refreshed durations level to ~212s each; the heaviest
# remaining single spec is chat-adapters-ui-messaging at ~242s.
- shard_index: 0
shard_count: 3
shard_label: 1/3
shard_count: 8
shard_label: 1/8
- shard_index: 1
shard_count: 3
shard_label: 2/3
shard_count: 8
shard_label: 2/8
- shard_index: 2
shard_count: 3
shard_label: 3/3
shard_count: 8
shard_label: 3/8
- shard_index: 3
shard_count: 8
shard_label: 4/8
- shard_index: 4
shard_count: 8
shard_label: 5/8
- shard_index: 5
shard_count: 8
shard_label: 6/8
- shard_index: 6
shard_count: 8
shard_label: 7/8
- shard_index: 7
shard_count: 8
shard_label: 8/8
steps:
- name: Checkout repository
@@ -994,15 +1241,18 @@ jobs:
key: node-cache-${{ runner.os }}-${{ steps.pnpm_store.outputs.arch }}-pnpm-${{ hashFiles('pnpm-lock.yaml') }}
restore-keys: node-cache-${{ runner.os }}-${{ steps.pnpm_store.outputs.arch }}-pnpm-
- name: Restore regenerated PR lockfile (if policy uploaded one)
if: needs.policy.outputs.lockfile_regenerated == '1'
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
with:
name: pr-lockfile
path: .
- name: Install dependencies
run: pnpm install --frozen-lockfile
# Manifest-changing and stacked PRs can hold a pnpm-lock.yaml that is
# stale for the merge tree. Resolve it inline instead of waiting on a
# policy-job artifact: that dependency put the policy job's queue and
# runtime (~60s) on every lane's critical path, and policy still
# validates resolution as a required check in parallel.
run: |
if ! pnpm install --frozen-lockfile; then
echo '::notice title=Lockfile::checked-in lockfile is stale for this merge tree; resolving it inline'
pnpm install --resolution-only --ignore-scripts --no-frozen-lockfile
pnpm install --frozen-lockfile
fi
- name: Verify runner Chrome
# GitHub's Ubuntu runner image already ships Google Chrome, so use that
@@ -153,7 +153,7 @@ Before opening a real provider, I run the deterministic Paperclip-side browser c
```sh
pnpm exec playwright test \
--config tests/e2e/playwright.config.ts \
tests/e2e/chat-adapters-ui.spec.ts \
tests/e2e/chat-adapters-ui-providers.spec.ts \
--grep '^Slack:'
```
+6 -1
View File
@@ -73,7 +73,9 @@
"typecheck:rust": "cargo fmt --manifest-path runner/Cargo.toml --all -- --check && cargo check --manifest-path runner/Cargo.toml --locked --workspace",
"typecheck:browser": "tsc -p tsconfig.browser.json --noEmit",
"test": "pnpm run test:typescript && pnpm run test:rust",
"test:typescript": "pnpm run ensure:eval-build-deps && pnpm run build:rust && node --test test/protocol-contract.test.mjs test/acpx-sidecar-contract.test.mjs test/acpx-codex-package-contract.test.mjs scripts/aws-agentcore-provisioning.test.mjs scripts/build-verified-provider-entrypoints.test.mjs scripts/local-provider-smoke-environment.test.mjs scripts/materialize-opencode-binary.test.mjs && vitest run",
"test:typescript": "pnpm run test:typescript:prep && vitest run",
"test:typescript:prep": "pnpm run ensure:eval-build-deps && pnpm run build:rust && node --test test/protocol-contract.test.mjs test/acpx-sidecar-contract.test.mjs test/acpx-codex-package-contract.test.mjs scripts/aws-agentcore-provisioning.test.mjs scripts/build-verified-provider-entrypoints.test.mjs scripts/local-provider-smoke-environment.test.mjs scripts/materialize-opencode-binary.test.mjs",
"test:typescript:vitest": "pnpm run ensure:eval-build-deps && pnpm run build:rust && vitest run",
"test:rust": "cargo test --release --manifest-path runner/Cargo.toml --locked --workspace",
"test:codex": "cargo test --manifest-path runner/Cargo.toml --locked -p paperclip-runner-core --test codex_provider",
"test:durable": "cargo test --manifest-path runner/Cargo.toml --locked -p paperclip-runner-core durable::",
@@ -95,8 +97,11 @@
"generate:semantic-action-catalog": "pnpm run build:typescript && node scripts/generate-semantic-action-catalog.mjs",
"check:semantic-action-catalog": "pnpm run build:typescript && node scripts/generate-semantic-action-catalog.mjs --check",
"check:protocol": "pnpm run typecheck:typescript && pnpm run check:protocol-manifest && pnpm run test:typescript && pnpm run check:replay-goldens",
"check:protocol-without-vitest": "pnpm run typecheck:typescript && pnpm run check:protocol-manifest && pnpm run test:typescript:prep && pnpm run check:replay-goldens",
"check:runner": "pnpm run typecheck:rust && pnpm run test:rust && pnpm run check:conformance-parity && pnpm run check:replay-parity",
"check:all": "pnpm run check:eval-kernel && pnpm run check:protocol && pnpm run check:runner && pnpm run check:api-authority",
"check:all-without-vitest": "pnpm run check:eval-kernel && pnpm run check:protocol-without-vitest && pnpm run check:runner && pnpm run check:api-authority",
"check:static": "pnpm run check:eval-kernel && pnpm run check:protocol-without-vitest && pnpm run check:api-authority",
"check:api-authority": "node scripts/check-api-authority.mjs",
"generate:replay-goldens": "pnpm run build:typescript && node scripts/generate-replay-goldens.mjs",
"check:replay-goldens": "pnpm run build:typescript && node scripts/generate-replay-goldens.mjs --check",
+14 -12
View File
@@ -17,7 +17,7 @@ const prCallerWorkflow = path.join(repoRoot, ".github", "workflows", "pr.yml");
const trustedPrWorkflowPath = ".github/workflows/pr-trusted.yml";
const trustedPrWorkflow = path.join(repoRoot, trustedPrWorkflowPath);
const SHARD_COUNT = 3;
const SHARD_COUNT = 8;
function runShard(args) {
const result = spawnSync(process.execPath, [script, ...args], { cwd: repoRoot, encoding: "utf8" });
@@ -317,23 +317,25 @@ test("the trusted PR workflow regenerates stale stacked lockfiles", () => {
"resolution-only policy must not restore or save a dependency store",
);
assert.match(
workflow,
policy,
/pnpm install --resolution-only --ignore-scripts --no-frozen-lockfile/,
"the policy job must resolve the complete merge tree without rewriting platform metadata",
);
assert.match(
workflow,
/cmp -s "\$RUNNER_TEMP\/pnpm-lock\.before\.yaml" pnpm-lock\.yaml/,
"the policy job must upload a lockfile only when regeneration changed it",
);
const restoreSteps = workflow.match(
/- name: Restore regenerated PR lockfile \(if policy uploaded one\)\n if: needs\.policy\.outputs\.lockfile_regenerated == '1'/g,
// Test lanes no longer wait on a policy-job artifact: each install step
// resolves a stale lockfile inline, so a manifest-changing or stacked PR
// still installs while the policy job validates resolution in parallel.
const fallbackInstalls = workflow.match(
/if ! pnpm install --frozen-lockfile; then\n[\s\S]{0,240}?pnpm install --resolution-only --ignore-scripts --no-frozen-lockfile\n\s+pnpm install --frozen-lockfile\n\s+fi/g,
) ?? [];
assert.equal(restoreSteps.length, 7, "every downstream install job must restore a required regenerated artifact");
assert.equal(
fallbackInstalls.length,
7,
"every downstream install job must resolve a stale lockfile inline",
);
assert.doesNotMatch(
workflow,
/- name: Restore regenerated PR lockfile \(if policy uploaded one\)[\s\S]{0,220}continue-on-error:/,
"a missing artifact must fail after the policy job says it uploaded one",
/Restore regenerated PR lockfile|lockfile_regenerated|name: pr-lockfile/,
"the policy lockfile artifact chain must stay removed; it put the policy job on every lane's critical path",
);
});
@@ -36,7 +36,14 @@ git_current_branch() { printf 'master\\n'; }
get_last_stable_tag() { printf 'v2026.709.0\\n'; }
get_current_stable_version() { printf '2026.709.0\\n'; }
utc_date_iso() { printf '2026-07-10\\n'; }
list_public_package_info() { printf 'cli\\tpaperclipai\\t0.0.0\\n'; }
list_public_package_info() {
if [ -n "\${FAKE_PACKAGE_INFO:-}" ]; then
printf '%b' "$FAKE_PACKAGE_INFO"
else
printf 'cli\\tpaperclipai\\t0.0.0\\n'
fi
}
package_publish_tool() { printf 'pnpm\\n'; }
next_stable_version() { printf '2026.710.0\\n'; }
next_prerelease_version() { printf '2026.710.0-%s.0\\n' "$1"; }
release_notes_file() { printf '%s/releases/v%s.md\\n' "$REPO_ROOT" "$1"; }
@@ -56,6 +63,7 @@ require_channel_tag_absent_at_head() {
fi
echo "[fixture] require_channel_tag_absent_at_head $1"
}
require_on_master_branch() { :; }
require_clean_worktree() { :; }
require_npm_publish_auth() { :; }
git_local_tag_exists() { return 1; }
@@ -112,9 +120,21 @@ exit 0
set -euo pipefail
printf 'pnpm %s\\n' "$*" >> "$FAKE_CALL_LOG"
if [ "$*" = "build" ]; then
if [ "\${FAKE_BUILD_OK:-}" = "1" ]; then
exit 0
fi
echo "fixture stopped at workspace build"
exit 42
fi
if [ "$1" = "publish" ]; then
echo "fixture publish preview in $PWD"
case "$PWD" in
*"\${FAKE_FAIL_PUBLISH_DIR:-__none__}")
echo "fixture preview failure" >&2
exit 3
;;
esac
fi
exit 0
`,
);
@@ -122,8 +142,11 @@ exit 0
return { binDir, callLog, fixtureDir, script: join(scriptsDir, "release.sh") };
}
function runRelease(args, extraEnv = {}) {
function runRelease(args, extraEnv = {}, prepare = null) {
const fixture = createReleaseFixture();
if (prepare) {
prepare(fixture);
}
const result = spawnSync(fixture.script, args, {
cwd: fixture.fixtureDir,
encoding: "utf8",
@@ -247,3 +270,92 @@ test("beta refuses commits that never shipped a nightly", () => {
assert.match(result.output, /HEAD has no nightly\/v\* tag/);
assert.doesNotMatch(result.calls, /^pnpm /m);
});
// --- Step 5 concurrent publish-payload previews (dry-run only) ---
const PREVIEW_PACKAGE_COUNT = 10; // more than one batch of 8
function preparePreviewFixture(fixture) {
const { fixtureDir } = fixture;
// Step 2 copies the skills tree into these package dirs.
mkdirSync(join(fixtureDir, "skills"), { recursive: true });
for (const dir of ["server", "packages/adapters/claude-local", "packages/adapters/codex-local"]) {
mkdirSync(join(fixtureDir, dir), { recursive: true });
}
// Step 2 build helpers that live outside the pnpm stub.
writeExecutable(
join(fixtureDir, "scripts", "build-standalone-public-packages.mjs"),
"#!/usr/bin/env node\nprocess.exit(0);\n",
);
writeExecutable(join(fixtureDir, "scripts", "prepare-server-ui-dist.sh"), "#!/usr/bin/env bash\nexit 0\n");
writeExecutable(join(fixtureDir, "scripts", "build-npm.sh"), "#!/usr/bin/env bash\nexit 0\n");
// Step 4 verifies the CLI package version against the target canary version.
mkdirSync(join(fixtureDir, "cli"), { recursive: true });
writeFileSync(
join(fixtureDir, "cli", "package.json"),
JSON.stringify({ name: "paperclipai", version: "2026.710.0-canary.0" }),
);
// Step 5 cds into each previewed package directory.
for (let index = 1; index <= PREVIEW_PACKAGE_COUNT; index += 1) {
mkdirSync(join(fixtureDir, previewPackageDir(index)), { recursive: true });
}
}
function previewPackageDir(index) {
return `pkg-${String(index).padStart(2, "0")}`;
}
function previewPackageInfo() {
return Array.from({ length: PREVIEW_PACKAGE_COUNT }, (_, i) => {
const dir = previewPackageDir(i + 1);
return `${dir}\\t@fixture/${dir}\\t0.0.0\\n`;
}).join("");
}
test("canary dry-run previews every publish payload and replays logs in package order", () => {
const result = runRelease(
["canary", "--skip-verify", "--dry-run"],
{ FAKE_BUILD_OK: "1", FAKE_PACKAGE_INFO: previewPackageInfo() },
preparePreviewFixture,
);
assert.equal(result.status, 0, result.output);
const headerOffsets = [];
for (let index = 1; index <= PREVIEW_PACKAGE_COUNT; index += 1) {
const offset = result.output.indexOf(`--- ${previewPackageDir(index)} ---`);
assert.notEqual(offset, -1, `missing preview header for ${previewPackageDir(index)}`);
headerOffsets.push(offset);
}
assert.deepEqual(
headerOffsets,
[...headerOffsets].sort((a, b) => a - b),
"preview logs must replay in package order even though previews run concurrently",
);
const previews = result.output.match(/fixture publish preview in /g) ?? [];
assert.equal(previews.length, PREVIEW_PACKAGE_COUNT, "every package must be previewed exactly once");
assert.match(result.output, /Would create git tag/);
});
test("canary dry-run fails when any concurrent payload preview fails, after replaying every log", () => {
const result = runRelease(
["canary", "--skip-verify", "--dry-run"],
{
FAKE_BUILD_OK: "1",
FAKE_PACKAGE_INFO: previewPackageInfo(),
FAKE_FAIL_PUBLISH_DIR: previewPackageDir(7),
},
preparePreviewFixture,
);
assert.notEqual(result.status, 0, "a failed preview must fail the dry run");
assert.match(result.output, /one or more publish payload previews failed/);
assert.match(result.output, /fixture preview failure/);
for (let index = 1; index <= PREVIEW_PACKAGE_COUNT; index += 1) {
assert.match(
result.output,
new RegExp(`--- ${previewPackageDir(index)} ---`),
"every package's log must still replay so the failure is diagnosable",
);
}
assert.doesNotMatch(result.output, /Would create git tag/, "a failed preview must not reach tagging");
});
@@ -35,8 +35,8 @@ function dryRunJson(args) {
return JSON.parse(result.stdout);
}
const SHARD_COUNT = 5;
const SERIALIZED_SHARD_COUNT = 5;
const SHARD_COUNT = 12;
const SERIALIZED_SHARD_COUNT = 9;
test("the serialized shards form a complete, non-overlapping partition", () => {
@@ -174,44 +174,17 @@ test("the checked-in manifest loads and covers most of the current suite set", (
);
});
test("the measured chat integration cohort does not share a general-server shard", () => {
test("the chat integration suite keeps a measured duration for duration-aware fallbacks", () => {
// The PR and release matrices both run the chat suite in dedicated
// line-sharded lanes, but the plain general-server group (local full runs)
// still weighs it into the LPT partition; a median-fallback weight there
// would silently overload whichever shard receives it.
const chatSuite = "server/src/__tests__/chat-channels.integration.test.ts";
const durations = loadShardDurations(durationsManifest);
assert.ok(
Number.isFinite(durations[chatSuite]),
"the full chat cohort must have a measured duration, not the median fallback",
);
const unsharded = dryRunJson([
"--mode",
"general",
"--group",
"general-server",
"--shard-index",
"0",
"--shard-count",
"1",
]);
const shards = partitionGeneralServerSuites(
unsharded.selectedGeneralServerSuites,
SHARD_COUNT,
durations,
);
const chatShards = shards.filter((shard) => shard.files.includes(chatSuite));
assert.equal(
chatShards.length,
1,
"the full chat cohort must run exactly once",
);
assert.deepEqual(
chatShards[0].files,
[chatSuite],
"its measured cost must reserve one existing shard without other suites",
);
assert.deepEqual(
shards.flatMap((shard) => shard.files).sort(),
[...unsharded.selectedGeneralServerSuites].sort(),
"duration balancing must not omit or duplicate any general-server suite",
);
});
test("the checked-in serialized manifest loads and covers most of the current suite set", () => {
@@ -248,10 +221,12 @@ test("the real serialized shard partition is duration-balanced", () => {
});
test("the real shard partition is duration-balanced", () => {
// Mirrors the PR matrix: general-server-without-chat across SHARD_COUNT
// runners, with the chat suite carried by the dedicated general-chat lanes.
const durations = loadShardDurations(durationsManifest);
const fallback = defaultSuiteWeight(durations);
const shards = Array.from({ length: SHARD_COUNT }, (_, index) =>
dryRunJson(["--mode", "general", "--group", "general-server", "--shard-index", String(index), "--shard-count", String(SHARD_COUNT)]),
dryRunJson(["--mode", "general", "--group", "general-server-without-chat", "--shard-index", String(index), "--shard-count", String(SHARD_COUNT)]),
);
const totals = shards.map((shard) =>
@@ -259,8 +234,12 @@ test("the real shard partition is duration-balanced", () => {
);
const maxTotal = Math.max(...totals);
const minTotal = Math.min(...totals);
// LPT keeps the spread within the heaviest single suite; use that as the bound.
const heaviest = Math.max(...Object.values(durations));
// LPT keeps the spread within the heaviest single suite; use that as the
// bound. The chat suite runs in its own lanes, so exclude it here.
const chat = "server/src/__tests__/chat-channels.integration.test.ts";
const heaviest = Math.max(
...Object.entries(durations).filter(([file]) => file !== chat).map(([, ms]) => ms),
);
assert.ok(
maxTotal - minTotal <= heaviest,
`shard weight spread ${maxTotal - minTotal}ms exceeds heaviest suite ${heaviest}ms: ${totals.join(", ")}`,
@@ -268,20 +247,23 @@ test("the real shard partition is duration-balanced", () => {
});
test("release server shards plus the dedicated chat file cover the original server group exactly", () => {
const full = dryRunJson(["--mode", "general", "--group", "general-server", "--shard-index", "0", "--shard-count", "1"]);
const shards = Array.from({ length: 10 }, (_, index) => dryRunJson([
"--mode", "general", "--group", "general-server-without-chat",
"--shard-index", String(index), "--shard-count", "10",
]));
const files = shards.flatMap((shard) => shard.selectedGeneralServerSuites);
const chat = "server/src/__tests__/chat-channels.integration.test.ts";
assert.ok(!files.includes(chat));
assert.deepEqual([...files, chat].sort(), full.selectedGeneralServerSuites.sort());
assert.equal(new Set(files).size, files.length);
const defaultRun = dryRunJson([]);
assert.ok(defaultRun.generalServerSuiteCount === full.generalServerSuiteCount);
});
// 12 mirrors pr-trusted.yml, 10 mirrors release-verify.yml.
for (const withoutChatShardCount of [10, 12]) {
test(`${withoutChatShardCount} without-chat shards plus the dedicated chat file cover the original server group exactly`, () => {
const full = dryRunJson(["--mode", "general", "--group", "general-server", "--shard-index", "0", "--shard-count", "1"]);
const shards = Array.from({ length: withoutChatShardCount }, (_, index) => dryRunJson([
"--mode", "general", "--group", "general-server-without-chat",
"--shard-index", String(index), "--shard-count", String(withoutChatShardCount),
]));
const files = shards.flatMap((shard) => shard.selectedGeneralServerSuites);
const chat = "server/src/__tests__/chat-channels.integration.test.ts";
assert.ok(!files.includes(chat));
assert.deepEqual([...files, chat].sort(), full.selectedGeneralServerSuites.sort());
assert.equal(new Set(files).size, files.length);
const defaultRun = dryRunJson([]);
assert.ok(defaultRun.generalServerSuiteCount === full.generalServerSuiteCount);
});
}
const lineShardFile = path.join(repoRoot, "server/src/__tests__/chat-channels.integration.test.ts");
const caseAt = (line, name) => ({ name, file: lineShardFile, projectName: "@paperclipai/server", location: { line, column: 3 } });
+32 -16
View File
@@ -1,21 +1,37 @@
{
"$comment": "Per-spec wall-clock durations (ms) for the Playwright e2e lane, used by scripts/e2e-shard.mjs to balance specs across the PR shard matrix. Averaged from two real PR runs of .github/workflows/pr.yml (actions runs 29775184389 and 29769465682, 2026-07-20). The split Smoke Lab weights scale its prior 168s CI duration by the measured local before/after ratio and scenario count; refresh them from the next two real PR runs. Specs missing here get the median weight, so the manifest only needs occasional refreshes: pull the 'Run e2e tests' logs from a recent PR run and sum the Playwright list-reporter durations per spec file.",
"$comment": "Per-spec wall-clock durations (ms) for the Playwright e2e lane, used by scripts/e2e-shard.mjs to balance specs across the PR shard matrix. Averaged from two real PR runs of .github/workflows/pr.yml (actions runs 35036001734 and 35024948947, 2026-09-15) by diffing consecutive test-completion timestamps in the 'Run e2e tests' logs, attributing each gap to the finishing test's spec — that captures fixture and navigation time the list-reporter durations alone miss. Smoke Lab tests report under tests/e2e/smoke-lab.shared.ts; their time is mapped back to the p1-p4/p5-p7 spec by scenario tag. Specs missing here get the median weight, so the manifest only needs occasional refreshes: regenerate the same way when e2e shard steps drift apart. chat-adapters-ui and agent-chat were split 2026-09-15 into providers/messaging specs; their weights are the per-describe sums from the same two runs. agent-chat halves are weighted by their per-test sums scaled to the measured spec total.",
"unit": "ms",
"durations": {
"tests/e2e/app-not-connected.spec.ts": 26750,
"tests/e2e/application-delete-screenshot.spec.ts": 6750,
"tests/e2e/applications-crud.spec.ts": 16100,
"tests/e2e/apps-dark-mode-shots.spec.ts": 18000,
"tests/e2e/apps-prosumer-mcp-flow.spec.ts": 16800,
"tests/e2e/conference-room-typing-intro.spec.ts": 4900,
"tests/e2e/mcp-user-stories.spec.ts": 39000,
"tests/e2e/nux-phase4-screenshots.spec.ts": 19650,
"tests/e2e/onboarding.spec.ts": 3200,
"tests/e2e/pipelines-tutorial-flow.spec.ts": 30198,
"tests/e2e/planning-mode-visual-verification.spec.ts": 23250,
"tests/e2e/sidebar-takeover.spec.ts": 16650,
"tests/e2e/signoff-policy.spec.ts": 9696,
"tests/e2e/smoke-lab-p1-p4.spec.ts": 71000,
"tests/e2e/smoke-lab-p5-p7.spec.ts": 53000
"tests/e2e/acp-stop-continuation.spec.ts": 44152,
"tests/e2e/action-test-agent-picker.spec.ts": 10050,
"tests/e2e/agent-chat-projects.spec.ts": 159100,
"tests/e2e/agent-chat-sessions.spec.ts": 151200,
"tests/e2e/agentmail.spec.ts": 8944,
"tests/e2e/app-not-connected.spec.ts": 15684,
"tests/e2e/application-delete-screenshot.spec.ts": 3585,
"tests/e2e/applications-crud.spec.ts": 10924,
"tests/e2e/apps-dark-mode-shots.spec.ts": 27798,
"tests/e2e/apps-prosumer-mcp-flow.spec.ts": 19557,
"tests/e2e/archived-company-url.spec.ts": 11149,
"tests/e2e/board-attachment-receipts.spec.ts": 102785,
"tests/e2e/chat-adapters-ui-messaging.spec.ts": 241600,
"tests/e2e/chat-adapters-ui-providers.spec.ts": 200200,
"tests/e2e/composer-stop.spec.ts": 57673,
"tests/e2e/conference-room-typing-intro.spec.ts": 15196,
"tests/e2e/connection-intents.spec.ts": 14451,
"tests/e2e/connection-reviews.spec.ts": 141760,
"tests/e2e/legacy-failure-continuation.spec.ts": 74050,
"tests/e2e/mcp-user-stories.spec.ts": 38899,
"tests/e2e/nux-phase4-screenshots.spec.ts": 18776,
"tests/e2e/onboarding.spec.ts": 21295,
"tests/e2e/paused-composer.spec.ts": 33211,
"tests/e2e/pipelines-tutorial-flow.spec.ts": 30545,
"tests/e2e/planning-mode-visual-verification.spec.ts": 26193,
"tests/e2e/project-repositories.spec.ts": 21634,
"tests/e2e/runner-e2e-dashboard.spec.ts": 2028,
"tests/e2e/sidebar-takeover.spec.ts": 19034,
"tests/e2e/signoff-policy.spec.ts": 12121,
"tests/e2e/smoke-lab-p1-p4.spec.ts": 95529,
"tests/e2e/smoke-lab-p5-p7.spec.ts": 64328
}
}
File diff suppressed because it is too large. Load diff
+57 -11
View File
@@ -314,21 +314,67 @@ VERIFY_DELAY_SECONDS="${NPM_PUBLISH_VERIFY_DELAY_SECONDS:-5}"
release_info ""
if [ "$dry_run" = true ]; then
release_info "==> Step 5/7: Previewing publish payloads (--dry-run)..."
# Previews are independent per package (each works in its own temp dir and
# never touches npm), so run them concurrently in batches. Serial previews
# took ~78s of every PR Canary Dry Run job (run 35036001734, 2026-09-15).
# The real publish path below stays strictly serial.
preview_logs_dir="$(mktemp -d "${TMPDIR:-/tmp}/paperclip-release-previews.XXXXXX")"
preview_batch_size=8
preview_pids=()
preview_dirs=()
preview_logs=()
preview_failed=false
wait_for_preview_batch() {
# Guard the expansion: "${preview_pids[@]}" on an empty array trips
# `set -u` under bash < 4.4 (the macOS system bash).
if [ "${#preview_pids[@]}" -eq 0 ]; then
return 0
fi
for pid in "${preview_pids[@]}"; do
if ! wait "$pid"; then
preview_failed=true
fi
done
preview_pids=()
}
while IFS=$'\t' read -r pkg_dir _pkg_name _pkg_version; do
[ -z "$pkg_dir" ] && continue
release_info " --- $pkg_dir ---"
cd "$REPO_ROOT/$pkg_dir"
publish_tool="$(package_publish_tool)"
if [ "$publish_tool" = "npm" ]; then
publish_dir="$(mktemp -d "${TMPDIR:-/tmp}/paperclip-release-package.XXXXXX")"
node "$REPO_ROOT/scripts/prepare-bundled-package.mjs" "$REPO_ROOT/$pkg_dir" "$publish_dir"
cd "$publish_dir"
run_bundled_npm_pack pack --pack-destination "$publish_dir" 2>&1 | tail -3
rm -rf "$publish_dir"
else
pnpm publish --dry-run --no-git-checks --tag "$DIST_TAG" 2>&1 | tail -3
preview_log="$preview_logs_dir/$(printf '%03d' "${#preview_logs[@]}")-${pkg_dir//\//_}.log"
preview_dirs+=("$pkg_dir")
preview_logs+=("$preview_log")
(
set -euo pipefail
cd "$REPO_ROOT/$pkg_dir"
publish_tool="$(package_publish_tool)"
if [ "$publish_tool" = "npm" ]; then
publish_dir="$(mktemp -d "${TMPDIR:-/tmp}/paperclip-release-package.XXXXXX")"
node "$REPO_ROOT/scripts/prepare-bundled-package.mjs" "$REPO_ROOT/$pkg_dir" "$publish_dir"
cd "$publish_dir"
run_bundled_npm_pack pack --pack-destination "$publish_dir" 2>&1 | tail -3
rm -rf "$publish_dir"
else
pnpm publish --dry-run --no-git-checks --tag "$DIST_TAG" 2>&1 | tail -3
fi
) > "$preview_log" 2>&1 &
preview_pids+=($!)
if [ "$(( ${#preview_pids[@]} % preview_batch_size ))" -eq 0 ]; then
wait_for_preview_batch
fi
done <<< "$VERSIONED_PACKAGE_INFO"
wait_for_preview_batch
preview_index=0
for pkg_dir in "${preview_dirs[@]}"; do
release_info " --- $pkg_dir ---"
cat "${preview_logs[$preview_index]}"
preview_index=$((preview_index + 1))
done
rm -rf "$preview_logs_dir"
if [ "$preview_failed" = true ]; then
release_fail "one or more publish payload previews failed; see the package output above"
fi
release_info " [dry-run] Would create git tag $tag_name on $CURRENT_SHA"
else
release_info "==> Step 5/7: Publishing packages to npm..."
@@ -1,381 +1,16 @@
import path from "node:path";
import { createLocalAgentJwt } from "../../server/src/agent-auth-jwt";
import {
test,
expect,
type APIRequestContext,
type Page,
} from "@playwright/test";
import { expect, test } from "@playwright/test";
import { idle, json, send, setup } from "./agent-chat.shared";
test.use({ trace: "retain-on-failure" });
test.setTimeout(120_000);
async function json(response: Awaited<ReturnType<APIRequestContext["get"]>>) {
expect(response.ok(), `${response.status()} ${await response.text()}`).toBe(
true,
);
return response.json();
}
async function setup(request: APIRequestContext) {
const company = await json(
await request.post("/api/companies", {
data: { name: `Agent Chat ${Date.now()}` },
}),
);
const original = await json(
await request.get("/api/instance/settings/experimental"),
);
await json(
await request.patch("/api/instance/settings/experimental", {
data: { enableAgentChat: true, enableClassicTaskInterface: false },
}),
);
const agents = [];
for (const name of ["Alpha", "Beta", "Gamma", "Delta", "Epsilon", "Zeta"])
agents.push(
await json(
await request.post(`/api/companies/${company.id}/agents`, {
data: {
name,
adapterType: "process",
adapterConfig: {
command: process.execPath,
args: [path.resolve("tests/e2e/fixtures/agent-chat.mjs")],
graceSec: 1,
},
runtimeConfig: {
heartbeat: { enabled: false, wakeOnDemand: true },
},
},
}),
),
);
const agent = agents[0];
const chatPath = `/api/companies/${company.id}/chats/${agent.id}`;
const route = `/${company.issuePrefix}/chats/${agent.id}`;
return {
company,
agents,
agent,
chatPath,
route,
restore: async () => {
try {
const chat = await request.get(chatPath);
const history = chat.ok() ? await chat.json() : null;
const tasks = await json(
await request.get(`/api/companies/${company.id}/issues`),
);
const projects = await json(
await request.get(`/api/companies/${company.id}/projects`),
);
const ledger = await json(
await request.get(`/api/companies/${company.id}/heartbeat-runs`),
);
const documents = await Promise.all(
[...(history ? [history] : []), ...tasks].map(async (task: any) => ({
taskId: task.id,
documents: await json(
await request.get(`/api/issues/${task.id}/documents`),
),
})),
);
await test.info().attach("chat-persisted-state", {
contentType: "application/json",
body: Buffer.from(
JSON.stringify(
{
chat: history,
tasks,
projects,
documents,
comments: history
? await json(
await request.get(`/api/issues/${history.id}/comments`),
)
: [],
runs: ledger.map((run: any) => ({
id: run.id,
status: run.status,
startedAt: run.startedAt,
finishedAt: run.finishedAt,
sessionIdBefore: run.sessionIdBefore,
sessionIdAfter: run.sessionIdAfter,
issueId: run.contextSnapshot?.issueId,
generation:
run.contextSnapshot?.conversationSessionGeneration,
reset: run.contextSnapshot?.conversationReset,
})),
},
null,
2,
),
),
});
} finally {
const runs = await json(
await request.get(`/api/companies/${company.id}/live-runs`),
);
for (const run of runs)
await json(
await request.post(`/api/heartbeat-runs/${run.id}/cancel`),
);
await json(
await request.patch("/api/instance/settings/experimental", {
data: {
enableAgentChat: original.enableAgentChat,
enableClassicTaskInterface: original.enableClassicTaskInterface,
},
}),
);
}
},
};
}
async function send(page: Page, value: unknown) {
await page
.getByTestId("task-chat-composer-input")
.last()
.locator('[contenteditable="true"],textarea')
.first()
.fill(
typeof value === "string"
? value
: `fixture:${Buffer.from(JSON.stringify(value)).toString("base64url")}`,
);
await page.getByTestId("task-chat-composer-send").last().click();
}
async function idle(
request: APIRequestContext,
chatPath: string,
minimumReplies = 1,
) {
let issue: any;
await expect
.poll(
async () => {
issue = await json(await request.get(chatPath));
if (!issue) return false;
const replies = await json(
await request.get(`/api/issues/${issue.id}/comments`),
);
const live = await json(
await request.get(`/api/issues/${issue.id}/live-runs`),
);
return (
live.length === 0 &&
issue.conversationState === "waiting" &&
issue.status === "in_review" &&
replies.filter((c: any) => c.authorAgentId).length >= minimumReplies
);
},
{ timeout: 60_000, intervals: [100, 250, 500] },
)
.toBe(true);
return issue;
}
test("chat first open is read-only; concurrent first sends and retries share one task", async ({
page,
context,
request,
}) => {
const f = await setup(request);
try {
await page.goto(f.route);
await expect(page.getByTestId("task-chat-composer-input")).toBeVisible();
expect(await json(await request.get(f.chatPath))).toBeNull();
expect(
await json(
await request.get(`/api/companies/${f.company.id}/heartbeat-runs`),
),
).toHaveLength(0);
const other = await context.newPage();
await other.goto(f.route);
await Promise.all([send(page, "Same first message"), send(other, "Same first message")]);
const issue = await idle(request, f.chatPath, 2);
const initialComments = await json(await request.get(`/api/issues/${issue.id}/comments`));
expect(initialComments.filter((comment: any) => !comment.authorAgentId && comment.body === "Same first message")).toHaveLength(2);
const resolved = await Promise.all(
Array.from({ length: 4 }, () =>
request.post(f.chatPath, { data: {} }).then(json),
),
);
expect(new Set(resolved.map((row) => row.id))).toEqual(new Set([issue.id]));
const body = {
body: "Retry once",
clientRequestId: "00000000-0000-4000-8000-000000000001",
};
const replies = await Promise.all([
request
.post(`/api/issues/${issue.id}/comments`, { data: body })
.then(json),
request
.post(`/api/issues/${issue.id}/comments`, { data: body })
.then(json),
]);
expect(replies[0].id).toBe(replies[1].id);
await idle(request, f.chatPath, 3);
await page.reload();
await expect(
page.getByText("Reply generation 0: Retry once", { exact: true }),
).toBeVisible();
expect(
await json(await request.get(`/api/companies/${f.company.id}/issues`)),
).toHaveLength(0);
const dashboard = await json(
await request.get(`/api/companies/${f.company.id}/dashboard`),
);
expect(dashboard.tasks).toEqual({
open: 0,
inProgress: 0,
blocked: 0,
done: 0,
});
const count = (
await json(
await request.get(`/api/companies/${f.company.id}/heartbeat-runs`),
)
).length;
await page.goto(`/${f.company.issuePrefix}/issues/${issue.identifier}`);
await page.goto(f.route);
expect(
await json(
await request.get(`/api/companies/${f.company.id}/heartbeat-runs`),
),
).toHaveLength(count);
await other.close();
} finally {
await f.restore();
}
});
test("feature flag blocks new sends and resets while preserving existing history", async ({
page,
request,
}) => {
const f = await setup(request);
try {
await page.goto(f.route);
await send(page, "Visible history");
const issue = await idle(request, f.chatPath);
await json(
await request.patch("/api/instance/settings/experimental", {
data: { enableAgentChat: false },
}),
);
for (const body of ["blocked message", "/new"])
expect(
(
await request.post(`/api/issues/${issue.id}/comments`, {
data: { body },
})
).status(),
).toBe(404);
await page.reload();
await expect(page.getByText(/Agent Chat is disabled/)).toBeVisible();
expect(
(await json(await request.get(`/api/issues/${issue.id}/comments`))).some(
(c: any) => c.body.includes("Visible history"),
),
).toBe(true);
expect((await request.post(f.chatPath, { data: {} })).status()).toBe(404);
} finally {
await f.restore();
}
});
test("Stop then queued /new resets unpause the conversation without losing history", async ({
page,
request,
}) => {
const f = await setup(request);
try {
await page.goto(f.route);
await send(page, { action: "hold" });
await expect
.poll(async () => {
const chat = await json(await request.get(f.chatPath));
return (
chat &&
(
await json(await request.get(`/api/issues/${chat.id}/comments`))
).some(
(c: any) => c.body === "Provider is streaming and ready to stop.",
)
);
})
.toBe(true);
const issue = await json(await request.get(f.chatPath));
const active = (
await json(await request.get(`/api/issues/${issue.id}/live-runs`))
)[0];
await page.getByTestId("task-chat-composer-stop").click();
await expect
.poll(
async () =>
(await json(await request.get(`/api/heartbeat-runs/${active.id}`)))
.status,
)
.toBe("cancelled");
const staleToken = createLocalAgentJwt(
f.agent.id,
f.company.id,
"process",
active.id,
);
expect(staleToken).toBeTruthy();
const late = await request.post(`/api/issues/${issue.id}/comments`, {
headers: { Authorization: `Bearer ${staleToken}` },
data: { body: "Forbidden late response" },
});
expect([403, 409]).toContain(late.status());
const mutation = await request.post(
`/api/companies/${f.company.id}/projects`,
{
headers: { Authorization: `Bearer ${staleToken}` },
data: { name: "Cancelled project" },
},
);
expect([403, 409]).toContain(mutation.status());
expect(
await json(await request.get(`/api/companies/${f.company.id}/projects`)),
).toHaveLength(0);
await send(page, "/new");
await send(page, "/new");
await send(page, "Fresh followup");
await idle(request, f.chatPath, 2);
const fresh = await json(await request.get(f.chatPath));
expect(fresh.id).toBe(issue.id);
expect(fresh.conversationSessionGeneration).toBe(2);
await expect(
page.getByText("Reply generation 2: Fresh followup", { exact: true }),
).toBeVisible();
await page.reload();
await expect(page.getByText("New session", { exact: true })).toHaveCount(2);
await expect(
page.getByRole("separator", { name: "Run completed", exact: true }),
).toHaveCount(0);
const runs = await json(
await request.get(`/api/companies/${f.company.id}/heartbeat-runs`),
);
const detailed = await Promise.all(
runs.map((run: any) =>
request.get(`/api/heartbeat-runs/${run.id}`).then(json),
),
);
expect(
detailed.filter((run) => run.contextSnapshot?.conversationReset),
).toHaveLength(2);
expect(
detailed
.filter((run) => run.contextSnapshot?.conversationReset)
.every((run) => !run.sessionIdAfter),
).toBe(true);
} finally {
await f.restore();
}
});
/**
* Agent chat project and content coverage: project cards, split handoff,
* uploads, Ask/Plan modes, repository selection, plan approval, and shared
* history. Shared fixtures live in ./agent-chat.shared.ts; session lifecycle
* flows run in agent-chat-sessions.spec.ts.
*/
for (const direct of [false, true])
test(`project card through ${direct ? "direct API" : "dedicated tool"} persists and deduplicates retries`, async ({
page,
@@ -554,93 +189,6 @@ for (const bad of [
}
});
test("sidebar discovery, stars, recent agents, configuration links, and drafts survive switching", async ({
page,
request,
}) => {
const f = await setup(request);
try {
await page.goto(`/${f.company.issuePrefix}/dashboard`);
const nav = page.getByRole("navigation");
const chatLinks = nav.locator('a[href*="/chats/"]');
await expect(chatLinks).toHaveText(["Alpha"]);
const compose = nav.getByRole("button", { name: "Chat with an agent", exact: true });
await compose.click();
const picker = page.getByRole("dialog", { name: "Chat with an agent", exact: true });
await expect(picker.getByRole("option")).toHaveCount(6);
await picker.getByRole("combobox").fill("Beta");
await picker.getByRole("combobox").press("Enter");
await expect(picker).not.toBeVisible();
await expect(page.getByRole("link", { name: "Configure Beta", exact: true })).toBeVisible();
expect(await json(await request.get(`/api/companies/${f.company.id}/chats/${f.agents[1].id}`))).toBeNull();
for (const agent of f.agents) {
await page.goto(`/${f.company.issuePrefix}/chats/${agent.id}`);
await expect(page.getByTestId("task-chat-composer-input")).toBeVisible();
}
await expect(chatLinks).toHaveText(["Alpha", "Zeta", "Epsilon", "Delta", "Gamma"]);
const star = page.getByRole("button", { name: "Star Zeta", exact: true });
await page.getByTestId("task-chat-composer-input").click();
await expect(star).toHaveCSS("opacity", "0");
await star.focus();
await page.keyboard.press("Tab");
await page.keyboard.press("Shift+Tab");
await expect(star).toHaveCSS("opacity", "1");
await star.click();
await page.goto(f.route);
await page.getByRole("button", { name: "Star Alpha", exact: true }).click();
await expect(nav.locator('a[href*="/chats/"]').first()).toHaveText("Alpha");
await expect(nav.locator('a[href*="/chats/"]').nth(1)).toHaveText("Zeta");
const editor = page
.getByTestId("task-chat-composer-input")
.locator('[contenteditable="true"]');
await editor.fill("Unsent draft for Alpha");
await nav.getByRole("link", { name: "Zeta", exact: true }).click();
await expect(editor).toHaveText("");
await nav.getByRole("link", { name: "Alpha", exact: true }).click();
await expect(editor).toContainText("Unsent draft for Alpha");
const recent = await page.evaluate(() =>
Object.fromEntries(
Object.entries(localStorage).filter(([key]) =>
key.startsWith("paperclip.recentAgentChats:"),
),
),
);
await page.getByRole("link", { name: /Configure Alpha/ }).click();
await expect(page).toHaveURL(/\/agents\/.*\/runtime/);
const backgroundPath = `/api/companies/${f.company.id}/chats/${f.agents[1].id}`;
const background = await json(
await request.post(backgroundPath, { data: {} }),
);
await json(
await request.post(`/api/issues/${background.id}/comments`, {
data: {
body: "Background activity",
clientRequestId: "00000000-0000-4000-8000-000000000099",
},
}),
);
await idle(request, backgroundPath);
expect(
await page.evaluate(() =>
Object.fromEntries(
Object.entries(localStorage).filter(([key]) =>
key.startsWith("paperclip.recentAgentChats:"),
),
),
),
).toEqual(recent);
await compose.click();
await expect(picker.getByRole("combobox")).toHaveValue("");
await picker.getByRole("combobox").fill("Beta");
await picker.getByRole("combobox").press("Enter");
await expect(picker).not.toBeVisible();
await expect(page.getByRole("link", { name: "Configure Beta", exact: true })).toBeVisible();
await expect(page.getByText("Background activity", { exact: true })).toBeVisible();
} finally {
await f.restore();
}
});
test("first upload creates the chat without invoking its agent; shared attachments persist", async ({
page,
request,
+303
View File
@@ -0,0 +1,303 @@
import { expect, test } from "@playwright/test";
import { createLocalAgentJwt } from "../../server/src/agent-auth-jwt";
import { idle, json, send, setup } from "./agent-chat.shared";
test.use({ trace: "retain-on-failure" });
test.setTimeout(120_000);
/**
* Agent chat session lifecycle coverage: first-open semantics, the feature
* flag, Stop + queued /new resets, and sidebar discovery. Shared fixtures
* live in ./agent-chat.shared.ts; project, attachment, and history flows run
* in agent-chat-projects.spec.ts.
*/
test("chat first open is read-only; concurrent first sends and retries share one task", async ({
page,
context,
request,
}) => {
const f = await setup(request);
try {
await page.goto(f.route);
await expect(page.getByTestId("task-chat-composer-input")).toBeVisible();
expect(await json(await request.get(f.chatPath))).toBeNull();
expect(
await json(
await request.get(`/api/companies/${f.company.id}/heartbeat-runs`),
),
).toHaveLength(0);
const other = await context.newPage();
await other.goto(f.route);
await Promise.all([send(page, "Same first message"), send(other, "Same first message")]);
const issue = await idle(request, f.chatPath, 2);
const initialComments = await json(await request.get(`/api/issues/${issue.id}/comments`));
expect(initialComments.filter((comment: any) => !comment.authorAgentId && comment.body === "Same first message")).toHaveLength(2);
const resolved = await Promise.all(
Array.from({ length: 4 }, () =>
request.post(f.chatPath, { data: {} }).then(json),
),
);
expect(new Set(resolved.map((row) => row.id))).toEqual(new Set([issue.id]));
const body = {
body: "Retry once",
clientRequestId: "00000000-0000-4000-8000-000000000001",
};
const replies = await Promise.all([
request
.post(`/api/issues/${issue.id}/comments`, { data: body })
.then(json),
request
.post(`/api/issues/${issue.id}/comments`, { data: body })
.then(json),
]);
expect(replies[0].id).toBe(replies[1].id);
await idle(request, f.chatPath, 3);
await page.reload();
await expect(
page.getByText("Reply generation 0: Retry once", { exact: true }),
).toBeVisible();
expect(
await json(await request.get(`/api/companies/${f.company.id}/issues`)),
).toHaveLength(0);
const dashboard = await json(
await request.get(`/api/companies/${f.company.id}/dashboard`),
);
expect(dashboard.tasks).toEqual({
open: 0,
inProgress: 0,
blocked: 0,
done: 0,
});
const count = (
await json(
await request.get(`/api/companies/${f.company.id}/heartbeat-runs`),
)
).length;
await page.goto(`/${f.company.issuePrefix}/issues/${issue.identifier}`);
await page.goto(f.route);
expect(
await json(
await request.get(`/api/companies/${f.company.id}/heartbeat-runs`),
),
).toHaveLength(count);
await other.close();
} finally {
await f.restore();
}
});
test("feature flag blocks new sends and resets while preserving existing history", async ({
page,
request,
}) => {
const f = await setup(request);
try {
await page.goto(f.route);
await send(page, "Visible history");
const issue = await idle(request, f.chatPath);
await json(
await request.patch("/api/instance/settings/experimental", {
data: { enableAgentChat: false },
}),
);
for (const body of ["blocked message", "/new"])
expect(
(
await request.post(`/api/issues/${issue.id}/comments`, {
data: { body },
})
).status(),
).toBe(404);
await page.reload();
await expect(page.getByText(/Agent Chat is disabled/)).toBeVisible();
expect(
(await json(await request.get(`/api/issues/${issue.id}/comments`))).some(
(c: any) => c.body.includes("Visible history"),
),
).toBe(true);
expect((await request.post(f.chatPath, { data: {} })).status()).toBe(404);
} finally {
await f.restore();
}
});
test("Stop then queued /new resets unpause the conversation without losing history", async ({
page,
request,
}) => {
const f = await setup(request);
try {
await page.goto(f.route);
await send(page, { action: "hold" });
await expect
.poll(async () => {
const chat = await json(await request.get(f.chatPath));
return (
chat &&
(
await json(await request.get(`/api/issues/${chat.id}/comments`))
).some(
(c: any) => c.body === "Provider is streaming and ready to stop.",
)
);
})
.toBe(true);
const issue = await json(await request.get(f.chatPath));
const active = (
await json(await request.get(`/api/issues/${issue.id}/live-runs`))
)[0];
await page.getByTestId("task-chat-composer-stop").click();
await expect
.poll(
async () =>
(await json(await request.get(`/api/heartbeat-runs/${active.id}`)))
.status,
)
.toBe("cancelled");
const staleToken = createLocalAgentJwt(
f.agent.id,
f.company.id,
"process",
active.id,
);
expect(staleToken).toBeTruthy();
const late = await request.post(`/api/issues/${issue.id}/comments`, {
headers: { Authorization: `Bearer ${staleToken}` },
data: { body: "Forbidden late response" },
});
expect([403, 409]).toContain(late.status());
const mutation = await request.post(
`/api/companies/${f.company.id}/projects`,
{
headers: { Authorization: `Bearer ${staleToken}` },
data: { name: "Cancelled project" },
},
);
expect([403, 409]).toContain(mutation.status());
expect(
await json(await request.get(`/api/companies/${f.company.id}/projects`)),
).toHaveLength(0);
await send(page, "/new");
await send(page, "/new");
await send(page, "Fresh followup");
await idle(request, f.chatPath, 2);
const fresh = await json(await request.get(f.chatPath));
expect(fresh.id).toBe(issue.id);
expect(fresh.conversationSessionGeneration).toBe(2);
await expect(
page.getByText("Reply generation 2: Fresh followup", { exact: true }),
).toBeVisible();
await page.reload();
await expect(page.getByText("New session", { exact: true })).toHaveCount(2);
await expect(
page.getByRole("separator", { name: "Run completed", exact: true }),
).toHaveCount(0);
const runs = await json(
await request.get(`/api/companies/${f.company.id}/heartbeat-runs`),
);
const detailed = await Promise.all(
runs.map((run: any) =>
request.get(`/api/heartbeat-runs/${run.id}`).then(json),
),
);
expect(
detailed.filter((run) => run.contextSnapshot?.conversationReset),
).toHaveLength(2);
expect(
detailed
.filter((run) => run.contextSnapshot?.conversationReset)
.every((run) => !run.sessionIdAfter),
).toBe(true);
} finally {
await f.restore();
}
});
test("sidebar discovery, stars, recent agents, configuration links, and drafts survive switching", async ({
page,
request,
}) => {
const f = await setup(request);
try {
await page.goto(`/${f.company.issuePrefix}/dashboard`);
const nav = page.getByRole("navigation");
const chatLinks = nav.locator('a[href*="/chats/"]');
await expect(chatLinks).toHaveText(["Alpha"]);
const compose = nav.getByRole("button", { name: "Chat with an agent", exact: true });
await compose.click();
const picker = page.getByRole("dialog", { name: "Chat with an agent", exact: true });
await expect(picker.getByRole("option")).toHaveCount(6);
await picker.getByRole("combobox").fill("Beta");
await picker.getByRole("combobox").press("Enter");
await expect(picker).not.toBeVisible();
await expect(page.getByRole("link", { name: "Configure Beta", exact: true })).toBeVisible();
expect(await json(await request.get(`/api/companies/${f.company.id}/chats/${f.agents[1].id}`))).toBeNull();
for (const agent of f.agents) {
await page.goto(`/${f.company.issuePrefix}/chats/${agent.id}`);
await expect(page.getByTestId("task-chat-composer-input")).toBeVisible();
}
await expect(chatLinks).toHaveText(["Alpha", "Zeta", "Epsilon", "Delta", "Gamma"]);
const star = page.getByRole("button", { name: "Star Zeta", exact: true });
await page.getByTestId("task-chat-composer-input").click();
await expect(star).toHaveCSS("opacity", "0");
await star.focus();
await page.keyboard.press("Tab");
await page.keyboard.press("Shift+Tab");
await expect(star).toHaveCSS("opacity", "1");
await star.click();
await page.goto(f.route);
await page.getByRole("button", { name: "Star Alpha", exact: true }).click();
await expect(nav.locator('a[href*="/chats/"]').first()).toHaveText("Alpha");
await expect(nav.locator('a[href*="/chats/"]').nth(1)).toHaveText("Zeta");
const editor = page
.getByTestId("task-chat-composer-input")
.locator('[contenteditable="true"]');
await editor.fill("Unsent draft for Alpha");
await nav.getByRole("link", { name: "Zeta", exact: true }).click();
await expect(editor).toHaveText("");
await nav.getByRole("link", { name: "Alpha", exact: true }).click();
await expect(editor).toContainText("Unsent draft for Alpha");
const recent = await page.evaluate(() =>
Object.fromEntries(
Object.entries(localStorage).filter(([key]) =>
key.startsWith("paperclip.recentAgentChats:"),
),
),
);
await page.getByRole("link", { name: /Configure Alpha/ }).click();
await expect(page).toHaveURL(/\/agents\/.*\/runtime/);
const backgroundPath = `/api/companies/${f.company.id}/chats/${f.agents[1].id}`;
const background = await json(
await request.post(backgroundPath, { data: {} }),
);
await json(
await request.post(`/api/issues/${background.id}/comments`, {
data: {
body: "Background activity",
clientRequestId: "00000000-0000-4000-8000-000000000099",
},
}),
);
await idle(request, backgroundPath);
expect(
await page.evaluate(() =>
Object.fromEntries(
Object.entries(localStorage).filter(([key]) =>
key.startsWith("paperclip.recentAgentChats:"),
),
),
),
).toEqual(recent);
await compose.click();
await expect(picker.getByRole("combobox")).toHaveValue("");
await picker.getByRole("combobox").fill("Beta");
await picker.getByRole("combobox").press("Enter");
await expect(picker).not.toBeVisible();
await expect(page.getByRole("link", { name: "Configure Beta", exact: true })).toBeVisible();
await expect(page.getByText("Background activity", { exact: true })).toBeVisible();
} finally {
await f.restore();
}
});
+176
View File
@@ -0,0 +1,176 @@
import path from "node:path";
import { createLocalAgentJwt } from "../../server/src/agent-auth-jwt";
import {
test,
expect,
type APIRequestContext,
type Page,
} from "@playwright/test";
// Per-file Playwright configuration (test.use, test.setTimeout) must stay in
// the spec files themselves: calling test.use() from a shared helper makes
// Playwright fail the whole worker with "inconsistent test.use() options".
export async function json(response: Awaited<ReturnType<APIRequestContext["get"]>>) {
expect(response.ok(), `${response.status()} ${await response.text()}`).toBe(
true,
);
return response.json();
}
export async function setup(request: APIRequestContext) {
const company = await json(
await request.post("/api/companies", {
data: { name: `Agent Chat ${Date.now()}` },
}),
);
const original = await json(
await request.get("/api/instance/settings/experimental"),
);
await json(
await request.patch("/api/instance/settings/experimental", {
data: { enableAgentChat: true, enableClassicTaskInterface: false },
}),
);
const agents = [];
for (const name of ["Alpha", "Beta", "Gamma", "Delta", "Epsilon", "Zeta"])
agents.push(
await json(
await request.post(`/api/companies/${company.id}/agents`, {
data: {
name,
adapterType: "process",
adapterConfig: {
command: process.execPath,
args: [path.resolve("tests/e2e/fixtures/agent-chat.mjs")],
graceSec: 1,
},
runtimeConfig: {
heartbeat: { enabled: false, wakeOnDemand: true },
},
},
}),
),
);
const agent = agents[0];
const chatPath = `/api/companies/${company.id}/chats/${agent.id}`;
const route = `/${company.issuePrefix}/chats/${agent.id}`;
return {
company,
agents,
agent,
chatPath,
route,
restore: async () => {
try {
const chat = await request.get(chatPath);
const history = chat.ok() ? await chat.json() : null;
const tasks = await json(
await request.get(`/api/companies/${company.id}/issues`),
);
const projects = await json(
await request.get(`/api/companies/${company.id}/projects`),
);
const ledger = await json(
await request.get(`/api/companies/${company.id}/heartbeat-runs`),
);
const documents = await Promise.all(
[...(history ? [history] : []), ...tasks].map(async (task: any) => ({
taskId: task.id,
documents: await json(
await request.get(`/api/issues/${task.id}/documents`),
),
})),
);
await test.info().attach("chat-persisted-state", {
contentType: "application/json",
body: Buffer.from(
JSON.stringify(
{
chat: history,
tasks,
projects,
documents,
comments: history
? await json(
await request.get(`/api/issues/${history.id}/comments`),
)
: [],
runs: ledger.map((run: any) => ({
id: run.id,
status: run.status,
startedAt: run.startedAt,
finishedAt: run.finishedAt,
sessionIdBefore: run.sessionIdBefore,
sessionIdAfter: run.sessionIdAfter,
issueId: run.contextSnapshot?.issueId,
generation:
run.contextSnapshot?.conversationSessionGeneration,
reset: run.contextSnapshot?.conversationReset,
})),
},
null,
2,
),
),
});
} finally {
const runs = await json(
await request.get(`/api/companies/${company.id}/live-runs`),
);
for (const run of runs)
await json(
await request.post(`/api/heartbeat-runs/${run.id}/cancel`),
);
await json(
await request.patch("/api/instance/settings/experimental", {
data: {
enableAgentChat: original.enableAgentChat,
enableClassicTaskInterface: original.enableClassicTaskInterface,
},
}),
);
}
},
};
}
export async function send(page: Page, value: unknown) {
await page
.getByTestId("task-chat-composer-input")
.last()
.locator('[contenteditable="true"],textarea')
.first()
.fill(
typeof value === "string"
? value
: `fixture:${Buffer.from(JSON.stringify(value)).toString("base64url")}`,
);
await page.getByTestId("task-chat-composer-send").last().click();
}
export async function idle(
request: APIRequestContext,
chatPath: string,
minimumReplies = 1,
) {
let issue: any;
await expect
.poll(
async () => {
issue = await json(await request.get(chatPath));
if (!issue) return false;
const replies = await json(
await request.get(`/api/issues/${issue.id}/comments`),
);
const live = await json(
await request.get(`/api/issues/${issue.id}/live-runs`),
);
return (
live.length === 0 &&
issue.conversationState === "waiting" &&
issue.status === "in_review" &&
replies.filter((c: any) => c.authorAgentId).length >= minimumReplies
);
},
{ timeout: 60_000, intervals: [100, 250, 500] },
)
.toBe(true);
return issue;
}
File diff suppressed because it is too large. Load diff
@@ -0,0 +1,833 @@
import { expect, test, type Page } from "@playwright/test";
import {
expectMinimumProviderSetup,
expectProviderTryInstructions,
expectSetupRail,
fillProviderSetup,
installChatControlPlaneMock,
seedCompanyAndAgent,
selectMaya,
PROVIDER_LIFECYCLE_COPY,
PROVIDERS,
expectedCredentialKeys,
type ProviderCase,
type Seed,
} from "./chat-adapters-ui.shared";
/**
* Native chat-connector provider setup coverage: adapter install/lifecycle
* flows and the iMessage Photon variant. Shared fixtures and the
* chat-control-plane mock live in ./chat-adapters-ui.shared.ts; the
* messaging-flow describes run in chat-adapters-ui-messaging.spec.ts.
*/
test.describe.serial("native chat adapter UI", () => {
test.setTimeout(180_000);
let seed: Seed;
test.beforeAll(async ({ request }) => {
seed = await seedCompanyAndAgent(request);
});
test("GitHub: the default-off gate keeps direct tool setup and fences chat routes", async ({
page,
}) => {
const github = PROVIDERS.find(
(provider) => provider.provider === "github",
)!;
const mock = await installChatControlPlaneMock(page, github, seed, {
enableChatConnectors: false,
});
await page.goto(`/${seed.prefix}/apps`);
await expect(page.getByRole("heading", { name: "Connectors" })).toBeVisible(
{ timeout: 30_000 },
);
const connector = page.locator(
`[role="listitem"][data-app-slug="${github.slug}"]`,
);
await expect(connector).toBeVisible();
await connector.getByRole("button", { name: "Connect GitHub" }).click();
await expect(page).toHaveURL(/\/apps\/connect\?/);
expect(new URL(page.url()).searchParams.get("source")).toBe("github");
await expect(
page.getByRole("heading", { name: "Connect GitHub as" }),
).toBeVisible();
await expect(
page.getByRole("heading", { name: "Choose how to connect" }),
).toHaveCount(0);
await expect(
page.getByText("Chat with an agent", { exact: true }),
).toHaveCount(0);
await page.goto(`/${seed.prefix}/apps/chat/connect?provider=github`);
await expect(page).toHaveURL(new RegExp(`/${seed.prefix}/apps$`));
await expect(
page.getByRole("heading", { name: "Connectors" }),
).toBeVisible();
await page.goto(`/${seed.prefix}/apps/chat/endpoint-github/settings`);
await expect(page).toHaveURL(new RegExp(`/${seed.prefix}/apps$`));
await expect.poll(() => mock.chatEndpointListReads).toBe(0);
expect(mock.createdWithAgentId).toBeNull();
});
for (const enabled of [false, true]) {
test(`Agent Channels: one heading and current experiment gate (${enabled})`, async ({
page,
}) => {
const github = PROVIDERS.find(
(provider) => provider.provider === "github",
)!;
const mock = await installChatControlPlaneMock(page, github, seed, {
enableChatConnectors: enabled,
});
await page.goto(`/${seed.prefix}/agents/${seed.agentId}/channels`);
const channelsHeading = page.getByRole("heading", {
name: "Channels",
exact: true,
});
if (enabled) {
await expect(page).toHaveURL(/\/channels$/);
await expect(channelsHeading).toHaveCount(1);
await expect(channelsHeading).toBeVisible();
await expect(
page.getByRole("link", { name: "Connect a channel" }),
).toBeVisible();
await expect.poll(() => mock.chatEndpointListReads).toBeGreaterThan(0);
} else {
await expect(page).toHaveURL(/\/overview$/);
await expect(channelsHeading).toHaveCount(0);
await expect(
page.getByRole("link", { name: "Channels", exact: true }),
).toHaveCount(0);
expect(mock.chatEndpointListReads).toBe(0);
}
});
}
for (const provider of PROVIDERS) {
test(`${provider.name}: catalog, setup, and connection management tabs`, async ({
page,
}) => {
const mock = await installChatControlPlaneMock(page, provider, seed, {
enableChatConnectors: true,
});
await page.goto(`/${seed.prefix}/apps`);
await expect(
page.getByRole("heading", { name: "Connectors" }),
).toBeVisible({ timeout: 30_000 });
const connector = page.locator(
`[role="listitem"][data-app-slug="${provider.slug}"]`,
);
await expect(connector).toBeVisible({ timeout: 30_000 });
await connector
.getByRole("button", { name: `Connect ${provider.name}` })
.click();
if (provider.chatAndTool) {
await expect(
page.getByRole("heading", { name: "Choose how to connect" }),
).toBeVisible();
await expect(
page.getByRole("button", { name: /Chat with an agent/ }),
).toBeVisible();
await expect(
page.getByRole("button", {
name: /Use this connection as an agent tool/,
}),
).toBeVisible();
const chatSetupUrl = page.url();
const toolHref = new URL(chatSetupUrl).searchParams.get("toolHref");
expect(toolHref).toBeTruthy();
await page
.getByRole("button", {
name: /Use this connection as an agent tool/,
})
.click();
await expect(page).toHaveURL(/\/apps\/connect\?/);
expect(new URL(page.url()).searchParams.get("source")).toBe(
provider.provider,
);
if (provider.provider === "github") {
await expect(
page.getByRole("heading", { name: "Connect GitHub as" }),
).toBeVisible();
await expect(
page.getByText("Chat with an agent", { exact: true }),
).toHaveCount(0);
}
await page.goto(chatSetupUrl);
await expect(
page.getByRole("heading", { name: "Choose how to connect" }),
).toBeVisible();
await page.getByRole("button", { name: /Chat with an agent/ }).click();
} else {
const chatOnlySetupUrl = new URL(page.url());
expect(chatOnlySetupUrl.searchParams.get("purpose")).toBe("chat");
expect(chatOnlySetupUrl.searchParams.get("toolHref")).toBeNull();
await expect(
page.getByRole("heading", { name: "Choose how to connect" }),
).toHaveCount(0);
}
await expect(
page.getByRole("heading", {
name: "Which agent do you want to chat with?",
}),
).toBeVisible();
await expectSetupRail(page);
await selectMaya(page);
await expect.poll(() => mock.createdWithAgentId).toBe(seed.agentId);
expect(mock.createdWithAgentId).not.toBe(seed.otherAgentId);
await expect(
page.getByRole("button", { name: "Choose an active agent" }),
).toHaveCount(0);
await expect(
page.getByRole("heading", { name: provider.setupHeading }),
).toBeVisible();
await expect(
page.getByText(PROVIDER_LIFECYCLE_COPY[provider.provider].reconnect, {
exact: false,
}),
).toHaveCount(0);
await expectSetupRail(page);
await expectMinimumProviderSetup(page, provider);
await fillProviderSetup(page, provider);
if (provider.provider === "github") {
expect(mock.githubPrivateKeyMatchedFile).toBe(true);
expect(mock.githubPrivateKeyMatchedPaste).toBe(true);
expect(mock.setupAttempts).toBe(2);
}
if (provider.provider === "telegram") {
const submittedToken = "123456:e2e-redacted";
// The next step intentionally has its own identity-readiness alert.
// Assert that this failed setup attempt clears, not that all alerts
// disappear during the transition between two valid wizard states.
const setupAlert = page.getByRole("alert").filter({
has: page.getByText("Connection failed", { exact: true }),
});
await expect(setupAlert).toContainText("Connection failed");
await expect(setupAlert).toContainText(
"Telegram rejected bot token [redacted]. Confirm the token in BotFather and try again.",
);
await expect(page.locator("body")).not.toContainText(submittedToken);
const tokenInput = page.getByLabel("Bot token");
await expect(tokenInput).toHaveAttribute("type", "password");
await expect(tokenInput).toHaveValue(submittedToken);
await tokenInput.focus();
await tokenInput.press("Tab");
await expect(setupAlert).toBeVisible();
await page.getByRole("button", { name: provider.setupButton }).click();
await expect(
page.getByRole("heading", { name: `Try Maya in ${provider.name}` }),
).toBeVisible();
await expect(setupAlert).toHaveCount(0);
expect(mock.setupAttempts).toBe(2);
}
if (provider.provider === "slack") {
await expect(
page.getByRole("heading", { name: "Finish Slack setup" }),
).toBeVisible();
await expect(
page.getByText(
`https://paperclip.example.test/api/chat-webhooks/public-slack/slack`,
{ exact: true },
),
).toBeVisible();
await expect(
page.getByText("/maya-public", { exact: true }),
).toBeVisible();
await expect(
page.getByText(
/Slack's bare \/status command is not a Paperclip control/,
),
).toBeVisible();
const saveChangesStep = page
.getByRole("listitem")
.filter({ hasText: "Save Changes" });
await expect(saveChangesStep).toHaveCount(1);
await expect(
saveChangesStep.locator("..").getByRole("listitem"),
).toHaveCount(1);
await expect(saveChangesStep).toHaveText(
"Return to App Manifest in Slack and click Save Changes. The copied manifest already contains the event, interaction, and slash-command URLs. Slack verifies the Events URL when you save; Paperclip records Interactivity and slash command health only after each signed callback is observed.",
);
for (const removedManualStep of [
"Event Subscriptions",
"Interactivity & Shortcuts",
"Slash Commands",
]) {
await expect(
page.getByText(removedManualStep, { exact: true }),
).toHaveCount(0);
}
await page
.getByRole("button", { name: "Start Slack message test" })
.click();
}
await expect(
page.getByRole("heading", { name: `Try Maya in ${provider.name}` }),
).toBeVisible();
await expect(
page.getByRole("button", { name: "I've sent the test message" }),
).toBeVisible();
await expect(
page.getByRole("heading", {
name: "Link the account you’re testing",
}),
).toBeVisible();
await expect(
page.getByText(
/An observed external account is unlinked, and isolated guest work is off, so it cannot safely start Maya/,
),
).toBeVisible();
await expect(
page.getByRole("button", { name: "Review identity access" }),
).toBeVisible();
await page
.getByRole("button", { name: "Review identity access" })
.click();
await expect(page).toHaveURL(
new RegExp(
`/${seed.prefix}/apps/chat/endpoint-${provider.provider}/access$`,
),
);
await expect(
page.getByRole("button", { name: "Continue setup" }),
).toBeVisible();
await page.getByRole("button", { name: "Continue setup" }).click();
expect(new URL(page.url()).searchParams.get("reconnect")).toBeNull();
await expect(
page.getByRole("heading", { name: `Try Maya in ${provider.name}` }),
).toBeVisible();
await expectSetupRail(page);
await expectProviderTryInstructions(page, provider);
expect(mock.configuredCredentialKeys).toEqual(
expectedCredentialKeys(provider.provider),
);
await page
.getByRole("button", { name: "I've sent the test message" })
.click();
await expect(page).toHaveURL(
new RegExp(
`/${seed.prefix}/apps/chat/endpoint-${provider.provider}/settings$`,
),
);
await expect(
page.getByRole("heading", { name: `Maya in ${provider.name}` }),
).toBeVisible();
await expect(
page.getByText(provider.accountLabel, { exact: true }),
).toBeVisible();
await expect(
page.getByRole("button", { name: "Continue setup" }),
).toHaveCount(0);
await expect(page.getByText("Change agent", { exact: true })).toHaveCount(
0,
);
await expect(page.getByRole("tab")).toHaveCount(4);
for (const tab of ["Settings", "Access", "Conversations", "Activity"]) {
await expect(page.getByRole("tab", { name: tab })).toBeVisible();
}
await expect(
page.getByRole("heading", { name: "Where this agent can work" }),
).toBeVisible();
if (provider.provider === "slack") {
await expect(
page.getByRole("heading", { name: "Slack command" }),
).toBeVisible();
await expect(
page.getByText("/maya-public", { exact: true }),
).toBeVisible();
await expect(
page.getByText(
/Slack's bare \/status command is not a Paperclip control/,
),
).toBeVisible();
}
await expect(
page.getByRole("switch", {
name: `Enable ${provider.resourceLabel}`,
}),
).toBeVisible();
await expect(
page.getByText(provider.resourceLabel, { exact: true }),
).toBeVisible();
await expect(
page.getByText(provider.secondaryResourceLabel, { exact: true }),
).toBeVisible();
await expect(
page.getByRole("switch", {
name: `Enable ${provider.secondaryResourceLabel}`,
}),
).not.toBeChecked();
await page
.getByRole("switch", { name: `Enable ${provider.resourceLabel}` })
.click();
await expect.poll(() => mock.updatedResource).toBe(true);
expect(mock.resourceUpdates.at(-1)).toEqual([
{ id: `resource-${provider.provider}`, enabled: true },
]);
if (provider.provider === "github") {
await expect(
page.getByRole("heading", { name: "Private conversations" }),
).toHaveCount(0);
} else {
const directMessages = page.getByRole("switch", {
name: "Allow direct messages",
});
await expect(directMessages).toBeVisible();
if (provider.provider === "discord") {
await expect(
page.getByText(
"People must also enable Direct Messages in their shared Discord server’s Privacy Settings.",
),
).toBeVisible();
}
await directMessages.click();
await expect.poll(() => mock.allowDirectMessages).toBe(true);
}
if (provider.provider === "microsoft-teams") {
const groupChats = page.getByRole("switch", {
name: "Allow group chats",
});
await expect(groupChats).toBeVisible();
await groupChats.click();
await expect.poll(() => mock.allowGroupChats).toBe(true);
}
for (const lifecycleAction of [
"Pause",
"Resume",
"Reconnect",
"Remove connection",
]) {
await expect(
page.getByRole("button", {
name: lifecycleAction,
exact: true,
}),
).toHaveCount(0);
}
await expect(
page.getByRole("switch", { name: "Allow unlinked people" }),
).toHaveCount(0);
await page.getByRole("tab", { name: "Access" }).click();
await expect(
page.getByRole("heading", { name: "External identity access" }),
).toBeVisible();
await expect(
page.getByText(
/Their tasks run only with an isolated workspace and sandbox environment; otherwise Paperclip safely refuses the request/,
),
).toBeVisible();
const allowUnlinked = page.getByRole("switch", {
name: "Allow unlinked people",
});
await expect(allowUnlinked).toBeVisible();
await allowUnlinked.click();
await expect.poll(() => mock.allowUnlinkedPeople).toBe(true);
await expect(
page.getByText("Ada Lovelace", { exact: true }),
).toBeVisible();
await expect(
page.getByText("Grace Hopper", { exact: true }),
).toBeVisible();
await expect(
page.getByText("Linked to Grace Hopper", { exact: true }),
).toBeVisible();
await expect(
page.getByRole("button", { name: "Create private link" }),
).toBeVisible();
await page
.context()
.grantPermissions(["clipboard-read", "clipboard-write"], {
origin: new URL(page.url()).origin,
});
await page.getByRole("button", { name: "Create private link" }).click();
await expect
.poll(() => mock.linkIntentPrincipalId)
.toBe(`principal-${provider.provider}`);
const confirmationUrl = `https://paperclip.example.test/${seed.prefix}/chat-identity/confirm?token=e2e-redacted`;
await expect(
page.getByText(confirmationUrl, { exact: true }),
).toBeVisible();
await page.getByRole("button", { name: "Copy link" }).click();
await expect
.poll(() => page.evaluate(() => navigator.clipboard.readText()))
.toBe(confirmationUrl);
await expect(
page.getByText("Confirmation link copied", { exact: true }),
).toBeVisible();
await page.getByRole("button", { name: "Revoke" }).click();
await expect
.poll(() => mock.revokedPrincipalId)
.toBe(`principal-${provider.provider}-linked`);
await expect(
page.getByText(`grace@${provider.provider}`, { exact: true }),
).toBeVisible();
await page.getByRole("tab", { name: "Conversations" }).click();
await expect(
page.getByRole("heading", { name: "Conversations" }),
).toBeVisible();
await expect(
page.getByText(`CHAT-123 · Investigate ${provider.name} delivery`),
).toBeVisible();
await expect(
page.getByText(provider.resourceLabel, { exact: true }),
).toHaveCount(1);
const providerLink = page.getByRole("link", {
name: `Open ${provider.name}`,
});
await expect(providerLink).toHaveAttribute("href", provider.externalUrl);
const taskLink = page.getByRole("link", { name: "Open task" });
await expect(taskLink).toHaveAttribute(
"href",
new RegExp(`/${seed.prefix}/issues/issue-${provider.provider}$`),
);
mock.conversationState = "waiting";
await expect(page.getByText("waiting", { exact: true })).toBeVisible({
timeout: 8_000,
});
await page.getByRole("tab", { name: "Activity" }).click();
await expect(
page.getByRole("heading", { name: "Connection activity" }),
).toBeVisible();
await expect(
page.getByText(
`Inbound ${provider.name} delivery could not be processed`,
),
).toBeVisible();
await expect(
page.getByText(`Published safe output to ${provider.name}`),
).toBeVisible();
await expect(
page.getByText("Credential values and request bodies are redacted."),
).toBeVisible();
const deliveryTimestamp = page
.getByText(`Inbound ${provider.name} delivery could not be processed`)
.locator("..")
.locator("time");
await expect(deliveryTimestamp).toHaveText(
/\w+ \d+, \d{4}, \d{1,2}:\d{2}:\d{2} [AP]M/,
);
await expect(deliveryTimestamp).toHaveAttribute(
"datetime",
/\d{4}-\d{2}-\d{2}T/,
);
await expect(deliveryTimestamp).toHaveAttribute(
"title",
(await deliveryTimestamp.getAttribute("datetime"))!,
);
await expect(page.getByText("xoxb-e2e-redacted")).toHaveCount(0);
await expect(page.getByText("teams-client-secret")).toHaveCount(0);
await expect(page.getByText("github-webhook-secret")).toHaveCount(0);
// A provider callback does not cause a Board mutation. Keep this tab
// mounted and focused: neither navigation nor Replay may refresh it.
mock.liveActivitySummary = `${provider.name} reaction removed while viewing Activity`;
mock.setStatus("paused");
await expect(
page.getByText(mock.liveActivitySummary, { exact: true }),
).toBeVisible({ timeout: 8_000 });
await expect(
page.getByRole("button", { name: "Resume", exact: true }),
).toBeVisible({ timeout: 8_000 });
mock.setStatus("active");
await expect(
page.getByRole("button", { name: "Pause", exact: true }),
).toBeVisible({ timeout: 8_000 });
await page.getByRole("button", { name: "Replay" }).click();
await expect.poll(() => mock.replayedDelivery).toBe(true);
await expect(
page.getByRole("button", { name: "Pause", exact: true }),
).toBeVisible();
await expect(
page.getByRole("button", { name: "Remove connection", exact: true }),
).toBeVisible();
await expect(
page.getByText(PROVIDER_LIFECYCLE_COPY[provider.provider].reconnect, {
exact: true,
}),
).toBeVisible();
await page.getByRole("button", { name: "Pause", exact: true }).click();
await expect(
page.getByRole("button", { name: "Resume", exact: true }),
).toBeVisible();
await page.getByRole("button", { name: "Resume", exact: true }).click();
await expect(
page.getByRole("button", { name: "Pause", exact: true }),
).toBeVisible();
expect(mock.lifecycleActions).toEqual(["pause", "resume"]);
mock.setStatus("attention");
await page.reload();
await expect(
page.getByRole("button", { name: "Reconnect", exact: true }),
).toBeVisible();
await expect(
page.getByRole("button", { name: "Remove connection" }),
).toBeVisible();
await page
.getByRole("button", { name: "Reconnect", exact: true })
.click();
await expect(page).toHaveURL(
new RegExp(
`/${seed.prefix}/apps/chat/connect\\?.*resume=endpoint-${provider.provider}`,
),
);
expect(new URL(page.url()).searchParams.get("reconnect")).toBe("1");
await expect(
page.getByRole("heading", {
name:
provider.provider === "github"
? "Reconnect GitHub App"
: provider.setupHeading,
}),
).toBeVisible();
await expect(
page.getByRole("button", { name: "Choose an active agent" }),
).toHaveCount(0);
if (provider.provider === "github") {
const connectButton = page.getByRole("button", {
name: provider.setupButton,
});
await expect(
page.getByText(
/Under the target user or organization, create a new GitHub App/,
),
).toHaveCount(0);
await expect(
page.getByText(
/Leave App ID and private key blank to reuse saved credentials/,
),
).toBeVisible();
await page.getByLabel("GitHub App ID").fill("123456");
await page
.getByLabel("Private key (PEM)")
.fill("reconnect-private-key");
await expect(connectButton).toBeEnabled();
await page.evaluate((buttonName) => {
const state = window as typeof window & {
__githubConnectEnabledAfterRotation?: boolean;
};
state.__githubConnectEnabledAfterRotation = false;
new MutationObserver(() => {
const button = [...document.querySelectorAll("button")].find(
(candidate) => candidate.textContent?.trim() === buttonName,
);
if (button instanceof HTMLButtonElement && !button.disabled) {
state.__githubConnectEnabledAfterRotation = true;
}
}).observe(document.body, {
attributes: true,
childList: true,
subtree: true,
});
}, provider.setupButton);
await page
.getByRole("button", { name: "Regenerate webhook secret" })
.click();
await expect(connectButton).toBeDisabled();
await page.waitForTimeout(100);
expect(
await page.evaluate(
() =>
(
window as typeof window & {
__githubConnectEnabledAfterRotation?: boolean;
}
).__githubConnectEnabledAfterRotation,
),
).toBe(false);
await page.goBack();
await expect(
page.getByRole("heading", { name: "Connection activity" }),
).toBeVisible();
await page
.getByRole("button", { name: "Reconnect", exact: true })
.click();
await expect(
page.getByRole("heading", { name: "Reconnect GitHub App" }),
).toBeVisible();
await page.getByLabel("GitHub App ID").fill("123456");
await page
.getByLabel("Private key (PEM)")
.fill("reconnect-private-key");
await expect(
page.getByRole("button", { name: provider.setupButton }),
).toBeDisabled();
mock.setGitHubWebhookVerified();
await expect(
page.getByRole("button", { name: provider.setupButton }),
).toBeEnabled();
}
mock.setStatus("active");
await page.goto(
`/${seed.prefix}/apps/chat/endpoint-${provider.provider}/activity`,
);
await page.getByRole("button", { name: "Remove connection" }).click();
const confirmation = page.getByRole("alertdialog");
await expect(confirmation).toContainText("Remove this connection?");
await expect(confirmation).toContainText(
PROVIDER_LIFECYCLE_COPY[provider.provider].remove,
);
await confirmation
.getByRole("button", { name: "Remove connection" })
.click();
await expect(page).toHaveURL(new RegExp(`/${seed.prefix}/apps$`));
await expect.poll(() => mock.removed).toBe(true);
expect(mock.lifecycleActions).toEqual(["pause", "resume", "remove"]);
});
}
});
test.describe("iMessage Photon setup and management", () => {
let seed: Seed;
const photon: ProviderCase = {
provider: "imessage-photon",
slug: "imessage-photon",
name: "iMessage Photon",
accountLabel: "Photon Test",
botLabel: "Maya",
botUsername: "+15555550100",
resourceLabel: "Family project",
secondaryResourceLabel: "Second group",
resourceType: "group_chat",
externalUrl: "https://app.photon.codes/",
setupHeading: /Connect iMessage Photon/,
setupButton: "Connect selected number",
chatAndTool: false,
};
test.beforeAll(async ({ request }) => {
seed = await seedCompanyAndAgent(request);
});
test("connects Pro shared DMs without presenting a fake owned number or enabling groups", async ({page}) => {
await installChatControlPlaneMock(page, photon, seed, { enableChatConnectors: true, photonShared: true });
await page.goto(`/${seed.prefix}/apps`);
await page.getByRole("button", {name:"Connect iMessage Photon",exact:true}).click();
await selectMaya(page);
await page.getByLabel("Project ID").fill("project-e2e");
await page.getByLabel("Project secret").fill("photon-test-secret");
await page.getByRole("button", {name:"Inspect Photon project"}).click();
await expect(page.getByText(/Groups cannot be enabled on this channel/)).toBeVisible();
await page.getByRole("button", {name:"Connect shared DMs"}).click();
await expect(page.getByRole("heading", {name:"Try Maya in iMessage Photon"})).toBeVisible();
await page.reload();
await expect(page.getByText(/enroll your sender in Users/)).toBeVisible();
await expect(page.getByRole("button", {name:/Copy \+1555/})).toHaveCount(0);
await page.getByRole("button", {name:"I've sent the test message"}).click();
await page.getByRole("tab", {name:"Settings"}).click();
await expect(page.getByText(/Shared Photon project · direct messages only/)).toBeVisible();
await expect(page.getByRole("switch", {name:"Enable Family project"})).toBeDisabled();
await expect(page.getByRole("button", {name:"Copy dedicated number"})).toHaveCount(0);
});
for (const theme of ["light", "dark"] as const) {
test(`discovers dedicated lines and completes the channel wizard (${theme})`, async ({
page,
}, testInfo) => {
const mock = await installChatControlPlaneMock(page, photon, seed, {
enableChatConnectors: true,
});
await page.addInitScript(
(value) => localStorage.setItem("paperclip.theme", value),
theme,
);
await page.goto(`/${seed.prefix}/apps`);
const card = page.locator(
'[role="listitem"][data-app-slug="imessage-photon"]',
);
await expect(card).toBeVisible();
await card
.getByRole("button", { name: "Connect iMessage Photon" })
.click();
await selectMaya(page);
await expectSetupRail(page);
await expect(
page.getByRole("heading", { name: "Connect iMessage Photon" }),
).toBeVisible();
await page.getByLabel("Project ID").fill("project-e2e");
await page.getByLabel("Project secret").fill("photon-test-secret");
await expect(page.getByLabel("Project secret")).toHaveAttribute(
"type",
"password",
);
await page
.getByRole("button", { name: "Inspect Photon project" })
.click();
await expect(
page.getByRole("button", { name: "Connect selected number" }),
).toBeDisabled();
await page
.getByRole("radio", { name: "+15555550100", exact: true })
.focus();
await page.keyboard.press("Space");
await page
.getByRole("button", { name: "Connect selected number" })
.click();
await expect(
page.getByRole("heading", { name: "Try Maya in iMessage Photon" }),
).toBeVisible();
expect(mock.configuredCredentialKeys).toEqual(["projectSecret"]);
await expect(
page.getByRole("button", { name: "Copy +15555550100" }),
).toBeVisible();
await expect(
page.getByText("Link your Messages identity", { exact: true }),
).toBeVisible();
await page
.getByRole("button", { name: "I've sent the test message" })
.click();
await page.getByRole("tab", { name: "Settings" }).click();
await expect(
page.getByText(/replies are visible to everyone in that group/),
).toBeVisible();
const group = page.getByRole("switch", { name: "Enable Family project" });
await expect(group).not.toBeChecked();
await group.click();
await expect(group).toBeChecked();
await page.setViewportSize({ width: 390, height: 844 });
await page
.getByRole("combobox", { name: "Page section" })
.selectOption("activity");
await expect(
page.getByRole("button", { name: "Pause", exact: true }),
).toBeVisible();
await page.screenshot({
path: testInfo.outputPath(`photon-${theme}-mobile.png`),
fullPage: true,
});
expect(
await page.evaluate(
() => document.documentElement.scrollWidth <= window.innerWidth,
),
).toBe(true);
await page.getByRole("button", { name: "Pause", exact: true }).click();
await expect(
page.getByRole("button", { name: "Resume", exact: true }),
).toBeVisible();
await page.getByRole("button", { name: "Resume", exact: true }).click();
await expect(
page.getByRole("button", { name: "Pause", exact: true }),
).toBeVisible();
});
}
});
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff