test(release-smoke): follow onboarding into the standing chat (#22259)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The release workflow runs a Docker onboarding smoke on the real npm
artifact before it publishes a nightly or beta
> - #22077 changed onboarding. "Get started" now opens the lead's
standing chat. It does not create a first task or project
> - That PR updated the e2e tests, but it did not update the release
smoke
> - The smoke still expects an `/issues/` landing, so every nightly
smoke fails, and no nightly can publish
> - This pull request changes the smoke to assert the new chat-first
contract
> - The benefit is that the nightly lane can publish again, and the
smoke still guards the first-run experience

## Linked Issues or Issue Description

Refs #22077

**What happened?**
The release smoke `docker-auth-onboarding.spec.ts` fails at
`expect(page).toHaveURL(/\/issues\//)`. The page is on
`/<prefix>/chats/<agent id>`. The failure occurs on the first attempt
and on the retry. The scheduled nightly on 2026-11-18 and a dispatched
nightly for `2026.1118.0-canary.828200` both failed at this step.

**Steps to reproduce**
1. Run the release workflow with `channel=nightly` on a canary that
contains #22077.
2. Look at the `smoke_nightly / smoke` job.

**Expected behavior**
The smoke follows onboarding into the chat and passes when the artifact
works.

## What Changed

- Expect the launch to land on `/chats/<lead agent id>`, not on
`/issues/`.
- Assert that the task list and the project list are empty after launch.
- Assert that exactly one chat exists, with `originKind:
"onboarding_chat"`, no project, and the lead as `conversationAgentId`.
- Assert that the chat composer renders.
- Replace the old "no run" check. Now the smoke polls until the live
welcome run is queued for the lead. The provider mock serves no
completions, so the smoke does not check the welcome text.
- Remove the `FIRST_TASK_TITLE` constant, because onboarding no longer
seeds a task.

## Verification

- `tsc --noEmit --strict` on the spec passes.
- I checked the asserted fields against
`server/src/services/onboarding-chat.ts`, `GET
/companies/:companyId/chats` and
`tests/e2e/conference-room-typing-intro.spec.ts`.
- Full check: the `Release Smoke` workflow (or a nightly dispatch after
merge) must pass the `smoke` job.

## Risks

- Low risk. The change is test-only.
- The welcome-run assertion depends on the UI requesting the welcome in
a visible headless tab. If that is not true in CI, the poll times out
after 30 seconds. Then remove or relax that one assertion.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), with tool use (shell, GitHub
CLI).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
authored and GitHub committed 2026-11-18 16:12:17 -08:00
1 parent 11d777b80f
commit 52cf30f131
1 file changed
+55 -51
@@ -37,9 +37,8 @@ const AGENT_NAME = "Release Smoke Lead";
// The arc asks for a name, not a role, so every onboarding hire is filed under
// the neutral role (DEFAULT_AGENT_ROLE in ui/src/lib/onboarding-agent-role.ts).
const AGENT_ROLE = "general";
// Seeded by the wizard's launch step (DEFAULT_TASK_TITLE in
// ui/src/components/OnboardingWizard.tsx).
const FIRST_TASK_TITLE = "Paperclip onboarding";
// ONBOARDING_CHAT_ORIGIN_KIND in packages/shared/src/constants.ts.
const ONBOARDING_CHAT_ORIGIN_KIND = "onboarding_chat";
async function signIn(page: Page) {
await page.goto("/");
@@ -219,9 +218,9 @@ test.describe("Docker authenticated onboarding smoke", () => {
await expect(connectButton).toBeEnabled({ timeout: 30_000 });
await connectButton.click();
// Step 5: review, then launch. "Get started" provisions the onboarding
// project and first task and, only on success, drops the user into the
// seeded first task's thread (not the dashboard).
// Step 5: review, then launch. Since #22077, "Get started" opens the lead's
// standing chat and, only on success, drops the user into it (not the
// dashboard). It creates no execution task or project.
const getStartedButton = page.getByRole("button", {
name: "Get started",
exact: true,
@@ -229,7 +228,7 @@ test.describe("Docker authenticated onboarding smoke", () => {
await expect(getStartedButton).toBeVisible({ timeout: 60_000 });
await expect(getStartedButton).toBeEnabled({ timeout: 10_000 });
await getStartedButton.click();
await expect(page).toHaveURL(/\/issues\//, { timeout: 30_000 });
await expect(page).toHaveURL(/\/chats\/[^/]+$/, { timeout: 30_000 });
const agents = await getJson<
Array<{ id: string; name: string; role: string; adapterType: string; pauseReason: string | null }>
@@ -251,53 +250,58 @@ test.describe("Docker authenticated onboarding smoke", () => {
);
expect(goals).toEqual([]);
const issues = await getJson<
Array<{
id: string;
identifier: string | null;
title: string;
assigneeAgentId: string | null;
}>
>(page, `${baseUrl}/api/companies/${company!.id}/issues`);
const seededIssue = issues.find((entry) => entry.title === FIRST_TASK_TITLE);
expect(seededIssue).toBeTruthy();
expect(seededIssue!.assigneeAgentId).toBe(leadAgent!.id);
// The launch must have landed on the seeded task itself, not merely on
// some issue route.
const seededRef = seededIssue!.identifier ?? seededIssue!.id;
expect(new URL(page.url()).pathname.endsWith(`/issues/${seededRef}`)).toBe(
// The launch must have landed on the lead's chat itself, not merely on
// some chat route.
expect(new URL(page.url()).pathname.endsWith(`/chats/${leadAgent!.id}`)).toBe(
true
);
// #13068 rebuilt the seeded first task as a chat with the lead: launch
// posts a deterministic, server-owned greeting plus an opening question
// card, and deliberately does not wake the assignee — "no run until the
// user answers". Assert the chat actually opened (the greeting and the
// card are seeded without an LLM, so their absence means the launch
// half-finished) …
await expect(
page.getByText("Welcome to Paperclip!").first()
).toBeVisible({ timeout: 20_000 });
// The timeline also summarizes the unanswered question. Check the open
// question card so a summary alone cannot satisfy this assertion.
await expect(
page
.getByTestId("task-chat-interaction")
.getByText("What would you like to do?", { exact: true })
).toBeVisible();
// … and that the no-run contract holds. This spec used to poll for an
// assignment-triggered heartbeat run here; a run appearing before the
// user's first answer is now the regression, not the success. Wake
// dispatch is asynchronous, so watch the endpoint over a bounded window
// rather than sampling it once — a launch-time wake that slips through
// lands well within this window.
const runsUrl = `${baseUrl}/api/companies/${company!.id}/heartbeat-runs?agentId=${leadAgent!.id}`;
const noRunDeadline = Date.now() + 15_000;
while (Date.now() < noRunDeadline) {
expect(await getJson<Array<{ id: string }>>(page, runsUrl)).toEqual([]);
await page.waitForTimeout(1_000);
// Execution work waits until the user accepts a proposal, so the wizard
// leaves the task and project lists empty. The chat record is excluded
// from task collections and is listed only under chats.
for (const resource of ["issues", "projects"]) {
expect(
await getJson<Array<{ id: string }>>(
page,
`${baseUrl}/api/companies/${company!.id}/${resource}`,
),
).toEqual([]);
}
const chats = await getJson<
Array<{
originKind: string;
projectId: string | null;
conversationAgentId: string | null;
}>
>(page, `${baseUrl}/api/companies/${company!.id}/chats`);
expect(chats).toEqual([
expect.objectContaining({
originKind: ONBOARDING_CHAT_ORIGIN_KIND,
projectId: null,
conversationAgentId: leadAgent!.id,
}),
]);
// Assert the chat actually opened: its composer renders only once the
// conversation loaded.
await expect(page.getByTestId("task-chat-composer-add")).toBeVisible({
timeout: 20_000,
});
// Once the chat paints in a visible tab, the UI requests a live welcome
// through the normal agent queue. The provider mock serves no completions,
// so assert only that the published artifact queued the welcome run, not
// what it says.
const runsUrl = `${baseUrl}/api/companies/${company!.id}/heartbeat-runs?agentId=${leadAgent!.id}`;
await expect
.poll(
async () =>
(await getJson<Array<{ id: string }>>(page, runsUrl)).length,
{
message: "The onboarding chat must queue the lead's live welcome",
timeout: 30_000,
},
)
.toBeGreaterThan(0);
});
});