Files
PaperClipAI/.github/workflows/release-smoke.yml
T
Devin Foley 0a01444514 test(release-smoke): cover the background-service leg of onboarding (#12151)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The release pipeline gates each nightly and beta on a smoke suite
that onboards the published npm artifact and drives the golden path
> - That smoke runs onboarding inside a Docker container, and containers
have no service manager, so the background-service leg of onboarding has
zero automated coverage
> - v2026.824.0 shipped a service install that crash-looped on a missing
shim, and every smoke check stayed green (#12148 fixed the defect
itself)
> - This pull request adds a `smoke_service` job that runs the same
published artifact directly on the runner VM's systemd and requires the
installed service to end up serving
> - The benefit is that a release with a broken service install can no
longer pass the release smoke suite

## Linked Issues or Issue Description

Refs #12148 — the fix for the defect this coverage gap let through. The
gap: the release smoke runs `onboard` with `--yes` inside Docker, which
both skips the service prompt and lacks systemd, so no CI job ever
executed `manager.install()` against a real service manager.

## What Changed

- New `scripts/service-onboard-smoke.sh`: onboards the published
artifact with `--yes --install-service` on a systemd host, then fails
unless the managed shim exists and is executable, `paperclipai.service`
is active, and `/api/health` answers. A health response while the unit
is not active also fails, because that is the signature of something
other than the service serving. The script refuses to run over an
existing managed install unless `SMOKE_FORCE=true`, and cleans up after
itself by default so it is safe to run locally.
- New `smoke_service` job in `.github/workflows/release-smoke.yml`:
starts a user systemd session on the hosted runner (`loginctl
enable-linger` + exported `XDG_RUNTIME_DIR`/`DBUS_SESSION_BUS_ADDRESS`),
runs the script against `inputs.paperclip_version`, and uploads
`systemctl status` + journal output as diagnostics.
- No `release.yml` changes needed: `smoke_nightly` and `smoke_beta` call
this reusable workflow, and a `workflow_call` result aggregates all
jobs, so the new job gates nightly promotion automatically.

## Verification

- `bash -n scripts/service-onboard-smoke.sh` passes and the workflow
YAML parses.
- End-to-end: dispatched this branch's Release Smoke workflow against
the published canary that contains #12148; the `smoke_service` job
onboards, installs the service, and verifies the service serves health.
(Run link in PR comments.)
- Negative case: the same assertions fail against v2026.824.0 —
reproduced in a systemd container during the #12148 investigation: shim
missing, unit in a 203/EXEC restart loop.

## Risks

- Low risk to the product: no application code changes.
- Pipeline risk: a flaky user-session setup on the hosted runner would
block nightly promotion. Mitigated by validating the job end-to-end from
this branch before merge, a 30-minute job timeout, and diagnostics
uploaded on every run.
- The service leg only covers systemd. launchd (macOS) still has no CI
coverage; a macOS runner job is a possible follow-up.

## Model Used

- Claude Fable 5 (Anthropic, model ID `claude-fable-5`), extended
thinking, agentic tool use via Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-08-25 01:05:27 -07:00

184 lines
5.8 KiB
YAML

name: Release Smoke
on:
workflow_dispatch:
inputs:
paperclip_version:
description: Published Paperclip dist-tag to test
required: true
default: canary
type: choice
options:
- canary
- nightly
- beta
- latest
host_port:
description: Host port for the Docker smoke container
required: false
default: "3232"
type: string
artifact_name:
description: Artifact name for uploaded diagnostics
required: false
default: release-smoke
type: string
workflow_call:
inputs:
paperclip_version:
required: true
type: string
host_port:
required: false
default: "3232"
type: string
artifact_name:
required: false
default: release-smoke
type: string
jobs:
# The Docker smoke below can never exercise the background-service leg of
# onboarding: containers have no service manager, so v2026.824.0 shipped a
# service install that crash-looped on a missing shim while every
# golden-path check stayed green. Run the same published artifact directly
# on the runner VM's systemd and require the installed service to end up
# serving.
smoke_service:
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- name: Checkout repository
uses: actions/checkout@v7
- name: Setup Node.js
uses: actions/setup-node@v7
with:
node-version: 24
- name: Start a user systemd session
# The hosted runner has no login session for the runner user, so
# `systemctl --user` cannot reach a user manager until lingering
# starts one. Export the session address for the steps below.
run: |
sudo loginctl enable-linger "$(id -un)"
uid="$(id -u)"
for _ in $(seq 1 30); do
[[ -S "/run/user/$uid/bus" ]] && break
sleep 1
done
[[ -S "/run/user/$uid/bus" ]]
{
echo "XDG_RUNTIME_DIR=/run/user/$uid"
echo "DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/$uid/bus"
} >> "$GITHUB_ENV"
- name: Onboard with the background service
env:
PAPERCLIPAI_VERSION: ${{ inputs.paperclip_version }}
DATA_DIR: ${{ runner.temp }}/service-smoke-data
SMOKE_CLEANUP: "false"
run: ./scripts/service-onboard-smoke.sh
- name: Capture service diagnostics
if: always()
run: |
{
systemctl --user --no-pager status paperclipai.service || true
journalctl --user -u paperclipai.service --no-pager || true
} > "$RUNNER_TEMP/paperclipai-service.log" 2>&1
- name: Upload service diagnostics
if: always()
uses: actions/upload-artifact@v7
with:
name: ${{ inputs.artifact_name }}-service
path: ${{ runner.temp }}/paperclipai-service.log
retention-days: 14
smoke:
runs-on: ubuntu-latest
timeout-minutes: 45
steps:
- name: Checkout repository
uses: actions/checkout@v7
- name: Setup pnpm
uses: pnpm/action-setup@v6
with:
version: 9.15.4
- name: Setup Node.js
uses: actions/setup-node@v7
with:
node-version: 24
cache: pnpm
- name: Install dependencies
run: pnpm install --no-frozen-lockfile
- name: Verify runner Chrome
# Release smoke also runs headless on GitHub's Ubuntu image, so use the
# runner's preinstalled Chrome instead of a Playwright browser download.
run: google-chrome --version
- name: Launch Docker smoke harness
run: |
metadata_file="$RUNNER_TEMP/release-smoke.env"
HOST_PORT="${{ inputs.host_port }}" \
DATA_DIR="$RUNNER_TEMP/release-smoke-data" \
PAPERCLIPAI_VERSION="${{ inputs.paperclip_version }}" \
SMOKE_READY_TIMEOUT_SECONDS=420 \
SMOKE_DETACH=true \
SMOKE_METADATA_FILE="$metadata_file" \
./scripts/docker-onboard-smoke.sh
set -a
source "$metadata_file"
set +a
{
echo "SMOKE_BASE_URL=$SMOKE_BASE_URL"
echo "SMOKE_ADMIN_EMAIL=$SMOKE_ADMIN_EMAIL"
echo "SMOKE_ADMIN_PASSWORD=$SMOKE_ADMIN_PASSWORD"
echo "SMOKE_CONTAINER_NAME=$SMOKE_CONTAINER_NAME"
echo "SMOKE_DATA_DIR=$SMOKE_DATA_DIR"
echo "SMOKE_IMAGE_NAME=$SMOKE_IMAGE_NAME"
echo "SMOKE_PAPERCLIPAI_VERSION=$SMOKE_PAPERCLIPAI_VERSION"
echo "SMOKE_METADATA_FILE=$metadata_file"
} >> "$GITHUB_ENV"
- name: Run release smoke Playwright suite
env:
PAPERCLIP_RELEASE_SMOKE_BASE_URL: ${{ env.SMOKE_BASE_URL }}
PAPERCLIP_RELEASE_SMOKE_EMAIL: ${{ env.SMOKE_ADMIN_EMAIL }}
PAPERCLIP_RELEASE_SMOKE_PASSWORD: ${{ env.SMOKE_ADMIN_PASSWORD }}
PAPERCLIP_PLAYWRIGHT_CHANNEL: "chrome"
run: pnpm run test:release-smoke
- name: Capture Docker logs
if: always()
run: |
if [[ -n "${SMOKE_CONTAINER_NAME:-}" ]]; then
docker logs "$SMOKE_CONTAINER_NAME" >"$RUNNER_TEMP/docker-onboard-smoke.log" 2>&1 || true
fi
- name: Upload diagnostics
if: always()
uses: actions/upload-artifact@v7
with:
name: ${{ inputs.artifact_name }}
path: |
${{ runner.temp }}/docker-onboard-smoke.log
${{ env.SMOKE_METADATA_FILE }}
tests/release-smoke/playwright-report/
tests/release-smoke/test-results/
retention-days: 14
- name: Stop Docker smoke container
if: always()
run: |
if [[ -n "${SMOKE_CONTAINER_NAME:-}" ]]; then
docker rm -f "$SMOKE_CONTAINER_NAME" >/dev/null 2>&1 || true
fi