Files
PaperClipAI/scripts/docker-onboard-smoke.sh
T
Devin Foley 30f6999cbe fix(release-smoke): configurable readiness timeout and diagnostics for slow containers (#11187)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The release subsystem's nightly lane (#11006) gates every nightly
publish on the release smoke suite, which boots the published artifact
in a Docker container
> - The suite's first CI execution failed at the health readiness check:
the harness hard-codes a 90 second budget, but a CI container
cold-installs paperclipai from npm and initializes embedded postgres
with no warm caches
> - When the timeout expired with the container still running, the
harness printed no container logs, so the failure gave no diagnostics
> - This pull request makes the readiness budget configurable, raises it
for CI, and dumps container logs on timeout
> - The benefit is that the nightly gate measures the artifact, not the
runner's cold caches, and a red smoke run is diagnosable from its logs

## Linked Issues or Issue Description

**Subsystem affected**

Release smoke testing: `scripts/docker-onboard-smoke.sh`,
`.github/workflows/release-smoke.yml`.

**Problem or motivation**

Run 31426044332 (first forced nightly after #11006) failed in
`smoke_nightly` with `server did not become ready at
http://localhost:3232/api/health` after exactly 90 seconds. The
harness's readiness window is hard-coded to 90 attempts at 1 second.
Locally that works because the npm cache is warm; in CI the container
downloads the full package set and embedded postgres first. The timeout
path also printed no container logs when the container was still
running, so there was no way to see how far boot had progressed.

**Proposed solution**

Make the readiness budget an environment variable
(`SMOKE_READY_TIMEOUT_SECONDS`, default unchanged at 90 for local use),
set it to 420 in the CI workflow, and dump the last 150 container log
lines when the readiness check times out on a still-running container.

## What Changed

- `scripts/docker-onboard-smoke.sh`: `SMOKE_READY_TIMEOUT_SECONDS` env
var (default 90) replaces the hard-coded readiness budget; timeout with
a still-running container now prints the tail of `docker logs`
- `.github/workflows/release-smoke.yml`: sets
`SMOKE_READY_TIMEOUT_SECONDS=420` for CI runs

## Verification

- `bash -n` on the harness and YAML parse of the workflow
- The real proof is the next `channel: nightly` dispatch of
`release.yml`, which re-runs this suite in CI with the new budget

## Risks

- Low. The local default is unchanged; CI runs simply wait longer before
declaring failure, and a genuinely broken artifact still fails (with
logs now)

## Model Used

Claude Fable 5 (`claude-fable-5`, Anthropic) in Claude Code, with
extended thinking and full tool use. Diagnosis from CI run logs; patch
model-authored under human direction.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green (pending — will confirm before
merge)
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
(pending — will confirm before merge)
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-08-10 13:19:33 -07:00

312 lines
9.9 KiB
Bash
Executable File

#!/usr/bin/env bash
set -euo pipefail
REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
IMAGE_NAME="${IMAGE_NAME:-paperclip-onboard-smoke}"
HOST_PORT="${HOST_PORT:-3131}"
PAPERCLIPAI_VERSION="${PAPERCLIPAI_VERSION:-latest}"
DATA_DIR="${DATA_DIR:-$REPO_ROOT/data/docker-onboard-smoke}"
HOST_UID="${HOST_UID:-$(id -u)}"
SMOKE_DETACH="${SMOKE_DETACH:-false}"
SMOKE_METADATA_FILE="${SMOKE_METADATA_FILE:-}"
PAPERCLIP_DEPLOYMENT_MODE="${PAPERCLIP_DEPLOYMENT_MODE:-authenticated}"
PAPERCLIP_DEPLOYMENT_EXPOSURE="${PAPERCLIP_DEPLOYMENT_EXPOSURE:-private}"
PAPERCLIP_PUBLIC_URL="${PAPERCLIP_PUBLIC_URL:-http://localhost:${HOST_PORT}}"
SMOKE_AUTO_BOOTSTRAP="${SMOKE_AUTO_BOOTSTRAP:-true}"
# Seconds to wait for /api/health after the container starts. The container
# cold-installs paperclipai from npm and initializes embedded postgres before
# it can serve health, so CI callers with no warm caches need far more than
# the local default.
SMOKE_READY_TIMEOUT_SECONDS="${SMOKE_READY_TIMEOUT_SECONDS:-90}"
SMOKE_ADMIN_NAME="${SMOKE_ADMIN_NAME:-Smoke Admin}"
SMOKE_ADMIN_EMAIL="${SMOKE_ADMIN_EMAIL:-smoke-admin@paperclip.local}"
SMOKE_ADMIN_PASSWORD="${SMOKE_ADMIN_PASSWORD:-paperclip-smoke-password}"
CONTAINER_NAME="${IMAGE_NAME//[^a-zA-Z0-9_.-]/-}"
LOG_PID=""
COOKIE_JAR=""
TMP_DIR=""
PRESERVE_CONTAINER_ON_EXIT="false"
mkdir -p "$DATA_DIR"
cleanup() {
if [[ -n "$LOG_PID" ]]; then
kill "$LOG_PID" >/dev/null 2>&1 || true
fi
if [[ "$PRESERVE_CONTAINER_ON_EXIT" != "true" ]]; then
docker stop "$CONTAINER_NAME" >/dev/null 2>&1 || true
fi
if [[ -n "$TMP_DIR" && -d "$TMP_DIR" ]]; then
rm -rf "$TMP_DIR"
fi
}
trap cleanup EXIT INT TERM
container_is_running() {
local running
running="$(docker inspect -f '{{.State.Running}}' "$CONTAINER_NAME" 2>/dev/null || true)"
[[ "$running" == "true" ]]
}
wait_for_http() {
local url="$1"
local attempts="${2:-60}"
local sleep_seconds="${3:-1}"
local i
for ((i = 1; i <= attempts; i += 1)); do
if curl -fsS "$url" >/dev/null 2>&1; then
return 0
fi
if ! container_is_running; then
echo "Smoke bootstrap failed: container $CONTAINER_NAME exited before $url became ready" >&2
docker logs "$CONTAINER_NAME" >&2 || true
return 1
fi
sleep "$sleep_seconds"
done
if ! container_is_running; then
echo "Smoke bootstrap failed: container $CONTAINER_NAME exited before readiness check completed" >&2
docker logs "$CONTAINER_NAME" >&2 || true
else
echo "Smoke bootstrap failed: $url not ready after ${attempts} attempts; container is still running. Last container logs:" >&2
docker logs --tail 150 "$CONTAINER_NAME" >&2 || true
fi
return 1
}
write_metadata_file() {
if [[ -z "$SMOKE_METADATA_FILE" ]]; then
return 0
fi
mkdir -p "$(dirname "$SMOKE_METADATA_FILE")"
{
printf 'SMOKE_BASE_URL=%q\n' "$PAPERCLIP_PUBLIC_URL"
printf 'SMOKE_ADMIN_EMAIL=%q\n' "$SMOKE_ADMIN_EMAIL"
printf 'SMOKE_ADMIN_PASSWORD=%q\n' "$SMOKE_ADMIN_PASSWORD"
printf 'SMOKE_CONTAINER_NAME=%q\n' "$CONTAINER_NAME"
printf 'SMOKE_DATA_DIR=%q\n' "$DATA_DIR"
printf 'SMOKE_IMAGE_NAME=%q\n' "$IMAGE_NAME"
printf 'SMOKE_PAPERCLIPAI_VERSION=%q\n' "$PAPERCLIPAI_VERSION"
} >"$SMOKE_METADATA_FILE"
}
generate_bootstrap_invite_url() {
local bootstrap_output
local bootstrap_status
if bootstrap_output="$(
docker exec \
-e PAPERCLIP_DEPLOYMENT_MODE="$PAPERCLIP_DEPLOYMENT_MODE" \
-e PAPERCLIP_DEPLOYMENT_EXPOSURE="$PAPERCLIP_DEPLOYMENT_EXPOSURE" \
-e PAPERCLIP_PUBLIC_URL="$PAPERCLIP_PUBLIC_URL" \
-e PAPERCLIP_HOME="/paperclip" \
"$CONTAINER_NAME" bash -lc \
'timeout 20s npx --yes "paperclipai@${PAPERCLIPAI_VERSION}" auth bootstrap-ceo --data-dir "$PAPERCLIP_HOME" --base-url "$PAPERCLIP_PUBLIC_URL"' \
2>&1
)"; then
bootstrap_status=0
else
bootstrap_status=$?
fi
if [[ $bootstrap_status -ne 0 && $bootstrap_status -ne 124 ]]; then
echo "Smoke bootstrap failed: could not run bootstrap-ceo inside container" >&2
printf '%s\n' "$bootstrap_output" >&2
return 1
fi
local invite_url
invite_url="$(
printf '%s\n' "$bootstrap_output" \
| grep -o 'https\?://[^[:space:]]*/invite/pcp_bootstrap_[[:alnum:]]*' \
| tail -n 1
)"
if [[ -z "$invite_url" ]]; then
echo "Smoke bootstrap failed: bootstrap-ceo did not print an invite URL" >&2
printf '%s\n' "$bootstrap_output" >&2
return 1
fi
if [[ $bootstrap_status -eq 124 ]]; then
echo " Smoke bootstrap: bootstrap-ceo timed out after printing invite URL; continuing" >&2
fi
printf '%s\n' "$invite_url"
}
post_json_with_cookies() {
local url="$1"
local body="$2"
local output_file="$3"
curl -sS \
-o "$output_file" \
-w "%{http_code}" \
-c "$COOKIE_JAR" \
-b "$COOKIE_JAR" \
-H "Content-Type: application/json" \
-H "Origin: $PAPERCLIP_PUBLIC_URL" \
-X POST \
"$url" \
--data "$body"
}
get_with_cookies() {
local url="$1"
curl -fsS \
-c "$COOKIE_JAR" \
-b "$COOKIE_JAR" \
-H "Accept: application/json" \
"$url"
}
sign_up_or_sign_in() {
local signup_response="$TMP_DIR/signup.json"
local signup_status
signup_status="$(post_json_with_cookies \
"$PAPERCLIP_PUBLIC_URL/api/auth/sign-up/email" \
"{\"name\":\"$SMOKE_ADMIN_NAME\",\"email\":\"$SMOKE_ADMIN_EMAIL\",\"password\":\"$SMOKE_ADMIN_PASSWORD\"}" \
"$signup_response")"
if [[ "$signup_status" =~ ^2 ]]; then
echo " Smoke bootstrap: created admin user $SMOKE_ADMIN_EMAIL"
return 0
fi
local signin_response="$TMP_DIR/signin.json"
local signin_status
signin_status="$(post_json_with_cookies \
"$PAPERCLIP_PUBLIC_URL/api/auth/sign-in/email" \
"{\"email\":\"$SMOKE_ADMIN_EMAIL\",\"password\":\"$SMOKE_ADMIN_PASSWORD\"}" \
"$signin_response")"
if [[ "$signin_status" =~ ^2 ]]; then
echo " Smoke bootstrap: signed in existing admin user $SMOKE_ADMIN_EMAIL"
return 0
fi
echo "Smoke bootstrap failed: could not sign up or sign in admin user" >&2
echo "Sign-up response:" >&2
cat "$signup_response" >&2 || true
echo >&2
echo "Sign-in response:" >&2
cat "$signin_response" >&2 || true
echo >&2
return 1
}
auto_bootstrap_authenticated_smoke() {
local health_url="$PAPERCLIP_PUBLIC_URL/api/health"
local health_json
health_json="$(curl -fsS "$health_url")"
if [[ "$health_json" != *'"deploymentMode":"authenticated"'* ]]; then
return 0
fi
sign_up_or_sign_in
if [[ "$health_json" == *'"bootstrapStatus":"ready"'* ]]; then
echo " Smoke bootstrap: instance already ready"
else
local invite_url
invite_url="$(generate_bootstrap_invite_url)"
echo " Smoke bootstrap: generated bootstrap invite via auth bootstrap-ceo"
local invite_token="${invite_url##*/}"
local accept_response="$TMP_DIR/accept.json"
local accept_status
accept_status="$(post_json_with_cookies \
"$PAPERCLIP_PUBLIC_URL/api/invites/$invite_token/accept" \
'{"requestType":"human"}' \
"$accept_response")"
if [[ ! "$accept_status" =~ ^2 ]]; then
echo "Smoke bootstrap failed: bootstrap invite acceptance returned HTTP $accept_status" >&2
cat "$accept_response" >&2 || true
echo >&2
return 1
fi
echo " Smoke bootstrap: accepted bootstrap invite"
fi
local session_json
session_json="$(get_with_cookies "$PAPERCLIP_PUBLIC_URL/api/auth/get-session")"
if [[ "$session_json" != *'"userId"'* ]]; then
echo "Smoke bootstrap failed: no authenticated session after bootstrap" >&2
echo "$session_json" >&2
return 1
fi
local companies_json
companies_json="$(get_with_cookies "$PAPERCLIP_PUBLIC_URL/api/companies")"
if [[ "${companies_json:0:1}" != "[" ]]; then
echo "Smoke bootstrap failed: board companies endpoint did not return JSON array" >&2
echo "$companies_json" >&2
return 1
fi
echo " Smoke bootstrap: board session verified"
echo " Smoke admin credentials: $SMOKE_ADMIN_EMAIL / $SMOKE_ADMIN_PASSWORD"
}
echo "==> Building onboard smoke image"
docker build \
--build-arg PAPERCLIPAI_VERSION="$PAPERCLIPAI_VERSION" \
--build-arg HOST_UID="$HOST_UID" \
-f "$REPO_ROOT/docker/Dockerfile.onboard-smoke" \
-t "$IMAGE_NAME" \
"$REPO_ROOT"
echo "==> Running onboard smoke container"
echo " UI should be reachable at: http://localhost:$HOST_PORT"
echo " Public URL: $PAPERCLIP_PUBLIC_URL"
echo " Smoke auto-bootstrap: $SMOKE_AUTO_BOOTSTRAP"
echo " Detached mode: $SMOKE_DETACH"
echo " Data dir: $DATA_DIR"
echo " Deployment: $PAPERCLIP_DEPLOYMENT_MODE/$PAPERCLIP_DEPLOYMENT_EXPOSURE"
if [[ "$SMOKE_DETACH" != "true" ]]; then
echo " Live output: onboard banner and server logs stream in this terminal (Ctrl+C to stop)"
fi
docker rm -f "$CONTAINER_NAME" >/dev/null 2>&1 || true
docker run -d --rm \
--name "$CONTAINER_NAME" \
-p "$HOST_PORT:3100" \
-e HOST=0.0.0.0 \
-e PORT=3100 \
-e PAPERCLIP_DEPLOYMENT_MODE="$PAPERCLIP_DEPLOYMENT_MODE" \
-e PAPERCLIP_DEPLOYMENT_EXPOSURE="$PAPERCLIP_DEPLOYMENT_EXPOSURE" \
-e PAPERCLIP_PUBLIC_URL="$PAPERCLIP_PUBLIC_URL" \
-v "$DATA_DIR:/paperclip" \
"$IMAGE_NAME" >/dev/null
if [[ "$SMOKE_DETACH" != "true" ]]; then
docker logs -f "$CONTAINER_NAME" &
LOG_PID=$!
fi
TMP_DIR="$(mktemp -d "${TMPDIR:-/tmp}/paperclip-onboard-smoke.XXXXXX")"
COOKIE_JAR="$TMP_DIR/cookies.txt"
if ! wait_for_http "$PAPERCLIP_PUBLIC_URL/api/health" "$SMOKE_READY_TIMEOUT_SECONDS" 1; then
echo "Smoke bootstrap failed: server did not become ready at $PAPERCLIP_PUBLIC_URL/api/health" >&2
exit 1
fi
if [[ "$SMOKE_AUTO_BOOTSTRAP" == "true" && "$PAPERCLIP_DEPLOYMENT_MODE" == "authenticated" ]]; then
auto_bootstrap_authenticated_smoke
fi
write_metadata_file
if [[ "$SMOKE_DETACH" == "true" ]]; then
PRESERVE_CONTAINER_ON_EXIT="true"
echo "==> Smoke container ready for automation"
echo " Smoke base URL: $PAPERCLIP_PUBLIC_URL"
echo " Smoke admin credentials: $SMOKE_ADMIN_EMAIL / $SMOKE_ADMIN_PASSWORD"
if [[ -n "$SMOKE_METADATA_FILE" ]]; then
echo " Smoke metadata file: $SMOKE_METADATA_FILE"
fi
exit 0
fi
wait "$LOG_PID"