Files
PaperClipAI/packages/paperclip-runner/scripts/aws-agentcore.sh
T
Dotta 5716fe907e test(runner): add full-stack acceptance and eval gates (#12700)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The runner subsystem executes agent work across local and managed
provider backends.
> - The lower pull requests restore the task runtime, provider backends,
and managed-provider control plane.
> - The restored system needs repeatable full-stack checks before it can
ship safely.
> - Paid live checks also need clear access, cost, and secret controls.
> - This pull request adds acceptance, live evaluation, chaos, and
release gates for the restored runner stack.
> - The benefit is measurable runner parity with safer release
decisions.

## Linked Issues or Issue Description

**Subsystem affected**

Cross-cutting. This change covers runner tests, release workflows,
server contracts, and evaluation tools.

**Problem or motivation**

The runner stack did not have one complete acceptance surface for native
Codex, ACPX, Claude Managed, and AWS AgentCore. Release checks could
miss provider drift, task-view regressions, cost-policy errors, and
destructive cleanup errors.

**Proposed solution**

Add a 57-cell full-stack catalog, a Daytona image, and opt-in paid
workflows. Add live evaluation, chaos, cost-limit, redaction, and
release contract checks. Add AWS AgentCore infrastructure and guarded
provisioning tools. Keep the native runner experimental flag off by
default.

**Alternatives considered**

We considered manual smoke tests only. They do not give repeatable
evidence and they do not protect release branches. We also considered
one large pull request. The stacked pull requests keep each review below
the Greptile file limit.

**Roadmap alignment**

This work supports the shipped Cloud / Sandbox agents milestone and the
shipped Agent evals & feedback milestone in `ROADMAP.md`.

Related stack:

- #12699 adds managed provider backends and lifecycle support.
- #12691 adds qualified OpenCode and ACPX provider backends.
- #12685 restores task runtime rendering and steering.

## What Changed

- Add the runner full-stack harness with 57 catalog cells and 60 unit
tests.
- Add a Daytona runner image with digest-pinned base images and
base-aware image-content checks.
- Add guarded live evaluation and chaos workflows with a fixed
40-execution matrix; live and full-stack paid schedules now run only on
Sundays or by manual dispatch.
- Add in-flight reported-usage cost stops, post-turn cost caps,
exact-threshold failure classification, secret redaction, retry
classification, and actor authorization.
- Reattach stream and hard-budget listeners before restart-recovery
continuations so restored paid sessions cannot bypass in-flight
interruption.
- Preserve OpenCode usage and cost across tool-loop messages and turns
while exposing an explicit current-run delta to durable accounting.
- Keep PNG/WebM evidence in access-controlled artifacts only, reject
SVG, and publish only pruned inert structured per-attempt evidence.
- Add AWS AgentCore infrastructure, provisioning checks, and smoke
tools; reject unsafe model identifiers, require exact stack ownership
markers, and make failed-stack replacement explicit.
- Add evaluation-session contracts and capability reports.
- Add release workflow checks for immutable action pins, frozen
dependency installs, exact weekly cron shape, paid-run guards,
provider-secret isolation, and chaos test paths.
- Reauthorize the original and triggering numeric actor IDs as the first
step of every provider-secret job, including partial reruns, before
checkout or provider access.
- Give each full-stack matrix cell only its matching provider
credential, expose Daytona only to Daytona cells, and disable shared
dependency caches anywhere paid credentials or OIDC write access are
present.
- Protect the legacy manual E2E workflow with the same default-branch,
allowlist, environment, and per-job authorization boundary.
- Rotate live-eval candidates by week and retain 120 days of compatible
history so the seven-week trend window remains viable.
- Restore the root runner-acceptance commands and reconcile reported
snapshots,
raw receipts, and terminal usage without double counting or losing late
usage.
- Mark ACPX token deltas exact only when every budget field is present,
keep
cumulative cost/request authority separate, reject non-USD cost
labeling,
  and include thought tokens in output-token budgets.
- Keep `enableNativeRunner` off by default. The acceptance harness
enables it only in its isolated test instance.

## Verification

Passed locally:

- `pnpm --filter @paperclipai/paperclip-runner typecheck`
- `pnpm test:runner-acceptance:typecheck`
- `pnpm test:runner-acceptance` (19 tests)
- focused OpenCode proxy, driver, runnerd transport, live-session, and
turn-stream tests (106 tests)
- `pnpm --filter @paperclipai/paperclip-runner exec vitest run
src/live/clean-room-server.test.ts` (22 tests)
- `pnpm test:e2e:runner:typecheck`
- `pnpm test:e2e:runner:unit` (62 tests)
- `node --test scripts/__tests__/release-verify-workflow.test.mjs`
- `pnpm --filter @paperclipai/paperclip-runner
test:runner-workflow-evals` (22 tests)
- `pnpm -r typecheck`
- `pnpm build`
- `node --test
packages/paperclip-runner/scripts/aws-agentcore-provisioning.test.mjs`
(6 tests)
- `git diff --check`
- `cargo test --manifest-path
packages/paperclip-runner/runner/Cargo.toml -p paperclip-runner-core
--lib --locked` (161 tests)
- focused ACPX provider-event tests (10 tests)
- The rebased PR changes 92 files. `pnpm-lock.yaml` is unchanged.

I did not run paid live provider jobs or provision AWS resources. Those
checks need credentials and can create cost.

## Risks

The paid workflows can create provider cost. They require an allowlisted
original and triggering actor, the protected `runner-e2e-paid`
environment, explicit opt-in variables, and cost limits. The four
provider credentials exist only in that master-only environment, which
requires allowlisted reviewer approval and disables administrator
bypass; repository and organization Actions scopes contain no copies.

Provider usage arrives after a billable request, so the live guard
cannot prevent one request from crossing a threshold. It interrupts
immediately on the first reported threshold hit and permits no
continuation.

Visual evidence can contain secrets rendered as pixels. PNG/WebM remain
only in access-controlled workflow artifacts; SVG and per-attempt XML
are excluded, and S3/Pages receive a pruned structured dashboard.

The AWS scripts can create cloud resources. They use explicit commands,
least-privilege roles, KMS encryption, saved nonsecret metadata, and
explicit teardown.

This pull request does not enable the experimental native runner for
existing instances.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

OpenAI Codex with GPT-5. The model used extended reasoning, tool use,
code execution, and parallel subagents.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-09-02 08:55:08 -05:00

466 lines
23 KiB
Bash

#!/usr/bin/env bash
set -euo pipefail
PACKAGE_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
TEMPLATE="$PACKAGE_DIR/infra/aws-agentcore-paperclip.yaml"
STACK_DESCRIPTION="Paperclip proof-of-concept Amazon Bedrock AgentCore Harness and least-privilege invocation roles."
LOCAL_DIR="$PACKAGE_DIR/.paperclip-local"
ENV_FILE="$LOCAL_DIR/aws-agentcore.env"
ACTIVE_FILE="$LOCAL_DIR/aws-agentcore-lab.pid"
ACTION="${1:-}"
if [[ -n "$ACTION" ]]; then shift; fi
AWS_PROFILE_NAME="${AWS_PROFILE:-default}"
AWS_REGION_NAME="${AWS_REGION:-${AWS_DEFAULT_REGION:-us-east-1}}"
AWS_PROFILE_EXPLICIT=false
AWS_REGION_EXPLICIT=false
DEPLOYMENT_MODE="development"
STACK_NAME="paperclip-agentcore-development"
STACK_NAME_EXPLICIT=false
ENDPOINT_NAME="paperclip"
MODEL_ID="global.anthropic.claude-sonnet-4-6"
MARKETPLACE_PRODUCT_ID="prod-ffvjxvh4ltq64"
MARKETPLACE_PRODUCT_ID_EXPLICIT=false
TRUSTED_PRINCIPAL=""
CONTEXT_PREFIX=""
DRY_RUN=false
FORCE=false
YES=false
REPLACE_FAILED_STACK=false
usage() {
printf '%s\n' \
"Usage: aws-agentcore.sh <provision|probe|lab|destroy> [options]" \
" --aws-profile NAME AWS CLI v2 profile (default: $AWS_PROFILE_NAME)" \
" --region REGION commercial AWS region (default: $AWS_REGION_NAME)" \
" --mode MODE development or private" \
" --stack-name NAME CloudFormation stack name" \
" --model MODEL_ID Bedrock-native model ID" \
" --marketplace-product-id ID exact AWS Marketplace product ID for the model" \
" --principal ARN stable IAM role/user trusted to assume runner role" \
" --context-prefix PFX S3 prefix dedicated to this qualified profile" \
" --dry-run validate and print changes without deployment" \
" --replace-failed-stack explicitly replace an owned ROLLBACK_COMPLETE stack" \
" --yes confirm destructive replacement or teardown" \
" --force destroy even when a recorded Lab process is active"
}
while [[ $# -gt 0 ]]; do
case "$1" in
--) shift ;;
--aws-profile) AWS_PROFILE_NAME="$2"; AWS_PROFILE_EXPLICIT=true; shift 2 ;;
--region) AWS_REGION_NAME="$2"; AWS_REGION_EXPLICIT=true; shift 2 ;;
--mode) DEPLOYMENT_MODE="$2"; shift 2 ;;
--stack-name) STACK_NAME="$2"; STACK_NAME_EXPLICIT=true; shift 2 ;;
--model) MODEL_ID="$2"; shift 2 ;;
--marketplace-product-id) MARKETPLACE_PRODUCT_ID="$2"; MARKETPLACE_PRODUCT_ID_EXPLICIT=true; shift 2 ;;
--principal) TRUSTED_PRINCIPAL="$2"; shift 2 ;;
--context-prefix) CONTEXT_PREFIX="$2"; shift 2 ;;
--dry-run) DRY_RUN=true; shift ;;
--replace-failed-stack) REPLACE_FAILED_STACK=true; shift ;;
--force) FORCE=true; shift ;;
--yes) YES=true; shift ;;
-h|--help) usage; exit 0 ;;
*) printf 'Unknown option: %s\n' "$1" >&2; usage >&2; exit 2 ;;
esac
done
if [[ "$ACTION" != "provision" && "$ACTION" != "probe" && "$ACTION" != "lab" && "$ACTION" != "destroy" ]]; then
usage >&2
exit 2
fi
if [[ "$DEPLOYMENT_MODE" != "development" && "$DEPLOYMENT_MODE" != "private" ]]; then
printf 'Mode must be development or private.\n' >&2
exit 2
fi
if [[ ! "$AWS_PROFILE_NAME" =~ ^[A-Za-z0-9_.@+-]+$ ]]; then
printf 'AWS profile contains unsupported characters.\n' >&2
exit 2
fi
if [[ ! "$AWS_REGION_NAME" =~ ^[a-z]{2}(-gov)?-[a-z]+-[0-9]+$ || "$AWS_REGION_NAME" == cn-* || "$AWS_REGION_NAME" == us-gov-* ]]; then
printf 'This proof of concept currently supports commercial AWS regions only.\n' >&2
exit 2
fi
if [[ ! "$MODEL_ID" =~ ^[A-Za-z0-9][A-Za-z0-9._:-]{0,255}$ ]]; then
printf 'Bedrock model ID must not contain ARN, path, glob, or wildcard syntax.\n' >&2
exit 2
fi
if [[ "$MODEL_ID" != "global.anthropic.claude-sonnet-4-6" && "$MARKETPLACE_PRODUCT_ID_EXPLICIT" != true ]]; then
printf 'A custom --model requires its exact --marketplace-product-id so subscription authority remains model-scoped.\n' >&2
exit 2
fi
if [[ ! "$MARKETPLACE_PRODUCT_ID" =~ ^prod-[a-z0-9]+$ ]]; then
printf 'Marketplace product ID must use the AWS prod-* form.\n' >&2
exit 2
fi
if [[ -z "$CONTEXT_PREFIX" ]]; then CONTEXT_PREFIX="paperclip/agentcore/$STACK_NAME"; fi
if [[ ! "$CONTEXT_PREFIX" =~ ^[a-z0-9][a-z0-9/_-]{1,127}$ || "$CONTEXT_PREFIX" == /* || "$CONTEXT_PREFIX" == */ || "$CONTEXT_PREFIX" == *//* || "/$CONTEXT_PREFIX/" == */./* || "/$CONTEXT_PREFIX/" == */../* ]]; then
printf 'Context prefix must be a safe, relative S3 key prefix.\n' >&2
exit 2
fi
aws_cli() {
aws --profile "$AWS_PROFILE_NAME" --region "$AWS_REGION_NAME" --no-cli-pager "$@"
}
require_aws() {
command -v aws >/dev/null || { printf 'AWS CLI v2 is required.\n' >&2; exit 1; }
local version
version="$(aws --version 2>&1)"
[[ "$version" == aws-cli/2.* ]] || { printf 'AWS CLI v2 is required; found %s\n' "$version" >&2; exit 1; }
}
json_field() {
local key="$1"
node -e 'const fs=require("fs");const o=JSON.parse(fs.readFileSync(0,"utf8"));let v=o;for(const k of process.argv[1].split("."))v=v?.[k];if(v!==undefined&&v!==null)process.stdout.write(String(v));' "$key"
}
stack_tag_value() {
local key="$1"
node -e 'const fs=require("fs");const o=JSON.parse(fs.readFileSync(0,"utf8"));const tag=o.Stacks?.[0]?.Tags?.find((candidate)=>candidate.Key===process.argv[1]);if(tag?.Value)process.stdout.write(tag.Value);' "$key"
}
stack_output() {
local key="$1"
aws_cli cloudformation describe-stacks --stack-name "$STACK_NAME" \
--query "Stacks[0].Outputs[?OutputKey=='$key'].OutputValue | [0]" --output text
}
wait_for_harness_status() {
local harness_id="$1" harness_version="$2" status=""
for _ in {1..60}; do
status="$(aws_cli bedrock-agentcore-control get-harness --harness-id "$harness_id" --harness-version "$harness_version" --query harness.status --output text 2>/dev/null || true)"
if [[ "$status" == "READY" ]]; then return 0; fi
if [[ "$status" == "FAILED" || "$status" == "CREATE_FAILED" || "$status" == "UPDATE_FAILED" ]]; then
printf 'Harness %s entered terminal status %s.\n' "$harness_id" "$status" >&2
return 1
fi
sleep 5
done
printf 'Timed out waiting for Harness %s (last status: %s).\n' "$harness_id" "${status:-unknown}" >&2
return 1
}
wait_for_memory_status() {
local memory_id="$1" status=""
for _ in {1..60}; do
status="$(aws_cli bedrock-agentcore-control get-memory --memory-id "$memory_id" --query memory.status --output text 2>/dev/null || true)"
[[ "$status" == "ACTIVE" ]] && return 0
[[ "$status" == "FAILED" ]] && { printf 'Memory %s entered FAILED.\n' "$memory_id" >&2; return 1; }
sleep 5
done
printf 'Timed out waiting for Memory %s (last status: %s).\n' "$memory_id" "${status:-unknown}" >&2
return 1
}
wait_for_endpoint_status() {
local harness_id="$1" endpoint_name="$2" status=""
for _ in {1..60}; do
status="$(aws_cli bedrock-agentcore-control get-harness-endpoint --harness-id "$harness_id" --endpoint-name "$endpoint_name" --query endpoint.status --output text 2>/dev/null || true)"
[[ "$status" == "READY" ]] && return 0
[[ "$status" == "FAILED" ]] && { printf 'Harness endpoint %s entered FAILED.\n' "$endpoint_name" >&2; return 1; }
sleep 5
done
printf 'Timed out waiting for Harness endpoint %s (last status: %s).\n' "$endpoint_name" "${status:-unknown}" >&2
return 1
}
normalize_principal() {
local arn="$1"
if [[ "$arn" =~ ^arn:(aws[^:]*):sts::([0-9]{12}):assumed-role/(.+)/[^/]+$ ]]; then
# STS omits an IAM role's path from assumed-role ARNs. This matters for
# Identity Center roles, whose canonical ARN lives under
# /aws-reserved/sso.amazonaws.com/. Resolve it instead of constructing a
# role ARN that may not exist.
local role_name canonical_arn
role_name="${BASH_REMATCH[3]##*/}"
canonical_arn="$(aws_cli iam get-role --role-name "$role_name" --query Role.Arn --output text)"
[[ "$canonical_arn" =~ ^arn:(aws[^:]*):iam::[0-9]{12}:role/.+$ ]] || {
printf 'Could not resolve the canonical IAM role ARN for %s.\n' "$role_name" >&2
exit 1
}
printf '%s\n' "$canonical_arn"
elif [[ "$arn" =~ ^arn:(aws[^:]*):iam::[0-9]{12}:(role|user)/.+$ ]]; then
printf '%s\n' "$arn"
else
printf 'The caller ARN cannot be normalized to a stable IAM role/user: %s\n' "$arn" >&2
exit 1
fi
}
load_local_env() {
[[ -f "$ENV_FILE" ]] || { printf 'Missing %s; run aws-agentcore:provision first.\n' "$ENV_FILE" >&2; exit 1; }
local requested_profile="$AWS_PROFILE_NAME" requested_region="$AWS_REGION_NAME" requested_stack="$STACK_NAME"
# This file is generated locally, mode 0600, and contains nonsecret metadata only.
set -a
# shellcheck disable=SC1090
source "$ENV_FILE"
set +a
if $AWS_PROFILE_EXPLICIT; then AWS_PROFILE_NAME="$requested_profile"; else AWS_PROFILE_NAME="${AWS_PROFILE:-$requested_profile}"; fi
if $AWS_REGION_EXPLICIT; then AWS_REGION_NAME="$requested_region"; else AWS_REGION_NAME="${AWS_REGION:-$requested_region}"; fi
if $STACK_NAME_EXPLICIT; then STACK_NAME="$requested_stack"; else STACK_NAME="${PAPERCLIP_AWS_AGENTCORE_STACK_NAME:-$requested_stack}"; fi
export AWS_PROFILE="$AWS_PROFILE_NAME"
export AWS_REGION="$AWS_REGION_NAME"
export AWS_DEFAULT_REGION="$AWS_REGION_NAME"
export PAPERCLIP_AWS_AGENTCORE_STACK_NAME="$STACK_NAME"
}
write_teardown_metadata() {
mkdir -p "$LOCAL_DIR"
umask 077
local tmp="$ENV_FILE.tmp.$$"
printf '%q\n' \
"AWS_PROFILE=$AWS_PROFILE_NAME" \
"AWS_REGION=$AWS_REGION_NAME" \
"AWS_DEFAULT_REGION=$AWS_REGION_NAME" \
"PAPERCLIP_AWS_AGENTCORE_STACK_NAME=$STACK_NAME" >"$tmp"
if [[ -n "${AWS_CONFIG_FILE:-}" ]]; then
printf 'AWS_CONFIG_FILE=%q\n' "$AWS_CONFIG_FILE" >>"$tmp"
fi
chmod 600 "$tmp"
mv "$tmp" "$ENV_FILE"
}
assume_runner_role() {
local role_arn="$1"
local response access secret token
if ! response="$(aws_cli sts assume-role --role-arn "$role_arn" --role-session-name paperclip-agentcore-probe --duration-seconds 900 --output json 2>&1)"; then
local caller
caller="$(aws_cli sts get-caller-identity --query Arn --output text 2>/dev/null || printf '<caller-arn>')"
printf 'Cannot assume %s. Add this statement to the caller policy for %s:\n' "$role_arn" "$caller" >&2
printf '{"Effect":"Allow","Action":"sts:AssumeRole","Resource":"%s"}\n' "$role_arn" >&2
printf '%s\n' "$response" | sed -E 's/(ASIA|AKIA)[A-Z0-9]+/[REDACTED]/g' >&2
return 1
fi
access="$(printf '%s' "$response" | json_field Credentials.AccessKeyId)"
secret="$(printf '%s' "$response" | json_field Credentials.SecretAccessKey)"
token="$(printf '%s' "$response" | json_field Credentials.SessionToken)"
export AWS_ACCESS_KEY_ID="$access" AWS_SECRET_ACCESS_KEY="$secret" AWS_SESSION_TOKEN="$token"
unset AWS_PROFILE
}
probe() {
load_local_env
local profile_save="$AWS_PROFILE_NAME" harness_json
assume_runner_role "$PAPERCLIP_AWS_AGENTCORE_INVOCATION_ROLE_ARN"
AWS_PROFILE_NAME=""
harness_json="$(aws --region "$AWS_REGION_NAME" --no-cli-pager bedrock-agentcore-control get-harness \
--harness-id "$PAPERCLIP_AWS_AGENTCORE_HARNESS_ID" \
--harness-version "$PAPERCLIP_AWS_AGENTCORE_HARNESS_VERSION" --output json)"
printf '%s' "$harness_json" | node -e '
const fs = require("fs");
const h = JSON.parse(fs.readFileSync(0, "utf8")).harness;
const expectedModel = process.env.PAPERCLIP_AWS_AGENTCORE_MODEL;
if (!h || h.model?.bedrockModelConfig?.modelId !== expectedModel) throw new Error("AgentCore model drift");
if (JSON.stringify(h.allowedTools) !== JSON.stringify(["@*/pc_*"])) throw new Error("AgentCore tool allowlist drift");
if (!Array.isArray(h.tools) || h.tools.length !== 0) throw new Error("AgentCore persistent tool drift");
if (!Array.isArray(h.skills) || h.skills.length !== 0) throw new Error("AgentCore skill drift");
'
aws --region "$AWS_REGION_NAME" --no-cli-pager bedrock-agentcore-control get-harness-endpoint \
--harness-id "$PAPERCLIP_AWS_AGENTCORE_HARNESS_ID" \
--endpoint-name "$PAPERCLIP_AWS_AGENTCORE_ENDPOINT_QUALIFIER" >/dev/null
aws --region "$AWS_REGION_NAME" --no-cli-pager bedrock-agentcore-control get-memory \
--memory-id "$PAPERCLIP_AWS_AGENTCORE_MEMORY_ID" --view full >/dev/null
[[ -n "$PAPERCLIP_AWS_AGENTCORE_CONTEXT_BUCKET" && -n "$PAPERCLIP_AWS_AGENTCORE_CONTEXT_PREFIX" && -n "$PAPERCLIP_AWS_AGENTCORE_CONTEXT_KMS_KEY_ARN" ]] || {
printf 'AWS AgentCore profile is missing its qualified S3/KMS runtime context.\n' >&2
exit 1
}
unset AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN
AWS_PROFILE_NAME="$profile_save"
printf 'AWS AgentCore profile qualified: %s (%s, Harness %s, Memory %s; 90-day retention).\n' \
"$PAPERCLIP_AWS_AGENTCORE_PROFILE_ID" "$AWS_REGION_NAME" \
"$PAPERCLIP_AWS_AGENTCORE_HARNESS_VERSION" "$PAPERCLIP_AWS_AGENTCORE_MEMORY_ID"
}
provision() {
require_aws
local identity caller_arn account_id partition model_resource_arn foundation_model_id foundation_model_resource_arn
identity="$(aws_cli sts get-caller-identity --output json)"
caller_arn="$(printf '%s' "$identity" | json_field Arn)"
account_id="$(printf '%s' "$identity" | json_field Account)"
partition="$(printf '%s' "$caller_arn" | cut -d: -f2)"
model_resource_arn="arn:$partition:bedrock:$AWS_REGION_NAME:$account_id:inference-profile/$MODEL_ID"
foundation_model_id="$(printf '%s' "$MODEL_ID" | sed -E 's/^(global|us|eu|apac)\.//')"
foundation_model_resource_arn="arn:$partition:bedrock:*::foundation-model/$foundation_model_id"
if [[ -z "$TRUSTED_PRINCIPAL" ]]; then TRUSTED_PRINCIPAL="$(normalize_principal "$caller_arn")"; fi
if [[ "$DEPLOYMENT_MODE" == "private" ]]; then
aws_cli bedrock-agentcore-control list-harnesses --max-results 1 >/dev/null || {
printf 'Private mode is unavailable because AgentCore Harness is not enabled in %s.\n' "$AWS_REGION_NAME" >&2
exit 1
}
local services
services="$(aws_cli ec2 describe-vpc-endpoint-services --query 'ServiceNames' --output text)"
for service in "com.amazonaws.$AWS_REGION_NAME.ecr.api" "com.amazonaws.$AWS_REGION_NAME.ecr.dkr" "com.amazonaws.$AWS_REGION_NAME.s3" "com.amazonaws.$AWS_REGION_NAME.bedrock-runtime"; do
[[ " $services " == *" $service "* ]] || { printf 'Private mode is unavailable: missing VPC endpoint service %s\n' "$service" >&2; exit 1; }
done
fi
aws_cli cloudformation validate-template --template-body "file://$TEMPLATE" >/dev/null
printf 'Stack: %s\nRegion: %s\nMode: %s\nModel: %s\nTrusted principal: %s\n' \
"$STACK_NAME" "$AWS_REGION_NAME" "$DEPLOYMENT_MODE" "$MODEL_ID" "$TRUSTED_PRINCIPAL"
printf 'Expected charges: Bedrock model tokens, AgentCore Runtime active time, Memory storage/requests, and (private mode) VPC endpoints.\n'
if $DRY_RUN; then
printf 'Dry run complete; the template is syntactically valid and no resources were changed.\n'
return
fi
local existing_stack="" existing_status=""
if existing_stack="$(aws_cli cloudformation describe-stacks --stack-name "$STACK_NAME" --output json 2>&1)"; then
local existing_description existing_owned existing_environment existing_cost_center
existing_status="$(printf '%s' "$existing_stack" | json_field Stacks.0.StackStatus)"
existing_description="$(printf '%s' "$existing_stack" | json_field Stacks.0.Description)"
existing_owned="$(printf '%s' "$existing_stack" | stack_tag_value paperclip:owned)"
existing_environment="$(printf '%s' "$existing_stack" | stack_tag_value paperclip:environment)"
existing_cost_center="$(printf '%s' "$existing_stack" | stack_tag_value paperclip:cost-center)"
if [[ -z "$existing_status" || "$existing_description" != "$STACK_DESCRIPTION" || "$existing_owned" != "true" || "$existing_environment" != "development" || "$existing_cost_center" != "runner-lab" ]]; then
printf 'Refusing to modify stack %s: Paperclip ownership tags or template provenance do not match. Inspect or remove it manually, or choose another --stack-name.\n' "$STACK_NAME" >&2
exit 1
fi
elif [[ "$existing_stack" == *"(ValidationError)"* && "$existing_stack" == *"does not exist"* ]]; then
existing_stack=""
else
printf 'Unable to verify whether stack %s exists and is owned by Paperclip; refusing to provision.\n' "$STACK_NAME" >&2
exit 1
fi
if [[ "$existing_status" == "ROLLBACK_COMPLETE" ]]; then
if ! $REPLACE_FAILED_STACK; then
printf 'Stack %s is ROLLBACK_COMPLETE. Re-run with --replace-failed-stack to explicitly authorize deleting this verified Paperclip stack.\n' "$STACK_NAME" >&2
exit 1
fi
if ! $YES; then
printf 'Delete verified Paperclip stack %s before retrying provisioning? [y/N] ' "$STACK_NAME"
read -r answer || true
[[ "$answer" == y || "$answer" == Y ]] || { printf 'Cancelled.\n'; return; }
fi
printf 'Removing verified failed Paperclip stack %s before retrying provisioning.\n' "$STACK_NAME"
aws_cli cloudformation delete-stack --stack-name "$STACK_NAME"
aws_cli cloudformation wait stack-delete-complete --stack-name "$STACK_NAME"
fi
aws_cli cloudformation deploy \
--template-file "$TEMPLATE" \
--stack-name "$STACK_NAME" \
--capabilities CAPABILITY_NAMED_IAM \
--parameter-overrides \
"EnvironmentName=development" "DeploymentMode=$DEPLOYMENT_MODE" \
"HarnessEndpointName=$ENDPOINT_NAME" \
"ContextPrefix=$CONTEXT_PREFIX" \
"TrustedRunnerPrincipalArn=$TRUSTED_PRINCIPAL" "BedrockModelId=$MODEL_ID" \
"BedrockModelResourceArn=$model_resource_arn" \
"BedrockFoundationModelResourceArn=$foundation_model_resource_arn" \
"BedrockMarketplaceProductId=$MARKETPLACE_PRODUCT_ID" \
--tags paperclip:owned=true paperclip:environment=development paperclip:cost-center=runner-lab
# CloudFormation has created the billable resources at this point. Persist
# the nonsecret coordinates needed by `destroy` before any qualification
# API call can fail, then replace this checkpoint with the full qualified
# profile only after every resource is ready.
write_teardown_metadata
local harness_id harness_arn harness_version runtime_arn memory_id memory_arn role_arn context_bucket context_prefix context_kms_key_arn qualification_revision
harness_id="$(stack_output HarnessId)"
harness_arn="$(stack_output HarnessArn)"
harness_version="$(stack_output HarnessVersion)"
runtime_arn="$(stack_output AgentRuntimeArn)"
memory_id="$(stack_output MemoryId)"
memory_arn="$(stack_output MemoryArn)"
role_arn="$(stack_output RunnerInvocationRoleArn)"
context_bucket="$(stack_output ContextBucketName)"
context_prefix="$(stack_output ContextPrefix)"
context_kms_key_arn="$(stack_output ContextKmsKeyArn)"
qualification_revision="$(stack_output QualificationRevision)"
wait_for_harness_status "$harness_id" "$harness_version"
wait_for_memory_status "$memory_id"
if aws_cli bedrock-agentcore-control get-harness-endpoint --harness-id "$harness_id" --endpoint-name "$ENDPOINT_NAME" >/dev/null 2>&1; then
aws_cli bedrock-agentcore-control update-harness-endpoint --harness-id "$harness_id" --endpoint-name "$ENDPOINT_NAME" \
--target-version "$harness_version" --description "Paperclip pinned Harness endpoint" \
--client-token "paperclip-update-$harness_id-$harness_version" >/dev/null
else
aws_cli bedrock-agentcore-control create-harness-endpoint --harness-id "$harness_id" --endpoint-name "$ENDPOINT_NAME" \
--target-version "$harness_version" --description "Paperclip pinned Harness endpoint" \
--client-token "paperclip-create-$harness_id-$harness_version" \
--tags paperclip:owned=true,paperclip:environment=development >/dev/null
fi
wait_for_endpoint_status "$harness_id" "$ENDPOINT_NAME"
local endpoint_json endpoint_arn
endpoint_json="$(aws_cli bedrock-agentcore-control get-harness-endpoint --harness-id "$harness_id" --endpoint-name "$ENDPOINT_NAME" --output json)"
endpoint_arn="$(printf '%s' "$endpoint_json" | node -e 'const fs=require("fs");const o=JSON.parse(fs.readFileSync(0,"utf8"));process.stdout.write(o.endpoint?.arn||o.harnessEndpointArn||o.endpointArn||o.arn||"")')"
[[ -n "$endpoint_arn" ]] || endpoint_arn="$harness_arn/endpoint/$ENDPOINT_NAME"
mkdir -p "$LOCAL_DIR"
umask 077
local tmp="$ENV_FILE.tmp.$$"
printf '%q\n' \
"AWS_PROFILE=$AWS_PROFILE_NAME" \
"AWS_REGION=$AWS_REGION_NAME" \
"AWS_DEFAULT_REGION=$AWS_REGION_NAME" \
"PAPERCLIP_AWS_AGENTCORE_STACK_NAME=$STACK_NAME" \
"PAPERCLIP_AWS_AGENTCORE_PROFILE_ID=$STACK_NAME" \
"PAPERCLIP_AWS_AGENTCORE_ACCOUNT_ID=$account_id" \
"PAPERCLIP_AWS_AGENTCORE_HARNESS_ARN=$harness_arn" \
"PAPERCLIP_AWS_AGENTCORE_HARNESS_ID=$harness_id" \
"PAPERCLIP_AWS_AGENTCORE_HARNESS_VERSION=$harness_version" \
"PAPERCLIP_AWS_AGENTCORE_ENDPOINT_ARN=$endpoint_arn" \
"PAPERCLIP_AWS_AGENTCORE_ENDPOINT_QUALIFIER=$ENDPOINT_NAME" \
"PAPERCLIP_AWS_AGENTCORE_RUNTIME_ARN=$runtime_arn" \
"PAPERCLIP_AWS_AGENTCORE_MEMORY_ARN=$memory_arn" \
"PAPERCLIP_AWS_AGENTCORE_MEMORY_ID=$memory_id" \
"PAPERCLIP_AWS_AGENTCORE_INVOCATION_ROLE_ARN=$role_arn" \
"PAPERCLIP_AWS_AGENTCORE_CONTEXT_BUCKET=$context_bucket" \
"PAPERCLIP_AWS_AGENTCORE_CONTEXT_PREFIX=$context_prefix" \
"PAPERCLIP_AWS_AGENTCORE_CONTEXT_KMS_KEY_ARN=$context_kms_key_arn" \
"PAPERCLIP_AWS_AGENTCORE_MODEL=$MODEL_ID" \
"PAPERCLIP_AWS_AGENTCORE_QUALIFICATION_REVISION=$qualification_revision" \
"PAPERCLIP_AWS_AGENTCORE_EVENT_EXPIRY_DAYS=90" >"$tmp"
if [[ -n "${AWS_CONFIG_FILE:-}" ]]; then
printf 'AWS_CONFIG_FILE=%q\n' "$AWS_CONFIG_FILE" >>"$tmp"
fi
chmod 600 "$tmp"
mv "$tmp" "$ENV_FILE"
probe
printf 'Wrote nonsecret local profile metadata to %s (0600).\n' "$ENV_FILE"
}
lab() {
load_local_env
if [[ -f "$ACTIVE_FILE" ]] && kill -0 "$(cat "$ACTIVE_FILE")" 2>/dev/null; then
printf 'Runner Lab is already active as PID %s.\n' "$(cat "$ACTIVE_FILE")" >&2
exit 1
fi
printf '%s\n' "$$" >"$ACTIVE_FILE"
chmod 600 "$ACTIVE_FILE"
trap 'rm -f "$ACTIVE_FILE"' EXIT INT TERM
cd "$PACKAGE_DIR"
pnpm console:issue-thread
}
destroy() {
require_aws
load_local_env
if [[ -f "$ACTIVE_FILE" ]] && kill -0 "$(cat "$ACTIVE_FILE")" 2>/dev/null && ! $FORCE; then
printf 'Runner Lab PID %s is active; stop it or pass --force.\n' "$(cat "$ACTIVE_FILE")" >&2
exit 1
fi
if ! $YES; then
printf 'Delete the pinned AgentCore endpoint and CloudFormation stack %s? [y/N] ' "$STACK_NAME"
read -r answer
[[ "$answer" == y || "$answer" == Y ]] || { printf 'Cancelled.\n'; return; }
fi
local harness_id context_bucket context_prefix profile_save
harness_id="$(stack_output HarnessId)"
context_bucket="$(stack_output ContextBucketName)"
context_prefix="$(stack_output ContextPrefix)"
aws_cli bedrock-agentcore-control delete-harness-endpoint --harness-id "$harness_id" --endpoint-name "$ENDPOINT_NAME" \
--client-token "paperclip-delete-$harness_id-$ENDPOINT_NAME" >/dev/null 2>&1 || true
profile_save="$AWS_PROFILE_NAME"
assume_runner_role "$(stack_output RunnerInvocationRoleArn)"
aws --region "$AWS_REGION_NAME" --no-cli-pager s3 rm "s3://$context_bucket/$context_prefix/assets/" --recursive >/dev/null
unset AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN
AWS_PROFILE_NAME="$profile_save"
aws_cli cloudformation delete-stack --stack-name "$STACK_NAME"
aws_cli cloudformation wait stack-delete-complete --stack-name "$STACK_NAME"
rm -f "$ENV_FILE" "$ACTIVE_FILE"
printf 'Deleted AgentCore endpoint and stack. Local profile metadata was removed.\n'
}
case "$ACTION" in
provision) provision ;;
probe) require_aws; probe ;;
lab) lab ;;
destroy) destroy ;;
esac