Add Paperclip page publishing skill (#8664)

Reviewed by CTO for PAP-12039. Additive operational skill with validated S3/CloudFront publishing helper, defensive overwrite/symlink/hidden-file checks, focused Node tests, and full green CI/security/Greptile checks. Merged via maintainer path after CODEOWNERS review request was accepted as good enough for now.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed 2026-06-26 13:49:13 -05:00
1 parent 574543d7d3
commit de3b143fef
4 files changed
+1278

No files matched your search

+621
View File
@@ -0,0 +1,621 @@
# Paperclip Page Skill
`paperclip-page` publishes static page directories to a Paperclip-controlled S3
bucket served through CloudFront. It is the durable Paperclip-owned replacement
for quick `here.now`-style page sharing.
The v1 security posture is:
- CloudFront + ACM + Origin Access Control in front of a private S3 REST origin.
- Public content only.
- Dedicated uploader IAM identity, separate from Paperclip attachment storage.
- No `s3:DeleteObject`, no `aws s3 sync --delete`, and no bucket/IAM/DNS changes
from the publish helper.
- Symlinks, hidden files, unsafe slugs, and accidental overwrites are rejected.
## Agent Quick Start
Build or prepare a static directory with `index.html` at its root:
```bash
site/
index.html
assets/app.css
assets/app.js
```
Validate without AWS writes:
```bash
skills/paperclip-page/scripts/publish.sh ./site --slug demo --dry-run
```
Publish:
```bash
skills/paperclip-page/scripts/publish.sh ./site --slug demo
```
Update an existing page from the same source directory:
```bash
skills/paperclip-page/scripts/publish.sh ./site --slug demo --update
```
The helper prints:
- public URL
- S3 key prefix
- local ownership state path
## Source Directory Rules
- `index.html` must exist at the directory root.
- Source directory itself must not be a symlink.
- No symlinks anywhere in the tree.
- No hidden files or dot paths in published content.
- `.paperclip-page/state.json` is allowed and excluded from uploads.
- Do not publish secrets, credentials, internal logs, private company material,
customer data, or regulated data.
Add this to the publishing repo or generated site `.gitignore` when the source
directory lives in a git checkout:
```gitignore
.paperclip-page/
```
## Environment Variables
Required for live publishes:
```bash
export AWS_REGION=us-east-1
export PAPERCLIP_PAGE_BUCKET=paperclip-pages-prod
export PAPERCLIP_PAGE_BASE_URL=https://pages.paperclip.ing
export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
```
Optional:
```bash
export PAPERCLIP_PAGE_DEFAULT_PREFIX=""
export PAPERCLIP_PAGE_AWS_PROFILE=paperclip-page-uploader
```
Recommended Paperclip secret names:
- `paperclip-page-aws-access-key-id`
- `paperclip-page-aws-secret-access-key`
Bind those secrets into publisher agents as `AWS_ACCESS_KEY_ID` and
`AWS_SECRET_ACCESS_KEY`. Do not reuse Paperclip's internal S3 attachment/object
storage credentials.
## AWS Setup
Run setup with an operator/admin AWS profile. Agents using this skill should not
create buckets, mutate IAM, change DNS, or manage CloudFront.
```bash
export AWS_PROFILE=paperclip-admin
export AWS_REGION=us-east-1
export BUCKET=paperclip-pages-prod
export DOMAIN=pages.paperclip.ing
export UPLOADER_USER=paperclip-page-uploader
export CLOUDFRONT_COMMENT="Paperclip pages"
aws sts get-caller-identity --profile "$AWS_PROFILE"
```
Create the bucket:
```bash
aws s3api create-bucket \
--profile "$AWS_PROFILE" \
--region "$AWS_REGION" \
--bucket "$BUCKET"
```
For regions other than `us-east-1`, add:
```bash
--create-bucket-configuration LocationConstraint="$AWS_REGION"
```
Disable ACLs and keep ownership bucket-enforced:
```bash
aws s3api put-bucket-ownership-controls \
--profile "$AWS_PROFILE" \
--bucket "$BUCKET" \
--ownership-controls '{"Rules":[{"ObjectOwnership":"BucketOwnerEnforced"}]}'
```
Block public access. CloudFront reads through OAC, so the bucket does not need a
public website policy:
```bash
aws s3api put-public-access-block \
--profile "$AWS_PROFILE" \
--bucket "$BUCKET" \
--public-access-block-configuration \
'BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true'
```
Enable versioning:
```bash
aws s3api put-bucket-versioning \
--profile "$AWS_PROFILE" \
--bucket "$BUCKET" \
--versioning-configuration Status=Enabled
```
Enable default encryption:
```bash
aws s3api put-bucket-encryption \
--profile "$AWS_PROFILE" \
--bucket "$BUCKET" \
--server-side-encryption-configuration '{"Rules":[{"ApplyServerSideEncryptionByDefault":{"SSEAlgorithm":"AES256"}}]}'
```
Upload an operator-managed root `404.html` before creating the distribution:
```bash
mkdir -p /tmp/paperclip-pages-bootstrap
printf '<!doctype html><title>Not found</title><h1>Not found</h1>\n' \
> /tmp/paperclip-pages-bootstrap/404.html
aws s3 cp /tmp/paperclip-pages-bootstrap/404.html "s3://$BUCKET/404.html" \
--profile "$AWS_PROFILE" \
--content-type text/html \
--cache-control 'public,max-age=60'
```
Create an ACM certificate in `us-east-1` for CloudFront:
```bash
export ACM_REGION=us-east-1
aws acm request-certificate \
--profile "$AWS_PROFILE" \
--region "$ACM_REGION" \
--domain-name "$DOMAIN" \
--validation-method DNS \
--idempotency-token paperclippages \
> /tmp/paperclip-pages-acm.json
export CERT_ARN="$(jq -r '.CertificateArn' /tmp/paperclip-pages-acm.json)"
aws acm describe-certificate \
--profile "$AWS_PROFILE" \
--region "$ACM_REGION" \
--certificate-arn "$CERT_ARN" \
--query 'Certificate.DomainValidationOptions[].ResourceRecord'
```
Add the returned DNS validation record in Cloudflare, then wait:
```bash
aws acm wait certificate-validated \
--profile "$AWS_PROFILE" \
--region "$ACM_REGION" \
--certificate-arn "$CERT_ARN"
```
Create a CloudFront Origin Access Control:
```bash
aws cloudfront create-origin-access-control \
--profile "$AWS_PROFILE" \
--origin-access-control-config "{
\"Name\":\"paperclip-pages-oac\",
\"Description\":\"OAC for $BUCKET\",
\"SigningProtocol\":\"sigv4\",
\"SigningBehavior\":\"always\",
\"OriginAccessControlOriginType\":\"s3\"
}" \
> /tmp/paperclip-pages-oac.json
export OAC_ID="$(jq -r '.OriginAccessControl.Id' /tmp/paperclip-pages-oac.json)"
```
Create and publish a CloudFront Function so clean page URLs such as `/demo/`
load `/demo/index.html` from the S3 REST origin:
```bash
cat > paperclip-pages-index-router.js <<'EOF'
function handler(event) {
var request = event.request;
var uri = request.uri;
if (uri.endsWith('/')) {
request.uri = uri + 'index.html';
return request;
}
var lastSegment = uri.substring(uri.lastIndexOf('/') + 1);
if (lastSegment.indexOf('.') === -1) {
request.uri = uri + '/index.html';
}
return request;
}
EOF
aws cloudfront create-function \
--profile "$AWS_PROFILE" \
--name paperclip-pages-index-router \
--function-config 'Comment=Rewrite clean page URLs to index.html,Runtime=cloudfront-js-2.0' \
--function-code fileb://paperclip-pages-index-router.js \
> /tmp/paperclip-pages-function.json
export FUNCTION_ETAG="$(jq -r '.ETag' /tmp/paperclip-pages-function.json)"
aws cloudfront publish-function \
--profile "$AWS_PROFILE" \
--name paperclip-pages-index-router \
--if-match "$FUNCTION_ETAG" \
> /tmp/paperclip-pages-function-live.json
export FUNCTION_ARN="$(jq -r '.FunctionSummary.FunctionMetadata.FunctionARN' /tmp/paperclip-pages-function-live.json)"
```
Create `cloudfront-config.json`:
```bash
export CALLER_REFERENCE="paperclip-pages-$(date +%s)"
jq -n \
--arg caller "$CALLER_REFERENCE" \
--arg comment "$CLOUDFRONT_COMMENT" \
--arg domain "$DOMAIN" \
--arg bucket "$BUCKET" \
--arg oac "$OAC_ID" \
--arg functionArn "$FUNCTION_ARN" \
--arg cert "$CERT_ARN" \
'{
CallerReference: $caller,
Comment: $comment,
Enabled: true,
IsIPV6Enabled: true,
Aliases: {Quantity: 1, Items: [$domain]},
Origins: {
Quantity: 1,
Items: [{
Id: "s3-origin",
DomainName: ($bucket + ".s3.amazonaws.com"),
OriginAccessControlId: $oac,
S3OriginConfig: {OriginAccessIdentity: ""}
}]
},
DefaultRootObject: "index.html",
DefaultCacheBehavior: {
TargetOriginId: "s3-origin",
ViewerProtocolPolicy: "redirect-to-https",
AllowedMethods: {Quantity: 2, Items: ["GET", "HEAD"], CachedMethods: {Quantity: 2, Items: ["GET", "HEAD"]}},
Compress: true,
CachePolicyId: "658327ea-f89d-4fab-a63d-7e88639e58f6",
OriginRequestPolicyId: "88a5eaf4-2fd4-4709-b370-b4c650ea3fcf",
FunctionAssociations: {
Quantity: 1,
Items: [{EventType: "viewer-request", FunctionARN: $functionArn}]
}
},
CustomErrorResponses: {
Quantity: 1,
Items: [{ErrorCode: 403, ResponsePagePath: "/404.html", ResponseCode: "404", ErrorCachingMinTTL: 60}]
},
ViewerCertificate: {
ACMCertificateArn: $cert,
SSLSupportMethod: "sni-only",
MinimumProtocolVersion: "TLSv1.2_2021"
},
Restrictions: {GeoRestriction: {RestrictionType: "none", Quantity: 0}}
}' > cloudfront-config.json
```
Create the distribution:
```bash
aws cloudfront create-distribution \
--profile "$AWS_PROFILE" \
--distribution-config file://cloudfront-config.json \
> /tmp/paperclip-pages-cloudfront.json
export DISTRIBUTION_ID="$(jq -r '.Distribution.Id' /tmp/paperclip-pages-cloudfront.json)"
export DISTRIBUTION_DOMAIN="$(jq -r '.Distribution.DomainName' /tmp/paperclip-pages-cloudfront.json)"
```
Grant CloudFront read access to the private bucket:
```bash
export ACCOUNT_ID="$(aws sts get-caller-identity --profile "$AWS_PROFILE" --query Account --output text)"
jq -n \
--arg bucket "$BUCKET" \
--arg account "$ACCOUNT_ID" \
--arg distribution "$DISTRIBUTION_ID" \
'{
Version: "2012-10-17",
Statement: [{
Sid: "AllowCloudFrontServicePrincipalReadOnly",
Effect: "Allow",
Principal: {Service: "cloudfront.amazonaws.com"},
Action: "s3:GetObject",
Resource: ("arn:aws:s3:::" + $bucket + "/*"),
Condition: {
StringEquals: {
"AWS:SourceArn": ("arn:aws:cloudfront::" + $account + ":distribution/" + $distribution)
}
}
}]
}' > bucket-policy.json
aws s3api put-bucket-policy \
--profile "$AWS_PROFILE" \
--bucket "$BUCKET" \
--policy file://bucket-policy.json
```
Create the uploader IAM user:
```bash
aws iam create-user \
--profile "$AWS_PROFILE" \
--user-name "$UPLOADER_USER"
```
Create `paperclip-page-uploader-policy.json`. This policy supports collision
checks and additive uploads under slug prefixes while protecting root bootstrap
objects such as `404.html`.
```bash
jq -n \
--arg bucket "$BUCKET" \
'{
Version: "2012-10-17",
Statement: [
{
Sid: "ListPublishedPagePrefixes",
Effect: "Allow",
Action: ["s3:ListBucket"],
Resource: ("arn:aws:s3:::" + $bucket)
},
{
Sid: "ReadPublishedPages",
Effect: "Allow",
Action: ["s3:GetObject"],
Resource: ("arn:aws:s3:::" + $bucket + "/*")
},
{
Sid: "WritePublishedPageObjects",
Effect: "Allow",
Action: ["s3:PutObject"],
Resource: ("arn:aws:s3:::" + $bucket + "/*/*")
},
{
Sid: "DenyReservedRootWrites",
Effect: "Deny",
Action: ["s3:PutObject", "s3:DeleteObject", "s3:PutObjectTagging"],
Resource: [
("arn:aws:s3:::" + $bucket + "/404.html"),
("arn:aws:s3:::" + $bucket + "/index.html")
]
}
]
}' > paperclip-page-uploader-policy.json
```
Attach it:
```bash
aws iam put-user-policy \
--profile "$AWS_PROFILE" \
--user-name "$UPLOADER_USER" \
--policy-name PaperclipPagePublisher \
--policy-document file://paperclip-page-uploader-policy.json
```
Create access keys and treat the output as secret material:
```bash
aws iam create-access-key \
--profile "$AWS_PROFILE" \
--user-name "$UPLOADER_USER" \
> /tmp/paperclip-page-uploader-key.json
chmod 600 /tmp/paperclip-page-uploader-key.json
```
## Cloudflare DNS
Use DNS-only or proxied CNAME to CloudFront. Do not point v1 at the S3 website
endpoint.
Cloudflare UI:
- Open the `paperclip.ing` zone.
- Add `CNAME`:
- Name: `pages`
- Target: the CloudFront domain, for example `d111111abcdef8.cloudfront.net`
- Proxy status: DNS only or Proxied
- TTL: Auto
API equivalent:
```bash
export CF_ZONE_ID=<paperclip.ing-zone-id>
export CF_API_TOKEN=<token-with-zone-dns-edit>
curl -sS -X POST "https://api.cloudflare.com/client/v4/zones/$CF_ZONE_ID/dns_records" \
-H "Authorization: Bearer $CF_API_TOKEN" \
-H "Content-Type: application/json" \
--data "$(jq -n \
--arg name pages \
--arg content "$DISTRIBUTION_DOMAIN" \
'{type:"CNAME", name:$name, content:$content, ttl:1, proxied:false}')"
```
Smoke check:
```bash
curl -I "https://$DOMAIN/404.html"
```
## Paperclip Secrets
Create secrets from environment variables so values do not land in shell history:
```bash
export PAPERCLIP_PAGE_AWS_ACCESS_KEY_ID="$(jq -r '.AccessKey.AccessKeyId' /tmp/paperclip-page-uploader-key.json)"
export PAPERCLIP_PAGE_AWS_SECRET_ACCESS_KEY="$(jq -r '.AccessKey.SecretAccessKey' /tmp/paperclip-page-uploader-key.json)"
pnpm paperclipai secrets create \
--company-id <company-id> \
--name paperclip-page-aws-access-key-id \
--value-env PAPERCLIP_PAGE_AWS_ACCESS_KEY_ID
pnpm paperclipai secrets create \
--company-id <company-id> \
--name paperclip-page-aws-secret-access-key \
--value-env PAPERCLIP_PAGE_AWS_SECRET_ACCESS_KEY
```
Bind runtime env to publishing agents:
```json
{
"AWS_ACCESS_KEY_ID": {
"type": "secret_ref",
"secretId": "<access-key-secret-id>",
"version": "latest"
},
"AWS_SECRET_ACCESS_KEY": {
"type": "secret_ref",
"secretId": "<secret-key-secret-id>",
"version": "latest"
},
"AWS_REGION": { "type": "plain", "value": "us-east-1" },
"PAPERCLIP_PAGE_BUCKET": { "type": "plain", "value": "paperclip-pages-prod" },
"PAPERCLIP_PAGE_BASE_URL": { "type": "plain", "value": "https://pages.paperclip.ing" },
"PAPERCLIP_PAGE_DEFAULT_PREFIX": { "type": "plain", "value": "" }
}
```
## Install And Attach
Create or update the company skill from this package:
```bash
pnpm paperclipai skills create \
--company-id <company-id> \
--name "Paperclip Page" \
--slug paperclip-page \
--description "Publish static pages to the Paperclip pages host" \
--body-file skills/paperclip-page/SKILL.md
```
Attach it to an agent:
```bash
pnpm paperclipai skills agent sync <agent-id-or-shortname> \
--company-id <company-id> \
--skill paperclip-page
```
Ensure the agent can read this directory or copy the package into the installed
company skill location with `scripts/publish.sh` preserved as executable.
## Credential Rotation
1. Create a second access key:
```bash
aws iam create-access-key \
--profile "$AWS_PROFILE" \
--user-name "$UPLOADER_USER" \
> /tmp/paperclip-page-uploader-key-rotation.json
chmod 600 /tmp/paperclip-page-uploader-key-rotation.json
```
2. Store new secret versions in Paperclip Secrets.
3. Update agent env bindings to the new versions or `latest`.
4. Run a dry-run and a small publish smoke.
5. Disable the old key:
```bash
aws iam update-access-key \
--profile "$AWS_PROFILE" \
--user-name "$UPLOADER_USER" \
--access-key-id <old-access-key-id> \
--status Inactive
```
6. Delete the old key after the next successful publish:
```bash
aws iam delete-access-key \
--profile "$AWS_PROFILE" \
--user-name "$UPLOADER_USER" \
--access-key-id <old-access-key-id>
```
## Troubleshooting
`AccessDenied` on upload:
- Confirm agent env contains the uploader key, not the admin key.
- Confirm the uploader policy allows `s3:ListBucket`, `s3:GetObject`, and
`s3:PutObject`.
- Confirm uploads target `<slug>/...` so the `arn:aws:s3:::<bucket>/*/*`
object ARN matches.
`Slug already exists`:
- Use a different slug.
- Or run `--update` from the original source directory that has
`.paperclip-page/state.json`.
Generated slug collides:
- The helper appends a short suffix for generated slugs when AWS reports a
collision. Explicit slugs fail instead of silently changing the URL.
URL 404s after upload:
- Check `curl -I https://<domain>/<slug>/`.
- Check CloudFront distribution deployment status.
- Check DNS CNAME target.
- Check the object exists at `s3://<bucket>/<slug>/index.html`.
Stale browser cache:
- The helper uses `Cache-Control: public,max-age=60`.
- Wait a minute or issue a CloudFront invalidation if the operator wants an
immediate refresh.
CloudFront returns 403:
- Confirm the bucket policy references the correct distribution ARN.
- Confirm OAC is attached to the S3 origin.
- Confirm the bucket is private and public access block is enabled.
## Public Content Security Notes
Anything published with this skill is public. The tool cannot reliably classify
generated files, so the publishing agent must inspect content before uploading.
Do not publish:
- API keys, OAuth tokens, cookies, or `.env` files
- internal customer data
- private company docs
- unpublished security reports
- raw transcripts that may contain secrets
Recovery after accidental overwrite uses S3 versioning. Because v1 uploader
credentials cannot delete objects, rollback should be performed by an operator
with admin credentials.
+90
View File
@@ -0,0 +1,90 @@
---
name: paperclip-page
description: >
Publish static HTML pages and asset folders to the Paperclip-approved S3 and
CloudFront website host. Use when asked to deploy, publish, host, or share a
persistent Paperclip page, wireframe viewer, prototype, report, or other static
site without using here.now.
---
# Paperclip Page
Use this skill to publish a static directory to the configured Paperclip pages
host, for example `https://pages.paperclip.ing/<slug>/`.
## Requirements
- Source directory contains `index.html` at its root.
- `aws` CLI v2, `curl`, and `jq` are available on PATH for live publishes.
- Environment variables are configured:
- `PAPERCLIP_PAGE_BUCKET`
- `PAPERCLIP_PAGE_BASE_URL`
- `AWS_REGION`
- AWS credentials via Paperclip Secrets or an approved AWS vault
- Optional environment variables:
- `PAPERCLIP_PAGE_DEFAULT_PREFIX`
- `PAPERCLIP_PAGE_AWS_PROFILE`
## Workflow
1. Inspect the source directory and confirm it is public static content only.
2. Run `scripts/publish.sh <dir> --dry-run` to validate local structure and see
the resolved URL/prefix.
3. Choose a slug:
- Use `--slug <slug>` when the user gave a stable URL path.
- Omit `--slug` to derive one from the source directory name.
4. Publish:
```bash
skills/paperclip-page/scripts/publish.sh ./site --slug my-page
```
5. Return the printed public URL and S3 prefix to the issue/user.
## Update Workflow
Updates are additive overwrites only. The helper never deletes remote objects.
```bash
skills/paperclip-page/scripts/publish.sh ./site --slug my-page --update
```
When the target prefix already exists, `--update` requires local ownership proof
from `./site/.paperclip-page/state.json` generated by an earlier publish from
that same source directory. Without that state, create a new slug instead of
overwriting another page.
## Safety Rules
- Publish public content only. Do not publish secrets, customer data, private
company material, credentials, or internal logs.
- Never print AWS secret values.
- Never change bucket policy, IAM, DNS, CloudFront, or ACM settings from this
skill. Setup belongs to an operator runbook, not the publish helper.
- Never upload outside the configured bucket and prefix.
- Never use `aws s3 sync --delete` or require `s3:DeleteObject` in v1.
- The helper forces `--no-follow-symlinks` and fails if any source symlink is
present.
- The helper rejects hidden files and dot-segment paths except its own
`.paperclip-page/state.json`.
- Slugs and prefix segments must use lowercase ASCII letters, digits, and
hyphens only.
- Keep site-wide root objects such as `404.html` operator-managed; publishes
always target `<slug>/...` or `<default-prefix>/<slug>/...`.
## Troubleshooting
- `Slug already exists`: choose a different slug or use `--update` from the
original source directory containing `.paperclip-page/state.json`.
- `Missing index.html`: build the static site first or point the helper at the
directory that contains the root HTML file.
- `Found symlink`: replace symlinks with real files before publishing.
- `AccessDenied`: confirm the uploader IAM policy allows `ListBucket`,
`GetObject`, and `PutObject` for the configured bucket/prefix, and that the
agent received the Paperclip Secrets.
- Public URL verification failed: check CloudFront deployment/DNS, object
existence, and that the distribution uses HTTPS with the private S3 REST
origin.
See `README.md` next to this skill for operator setup, AWS policy examples,
credential rotation, and install/attach commands.
+388
View File
@@ -0,0 +1,388 @@
#!/usr/bin/env bash
set -euo pipefail
usage() {
cat <<'EOF'
Usage:
publish.sh <dir> [--slug slug] [--update] [--dry-run]
Publishes a static directory with a root index.html to the configured Paperclip
pages bucket and prints the public URL and S3 prefix.
Required environment for live publish:
PAPERCLIP_PAGE_BUCKET, PAPERCLIP_PAGE_BASE_URL, AWS_REGION, AWS credentials
Optional environment:
PAPERCLIP_PAGE_DEFAULT_PREFIX, PAPERCLIP_PAGE_AWS_PROFILE
Options:
--slug SLUG Lowercase URL slug. Allowed: a-z, 0-9, hyphen.
--update Additively overwrite an owned existing prefix. Never deletes.
--dry-run Validate and print the planned target without AWS writes.
--help, -h Show this help.
EOF
}
die() {
printf 'paperclip-page: %s\n' "$*" >&2
exit 1
}
require_command() {
if ! command -v "$1" >/dev/null 2>&1; then
die "missing required command: $1"
fi
}
normalize_base_url() {
local value="$1"
value="${value%/}"
[[ "$value" == https://* ]] || die "PAPERCLIP_PAGE_BASE_URL must be an https URL"
[[ ! "$value" =~ [[:space:]] ]] || die "PAPERCLIP_PAGE_BASE_URL cannot contain whitespace"
printf '%s\n' "$value"
}
validate_segment() {
local value="$1"
local label="$2"
[[ -n "$value" ]] || die "$label cannot be empty"
[[ "${#value}" -le 64 ]] || die "$label is too long; max length is 64 characters"
[[ "$value" =~ ^[a-z0-9]([a-z0-9-]*[a-z0-9])?$ ]] || die "$label must use lowercase letters, digits, and hyphens only"
[[ "$value" != "." && "$value" != ".." ]] || die "$label cannot be a dot segment"
}
normalize_slug() {
local value="$1"
value="${value#/}"
value="${value%/}"
[[ "$value" != *"/"* ]] || die "slug must be one path segment, not a nested path"
validate_segment "$value" "slug"
case "$value" in
404|404-html|index|index-html|root|assets)
die "slug '$value' is reserved"
;;
esac
printf '%s\n' "$value"
}
derive_slug() {
local source_dir="$1"
local base
base="$(basename "$source_dir")"
base="$(printf '%s' "$base" | tr '[:upper:]' '[:lower:]' | sed -E 's/[^a-z0-9]+/-/g; s/^-+//; s/-+$//; s/-{2,}/-/g')"
if [[ -z "$base" ]]; then
base="paperclip-page"
fi
printf '%.48s\n' "$base" | sed -E 's/-+$//'
}
random_suffix() {
if command -v openssl >/dev/null 2>&1; then
openssl rand -hex 3
else
od -An -N3 -tx1 /dev/urandom | tr -d ' \n'
fi
}
normalize_default_prefix() {
local raw="${1:-}"
local segment
local normalized=""
raw="${raw#/}"
raw="${raw%/}"
[[ "$raw" != *"//"* ]] || die "PAPERCLIP_PAGE_DEFAULT_PREFIX cannot contain empty path segments"
if [[ -z "$raw" ]]; then
printf '\n'
return
fi
IFS='/' read -r -a segments <<<"$raw"
for segment in "${segments[@]}"; do
validate_segment "$segment" "prefix segment"
if [[ -z "$normalized" ]]; then
normalized="$segment"
else
normalized="$normalized/$segment"
fi
done
printf '%s\n' "$normalized"
}
join_prefix() {
local default_prefix="$1"
local slug="$2"
if [[ -n "$default_prefix" ]]; then
printf '%s/%s/\n' "$default_prefix" "$slug"
else
printf '%s/\n' "$slug"
fi
}
aws_base_args=()
aws_cli() {
aws "${aws_base_args[@]}" "$@"
}
object_exists() {
local bucket="$1"
local prefix="$2"
local key
key="$(aws_cli s3api list-objects-v2 \
--bucket "$bucket" \
--prefix "$prefix" \
--max-keys 1 \
--query 'Contents[0].Key' \
--output text)"
[[ "$key" != "None" && -n "$key" ]]
}
assert_safe_source_tree() {
local source_dir="$1"
local found
[[ ! -L "$source_dir" ]] || die "source directory must not be a symlink"
[[ -f "$source_dir/index.html" ]] || die "source directory must contain root index.html"
found="$(find "$source_dir" -type l -print -quit)"
[[ -z "$found" ]] || die "found symlink in source tree: $found"
found="$(
cd "$source_dir"
find . -mindepth 1 \
\( -path './.paperclip-page' -o -path './.paperclip-page/*' \) -prune -o \
\( -name '.*' -o -path '*/.*' \) -print -quit
)"
[[ -z "$found" ]] || die "hidden files and dot paths are not allowed in published content: $found"
}
read_state_value() {
local state_file="$1"
local expression="$2"
jq -r "$expression // empty" "$state_file"
}
assert_update_ownership() {
local source_dir="$1"
local bucket="$2"
local prefix="$3"
local state_file="$source_dir/.paperclip-page/state.json"
local state_bucket
local state_prefix
[[ -f "$state_file" ]] || die "update of an existing prefix requires ownership state at $state_file"
state_bucket="$(read_state_value "$state_file" '.bucket')"
state_prefix="$(read_state_value "$state_file" '.prefix')"
[[ "$state_bucket" == "$bucket" ]] || die "state bucket does not match target bucket"
[[ "$state_prefix" == "$prefix" ]] || die "state prefix does not match target prefix"
}
compute_source_hash() {
local source_dir="$1"
if ! command -v sha256sum >/dev/null 2>&1; then
printf 'unavailable\n'
return
fi
(
cd "$source_dir"
find . -type f ! -path './.paperclip-page/*' -print0 |
LC_ALL=C sort -z |
while IFS= read -r -d '' path; do
sha256sum "$path"
done
) | sha256sum | awk '{print $1}'
}
write_state() {
local source_dir="$1"
local bucket="$2"
local prefix="$3"
local slug="$4"
local url="$5"
local base_url="$6"
local source_hash="$7"
local state_dir="$source_dir/.paperclip-page"
local state_file="$state_dir/state.json"
local temp_file
mkdir -p "$state_dir"
temp_file="$(mktemp "$state_dir/state.json.tmp.XXXXXX")"
jq -n \
--arg bucket "$bucket" \
--arg prefix "$prefix" \
--arg slug "$slug" \
--arg url "$url" \
--arg baseUrl "$base_url" \
--arg publishedAt "$(date -u +'%Y-%m-%dT%H:%M:%SZ')" \
--arg sourceHash "$source_hash" \
'{
bucket: $bucket,
prefix: $prefix,
slug: $slug,
url: $url,
baseUrl: $baseUrl,
publishedAt: $publishedAt,
sourceHash: $sourceHash,
version: 1
}' >"$temp_file"
mv "$temp_file" "$state_file"
}
source_arg=""
slug_arg=""
update=0
dry_run=0
while [[ $# -gt 0 ]]; do
case "$1" in
--slug)
slug_arg="${2:-}"
shift 2
;;
--update)
update=1
shift
;;
--dry-run)
dry_run=1
shift
;;
--help|-h)
usage
exit 0
;;
--*)
die "unknown argument: $1"
;;
*)
if [[ -n "$source_arg" ]]; then
die "unexpected positional argument: $1"
fi
source_arg="$1"
shift
;;
esac
done
[[ -n "$source_arg" ]] || {
usage >&2
exit 1
}
require_command jq
require_command find
require_command sed
[[ -d "$source_arg" ]] || die "source path is not a directory: $source_arg"
source_dir="$(cd "$source_arg" && pwd -P)"
assert_safe_source_tree "$source_dir"
bucket="${PAPERCLIP_PAGE_BUCKET:-}"
base_url="${PAPERCLIP_PAGE_BASE_URL:-}"
region="${AWS_REGION:-}"
default_prefix="$(normalize_default_prefix "${PAPERCLIP_PAGE_DEFAULT_PREFIX:-}")"
[[ -n "$bucket" ]] || die "PAPERCLIP_PAGE_BUCKET is required"
[[ "$bucket" =~ ^[a-z0-9][a-z0-9.-]{1,61}[a-z0-9]$ ]] || die "PAPERCLIP_PAGE_BUCKET does not look like a valid S3 bucket name"
[[ -n "$base_url" ]] || die "PAPERCLIP_PAGE_BASE_URL is required"
base_url="$(normalize_base_url "$base_url")"
explicit_slug=0
if [[ -n "$slug_arg" ]]; then
explicit_slug=1
slug="$(normalize_slug "$slug_arg")"
else
slug="$(normalize_slug "$(derive_slug "$source_dir")")"
fi
if [[ "$dry_run" == "0" ]]; then
require_command aws
require_command curl
[[ -n "$region" ]] || die "AWS_REGION is required for live publish"
aws_base_args=(--region "$region")
if [[ -n "${PAPERCLIP_PAGE_AWS_PROFILE:-}" ]]; then
aws_base_args+=(--profile "$PAPERCLIP_PAGE_AWS_PROFILE")
fi
fi
prefix="$(join_prefix "$default_prefix" "$slug")"
target_exists=0
if [[ "$update" == "1" ]]; then
assert_update_ownership "$source_dir" "$bucket" "$prefix"
fi
if [[ "$dry_run" == "0" ]]; then
if object_exists "$bucket" "$prefix"; then
target_exists=1
fi
if [[ "$target_exists" == "1" && "$update" == "0" ]]; then
if [[ "$explicit_slug" == "1" ]]; then
die "slug already exists: $slug. Use --update from the owning source directory or choose a new slug."
fi
for _ in 1 2 3 4 5; do
candidate="${slug}-$(random_suffix)"
candidate="$(printf '%.64s' "$candidate" | sed -E 's/-+$//')"
validate_segment "$candidate" "generated slug"
candidate_prefix="$(join_prefix "$default_prefix" "$candidate")"
if ! object_exists "$bucket" "$candidate_prefix"; then
slug="$candidate"
prefix="$candidate_prefix"
target_exists=0
break
fi
done
[[ "$target_exists" == "0" ]] || die "could not find an unused generated slug after 5 attempts"
fi
fi
url="${base_url}/${prefix}"
mode="publish"
if [[ "$update" == "1" ]]; then
mode="update"
fi
if [[ "$dry_run" == "1" ]]; then
cat <<EOF
paperclip-page dry run
mode: $mode
source: $source_dir
bucket: $bucket
prefix: $prefix
url: $url
EOF
exit 0
fi
aws_cli s3 sync "$source_dir/" "s3://$bucket/$prefix" \
--no-follow-symlinks \
--exclude '.paperclip-page/*' \
--cache-control 'public,max-age=60' \
--only-show-errors
source_hash="$(compute_source_hash "$source_dir")"
write_state "$source_dir" "$bucket" "$prefix" "$slug" "$url" "$base_url" "$source_hash"
curl -fsSIL --max-time 20 "$url" >/dev/null
cat <<EOF
paperclip-page published
mode: $mode
url: $url
bucket: $bucket
prefix: $prefix
state: $source_dir/.paperclip-page/state.json
EOF
@@ -0,0 +1,179 @@
import assert from "node:assert/strict";
import { execFileSync } from "node:child_process";
import {
chmodSync,
existsSync,
mkdirSync,
mkdtempSync,
readFileSync,
rmSync,
statSync,
writeFileSync,
} from "node:fs";
import { tmpdir } from "node:os";
import { dirname, join } from "node:path";
import { fileURLToPath } from "node:url";
import { after, test } from "node:test";
const scriptPath = join(dirname(fileURLToPath(import.meta.url)), "publish.sh");
const tempDirs = new Set();
after(() => {
for (const dir of tempDirs) {
rmSync(dir, { force: true, recursive: true });
}
});
function createSite(name = "paperclip-page-test") {
const siteDir = mkdtempSync(join(tmpdir(), `${name}-`));
tempDirs.add(siteDir);
writeFileSync(join(siteDir, "index.html"), "<!doctype html><title>Paperclip</title>\n");
return siteDir;
}
function writeExecutable(path, body) {
writeFileSync(path, body, { mode: 0o755 });
chmodSync(path, 0o755);
}
function writeState(siteDir, state) {
mkdirSync(join(siteDir, ".paperclip-page"), { recursive: true });
writeFileSync(join(siteDir, ".paperclip-page", "state.json"), `${JSON.stringify(state)}\n`);
}
function runPublish(args, env = {}) {
try {
return {
output: execFileSync("bash", [scriptPath, ...args], {
encoding: "utf8",
env: {
...process.env,
PAPERCLIP_PAGE_BUCKET: "paperclip-pages-test",
PAPERCLIP_PAGE_BASE_URL: "https://pages.example.test/",
...env,
},
stdio: ["ignore", "pipe", "pipe"],
}),
status: 0,
};
} catch (error) {
return {
output: `${error.stdout ?? ""}${error.stderr ?? ""}`,
status: error.status ?? 1,
};
}
}
test("publish helper stays executable", () => {
assert.equal(statSync(scriptPath).mode & 0o111, 0o111);
});
test("dry run validates and prints the planned target without requiring AWS", () => {
const result = runPublish([
createSite(),
"--slug",
"demo-page",
"--dry-run",
]);
assert.equal(result.status, 0);
assert.match(result.output, /^paperclip-page dry run$/m);
assert.match(result.output, /^mode: publish$/m);
assert.match(result.output, /^bucket: paperclip-pages-test$/m);
assert.match(result.output, /^prefix: demo-page\/$/m);
assert.match(result.output, /^url: https:\/\/pages\.example\.test\/demo-page\/$/m);
});
test("dry run normalizes a safe default prefix", () => {
const result = runPublish(
[createSite(), "--slug", "demo-page", "--dry-run"],
{ PAPERCLIP_PAGE_DEFAULT_PREFIX: "/reports/launches/" },
);
assert.equal(result.status, 0);
assert.match(result.output, /^prefix: reports\/launches\/demo-page\/$/m);
assert.match(result.output, /^url: https:\/\/pages\.example\.test\/reports\/launches\/demo-page\/$/m);
});
test("dry run update requires matching local ownership state", () => {
const siteDir = createSite();
const missingState = runPublish([siteDir, "--slug", "demo-page", "--update", "--dry-run"]);
assert.notEqual(missingState.status, 0);
assert.match(missingState.output, /requires ownership state/);
writeState(siteDir, {
bucket: "paperclip-pages-test",
prefix: "demo-page/",
});
const result = runPublish([siteDir, "--slug", "demo-page", "--update", "--dry-run"]);
assert.equal(result.status, 0);
assert.match(result.output, /^mode: update$/m);
});
test("rejects nested slugs", () => {
const result = runPublish([createSite(), "--slug", "nested/path", "--dry-run"]);
assert.notEqual(result.status, 0);
assert.match(result.output, /slug must be one path segment/);
});
test("rejects hidden files in the source tree", () => {
const siteDir = createSite();
mkdirSync(join(siteDir, "assets"));
writeFileSync(join(siteDir, "assets", ".secret"), "do not publish\n");
const result = runPublish([siteDir, "--slug", "demo-page", "--dry-run"]);
assert.notEqual(result.status, 0);
assert.match(result.output, /hidden files and dot paths are not allowed/);
});
test("live publish writes state before URL verification", () => {
const siteDir = createSite();
const binDir = mkdtempSync(join(tmpdir(), "paperclip-page-bin-"));
tempDirs.add(binDir);
writeExecutable(
join(binDir, "aws"),
`#!/usr/bin/env bash
set -euo pipefail
while [[ "$1" == "--region" || "$1" == "--profile" ]]; do
shift 2
done
if [[ "$1" == "s3api" ]]; then
echo "None"
exit 0
fi
if [[ "$1" == "s3" && "$2" == "sync" ]]; then
exit 0
fi
echo "unexpected aws call: $*" >&2
exit 1
`,
);
writeExecutable(
join(binDir, "curl"),
`#!/usr/bin/env bash
set -euo pipefail
echo "simulated CloudFront propagation miss" >&2
exit 22
`,
);
const result = runPublish([siteDir, "--slug", "demo-page"], {
AWS_REGION: "us-east-1",
PATH: `${binDir}:${process.env.PATH}`,
});
assert.notEqual(result.status, 0);
assert.match(result.output, /simulated CloudFront propagation miss/);
assert.equal(existsSync(join(siteDir, ".paperclip-page", "state.json")), true);
const state = JSON.parse(readFileSync(join(siteDir, ".paperclip-page", "state.json"), "utf8"));
assert.equal(state.bucket, "paperclip-pages-test");
assert.equal(state.prefix, "demo-page/");
assert.equal(state.url, "https://pages.example.test/demo-page/");
});