Files
PaperClipAI/packages/paperclip-runner/protocol
DottaandPaperclip 612d9931a6 Support per-turn Runner sessions on ephemeral web servers (#18932)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The web server controls native Runner sessions in remote sandboxes.
> - An ephemeral web server must recover without its previous local
filesystem.
> - The existing admission rule accepts warm sessions. Managed Claude,
Grok, and OpenCode sessions resolve to per-turn execution.
> - Per-turn execution also needs durable provider-session bytes after
the sandbox is retired.
> - This pull request adds encrypted backup publication and removes the
warm-only admission rule.
> - Both lifecycle modes can use the existing native ownership and
checkpoint contracts during a web deployment.

## Linked Issues or Issue Description

Refs: #18518, #18529, #18826, #18835, #18841, #18864, #18887.

**What existing behavior does this improve?**

Native Runner execution on web servers with
`PAPERCLIP_EPHEMERAL_SERVER=true`.

**Current behavior**

The admission rule requires a warm Runner lifecycle. Managed provider
credentials can resolve an agent to per-turn execution, so enabling the
ephemeral deployment profile blocks those turns. A database controller
checkpoint does not preserve provider files after the old web home and
per-turn sandbox are removed.

**Proposed behavior**

Admit native execution in either resolved lifecycle. Keep provider and
credential choices unchanged. Publish encrypted provider state to shared
storage before sandbox cleanup. Restore it from a fresh web home for the
next turn. Use the existing controller lease and generation checks for
active takeover.

**Reason and benefit**

Operators can remove the web server volume without excluding per-turn
providers or forcing managed accounts into warm sessions. New native
Daytona profiles automatically enter both rolling-deployment
qualification lanes.

**Breaking changes**

The additive migration creates a backup publication ledger. Existing
volume-backed local-disk installations retain their local backup path.
S3 installations publish encrypted backups and require a stable secrets
master key. Existing-volume enrollment must import surviving session
data before detaching the volume.

## What Changed

- Admit native Runner execution independently of its warm or per-turn
lifecycle. Keep host-owned legacy execution excluded from ephemeral
deployments.
- Store private provider state as encrypted objects. Journal uploads
before I/O, read them back, and fence publication against the current
native owner. Keep the previous published backup when storage or
ownership checks fail.
- Restore exact provider bytes into a private cache. Require an exact
durable backup before sandbox cleanup. Extend the existing bounded
retention pass to retire stored objects without adding idle polling.
- Import selected legacy controller and provider state in the existing
offline maintenance process. Preserve queued work, task pointers,
pending recovery, and original files. Prefer canonical database
checkpoints.
- Preserve already-failed legacy controllers with missing original
artifact proof as encrypted archives. These archives do not become
execution authority. Successful retained sessions still require proof.
Report archival and executable imports separately.
- Verify unmanaged Grok API keys through the existing xAI account
validator. The live test exposed a readiness gap that kept remote Grok
agents paused after platform checks. Preserve failed credentials and the
separate runtime warning.
- Ingest Grok whole-prompt usage at the owning ACP boundary. Bind the
receipt to the exact outbound request/session; reject replay, conflicts,
and incomplete authority. Separate turn cost from cumulative session
cost in the sidecar schema and Rust normalization. Preserve managed
billing identity and identify direct xAI API credentials.
- Preserve published workspace directory names across cold allocations,
including destroyed-allocation cleanup. Keep unpublished epoch/lease
directories isolated and refuse to overwrite an occupied preparation.
This fixes the cold-session identity regression exposed by #18922.
- Increase bounded checkpoint-header inspection to 1 MiB. Retain the
aggregate inspection limit.
- Register all nine native Daytona profiles in per-turn and warm
rolling-deployment suites. Delete the entire departing server home,
exercise ordinary reads and writes, require active ownership transfer,
then check a later turn on another fresh server. Permit only exact
notification replay after reconnect.

Ownership stays with the existing modules: Runner validates its protocol
state; Core validates run scope and controller ownership; storage owns
object I/O; the maintenance process owns offline conversion; the E2E
wrapper owns process replacement and probes. No provider protocol is
added to the shared admission rule. Grok receipt parsing stays in its
provider module. The shared sidecar gains an explicit cost basis because
its existing price field means session-cumulative spend, while Grok
reports prompt totals. Existing consumers retain cumulative behavior
when the new field is absent.

## Verification

- Focused native admission, archive validation, publication fencing,
cleanup, readiness, legacy import, and E2E observer tests pass. Real
PostgreSQL cases verify that queued intent and historical recovery rows
stay unchanged.
- Full typecheck and build pass on the rebased branch. The review
follow-up passes server typecheck, 29 focused
PostgreSQL/archive/readiness tests, and module-boundary checks. Revision
fb8d39a7a1 reached 55 successful CI checks and Greptile 5/5 with no
unresolved threads. The Grok follow-up passes Runner/server typechecks,
238 ACP runtime/projection/usage tests, 16 sidecar-schema tests, the
Rust cost regression, and Core billing regression. The e9aae9cf revision
passed all 57 successful checks, with 5 skips and 2 neutral checks, and
exact-head Greptile 5/5. The latest 9cbec30a4f follow-up changes only
E2E catalog/admission controls; its 291 affected tests pass, all 57
check contexts succeeded (5 skips, 2 neutral), and exact-head Greptile
is 5/5 with no open review threads.
- Migration numbering and safety checks pass.
- A local real PostgreSQL/S3 roundtrip preserved 2,870,101,591 payload
bytes across 89,100 archive entries after deleting the source directory.
Restored SHA-256 matched; upload/readback took 56.5 seconds, restore
74.8 seconds, process peak RSS was 1.40 GB, and all fixture resources
were removed.
- Archives preserve contained relative links, reject link
chains/traversal, and enforce independent byte/entry/object limits.
Offline verification streams hashes. Orphaned deleted-run backups stay
on the original volume. Obsolete object retention preserves the selected
backup, live lease references, and pending uploads with owners.
- Isolated API-key Codex deployment: original active runner/session
transferred to the replacement and committed. All 567 reads, 57 writes,
and 57 readbacks succeeded. A second replacement completed a
same-session continuation with all 287 reads and 29 writes succeeding.
The strict cell failed two agent-authored shell commands; this result is
retained as failed.
- Isolated API-key Claude deployment: original active runner/session
transferred and committed. All 584 reads, 59 writes, and 59 readbacks
succeeded. A second replacement completed a same-provider-session
continuation, with 211 reads and 22 writes succeeding. The strict cell
failed an initial completion call that omitted a registered attachment
citation; the agent corrected it. This result is retained as failed.
- Isolated API-key OpenCode deployment: the same active runner/session
transferred and committed. All 481 reads, 49 writes, and 49 readbacks
succeeded. A second replacement retained the provider session and
completed useful work, with 156 reads and 16 writes succeeding. The
strict cell failed agent tool calls: a denied external-directory write,
malformed artifact hashes, and a second completion after the first
result was committed. There was one submitted/completed turn and one
committed result; reconnect notification replay was identical. This
result is retained as failed.
- Grok's real first turn now completes with settled usage and provider
cost ($0.41397); the next turn is admitted normally. That attempt then
exposed a shared workspace-path identity regression before deployment.
Three cold-restore cases reproduced it before the fix; 106
workspace/transport tests and four focused cold-restore cases pass
afterward, including destroyed allocation and occupied-directory
preservation. Server typecheck and build pass. At b5b0dbb03f both
controller switches passed the read/write probe without failure, and the
active runner retained its process while the controller generation
advanced. The third turn exposed an invalid reasoning-token subset
check; ad07e681ba fixes it with the actual receipt regression. Two later
live attempts failed with an upstream Grok rate limit before deployment.
A tiny preflight prompt succeeded before the second attempt, but the
full workload was still rate-limited. The final live gate remains
incomplete. No accounting or identity guard was bypassed. Earlier failed
attempts and their exact resource-cleanup receipts remain recorded.
- Earlier Codex/Claude/OpenCode receipts predate #18922; they establish
their tested revisions. Current-head live qualification remains pending;
#18943 separately preserves pre-enrollment logical/provider workspace
addresses and adds actual legacy migration proof. Its first Codex
migration attempt stopped safely at skill-version verification.
- Earlier attempts remain recorded: missing local provider-pack setup
blocked one Claude launch, and the observer rejected identical reconnect
replay in another. Setup and the observer were corrected before the
latest run.
- Named staging enrollment has not started. Its serving revision and
post-deployment journeys remain separate delivery gates.

- Broader E2E harness validation exposed deployment-catalog and
installed-lane coverage drift; the affected 291 controls now pass.
Twelve other failures reproduce on unchanged base 1a39dea9aa. They
concern cleanup call expectations, dated operating-overhead fixtures, a
Pi profile-version expectation and workflow-security expectations. No
baseline failure is reported as passing.

## Risks

- Provider archives can contain credentials. Objects use authenticated
encryption with scoped associated data and a wrapped data key. The
original stable master key remains required.
- Storage failure prevents publication and sandbox cleanup. Operators
must keep shared storage available and preserve the previous volume
during enrollment.
- Legacy state without verified identity cannot become executable
recovery authority. Archival preserves evidence and existing failures;
it does not recover already-lost provider files or clear blocked tasks.
- Local overlap probes establish the tested application behavior. They
do not establish Railway routing behavior or live managed-subscription
qualification.

## Model Used

OpenAI Codex, based on GPT-6, with reasoning, tool use, code execution,
and live integration tests. The exact runtime variant and context-window
size are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-04 02:12:51 -06:00
..

PRP v1/v2 Contract

The JSON Schema files in schemas/ are the language-neutral source of truth for Paperclip Runner Protocol versions 1 and 2. The fixtures in fixtures/ define accepted and rejected compatibility cases.

Compatibility

  • protocolVersion, fixtureVersion, and event.schemaVersion are required.
  • A consumer fails closed when a required version or schema discriminator is not supported.
  • A v1 envelope can contain an unknown optional property when its schema marks that object as extensible.
  • A consumer ignores an unknown optional property until a later contract gives it meaning.
  • A required field, enum value, or typed structured-input field is not optional.
  • Question and answer identifiers are stable across the provider boundary.
  • Peers negotiate the highest mutually supported protocol version. Existing v1 runners remain compatible but cannot receive v2-only session-goal commands.

The unknown-optional-fields.json fixture must be accepted. The unsupported-required-version.json fixture must be rejected.

Session goals (v2)

PRP v2 adds a provider-neutral durable session-goal lifecycle. Every v2 capability snapshot includes sessionGoals, even when its availability is unsupported or policy_disabled. Paperclip sends controls only when the negotiated capability advertises the corresponding action.

Commands:

  • session.goal.get
  • session.goal.set for objective, status, and optional token budget changes
  • session.goal.clear

Events:

  • session.capabilities.updated
  • session.goal.snapshot
  • session.goal.updated
  • session.goal.cleared

Goal state is separate from Paperclip's company/business goal hierarchy. The snapshot distinguishes the durable status from workingNow, because an active goal can be idle between autonomous turns. A runner emits the full capability and authoritative snapshot after every session open or resume. Missing v1 capability is unsupported; clients do not infer support from an adapter name.

Harness credential updates

auth.credentials.updated is a P0 runner event available in the v1, v2, and v3 event schemas. Its payload uses paperclip.auth.credentials.updated.v1 and contains only provider, bindingId, revision, and detection. No tokens, account identifiers, credential paths, or credential fingerprints are permitted.

An updated consumer opts in with observeCredentials: true in run.prepare, before session.open. The bundled runner transport opts in for fresh sessions; set its observeCredentials option to false to disable observation for a fresh session. Older hosts that omit the field receive no new event. Consumers must update their schemas before opting in: an event-enum addition is not an unknown optional property. Existing runner binaries ignore the optional flag and do not provide observation; absence of an event does not establish support.

The current Unix runner observes native Codex's managed CODEX_HOME/auth.json and the Codex/Grok ACPX session homes. It uses a non-recursive directory watch plus a one-second polling fallback, so atomic file replacement is handled. Grok's initial inline subscription credential is the baseline, allowing a refresh during launcher startup to be detected. A first valid credential file without a prior baseline establishes the baseline without claiming a refresh. Account identity changes also emit updates, even when token values stay the same, and establish the baseline for later refreshes on the new account. Hosts must evaluate the current credential through their private credential channel and account checks; the event does not authorize an account switch. Metadata-only changes, partial writes, and deletion do not emit updates. Unsupported harnesses and refreshes retained only in memory need their own observer or native-hook integration; adding this event does not make them observable.

bindingId: "harness-auth" identifies the selected harness authentication within the envelope's normalized session. Revisions increase for that binding and survive runner restarts and warm run attachments. The baseline fingerprint and pending event are persisted together in a private runner state file. Source retries retain the same event identity until the durable PRP outbox accepts it. Hosts should commit an idempotent reaction or durable work item through onCommittedEvent before returning and advancing the cumulative ACK. The bundled transport exposes onCredentialsUpdated(event) for this commit. It awaits the callback before ACK, including during warm idle, attachment, and provider cleanup. Callback failures leave delivery unacknowledged for retry; callbacks must therefore deduplicate by the original source event identity. When configured, credential updates bypass the turn notification queue. Paperclip commits them directly to the run log in a separate credential source namespace, preserving the original PRP envelope and avoiding collisions with the driver's normalized event sequence. A turn-only notification consumer cannot reliably handle idle or cleanup updates; use this callback instead.

detection: "credential_store" reports an observed credential change, not proof that an OAuth refresh succeeded. Multiple changes between observations can be coalesced; this is not an audit of every refresh exchange. The schema also allows native_hook for integrations with an explicit signal. Observation continues during warm idle and checks at turn terminal and close boundaries, including Grok's private refresh handoff after live-store cleanup. Retained runner observations enter the durable outbox before a suspend or shutdown command completes, without draining the provider backlog. A final check after provider cleanup also enters the outbox and waits for the host ACK. The runner retains its terminal recovery fence if that delivery fails.

The event authorizes no credential access or write on its own. A host that needs rotated token values must use a separately authorized handoff with its existing account-identity and freshness safeguards. The default Paperclip credential copy-back behavior remains in place.

Scope

The first provider descriptor and adapter fixture cover Codex only. The schemas for provider-neutral events and semantic receipts do not enable those actions. Discovery and authorization are separate contracts.

The conformance manifest records every source file and its SHA-256 digest. Run pnpm generate:protocol-manifest from this package after a source change. CI runs the same generator with --check to reject drift. This check also compiles the JSON Schemas and validates every replay, question, and cross-language conformance fixture against its declared schema.

The files in fixtures/replay/golden/ are deterministic reducer oracles. Each accepted replay fixture has a complete session snapshot and a compact parity summary. pnpm generate:replay-goldens updates them after an intentional reducer change; package build and CI fail when they drift.

The files in fixtures/local-runner/scripts/ drive the package-local fake harness. They cover successful, failed, interrupted, interactive, duplicate terminal, process-cleanup, and oversized-frame behavior without starting a production adapter.

runtime.activity.recorded is a P1 Runner event in v1, v2 and v3 with exactly { namespace, activityKey }. Both opaque identifiers contain 1–160 Unicode code points and exclude C0 controls and DEL. namespace groups a producer's activity types; activityKey names an activity within that group. Consumers use the pair to select a response to activity in the run. The authenticated outer envelope provides run and session identity, source sequence and emission time. Core resolves the run's company, task and agent through its existing authorized run binding; callers cannot supply those identities in the payload. Recording activity does not independently verify the reported action or grant access to any resource. The event uses the ordinary durable outbox, acknowledgement and replay contract.

Local clients capture an opaque context with paperclip-runnerd notification-context before an operation and invoke record-activity CONTEXT NAMESPACE ACTIVITY_KEY after success. Runner publishes its matching absolute executable as host-owned PAPERCLIP_RUNTIME_NOTIFICATION_HELPER and the stable endpoint descriptor path as PAPERCLIP_RUNTIME_NOTIFICATION_CONTEXT. Configured task environment cannot override either value. Context is invalidated on warm attachment and process recovery, so a caller cannot attribute a later completion to a different run.