Files
PaperClipAI/tests/runner-e2e/ANTIGRAVITY-AGENT-JOURNEY.md
T
DottaandPaperclip ebed7f5237 test(evals): protect Antigravity setup and two-turn delivery (#18190)
## Thinking Path

> - Paperclip manages agents and their work.
> - A provider integration must work through ordinary agent setup and
task controls.
> - The recorded Antigravity walkthrough exposed setup, delivery, and
continuation failures.
> - A repeatable check must verify real work and retain failed attempts.
> - This pull request converts all 47 recorded steps into one fixed
Product E2E journey.
> - Maintainers can rerun the case and inspect independent delivery,
cleanup, and accounting evidence.

## Linked Issues or Issue Description

Builds on merged #18187, #18188, and #18189, and consumes the shared
repairs from merged #18230 and the equivalent manual-policy declaration
from #18222. Refs #18214 for the CI task wrapper. Refs #12503 and #14938
for related legacy adapter work.

**Agent or provider**

Google Antigravity native Runner with a saved managed Google
subscription and the exact Low model.

**Why this adapter is useful**

A user must be able to add an agent, connect its account, and receive
real task results across a follow-up. Picker visibility alone is
insufficient.

**How the agent is invoked**

A fixed Playwright driver uses ordinary product controls. The agent runs
through native ACP. Independent checks use public APIs, actual
downloads, owned process observations, and native lifetime fences.

## What Changed

- Register one explicit-only case:
`antigravity-agent-journey.runner-antigravity.local.saved-subscription-two-turn`.
- Map all 47 source steps and declare saved-account prerequisites,
finite limits, and same-host macOS ARM64 scope.
- Create an agent through the ordinary picker, connect the saved
account, select Low, run Test, and wait for readiness. Save an
instruction marker, deliver a file, and update it through an ordinary
follow-up.
- Require each run's visible completed file activity, exact downloaded
bytes, committed Done result, same tagged conversation, distinct
physical owner birth, released lease, and free native fences. Verify
reload persistence.
- Stop failed native runs that still own a retry. Replay an existing
Stop with its durable request identity and wait for the matching audited
acknowledgement. A terminal reservation without a native intent remains
unconfirmed.
- Keep private evidence in the existing result, report, dashboard, and
accounting pipeline. Retain the paused QA agent and financial history
while preserving the shared account and budget policies.
- Journal only the ordinary wizard's future ID, target, name, intended
harness and expected creator before forwarding its POST unchanged. Bind
the response and interrupted recovery to exact identity reads; a pending
404 stays unconfirmed.
- Serialize private resource snapshots through temporary-file fsync,
atomic rename and directory fsync. Abort forwarding on storage failure
and preserve legacy receipts. If later journal or diagnostic writes
fail, retain the primary error, always close the browser, and attempt
the ordinary failed report.
- Add production-component browser controls and harden task-menu
readiness, alias selection, scroll restoration, failure packaging, and
one hermetic auth-test budget.
- Wait for the existing GitHub recovery fixture to register its next
deadline before asserting the original range; preserve its
nested-transaction and zero-request checks. In the decision-expiry
fixture, verify the exact saved expiry separately from the earlier
global wake of another queue.

## Verification

- Fresh registered acceptance passed on eval source
`9060f3cd01b7c3445ecdf703d72e9cf30fe42b59` and merged serving product
`61bd4ac6919c2309f766a25e62273b074cee3261`. It completed ordinary setup
and two turns with native `.5` and `gemini-3.8-flash-low`. All 16
matchers, 14 checkpoints, and cleanup passed in **165,613 ms**. Actual
downloads were **61 bytes** and **119 bytes**. Both turns passed their
own run-bound browser activity checks and reload persistence.
- Independent read-back confirms Done, a paused retained agent, the same
connected saved default account, and no active owned run. Both accepted
results have the same full tagged conversation and distinct primary
process births. All 21 observed processes retired; both leases released
and all three native fences were free after each turn. The selected `.5`
checkpoints also showed equal agent/backend session IDs; this is an
observed case requirement, not a generic ACP invariant.
- Receipts report 339,273 input tokens, 7,224 output tokens, and 202,846
cached input tokens. Subscription prices are unavailable. Billing
coverage remains incomplete, not free.
- The normal report CLI passed. All 310 files from nine original
attempts and two separate recoveries retained their hashes. The twelve
current screenshots are byte-identical, and 194 local report links are
contained and valid. The eight earlier reports were copied without
regeneration.
- Delivery source `1bf4fb04df04710b82844b3c9355767f46b5f68f` branches
from `d3f2a60e795fc60d64c8b19d1eed4d48657d5492` and merges cleanly with
current master `1d571de5f9f886d66e520800baec8fc2f67d0948`. All 41 prior
`c8b332` authored file blobs remain identical; the 42nd path is the
decision-expiry fixture contract correction. Creation ownership, the
strict durable writer, and the ordinary successful workflow remain
unchanged. Of 43 measured file blobs, 30 remain identical. The
Antigravity oracle, artifact, billing and report gates and its
suite-specific validation block remain unchanged. The actual retained
agent matches its original creator; its original request `setupId` was
not captured and is not reconstructed.
- Later master changes include model-visible workspace guidance and
rejected-terminal-report handling. Their bounded source assessment is
separate: the first actual finish failed, then a same-turn replacement
was accepted and committed. Later read-only results match both original
native-result hashes, and the complete event catalogs have no
`session.failed` event. Master #18228 also expands detection of explicit
JSON file requests. The measured create/update-and-attach `.txt` task
already required publication under both parsers; the publication
validation gate itself is unchanged. Master #18240 changes remote
workspace cache staging and adds an OpenCode index refresh. The selected
local lane returns before those paths. Master #18213 adds optional
numeric turn timing and trace boundaries. Its selected Antigravity
model, text, IDs, result and retirement semantics remain unchanged in
the bounded source review. Local diagnostic rows and awaits can change
event ordinals and latency; the oracle checks relative semantic order.
Collector coverage and latency improvement remain outside this proof.
This does not claim that the whole current runtime is byte-identical to
serving `61bd`, or relabel the measured result as a new run.
- The adopted upstream cache controls passed 3/3 with pinned real Turbo.
Forced work disables local and remote reads and writes; ordinary work
reuses local results. The original genuine-pnpm/Turbo controls remain
historical to `9060`'s earlier write-only policy. The duplicate cache
patch is superseded by #18230. Its qualification-only pnpm addition is
removed because the adopted upstream controls do not require it.
- The E2E typecheck, map checks and 277-control replay are carried from
the unchanged eval source paths. That replay passed, including the 183
Antigravity controls, catalog/connection checks and the separate
product-journey validator. The carried creation controls include 91
config/recovery cases. The final launcher suite passed 19/19, including
five real campaign finalization fault cases; the report suite passed
7/7. These final checks did not rerun the earlier 277-control group. The
canonical production-component browser spec passed in 27.4 s under its
unchanged 60 s limit before the replay. Four paused-composer cases and
the product base's build/typecheck plus 376 focused regressions were
verified earlier.
- The shipped wizard and actual API client cover persistence before
forwarding, lost response JSON, foreign/malformed requests, wrong
response identity/creator and cleanup-error preservation. Real
filesystem controls failed 5/5 before the writer repair and passed 5/5
afterward, preserving the old journal on partial-write, temporary-file
sync and rename faults. Three persistent-sync campaign controls failed
before the finalization repair and passed afterward. All five final
campaign fault cases preserve the primary failure, cleanup failure,
usage and failed report while attempting browser closure. The strict
journal still blocks the next mutation. These are provider-free
ownership controls; the Google task proof remains the original `9060` →
`61bd` acceptance.
- The decision restart check passed in the local 20-case fixture run. A
current-source real-PostgreSQL rollback control reproduced the exact
90,000 ms earlier global wake: the old equality rejected it, while the
exact durable expiry and new future-wake bound passed. The original 60 s
open-state, exact-timestamp, continuation and quiet checks remain. The
broader local run was **FAILED: 19 passed / 1 failed**, with a 15 s
timeout and a 30 s cleanup timeout in an unchanged timestamp-pagination
case. That case passed in canonical `c8b332` CI; the local timeout cause
remains inconclusive. An extra local server typecheck failed against
stale cached Runner declarations; it did not check this excluded test
file. Both limits remain recorded for fresh canonical CI.
- A broader local GitHub recovery group passed 21/22. Its unchanged
pagination case observed an unexpected config request; the log does not
establish the cause. That result remains retained, and fresh
current-head CI is the final broader gate.

- Exact delivery-head CI completed with 55 successes, two skips, and one
optional neutral review. Required `ci / verify` and `ci / e2e`, build,
typecheck, and Runner checks passed. Fresh Greptile is 5/5 with zero
unresolved threads. Optional Paperclip is neutral 4/5; its carried
recommendations are dispositioned against the public Stop contract and
selected live identity evidence.

Seven earlier converted attempts retain their failed grades and partial
evidence. Historical Paid8 and its later read-only per-run UI
observation remain separate from the fresh acceptance above. Unknown
prices and the older partial startup accounting remain explicit. Full
lineage and raw evidence remain private in the normal report bundle.

## Risks

- This case requires an authorized saved personal default Google
subscription and finite operator candidate admission. It does not test
fresh OAuth or establish global qualification.
- The physical observer supports a same-host loopback macOS ARM64
target. Cloud, Linux, and other models remain outside this case.
- A known-cost stop cannot cap missing prices or in-flight subscription
charges. Existing operator policies remain unchanged.
- Raw account/native identities stay private. Missing retirement proof
keeps resources retained. The selected cleanup mode does not test agent
purge.

## Model Used

OpenAI Codex (GPT-6). The host did not expose the exact model revision
or context limit. The model used tools for code edits, tests, browser
automation, and review.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-29 19:51:32 -05:00

16 KiB

Antigravity additional-agent journey

antigravity-agent-journey.runner-antigravity.local.saved-subscription-two-turn converts the recorded 38-step setup/first-task and 9-step follow-up into one explicit-only Product E2E cell. It is excluded from --all and scheduled paid matrices. The conversion brief records all 47 source-step dispositions, immutable source hashes, original grades, earlier failures and evidence limits.

The fixed browser driver starts on the company dashboard, follows Agents → New Agent, selects Google Antigravity in the ordinary grid with Advanced collapsed, reuses the saved personal Google subscription, selects the exact low model and Local environment, and runs the native prompt-free Test. Finish must reach actual readiness before task assignment is enabled. It checks the first composer for a stale paused warning, closes that empty composer, saves a nonce only through the agent's Instructions, then submits a task asking for the configured marker without supplying its value. An ordinary follow-up asks the same agent/task to preserve the first line and append a different nonce. Each real file and Done state must remain available after reload.

Selected setup and dependencies

The runnable setup attaches to an existing self-hosted loopback target on the same macOS ARM64 host. This narrower implementation lets the independent native observer verify owned process retirement and all three public-checkpoint provider lifetime fences. .local. identifies agent execution. It does not itself identify application hosting. Compatible Cloud/staging behavior remains a portability target, not an implemented or verified remote observer path.

Required existing state:

  • A dedicated test company and local execution environment; expected and observed serving commit and deployment mode must match. The harness never resets or restarts an attached target. Its roster already contains a nonterminated agent, so the measured creation is an additional-agent path.
  • A dedicated signed-in QA browser profile for the intended board operator, with company agent/instruction/task creation and saved-connection use rights. The normal profile ownership marker and exclusive lock apply; do not use a normal personal Chrome profile.
  • The same person's saved managed Google subscription, with its existing personal default grant. The private setup names exact connection, owner, company and environment references. Names alone do not establish the binding.
  • A current, finite trusted operator startup admission for that company, gemini-3.8-flash-low, local darwin-arm64, native version, profile and closure. The config's pins assert provenance; they cannot grant admission. The global release remains pending and closed by default.
  • Existing operator accounting policy permitting the two task turns, with held and unpriced usage reconciled through its ordinary owner. The harness preserves that policy and the new agent's inherited policy; it creates no agent budget policy or QA override. Missing subscription prices remain unknown.

No MCP connection is required by the browser driver, product agent or independent observer. Native product tools are used by the agent. No API key, mailbox, billing-management credential, fresh Google login or OAuth registration is part of this case. Authentication freshness is saved browser session + saved product connection, not fresh provider consent. Product-managed refresh stays inside the normal managed-account boundary.

After verified native retirement, a refreshed Google credential transfers to the existing encrypted, fsynced recovery journal before database rotation. The next authorized task, Test or model discovery recovers it with the same grant, generation and account checks. A failed journal write or unconfirmed retirement keeps the private home and blocks cleanup; a retry does not grant new authority.

One-time test-drive/bootstrap setup remains the operator's supported workspace owner responsibility. This suite does not implement a managed-local target bootstrap, mint candidate authority, install a provider through a test-only path, or import an operator's CLI credential home. Install the maintained native assets through paperclipai runtime setup antigravity as the OS user running Paperclip. Then use the existing product UI to establish the saved account once. Per-attempt preflight verifies those declared prerequisites; the measured new agent must remain absent until the visible creation flow.

Private setup file

Create an owner-only regular JSON file outside the repository (chmod 600). Replace every placeholder with an authorized binding; never put token/cookie, password, OAuth code or startup authority bytes in it. Relative profileDir is resolved against this file's directory. The browser must declare signed-in. If board sign-in needs assistance, use setup-only --antigravity-journey-preflight to save the dedicated browser session, then start a fresh measured attempt. A measured attempt that needed assisted sign-in stops before agent or provider setup and retains a blocked report.

{
  "version": 1,
  "setupId": "local-antigravity-saved-subscription",
  "hosting": "self-hosted",
  "executionEnvironment": "local",
  "target": {
    "mode": "attach",
    "baseURL": "http://127.0.0.1:YOUR_PORT",
    "expectedCommit": "EXACT_40_CHARACTER_SERVING_COMMIT",
    "deploymentMode": "local_trusted",
    "companyId": "EXISTING_TEST_COMPANY_UUID",
    "environmentId": "EXISTING_LOCAL_ENVIRONMENT_UUID"
  },
  "browser": {
    "headed": true,
    "channel": "chrome",
    "profileDir": "./qa-browser",
    "accountAlias": "browser-operator",
    "freshness": "signed-in",
    "loginTimeoutMs": 600000
  },
  "savedConnection": {
    "id": "EXISTING_GOOGLE_CONNECTION_UUID",
    "name": "YOUR_SAVED_SUBSCRIPTION_LABEL",
    "ownerUserId": "EXPECTED_BROWSER_USER_ID"
  },
  "candidate": {
    "platform": "darwin",
    "architecture": "arm64",
    "nativeVersion": "1.4.3-paperclip.5",
    "profileDigest": "sha256:acf5f574b6b9ec4388ff9d081890caa330e66bf1ae3fe3b08e7f0fb0e851b9ce",
    "nativeClosureDigest": "sha256:044792d793e203bc8b4524a418f1816c4d34db67bb41fa4bae5947a050c49b4a"
  },
  "limits": {
    "durationMs": 1200000,
    "turnTimeoutMs": 300000,
    "maxRuns": 2,
    "budgetCents": 200
  },
  "retainAgent": true
}

Use authenticated when it is the target's actual deployment mode; do not change the server to match the example. The maximum whole attempt is 30 minutes and each turn at most 10 minutes. budgetCents is an observed known-cost stop threshold (at most 1,000 cents), not a hard currency cap or a product-policy write. It cannot cap missing prices, unknown subscription charges or already in-flight usage. The finite execution boundary is one selected case, one attempt, exactly two task runs, finite deadlines and the startup authority's TTL. Native Test performs account/model admission and cleanup without a model prompt. An unexpected work run or inference edge must be recorded, not omitted from accounting.

Commands

Run from the Paperclip repository root, with its normal workspace dependencies installed. The maintained source files own all behavior; no author-local helper script is required. Discovery and credential-free checks do no provider work:

pnpm test:e2e:runner -- --list --suite antigravity-agent-journey
pnpm test:e2e:runner:typecheck
pnpm test:e2e:runner:unit
pnpm test:e2e:runner:browser-support -- antigravity-journey-browser.spec.ts

The browser calibration is included in the canonical browser-support target and uses local fixtures, including the real activity renderer. It checks selectors and stale UI states without an account connection, an agent setup or a provider run; it is separate from live qualification.

Read-only readiness validates the target revision/mode, signed-in consuming identity, exact personal default grant, local environment and candidate catalog. It does not run native Test, create a task or dispatch a provider prompt:

pnpm test:e2e:runner -- \
  --id antigravity-agent-journey.runner-antigravity.local.saved-subscription-two-turn \
  --antigravity-journey-config /absolute/private/antigravity.json \
  --antigravity-journey-preflight \
  --max-automatic-retries 0

After readiness, run only the exact authorized cell. A new campaign preserves previous attempts and uses no automatic retry:

pnpm test:e2e:runner -- \
  --id antigravity-agent-journey.runner-antigravity.local.saved-subscription-two-turn \
  --antigravity-journey-config /absolute/private/antigravity.json \
  --max-automatic-retries 0

Campaigns live under tests/runner-e2e/results/antigravity-journey-<timestamp>-<nonce>/. Retain their normal result, evidence manifest, checkpoint images, actual byte/hash receipts, timings, effective setup/revisions, billing and cleanup alongside failures. Run the normal importer separately to verify a retained report; it makes zero new model calls. Environment variables, not positional report arguments, select it:

PAPERCLIP_RUNNER_E2E_REPORT_ROOT=/absolute/private/campaign \
PAPERCLIP_RUNNER_E2E_REPORT_OUT=/absolute/private/rebuilt-report \
PAPERCLIP_RUNNER_E2E_EXPECTED_IDS='["antigravity-agent-journey.runner-antigravity.local.saved-subscription-two-turn"]' \
  pnpm test:e2e:runner:report

The importer must preserve the attempt verdict, cleanup outcome, duration, known input/output/cache tokens and unknown prices. A rebuilt dashboard is a presentation check, not another provider measurement. Missing final proof cannot be repaired by inventing a screenshot or rewriting an old grade.

If packaging fails, attempt-result.json preserves the primary measurement and packaging-error.json records the separate reporting failure. Any partially written result and manifest are copied before the retained presentation is marked failed with journey_report_packaging_failed. Cleanup, usage and original screenshots remain available; the normal importer can rebuild that failed presentation without changing the original outcome or supplying missing proof.

Independent acceptance and retained evidence

Success requires the correct browser controls and persisted object identity, exact actual downloaded bytes for both distinct run-bound artifacts, requested and effective low model, same full tagged native conversation, accepted result, committed finalization and an applied Done decision for each own settlement. The session-ID equality gate is an observed property of this exact native .5 integration, corroborated by both retained real task checkpoints; it is not a general ACP invariant. The first settlement is captured before reopening; later legitimate reassessment may supersede its current pointer without invalidating that historical lineage. The final task's visible status and both outputs must survive reload. Narrative claims, hidden status labels, a matching filename or a stale first artifact do not satisfy those gates.

Native public lease release, exact owned process absence and reacquisition/release of all three observed provider fence ports establish local retirement. The two turns must have different independently captured primary Runner PID/OS-birth identities, each verified against the exact run and per_turn argv. Their session-scoped Runner instance ID may remain the same; a different sidecar does not prove a fresh primary. The observer reads no private database or raw provider reasoning. Its exact local target constraint is part of the case, not portable host proof. A genuine failed finish remains failed while prior successful native file activities stay completed; the happy path does not deliberately manufacture that rejection.

Synthetic positive/negative/missing-proof controls use the same oracle and normal report boundary. They cover wrong object/run/model/conversation, stale or wrong bytes, incomplete settlement/retirement, missing final screenshot and unknown price preservation. Browser calibration includes hidden duplicate/menu/SVG Done text beside the intended visible task status. Synthetic fixtures establish these gates, not Google behavior or discoverability.

Raw authenticated setup screenshots, concrete account/company/agent/task/native session identifiers, browser state and ownership receipts stay private. Do not mark these captures public-runner-fixture, publish original walkthrough HTML, or copy provider payloads into public evidence. Source aliases and hashes are safe provenance; any public projection remains the existing sanitizer's allowlist.

Cleanup and interrupted recovery

Before forwarding the ordinary wizard's draft POST unchanged, the campaign journals only its future setupId, exact company/name/origin, intended harness and preflight browser-user identity. It does not store the full request body or credentials. The private journal serializes snapshots, fsyncs a private temporary file, atomically renames it, and fsyncs its directory; failure aborts forwarding. The response must match that future ID and creator. If the response is lost, recovery reads that exact ID, verifies company/name/creator and uniqueness, then persists the verified ID before cleanup mutations. A pending creation returning 404 remains unconfirmed; a name match alone grants no ownership. Existing receipts with a known response and no new optional witness remain supported. The receipt binds target/company, execution/attempt and immutable created resources before later task dispatch. Cleanup stops only owned active work, verifies settlement and native retirement, then pauses/removes the owned agent according to retainAgent. Supplied company/environment/browser identity and saved Google connection/default grant remain intact and are reverified. It never revokes a shared credential or resets the target. Retained task history remains inspectable. Use retainAgent: true for accounted work: cleanup leaves the owned agent paused and preserves its task and cost history. Purge remains subject to the product's ledger constraints; the fixture never removes accounting rows to force deletion. For cancellation before native selection, cleanup instead persists a closed no-launch receipt from the server's settled preparation and explicit null native ownership fields, settled accounting, and exact company/run/environment lease. That lease must have a concrete release timestamp no later than preparation settlement; expired is accepted only in this no-launch variant. Missing or pending receipts retain the agent. This branch creates no physical capture proof.

A Test that fails or times out before the product confirms probe retirement leaves its setup probe marked unconfirmed. The harness keeps that agent paused and retained. Public task-run absence cannot prove that a setup probe has retired; recovery needs authoritative proof from the product cleanup owner.

On interruption, preserve the result and receipt first. Recover with the same private target binding and the exact receipt; no name-based broad cleanup:

pnpm test:e2e:runner -- \
  --id antigravity-agent-journey.runner-antigravity.local.saved-subscription-two-turn \
  --antigravity-journey-config /absolute/private/antigravity.json \
  --antigravity-journey-recover /absolute/private/campaign/owned-resources.json \
  --max-automatic-retries 0

Recovery validates ownership before stopping recorded/new owned-agent runs or removing its agent. An uncertain mutation is reconciled before retrying. Cleanup failure stays separate from primary failure; late recovery produces a separate receipt without upgrading the original cleanup grade. Do not start another paid attempt while owned work or the browser profile lock remains unresolved.

Verification scope

The source .5 browser walkthrough and independent retained oracle passed on the declared attended local saved-subscription/low-model candidate. Native instruction wire receipts were not archived; Instructions-only provenance plus actual file bytes proves resulting behavior. HTML gallery navigation was blocked by the host's local-file preview policy and is unverified. Earlier failed/partial .4 and package-admission attempts retain their original grades.

The converted fixed driver, credential-free calibration, fresh live attempt and normal report results are recorded separately in PR/attempt evidence. A registry listing or unit pass does not qualify a live provider. Fresh OAuth, API-key live operation, Cloud, staging/remote targets, other models/platforms, failure controls, reliability repetitions and unattended execution remain outside this selection.