Files
PaperClipAI/maps/feature-map/runs-adapters.md
T
DottaandPaperclip 820c458465 fix(runner): restore per-turn controller checkpoints before resume (#18018)
## Thinking Path

> - Paperclip coordinates agents and preserves their work between turns.
> - Remote native sessions save controller checkpoints in the database.
> - External persistence leaves a small identity marker on local disk.
> - A later `per_turn` continuation must restore that checkpoint before
it reads the controller identity.
> - The restore path handled warm sessions only, so valid per-turn
continuations failed before provider execution.
> - This PR restores the referenced checkpoint and retains the existing
ownership and lifecycle checks.

## Linked Issues or Issue Description

Related: #17439 added external controller persistence and the warm
restore path. #18011 repairs warm preparation and fresh bootstrap
checks; it does not cover ordinary per-turn continuation.

**What happened?**

A remote OpenCode run completed with `per_turn` lifecycle and saved its
controller checkpoint in the database. Its next turn reused the
normalized session. Admission read the remaining external-persistence
marker as a full controller file and failed with
`runner_state_identity_mismatch: durable_identity_unreadable`. The prior
run's complete checkpoint and harness backup still existed. The failure
occurred before provider work started.

**Expected behavior**

Restore the exact prior controller checkpoint before identity
inspection. Then apply the existing company, task, session, prior-owner
and suspended-state checks before a provider can start.

**Steps to reproduce**

1. Complete a remote native OpenCode task with `per_turn` lifecycle and
external controller persistence.
2. Send a follow-up that resumes the same saved session without a
session reset.
3. Observe admission reject the valid external controller marker before
provider execution.

**Paperclip version or commit**

Observed at `40c1ab36cdac70c85dc8fe5e912431834fdb691b`. The same
warm-only restore remains on current master.

**Deployment mode**

Hosted Linux with Daytona and native OpenCode. The original model was
`openrouter/deepseek/deepseek-v4.1-flash-1017`.

## What Changed

- Restore external controller state for ordinary remote per-turn
continuations through the existing checkpoint loader.
- Bind that read to the current company and task, the marker's run, and
the expected normalized session.
- Keep local, warm and restart-recovery paths, prior-owner checks and
harness backup checks intact.
- Refresh a prior-run projection from the latest checkpoint after a
rejected active-owner attempt. Keep current-run full state and the
existing legacy file-backed fallback intact.
- Add restoration and denial regressions, execution documentation and
session verification recipes.

## Verification

- Red: the real controller generated an external marker; a valid
per-turn OpenCode continuation failed with the exact
`durable_identity_unreadable` error before tool-authority construction.
The repair passed the same case and its ownership guards.
- Review regression red/green: a real controller advanced its checkpoint
after an active-owner rejection. Before the follow-up fix, a later valid
continuation retained a null attach seed. After the fix, it restored the
newer attach template and prepare command. Legacy full-file roots with
no database checkpoint retain their existing behavior.
- At current head `1f6589233e6121d082a0c0c4444ea895b6e8cd7c`, 86 focused
executor cases and all 14 execution-driver cases pass. Coverage includes
checkpoint restoration, direct/backup suspension, Codex, legacy files,
warm admission and canonical bootstrap, missing or malformed evidence,
wrong bindings, active/foreign owners and unsuspended runners. Tests
stop before provider launch.
- The real PostgreSQL encrypted-checkpoint integration passes, including
normalized-session refusal before write and stale-writer fencing.
Independent source review found no actionable safety defect.
- `node scripts/check-maps.mjs`, `git diff --check`, full `pnpm -r
typecheck` and `pnpm build` pass on current head.
- The local full suite did not complete: native PostgreSQL startup
failed under the host sandbox, an initial Docker rerun lacked
migrations, and the migrated retry crossed the review-fix revision
change. Logs are retained; that mixed-revision run was stopped and its
owned database removed. Full test evidence comes from current-head CI.
- Current-head CI passes: 53 successful checks, two intended Storybook
skips, none pending or failing. Greptile is 5/5 on that exact head with
no open review threads. The only manual merge conflict with #18011 was
in the verification recipes; both recipes and the unchanged per-turn
restore helper are preserved.

**Live continuation acceptance remains unverified, so this PR stays
draft.** Two disposable OpenCode/Daytona attempts never reached turns
two and three:

- At `72b49208b1`, the first turn executed tools and produced the exact
33-byte workspace note, but exceeded its four-minute settlement
deadline. The first completion lacked a registered attachment; the
second decorated a typed attachment reference with filename/size/hash
text. The strict UUID parser rejected that reference before checking a
valid criterion reference. The scoped attachment, asset and publication
receipt exist. Replaying the actual guard rejects the original input
before a database lookup; correcting only that reference passes against
the recovered row and receipt. This did not show a
publication-persistence or controller defect. The task remained
unsettled.
- At `1f6589233e`, a separately approved replacement used a temporary
prompt-only fixture change requiring exact `deliverable:<attachmentId>`
references, with metadata in `verification.detail`. Production code,
grading and deadlines were unchanged; the fixture patch was removed
afterward. This attempt had no recorded provider authentication,
harness-ready, model, tool or result event. Ingress acquisition took
84.072 seconds; transport connection took 205.835 seconds within the
239.832-second failed native execution. A local settlement GET also
timed out after 30 seconds. The underlying startup stall is unresolved;
these observations do not establish a remote provider outage. Final
recovery was blocked with `ambiguous_remote_recovery_evidence`; no
successful settlement or continuation is claimed.

Both attempts used a Darwin arm64 source host with Node 26.4.0, isolated
PostgreSQL 18, a Linux x64 runner and OpenCode 1.18.34. The
representative qualified model was
`openrouter/deepseek/deepseek-v4-flash-0731`; the original incident used
`openrouter/deepseek/deepseek-v4.1-flash-1017`. The controller defect
occurs before provider work. Runtime artifacts came from immutable build
`e7ea2e2961`, whose Runner package source is identical to the candidate.
The current-source manifest, artifact, distribution, asset and all 110
archive-hash checks passed. Each attempt had $10 company/agent model
budgets and zero automatic workflow/task retries. Accounting was
incomplete; zero recorded spend is not proof of zero cost.

Earlier setup failures are also retained: host PostgreSQL bootstrap
denial, cached-pack compatibility admission before provider work, and an
[official artifact
build](https://github.com/paperclipai/paperclip/actions/runs/40696838122)
that stopped during an unrelated Pi npm install at its 300-second bound.
No manifest or integrity guard was weakened. Owned local processes and
databases are stopped, and the temporary fixture patch is removed.
Standard fixture cleanup had left one replacement sandbox started; it
was stopped and deleted through the SDK. Every owned sandbox was then
verified absent with a final 404 readback. Resource cleanup does not
establish successful task finalization.

An earlier head retained one [voice-recovery timing
failure](https://github.com/paperclipai/paperclip/actions/runs/40695892339/job/122843005098):
a deadline test observed two failed-provider inspections instead of one.
A single isolated shard rerun passed all 1,579 tests without a source
change. Its timing explanation remains an inference. The current
reconciled head passed without a retry.

## Risks

Restoration writes a local projection before the existing admission
checks. The encrypted checkpoint loader validates company/task/run and
normalized session before writing. A later ordinary continuation
refreshes a prior-run projection so rejected admission cannot freeze an
active owner's older checkpoint. Active, foreign or unsuspended prior
owners still cannot start a successor. Missing or malformed evidence
remains blocked. The change adds no retry, credential, provider binary
or lifecycle-policy change; it preserves the existing legacy fallback.

This PR does not deploy the repair or recover the original hosted tasks.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, tool use, code execution and
collaborating agents. The session exposes GPT-6 as its model version; a
more specific model ID and context-window size are unavailable.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-28 09:49:32 -05:00

20 KiB
Raw Blame History

navigation
navigation
section sectionOrder order
Agents and reusable capabilities 30 20

Runs, harnesses, and model accounts

Operators choose an agent harness and model account, inspect execution history and transcripts, and distinguish a working configuration from a successful task result.

Feature ID: runs-adapters.

Implementation: adapter manager, run host, configuration.

Agent access: companion.

Sub-features

  • model-accounts: runs-adapters/model-accounts — configure AI accounts, login attempts, defaults, bindings, usage and pools through delegated connection APIs.
  • availability: runs-adapters/availability — discover installed adapters and available models/accounts.
  • validation: runs-adapters/validation — test an adapter in the selected execution environment and surface authentication failures.
  • runs: runs-adapters/runs — inspect run status, events, output, costs, and the source task.
  • session: runs-adapters/session — preserve or reset execution context through supported lifecycle actions.

How to get to it (user POV)

adapter-setup

Open an agent’s model/adapter configuration or the instance adapter settings page when permitted.

managed-account-setup

Ask an authorized agent to save or connect a managed AI account. Open its saved aiLogin setup link as the responsible human when sign-in or a code is required. The agent can inspect the saved attempt and manage authorized account/pool bindings; human authentication and raw host administration retain their gates.

run-history

Open an agent run at /agents/:agentId/runs/:runId from the agent or task activity. The agent header has one Run now action. Administrators can enable Capture raw provider traces under the agent configuration’s Debugging section for future runs. The header shows Raw tracing on while that setting is enabled.

For latency diagnosis with an operator OTLP endpoint, search the hashed run ID without restricting the root span name. Compare workspace preparation with its workspace_<step> intervals. Measure controller readiness separately through controller.startup.<phase>. A recovery pause or deployment drain is not model execution time. See the tracing contract. These source-level diagnostics do not establish a live response-time budget. task.time_to_first_token measures accepted ingress through the first committed text that the task transcript can show, including thinking and completed responses. It includes preparation and startup, and excludes browser latency. Inspect native.controller.<phase> for backend construction, assigned-tool discovery and post-turn cleanup. Compare those intervals with the actual model turn; do not count a nested discovery interval twice.

Driving it with Playwright and the Paperclip CLI

Preconditions: follow the baseline. Record adapter, model, account, native/legacy runtime, and environment. Capabilities vary; one harness result is not a matrix-wide qualification.

managed-account-setup

Automated: actual-auth connection integration exercises account/pool persistence, responsible-user ownership, agent attribution, saved local-login access and native secret-response protection with provider fixtures. Saved handoff UI tests exercise setup routing with mocked APIs. Neither establishes live provider sign-in.

Manual: Start delegated managed-account setup, reload the exact saved human handoff, finish the required login, and inspect the saved account and original task continuation. Bind the account or pool only to authorized agents, reload their configuration, and verify the selected owner and account remain correct. Reject a different user's saved attempt, revoked management authority and Ask/Plan mutations. Qualify a useful model task separately from saving the account.

session

For a remote OpenCode agent configured with per_turn, complete a task, then send two follow-ups without resetting its session. Verify that each continuation restores its external controller checkpoint, retains provider context and saved files, and finishes with no stranded task lock. A missing checkpoint, an active prior owner or a mismatched session must still stop admission. The rejected active-owner attempt must not prevent a later continuation from reading the owner's newer completed checkpoint. The executor controls cover checkpoint restoration and its guards; real-provider acceptance remains separate evidence.

After a completed warm turn, change an agent setting that invalidates its instruction copy, then send a follow-up. Verify the old runner suspends during preparation, saved files remain available, and the follow-up completes once. Repeat with an expired or foreign preparation owner and verify that execution stays held. The checkpoint integration regression covers the ownership boundary; a real provider run remains separate acceptance evidence.

For a repository project that inherits storage-plugin defaults, run a task and send two follow-ups. Start with unset task workspace settings, then repeat with an explicit isolated mode or strategy. Binding must save reuse_existing and the resolved mode without changing the effective policy, session fingerprint, and workspace identity. An actual project-policy change must still reset the session and preserve the stored workspace owner. The session admission regression checks this with PostgreSQL and synthetic provider execution; it is not live warm-process or latency evidence.

For runs-adapters/session, complete a task in a disposable warm sandbox and retain its saved workspace file. Let the warm session close normally and verify that the retained lease has a complete, lease-bound harness backup before injecting sandbox loss. Removing a still-warm sandbox before that checkpoint exercises the separate acquisition safety gate, which must remain blocked. Use a board-authorized session reset, then remove only that fixture's inactive provider sandbox. Send a follow-up that reads the retained file and appends one new line. Verify a replacement sandbox, a new provider session, the exact file contents, successful finalization and no stranded task lock. Send another follow-up and verify ordinary session reuse.

With external controller checkpoints enabled, also admit a fresh session on a reused sandbox lease. Verify startup from canonical controller evidence and reject a marker without a registered controller, changed identity or prior turn activity. An existing remote session directory must retain its recovery gate.

Repeat without resetting the session: its replacement must restore verified history or remain blocked. Missing backup state must not silently start an existing session again. The executor controls cover the bootstrap identity, activity and atomic directory claim; those controls do not establish live provider acceptance.

adapter-setup

Automated: adapter validation routes and model refresh cover server behavior.

Claude native setup uses the existing Anthropic API-key or subscription connection flow. Record the selected account separately from successful execution; unavailable models/authentication and unsupported Legacy-only options must fail visibly.

Claude CLI tests and runs also prepare a compatible private executable when a managed sandbox image's default CLI is too old for the selected model. Verify the command used for the hello test and task, reuse of a compatible private installation, and a blocking failure when installation cannot establish a valid version. Local, SSH and explicit custom commands retain operator management. Cloud onboarding must not offer shell commands the customer cannot run.

Manual: Select the intended adapter and account, refresh model choices, and run its environment test. Save, reload, then execute a small task. Repeat with a deliberately unavailable account and verify the failure remains visible.

For Legacy Codex, assign a service secret such as SLACK_BOT_TOKEN through agent configuration and run a task that checks its presence without printing its value. Check both ACP and CLI execution. Remove the binding and repeat on the next run. The assigned secret must reach shell tools; an unassigned host token and Codex's own login keys must remain unavailable. Verify the same delivery without an agent identity. Shell-policy metadata must contain names rather than values.

Choose Codex as the harness; the UI displays its name/icon for both runners. Advanced identifies the automatic default and offers Paperclip Runner or Legacy. New Codex agents default to the native runner on supported macOS or GNU/Linux targets; the shared selection contract and runtime prerequisites define the platform/libc boundary. Windows and Alpine/musl remain Legacy. Claude follows its qualified rollout described above. Disabled adapters, external overrides, and selected-target qualification also affect availability.

Choose OpenCode with the native-runner experimental flag off on a qualified macOS or GNU/Linux x64 target. Linux ARM64 must retain Legacy because its pinned OpenCode runtime is not qualified. Advanced should show Paperclip Runner as the automatic default on qualified targets. Save an inactive draft without a model, resume it, select an explicit provider/model, test, finish setup, and reload. Verify the saved provider is opencode. Repeat with explicit Legacy and an existing Legacy agent; neither should migrate on an unrelated edit. Check that saved native permission/lifecycle choices survive resuming setup. Custom commands, arguments, variants, and inherited permissions must give Legacy remediation when native is selected. A live task with the selected account is separate proof from these setup and persistence controls. Repeat with Pi on macOS or GNU/Linux x64 after explicit operator setup of the pinned runtime (paperclipai runtime setup pi). Linux ARM64 retains Legacy. Resume an inactive model-free draft, connect an account, choose a provider/model, and finish setup with the generic native flag off. Verify Pi branding and native ACPX Pi execution by completing a simple assigned task. Preserve saved native thinking/permission/lifecycle settings on resume and unrelated edits. Native thinking supports off, low, high, max; legacy aliases must give remediation, and returning to Legacy must reject settings it cannot honor. On a fresh forced-Kubernetes instance without an environment row, inspect discovery and create with automatic selection. Both must retain Legacy without provisioning; an explicit Paperclip request must give target-qualification guidance. Begin the ordinary task to exercise the existing lazy environment path. This live journey remains separate from the resolver's source controls.

For sandbox defaults, follow the environment probe recipe first. Discovery/creation read company-bound immutable-image evidence without allocating a target. A mutable image or snapshot name does not establish native compatibility. Then test the actual selected runtime with its selected account/model: a legacy CLI or authentication probe alone cannot establish native readiness. For an image-provided Codex harness, verify both final executables after concurrent artifact preparation on an eligible provider. Repeat with a failed runner probe and confirm the harness work has settled before failure returns. Providers without concurrent sync and explicit harness installs remain serial. The real-shell artifact controls cover these source contracts; they do not establish live reply latency. For remote native Codex with managed GitHub access, inspect the controller preparation spans for overlapping GitHub setup and assigned MCP discovery. Provider startup must wait for both. Force either preparation to fail while the other is pending; confirm that the pending operation settles and run-owned access is released before failure returns. Repeat a warm follow-up and retain its actual reply time. The executor controls cover ordering, failure drain and credential projection; they do not qualify a live response-time target. For native OpenCode with assigned connections, inspect the provider tool surface on both local and sandbox execution. It should contain bounded assigned-tool search/call schemas. Discover an allowed tool, call its returned alias, revoke its grant, and confirm discovery and execution enforce that revocation. Repeat in planning/ask mode with a write tool and confirm denial. The controller must retain the assigned gateway URL and token. An older direct-catalog OpenCode checkpoint must refresh without changing another provider's fingerprint. The assigned-tool and executor controls cover source behavior; live sandbox discovery and a real call remain separate acceptance evidence. Missing dependencies, failed authentication, invalid models and incompatible custom settings must remain actionable errors without silent runner switching. Saved unchanged execution can still be tested without applying new selection policy, but runtime/account validation must still run.

Verify an automatic Codex hire and an explicit Legacy control through a useful file artifact, follow-up and reload. Preserve Codex reasoning effort and account isolation; retain explicit Legacy permission choices, including false. Reopen an existing agent, make an unrelated edit, and confirm both its persisted runner and next run stay unchanged. Use hiring/import preservation for approval and portability controls.

Test native Codex with a saved subscription whose setup auth file is a managed link, as well as a regular file and an API-key account. Setup must seed the isolated probe from the checked account without exposing credentials or changing its saved runner. With subscription authentication, an unbound configured CODEX_HOME with linked credentials must fail setup with managed-account or explicit Legacy guidance; it must not claim readiness when task isolation would omit the file. Dangling or incorrectly owned credentials must also fail clearly. A selected CODEX_API_KEY or OPENAI_API_KEY takes precedence over an unrelated subscription file. Test valid, dangling and invalid subscription links with that selected key; setup must test the key without reading or refreshing the unrelated account. The private probe copy must be removed after confirmed retirement following success, failure or cancellation. Unconfirmed shutdown remains a cleanup failure. Exercise an unavailable local daemon and a preparation failure before probe handoff. No owned temporary credential copy may remain, and original account files must be unchanged. A probe whose shutdown is unconfirmed must retain its recovery state; do not treat it as a preparation-only failure. Existing setup controls use fake credentials; real account execution remains a separate acceptance requirement.

run-history

Automated: live run routes and run event sequencing cover run data contracts.

Manual: Start a disposable task, open its running transcript, and follow it to a terminal outcome. Reload and compare task/run IDs, status, output, and cost. Verify a failed run remains distinguishable from a completed run with useful output.

For Legacy OpenCode on a disposable Linux sandbox, run a Bash tool that reads /proc/self/oom_score_adj and its parent's score. The tool should report 1000; OpenCode should retain its original score. Repeat with an operator BASH_ENV and confirm that it takes precedence. Repeat with a linked operator plugins directory; it must remain unchanged, and the uploaded runtime must contain a real plugins directory with the guard and existing plugins. Under controlled memory pressure, record the cgroup OOM counters and verify that a killed tool can return an error while the agent continues. Local and SSH execution must keep their original shell environment. See the adapter contract. These controls do not guarantee that every memory exhaustion preserves the agent.

An explicit runner change starts a configuration revision and invalidates the eligible session. Active runs, restart recovery and historical transcript parsing retain their recorded execution configuration; do not infer their runner from a later edit to the agent. Account/harness branding and execution identity are different observations, both required in the evidence.

For an onboarding child-completion callback, verify actual session/turn events and the source-correlated reply. A succeeded row alone is insufficient: the literal ea6c8eb5e16d61111968f633a66748b397b64a3e managed attempt was marked successful during startup without a provider turn. The recovery correction preserves the existing validated Done-parent handoff; process-death and ownership checks remain active. Its PostgreSQL controls are separate from the required revised-source live onboarding and follow-up proof.

Gotchas

  • Model accounts are managed in Apps and inline agent account controls. /company/settings/connections configures decision/fast-response models, not the general model-account library.

  • The run log is instance database data; it is not the opt-out first-party Telemetry system or operator-configured OpenTelemetry tracing.

  • Authentication/model discovery does not establish tool, image, steering, or resumption capability parity.

  • Use recovery for interrupted runs and execution environments for local/SSH/sandbox differences.

Native setup retirement joins owned startup before cleanup can be confirmed. Transport ownership controls hold preparation or registration without launching a provider, then check bounded close, rejected duplicate requests, no later launch and retention of unconfirmed state. Readiness observes shutdown; resumed startup cannot reactivate ingress after close. These are lifecycle controls, separate from installed provider proof. The real daemon readiness and recovery controls require their existing native fixtures in hosted CI; passing credential-free controls does not qualify them. A primary setup timeout remains a failure even when teardown also fails.

For remote setup shutdown, verify that each durable-state read respects the remaining suspension deadline. A late result cannot establish suspension or queue a replacement command after that deadline. An intentional ingress close must observe physical socket closure before retiring its connection owner and releasing the endpoint; an unresponsive peer must not extend the setup budget. The existing transport and ingress suites exercise these conditions without a provider. Repeat the actual managed setup and useful task journey separately; the literal 651a3de64fc943216cc414b64b1443809f9499da Test attempt failed cleanup before hire and retains its unconfirmed recovery state.

delegated-configuration

See the canonical delegated configuration guidance and acceptance evidence.