mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-08 01:57:31 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native agents use assigned connection tools through a governed gateway. > - OpenCode received the gateway catalog directly, outside the bounded server tool projection. > - A large catalog can exceed a provider grammar compiler's capacity even when it fits the transport limits. > - Synthetic requests reproduced the Fireworks grammar timeout on the configured DeepSeek model. > - This pull request gives OpenCode two assigned-tool discovery and call schemas through the existing server relay. > - The gateway keeps the same grants, validation, approvals, revocation, and audit rules. ## Linked Issues or Issue Description **What happened?** A native OpenCode run could fail during provider grammar compilation before its first assigned-tool call. OpenCode advertised the full assigned gateway catalog. A synthetic catalog of 741 tool schemas reproduced `Grammar processing took too much time` on Fireworks through OpenRouter with `deepseek/deepseek-v4.1-flash-1017`. The same model and backend accepted a compact two-tool discovery/call catalog. This reproduces the mechanism; it does not replay a historical private request. **Expected behavior** Assigned-tool discovery must keep provider grammar size bounded. Every call must retain the run's original gateway binding and permissions. **Steps to reproduce** 1. Give a native OpenCode run a large assigned connection catalog. 2. Use a provider that compiles tool schemas into a constrained output grammar. 3. Observe the full catalog in the provider tool surface before this change. 4. After this change, observe only `paperclip_search_assigned_tools` and `paperclip_call_assigned_tool` for assigned tools. Discover and call a permitted tool through its pinned alias. **Paperclip version or commit** Source checkout based on current `master`. **Deployment mode** Native local or sandbox execution with OpenCode 1.18.34. Related work: Refs #14258 (discovery paging), #16678 (local adapter MCP launch), and #17688 (gateway capability advertisement). This change covers native OpenCode catalog projection. ## What Changed - Route native OpenCode assigned tools through the existing server-owned authenticated relay. - Use bounded discovery/call schemas for OpenCode, including small catalogs. Preserve transport capacity checks. - Keep assigned gateway credentials on the controller. Ignore gateway and credential values in tool arguments. - Preserve exact declared MCP operation names through admission and driver dispatch. Resolve transport prefixes only against the closed catalog. Reject unknown or shortened names. - Refresh former OpenCode direct-catalog and shortened-name checkpoints. Keep the existing fingerprints for other providers. - Add local and remote wiring, catalog capacity, alias, scope, work-mode, approval, and revocation tests. - Allow the native test files to use a validated dedicated external test database when embedded PostgreSQL is unavailable. - Document the assigned-tool execution contract. - Bound OpenCode discovery pages to 32 KiB and large-schema chunks to 16K characters. Keep other harness budgets unchanged. - Test medium pages and escaped large schemas through the actual HTTP MCP bridge on the explicit Runner testing surface. ## Verification - Red before the catalog fix: OpenCode did not create the server relay, and forced compact delivery advertised 1 or 741 direct schemas. - Red before the alias repair: HTTP MCP changed declared operation names, and the driver sent `search_assigned_tools` instead of the authorized `paperclip_search_assigned_tools`. - Current repair tests: 112 OpenCode/proxy tests pass, plus one actual Rust/proxy/HTTP MCP regression. Two checkpoint regressions fail with the former scoped revision and pass with the new revision. Other providers keep their prior fingerprints. - The combined QA candidate with this exact PR head passes all 916 native tests and all 113 transport tests on supported Node 24.22. An earlier Node 26.4/macOS run passed the 113 assertions but had an uncaught networking error. That non-clean run remains recorded; its stack matches [the upstream Undici defect](https://github.com/nodejs/undici/issues/5544). No production workaround or error suppression was added. - `pnpm -r typecheck`, `pnpm build`, and `git diff --check` pass on `bc41ba72be2195eabeb0d7727227ee0cd02e8532`. The maps structural check passes. - The local `pnpm test:run` attempt stopped after host embedded PostgreSQL initialization failures. The relevant database suites pass with isolated PostgreSQL 18. All repository CI gates, including aggregate `ci / verify`, pass on the current head. - Synthetic live provider comparison: 741 invented schemas produced the exact Fireworks grammar timeout in 20.9 seconds. Two compact invented schemas produced a discovery call in 585 ms on the same model and backend. - The first hosted QA attempt exposed the alias failure described above. On the patched source, a real local Rust runner → proxy → OpenCode 1.18.34 → exact DeepSeek smoke completed discovery, one synthetic gateway read, and `paperclip_finish`, then emitted `turn/completed`. It used the production compact projection and unchanged Rust allowlist. - Two earlier local smoke attempts exposed missing terminal authority and completion-contract fields in the smoke harness. Those failures remain recorded. The corrected ordinary bindings produced the clean result. - Fresh Greptile: 5/5 on `bc41ba72be2195eabeb0d7727227ee0cd02e8532`, with no new actionable findings or unresolved threads. - Hosted QA publication and lifecycle acceptance remain pending. The separate deployed-agent `Paperclip Review` check is also pending; it follows a task that failed on the earlier deployed runtime. - Local OpenCode/OpenRouter acceptance at `bc41ba72`: 530 synthetic tools were authorized and installed. A simple model-ID reply completed in 17.3 seconds with no app calls. A discovery task completed in 37.6 seconds, used the search/call relay, made one upstream call, and returned the exact synthetic value. This is local source acceptance, not a staging sandbox timing comparison. - The large-result review finding has a failing-before/passing-after HTTP MCP regression. Five medium schemas now page without invalid JSON. A large schema with escaped text reconstructs exactly through bounded chunks. All 35 assigned-tool tests and the server typecheck pass after this follow-up. - Final head `d84ce4856a6e65a35ad5fb8b561492598b525f4e`: all CI gates pass. Greptile and Paperclip Review both rate the change 5/5. No review threads remain open. The final full build, server typecheck, and 35 assigned-tool tests pass locally. ## Risks - OpenCode uses discovery before it calls an assigned tool, including for small catalogs. This can add a model turn. - An existing OpenCode provider checkpoint rotates once because it retains the former tool declarations. Paperclip task history remains available. - The historical filtered catalog was not captured. The synthetic comparison proves the failure mechanism, not the exact cardinality of an earlier private request. - No model, provider route, permission, or automatic retry policy changes. ## Model Used The bounded-result follow-up also used OpenAI Codex (GPT-6 family). The exact backend model ID and context-window size were not exposed for that session. It used reasoning, code execution, and tool use. OpenAI `gpt-6.1-sol` in Codex, with reasoning, tool use, code execution, and source inspection. The runtime does not expose the context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
248 lines
16 KiB
Markdown
248 lines
16 KiB
Markdown
---
|
||
navigation: {"section":"Agents and reusable capabilities","sectionOrder":30,"order":20}
|
||
---
|
||
|
||
# Runs, harnesses, and model accounts
|
||
|
||
Operators choose an agent harness and model account, inspect execution history and transcripts, and distinguish a working configuration from a successful task result.
|
||
|
||
Feature ID: `runs-adapters`.
|
||
|
||
Implementation: [adapter manager](../../ui/src/pages/AdapterManager.tsx), [run host](../../ui/src/pages/AgentDetail.tsx), [configuration](../../ui/src/components/AgentConfigForm.tsx).
|
||
|
||
Agent access: [companion](../agent-access-map/runs-adapters.md).
|
||
|
||
## Sub-features
|
||
|
||
- `availability`: `runs-adapters/availability` — discover installed adapters and available models/accounts.
|
||
- `validation`: `runs-adapters/validation` — test an adapter in the selected execution environment and surface authentication failures.
|
||
- `runs`: `runs-adapters/runs` — inspect run status, events, output, costs, and the source task.
|
||
- `session`: `runs-adapters/session` — preserve or reset execution context through supported lifecycle actions.
|
||
|
||
## How to get to it (user POV)
|
||
|
||
### `adapter-setup`
|
||
|
||
Open an agent’s model/adapter configuration or the instance adapter settings page when permitted.
|
||
|
||
### `run-history`
|
||
|
||
Open an agent run at `/agents/:agentId/runs/:runId` from the agent or task activity.
|
||
The agent header has one Run now action. Administrators can enable
|
||
Capture raw provider traces under the agent configuration’s Debugging section
|
||
for future runs. The header shows Raw tracing on while that setting is enabled.
|
||
|
||
For latency diagnosis with an operator OTLP endpoint, search the hashed run ID
|
||
without restricting the root span name. Compare workspace preparation with its
|
||
`workspace_<step>` intervals. Measure controller readiness separately through
|
||
`controller.startup.<phase>`. A recovery pause or deployment drain is not model
|
||
execution time. See the [tracing contract](../../doc/observability.md#controller-preparation-phases).
|
||
These source-level diagnostics do not establish a live response-time budget.
|
||
`task.time_to_first_token` measures accepted ingress through the first committed
|
||
text that the task transcript can show, including thinking and completed
|
||
responses. It includes preparation and startup, and excludes browser latency.
|
||
Inspect `native.controller.<phase>` for backend construction, assigned-tool
|
||
discovery and post-turn cleanup. Compare those intervals with the actual model
|
||
turn; do not count a nested discovery interval twice.
|
||
|
||
## Driving it with Playwright and the Paperclip CLI
|
||
|
||
Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Record adapter, model, account, native/legacy runtime, and environment. Capabilities vary; one harness result is not a matrix-wide qualification.
|
||
|
||
### `session`
|
||
|
||
For `runs-adapters/session`, complete a task in a disposable warm sandbox and
|
||
retain its saved workspace file. Let the warm session close normally and verify
|
||
that the retained lease has a complete, lease-bound harness backup before
|
||
injecting sandbox loss. Removing a still-warm sandbox before that checkpoint
|
||
exercises the separate acquisition safety gate, which must remain blocked.
|
||
Use a board-authorized session reset, then remove only that fixture's inactive
|
||
provider sandbox. Send a follow-up that
|
||
reads the retained file and appends one new line. Verify a replacement sandbox,
|
||
a new provider session, the exact file contents, successful finalization and no
|
||
stranded task lock. Send another follow-up and verify ordinary session reuse.
|
||
|
||
Repeat without resetting the session: its replacement must restore verified
|
||
history or remain blocked. Missing backup state must not silently start an
|
||
existing session again. The [executor controls](../../server/src/services/native-runtime/native-session-executor.test.ts)
|
||
cover the bootstrap identity, activity and atomic directory claim; those
|
||
controls do not establish live provider acceptance.
|
||
|
||
### `adapter-setup`
|
||
|
||
Automated: [adapter validation routes](../../server/src/__tests__/agent-adapter-validation-routes.test.ts) and [model refresh](../../server/src/__tests__/adapter-model-refresh-routes.test.ts) cover server behavior.
|
||
|
||
Claude native setup uses the existing Anthropic API-key or subscription connection
|
||
flow. Record the selected account separately from successful execution; unavailable
|
||
models/authentication and unsupported Legacy-only options must fail visibly.
|
||
|
||
Claude CLI tests and runs also prepare a compatible private executable when a
|
||
managed sandbox image's default CLI is too old for the selected model. Verify
|
||
the command used for the hello test and task, reuse of a compatible private
|
||
installation, and a blocking failure when installation cannot establish a valid
|
||
version. Local, SSH and explicit custom commands retain operator management.
|
||
Cloud onboarding must not offer shell commands the customer cannot run.
|
||
|
||
Manual: Select the intended adapter and account, refresh model choices, and run its environment test. Save, reload, then execute a small task. Repeat with a deliberately unavailable account and verify the failure remains visible.
|
||
|
||
For Legacy Codex, assign a service secret such as `SLACK_BOT_TOKEN` through agent
|
||
configuration and run a task that checks its presence without printing its value.
|
||
Check both ACP and CLI execution. Remove the binding and repeat on the next run.
|
||
The assigned secret must reach shell tools; an unassigned host token and Codex's
|
||
own login keys must remain unavailable. Verify the same delivery without an
|
||
agent identity. Shell-policy metadata must contain names rather than values.
|
||
|
||
Choose Codex as the harness; the UI displays its name/icon for both runners.
|
||
Advanced identifies the automatic default and offers Paperclip Runner or Legacy.
|
||
New Codex agents default to the native runner on supported macOS or GNU/Linux
|
||
targets; the [shared selection contract](../../packages/shared/src/agent-runner.ts)
|
||
and [runtime prerequisites](../../packages/paperclip-runner/README.md) define the
|
||
platform/libc boundary. Windows and Alpine/musl remain Legacy. Claude follows
|
||
its qualified rollout described above. Disabled adapters, external overrides, and selected-target
|
||
qualification also affect availability.
|
||
|
||
Choose OpenCode with the native-runner experimental flag off on a qualified
|
||
macOS or GNU/Linux x64 target. Linux ARM64 must retain Legacy because its pinned
|
||
OpenCode runtime is not qualified. Advanced should show Paperclip Runner as the
|
||
automatic default on qualified targets. Save an inactive draft without a model,
|
||
resume it, select an explicit `provider/model`,
|
||
test, finish setup, and reload. Verify the saved provider is `opencode`. Repeat
|
||
with explicit Legacy and an existing Legacy agent; neither should migrate on an
|
||
unrelated edit. Check that saved native permission/lifecycle choices survive
|
||
resuming setup. Custom commands, arguments, variants, and inherited permissions
|
||
must give Legacy remediation when native is selected. A live task with the
|
||
selected account is separate proof from these setup and persistence controls.
|
||
Repeat with Pi on macOS or GNU/Linux x64 after explicit operator setup of the
|
||
pinned runtime (`paperclipai runtime setup pi`). Linux ARM64 retains Legacy.
|
||
Resume an inactive model-free draft, connect an account, choose a `provider/model`,
|
||
and finish setup with the generic native flag off. Verify Pi branding and native
|
||
ACPX Pi execution by completing a simple assigned task. Preserve saved native
|
||
thinking/permission/lifecycle settings on resume and unrelated edits. Native
|
||
thinking supports `off`, `low`, `high`, `max`; legacy aliases must give remediation,
|
||
and returning to Legacy must reject settings it cannot honor.
|
||
On a fresh forced-Kubernetes instance without an environment row, inspect discovery
|
||
and create with automatic selection. Both must retain Legacy without provisioning;
|
||
an explicit Paperclip request must give target-qualification guidance. Begin the
|
||
ordinary task to exercise the existing lazy environment path. This live journey
|
||
remains separate from the resolver's source controls.
|
||
|
||
For sandbox defaults, follow the [environment probe](./execution-environments.md#environment-settings)
|
||
recipe first. Discovery/creation read company-bound immutable-image evidence
|
||
without allocating a target. A mutable image or snapshot name does not establish
|
||
native compatibility. Then test the actual selected runtime with its selected
|
||
account/model: a legacy CLI or authentication probe alone cannot establish native
|
||
readiness. For an image-provided Codex harness, verify both final executables
|
||
after concurrent artifact preparation on an eligible provider. Repeat with a
|
||
failed runner probe and confirm the harness work has settled before failure
|
||
returns. Providers without concurrent sync and explicit harness installs remain
|
||
serial. The [real-shell artifact controls](../../server/src/services/native-runtime/native-artifact-preparation.test.ts)
|
||
cover these source contracts; they do not establish live reply latency.
|
||
For remote native Codex with managed GitHub access, inspect the controller
|
||
preparation spans for overlapping GitHub setup and assigned MCP discovery.
|
||
Provider startup must wait for both. Force either preparation to fail while the
|
||
other is pending; confirm that the pending operation settles and run-owned access
|
||
is released before failure returns. Repeat a warm follow-up and retain its actual
|
||
reply time. The [executor controls](../../server/src/services/native-runtime/native-session-executor.test.ts)
|
||
cover ordering, failure drain and credential projection; they do not qualify a
|
||
live response-time target.
|
||
For native OpenCode with assigned connections, inspect the provider tool surface
|
||
on both local and sandbox execution. It should contain bounded assigned-tool
|
||
search/call schemas. Discover an allowed tool, call its returned alias, revoke
|
||
its grant, and confirm discovery and execution enforce that revocation. Repeat
|
||
in planning/ask mode with a write tool and confirm denial. The controller must
|
||
retain the assigned gateway URL and token. An older direct-catalog OpenCode
|
||
checkpoint must refresh without changing another provider's fingerprint. The
|
||
[assigned-tool](../../server/src/services/native-runtime/assigned-mcp-tools.test.ts)
|
||
and executor controls cover source behavior; live sandbox discovery and a real
|
||
call remain separate acceptance evidence.
|
||
Missing dependencies, failed authentication, invalid models and
|
||
incompatible custom settings must remain actionable errors without silent runner
|
||
switching. Saved unchanged execution can still be tested without applying new
|
||
selection policy, but runtime/account validation must still run.
|
||
|
||
Verify an automatic Codex hire and an explicit Legacy control through a useful
|
||
file artifact, follow-up and reload. Preserve Codex reasoning effort and account
|
||
isolation; retain explicit Legacy permission choices, including false. Reopen an
|
||
existing agent, make an unrelated edit, and confirm both its persisted runner and
|
||
next run stay unchanged. Use [hiring/import preservation](./agents.md#hire-agent)
|
||
for approval and portability controls.
|
||
|
||
Test native Codex with a saved subscription whose setup auth file is a managed
|
||
link, as well as a regular file and an API-key account. Setup must seed the
|
||
isolated probe from the checked account without exposing credentials or changing
|
||
its saved runner. With subscription authentication, an unbound configured
|
||
`CODEX_HOME` with linked credentials must fail setup with managed-account or explicit Legacy guidance; it must not claim
|
||
readiness when task isolation would omit the file. Dangling or incorrectly owned
|
||
credentials must also fail clearly. A selected `CODEX_API_KEY` or
|
||
`OPENAI_API_KEY` takes precedence over an unrelated subscription file. Test valid,
|
||
dangling and invalid subscription links with that selected key; setup must test
|
||
the key without reading or refreshing the unrelated account. The private probe copy must be removed after confirmed retirement following success,
|
||
failure or cancellation. Unconfirmed shutdown remains a cleanup failure.
|
||
Exercise an unavailable local daemon and a preparation failure before probe
|
||
handoff. No owned temporary credential copy may remain, and original account
|
||
files must be unchanged. A probe whose shutdown is unconfirmed must retain its
|
||
recovery state; do not treat it as a preparation-only failure.
|
||
Existing setup controls use fake credentials; real account execution remains
|
||
a separate acceptance requirement.
|
||
|
||
### `run-history`
|
||
|
||
Automated: [live run routes](../../server/src/__tests__/agent-live-run-routes.test.ts) and [run event sequencing](../../server/src/__tests__/heartbeat-run-event-sequencing.test.ts) cover run data contracts.
|
||
|
||
Manual: Start a disposable task, open its running transcript, and follow it to a terminal outcome. Reload and compare task/run IDs, status, output, and cost. Verify a failed run remains distinguishable from a completed run with useful output.
|
||
|
||
For Legacy OpenCode on a disposable Linux sandbox, run a Bash tool that reads
|
||
`/proc/self/oom_score_adj` and its parent's score. The tool should report `1000`;
|
||
OpenCode should retain its original score. Repeat with an operator `BASH_ENV`
|
||
and confirm that it takes precedence. Repeat with a linked operator plugins
|
||
directory; it must remain unchanged, and the uploaded runtime must contain a
|
||
real plugins directory with the guard and existing plugins. Under controlled memory pressure, record
|
||
the cgroup OOM counters and verify that a killed tool can return an error while
|
||
the agent continues. Local and SSH execution must keep their original shell
|
||
environment. See the [adapter contract](../../docs/adapters/overview.md#opencode-sandbox-memory-pressure).
|
||
These controls do not guarantee that every memory exhaustion preserves the agent.
|
||
|
||
An explicit runner change starts a configuration revision and invalidates the
|
||
eligible session. Active runs, restart recovery and historical transcript parsing
|
||
retain their recorded execution configuration; do not infer their runner from a
|
||
later edit to the agent. Account/harness branding and execution identity are
|
||
different observations, both required in the evidence.
|
||
|
||
For an onboarding child-completion callback, verify actual session/turn events
|
||
and the source-correlated reply. A `succeeded` row alone is insufficient: the
|
||
literal `ea6c8eb5e16d61111968f633a66748b397b64a3e` managed attempt was marked
|
||
successful during startup without a provider turn. The recovery correction
|
||
preserves the existing validated Done-parent handoff; process-death and ownership
|
||
checks remain active. Its PostgreSQL controls are separate from the required
|
||
revised-source live onboarding and follow-up proof.
|
||
|
||
## Gotchas
|
||
|
||
- Model accounts are managed in Apps and inline agent account controls.
|
||
`/company/settings/connections` configures decision/fast-response models,
|
||
not the general model-account library.
|
||
|
||
- The run log is instance database data; it is not the opt-out first-party Telemetry system or operator-configured OpenTelemetry tracing.
|
||
- Authentication/model discovery does not establish tool, image, steering, or resumption capability parity.
|
||
- Use [recovery](./recovery.md) for interrupted runs and [execution environments](./execution-environments.md) for local/SSH/sandbox differences.
|
||
|
||
Native setup retirement joins owned startup before cleanup can be confirmed.
|
||
[Transport ownership controls](../../packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts)
|
||
hold preparation or registration without launching a provider, then check bounded
|
||
close, rejected duplicate requests, no later launch and retention of unconfirmed
|
||
state. Readiness observes shutdown; resumed startup cannot reactivate ingress
|
||
after close. These are lifecycle controls, separate from installed provider proof.
|
||
The real daemon readiness and recovery controls require their existing native
|
||
fixtures in hosted CI; passing credential-free controls does not qualify them.
|
||
A primary setup timeout remains a failure even when teardown also fails.
|
||
|
||
For remote setup shutdown, verify that each durable-state read respects the
|
||
remaining suspension deadline. A late result cannot establish suspension or
|
||
queue a replacement command after that deadline. An intentional ingress close
|
||
must observe physical socket closure before retiring its connection owner and
|
||
releasing the endpoint; an unresponsive peer must not extend the setup budget.
|
||
The existing transport and ingress suites exercise these conditions without a
|
||
provider. Repeat the actual managed setup and useful task journey separately;
|
||
the literal `651a3de64fc943216cc414b64b1443809f9499da` Test attempt failed
|
||
cleanup before hire and retains its unconfirmed recovery state.
|