Jannik/PaperClipAI
1
0 Fork 0
mirror of https://github.com/paperclipai/paperclip.git synced 2026-11-04 02:32:07 +01:00
Code Issues Packages Projects Releases Wiki Activity
Files
ea4aa185b0f3bc30be1c5fc160cdfd0eb433fe41
PaperClipAI/maps/feature-map/runs-adapters.md
T
DottaandPaperclip ebceb2a744 trace(server): time workspace, native setup and startup recovery (#16330) 
## Thinking Path

> - Paperclip manages agents and their work.
> - The controller prepares workspaces before it starts a Runner.
> - A slow run can spend 30–80 seconds inside the combined workspace phase.
> - Startup recovery can also delay queued work for minutes.
> - Native runs also have gaps before Runner startup and after the provider result.
> - Existing traces do not identify the awaited steps inside those intervals.
> - This change adds closed timing boundaries around the existing operations.
> - Operators can locate the wait without changing execution or exposing content.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Operator-configured OpenTelemetry diagnosis of controller preparation and startup recovery. Related: #16178, #16236 and #16285 introduced the existing preparation timing and sampling contract. #16306 and #16312 reduce separate configuration and copyback work.

**Current behavior**

The workspace interval combines source resolution, trust checks, provisioning, database persistence, binding and repository admission. Startup awaits accounting and native recovery before queued work resumes, but neither has a separate duration. Native backend setup and post-turn cleanup also contain unmeasured awaits. CPU profiles and a fast database probe cannot identify which await is slow.

**Proposed behavior**

Record workspace steps and native controller setup and cleanup with the existing hashed run ID. Record startup phases as independent, operator-sampled root spans without custom attributes. Keep the same operation order, results, failures and authorization.

**Reason and benefit**

These boundaries distinguish source work, database writes, repository admission and restart recovery. This is a diagnosis change. It does not claim a response-time improvement or a solved latency budget.

## What Changed

- Add 14 closed workspace step names around 16 existing awaits. Repeated updates and bindings produce separate spans.
- Time startup tool reviews, gateway sessions, delivery-worker readiness, accounting, plugin readiness, native recovery, hot restart, orphan reaping, retries, queued work and goal recovery.
- Time native GitHub setup, wake binding, backend construction, connector assignments, MCP discovery, tool authority, companion selection and binary resolution. Time selected post-turn GitHub, coordinator and instruction cleanup awaits.
- Move preparation timing to a pure shared module. Keep the existing heartbeat import as an alias. Both heartbeat and native timing skip tracer lookup when the endpoint is unset or blank, including when another library registered a provider.
- Tracing lookup, callback, status and end faults cannot repeat work or replace its value or error.
- Add real PostgreSQL workspace timing coverage and silent no-op/fault controls. Update the tracing contract and paired run maps.
- Keep span attributes closed. Startup spans add none; workspace steps use only the existing run hash. No commands, paths, returned records or exception messages are recorded.

## Verification

- Diagnostic-gap baseline: the new real workspace test fails against the unchanged workspace implementation because the expected step spans are absent. It passes with the wrappers.
- Final affected coverage: **804 distinct cases pass** across seven complete suites: 698 native-executor cases, 39 real PostgreSQL workspace cases, 61 timing/context/startup cases, and six execution-configuration integration cases. The final native and unit invocations pass with no selected skips. The workspace suite passed in the initial six-file invocation on the same final workspace source.
- Preserve the initial six-file attempt as failed: **7 failed and 789 passed**. All seven errors occurred during embedded PostgreSQL startup. The log reports local macOS shared-memory exhaustion. A fresh 15-case database group passes, followed by the complete 698-case native suite. These runs overlap; do not add their counts.
- The new native timing assertion fails against unchanged native code because the expected spans are absent. It passes with the wrappers and retains the credential-isolation assertions.
- Full `pnpm -r typecheck` and `pnpm build`: passed on the final production source. Earlier nullable-closure errors are fixed with erased assertions inside existing guards. The failed attempts remain recorded.
- Normalized AST comparison: removing only timing wrappers, timing imports/local handles and erased assertions restores the original workspace, startup and native executor programs. This checks call order and policy expressions; it is not live acceptance.
- The expanded head first failed two configuration integration assertions because their test providers had no endpoint. Configure the endpoint explicitly in those two tracing tests, and restore it after each test. All six cases then pass with the original credential cleanup, redaction and exact span assertions. Production is unchanged by this repair.
- The first PR head failed one unchanged attachment test at its 15-second deadline in CI. The complete 32-case attachment suite passes locally with that deadline. Do not label the failed CI attempt green. The expanded head needs fresh full CI.
- `node scripts/check-maps.mjs`: passed, 35 areas and 176 capability IDs. `node scripts/storybook-catalog.mjs`: 283 files and 2,728 stories. Structural checks only; no browser, play or Storybook build claim.
- `git diff --check`: passed. No added prose comments, lockfile changes or new 1,000-line crossings.
- A duplicate full local test run was not started. Fresh exact-head full CI and E2E pass on `15a683bb2419960f611584f8c07cdcfc9ce3af0b`. All 56 check runs are successful or optional skipped. Superagent and legacy Snyk pass. Completed Apex review is 5/5 on this same head, with zero unresolved review threads.

Commands:

```sh
pnpm exec vitest run --project @paperclipai/server --maxWorkers=1 --fileParallelism=false server/src/services/native-runtime/native-session-executor.test.ts server/src/services/execution-preparation-timing.test.ts server/src/startup-recovery-timing.test.ts server/src/services/heartbeat/preparation-timing.test.ts server/src/services/heartbeat/preparation-timing-context.test.ts server/src/services/heartbeat/workspace-preparation.test.ts
pnpm -r typecheck
pnpm build
node scripts/check-maps.mjs
node scripts/storybook-catalog.mjs
```

## Risks

- Configured operators receive more spans. They still follow the configured sampler and exporter.
- Startup spans measure the awaited phase, including remaining readiness waits. They do not measure earlier worker construction or a provider turn.
- Workspace and native setup intervals overlap their containing phases. Do not add them to the parent duration. Native timing covers named awaits, not every statement or every error path.
- These spans are not yet deployed or tested against a live latency run. No first-party Telemetry event, run-log write, schema or API changes are included.

## Model Used

OpenAI GPT-6 through Codex, with tool use and code execution. The exact served variant and context-window size are not exposed to this task.

## Checklist

- [x] I have included a thinking path that traces from project context to this change
- [x] I have specified the model used (with version and capability details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before requesting merge

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-14 17:16:07 -05:00

147 lines
9.2 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
navigation: {"section":"Agents and reusable capabilities","sectionOrder":30,"order":20}
---
# Runs, harnesses, and model accounts
Operators choose an agent harness and model account, inspect execution history and transcripts, and distinguish a working configuration from a successful task result.
Implementation: [adapter manager](../../ui/src/pages/AdapterManager.tsx), [run host](../../ui/src/pages/AgentDetail.tsx), [configuration](../../ui/src/components/AgentConfigForm.tsx).
Agent access: [companion](../agent-access-map/runs-adapters.md).
## Sub-features
- `availability`: discover installed adapters and available models/accounts.
- `validation`: test an adapter in the selected execution environment and surface authentication failures.
- `runs`: inspect run status, events, output, costs, and the source task.
- `session`: preserve or reset execution context through supported lifecycle actions.
## How to get to it (user POV)
### `adapter-setup`
Open an agent’s model/adapter configuration or the instance adapter settings page when permitted.
### `run-history`
Open an agent run at `/agents/:agentId/runs/:runId` from the agent or task activity.
For latency diagnosis with an operator OTLP endpoint, search the hashed run ID
without restricting the root span name. Compare workspace preparation with its
`workspace_<step>` intervals. Measure controller readiness separately through
`controller.startup.<phase>`. A recovery pause or deployment drain is not model
execution time. See the [tracing contract](../../doc/observability.md#controller-preparation-phases).
These source-level diagnostics do not establish a live response-time budget.
Inspect `native.controller.<phase>` for backend construction, assigned-tool
discovery and post-turn cleanup. Compare those intervals with the actual model
turn; do not count a nested discovery interval twice.
## Driving it with Playwright and the Paperclip CLI
Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Record adapter, model, account, native/legacy runtime, and environment. Capabilities vary; one harness result is not a matrix-wide qualification.
### `adapter-setup`
Automated: [adapter validation routes](../../server/src/__tests__/agent-adapter-validation-routes.test.ts) and [model refresh](../../server/src/__tests__/adapter-model-refresh-routes.test.ts) cover server behavior.
Manual: Select the intended adapter and account, refresh model choices, and run its environment test. Save, reload, then execute a small task. Repeat with a deliberately unavailable account and verify the failure remains visible.
Choose Codex as the harness; the UI displays its name/icon for both runners.
Advanced identifies the automatic default and offers Paperclip Runner or Legacy.
Only new Codex agents default to the native runner on supported macOS or GNU/Linux
targets; the [shared selection contract](../../packages/shared/src/agent-runner.ts)
and [runtime prerequisites](../../packages/paperclip-runner/README.md) define the
platform/libc boundary. Windows and Alpine/musl remain Legacy; other harness
defaults are unchanged. Disabled adapters, external overrides, and selected-target
qualification also affect availability.
On a fresh forced-Kubernetes instance without an environment row, inspect discovery
and create with automatic selection. Both must retain Legacy without provisioning;
an explicit Paperclip request must give target-qualification guidance. Begin the
ordinary task to exercise the existing lazy environment path. This live journey
remains separate from the resolver's source controls.
For sandbox defaults, follow the [environment probe](./execution-environments.md#environment-settings)
recipe first. Discovery/creation read company-bound immutable-image evidence
without allocating a target. A mutable image or snapshot name does not establish
native compatibility. Then test the actual selected runtime with its selected
account/model: a legacy CLI or authentication probe alone cannot establish native
readiness. Missing dependencies, failed authentication, invalid models and
incompatible custom settings must remain actionable errors without silent runner
switching. Saved unchanged execution can still be tested without applying new
selection policy, but runtime/account validation must still run.
Verify an automatic Codex hire and an explicit Legacy control through a useful
file artifact, follow-up and reload. Preserve Codex reasoning effort and account
isolation; retain explicit Legacy permission choices, including false. Reopen an
existing agent, make an unrelated edit, and confirm both its persisted runner and
next run stay unchanged. Use [hiring/import preservation](./agents.md#hire-agent)
for approval and portability controls.
Test native Codex with a saved subscription whose setup auth file is a managed
link, as well as a regular file and an API-key account. Setup must seed the
isolated probe from the checked account without exposing credentials or changing
its saved runner. With subscription authentication, an unbound configured
`CODEX_HOME` with linked credentials must fail setup with managed-account or explicit Legacy guidance; it must not claim
readiness when task isolation would omit the file. Dangling or incorrectly owned
credentials must also fail clearly. A selected `CODEX_API_KEY` or
`OPENAI_API_KEY` takes precedence over an unrelated subscription file. Test valid,
dangling and invalid subscription links with that selected key; setup must test
the key without reading or refreshing the unrelated account. The private probe copy must be removed after confirmed retirement following success,
failure or cancellation. Unconfirmed shutdown remains a cleanup failure.
Exercise an unavailable local daemon and a preparation failure before probe
handoff. No owned temporary credential copy may remain, and original account
files must be unchanged. A probe whose shutdown is unconfirmed must retain its
recovery state; do not treat it as a preparation-only failure.
Existing setup controls use fake credentials; real account execution remains
a separate acceptance requirement.
### `run-history`
Automated: [live run routes](../../server/src/__tests__/agent-live-run-routes.test.ts) and [run event sequencing](../../server/src/__tests__/heartbeat-run-event-sequencing.test.ts) cover run data contracts.
Manual: Start a disposable task, open its running transcript, and follow it to a terminal outcome. Reload and compare task/run IDs, status, output, and cost. Verify a failed run remains distinguishable from a completed run with useful output.
An explicit runner change starts a configuration revision and invalidates the
eligible session. Active runs, restart recovery and historical transcript parsing
retain their recorded execution configuration; do not infer their runner from a
later edit to the agent. Account/harness branding and execution identity are
different observations, both required in the evidence.
For an onboarding child-completion callback, verify actual session/turn events
and the source-correlated reply. A `succeeded` row alone is insufficient: the
literal `ea6c8eb5e16d61111968f633a66748b397b64a3e` managed attempt was marked
successful during startup without a provider turn. The recovery correction
preserves the existing validated Done-parent handoff; process-death and ownership
checks remain active. Its PostgreSQL controls are separate from the required
revised-source live onboarding and follow-up proof.
## Gotchas
- Model accounts are managed in Apps and inline agent account controls.
`/company/settings/connections` configures decision/fast-response models,
not the general model-account library.
- The run log is instance database data; it is not the opt-out first-party Telemetry system or operator-configured OpenTelemetry tracing.
- Authentication/model discovery does not establish tool, image, steering, or resumption capability parity.
- Use [recovery](./recovery.md) for interrupted runs and [execution environments](./execution-environments.md) for local/SSH/sandbox differences.
Native setup retirement joins owned startup before cleanup can be confirmed.
[Transport ownership controls](../../packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts)
hold preparation or registration without launching a provider, then check bounded
close, rejected duplicate requests, no later launch and retention of unconfirmed
state. Readiness observes shutdown; resumed startup cannot reactivate ingress
after close. These are lifecycle controls, separate from installed provider proof.
The real daemon readiness and recovery controls require their existing native
fixtures in hosted CI; passing credential-free controls does not qualify them.
For remote setup shutdown, verify that each durable-state read respects the
remaining suspension deadline. A late result cannot establish suspension or
queue a replacement command after that deadline. An intentional ingress close
must observe physical socket closure before retiring its connection owner and
releasing the endpoint; an unresponsive peer must not extend the setup budget.
The existing transport and ingress suites exercise these conditions without a
provider. Repeat the actual managed setup and useful task journey separately;
the literal `651a3de64fc943216cc414b64b1443809f9499da` Test attempt failed
cleanup before hire and retains its unconfirmed recovery state.
Reference in new issue View Git Blame Copy Permalink