Files
PaperClipAI/maps/agent-access-map/runs-adapters.md
T
DottaandPaperclip ebceb2a744 trace(server): time workspace, native setup and startup recovery (#16330)
## Thinking Path

> - Paperclip manages agents and their work.
> - The controller prepares workspaces before it starts a Runner.
> - A slow run can spend 30–80 seconds inside the combined workspace phase.
> - Startup recovery can also delay queued work for minutes.
> - Native runs also have gaps before Runner startup and after the provider result.
> - Existing traces do not identify the awaited steps inside those intervals.
> - This change adds closed timing boundaries around the existing operations.
> - Operators can locate the wait without changing execution or exposing content.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Operator-configured OpenTelemetry diagnosis of controller preparation and startup recovery. Related: #16178, #16236 and #16285 introduced the existing preparation timing and sampling contract. #16306 and #16312 reduce separate configuration and copyback work.

**Current behavior**

The workspace interval combines source resolution, trust checks, provisioning, database persistence, binding and repository admission. Startup awaits accounting and native recovery before queued work resumes, but neither has a separate duration. Native backend setup and post-turn cleanup also contain unmeasured awaits. CPU profiles and a fast database probe cannot identify which await is slow.

**Proposed behavior**

Record workspace steps and native controller setup and cleanup with the existing hashed run ID. Record startup phases as independent, operator-sampled root spans without custom attributes. Keep the same operation order, results, failures and authorization.

**Reason and benefit**

These boundaries distinguish source work, database writes, repository admission and restart recovery. This is a diagnosis change. It does not claim a response-time improvement or a solved latency budget.

## What Changed

- Add 14 closed workspace step names around 16 existing awaits. Repeated updates and bindings produce separate spans.
- Time startup tool reviews, gateway sessions, delivery-worker readiness, accounting, plugin readiness, native recovery, hot restart, orphan reaping, retries, queued work and goal recovery.
- Time native GitHub setup, wake binding, backend construction, connector assignments, MCP discovery, tool authority, companion selection and binary resolution. Time selected post-turn GitHub, coordinator and instruction cleanup awaits.
- Move preparation timing to a pure shared module. Keep the existing heartbeat import as an alias. Both heartbeat and native timing skip tracer lookup when the endpoint is unset or blank, including when another library registered a provider.
- Tracing lookup, callback, status and end faults cannot repeat work or replace its value or error.
- Add real PostgreSQL workspace timing coverage and silent no-op/fault controls. Update the tracing contract and paired run maps.
- Keep span attributes closed. Startup spans add none; workspace steps use only the existing run hash. No commands, paths, returned records or exception messages are recorded.

## Verification

- Diagnostic-gap baseline: the new real workspace test fails against the unchanged workspace implementation because the expected step spans are absent. It passes with the wrappers.
- Final affected coverage: **804 distinct cases pass** across seven complete suites: 698 native-executor cases, 39 real PostgreSQL workspace cases, 61 timing/context/startup cases, and six execution-configuration integration cases. The final native and unit invocations pass with no selected skips. The workspace suite passed in the initial six-file invocation on the same final workspace source.
- Preserve the initial six-file attempt as failed: **7 failed and 789 passed**. All seven errors occurred during embedded PostgreSQL startup. The log reports local macOS shared-memory exhaustion. A fresh 15-case database group passes, followed by the complete 698-case native suite. These runs overlap; do not add their counts.
- The new native timing assertion fails against unchanged native code because the expected spans are absent. It passes with the wrappers and retains the credential-isolation assertions.
- Full `pnpm -r typecheck` and `pnpm build`: passed on the final production source. Earlier nullable-closure errors are fixed with erased assertions inside existing guards. The failed attempts remain recorded.
- Normalized AST comparison: removing only timing wrappers, timing imports/local handles and erased assertions restores the original workspace, startup and native executor programs. This checks call order and policy expressions; it is not live acceptance.
- The expanded head first failed two configuration integration assertions because their test providers had no endpoint. Configure the endpoint explicitly in those two tracing tests, and restore it after each test. All six cases then pass with the original credential cleanup, redaction and exact span assertions. Production is unchanged by this repair.
- The first PR head failed one unchanged attachment test at its 15-second deadline in CI. The complete 32-case attachment suite passes locally with that deadline. Do not label the failed CI attempt green. The expanded head needs fresh full CI.
- `node scripts/check-maps.mjs`: passed, 35 areas and 176 capability IDs. `node scripts/storybook-catalog.mjs`: 283 files and 2,728 stories. Structural checks only; no browser, play or Storybook build claim.
- `git diff --check`: passed. No added prose comments, lockfile changes or new 1,000-line crossings.
- A duplicate full local test run was not started. Fresh exact-head full CI and E2E pass on `15a683bb2419960f611584f8c07cdcfc9ce3af0b`. All 56 check runs are successful or optional skipped. Superagent and legacy Snyk pass. Completed Apex review is 5/5 on this same head, with zero unresolved review threads.

Commands:

```sh
pnpm exec vitest run --project @paperclipai/server --maxWorkers=1 --fileParallelism=false server/src/services/native-runtime/native-session-executor.test.ts server/src/services/execution-preparation-timing.test.ts server/src/startup-recovery-timing.test.ts server/src/services/heartbeat/preparation-timing.test.ts server/src/services/heartbeat/preparation-timing-context.test.ts server/src/services/heartbeat/workspace-preparation.test.ts
pnpm -r typecheck
pnpm build
node scripts/check-maps.mjs
node scripts/storybook-catalog.mjs
```

## Risks

- Configured operators receive more spans. They still follow the configured sampler and exporter.
- Startup spans measure the awaited phase, including remaining readiness waits. They do not measure earlier worker construction or a provider turn.
- Workspace and native setup intervals overlap their containing phases. Do not add them to the parent duration. Native timing covers named awaits, not every statement or every error path.
- These spans are not yet deployed or tested against a live latency run. No first-party Telemetry event, run-log write, schema or API changes are included.

## Model Used

OpenAI GPT-6 through Codex, with tool use and code execution. The exact served variant and context-window size are not exposed to this task.

## Checklist

- [x] I have included a thinking path that traces from project context to this change
- [x] I have specified the model used (with version and capability details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before requesting merge

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-14 17:16:07 -05:00

12 KiB

Runs, harnesses, and model accounts — agent access

Product map: Runs, harnesses, and model accounts.

Review date: 2026-10-12. Source commit: 58e16dbddfc9c8d0cf60867f9d9847f0feda055b. Native REST transport reconciled in this change; see the shared transport contract.

This is a source-reviewed baseline of representative operations and known gaps. Each capability needs operation-level review before its entire scope can be qualified. No live agent execution is claimed. Follow the map contract for format, permission checks, evidence and maintenance.

Scoped runner-default source review: 2026-10-12 at eee0534154eaf5d0d3f2bbc97c1e24d86f34a5a5, limited to runner discovery, selected-runtime testing and saved execution/session preservation. The original baseline source and review depth remain unchanged for unrelated operations.

Scoped native cancellation source review: 2026-10-12 at ce23716503a2c9158b5a7ab74ab61571ff4ee7a3, limited to owned startup and shutdown/readiness ordering. The original baseline and review depth remain for unrelated operations; no live provider or agent authorization is claimed.

availability

Review: baseline.

API: present — representative operation IDs:

  • GET /api/adapters.
  • GET /api/companies/{companyId}/adapters/{type}/models.

Agent access: partial — Agent company-model REST / native call_api discovery and bound runtime context; adapter/account inventories remain board-only.

Policy: Run/cost visibility and agent configuration policy; adapter installation/account login/session control require board/admin.

Evidence: source-only — implementation at the reviewed commit above.

Verification: not-run — exercise a listed permitted operation with actual agent credentials/run context; inspect its result and applicable persistence/attribution. Assert the relevant denied actor, owner, company or mode from Policy. Review the remaining operations before qualifying the whole capability.

Gaps: Raw instance adapter listing requires board, while company model discovery exists for eligible agents.

Scoped source-only evidence: adapter discovery reports supported runners/effective default for the selected company/environment; selection policy reads server-owned immutable sandbox facts without provisioning. New Codex defaults and explicit overrides follow the shared runner contract; other harness defaults remain unchanged. The board inventory does not grant a native agent an equivalent discovery binding. See probe restrictions for acquiring qualification. An absent forced-Kubernetes environment is an unqualified sandbox target, never a local target. Discovery and automatic creation keep the Legacy path without provisioning it; explicit Paperclip selection returns qualification guidance. The existing execution path owns lazy environment creation. Saved runners remain unchanged. This is source/control evidence, not live Kubernetes qualification.

validation

Review: baseline.

API: present — representative operation IDs:

  • POST /api/companies/{companyId}/adapters/{type}/test-environment.

Agent access: partial — Permitted agent-creation validation through REST / native call_api / CLI; account login requires its operator.

Policy: Run/cost visibility and agent configuration policy; adapter installation/account login/session control require board/admin.

Evidence: source-only — implementation at the reviewed commit above.

Verification: not-run — exercise a listed permitted operation with actual agent credentials/run context; inspect its result and applicable persistence/attribution. Assert the relevant denied actor, owner, company or mode from Policy. Review the remaining operations before qualifying the whole capability.

Gaps: Environment test path checks creation/configuration policy; missing provider auth must use its owner setup flow.

Scoped source review: test authorization and resolution requires same-company agents:create, saved-agent update access when supplied, allowed environment selection, and normal secret/responsible-user checks. An unchanged saved runner avoids new-creation policy but still receives selected runtime validation. Native readiness tests native Codex startup/model/account and bounded teardown; an authentication or Legacy CLI check cannot replace that runtime evidence. Missing dependencies, credentials, invalid models and incompatible settings fail without silently changing runners. Account login remains board/owner-bound.

For a linked Codex subscription with server-resolved managed credential provenance, setup checks the resolved owner and stages a private probe copy before local or remote isolation. Without a selected API key, an unbound linked CODEX_HOME is rejected with explicit Legacy or managed-account guidance. Caller configuration cannot supply that provenance. Selected CODEX_API_KEY and OPENAI_API_KEY authentication skips unrelated subscription reads and refresh bindings, matching task materialization. The fake-credential controls verify key precedence without changing the account source or exposing either credential. The general native home materializer retains its no-follow symlink guard. Refresh still targets the checked account source, and confirmed bounded retirement removes its private copy. Unconfirmed retirement retains the existing cleanup block and owned state. Failures before probe handoff remove only the owned temporary preparation root; local daemon resolution occurs before credential staging. Once a probe is handed off, unconfirmed retirement still retains its recovery state. Existing fake-credential setup controls exercise linked/regular sources and failure cleanup. This adds no credential-read API, agent login authority or cross-company account access. Its source-only evidence does not qualify a live permitted/denied agent operation.

Verification for this scope remains not-run: use an admitted agent REST/CLI validation path, assert missing grants and cross-company targets/accounts are denied, then prove the useful task/follow-up separately. hire_agent inherits its caller's runtime; do not assume its binding accepts arbitrary runner or account settings. See creation access.

runs

Review: baseline.

Scoped source-only timing evidence: workspace preparation spans carry only the existing hashed run ID. Controller startup phase spans carry no custom attributes. Native controller setup and settlement phases use that same run hash, including assigned MCP discovery; neither tool definitions nor credentials are recorded. Both retain operator-controlled tracing; unconfigured clients stay silent. This adds no agent API, credential access, run-log write, or recovery authority. See the tracing contract. Live latency verification remains separate from the existing access baseline.

API: present — representative operation IDs:

  • GET /api/heartbeat-runs/{runId}.
  • GET /api/heartbeat-runs/{runId}/log.

Agent access: available — REST / native call_api / CLI run.

Policy: Run/cost visibility and agent configuration policy; adapter installation/account login/session control require board/admin.

Evidence: source-only — implementation at the reviewed commit above.

Verification: not-run — exercise a listed permitted operation with actual agent credentials/run context; inspect its result and applicable persistence/attribution. Assert the relevant denied actor, owner, company or mode from Policy. Review the remaining operations before qualifying the whole capability.

Gaps: Provider trace reveals/downloads require instance-admin and cannot be counted as ordinary run output access.

session

Review: baseline.

API: present — representative operation IDs:

  • POST /api/agents/{id}/runtime-state/reset-session.
  • GET /api/agents/{id}/task-sessions.

Agent access: restricted — Board session controls; native server-owned session continuation.

Policy: Run/cost visibility and agent configuration policy; adapter installation/account login/session control require board/admin.

Evidence: source-only — implementation at the reviewed commit above.

Verification: not-run — assert the documented agent denial with an owned test fixture; verify any listed agent request/result path separately. A delegated operation needs its own policy and access review.

Gaps: Task-session listing and execution-context reset require board agent management at their HTTP routes. Native call_api preserves these denials; native continuation remains server-owned.

Scoped source-only evidence: configuration changes and native execution/recovery retain existing execution on unrelated edits and use recorded run configuration for active/recovered runs. Explicit runner changes use revisions and session invalidation. Verify saved configuration, current versus historical run identity, restart and follow-up without granting session-reset authority to an agent.

Scoped source review: 2026-10-13, the recovery correction against ea6c8eb5e16d61111968f633a66748b397b64a3e reuses completed onboarding handoff validation before applying the issue-terminal backstop. The company, parent origin/status, assignee, completed child and remaining children must match. Only that backstop is suppressed for the validated report; process-death and controller/native ownership checks still apply. No agent gains session or recovery control. PostgreSQL regressions cover valid native/Legacy callbacks and rejected bindings; revised live execution and actor authorization remain not-run.

Scoped Evidence: source-only — the native transport joins owned startup before certifying cleanup. Rejected duplicates cannot replace that owner or mutate recovery state. Unconfirmed preparation or registration retains recovery state; late registration may release only its own route and cannot launch after close. Shutdown interrupts readiness without granting authority over an unauthenticated adopted process.

Scoped Verification: not-run — verify allowed/denied access using the actual actor independently from the credential-free ownership controls. This correction adds no account-setup, cleanup-override or session-reset authority. The real native readiness/recovery fixtures and installed provider acceptance remain separate gates.

Scoped shutdown source review: 2026-10-14, against literal 651a3de64fc943216cc414b64b1443809f9499da. The suspension barrier bounds remote state reads by its remaining deadline and rejects late proof. Intentional provider ingress shutdown joins physical socket closure and retires the connection owner before endpoint release. Unconfirmed termination stays blocked. Existing fake-peer and delayed read controls are source evidence; this changes no actor, company, account or session-reset permission. Operation-level verification remains not-run and the failed Test setup does not qualify successful live cleanup.