## Thinking Path > - Paperclip manages agents and their work. > - The controller prepares workspaces before it starts a Runner. > - A slow run can spend 30–80 seconds inside the combined workspace phase. > - Startup recovery can also delay queued work for minutes. > - Native runs also have gaps before Runner startup and after the provider result. > - Existing traces do not identify the awaited steps inside those intervals. > - This change adds closed timing boundaries around the existing operations. > - Operators can locate the wait without changing execution or exposing content. ## Linked Issues or Issue Description **What existing behavior does this improve?** Operator-configured OpenTelemetry diagnosis of controller preparation and startup recovery. Related: #16178, #16236 and #16285 introduced the existing preparation timing and sampling contract. #16306 and #16312 reduce separate configuration and copyback work. **Current behavior** The workspace interval combines source resolution, trust checks, provisioning, database persistence, binding and repository admission. Startup awaits accounting and native recovery before queued work resumes, but neither has a separate duration. Native backend setup and post-turn cleanup also contain unmeasured awaits. CPU profiles and a fast database probe cannot identify which await is slow. **Proposed behavior** Record workspace steps and native controller setup and cleanup with the existing hashed run ID. Record startup phases as independent, operator-sampled root spans without custom attributes. Keep the same operation order, results, failures and authorization. **Reason and benefit** These boundaries distinguish source work, database writes, repository admission and restart recovery. This is a diagnosis change. It does not claim a response-time improvement or a solved latency budget. ## What Changed - Add 14 closed workspace step names around 16 existing awaits. Repeated updates and bindings produce separate spans. - Time startup tool reviews, gateway sessions, delivery-worker readiness, accounting, plugin readiness, native recovery, hot restart, orphan reaping, retries, queued work and goal recovery. - Time native GitHub setup, wake binding, backend construction, connector assignments, MCP discovery, tool authority, companion selection and binary resolution. Time selected post-turn GitHub, coordinator and instruction cleanup awaits. - Move preparation timing to a pure shared module. Keep the existing heartbeat import as an alias. Both heartbeat and native timing skip tracer lookup when the endpoint is unset or blank, including when another library registered a provider. - Tracing lookup, callback, status and end faults cannot repeat work or replace its value or error. - Add real PostgreSQL workspace timing coverage and silent no-op/fault controls. Update the tracing contract and paired run maps. - Keep span attributes closed. Startup spans add none; workspace steps use only the existing run hash. No commands, paths, returned records or exception messages are recorded. ## Verification - Diagnostic-gap baseline: the new real workspace test fails against the unchanged workspace implementation because the expected step spans are absent. It passes with the wrappers. - Final affected coverage: **804 distinct cases pass** across seven complete suites: 698 native-executor cases, 39 real PostgreSQL workspace cases, 61 timing/context/startup cases, and six execution-configuration integration cases. The final native and unit invocations pass with no selected skips. The workspace suite passed in the initial six-file invocation on the same final workspace source. - Preserve the initial six-file attempt as failed: **7 failed and 789 passed**. All seven errors occurred during embedded PostgreSQL startup. The log reports local macOS shared-memory exhaustion. A fresh 15-case database group passes, followed by the complete 698-case native suite. These runs overlap; do not add their counts. - The new native timing assertion fails against unchanged native code because the expected spans are absent. It passes with the wrappers and retains the credential-isolation assertions. - Full `pnpm -r typecheck` and `pnpm build`: passed on the final production source. Earlier nullable-closure errors are fixed with erased assertions inside existing guards. The failed attempts remain recorded. - Normalized AST comparison: removing only timing wrappers, timing imports/local handles and erased assertions restores the original workspace, startup and native executor programs. This checks call order and policy expressions; it is not live acceptance. - The expanded head first failed two configuration integration assertions because their test providers had no endpoint. Configure the endpoint explicitly in those two tracing tests, and restore it after each test. All six cases then pass with the original credential cleanup, redaction and exact span assertions. Production is unchanged by this repair. - The first PR head failed one unchanged attachment test at its 15-second deadline in CI. The complete 32-case attachment suite passes locally with that deadline. Do not label the failed CI attempt green. The expanded head needs fresh full CI. - `node scripts/check-maps.mjs`: passed, 35 areas and 176 capability IDs. `node scripts/storybook-catalog.mjs`: 283 files and 2,728 stories. Structural checks only; no browser, play or Storybook build claim. - `git diff --check`: passed. No added prose comments, lockfile changes or new 1,000-line crossings. - A duplicate full local test run was not started. Fresh exact-head full CI and E2E pass on `15a683bb2419960f611584f8c07cdcfc9ce3af0b`. All 56 check runs are successful or optional skipped. Superagent and legacy Snyk pass. Completed Apex review is 5/5 on this same head, with zero unresolved review threads. Commands: ```sh pnpm exec vitest run --project @paperclipai/server --maxWorkers=1 --fileParallelism=false server/src/services/native-runtime/native-session-executor.test.ts server/src/services/execution-preparation-timing.test.ts server/src/startup-recovery-timing.test.ts server/src/services/heartbeat/preparation-timing.test.ts server/src/services/heartbeat/preparation-timing-context.test.ts server/src/services/heartbeat/workspace-preparation.test.ts pnpm -r typecheck pnpm build node scripts/check-maps.mjs node scripts/storybook-catalog.mjs ``` ## Risks - Configured operators receive more spans. They still follow the configured sampler and exporter. - Startup spans measure the awaited phase, including remaining readiness waits. They do not measure earlier worker construction or a provider turn. - Workspace and native setup intervals overlap their containing phases. Do not add them to the parent duration. Native timing covers named awaits, not every statement or every error path. - These spans are not yet deployed or tested against a live latency run. No first-party Telemetry event, run-log write, schema or API changes are included. ## Model Used OpenAI GPT-6 through Codex, with tool use and code execution. The exact served variant and context-window size are not exposed to this task. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-Authored-By: Paperclip <noreply@paperclip.ing>
12 KiB
Runs, harnesses, and model accounts — agent access
Product map: Runs, harnesses, and model accounts.
Review date: 2026-10-12. Source commit: 58e16dbddfc9c8d0cf60867f9d9847f0feda055b.
Native REST transport reconciled in this change; see the
shared transport contract.
This is a source-reviewed baseline of representative operations and known gaps. Each capability needs operation-level review before its entire scope can be qualified. No live agent execution is claimed. Follow the map contract for format, permission checks, evidence and maintenance.
Scoped runner-default source review: 2026-10-12 at
eee0534154eaf5d0d3f2bbc97c1e24d86f34a5a5, limited to runner discovery,
selected-runtime testing and saved execution/session preservation. The original
baseline source and review depth remain unchanged for unrelated operations.
Scoped native cancellation source review: 2026-10-12 at
ce23716503a2c9158b5a7ab74ab61571ff4ee7a3, limited to owned startup and
shutdown/readiness ordering. The original baseline and review depth remain for
unrelated operations; no live provider or agent authorization is claimed.
availability
Review: baseline.
API: present — representative operation IDs:
GET /api/adapters.GET /api/companies/{companyId}/adapters/{type}/models.
Agent access: partial — Agent company-model REST / native call_api discovery and
bound runtime context; adapter/account inventories remain board-only.
Policy: Run/cost visibility and agent configuration policy; adapter installation/account login/session control require board/admin.
Evidence: source-only — implementation at
the reviewed commit above.
Verification: not-run — exercise a listed permitted operation with actual agent
credentials/run context; inspect its result and applicable persistence/attribution.
Assert the relevant denied actor, owner, company or mode from Policy. Review the
remaining operations before qualifying the whole capability.
Gaps: Raw instance adapter listing requires board, while company model discovery exists for eligible agents.
Scoped source-only evidence: adapter discovery reports supported runners/effective default for the selected company/environment; selection policy reads server-owned immutable sandbox facts without provisioning. New Codex defaults and explicit overrides follow the shared runner contract; other harness defaults remain unchanged. The board inventory does not grant a native agent an equivalent discovery binding. See probe restrictions for acquiring qualification. An absent forced-Kubernetes environment is an unqualified sandbox target, never a local target. Discovery and automatic creation keep the Legacy path without provisioning it; explicit Paperclip selection returns qualification guidance. The existing execution path owns lazy environment creation. Saved runners remain unchanged. This is source/control evidence, not live Kubernetes qualification.
validation
Review: baseline.
API: present — representative operation IDs:
POST /api/companies/{companyId}/adapters/{type}/test-environment.
Agent access: partial — Permitted agent-creation validation through REST / native
call_api / CLI; account login requires its operator.
Policy: Run/cost visibility and agent configuration policy; adapter installation/account login/session control require board/admin.
Evidence: source-only — implementation at
the reviewed commit above.
Verification: not-run — exercise a listed permitted operation with actual agent
credentials/run context; inspect its result and applicable persistence/attribution.
Assert the relevant denied actor, owner, company or mode from Policy. Review the
remaining operations before qualifying the whole capability.
Gaps: Environment test path checks creation/configuration policy; missing provider auth must use its owner setup flow.
Scoped source review: test authorization and resolution
requires same-company agents:create, saved-agent update access when supplied,
allowed environment selection, and normal secret/responsible-user checks. An
unchanged saved runner avoids new-creation policy but still receives selected
runtime validation. Native readiness
tests native Codex startup/model/account and bounded teardown; an authentication
or Legacy CLI check cannot replace that runtime evidence. Missing dependencies,
credentials, invalid models and incompatible settings fail without silently
changing runners. Account login remains board/owner-bound.
For a linked Codex subscription with server-resolved managed credential provenance,
setup checks the resolved owner and stages a private probe copy before local or
remote isolation. Without a selected API key, an unbound linked CODEX_HOME is
rejected with explicit Legacy
or managed-account guidance. Caller configuration cannot supply that provenance. Selected CODEX_API_KEY and
OPENAI_API_KEY authentication skips unrelated subscription reads and refresh
bindings, matching task materialization. The fake-credential controls verify key
precedence without changing the account source or exposing either credential.
The general native
home materializer retains its no-follow symlink guard. Refresh still targets the
checked account source, and confirmed bounded retirement removes its private
copy. Unconfirmed retirement retains the existing cleanup block and owned state.
Failures before probe handoff remove only the owned temporary preparation root;
local daemon resolution occurs before credential staging. Once a probe is handed
off, unconfirmed retirement still retains its recovery state.
Existing fake-credential setup controls
exercise linked/regular sources and failure cleanup. This adds no credential-read
API, agent login authority or cross-company account access. Its source-only
evidence does not qualify a live permitted/denied agent operation.
Verification for this scope remains not-run: use an admitted agent REST/CLI
validation path, assert missing grants and cross-company targets/accounts are
denied, then prove the useful task/follow-up separately. hire_agent inherits its
caller's runtime; do not assume its binding accepts arbitrary runner or account
settings. See creation access.
runs
Review: baseline.
Scoped source-only timing evidence: workspace preparation spans carry only the existing hashed run ID. Controller startup phase spans carry no custom attributes. Native controller setup and settlement phases use that same run hash, including assigned MCP discovery; neither tool definitions nor credentials are recorded. Both retain operator-controlled tracing; unconfigured clients stay silent. This adds no agent API, credential access, run-log write, or recovery authority. See the tracing contract. Live latency verification remains separate from the existing access baseline.
API: present — representative operation IDs:
GET /api/heartbeat-runs/{runId}.GET /api/heartbeat-runs/{runId}/log.
Agent access: available — REST / native call_api / CLI run.
Policy: Run/cost visibility and agent configuration policy; adapter installation/account login/session control require board/admin.
Evidence: source-only — implementation at
the reviewed commit above.
Verification: not-run — exercise a listed permitted operation with actual agent
credentials/run context; inspect its result and applicable persistence/attribution.
Assert the relevant denied actor, owner, company or mode from Policy. Review the
remaining operations before qualifying the whole capability.
Gaps: Provider trace reveals/downloads require instance-admin and cannot be counted as ordinary run output access.
session
Review: baseline.
API: present — representative operation IDs:
POST /api/agents/{id}/runtime-state/reset-session.GET /api/agents/{id}/task-sessions.
Agent access: restricted — Board session controls; native server-owned session
continuation.
Policy: Run/cost visibility and agent configuration policy; adapter installation/account login/session control require board/admin.
Evidence: source-only — implementation at
the reviewed commit above.
Verification: not-run — assert the documented agent denial with an owned test fixture;
verify any listed agent request/result path separately. A delegated operation needs its
own policy and access review.
Gaps: Task-session listing and execution-context reset require board agent
management at their HTTP routes. Native call_api preserves these denials; native
continuation remains server-owned.
Scoped source-only evidence: configuration changes and native execution/recovery retain existing execution on unrelated edits and use recorded run configuration for active/recovered runs. Explicit runner changes use revisions and session invalidation. Verify saved configuration, current versus historical run identity, restart and follow-up without granting session-reset authority to an agent.
Scoped source review: 2026-10-13, the recovery correction against
ea6c8eb5e16d61111968f633a66748b397b64a3e reuses
completed onboarding handoff validation
before applying the issue-terminal backstop. The company, parent origin/status,
assignee, completed child and remaining children must match. Only that backstop
is suppressed for the validated report; process-death and controller/native
ownership checks still apply. No agent gains session or recovery control.
PostgreSQL regressions cover valid native/Legacy callbacks and rejected bindings;
revised live execution and actor authorization remain not-run.
Scoped Evidence: source-only — the
native transport
joins owned startup before certifying cleanup. Rejected duplicates cannot replace
that owner or mutate recovery state. Unconfirmed preparation or registration
retains recovery state; late registration may release only its own route and
cannot launch after close. Shutdown interrupts readiness without granting
authority over an unauthenticated adopted process.
Scoped Verification: not-run — verify allowed/denied access using the actual
actor independently from the credential-free ownership controls. This correction
adds no account-setup, cleanup-override or session-reset authority. The real native
readiness/recovery fixtures and installed provider acceptance remain separate gates.
Scoped shutdown source review: 2026-10-14, against literal
651a3de64fc943216cc414b64b1443809f9499da. The suspension barrier bounds remote
state reads by its remaining deadline and rejects late proof. Intentional
provider ingress shutdown
joins physical socket closure and retires the connection owner before endpoint
release. Unconfirmed termination stays blocked. Existing fake-peer and delayed
read controls are source evidence; this changes no actor, company, account or
session-reset permission. Operation-level verification remains not-run and
the failed Test setup does not qualify successful live cleanup.