From cccde7122b9717b26737ebf4dd969c214191a71d Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Thu, 15 Oct 2026 08:13:53 -0500 Subject: [PATCH 01/19] Trace execution preparation and durable response persistence (#16419) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The server prepares an execution environment before it starts an agent. > - Operators need to distinguish preparation, provider wait, and response persistence. > - Existing preparation phases start separate traces, and the native session trace ends before the final response is saved. > - This pull request adds an execution-attempt trace and detailed spans at the existing work boundaries. > - The benefit is a measurable startup and settlement path without changing execution policy. ## Linked Issues or Issue Description **What existing behavior does this improve?** Operator-configured OpenTelemetry tracing for agent preparation, startup, and durable response storage. **Subsystem affected** Server orchestration, native Runner startup, and the Honeycomb run link. **Current behavior** Preparation phases start independent roots. Workspace lock waits, authorization groups, runtime asset work, provider-pack fallback, and the gap after provider completion are hard to attribute. **Proposed behavior** Use a server-owned async context for each execution attempt. Parent nested work to that attempt. Link the separately sampled native session because its ingress and queue history can begin earlier. Record transaction entry, lock waits, grouped authorization, asset preparation, artifact fallback, copyback, and response commit. **Reason and benefit** Operators can identify expensive work from recorded boundaries instead of inferring the cause from elapsed time. Nested and concurrent durations remain visible and must not be added together. **Breaking changes** No API or execution-policy change. New Observability span names and closed count attributes are documented. The existing local run-performance schema also records the additional provider-pack and harness measurements. Refs: #16178, #16205, #16330. ## What Changed - Add `task.execution_attempt` with same-run async ownership, independent root sampling, and native-session links. - Measure workspace transaction entry and settlement, advisory and row waits, source scans, reader groups, notifications, and writes. - Measure runtime selection and each concurrent instructions, skills, and MCP branch. Separate manifest, cache, materialization, and verification work. - Measure provider-pack discovery, verified reuse or upload, final verification, and remote harness state checks. - Measure workspace settlement, accounting, liveness, response authorization, comment commit, and publication enqueue. - Preserve the silent endpoint-off path, including when another library has registered a tracer. Bound repeated detail spans to 16 per name per attempt. - Extend the Honeycomb run link and document the actual coverage and remaining gaps. ## Verification - After resolving the conflict with retained-reader batching on `b98cdaea531c8447637cd4dcc19c61b3f5f41844`, all 61 targeted query, authorization, workspace-preparation and trace-boundary tests passed. The server typecheck passed. Reader tests still bound query count, avoid loading retained run payloads and enforce fresh project grants on every publication attempt. - Before that base integration, the tracing/exporter group passed 855 tests with 2 optional-SDK skips across 10 files and no inherited OTLP endpoint. It covers real PostgreSQL waits/rollback, durable response commit, assembled heartbeat preparation, concurrent parentage, sampling and tracer failure isolation. Native workspace/history tests passed 23 cases. These results remain supporting evidence for the unchanged tracing paths. - Full workspace typecheck and build passed during implementation. Full sharded CI, release validation, E2E and Apex 5/5 passed on prior head `8cebad80f3856dbcff6f3a6d5d34880ff77a20f2`. Fresh CI, security and Apex review are required on the current head before merge. - `pnpm check:token-gates` passed. No styles changed. - Earlier CI runs exposed exact-source-expression assertions changed by wrappers, missing run hashes on transaction-boundary spans, and sampling fixtures without an enabled endpoint. The fixes preserve the calls, add the declared hash and span cap, and enable tracing explicitly in the sampling fixtures. Focused repairs passed; the full prior-head CI passed after those repairs. Earlier failed run: https://github.com/paperclipai/paperclip/actions/runs/38729131366. - The full local `pnpm test:run` was stopped after the base advanced during that process. Its mixed-version results are not final-head evidence. A full local-suite pass is not claimed. - The old-head Superagent scan stalled at 35%. The current head has a fresh scan; no security pass is inferred from the old pending result. - No deployed Honeycomb acceptance is claimed by this code change. ## Risks - Enabled tracing creates more spans. Repeated detail spans are bounded, and inventories use group spans. - The attempt and native session remain linked traces. Historical queue time can precede the attempt, and nested intervals overlap. - A committed comment proves durable server storage. Publication enqueue and realtime dispatch do not prove external delivery or browser paint. - New custom attributes use only the existing run hash and validated file/byte counts. No SQL, paths, permissions, content, or exception text is added. ## Model Used OpenAI GPT-6 through Codex. The serving model ID and context window size are not exposed in this session. The agent used reasoning, tools, code editing, and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/observability.md | 69 +++- doc/run-log-events.md | 14 + .../src/acpx-engine/startup-timing.ts | 6 + server/src/__tests__/redaction.test.ts | 4 + server/src/instrumentation.ts | 4 +- server/src/redaction.ts | 11 + server/src/services/activity-log.ts | 7 + .../chat-publication-reconciliation.test.ts | 2 +- .../services/execution-preparation-timing.ts | 154 ++++++-- .../services/execution-timing-context.test.ts | 192 ++++++++++ .../execution-transaction-timing.test.ts | 127 +++++++ .../services/execution-transaction-timing.ts | 31 ++ .../execution-workspace-repositories.ts | 328 +++++++++--------- server/src/services/execution-workspaces.ts | 64 ++-- server/src/services/heartbeat.ts | 18 +- .../src/services/heartbeat/run-completion.ts | 57 +-- .../services/heartbeat/runtime-execution.ts | 19 +- .../services/heartbeat/runtime-selection.ts | 90 ++--- server/src/services/issues.ts | 20 +- .../native-run-trace-context.test.ts | 5 +- .../native-runtime/native-run-trace.test.ts | 26 ++ .../native-runtime/native-run-trace.ts | 31 +- .../native-runtime/native-session-executor.ts | 33 +- .../native-session-resume.test.ts | 4 +- ...native-workspace-finalization-ownership.ts | 7 + .../native-runtime/native-workspace-sync.ts | 65 ++-- .../native-runtime/remote-provider-pack.ts | 30 +- .../native-runtime/runtime-context.test.ts | 54 +++ .../native-runtime/runtime-context.ts | 106 +++--- .../services/task-workspace-source-access.ts | 119 +++---- ui/src/lib/honeycomb-run-link.test.ts | 2 +- ui/src/lib/honeycomb-run-link.ts | 2 +- 32 files changed, 1187 insertions(+), 514 deletions(-) create mode 100644 server/src/services/execution-timing-context.test.ts create mode 100644 server/src/services/execution-transaction-timing.test.ts create mode 100644 server/src/services/execution-transaction-timing.ts diff --git a/doc/observability.md b/doc/observability.md index 741dced29a7..8d15d11be67 100644 --- a/doc/observability.md +++ b/doc/observability.md @@ -179,13 +179,15 @@ connection instructions, and agent identity. Their names are `execution_configuration`; do not add them to its duration. They remain available when a later setup step fails before the provider lifecycle trace starts. -Each phase starts a new root trace and links to the active trigger span when -one exists. An unsampled HTTP request must not suppress diagnostics for the -background run it starts. Root traces still follow the operator's configured -sampler. The trigger's trace remains unchanged, and overlapping runs keep -their own phase and child contexts. +`task.execution_attempt` owns one invocation of the server executor, from its +admission reads through preparation, provider dispatch, completion, reply persistence, +and cleanup. It starts a new root trace with a link to the active trigger. It +therefore follows the operator's root sampler even when the HTTP trigger was not +sampled. Async-local ownership parents same-run preparation and nested phases to +the current attempt. Concurrent runs have separate contexts. A phase invoked +without a live matching attempt still starts an independent linked root. -These spans exist before provider dispatch, including setup failures. Each has +These spans exist before provider dispatch, including setup failures. Preparation phases have only the existing `paperclip.task.run.run_id` attribute: the first 12 hexadecimal characters of the run UUID's SHA-256 hash. Failed operations set error status without recording exception messages, paths, inputs, or returned content. @@ -203,8 +205,8 @@ project repositories, and recovery-root reservation. These intervals are inside Do not add these intervals to the containing workspace duration. They use the same run hash and contain no workspace paths, commands, or returned records. -The `paperclip.native-controller` tracer records separate -`native.controller.` root spans around GitHub access setup, wake binding, +The `paperclip.native-controller` tracer records +`native.controller.` spans around GitHub access setup, wake binding, backend construction, connector assignments, assigned MCP discovery, tool authority, companion selection and Runner binary resolution. It also records GitHub shutdown, coordinator release, instruction checkpoints and collection, @@ -240,9 +242,51 @@ once known, `paperclip.native_reconciliation.candidates` (a count). A failed pass marks the pass span and the phase that was running with an error status. Faster passes record nothing. +### Detailed execution phases + +The `paperclip.execution` tracer adds these content-free spans beneath the current +attempt or owning phase: + +| Span family | Measured work | +| --- | --- | +| `workspace.transaction`, `.entry`, `.commit`, `.rollback` | Transaction lifetime, connection acquisition plus BEGIN, and settlement after the callback returns. Entry cannot distinguish pool wait from BEGIN. | +| `workspace.provenance_lock`, `workspace.row_lock` | The awaited advisory-lock statement and current workspace row read with its row lock. | +| `workspace.sources.*` | Source inventory queries, canonicalization, one filesystem alias scan, repository scan, and configured project scan. | +| `workspace.readers`, `workspace.readers.*`, `workspace.access.*` | Retained-reader groups, identity, trust, project and source authorization, privacy and inventory reads. | +| `workspace.provenance`, `workspace.notification`, `workspace.write` | Preserved ownership, notification statements, and record writes. | +| `runtime_selection.*` | Completion contracts, session/recovery reads, profiles, interaction materialization, backend discovery, bootstrap, run/coordinator writes, transaction entry/settlement, and workspace synchronization. | +| `runtime_context.instructions`, `.skills`, `.mcp` | The three concurrent runtime-context branches. Their durations overlap. | +| `runtime_context.asset_*` | Source reads, manifest hashing, cache lookup, cold materialization, and exact file verification on cold and warm paths. | +| `workspace_sync.*` | Existing Git and baseline snapshot, pack, seed verification, transfer, and restore operations. Pack and verification can be inside transfer; do not add nested durations. | +| `execution.runtime` | Runtime dispatch, usage, workspace finalization and runtime cleanup; this includes work after provider execution. | +| `execution.workspace_ownership`, `.workspace_ownership_wait`, `.workspace_restore`, `.workspace_commit`, `.native_finalize` | Ownership coordination including its action, acquisition before the action, copyback, final snapshot/stamp/descriptor persistence, and native settlement. | +| `completion.*` | Completion handler, run-log finalization, terminal persistence, accounting, liveness, and continuation summaries. | +| `response.resolve`, `.authorize`, `.add_comment`, `.transaction*`, `.row_lock`, `.write`, `.publish` | Response selection, existing authorization, comment transaction including commit or rollback, row wait, insert, and external publication enqueue. | +| `execution.realtime` | Synchronous live-event and plugin publication dispatch. It does not measure browser receipt or paint. | + +Detail phases emit at most 16 spans of each name per attempt. Larger reader and +asset inventories continue under the containing group without unbounded per-row +or per-file spans. Work, query order, authorization, and locks are unchanged. +Manual transaction and dispatch boundaries retain their actual lifetime. +Asset writes and verification may add only non-negative integer +`paperclip.execution.fileCount` and `paperclip.execution.bytes` attributes. +The closed keys are declared in `EXECUTION_PREPARATION_SPAN_ATTRS` in +`packages/adapter-utils/src/acpx-engine/startup-timing.ts`. +Other execution phases carry only the existing hashed run ID; manual boundaries +have no attributes. Exceptions set status without exporting their messages. + +A committed comment transaction establishes durable server storage. A publication +enqueue establishes pending delivery, and realtime dispatch establishes a server +notification call. None establishes browser paint or external delivery. The attempt +can continue through later bookkeeping and resource cleanup. A later recovery +invocation has its own attempt; this does not manufacture one continuous trace +across queueing, process restarts, or an unavailable controller. + + ## Native Runner Trace Spans -Paperclip Runner task runs emit a single foldable OpenTelemetry trace. This is +The native provider lifecycle emits one foldable OpenTelemetry trace, linked to +the server attempt described above. This is the native-run trace schema version `3`. `task.provider_session` is the native provider lifecycle root; it includes preparation and provider execution, but ends before outer workspace persistence. The heartbeat-owned local run-log @@ -251,12 +295,15 @@ every other native span carries a real OpenTelemetry parent context rather than only a descriptive `parentName` field. The native lifecycle root uses the operator's root sampling policy. It links -to a valid active trigger trace without inheriting its sampling decision. +to the matching live server attempt (or a valid active trigger) without inheriting +its sampling decision. This is a link because historical ingress and queue spans +can predate this invocation. It does not reparent earlier work beneath a later span. An unsampled HTTP request therefore cannot suppress an otherwise sampled background run. An operator who disables root sampling still gets no spans. Without an OTLP endpoint, tracing remains silent and does not export data. -The run link in Honeycomb matches both `task.provider_session` for native runs and `task.run` for legacy adapters. It retains the exact hashed run-ID and service filters and groups by trace ID. The URL builder uses Honeycomb’s supported `in` operator for the two root names; see the [query specification](https://github.com/honeycombio/honeycomb-mcp#query-specification-for-run_query). This changes only the query link. The operator endpoint gate and span attribute allowlist remain unchanged. +The run link in Honeycomb matches `task.provider_session` for native runs, `task.run` for legacy adapters, and +`task.execution_attempt` for server preparation and settlement. It retains the exact hashed run-ID and service filters and groups by trace ID. The URL builder uses Honeycomb’s supported `in` operator for the three root names; see the [query specification](https://github.com/honeycombio/honeycomb-mcp#query-specification-for-run_query). This changes only the query link. The operator endpoint gate and span attribute allowlist remain unchanged. The canonical lifecycle is: diff --git a/doc/run-log-events.md b/doc/run-log-events.md index 9a9dfe7507e..47c8a4d16ab 100644 --- a/doc/run-log-events.md +++ b/doc/run-log-events.md @@ -5,6 +5,20 @@ Run-log events write to the `heartbeat_run_events` table Paperclip Telemetry events, and they are not OpenTelemetry exports. A run-log event needs no operator endpoint. +## Provider-pack preparation timing + +The existing `run.performance.span` event also records `provider_pack.select`, +`provider_pack.discover`, `provider_pack.verify`, `provider_pack.verify_preinstalled`, +`provider_pack.link_preinstalled`, `provider_pack.clear_stale`, `provider_pack.upload`, +and `provider_pack.verify_uploaded`. Selection records `stateSource` as `staged`, +`preinstalled`, or `uploaded`. Its closed reason is `verified_pack_reused` or +`cached_packs_unverified`. A verification failure describes an unusable candidate; +it does not identify a particular compatibility mismatch or a failed run. +Harness startup also records `harness_state.inspect`, `harness_state.path_check`, +and `harness_state.launch_material` for remote state inspection and fresh launch +material. These events retain the existing schema and carry no paths or exception strings. +They also become native Observability spans only with an operator OTLP endpoint. + ## Native PRP Run-Log Events `native.recovery.execution_reclaimed` records an atomic same-run recovery claim diff --git a/packages/adapter-utils/src/acpx-engine/startup-timing.ts b/packages/adapter-utils/src/acpx-engine/startup-timing.ts index ebbd237befb..590d1f4f2ab 100644 --- a/packages/adapter-utils/src/acpx-engine/startup-timing.ts +++ b/packages/adapter-utils/src/acpx-engine/startup-timing.ts @@ -47,6 +47,12 @@ export function normalizeProviderFamily(key: string | undefined): string { */ export const SANDBOX_STARTUP_SPAN_ATTR_PREFIX = "paperclip.sandbox.startup."; +export const EXECUTION_PREPARATION_SPAN_ATTRS = { + runId: "paperclip.task.run.run_id", + fileCount: "paperclip.execution.fileCount", + bytes: "paperclip.execution.bytes", +} as const; + /** * The closed attribute-name contract for every sandbox-startup span. This is * the single source of truth for the harness span attributes. Each name uses diff --git a/server/src/__tests__/redaction.test.ts b/server/src/__tests__/redaction.test.ts index 1075c05ea18..c95fb8ae321 100644 --- a/server/src/__tests__/redaction.test.ts +++ b/server/src/__tests__/redaction.test.ts @@ -380,6 +380,10 @@ describe("redaction", () => { "runner.session.bootstrap", "runner.turn.submit", "runner.session.startup", + "provider_pack.select", + "provider_pack.discover", + "provider_pack.verify_uploaded", + "provider_pack.upload", "provider.turn.queue", "question_response.to_run_created", "native.session.execute", diff --git a/server/src/instrumentation.ts b/server/src/instrumentation.ts index 34275e1dbdc..fe8f0665bfd 100644 --- a/server/src/instrumentation.ts +++ b/server/src/instrumentation.ts @@ -39,7 +39,7 @@ import { checkExactPeerVersions } from "./peer-version-check.js"; export { checkExactPeerVersions } from "./peer-version-check.js"; -const endpoint = process.env.OTEL_EXPORTER_OTLP_ENDPOINT; +const endpoint = process.env.OTEL_EXPORTER_OTLP_ENDPOINT?.trim(); let sdkShutdown: (() => Promise) | null = null; let shutdownPromise: Promise | null = null; @@ -96,6 +96,7 @@ let tracerApiLoadFailed = false; * local no-op tracer (fail open). */ export function getStartupTracer(name = "paperclip.startup"): StartupTracerHandle { + if (!process.env.OTEL_EXPORTER_OTLP_ENDPOINT?.trim()) return NOOP_TRACER; try { const require = createRequire(import.meta.url); const api = require("@opentelemetry/api") as { @@ -151,6 +152,7 @@ let traceContextApiLoadFailed = false; * local no-op trace context (fail open). */ export function getStartupTraceContext(name = "paperclip.startup"): StartupTraceContextHandle { + if (!process.env.OTEL_EXPORTER_OTLP_ENDPOINT?.trim()) return NOOP_TRACE_CONTEXT; try { const require = createRequire(import.meta.url); const api = require("@opentelemetry/api") as { diff --git a/server/src/redaction.ts b/server/src/redaction.ts index 03efe939d53..fd7998073dd 100644 --- a/server/src/redaction.ts +++ b/server/src/redaction.ts @@ -328,6 +328,17 @@ const NATIVE_RUN_SPAN_NAMES = new Set([ "provider.time_to_first_agent_event", "provider.turn.queue", "question_response.to_run_created", + "harness_state.inspect", + "harness_state.path_check", + "harness_state.launch_material", + "provider_pack.select", + "provider_pack.discover", + "provider_pack.verify", + "provider_pack.verify_preinstalled", + "provider_pack.verify_uploaded", + "provider_pack.link_preinstalled", + "provider_pack.clear_stale", + "provider_pack.upload", "runner.artifact.discover", "runner.artifact.prepare", "runner.prp.authenticate", diff --git a/server/src/services/activity-log.ts b/server/src/services/activity-log.ts index a7e4376dbdb..4657c4eb426 100644 --- a/server/src/services/activity-log.ts +++ b/server/src/services/activity-log.ts @@ -1,3 +1,4 @@ +import { beginRunPhase } from "./execution-preparation-timing.js"; import type { ActivityPublication } from "../types/activity-publication.js"; export type { ActivityPublication } from "../types/activity-publication.js"; import { randomUUID } from "node:crypto"; @@ -146,12 +147,18 @@ export async function resolveResponsibleUserIdForActivity(db: Db, input: LogActi } export function publishActivity(publication: ActivityPublication) { + const finish = beginRunPhase("execution.realtime"); + try { publishLiveEvent({ companyId: publication.companyId, type: "activity.logged", payload: publication.payload, }); if (publication.pluginEvent) publishPluginDomainEvent(publication.pluginEvent); + } catch (error) { + finish(true); + throw error; + } finally { finish(); } } export async function persistActivity(db: Db, input: LogActivityInput) { diff --git a/server/src/services/chat-publication-reconciliation.test.ts b/server/src/services/chat-publication-reconciliation.test.ts index aa3fad2a36a..3311bd77baf 100644 --- a/server/src/services/chat-publication-reconciliation.test.ts +++ b/server/src/services/chat-publication-reconciliation.test.ts @@ -247,7 +247,7 @@ describe("chat publication commit signals", () => { 'eventType: "run.presentation.resolved"', ); const committedComment = completionSource.lastIndexOf( - "await issuesSvc.addComment", + "issuesSvc.addComment", presentationMarker, ); expect(presentationMarker).toBeGreaterThanOrEqual(0); diff --git a/server/src/services/execution-preparation-timing.ts b/server/src/services/execution-preparation-timing.ts index ace73f626bc..c20c5ca3737 100644 --- a/server/src/services/execution-preparation-timing.ts +++ b/server/src/services/execution-preparation-timing.ts @@ -1,5 +1,7 @@ +import { AsyncLocalStorage } from "node:async_hooks"; +import { EXECUTION_PREPARATION_SPAN_ATTRS, runWithRuntimeParent } from "@paperclipai/adapter-utils/acpx-engine/startup-timing"; import { createHash } from "node:crypto"; -import { isSpanContextValid, ROOT_CONTEXT, SpanStatusCode, trace } from "@opentelemetry/api"; +import { isSpanContextValid, ROOT_CONTEXT, SpanStatusCode, trace, type Context } from "@opentelemetry/api"; type PreparationPhase = "environment_selection" | "execution_configuration" | "session" | "workspace" | "github_environment" | "github_launchers" | "run_output" @@ -27,37 +29,125 @@ type PreparationPhase = "environment_selection" | "execution_configuration" | "s | "github_stop" | "coordinator_release" | "instruction_checkpoint" | "instruction_changes" | "warm_release" | "instruction_collection"; +export type ExecutionPhase = + | "workspace.transaction" | "workspace.transaction.entry" | "workspace.transaction.commit" | "workspace.transaction.rollback" + | "workspace.identity_read" | "workspace.row_lock" | "workspace.provenance" | "workspace.notification" | "workspace.write" + | "workspace.sources" | "workspace.sources.canonicalize" | "workspace.sources.inventory" | "workspace.sources.alias_scan" + | "workspace.sources.repository_scan" | "workspace.sources.project_scan" | "workspace.provenance_lock" + | "workspace.readers" | "workspace.readers.identity" | "workspace.readers.trust" | "workspace.readers.projects" | "workspace.readers.sources" + | "workspace.access" | "workspace.access.privacy" | "workspace.access.inventory" | "workspace.access.authorization" + | "runtime_selection.review" | "runtime_selection.contract_read" | "runtime_selection.contract_ensure" + | "runtime_selection.resume" | "runtime_selection.recovery_profile" | "runtime_selection.profile" | "runtime_selection.interactions" + | "runtime_selection.provider_discovery" | "runtime_selection.bootstrap" | "runtime_selection.run_write" | "runtime_selection.finalization_write" + | "runtime_selection.transaction" | "runtime_selection.transaction.entry" | "runtime_selection.transaction.commit" | "runtime_selection.transaction.rollback" + | "runtime_selection.row_lock" | "runtime_selection.workspace_sync" + | "runtime_context.instructions" | "runtime_context.skills" | "runtime_context.mcp" + | "runtime_context.asset_read" | "runtime_context.asset_manifest" | "runtime_context.asset_cache" | "runtime_context.asset_materialize" | "runtime_context.asset_verify" + | "workspace_sync.snapshot_git" | "workspace_sync.snapshot_baseline" | "workspace_sync.pack" | "workspace_sync.seed_verify" | "workspace_sync.stage_workspace" | "workspace_sync.restore" + | "execution.runtime" | "execution.workspace_ownership" | "execution.workspace_restore" | "execution.workspace_commit" + | "execution.workspace_ownership_wait" | "execution.realtime" + | "execution.usage" | "execution.native_finalize" + | "completion.run" | "completion.output" | "completion.terminal_write" | "completion.cost" | "completion.liveness" | "completion.summary" + | "response.transaction" | "response.transaction.entry" | "response.transaction.commit" | "response.transaction.rollback" | "response.row_lock" | "response.write" + | "response.resolve" | "response.authorize" | "response.add_comment" | "response.activity" | "response.publish"; + +type RunScope = { runId: string; parent?: Context; ended: boolean; details: Map }; +const ownership = new AsyncLocalStorage(); +const endpointEnabled = () => Boolean(process.env.OTEL_EXPORTER_OTLP_ENDPOINT?.trim()); + +export function getRunExecutionTraceContext(runId: string): Context | undefined { + const owned = ownership.getStore(); + return endpointEnabled() && owned?.runId === runId && !owned.ended ? owned.parent : undefined; +} + +function measure(runId: string, name: string, work: () => T | Promise, independent = false, counts: { fileCount?: number; bytes?: number } = {}): Promise { + let operation: Promise | undefined; + const run = (span?: { setStatus(status: { code: number }): void; end(): void }, parent?: Context) => { + if (operation) return operation; + const previous = ownership.getStore(); + const owned: RunScope = { runId, + parent: parent ?? (!independent && previous?.runId === runId && !previous.ended ? previous.parent : undefined), + ended: false, + details: !independent && previous?.runId === runId ? previous.details : new Map(), + }; + operation = Promise.resolve().then(() => + ownership.run(owned, () => runWithRuntimeParent(owned.parent, work))).then(value => { + try { span?.end(); } catch {} + owned.ended = true; + return value; + }, error => { + try { span?.setStatus({ code: SpanStatusCode.ERROR }); } catch {} + try { span?.end(); } catch {} + owned.ended = true; + throw error; + }); + return operation; + }; + if (!endpointEnabled()) return (async () => work())(); + try { + const parent = independent ? undefined : getRunExecutionTraceContext(runId); + const trigger = parent ? undefined : trace.getActiveSpan()?.spanContext(); + const traced = trace.getTracer(name.startsWith("heartbeat.prepare.") ? "paperclip.heartbeat-preparation" + : name.startsWith("native.controller.") ? "paperclip.native-controller" : "paperclip.execution").startActiveSpan(name, { + attributes: { [EXECUTION_PREPARATION_SPAN_ATTRS.runId]: createHash("sha256").update(runId).digest("hex").slice(0, 12), + ...Object.fromEntries((["fileCount", "bytes"] as const).flatMap(key => + typeof counts[key] === "number" && Number.isSafeInteger(counts[key]) && counts[key]! >= 0 + ? [[EXECUTION_PREPARATION_SPAN_ATTRS[key], counts[key]!]] : [])), + }, + ...(trigger && isSpanContextValid(trigger) ? { links: [{ context: trigger }] } : {}), + }, parent ?? ROOT_CONTEXT, span => { + let childContext: Context | undefined; + try { childContext = trace.setSpan(parent ?? ROOT_CONTEXT, span); } catch {} + return run(span, childContext); + }); + void Promise.resolve(traced).catch(() => undefined); + } catch {} + return operation ?? run(); +} + +export function withRunExecutionTiming(runId: string, work: () => T | Promise): Promise { + return measure(runId, "task.execution_attempt", work, true); +} + +export async function measureRunPhase(phase: ExecutionPhase, work: () => T | Promise, counts?: { fileCount?: number; bytes?: number }): Promise { + const owned = ownership.getStore(); + if (!endpointEnabled() || !owned || owned.ended) return work(); + const count = owned.details.get(phase) ?? 0; + if (count >= 16) return work(); + owned.details.set(phase, count + 1); + return measure(owned.runId, phase, work, false, counts); +} + +export function beginRunPhase(phase: ExecutionPhase): (failed?: boolean) => void { + const owned = ownership.getStore(); + if (!endpointEnabled() || !owned?.parent || owned.ended) return () => {}; + const count = owned.details.get(phase) ?? 0; + if (count >= 16) return () => {}; + owned.details.set(phase, count + 1); + try { + const span = trace.getTracer("paperclip.execution").startSpan(phase, { + attributes: { [EXECUTION_PREPARATION_SPAN_ATTRS.runId]: createHash("sha256").update(owned.runId).digest("hex").slice(0, 12) }, + }, owned.parent); + let ended = false; + return failed => { + if (ended) return; + ended = true; + if (failed) { try { span.setStatus({ code: SpanStatusCode.ERROR }); } catch {} } + try { span.end(); } catch {} + }; + } catch { return () => {}; } +} + export function createRunPreparationTiming(runId: string, scope: "heartbeat.prepare" | "native.controller" = "heartbeat.prepare") { - let runIdHash: string | undefined; - return (phase: PreparationPhase, work: () => T | Promise): Promise => { - let operation: Promise | undefined; - const run = (span?: { setStatus(status: { code: number }): void; end(): void }) => { - operation ??= (async () => { - try { return await work(); } - catch (error) { - try { span?.setStatus({ code: SpanStatusCode.ERROR }); } catch {} - throw error; - } finally { - try { span?.end(); } catch {} - } - })(); - return operation; - }; - if (!process.env.OTEL_EXPORTER_OTLP_ENDPOINT?.trim()) return run(); - try { - runIdHash ??= createHash("sha256").update(runId).digest("hex").slice(0, 12); - const trigger = trace.getActiveSpan()?.spanContext(); - const traced = trace.getTracer(scope === "heartbeat.prepare" ? "paperclip.heartbeat-preparation" : "paperclip.native-controller").startActiveSpan( - `${scope}.${phase}`, - { - attributes: { "paperclip.task.run.run_id": runIdHash }, - ...(trigger && isSpanContextValid(trigger) ? { links: [{ context: trigger }] } : {}), - }, - ROOT_CONTEXT, - span => run(span), - ); - void Promise.resolve(traced).catch(() => undefined); - } catch {} - return operation ?? run(); + return (phase: PreparationPhase, work: () => T | Promise): Promise => measure(runId, `${scope}.${phase}`, work); +} + +export function measureWorkspaceSyncPhase(name: string, work: () => Promise): Promise { + const phases: Readonly> = { + "snapshot.git": "workspace_sync.snapshot_git", "snapshot.baseline": "workspace_sync.snapshot_baseline", + pack: "workspace_sync.pack", "seed.verify": "workspace_sync.seed_verify", + "stage.workspace": "workspace_sync.stage_workspace", "restore.workspace": "workspace_sync.restore", + "workspace.restore": "execution.workspace_restore", }; + return Object.prototype.hasOwnProperty.call(phases, name) ? measureRunPhase(phases[name]!, work) : work(); } diff --git a/server/src/services/execution-timing-context.test.ts b/server/src/services/execution-timing-context.test.ts new file mode 100644 index 00000000000..864b07ccf43 --- /dev/null +++ b/server/src/services/execution-timing-context.test.ts @@ -0,0 +1,192 @@ +import { getActiveStepContext } from "@paperclipai/adapter-utils/acpx-engine/startup-timing"; +import { createHash } from "node:crypto"; +import { context, trace, TraceFlags } from "@opentelemetry/api"; +import { AsyncLocalStorageContextManager } from "@opentelemetry/context-async-hooks"; +import { InMemorySpanExporter, SimpleSpanProcessor, TracerProvider } from "@opentelemetry/sdk-trace"; +import { afterEach, beforeEach, expect, it, vi } from "vitest"; +import { getStartupTraceContext, getStartupTracer } from "../instrumentation.js"; +import { prepareVerifiedRemoteProviderPack } from "./native-runtime/remote-provider-pack.js"; +import { createNativeRunTrace } from "./native-runtime/native-run-trace.js"; +import { beginRunPhase, createRunPreparationTiming, getRunExecutionTraceContext, measureRunPhase, withRunExecutionTiming } from "./execution-preparation-timing.js"; + +let exporter: InMemorySpanExporter; +let provider: TracerProvider; +let manager: AsyncLocalStorageContextManager; +beforeEach(() => { + vi.stubEnv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://collector.test"); + exporter = new InMemorySpanExporter(); + provider = new TracerProvider({ spanProcessors: [new SimpleSpanProcessor({ exporter })] }); + manager = new AsyncLocalStorageContextManager().enable(); + expect(context.setGlobalContextManager(manager)).toBe(true); + expect(trace.setGlobalTracerProvider(provider)).toBe(true); +}); +afterEach(async () => { + vi.restoreAllMocks(); + vi.unstubAllEnvs(); + trace.disable(); + context.disable(); + manager.disable(); + await provider.shutdown(); +}); + +it("parents overlapping preparation and response work to their own execution attempts independently of ingress sampling", async () => { + const trigger = { traceId: "1".repeat(32), spanId: "2".repeat(16), traceFlags: TraceFlags.NONE, isRemote: true }; + await context.with(trace.setSpanContext(context.active(), trigger), () => Promise.all(["one", "two"].map(runId => + withRunExecutionTiming(runId, async () => { + await createRunPreparationTiming(runId)("workspace", () => Promise.all([ + measureRunPhase("workspace.sources", async () => { + await Promise.resolve(); + await measureRunPhase("workspace.sources.alias_scan", () => { + const parent = getActiveStepContext()?.parentContext; + expect(parent).toBeDefined(); + expect(trace.getSpan(parent as import("@opentelemetry/api").Context)?.spanContext()).toEqual(trace.getActiveSpan()?.spanContext()); + expect(getRunExecutionTraceContext(runId === "one" ? "two" : "one")).toBeUndefined(); + }); + }), + measureRunPhase("workspace.readers", async () => { await Promise.resolve(); }), + ])); + await measureRunPhase("response.add_comment", () => undefined); + }), + ))); + const spans = exporter.getFinishedSpans(); + const attempts = spans.filter(span => span.name === "task.execution_attempt"); + expect(attempts).toHaveLength(2); + expect(new Set(attempts.map(span => span.spanContext().traceId)).size).toBe(2); + for (const runId of ["one", "two"]) { + const attempt = attempts.find(span => span.attributes["paperclip.task.run.run_id"] === createHash("sha256").update(runId).digest("hex").slice(0, 12))!; + expect(attempt.parentSpanContext).toBeUndefined(); + expect(attempt.links[0].context).toEqual(trigger); + const owned = spans.filter(span => span.spanContext().traceId === attempt.spanContext().traceId); + const phase = owned.find(span => span.name === "heartbeat.prepare.workspace")!; + const source = owned.find(span => span.name === "workspace.sources")!; + const aliases = owned.find(span => span.name === "workspace.sources.alias_scan")!; + const readers = owned.find(span => span.name === "workspace.readers")!; + const reply = owned.find(span => span.name === "response.add_comment")!; + expect(phase.parentSpanContext?.spanId).toBe(attempt.spanContext().spanId); + expect(source.parentSpanContext?.spanId).toBe(phase.spanContext().spanId); + expect(readers.parentSpanContext?.spanId).toBe(phase.spanContext().spanId); + expect(aliases.parentSpanContext?.spanId).toBe(source.spanContext().spanId); + expect(reply.parentSpanContext?.spanId).toBe(attempt.spanContext().spanId); + const ns = (time: readonly [number, number]) => time[0] * 1e9 + time[1]; + expect(ns(readers.startTime)).toBeLessThanOrEqual(ns(source.endTime)); + expect(ns(source.startTime)).toBeLessThanOrEqual(ns(readers.endTime)); + expect(owned).toHaveLength(6); + } + expect(getRunExecutionTraceContext("one")).toBeUndefined(); +}); + +it("uses a link for a native session whose queue history predates the current attempt", async () => { + await withRunExecutionTiming("native", async () => { + const native = createNativeRunTrace({ runId: "native", startedAtMs: Date.now() - 10_000 }); + await native.measure("agent.turn", async () => {}); + await native.finish("ok"); + await measureRunPhase("response.add_comment", () => undefined); + }); + const spans = exporter.getFinishedSpans(); + const attempt = spans.find(span => span.name === "task.execution_attempt")!; + const session = spans.find(span => span.name === "task.provider_session")!; + const response = spans.find(span => span.name === "response.add_comment")!; + expect(session.parentSpanContext).toBeUndefined(); + expect(session.links.map(link => link.context.spanId)).toContain(attempt.spanContext().spanId); + expect(response.parentSpanContext?.spanId).toBe(attempt.spanContext().spanId); + expect(session.startTime[0]).toBeLessThan(attempt.startTime[0]); +}); + +it("caps repeated detail spans and ends failed scopes without content or exception text", async () => { + const failure = new Error("private path and credential"); + const work = vi.fn(() => undefined); + await expect(withRunExecutionTiming("bounded", async () => { + for (let index = 0; index < 100; index++) await measureRunPhase("workspace.readers.identity", work); + for (let index = 0; index < 100; index++) beginRunPhase("workspace.transaction.entry")(); + await measureRunPhase("workspace.readers.trust", () => { throw failure; }); + })).rejects.toBe(failure); + expect(work).toHaveBeenCalledTimes(100); + const spans = exporter.getFinishedSpans(); + expect(spans.filter(span => span.name === "workspace.readers.identity")).toHaveLength(16); + expect(spans.filter(span => span.name === "workspace.transaction.entry")).toHaveLength(16); + for (const span of spans) expect(Object.keys(span.attributes)).toEqual(["paperclip.task.run.run_id"]); + expect(spans.filter(span => span.status.code === 2).map(span => span.name).sort()).toEqual(["task.execution_attempt", "workspace.readers.trust"]); + expect(JSON.stringify(spans.map(span => ({ attributes: span.attributes, events: span.events, status: span.status })))).not.toContain(failure.message); + expect(getRunExecutionTraceContext("bounded")).toBeUndefined(); +}); + +it.each([undefined, "", " "])("does not consult tracers or warn without endpoint %s even with a registered provider", async endpoint => { + vi.stubEnv("OTEL_EXPORTER_OTLP_ENDPOINT", endpoint); + const lookup = vi.spyOn(trace, "getTracer"); + const warnings = (["log", "info", "warn", "error", "debug"] as const).map(method => vi.spyOn(console, method)); + await expect(withRunExecutionTiming("off", () => measureRunPhase("response.add_comment", () => 42))).resolves.toBe(42); + getStartupTracer().startSpan("disabled").end(); + getStartupTraceContext().tracer.startSpan("disabled").end(); + const native = createNativeRunTrace({ runId: "off" }); + await native.measure("agent.turn", async () => undefined); + await native.finish("ok"); + expect(lookup).not.toHaveBeenCalled(); + expect(exporter.getFinishedSpans()).toEqual([]); + for (const warning of warnings) expect(warning).not.toHaveBeenCalled(); +}); + +it.each(["staged", "preinstalled", "uploaded", "failed"] as const)("records provider-pack selection %s with closed source and fallback reasons", async source => { + const failure = new Error("private manifest path and hash"); + const native = createNativeRunTrace({ runId: "pack" }); + const select = prepareVerifiedRemoteProviderPack({ + trace: native, + verifyStaged: async () => { if (source !== "staged") throw failure; }, + usePreinstalled: async () => source === "preinstalled", + stageAndVerify: async () => { if (source === "failed") throw failure; }, + }); + if (source === "failed") await expect(select).rejects.toBe(failure); + else await expect(select).resolves.toBe(source); + await native.finish(source === "failed" ? "failed" : "ok"); + const span = exporter.getFinishedSpans().find(span => span.name === "provider_pack.select")!; + expect(span.status.code).toBe(source === "failed" ? 2 : 0); + if (source !== "failed") expect(span.attributes["paperclip.native.span.stateSource"]).toBe(source); + expect(JSON.stringify({ attributes: span.attributes, events: span.events })).not.toContain(failure.message); +}); + +it("exports only the declared count attributes and rejects unsafe numbers or extra fields", async () => { + await withRunExecutionTiming("counts", async () => { + await measureRunPhase("runtime_context.asset_materialize", () => undefined, + { fileCount: 3, bytes: 99, privatePath: "/private/fixture" } as { fileCount: number; bytes: number }); + await measureRunPhase("runtime_context.asset_verify", () => undefined, { fileCount: -1, bytes: Infinity }); + }); + const spans = exporter.getFinishedSpans(); + const materialize = spans.find(span => span.name === "runtime_context.asset_materialize")!; + expect(materialize.attributes).toEqual({ + "paperclip.task.run.run_id": createHash("sha256").update("counts").digest("hex").slice(0, 12), + "paperclip.execution.fileCount": 3, + "paperclip.execution.bytes": 99, + }); + const verify = spans.find(span => span.name === "runtime_context.asset_verify")!; + expect(Object.keys(verify.attributes)).toEqual(["paperclip.task.run.run_id"]); +}); + +it("does not inherit another run when a new attempt's tracer lookup fails", async () => { + await withRunExecutionTiming("outer", async () => { + const lookup = vi.spyOn(trace, "getTracer").mockImplementationOnce(() => { throw new Error("tracer unavailable"); }); + try { + await withRunExecutionTiming("inner", () => measureRunPhase("workspace.sources", () => 42)); + } finally { lookup.mockRestore(); } + }); + const spans = exporter.getFinishedSpans(); + const outer = spans.find(span => span.name === "task.execution_attempt")!; + const inner = spans.find(span => span.name === "workspace.sources")!; + expect(inner.parentSpanContext).toBeUndefined(); + expect(inner.spanContext().traceId).not.toBe(outer.spanContext().traceId); + expect(inner.attributes["paperclip.task.run.run_id"]).toBe(createHash("sha256").update("inner").digest("hex").slice(0, 12)); +}); + +it("does not attach native children to an ambient run when native context construction fails", async () => { + await withRunExecutionTiming("ambient", async () => { + const native = createNativeRunTrace({ runId: "native-context-failure", traceContext: { + tracer: getStartupTracer("broken-native-context"), + contextWithSpan() { throw new Error("context unavailable"); }, + } }); + await native.measure("agent.turn", async () => undefined); + await native.finish("ok"); + }); + const spans = exporter.getFinishedSpans(); + const ambient = spans.find(span => span.name === "task.execution_attempt")!; + const child = spans.find(span => span.name === "agent.turn")!; + expect(child.parentSpanContext).toBeUndefined(); + expect(child.spanContext().traceId).not.toBe(ambient.spanContext().traceId); +}); diff --git a/server/src/services/execution-transaction-timing.test.ts b/server/src/services/execution-transaction-timing.test.ts new file mode 100644 index 00000000000..a7649e37ec6 --- /dev/null +++ b/server/src/services/execution-transaction-timing.test.ts @@ -0,0 +1,127 @@ +import { context, trace } from "@opentelemetry/api"; +import { AsyncLocalStorageContextManager } from "@opentelemetry/context-async-hooks"; +import { InMemorySpanExporter, SimpleSpanProcessor, TracerProvider } from "@opentelemetry/sdk-trace"; +import { companies, createDb, executionWorkspaces, issueComments, issues, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS } from "@paperclipai/db"; +import { eq, sql } from "drizzle-orm"; +import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../__tests__/helpers/embedded-postgres.js"; +import { measureRunTransaction } from "./execution-transaction-timing.js"; +import { withRunExecutionTiming } from "./execution-preparation-timing.js"; +import { issueService } from "./issues.js"; +import { logActivity } from "./activity-log.js"; +import { executionWorkspaceService } from "./execution-workspaces.js"; +import { lockExecutionWorkspaceRootProvenance } from "./execution-workspace-repositories.js"; +import { assertTaskWorkspaceAccess } from "./task-workspace-source-access.js"; + +const support = await getEmbeddedPostgresTestSupport(); +(support.supported ? describe : describe.skip)("execution timing with real database waits", () => { + let temporary: Awaited>; + let db: ReturnType; + let companyId: string, workspaceId: string; + let exporter: InMemorySpanExporter, provider: TracerProvider, manager: AsyncLocalStorageContextManager; + beforeAll(async () => { + temporary = await startEmbeddedPostgresTestDatabase("paperclip-execution-timing-"); + db = createDb(temporary.connectionString); + const [company] = await db.insert(companies).values({ name: "Timing fixture", issuePrefix: "TIME" }).returning(); + companyId = company.id; + const [workspace] = await db.insert(executionWorkspaces).values({ companyId, name: "Files", mode: "shared_workspace", strategyType: "task_directory" }).returning(); + workspaceId = workspace.id; + }, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS); + beforeEach(() => { + vi.stubEnv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://collector.test"); + exporter = new InMemorySpanExporter(); + provider = new TracerProvider({ spanProcessors: [new SimpleSpanProcessor({ exporter })] }); + manager = new AsyncLocalStorageContextManager().enable(); + expect(context.setGlobalContextManager(manager)).toBe(true); + expect(trace.setGlobalTracerProvider(provider)).toBe(true); + }); + afterEach(async () => { + vi.unstubAllEnvs(); + trace.disable(); context.disable(); manager.disable(); + await provider.shutdown(); + }); + afterAll(async () => { await temporary?.cleanup(); }); + + it.each(["advisory", "row"] as const)("attributes a controlled %s lock wait to its leaf and commits the original update", async kind => { + let release!: () => void, acquired!: () => void; + const released = new Promise(resolve => { release = resolve; }); + const ready = new Promise(resolve => { acquired = resolve; }); + const blocker = db.transaction(async tx => { + if (kind === "advisory") await lockExecutionWorkspaceRootProvenance(tx, companyId); + else await tx.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspaceId)).for("update"); + acquired(); + await released; + }); + await ready; + const update = withRunExecutionTiming(`wait-${kind}`, () => executionWorkspaceService(db).update(workspaceId, { metadata: { verified: kind } })); + try { + await vi.waitFor(async () => { + const pending = await db.execute(sql`select count(*)::int as count from pg_locks where not granted and locktype = ${kind === "advisory" ? "advisory" : "transactionid"}`); + expect(pending[0]?.count).toBeGreaterThan(0); + }); + expect(exporter.getFinishedSpans().some(span => span.name === (kind === "advisory" ? "workspace.provenance_lock" : "workspace.row_lock"))).toBe(false); + } finally { + release(); + await blocker; + await update; + } + const [saved] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspaceId)); + expect(saved.metadata).toMatchObject({ verified: kind }); + const spans = exporter.getFinishedSpans(); + const transaction = spans.find(span => span.name === "workspace.transaction")!; + const wait = spans.find(span => span.name === (kind === "advisory" ? "workspace.provenance_lock" : "workspace.row_lock"))!; + expect(wait.parentSpanContext?.spanId).toBe(transaction.spanContext().spanId); + expect(wait.duration[0] * 1e9 + wait.duration[1]).toBeGreaterThan(0); + expect(spans.map(span => span.name)).toEqual(expect.arrayContaining(["workspace.transaction.entry", "workspace.transaction.commit", "workspace.notification", "workspace.write"])); + expect(spans.some(span => span.name === "workspace.transaction.rollback")).toBe(false); + expect(JSON.stringify(spans.map(span => span.attributes))).not.toContain(workspaceId); + }); + + it("preserves transaction rollback and the original error without replaying writes", async () => { + const [before] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspaceId)); + const failure = new Error("private rollback fixture"); + let calls = 0; + await expect(withRunExecutionTiming("rollback", () => measureRunTransaction(db, "workspace", async tx => { + calls++; + await tx.update(executionWorkspaces).set({ metadata: { rolledBack: true } }).where(eq(executionWorkspaces.id, workspaceId)); + throw failure; + }))).rejects.toBe(failure); + expect(calls).toBe(1); + const [after] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspaceId)); + expect(after.metadata).toEqual(before.metadata); + const spans = exporter.getFinishedSpans(); + expect(spans.find(span => span.name === "workspace.transaction.rollback")?.status.code).toBe(2); + expect(spans.some(span => span.name === "workspace.transaction.commit")).toBe(false); + expect(JSON.stringify(spans.map(span => ({ attributes: span.attributes, events: span.events })))).not.toContain(failure.message); + }); + + it("records durable response commit and server publication under the same attempt", async () => { + const [issue] = await db.insert(issues).values({ companyId, title: "Response timing" }).returning(); + const body = "Private response fixture"; + const comment = await withRunExecutionTiming("response", async () => { + const saved = await issueService(db).addComment(issue.id, body, { userId: "local-board" }); + await logActivity(db, { companyId, actorType: "user", actorId: "local-board", action: "issue.comment_added", entityType: "issue", entityId: issue.id }); + return saved; + }); + const [stored] = await db.select().from(issueComments).where(eq(issueComments.id, comment.id)); + expect(stored.body).toBe(body); + const spans = exporter.getFinishedSpans(); + const attempt = spans.find(span => span.name === "task.execution_attempt")!; + const transaction = spans.find(span => span.name === "response.transaction")!; + const commit = spans.find(span => span.name === "response.transaction.commit")!; + expect(transaction.parentSpanContext?.spanId).toBe(attempt.spanContext().spanId); + expect(commit.parentSpanContext?.spanId).toBe(transaction.spanContext().spanId); + expect(spans.map(span => span.name)).toEqual(expect.arrayContaining(["response.row_lock", "response.write", "execution.realtime"])); + expect(spans.every(span => span.spanContext().traceId === attempt.spanContext().traceId)).toBe(true); + expect(JSON.stringify(spans.map(span => span.attributes))).not.toContain(body); + }); + + it("preserves a real access denial and exports only failed status", async () => { + await expect(withRunExecutionTiming("denied", () => assertTaskWorkspaceAccess(db, + { type: "board", userId: "not-a-company-member" }, companyId, workspaceId, + ))).rejects.toMatchObject({ status: 403 }); + const spans = exporter.getFinishedSpans(); + expect(spans.find(span => span.name === "task.execution_attempt")?.status.code).toBe(2); + expect(JSON.stringify(spans.map(span => ({ attributes: span.attributes, events: span.events })))).not.toContain("not-a-company-member"); + }); +}); diff --git a/server/src/services/execution-transaction-timing.ts b/server/src/services/execution-transaction-timing.ts new file mode 100644 index 00000000000..bb22c861be0 --- /dev/null +++ b/server/src/services/execution-transaction-timing.ts @@ -0,0 +1,31 @@ +import type { Db } from "@paperclipai/db"; +import { beginRunPhase, measureRunPhase } from "./execution-preparation-timing.js"; + +type Transaction = Parameters[0]>[0]; + +export function measureRunTransaction(db: Db, scope: "workspace" | "runtime_selection" | "response", work: (tx: Transaction) => Promise): Promise { + return measureRunPhase(`${scope}.transaction`, async () => { + const entry = beginRunPhase(`${scope}.transaction.entry`); + let settle: ((failed?: boolean) => void) | undefined; + let failed = false; + try { + return await db.transaction(async tx => { + entry(); + try { + const value = await work(tx); + settle = beginRunPhase(`${scope}.transaction.commit`); + return value; + } catch (error) { + settle = beginRunPhase(`${scope}.transaction.rollback`); + throw error; + } + }); + } catch (error) { + failed = true; + throw error; + } finally { + entry(failed); + settle?.(failed); + } + }); +} diff --git a/server/src/services/execution-workspace-repositories.ts b/server/src/services/execution-workspace-repositories.ts index 47d94e6e80e..60acd1f81d8 100644 --- a/server/src/services/execution-workspace-repositories.ts +++ b/server/src/services/execution-workspace-repositories.ts @@ -1,3 +1,5 @@ +import { measureRunTransaction } from "./execution-transaction-timing.js"; +import { measureRunPhase } from "./execution-preparation-timing.js"; import { createHash, randomUUID } from "node:crypto"; import { execFile as execFileCallback } from "node:child_process"; import { isDeepStrictEqual, promisify } from "node:util"; @@ -65,10 +67,10 @@ export function assertRepositoryRetryMatches(row: Pick; }) { - const canonicalCwd = await canonicalRepositoryRoot(input.cwd); + const canonicalCwd = await measureRunPhase("workspace.sources.canonicalize", () => canonicalRepositoryRoot(input.cwd)); const rootKey = createHash("sha256").update(canonicalCwd).digest("hex"); - const companyEntries = await tx.select().from(executionWorkspaceRepositoryAccess) - .where(eq(executionWorkspaceRepositoryAccess.companyId, input.companyId)); + const companyEntries = await measureRunPhase("workspace.sources.inventory", async () => tx.select().from(executionWorkspaceRepositoryAccess) + .where(eq(executionWorkspaceRepositoryAccess.companyId, input.companyId))); const entries = companyEntries.filter(entry => entry.rootKey === rootKey); const overlappingEntries = companyEntries.filter(entry => repositoryRootsOverlap(entry.canonicalCwd, canonicalCwd)); if (entries.some(entry => entry.canonicalCwd !== canonicalCwd) @@ -94,22 +96,24 @@ async function resolveRepositoryRootSources(tx: Db | DbTransaction, input: { const isolatedSourceIds = new Set(overlappingEntries.filter(entry => entry.kind === "isolated_source").map(entry => entry.scopeId)); const sourceIds = new Set([...sharedSourceIds, ...isolatedSourceIds]); if (input.sourceWorkspaceId) { sourceIds.add(input.sourceWorkspaceId); sharedSourceIds.add(input.sourceWorkspaceId); addSharedSource(input.sourceWorkspaceId, canonicalCwd); } - const workspaces = await tx.select({ id: executionWorkspaces.id, cwd: executionWorkspaces.cwd }).from(executionWorkspaces) - .where(eq(executionWorkspaces.companyId, input.companyId)); - for (const workspace of workspaces) { - if (!workspace.cwd) continue; - const peerRoot = await canonicalRepositoryRoot(workspace.cwd).catch((error: NodeJS.ErrnoException) => { - if (["ENOTDIR", "EACCES", "EPERM", "ELOOP", "ENAMETOOLONG"].includes(error.code ?? "")) return null; - throw error; - }); - if (peerRoot && repositoryRootsOverlap(peerRoot, canonicalCwd)) { - sourceIds.add(workspace.id); sharedSourceIds.add(workspace.id); addSharedSource(workspace.id, peerRoot); + const workspaces = await measureRunPhase("workspace.sources.inventory", async () => tx.select({ id: executionWorkspaces.id, cwd: executionWorkspaces.cwd }).from(executionWorkspaces) + .where(eq(executionWorkspaces.companyId, input.companyId))); + await measureRunPhase("workspace.sources.alias_scan", async () => { + for (const workspace of workspaces) { + if (!workspace.cwd) continue; + const peerRoot = await canonicalRepositoryRoot(workspace.cwd!).catch((error: NodeJS.ErrnoException) => { + if (["ENOTDIR", "EACCES", "EPERM", "ELOOP", "ENAMETOOLONG"].includes(error.code ?? "")) return null; + throw error; + }); + if (peerRoot && repositoryRootsOverlap(peerRoot, canonicalCwd)) { + sourceIds.add(workspace.id); sharedSourceIds.add(workspace.id); addSharedSource(workspace.id, peerRoot); + } } - } - const inventory = await tx.select({ id: executionWorkspaceRepositories.id, workspaceId: executionWorkspaces.id, cwd: executionWorkspaces.cwd, + }); + const inventory = await measureRunPhase("workspace.sources.inventory", async () => tx.select({ id: executionWorkspaceRepositories.id, workspaceId: executionWorkspaces.id, cwd: executionWorkspaces.cwd, identity: executionWorkspaceRepositories.repositoryIdentity }).from(executionWorkspaceRepositories) .innerJoin(executionWorkspaces, eq(executionWorkspaces.id, executionWorkspaceRepositories.executionWorkspaceId)) - .where(and(eq(executionWorkspaceRepositories.companyId, input.companyId), eq(executionWorkspaces.companyId, input.companyId))); + .where(and(eq(executionWorkspaceRepositories.companyId, input.companyId), eq(executionWorkspaces.companyId, input.companyId)))); const namespace = path.join(canonicalCwd, ".paperclip-repositories"); const namespaceEntry = await fs.lstat(namespace).catch((error: NodeJS.ErrnoException) => { if (error.code === "ENOENT") return null; @@ -122,37 +126,39 @@ async function resolveRepositoryRootSources(tx: Db | DbTransaction, input: { } for (const name of await fs.readdir(namespace)) if (name.startsWith("task-repo-")) publishedNames.add(name); } - const repositoryOrigins = await tx.select({ id: executionWorkspaceRepositoryAccess.scopeId }).from(executionWorkspaceRepositoryAccess) - .where(and(eq(executionWorkspaceRepositoryAccess.companyId, input.companyId), eq(executionWorkspaceRepositoryAccess.kind, "repository"))); + const repositoryOrigins = await measureRunPhase("workspace.sources.inventory", async () => tx.select({ id: executionWorkspaceRepositoryAccess.scopeId }).from(executionWorkspaceRepositoryAccess) + .where(and(eq(executionWorkspaceRepositoryAccess.companyId, input.companyId), eq(executionWorkspaceRepositoryAccess.kind, "repository")))); const knownOrigins = new Set(repositoryOrigins.map(origin => origin.id)); const recognizedNames = new Set(); - for (const peer of inventory) { - const name = path.basename(taskRepositoryRelativePath(peer.identity)); - const published = publishedNames.has(name); - const peerRoot = peer.cwd ? await fs.realpath(peer.cwd).catch((error: NodeJS.ErrnoException) => { - if (["ENOENT", "ENOTDIR", "EACCES", "EPERM", "ELOOP", "ENAMETOOLONG"].includes(error.code ?? "")) return null; - throw error; - }) : null; - const matches = sourceIds.has(peer.workspaceId) || (peerRoot !== null && repositoryRootsOverlap(peerRoot, canonicalCwd)) - || (published && peerRoot === null && !knownOrigins.has(peer.id)) - || (peerRoot === null && !entries.some(entry => entry.kind === "reader") && peer.cwd && await legacyRepositoryRootMayMatch(peer.cwd, canonicalCwd)); - if (matches) { - sourceIds.add(peer.workspaceId); - if (!isolatedSourceIds.has(peer.workspaceId) || (peerRoot && repositoryRootsOverlap(peerRoot, canonicalCwd))) { - sharedSourceIds.add(peer.workspaceId); addSharedSource(peer.workspaceId, peerRoot ?? canonicalCwd); + await measureRunPhase("workspace.sources.repository_scan", async () => { + for (const peer of inventory) { + const name = path.basename(taskRepositoryRelativePath(peer.identity)); + const published = publishedNames.has(name); + const peerRoot = peer.cwd ? await fs.realpath(peer.cwd).catch((error: NodeJS.ErrnoException) => { + if (["ENOENT", "ENOTDIR", "EACCES", "EPERM", "ELOOP", "ENAMETOOLONG"].includes(error.code ?? "")) return null; + throw error; + }) : null; + const matches = sourceIds.has(peer.workspaceId) || (peerRoot !== null && repositoryRootsOverlap(peerRoot, canonicalCwd)) + || (published && peerRoot === null && !knownOrigins.has(peer.id)) + || (peerRoot === null && !entries.some(entry => entry.kind === "reader") && peer.cwd && await legacyRepositoryRootMayMatch(peer.cwd, canonicalCwd)); + if (matches) { + sourceIds.add(peer.workspaceId); + if (!isolatedSourceIds.has(peer.workspaceId) || (peerRoot && repositoryRootsOverlap(peerRoot, canonicalCwd))) { + sharedSourceIds.add(peer.workspaceId); addSharedSource(peer.workspaceId, peerRoot ?? canonicalCwd); + } + if (published) recognizedNames.add(name); } - if (published) recognizedNames.add(name); } - } + }); if ([...publishedNames].some(name => !recognizedNames.has(name))) throw forbidden("Published repository has no retained workspace provenance"); - const configured = await tx.select({ projectId: projectWorkspaces.projectId, cwd: projectWorkspaces.cwd, repoUrl: projectWorkspaces.repoUrl }) - .from(projectWorkspaces).where(eq(projectWorkspaces.companyId, input.companyId)); + const configured = await measureRunPhase("workspace.sources.inventory", async () => tx.select({ projectId: projectWorkspaces.projectId, cwd: projectWorkspaces.cwd, repoUrl: projectWorkspaces.repoUrl }) + .from(projectWorkspaces).where(eq(projectWorkspaces.companyId, input.companyId))); const sourceProjectIds = new Set(overlappingEntries.filter(entry => (entry.kind === "project_source" || entry.kind === "isolated_project_source")).map(entry => entry.scopeId)); const sharedProjectIds = new Set(overlappingEntries.filter(entry => entry.kind === "shared_project_source" && !isCopiedAncestor(entry.scopeId, entry.canonicalCwd)).map(entry => entry.scopeId)); const isolatedProjectIds = new Set(overlappingEntries.filter(entry => entry.kind === "isolated_project_source").map(entry => entry.scopeId)); if (sourceIds.size) { - const owners = await tx.select({ id: executionWorkspaces.id, projectId: executionWorkspaces.projectId, strategyType: executionWorkspaces.strategyType }).from(executionWorkspaces).where(and( - eq(executionWorkspaces.companyId, input.companyId), or(...[...sourceIds].map(id => eq(executionWorkspaces.id, id))))); + const owners = await measureRunPhase("workspace.sources.inventory", async () => tx.select({ id: executionWorkspaces.id, projectId: executionWorkspaces.projectId, strategyType: executionWorkspaces.strategyType }).from(executionWorkspaces).where(and( + eq(executionWorkspaces.companyId, input.companyId), or(...[...sourceIds].map(id => eq(executionWorkspaces.id, id)))))); for (const owner of owners) if (owner.projectId) { sourceProjectIds.add(owner.projectId); if (owner.strategyType === "git_worktree" || !sharedSourceIds.has(owner.id) @@ -160,13 +166,15 @@ async function resolveRepositoryRootSources(tx: Db | DbTransaction, input: { else sharedProjectIds.add(owner.projectId); } } - for (const source of configured) { - if (sharedProjectIds.has(source.projectId)) continue; - const sourceRoot = await configuredWorkspaceRoot(source, true); - if (!sourceRoot || !repositoryRootsOverlap(sourceRoot, canonicalCwd)) continue; - sourceProjectIds.add(source.projectId); - if (!isCopiedAncestor(source.projectId, sourceRoot)) sharedProjectIds.add(source.projectId); - } + await measureRunPhase("workspace.sources.project_scan", async () => { + for (const source of configured) { + if (sharedProjectIds.has(source.projectId)) continue; + const sourceRoot = await configuredWorkspaceRoot(source, true); + if (!sourceRoot || !repositoryRootsOverlap(sourceRoot, canonicalCwd)) continue; + sourceProjectIds.add(source.projectId); + if (!isCopiedAncestor(source.projectId, sourceRoot)) sharedProjectIds.add(source.projectId); + } + }); for (const entry of overlappingEntries) if (entry.kind === "project_source" && !isolatedProjectIds.has(entry.scopeId)) { sharedProjectIds.add(entry.scopeId); } @@ -195,9 +203,9 @@ async function assertRootProjectsReadAccessForActors(db: Db | DbTransaction, act } async function retainedRootReader(tx: DbTransaction, companyId: string, reader: Pick) { - const [run] = await tx.select(rootReaderRunColumns).from(heartbeatRuns).where(and(eq(heartbeatRuns.id, reader.scopeId), eq(heartbeatRuns.companyId, companyId))); - const [agent] = reader.agentId ? await tx.select({ id: agents.id }).from(agents).where(and(eq(agents.id, reader.agentId), eq(agents.companyId, companyId))) : []; - const [issue] = reader.issueId ? await tx.select().from(issues).where(and(eq(issues.id, reader.issueId), eq(issues.companyId, companyId))) : []; + const [run] = await measureRunPhase("workspace.sources.inventory", async () => tx.select(rootReaderRunColumns).from(heartbeatRuns).where(and(eq(heartbeatRuns.id, reader.scopeId), eq(heartbeatRuns.companyId, companyId)))); + const [agent] = reader.agentId ? await measureRunPhase("workspace.sources.inventory", async () => tx.select({ id: agents.id }).from(agents).where(and(eq(agents.id, reader.agentId!), eq(agents.companyId, companyId)))) : []; + const [issue] = reader.issueId ? await measureRunPhase("workspace.sources.inventory", async () => tx.select().from(issues).where(and(eq(issues.id, reader.issueId!), eq(issues.companyId, companyId)))) : []; if (!run || !agent || !issue || run.agentId !== reader.agentId || run.responsibleUserId !== reader.responsibleUserId || (run.nativeIssueId ?? run.contextSnapshot?.issueId ?? run.contextSnapshot?.taskId ?? run.issueId) !== reader.issueId) { throw forbidden("Repository root retains a reader whose identity is no longer available"); @@ -208,7 +216,7 @@ async function retainedRootReader(tx: DbTransaction, companyId: string, reader: } export async function lockExecutionWorkspaceRootProvenance(tx: DbTransaction, companyId: string) { - await tx.execute(sql`select pg_advisory_xact_lock(hashtextextended(${`task-repository-roots:${companyId}`}, 0))`); + await measureRunPhase("workspace.provenance_lock", () => tx.execute(sql`select pg_advisory_xact_lock(hashtextextended(${`task-repository-roots:${companyId}`}, 0))`)); } export async function preserveConfiguredProjectWorkspaceRootProvenance(tx: DbTransaction, input: { @@ -217,12 +225,12 @@ export async function preserveConfiguredProjectWorkspaceRootProvenance(tx: DbTra const canonicalCwd = await configuredWorkspaceRoot(input, false); if (!canonicalCwd) return; const rootKey = createHash("sha256").update(canonicalCwd).digest("hex"); - const entries = await tx.select().from(executionWorkspaceRepositoryAccess).where(and( - eq(executionWorkspaceRepositoryAccess.companyId, input.companyId), eq(executionWorkspaceRepositoryAccess.rootKey, rootKey))); + const entries = await measureRunPhase("workspace.sources.inventory", async () => tx.select().from(executionWorkspaceRepositoryAccess).where(and( + eq(executionWorkspaceRepositoryAccess.companyId, input.companyId), eq(executionWorkspaceRepositoryAccess.rootKey, rootKey)))); if (entries.some(entry => entry.canonicalCwd !== canonicalCwd)) throw forbidden("Repository root identity conflicts with its retained access record"); for (const kind of ["project_source", "shared_project_source"] as const) { if (!entries.some(entry => entry.kind === kind && entry.scopeId === input.projectId)) { - await tx.insert(executionWorkspaceRepositoryAccess).values({ companyId: input.companyId, canonicalCwd, rootKey, kind, scopeId: input.projectId }); + await measureRunPhase("workspace.write", () => tx.insert(executionWorkspaceRepositoryAccess).values({ companyId: input.companyId, canonicalCwd, rootKey, kind, scopeId: input.projectId })); } } } @@ -237,13 +245,13 @@ export async function preserveExecutionWorkspaceRootProvenance(tx: DbTransaction }); if (!directory) return; if (!directory.isDirectory()) throw forbidden("Workspace files cannot be detached from unverifiable provenance"); - const { canonicalCwd, rootKey, entries, sharedSourceIds, isolatedSourceIds, sourceProjectIds, sharedProjectIds, isolatedProjectIds } = await resolveRepositoryRootSources(tx, { - companyId: input.companyId, cwd: input.cwd, sourceWorkspaceId: input.workspaceId, inspectCurrentAliases: true, - }); + const { canonicalCwd, rootKey, entries, sharedSourceIds, isolatedSourceIds, sourceProjectIds, sharedProjectIds, isolatedProjectIds } = await measureRunPhase("workspace.sources", () => resolveRepositoryRootSources(tx, { + companyId: input.companyId, cwd: input.cwd!, sourceWorkspaceId: input.workspaceId, inspectCurrentAliases: true, + })); for (const [kind, ids] of [["source", sharedSourceIds], ["isolated_source", isolatedSourceIds], ["project_source", sourceProjectIds], ["shared_project_source", sharedProjectIds], ["isolated_project_source", isolatedProjectIds]] as const) { for (const scopeId of ids) { if (!entries.some(entry => entry.kind === kind && entry.scopeId === scopeId)) { - await tx.insert(executionWorkspaceRepositoryAccess).values({ companyId: input.companyId, canonicalCwd, rootKey, kind, scopeId }); + await measureRunPhase("workspace.write", () => tx.insert(executionWorkspaceRepositoryAccess).values({ companyId: input.companyId, canonicalCwd, rootKey, kind, scopeId })); } } } @@ -252,9 +260,9 @@ export async function preserveExecutionWorkspaceRootProvenance(tx: DbTransaction export async function assertExecutionWorkspaceRootReadAccess(db: Db | DbTransaction, input: { companyId: string; cwd: string; actor: AuthorizationActor; sourceWorkspaceId?: string; }): Promise { - const { sourceIds, sourceProjectIds } = await resolveRepositoryRootSources(db, { ...input, inspectCurrentAliases: true }); - await assertTaskWorkspacesReadAccess(db, input.actor, input.companyId, sourceIds); - await assertRootProjectAccess(db, input.actor, input.companyId, sourceProjectIds); + const { sourceIds, sourceProjectIds } = await measureRunPhase("workspace.sources", () => resolveRepositoryRootSources(db, { ...input, inspectCurrentAliases: true })); + await measureRunPhase("workspace.readers.sources", () => assertTaskWorkspacesReadAccess(db, input.actor, input.companyId, sourceIds)); + await measureRunPhase("workspace.readers.projects", () => assertRootProjectAccess(db, input.actor, input.companyId, sourceProjectIds)); } export type IsolatedWorkspaceSourceAccess = { @@ -267,20 +275,20 @@ export type IsolatedWorkspaceSourceAccess = { async function retainGitMetadataOwners(tx: DbTransaction, input: { companyId: string; commonRoot: string; workspaceIds: Set; projectIds: Set; }): Promise { - const closure = await resolveRepositoryRootSources(tx, { companyId: input.companyId, cwd: input.commonRoot, inspectCurrentAliases: true }); + const closure = await measureRunPhase("workspace.sources", () => resolveRepositoryRootSources(tx, { companyId: input.companyId, cwd: input.commonRoot, inspectCurrentAliases: true })); const workspaceIds = new Set([...closure.sourceIds, ...input.workspaceIds]); const projectIds = new Set([...closure.sourceProjectIds, ...input.projectIds]); const readerActors: AuthorizationActor[] = []; for (const reader of closure.overlappingEntries.filter(entry => entry.kind === "reader")) { - const { actor } = await retainedRootReader(tx, input.companyId, reader); + const { actor } = await measureRunPhase("workspace.readers.identity", () => retainedRootReader(tx, input.companyId, reader)); readerActors.push(actor); } - await assertRootProjectsReadAccessForActors(tx, readerActors, input.companyId, projectIds); - await assertTaskWorkspacesReadAccessForActors(tx, readerActors, input.companyId, workspaceIds); + await measureRunPhase("workspace.readers.projects", () => assertRootProjectsReadAccessForActors(tx, readerActors, input.companyId, projectIds)); + await measureRunPhase("workspace.readers.sources", () => assertTaskWorkspacesReadAccessForActors(tx, readerActors, input.companyId, workspaceIds)); for (const [kind, ids] of [["source", workspaceIds], ["project_source", projectIds]] as const) { for (const scopeId of ids) if (!closure.entries.some(entry => entry.kind === kind && entry.scopeId === scopeId)) { - await tx.insert(executionWorkspaceRepositoryAccess).values({ companyId: input.companyId, - canonicalCwd: closure.canonicalCwd, rootKey: closure.rootKey, kind, scopeId }); + await measureRunPhase("workspace.write", () => tx.insert(executionWorkspaceRepositoryAccess).values({ companyId: input.companyId, + canonicalCwd: closure.canonicalCwd, rootKey: closure.rootKey, kind, scopeId })); } } } @@ -289,14 +297,14 @@ export async function assertIsolatedWorkspaceSourceReadAccess(db: Db, input: { companyId: string; sourceCwd: string; actor: AuthorizationActor; copyKind?: WorkspaceSourceCopyKind; }): Promise { const { roots, commonRoot } = await isolatedWorkspaceSourceRoots(db, input.companyId, input.sourceCwd, input.copyKind); - return db.transaction(async tx => { + return measureRunTransaction(db, "workspace", async tx => { await lockExecutionWorkspaceRootProvenance(tx, input.companyId); const workspaceIds = new Set(), projectIds = new Set(); for (const cwd of roots) { - const closure = await resolveRepositoryRootSources(tx, { companyId: input.companyId, cwd, inspectCurrentAliases: true }); - await assertTaskWorkspacesReadAccess(tx, input.actor, input.companyId, closure.sourceIds); + const closure = await measureRunPhase("workspace.sources", () => resolveRepositoryRootSources(tx, { companyId: input.companyId, cwd, inspectCurrentAliases: true })); + await measureRunPhase("workspace.readers.sources", () => assertTaskWorkspacesReadAccess(tx, input.actor, input.companyId, closure.sourceIds)); for (const id of closure.sourceIds) workspaceIds.add(id); - await assertRootProjectAccess(tx, input.actor, input.companyId, closure.sourceProjectIds); + await measureRunPhase("workspace.readers.projects", () => assertRootProjectAccess(tx, input.actor!, input.companyId, closure.sourceProjectIds)); for (const id of closure.sourceProjectIds) projectIds.add(id); } await retainGitMetadataOwners(tx, { companyId: input.companyId, commonRoot, workspaceIds, projectIds }); @@ -313,31 +321,31 @@ export async function retainIsolatedWorkspaceSourceAccess(db: Db, input: { const roots = new Set(input.captured.roots); try { for (const cwd of (await isolatedWorkspaceSourceRoots(db, input.companyId, input.sourceCwd, input.copyKind)).roots) roots.add(cwd); } catch (error) { sourceFailure = error; } - const canonicalCwd = await canonicalRepositoryRoot(input.targetCwd); + const canonicalCwd = await measureRunPhase("workspace.sources.canonicalize", () => canonicalRepositoryRoot(input.targetCwd)); const rootKey = createHash("sha256").update(canonicalCwd).digest("hex"); - const denied = await db.transaction(async tx => { + const denied = await measureRunTransaction(db, "workspace", async tx => { await lockExecutionWorkspaceRootProvenance(tx, input.companyId); - const existing = await tx.select().from(executionWorkspaceRepositoryAccess).where(and( - eq(executionWorkspaceRepositoryAccess.companyId, input.companyId), eq(executionWorkspaceRepositoryAccess.rootKey, rootKey))); + const existing = await measureRunPhase("workspace.sources.inventory", async () => tx.select().from(executionWorkspaceRepositoryAccess).where(and( + eq(executionWorkspaceRepositoryAccess.companyId, input.companyId), eq(executionWorkspaceRepositoryAccess.rootKey, rootKey)))); if (existing.some(entry => entry.canonicalCwd !== canonicalCwd)) throw forbidden("Repository root identity conflicts with its retained access record"); const workspaceIds = new Set(input.captured.workspaceIds), projectIds = new Set(input.captured.projectIds); let failure = sourceFailure; for (const cwd of roots) { try { - const closure = await resolveRepositoryRootSources(tx, { companyId: input.companyId, cwd, inspectCurrentAliases: true }); + const closure = await measureRunPhase("workspace.sources", () => resolveRepositoryRootSources(tx, { companyId: input.companyId, cwd, inspectCurrentAliases: true })); for (const id of closure.sourceIds) workspaceIds.add(id); for (const id of closure.sourceProjectIds) projectIds.add(id); } catch (error) { failure ??= error; } } - try { await assertTaskWorkspacesReadAccess(tx, input.actor, input.companyId, workspaceIds); } + try { await measureRunPhase("workspace.readers.sources", () => assertTaskWorkspacesReadAccess(tx, input.actor, input.companyId, workspaceIds)); } catch (error) { failure ??= error; } for (const id of projectIds) { - try { await assertRootProjectAccess(tx, input.actor, input.companyId, new Set([id])); } + try { await measureRunPhase("workspace.readers.projects", () => assertRootProjectAccess(tx, input.actor!, input.companyId, new Set([id]))); } catch (error) { failure ??= error; } } for (const [kind, ids] of [["isolated_source", workspaceIds], ["isolated_project_source", projectIds]] as const) { for (const scopeId of ids) if (!existing.some(entry => entry.kind === kind && entry.scopeId === scopeId)) { - await tx.insert(executionWorkspaceRepositoryAccess).values({ companyId: input.companyId, rootKey, canonicalCwd, kind, scopeId }); + await measureRunPhase("workspace.write", () => tx.insert(executionWorkspaceRepositoryAccess).values({ companyId: input.companyId, rootKey, canonicalCwd, kind, scopeId })); } } const phase = input.phase ?? "preparing"; @@ -360,23 +368,23 @@ export async function retainIsolatedWorkspaceSourceAccess(db: Db, input: { } } catch (error) { failure ??= error; } try { - const target = await resolveRepositoryRootSources(tx, { companyId: input.companyId, cwd: canonicalCwd, preparingCopyProjectIds }); - await assertRootProjectTrust(tx, { companyId: input.companyId, actor: input.actor, issueId: input.issueId, projectIds: target.sourceProjectIds }); - await assertTaskWorkspacesReadAccess(tx, input.actor, input.companyId, target.sourceIds); - await assertRootProjectAccess(tx, input.actor, input.companyId, target.sourceProjectIds, - { sharedProjectIds: target.sharedProjectIds, issueId: input.issueId }); + const target = await measureRunPhase("workspace.sources", () => resolveRepositoryRootSources(tx, { companyId: input.companyId, cwd: canonicalCwd, preparingCopyProjectIds })); + await measureRunPhase("workspace.readers.trust", () => assertRootProjectTrust(tx, { companyId: input.companyId, actor: input.actor, issueId: input.issueId, projectIds: target.sourceProjectIds })); + await measureRunPhase("workspace.readers.sources", () => assertTaskWorkspacesReadAccess(tx, input.actor, input.companyId, target.sourceIds)); + await measureRunPhase("workspace.readers.projects", () => assertRootProjectAccess(tx, input.actor!, input.companyId, target.sourceProjectIds, + { sharedProjectIds: target.sharedProjectIds, issueId: input.issueId })); const readerActors: AuthorizationActor[] = []; for (const reader of target.overlappingEntries.filter(entry => entry.kind === "reader")) { - const { actor, issue } = await retainedRootReader(tx, input.companyId, reader); + const { actor, issue } = await measureRunPhase("workspace.readers.identity", () => retainedRootReader(tx, input.companyId, reader)); readerActors.push(actor); - await assertRootProjectTrust(tx, { companyId: input.companyId, actor, issueId: issue.id, projectIds: target.sourceProjectIds }); + await measureRunPhase("workspace.readers.trust", () => assertRootProjectTrust(tx, { companyId: input.companyId, actor, issueId: issue.id, projectIds: target.sourceProjectIds })); } - await assertRootProjectsReadAccessForActors(tx, readerActors, input.companyId, target.sourceProjectIds); - await assertTaskWorkspacesReadAccessForActors(tx, readerActors, input.companyId, target.sourceIds); + await measureRunPhase("workspace.readers.projects", () => assertRootProjectsReadAccessForActors(tx, readerActors, input.companyId, target.sourceProjectIds)); + await measureRunPhase("workspace.readers.sources", () => assertTaskWorkspacesReadAccessForActors(tx, readerActors, input.companyId, target.sourceIds)); } catch (error) { failure ??= error; } if (!failure && verifiedCopyProjectIds) { for (const scopeId of verifiedCopyProjectIds) if (!existing.some(entry => entry.kind === "isolated_copy" && entry.scopeId === scopeId)) { - await tx.insert(executionWorkspaceRepositoryAccess).values({ companyId: input.companyId, rootKey, canonicalCwd, kind: "isolated_copy", scopeId }); + await measureRunPhase("workspace.write", () => tx.insert(executionWorkspaceRepositoryAccess).values({ companyId: input.companyId, rootKey, canonicalCwd, kind: "isolated_copy", scopeId })); } } return failure; @@ -464,10 +472,10 @@ async function assertRootProjectTrust(db: Db | DbTransaction, input: { export async function assertExecutionWorkspaceRootWriteAccess(db: Db | DbTransaction, input: { companyId: string; cwd: string; actor: AuthorizationActor; sourceWorkspaceId?: string; taskCreation?: TaskWorkspaceCreationTrustContext; } & Omit[4]>, "write">): Promise { - const { sourceIds, sourceProjectIds, sharedProjectIds } = await resolveRepositoryRootSources(db, { ...input, inspectCurrentAliases: true }); - await assertTaskWorkspacesReadAccess(db, input.actor, input.companyId, sourceIds); - await assertRootProjectAccess(db, input.actor, input.companyId, sourceProjectIds, { ...input, sharedProjectIds }); - await assertRootProjectTrust(db, { ...input, projectIds: sourceProjectIds }); + const { sourceIds, sourceProjectIds, sharedProjectIds } = await measureRunPhase("workspace.sources", () => resolveRepositoryRootSources(db, { ...input, inspectCurrentAliases: true })); + await measureRunPhase("workspace.readers.sources", () => assertTaskWorkspacesReadAccess(db, input.actor, input.companyId, sourceIds)); + await measureRunPhase("workspace.readers.projects", () => assertRootProjectAccess(db, input.actor, input.companyId, sourceProjectIds, { ...input, sharedProjectIds })); + await measureRunPhase("workspace.readers.trust", () => assertRootProjectTrust(db, { ...input, projectIds: sourceProjectIds })); } export function executionWorkspaceRepositoryService(db: Db) { @@ -501,10 +509,10 @@ export function executionWorkspaceRepositoryService(db: Db) { if (await requiresAnonymousAcquisition(db, { issue, workspace, agentId: input.agentId, runId: input.runId })) throw forbidden(controllerAcquisitionMessage); } async function reserveRootPolicy(tx: DbTransaction, companyId: string, cwd: string, mode: "anonymous_only" | "managed") { - const canonicalCwd = await canonicalRepositoryRoot(cwd); + const canonicalCwd = await measureRunPhase("workspace.sources.canonicalize", () => canonicalRepositoryRoot(cwd)); const rootKey = createHash("sha256").update(canonicalCwd).digest("hex"); - const reservations = await tx.select().from(executionWorkspaceRepositoryRoots) - .where(eq(executionWorkspaceRepositoryRoots.companyId, companyId)); + const reservations = await measureRunPhase("workspace.sources.inventory", async () => tx.select().from(executionWorkspaceRepositoryRoots) + .where(eq(executionWorkspaceRepositoryRoots.companyId, companyId))); const existing = reservations.find(entry => entry.rootKey === rootKey); const overlapping = reservations.filter(entry => repositoryRootsOverlap(entry.canonicalCwd, canonicalCwd)); if ((existing && existing.canonicalCwd !== canonicalCwd) @@ -512,10 +520,10 @@ export function executionWorkspaceRepositoryService(db: Db) { throw forbidden(lowTrustRepositoryMessage); } if (mode === "anonymous_only") { - const managed = await tx.select({ cwd: executionWorkspaces.cwd, repositoryIdentity: executionWorkspaceRepositories.repositoryIdentity }).from(executionWorkspaceRepositories) + const managed = await measureRunPhase("workspace.sources.inventory", async () => tx.select({ cwd: executionWorkspaces.cwd, repositoryIdentity: executionWorkspaceRepositories.repositoryIdentity }).from(executionWorkspaceRepositories) .innerJoin(executionWorkspaces, eq(executionWorkspaces.id, executionWorkspaceRepositories.executionWorkspaceId)) .where(and(eq(executionWorkspaceRepositories.companyId, companyId), eq(executionWorkspaces.companyId, companyId), - isNotNull(executionWorkspaceRepositories.catalogRepositoryId))); + isNotNull(executionWorkspaceRepositories.catalogRepositoryId)))); if (managed.length) { const namespace = path.join(canonicalCwd, ".paperclip-repositories"); const namespaceEntry = await fs.lstat(namespace).catch((error: NodeJS.ErrnoException) => { @@ -539,79 +547,83 @@ export function executionWorkspaceRepositoryService(db: Db) { if (peer.cwd && await legacyRepositoryRootMayMatch(peer.cwd, canonicalCwd)) throw forbidden(lowTrustRepositoryMessage); } } - if (!existing) await tx.insert(executionWorkspaceRepositoryRoots).values({ companyId, canonicalCwd, rootKey, mode }); + if (!existing) await measureRunPhase("workspace.write", () => tx.insert(executionWorkspaceRepositoryRoots).values({ companyId, canonicalCwd, rootKey, mode })); return canonicalCwd; } type RootReader = { runId: string; issueId: string; agentId: string }; async function authorizeRootAccess(tx: DbTransaction, input: { companyId: string; cwd: string; sourceWorkspaceId?: string; reader?: RootReader; actor?: AuthorizationActor; issueId?: string; preserveOwnAdmission?: boolean; }) { - const { canonicalCwd, rootKey, entries, overlappingEntries, sourceIds, sharedSourceIds, isolatedSourceIds, sourceProjectIds, sharedProjectIds, isolatedProjectIds } = await resolveRepositoryRootSources(tx, input); + const { canonicalCwd, rootKey, entries, overlappingEntries, sourceIds, sharedSourceIds, isolatedSourceIds, sourceProjectIds, sharedProjectIds, isolatedProjectIds } = await measureRunPhase("workspace.sources", () => resolveRepositoryRootSources(tx, input)); const readers = overlappingEntries.filter(entry => entry.kind === "reader"); if (input.reader && !entries.some(entry => entry.kind === "reader" && entry.scopeId === input.reader!.runId)) { - const [run] = await tx.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.id, input.reader.runId), eq(heartbeatRuns.companyId, input.companyId))); + const [run] = await measureRunPhase("workspace.sources.inventory", async () => tx.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.id, input.reader!.runId), eq(heartbeatRuns.companyId, input.companyId)))); readers.push({ id: randomUUID(), companyId: input.companyId, rootKey, canonicalCwd, kind: "reader", scopeId: input.reader.runId, issueId: input.reader.issueId, agentId: input.reader.agentId, responsibleUserId: run?.responsibleUserId ?? null, createdAt: new Date(), updatedAt: new Date() }); } const ownAdmissionOnly = input.preserveOwnAdmission && input.reader && input.sourceWorkspaceId && sourceIds.size === 1 && sourceIds.has(input.sourceWorkspaceId) && readers.every(reader => reader.issueId === input.reader!.issueId); - const sourceProjects = sourceIds.size ? await tx.selectDistinct({ projectId: executionWorkspaces.projectId }).from(executionWorkspaces).where(and( - eq(executionWorkspaces.companyId, input.companyId), inArray(executionWorkspaces.id, [...sourceIds]))) : []; + const sourceProjects = sourceIds.size ? await measureRunPhase("workspace.sources.inventory", async () => tx.selectDistinct({ projectId: executionWorkspaces.projectId }).from(executionWorkspaces).where(and( + eq(executionWorkspaces.companyId, input.companyId), inArray(executionWorkspaces.id, [...sourceIds])))) : []; const sourceActors: AuthorizationActor[] = []; const projectReaders: AuthorizationActor[] = []; - for (const reader of readers) { - const { actor, issue } = await retainedRootReader(tx, input.companyId, reader); - await assertRootProjectTrust(tx, { companyId: input.companyId, projectIds: sourceProjectIds, actor, issueId: issue.id, - sourceWorkspaceId: input.reader?.runId === reader.scopeId ? input.sourceWorkspaceId : undefined }); - if (input.reader?.runId === reader.scopeId) { - await assertRootProjectAccess(tx, actor, input.companyId, sourceProjectIds, { sharedProjectIds, issueId: issue.id }); - } else projectReaders.push(actor); - if (ownAdmissionOnly) { - const [ownSource] = await tx.select({ id: executionWorkspaces.id }).from(executionWorkspaces).where(and( - eq(executionWorkspaces.companyId, input.companyId), eq(executionWorkspaces.id, input.sourceWorkspaceId!), - await executionWorkspaceReadSqlCondition(tx, { ...actor, onBehalfOfUserId: null }), - actor.onBehalfOfUserId ? await executionWorkspaceReadSqlCondition(tx, { - type: "board", userId: actor.onBehalfOfUserId, ignoreInstanceAdmin: true, - }) : undefined)); - if (!ownSource) throw forbidden("Task workspace access is no longer available"); - } else sourceActors.push(actor); - for (const sourceWorkspace of sourceProjects.length ? sourceProjects : [null]) { - if (await requiresAnonymousAcquisition(tx, { issue, workspace: sourceWorkspace, agentId: reader.agentId!, runId: reader.scopeId })) { - await reserveRootPolicy(tx, input.companyId, canonicalCwd, "anonymous_only"); + await measureRunPhase("workspace.readers", async () => { + for (const reader of readers) { + const { actor, issue } = await measureRunPhase("workspace.readers.identity", () => retainedRootReader(tx, input.companyId, reader)); + await measureRunPhase("workspace.readers.trust", () => assertRootProjectTrust(tx, { companyId: input.companyId, projectIds: sourceProjectIds, actor, issueId: issue.id, + sourceWorkspaceId: input.reader?.runId === reader.scopeId ? input.sourceWorkspaceId : undefined })); + if (input.reader?.runId === reader.scopeId) { + await measureRunPhase("workspace.readers.projects", () => assertRootProjectAccess(tx, actor, input.companyId, sourceProjectIds, { sharedProjectIds, issueId: issue.id })); + } else projectReaders.push(actor); + if (ownAdmissionOnly) { + const [ownSource] = await measureRunPhase("workspace.sources.inventory", async () => tx.select({ id: executionWorkspaces.id }).from(executionWorkspaces).where(and( + eq(executionWorkspaces.companyId, input.companyId), eq(executionWorkspaces.id, input.sourceWorkspaceId!), + await executionWorkspaceReadSqlCondition(tx, { ...actor, onBehalfOfUserId: null }), + actor.onBehalfOfUserId ? await executionWorkspaceReadSqlCondition(tx, { + type: "board", userId: actor.onBehalfOfUserId, ignoreInstanceAdmin: true, + }) : undefined))); + if (!ownSource) throw forbidden("Task workspace access is no longer available"); + } else sourceActors.push(actor); + for (const sourceWorkspace of sourceProjects.length ? sourceProjects : [null]) { + if (await requiresAnonymousAcquisition(tx, { issue, workspace: sourceWorkspace, agentId: reader.agentId!, runId: reader.scopeId })) { + await reserveRootPolicy(tx, input.companyId, canonicalCwd, "anonymous_only"); + } } } - } - await assertRootProjectsReadAccessForActors(tx, projectReaders, input.companyId, sourceProjectIds); + await measureRunPhase("workspace.readers.projects", () => assertRootProjectsReadAccessForActors(tx, projectReaders, input.companyId, sourceProjectIds)); + }); if (input.actor) { if (!input.issueId) throw forbidden("Workspace write authorization requires a task"); - await assertRootProjectTrust(tx, { ...input, projectIds: sourceProjectIds, actor: input.actor }); - await assertRootProjectAccess(tx, input.actor, input.companyId, sourceProjectIds, { sharedProjectIds, issueId: input.issueId }); + await measureRunPhase("workspace.readers.trust", () => assertRootProjectTrust(tx, { ...input, projectIds: sourceProjectIds, actor: input.actor! })); + await measureRunPhase("workspace.readers.projects", () => assertRootProjectAccess(tx, input.actor!, input.companyId, sourceProjectIds, { sharedProjectIds, issueId: input.issueId })); sourceActors.push(input.actor); } - await assertTaskWorkspacesReadAccessForActors(tx, sourceActors, input.companyId, sourceIds); + await measureRunPhase("workspace.readers.sources", () => assertTaskWorkspacesReadAccessForActors(tx, sourceActors, input.companyId, sourceIds)); for (const [kind, ids] of [["source", sharedSourceIds], ["isolated_source", isolatedSourceIds], ["project_source", sourceProjectIds], ["shared_project_source", sharedProjectIds], ["isolated_project_source", isolatedProjectIds]] as const) { for (const scopeId of ids) { if (!entries.some(entry => entry.kind === kind && entry.scopeId === scopeId)) { - await tx.insert(executionWorkspaceRepositoryAccess).values({ companyId: input.companyId, rootKey, canonicalCwd, kind, scopeId }); + await measureRunPhase("workspace.write", () => tx.insert(executionWorkspaceRepositoryAccess).values({ companyId: input.companyId, rootKey, canonicalCwd, kind, scopeId })); } } } - for (const reader of readers) { - if (reader.rootKey === rootKey && !entries.some(entry => entry.kind === "reader" && entry.scopeId === reader.scopeId)) { - await tx.insert(executionWorkspaceRepositoryAccess).values(reader); + await measureRunPhase("workspace.readers", async () => { + for (const reader of readers) { + if (reader.rootKey === rootKey && !entries.some(entry => entry.kind === "reader" && entry.scopeId === reader.scopeId)) { + await measureRunPhase("workspace.write", () => tx.insert(executionWorkspaceRepositoryAccess).values(reader)); + } } - } + }); } async function reserveRootAccess(input: Parameters[1]) { - return db.transaction(async tx => { + return measureRunTransaction(db, "workspace", async tx => { await lockExecutionWorkspaceRootProvenance(tx, input.companyId); await authorizeRootAccess(tx, input); }); } async function reserveAdmissionRoot(companyId: string, cwd: string, mode: "anonymous_only" | "managed") { - return db.transaction(async tx => { + return measureRunTransaction(db, "workspace", async tx => { await lockExecutionWorkspaceRootProvenance(tx, companyId); return reserveRootPolicy(tx, companyId, cwd, mode); }); @@ -637,16 +649,16 @@ export function executionWorkspaceRepositoryService(db: Db) { } const available = await catalog(input.companyId, userId, localTrusted); const requestedRef = parsed.ref ?? "HEAD"; - const receipt = await db.transaction(async tx => { + const receipt = await measureRunTransaction(db, "workspace", async tx => { await lockExecutionWorkspaceRootProvenance(tx, input.companyId); - const [issue] = await tx.select().from(issues).where(and(eq(issues.id, input.issueId), eq(issues.companyId, input.companyId), await issueReadSqlCondition(tx, input.actor))).for("no key update"); + const [issue] = await measureRunPhase("workspace.sources.inventory", async () => tx.select().from(issues).where(and(eq(issues.id, input.issueId), eq(issues.companyId, input.companyId), await issueReadSqlCondition(tx, input.actor))).for("no key update")); if (!issue) throw notFound("Task not found"); const currentDecision = await authorizationService(tx).decide({ actor: input.actor, action: "issue:mutate", resource: issueResource(issue) }); if (!currentDecision.allowed) throw forbidden("Task workspace modification is not allowed"); if (input.actor.type === "agent" && !["standard", "skill_test"].includes(issue.workMode)) throw forbidden("Repository preparation is unavailable in Ask or Plan mode"); if (!issue.executionWorkspaceId) throw conflict("The task has no admitted workspace yet. Start it once before preparing a repository."); if (!(await canActorReadExecutionWorkspace(db, input.actor, issue.executionWorkspaceId))) throw notFound("Execution workspace not found"); - const [workspace] = await tx.select().from(executionWorkspaces).where(and(eq(executionWorkspaces.id, issue.executionWorkspaceId), eq(executionWorkspaces.companyId, input.companyId), taskWorkspaceSelectableCondition())).for("no key update"); + const [workspace] = await measureRunPhase("workspace.sources.inventory", async () => tx.select().from(executionWorkspaces).where(and(eq(executionWorkspaces.id, issue.executionWorkspaceId!), eq(executionWorkspaces.companyId, input.companyId), taskWorkspaceSelectableCondition())).for("no key update")); if (!workspace) throw conflict("Execution workspace is unavailable"); await assertTaskWorkspaceAccess(tx, input.actor, input.companyId, workspace.id, { write: true, issueId: input.issueId }); const anonymousOnly = await requiresAnonymousAcquisition(tx, { issue, workspace, @@ -659,7 +671,7 @@ export function executionWorkspaceRepositoryService(db: Db) { if (!workspace.cwd) throw conflict("The task workspace has no directory for repository preparation"); await authorizeRootAccess(tx, { companyId: input.companyId, cwd: workspace.cwd, sourceWorkspaceId: workspace.id, actor: input.actor, issueId: input.issueId }); if (anonymousOnly) throw forbidden(controllerAcquisitionMessage); - const existing = await tx.select().from(executionWorkspaceRepositories).where(and(eq(executionWorkspaceRepositories.executionWorkspaceId, workspace.id), or(eq(executionWorkspaceRepositories.repositoryIdentity, source.repositoryIdentity), sql`${executionWorkspaceRepositories.requestKeys} ? ${parsed.requestKey}`))); + const existing = await measureRunPhase("workspace.sources.inventory", async () => tx.select().from(executionWorkspaceRepositories).where(and(eq(executionWorkspaceRepositories.executionWorkspaceId, workspace.id), or(eq(executionWorkspaceRepositories.repositoryIdentity, source.repositoryIdentity), sql`${executionWorkspaceRepositories.requestKeys} ? ${parsed.requestKey}`)))); for (const row of existing) { assertAcquisitionReceipt(row, anonymousOnly); assertRepositoryRetryMatches(row, source.repositoryIdentity, requestedRef); @@ -672,11 +684,11 @@ export function executionWorkspaceRepositoryService(db: Db) { } return row; } - const [row] = await tx.insert(executionWorkspaceRepositories).values({ companyId: input.companyId, executionWorkspaceId: workspace.id, requestedByIssueId: input.issueId, ...source, relativePath: taskRepositoryRelativePath(source.repositoryIdentity), requestedRef, requestKey: parsed.requestKey, requestKeys: [parsed.requestKey] }).returning(); - const canonicalCwd = await canonicalRepositoryRoot(workspace.cwd); - await tx.insert(executionWorkspaceRepositoryAccess).values({ companyId: input.companyId, - rootKey: createHash("sha256").update(canonicalCwd).digest("hex"), canonicalCwd, kind: "repository", scopeId: row.id }); - await tx.insert(activityLog).values({ companyId: input.companyId, actorType: input.actor.type === "agent" ? "agent" : "user", actorId: input.actor.agentId ?? input.actor.userId ?? "local-board", action: "workspace.repository.requested", entityType: "execution_workspace", entityId: workspace.id, runId: input.actor.runId ?? null, responsibleUserId: userId, details: { repositoryId: row.id, issueId: input.issueId, applies: "next_normal_admission" } }); + const [row] = await measureRunPhase("workspace.write", () => tx.insert(executionWorkspaceRepositories).values({ companyId: input.companyId, executionWorkspaceId: workspace.id, requestedByIssueId: input.issueId, ...source, relativePath: taskRepositoryRelativePath(source.repositoryIdentity), requestedRef, requestKey: parsed.requestKey, requestKeys: [parsed.requestKey] }).returning()); + const canonicalCwd = await measureRunPhase("workspace.sources.canonicalize", () => canonicalRepositoryRoot(workspace.cwd!)); + await measureRunPhase("workspace.write", () => tx.insert(executionWorkspaceRepositoryAccess).values({ companyId: input.companyId, + rootKey: createHash("sha256").update(canonicalCwd).digest("hex"), canonicalCwd, kind: "repository", scopeId: row.id })); + await measureRunPhase("workspace.write", () => tx.insert(activityLog).values({ companyId: input.companyId, actorType: input.actor.type === "agent" ? "agent" : "user", actorId: input.actor.agentId ?? input.actor.userId ?? "local-board", action: "workspace.repository.requested", entityType: "execution_workspace", entityId: workspace.id, runId: input.actor.runId ?? null, responsibleUserId: userId, details: { repositoryId: row.id, issueId: input.issueId, applies: "next_normal_admission" } })); return row; }); return { kind: "requires_next_admission" as const, applies: "next_normal_admission" as const, operationId: receipt.id, repository: receipt, reason: "immutable_admitted_workspace" }; @@ -686,24 +698,24 @@ export function executionWorkspaceRepositoryService(db: Db) { } type PublicationInput = { companyId: string; issueId: string; workspaceId: string; cwd: string; agentId: string; runId: string }; async function publicationBinding(tx: DbTransaction, input: PublicationInput) { - const [issue] = await tx.select().from(issues).where(and(eq(issues.id, input.issueId), eq(issues.companyId, input.companyId))); - const [workspace] = await tx.select().from(executionWorkspaces).where(and(eq(executionWorkspaces.id, input.workspaceId), eq(executionWorkspaces.companyId, input.companyId))); + const [issue] = await measureRunPhase("workspace.sources.inventory", async () => tx.select().from(issues).where(and(eq(issues.id, input.issueId), eq(issues.companyId, input.companyId)))); + const [workspace] = await measureRunPhase("workspace.sources.inventory", async () => tx.select().from(executionWorkspaces).where(and(eq(executionWorkspaces.id, input.workspaceId), eq(executionWorkspaces.companyId, input.companyId)))); if (!issue || !workspace || issue.executionWorkspaceId !== workspace.id || !workspace.cwd - || await canonicalRepositoryRoot(workspace.cwd) !== await canonicalRepositoryRoot(input.cwd)) { + || await measureRunPhase("workspace.sources.canonicalize", () => canonicalRepositoryRoot(workspace.cwd!)) !== await measureRunPhase("workspace.sources.canonicalize", () => canonicalRepositoryRoot(input.cwd))) { throw forbidden("Task repository publication binding is unavailable"); } await requiresAnonymousAcquisition(tx, { issue, workspace, agentId: input.agentId, runId: input.runId }); - const [run] = await tx.select({ responsibleUserId: heartbeatRuns.responsibleUserId }).from(heartbeatRuns).where(and(eq(heartbeatRuns.id, input.runId), eq(heartbeatRuns.companyId, input.companyId))); - const { actor } = await retainedRootReader(tx, input.companyId, { + const [run] = await measureRunPhase("workspace.sources.inventory", async () => tx.select({ responsibleUserId: heartbeatRuns.responsibleUserId }).from(heartbeatRuns).where(and(eq(heartbeatRuns.id, input.runId), eq(heartbeatRuns.companyId, input.companyId)))); + const { actor } = await measureRunPhase("workspace.readers.identity", () => retainedRootReader(tx, input.companyId, { scopeId: input.runId, issueId: input.issueId, agentId: input.agentId, responsibleUserId: run?.responsibleUserId ?? null, - }); + })); return { workspace, actor }; } async function guardForPublication(input: PublicationInput) { - await db.transaction(async tx => { + await measureRunTransaction(db, "workspace", async tx => { await lockExecutionWorkspaceRootProvenance(tx, input.companyId); const { workspace } = await publicationBinding(tx, input); - await authorizeRootAccess(tx, { companyId: input.companyId, cwd: input.cwd, sourceWorkspaceId: workspace.id, reader: input }); + await authorizeRootAccess(tx, { companyId: input.companyId, cwd: input.cwd!, sourceWorkspaceId: workspace.id, reader: input }); }); } async function prepareConfiguredRepositorySource(input: PublicationInput & { sourceCwd: string }): Promise<() => Promise> { @@ -717,35 +729,35 @@ export function executionWorkspaceRepositoryService(db: Db) { commonRoot = source.commonRoot; for (const cwd of source.roots) roots.add(cwd); } catch (error) { sourceFailure = error; } - const denied = await db.transaction(async tx => { + const denied = await measureRunTransaction(db, "workspace", async tx => { await lockExecutionWorkspaceRootProvenance(tx, input.companyId); const { workspace, actor } = await publicationBinding(tx, input); - const canonicalCwd = await canonicalRepositoryRoot(input.cwd); + const canonicalCwd = await measureRunPhase("workspace.sources.canonicalize", () => canonicalRepositoryRoot(input.cwd)); const rootKey = createHash("sha256").update(canonicalCwd).digest("hex"); - const existing = await tx.select().from(executionWorkspaceRepositoryAccess).where(and( - eq(executionWorkspaceRepositoryAccess.companyId, input.companyId), eq(executionWorkspaceRepositoryAccess.rootKey, rootKey))); + const existing = await measureRunPhase("workspace.sources.inventory", async () => tx.select().from(executionWorkspaceRepositoryAccess).where(and( + eq(executionWorkspaceRepositoryAccess.companyId, input.companyId), eq(executionWorkspaceRepositoryAccess.rootKey, rootKey)))); if (existing.some(entry => entry.canonicalCwd !== canonicalCwd)) throw forbidden("Repository root identity conflicts with its retained access record"); let failure = sourceFailure; for (const cwd of roots) { try { - const source = await resolveRepositoryRootSources(tx, { companyId: input.companyId, cwd }); + const source = await measureRunPhase("workspace.sources", () => resolveRepositoryRootSources(tx, { companyId: input.companyId, cwd })); for (const id of source.sourceIds) workspaceIds.add(id); for (const id of source.sourceProjectIds) projectIds.add(id); } catch (error) { failure ??= error; } } - try { await assertTaskWorkspacesReadAccess(tx, actor, input.companyId, workspaceIds); } + try { await measureRunPhase("workspace.readers.sources", () => assertTaskWorkspacesReadAccess(tx, actor, input.companyId, workspaceIds)); } catch (error) { failure ??= error; } for (const id of projectIds) { - try { await assertRootProjectAccess(tx, actor, input.companyId, new Set([id])); } + try { await measureRunPhase("workspace.readers.projects", () => assertRootProjectAccess(tx, actor, input.companyId, new Set([id]))); } catch (error) { failure ??= error; } } for (const [kind, ids] of [["isolated_source", workspaceIds], ["isolated_project_source", projectIds]] as const) { for (const scopeId of ids) if (!existing.some(entry => entry.kind === kind && entry.scopeId === scopeId)) { - await tx.insert(executionWorkspaceRepositoryAccess).values({ companyId: input.companyId, rootKey, canonicalCwd, kind, scopeId }); + await measureRunPhase("workspace.write", () => tx.insert(executionWorkspaceRepositoryAccess).values({ companyId: input.companyId, rootKey, canonicalCwd, kind, scopeId })); } } try { - await authorizeRootAccess(tx, { companyId: input.companyId, cwd: input.cwd, sourceWorkspaceId: workspace.id, + await authorizeRootAccess(tx, { companyId: input.companyId, cwd: input.cwd!, sourceWorkspaceId: workspace.id, actor, issueId: input.issueId }); } catch (error) { failure ??= error; } if (!failure && commonRoot) { diff --git a/server/src/services/execution-workspaces.ts b/server/src/services/execution-workspaces.ts index 42dbad1b387..d3a80e3aa94 100644 --- a/server/src/services/execution-workspaces.ts +++ b/server/src/services/execution-workspaces.ts @@ -1,3 +1,5 @@ +import { measureRunTransaction } from "./execution-transaction-timing.js"; +import { measureRunPhase } from "./execution-preparation-timing.js"; import { assertTaskWorkspaceAccess, assertTaskWorkspaceSourceProjectAccess } from "./task-workspace-source-access.js"; import { verifiedRecordedIsolatedRemoteSource } from "./isolated-remote-workspace-source.js"; import { parseObject } from "../adapters/utils.js"; @@ -251,9 +253,9 @@ async function preserveWorkspaceOwnershipForPatch( tx: DbTransaction, current: typeof executionWorkspaces.$inferSelect, patch: Partial, ) { if (workspaceSourceKeys.some(key => patch[key] !== undefined && patch[key] !== current[key])) { - await preserveExecutionWorkspaceRootProvenance(tx, { + await measureRunPhase("workspace.provenance", () => preserveExecutionWorkspaceRootProvenance(tx, { companyId: current.companyId, workspaceId: current.id, cwd: current.cwd, - }); + })); } const nextPatch = { ...patch }; const retainedSources = { @@ -1531,8 +1533,8 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic async function bindTaskWorkspace(companyId: string, issueId: string, workspaceId: string, patch: TaskWorkspaceBindingPatch = {}) { return db.transaction(async tx => { - const [task] = await tx.select().from(issues).where(and(eq(issues.id, issueId), eq(issues.companyId, companyId))).for("no key update"); - const [workspace] = await tx.select().from(executionWorkspaces).where(and(eq(executionWorkspaces.id, workspaceId), eq(executionWorkspaces.companyId, companyId))).for("no key update"); + const [task] = await measureRunPhase("workspace.row_lock", () => tx.select().from(issues).where(and(eq(issues.id, issueId), eq(issues.companyId, companyId))).for("no key update")); + const [workspace] = await measureRunPhase("workspace.row_lock", () => tx.select().from(executionWorkspaces).where(and(eq(executionWorkspaces.id, workspaceId), eq(executionWorkspaces.companyId, companyId))).for("no key update")); if (!task || !workspace) throw notFound("Task workspace unavailable"); await tx.update(executionWorkspaces).set({ metadata: { ...workspace.metadata, _issuePrivacySources: { ...(workspace.metadata?._issuePrivacySources as Record ?? {}), [task.id]: true } } }).where(eq(executionWorkspaces.id, workspaceId)); @@ -1572,9 +1574,9 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic async function prepareTaskWorkspaceUpdate(input: { companyId: string; issueId: string; patch: TaskWorkspaceBindingPatch; }, tx: DbTransaction) { - const [task] = await tx.select().from(issues).where(and( + const [task] = await measureRunPhase("workspace.row_lock", () => tx.select().from(issues).where(and( eq(issues.id, input.issueId), eq(issues.companyId, input.companyId), - )).for("no key update"); + )).for("no key update")); if (!task) throw notFound("Task not found"); const fields = ["executionWorkspaceId", "projectWorkspaceId", "executionWorkspacePreference", "executionWorkspaceSettings"] as const; if (!fields.some(field => input.patch[field] !== undefined && !isDeepStrictEqual(input.patch[field], task[field]))) return {}; @@ -1929,7 +1931,7 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic } return input.onSkip(); } - await notifyTerminalWorkspaceWork(tx, `workspace:${input.workspaceId}`); + await measureRunPhase("workspace.notification", () => notifyTerminalWorkspaceWork(tx, `workspace:${input.workspaceId}`)); return input.write({ tx, fresh }); }); } @@ -3582,36 +3584,36 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic }, create: async (data: typeof executionWorkspaces.$inferInsert) => { - const row = await db.transaction(async tx => { - await notifyTerminalWorkspaceWork(tx, data.id ? `workspace:${data.id}` - : data.sourceIssueId ? `issue:${data.sourceIssueId}` : undefined); - return tx.insert(executionWorkspaces).values(data).returning().then(rows => rows[0] ?? null); + const row = await measureRunTransaction(db, "workspace", async tx => { + await measureRunPhase("workspace.notification", () => notifyTerminalWorkspaceWork(tx, data.id ? `workspace:${data.id}` + : data.sourceIssueId ? `issue:${data.sourceIssueId}` : undefined)); + return measureRunPhase("workspace.write", () => tx.insert(executionWorkspaces).values(data).returning().then(rows => rows[0] ?? null)); }); return row ? toExecutionWorkspace(row) : null; }, update: async (id: string, patch: Partial) => { if (patch.metadata === undefined && !workspaceSourceKeys.some(key => patch[key] !== undefined)) { - const row = await db.transaction(async tx => { - await notifyTerminalWorkspaceWork(tx, `workspace:${id}`); - return tx.update(executionWorkspaces).set({ ...patch, updatedAt: new Date() }) - .where(eq(executionWorkspaces.id, id)).returning().then(rows => rows[0] ?? null); + const row = await measureRunTransaction(db, "workspace", async tx => { + await measureRunPhase("workspace.notification", () => notifyTerminalWorkspaceWork(tx, `workspace:${id}`)); + return measureRunPhase("workspace.write", () => tx.update(executionWorkspaces).set({ ...patch, updatedAt: new Date() }) + .where(eq(executionWorkspaces.id, id)).returning().then(rows => rows[0] ?? null)); }); return row ? toExecutionWorkspace(row) : null; } - const [identity] = await db.select({ companyId: executionWorkspaces.companyId }).from(executionWorkspaces) - .where(eq(executionWorkspaces.id, id)); + const [identity] = await measureRunPhase("workspace.identity_read", () => db.select({ companyId: executionWorkspaces.companyId }).from(executionWorkspaces) + .where(eq(executionWorkspaces.id, id))); if (!identity) return null; - return db.transaction(async tx => { + return measureRunTransaction(db, "workspace", async tx => { await lockExecutionWorkspaceRootProvenance(tx, identity.companyId); - const [current] = await tx.select().from(executionWorkspaces).where(and( + const [current] = await measureRunPhase("workspace.row_lock", () => tx.select().from(executionWorkspaces).where(and( eq(executionWorkspaces.id, id), eq(executionWorkspaces.companyId, identity.companyId), - )).for("no key update"); + )).for("no key update")); if (!current) return null; - const nextPatch = await preserveWorkspaceOwnershipForPatch(tx, current, patch); - await notifyTerminalWorkspaceWork(tx, `workspace:${id}`); - const [row] = await tx.update(executionWorkspaces).set({ ...nextPatch, updatedAt: new Date() }) - .where(and(eq(executionWorkspaces.id, id), eq(executionWorkspaces.companyId, current.companyId))).returning(); + const nextPatch = await measureRunPhase("workspace.provenance", () => preserveWorkspaceOwnershipForPatch(tx, current, patch)); + await measureRunPhase("workspace.notification", () => notifyTerminalWorkspaceWork(tx, `workspace:${id}`)); + const [row] = await measureRunPhase("workspace.write", () => tx.update(executionWorkspaces).set({ ...nextPatch, updatedAt: new Date() }) + .where(and(eq(executionWorkspaces.id, id), eq(executionWorkspaces.companyId, current.companyId))).returning()); return row ? toExecutionWorkspace(row) : null; }); }, @@ -3866,7 +3868,7 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic // generation so a queued cleanup that captured the old generation does // nothing. Clear cleanupEligibleAt so the reaper does not destroy the // half-built worktree while a later reopen retries. - await notifyTerminalWorkspaceWork(tx, `workspace:${row.id}`); + await measureRunPhase("workspace.notification", () => notifyTerminalWorkspaceWork(tx, `workspace:${row.id}`)); await tx .update(executionWorkspaces) .set({ @@ -3917,7 +3919,7 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic // flag stops the terminal reaper and the archive route from archiving and // destroying the rebuilt worktree in that window. const activeMetadata = setMetadataReopenPendingConsumption(nextMetadata, now()); - await notifyTerminalWorkspaceWork(tx, `workspace:${row.id}`); + await measureRunPhase("workspace.notification", () => notifyTerminalWorkspaceWork(tx, `workspace:${row.id}`)); const activeRow = await tx .update(executionWorkspaces) .set({ @@ -4085,12 +4087,12 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic return db.transaction(async (tx) => { await lockExecutionWorkspaceRootProvenance(tx, identity.companyId); await acquireExecutionWorkspaceLifecycleLock(tx, input.id); - const fresh = await tx + const fresh = await measureRunPhase("workspace.row_lock", () => tx .select() .from(executionWorkspaces) .where(and(eq(executionWorkspaces.id, input.id), eq(executionWorkspaces.companyId, identity.companyId))) .for("no key update") - .then((rows) => rows[0] ?? null); + .then((rows) => rows[0] ?? null)); if (!fresh || isClosedExecutionWorkspaceStatus(fresh.status)) { // The row is missing or already closed by a concurrent path. Do not // archive again. @@ -4103,14 +4105,14 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic // rebuilt worktree. The route maps this to HTTP 409. return { outcome: "reopen_pending" }; } - const nextPatch = await preserveWorkspaceOwnershipForPatch(tx, fresh, input.patch); + const nextPatch = await measureRunPhase("workspace.provenance", () => preserveWorkspaceOwnershipForPatch(tx, fresh, input.patch)); const baseMetadata = (nextPatch.metadata as Record | null | undefined) ?? (fresh.metadata as Record | null); const archiveMetadata = clearMetadataReopenPendingConsumption( bumpExecutionWorkspaceLifecycleGeneration(baseMetadata), ); - await notifyTerminalWorkspaceWork(tx, `workspace:${input.id}`); + await measureRunPhase("workspace.notification", () => notifyTerminalWorkspaceWork(tx, `workspace:${input.id}`)); const archived = await tx .update(executionWorkspaces) .set({ @@ -4375,7 +4377,7 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic updatePatch.name = inspection.toBranch; } - await notifyTerminalWorkspaceWork(tx, `workspace:${lockedWorkspace.id}`); + await measureRunPhase("workspace.notification", () => notifyTerminalWorkspaceWork(tx, `workspace:${lockedWorkspace.id}`)); const [branchUpdatedRow] = await tx .update(executionWorkspaces) .set(updatePatch) diff --git a/server/src/services/heartbeat.ts b/server/src/services/heartbeat.ts index c8544c5bdb9..6255be11197 100644 --- a/server/src/services/heartbeat.ts +++ b/server/src/services/heartbeat.ts @@ -1,3 +1,4 @@ +import { withRunExecutionTiming, measureRunPhase, measureWorkspaceSyncPhase } from "./execution-preparation-timing.js"; import { createHeartbeatPreparationTiming } from "./heartbeat/preparation-timing.js"; import { createHeartbeatInstructionCopyRelease, createHeartbeatInstructionWorkingCopy } from "./heartbeat/instruction-working-copy.js"; import { prepareHeartbeatRuntimeServiceIntents, startHeartbeatRuntimeServices, recordHeartbeatAdapterRuntimeServices } from "./heartbeat/runtime-services.js"; @@ -1795,6 +1796,13 @@ export function heartbeatService( nativeLeaseOwner?: string; nativeRestartRecovery?: NativeRestartRecoveryClaim; } = {}, + ) { + return withRunExecutionTiming(runId, () => executeRunAttempt(runId, runOptions)); + } + + async function executeRunAttempt( + runId: string, + runOptions: { nativeLeaseOwner?: string; nativeRestartRecovery?: NativeRestartRecoveryClaim }, ) { const attemptStartedAtMs = Date.now(); let attestedQuestionResponseAtMs: number | null = null; @@ -3304,7 +3312,7 @@ export function heartbeatService( }, onPhase: async (name, work) => { const started = Date.now(); - try { return await work(); } + try { return await measureWorkspaceSyncPhase(name, work); } finally { await appendRunEvent(run, { eventType: "lifecycle", stream: "system", level: "info", message: name, payload: { phase: name, durationMs: Date.now() - started } }); } }, @@ -3339,7 +3347,7 @@ export function heartbeatService( if (!runtimeSelection.selected) return; const { nativeExecution, nativeRunnerInstanceId, getNativeFreshSessionHandoff } = runtimeSelection; nativeWorkspaceSync = runtimeSelection.nativeWorkspaceSync; - const execution = await executeHeartbeatRuntime(db, { + const execution = await measureRunPhase("execution.runtime", () => executeHeartbeatRuntime(db, { run, agent, options, task: { issueRef, @@ -3436,7 +3444,7 @@ export function heartbeatService( onProviderResourceDisposition: (value) => { providerResourceDispositionForRun = value; }, onWorkspaceRestoreFailure: (evidence) => { requiredWorkspaceRestoreEvidence = evidence; }, }, - }); + })); if (!execution.dispatched) return; const { adapterResult } = execution; // Reconcile the referenced-project set against the real remote staging outcome. A referenced @@ -3479,7 +3487,7 @@ export function heartbeatService( run, agent, issueId, issueRef, executionWorkspace, context, issuesSvc, onLog, adapterResult, runtimeServices, }); - await runCompletion.completeRun({ + await measureRunPhase("completion.run", () => runCompletion.completeRun({ run, agent, issueId, issueRef, signal: executionControl.controller.signal, output: runOutput, session: { runtimeForAdapter, previousSessionParams, taskKey, configuredModel, sessionConfigMetadata }, @@ -3489,7 +3497,7 @@ export function heartbeatService( providerTraceCapture: providerTraceCapture !== null, onProviderTraceFinalized: () => { providerTraceFinalized = true; }, issueContext, onLog, - }); + })); } catch (err) { await persistUsageCaptureFailure?.(); if (err instanceof NativeControllerDetachedForRestartError) { diff --git a/server/src/services/heartbeat/run-completion.ts b/server/src/services/heartbeat/run-completion.ts index e1a3d0a3738..bef970a1e5d 100644 --- a/server/src/services/heartbeat/run-completion.ts +++ b/server/src/services/heartbeat/run-completion.ts @@ -1,3 +1,4 @@ +import { measureRunPhase } from "../execution-preparation-timing.js"; import { NON_RETRYABLE_PREFLIGHT_FAILURE_CODES, isHeartbeatRunTerminalStatus } from "./run-lifecycle.js"; import { ChatControlRecoveryUnresolvedError } from "./queue.js"; import { readTransientRecoveryContractFromRun, readHeartbeatRunErrorFamily, isTransientWorkspaceGitScanCode } from "./retries.js"; @@ -465,7 +466,7 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC compressed: boolean; } | null = null; if (handle) { - logSummary = await runLogStore.finalize(handle); + logSummary = await measureRunPhase("completion.output", () => runLogStore.finalize(handle)); } const finalLogBytes = logSummary?.bytes; if (outputProgressState.pending && typeof finalLogBytes === "number") { @@ -599,12 +600,12 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC logSha256: logSummary?.sha256, logCompressed: logSummary?.compressed ?? false, }; - const persistedRunWrite = await setRunStatusIfRunning( + const persistedRunWrite = await measureRunPhase("completion.terminal_write", () => setRunStatusIfRunning( run.id, status, finalRunPatch, { adapterErrorMeta: adapterResult.errorMeta, secretValues: readFailureReportSecrets() }, - ); + )); let persistedRun: typeof heartbeatRuns.$inferSelect | null = persistedRunWrite.run; if (!persistedRunWrite.updated) { @@ -645,7 +646,7 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC .then((rows) => rows[0] ?? null); } if (!persistedRun) { - await accountRunCost(db, run.id, budgetHooks).catch((err) => { + await measureRunPhase("completion.cost", () => accountRunCost(db, run.id, budgetHooks)).catch((err) => { logger.error({ err, runId: run.id }, "Late run accounting queued for recovery"); }); logger.info( @@ -661,14 +662,14 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC } if (persistedRun) { // Accounting recovery is independent of workspace or issue finalization. - await accountRunCost(db, persistedRun.id, budgetHooks).catch((err) => { + await measureRunPhase("completion.cost", () => accountRunCost(db, persistedRun!.id, budgetHooks)).catch((err) => { logger.error({ err, runId: run.id }, "Run accounting queued for recovery"); }); persistedRun = - (await classifyAndPersistRunLiveness( - persistedRun, + (await measureRunPhase("completion.liveness", () => classifyAndPersistRunLiveness( + persistedRun!, persistedResultJson, - )) ?? persistedRun; + ))) ?? persistedRun; } await setWakeupStatus( @@ -712,25 +713,25 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC ); } const livenessRun = finalizedRun; - await refreshContinuationSummaryForRun(livenessRun, agent); + await measureRunPhase("completion.summary", () => refreshContinuationSummaryForRun(livenessRun, agent)); const skipRunIssueComment = parseObject(livenessRun.contextSnapshot).skipIssueComment === true; let resolvedPresentationDecision: RunPresentationDecision | null = null; try { const existingRunComment = issueId - ? await findRunIssueComment( + ? await measureRunPhase("response.resolve", () => findRunIssueComment( livenessRun.id, livenessRun.companyId, issueId, persistedResultJson, - ) + )) : null; const finalAgentMessage = - await findLatestCompletedFinalAgentMessage( + await measureRunPhase("response.resolve", () => findLatestCompletedFinalAgentMessage( livenessRun.id, livenessRun.companyId, - ); + )); const externalChatPresentationCandidate = isExternalChatPresentationContext(livenessRun.contextSnapshot) || parseObject(livenessRun.contextSnapshot).source === "tool_action_review" || @@ -738,11 +739,11 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC parseObject(livenessRun.contextSnapshot).source === "issue.update"; const externalChatPresentationAuthorization = issueId && externalChatPresentationCandidate - ? await resolveChatRunPresentationAuthorizationReason(db, { + ? await measureRunPhase("response.authorize", () => resolveChatRunPresentationAuthorizationReason(db, { companyId: livenessRun.companyId, issueId, runId: livenessRun.id, - }) + })) : null; const externalChatPresentationContext = isExternalChatPresentationContext( livenessRun.contextSnapshot, @@ -781,17 +782,17 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC // authorize that narrow presentation as the provider reply; // ordinary internal runs retain the private default. const presentationAuthorizationReason = - await resolveChatRunPresentationAuthorizationReason(db, { + await measureRunPhase("response.authorize", () => resolveChatRunPresentationAuthorizationReason(db, { companyId: livenessRun.companyId, issueId, runId: livenessRun.id, - }); - const comment = await issuesSvc.addComment( - issueId, - resolved.text, + })); + const comment = await measureRunPhase("response.add_comment", () => issuesSvc.addComment( + issueId!, + resolved.text!, { agentId: agent.id, runId: livenessRun.id }, { authorizationReason: presentationAuthorizationReason, completionReply: true }, - ); + )); presentationDecision = { ...presentationDecision, commentId: comment.id, @@ -800,7 +801,7 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC "resolved_response_materialized", ], }; - await logActivity(db, { + await measureRunPhase("response.activity", () => logActivity(db, { companyId: livenessRun.companyId, actorType: "agent", actorId: agent.id, @@ -819,7 +820,7 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC source: "run_presentation_resolver", presentationSource: presentationDecision.chosenSource, }, - }); + })); } else if (presentationDecision.commentAction === "create") { presentationDecision = { ...presentationDecision, @@ -1092,7 +1093,7 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC } | null = null; if (handle) { try { - logSummary = await runLogStore.finalize(handle); + logSummary = await measureRunPhase("completion.output", () => runLogStore.finalize(handle)); } catch (finalizeErr) { logger.warn( { err: finalizeErr, runId }, @@ -1114,7 +1115,7 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC const stoppedDuringFailure = signal.aborted; const stopSnapshot = stoppedDuringFailure ? await getRun(run.id) : null; const failureOutcome = stoppedDuringFailure ? "cancelled" : "failed"; - const failedRunWrite = await setRunStatusIfRunning(run.id, failureOutcome, { + const failedRunWrite = await measureRunPhase("completion.terminal_write", () => setRunStatusIfRunning(run.id, failureOutcome, { error: message, errorCode: stopSnapshot?.errorCode ?? failureErrorCode, finishedAt: new Date(), @@ -1143,7 +1144,7 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC logBytes: logSummary?.bytes, logSha256: logSummary?.sha256, logCompressed: logSummary?.compressed ?? false, - }, { error: err, phase: "execute", secretValues: readFailureReportSecrets() }); + }, { error: err, phase: "execute", secretValues: readFailureReportSecrets() })); if ( !failedRunWrite.updated && !( @@ -1175,7 +1176,7 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC message, }); const livenessRun = - (await classifyAndPersistRunLiveness(failedRun)) ?? failedRun; + (await measureRunPhase("completion.liveness", () => classifyAndPersistRunLiveness(failedRun))) ?? failedRun; try { await completeSkillTestRunForHeartbeatOutcome({ run: livenessRun, @@ -1190,7 +1191,7 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC "failed to complete skill test run after heartbeat adapter failure", ); } - await refreshContinuationSummaryForRun(livenessRun, agent); + await measureRunPhase("completion.summary", () => refreshContinuationSummaryForRun(livenessRun, agent)); if ( !isWorkspaceValidationFailedRun(livenessRun) && !isConfigurationIncompleteFailedRun(livenessRun) diff --git a/server/src/services/heartbeat/runtime-execution.ts b/server/src/services/heartbeat/runtime-execution.ts index fe601c505b0..cd8e9fddb87 100644 --- a/server/src/services/heartbeat/runtime-execution.ts +++ b/server/src/services/heartbeat/runtime-execution.ts @@ -1,3 +1,4 @@ +import { measureRunPhase } from "../execution-preparation-timing.js"; import { parseNativeSessionGoalControl } from "./scheduling.js"; import { isHeartbeatRunTerminalStatus } from "./run-lifecycle.js"; import { @@ -890,7 +891,7 @@ export async function executeHeartbeatRuntime(db: Db, input: HeartbeatRuntimeExe if (instructionSave) adapterResult.resultJson = { ...adapterResult.resultJson, instructionSave }; if (parseObject(adapterResult.executionRecovery).providerWorkStarted !== false) { - const captured = await usageRecorder.complete(adapterResult); + const captured = await measureRunPhase("execution.usage", () => usageRecorder.complete(adapterResult)); adapterResult = { ...adapterResult, ...captured, usageComplete: captured.complete }; } else { // Stop may already own the terminal result. Preserve its metadata @@ -916,10 +917,10 @@ export async function executeHeartbeatRuntime(db: Db, input: HeartbeatRuntimeExe )).limit(1); if (exported.length) adapterFinalizeOutcome = "succeeded"; else await recordWorkspaceFinalize(hasWorkspaceRestoreFailure(adapterResult.resultJson) ? "failed" : "succeeded", undefined, - async () => { await restoreNativeWorkspaceBestEffort({ + async () => { await measureRunPhase("execution.workspace_restore", () => restoreNativeWorkspaceBestEffort({ db, runId: run.id, assertOwnership: ownership?.assertHeld, restore: () => nativeWorkspaceSync!.restoreWorkspace(ownership?.assertHeld), - }); }); + })); }); } await ownership?.assertHeld(); await db @@ -937,12 +938,12 @@ export async function executeHeartbeatRuntime(db: Db, input: HeartbeatRuntimeExe adapterResult.nativeFinalization.workspaceFinalizeStatus = workspaceFinalizeStatus; try { - const finalized = await finalizeNativeRun({ + const finalized = await measureRunPhase("execution.native_finalize", () => finalizeNativeRun({ db, runId: run.id, workspaceFinalizeStatus, preserveProviderAttempt: Boolean(nativeWorkspaceSync), - }); + })); await dispatchPendingNativeStatusWakeups({ companyId: run.companyId, }); @@ -965,9 +966,9 @@ export async function executeHeartbeatRuntime(db: Db, input: HeartbeatRuntimeExe } }; if (nativeWorkspaceSync) { - const owned = await withNativeWorkspaceFinalizationOwnership({ + const owned = await measureRunPhase("execution.workspace_ownership", () => withNativeWorkspaceFinalizationOwnership({ db, companyId: run.companyId, runId: run.id, - }, completeWorkspace); + }, completeWorkspace)); if (!owned.acquired) throw new NativeWorkspaceFinalizationBusyError(); } else { await completeWorkspace(); @@ -1115,11 +1116,11 @@ export async function executeHeartbeatRuntime(db: Db, input: HeartbeatRuntimeExe ); } try { - await finalizeNativeRun({ + await measureRunPhase("execution.native_finalize", () => finalizeNativeRun({ db, runId: run.id, workspaceFinalizeStatus: "failed", - }); + })); await dispatchPendingNativeStatusWakeups({ companyId: run.companyId, }); diff --git a/server/src/services/heartbeat/runtime-selection.ts b/server/src/services/heartbeat/runtime-selection.ts index 27aad33d9ee..a4fc9fb03fa 100644 --- a/server/src/services/heartbeat/runtime-selection.ts +++ b/server/src/services/heartbeat/runtime-selection.ts @@ -1,3 +1,5 @@ +import { measureRunTransaction } from "../execution-transaction-timing.js"; +import { measureRunPhase } from "../execution-preparation-timing.js"; import { ConfigurationIncompleteFailure, type buildPaperclipWakePayload, @@ -273,10 +275,10 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele const nativeExecutionWorkspaceId = persistedExecutionWorkspace?.id ?? run.id; const nativeReviewContext = readNativeReviewAssignmentContext(context); - const nativeReview = nativeReviewContext ? await getNativeReviewAssignment(db, { + const nativeReview = nativeReviewContext ? await measureRunPhase("runtime_selection.review", () => getNativeReviewAssignment(db, { companyId: agent.companyId, issueId: issueRef.id, agentId: agent.id, contextSnapshot: nativeReviewContext, - }) : null; + })) : null; if (nativeReviewContext && !nativeReview) throw new Error("native_review_assignment_no_longer_available"); const nativeReviewRequest = nativeReview ? buildNativeReviewRequest({ @@ -286,18 +288,18 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele }) : null; const persistedContract = run.completionContractId - ? await db + ? await measureRunPhase("runtime_selection.contract_read", () => db .select() .from(completionContracts) .where( and( - eq(completionContracts.id, run.completionContractId), + eq(completionContracts.id, run.completionContractId!), eq(completionContracts.companyId, agent.companyId), eq(completionContracts.issueId, issueRef.id), ), ) .limit(1) - .then((rows) => rows[0] ?? null) + .then((rows) => rows[0] ?? null)) : null; // Only a server-verified human resolution may supply a current answer // reference. Tool/agent results and generated summaries stay evidence. @@ -359,7 +361,7 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele row: persistedContract, contract: persistedContract.contractJson as never, } - : await ensureNativeCompletionContract({ + : await measureRunPhase("runtime_selection.contract_ensure", () => ensureNativeCompletionContract({ db, companyId: agent.companyId, issue: issueRef, @@ -380,7 +382,7 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele humanResponseId: currentHumanResponseId, immediateRequests: immediateCompletion.requests, immediateRequestSources: immediateCompletion.sources, - }); + })); const taskNativeSessionId = !taskSessionCredentialCompatible ? null : readNonEmptyString( taskSessionDecodedParams?.sessionId, ); @@ -390,7 +392,7 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele // replacement makes the ownership ambiguous and therefore ineligible. const legacyRetrySource = run.retryOfRunId && !isFailedChatRunRetry - ? await db + ? await measureRunPhase("runtime_selection.resume", () => db .select({ id: heartbeatRuns.id, companyId: heartbeatRuns.companyId, @@ -404,19 +406,19 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele .from(heartbeatRuns) .where( and( - eq(heartbeatRuns.id, run.retryOfRunId), + eq(heartbeatRuns.id, run.retryOfRunId!), eq(heartbeatRuns.companyId, agent.companyId), eq(heartbeatRuns.agentId, agent.id), ), ) .limit(1) - .then((rows) => rows[0] ?? null) + .then((rows) => rows[0] ?? null)) : null; const nativeBootstrapHasProviderEvidence = legacyRetrySource || run.nativeSessionId || persistedNativeExecutionInput - ? await db + ? await measureRunPhase("runtime_selection.resume", () => db .select({ id: heartbeatRunEvents.id }) .from(heartbeatRunEvents) .where( @@ -434,7 +436,7 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele ), ) .limit(1) - .then((rows) => rows.length > 0) + .then((rows) => rows.length > 0)) : false; const compatibleLegacyRetrySource = !managedAiRuntime && !isConversation(issueContext) && context.forceFreshSession !== true && isUnusedLegacyNativeRetryReplacement({ @@ -467,14 +469,14 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele run, nativeBootstrapHasProviderEvidence, ) - ? await findNativeSessionResumeRun(db, { + ? await measureRunPhase("runtime_selection.resume", () => findNativeSessionResumeRun(db, { companyId: agent.companyId, agentId: agent.id, issueId: issueRef.id, normalizedSessionId: requestedNativeSessionId, currentRunId: run.id, beforeCreatedAt: run.createdAt, - }) + })) : null; nativeRunnerInstanceId = previousNativeRun?.runnerInstanceId && @@ -569,12 +571,13 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele }); } if (nativeExecution.provider.kind === "claude_managed") { - const recoveryProfile = await managedAgentProfileService( + const recoveryProfileId = nativeExecution.provider.managedProfile.profileId; + const recoveryProfile = await measureRunPhase("runtime_selection.profile", () => managedAgentProfileService( db, ).requireQualified( agent.companyId, - nativeExecution.provider.managedProfile.profileId, - ); + recoveryProfileId, + )); assertManagedProfileRecoveryBinding({ adapterConfig: agent.adapterConfig, snapshot: nativeExecution.provider.managedProfile, @@ -582,13 +585,14 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele }); } if (nativeExecution.provider.kind === "aws_agentcore") { - const recoveryProfile = await remoteAgentProfileService( + const recoveryProfileId = nativeExecution.provider.agentCoreProfile.profileId; + const recoveryProfile = await measureRunPhase("runtime_selection.profile", () => remoteAgentProfileService( db, ).requireQualified( agent.companyId, - nativeExecution.provider.agentCoreProfile.profileId, + recoveryProfileId, "aws_bedrock_agentcore_harness", - ); + )); assertAgentCoreProfileRecoveryBinding({ snapshot: nativeExecution.provider.agentCoreProfile, stored: recoveryProfile, @@ -599,7 +603,7 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele const interactionResponses = context[ EXTERNAL_CHAT_QUESTION_RESPONSE_KEY ] - ? await materializeExternalChatQuestionResponseInput({ + ? await measureRunPhase("runtime_selection.interactions", () => materializeExternalChatQuestionResponseInput({ db, binding: { companyId: agent.companyId, @@ -608,8 +612,8 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele agentId: agent.id, }, contextSnapshot: context, - }) - : await materializeNativeInteractionResponses({ + })) + : await measureRunPhase("runtime_selection.interactions", () => materializeNativeInteractionResponses({ db, companyId: agent.companyId, issueId: issueRef.id, @@ -627,27 +631,27 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele : interactionId ? [interactionId] : [], - }); + })); const runnerAdapterConfig = parseObject(agent.adapterConfig); const managedProfile = nativeRuntimeResolution.profile.backend === "claude_managed_agents_api" - ? await managedAgentProfileService(db).requireQualified( + ? await measureRunPhase("runtime_selection.profile", () => managedAgentProfileService(db).requireQualified( agent.companyId, readNonEmptyString(runnerAdapterConfig.managedProfileId) ?? "", - ) + )) : null; const agentCoreProfile = nativeRuntimeResolution.profile.backend === "aws_agentcore_harness_api" - ? await remoteAgentProfileService(db).requireQualified( + ? await measureRunPhase("runtime_selection.profile", () => remoteAgentProfileService(db).requireQualified( agent.companyId, readNonEmptyString( runnerAdapterConfig.agentCoreProfileId, ) ?? "", "aws_bedrock_agentcore_harness", - ) + )) : null; if (managedProfile) { const rawApiKeyBinding = parseObject( @@ -827,9 +831,9 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele }, runtimeContext: nativeRuntimeContext, }); - const backendDescriptor = await describeRunnerdNativeSessionBackend(buildExecution({ + const backendDescriptor = await measureRunPhase("runtime_selection.provider_discovery", () => describeRunnerdNativeSessionBackend(buildExecution({ normalizedSessionId: nativeSessionId, resumedSession: previousNativeRun !== null, - })); + }))); const nativeExecutionWithCheckpoint = buildNativeExecutionWithCheckpoint({ previousRun: previousNativeRun, normalizedSessionId: nativeSessionId, @@ -888,14 +892,14 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele : undefined; onProviderResourceDisposition(providerResourceDispositionForRun); await beforeNativeRuntimeSelection?.(run.id); - const nativeSelected = await db.transaction(async (tx) => { - const lockedRun = await tx + const nativeSelected = await measureRunTransaction(db, "runtime_selection", async (tx) => { + const lockedRun = await measureRunPhase("runtime_selection.row_lock", () => tx .select() .from(heartbeatRuns) .where(eq(heartbeatRuns.id, run.id)) .for("update") .limit(1) - .then((rows) => rows[0] ?? null); + .then((rows) => rows[0] ?? null)); if (!lockedRun) throw new Error("native_runtime_run_missing"); // Cancellation and runtime selection serialize on this row. A // stopped preparation must never create a new native coordinator. @@ -913,13 +917,13 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele } const lockedProfile = parseObject(lockedRun.runnerProfileJson); const persistedNativeSessionId = - await prepareNativeSessionBootstrapPersistence(tx, { + await measureRunPhase("runtime_selection.bootstrap", () => prepareNativeSessionBootstrapPersistence(tx, { run: lockedRun, selectedSessionId: nativeSessionId, execution: nativeExecution!, restoringCheckpoint: nativeResumeCheckpoint !== null, - }); - await tx + })); + await measureRunPhase("runtime_selection.run_write", () => tx .update(heartbeatRuns) .set({ runtimeMode: "native", @@ -1005,8 +1009,8 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele lockedRun.nativePhaseUpdatedAt ?? new Date(), updatedAt: new Date(), }) - .where(eq(heartbeatRuns.id, run.id)); - await tx + .where(eq(heartbeatRuns.id, run.id))); + await measureRunPhase("runtime_selection.finalization_write", () => tx .insert(nativeRunFinalizations) .values({ runId: run.id, @@ -1014,12 +1018,12 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele issueId: issueRef.id, phase: "observed", }) - .onConflictDoNothing(); + .onConflictDoNothing()); return true; }); if (!nativeSelected) return { selected: false }; stopControllerLease(); - nativeWorkspaceSync = await prepareNativeWorkspaceSync({ + nativeWorkspaceSync = await measureRunPhase("runtime_selection.workspace_sync", () => prepareNativeWorkspaceSync({ db, runId: run.id, companyId: agent.companyId, @@ -1032,7 +1036,7 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele resourceDisposition: providerResourceDispositionForRun, onCheckpoint: input.workspace.onCheckpoint, onPhase: input.workspace.onPhase, - }); + })); } else { const legacyWarmLifecycle = executionTarget?.kind === "remote" && @@ -1047,7 +1051,7 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele providerResourceDispositionForRun = "keep_running"; onProviderResourceDisposition(providerResourceDispositionForRun); } - await db + await measureRunPhase("runtime_selection.run_write", () => db .update(heartbeatRuns) .set({ runtimeMode: "legacy", @@ -1065,7 +1069,7 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele else '{}'::jsonb end) || ${JSON.stringify(providerTraceRequested ? { providerTrace: { mode: "raw", traceId: providerTraceCapture?.metadata.id ?? null, maxBytes: PROVIDER_TRACE_MAX_BYTES } } : {})}::jsonb`, updatedAt: new Date(), }) - .where(eq(heartbeatRuns.id, run.id)); + .where(eq(heartbeatRuns.id, run.id))); } return { selected: true, nativeExecution, nativeRunnerInstanceId, getNativeFreshSessionHandoff, nativeWorkspaceSync }; } diff --git a/server/src/services/issues.ts b/server/src/services/issues.ts index b8eb1e51c1f..c2415b178dc 100644 --- a/server/src/services/issues.ts +++ b/server/src/services/issues.ts @@ -1,3 +1,5 @@ +import { measureRunTransaction } from "./execution-transaction-timing.js"; +import { measureRunPhase } from "./execution-preparation-timing.js"; import { automaticReadRecoveryCleanupCondition, touchedByUserCondition } from "./issue-participation.js"; import { HUMAN_TASK_ORIGIN_KINDS } from "@paperclipai/shared"; import { resolveProjectDefaultWorkspaceId } from "./project-default-workspace.js"; @@ -12439,16 +12441,16 @@ export function issueService(db: Db) { ): Promise { if (dbOrTx === db && (actor.runId || actor.userId)) { const append = () => - db.transaction(async (tx) => { + measureRunTransaction(db, "response", async (tx) => { // Serialize run-authored comments on the issue so a provider retry // cannot publish the same visible result twice. This needs no schema // change: the issue row is the transaction fence, and the recursive // call below performs the lookup and insert while holding it. - await tx + await measureRunPhase("response.row_lock", () => tx .select({ id: issues.id }) .from(issues) .where(eq(issues.id, issueId)) - .for("update"); + .for("update")); return addComment(issueId, body, actor, options, tx); }); return options?.authorizationReason === @@ -12710,7 +12712,7 @@ export function issueService(db: Db) { } } if (!comment) { - [comment] = await dbOrTx + [comment] = await measureRunPhase("response.write", () => dbOrTx .insert(issueComments) .values({ companyId: issue.companyId, @@ -12730,7 +12732,7 @@ export function issueService(db: Db) { ? { updatedAt: currentInsertTimestamp } : {}), }) - .returning(); + .returning()); } if (!comment) throw new Error("Failed to create issue comment"); if (options?.completionReply && actor.agentId && createdByRunId) await acknowledgeChatCompletionReply(dbOrTx, createdByRunId, comment.id); @@ -12983,7 +12985,7 @@ export function issueService(db: Db) { ); const publicationCreatedAt = new Date(); for (const binding of bindings) { - await dbOrTx.transaction(async (tx: DbTransaction) => { + await measureRunPhase("response.publish", () => dbOrTx.transaction(async (tx: DbTransaction) => { await notifyChatPublicationWork(tx); return tx .insert(chatPublications) @@ -13004,12 +13006,12 @@ export function issueService(db: Db) { updatedAt: publicationCreatedAt, }) .onConflictDoNothing(); - }); + })); for (const [index, attachment] of boundAttachments.entries()) { const attachmentCreatedAt = new Date( publicationCreatedAt.getTime() + index + 1, ); - await dbOrTx.transaction(async (tx: DbTransaction) => { + await measureRunPhase("response.publish", () => dbOrTx.transaction(async (tx: DbTransaction) => { await notifyChatPublicationWork(tx); return tx .insert(chatPublications) @@ -13031,7 +13033,7 @@ export function issueService(db: Db) { updatedAt: attachmentCreatedAt, }) .onConflictDoNothing(); - }); + })); } } } diff --git a/server/src/services/native-runtime/native-run-trace-context.test.ts b/server/src/services/native-runtime/native-run-trace-context.test.ts index 6277ba82e61..171f1209994 100644 --- a/server/src/services/native-runtime/native-run-trace-context.test.ts +++ b/server/src/services/native-runtime/native-run-trace-context.test.ts @@ -1,9 +1,12 @@ import { context, trace, TraceFlags } from "@opentelemetry/api"; import { AsyncLocalStorageContextManager } from "@opentelemetry/context-async-hooks"; import { AlwaysOffSampler, ParentBasedSampler, TracerProvider, InMemorySpanExporter, SimpleSpanProcessor } from "@opentelemetry/sdk-trace"; -import { expect, it, vi } from "vitest"; +import { afterEach, beforeEach, expect, it, vi } from "vitest"; import { createNativeRunTrace } from "./native-run-trace.js"; +beforeEach(() => vi.stubEnv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://collector.test")); +afterEach(() => vi.unstubAllEnvs()); + it.each([TraceFlags.NONE, TraceFlags.SAMPLED])("exports a complete native run independently of trigger flags %i", async traceFlags => { const manager = new AsyncLocalStorageContextManager().enable(); const exporter = new InMemorySpanExporter(); diff --git a/server/src/services/native-runtime/native-run-trace.test.ts b/server/src/services/native-runtime/native-run-trace.test.ts index a5554167c36..2077e64a2ce 100644 --- a/server/src/services/native-runtime/native-run-trace.test.ts +++ b/server/src/services/native-runtime/native-run-trace.test.ts @@ -444,3 +444,29 @@ describe("native runner performance trace", () => { ).toBe(true); }); }); + +it.each(["start", "context", "attribute", "status", "end"])("keeps execution and cleanup independent of a throwing %s tracer method", async broken => { + const failure = new Error("original private execution failure"); + const spans: Array<{ end: ReturnType }> = []; + const traceContext: StartupTraceContextHandle = { + tracer: { startSpan() { + if (broken === "start") throw new Error("tracer failed"); + const span = { + setAttribute() { if (broken === "attribute") throw new Error("attribute failed"); }, + setStatus() { if (broken === "status") throw new Error("status failed"); }, + end: vi.fn(() => { if (broken === "end") throw new Error("end failed"); }), + }; + spans.push(span); + return span; + } }, + contextWithSpan() { if (broken === "context") throw new Error("context failed"); return undefined; }, + }; + const native = createNativeRunTrace({ runId: "throwing-tracer", traceContext }); + const result = {}, work = vi.fn(async () => result), failedWork = vi.fn(async () => { throw failure; }); + await expect(native.measure("runner.turn.submit", work)).resolves.toBe(result); + await expect(native.measure("agent.turn", failedWork)).rejects.toBe(failure); + await expect(native.finish("failed")).resolves.toBeUndefined(); + expect(work).toHaveBeenCalledOnce(); + expect(failedWork).toHaveBeenCalledOnce(); + for (const span of spans) expect(span.end).toHaveBeenCalledOnce(); +}); diff --git a/server/src/services/native-runtime/native-run-trace.ts b/server/src/services/native-runtime/native-run-trace.ts index 83080774d83..8594de2d31b 100644 --- a/server/src/services/native-runtime/native-run-trace.ts +++ b/server/src/services/native-runtime/native-run-trace.ts @@ -1,3 +1,4 @@ +import { getRunExecutionTraceContext } from "../execution-preparation-timing.js"; import { AsyncLocalStorage } from "node:async_hooks"; import { createHash } from "node:crypto"; import { isSpanContextValid, ROOT_CONTEXT, trace } from "@opentelemetry/api"; @@ -211,7 +212,10 @@ export function createNativeRunTrace(input: { const runIdHash = hashedId(input.runId); let rootSpan: SpanHandle; try { - const trigger = trace.getActiveSpan()?.spanContext(); + const owned = getRunExecutionTraceContext(input.runId); + const trigger = process.env.OTEL_EXPORTER_OTLP_ENDPOINT?.trim() + ? (owned ? trace.getSpan(owned) : trace.getActiveSpan())?.spanContext() + : undefined; rootSpan = tracing.tracer.startSpan("task.provider_session", { startTime: traceStartedAtMs, attributes: { @@ -341,7 +345,7 @@ export function createNativeRunTrace(input: { ), }, }, - parent.context, + parent.context ?? ROOT_CONTEXT, ); } catch { span = NOOP_SPAN; @@ -407,12 +411,8 @@ export function createNativeRunTrace(input: { duration_ms: durationMs, outcome, }); - try { - if (outcome === "failed") state.span.setStatus({ code: 2 }); - state.span.end(endedAtMs); - } catch { - // Tracing is diagnostic-only and must never change runner control flow. - } + if (outcome === "failed") { try { state.span.setStatus({ code: 2 }); } catch {} } + try { state.span.end(endedAtMs); } catch {} await emitCompletedSpanEvent({ name: state.name, parentName: state.parentName, @@ -497,17 +497,10 @@ export function createNativeRunTrace(input: { for (const scope of [...openScopes].reverse()) { await end(scope, { endedAtMs, outcome }); } - try { - rootSpan.setAttribute( - "paperclip.task.run.wall_ms", - endedAtMs - traceStartedAtMs, - ); - rootSpan.setAttribute("paperclip.task.run.outcome", outcome); - if (outcome === "failed") rootSpan.setStatus({ code: 2 }); - rootSpan.end(endedAtMs); - } catch { - // Tracing is diagnostic-only and must never change runner control flow. - } + try { rootSpan.setAttribute("paperclip.task.run.wall_ms", endedAtMs - traceStartedAtMs); } catch {} + try { rootSpan.setAttribute("paperclip.task.run.outcome", outcome); } catch {} + if (outcome === "failed") { try { rootSpan.setStatus({ code: 2 }); } catch {} } + try { rootSpan.end(endedAtMs); } catch {} // Preserve the historical run-log record without emitting a duplicate // full-width child span into the OpenTelemetry waterfall. await emitCompletedSpanEvent({ diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index 9e88e5b6314..24d5198972f 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -11747,13 +11747,14 @@ async function createRunnerdBackendWithinSessionClaim( stagedRemoteProviderPackRoot ) { const packSource = await prepareVerifiedRemoteProviderPack({ + trace: input.trace, verifyStaged: () => measureNativeRunnerSpan( input.trace, "provider_pack.verify", () => verifyRemoteProviderPack(stagedRemoteProviderPackRoot), ), usePreinstalled: async () => { - let preinstalledProviderPack = await discoverPreinstalledProviderPack(); + let preinstalledProviderPack = await measureNativeRunnerSpan(input.trace, "provider_pack.discover", discoverPreinstalledProviderPack); if (preinstalledProviderPack) { try { await measureNativeRunnerSpan( @@ -11772,7 +11773,7 @@ async function createRunnerdBackendWithinSessionClaim( const escapedParent = posix .dirname(stagedRemoteProviderPackRoot) .replaceAll("'", "'\\''"); - const linked = await remoteCommandRunner.execute({ + const linked = await measureNativeRunnerSpan(input.trace, "provider_pack.link_preinstalled", () => remoteCommandRunner.execute({ command: "sh", args: [ "-c", @@ -11781,7 +11782,7 @@ async function createRunnerdBackendWithinSessionClaim( cwd: remoteTarget.remoteCwd, bypassSession: true, timeoutMs: 10_000, - }); + })); if (linked.exitCode !== 0 || linked.timedOut) { throw new Error( "runner_remote_provider_artifact_incompatible: preinstalled provider pack could not be linked", @@ -11811,26 +11812,26 @@ async function createRunnerdBackendWithinSessionClaim( "'", "'\\''", ); - const cleared = await remoteCommandRunner.execute({ + const cleared = await measureNativeRunnerSpan(input.trace, "provider_pack.clear_stale", () => remoteCommandRunner.execute({ command: "sh", args: ["-c", `rm -rf '${escapedPackRoot}'`], cwd: remoteTarget.remoteCwd, bypassSession: true, timeoutMs: 10_000, - }); + })); if (cleared.exitCode !== 0 || cleared.timedOut) { throw new Error( "runner_remote_provider_artifact_incompatible: stale provider pack could not be replaced", ); } - await stageRemoteRunnerDirectory({ + await measureNativeRunnerSpan(input.trace, "provider_pack.upload", () => stageRemoteRunnerDirectory({ target: remoteTarget, runner: remoteCommandRunner, sourcePath: configuredProviderPackRoot, targetPath: stagedRemoteProviderPackRoot, mode: 0o700, - }); - await measureNativeRunnerSpan(input.trace, "provider_pack.verify", () => + })); + await measureNativeRunnerSpan(input.trace, "provider_pack.verify_uploaded", () => verifyRemoteProviderPack(stagedRemoteProviderPackRoot), ); activeRemoteProviderPackRoot = stagedRemoteProviderPackRoot; @@ -12096,7 +12097,7 @@ async function createRunnerdBackendWithinSessionClaim( }, }, ); - const restored = await inspectRemoteHarnessState(); + const restored = await measureNativeRunnerSpan(input.trace, "harness_state.inspect", () => inspectRemoteHarnessState()); if ( !restored.complete || !restored.runnerState || @@ -12283,7 +12284,7 @@ async function createRunnerdBackendWithinSessionClaim( ); } else { const reuseStartedAtMs = Date.now(); - const state = await inspectRemoteHarnessState(); + const state = await measureNativeRunnerSpan(input.trace, "harness_state.inspect", () => inspectRemoteHarnessState()); if ( state.complete && state.runnerState && @@ -12312,7 +12313,7 @@ async function createRunnerdBackendWithinSessionClaim( if ( remoteDirectory && existsSync(localDirectory) && - !(await remoteRunnerPathExists({ + !(await measureNativeRunnerSpan(input.trace, "harness_state.path_check", () => remoteRunnerPathExists({ runner: remoteCommandRunner, path: directory.name === "runner" @@ -12320,7 +12321,7 @@ async function createRunnerdBackendWithinSessionClaim( : remoteDirectory, kind: directory.name === "runner" ? "file" : "directory", - })) + }))) ) { await stageRemoteRunnerDirectory({ target: remoteTarget, @@ -12352,7 +12353,7 @@ async function createRunnerdBackendWithinSessionClaim( // this invocation's isolated Codex auth/config. The later launch must // still stage those fresh files even when history was already restored. // Launch material is never recovered from a failover backup. - await materializeRemoteHarnessLaunchState(); + await measureNativeRunnerSpan(input.trace, "harness_state.launch_material", () => materializeRemoteHarnessLaunchState()); if ( remoteTarget && remoteCommandRunner && @@ -12440,14 +12441,14 @@ async function createRunnerdBackendWithinSessionClaim( // `runner_harness_state_mismatch`. Only checkpoint a harness that runnerd // has proved complete. A malformed or identity-conflicting state still // throws from inspectRemoteHarnessState and therefore fails closed. - let checkpointable = await inspectRemoteHarnessState(); + let checkpointable = await measureNativeRunnerSpan(input.trace, "harness_state.inspect", () => inspectRemoteHarnessState()); // The remote runner writes its suspended lifecycle and provider state // before the outer process-owner RPC necessarily observes completion. // Allow a very small bounded visibility window without ever accepting an // active, incomplete, malformed, or identity-conflicting checkpoint. for (let attempt = 1; !checkpointable.complete && attempt < 3; attempt++) { await new Promise((resolve) => setTimeout(resolve, 100)); - checkpointable = await inspectRemoteHarnessState(); + checkpointable = await measureNativeRunnerSpan(input.trace, "harness_state.inspect", () => inspectRemoteHarnessState()); } if (!checkpointable.complete) { const incompleteFailure = remoteCheckpointIncompleteFailure( @@ -12477,7 +12478,7 @@ async function createRunnerdBackendWithinSessionClaim( input.trace, "harness_state.backup.persist", async () => { - const verified = await inspectRemoteHarnessState(); + const verified = await measureNativeRunnerSpan(input.trace, "harness_state.inspect", () => inspectRemoteHarnessState()); if ( !verified.complete || !verified.runnerState || diff --git a/server/src/services/native-runtime/native-session-resume.test.ts b/server/src/services/native-runtime/native-session-resume.test.ts index b0ba4e194f6..ced2d0f819b 100644 --- a/server/src/services/native-runtime/native-session-resume.test.ts +++ b/server/src/services/native-runtime/native-session-resume.test.ts @@ -268,9 +268,9 @@ it("wires exact-session recovery and guarded selected identity into heartbeat pe new URL("../heartbeat/runtime-selection.ts", import.meta.url), "utf8", ); - expect(source).toContain("await findNativeSessionResumeRun(db,"); + expect(source).toContain("findNativeSessionResumeRun(db,"); expect(source).toContain( - "await prepareNativeSessionBootstrapPersistence(tx,", + "prepareNativeSessionBootstrapPersistence(tx,", ); expect(source).toContain("nativeSessionId: persistedNativeSessionId,"); expect(source).toContain("lockedProfile.sessionCheckpoint != null"); diff --git a/server/src/services/native-runtime/native-workspace-finalization-ownership.ts b/server/src/services/native-runtime/native-workspace-finalization-ownership.ts index 9d39705f5b5..887300d7406 100644 --- a/server/src/services/native-runtime/native-workspace-finalization-ownership.ts +++ b/server/src/services/native-runtime/native-workspace-finalization-ownership.ts @@ -1,3 +1,4 @@ +import { beginRunPhase } from "../execution-preparation-timing.js"; import { logActivity } from "../activity-log.js"; import { issueRecoveryActionService } from "../issue-recovery-actions.js"; import { nativeSha256 } from "./canonical.js"; @@ -70,6 +71,7 @@ export async function withNativeWorkspaceFinalizationOwnership( const owned = and(scope, sql`${heartbeatRuns.runnerProfileJson}->${OWNER_KEY}->>'token' = ${owner.token}`); let claimed = false; let actionPromise: Promise | undefined; + const finishAcquisition = beginRunPhase("execution.workspace_ownership_wait"); try { return await withDedicatedDbConnection(input.db, (dedicated) => dedicated.transaction(async (lock) => { const rows = await lock.execute(sql`select pg_try_advisory_xact_lock(hashtextextended(${`native-workspace-finalization:${input.companyId}:${input.runId}`}, 0)) as acquired, @@ -213,13 +215,18 @@ export async function withNativeWorkspaceFinalizationOwnership( eq(issueRecoveryActions.fingerprint, nativeSha256({ runId: input.runId, owner: rawPrior })), inArray(issueRecoveryActions.status, ["active", "escalated"]), )); + finishAcquisition(); actionPromise = action(ownership); const value = await actionPromise; await ownership.assertHeld(); return { acquired: true, value } as const; }); })); + } catch (error) { + finishAcquisition(true); + throw error; } finally { + finishAcquisition(); // postgres.js may reject the transaction when the socket dies before the // callback settles. Join that callback before releasing its durable fence. await actionPromise?.catch(() => undefined); diff --git a/server/src/services/native-runtime/native-workspace-sync.ts b/server/src/services/native-runtime/native-workspace-sync.ts index cb8ea5be2e2..8c1399b64fc 100644 --- a/server/src/services/native-runtime/native-workspace-sync.ts +++ b/server/src/services/native-runtime/native-workspace-sync.ts @@ -1,3 +1,4 @@ +import { measureRunPhase } from "../execution-preparation-timing.js"; import type { WorkspaceCheckpointMetrics } from "@paperclipai/adapter-utils/workspace-checkpoint"; import type { RuntimeSpanRunner } from "@paperclipai/adapter-utils/acpx-engine/startup-timing"; import fs from "node:fs/promises"; @@ -826,38 +827,40 @@ async function finalizePreparedRuntime(input: { }): Promise { if (input.onPhase) await input.onPhase("workspace.restore", () => input.runtime.restoreWorkspace()); else await input.runtime.restoreWorkspace(); - await input.assertOwnership?.(); - const finalSnapshot = - await import("@paperclipai/adapter-utils/workspace-restore-merge").then( - ({ captureDirectorySnapshotOffThread: captureDirectorySnapshot }) => - captureDirectorySnapshot(input.descriptor.binding.localCwd, { - exclude: input.runtime.workspaceSyncSnapshot?.baseline.exclude ?? [], - ignoredPaths: input.runtime.workspaceSyncSnapshot?.baseline.ignoredPaths, - diskBacked: true, - }), - ); - const finalHostSha256 = directorySnapshotSha256(finalSnapshot); - await disposeDirectorySnapshot(finalSnapshot); - const stamp = finalizedWorkspaceStamp({ - descriptor: input.descriptor, - hostSha256: finalHostSha256, + return measureRunPhase("execution.workspace_commit", async () => { + await input.assertOwnership?.(); + const finalSnapshot = + await import("@paperclipai/adapter-utils/workspace-restore-merge").then( + ({ captureDirectorySnapshotOffThread: captureDirectorySnapshot }) => + captureDirectorySnapshot(input.descriptor.binding.localCwd, { + exclude: input.runtime.workspaceSyncSnapshot?.baseline.exclude ?? [], + ignoredPaths: input.runtime.workspaceSyncSnapshot?.baseline.ignoredPaths, + diskBacked: true, + }), + ); + const finalHostSha256 = directorySnapshotSha256(finalSnapshot); + await disposeDirectorySnapshot(finalSnapshot); + const stamp = finalizedWorkspaceStamp({ + descriptor: input.descriptor, + hostSha256: finalHostSha256, + }); + await writeRemoteStamp({ target: input.target, stamp }); + const finalizedDescriptor: NativeWorkspaceSyncDescriptor = { + ...input.descriptor, + state: "finalized", + finalizedAt: new Date().toISOString(), + finalHostSha256, + }; + await input.assertOwnership?.(); + const reference = await writeDescriptor(finalizedDescriptor); + await persistRunReference(input.db, input.runId, reference); + await persistLeaseStamp({ + db: input.db, + leaseId: input.descriptor.binding.leaseId, + stamp, + }); + return reference; }); - await writeRemoteStamp({ target: input.target, stamp }); - const finalizedDescriptor: NativeWorkspaceSyncDescriptor = { - ...input.descriptor, - state: "finalized", - finalizedAt: new Date().toISOString(), - finalHostSha256, - }; - await input.assertOwnership?.(); - const reference = await writeDescriptor(finalizedDescriptor); - await persistRunReference(input.db, input.runId, reference); - await persistLeaseStamp({ - db: input.db, - leaseId: input.descriptor.binding.leaseId, - stamp, - }); - return reference; } export async function prepareNativeWorkspaceSync(input: { diff --git a/server/src/services/native-runtime/remote-provider-pack.ts b/server/src/services/native-runtime/remote-provider-pack.ts index adf8459060e..2a01b380f33 100644 --- a/server/src/services/native-runtime/remote-provider-pack.ts +++ b/server/src/services/native-runtime/remote-provider-pack.ts @@ -1,16 +1,32 @@ +import type { NativeRunTrace } from "./native-run-trace.js"; + /** Reuse only a pack that passes the same full verification as a new upload. */ export async function prepareVerifiedRemoteProviderPack(input: { + trace?: NativeRunTrace; verifyStaged: () => Promise; usePreinstalled: () => Promise; stageAndVerify: () => Promise; }): Promise<"staged" | "preinstalled" | "uploaded"> { + const scope = input.trace?.start("provider_pack.select"); + let failed = false; + const select = async () => { + try { + await input.verifyStaged(); + return "staged" as const; + } catch {} + if (await input.usePreinstalled()) return "preinstalled" as const; + await input.stageAndVerify(); + return "uploaded" as const; + }; try { - await input.verifyStaged(); - return "staged"; - } catch { - // Missing, stale, or modified packs are never executable cache hits. + const source = scope ? await input.trace!.run(scope, select) : await select(); + if (scope) input.trace!.annotate(scope, { stateSource: source, + reason: source === "uploaded" ? "cached_packs_unverified" : "verified_pack_reused" }); + return source; + } catch (error) { + failed = true; + throw error; + } finally { + if (scope) await input.trace!.end(scope, { outcome: failed ? "failed" : "ok" }); } - if (await input.usePreinstalled()) return "preinstalled"; - await input.stageAndVerify(); - return "uploaded"; } diff --git a/server/src/services/native-runtime/runtime-context.test.ts b/server/src/services/native-runtime/runtime-context.test.ts index dcacd4e5010..9b14540b327 100644 --- a/server/src/services/native-runtime/runtime-context.test.ts +++ b/server/src/services/native-runtime/runtime-context.test.ts @@ -37,6 +37,10 @@ vi.mock("../tool-access.js", () => ({ })); import { createHash } from "node:crypto"; +import { context as otelContext, trace } from "@opentelemetry/api"; +import { AsyncLocalStorageContextManager } from "@opentelemetry/context-async-hooks"; +import { InMemorySpanExporter, SimpleSpanProcessor, TracerProvider } from "@opentelemetry/sdk-trace"; +import { withRunExecutionTiming } from "../execution-preparation-timing.js"; import { buildNativeRuntimeContext, resolveNativeRuntimeMcpSnapshot } from "./runtime-context.js"; const temporaryRoots: string[] = []; @@ -420,3 +424,53 @@ it("pins a voice tool only when the current requester can call through that assi expect((await resolveNativeRuntimeMcpSnapshot(input)).bindingId).toBe("native-mcp:run-1"); expect(serviceMocks.spekoToolsForSession).toHaveBeenLastCalledWith(db, { companyId: "company-1", agentId: "agent-1", runId: "run-1", issueId: "task-1", identityContextId: "identity-1" }); }); + +it("keeps instructions, selected skills, and MCP concurrent and distinguishes cold materialization from warm verification", async () => { + vi.stubEnv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://collector.test"); + const exporter = new InMemorySpanExporter(); + const provider = new TracerProvider({ spanProcessors: [new SimpleSpanProcessor({ exporter })] }); + const manager = new AsyncLocalStorageContextManager().enable(); + expect(otelContext.setGlobalContextManager(manager)).toBe(true); + expect(trace.setGlobalTracerProvider(provider)).toBe(true); + let release!: () => void; + const gate = new Promise(resolve => { release = resolve; }); + serviceMocks.exportFiles.mockImplementation(async () => { + await gate; + return { entryFile: "AGENTS.md", files: { "AGENTS.md": "Private fixture instructions" } }; + }); + const source = path.join(temporaryRoots.at(-1)!, "selected-skill"); + await mkdir(source); + await writeFile(path.join(source, "SKILL.md"), "Private skill fixture"); + const input = { + db: {} as Db, agent: { id: "agent-1", companyId: "company-1", name: "Reviewer", adapterConfig: {} }, + runId: "concurrent-context", runtimeConfig: { paperclipSkillSync: { desiredSkills: ["selected"] } }, + runtimeSkillEntries: [{ key: "selected", runtimeName: "selected", source }], + }; + const pending = withRunExecutionTiming(input.runId, () => buildNativeRuntimeContext(input)); + try { + await vi.waitFor(() => { + expect(exporter.getFinishedSpans().map(span => span.name)).toEqual(expect.arrayContaining(["runtime_context.skills", "runtime_context.mcp"])); + }); + expect(exporter.getFinishedSpans().some(span => span.name === "runtime_context.instructions")).toBe(false); + release(); + const cold = await pending; + expect(cold.skills).toHaveLength(1); + expect(exporter.getFinishedSpans().some(span => span.name === "runtime_context.asset_materialize")).toBe(true); + exporter.reset(); + const warm = await withRunExecutionTiming(input.runId, () => buildNativeRuntimeContext(input)); + expect(warm.aggregateDigest).toBe(cold.aggregateDigest); + const spans = exporter.getFinishedSpans(); + expect(spans.some(span => span.name === "runtime_context.asset_verify")).toBe(true); + expect(spans.some(span => span.name === "runtime_context.asset_materialize")).toBe(false); + expect(JSON.stringify(spans.map(span => span.attributes))).not.toContain("Private"); + const siblings = spans.filter(span => ["runtime_context.instructions", "runtime_context.skills", "runtime_context.mcp"].includes(span.name)); + expect(siblings).toHaveLength(3); + expect(new Set(siblings.map(span => span.parentSpanContext?.spanId)).size).toBe(1); + } finally { + release(); + await pending.catch(() => undefined); + trace.disable(); otelContext.disable(); manager.disable(); + await provider.shutdown(); + vi.unstubAllEnvs(); + } +}); diff --git a/server/src/services/native-runtime/runtime-context.ts b/server/src/services/native-runtime/runtime-context.ts index f6a6e184ee2..03693fdbe94 100644 --- a/server/src/services/native-runtime/runtime-context.ts +++ b/server/src/services/native-runtime/runtime-context.ts @@ -1,3 +1,4 @@ +import { measureRunPhase } from "../execution-preparation-timing.js"; import { spekoToolsForSession } from "../voice/speko-agent-tools.js"; import { githubBotConnectionIdsForRun } from "../chat-github-tools.js"; import { isBrowserUseConnection } from "../browser-use-client.js"; @@ -80,7 +81,7 @@ async function verifyMaterializedAsset( rootPath: string, manifestFiles: Array<{ path: string; sha256: string; mode: number; size: number }>, ): Promise { - const actual = await collectDirectoryFiles(rootPath); + const actual = await measureRunPhase("runtime_context.asset_read", () => collectDirectoryFiles(rootPath)); const actualByPath = new Map(actual.map((file) => [file.path, file])); if (actualByPath.size !== manifestFiles.length) throw new Error("runtime context asset file count mismatch"); for (const expected of manifestFiles) { @@ -92,59 +93,64 @@ async function verifyMaterializedAsset( } export async function materializeAsset(files: AssetFile[]): Promise { - const sorted = [...files].sort((a, b) => a.path.localeCompare(b.path)); - const manifestFiles = sorted.map((file) => ({ path: safeRelativePath(file.path, "runtime context path"), sha256: sha256(file.content), mode: file.mode & 0o555, size: file.content.byteLength })); - const totalBytes = manifestFiles.reduce((sum, file) => sum + file.size, 0); - if (!manifestFiles.length || manifestFiles.length > MAX_ASSET_FILES || totalBytes > MAX_ASSET_BYTES) throw new Error("runtime context bundle is empty or exceeds its bound"); - const assetDigest = sha256(JSON.stringify(manifestFiles)); - const manifestText = `${JSON.stringify({ schema: "paperclip.runtime-asset-manifest.v1", digest: assetDigest, fileCount: manifestFiles.length, totalBytes, files: manifestFiles })}\n`; - const manifestDigest = sha256(manifestText); + const { sorted, manifestFiles, totalBytes, assetDigest, manifestText, manifestDigest } = await measureRunPhase("runtime_context.asset_manifest", () => { + const sorted = [...files].sort((a, b) => a.path.localeCompare(b.path)); + const manifestFiles = sorted.map((file) => ({ path: safeRelativePath(file.path, "runtime context path"), sha256: sha256(file.content), mode: file.mode & 0o555, size: file.content.byteLength })); + const totalBytes = manifestFiles.reduce((sum, file) => sum + file.size, 0); + if (!manifestFiles.length || manifestFiles.length > MAX_ASSET_FILES || totalBytes > MAX_ASSET_BYTES) throw new Error("runtime context bundle is empty or exceeds its bound"); + const assetDigest = sha256(JSON.stringify(manifestFiles)); + const manifestText = `${JSON.stringify({ schema: "paperclip.runtime-asset-manifest.v1", digest: assetDigest, fileCount: manifestFiles.length, totalBytes, files: manifestFiles })}\n`; + const manifestDigest = sha256(manifestText); + return { sorted, manifestFiles, totalBytes, assetDigest, manifestText, manifestDigest }; + }); const assetsRoot = path.join(resolvePaperclipInstanceRoot(), "runtime-context-assets"); const rootPath = path.join(assetsRoot, "bundles", assetDigest); const manifestPath = path.join(assetsRoot, "manifests", `${assetDigest}.json`); - const existing = await fs.readFile(manifestPath, "utf8").catch(() => null); + const existing = await measureRunPhase("runtime_context.asset_cache", () => fs.readFile(manifestPath, "utf8").catch(() => null)); if (existing !== null) { if (sha256(existing) !== manifestDigest || !(await fs.stat(rootPath).catch(() => null))?.isDirectory()) throw new Error(`runtime context asset verification failed: ${assetDigest}`); - await verifyMaterializedAsset(rootPath, manifestFiles); + await measureRunPhase("runtime_context.asset_verify", () => verifyMaterializedAsset(rootPath, manifestFiles), { fileCount: manifestFiles.length, bytes: totalBytes }); return { schema: NATIVE_RUNTIME_ASSET_SCHEMA, digest: assetDigest, manifestDigest, rootPath, fileCount: manifestFiles.length, totalBytes }; } - const stagingRoot = path.join(assetsRoot, ".staging", randomUUID()); - await fs.mkdir(stagingRoot, { recursive: true, mode: 0o700 }); - try { - for (const file of sorted) { - const target = path.join(stagingRoot, file.path); - await fs.mkdir(path.dirname(target), { recursive: true, mode: 0o700 }); - await fs.writeFile(target, file.content, { mode: file.mode & 0o555 }); - await fs.chmod(target, file.mode & 0o555); + await measureRunPhase("runtime_context.asset_materialize", async () => { + const stagingRoot = path.join(assetsRoot, ".staging", randomUUID()); + await fs.mkdir(stagingRoot, { recursive: true, mode: 0o700 }); + try { + for (const file of sorted) { + const target = path.join(stagingRoot, file.path); + await fs.mkdir(path.dirname(target), { recursive: true, mode: 0o700 }); + await fs.writeFile(target, file.content, { mode: file.mode & 0o555 }); + await fs.chmod(target, file.mode & 0o555); + } + await fs.mkdir(path.dirname(rootPath), { recursive: true, mode: 0o700 }); + let publishedRoot = false; + await fs.rename(stagingRoot, rootPath).then(() => { + publishedRoot = true; + }).catch(async (error: NodeJS.ErrnoException) => { + // Concurrent publishers can surface EACCES on macOS when the winning + // destination has already been made read-only. Treat that as a race only + // when the content-addressed destination now exists; a genuine permission + // failure with no published destination must still fail loudly. + const publishedByPeer = (await fs.stat(rootPath).catch(() => null))?.isDirectory() === true; + if (error.code !== "EEXIST" && error.code !== "ENOTEMPTY" && !publishedByPeer) throw error; + await fs.rm(stagingRoot, { recursive: true, force: true }); + }); + // Publish the writable staging directory first. Renaming a read-only + // directory can fail with EACCES on macOS even when both parents are owned + // and writable. Files are already immutable; directories become immutable + // immediately after this process wins publication. + if (publishedRoot) await makeDirectoriesReadOnly(rootPath); + await fs.mkdir(path.dirname(manifestPath), { recursive: true, mode: 0o700 }); + await fs.writeFile(manifestPath, manifestText, { flag: "wx", mode: 0o400 }).catch(async (error: NodeJS.ErrnoException) => { + if (error.code !== "EEXIST") throw error; + if (sha256(await measureRunPhase("runtime_context.asset_cache", () => fs.readFile(manifestPath, "utf8"))) !== manifestDigest) throw new Error(`runtime context manifest race mismatch: ${assetDigest}`); + }); + await measureRunPhase("runtime_context.asset_verify", () => verifyMaterializedAsset(rootPath, manifestFiles), { fileCount: manifestFiles.length, bytes: totalBytes }); + } catch (error) { + await fs.rm(stagingRoot, { recursive: true, force: true }).catch(() => undefined); + throw error; } - await fs.mkdir(path.dirname(rootPath), { recursive: true, mode: 0o700 }); - let publishedRoot = false; - await fs.rename(stagingRoot, rootPath).then(() => { - publishedRoot = true; - }).catch(async (error: NodeJS.ErrnoException) => { - // Concurrent publishers can surface EACCES on macOS when the winning - // destination has already been made read-only. Treat that as a race only - // when the content-addressed destination now exists; a genuine permission - // failure with no published destination must still fail loudly. - const publishedByPeer = (await fs.stat(rootPath).catch(() => null))?.isDirectory() === true; - if (error.code !== "EEXIST" && error.code !== "ENOTEMPTY" && !publishedByPeer) throw error; - await fs.rm(stagingRoot, { recursive: true, force: true }); - }); - // Publish the writable staging directory first. Renaming a read-only - // directory can fail with EACCES on macOS even when both parents are owned - // and writable. Files are already immutable; directories become immutable - // immediately after this process wins publication. - if (publishedRoot) await makeDirectoriesReadOnly(rootPath); - await fs.mkdir(path.dirname(manifestPath), { recursive: true, mode: 0o700 }); - await fs.writeFile(manifestPath, manifestText, { flag: "wx", mode: 0o400 }).catch(async (error: NodeJS.ErrnoException) => { - if (error.code !== "EEXIST") throw error; - if (sha256(await fs.readFile(manifestPath, "utf8")) !== manifestDigest) throw new Error(`runtime context manifest race mismatch: ${assetDigest}`); - }); - await verifyMaterializedAsset(rootPath, manifestFiles); - } catch (error) { - await fs.rm(stagingRoot, { recursive: true, force: true }).catch(() => undefined); - throw error; - } + }, { fileCount: manifestFiles.length, bytes: totalBytes }); return { schema: NATIVE_RUNTIME_ASSET_SCHEMA, digest: assetDigest, manifestDigest, rootPath, fileCount: manifestFiles.length, totalBytes }; } @@ -181,7 +187,7 @@ async function materializeSelectedSkills(runtimeConfig: Record, const entry = byKey.get(key); if (!entry) throw new Error(`assigned runtime skill is missing from the company library: ${key}`); if (entry.sourceStatus === "missing") throw new Error(entry.missingDetail ?? `assigned runtime skill source is missing: ${key}`); - return { key: entry.key, runtimeName: safeRelativePath(entry.runtimeName, `runtime name for ${key}`), versionId: entry.versionId ?? entry.currentVersionId ?? null, bundle: await materializeAsset(await collectDirectoryFiles(entry.source)) }; + return { key: entry.key, runtimeName: safeRelativePath(entry.runtimeName, `runtime name for ${key}`), versionId: entry.versionId ?? entry.currentVersionId ?? null, bundle: await materializeAsset(await measureRunPhase("runtime_context.asset_read", () => collectDirectoryFiles(entry.source))) }; })); } @@ -246,9 +252,9 @@ export async function resolveNativeRuntimeMcpSnapshot(input: { db: Db; agent: Pi export async function buildNativeRuntimeContext(input: { db: Db; agent: RuntimeAgent; runId: string; runtimeConfig: Record; runtimeSkillEntries: PaperclipSkillEntry[]; instructionWorkingCopy?: { rootPath: string; entryPath: string; kind?: "agent_files" } }): Promise { const [instructions, skills, mcp] = await Promise.all([ - materializeInstructionBundle(input.db, input.agent, input.instructionWorkingCopy?.kind === "agent_files"), - materializeSelectedSkills(input.runtimeConfig, input.runtimeSkillEntries, input.agent.adapterType === "paperclip_runner"), - resolveNativeRuntimeMcpSnapshot({ db: input.db, agent: input.agent, runId: input.runId }), + measureRunPhase("runtime_context.instructions", () => materializeInstructionBundle(input.db, input.agent, input.instructionWorkingCopy?.kind === "agent_files")), + measureRunPhase("runtime_context.skills", () => materializeSelectedSkills(input.runtimeConfig, input.runtimeSkillEntries, input.agent.adapterType === "paperclip_runner")), + measureRunPhase("runtime_context.mcp", () => resolveNativeRuntimeMcpSnapshot({ db: input.db, agent: input.agent, runId: input.runId })), ]); const snapshot = { prompt: { revision: PAPERCLIP_EXECUTION_PROMPT_REVISION, text: PAPERCLIP_EXECUTION_PROMPT, digest: nativeRuntimePromptDigest() }, diff --git a/server/src/services/task-workspace-source-access.ts b/server/src/services/task-workspace-source-access.ts index 42752933739..933855d0ee1 100644 --- a/server/src/services/task-workspace-source-access.ts +++ b/server/src/services/task-workspace-source-access.ts @@ -1,3 +1,4 @@ +import { measureRunPhase } from "./execution-preparation-timing.js"; import { executionWorkspaces, issues, type Db } from "@paperclipai/db"; import { and, eq, inArray, or, sql } from "drizzle-orm"; import { forbidden } from "../errors.js"; @@ -12,23 +13,23 @@ export async function assertTaskWorkspaceSourceProjectAccess( db: Db | DbTransaction, actor: AuthorizationActor, companyId: string, projectId: string | null, options: WorkspaceAccessContext = {}, ) { if (!projectId) return; - const decision = await accessService(db as Db).decide({ + const decision = await measureRunPhase("workspace.access.authorization", () => accessService(db as Db).decide({ actor, action: "project:read", resource: { type: "project", companyId, projectId }, - }); + })); if (!decision.allowed) throw forbidden(decision.explanation); if (options.write) { const [task] = actor.type === "board" && options.issueId - ? await db.select({ assigneeAgentId: issues.assigneeAgentId, assigneeUserId: issues.assigneeUserId }).from(issues) - .where(and(eq(issues.id, options.issueId), eq(issues.companyId, companyId))) + ? await measureRunPhase("workspace.access.inventory", async () => db.select({ assigneeAgentId: issues.assigneeAgentId, assigneeUserId: issues.assigneeUserId }).from(issues) + .where(and(eq(issues.id, options.issueId!), eq(issues.companyId, companyId)))) : []; const assigneeAgentId = actor.type === "agent" ? actor.agentId ?? null : task?.assigneeAgentId ?? options.assigneeAgentId ?? null; const assigneeUserId = actor.type === "agent" ? null : task?.assigneeUserId ?? options.assigneeUserId ?? null; - const assignment = await accessService(db as Db).decide({ + const assignment = await measureRunPhase("workspace.access.authorization", () => accessService(db as Db).decide({ actor, action: "project:write_workspace", resource: { type: "issue", companyId, projectId, assigneeAgentId, assigneeUserId, issueId: options.issueId, parentIssueId: options.parentIssueId }, scope: { projectId, assigneeAgentId, assigneeUserId }, - }); + })); if (!assignment.allowed) throw forbidden(assignment.explanation); } } @@ -36,28 +37,28 @@ export async function assertTaskWorkspaceSourceProjectAccess( export async function assertTaskWorkspaceAccess( db: Db | DbTransaction, actor: AuthorizationActor, companyId: string, workspaceId: string, options: WorkspaceAccessContext = {}, ) { - const [workspace] = await db.select().from(executionWorkspaces).where(and( + const [workspace] = await measureRunPhase("workspace.access.inventory", async () => db.select().from(executionWorkspaces).where(and( eq(executionWorkspaces.id, workspaceId), eq(executionWorkspaces.companyId, companyId), - await executionWorkspaceReadSqlCondition(db, actor), - )); + await measureRunPhase("workspace.access.privacy", () => executionWorkspaceReadSqlCondition(db, actor)), + ))); if (!workspace) throw forbidden("Task workspace access is no longer available"); await assertTaskWorkspaceSourceProjectAccess(db, actor, companyId, workspace.projectId, { ...options, write: options.write && workspace.strategyType !== "git_worktree" }); const retainedIssueIds = Object.keys(workspace.metadata?._issuePrivacySources ?? {}); if (workspace.sourceIssueId) retainedIssueIds.push(workspace.sourceIssueId); - const sources = await db.select({ id: issues.id }).from(issues).where(and( + const sources = await measureRunPhase("workspace.access.inventory", async () => db.select({ id: issues.id }).from(issues).where(and( eq(issues.companyId, companyId), or(eq(issues.executionWorkspaceId, workspaceId), retainedIssueIds.length > 0 ? inArray(sql`${issues.id}`, retainedIssueIds.map(id => canonicalExecutionUuid(sql`${id}::text`))) : undefined), - )); + ))); const sourceIds = new Set(sources.map(source => source.id)); if (retainedIssueIds.some(id => !sourceIds.has(id))) throw forbidden("Task workspace source is no longer accessible"); const access = accessService(db as Db); if (sources.length === 0 && !workspace.projectId) { - const decision = await access.decide({ actor, action: "company_scope:read", resource: { type: "company", companyId } }); + const decision = await measureRunPhase("workspace.access.authorization", () => access.decide({ actor, action: "company_scope:read", resource: { type: "company", companyId } })); if (!decision.allowed) throw forbidden("An unscoped workspace is outside this actor's authorization boundary"); } for (const source of sources) { - const decision = await access.decide({ actor, action: "issue:read", resource: { type: "issue", companyId, issueId: source.id } }); + const decision = await measureRunPhase("workspace.access.authorization", () => access.decide({ actor, action: "issue:read", resource: { type: "issue", companyId, issueId: source.id } })); if (!decision.allowed) throw forbidden("Task workspace retains files outside this actor's authorization boundary"); } } @@ -71,51 +72,53 @@ export async function assertTaskWorkspacesReadAccess( export async function assertTaskWorkspacesReadAccessForActors( db: Db | DbTransaction, actors: AuthorizationActor[], companyId: string, workspaceIds: Iterable, ) { - const ids = [...new Set(workspaceIds)]; - if (!ids.length || !actors.length) return; - const privacyActors = [...new Map(actors.map(actor => [JSON.stringify({ ...actor, runId: undefined }), actor])).values()]; - const privacy = and(...await Promise.all(privacyActors.map(actor => executionWorkspaceReadSqlCondition(db, actor)))); - const projectIds = new Set(), sourceIds = new Set(); - let needsCompanyRead = false; - for (let offset = 0; offset < ids.length; offset += 500) { - const batch = ids.slice(offset, offset + 500); - const workspaces = await db.select({ id: executionWorkspaces.id, projectId: executionWorkspaces.projectId, - sourceIssueId: executionWorkspaces.sourceIssueId, metadata: executionWorkspaces.metadata, - }).from(executionWorkspaces).where(and(eq(executionWorkspaces.companyId, companyId), inArray(executionWorkspaces.id, batch), privacy)); - if (workspaces.length !== batch.length) throw forbidden("Task workspace access is no longer available"); - const retained = new Set(workspaces.flatMap(workspace => [ - ...Object.keys(workspace.metadata?._issuePrivacySources ?? {}), ...(workspace.sourceIssueId ? [workspace.sourceIssueId] : []), - ])); - const attached = await db.select({ id: issues.id, workspaceId: issues.executionWorkspaceId }).from(issues) - .where(and(eq(issues.companyId, companyId), inArray(issues.executionWorkspaceId, batch))); - const attachedWorkspaceIds = new Set(attached.map(source => source.workspaceId)); - for (const source of attached) sourceIds.add(source.id); - const retainedIds = [...retained]; - for (let start = 0; start < retainedIds.length; start += 500) { - const retainedBatch = retainedIds.slice(start, start + 500); - const sources = await db.select({ id: issues.id }).from(issues).where(and(eq(issues.companyId, companyId), - inArray(sql`${issues.id}`, retainedBatch.map(id => canonicalExecutionUuid(sql`${id}::text`))))); - const found = new Set(sources.map(source => source.id)); - if (retainedBatch.some(id => !found.has(id))) throw forbidden("Task workspace source is no longer accessible"); - for (const source of sources) sourceIds.add(source.id); + return measureRunPhase("workspace.access", async () => { + const ids = [...new Set(workspaceIds)]; + if (!ids.length || !actors.length) return; + const privacyActors = [...new Map(actors.map(actor => [JSON.stringify({ ...actor, runId: undefined }), actor])).values()]; + const privacy = and(...await Promise.all(privacyActors.map(actor => executionWorkspaceReadSqlCondition(db, actor)))); + const projectIds = new Set(), sourceIds = new Set(); + let needsCompanyRead = false; + for (let offset = 0; offset < ids.length; offset += 500) { + const batch = ids.slice(offset, offset + 500); + const workspaces = await measureRunPhase("workspace.access.inventory", async () => db.select({ id: executionWorkspaces.id, projectId: executionWorkspaces.projectId, + sourceIssueId: executionWorkspaces.sourceIssueId, metadata: executionWorkspaces.metadata, + }).from(executionWorkspaces).where(and(eq(executionWorkspaces.companyId, companyId), inArray(executionWorkspaces.id, batch), privacy))); + if (workspaces.length !== batch.length) throw forbidden("Task workspace access is no longer available"); + const retained = new Set(workspaces.flatMap(workspace => [ + ...Object.keys(workspace.metadata?._issuePrivacySources ?? {}), ...(workspace.sourceIssueId ? [workspace.sourceIssueId] : []), + ])); + const attached = await measureRunPhase("workspace.access.inventory", async () => db.select({ id: issues.id, workspaceId: issues.executionWorkspaceId }).from(issues) + .where(and(eq(issues.companyId, companyId), inArray(issues.executionWorkspaceId, batch)))); + const attachedWorkspaceIds = new Set(attached.map(source => source.workspaceId)); + for (const source of attached) sourceIds.add(source.id); + const retainedIds = [...retained]; + for (let start = 0; start < retainedIds.length; start += 500) { + const retainedBatch = retainedIds.slice(start, start + 500); + const sources = await measureRunPhase("workspace.access.inventory", async () => db.select({ id: issues.id }).from(issues).where(and(eq(issues.companyId, companyId), + inArray(sql`${issues.id}`, retainedBatch.map(id => canonicalExecutionUuid(sql`${id}::text`)))))); + const found = new Set(sources.map(source => source.id)); + if (retainedBatch.some(id => !found.has(id))) throw forbidden("Task workspace source is no longer accessible"); + for (const source of sources) sourceIds.add(source.id); + } + for (const workspace of workspaces) { + if (workspace.projectId) projectIds.add(workspace.projectId); + else if (!workspace.sourceIssueId && !Object.keys(workspace.metadata?._issuePrivacySources ?? {}).length + && !attachedWorkspaceIds.has(workspace.id)) needsCompanyRead = true; + } } - for (const workspace of workspaces) { - if (workspace.projectId) projectIds.add(workspace.projectId); - else if (!workspace.sourceIssueId && !Object.keys(workspace.metadata?._issuePrivacySources ?? {}).length - && !attachedWorkspaceIds.has(workspace.id)) needsCompanyRead = true; + const resources = [ + ...[...projectIds].map(projectId => ({ type: "project" as const, companyId, projectId })), + ...[...sourceIds].map(issueId => ({ type: "issue" as const, companyId, issueId })), + ...(needsCompanyRead ? [{ type: "company" as const, companyId }] : []), + ]; + const batches = await measureRunPhase("workspace.access.authorization", () => authorizationService(db).decideReadsForActors(actors, resources)); + for (const decisions of batches) for (let index = 0; index < decisions.length; index++) { + if (decisions[index].allowed) continue; + throw forbidden(resources[index].type === "issue" + ? "Task workspace retains files outside this actor's authorization boundary" + : resources[index].type === "company" + ? "An unscoped workspace is outside this actor's authorization boundary" : decisions[index].explanation); } - } - const resources = [ - ...[...projectIds].map(projectId => ({ type: "project" as const, companyId, projectId })), - ...[...sourceIds].map(issueId => ({ type: "issue" as const, companyId, issueId })), - ...(needsCompanyRead ? [{ type: "company" as const, companyId }] : []), - ]; - const batches = await authorizationService(db).decideReadsForActors(actors, resources); - for (const decisions of batches) for (let index = 0; index < decisions.length; index++) { - if (decisions[index].allowed) continue; - throw forbidden(resources[index].type === "issue" - ? "Task workspace retains files outside this actor's authorization boundary" - : resources[index].type === "company" - ? "An unscoped workspace is outside this actor's authorization boundary" : decisions[index].explanation); - } + }); } diff --git a/ui/src/lib/honeycomb-run-link.test.ts b/ui/src/lib/honeycomb-run-link.test.ts index 61ca8a0c221..46a81c502d4 100644 --- a/ui/src/lib/honeycomb-run-link.test.ts +++ b/ui/src/lib/honeycomb-run-link.test.ts @@ -31,7 +31,7 @@ describe("Honeycomb run links", () => { expect(query.filter_combination).toBe("AND"); expect(query.filters).toEqual([ { column: "service.name", op: "=", value: "paperclip" }, - { column: "name", op: "in", value: ["task.run", "task.provider_session"] }, + { column: "name", op: "in", value: ["task.run", "task.provider_session", "task.execution_attempt"] }, { column: HONEYCOMB_RUN_HASH_ATTRIBUTE, op: "=", diff --git a/ui/src/lib/honeycomb-run-link.ts b/ui/src/lib/honeycomb-run-link.ts index 5478a8a0032..d7bc29ecb61 100644 --- a/ui/src/lib/honeycomb-run-link.ts +++ b/ui/src/lib/honeycomb-run-link.ts @@ -26,7 +26,7 @@ export function buildHoneycombRunQueryUrl(runIdHash: string): string { breakdowns: ["trace.trace_id"], filters: [ { column: "service.name", op: "=", value: "paperclip" }, - { column: "name", op: "in", value: ["task.run", "task.provider_session"] }, + { column: "name", op: "in", value: ["task.run", "task.provider_session", "task.execution_attempt"] }, { column: HONEYCOMB_RUN_HASH_ATTRIBUTE, op: "=", value: runIdHash }, ], filter_combination: "AND", From 57fa6b358de62922cfa9a67766b2a227c434fcdf Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Thu, 15 Oct 2026 08:35:02 -0500 Subject: [PATCH 02/19] fix(observability): separate workspace finalization and copyback spans (#16428) Give native workspace finalization its own span name. Keep copyback and commit as separate child measurements, with an exported-span regression that preserves the workspace operation result. Validation: 40 runtime/tracing tests and server typecheck passed locally. Final-head CI, E2E and security checks passed. Apex review: 5/5, zero findings and no unresolved threads. Co-Authored-By: Paperclip --- doc/observability.md | 1 + .../services/execution-preparation-timing.ts | 2 +- .../heartbeat/runtime-execution.test.ts | 35 ++++++++++++++++--- .../services/heartbeat/runtime-execution.ts | 2 +- 4 files changed, 33 insertions(+), 7 deletions(-) diff --git a/doc/observability.md b/doc/observability.md index 8d15d11be67..2fcf0ade1b8 100644 --- a/doc/observability.md +++ b/doc/observability.md @@ -259,6 +259,7 @@ attempt or owning phase: | `runtime_context.asset_*` | Source reads, manifest hashing, cache lookup, cold materialization, and exact file verification on cold and warm paths. | | `workspace_sync.*` | Existing Git and baseline snapshot, pack, seed verification, transfer, and restore operations. Pack and verification can be inside transfer; do not add nested durations. | | `execution.runtime` | Runtime dispatch, usage, workspace finalization and runtime cleanup; this includes work after provider execution. | +| `execution.workspace_finalize` | Best-effort native workspace finalization, including restore recovery, copyback and commit. Its nested restore and commit spans measure distinct steps. | | `execution.workspace_ownership`, `.workspace_ownership_wait`, `.workspace_restore`, `.workspace_commit`, `.native_finalize` | Ownership coordination including its action, acquisition before the action, copyback, final snapshot/stamp/descriptor persistence, and native settlement. | | `completion.*` | Completion handler, run-log finalization, terminal persistence, accounting, liveness, and continuation summaries. | | `response.resolve`, `.authorize`, `.add_comment`, `.transaction*`, `.row_lock`, `.write`, `.publish` | Response selection, existing authorization, comment transaction including commit or rollback, row wait, insert, and external publication enqueue. | diff --git a/server/src/services/execution-preparation-timing.ts b/server/src/services/execution-preparation-timing.ts index c20c5ca3737..58fc7a33cc5 100644 --- a/server/src/services/execution-preparation-timing.ts +++ b/server/src/services/execution-preparation-timing.ts @@ -44,7 +44,7 @@ export type ExecutionPhase = | "runtime_context.instructions" | "runtime_context.skills" | "runtime_context.mcp" | "runtime_context.asset_read" | "runtime_context.asset_manifest" | "runtime_context.asset_cache" | "runtime_context.asset_materialize" | "runtime_context.asset_verify" | "workspace_sync.snapshot_git" | "workspace_sync.snapshot_baseline" | "workspace_sync.pack" | "workspace_sync.seed_verify" | "workspace_sync.stage_workspace" | "workspace_sync.restore" - | "execution.runtime" | "execution.workspace_ownership" | "execution.workspace_restore" | "execution.workspace_commit" + | "execution.runtime" | "execution.workspace_ownership" | "execution.workspace_finalize" | "execution.workspace_restore" | "execution.workspace_commit" | "execution.workspace_ownership_wait" | "execution.realtime" | "execution.usage" | "execution.native_finalize" | "completion.run" | "completion.output" | "completion.terminal_write" | "completion.cost" | "completion.liveness" | "completion.summary" diff --git a/server/src/services/heartbeat/runtime-execution.test.ts b/server/src/services/heartbeat/runtime-execution.test.ts index 20f2ff1ec4d..692ab07b2ec 100644 --- a/server/src/services/heartbeat/runtime-execution.test.ts +++ b/server/src/services/heartbeat/runtime-execution.test.ts @@ -5,6 +5,9 @@ import { join } from "node:path"; import { agents, companies, createDb, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, heartbeatRuns, issues, nativeRunFinalizations, runUsageReceipts, workspaceOperations, type Db } from "@paperclipai/db"; import { eq, sql } from "drizzle-orm"; import { afterAll, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; +import { trace } from "@opentelemetry/api"; +import { InMemorySpanExporter, SimpleSpanProcessor, TracerProvider } from "@opentelemetry/sdk-trace"; +import { measureRunPhase, measureWorkspaceSyncPhase, withRunExecutionTiming } from "../execution-preparation-timing.js"; import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js"; import * as legacyRecovery from "../legacy-execution-recovery.js"; import { createAgentIdentityRedactor } from "../agent-identity-redaction.js"; @@ -214,10 +217,17 @@ describe.skipIf(!support.supported)("heartbeat runtime execution boundary", () = it("records native restoration inside the workspace finalization operation", async () => { const { input, run } = await fixture(true); + const endpoint = process.env.OTEL_EXPORTER_OTLP_ENDPOINT; + vi.stubEnv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://collector.test"); + const exporter = new InMemorySpanExporter(); + const provider = new TracerProvider({ spanProcessors: [new SimpleSpanProcessor({ exporter })] }); + expect(trace.setGlobalTracerProvider(provider)).toBe(true); const restore = vi.fn(async () => { expect(await finalizations()).toMatchObject([{ status: "running", metadata: { owningService: "native_workspace_finalizer" }, }]); + await measureWorkspaceSyncPhase("workspace.restore", async () => {}); + await measureRunPhase("execution.workspace_commit", async () => {}); }); input.workspace.nativeWorkspaceSync = { mode: "host_current", @@ -227,11 +237,26 @@ describe.skipIf(!support.supported)("heartbeat runtime execution boundary", () = restoreWorkspace: restore, cleanup: vi.fn(async () => {}), }; - expect(await executeHeartbeatRuntime(db, input)).toMatchObject({ dispatched: true }); - expect(restore).toHaveBeenCalledOnce(); - expect(await finalizations()).toMatchObject([{ - status: "succeeded", metadata: { owningService: "native_workspace_finalizer" }, - }]); + try { + expect(await withRunExecutionTiming(run.id, () => executeHeartbeatRuntime(db, input))).toMatchObject({ dispatched: true }); + expect(restore).toHaveBeenCalledOnce(); + expect(await finalizations()).toMatchObject([{ + status: "succeeded", metadata: { owningService: "native_workspace_finalizer" }, + }]); + await provider.forceFlush(); + const spans = exporter.getFinishedSpans(); + const copyback = spans.filter(span => span.name === "execution.workspace_restore"); + const commit = spans.filter(span => span.name === "execution.workspace_commit"); + expect(copyback).toHaveLength(1); + expect(commit).toHaveLength(1); + const finalization = spans.find(span => span.name === "execution.workspace_finalize")!; + expect(finalization).toBeDefined(); + for (const child of [...copyback, ...commit]) expect(child.parentSpanContext?.spanId).toBe(finalization.spanContext().spanId); + } finally { + vi.stubEnv("OTEL_EXPORTER_OTLP_ENDPOINT", endpoint); + trace.disable(); + await provider.shutdown(); + } }); it("retains an unexported native source before provider-error teardown", async () => { diff --git a/server/src/services/heartbeat/runtime-execution.ts b/server/src/services/heartbeat/runtime-execution.ts index cd8e9fddb87..bbd31e4ee8a 100644 --- a/server/src/services/heartbeat/runtime-execution.ts +++ b/server/src/services/heartbeat/runtime-execution.ts @@ -917,7 +917,7 @@ export async function executeHeartbeatRuntime(db: Db, input: HeartbeatRuntimeExe )).limit(1); if (exported.length) adapterFinalizeOutcome = "succeeded"; else await recordWorkspaceFinalize(hasWorkspaceRestoreFailure(adapterResult.resultJson) ? "failed" : "succeeded", undefined, - async () => { await measureRunPhase("execution.workspace_restore", () => restoreNativeWorkspaceBestEffort({ + async () => { await measureRunPhase("execution.workspace_finalize", () => restoreNativeWorkspaceBestEffort({ db, runId: run.id, assertOwnership: ownership?.assertHeld, restore: () => nativeWorkspaceSync!.restoreWorkspace(ownership?.assertHeld), })); }); From 5a9e6b631563dcd7762844fc0f544e0112384d90 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Thu, 15 Oct 2026 08:52:50 -0500 Subject: [PATCH 03/19] fix(grok): avoid false file recovery after Stop (#16427) ## Thinking Path > - Paperclip lets people manage AI agents and their work. > - Grok Bot uses a managed Runner and a fixed external work session. > - Its admitted contract gives it no access to Paperclip workspace files. > - The Runner still has a sync descriptor to support durable recovery. > - Cancellation treated that descriptor as proof that files needed repair. > - This pull request uses the admitted access contract while preserving ownership settlement. > - Users get the correct recovery action after Stop, with native file safeguards intact. ## Linked Issues or Issue Description Refs #16219 and #16422. The duplicate search found no open Grok workspace correction. **What happened?** A real Grok Stop test cancelled its run, fenced its assignment, denied a stale progress update and cleaned up the sandbox. Paperclip also created a file-repair hold. The recovery panel told the user to recover workspace files, although the admitted external contract has workspace access set to none. **Expected behavior** Keep the stopped session fenced and require inspection before new work. Keep failed-run ownership settlement. Do not invent a file export obligation for a session that cannot access the workspace. Native sessions with writable or unknown workspace access must keep their file safeguards. **Steps to reproduce** 1. Connect a Grok Bot and assign a task. 2. Let it accept the task and save a Paperclip checkpoint document. 3. Use Stop while it is working. 4. Inspect the task recovery panel. It asks for file repair after sandbox cleanup succeeds. **Paperclip version or commit** Observed on Core 11d69d04c7531cf7eb37f7994b7112797e145837 through the actual Grok client. Reproduced against master with database-backed regressions. ## What Changed - Match the admitted native input to its run, company, agent, task and external driver before treating workspace access as disabled. - Keep coordinator ownership checks and failure settlement. Skip file retention and the false file-repair obligation for that exact contract. - Add inspection-only settlement for old Grok file holds through Runtime recovery. Verify exact run ownership, fenced assignments, terminal coordinators and managed lease stop receipts. Archive the false file evidence and preserve no-replay. - Keep historical Dot recovery controls. Do not claim global external-provider termination. - Fix the later dispatch-failure owner so it cannot retain a no-workspace external sandbox for a false file obligation. - Cover both Grok and Dot, mismatched access evidence, native file recovery and the assembled heartbeat cleanup path. - Update the Grok guide and the existing release evidence plan. ## Verification - Four database regressions fail before the correction. - 68 file-recovery and cancellation checks pass. Grok, Dot and the shared recovery notice pass 98 checks. The assembled heartbeat selection passes both cases; 404 unrelated cases were not selected in that local command. - The lifecycle regression caught an unsafe early-return prototype. The final code keeps coordinator settlement for both workspace cases. A separate assertion was corrected for the run-state projection's explicit nulls. The failed attempts remain in the private evidence record. - Repository typecheck and build, token gates and Storybook build pass. The branch was rebased from ea4aa185b0f3bc30be1c5fc160cdfd0eb433fe41 onto 7420af26fc9a80d6252ababd879e2fc991eebd07, whose additional change affects CI profile exports. The core recovery checks are repeated after rebase. - On the earlier deployed source, a real 824-second task survived an actual same-image controller restart with its original run, assignment, process and lease. It completed canonically and cleaned up without an extra wake. The active Stop test denied a real stale update and completed connection inspection and authenticated re-verification. These results establish the observed baseline and defect; they do not qualify this patch's deployed behavior. - Review found two valid gaps: the old file obligation still blocked fresh input, and dispatch failure could request retention earlier. Both are corrected. The follow-up passes 128 focused tests and server typecheck. The recovery test proves fresh user continuation after inspection, authenticated re-verification, and managed Runner cleanup while the old assignment stays fenced. - A second review found a partial-transaction gap. Routine readiness now commits as invalid before task holds retire. All 43 Grok routine tests pass, including a held task lock and an unrelated active assignment. - Repository typecheck and build pass after the first review corrections; final server typecheck also passes. The full local suite is still running and is not claimed as passed. The first follow-up CI timed out at 15 seconds in one unchanged run-history case; all 91 cases pass locally. The cause is unproven. - The final head, 33e436191160686d95101eb784e07b15100bde1d, passes 54 checks plus one successful status context. Two optional Storybook jobs are skipped. Greptile reviewed that exact head at 5/5 with no unresolved threads. All source CI suites pass on this final head without retry. The earlier runner-shutdown and timeout attempts remain recorded. - An actual deployed historical recovery and fresh Stop repeat are still required. ## Risks An incomplete or mismatched contract must retain file safeguards. The change uses stored admission evidence rather than current agent configuration. Historical records stay auditable and do not gain automatic replay or a generic Retry action. Operator inspection and a fresh authenticated verification remain required before new Grok work. Global Grok termination remains unconfirmed. No schema migration is needed. ## Model Used OpenAI GPT-6 in Codex, with reasoning, repository editing, shell tools and computer use. The exact serving model ID, snapshot and context window are not exposed by this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/grok-bot-agent.md | 15 ++ ...026-10-13-grok-bot-production-readiness.md | 38 ++++- server/src/__tests__/grok-bot-routine.test.ts | 143 +++++++++++++++++- .../heartbeat-process-recovery.test.ts | 33 ++-- server/src/services/execution-blocker.ts | 16 +- server/src/services/grok-bot/routine.ts | 8 +- .../services/grok-bot/workspace-inspection.ts | 87 +++++++++++ server/src/services/heartbeat.ts | 3 +- .../heartbeat/runtime-execution.test.ts | 16 +- .../services/heartbeat/runtime-execution.ts | 3 +- .../native-failed-workspace-recovery.test.ts | 52 +++++++ .../native-failed-workspace-recovery.ts | 11 +- .../workspace-restore-recovery-state.ts | 18 +++ 13 files changed, 413 insertions(+), 30 deletions(-) create mode 100644 server/src/services/grok-bot/workspace-inspection.ts diff --git a/doc/grok-bot-agent.md b/doc/grok-bot-agent.md index ebea2c20524..ffd37093d98 100644 --- a/doc/grok-bot-agent.md +++ b/doc/grok-bot-agent.md @@ -80,3 +80,18 @@ inspected recovery hold before admitting another assignment. stops future dispatch. Remote Grok execution may continue. Open the Grok Bot app to inspect it. Paperclip shows reported activity, not a complete remote transcript or computer recording. + +The admitted Grok execution contract has no access to the Runner's workspace files. Its managed Runner still keeps durable recovery state, but stopping that session does not require workspace file repair. Inspect the Bot in Grok, clear the connection's recovery hold in the agent's Runtime settings, and verify the routine again before assigning new work. This records inspection without asserting global Grok termination. + +Historical stopped Grok runs can have an incorrect workspace-file hold from an +earlier release. Runtime shows Recovery needed for these runs even when the +routine is healthy. After inspecting Grok, use Clear recovery hold and verify +the connection again. Paperclip retires the false file obligation only when the +original admission denied workspace access, the assignment is fenced, the +coordinator has settled, and the exact managed Runner leases have stop receipts. +Connection readiness is invalidated before task inspection, so a partial failure +cannot retain an old verified connection while removing a task hold. +The inspection preserves the old result, action uncertainty, and no-replay hold. +Send a new task message to request fresh work. Inspection does not certify global +Grok termination or delete any previously retained allocation. Such allocations +still use the existing environment cleanup controls. diff --git a/doc/plans/2026-10-13-grok-bot-production-readiness.md b/doc/plans/2026-10-13-grok-bot-production-readiness.md index 05de9675f3e..438cfca7607 100644 --- a/doc/plans/2026-10-13-grok-bot-production-readiness.md +++ b/doc/plans/2026-10-13-grok-bot-production-readiness.md @@ -1,6 +1,6 @@ # Grok Bot production readiness -Status as of October 15: real assigned and Bot-initiated tasks reach canonical completion. The active-session wake correction is merged. A real proactive test found a delayed-admission echo; its correction is under verification. Production acceptance is incomplete. The current release goal and matrix below supersede the earlier dated campaign entries. Historical successes and failures remain evidence for the revisions actually tested. +Status as of October 15: real assigned and Bot-initiated tasks reach canonical completion. The active-session wake correction is merged. The proactive admission echo correction is merged and source-verified; image publication and a real no-echo repeat remain pending. A real controller restart and active Stop diagnostic passed their authority checks; Stop also exposed a false workspace-repair hold, now under correction. Production acceptance is incomplete. The current release goal and matrix below supersede the earlier dated campaign entries. Historical successes and failures remain evidence for the revisions actually tested. ## Current production finish line — October 14 @@ -13,7 +13,7 @@ There is no testing call, case-count, token or spending cap. The former six-case | Ordinary onboarding works | Actual Grok UI, fixed agent consent, saved setup, wrong-key behavior, authenticated challenge, refresh, reconnect and revocation | Resumed real Grok authorization, saved setup Done, authenticated routine challenge and automatic refresh passed on the October 15 serving revision. Runtime reconnect copy fix is merged; verify its published UI and fresh production setup next. | | Assigned work reaches a usable result | One actual managed workflow: progress, question and answer, document, downloaded file/hash, canonical completion or review and cleanup | A preliminary comparison task on serving Core ff0 completed progress → question → answer → report/CSV → canonical Done, with both cleanup receipts. The cited CSV downloads with the exact publication hash. Two distinct CSV uploads accompanied an active-session result wake; correct that wake behavior and requalify the final candidate. | | Bot initiates governed work | Real-client task creation/claim/completion with stable receipts and no outbound echo or duplicate effects | A real idle-conversation request created one task and one managed run, saved a 100-word report, reached accepted Done and cleaned up its lease. A delayed-admission wake and two acceptance attempts were also recorded. Functional completion passed; no-echo acceptance failed. Requalify the correction. | -| Recovery and authority are reliable | Healthy work past the prior five-minute failure window; actual controller restart preserving assignment/session/process and no duplicate wake; Stop, pause, reassignment, revocation, stale calls, queues and cleanup | The old hold was inspected and cleared. The successful comparison continuation ended below five minutes. Healthy work beyond that window, an actual restart and active Stop remain unverified; protocol tests are separate evidence. | +| Recovery and authority are reliable | Healthy work past the prior five-minute failure window; actual controller restart preserving assignment/session/process and no duplicate wake; Stop, pause, reassignment, revocation, stale calls, queues and cleanup | A real managed diagnostic ran for 13 minutes 44 seconds, survived an actual same-image controller restart with the original assignment/session/process/lease, completed canonically and cleaned up without another wake. Active Stop fenced its work, rejected a real stale progress update, kept its checkpoint unchanged and cleaned up. Cancellation also created a false file-repair hold despite an admitted workspace-access-none contract; correct and requalify that recovery behavior. These diagnostics ran on Core 11d, not the final release image. | | One identifiable reviewed candidate passes source gates | Focused protocol/auth/Runner/UI tests, full CI/typecheck/tests/build/token/Storybook evidence and exact artifacts | Active-session wake correction #16394 merged with 55 passing checks, two optional skips and review 5/5. The proactive echo correction started from master 7fc2dc8b3236e1149ae32756b6153a503623404b and was rebased onto freshly fetched master 250429942e27d568f20d3bc7dddcfc84a666480a before submission. Two regression cases reproduce the real delayed-admission wake before the fix. All 214 focused protocol/auth/question checks pass, including 80 Grok and Dot checks. Repository typecheck/build, token gates, the 2,731-story structure and Storybook build pass. Fresh CI, review, publication and the real no-echo repeat remain required. | | Production opt-in is delivered and smoke-tested | Staging acceptance, correct served Core/distribution/Runner, normal editable opt-in controls/defaults, actual production Bot workflow and inspectable task/output links | Cloud opt-in policy and release-guide corrections are merged. Staging Cloud provenance and minimal feature values were verified, with Dot disabled and defaults unchanged. The corrected wake candidate, remaining real-client cases, ordinary production opt-in controls and production smoke are outstanding. | @@ -33,6 +33,12 @@ The proactive run also received an authenticated delayed-admission webhook while Normal staging automation has advanced beyond the older prepared candidate. No shared CI routing switch was changed. The earlier temporary fallback request is no longer needed for the already served successor. The dedicated production workspace still has no qualification agent grant or smoke task. +The proactive no-echo correction merged as #16422 after 55 successful checks, two optional Storybook skips and fresh review 5/5. Canonical source verification and exact-source migrator publication passed. The first image attempt was displaced by an older rerun before any build job started; its single retry was then superseded by a newer master push. The successor contains the fix. Neither cancellation is a successful image publication, and no shared CI routing flag changed. + +The controlled recovery diagnostic preserved one original run for 824 seconds. The operator created the task, performed one same-image controller redeploy after checking for other work and rollouts, and sent one planned normal resume comment. The Bot read that queued comment at a safe boundary, read its checkpoint back, saved revision two and reached accepted Done. Its exact source identities and successful cleanup receipt are retained privately. The Bot corrected its own rejected document operation and reported correcting an invalid finish attachment; neither was repaired by the operator. This proves observed recovery on Core 11d, not the cause of the older failure or final-image acceptance. + +The active Stop diagnostic cancelled its run, fenced the assignment, kept the checkpoint at revision one and recorded successful sandbox cleanup. One planned negative probe in the real Grok client returned “Assignment authority expired or was revoked.” No late completion was accepted. Inspection found a generic workspace-repair obligation attached to an external native input that explicitly forbids workspace access. Four database regressions reproduce that obligation and its misleading recovery guidance. The correction keeps coordinator settlement, no-replay holds and native file safeguards while omitting file retention for an exact admitted workspace-less external contract. Repository typecheck and 68 focused file-recovery/cancellation checks pass; an assembled heartbeat regression exposed and preserved the coordinator-settlement requirement. Fresh review, full checks and deployed recovery remain required. + Keep native providers and person-purpose MCP connections unchanged. No browser automation belongs in normal operation. Defer localhost relays, internal Grok APIs, marketplace distribution and Paperclip-connected third-party tools. Completion of a Paperclip work session does not assert remote Grok termination. ## Candidate and release contract @@ -255,3 +261,31 @@ assignment authority. Disconnect completes while storage is pending, and the late upload returns 403 with no attachment or publication receipt. Person write checks remain in place. The full repository command was interrupted when this new review finding arrived; its partial output is not a passing test result. + +## October 15 — Stop review follow-up + +Review found that hiding the historical file-repair card alone left the stored +file obligation blocking fresh task input. Runtime inspection now finds these +holds even after routine recovery. It retires only the false file fields after +exact run, assignment, coordinator and managed-lease cleanup checks, archives +the previous evidence, and preserves no-replay. A real authenticated readiness +exchange and new user input are still required. No global provider-stop claim is +created. Dot historical recovery controls remain available. + +The later dispatch-failure owner also now honors the validated no-workspace +contract before asking for sandbox retention. Both external providers and the +writable native path pass the runtime boundary regression. The review follow-up +passes 128 focused protocol, recovery, cancellation and runtime checks. An initial +negative assertion saw the earlier readiness gate instead of the revoked-session +gate; the corrected test checks both before and after re-verification. Server +typecheck passes after narrowing the native workspace union. Final current-head +CI, review, deployed historical recovery and fresh Stop are still release gates. + +A second review caught a partial-transaction ordering gap. Routine readiness now +commits as invalid before the separate task inspection transactions. New tests +hold the task lock to prove readiness is already gone and verification remains +blocked; an unrelated active assignment cannot cause a task hold to disappear. +All 43 Grok routine tests pass. The first follow-up CI attempt also timed out in +one unchanged run-history case at 15 seconds; all 91 tests in that file pass +locally. The timeout remains a recorded failure with cause unproven. The new +source head requires fresh CI and review. diff --git a/server/src/__tests__/grok-bot-routine.test.ts b/server/src/__tests__/grok-bot-routine.test.ts index e54a3f286f2..c49b846a6d8 100644 --- a/server/src/__tests__/grok-bot-routine.test.ts +++ b/server/src/__tests__/grok-bot-routine.test.ts @@ -8,7 +8,7 @@ import { createHash, randomBytes, randomUUID } from "node:crypto"; import { and, eq, sql } from "drizzle-orm"; import { beforeAll, afterAll, afterEach, describe, expect, it, vi } from "vitest"; import { agents, authUsers, companies, companyMemberships, createDb, dotAgentBindings, grokBotDispatches, - grokBotRoutines, mcpOauthGrants, activityLog, issues, heartbeatRuns, dotRunnerAssignments, dotRunnerOperations, dotMailboxItems, nativeRunFinalizations } from "@paperclipai/db"; + grokBotRoutines, mcpOauthGrants, activityLog, issues, heartbeatRuns, dotRunnerAssignments, dotRunnerOperations, dotMailboxItems, nativeRunFinalizations, environmentLeases, environments, issueRecoveryActions, issueComments } from "@paperclipai/db"; import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; import { accessService } from "../services/access.js"; import { instanceSettingsService } from "../services/instance-settings.js"; @@ -20,6 +20,10 @@ import * as secretModule from "../services/secrets.js"; import * as externalAuthority from "../services/external-bot-authority.js"; import { type GrokRoutineWake } from "../services/grok-bot/webhook.js"; +import { remoteTerminationReceipt } from "../services/remote-execution-termination.js"; +import { admitExplicitNativeContinuation } from "../services/explicit-native-continuation.js"; +import { hasRequiredWorkspaceRecovery } from "../services/workspace-restore-recovery-state.js"; + let temporary: Awaited>; let db: ReturnType; beforeAll(async () => { @@ -479,3 +483,140 @@ describe("durable Grok routine qualification", () => { }); }); + +// Exercise the public recovery service and the actual native admission boundary, +// including a stale file obligation persisted before this fix. +describe("Grok inspection of historical workspace-less Stop", () => { + async function historicalHold() { + const f = await fixture(); + const { run, task, work } = await admittedWork(f, "accepted"); + const finishedAt = new Date(Date.now() - 1000); + const fileHold = { workspaceRestoreFailure: "restore_failed", workspaceRestoreRecovery: { + schema: "paperclip.workspace-restore-recovery.v1", runId: run.id, leaseIds: [] as string[], + } }; + const [environment] = await db.insert(environments).values({ name: `Grok ${run.id}`, driver: "sandbox" }).returning(); + const [lease] = await db.insert(environmentLeases).values({ companyId: f.company.id, issueId: task.id, + heartbeatRunId: run.id, environmentId: environment!.id, provider: "daytona", providerLeaseId: randomUUID(), + status: "released", releasedAt: new Date(), cleanupStatus: "success" }).returning(); + await db.update(environmentLeases).set({ metadata: { remoteExecutionTermination: + remoteTerminationReceipt(lease!, { providerLeaseId: lease!.providerLeaseId, state: "destroyed" }) } }).where(eq(environmentLeases.id, lease!.id)); + fileHold.workspaceRestoreRecovery.leaseIds.push(lease!.id); + await db.update(heartbeatRuns).set({ status: "cancelled", driverKind: "grok_bot_mcp", finishedAt, + nativeIssueId: task.id, issueId: task.id, contextSnapshot: { issueId: task.id }, + runnerProfileJson: { nativeExecutionInput: { schema: "paperclip.native-execution-input.v6", + binding: { runId: run.id, companyId: f.company.id, agentId: f.agent.id, issueId: task.id }, + provider: { kind: "grok_bot" }, workspace: { access: "none", cwd: null } } }, + resultJson: { ...fileHold, cancellation: { source: "user", reason: "Paperclip work stopped" } }, + }).where(eq(heartbeatRuns.id, run.id)); + await db.update(issues).set({ status: "blocked", executionRunId: null, checkoutRunId: null }).where(eq(issues.id, task.id)); + await db.update(nativeRunFinalizations).set({ phase: "terminal_failure", leaseOwner: null, + leaseExpiresAt: null, recoveryState: "blocked" }).where(eq(nativeRunFinalizations.runId, run.id)); + await db.update(dotRunnerAssignments).set({ status: "fenced" }).where(eq(dotRunnerAssignments.id, work.id)); + const [action] = await db.insert(issueRecoveryActions).values({ companyId: f.company.id, sourceIssueId: task.id, + kind: "active_run_watchdog", cause: "native_continuation_requires_reconciliation", fingerprint: run.id, + status: "resolved", outcome: "blocked", nextAction: "Recover files.", + evidence: { ...fileHold, runId: run.id, automaticRecovery: { replay: "blocked", actionOutcome: "unknown" } }, + }).returning(); + const [comment] = await db.insert(issueComments).values({ companyId: f.company.id, issueId: task.id, + authorType: "user", authorUserId: f.principal.grant.userId, body: "Start a fresh task turn after inspection." }).returning(); + const continueWork = (dryRun = true) => db.transaction(async tx => { + await tx.select().from(issues).where(eq(issues.id, task.id)).for("update"); + return admitExplicitNativeContinuation({ db: tx as unknown as typeof db, companyId: f.company.id, + issueId: task.id, agentId: f.agent.id, actorType: "user", actorId: f.principal.grant.userId, + reason: "issue_commented", commentId: comment!.id, successorRunId: randomUUID(), dryRun }); + }); + return { ...f, run, task, work, lease: lease!, action: action!, continueWork }; + } + + it("offers inspection for an old task hold and admits only new explicit input after inspection and Runner cleanup", async () => { + const f = await historicalHold(); + f.delivery.mockClear(); + expect(await f.service.recoveryNeeded(f.company.id, f.agent.id)).toBe(true); + expect(await f.continueWork()).toBeNull(); + expect(await f.service.recover(f.company.id, f.agent.id, f.principal.grant.userId)).toMatchObject({ + requiresVerification: true, remoteTerminationConfirmed: false, + }); + expect(await f.service.state(f.company.id, f.agent.id, f.pairing.bindingId)).toMatchObject({ verified: false, recoveryNeeded: false }); + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.run.id)); + expect(hasRequiredWorkspaceRecovery(run!.resultJson)).toBe(false); + expect(run).toMatchObject({ status: "cancelled", resultJson: { externalWorkspaceInspection: { + inspectedBy: f.principal.grant.userId, remoteTerminationConfirmed: false, + previousWorkspaceRestoreFailure: "restore_failed", runnerLeaseIds: [f.lease.id], + } } }); + const [action] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.action.id)); + expect(action).toMatchObject({ status: "resolved", outcome: "blocked", evidence: { + automaticRecovery: { replay: "blocked", actionOutcome: "unknown" }, externalWorkspaceInspection: { workspaceAccess: "none" }, + } }); + expect(action!.evidence).not.toHaveProperty("workspaceRestoreRecovery"); + expect(f.delivery).not.toHaveBeenCalled(); + expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.company.id))).toHaveLength(1); + await expect(dotRunnerBroker(db).operation(f.principal, f.work.id, randomUUID(), "progress", { text: "late effect" })).rejects.toThrow("wake verification"); + // Connection recovery still requires a real authenticated challenge exchange. + const row = await verification(f); await f.service.send(f.company.id, f.agent.id, row.id); + await f.service.acknowledge(f.principal, acknowledgement(row)); + await expect(dotRunnerBroker(db).operation(f.principal, f.work.id, randomUUID(), "progress", { text: "late effect" })).rejects.toThrow("revoked"); + expect(await f.continueWork(false)).toMatchObject({ previousRunId: f.run.id }); + const [settled] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.action.id)); + expect(settled!.evidence.automaticRecovery).toMatchObject({ replay: "explicit_user_continuation", actionOutcome: "unknown" }); + expect((await db.select().from(dotRunnerAssignments).where(eq(dotRunnerAssignments.id, f.work.id)))[0]!.status).toBe("fenced"); + await f.service.recover(f.company.id, f.agent.id, f.principal.grant.userId); + expect(await db.select().from(activityLog).where(and(eq(activityLog.companyId, f.company.id), eq(activityLog.action, "grok.workspace_obligation_inspected")))).toHaveLength(1); + }); + + it("does not retire task holds if another assignment prevents recovery", async () => { + const f = await historicalHold(); + const [other] = await db.insert(heartbeatRuns).values({ companyId: f.company.id, agentId: f.agent.id, + status: "running", invocationSource: "assignment", triggerDetail: "system" }).returning(); + await db.insert(dotRunnerAssignments).values({ ...f.work, id: randomUUID(), runId: other!.id, + normalizedSessionId: randomUUID(), turnId: randomUUID(), status: "accepted" }); + await expect(f.service.recover(f.company.id, f.agent.id, f.principal.grant.userId)).rejects.toThrow("current Paperclip work session"); + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.run.id)); + expect(hasRequiredWorkspaceRecovery(run!.resultJson)).toBe(true); + expect(await f.service.recoveryNeeded(f.company.id, f.agent.id)).toBe(true); + }); + + it("commits verification invalidation before waiting for historical task inspection", async () => { + const f = await historicalHold(); + let recovery: Promise | undefined; + try { + await db.transaction(async tx => { + await tx.select().from(issues).where(eq(issues.id, f.task.id)).for("update"); + recovery = f.service.recover(f.company.id, f.agent.id, f.principal.grant.userId); + void recovery.catch(() => {}); + await vi.waitFor(async () => { + expect(await f.service.state(f.company.id, f.agent.id, f.pairing.bindingId)).toMatchObject({ verified: false, recoveryNeeded: true }); + }); + await expect(verification(f)).rejects.toThrow("recover"); + const [binding] = await db.select().from(dotAgentBindings).where(eq(dotAgentBindings.id, f.pairing.bindingId)); + expect(binding).toMatchObject({ status: "connected", readyAt: null }); + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.run.id)); + expect(hasRequiredWorkspaceRecovery(run!.resultJson)).toBe(true); + }); + } finally { await recovery; } + expect(await f.service.state(f.company.id, f.agent.id, f.pairing.bindingId)).toMatchObject({ verified: false, recoveryNeeded: false }); + }); + + it.each(["cleanup_pending", "receipt_mismatch", "controller_owned", "assignment_active", "running", "operator_revoked", "foreign_company"])("does not retire an old obligation when %s", async condition => { + const f = await historicalHold(); + if (condition === "cleanup_pending") await db.update(environmentLeases).set({ cleanupStatus: "failed" }).where(eq(environmentLeases.id, f.lease.id)); + if (condition === "receipt_mismatch") await db.update(environmentLeases).set({ metadata: { remoteExecutionTermination: { schema: "paperclip.remote-termination.v1", runId: randomUUID() } } }).where(eq(environmentLeases.id, f.lease.id)); + if (condition === "controller_owned") await db.update(nativeRunFinalizations).set({ leaseOwner: "other-owner" }).where(eq(nativeRunFinalizations.runId, f.run.id)); + if (condition === "assignment_active") await db.update(dotRunnerAssignments).set({ status: "accepted" }).where(eq(dotRunnerAssignments.id, f.work.id)); + if (condition === "running") await db.update(heartbeatRuns).set({ status: "running" }).where(eq(heartbeatRuns.id, f.run.id)); + if (condition === "operator_revoked") await accessService(db).setPrincipalPermission(f.company.id, "user", f.principal.grant.userId, "agents:create", false, f.principal.grant.userId); + if (condition === "foreign_company") { + const foreign = await fixture(); + await f.service.recover(foreign.company.id, foreign.agent.id, foreign.principal.grant.userId); + } else if (condition === "running") { + // Not an eligible historical source. Its active assignment is fenced in + // this deliberately malformed fixture, but no file state may be removed. + await f.service.recover(f.company.id, f.agent.id, f.principal.grant.userId); + } else await expect(f.service.recover(f.company.id, f.agent.id, f.principal.grant.userId)).rejects.toThrow(); + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.run.id)); + expect(hasRequiredWorkspaceRecovery(run!.resultJson)).toBe(true); + if (["cleanup_pending", "receipt_mismatch", "controller_owned"].includes(condition)) { + expect(await f.service.state(f.company.id, f.agent.id, f.pairing.bindingId)).toMatchObject({ verified: false, recoveryNeeded: true }); + } + expect(await f.continueWork()).toBeNull(); + }); +}); diff --git a/server/src/__tests__/heartbeat-process-recovery.test.ts b/server/src/__tests__/heartbeat-process-recovery.test.ts index 9d24caf061d..4b8a38f3e3b 100644 --- a/server/src/__tests__/heartbeat-process-recovery.test.ts +++ b/server/src/__tests__/heartbeat-process-recovery.test.ts @@ -3521,7 +3521,7 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => { } } - it("retains the source when a saved native descriptor fails inside runtime selection", async () => { + it.each([false, true])("settles a failed saved native descriptor without inventing files when workspace access is disabled: %s", async withoutWorkspace => { await withTempPaperclipHome(async () => { const f = await seedQueuedIssueRunFixture(); await db.update(agents).set({ adapterType: "paperclip_runner", @@ -3547,7 +3547,12 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => { descriptorSha256: "a".repeat(64), baselineSha256: "b".repeat(64), finalHostSha256: null, workspaceId: randomUUID(), leaseId: lease!.id, providerLeaseId: lease!.providerLeaseId!, remoteCwd: "/work", resourceDisposition: "destroy" as const }; await db.update(heartbeatRuns).set({ runtimeMode: "native", nativeIssueId: f.issueId, nativePhase: "observed", - runnerProfileJson: { nativeWorkspaceSync: reference }, + driverKind: withoutWorkspace ? "grok_bot_mcp" : null, + runnerProfileJson: { nativeWorkspaceSync: reference, ...(withoutWorkspace ? { nativeExecutionInput: { + schema: "paperclip.native-execution-input.v6", provider: { kind: "grok_bot" }, + binding: { companyId: f.companyId, agentId: f.agentId, runId, issueId: f.issueId }, + workspace: { access: "none", cwd: null, repoUrl: null, repoRef: null, branchName: null }, + } } : {}) }, }).where(eq(heartbeatRuns.id, runId)); await db.insert(nativeRunFinalizations).values({ runId, companyId: f.companyId, issueId: f.issueId, phase: "observed" }); await nativeWorkspaceSyncInternals.readDescriptor({ companyId: f.companyId, runId, workspaceId: reference.workspaceId, reference }); @@ -3556,15 +3561,21 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => { await heartbeat.resumeQueuedRuns(); await heartbeat.drainActiveRunExecutions(); expect(leaseId).toBeDefined(); - expect(await heartbeat.getRun(f.runId, { includeExecutionEvidence: true })).toMatchObject({ status: "failed", - runtimeMode: "native", nativePhase: "terminal_failure", resultJson: { - workspaceRestoreFailure: "restore_failed", workspaceRestoreRecovery: { leaseIds: [leaseId] }, - }, - }); - expect(await db.select().from(environmentLeases).where(eq(environmentLeases.id, leaseId!))) - .toMatchObject([{ status: "pending_cleanup", leasePolicy: "retain_on_failure" }]); - expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toMatchObject({ workspaceRepairRequired: true, canRetry: false }); - expect(release).toHaveBeenCalledWith(expect.objectContaining({ providerResourceDisposition: "stop_and_retain" })); + const failed = await heartbeat.getRun(f.runId, { includeExecutionEvidence: true }); + expect(failed).toMatchObject({ status: "failed", runtimeMode: "native", nativePhase: "terminal_failure" }); + if (withoutWorkspace) { + expect(failed?.resultJson?.workspaceRestoreFailure ?? null).toBeNull(); + expect(failed?.resultJson?.workspaceRestoreRecovery ?? null).toBeNull(); + expect((await getExecutionBlocker(db, f.companyId, f.issueId))?.workspaceRepairRequired).not.toBe(true); + expect(release).toHaveBeenCalled(); + expect(release).not.toHaveBeenCalledWith(expect.objectContaining({ providerResourceDisposition: "stop_and_retain" })); + } else { + expect(failed?.resultJson).toMatchObject({ workspaceRestoreFailure: "restore_failed", workspaceRestoreRecovery: { leaseIds: [leaseId] } }); + expect(await db.select().from(environmentLeases).where(eq(environmentLeases.id, leaseId!))) + .toMatchObject([{ status: "pending_cleanup", leasePolicy: "retain_on_failure" }]); + expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toMatchObject({ workspaceRepairRequired: true, canRetry: false }); + expect(release).toHaveBeenCalledWith(expect.objectContaining({ providerResourceDisposition: "stop_and_retain" })); + } expect(provider).not.toHaveBeenCalled(); expect(mockAdapterExecute).not.toHaveBeenCalled(); } finally { factory.mockRestore(); } diff --git a/server/src/services/execution-blocker.ts b/server/src/services/execution-blocker.ts index 9b3a16a3e0c..b451fd41e30 100644 --- a/server/src/services/execution-blocker.ts +++ b/server/src/services/execution-blocker.ts @@ -7,7 +7,7 @@ import { canContinueCancelledRun, readRunCancellation } from "./run-cancellation import { canRetryStoppedRun } from "./cancelled-native-startup.js"; import { queuedCommentIdsFromWakePayload } from "./issue-queued-comment-queue.js"; import { EXECUTION_RECONCILIATION_CAUSES, type ExecutionBlocker } from "@paperclipai/shared"; -import { hasRequiredWorkspaceRecovery, LEGACY_WORKSPACE_RECOVERY_SCHEMA } from "./workspace-restore-recovery-state.js"; +import { hasRequiredWorkspaceRecovery, LEGACY_WORKSPACE_RECOVERY_SCHEMA, nativeRunHasNoWorkspaceAccess } from "./workspace-restore-recovery-state.js"; /** Resolved recovery bookkeeping can still carry an effective no-replay hold. */ export function executionBlockerPredicate() { @@ -65,6 +65,8 @@ export async function getExecutionBlocker(db: Db, companyId: string, issueId: st const [run] = runId ? await db.select().from(heartbeatRuns).where(and( eq(heartbeatRuns.companyId, companyId), eq(heartbeatRuns.id, runId), )).limit(1) : []; + const withoutWorkspace = Boolean(run?.driverKind === "grok_bot_mcp" && nativeRunHasNoWorkspaceAccess(run)); + const workspaceRepairRequired = Boolean(restoreHold && !withoutWorkspace); const queue = await db.select({ payload: agentWakeupRequests.payload }).from(agentWakeupRequests).where(and( eq(agentWakeupRequests.companyId, companyId), eq(agentWakeupRequests.status, "deferred_issue_execution"), sql`${agentWakeupRequests.payload}->>'issueId' = ${issueId}`, @@ -118,6 +120,16 @@ export async function getExecutionBlocker(db: Db, companyId: string, issueId: st nextAction += " Inspect the run before sending a new message to request continuation."; } + if (withoutWorkspace && (restoreHold || run?.status === "cancelled") && !run?.resultJson?.externalWorkspaceInspection) { + nextAction = run?.driverKind === "grok_bot_mcp" + ? "Inspect this Bot in Grok Bot, then clear the recovery hold and verify the connection in the agent's Runtime settings before assigning more work." + : "Inspect the stopped work in the external agent and its connection before requesting continuation."; + } + + if (run?.resultJson?.externalWorkspaceInspection && !restoreHold) { + nextAction = "Verify the Grok connection, then send a new message to continue. The stopped work will not be replayed."; + } + return { recoveryActionId: action.id, runId, @@ -129,7 +141,7 @@ export async function getExecutionBlocker(db: Db, companyId: string, issueId: st runError: runError?.slice(0, 1024) ?? null, canContinue: eligibleContinuation && !chatBinding, canRetry: canRetry && !chatBinding, - ...(restoreHold ? { workspaceRepairRequired: true, canRecordWorkspaceRepair: hasRequiredWorkspaceRecovery(action.evidence) } : {}), + ...(restoreHold ? { workspaceRepairRequired, canRecordWorkspaceRepair: workspaceRepairRequired && hasRequiredWorkspaceRecovery(action.evidence) } : {}), savedMessageCount: saved.length, }; } diff --git a/server/src/services/grok-bot/routine.ts b/server/src/services/grok-bot/routine.ts index 791a94357fb..6e6acb5ac62 100644 --- a/server/src/services/grok-bot/routine.ts +++ b/server/src/services/grok-bot/routine.ts @@ -10,6 +10,8 @@ import { scheduleAgentLifecycle } from "../agent-lifecycle.js"; import { logActivity } from "../activity-log.js"; import { deliverGrokRoutineWake, validateGrokRoutineCredential, type GrokRoutineDelivery, type GrokRoutineWake } from "./webhook.js"; +import { grokWorkspaceInspectionCandidates, inspectGrokWorkspaceObligations } from "./workspace-inspection.js"; + const digest = (value: string) => createHash("sha256").update(value).digest("hex"); const fail = (message: string) => new McpOAuthError("access_denied", message, 409); export const grokWakeAcknowledgementSchema = z.object({ correlationId: z.uuid(), bindingId: z.uuid(), @@ -44,7 +46,7 @@ export function grokRoutineService(db: Db, deliver: Delivery = deliverGrokRoutin const [uncertain] = await db.select({ id: dispatches.id }).from(dispatches).innerJoin(routines, eq(routines.id, dispatches.routineId)) .where(and(eq(dispatches.companyId, companyId), eq(routines.agentId, agentId), or(eq(dispatches.status, "unknown"), and(eq(dispatches.status, "sending"), lte(dispatches.updatedAt, new Date(Date.now() - 30_000))), and(eq(dispatches.status, "accepted"), lte(dispatches.expiresAt, new Date()))))).limit(1); - return !!held || !!uncertain; + return !!held || !!uncertain || (await grokWorkspaceInspectionCandidates(db, companyId, agentId)).length > 0; } async function recover(companyId: string, agentId: string, operatorId: string) { @@ -65,6 +67,10 @@ export function grokRoutineService(db: Db, deliver: Delivery = deliverGrokRoutin await logActivity(tx as unknown as Db, { companyId, actorType: "user", actorId: operatorId, action: "grok.recovery_inspected", entityType: "agent", entityId: agentId, details: { inspectedRemoteWork: true, remoteTerminationConfirmed: false, requiresVerification: true } }); }); + // Commit loss of readiness before retiring task holds. If inspection fails + // or waits for a task lock, the old verification cannot admit new work. + // The remaining task hold also prevents verification during this gap. + await inspectGrokWorkspaceObligations(db, companyId, agentId, operatorId); return { recovered: true, requiresVerification: true, remoteTerminationConfirmed: false }; } diff --git a/server/src/services/grok-bot/workspace-inspection.ts b/server/src/services/grok-bot/workspace-inspection.ts new file mode 100644 index 00000000000..f1db9a4a851 --- /dev/null +++ b/server/src/services/grok-bot/workspace-inspection.ts @@ -0,0 +1,87 @@ +import { and, eq, inArray, isNotNull, or, sql } from "drizzle-orm"; +import { dotRunnerAssignments, environmentLeases, heartbeatRuns, issueRecoveryActions, issues, nativeRunFinalizations, type Db } from "@paperclipai/db"; +import { authorizeExternalBotOperator } from "../external-bot-authority.js"; +import { logActivity } from "../activity-log.js"; +import { McpOAuthError } from "../public-mcp/oauth.js"; +import { hasRemoteTerminationReceipt } from "../remote-execution-termination.js"; +import { hasRequiredWorkspaceRecovery, nativeRunHasNoWorkspaceAccess } from "../workspace-restore-recovery-state.js"; + +/** Old Runner synchronization descriptors could incorrectly require a file + * export for a Bot explicitly admitted without workspace access. Keep the old + * execution fenced and its no-replay hold; inspection only retires that false + * file obligation. Native Runner cleanup never certifies global Grok exit. */ +export async function grokWorkspaceInspectionCandidates(db: Db, companyId: string, agentId: string) { + const runs = await db.select().from(heartbeatRuns).where(and( + eq(heartbeatRuns.companyId, companyId), eq(heartbeatRuns.agentId, agentId), + eq(heartbeatRuns.driverKind, "grok_bot_mcp"), isNotNull(heartbeatRuns.finishedAt), + inArray(heartbeatRuns.status, ["cancelled", "failed", "timed_out", "interrupted"]), + sql`${heartbeatRuns.resultJson}->'workspaceRestoreRecovery'->>'schema' = 'paperclip.workspace-restore-recovery.v1'`, + )); + return runs.filter(run => nativeRunHasNoWorkspaceAccess(run) && hasRequiredWorkspaceRecovery(run.resultJson)); +} + +export async function inspectGrokWorkspaceObligations(db: Db, companyId: string, agentId: string, operatorId: string) { + for (const selected of await grokWorkspaceInspectionCandidates(db, companyId, agentId)) { + // Independent task transactions keep the native task/coordinator/run lock + // order out of the routine writer's binding/routine lock order. A partial + // inspection grants no authority and never schedules work. + await db.transaction(async tx => { + const database = tx as unknown as Db; + await authorizeExternalBotOperator(database, companyId, operatorId); + const issueId = selected.nativeIssueId ?? selected.issueId; + if (!issueId) return; + const [task] = await tx.select().from(issues).where(and( + eq(issues.companyId, companyId), eq(issues.id, issueId), + )).for("update"); + const [coordinator] = await tx.select().from(nativeRunFinalizations).where(and( + eq(nativeRunFinalizations.companyId, companyId), eq(nativeRunFinalizations.runId, selected.id), + eq(nativeRunFinalizations.issueId, issueId), + )).for("update"); + const [run] = await tx.select().from(heartbeatRuns).where(and( + eq(heartbeatRuns.companyId, companyId), eq(heartbeatRuns.agentId, agentId), eq(heartbeatRuns.id, selected.id), + )).for("update"); + if (!run || !hasRequiredWorkspaceRecovery(run.resultJson)) return; + const fail = () => new McpOAuthError("access_denied", "The stopped task still needs verified Runner cleanup before inspection can clear its file obligation.", 409); + if (!task || !nativeRunHasNoWorkspaceAccess(run) || run.driverKind !== "grok_bot_mcp" + || !run.finishedAt || !["cancelled", "failed", "timed_out", "interrupted"].includes(run.status) + || !coordinator || coordinator.phase !== "terminal_failure" || coordinator.leaseOwner + || coordinator.failureDetail?.successorRunId) throw fail(); + const [active] = await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and( + eq(heartbeatRuns.companyId, companyId), inArray(heartbeatRuns.status, ["running", "queued", "scheduled_retry"]), + or(eq(heartbeatRuns.nativeIssueId, issueId), eq(heartbeatRuns.issueId, issueId)), + )).limit(1); + const work = await tx.select().from(dotRunnerAssignments).where(and( + eq(dotRunnerAssignments.companyId, companyId), eq(dotRunnerAssignments.agentId, agentId), + eq(dotRunnerAssignments.runId, run.id), + )); + const leases = await tx.select().from(environmentLeases).where(and( + eq(environmentLeases.companyId, companyId), eq(environmentLeases.heartbeatRunId, run.id), + )); + if (active || !work.length || work.some(row => row.status !== "fenced") || !leases.length + || !leases.every(lease => lease.issueId === issueId && hasRemoteTerminationReceipt(lease))) throw fail(); + const actions = await tx.select().from(issueRecoveryActions).where(and( + eq(issueRecoveryActions.companyId, companyId), eq(issueRecoveryActions.sourceIssueId, issueId), + or(sql`${issueRecoveryActions.evidence}->>'runId' = ${run.id}`, + sql`${issueRecoveryActions.evidence}->>'sourceRunId' = ${run.id}`), + )).for("update"); + const inspection = { schema: "paperclip.grok-workspace-inspection.v1", runId: run.id, + inspectedBy: operatorId, inspectedAt: new Date().toISOString(), workspaceAccess: "none", + remoteTerminationConfirmed: false, runnerLeaseIds: leases.map(lease => lease.id), + previousWorkspaceRestoreFailure: run.resultJson?.workspaceRestoreFailure, + previousWorkspaceRestoreRecovery: run.resultJson?.workspaceRestoreRecovery }; + const retireFileObligation = (value: Record) => { + const { workspaceRestoreFailure: _failure, workspaceRestoreRecovery: _recovery, ...rest } = value; + return { ...rest, externalWorkspaceInspection: inspection }; + }; + await tx.update(heartbeatRuns).set({ resultJson: retireFileObligation(run.resultJson!) }).where(eq(heartbeatRuns.id, run.id)); + for (const action of actions) { + await tx.update(issueRecoveryActions).set({ evidence: retireFileObligation(action.evidence), updatedAt: new Date(), + nextAction: "Inspect Grok and verify its connection before sending a new task message. The old work will not be replayed.", + }).where(eq(issueRecoveryActions.id, action.id)); + } + await logActivity(database, { companyId, actorType: "user", actorId: operatorId, + action: "grok.workspace_obligation_inspected", entityType: "issue", entityId: issueId, + agentId, runId: run.id, details: inspection }); + }); + } +} diff --git a/server/src/services/heartbeat.ts b/server/src/services/heartbeat.ts index 6255be11197..2525be9a261 100644 --- a/server/src/services/heartbeat.ts +++ b/server/src/services/heartbeat.ts @@ -10,6 +10,7 @@ import { selectHeartbeatEnvironment } from "./heartbeat/environment-selection.js import { prepareHeartbeatWorkspace } from "./heartbeat/workspace-preparation.js"; import { executeHeartbeatRuntime, NativeSessionResumeScheduledError, NativeWorkspaceFinalizeScheduledError } from "./heartbeat/runtime-execution.js"; import { preserveNativeFailedWorkspace } from "./native-runtime/native-failed-workspace-recovery.js"; +import { nativeRunHasNoWorkspaceAccess } from "./workspace-restore-recovery-state.js"; import { selectHeartbeatRuntime } from "./heartbeat/runtime-selection.js"; import { createHeartbeatRunCompletion } from "./heartbeat/run-completion.js"; export { @@ -1948,7 +1949,7 @@ export function heartbeatService( const failureRun = await getRun(run.id); if (failureRun?.runtimeMode === "native" && parseObject(failureRun.runnerProfileJson?.nativeWorkspaceSync).state === "prepared") { - providerResourceDispositionForRun = "stop_and_retain"; + if (!nativeRunHasNoWorkspaceAccess(failureRun)) providerResourceDispositionForRun = "stop_and_retain"; try { await preserveNativeFailedWorkspace(db, failureRun, { allowPreterminal: true, diff --git a/server/src/services/heartbeat/runtime-execution.test.ts b/server/src/services/heartbeat/runtime-execution.test.ts index 692ab07b2ec..f124baeb711 100644 --- a/server/src/services/heartbeat/runtime-execution.test.ts +++ b/server/src/services/heartbeat/runtime-execution.test.ts @@ -59,9 +59,9 @@ describe.skipIf(!support.supported)("heartbeat runtime execution boundary", () = vi.mocked(executePaperclipNativeSession).mockResolvedValue({ exitCode: 0, signal: null, timedOut: false }); }); - async function fixture(native = false) { + async function fixture(native = false, provider: "codex" | "grok_bot" | "openai_dot" = "codex") { const [company] = await db.insert(companies).values({ name: "Runtime execution", issuePrefix: "EXE" }).returning(); - const [agent] = await db.insert(agents).values({ companyId: company.id, name: "Executing agent", status: "running", adapterType: native ? "paperclip_runner" : "process", adapterConfig: { provider: "codex" } }).returning(); + const [agent] = await db.insert(agents).values({ companyId: company.id, name: "Executing agent", status: "running", adapterType: native ? "paperclip_runner" : "process", adapterConfig: provider === "codex" ? { provider } : { provider, lifecycleMode: "per_turn", allowUnmeteredProvider: true, [provider === "grok_bot" ? "externalBindingId" : "dotBindingId"]: randomUUID() } }).returning(); const [issue] = await db.insert(issues).values({ companyId: company.id, title: "Complete this task", assigneeAgentId: agent.id }).returning(); const [run] = await db.insert(heartbeatRuns).values({ companyId: company.id, agentId: agent.id, issueId: issue.id, scopeKind: "issue", status: "running", runtimeMode: native ? "native" : "legacy" }).returning(); const state = createHeartbeatRunState(db); @@ -71,14 +71,15 @@ describe.skipIf(!support.supported)("heartbeat runtime execution boundary", () = const nativeExecution = native ? buildNativeExecutionInput({ companyId: company.id, agentId: agent.id, runId: run.id, issue, taskPrompt: issue.title, workspace: { id: run.id, cwd: home, repoUrl: null, repoRef: null, branchName: null }, - normalizedSessionId: null, provider: "codex", runtimeContext: nativeRuntimeContextFixture(), + normalizedSessionId: null, provider, runtimeContext: nativeRuntimeContextFixture(), + dotBinding: { bindingId: randomUUID(), bindingGeneration: 1, companyId: company.id, agentId: agent.id, acceptByUnixMs: Date.now() + 60_000, expiresAtUnixMs: Date.now() + 120_000 }, completionContract: { id: randomUUID(), sha256: `sha256:${"a".repeat(64)}`, schemaVersion: "paperclip.run-result.v1", contract: { revision: "1", objective: issue.title, criteria: [{ id: "output", requirement: issue.title }] } }, }) : null; const input: HeartbeatRuntimeExecutionInput = { run, agent, options: {}, task: { issueRef: issue, issueContext: null, context: {}, issueId: issue.id, taskKey: issue.id, executionContinuation: null }, runtime: { nativeExecution, nativeRunnerInstanceId: native ? randomUUID() : null, getNativeFreshSessionHandoff: undefined, - nativeRuntimeResolution: resolveHeartbeatNativeRuntimeMode({ persisted: run, enabled: native, runtimeConfig: {}, adapterConfig: agent.adapterConfig, agent, issue, target: null, workspaceId: null }), + nativeRuntimeResolution: resolveHeartbeatNativeRuntimeMode({ persisted: run, enabled: native, grokEnabled: true, dotEnabled: true, runtimeConfig: {}, adapterConfig: agent.adapterConfig, agent, issue, target: null, workspaceId: null }), adapter, runtimeForAdapter: { taskKey: issue.id, sessionId: null, sessionParams: null, sessionDisplayId: null }, getFreshSessionHandoff: undefined, runOptions: {}, }, workspace: { persistedExecutionWorkspace: null, executionWorkspace: { baseCwd: home, cwd: home, source: "agent_home", projectId: null, workspaceId: null, repoUrl: null, repoRef: null, strategy: "project_primary", branchName: null, worktreePath: null, warnings: [], created: false, branchCreatedByRuntime: false }, @@ -259,8 +260,8 @@ describe.skipIf(!support.supported)("heartbeat runtime execution boundary", () = } }); - it("retains an unexported native source before provider-error teardown", async () => { - const { input, run } = await fixture(true); + it.each(["codex", "grok_bot", "openai_dot"] as const)("retains user files only when %s can access the workspace after a dispatch failure", async provider => { + const { input, run } = await fixture(true, provider); input.workspace.nativeWorkspaceSync = { mode: "host_current", reference: { schema: "paperclip.native-workspace-sync/v3", state: "prepared", @@ -271,7 +272,8 @@ describe.skipIf(!support.supported)("heartbeat runtime execution boundary", () = const error = new Error("runner_remote_process_identity_unavailable"); vi.mocked(executePaperclipNativeSession).mockRejectedValueOnce(error); input.instructions.releaseInstructionCopy = vi.fn(async () => { - expect(input.effects.onProviderResourceDisposition).toHaveBeenCalledWith("stop_and_retain"); + if (provider === "codex") expect(input.effects.onProviderResourceDisposition).toHaveBeenCalledWith("stop_and_retain"); + else expect(input.effects.onProviderResourceDisposition).not.toHaveBeenCalledWith("stop_and_retain"); }); await expect(executeHeartbeatRuntime(db, input)).rejects.toBe(error); expect(input.workspace.nativeWorkspaceSync.restoreWorkspace).not.toHaveBeenCalled(); diff --git a/server/src/services/heartbeat/runtime-execution.ts b/server/src/services/heartbeat/runtime-execution.ts index bbd31e4ee8a..c2971935867 100644 --- a/server/src/services/heartbeat/runtime-execution.ts +++ b/server/src/services/heartbeat/runtime-execution.ts @@ -995,7 +995,8 @@ export async function executeHeartbeatRuntime(db: Db, input: HeartbeatRuntimeExe input.effects.onNativeOwnershipHeld(); throw adapterErr; } - if (nativeRuntimeResolution.kind === "native" && nativeWorkspaceSync?.reference.state === "prepared") { + if (nativeRuntimeResolution.kind === "native" && nativeWorkspaceSync?.reference.state === "prepared" + && !(nativeExecution && "access" in nativeExecution.workspace && nativeExecution.workspace.access === "none")) { input.effects.onProviderResourceDisposition("stop_and_retain"); } await db diff --git a/server/src/services/native-runtime/native-failed-workspace-recovery.test.ts b/server/src/services/native-runtime/native-failed-workspace-recovery.test.ts index bd48f561e43..aaf7d5d88e5 100644 --- a/server/src/services/native-runtime/native-failed-workspace-recovery.test.ts +++ b/server/src/services/native-runtime/native-failed-workspace-recovery.test.ts @@ -76,6 +76,58 @@ const support = externalUrl ? { supported: true } : await getEmbeddedPostgresTes eq(activityLog.action, "issue.workspace_recovery_required")))).toHaveLength(1); }); + async function bindExternalInput(f: Awaited>, provider: "grok_bot" | "openai_dot", change?: (input: { schema: string; binding: { runId: string; companyId: string; agentId: string }; workspace: { access: string } }) => void) { + const run = await readRun(f.runId); + const input = { schema: "paperclip.native-execution-input.v6", + binding: { runId: f.runId, companyId: f.companyId, agentId: f.agentId, issueId: f.issueId }, + provider: { kind: provider }, workspace: { access: "none", cwd: null, repoUrl: null, repoRef: null, branchName: null } }; + change?.(input); + const [updated] = await db.update(heartbeatRuns).set({ driverKind: provider === "grok_bot" ? "grok_bot_mcp" : "openai_dot_mcp", + runnerProfileJson: { ...run.runnerProfileJson, nativeExecutionInput: input } }).where(eq(heartbeatRuns.id, f.runId)).returning(); + return updated; + } + + it.each(["grok_bot", "openai_dot"] as const)("does not invent file recovery for %s admitted without workspace access", async provider => { + const f = await seed(), run = await bindExternalInput(f, provider); + await preserveNativeFailedWorkspace(db, run); + expect((await readRun(f.runId)).resultJson).not.toHaveProperty("workspaceRestoreFailure"); + expect(await readLease(f.lease.id)).toMatchObject({ status: "active", leasePolicy: "ephemeral" }); + expect(await actions(f.issueId)).toHaveLength(0); + }); + + it.each(["run", "company", "agent", "schema", "write_access"])("preserves file safeguards when no-workspace evidence has an invalid %s", async mismatch => { + const f = await seed(); + const run = await bindExternalInput(f, "grok_bot", input => { + if (mismatch === "run") input.binding.runId = randomUUID(); + if (mismatch === "company") input.binding.companyId = randomUUID(); + if (mismatch === "agent") input.binding.agentId = randomUUID(); + if (mismatch === "schema") input.schema = "unknown"; + if (mismatch === "write_access") input.workspace.access = "read_write"; + }); + await preserveNativeFailedWorkspace(db, run); + expect((await readRun(f.runId)).resultJson).toHaveProperty("workspaceRestoreFailure", "restore_failed"); + expect(await readLease(f.lease.id)).toMatchObject({ leasePolicy: "retain_on_failure" }); + }); + + it.each(["grok_bot", "openai_dot"] as const)("keeps historical %s inspection holds with an available recovery path", async provider => { + const f = await seed(); + // Reproduce a hold persisted by an older cancellation path, then attach its admitted contract. + await preserveNativeFailedWorkspace(db, f.run); + const lease = await readLease(f.lease.id); + await settleStopOnlyCleanup(db, lease, { attemptId: String(lease.metadata?.pendingCleanupAttemptId), + receipt: { providerLeaseId: lease.providerLeaseId, state: "stopped" } }); + await bindExternalInput(f, provider); + await db.update(heartbeatRuns).set({ status: "cancelled" }).where(eq(heartbeatRuns.id, f.runId)); + const blocker = await getExecutionBlocker(db, f.companyId, f.issueId); + expect(blocker).toMatchObject({ runId: f.runId, canRetry: false, canContinue: false, + workspaceRepairRequired: provider !== "grok_bot", canRecordWorkspaceRepair: provider !== "grok_bot" }); + if (provider === "grok_bot") { + expect(blocker?.nextAction).not.toContain("files"); + expect(blocker?.nextAction).toContain("Runtime"); + } + expect(await actions(f.issueId)).toHaveLength(1); + }); + it("records verified file repair without retrying the native run or changing its task", async () => { const f = await seed(); await db.update(issues).set({ status: "done", assigneeAgentId: null }).where(eq(issues.id, f.issueId)); diff --git a/server/src/services/native-runtime/native-failed-workspace-recovery.ts b/server/src/services/native-runtime/native-failed-workspace-recovery.ts index 348c016309d..b36df60f526 100644 --- a/server/src/services/native-runtime/native-failed-workspace-recovery.ts +++ b/server/src/services/native-runtime/native-failed-workspace-recovery.ts @@ -1,7 +1,7 @@ import { and, eq, gt, isNull, sql } from "drizzle-orm"; import { environmentLeases, heartbeatRuns, issues, nativeRunFinalizations, type Db } from "@paperclipai/db"; import { recordRequiredWorkspaceRestoreFailure } from "../legacy-execution-recovery.js"; -import { hasRequiredWorkspaceRecovery } from "../workspace-restore-recovery-state.js"; +import { hasRequiredWorkspaceRecovery, nativeRunHasNoWorkspaceAccess } from "../workspace-restore-recovery-state.js"; import { logActivity } from "../activity-log.js"; import { readNativeWorkspaceSyncReference } from "./native-workspace-sync.js"; import type { NativeRestartRecoveryClaim } from "./native-restart-recovery.js"; @@ -45,9 +45,12 @@ export async function preserveNativeFailedWorkspace(db: Db, selected: typeof hea )).limit(1); if (!source || source.issueId !== issueId || source.metadata?.remoteCwd !== currentReference.remoteCwd || (source.executionWorkspaceId && source.executionWorkspaceId !== currentReference.workspaceId)) return; + // Ownership settlement still applies to workspace-less external sessions. + // Their Runner descriptor is not evidence that user files need exporting. + const retainFiles = !nativeRunHasNoWorkspaceAccess(run); const alreadyRetained = hasRequiredWorkspaceRecovery(run.resultJson); - let retained = alreadyRetained ? run : await recordRequiredWorkspaceRestoreFailure(tx as unknown as Db, run, { workspaceRestoreFailure: "restore_failed" }, source); - if (!retained || !hasRequiredWorkspaceRecovery(retained.resultJson)) return; + let retained = !retainFiles || alreadyRetained ? run : await recordRequiredWorkspaceRestoreFailure(tx as unknown as Db, run, { workspaceRestoreFailure: "restore_failed" }, source); + if (!retained || retainFiles && !hasRequiredWorkspaceRecovery(retained.resultJson)) return; if (options.allowPreterminal && coordinator.phase === "observed") { const [settled] = await tx.update(nativeRunFinalizations).set({ phase: "terminal_failure", leaseOwner: null, leaseExpiresAt: null, controlDeadlineAt: null, nextAttemptAt: null, @@ -62,7 +65,7 @@ export async function preserveNativeFailedWorkspace(db: Db, selected: typeof hea [retained] = await tx.update(heartbeatRuns).set({ nativePhase: "terminal_failure", nativePhaseUpdatedAt: new Date() }) .where(and(eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, run.companyId))).returning(); } - if (!alreadyRetained) await logActivity(tx as unknown as Db, { + if (retainFiles && !alreadyRetained) await logActivity(tx as unknown as Db, { companyId: run.companyId, actorType: "system", actorId: "native-workspace-recovery", action: "issue.workspace_recovery_required", entityType: "issue", entityId: issueId, diff --git a/server/src/services/workspace-restore-recovery-state.ts b/server/src/services/workspace-restore-recovery-state.ts index 9c532a9ca77..615bd13c54c 100644 --- a/server/src/services/workspace-restore-recovery-state.ts +++ b/server/src/services/workspace-restore-recovery-state.ts @@ -1,8 +1,26 @@ +import type { heartbeatRuns } from "@paperclipai/db"; import { hasWorkspaceRestoreFailure } from "@paperclipai/shared"; export const LEGACY_WORKSPACE_RECOVERY_KEY = "workspaceRestoreRecovery"; export const LEGACY_WORKSPACE_RECOVERY_SCHEMA = "paperclip.workspace-restore-recovery.v1"; +/** The admitted external contract can prohibit workspace access even though its + * managed Runner has a synchronization descriptor for durable recovery. That + * descriptor alone does not create a user-file export obligation. */ +export function nativeRunHasNoWorkspaceAccess(run: Pick): boolean { + const input = run.runnerProfileJson?.nativeExecutionInput as Record | undefined; + if (run.runtimeMode !== "native" || input?.schema !== "paperclip.native-execution-input.v6") return false; + const binding = input.binding as Record | undefined; + const workspace = input.workspace as Record | undefined; + const provider = input.provider as Record | undefined; + const driver = provider?.kind === "grok_bot" ? "grok_bot_mcp" + : provider?.kind === "openai_dot" ? "openai_dot_mcp" : null; + return driver !== null && run.driverKind === driver && workspace?.access === "none" && workspace.cwd === null + && binding?.runId === run.id && binding.companyId === run.companyId && binding.agentId === run.agentId + && binding.issueId === (run.nativeIssueId ?? run.issueId); +} + export function hasRequiredWorkspaceRecovery(result: Record | null | undefined): boolean { const value = result?.[LEGACY_WORKSPACE_RECOVERY_KEY]; return hasWorkspaceRestoreFailure(result) && !!value && typeof value === "object" && !Array.isArray(value) From 69eb3b6eff4ef00453b623f89abc8a34c8172a9f Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Thu, 15 Oct 2026 09:01:04 -0500 Subject: [PATCH 04/19] perf: reduce repository admission filesystem and write overhead (#16425) Reduce repository preparation latency by resolving each distinct workspace path once per scan, loading overlapping root provenance, and batching source retention writes. Preserve company authorization, fresh alias checks, and preparation timing. Add a reproducible benchmark and regressions for source identity retention, symlink retargeting, and bounded filesystem and database work. Co-Authored-By: Paperclip --- doc/DATABASE.md | 2 + doc/DEVELOPING.md | 23 ++ package.json | 1 + .../scripts/benchmark-workspace-admission.ts | 282 ++++++++++++++++++ .../repository-root-reader-queries.test.ts | 22 ++ ...task-workspace-source-copy-privacy.test.ts | 27 ++ .../execution-workspace-repositories.ts | 115 +++---- .../execution-workspace-source-roots.ts | 21 ++ 8 files changed, 442 insertions(+), 51 deletions(-) create mode 100644 server/scripts/benchmark-workspace-admission.ts diff --git a/doc/DATABASE.md b/doc/DATABASE.md index d684b1c7e13..18a686ce799 100644 --- a/doc/DATABASE.md +++ b/doc/DATABASE.md @@ -884,6 +884,8 @@ Migration [0338_absurd_vengeance.sql](../packages/db/src/migrations/0338_absurd_ Repository admission loads retained workspace and task sources in bounded batches. Readers share source rows and privacy evaluation for identical principals, but each retained run receives its own trust and responsible-user decision. The snapshot lasts only for one read batch; later admission and publication reload current authority. Keep source-loading query counts bounded as both workspace history and retained reader counts grow. +Each physical-root scan resolves duplicate workspace paths once, with bounded concurrency for distinct paths. Alias results are discarded after that scan so later admission and publication recheck symlink targets. Provenance identities select overlapping root records through the company/root index; source retention writes use bounded batches under the existing company lock, including deny-only ownership retained after a failed copy authorization. + Git-source capture checks the actual common Git storage directory and verified checkout owners. Separately stored metadata does not make its parent a checkout. When a linked worktree cannot identify its primary checkout, the service verifies diff --git a/doc/DEVELOPING.md b/doc/DEVELOPING.md index b82af99d5b5..b4b8437a284 100644 --- a/doc/DEVELOPING.md +++ b/doc/DEVELOPING.md @@ -674,6 +674,29 @@ Set `SEARCH_EVAL_SCALE=1` to additionally measure a disposable 10,000-task, 30,000-comment dataset. `SEARCH_EVAL_REPORT=/tmp/search-quality.json` saves per-query results and latency measurements; scale measurements are opt-in. +### Repository workspace admission benchmark + +Measure the real repository preparation service with a disposable embedded +PostgreSQL database and synthetic history, without invoking an agent: + +```sh +pnpm perf:workspace-admission +pnpm perf:workspace-admission -- --realpath-delay-ms 2.3 +pnpm perf:workspace-admission -- --distinct-cwds --realpath-delay-ms 2.3 +``` + +Defaults model 8,516 workspaces, 27,971 provenance records and 28 retained +readers on the shared root. The first command measures ordinary local I/O; +the others inject filesystem metadata latency for sensitivity comparisons. +Injected latency does not establish the actual latency of a hosted volume. +`--distinct-cwds` creates separate folders to exercise bounded concurrent I/O. +`--service /absolute/path/to/baseline/server/src/services/execution-workspace-repositories.ts` +compares another checkout, which needs its workspace dependencies installed. +The JSON report times only the service call, counts filesystem lookups and SQL +statements, and verifies retained source identities. Setup and cleanup are +excluded. Source tasks and permissions are synthetic; this is not an end-to-end +agent latency measurement. + ### Recent task ordering The streamlined sidebar keeps five recent tasks per company and account in browser diff --git a/package.json b/package.json index 117c6439ecc..32ed9601455 100644 --- a/package.json +++ b/package.json @@ -3,6 +3,7 @@ "private": true, "type": "module", "scripts": { + "perf:workspace-admission": "node --import ./server/node_modules/tsx/dist/loader.mjs server/scripts/benchmark-workspace-admission.ts", "postinstall": "node scripts/link-plugin-dev-sdk.mjs", "preflight:workspace-links": "node cli/node_modules/tsx/dist/cli.mjs scripts/ensure-workspace-package-links.ts", "dev": "pnpm --filter @paperclipai/server exec tsx ../scripts/dev-runner.ts watch", diff --git a/server/scripts/benchmark-workspace-admission.ts b/server/scripts/benchmark-workspace-admission.ts new file mode 100644 index 00000000000..d48ce416f3c --- /dev/null +++ b/server/scripts/benchmark-workspace-admission.ts @@ -0,0 +1,282 @@ +import assert from "node:assert/strict"; +import { randomUUID, createHash } from "node:crypto"; +import { execFile as execFileCallback } from "node:child_process"; +import { promisify } from "node:util"; +import { mkdtemp, rm, mkdir } from "node:fs/promises"; +import fs from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; +import { pathToFileURL } from "node:url"; +import { and, eq, inArray } from "drizzle-orm"; +import { drizzle } from "drizzle-orm/postgres-js"; +import { + companies, + createDb, + executionWorkspaceRepositoryAccess, + executionWorkspaces, + heartbeatRuns, + issues, + startEmbeddedPostgresTestDatabase, +} from "@paperclipai/db"; + +import { createAgentLifecycle } from "../src/services/agent-lifecycle.js"; + +const execFile = promisify(execFileCallback); + +function option(name: string, fallback: string) { + const index = process.argv.indexOf(name); + return index < 0 ? fallback : process.argv[index + 1] ?? fallback; +} + +function latency(name: string, fallback: number) { + const value = Number(option(name, String(fallback))); + if (!Number.isFinite(value) || value < 0) throw new Error(`${name} must be a non-negative number`); + return value; +} + +function count(name: string, fallback: number) { + const value = Number(option(name, String(fallback))); + if (!Number.isSafeInteger(value) || value < 0) throw new Error(`${name} must be a non-negative integer`); + return value; +} + +async function insertBatches>( + insert: (values: T[]) => Promise, + values: T[], +) { + for (let offset = 0; offset < values.length; offset += 500) { + await insert(values.slice(offset, offset + 500)); + } +} + +const workspaceCount = count("--workspaces", 8_516); +const rootWorkspaceCount = count("--root-workspaces", 5_315); +const sourceEntryCount = count("--source-entries", 17_256); +const isolatedSourceEntryCount = count("--isolated-source-entries", 10_630); +const readerEntryCount = count("--readers", 51); +const rootReaderCount = count("--root-readers", 28); +const otherAccessEntryCount = count("--other-access-entries", 34); +const realpathDelayMs = latency("--realpath-delay-ms", 0); +const distinctCwds = process.argv.includes("--distinct-cwds"); +if (workspaceCount < 1 || rootWorkspaceCount < 1 || rootWorkspaceCount > workspaceCount) { + throw new Error("Need at least one workspace; root workspaces must be between one and the workspace total"); +} +if (readerEntryCount < rootReaderCount) { + throw new Error("Root reader count cannot exceed the company-wide reader total"); +} + +const configuredService = option("--service", ""); +const serviceUrl = configuredService + ? pathToFileURL(path.resolve(configuredService)) + : new URL("../src/services/execution-workspace-repositories.ts", import.meta.url); + +let measuring = false; +let realpathCalls = 0; +let realpathIoMs = 0; +let injectedWaitMs = 0; +const actualRealpath = fs.realpath.bind(fs); +fs.realpath = (async (...args: Parameters) => { + if (!measuring) return actualRealpath(...args); + const started = performance.now(); + try { + return await actualRealpath(...args); + } finally { + realpathCalls += 1; + realpathIoMs += performance.now() - started; + if (realpathDelayMs > 0) { + const waitStarted = performance.now(); + await new Promise((resolve) => setTimeout(resolve, realpathDelayMs)); + injectedWaitMs += performance.now() - waitStarted; + } + } +}) as typeof fs.realpath; + +const { executionWorkspaceRepositoryService } = await import(serviceUrl.href); + +const temp = await startEmbeddedPostgresTestDatabase("paperclip-workspace-admission-bench-"); +const db = createDb(temp.connectionString); +const scratchRoot = await fs.realpath(await mkdtemp(path.join(os.tmpdir(), "paperclip-workspace-admission-"))); +const repoRoot = path.join(scratchRoot, "shared-repository"); +const unrelatedRoot = path.join(scratchRoot, "unrelated-workspaces"); + +try { + await mkdir(repoRoot, { recursive: true }); + await mkdir(unrelatedRoot, { recursive: true }); + await execFile("git", ["init", "--quiet", "--initial-branch=main", repoRoot]); + + const companyId = randomUUID(); + const agentId = randomUUID(); + const issueId = randomUUID(); + const workspaceIds = Array.from({ length: workspaceCount }, () => randomUUID()); + const workspaceCwds = workspaceIds.map((_, index) => { + const root = index < rootWorkspaceCount ? repoRoot : unrelatedRoot; + return distinctCwds && index !== 0 ? path.join(root, `workspace-${index}`) : root; + }); + if (distinctCwds) for (let offset = 0; offset < workspaceCwds.length; offset += 64) { + await Promise.all(workspaceCwds.slice(offset, offset + 64).map(cwd => mkdir(cwd, { recursive: true }))); + } + const runIds = Array.from({ length: Math.max(1, rootReaderCount) }, () => randomUUID()); + const now = new Date(); + const rootKey = createHash("sha256").update(repoRoot).digest("hex"); + const unrelatedKey = createHash("sha256").update(unrelatedRoot).digest("hex"); + + await db.insert(companies).values({ id: companyId, name: "Synthetic workspace-admission benchmark", issuePrefix: "WAB" }); + await createAgentLifecycle(db).requestHire(companyId, { + id: agentId, + name: "Synthetic Codie", + role: "engineer", + status: "idle", + adapterType: "codex_local", + adapterConfig: {}, + runtimeConfig: {}, + permissions: {}, + }, { runnerResolved: true }); + + await insertBatches((rows) => db.insert(executionWorkspaces).values(rows as never), workspaceIds.map((id, index) => ({ + id, + companyId, + name: `Synthetic workspace ${index}`, + mode: "shared_workspace", + strategyType: "project_primary", + status: "active", + cwd: workspaceCwds[index], + providerType: "local_fs", + }))); + + await db.insert(issues).values({ + id: issueId, + companyId, + title: "Synthetic assigned task", + status: "in_progress", + assigneeAgentId: agentId, + executionWorkspaceId: workspaceIds[0], + responsibleUserId: "local-board", + }); + for (let offset = 0; offset < rootWorkspaceCount; offset += 500) { + await db.update(executionWorkspaces).set({ sourceIssueId: issueId }) + .where(and(eq(executionWorkspaces.companyId, companyId), inArray(executionWorkspaces.id, workspaceIds.slice(offset, offset + 500)))); + } + + if (runIds.length) { + await db.insert(heartbeatRuns).values(runIds.map((id) => ({ + id, + companyId, + agentId, + scopeKind: "issue" as const, + issueId, + nativeIssueId: issueId, + invocationSource: "on_demand", + status: "completed", + responsibleUserId: "local-board", + runtimeMode: "legacy", + contextSnapshot: { issueId }, + startedAt: now, + finishedAt: now, + }))); + } + + const accessRows: Array = []; + for (let index = 0; index < sourceEntryCount; index++) { + const isRootSource = index < rootWorkspaceCount; + accessRows.push({ + companyId, + rootKey: isRootSource ? rootKey : unrelatedKey, + canonicalCwd: isRootSource ? repoRoot : unrelatedRoot, + kind: "source", + scopeId: isRootSource ? workspaceIds[index] : randomUUID(), + }); + } + for (let index = 0; index < isolatedSourceEntryCount; index++) { + accessRows.push({ companyId, rootKey: unrelatedKey, canonicalCwd: unrelatedRoot, kind: "isolated_source", scopeId: randomUUID() }); + } + for (let index = 0; index < readerEntryCount; index++) { + const rootReader = index < rootReaderCount; + accessRows.push({ + companyId, + rootKey: rootReader ? rootKey : unrelatedKey, + canonicalCwd: rootReader ? repoRoot : unrelatedRoot, + kind: "reader", + scopeId: rootReader ? runIds[index] : randomUUID(), + issueId: rootReader ? issueId : randomUUID(), + agentId: rootReader ? agentId : randomUUID(), + responsibleUserId: "local-board", + }); + } + for (let index = 0; index < otherAccessEntryCount; index++) { + accessRows.push({ companyId, rootKey: unrelatedKey, canonicalCwd: unrelatedRoot, kind: "repository", scopeId: randomUUID() }); + } + await insertBatches((rows) => db.insert(executionWorkspaceRepositoryAccess).values(rows as never), accessRows); + + let queryCount = 0; + let provenanceInsertCount = 0; + const measuredDb = drizzle(db.$client, { + schema: db._.fullSchema, + logger: { logQuery(query: string) { + if (!measuring) return; + queryCount++; + if (query.startsWith('insert into "execution_workspace_repository_access"')) provenanceInsertCount++; + } }, + }) as typeof db; + const service = executionWorkspaceRepositoryService(measuredDb); + realpathCalls = 0; + realpathIoMs = 0; + process.stderr.write(`Starting admission benchmark: ${workspaceCount} workspaces, ${rootReaderCount} retained readers, ${realpathDelayMs} ms injected per realpath.\n`); + measuring = true; + const startedAt = performance.now(); + await service.prepareConfiguredRepositorySource({ + companyId, + issueId, + workspaceId: workspaceIds[0], + cwd: repoRoot, + sourceCwd: repoRoot, + agentId, + runId: runIds[0] ?? randomUUID(), + }); + const elapsedMs = Math.round(performance.now() - startedAt); + measuring = false; + process.stderr.write(`Admission service call completed in ${elapsedMs} ms.\n`); + + const retained = await db.select({ kind: executionWorkspaceRepositoryAccess.kind, scopeId: executionWorkspaceRepositoryAccess.scopeId }) + .from(executionWorkspaceRepositoryAccess).where(and(eq(executionWorkspaceRepositoryAccess.companyId, companyId), + eq(executionWorkspaceRepositoryAccess.rootKey, rootKey), eq(executionWorkspaceRepositoryAccess.kind, "isolated_source"))); + const retainedIds = new Set(retained.map(row => row.scopeId)); + assert.equal(retainedIds.size, rootWorkspaceCount, "Every source must retain isolated-copy provenance"); + assert.ok(workspaceIds.slice(0, rootWorkspaceCount).every(id => retainedIds.has(id)), "Source provenance identities must match"); + const metadataRoot = await fs.realpath(path.join(repoRoot, ".git")); + const metadataOwners = await db.select({ scopeId: executionWorkspaceRepositoryAccess.scopeId }) + .from(executionWorkspaceRepositoryAccess).where(and(eq(executionWorkspaceRepositoryAccess.companyId, companyId), + eq(executionWorkspaceRepositoryAccess.rootKey, createHash("sha256").update(metadataRoot).digest("hex")), + eq(executionWorkspaceRepositoryAccess.kind, "source"))); + const metadataOwnerIds = new Set(metadataOwners.map(row => row.scopeId)); + assert.equal(metadataOwnerIds.size, rootWorkspaceCount, "Git metadata must retain every source owner"); + assert.ok(workspaceIds.slice(0, rootWorkspaceCount).every(id => metadataOwnerIds.has(id)), "Git metadata source identities must match"); + const report = { + elapsedMs, + database: { queries: queryCount, provenanceInserts: provenanceInsertCount }, + realpath: { + calls: realpathCalls, + measuredIoMs: Math.round(realpathIoMs), + injectedDelayPerCallMs: realpathDelayMs, + summedInjectedWaitMs: Math.round(injectedWaitMs), + }, + fixture: { + workspaces: workspaceCount, + sharedRootWorkspaces: rootWorkspaceCount, + distinctWorkspaceCwds: new Set(workspaceCwds).size, + repositoryAccessRows: accessRows.length, + sourceRows: sourceEntryCount, + isolatedSourceRows: isolatedSourceEntryCount, + retainedReaders: rootReaderCount, + allCompanyReaders: readerEntryCount, + otherAccessRows: otherAccessEntryCount, + }, + service: serviceUrl.pathname, + note: "Synthetic local service benchmark only; no provider invocation.", + }; + await new Promise((resolve, reject) => { + process.stdout.write(`${JSON.stringify(report, null, 2)}\n`, (error) => error ? reject(error) : resolve()); + }); +} finally { + await temp.cleanup(); + await rm(scratchRoot, { recursive: true, force: true }); +} diff --git a/server/src/__tests__/repository-root-reader-queries.test.ts b/server/src/__tests__/repository-root-reader-queries.test.ts index 4433b2f50f0..5bae7040a4b 100644 --- a/server/src/__tests__/repository-root-reader-queries.test.ts +++ b/server/src/__tests__/repository-root-reader-queries.test.ts @@ -1,5 +1,6 @@ import { createHash, randomUUID } from "node:crypto"; import { mkdtemp, realpath, rm } from "node:fs/promises"; +import fs from "node:fs/promises"; import { tmpdir } from "node:os"; import path from "node:path"; import { and, eq, inArray } from "drizzle-orm"; @@ -113,3 +114,24 @@ it("keeps supplied membership snapshots separate from fresh database snapshots", await db.update(companyMemberships).set({ status: "active" }).where(and(eq(companyMemberships.companyId, companyId), eq(companyMemberships.principalId, userId))); } }); + +it("batches source retention and duplicate path reads without losing owners", async () => { + const sourceIds = Array.from({ length: 501 }, () => randomUUID()); + await db.insert(executionWorkspaces).values(sourceIds.map(id => ({ id, companyId, projectId, cwd, + sourceIssueId: issueId, name: "Shared source", mode: "shared_workspace", strategyType: "operator_directory" }))); + const queries: string[] = []; + const recorded = drizzle(db.$client, { schema: db._.fullSchema, logger: { logQuery: query => queries.push(query) } }) as unknown as typeof db; + const paths = vi.spyOn(fs, "realpath"); + try { + await executionWorkspaceRepositoryService(recorded).guardForPublication(input()); + expect(queries.filter(query => query.startsWith('insert into "execution_workspace_repository_access"')).length).toBeLessThanOrEqual(2); + expect(paths.mock.calls.length).toBeLessThanOrEqual(8); + } finally { + paths.mockRestore(); + } + const retained = await db.select({ scopeId: executionWorkspaceRepositoryAccess.scopeId }).from(executionWorkspaceRepositoryAccess) + .where(and(eq(executionWorkspaceRepositoryAccess.companyId, companyId), + eq(executionWorkspaceRepositoryAccess.rootKey, createHash("sha256").update(cwd).digest("hex")), + eq(executionWorkspaceRepositoryAccess.kind, "source"))); + expect(new Set(retained.map(row => row.scopeId))).toEqual(new Set([workspaceId, ...sourceIds])); +}); diff --git a/server/src/__tests__/task-workspace-source-copy-privacy.test.ts b/server/src/__tests__/task-workspace-source-copy-privacy.test.ts index ad2e87b789b..5c6e7e7a851 100644 --- a/server/src/__tests__/task-workspace-source-copy-privacy.test.ts +++ b/server/src/__tests__/task-workspace-source-copy-privacy.test.ts @@ -691,6 +691,33 @@ exec "$PROBE_REAL_GIT" "$@" await expect(f.service.prepareForAdmission(f.admission)).resolves.toEqual([]); }); + it("rechecks retargeted execution-workspace aliases at admission and publication", async () => { + const f = await fixture(), peer = await standardPeer(f); + const unrelated = path.join(root, `${f.issueId}-unrelated`); await mkdir(unrelated); + const alias = path.join(root, `${f.issueId}-alias`); await symlink(unrelated, alias); + await db.update(executionWorkspaces).set({ cwd: alias }).where(eq(executionWorkspaces.id, peer.workspace.id)); + await db.update(issues).set({ visibility: "private", privacyRootIssueId: peer.issueId, responsibleUserId: peer.userId }) + .where(eq(issues.id, peer.issueId)); + await expect(f.service.prepareForAdmission(f.admission)).resolves.toEqual([]); + await rm(alias); + await symlink(f.admission.cwd, alias); + await expect(assertExecutionWorkspaceRootReadAccess(db, { companyId: f.companyId, cwd: f.admission.cwd, actor: f.actor })) + .rejects.toThrow(); + await expect(f.service.prepareForAdmission(f.admission)).rejects.toThrow(); + await expect(f.service.guardForPublication(f.admission)).rejects.toThrow(); + }); + + it.each(["conflicting_path", "overlapping_invalid_key"] as const)("rejects retained root identity corruption (%s)", async scenario => { + const f = await fixture(); + const canonicalCwd = await realpath(f.admission.cwd); + const unrelated = path.join(root, `${f.issueId}-unrelated`); await mkdir(unrelated); + await db.insert(executionWorkspaceRepositoryAccess).values({ companyId: f.companyId, kind: "source", scopeId: f.workspace.id, + rootKey: createHash("sha256").update(scenario === "conflicting_path" ? canonicalCwd : await realpath(unrelated)).digest("hex"), + canonicalCwd: scenario === "conflicting_path" ? await realpath(unrelated) : path.join(canonicalCwd, "child") }); + await expect(f.service.prepareForAdmission(f.admission)).rejects.toThrow("Repository root identity conflicts"); + await expect(f.service.guardForPublication(f.admission)).rejects.toThrow("Repository root identity conflicts"); + }); + it.each(["revoked", "deleted_source", "deleted_owner", "retained_reader"] as const)("protects already-copied bytes when post-copy authorization fails (%s)", async failure => { const f = await fixture(), exec = promisify(execFileCallback); await exec("git", ["init", "--initial-branch=main", f.admission.cwd]); diff --git a/server/src/services/execution-workspace-repositories.ts b/server/src/services/execution-workspace-repositories.ts index 60acd1f81d8..aeef482fbdf 100644 --- a/server/src/services/execution-workspace-repositories.ts +++ b/server/src/services/execution-workspace-repositories.ts @@ -22,13 +22,34 @@ import { normalizeProjectRepositoryUrl } from "./project-repositories.js"; import { buildGitAuthInvocation, createGitRemoteAuthProvider } from "./git-credentials.js"; import { materializeManagedProjectWorkspace, ensureManagedRepositoriesIgnored } from "./managed-repository-checkout.js"; -import { canonicalRepositoryRoot, repositoryRootsOverlap, legacyRepositoryRootMayMatch, configuredWorkspaceRoot, verifiedIsolatedWorktree, unusedIsolatedCopyTarget, isolatedWorkspaceSourceRoots, type WorkspaceSourceCopyKind } from "./execution-workspace-source-roots.js"; +import { canonicalRepositoryRoot, resolveRepositoryRoots, repositoryRootsOverlap, legacyRepositoryRootMayMatch, configuredWorkspaceRoot, verifiedIsolatedWorktree, unusedIsolatedCopyTarget, isolatedWorkspaceSourceRoots, type WorkspaceSourceCopyKind } from "./execution-workspace-source-roots.js"; export { canonicalRepositoryRoot } from "./execution-workspace-source-roots.js"; const execFile = promisify(execFileCallback); type RepositoryRow = typeof executionWorkspaceRepositories.$inferSelect; type DbTransaction = Parameters[0]>[0]; +function repositoryAccessKeys(entries: Array>) { + return new Set(entries.map(entry => `${entry.kind}:${entry.scopeId}`)); +} + +async function retainRepositorySources(tx: DbTransaction, + root: { companyId: string; canonicalCwd: string; rootKey: string }, + existingKeys: Set, + groups: ReadonlyArray]>, +): Promise { + const additions: Array = []; + for (const [kind, ids] of groups) for (const scopeId of ids) { + const key = `${kind}:${scopeId}`; + if (existingKeys.has(key)) continue; + additions.push({ ...root, kind, scopeId }); + existingKeys.add(key); + } + for (let offset = 0; offset < additions.length; offset += 500) { + await measureRunPhase("workspace.write", () => tx.insert(executionWorkspaceRepositoryAccess).values(additions.slice(offset, offset + 500))); + } +} + const controllerAcquisitionMessage = "New controller repository acquisition requires a trusted task. Existing verified public checkouts remain available."; const lowTrustRepositoryMessage = "Low-trust repository reuse permits anonymous public URLs only; new acquisition is unavailable. Use an authorized configured source for private repositories."; const rootReaderRunColumns = { @@ -69,9 +90,22 @@ async function resolveRepositoryRootSources(tx: Db | DbTransaction, input: { }) { const canonicalCwd = await measureRunPhase("workspace.sources.canonicalize", () => canonicalRepositoryRoot(input.cwd)); const rootKey = createHash("sha256").update(canonicalCwd).digest("hex"); - const companyEntries = await measureRunPhase("workspace.sources.inventory", async () => tx.select().from(executionWorkspaceRepositoryAccess) - .where(eq(executionWorkspaceRepositoryAccess.companyId, input.companyId))); + const companyEntries = await measureRunPhase("workspace.sources.inventory", async () => { + const identities = await tx.selectDistinct({ rootKey: executionWorkspaceRepositoryAccess.rootKey, + canonicalCwd: executionWorkspaceRepositoryAccess.canonicalCwd }).from(executionWorkspaceRepositoryAccess) + .where(eq(executionWorkspaceRepositoryAccess.companyId, input.companyId)); + const rootKeys = [...new Set([rootKey, ...identities.filter(identity => repositoryRootsOverlap(identity.canonicalCwd, canonicalCwd)) + .map(identity => identity.rootKey)])]; + const retained: Array = []; + for (let offset = 0; offset < rootKeys.length; offset += 500) { + retained.push(...await tx.select().from(executionWorkspaceRepositoryAccess).where(and( + eq(executionWorkspaceRepositoryAccess.companyId, input.companyId), + inArray(executionWorkspaceRepositoryAccess.rootKey, rootKeys.slice(offset, offset + 500))))); + } + return retained; + }); const entries = companyEntries.filter(entry => entry.rootKey === rootKey); + const entryKeys = repositoryAccessKeys(entries); const overlappingEntries = companyEntries.filter(entry => repositoryRootsOverlap(entry.canonicalCwd, canonicalCwd)); if (entries.some(entry => entry.canonicalCwd !== canonicalCwd) || overlappingEntries.some(entry => createHash("sha256").update(entry.canonicalCwd).digest("hex") !== entry.rootKey)) { @@ -99,12 +133,10 @@ async function resolveRepositoryRootSources(tx: Db | DbTransaction, input: { const workspaces = await measureRunPhase("workspace.sources.inventory", async () => tx.select({ id: executionWorkspaces.id, cwd: executionWorkspaces.cwd }).from(executionWorkspaces) .where(eq(executionWorkspaces.companyId, input.companyId))); await measureRunPhase("workspace.sources.alias_scan", async () => { + const workspaceRoots = await resolveRepositoryRoots(workspaces.flatMap(workspace => workspace.cwd ? [workspace.cwd] : [])); for (const workspace of workspaces) { if (!workspace.cwd) continue; - const peerRoot = await canonicalRepositoryRoot(workspace.cwd!).catch((error: NodeJS.ErrnoException) => { - if (["ENOTDIR", "EACCES", "EPERM", "ELOOP", "ENAMETOOLONG"].includes(error.code ?? "")) return null; - throw error; - }); + const peerRoot = workspaceRoots.get(workspace.cwd); if (peerRoot && repositoryRootsOverlap(peerRoot, canonicalCwd)) { sourceIds.add(workspace.id); sharedSourceIds.add(workspace.id); addSharedSource(workspace.id, peerRoot); } @@ -126,21 +158,20 @@ async function resolveRepositoryRootSources(tx: Db | DbTransaction, input: { } for (const name of await fs.readdir(namespace)) if (name.startsWith("task-repo-")) publishedNames.add(name); } - const repositoryOrigins = await measureRunPhase("workspace.sources.inventory", async () => tx.select({ id: executionWorkspaceRepositoryAccess.scopeId }).from(executionWorkspaceRepositoryAccess) - .where(and(eq(executionWorkspaceRepositoryAccess.companyId, input.companyId), eq(executionWorkspaceRepositoryAccess.kind, "repository")))); + const repositoryOrigins = inventory.length ? await measureRunPhase("workspace.sources.inventory", async () => tx.select({ id: executionWorkspaceRepositoryAccess.scopeId }).from(executionWorkspaceRepositoryAccess) + .where(and(eq(executionWorkspaceRepositoryAccess.companyId, input.companyId), eq(executionWorkspaceRepositoryAccess.kind, "repository")))) : []; const knownOrigins = new Set(repositoryOrigins.map(origin => origin.id)); const recognizedNames = new Set(); await measureRunPhase("workspace.sources.repository_scan", async () => { + const inventoryRoots = await resolveRepositoryRoots(inventory.flatMap(peer => peer.cwd ? [peer.cwd] : []), true); + const hasReader = entries.some(entry => entry.kind === "reader"); for (const peer of inventory) { const name = path.basename(taskRepositoryRelativePath(peer.identity)); const published = publishedNames.has(name); - const peerRoot = peer.cwd ? await fs.realpath(peer.cwd).catch((error: NodeJS.ErrnoException) => { - if (["ENOENT", "ENOTDIR", "EACCES", "EPERM", "ELOOP", "ENAMETOOLONG"].includes(error.code ?? "")) return null; - throw error; - }) : null; + const peerRoot = peer.cwd ? inventoryRoots.get(peer.cwd)! : null; const matches = sourceIds.has(peer.workspaceId) || (peerRoot !== null && repositoryRootsOverlap(peerRoot, canonicalCwd)) || (published && peerRoot === null && !knownOrigins.has(peer.id)) - || (peerRoot === null && !entries.some(entry => entry.kind === "reader") && peer.cwd && await legacyRepositoryRootMayMatch(peer.cwd, canonicalCwd)); + || (peerRoot === null && !hasReader && peer.cwd && await legacyRepositoryRootMayMatch(peer.cwd, canonicalCwd)); if (matches) { sourceIds.add(peer.workspaceId); if (!isolatedSourceIds.has(peer.workspaceId) || (peerRoot && repositoryRootsOverlap(peerRoot, canonicalCwd))) { @@ -158,7 +189,7 @@ async function resolveRepositoryRootSources(tx: Db | DbTransaction, input: { const isolatedProjectIds = new Set(overlappingEntries.filter(entry => entry.kind === "isolated_project_source").map(entry => entry.scopeId)); if (sourceIds.size) { const owners = await measureRunPhase("workspace.sources.inventory", async () => tx.select({ id: executionWorkspaces.id, projectId: executionWorkspaces.projectId, strategyType: executionWorkspaces.strategyType }).from(executionWorkspaces).where(and( - eq(executionWorkspaces.companyId, input.companyId), or(...[...sourceIds].map(id => eq(executionWorkspaces.id, id)))))); + eq(executionWorkspaces.companyId, input.companyId), inArray(executionWorkspaces.id, [...sourceIds])))); for (const owner of owners) if (owner.projectId) { sourceProjectIds.add(owner.projectId); if (owner.strategyType === "git_worktree" || !sharedSourceIds.has(owner.id) @@ -178,7 +209,7 @@ async function resolveRepositoryRootSources(tx: Db | DbTransaction, input: { for (const entry of overlappingEntries) if (entry.kind === "project_source" && !isolatedProjectIds.has(entry.scopeId)) { sharedProjectIds.add(entry.scopeId); } - return { canonicalCwd, rootKey, entries, overlappingEntries, sourceIds, sharedSourceIds, isolatedSourceIds, sourceProjectIds, sharedProjectIds, isolatedProjectIds }; + return { canonicalCwd, rootKey, entries, entryKeys, overlappingEntries, sourceIds, sharedSourceIds, isolatedSourceIds, sourceProjectIds, sharedProjectIds, isolatedProjectIds }; } async function assertRootProjectAccess(db: Db | DbTransaction, actor: AuthorizationActor, companyId: string, projectIds: Set, @@ -245,16 +276,11 @@ export async function preserveExecutionWorkspaceRootProvenance(tx: DbTransaction }); if (!directory) return; if (!directory.isDirectory()) throw forbidden("Workspace files cannot be detached from unverifiable provenance"); - const { canonicalCwd, rootKey, entries, sharedSourceIds, isolatedSourceIds, sourceProjectIds, sharedProjectIds, isolatedProjectIds } = await measureRunPhase("workspace.sources", () => resolveRepositoryRootSources(tx, { + const { canonicalCwd, rootKey, entryKeys, sharedSourceIds, isolatedSourceIds, sourceProjectIds, sharedProjectIds, isolatedProjectIds } = await measureRunPhase("workspace.sources", () => resolveRepositoryRootSources(tx, { companyId: input.companyId, cwd: input.cwd!, sourceWorkspaceId: input.workspaceId, inspectCurrentAliases: true, })); - for (const [kind, ids] of [["source", sharedSourceIds], ["isolated_source", isolatedSourceIds], ["project_source", sourceProjectIds], ["shared_project_source", sharedProjectIds], ["isolated_project_source", isolatedProjectIds]] as const) { - for (const scopeId of ids) { - if (!entries.some(entry => entry.kind === kind && entry.scopeId === scopeId)) { - await measureRunPhase("workspace.write", () => tx.insert(executionWorkspaceRepositoryAccess).values({ companyId: input.companyId, canonicalCwd, rootKey, kind, scopeId })); - } - } - } + await retainRepositorySources(tx, { companyId: input.companyId, canonicalCwd, rootKey }, entryKeys, + [["source", sharedSourceIds], ["isolated_source", isolatedSourceIds], ["project_source", sourceProjectIds], ["shared_project_source", sharedProjectIds], ["isolated_project_source", isolatedProjectIds]]); } export async function assertExecutionWorkspaceRootReadAccess(db: Db | DbTransaction, input: { @@ -285,12 +311,8 @@ async function retainGitMetadataOwners(tx: DbTransaction, input: { } await measureRunPhase("workspace.readers.projects", () => assertRootProjectsReadAccessForActors(tx, readerActors, input.companyId, projectIds)); await measureRunPhase("workspace.readers.sources", () => assertTaskWorkspacesReadAccessForActors(tx, readerActors, input.companyId, workspaceIds)); - for (const [kind, ids] of [["source", workspaceIds], ["project_source", projectIds]] as const) { - for (const scopeId of ids) if (!closure.entries.some(entry => entry.kind === kind && entry.scopeId === scopeId)) { - await measureRunPhase("workspace.write", () => tx.insert(executionWorkspaceRepositoryAccess).values({ companyId: input.companyId, - canonicalCwd: closure.canonicalCwd, rootKey: closure.rootKey, kind, scopeId })); - } - } + await retainRepositorySources(tx, { companyId: input.companyId, canonicalCwd: closure.canonicalCwd, rootKey: closure.rootKey }, closure.entryKeys, + [["source", workspaceIds], ["project_source", projectIds]]); } export async function assertIsolatedWorkspaceSourceReadAccess(db: Db, input: { @@ -328,6 +350,7 @@ export async function retainIsolatedWorkspaceSourceAccess(db: Db, input: { const existing = await measureRunPhase("workspace.sources.inventory", async () => tx.select().from(executionWorkspaceRepositoryAccess).where(and( eq(executionWorkspaceRepositoryAccess.companyId, input.companyId), eq(executionWorkspaceRepositoryAccess.rootKey, rootKey)))); if (existing.some(entry => entry.canonicalCwd !== canonicalCwd)) throw forbidden("Repository root identity conflicts with its retained access record"); + const existingKeys = repositoryAccessKeys(existing); const workspaceIds = new Set(input.captured.workspaceIds), projectIds = new Set(input.captured.projectIds); let failure = sourceFailure; for (const cwd of roots) { @@ -343,11 +366,8 @@ export async function retainIsolatedWorkspaceSourceAccess(db: Db, input: { try { await measureRunPhase("workspace.readers.projects", () => assertRootProjectAccess(tx, input.actor!, input.companyId, new Set([id]))); } catch (error) { failure ??= error; } } - for (const [kind, ids] of [["isolated_source", workspaceIds], ["isolated_project_source", projectIds]] as const) { - for (const scopeId of ids) if (!existing.some(entry => entry.kind === kind && entry.scopeId === scopeId)) { - await measureRunPhase("workspace.write", () => tx.insert(executionWorkspaceRepositoryAccess).values({ companyId: input.companyId, rootKey, canonicalCwd, kind, scopeId })); - } - } + await retainRepositorySources(tx, { companyId: input.companyId, canonicalCwd, rootKey }, existingKeys, + [["isolated_source", workspaceIds], ["isolated_project_source", projectIds]]); const phase = input.phase ?? "preparing"; let preparingCopyProjectIds: Set | undefined; let verifiedCopyProjectIds: Set | undefined; @@ -554,9 +574,9 @@ export function executionWorkspaceRepositoryService(db: Db) { async function authorizeRootAccess(tx: DbTransaction, input: { companyId: string; cwd: string; sourceWorkspaceId?: string; reader?: RootReader; actor?: AuthorizationActor; issueId?: string; preserveOwnAdmission?: boolean; }) { - const { canonicalCwd, rootKey, entries, overlappingEntries, sourceIds, sharedSourceIds, isolatedSourceIds, sourceProjectIds, sharedProjectIds, isolatedProjectIds } = await measureRunPhase("workspace.sources", () => resolveRepositoryRootSources(tx, input)); + const { canonicalCwd, rootKey, entryKeys, overlappingEntries, sourceIds, sharedSourceIds, isolatedSourceIds, sourceProjectIds, sharedProjectIds, isolatedProjectIds } = await measureRunPhase("workspace.sources", () => resolveRepositoryRootSources(tx, input)); const readers = overlappingEntries.filter(entry => entry.kind === "reader"); - if (input.reader && !entries.some(entry => entry.kind === "reader" && entry.scopeId === input.reader!.runId)) { + if (input.reader && !entryKeys.has(`reader:${input.reader.runId}`)) { const [run] = await measureRunPhase("workspace.sources.inventory", async () => tx.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.id, input.reader!.runId), eq(heartbeatRuns.companyId, input.companyId)))); readers.push({ id: randomUUID(), companyId: input.companyId, rootKey, canonicalCwd, kind: "reader", scopeId: input.reader.runId, issueId: input.reader.issueId, agentId: input.reader.agentId, responsibleUserId: run?.responsibleUserId ?? null, createdAt: new Date(), updatedAt: new Date() }); @@ -600,17 +620,12 @@ export function executionWorkspaceRepositoryService(db: Db) { sourceActors.push(input.actor); } await measureRunPhase("workspace.readers.sources", () => assertTaskWorkspacesReadAccessForActors(tx, sourceActors, input.companyId, sourceIds)); - for (const [kind, ids] of [["source", sharedSourceIds], ["isolated_source", isolatedSourceIds], ["project_source", sourceProjectIds], - ["shared_project_source", sharedProjectIds], ["isolated_project_source", isolatedProjectIds]] as const) { - for (const scopeId of ids) { - if (!entries.some(entry => entry.kind === kind && entry.scopeId === scopeId)) { - await measureRunPhase("workspace.write", () => tx.insert(executionWorkspaceRepositoryAccess).values({ companyId: input.companyId, rootKey, canonicalCwd, kind, scopeId })); - } - } - } + await retainRepositorySources(tx, { companyId: input.companyId, canonicalCwd, rootKey }, entryKeys, + [["source", sharedSourceIds], ["isolated_source", isolatedSourceIds], ["project_source", sourceProjectIds], + ["shared_project_source", sharedProjectIds], ["isolated_project_source", isolatedProjectIds]]); await measureRunPhase("workspace.readers", async () => { for (const reader of readers) { - if (reader.rootKey === rootKey && !entries.some(entry => entry.kind === "reader" && entry.scopeId === reader.scopeId)) { + if (reader.rootKey === rootKey && !entryKeys.has(`reader:${reader.scopeId}`)) { await measureRunPhase("workspace.write", () => tx.insert(executionWorkspaceRepositoryAccess).values(reader)); } } @@ -737,6 +752,7 @@ export function executionWorkspaceRepositoryService(db: Db) { const existing = await measureRunPhase("workspace.sources.inventory", async () => tx.select().from(executionWorkspaceRepositoryAccess).where(and( eq(executionWorkspaceRepositoryAccess.companyId, input.companyId), eq(executionWorkspaceRepositoryAccess.rootKey, rootKey)))); if (existing.some(entry => entry.canonicalCwd !== canonicalCwd)) throw forbidden("Repository root identity conflicts with its retained access record"); + const existingKeys = repositoryAccessKeys(existing); let failure = sourceFailure; for (const cwd of roots) { try { @@ -751,11 +767,8 @@ export function executionWorkspaceRepositoryService(db: Db) { try { await measureRunPhase("workspace.readers.projects", () => assertRootProjectAccess(tx, actor, input.companyId, new Set([id]))); } catch (error) { failure ??= error; } } - for (const [kind, ids] of [["isolated_source", workspaceIds], ["isolated_project_source", projectIds]] as const) { - for (const scopeId of ids) if (!existing.some(entry => entry.kind === kind && entry.scopeId === scopeId)) { - await measureRunPhase("workspace.write", () => tx.insert(executionWorkspaceRepositoryAccess).values({ companyId: input.companyId, rootKey, canonicalCwd, kind, scopeId })); - } - } + await retainRepositorySources(tx, { companyId: input.companyId, canonicalCwd, rootKey }, existingKeys, + [["isolated_source", workspaceIds], ["isolated_project_source", projectIds]]); try { await authorizeRootAccess(tx, { companyId: input.companyId, cwd: input.cwd!, sourceWorkspaceId: workspace.id, actor, issueId: input.issueId }); diff --git a/server/src/services/execution-workspace-source-roots.ts b/server/src/services/execution-workspace-source-roots.ts index 36830eac7cf..33511fc3fae 100644 --- a/server/src/services/execution-workspace-source-roots.ts +++ b/server/src/services/execution-workspace-source-roots.ts @@ -21,6 +21,27 @@ export async function canonicalRepositoryRoot(cwd: string): Promise { } } +export async function resolveRepositoryRoots(cwds: Iterable, requireExisting = false): Promise> { + const paths = [...new Set(cwds)]; + const roots = new Map(); + let next = 0; + const settled = await Promise.allSettled(Array.from({ length: Math.min(16, paths.length) }, async () => { + while (next < paths.length) { + const cwd = paths[next++]; + const root = await (requireExisting ? fs.realpath(cwd) : canonicalRepositoryRoot(cwd)) + .catch((error: NodeJS.ErrnoException) => { + if (["ENOTDIR", "EACCES", "EPERM", "ELOOP", "ENAMETOOLONG"].includes(error.code ?? "") + || (requireExisting && error.code === "ENOENT")) return null; + throw error; + }); + roots.set(cwd, root); + } + })); + const rejected = settled.find(result => result.status === "rejected"); + if (rejected?.status === "rejected") throw rejected.reason; + return roots; +} + export function repositoryRootsOverlap(left: string, right: string): boolean { const contains = (parent: string, child: string) => { const relative = path.relative(parent, child); From 7ec6fd09725fdcd771425e31ce900f8db9aae07d Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Thu, 15 Oct 2026 09:23:47 -0500 Subject: [PATCH 05/19] perf(ci): remove repeated test setup and fixture work (#16429) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - CI must check changes without making contributors wait for repeated setup. > - A complete CI profile showed 158 serialized Vitest launches and repeated dependency installs. > - A large fixture also transferred 40,000 files to prove two different size boundaries. > - This PR removes repeated controller, install, build, transfer, and wait work. > - It keeps isolated test processes and all useful regression boundaries. > - Paired command benchmarks check speed and test inventories before we claim a benefit. ## Linked Issues or Issue Description Refs #16423 for the complete CI profiling data path. No duplicate optimization PR was found. **Current behavior** Each serialized file starts a new Vitest controller. Each CI lane restores the pnpm store and installs dependencies. Large Git scanning and staging share one 40,000-file fixture. A readiness regression waits over a minute for the default database idle deadline. **Proposed behavior** Use one controller per serialized shard, with a fresh fork for each file. Restore exact installed dependencies and certified shared/SDK build outputs when inputs match. Keep the normal install and build paths on a miss. Separate the 32 MiB Git scan proof from staging above command argument limits. Cross a real database idle deadline with a one-second fixture setting. **Reason and benefit** The paired Linux CI benchmark reduced the complete 566-case serialized shard from 303.43s to 246.96s (18.6%). The Git/database fixture command fell from 156.54s to 33.78s (78.4%). Exact dependency/build restoration reduced setup from 56s to 19s (66.1%). These are measured command/setup times. They do not establish a controlled reduction in the full workflow's queue-to-completion time. **Breaking changes** None to application behavior. Tests still run in full. Test results are not cached. Cache misses preserve the existing lockfile resolution fallback. ## What Changed - Batch serialized files into one isolated, single-worker Vitest invocation per shard. Give each worker its own home, config and instance ID. - Add a real two-order isolation check for processes, environment, globals, timers and filesystem state. - Add exact dependency keys for all manifests, lockfile, workspace settings, patches, installer, runtime, platform, libc, pnpm, store and checkout path. Cache only installed package directories, not source fixtures. - Add a separate content key for the shared/SDK dependency graph. Retain the existing source/output fingerprint verifier and rebuild path. Only an exact canonical master push can populate normal caches. Stale lockfile fallback runs cannot publish entries. - Keep four real Git filename lanes above 32 MiB. Use a separate 4,000-file staging/deletion case above 2 MiB. Check every staged file, unusual names, cleanup and bounded commands. - Configure a real one-second PostgreSQL idle timeout in the readiness fixture. Assert that the backend connection changes and that another connection can write the parent row. - Add an opt-in, unprivileged Linux benchmark under `profile-ci`. It compares the tested merge tree with its actual first parent on one runner. It also removes installed outputs and qualifies a real cache restore using temporary PR-scoped entries. ## Verification - Local Node 24.19.0: unchanged serialized shard 255.69s; candidate 231.44s; all 566 cases pass in both. - Local fixture command: unchanged 189.13s; candidate 43.39s. Candidate scan 18.58s, staging 13.41s, database case 2.80s. All three candidate cases pass. - Negative control: moving readiness back inside the real parent-lock transaction fails the shortened regression at the parent-write assertion. The production source was restored and is absent from the diff. - Worker isolation check passes in both file orders. - 565 CI policy, partition and workflow regression checks pass with local HTTP/IPC access enabled. The first sandboxed attempt was denied listen access; it was not a code failure. - `actionlint` passes for the three changed workflows. - [All 48 main CI jobs passed](https://github.com/paperclipai/paperclip/actions/runs/38745643045) on head `5bec2edb2b194e74352ca697f3fd1d0ac03ce967`, including full typecheck, tests, build, browser E2E and release checks. All 158 serialized files and 3,250 cases passed through 9 controller invocations. Full typecheck/test/build were verified in CI; they were not all repeated locally. - [Paired Linux benchmark passed](https://github.com/paperclipai/paperclip/actions/runs/38745643127) on Node 24.21.0. Its baseline is `57fa6b358de62922cfa9a67766b2a227c434fcdf`; its candidate merge tree is `f68f1b32966090f11647502e31852130c20237cf`. The artifact contains results, exact case inventories, setup times and raw logs. Serialized case identities match exactly. All three fixture contracts execute in both variants. | Measured work | Baseline | Candidate | Reduction | |---|---:|---:|---:| | Complete serialized shard, 566 cases | 303.43s | 246.96s | 18.6% | | Git/database fixture command | 156.54s | 33.78s | 78.4% | | Dependency/build setup, cold vs exact hit | 56s | 19s | 66.1% | - [Honeycomb profile export succeeded on its second attempt](https://github.com/paperclipai/paperclip/actions/runs/38747237615/attempts/2). The first attempt failed on GitHub HTTP 503 while downloading an artifact, before export. Only the export was retried; tests were not repeated. The temporary PR-scoped qualification caches were removed after validation, reclaiming 3.34 GB. Timing artifacts remain available. - Greptile is 5/5 on the current head, with no unresolved review threads. Its P1 checkout-prefix finding was fixed in `5bec2edb2`; the corrected reporter paths and paired comparison passed. Earlier runs were superseded while this was fixed. A copy-batching experiment did not speed up the Git fixture and was replaced by the two-boundary fixture split. ## Risks - A persistent controller retains transform state longer. The local shard's maximum RSS rose from about 1.52 GB to 1.99 GB. Full CI must confirm all shards fit the runner budget. - Materialized caches are larger than metadata caches. The Linux benchmark measured real archive transfer and validation after removing installed outputs. Its hit took 19s versus 56s cold. Normal PR lanes cannot publish cache entries. Exact misses run installation. Canonical cache warming requires a frozen-lockfile install; the existing stale-lockfile fallback remains uncached. - The PR uses the trusted reusable workflow from master. The paired benchmark directly exercises the candidate dependency action; normal workflow adoption still requires this PR to merge. This PR does not change the trust references. The completed main workflow took 11m02 including queue time. No controlled full-workflow latency improvement is claimed before this workflow adopts the cache change. - Only deterministic dependency/build outputs are reused. Broad dependency-based test omission and caching successful test results are deliberately excluded from this change. - The real Runner lease minimum and cold-admission waits remain. They protect protocol boundaries and were not shortened by weakening production limits. ## Model Used OpenAI GPT-6 through Codex, with reasoning, shell execution and code tools. The runtime does not expose a more specific deployment ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- .../actions/setup-ci-dependencies/action.yml | 102 +++++++++++++ .../tests/ci-dependency-cache.test.mjs | 88 +++++++++++ .github/workflows/ci-work-benchmark.yml | 139 ++++++++++++++++++ .github/workflows/pr-trusted.yml | 49 ++++++ .github/workflows/release-verify.yml | 9 +- doc/ci-profiling.md | 41 ++++++ package.json | 2 +- .../vitest-worker-isolation.test.mjs | 66 +++++++++ scripts/benchmark-ci-work.mjs | 76 ++++++++++ scripts/ci-dependency-cache.mjs | 83 +++++++++++ scripts/run-vitest-stable.mjs | 7 +- server/src/__tests__/setup-supertest.ts | 2 + .../src/__tests__/workspace-runtime.test.ts | 15 +- .../workspace-git-snapshot-streaming.test.ts | 50 ++++--- 14 files changed, 698 insertions(+), 31 deletions(-) create mode 100644 .github/actions/setup-ci-dependencies/action.yml create mode 100644 .github/scripts/tests/ci-dependency-cache.test.mjs create mode 100644 .github/workflows/ci-work-benchmark.yml create mode 100644 scripts/__tests__/vitest-worker-isolation.test.mjs create mode 100644 scripts/benchmark-ci-work.mjs create mode 100644 scripts/ci-dependency-cache.mjs diff --git a/.github/actions/setup-ci-dependencies/action.yml b/.github/actions/setup-ci-dependencies/action.yml new file mode 100644 index 00000000000..3367001b4b5 --- /dev/null +++ b/.github/actions/setup-ci-dependencies/action.yml @@ -0,0 +1,102 @@ +name: Restore materialized CI dependencies +description: Reuse exact installed dependencies and certified shared build outputs; install on a miss. +inputs: + working-directory: + description: Checkout directory relative to the workspace. + default: '.' + save-cache: + description: Allow a canonical master push to populate the caches. + default: 'false' + restore-cache: + description: Disable materialized restores for a controlled cold-path benchmark. + default: 'true' +outputs: + dependencies-restored: + description: Whether the exact materialized dependency cache was restored. + value: ${{ steps.installed.outputs.cache-hit }} + builds-restored: + description: Whether the exact certified build cache was restored. + value: ${{ steps.builds.outputs.cache-hit }} +runs: + using: composite + steps: + - name: Compute dependency inputs + id: inputs + shell: bash + working-directory: ${{ inputs.working-directory }} + run: node scripts/ci-dependency-cache.mjs + + - name: Restore installed dependencies (read only) + if: inputs.restore-cache == 'true' && steps.inputs.outputs.cacheable == 'true' + id: installed + uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 + with: + key: ${{ steps.inputs.outputs.dependency-key }} + path: ${{ steps.inputs.outputs.dependency-paths }} + + - name: Locate pnpm store + if: steps.installed.outputs.cache-hit != 'true' + id: pnpm_store + shell: bash + working-directory: ${{ inputs.working-directory }} + run: | + echo "path=$(pnpm store path --silent)" >> "$GITHUB_OUTPUT" + echo "arch=$(node -p 'process.arch')" >> "$GITHUB_OUTPUT" + + - name: Restore pnpm store (read only) + if: steps.installed.outputs.cache-hit != 'true' + uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 + with: + path: ${{ steps.pnpm_store.outputs.path }} + key: node-cache-${{ runner.os }}-${{ steps.pnpm_store.outputs.arch }}-pnpm-${{ hashFiles(format('{0}/pnpm-lock.yaml', inputs.working-directory)) }} + restore-keys: node-cache-${{ runner.os }}-${{ steps.pnpm_store.outputs.arch }}-pnpm- + + - name: Install dependencies + if: steps.installed.outputs.cache-hit != 'true' + id: install + shell: bash + working-directory: ${{ inputs.working-directory }} + run: | + if pnpm install --frozen-lockfile; then + echo 'frozen=true' >> "$GITHUB_OUTPUT" + else + echo '::notice title=Lockfile::checked-in lockfile is stale for this merge tree; resolving it inline' + pnpm install --resolution-only --ignore-scripts --no-frozen-lockfile + pnpm install --frozen-lockfile + fi + + - name: Restore certified shared build outputs (read only) + if: inputs.restore-cache == 'true' && steps.inputs.outputs.cacheable == 'true' + id: builds + uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 + with: + key: ${{ steps.inputs.outputs.build-key }} + path: ${{ steps.inputs.outputs.build-paths }} + + - name: Validate workspace links and shared build outputs + shell: bash + working-directory: ${{ inputs.working-directory }} + run: | + pnpm run preflight:workspace-links + node scripts/ensure-plugin-build-deps.mjs + + - name: Save installed dependencies + if: >- + inputs.save-cache == 'true' && steps.inputs.outputs.cacheable == 'true' && github.repository == 'paperclipai/paperclip' && + github.event_name == 'push' && github.ref == 'refs/heads/master' && + steps.installed.outputs.cache-hit != 'true' && steps.install.outputs.frozen == 'true' + uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 + with: + key: ${{ steps.inputs.outputs.dependency-key }} + path: ${{ steps.inputs.outputs.dependency-paths }} + + - name: Save certified shared build outputs + if: >- + inputs.save-cache == 'true' && steps.inputs.outputs.cacheable == 'true' && github.repository == 'paperclipai/paperclip' && + github.event_name == 'push' && github.ref == 'refs/heads/master' && + steps.builds.outputs.cache-hit != 'true' && + (steps.installed.outputs.cache-hit == 'true' || steps.install.outputs.frozen == 'true') + uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 + with: + key: ${{ steps.inputs.outputs.build-key }} + path: ${{ steps.inputs.outputs.build-paths }} diff --git a/.github/scripts/tests/ci-dependency-cache.test.mjs b/.github/scripts/tests/ci-dependency-cache.test.mjs new file mode 100644 index 00000000000..17c1f274061 --- /dev/null +++ b/.github/scripts/tests/ci-dependency-cache.test.mjs @@ -0,0 +1,88 @@ +import assert from "node:assert/strict"; +import test from "node:test"; +import { readFileSync } from "node:fs"; +import { cachePlan } from "../../../scripts/ci-dependency-cache.mjs"; + +const entries = { + "package.json": '{"scripts":{"postinstall":"node scripts/link-plugin-dev-sdk.mjs"}}', + "server/package.json": '{"dependencies":{"external":"1.0.0"}}', + "packages/shared/package.json": '{}', + "packages/plugins/sdk/package.json": '{}', + "pnpm-lock.yaml": 'lockfile', + "pnpm-workspace.yaml": 'workspace', + ".npmrc": 'auto-install-peers=false', + "patches/external.patch": 'patch', + "scripts/link-plugin-dev-sdk.mjs": 'postinstall', + "scripts/ci-dependency-cache.mjs": 'cache schema', + "scripts/ensure-plugin-build-deps.mjs": 'build verifier', + "tsconfig.base.json": '{}', + "packages/shared/src/index.ts": 'shared API', + "packages/plugins/sdk/src/index.ts": 'SDK API', + "server/src/server.ts": 'server', +}; +const scope = { node: 'v24.19.0', root: '/work/paperclip', arch: 'x64', libc: '2.39', pnpm: '9.15.4', store: '/store' }; +const plan = (files = entries, runtime = scope) => cachePlan({ + files: Object.keys(files), read: (file) => files[file] === null ? null : Buffer.from(files[file]), scope: runtime, + packageDirs: ['.', 'packages/plugins/sdk', 'packages/shared', 'server'], +}); + +test('dependency identity includes manifests, resolution, patches, installer and runtime boundaries', () => { + const original = plan(); + for (const file of Object.keys(entries).filter((file) => + /package\.json$/.test(file) || /^(pnpm-|\.npmrc|patches\/|scripts\/(link-plugin-dev-sdk|ci-dependency-cache))/.test(file))) { + for (const value of [file.endsWith('package.json') ? '{"changed":true}' : 'changed', null]) { + const changed = plan({ ...entries, [file]: value }); + assert.notEqual(changed.dependencyKey, original.dependencyKey, file); + assert.notEqual(changed.buildKey, original.buildKey, file); + } + } + assert.notEqual(plan({ ...entries, 'packages/new/package.json': '{}' }).dependencyKey, original.dependencyKey); + for (const key of Object.keys(scope)) { + assert.notEqual(plan(entries, { ...scope, [key]: 'different' }).dependencyKey, original.dependencyKey, key); + } +}); + +test('shared and SDK source changes invalidate dependent builds without reinstalling packages', () => { + const original = plan(); + for (const file of ['packages/shared/src/index.ts', 'packages/plugins/sdk/src/index.ts', + 'tsconfig.base.json']) { + const changed = plan({ ...entries, [file]: 'changed' }); + assert.equal(changed.dependencyKey, original.dependencyKey, file); + assert.notEqual(changed.buildKey, original.buildKey, file); + } + assert.deepEqual(plan({ ...entries, 'server/src/server.ts': 'different' }), original); + assert.deepEqual(plan(Object.fromEntries(Object.entries(entries).reverse())), original); + assert.deepEqual(original.dependencyPaths, ['node_modules', 'packages/plugins/sdk/node_modules', + 'packages/shared/node_modules', 'server/node_modules']); + assert.deepEqual(plan({ ...entries, 'tests/fixtures/app/package.json': '{}' }).dependencyPaths, original.dependencyPaths); + for (const directory of ['/outside', '../outside', 'server/../../outside']) { + assert.throws(() => cachePlan({ files: [], read: () => null, scope, packageDirs: [directory] })); + } +}); + +test('unmodelled workspace lifecycle scripts disable materialized reuse', () => { + assert.equal(plan().cacheable, true); + for (const hook of ['preinstall', 'install', 'postinstall', 'prepare']) { + assert.equal(plan({ ...entries, 'server/package.json': JSON.stringify({ scripts: { [hook]: 'node custom.mjs' } }) }).cacheable, false); + } + assert.equal(plan({ ...entries, 'package.json': '{"scripts":{"postinstall":"node other.mjs"}}' }).cacheable, false); + for (const file of ['scripts/ensure-plugin-build-deps.mjs', 'scripts/new-install-helper.mjs', '.pnpmfile.cjs']) { + assert.notEqual(plan({ ...entries, [file]: 'changed' }).dependencyKey, plan().dependencyKey); + } +}); + +test('cache restores are exact and only a frozen canonical master push can publish entries', () => { + const action = readFileSync(new URL('../../actions/setup-ci-dependencies/action.yml', import.meta.url), 'utf8'); + const installed = action.slice(action.indexOf(' - name: Restore installed'), action.indexOf(' - name: Locate pnpm')); + const builds = action.slice(action.indexOf(' - name: Restore certified'), action.indexOf(' - name: Validate workspace')); + for (const restore of [installed, builds]) assert.doesNotMatch(restore, /restore-keys:/); + for (const step of action.split(' - name: Save ').slice(1)) { + assert.match(step, /inputs\.save-cache == 'true'/); + assert.match(step, /github\.repository == 'paperclipai\/paperclip'/); + assert.match(step, /github\.event_name == 'push'/); + assert.match(step, /github\.ref == 'refs\/heads\/master'/); + assert.match(step, /steps\.install\.outputs\.frozen == 'true'/); + } + const workflow = readFileSync(new URL('../../workflows/release-verify.yml', import.meta.url), 'utf8'); + assert.match(workflow, /save-cache: .*inputs\.ref == github\.sha/); +}); diff --git a/.github/workflows/ci-work-benchmark.yml b/.github/workflows/ci-work-benchmark.yml new file mode 100644 index 00000000000..2ff13f8c7cc --- /dev/null +++ b/.github/workflows/ci-work-benchmark.yml @@ -0,0 +1,139 @@ +name: CI work benchmark + +on: + pull_request: + types: [opened, synchronize, reopened, labeled] + paths: + - .github/actions/setup-ci-dependencies/** + - .github/workflows/ci-work-benchmark.yml + - scripts/ci-dependency-cache.mjs + - scripts/benchmark-ci-work.mjs + - scripts/run-vitest-stable.mjs + - server/src/__tests__/setup-supertest.ts + - server/src/__tests__/workspace-runtime.test.ts + - server/src/services/workspace-git-snapshot-streaming.test.ts + +permissions: + contents: read + +concurrency: + group: ci-work-benchmark-${{ github.event.pull_request.number }} + cancel-in-progress: true + +jobs: + paired: + if: >- + contains(github.event.pull_request.labels.*.name, 'profile-ci') && + (github.event.action != 'labeled' || github.event.label.name == 'profile-ci') + runs-on: ubuntu-latest + timeout-minutes: 25 + defaults: + run: + working-directory: candidate + steps: + - name: Checkout candidate + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 + with: + ref: ${{ github.sha }} + path: candidate + fetch-depth: 2 + persist-credentials: false + + - name: Setup Node.js + uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 + with: + node-version: 24 + package-manager-cache: false + + - name: Setup pnpm + uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 + with: + version: 9.15.4 + run_install: false + + - name: Start cold setup measurement + run: echo "CI_COLD_STARTED=$(date +%s)" >> "$GITHUB_ENV" + + - name: Qualify candidate dependency setup + uses: ./candidate/.github/actions/setup-ci-dependencies + with: + working-directory: candidate + restore-cache: 'false' + + - name: Record cold setup and compute qualification keys + id: qualification + run: | + echo "CI_COLD_SECONDS=$(($(date +%s) - CI_COLD_STARTED))" >> "$GITHUB_ENV" + node scripts/ci-dependency-cache.mjs + + - name: Save PR-scoped qualification dependencies + uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 + with: + key: ${{ steps.qualification.outputs.dependency-key }} + path: ${{ steps.qualification.outputs.dependency-paths }} + + - name: Save PR-scoped qualification builds + uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 + with: + key: ${{ steps.qualification.outputs.build-key }} + path: ${{ steps.qualification.outputs.build-paths }} + + - name: Remove materialized outputs before restore + run: | + node --input-type=module <<'NODE' + import { execFileSync } from 'node:child_process'; + import { rmSync } from 'node:fs'; + const plan = JSON.parse(execFileSync(process.execPath, ['scripts/ci-dependency-cache.mjs'], { + env: { ...process.env, GITHUB_OUTPUT: '' }, encoding: 'utf8', + })); + for (const directory of [...plan.dependencyPaths, 'packages/shared/dist', 'packages/plugins/sdk/dist']) { + rmSync(directory, { recursive: true, force: true }); + } + NODE + echo "CI_WARM_STARTED=$(date +%s)" >> "$GITHUB_ENV" + + - name: Qualify a real warm restore + id: warm + uses: ./candidate/.github/actions/setup-ci-dependencies + with: + working-directory: candidate + + - name: Verify exact cache hits and record setup comparison + env: + DEPS_RESTORED: ${{ steps.warm.outputs.dependencies-restored }} + BUILDS_RESTORED: ${{ steps.warm.outputs.builds-restored }} + OUTPUT: ${{ runner.temp }}/ci-work-benchmark + run: | + test "$DEPS_RESTORED" = true + test "$BUILDS_RESTORED" = true + mkdir -p "$OUTPUT" + warm_seconds=$(($(date +%s) - CI_WARM_STARTED)) + printf '{"coldSeconds":%s,"warmSeconds":%s}\n' "$CI_COLD_SECONDS" "$warm_seconds" > "$OUTPUT/dependency-setup.json" + echo "Dependency setup: ${CI_COLD_SECONDS}s cold, ${warm_seconds}s with exact materialized restores." >> "$GITHUB_STEP_SUMMARY" + + - name: Prepare unchanged baseline + run: | + base_sha=$(git rev-parse HEAD^1) + git worktree add --detach ../baseline "$base_sha" + cd ../baseline + if ! pnpm install --frozen-lockfile; then + pnpm install --resolution-only --ignore-scripts --no-frozen-lockfile + pnpm install --frozen-lockfile + fi + pnpm run preflight:workspace-links + pnpm --filter @paperclipai/plugin-sdk ensure-build-deps + + - name: Compare complete commands and executed case inventories + env: + BENCHMARK_OUTPUT: ${{ runner.temp }}/ci-work-benchmark + run: | + node scripts/benchmark-ci-work.mjs ../baseline "$BENCHMARK_OUTPUT" + cat "$BENCHMARK_OUTPUT/summary.md" >> "$GITHUB_STEP_SUMMARY" + + - name: Retain timings, inventories and failure logs + if: always() + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a + with: + name: ci-work-benchmark + path: ${{ runner.temp }}/ci-work-benchmark + retention-days: 14 diff --git a/.github/workflows/pr-trusted.yml b/.github/workflows/pr-trusted.yml index 3a4db1c85ac..e225da5cebd 100644 --- a/.github/workflows/pr-trusted.yml +++ b/.github/workflows/pr-trusted.yml @@ -186,13 +186,19 @@ jobs: # Share the checked-in lockfile key with master. PR merge refs must not # save full copies of the store or evict the post-merge build caches. + - name: Restore materialized dependencies and shared builds + if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') != '' + uses: ./.github/actions/setup-ci-dependencies + - name: Locate pnpm store + if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') == '' id: pnpm_store run: | echo "path=$(pnpm store path --silent)" >> "$GITHUB_OUTPUT" echo "arch=$(node -p 'process.arch')" >> "$GITHUB_OUTPUT" - name: Restore pnpm store (read only) + if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') == '' uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 with: path: ${{ steps.pnpm_store.outputs.path }} @@ -200,6 +206,7 @@ jobs: restore-keys: node-cache-${{ runner.os }}-${{ steps.pnpm_store.outputs.arch }}-pnpm- - name: Install dependencies + if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') == '' # Manifest-changing and stacked PRs can hold a pnpm-lock.yaml that is # stale for the merge tree. Resolve it inline instead of waiting on a # policy-job artifact: that dependency put the policy job's queue and @@ -484,13 +491,19 @@ jobs: # Share the checked-in lockfile key with master. PR merge refs must not # save full copies of the store or evict the post-merge build caches. + - name: Restore materialized dependencies and shared builds + if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') != '' + uses: ./.github/actions/setup-ci-dependencies + - name: Locate pnpm store + if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') == '' id: pnpm_store run: | echo "path=$(pnpm store path --silent)" >> "$GITHUB_OUTPUT" echo "arch=$(node -p 'process.arch')" >> "$GITHUB_OUTPUT" - name: Restore pnpm store (read only) + if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') == '' uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 with: path: ${{ steps.pnpm_store.outputs.path }} @@ -498,6 +511,7 @@ jobs: restore-keys: node-cache-${{ runner.os }}-${{ steps.pnpm_store.outputs.arch }}-pnpm- - name: Install dependencies + if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') == '' # Manifest-changing and stacked PRs can hold a pnpm-lock.yaml that is # stale for the merge tree. Resolve it inline instead of waiting on a # policy-job artifact: that dependency put the policy job's queue and @@ -684,13 +698,19 @@ jobs: # Share the checked-in lockfile key with master. PR merge refs must not # save full copies of the store or evict the post-merge build caches. + - name: Restore materialized dependencies and shared builds + if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') != '' + uses: ./.github/actions/setup-ci-dependencies + - name: Locate pnpm store + if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') == '' id: pnpm_store run: | echo "path=$(pnpm store path --silent)" >> "$GITHUB_OUTPUT" echo "arch=$(node -p 'process.arch')" >> "$GITHUB_OUTPUT" - name: Restore pnpm store (read only) + if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') == '' uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 with: path: ${{ steps.pnpm_store.outputs.path }} @@ -698,6 +718,7 @@ jobs: restore-keys: node-cache-${{ runner.os }}-${{ steps.pnpm_store.outputs.arch }}-pnpm- - name: Install dependencies + if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') == '' # Manifest-changing and stacked PRs can hold a pnpm-lock.yaml that is # stale for the merge tree. Resolve it inline instead of waiting on a # policy-job artifact: that dependency put the policy job's queue and @@ -885,13 +906,19 @@ jobs: # Share the checked-in lockfile key with master. PR merge refs must not # save full copies of the store or evict the post-merge build caches. + - name: Restore materialized dependencies and shared builds + if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') != '' + uses: ./.github/actions/setup-ci-dependencies + - name: Locate pnpm store + if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') == '' id: pnpm_store run: | echo "path=$(pnpm store path --silent)" >> "$GITHUB_OUTPUT" echo "arch=$(node -p 'process.arch')" >> "$GITHUB_OUTPUT" - name: Restore pnpm store (read only) + if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') == '' uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 with: path: ${{ steps.pnpm_store.outputs.path }} @@ -899,6 +926,7 @@ jobs: restore-keys: node-cache-${{ runner.os }}-${{ steps.pnpm_store.outputs.arch }}-pnpm- - name: Install dependencies + if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') == '' # Manifest-changing and stacked PRs can hold a pnpm-lock.yaml that is # stale for the merge tree. Resolve it inline instead of waiting on a # policy-job artifact: that dependency put the policy job's queue and @@ -1160,13 +1188,19 @@ jobs: # Share the checked-in lockfile key with master. PR merge refs must not # save full copies of the store or evict the post-merge build caches. + - name: Restore materialized dependencies and shared builds + if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') != '' + uses: ./.github/actions/setup-ci-dependencies + - name: Locate pnpm store + if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') == '' id: pnpm_store run: | echo "path=$(pnpm store path --silent)" >> "$GITHUB_OUTPUT" echo "arch=$(node -p 'process.arch')" >> "$GITHUB_OUTPUT" - name: Restore pnpm store (read only) + if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') == '' uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 with: path: ${{ steps.pnpm_store.outputs.path }} @@ -1174,6 +1208,7 @@ jobs: restore-keys: node-cache-${{ runner.os }}-${{ steps.pnpm_store.outputs.arch }}-pnpm- - name: Install dependencies + if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') == '' # Manifest-changing and stacked PRs can hold a pnpm-lock.yaml that is # stale for the merge tree. Resolve it inline instead of waiting on a # policy-job artifact: that dependency put the policy job's queue and @@ -1239,13 +1274,19 @@ jobs: # Share the checked-in lockfile key with master. PR merge refs must not # save full copies of the store or evict the post-merge build caches. + - name: Restore materialized dependencies and shared builds + if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') != '' + uses: ./.github/actions/setup-ci-dependencies + - name: Locate pnpm store + if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') == '' id: pnpm_store run: | echo "path=$(pnpm store path --silent)" >> "$GITHUB_OUTPUT" echo "arch=$(node -p 'process.arch')" >> "$GITHUB_OUTPUT" - name: Restore pnpm store (read only) + if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') == '' uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 with: path: ${{ steps.pnpm_store.outputs.path }} @@ -1253,6 +1294,7 @@ jobs: restore-keys: node-cache-${{ runner.os }}-${{ steps.pnpm_store.outputs.arch }}-pnpm- - name: Install dependencies + if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') == '' # Manifest-changing and stacked PRs can hold a pnpm-lock.yaml that is # stale for the merge tree. Resolve it inline instead of waiting on a # policy-job artifact: that dependency put the policy job's queue and @@ -1486,13 +1528,19 @@ jobs: # Share the checked-in lockfile key with master. PR merge refs must not # save full copies of the store or evict the post-merge build caches. + - name: Restore materialized dependencies and shared builds + if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') != '' + uses: ./.github/actions/setup-ci-dependencies + - name: Locate pnpm store + if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') == '' id: pnpm_store run: | echo "path=$(pnpm store path --silent)" >> "$GITHUB_OUTPUT" echo "arch=$(node -p 'process.arch')" >> "$GITHUB_OUTPUT" - name: Restore pnpm store (read only) + if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') == '' uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 with: path: ${{ steps.pnpm_store.outputs.path }} @@ -1500,6 +1548,7 @@ jobs: restore-keys: node-cache-${{ runner.os }}-${{ steps.pnpm_store.outputs.arch }}-pnpm- - name: Install dependencies + if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') == '' # Manifest-changing and stacked PRs can hold a pnpm-lock.yaml that is # stale for the merge tree. Resolve it inline instead of waiting on a # policy-job artifact: that dependency put the policy job's queue and diff --git a/.github/workflows/release-verify.yml b/.github/workflows/release-verify.yml index a4632dfb442..cf333aaa618 100644 --- a/.github/workflows/release-verify.yml +++ b/.github/workflows/release-verify.yml @@ -52,7 +52,7 @@ jobs: uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7 with: node-version: 24 - cache: pnpm + package-manager-cache: false # rust-cache hashes its absolute cache paths into the entry's version, # and GitHub only serves an entry whose key and version both match. The @@ -102,7 +102,14 @@ jobs: - name: Validate release package manifest run: node ./scripts/release-package-map.mjs check + - name: Restore materialized dependencies and shared builds + if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') != '' + uses: ./.github/actions/setup-ci-dependencies + with: + save-cache: ${{ github.repository == 'paperclipai/paperclip' && github.event_name == 'push' && github.ref == 'refs/heads/master' && inputs.ref == github.sha }} + - name: Install dependencies + if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') == '' run: pnpm install --no-frozen-lockfile - name: Typecheck diff --git a/doc/ci-profiling.md b/doc/ci-profiling.md index 9d836537bd8..fe8472a7577 100644 --- a/doc/ci-profiling.md +++ b/doc/ci-profiling.md @@ -154,3 +154,44 @@ References: [Honeycomb trace fields](https://docs.honeycomb.io/get-started/best- [Events API](https://docs.honeycomb.io/api/events/create-events), [large trace display limit](https://docs.honeycomb.io/troubleshoot/common-issues/data-in-honeycomb/#your-very-large-trace-displays-only-32000-spans), [GitHub workflow_run](https://docs.github.com/en/actions/reference/workflows-and-actions/events-that-trigger-workflows#workflow_run). + +## Qualify reductions in CI work + +For changes to the stable test launcher, dependency setup, or the large Git and +readiness fixtures, `profile-ci` also runs **CI work benchmark**. It uses one +GitHub-hosted runner without secrets or AWS access. It compares the unchanged +PR base and candidate sequentially. It records complete command times, test +inventories, revisions, and failure logs in the `ci-work-benchmark` artifact. +The serialized shard must retain the same cases. The fixture comparison retains +the database contract and splits large Git scanning from bounded staging. +These command timings exclude queue time and setup; use the full Honeycomb +waterfall to assess the complete CI critical path. + +The benchmark also exercises cold dependency setup and a real materialized +cache restore after removing the installed directories. Its two qualification +entries are scoped to the PR merge ref and a separate checkout path. They cannot +populate master's caches. Normal PR lanes restore only; the exact canonical +master push in release verification is the normal cache writer. + +Materialized dependency keys include every tracked package manifest, lockfile, +workspace configuration, patches, repository scripts, Node version/ABI, platform, +architecture, libc, pnpm version, store location, and physical checkout path. +There are no prefix restores. An input change takes the normal install path. +A stale lockfile uses the existing resolution fallback but cannot publish an +entry. An additional workspace install/prepare hook disables materialized reuse +until its inputs are modeled. Shared/SDK build outputs have a separate key that also includes both +packages' source/configuration and the build verifier. The existing verifier +checks source and output fingerprints and rebuilds incomplete or changed output. +Tests and their results are never cached or omitted by this mechanism. + +Serialized shards share one Vitest controller with `forks`, `isolate`, one +worker, and file parallelism disabled. Each file gets a separate process and +test home/configuration. The isolation regression runs files in both orders and +checks process IDs, environment, globals, fake timers, and filesystem state. + +The readiness regression uses a real one-second PostgreSQL idle timeout and +verifies that the backend connection changes while the parent write can commit. +Its configured readiness window stays long. The Git regression keeps all four +filename lanes above 32 MiB, while the staging case transfers and deletes 4,000 +files whose filename list exceeds 2 MiB. It still checks every staged file, +unusual names, snapshot ownership, cleanup, and bounded command lengths. diff --git a/package.json b/package.json index 32ed9601455..93ef1dee218 100644 --- a/package.json +++ b/package.json @@ -62,7 +62,7 @@ "smoke:posthog-live": "node scripts/smoke/posthog-live.mjs", "smoke:pipelines-tutorial": "./scripts/smoke/pipelines-tutorial-smoke.sh", "smoke:terminal-bench-loop-skill": "node scripts/smoke/terminal-bench-loop-skill-smoke.mjs", - "test:release-registry": "node scripts/ci-profile/node-test.mjs scripts/ci-profile/reporters.test.mjs scripts/__tests__/canary-startup-playwright.test.mjs scripts/__tests__/cursor-public-install-sandbox.test.mjs scripts/__tests__/grok-public-install-sandbox.test.mjs scripts/__tests__/public-install-closure.test.mjs scripts/__tests__/prefetch-child.test.mjs scripts/__tests__/stage-bundled-packages.test.mjs scripts/__tests__/pack-staged-packages.test.mjs scripts/__tests__/retain-runner-qualification-packages.test.mjs scripts/__tests__/runner-release-crate-cache.test.mjs scripts/verify-release-registry-state.test.mjs scripts/release-package-map.test.mjs scripts/check-release-package-bootstrap.test.mjs scripts/check-no-git-push.test.mjs scripts/release-lib.test.mjs scripts/release-registry-versions.test.mjs scripts/link-plugin-dev-sdk.test.js scripts/acpx-patch-packaging.test.mjs scripts/service-onboard-smoke.test.mjs scripts/docker-onboard-smoke.test.mjs scripts/preview-artifacts.test.mjs scripts/cloud-migrator-artifacts.test.mjs", + "test:release-registry": "node scripts/ci-profile/node-test.mjs scripts/ci-profile/reporters.test.mjs scripts/__tests__/canary-startup-playwright.test.mjs scripts/__tests__/cursor-public-install-sandbox.test.mjs scripts/__tests__/grok-public-install-sandbox.test.mjs scripts/__tests__/public-install-closure.test.mjs scripts/__tests__/prefetch-child.test.mjs scripts/__tests__/stage-bundled-packages.test.mjs scripts/__tests__/pack-staged-packages.test.mjs scripts/__tests__/retain-runner-qualification-packages.test.mjs scripts/__tests__/runner-release-crate-cache.test.mjs scripts/__tests__/vitest-worker-isolation.test.mjs scripts/verify-release-registry-state.test.mjs scripts/release-package-map.test.mjs scripts/check-release-package-bootstrap.test.mjs scripts/check-no-git-push.test.mjs scripts/release-lib.test.mjs scripts/release-registry-versions.test.mjs scripts/link-plugin-dev-sdk.test.js scripts/acpx-patch-packaging.test.mjs scripts/service-onboard-smoke.test.mjs scripts/docker-onboard-smoke.test.mjs scripts/preview-artifacts.test.mjs scripts/cloud-migrator-artifacts.test.mjs", "storybook-visual:baseline": "node scripts/storybook-visual-baseline.mjs", "test:storybook-visual": "node scripts/storybook-visual-baseline.mjs download && node scripts/storybook-visual-baseline.mjs verify && pnpm build-storybook && npx playwright test --config tests/storybook-visual/playwright.config.ts", "test:storybook-visual:update": "node scripts/storybook-visual-baseline.mjs download && pnpm build-storybook && npx playwright test --config tests/storybook-visual/playwright.config.ts --update-snapshots && node scripts/storybook-visual-baseline.mjs pack", diff --git a/scripts/__tests__/vitest-worker-isolation.test.mjs b/scripts/__tests__/vitest-worker-isolation.test.mjs new file mode 100644 index 00000000000..bd70f733df4 --- /dev/null +++ b/scripts/__tests__/vitest-worker-isolation.test.mjs @@ -0,0 +1,66 @@ +import assert from "node:assert/strict"; +import { spawnSync } from "node:child_process"; +import { mkdtempSync, readFileSync, realpathSync, rmSync, writeFileSync } from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { fileURLToPath } from "node:url"; +import test from "node:test"; + +const root = fileURLToPath(new URL('../../', import.meta.url)); +const vitest = path.join(root, 'node_modules/vitest/vitest.mjs'); +const api = fileURLToPath(new URL('../../node_modules/vitest/dist/index.js', import.meta.url)); +const setup = path.join(root, 'server/src/__tests__/setup-supertest.ts'); + +test('one controller retains fresh fork, globals, environment, timers and home per serialized file', () => { + const directory = realpathSync(mkdtempSync(path.join(os.tmpdir(), 'pc-fork-proof-'))); + try { + const record = path.join(directory, 'workers.ndjson'); + const config = path.join(directory, 'vitest.config.mjs'); + const files = ['first.test.ts', 'second.test.ts']; + for (const file of files) writeFileSync(path.join(directory, file), ` + import { expect, it, vi } from ${JSON.stringify(api)}; + import { appendFileSync, existsSync, writeFileSync } from 'node:fs'; + it('starts clean and can contaminate only its own worker', () => { + expect(globalThis.serializedLeak).toBeUndefined(); + expect(process.env.PAPERCLIP_SERIALIZED_LEAK).toBeUndefined(); + expect(vi.isFakeTimers()).toBe(false); + expect(existsSync(process.env.PAPERCLIP_CONFIG)).toBe(false); + appendFileSync(${JSON.stringify(record)}, JSON.stringify({ + name: ${JSON.stringify(file)}, pid: process.pid, home: process.env.PAPERCLIP_HOME, + config: process.env.PAPERCLIP_CONFIG, instance: process.env.PAPERCLIP_INSTANCE_ID, + }) + '\\n'); + writeFileSync(process.env.PAPERCLIP_CONFIG, '{}'); + globalThis.serializedLeak = true; + process.env.PAPERCLIP_SERIALIZED_LEAK = 'leaked'; + vi.useFakeTimers(); + }); + `); + for (const order of [files, [...files].reverse()]) { + rmSync(record, { force: true }); + writeFileSync(config, `import Sequencer from './sequencer.mjs'; export default ${JSON.stringify({ + root: directory, + test: { + include: order, pool: 'forks', isolate: true, fileParallelism: false, + maxWorkers: 1, setupFiles: [setup], + sequence: { sequencer: '__SEQUENCER__' }, + }, + }).replace('"__SEQUENCER__"', 'Sequencer')};`); + writeFileSync(path.join(directory, 'sequencer.mjs'), `export default class { + shard(tests) { return tests; } + sort(tests) { return tests.sort((a, b) => ${JSON.stringify(order)}.indexOf(a.moduleId.split('/').pop()) - ${JSON.stringify(order)}.indexOf(b.moduleId.split('/').pop())); } + }`); + const result = spawnSync(process.execPath, [vitest, 'run', '--config', config], { + cwd: root, encoding: 'utf8', timeout: 60_000, + env: { ...process.env, PAPERCLIP_HOME: path.join(directory, 'controller-home'), + PAPERCLIP_CONFIG: path.join(directory, 'controller-config.json'), PAPERCLIP_INSTANCE_ID: 'controller' }, + }); + assert.equal(result.status, 0, `${result.error ?? ''}\n${result.stdout}\n${result.stderr}`); + const workers = readFileSync(record, 'utf8').trim().split('\n').map((line) => JSON.parse(line)); + assert.equal(workers.length, 2); + assert.deepEqual(workers.map((worker) => worker.name), order); + for (const key of ['pid', 'home', 'config', 'instance']) assert.equal(new Set(workers.map((worker) => worker[key])).size, 2, key); + } + } finally { + rmSync(directory, { recursive: true, force: true }); + } +}); diff --git a/scripts/benchmark-ci-work.mjs b/scripts/benchmark-ci-work.mjs new file mode 100644 index 00000000000..7fa6dec7661 --- /dev/null +++ b/scripts/benchmark-ci-work.mjs @@ -0,0 +1,76 @@ +import assert from "node:assert/strict"; +import { execFileSync, spawnSync } from "node:child_process"; +import { mkdirSync, readdirSync, readFileSync, writeFileSync } from "node:fs"; +import path from "node:path"; +import { fileURLToPath } from "node:url"; + +const candidate = path.resolve(path.dirname(fileURLToPath(import.meta.url)), ".."); +const baseline = path.resolve(process.argv[2] ?? ""); +assert.ok(process.argv[2] && baseline !== candidate, 'Pass a separately installed unchanged baseline checkout'); +const output = path.resolve(process.argv[3] ?? path.join(candidate, '.ci-benchmark')); +mkdirSync(output, { recursive: true }); +const results = []; +const commands = [ + ['serialized', process.execPath, ['scripts/run-vitest-stable.mjs', '--mode', 'serialized', '--shard-index', '0', '--shard-count', '9']], + ['fixtures', 'pnpm', ['exec', 'vitest', 'run', '--project', '@paperclipai/server', + 'server/src/services/workspace-git-snapshot-streaming.test.ts', 'server/src/__tests__/workspace-runtime.test.ts', + '-t', 'streams.*all four real Git|commits fixed-port reservations']], +]; +for (const [variant, cwd] of [['baseline', baseline], ['candidate', candidate]]) { + for (const [name, command, args] of commands) { + const profile = path.join(output, `${variant}-${name}`); + mkdirSync(profile, { recursive: true }); + const started = performance.now(); + const result = spawnSync(command, args, { + cwd, encoding: 'utf8', timeout: 15 * 60_000, maxBuffer: 64 * 1024 * 1024, + env: { ...process.env, GITHUB_WORKSPACE: cwd, NODE_ENV: 'test', PAPERCLIP_CI_PROFILE: 'true', + PAPERCLIP_CI_PROFILE_DIR: profile, PAPERCLIP_CI_PROFILE_JOB: `benchmark-${variant}-${name}` }, + }); + const wallMs = performance.now() - started; + writeFileSync(path.join(output, `${variant}-${name}.log`), `${result.stdout ?? ''}\n${result.stderr ?? ''}`); + const records = readdirSync(profile).filter((file) => file.endsWith('.ndjson')) + .flatMap((file) => readFileSync(path.join(profile, file), 'utf8').split('\n').filter(Boolean).map((line) => JSON.parse(line))); + results.push({ variant, name, wallMs, status: result.status, + sha: execFileSync('git', ['rev-parse', 'HEAD'], { cwd, encoding: 'utf8' }).trim(), + tests: records.filter((record) => record.kind === 'test' && record.status !== 'skipped') + .map((record) => [record.file, record.name, record.status]).sort(), + files: records.filter((record) => record.kind === 'file').map((record) => record.file).sort(), + }); + writeFileSync(path.join(output, 'results.json'), JSON.stringify({ node: process.version, results }, null, 2)); + assert.equal(result.status, 0, `${variant}/${name} failed: ${result.error ?? ''}; see ${output}`); + console.log(`${variant}/${name}: ${(wallMs / 1000).toFixed(2)}s`); + } +} +const rows = []; +for (const [name] of commands) { + const before = results.find((result) => result.variant === 'baseline' && result.name === name); + const after = results.find((result) => result.variant === 'candidate' && result.name === name); + assert.ok(before.tests.length > 0, `${name}: missing test evidence`); + assert.deepEqual(after.files, before.files, `${name}: file inventory changed`); + if (name === 'serialized') assert.deepEqual(after.tests, before.tests, 'serialized case inventory changed'); + else { + const contracts = (tests) => { + const found = new Set(); + for (const [file, title, status] of tests) { + assert.equal(status, 'passed'); + if (file.endsWith('/workspace-runtime.test.ts') && title.includes('commits fixed-port reservations')) found.add('database'); + if (file.endsWith('/workspace-git-snapshot-streaming.test.ts')) { + if (title.includes('above 32 MiB')) found.add('large Git lanes'); + if (/stag(?:es|ing)/.test(title)) found.add('bounded staging'); + } + } + return [...found].sort(); + }; + for (const result of [before, after]) assert.deepEqual(contracts(result.tests), + ['bounded staging', 'database', 'large Git lanes'], 'fixture contracts must all execute'); + } + rows.push(`| ${name} | ${(before.wallMs / 1000).toFixed(2)}s | ${(after.wallMs / 1000).toFixed(2)}s | ${((1 - after.wallMs / before.wallMs) * 100).toFixed(1)}% | ${after.tests.length} |`); +} +const summary = ['| Work | Baseline | Candidate | Reduction | Executed cases |', + '|---|---:|---:|---:|---:|', ...rows, + '', 'Paired sequential runs on one host with Node ' + process.version + '. Setup, cache transfer, queue time and the full CI critical path are outside these command timings.', +].join('\n'); +writeFileSync(path.join(output, 'summary.md'), summary + '\n'); +console.log(summary); +const total = (variant) => results.filter((result) => result.variant === variant).reduce((sum, result) => sum + result.wallMs, 0); +assert.ok(total('candidate') < total('baseline'), 'No combined speedup measured; inspect the paired evidence before claiming an improvement'); diff --git a/scripts/ci-dependency-cache.mjs b/scripts/ci-dependency-cache.mjs new file mode 100644 index 00000000000..b447c01d362 --- /dev/null +++ b/scripts/ci-dependency-cache.mjs @@ -0,0 +1,83 @@ +import { execFileSync } from "node:child_process"; +import { createHash } from "node:crypto"; +import { appendFileSync, existsSync, readFileSync, realpathSync } from "node:fs"; +import path from "node:path"; +import { fileURLToPath } from "node:url"; +import { readPluginsUnder } from "./link-plugin-dev-sdk.mjs"; + +const root = path.resolve(path.dirname(fileURLToPath(import.meta.url)), ".."); + +export function cachePlan({ files, read, scope, packageDirs }) { + const dependencyInputs = files.filter((file) => + /(^|\/)package\.json$/.test(file) + || ["pnpm-lock.yaml", "pnpm-workspace.yaml", ".npmrc", ".pnpmfile.cjs", "pnpmfile.cjs"].includes(file) + || file.startsWith("scripts/") + || file.startsWith("patches/")); + const digest = (inputs, salt) => { + const hash = createHash("sha256").update(JSON.stringify(salt)); + for (const file of [...inputs].sort()) { + const bytes = read(file); + hash.update(JSON.stringify([file, bytes === null ? null : bytes.length])); + if (bytes !== null) hash.update(bytes); + } + return hash.digest("hex"); + }; + const dependencyKey = `ci-dependencies-v1-${digest(dependencyInputs, scope)}`; + const cacheable = packageDirs.every((directory) => { + const manifest = read(path.posix.join(directory, 'package.json')); + if (manifest === null) return false; + const scripts = JSON.parse(manifest).scripts ?? {}; + return ['preinstall', 'install', 'postinstall', 'prepare'].every((hook) => + scripts[hook] === undefined || (directory === '.' && hook === 'postinstall' + && scripts[hook] === 'node scripts/link-plugin-dev-sdk.mjs')); + }); + const buildInputs = files.filter((file) => + file.startsWith("packages/shared/") || file.startsWith("packages/plugins/sdk/") + || /^tsconfig[^/]*\.json$/.test(file) + || file === "scripts/ensure-plugin-build-deps.mjs"); + return { + cacheable, + dependencyKey, + buildKey: `ci-build-dependencies-v1-${digest(buildInputs, dependencyKey)}`, + dependencyPaths: [...new Set(packageDirs.map((directory) => { + if (path.isAbsolute(directory) || directory.split('/').includes('..')) throw new Error('Package directory escapes the checkout'); + return path.posix.join(directory, 'node_modules'); + }))].sort(), + }; +} + +if (process.argv[1] && path.resolve(process.argv[1]) === fileURLToPath(import.meta.url)) { + const files = execFileSync("git", ["ls-files", "-z"], { cwd: root }).toString().split("\0").filter(Boolean); + const command = (args) => execFileSync("pnpm", args, { cwd: root, encoding: "utf8" }).trim(); + const workspaces = JSON.parse(command(['-r', 'list', '--depth=-1', '--json'])); + const packageDirs = ['.', + ...workspaces.map((workspace) => path.relative(root, workspace.path).split(path.sep).join('/') || '.'), + ...readPluginsUnder(path.join(root, 'packages/plugins/sandbox-providers')).map((directory) => path.relative(root, directory).split(path.sep).join('/')), + 'packages/plugins/examples/plugin-orchestration-smoke-example', + ]; + const plan = cachePlan({ + files, + read: (file) => existsSync(path.join(root, file)) ? readFileSync(path.join(root, file)) : null, + packageDirs, + scope: { + root: realpathSync(root), node: process.version, abi: process.versions.modules, + platform: process.platform, arch: process.arch, + libc: process.report.getReport().header.glibcVersionRuntime ?? null, + pnpm: command(["--version"]), store: command(["store", "path", "--silent"]), + }, + }); + const output = [ + `cacheable=${plan.cacheable}`, + `dependency-key=${plan.dependencyKey}`, + `build-key=${plan.buildKey}`, + "dependency-paths< path.join(root, file)), + "PAPERCLIP_CACHE_PATHS", + "build-paths< 0) { runVitest( [ "--project", "@paperclipai/server", - routeTest.repoPath, + ...shardTests.map((test) => test.repoPath), "--pool=forks", "--isolate", + ...serializedServerVitestArgs, ], - routeTest.repoPath, + `serialized shard ${shardIndex + 1}/${shardCount}`, ); } } diff --git a/server/src/__tests__/setup-supertest.ts b/server/src/__tests__/setup-supertest.ts index 82472092bf3..be189a6abe0 100644 --- a/server/src/__tests__/setup-supertest.ts +++ b/server/src/__tests__/setup-supertest.ts @@ -29,6 +29,8 @@ process.env.PAPERCLIP_TEST_HOST_HOME ??= process.env.PAPERCLIP_HOME?.trim() || p // Receipt-spool recovery must never scan a developer's instance during tests. const { home: paperclipTestHome, cleanup: cleanupPaperclipTestHome } = createIsolatedTestHome(); process.env.PAPERCLIP_HOME = paperclipTestHome; +process.env.PAPERCLIP_CONFIG = path.join(paperclipTestHome, "config.json"); +process.env.PAPERCLIP_INSTANCE_ID = `vt-${process.pid}`; // Setup-file afterAll hooks run before suites drain asynchronous heartbeats. // Cleanup at worker exit, after all suite teardown, avoids deleting live state. process.once("exit", cleanupPaperclipTestHome); diff --git a/server/src/__tests__/workspace-runtime.test.ts b/server/src/__tests__/workspace-runtime.test.ts index 3e193c74a16..1969d8a4bf7 100644 --- a/server/src/__tests__/workspace-runtime.test.ts +++ b/server/src/__tests__/workspace-runtime.test.ts @@ -7326,8 +7326,10 @@ describeEmbeddedPostgres("workspace runtime service control persistence", () => } }, 20_000); - it("commits fixed-port reservations before readiness windows longer than 60 seconds", async () => { + it("commits fixed-port reservations across the configured database idle timeout", async () => { const fixture = await createRuntimeFixture(); + const readinessDb = createDb(tempDb!.connectionString, { idleTimeoutSeconds: 1, maxConnections: 1 }); + const [initialConnection] = await readinessDb.$client<{ pid: number }[]>`select pg_backend_pid() as pid`; const cleanupRuntimeHome = await createRuntimeHome(); const workspace = fixture.workspaces[0]!; const basePort = await findFreePort(); @@ -7353,7 +7355,7 @@ describeEmbeddedPostgres("workspace runtime service control persistence", () => try { startPromise = startRuntimeServicesForWorkspaceControl({ - db, + db: readinessDb, actor: fixture.actor, issue: null, workspace: fixture.realizedWorkspace(workspace), @@ -7377,16 +7379,18 @@ describeEmbeddedPostgres("workspace runtime service control persistence", () => parentWrite.then(() => "committed" as const), new Promise<"locked">((resolve) => setTimeout(() => resolve("locked"), 2_000)), ]); + expect(parentWriteOutcome).toBe("committed"); const readinessWaitStartedAt = Date.now(); - await new Promise((resolve) => setTimeout(resolve, 60_250)); - expect(Date.now() - readinessWaitStartedAt).toBeGreaterThanOrEqual(60_000); + await new Promise((resolve) => setTimeout(resolve, 1_250)); + expect(Date.now() - readinessWaitStartedAt).toBeGreaterThanOrEqual(1_000); + const [reconnected] = await readinessDb.$client<{ pid: number }[]>`select pg_backend_pid() as pid`; + expect(reconnected!.pid).not.toBe(initialConnection!.pid); await fs.writeFile(releaseMarkerPath, "release"); const started = (await startPromise)[0]!; await parentWrite; - expect(parentWriteOutcome).toBe("committed"); expect(started).toMatchObject({ port: basePort, url: `http://127.0.0.1:${basePort}`, @@ -7413,6 +7417,7 @@ describeEmbeddedPostgres("workspace runtime service control persistence", () => }).catch(() => undefined); await cleanupRuntimeHome(); await fixture.cleanup(); + await readinessDb.$client.end({ timeout: 1 }); } }, 80_000); diff --git a/server/src/services/workspace-git-snapshot-streaming.test.ts b/server/src/services/workspace-git-snapshot-streaming.test.ts index fa6a8247cb6..6dbfe303ed3 100644 --- a/server/src/services/workspace-git-snapshot-streaming.test.ts +++ b/server/src/services/workspace-git-snapshot-streaming.test.ts @@ -53,7 +53,10 @@ function localClient(commands: string[]): SandboxManagedRuntimeClient { // macOS adds per-file process and filesystem overhead to the real deletion lane. const realGitTimeoutMs = process.platform === "darwin" ? 300_000 : 180_000; -it("streams and stages all four real Git filename lanes above 32 MiB through the shared scheduler", async () => { +it.each([ + { label: "above 32 MiB", fileCount: 40_000, expectedBytes: 34_988_890, minimumBytes: 32 * 1024 * 1024, stageFiles: false }, + { label: "with staging above argument limits", fileCount: 4_000, expectedBytes: 3_494_890, minimumBytes: 2 * 1024 * 1024, stageFiles: true }, +])("streams all four real Git filename lanes $label through the shared scheduler", async ({ fileCount, expectedBytes, minimumBytes, stageFiles }) => { const root = await fs.realpath(await fs.mkdtemp(path.join(os.tmpdir(), "pc-stream-git-"))); directories.push(root); const repo = path.join(root, "repo"); @@ -68,7 +71,7 @@ it("streams and stages all four real Git filename lanes above 32 MiB through the await fs.mkdir(parent, { recursive: true }); const fileName = (index: number) => `${"asset-".repeat(36)}${index}.js`; const writeFiles = async (content: string) => { - for (let start = 0; start < 40_000; start += 100) { + for (let start = 0; start < fileCount; start += 100) { await Promise.all(Array.from({ length: 100 }, (_, offset) => fs.writeFile(path.join(parent, fileName(start + offset)), content))); } }; @@ -95,9 +98,9 @@ it("streams and stages all four real Git filename lanes above 32 MiB through the count++; bytes += Buffer.byteLength(relative) + 1; } - expect(count).toBe(40_000); - expect(bytes).toBe(34_988_890); - expect(bytes).toBeGreaterThan(32 * 1024 * 1024); + expect(count).toBe(fileCount); + expect(bytes).toBe(expectedBytes); + expect(bytes).toBeGreaterThan(minimumBytes); }; // Standalone callers must use the same streaming semantics. setExpensiveWorkspaceGitExecutor(null); @@ -107,7 +110,6 @@ it("streams and stages all four real Git filename lanes above 32 MiB through the bindScheduler(); const snapshot = await scan(); checkLargeLane(snapshot.overlayPaths); - await fs.writeFile(path.join(parent, "late.js"), "must not stage"); const commands: string[] = []; const remote = path.join(root, "remote"); const stage = async (gitSnapshot: GitWorkspaceSnapshot) => { @@ -118,19 +120,22 @@ it("streams and stages all four real Git filename lanes above 32 MiB through the runtimes.push(runtime); return runtime; }; - const runtime = await stage(snapshot); const remoteParent = path.join(remote, path.relative(repo, parent)); - let stagedCount = 0; - for await (const entry of await fs.opendir(remoteParent)) { - expect(await fs.readFile(path.join(remoteParent, entry.name), "utf8")).toBe("base"); - stagedCount++; + if (stageFiles) { + await fs.writeFile(path.join(parent, "late.js"), "must not stage"); + const runtime = await stage(snapshot); + let stagedCount = 0; + for await (const entry of await fs.opendir(remoteParent)) { + expect(await fs.readFile(path.join(remoteParent, entry.name), "utf8")).toBe("base"); + stagedCount++; + } + expect(stagedCount).toBe(fileCount); + for (const name of unusual) expect(await fs.readFile(path.join(remote, name), "utf8")).toBe(name); + const manifestFile = isPathManifest(snapshot.overlayPaths) ? snapshot.overlayPaths.filePath : ""; + await runtime.cleanupWorkspaceSnapshot(); + await expect(fs.stat(manifestFile)).rejects.toMatchObject({ code: "ENOENT" }); + await fs.rm(path.join(parent, "late.js")); } - expect(stagedCount).toBe(40_000); - for (const name of unusual) expect(await fs.readFile(path.join(remote, name), "utf8")).toBe(name); - const manifestFile = isPathManifest(snapshot.overlayPaths) ? snapshot.overlayPaths.filePath : ""; - await runtime.cleanupWorkspaceSnapshot(); - await expect(fs.stat(manifestFile)).rejects.toMatchObject({ code: "ENOENT" }); - await fs.rm(path.join(parent, "late.js")); await fs.writeFile(path.join(repo, ".gitignore"), "*.js\n"); const ignored = await scan(); @@ -146,10 +151,13 @@ it("streams and stages all four real Git filename lanes above 32 MiB through the await fs.rm(parent, { recursive: true }); const deleted = await scan(); checkLargeLane(deleted.deletedPaths); - await stage(deleted); - let remaining = 0; - for await (const _entry of await fs.opendir(remoteParent)) remaining++; - expect(remaining).toBe(0); + if (stageFiles) { + await stage(deleted); + let remaining = 0; + for await (const _entry of await fs.opendir(remoteParent)) remaining++; + expect(remaining).toBe(0); + expect(commands.length).toBeGreaterThan(0); + } expect(commands.every((command) => command.length < 16 * 1024)).toBe(true); expect(scheduler.snapshot()).toMatchObject({ activeCount: 0, queuedCount: 0, inFlightCount: 0, cacheBytes: 0 }); }, realGitTimeoutMs); From d5b19756214ff9d2d348e26ddcaa97c187907e7d Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Thu, 15 Oct 2026 08:09:38 -0700 Subject: [PATCH 06/19] fix(native): serialize warm idle expiry with turn admission (#16432) The warm idle timer no longer closes a warm native session while a continuation has reserved or is preparing the same session scope; it re-arms instead. An idle close that has already started is registered in closingWarmNativeSessions so turn admission waits for the checkpoint, matching the other closers. Co-Authored-By: Paperclip Co-Authored-By: Claude Opus 5.5 --- .../native-session-executor.test.ts | 80 +++++++++++++++++++ .../native-runtime/native-session-executor.ts | 24 +++++- 2 files changed, 102 insertions(+), 2 deletions(-) diff --git a/server/src/services/native-runtime/native-session-executor.test.ts b/server/src/services/native-runtime/native-session-executor.test.ts index f90b4ffeb9d..fac9f420114 100644 --- a/server/src/services/native-runtime/native-session-executor.test.ts +++ b/server/src/services/native-runtime/native-session-executor.test.ts @@ -7906,6 +7906,86 @@ describe("native warm session supervision", () => { } }); + it.each(["idle expiry first", "turn first"])("serializes idle expiry and turn admission: %s", async (order) => { + const stateBase = await mkdtemp(join(tmpdir(), "paperclip-idle-race-")); + const previousPaperclipHome = process.env.PAPERCLIP_HOME; + process.env.PAPERCLIP_HOME = stateBase; + let finishClose!: () => void; + const closing = new Promise((resolve) => { finishClose = resolve; }); + const close = vi.fn(() => closing); + const idleTimeoutMs = 200; + try { + const first = { + ...execution, + binding: { + ...execution.binding, + runId: "run-idle-race-first", + executionWorkspaceId: "workspace-idle-race", + }, + session: { + ...execution.session, + normalizedSessionId: "session-idle-race", + lifecyclePolicy: { mode: "warm" as const, idleTimeoutMs }, + }, + } as NativeExecutionInputV1; + const result = { + result: { summary: "completed" }, + terminal: { runTerminalState: "succeeded" }, + turnId: "turn-idle-race", + normalizedSessionId: first.session.normalizedSessionId, + providerSessionId: "provider-idle-race", + driverKind: "test", + driverVersion: "1", + nativeEventCount: 1, + highestContiguousSourceSeq: 1, + usage: null, + }; + state.execute.mockReset().mockImplementationOnce(async (options) => { + await options.onSession?.({ close }); + return result; + }); + await executePaperclipNativeSession({ + db: leaseDb(first), execution: first, + runnerInstanceId: "runner-idle-race", useRunnerd: true, + }); + const second = { ...first, binding: { ...first.binding, runId: "run-idle-race-second" } }; + const reachedAdmission = new Error("test reached post-checkpoint admission"); + state.stageNativeRunnerWakeAttachments.mockClear(); + state.stageNativeRunnerWakeAttachments.mockImplementationOnce(async () => { + if (order === "turn first") { + // The idle window elapses while the continuation still prepares. + await new Promise((resolve) => setTimeout(resolve, idleTimeoutMs * 2)); + expect(close).not.toHaveBeenCalled(); + } + throw reachedAdmission; + }); + if (order === "idle expiry first") { + await vi.waitFor(() => expect(close).toHaveBeenCalledOnce(), { timeout: 2_000 }); + } + const continuation = executePaperclipNativeSession({ + db: leaseDb(second), execution: second, + runnerInstanceId: "runner-idle-race", useRunnerd: true, + }).catch((error) => error); + if (order === "idle expiry first") { + await new Promise((resolve) => setTimeout(resolve, 50)); + expect(state.stageNativeRunnerWakeAttachments).not.toHaveBeenCalled(); + finishClose(); + } + expect(await continuation).toBe(reachedAdmission); + expect(state.stageNativeRunnerWakeAttachments).toHaveBeenCalledOnce(); + // A deferred idle owner still retires once the scope is released. + await vi.waitFor(() => expect(close).toHaveBeenCalledExactlyOnceWith({ + reason: "warm native session idle timeout", + }), { timeout: 2_000 }); + } finally { + finishClose(); + await closeIdleWarmNativeSessionsForRestart(); + if (previousPaperclipHome === undefined) delete process.env.PAPERCLIP_HOME; + else process.env.PAPERCLIP_HOME = previousPaperclipHome; + await rm(stateBase, { recursive: true, force: true }); + } + }); + it.each([false, true])("checkpoints a busy warm session on release after the restart sweep: checkpoint fails=%s", async (checkpointFails) => { const checkpointError = new Error("restart checkpoint failed"); let finishCheckpoint!: () => void; diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index 24d5198972f..04dcb09e8af 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -6162,15 +6162,35 @@ async function releaseWarmNativeSession( } finally { entry.busy = false; } return; } + armWarmNativeSessionIdleTimer(sessionId, entry, idleTimeoutMs); +} + +function armWarmNativeSessionIdleTimer( + sessionId: string, + entry: WarmNativeSession, + idleTimeoutMs: number, +): void { entry.idleTimer = setTimeout(() => { const current = warmNativeSessions.get(sessionId); if (current !== entry || current.busy) return; + if (current.preparingRunId || executingRunnerdSessionScopes.has(sessionId)) { + // A continuation has reserved this scope and is still preparing. It + // verifies this idle owner as the prior run's authority; suspending it + // underneath that check makes the ready state look indeterminate. + armWarmNativeSessionIdleTimer(sessionId, current, idleTimeoutMs); + return; + } current.busy = true; current.closeOnReleaseReason = "warm native session idle timeout"; - void closeWarmNativeSession(current, current.closeOnReleaseReason) + const closing = closeWarmNativeSession(current, current.closeOnReleaseReason) .then(() => { if (warmNativeSessions.get(sessionId) === current) warmNativeSessions.delete(sessionId); }) .catch(() => { if (!current.instructionWorkingCopy?.runId && warmNativeSessions.get(sessionId) === current) warmNativeSessions.delete(sessionId); }) - .finally(() => { current.busy = false; }); + .finally(() => { + current.busy = false; + if (closingWarmNativeSessions.get(sessionId) === closing) closingWarmNativeSessions.delete(sessionId); + }); + // Admission waits for this checkpoint like a restart sweep's close. + closingWarmNativeSessions.set(sessionId, closing); }, idleTimeoutMs); entry.idleTimer.unref(); } From 03f22f4cb9be9a34906bda0e4906262dc359e9ce Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Thu, 15 Oct 2026 10:12:31 -0500 Subject: [PATCH 07/19] chore(lockfile): refresh pnpm-lock.yaml (#16189) Auto-generated lockfile refresh after dependencies changed on master. This PR only updates pnpm-lock.yaml. Co-authored-by: lockfile-bot --- pnpm-lock.yaml | 150 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 150 insertions(+) diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 0fd19271bf0..c20b065b64a 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -166,6 +166,9 @@ importers: '@types/node': specifier: ^24.0.0 version: 24.13.3 + tsx: + specifier: ^4.23.15 + version: 4.23.15 typescript: specifier: ^7.0.2 version: 7.0.2 @@ -1099,6 +1102,12 @@ importers: specifier: 1.0.88 version: 1.0.88 devDependencies: + '@opentelemetry/context-async-hooks': + specifier: 2.10.0 + version: 2.10.0(@opentelemetry/api@1.9.1) + '@opentelemetry/sdk-trace': + specifier: 2.10.0 + version: 2.10.0(@opentelemetry/api@1.9.1) '@paperclipai/paperclip-runner': specifier: workspace:* version: link:../packages/paperclip-runner @@ -1213,6 +1222,9 @@ importers: '@radix-ui/react-slot': specifier: ^1.3.3 version: 1.3.3(@types/react@19.2.18)(react@19.2.8) + '@spekoai/client': + specifier: 0.0.13 + version: 0.0.13 '@tailwindcss/typography': specifier: ^0.5.20 version: 0.5.20(tailwindcss@4.3.3) @@ -3164,6 +3176,12 @@ packages: '@lezer/yaml@1.0.4': resolution: {integrity: sha512-2lrrHqxalACEbxIbsjhqGpSW8kWpUKuY6RHgnSAFZa6qK62wvnPxA8hGOwOoDbwHcOFs5M4o27mjGu+P7TvBmw==} + '@livekit/mutex@1.1.1': + resolution: {integrity: sha512-EsshAucklmpuUAfkABPxJNhzj9v2sG7JuzFDL4ML1oJQSV14sqrpTYnsaOudMAw9yOaW53NU3QQTlUQoRs4czw==} + + '@livekit/protocol@1.52.1': + resolution: {integrity: sha512-X8DxKlNbzk6Lh+v630+mjdFHSotdAFwIyzxSRSUql2DbTxCnlpa7JsvFGc6Na3mXiex2OFh0iiQtWwOBP0rIYA==} + '@marijn/find-cluster-break@1.0.4': resolution: {integrity: sha512-Wy0V7+SGUjnF9/TkiM1hKVDPj7jKXduPNboMVtHTA8dySMURWqfg/JZ9E2Sq8JgSJmkl7k7Qe9FLeMSrSraWmQ==} @@ -3384,6 +3402,30 @@ packages: resolution: {integrity: sha512-gLyJlPHPZYdAk1JENA9LeHejZe1Ti77/pTeFm/nMXmQH/HFZlcS/O2XJB+L8fkbrNSqhdtlvjBVjxwUYanNH5Q==} engines: {node: '>=8.0.0'} + '@opentelemetry/context-async-hooks@2.10.0': + resolution: {integrity: sha512-bvyMcgLEkozzSzpEEEo1OMoeQ97bxj6Qs2uN3mPrSdDvObMI1myffD/BPqcLlzZO9//d1SqQA/WPw7Cz2AiqhA==} + engines: {node: ^18.19.0 || >=20.6.0} + peerDependencies: + '@opentelemetry/api': '>=1.0.0 <1.10.0' + + '@opentelemetry/core@2.10.0': + resolution: {integrity: sha512-/wNZ8twnEQQA4HoHu22+vcsdru6pWPWxW+7w+FlxT6Id7PE/WIbZmVKkte+PF72e0F2dnImFeHD2syyE1Mw6MQ==} + engines: {node: ^18.19.0 || >=20.6.0} + peerDependencies: + '@opentelemetry/api': '>=1.0.0 <1.10.0' + + '@opentelemetry/resources@2.10.0': + resolution: {integrity: sha512-q6MMm2zhggzsHVNbabYwut+a6nbuQQe3URUoxaojM/8K1IBfwwPzvxIjNi2/lI1TFe+fMHMW9MWhrtDLEXEnkA==} + engines: {node: ^18.19.0 || >=20.6.0} + peerDependencies: + '@opentelemetry/api': '>=1.3.0 <1.10.0' + + '@opentelemetry/sdk-trace@2.10.0': + resolution: {integrity: sha512-MfQGq3GRmTh5fM/y+OjaO0vj6+luCB1XO2gfXCalKCfgKw0eHL++sm75DNweC6ohlp+aFvACqeE0fYayqdRaoQ==} + engines: {node: ^18.19.0 || >=20.6.0} + peerDependencies: + '@opentelemetry/api': '>=1.3.0 <1.10.0' + '@opentelemetry/semantic-conventions@1.43.0': resolution: {integrity: sha512-eSYWTm620tTk45EKSedaUL8MFYI8hW164hIXsgIHyxu3VobUB3fFCu5t0hQby6OoWRPsG1KkKUG2M5UadiLiVg==} engines: {node: '>=14'} @@ -4947,6 +4989,9 @@ packages: resolution: {integrity: sha512-7wNWV7SugHpcMA7uzEawJNpE0GrasXM7a9E+1+Wm6NxVuDClESac/AKt+G7jMZNUz5vBLWqKlqVV7Sv7AtUr6Q==} engines: {node: '>=18.0.0'} + '@spekoai/client@0.0.13': + resolution: {integrity: sha512-Jodmw5jE9UPY0k2UrVJETFMahtSbwpImHsJ5eWrk7nWjPi0BgOOzlUVaNQsGR9153SIyeY/pNUu8RmDx4ryKaw==} + '@stablelib/base64@1.0.1': resolution: {integrity: sha512-1bnPQqSxSuc3Ii6MhBysoWCg58j97aUjuCSZrGSmDxNqtytIi0k8utUenAwTZN4V5mXXYGsVUI9zeBqy+jBOSQ==} @@ -6732,6 +6777,10 @@ packages: events-universal@1.0.1: resolution: {integrity: sha512-LUd5euvbMLpwOF8m6ivPCbhQeSiYVNb8Vs0fQ8QjXo0JTkEHpz8pxdQf0gStltaPpw0Cca8b39KxvK9cfKRiAw==} + events@3.3.0: + resolution: {integrity: sha512-mQw+2fkQbALzQ7V0MY0IqdnXNOeTtP4r0lN9z7AAawCXgqea7bDii20AYrIBrFd/Hx0M2Ocz6S111CaFkUcb0Q==} + engines: {node: '>=0.8.x'} + eventsource-parser@3.1.1: resolution: {integrity: sha512-EKN1vKAMcZ8MlYMpaNuxN6R9yakzH6uajHcHVTqWJzvu5pWw9DyhbP35HH8MVBQ+dZjAfDxk+A8NiR9KWaXiyQ==} engines: {node: '>=18.0.0'} @@ -7384,6 +7433,11 @@ packages: limiter@1.1.5: resolution: {integrity: sha512-FWWMIEOxz3GwUI4Ts/IvgVy6LPvoMPgjMdQ185nN6psJyBJ4yOpzqm695/h5umdLJg2vW3GR5iG11MAkR2AzJA==} + livekit-client@2.22.4: + resolution: {integrity: sha512-UJ5K8gqP8uIO+tjhGeF+s6SzgLfxZKPxQjSqzzkF298QNYMnhYmXGQRMMh3GnsfaeswnsHHMSogMeb7SocH2zw==} + peerDependencies: + '@types/dom-mediacapture-record': ^1 + lodash-es@4.18.1: resolution: {integrity: sha512-J8xewKD/Gk22OZbhpOVSwcs60zhd95ESDwezOFuA3/099925PdHJ7OFHNTGtajL3AlZkykD32HykiMo+BIBI8A==} @@ -7420,6 +7474,10 @@ packages: lodash@4.18.1: resolution: {integrity: sha512-dMInicTPVE8d1e5otfwmmjlxkZoUpiVLwyeTdUsi/Caj/gfzzblBcCE5sRHV/AsjuCmxWrte2TNGSYuCeCq+0Q==} + loglevel@1.9.2: + resolution: {integrity: sha512-HgMmCqIJSAKqo68l0rS2AanEWfkxaZ5wNiEFb5ggm08lDs9Xl2KxBlX3PTcaD2chBM1gXAYf491/M2Rv8Jwayg==} + engines: {node: '>= 0.6.0'} + long@5.3.2: resolution: {integrity: sha512-mNAgZ1GmyNhD7AuqnTG3/VQ26o760+ZYBPKjPvugO8+nLbYfX6TVpJPseBvopbdY+qpZ/lKUnmEc1LeZYS3QAA==} @@ -7459,6 +7517,10 @@ packages: resolution: {integrity: sha512-h5bgJWpxJNswbU7qCrV0tIKQCaS3blPDrqKWx+QxzuzL1zGUzij9XCWLrSLsJPu5t+eWA/ycetzYAO5IOMcWAQ==} hasBin: true + machina@7.0.1: + resolution: {integrity: sha512-0Xkgs8NeBvO2hW6bjJT8ewLWrMfh1haMPstaghpASWjXkLjRhtDiaJG1Kb7mslaOHxZYxpH7PIjpmirRVYbMMA==} + engines: {node: '>=22.22'} + magic-bytes.js@1.13.1: resolution: {integrity: sha512-x5sn4UX2k5gCWlcfmoFwG4TPie8+dctESyqOBdhB5p6MsgWXdBKGmt9nXPObj/JI50TTL928lc5Yt1WntMn1bw==} @@ -8389,6 +8451,9 @@ packages: rw@1.3.3: resolution: {integrity: sha512-PdhdWy89SiZogBLaw42zdeqtRJ//zFd2PgQavcICDUgJT5oW10QCRKbJ6bg4r0/UY2M6BWd5tkxuGFRvCkgfHQ==} + rxjs@7.8.2: + resolution: {integrity: sha512-dhKf903U/PQZY6boNNtAGdWbG85WAbjT/1xYoZIC7FAY0yWapOBQVsVrDl58W86//e1VpMNBtRV4MaXfdMySFA==} + sade@1.8.1: resolution: {integrity: sha512-xal3CZX1Xlo/k4ApwCFrHVACi9fBqJ7V+mwhBsuf/1IOKbBy098Fex+Wa/5QMubw09pSZ/u8EY8PWgevJsXp1A==} engines: {node: '>=6'} @@ -8413,6 +8478,13 @@ packages: scheduler@0.27.0: resolution: {integrity: sha512-eNv+WrVbKu1f3vbYJT/xtiF5syA5HPIMtf9IgY/nKg0sWqzAUEvqY/xm7OcZc/qafLx/iO9FgOmeSAp4v5ti/Q==} + sdp-transform@2.15.0: + resolution: {integrity: sha512-KrOH82c/W+GYQ0LHqtr3caRpM3ITglq3ljGUIb8LTki7ByacJZ9z+piSGiwZDsRyhQbYBOBJgr2k6X4BZXi3Kw==} + hasBin: true + + sdp@3.2.2: + resolution: {integrity: sha512-xZocWwfyp4hkbN4hLWxMjmv2Q8aNa9MhmOZ7L9aCZPT+dZsgRr6wZRrSYE3HTdyk/2pZKPSgqI7ns7Een1xMSA==} + secure-json-parse@4.1.0: resolution: {integrity: sha512-l4KnYfEyqYJxDwlNVyRfO2E4NTHfMKAWdUuA8J0yve2Dz/E/PdBepY03RvyJpssIpRFwJoCD55wA+mEDs6ByWA==} @@ -8742,6 +8814,9 @@ packages: resolution: {integrity: sha512-OZs6gsjF4vMp32qrCbiVSkrFmXtG/AZhY3t0iAMrMBiAZyV9oALtXO8hsrHbMXF9x6L3grlFuwW2oAz7cav+Gw==} engines: {node: '>= 0.6'} + typed-emitter@2.1.0: + resolution: {integrity: sha512-g/KzbYKbH5C2vPkaXGu8DJlHrGKHLsM25Zg9WuC9pMGfuvT+X25tZQWo5fK1BjBm8+UrVE9LDCvaY0CQk+fXDA==} + typescript@7.0.2: resolution: {integrity: sha512-8FYau96o3NKOhbjKi/qNvG/W5jhzxkbdm5sj9AbZ/5T5sWqn3hJgLfGx27sRKZWTvyzCP8dLRBTf5tBTSRVUNA==} engines: {node: '>=16.20.0'} @@ -9052,6 +9127,10 @@ packages: webpack-virtual-modules@0.6.2: resolution: {integrity: sha512-66/V2i5hQanC51vBQKPH4aI8NMAcBW59FVBs+rC7eGHupMyfn34q7rZIE+ETlJ+XTevqfUhVVBgSUNSW2flEUQ==} + webrtc-adapter@9.0.6: + resolution: {integrity: sha512-CHbl2ZQbxx164IgWRgzJno4hWtM4tFbRam1QfI3Yxhs3w/DvqluVxVWeXs3oL5/fbGkSNLKo0Ty5MgUWceNhog==} + engines: {node: '>=6.0.0', npm: '>=3.10.0'} + whatwg-mimetype@5.0.0: resolution: {integrity: sha512-sXcNcHOC51uPGF0P/D4NVtrkjSU2fNsm9iog4ZvZJsL3rjoDAzXZhkm2MWt1y+PUdggKAYVoMAIYcs78wJ51Cw==} engines: {node: '>=20'} @@ -11198,6 +11277,12 @@ snapshots: '@lezer/highlight': 1.2.3 '@lezer/lr': 1.4.10 + '@livekit/mutex@1.1.1': {} + + '@livekit/protocol@1.52.1': + dependencies: + '@bufbuild/protobuf': 1.10.0 + '@marijn/find-cluster-break@1.0.4': {} '@mdx-js/react@3.1.1(@types/react@19.2.18)(react@19.2.8)': @@ -11543,6 +11628,28 @@ snapshots: '@opentelemetry/api@1.9.1': {} + '@opentelemetry/context-async-hooks@2.10.0(@opentelemetry/api@1.9.1)': + dependencies: + '@opentelemetry/api': 1.9.1 + + '@opentelemetry/core@2.10.0(@opentelemetry/api@1.9.1)': + dependencies: + '@opentelemetry/api': 1.9.1 + '@opentelemetry/semantic-conventions': 1.43.0 + + '@opentelemetry/resources@2.10.0(@opentelemetry/api@1.9.1)': + dependencies: + '@opentelemetry/api': 1.9.1 + '@opentelemetry/core': 2.10.0(@opentelemetry/api@1.9.1) + '@opentelemetry/semantic-conventions': 1.43.0 + + '@opentelemetry/sdk-trace@2.10.0(@opentelemetry/api@1.9.1)': + dependencies: + '@opentelemetry/api': 1.9.1 + '@opentelemetry/core': 2.10.0(@opentelemetry/api@1.9.1) + '@opentelemetry/resources': 2.10.0(@opentelemetry/api@1.9.1) + '@opentelemetry/semantic-conventions': 1.43.0 + '@opentelemetry/semantic-conventions@1.43.0': {} '@oxc-parser/binding-android-arm-eabi@0.127.0': @@ -12962,6 +13069,13 @@ snapshots: '@smithy/core': 3.35.1 tslib: 2.8.1 + '@spekoai/client@0.0.13': + dependencies: + livekit-client: 2.22.4 + tslib: 2.8.1 + transitivePeerDependencies: + - '@types/dom-mediacapture-record' + '@stablelib/base64@1.0.1': {} '@standard-schema/spec@1.1.0': {} @@ -14658,6 +14772,8 @@ snapshots: transitivePeerDependencies: - bare-abort-controller + events@3.3.0: {} + eventsource-parser@3.1.1: {} eventsource@3.0.7: @@ -15324,6 +15440,19 @@ snapshots: limiter@1.1.5: {} + livekit-client@2.22.4: + dependencies: + '@livekit/mutex': 1.1.1 + '@livekit/protocol': 1.52.1 + events: 3.3.0 + jose: 6.2.12 + loglevel: 1.9.2 + machina: 7.0.1 + sdp-transform: 2.15.0 + tslib: 2.8.1 + typed-emitter: 2.1.0 + webrtc-adapter: 9.0.6 + lodash-es@4.18.1: {} lodash.camelcase@4.3.0: {} @@ -15348,6 +15477,8 @@ snapshots: lodash@4.18.1: {} + loglevel@1.9.2: {} + long@5.3.2: {} longest-streak@3.1.0: {} @@ -15381,6 +15512,8 @@ snapshots: lz-string@1.5.0: {} + machina@7.0.1: {} + magic-bytes.js@1.13.1: {} magic-string@0.30.21: @@ -16766,6 +16899,11 @@ snapshots: rw@1.3.3: {} + rxjs@7.8.2: + dependencies: + tslib: 2.8.1 + optional: true + sade@1.8.1: dependencies: mri: 1.2.0 @@ -16784,6 +16922,10 @@ snapshots: scheduler@0.27.0: {} + sdp-transform@2.15.0: {} + + sdp@3.2.2: {} + secure-json-parse@4.1.0: {} semver@6.3.1: {} @@ -17191,6 +17333,10 @@ snapshots: media-typer: 1.1.0 mime-types: 3.0.2 + typed-emitter@2.1.0: + optionalDependencies: + rxjs: 7.8.2 + typescript@7.0.2: optionalDependencies: '@typescript/typescript-aix-ppc64': 7.0.2 @@ -17453,6 +17599,10 @@ snapshots: webpack-virtual-modules@0.6.2: {} + webrtc-adapter@9.0.6: + dependencies: + sdp: 3.2.2 + whatwg-mimetype@5.0.0: {} whatwg-url@16.0.1(@noble/hashes@2.4.0): From afe5b3418dbdf48aca5791f8fc8df12c7a75a246 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Thu, 15 Oct 2026 10:38:46 -0500 Subject: [PATCH 08/19] fix(native): verify warm ownership before configuration rotation Track the exact prior warm-owner run ID separately from mutable configuration. Keep configuration retirement, terminal-state checks and failed-close fences. Add current-v5 handoff regressions and document the recovery boundary. Co-Authored-By: Paperclip --- doc/DEVELOPING.md | 8 ++ .../2026-10-15-native-warm-owner-proof.md | 38 ++++++ .../native-session-executor.test.ts | 108 +++++++++++++++++- .../native-runtime/native-session-executor.ts | 43 +++---- 4 files changed, 169 insertions(+), 28 deletions(-) create mode 100644 doc/plans/2026-10-15-native-warm-owner-proof.md diff --git a/doc/DEVELOPING.md b/doc/DEVELOPING.md index b4b8437a284..65f279c1661 100644 --- a/doc/DEVELOPING.md +++ b/doc/DEVELOPING.md @@ -1892,6 +1892,14 @@ Only successful turns retain a live warm provider session. A structured failed or cancelled result must retire that session and collect its managed files before heartbeat stops the reusable sandbox. A later retry can resume the same sandbox without inheriting the stopped provider transport. + +An idle warm owner proves only the exact prior run in the same company, session, +workspace and environment. Mutable configuration compatibility is checked +separately: changed settings retire the owned provider before a replacement can +execute. A copied older owner record, an active or missing owner, and a different +environment cannot use this in-memory proof. Database terminal status alone is +not proof that an unsuspended remote runner can be adopted. + The shared lease-release boundary rechecks the durable run status for recovery and ordinary teardown. Successful workspace copy-back alone must not keep a failed turn's sandbox running. diff --git a/doc/plans/2026-10-15-native-warm-owner-proof.md b/doc/plans/2026-10-15-native-warm-owner-proof.md new file mode 100644 index 00000000000..ac74ccb8a35 --- /dev/null +++ b/doc/plans/2026-10-15-native-warm-owner-proof.md @@ -0,0 +1,38 @@ +# Exact warm-owner proof across native configuration changes + +A follow-up can fail with `prior_owner_terminal_state_indeterminate` before +provider execution even when its prior heartbeat succeeded. The old predicate +required the next configuration digest to match before recognizing the idle +owner; this prevented the existing compatibility step from retiring a process +whose configuration changed. + +Keep the session cache, durable-root admission and compatibility retirement in +their current coordinator. They share process and instruction-copy ownership; +this fix does not move their lifetime or weaken their gates. Cache each actual +owner's heartbeat run ID and require it to match the durable prior-run identity. +Keep company/session/workspace/environment checks and the terminal database +check. Let the later compatibility step decide reuse versus retirement, and +continue to reject unconfirmed retirement before another provider executes. + +Verification is through the production session entry point with scripted native +backends and durable local owner files. The six existing selected controls pass. +The initial three-case reproduction has one pass and two failures at the same +prior-owner guard; after the fix all three pass. Eight focused controls now cover +unchanged reuse through three turns, changed settings, changed runtime context, +failed retirement, stale recorded identity, active owner, missing owner and a +different environment. The runtime-context case uses the current v5 input and +fails on the baseline at the same prior-owner guard before passing the repair. Replacing the exact +run-ID comparison with an any-owner check makes the stale-owner control fail; +the source is restored afterward. + +The first complete affected executor and resume suites passed 764 cases before +the additional v5 runtime-context case. Full typecheck initially found an +optional-argument error in the test database wrapper; after that correction, +full typecheck and build passed. The v5 fixture expansion then exposed a legacy +v1-only type on the shared database fixture. Its implementation only uses the +common binding, so it now accepts the full native input union. After integrating the separate idle-expiry admission repair from #16432, the +complete executor and resume suites passed 767 cases with zero selected skips. +Full repository typecheck and build passed on the same frozen code and tests. Scripted-provider results do not establish real Daytona recovery, +managed subscription behavior or hosted latency. Recheck the failed hosted +continuation after normal deployment, preserving its existing recovery evidence. +The broader response-latency target remains open independently of this repair. diff --git a/server/src/services/native-runtime/native-session-executor.test.ts b/server/src/services/native-runtime/native-session-executor.test.ts index fac9f420114..572e126883b 100644 --- a/server/src/services/native-runtime/native-session-executor.test.ts +++ b/server/src/services/native-runtime/native-session-executor.test.ts @@ -35,6 +35,7 @@ import { type Db, } from "@paperclipai/db"; import { + canonicalNativeRuntimeContextDigest, CURSOR_DISTRIBUTION_PINS, QUALIFIED_ACPX_PROFILES, acpxRuntimeSessionDirectoryName, @@ -5158,7 +5159,7 @@ type LeaseCoordinator = { }; function leaseDb( - boundExecution: NativeExecutionInputV1 = execution, + boundExecution: NativeExecutionInput = execution, coordinatorOverrides: Partial = {}, runResultJson: Record = {}, updates: Array<{ table: unknown; values: Record }> = [], @@ -9727,7 +9728,7 @@ describe("runnerd provider runtime wiring", () => { }; } - function priorNativeRunDb(priorExecution: NativeExecutionInputV1, status = "succeeded"): Db { + function priorNativeRunDb(priorExecution: NativeExecutionInput, status = "succeeded"): Db { return { select: () => ({ from: () => ({ where: () => ({ limit: async () => [{ status, @@ -9736,7 +9737,7 @@ describe("runnerd provider runtime wiring", () => { } as unknown as Db; } - async function initializeScopedRunnerRoot(current: NativeExecutionInputV1, runnerInstanceId: string) { + async function initializeScopedRunnerRoot(current: NativeExecutionInput, runnerInstanceId: string) { state.createBackend.mockClear(); state.createTransport.mockClear(); await createRunnerdBackend({ db: leaseDb(current), execution: current, runnerInstanceId }); @@ -9771,6 +9772,107 @@ describe("runnerd provider runtime wiring", () => { await rm(isolatedStateDirectory, { recursive: true, force: true }); }); + it.each(["unchanged", "changed", "runtime-context", "retirement-failed", "stale-owner", "active-owner", "missing-owner", "foreign-environment"] as const)("rotates an owned warm remote runner before checking changed configuration: %s", async scenario => { + const first = { ...scopedNativeExecution(`warm-admission-${scenario}`), + schema: "paperclip.native-execution-input.v5" as const, + provider: { kind: "codex" as const, model: null, approvalPolicy: "never" as const }, + executionMode: "default" as const, planningContext: null, runtimeContext: nativeRuntimeContextFixture() }; + first.session.lifecyclePolicy = { mode: "warm", idleTimeoutMs: 60_000 }; + const next = { ...first, binding: { ...first.binding, runId: `${first.binding.runId}-next` }, + session: { ...first.session, lifecyclePolicy: { mode: "warm" as const, idleTimeoutMs: ["unchanged", "runtime-context", "stale-owner", "missing-owner"].includes(scenario) ? 60_000 : 90_000 } } }; + if (scenario === "runtime-context") { + next.runtimeContext = { ...first.runtimeContext, + mcp: { ...first.runtimeContext.mcp, digest: "1".repeat(64) } }; + next.runtimeContext.aggregateDigest = canonicalNativeRuntimeContextDigest(next.runtimeContext); + } + const environmentId = `warm-admission-${scenario}`; + const target = { kind: "remote" as const, transport: "sandbox" as const, environmentId, + remoteCwd: "/home/daytona/workspace", runner: { execute: vi.fn() } }; + const identity = { runId: first.binding.runId, normalizedSessionId: first.session.normalizedSessionId, + runnerInstanceId: `runner-${scenario}`, environmentLeaseId: first.binding.executionWorkspaceId }; + const root = await initializeScopedRunnerRoot(first, identity.runnerInstanceId); + await writeRunnerAuthority(root, identity, "ready"); + const order: string[] = []; + let failRetirement = scenario === "retirement-failed"; + const firstSession = { close: vi.fn(async () => { + order.push("retire"); + if (failRetirement) throw new Error("retirement unconfirmed"); + await writeRunnerAuthority(root, identity, "suspended"); + }) }; + const nextSession = { close: vi.fn(async () => undefined) }; + const result = { result: { summary: "done" }, terminal: { runTerminalState: "succeeded" }, + turnId: "turn", normalizedSessionId: first.session.normalizedSessionId, + providerSessionId: "provider", driverKind: "test", driverVersion: "1", + nativeEventCount: 1, highestContiguousSourceSeq: 1, usage: null }; + state.execute.mockReset().mockImplementationOnce(async options => { + await options.onSession?.(firstSession); + return result; + }).mockImplementationOnce(async options => { + order.push("execute-next"); + expect(options.existingSession).toBe(scenario === "unchanged" ? firstSession : undefined); + await options.onSession?.(scenario === "unchanged" ? firstSession : nextSession); + return result; + }); + const older = { ...first, binding: { ...first.binding, runId: `${first.binding.runId}-older` } }; + const nextDb = leaseDb(next); + const ordinarySelect = nextDb.select.bind(nextDb); + nextDb.select = ((fields?: Record) => fields?.status === heartbeatRuns.status + && fields?.runnerProfileJson === heartbeatRuns.runnerProfileJson + ? priorNativeRunDb(scenario === "stale-owner" ? older : first, scenario === "active-owner" ? "running" : "succeeded").select(fields as never) + : ordinarySelect(fields as never)) as Db["select"]; + try { + await executePaperclipNativeSession({ db: leaseDb(first), execution: first, + runnerInstanceId: identity.runnerInstanceId, useRunnerd: true, runnerExecutionTarget: target as never }); + if (scenario === "stale-owner") await writeRunnerAuthority(root, { ...identity, runId: older.binding.runId }, "ready"); + if (scenario === "missing-owner") { + await closeWarmNativeSessionsForEnvironment({ environmentId, reason: "fixture owner loss" }); + await writeRunnerAuthority(root, identity, "ready"); + order.length = 0; + firstSession.close.mockClear(); + } + const continuation = executePaperclipNativeSession({ db: nextDb, execution: next, + runnerInstanceId: `runner-next-${scenario}`, useRunnerd: true, + runnerExecutionTarget: (scenario === "foreign-environment" ? { ...target, environmentId: `${environmentId}-other` } : target) as never }); + if (["stale-owner", "active-owner", "missing-owner", "foreign-environment"].includes(scenario)) { + await expect(continuation).rejects.toThrow(scenario === "active-owner" + ? "prior_owner_active" : "prior_owner_terminal_state_indeterminate"); + expect(state.execute).toHaveBeenCalledOnce(); + expect(firstSession.close).not.toHaveBeenCalled(); + expect(order).toEqual([]); + } else if (scenario === "retirement-failed") { + await expect(continuation).rejects.toThrow("retirement unconfirmed"); + expect(state.execute).toHaveBeenCalledOnce(); + expect(order).toEqual(["retire"]); + } else { + await expect(continuation).resolves.toMatchObject({ exitCode: 0 }); + expect(order).toEqual(scenario === "unchanged" ? ["execute-next"] : ["retire", "execute-next"]); + if (scenario === "unchanged") { + await writeRunnerAuthority(root, { ...identity, runId: next.binding.runId }, "ready"); + const third = { ...next, binding: { ...next.binding, runId: `${next.binding.runId}-third` } }; + const thirdDb = leaseDb(third); + const thirdSelect = thirdDb.select.bind(thirdDb); + thirdDb.select = ((fields?: Record) => fields?.status === heartbeatRuns.status + && fields?.runnerProfileJson === heartbeatRuns.runnerProfileJson + ? priorNativeRunDb(next).select(fields as never) + : thirdSelect(fields as never)) as Db["select"]; + state.execute.mockImplementationOnce(async options => { + expect(options.existingSession).toBe(firstSession); + await options.onSession?.(firstSession); + return result; + }); + await expect(executePaperclipNativeSession({ db: thirdDb, execution: third, + runnerInstanceId: `runner-third-${scenario}`, useRunnerd: true, + runnerExecutionTarget: target as never })).resolves.toMatchObject({ exitCode: 0 }); + expect(firstSession.close).not.toHaveBeenCalled(); + expect(state.execute).toHaveBeenCalledTimes(3); + } + } + } finally { + failRetirement = false; + await closeWarmNativeSessionsForEnvironment({ environmentId, reason: "test cleanup" }); + } + }); + it.each([ ["open", "before-close"], ["open", "during-close"], diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index 04dcb09e8af..f90d6a55060 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -435,6 +435,7 @@ export type NativeInstructionWorkingCopy = { }; type WarmNativeSession = { + runId: string; instructionWorkingCopy?: NativeInstructionWorkingCopy; // Keep the admitted physical root independent of the per-run collection // capability, which instruction preparation adopts before final admission. @@ -4244,7 +4245,7 @@ async function verifyPriorRunnerdStateForSessionScope(input: { identity: RunnerdDurableIdentity; execution: NativeExecutionInput; allowVerifiedBackup: boolean; - allowRetainedWarmRunner: boolean; + retainedWarmOwnerRunId?: string; }): Promise { let priorRun: { status: string; @@ -4291,7 +4292,7 @@ async function verifyPriorRunnerdStateForSessionScope(input: { input.identity, ); if ( - input.allowRetainedWarmRunner && + input.retainedWarmOwnerRunId === input.identity.runId && (lifecycle === "not_suspended" || // A remote runner keeps runner-state.json in the sandbox rather than // beside the controller's PRP journal. During a genuinely warm handoff @@ -4753,7 +4754,7 @@ async function migrateRunnerdStateRootForExecution(input: { db: Db; execution: NativeExecutionInput; allowVerifiedBackup: boolean; - allowRetainedWarmRunner: boolean; + retainedWarmOwnerRunId?: string; allowLocalRecovery: boolean; onLog?: (stream: "stdout" | "stderr", chunk: string) => Promise; restartRecovery?: NativeRestartRecoveryClaim; @@ -4770,7 +4771,7 @@ async function migrateRunnerdStateRootForExecution(input: { }); if ( input.allowLocalRecovery && - !input.allowRetainedWarmRunner && + !input.retainedWarmOwnerRunId && !input.restartRecovery ) { await recoverQuiescentRunnerdState({ ...input, scoped }); @@ -4843,7 +4844,7 @@ async function migrateRunnerdStateRootForExecution(input: { identity, execution: input.execution, allowVerifiedBackup: input.allowVerifiedBackup, - allowRetainedWarmRunner: input.allowRetainedWarmRunner, + retainedWarmOwnerRunId: input.retainedWarmOwnerRunId, }); if ( verification !== "verified" && @@ -4885,7 +4886,7 @@ async function migrateRunnerdStateRootForExecution(input: { identity, execution: input.execution, allowVerifiedBackup: input.allowVerifiedBackup, - allowRetainedWarmRunner: input.allowRetainedWarmRunner, + retainedWarmOwnerRunId: input.retainedWarmOwnerRunId, }); if ( verification !== "verified" && @@ -5406,27 +5407,17 @@ function nativeSessionConfigDigest( .digest("hex")}`; } -function hasIdleWarmNativeSessionOwner(input: { +function idleWarmNativeSessionOwnerRunId(input: { execution: NativeExecutionInput; runnerExecutionTarget?: AdapterExecutionTarget | null; -}): boolean { - if (input.execution.session.lifecyclePolicy.mode !== "warm") return false; +}): string | undefined { + if (input.execution.session.lifecyclePolicy.mode !== "warm") return undefined; const entry = warmNativeSessions.get(nativeSessionScopeKey(input.execution)); - if (!entry || (entry.busy && entry.instructionPreparationRunId !== input.execution.binding.runId)) return false; - // A verified idle owner proves the checkpoint belongs to this session even - // when its process must later rotate to a new run-scoped broker capability. - // Local environments also have an id. Compare the same environment binding - // stored at acquisition instead of treating every local target as unbound. + if (!entry || (entry.busy && entry.instructionPreparationRunId !== input.execution.binding.runId)) return undefined; const environmentId = input.runnerExecutionTarget?.environmentId ?? null; - return ( - entry.companyId === input.execution.binding.companyId && - entry.environmentId === environmentId && - entry.configDigest === - nativeSessionConfigDigest( - input.execution, - input.runnerExecutionTarget?.kind ?? "local", - ) - ); + return entry.companyId === input.execution.binding.companyId && entry.environmentId === environmentId + ? entry.runId + : undefined; } function nativeHarnessEnvironmentFingerprint( @@ -7762,7 +7753,7 @@ async function executePaperclipNativeSessionWithinScope( // suspended. Only the exact idle in-process owner may rotate that // prior-run authority; after a hard restart the map is empty and the // durable-state verifier continues to require a suspended runner. - allowRetainedWarmRunner: hasIdleWarmNativeSessionOwner(input), + retainedWarmOwnerRunId: idleWarmNativeSessionOwnerRunId(input), allowLocalRecovery: input.runnerExecutionTarget?.kind !== "remote", onLog: input.onLog, restartRecovery: input.restartRecovery, @@ -8864,8 +8855,10 @@ async function executePaperclipNativeSessionWithinScope( throw new Error("native_session_supervisor_busy"); } existing.session = session; + existing.runId = input.execution.binding.runId; } else warmNativeSessions.set(warmSessionId, { + runId: input.execution.binding.runId, instructionWorkingCopy: input.instructionWorkingCopy?.checkpointWarm ? undefined : input.instructionWorkingCopy, instructionRoot: input.instructionWorkingCopy?.root, agentId: input.execution.binding.agentId, @@ -11220,7 +11213,7 @@ export async function createRunnerdBackend(input: { db: input.db, execution: input.execution, allowVerifiedBackup: input.runnerExecutionTarget?.kind === "remote", - allowRetainedWarmRunner: false, + retainedWarmOwnerRunId: undefined, allowLocalRecovery: input.runnerExecutionTarget?.kind !== "remote", onLog: input.onLog, restartRecovery: input.restartRecovery, From e4ce989099388abd7c661445dc267183da69ef77 Mon Sep 17 00:00:00 2001 From: Leia Marshall Date: Thu, 15 Oct 2026 09:24:31 -0700 Subject: [PATCH 09/19] Add opt-in Gmail archive capability (#16331) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip separates Gmail capabilities by reviewed OAuth scope and tool policy. > - Existing Gmail connections support reading and draft creation but not bounded archiving. > - Gmail archive requires `gmail.modify`, so it must not widen existing read or draft grants. > - This change adds an opt-in archive profile and limits it to removing only the `INBOX` label. > - A final pre-dispatch check preserves that boundary for approved and replayed calls. ## Linked Issues or Issue Description **What existing behavior does this improve?** This improves the existing Gmail connection and tool-governance behavior. **Subsystem affected** Cross-cutting: `packages/shared` and `server`. **Current behavior** Paperclip can read Gmail and create drafts. It does not provide an opt-in, bounded action for archiving a thread. **Proposed behavior** Paperclip offers a separate Read & archive connection. It exposes `unlabel_thread` only for that connection and accepts only `threadId` with `labelIds: ["INBOX"]`. **Reason and benefit** Users can archive processed Gmail threads while sending, trashing, and other label changes remain disabled. **Breaking changes** None. Existing read and draft methods keep their current scopes and defaults. Related work: #15545 adds a unified Google Workspace connector and keeps Gmail draft-only. #15412 adds a separate Gmail REST server without label mutations. Neither PR adds this bounded archive capability. ## What Changed - Added the `gmail.archive` connector profile. - Added managed and customer-owned archive connection methods. - Added an Inbox-only schema and final pre-dispatch argument guard. - Classified archive as destructive and added Ask-first defaults for the archive methods. - Added permission-review records and focused tests. ## Verification - `pnpm --filter @paperclipai/server typecheck` - `pnpm --filter @paperclipai/server build` - `pnpm --filter @paperclipai/shared typecheck` - Focused Vitest suites: 42 tests passed. - App-definition ingestion: 103 definitions and 0 review flags. - `git diff --check` ## Risks The main risk is accidental scope or operation expansion. The new profile is opt-in. Existing profiles retain their scopes. The gateway validates the exact Inbox-only payload immediately before dispatch. Production rollout requires separate OAuth configuration and fresh consent. ## Model Used Unverified — reliable runtime model metadata was not available. Confirm the model attribution before publishing. ## Checklist - [x] I have included a concise design rationale - [x] I have specified that model attribution is unverified - [x] I have checked ROADMAP.md and found no Gmail archive entry - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in this PR - [x] I have not referenced internal or instance-local issues - [x] My branch name describes the change - [x] Local tests and builds pass - [x] I have added or updated tests - [x] I have updated relevant documentation and permission records - [x] I have documented risks - [ ] All hosted CI gates are green - [ ] Greptile and human review are complete --------- Co-authored-by: Leia Marshall --- .../tool-method-permission-reviews.json | 36 ++++++ packages/shared/src/app-definitions.test.ts | 17 ++- packages/shared/src/app-definitions.ts | 10 +- .../shared/src/app-definitions/gmail.json | 77 ++++++++++++ .../shared/src/google-workspace-connectors.ts | 3 +- .../src/__tests__/tool-access-service.test.ts | 34 +++++- .../gmail-archive-tool-policy.test.ts | 78 +++++++++++++ .../src/services/gmail-archive-tool-policy.ts | 110 ++++++++++++++++++ .../services/gmail-tool-governance.test.ts | 6 + server/src/services/tool-access.ts | 13 ++- server/src/services/tool-gateway.ts | 8 +- 11 files changed, 384 insertions(+), 8 deletions(-) create mode 100644 server/src/services/gmail-archive-tool-policy.test.ts create mode 100644 server/src/services/gmail-archive-tool-policy.ts diff --git a/doc/connections/tool-method-permission-reviews.json b/doc/connections/tool-method-permission-reviews.json index d83dd6a8e43..eb07d0400df 100644 --- a/doc/connections/tool-method-permission-reviews.json +++ b/doc/connections/tool-method-permission-reviews.json @@ -651,6 +651,42 @@ "reviewedAt": "2026-09-30", "liveProof": "not-run" }, + { + "app": "gmail", + "method": "paperclip-archive", + "auth": "oauth", + "policy": "explicit", + "requestedScopes": [ + "https://www.googleapis.com/auth/gmail.readonly", + "https://www.googleapis.com/auth/gmail.modify" + ], + "capability": "write", + "supportedActions": "Read Gmail and archive a thread by removing only the INBOX label.", + "evidence": [ + "https://developers.google.com/workspace/gmail/api/reference/mcp/tools_list/unlabel_thread", + "https://gmailmcp.googleapis.com/.well-known/oauth-protected-resource/mcp/v1" + ], + "reviewedAt": "2026-10-14", + "liveProof": "not-run" + }, + { + "app": "gmail", + "method": "customer-archive-oauth", + "auth": "oauth", + "policy": "explicit", + "requestedScopes": [ + "https://www.googleapis.com/auth/gmail.readonly", + "https://www.googleapis.com/auth/gmail.modify" + ], + "capability": "write", + "supportedActions": "Read Gmail and archive a thread by removing only the INBOX label.", + "evidence": [ + "https://developers.google.com/workspace/gmail/api/reference/mcp/tools_list/unlabel_thread", + "https://gmailmcp.googleapis.com/.well-known/oauth-protected-resource/mcp/v1" + ], + "reviewedAt": "2026-10-14", + "liveProof": "not-run" + }, { "app": "google-calendar", "method": "paperclip-read", diff --git a/packages/shared/src/app-definitions.test.ts b/packages/shared/src/app-definitions.test.ts index 52f73c5f0b7..bbe8d1cf8ee 100644 --- a/packages/shared/src/app-definitions.test.ts +++ b/packages/shared/src/app-definitions.test.ts @@ -51,6 +51,15 @@ const GOOGLE_WORKSPACE_PROFILE_EXPECTATIONS = [ scopes: [googleScope("gmail.readonly"), googleScope("gmail.compose")], writeTools: ["create_draft"], }, + { + profile: "gmail.archive", + appSlug: "gmail", + serverUrl: "https://gmailmcp.googleapis.com/mcp/v1", + capability: "archive", + riskTier: "S4", + scopes: [googleScope("gmail.readonly"), googleScope("gmail.modify")], + writeTools: ["unlabel_thread"], + }, { profile: "drive.read", appSlug: "google-drive", @@ -214,7 +223,7 @@ const GOOGLE_WORKSPACE_PROFILE_EXPECTATIONS = [ profile: GoogleWorkspaceConnectorProfileId; appSlug: string; serverUrl: string; - capability: "read" | "write" | "draft"; + capability: "read" | "write" | "draft" | "archive"; riskTier: "S3" | "S4"; scopes: readonly string[]; writeTools: readonly string[]; @@ -763,10 +772,10 @@ describe("AppDefinition catalog", () => { APP_DEFINITIONS.find((app) => app.slug === "linear")?.methods[0]?.defaults ?.scopesHint, ).toEqual(["read", "write"])); - it("defaults every new connection action to allowed except Enterpret writes", () => { + it("limits Ask-first defaults to reviewed high-risk connection methods", () => { for (const app of APP_DEFINITIONS) for (const method of app.methods) { - if (app.slug === "enterpret") { + if (app.slug === "enterpret" || (app.slug === "gmail" && method.capabilityProfile?.key === "archive")) { expect(recommendedDefaultsForApp(app, method.key)).toEqual({ access: "all_agents", askFirstRiskLevels: ["write", "destructive"], @@ -788,6 +797,8 @@ describe("AppDefinition catalog", () => { expect(getAvailableConnectionMethod(gmail)?.key).toBe( "customer-draft-oauth", ); + expect(gmail.methods.filter((method) => method.capabilityProfile?.key === "archive") + .map((method) => method.key)).toEqual(["paperclip-archive", "customer-archive-oauth"]); expect( getRecommendedConnectionMethod( drive.methods.filter((candidate) => diff --git a/packages/shared/src/app-definitions.ts b/packages/shared/src/app-definitions.ts index 4d7241b643c..e1f56841a94 100644 --- a/packages/shared/src/app-definitions.ts +++ b/packages/shared/src/app-definitions.ts @@ -262,7 +262,15 @@ export function recommendedDefaultsForApp(app: AppDefinition, methodKey?: string // concrete app/method. Connection finalization remains a configure-authorized, // audited operation, and Ask first stays available as an operator-selected // policy for any action after the connection is made. - void methodKey; + const method = methodKey + ? app.methods.find((candidate) => candidate.key === methodKey) + : null; + if (app.slug === "gmail" && method?.capabilityProfile?.key === "archive") { + return { + access: "all_agents", + askFirstRiskLevels: ["write", "destructive"], + }; + } // Enterpret's run_graph_query self-reports readOnlyHint but executes Cypher // and has not been established as read-only. Classify it as write (server // classifyRisk) and require Ask first so new token connections do not ship diff --git a/packages/shared/src/app-definitions/gmail.json b/packages/shared/src/app-definitions/gmail.json index a5d4b12cdeb..a0f2dcfaaec 100644 --- a/packages/shared/src/app-definitions/gmail.json +++ b/packages/shared/src/app-definitions/gmail.json @@ -177,6 +177,83 @@ "Google Workspace MCP servers are in Developer Preview." ], "riskTier": "S4" + }, + { + "key": "paperclip-archive", + "label": "Connect with Paperclip", + "transport": "mcp_remote", + "auth": "oauth", + "oauthStrategy": "paperclip_cloud_connector", + "connectorProfile": "gmail.archive", + "capabilityProfile": { + "key": "archive", + "label": "Read & archive", + "description": "Read Gmail and archive threads by removing only the Inbox label." + }, + "grantKinds": [ + "user", + "organization" + ], + "ownershipModes": [ + "platform_shared" + ], + "whenToUse": "Use Paperclip-managed Google OAuth to read Gmail and archive threads.", + "defaults": { + "serverUrl": "https://gmailmcp.googleapis.com/mcp/v1", + "scopesHint": [ + "https://www.googleapis.com/auth/gmail.readonly", + "https://www.googleapis.com/auth/gmail.modify" + ] + }, + "guidanceMd": "Connect Gmail to search and read mail and archive threads. Archive removes only the Inbox label; sending, trashing, and other label changes remain disabled.", + "warnings": [ + "Archiving requires approval and only removes the Inbox label.", + "Google Workspace MCP servers are in Developer Preview." + ], + "riskTier": "S4" + }, + { + "key": "customer-archive-oauth", + "label": "Use your own Google OAuth app", + "transport": "mcp_remote", + "auth": "oauth", + "capabilityProfile": { + "key": "archive", + "label": "Read & archive", + "description": "Read Gmail and archive threads by removing only the Inbox label." + }, + "grantKinds": [ + "user", + "organization" + ], + "ownershipModes": [ + "customer" + ], + "whenToUse": "Use a customer-owned Google OAuth client to read Gmail and archive threads.", + "defaults": { + "serverUrl": "https://gmailmcp.googleapis.com/mcp/v1", + "authorizationEndpoint": "https://accounts.google.com/o/oauth2/v2/auth", + "tokenEndpoint": "https://oauth2.googleapis.com/token", + "metadataUrl": "https://accounts.google.com/.well-known/openid-configuration", + "scopesHint": [ + "https://www.googleapis.com/auth/gmail.readonly", + "https://www.googleapis.com/auth/gmail.modify" + ], + "oauthAuthorizationParams": { + "access_type": "offline", + "prompt": "consent" + } + }, + "guidanceMd": "Enable the Gmail and Gmail MCP APIs in your Google Cloud project, then register Paperclip's callback URI. Archive removes only the Inbox label.", + "consoleLinks": { + "register": "https://console.cloud.google.com/auth/clients", + "docs": "https://developers.google.com/workspace/guides/configure-mcp-servers" + }, + "warnings": [ + "Archiving requires approval and only removes the Inbox label.", + "Google Workspace MCP servers are in Developer Preview." + ], + "riskTier": "S4" } ] } diff --git a/packages/shared/src/google-workspace-connectors.ts b/packages/shared/src/google-workspace-connectors.ts index 5800309de16..601f10314c4 100644 --- a/packages/shared/src/google-workspace-connectors.ts +++ b/packages/shared/src/google-workspace-connectors.ts @@ -1,5 +1,5 @@ export const GOOGLE_WORKSPACE_CONNECTOR_PROFILE_IDS = [ - "gmail.read", "gmail.draft", "drive.read", "drive.write", "docs.read", "docs.write", + "gmail.read", "gmail.draft", "gmail.archive", "drive.read", "drive.write", "docs.read", "docs.write", "sheets.read", "sheets.write", "slides.read", "slides.write", "calendar.read", "calendar.write", "chat.read", "chat.write", "people.read", "workspace-search.read", ] as const; @@ -16,6 +16,7 @@ export const GOOGLE_WORKSPACE_CONNECTOR_PROFILES: Readonly> = { "gmail.read": def("gmail", "https://gmailmcp.googleapis.com/mcp/v1", [auth("gmail.readonly")]), "gmail.draft": def("gmail", "https://gmailmcp.googleapis.com/mcp/v1", [auth("gmail.readonly"), auth("gmail.compose")], ["create_draft"]), + "gmail.archive": def("gmail", "https://gmailmcp.googleapis.com/mcp/v1", [auth("gmail.readonly"), auth("gmail.modify")], ["unlabel_thread"]), "drive.read": def("google-drive", "https://drivemcp.googleapis.com/mcp/v1", [auth("drive.readonly")]), "drive.write": def("google-drive", "https://drivemcp.googleapis.com/mcp/v1", [auth("drive.readonly"), auth("drive.file")], ["copy_file", "create_file"]), "docs.read": def("google-docs", "https://docsmcp.googleapis.com/mcp/v1", [auth("documents.readonly")]), diff --git a/server/src/__tests__/tool-access-service.test.ts b/server/src/__tests__/tool-access-service.test.ts index 5ea8b6ebba4..8b39407897a 100644 --- a/server/src/__tests__/tool-access-service.test.ts +++ b/server/src/__tests__/tool-access-service.test.ts @@ -5363,7 +5363,11 @@ describeEmbeddedPostgres("tool access service", () => { ).toBe(false); expect( nonPilotGmail.methods.map((method: { key: string }) => method.key), - ).toEqual(["customer-read-oauth", "customer-draft-oauth"]); + ).toEqual([ + "customer-read-oauth", + "customer-draft-oauth", + "customer-archive-oauth", + ]); const pilot = await request( createRouteApp( @@ -5384,6 +5388,34 @@ describeEmbeddedPostgres("tool access service", () => { "paperclip-read", "customer-read-oauth", "customer-draft-oauth", + "customer-archive-oauth", + ]); + + const archivePilotConnector = fakeGoogleWorkspaceConnector( + company.id, + userId, + "gmail.archive", + ); + const archivePilot = await request( + createRouteApp( + db, + boardSessionActor(company.id, "owner", userId), + undefined, + { paperclipCloudConnector: archivePilotConnector }, + ), + ).get(`/api/companies/${company.id}/tools/gallery`); + expect(archivePilot.status).toBe(200); + const archivePilotGmail = archivePilot.body.apps.find( + (app: { slug: string }) => app.slug === "gmail", + ); + expect(archivePilotGmail.ownershipAvailability.platform_shared).toBe(true); + expect( + archivePilotGmail.methods.map((method: { key: string }) => method.key), + ).toEqual([ + "customer-read-oauth", + "customer-draft-oauth", + "paperclip-archive", + "customer-archive-oauth", ]); }); diff --git a/server/src/services/gmail-archive-tool-policy.test.ts b/server/src/services/gmail-archive-tool-policy.test.ts new file mode 100644 index 00000000000..78474f44d3f --- /dev/null +++ b/server/src/services/gmail-archive-tool-policy.test.ts @@ -0,0 +1,78 @@ +import { describe, expect, it } from "vitest"; + +import { + assertGmailArchiveToolArgumentsSupported, + gmailArchiveToolDescription, + gmailArchiveToolInputSchema, +} from "./gmail-archive-tool-policy.js"; + +const archive = { config: { + sourceTemplateKey: "gmail", + connectionMethodKey: "paperclip-archive", + oauth: { connectorProfile: "gmail.archive" }, +} }; + +describe("Gmail archive tool policy", () => { + it("accepts only removal of the Inbox label from one thread", () => { + expect(() => assertGmailArchiveToolArgumentsSupported(archive, "unlabel_thread", { + threadId: "thread-123", + labelIds: ["INBOX"], + })).not.toThrow(); + + for (const parameters of [ + { threadId: "thread-123", labelIds: ["STARRED"] }, + { threadId: "thread-123", labelIds: ["INBOX", "UNREAD"] }, + { threadId: "thread-123", label_ids: ["INBOX"] }, + { threadId: "", labelIds: ["INBOX"] }, + { threadId: "thread-123", labelIds: ["INBOX"], extra: true }, + "encoded", + ]) { + expect(() => assertGmailArchiveToolArgumentsSupported(archive, "unlabel_thread", parameters)) + .toThrow(expect.objectContaining({ + status: 400, + details: { code: "gmail_archive_arguments_unsupported" }, + })); + } + }); + + it("requires an explicit archive connection even for the official Gmail endpoint", () => { + for (const config of [ + { sourceTemplateKey: "gmail", connectionMethodKey: "paperclip-read", oauth: { connectorProfile: "gmail.read" } }, + { sourceTemplateKey: "gmail", connectionMethodKey: "paperclip-draft", oauth: { connectorProfile: "gmail.draft" } }, + { url: "https://gmailmcp.googleapis.com/mcp/v1" }, + ]) { + expect(() => assertGmailArchiveToolArgumentsSupported({ config }, "unlabel_thread", { + threadId: "thread-123", labelIds: ["INBOX"], + })).toThrow(expect.objectContaining({ + status: 400, + details: { code: "gmail_archive_profile_required" }, + })); + } + }); + + it("publishes an Inbox-only schema without mutating the provider schema", () => { + const providerSchema = { type: "object", properties: { + threadId: { type: "string" }, + labelIds: { type: "array", items: { type: "string" } }, + } }; + const projected = gmailArchiveToolInputSchema(archive, "gmail/unlabel_thread", providerSchema); + expect(projected).toMatchObject({ + properties: { labelIds: { items: { enum: ["INBOX"] }, minItems: 1, maxItems: 1 } }, + required: ["threadId", "labelIds"], + additionalProperties: false, + }); + expect(providerSchema.properties.labelIds.items).toEqual({ type: "string" }); + }); + + it("describes archive narrowly and leaves unrelated tools unchanged", () => { + expect(gmailArchiveToolDescription(archive, "unlabel_thread", "Remove labels.")) + .toContain("removing only its Inbox label"); + expect(gmailArchiveToolDescription(archive, "search_threads", "Search mail.")) + .toBe("Search mail."); + expect(() => assertGmailArchiveToolArgumentsSupported( + { config: { sourceTemplateKey: "other" } }, + "unlabel_thread", + { anything: true }, + )).not.toThrow(); + }); +}); diff --git a/server/src/services/gmail-archive-tool-policy.ts b/server/src/services/gmail-archive-tool-policy.ts new file mode 100644 index 00000000000..1bc9462de92 --- /dev/null +++ b/server/src/services/gmail-archive-tool-policy.ts @@ -0,0 +1,110 @@ +import { badRequest } from "../errors.js"; + +type Connection = { config: Record }; + +function record(value: unknown): Record { + return value !== null && typeof value === "object" && !Array.isArray(value) + ? value as Record + : {}; +} + +function leafName(name: string): string { + return (name.split(/[.:/]/).pop() ?? name) + .replace(/([a-z0-9])([A-Z])/g, "$1-$2") + .replace(/_/g, "-") + .toLowerCase(); +} + +function isOfficialGmailConnection(connection: Connection): boolean { + const config = connection.config; + const profile = record(config.oauth).connectorProfile; + if ( + config.sourceTemplateKey === "gmail" + || profile === "gmail.read" + || profile === "gmail.draft" + || profile === "gmail.archive" + ) return true; + try { + return new URL(String(config.url)).hostname === "gmailmcp.googleapis.com"; + } catch { + return false; + } +} + +function isArchiveConnection(connection: Connection): boolean { + const config = connection.config; + return record(config.oauth).connectorProfile === "gmail.archive" + || (config.sourceTemplateKey === "gmail" + && (config.connectionMethodKey === "paperclip-archive" + || config.connectionMethodKey === "customer-archive-oauth")); +} + +function isUnlabelThread(connection: Connection, toolName: string): boolean { + return isOfficialGmailConnection(connection) && leafName(toolName) === "unlabel-thread"; +} + +export function assertGmailArchiveToolArgumentsSupported( + connection: Connection, + toolName: string, + parameters: unknown, +): void { + if (!isUnlabelThread(connection, toolName)) return; + if (!isArchiveConnection(connection)) { + throw badRequest("Gmail archiving requires an explicit Read & archive connection.", { + code: "gmail_archive_profile_required", + }); + } + const supplied = record(parameters); + const keys = Object.keys(supplied).sort(); + if ( + keys.length !== 2 + || keys[0] !== "labelIds" + || keys[1] !== "threadId" + || typeof supplied.threadId !== "string" + || supplied.threadId.trim().length === 0 + || !Array.isArray(supplied.labelIds) + || supplied.labelIds.length !== 1 + || supplied.labelIds[0] !== "INBOX" + ) { + throw badRequest("Gmail archive accepts only threadId and labelIds: [\"INBOX\"].", { + code: "gmail_archive_arguments_unsupported", + }); + } +} + +export function gmailArchiveToolInputSchema( + connection: Connection, + toolName: string, + schema: Record, +): Record { + if (!isUnlabelThread(connection, toolName) || !isArchiveConnection(connection)) return schema; + return { + type: "object", + properties: { + threadId: { + type: "string", + minLength: 1, + description: "The Gmail thread ID to archive.", + }, + labelIds: { + type: "array", + items: { type: "string", enum: ["INBOX"] }, + minItems: 1, + maxItems: 1, + description: "Must be exactly [\"INBOX\"].", + }, + }, + required: ["threadId", "labelIds"], + additionalProperties: false, + }; +} + +export function gmailArchiveToolDescription( + connection: Connection, + toolName: string, + description: string | null, +): string | null { + return isUnlabelThread(connection, toolName) && isArchiveConnection(connection) + ? "Archive a Gmail thread by removing only its Inbox label. Requires human approval." + : description; +} diff --git a/server/src/services/gmail-tool-governance.test.ts b/server/src/services/gmail-tool-governance.test.ts index 526f7b5b516..9d0b0b27396 100644 --- a/server/src/services/gmail-tool-governance.test.ts +++ b/server/src/services/gmail-tool-governance.test.ts @@ -12,6 +12,8 @@ describe("Gmail tool governance", () => { expect(isGmailToolPermanentlyBlocked({ name: "get_message" })).toBe(false); expect(classifyRisk({ name: "create_draft" }, "gmail")).toBe("write"); expect(isGmailToolPermanentlyBlocked({ name: "create_draft" })).toBe(false); + expect(classifyRisk({ name: "unlabel_thread" }, "gmail")).toBe("destructive"); + expect(isGmailToolPermanentlyBlocked({ name: "unlabel_thread" })).toBe(false); expect(isGmailToolPermanentlyBlocked({ name: "send_message" })).toBe(true); expect(isGmailToolPermanentlyBlocked({ name: "trash_thread" })).toBe(true); @@ -28,6 +30,10 @@ describe("Google Workspace tool governance", () => { expect(isGoogleWorkspaceToolAllowed("gmail.draft", { name: "create_draft" })).toBe(true); expect(isGoogleWorkspaceToolAllowed("gmail.draft", { name: "send_message" })).toBe(false); + expect(isGoogleWorkspaceToolAllowed("gmail.archive", { name: "search_threads" })).toBe(true); + expect(isGoogleWorkspaceToolAllowed("gmail.archive", { name: "unlabel_thread" })).toBe(true); + expect(isGoogleWorkspaceToolAllowed("gmail.archive", { name: "label_thread" })).toBe(false); + expect(isGoogleWorkspaceToolAllowed("gmail.archive", { name: "send_message" })).toBe(false); expect(isGoogleWorkspaceToolAllowed("calendar.write", { name: "delete_event" })).toBe(true); expect(isGoogleWorkspaceToolAllowed("calendar.write", { name: "publish_calendar" })).toBe(false); }); diff --git a/server/src/services/tool-access.ts b/server/src/services/tool-access.ts index 7a3d42eed73..f37c81032fe 100644 --- a/server/src/services/tool-access.ts +++ b/server/src/services/tool-access.ts @@ -253,6 +253,7 @@ import { profileIdsInBindingOrder, } from "./tool-profile-binding-precedence.js"; import { assertGoogleChatToolArgumentsSupported, googleChatToolDescription, googleChatToolInputSchema } from "./google-chat-tool-policy.js"; +import { gmailArchiveToolDescription, gmailArchiveToolInputSchema } from "./gmail-archive-tool-policy.js"; import { recordToolRuntimeAuditWriteFailure, TOOL_RUNTIME_AUDIT_WRITE_FAILURE_METRIC, @@ -1343,6 +1344,7 @@ export function projectedConnectionToolInputSchema( const managed = honchoManagedArguments(connection, inputSchema); if (managed) inputSchema = stripManagedToolArgumentSchema(inputSchema, managed); inputSchema = googleChatToolInputSchema(connection, toolName, inputSchema); + inputSchema = gmailArchiveToolInputSchema(connection, toolName, inputSchema); const sourceTemplateKey = typeof connection.config.sourceTemplateKey === "string" ? connection.config.sourceTemplateKey @@ -1737,7 +1739,11 @@ function toCatalogEntryForConnection( rawCatalogEntry.inputSchema ?? {}, row.toolName, ), - description: googleChatToolDescription(connection, row.toolName, rawCatalogEntry.description), + description: gmailArchiveToolDescription( + connection, + row.toolName, + googleChatToolDescription(connection, row.toolName, rawCatalogEntry.description), + ), }; if ( connection.transport === "local_stdio" && @@ -2409,6 +2415,10 @@ export function classifyRisk( // safe. Treat it as write so Ask-first defaults can restrict it. if (sourceTemplateKey === "enterpret" && normalizedToolName === "run-graph-query") return "write"; + // Gmail archive removes a system label. Keep it behind the archive method's + // destructive Ask-first default even when the preview provider omits hints. + if (sourceTemplateKey === "gmail" && normalizedToolName === "unlabel-thread") + return "destructive"; if (sourceTemplateKey === "posthog" && normalizedToolName === "exec") return "destructive"; // Superagent mirrors its REST API, and several mutations use verbs the @@ -2460,6 +2470,7 @@ export function classifyRisk( export function isGmailToolPermanentlyBlocked( tool: McpToolDescriptor, ): boolean { + if (googleWorkspaceToolLeafName(tool.name) === "unlabel-thread") return false; const riskLevel = classifyRisk(tool, "gmail"); return ( verbMatches( diff --git a/server/src/services/tool-gateway.ts b/server/src/services/tool-gateway.ts index f7fe7f1617b..3c28f51db62 100644 --- a/server/src/services/tool-gateway.ts +++ b/server/src/services/tool-gateway.ts @@ -124,6 +124,7 @@ import { projectedConnectionToolInputSchema, } from "./tool-access.js"; import { assertGoogleChatToolArgumentsSupported, googleChatToolDescription } from "./google-chat-tool-policy.js"; +import { assertGmailArchiveToolArgumentsSupported, gmailArchiveToolDescription } from "./gmail-archive-tool-policy.js"; import { parseRemoteHttpEndpoint } from "./remote-http-endpoint-guard.js"; import { guardedRemoteHttpFetch, @@ -1470,7 +1471,11 @@ export function createToolGatewayService( name: gatewayToolName, displayName: catalogEntry.title ?? catalogEntry.toolName, description: - googleChatToolDescription(connection, catalogEntry.toolName, catalogEntry.description) ?? + gmailArchiveToolDescription( + connection, + catalogEntry.toolName, + googleChatToolDescription(connection, catalogEntry.toolName, catalogEntry.description), + ) ?? `Connected MCP tool ${catalogEntry.toolName} from ${connection.name}.`, parametersSchema: inputSchema, pluginId: `mcp:${applicationKey ?? application.id}`, @@ -5972,6 +5977,7 @@ export function createToolGatewayService( // Recheck immediately before dispatch, including previously approved calls // and connections whose stored grant/catalog predates scope reduction. assertGoogleChatToolArgumentsSupported(connection, entry.toolName, parameters); + assertGmailArchiveToolArgumentsSupported(connection, entry.toolName, parameters); if (useDefaultTimeout && isRailwayConnection(connection) && entry.toolName === `${RAILWAY_TOOL_PREFIX}run-command`) { ms = railwayCommandBudgetMs(parameters); } From 60e03c002da9451326fdf6abc5d8846a8eb30328 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Thu, 15 Oct 2026 11:32:46 -0500 Subject: [PATCH 10/19] fix(grok): hold new runs while connection needs recovery (#16445) ## Thinking Path > - Paperclip manages agents and their work. > - Grok Bot assignments use a managed Runner and a verified connection. > - Stop can leave the connection held for remote-work inspection. > - The queue started new runs before it checked that connection. > - Those runs allocated Runners, failed, and caused more preparation attempts. > - This pull request checks readiness at queue admission. > - Pending input waits for recovery without using managed compute. ## Linked Issues or Issue Description Refs #16427 and #16422. The open Grok changes cover the coding adapter and do not duplicate this fix. **What happened?** A real Stop test held the Grok connection for inspection. A second task was pending. Four new preparation attempts acquired managed Runners, then failed because the connection was held. No new external assignment or webhook was sent. Pause stopped the loop, and all leases completed cleanup. **Expected behavior** Keep fresh input in the durable queue until its connection is ready. Do not allocate a Runner to discover a known connection hold. Keep existing session reconciliation under its recorded authority. **Steps to reproduce** 1. Connect Grok Bot and start an assignment. 2. Queue another task for that agent. 3. Stop the first assignment and leave the inspection hold in place. 4. Run queue recovery. The pending task repeatedly starts preparation and fails. **Paperclip version or commit** Observed on packaged Core 5a9e6b631563dcd7762844fc0f544e0112384d90 in Cloud staging. This branch includes current master afe5b3418dbdf48aca5791f8fc8df12c7a75a246. ## What Changed - Reuse the broker's current connection and authority checks before fresh Grok run claims. - Let valid connection-pool routing select a native provider before applying any unused Grok connection hold. - Leave pending input queued while readiness is denied. Recheck at the claim boundary if Stop races the queue scan. - Preserve already-started reconciliation and all later runtime-selection and dispatch checks. - Propagate database failures. Treat only explicit connection-authority errors as a hold. - Add queue, OAuth, broker and assembled heartbeat regressions. - Update the Grok guide and existing release evidence plan. ## Verification - Two queue admission assertions fail before the correction. - Six focused suites pass all 813 checks, including current master's native ownership coverage and a real healthy Codex pool member with a held Grok connection. The routing test reaches native runtime selection and then injects a controlled stop before provider start. - The assembled test uses real OAuth and broker state, repeats public queue recovery, and checks zero leases, assignments and extra webhooks. - Repository typecheck and build pass on final head 83664a91ce276c28346f3bcd0c4a732c13d03808. Its full local suite is still running. The earlier full attempt was stopped as superseded after the valid router review. - The full suite uses a clean tool PATH. On the earlier baseline, mocked Gemini tests reached the installed host CLI and exceeded their five-second deadline. Both tests pass with that host CLI absent. Test source and deadlines are unchanged. - The existing native cleanup and queue claim-isolation tests pass. The process-agent Stop browser journey passes locally. Its first CI attempt queried the queue 67 ms after comment creation, before asynchronous publication. The test now polls for the same durable entry with its existing deadline. - One server CI worker shut down after 1,047 tests passed. The failed attempt remains recorded. No routing flags or gates were changed. No UI component or schema changes. - The final head passes 55 CI checks, with two optional skips. Greptile reviewed that exact head at 5/5, with the earlier thread resolved and no new actionable finding. - Actual packaged recovery admission still needs a live repeat. The provider currently rejects routine starts with an external rate-limit error. This patch does not remove that provider limit. ## Risks Readiness can change after preflight. Runtime selection and dispatch retain their current authority checks. Already-started runs bypass this fresh-work check so they can reconcile their original session. Other providers keep their current queue behavior. Expected connection denials hold input; unexpected database errors still propagate. No migration is needed. ## Model Used OpenAI GPT-6 in Codex, with reasoning, repository editing, shell tools and computer use. The exact serving model ID, snapshot and context window are not exposed by this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/grok-bot-agent.md | 7 +++ ...026-10-13-grok-bot-production-readiness.md | 26 +++++++- .../__tests__/ai-connection-router.test.ts | 58 ++++++++++++++++-- server/src/__tests__/grok-bot-routine.test.ts | 59 +++++++++++++++++++ server/src/services/dot-runner-broker.ts | 9 +++ server/src/services/heartbeat.ts | 8 +++ server/src/services/heartbeat/queue.test.ts | 51 ++++++++++++++++ server/src/services/heartbeat/queue.ts | 4 ++ tests/e2e/composer-stop.spec.ts | 12 ++-- 9 files changed, 220 insertions(+), 14 deletions(-) diff --git a/doc/grok-bot-agent.md b/doc/grok-bot-agent.md index ffd37093d98..0da45937194 100644 --- a/doc/grok-bot-agent.md +++ b/doc/grok-bot-agent.md @@ -76,6 +76,13 @@ outcome needs inspection and explicit recovery; it is never sent again automatic Saved work remains on the task if execution fails. Reverify after clearing an inspected recovery hold before admitting another assignment. +New assignments remain in the durable queue while the Grok connection is +unverified, revoked, or held for recovery. Readiness is checked before claiming +a new run or allocating its Runner. Already-admitted session reconciliation +keeps its recorded authority; it does not gain permission to start new work. +A configured AI connection pool selects its own provider. Its native tasks use +the pool's normal admission checks and do not require an unused Grok connection. + **Stop Paperclip work** immediately revokes the session's Paperclip authority and stops future dispatch. Remote Grok execution may continue. Open the Grok Bot app to inspect it. Paperclip shows reported activity, not a complete remote transcript diff --git a/doc/plans/2026-10-13-grok-bot-production-readiness.md b/doc/plans/2026-10-13-grok-bot-production-readiness.md index 438cfca7607..dec52786c75 100644 --- a/doc/plans/2026-10-13-grok-bot-production-readiness.md +++ b/doc/plans/2026-10-13-grok-bot-production-readiness.md @@ -1,6 +1,6 @@ # Grok Bot production readiness -Status as of October 15: real assigned and Bot-initiated tasks reach canonical completion. The active-session wake correction is merged. The proactive admission echo correction is merged and source-verified; image publication and a real no-echo repeat remain pending. A real controller restart and active Stop diagnostic passed their authority checks; Stop also exposed a false workspace-repair hold, now under correction. Production acceptance is incomplete. The current release goal and matrix below supersede the earlier dated campaign entries. Historical successes and failures remain evidence for the revisions actually tested. +Status as of October 15: a real Bot-initiated repeat passed with one task, one run, seven completed operations, canonical Done, no assignment webhook and successful cleanup on packaged Core ea4aa185. The false file-recovery correction is merged and deployed as Core 5a9e6b63; historical inspection archived the incorrect obligation and fresh verification passed. Further assigned work is blocked by a provider-reported routine-start rate limit. Stop then exposed a separate queue-admission defect: pending work repeatedly allocated a Runner before discovering the connection hold. The qualification agent is paused and all affected leases are cleaned up. Production acceptance remains incomplete. ## Current production finish line — October 14 @@ -12,8 +12,8 @@ There is no testing call, case-count, token or spending cap. The former six-case | --- | --- | --- | | Ordinary onboarding works | Actual Grok UI, fixed agent consent, saved setup, wrong-key behavior, authenticated challenge, refresh, reconnect and revocation | Resumed real Grok authorization, saved setup Done, authenticated routine challenge and automatic refresh passed on the October 15 serving revision. Runtime reconnect copy fix is merged; verify its published UI and fresh production setup next. | | Assigned work reaches a usable result | One actual managed workflow: progress, question and answer, document, downloaded file/hash, canonical completion or review and cleanup | A preliminary comparison task on serving Core ff0 completed progress → question → answer → report/CSV → canonical Done, with both cleanup receipts. The cited CSV downloads with the exact publication hash. Two distinct CSV uploads accompanied an active-session result wake; correct that wake behavior and requalify the final candidate. | -| Bot initiates governed work | Real-client task creation/claim/completion with stable receipts and no outbound echo or duplicate effects | A real idle-conversation request created one task and one managed run, saved a 100-word report, reached accepted Done and cleaned up its lease. A delayed-admission wake and two acceptance attempts were also recorded. Functional completion passed; no-echo acceptance failed. Requalify the correction. | -| Recovery and authority are reliable | Healthy work past the prior five-minute failure window; actual controller restart preserving assignment/session/process and no duplicate wake; Stop, pause, reassignment, revocation, stale calls, queues and cleanup | A real managed diagnostic ran for 13 minutes 44 seconds, survived an actual same-image controller restart with the original assignment/session/process/lease, completed canonically and cleaned up without another wake. Active Stop fenced its work, rejected a real stale progress update, kept its checkpoint unchanged and cleaned up. Cancellation also created a false file-repair hold despite an admitted workspace-access-none contract; correct and requalify that recovery behavior. These diagnostics ran on Core 11d, not the final release image. | +| Bot initiates governed work | Real-client task creation/claim/completion with stable receipts and no outbound echo or duplicate effects | A real idle-conversation repeat on packaged Core ea4aa185 created one task and one run, read a public source marker with Grok's own tools, saved its report, reached canonical Done and cleaned up. One acceptance and seven operations completed, with no assignment webhook. The earlier duplicate-wake failure remains historical evidence; its correction passed this repeat. | +| Recovery and authority are reliable | Healthy work past the prior five-minute failure window; actual controller restart preserving assignment/session/process and no duplicate wake; Stop, pause, reassignment, revocation, stale calls, queues and cleanup | The Core 11d diagnostic survived an actual restart and 13 minutes 44 seconds, then completed and cleaned up. Active Stop denied a real stale mutation. Packaged Core 5a9e6b63 later archived the historical false file hold and stopped an unaccepted session without another file obligation. Queued preparation then churned behind the connection recovery hold; correct that admission gate and repeat the final workflow, restart and active Stop. The original October 14 exit cause remains unproven. | | One identifiable reviewed candidate passes source gates | Focused protocol/auth/Runner/UI tests, full CI/typecheck/tests/build/token/Storybook evidence and exact artifacts | Active-session wake correction #16394 merged with 55 passing checks, two optional skips and review 5/5. The proactive echo correction started from master 7fc2dc8b3236e1149ae32756b6153a503623404b and was rebased onto freshly fetched master 250429942e27d568f20d3bc7dddcfc84a666480a before submission. Two regression cases reproduce the real delayed-admission wake before the fix. All 214 focused protocol/auth/question checks pass, including 80 Grok and Dot checks. Repository typecheck/build, token gates, the 2,731-story structure and Storybook build pass. Fresh CI, review, publication and the real no-echo repeat remain required. | | Production opt-in is delivered and smoke-tested | Staging acceptance, correct served Core/distribution/Runner, normal editable opt-in controls/defaults, actual production Bot workflow and inspectable task/output links | Cloud opt-in policy and release-guide corrections are merged. Staging Cloud provenance and minimal feature values were verified, with Dot disabled and defaults unchanged. The corrected wake candidate, remaining real-client cases, ordinary production opt-in controls and production smoke are outstanding. | @@ -41,6 +41,26 @@ The active Stop diagnostic cancelled its run, fenced the assignment, kept the ch Keep native providers and person-purpose MCP connections unchanged. No browser automation belongs in normal operation. Defer localhost relays, internal Grok APIs, marketplace distribution and Paperclip-connected third-party tools. Completion of a Paperclip work session does not assert remote Grok termination. +## October 15 — queued admission during Grok recovery + +A real historical inspection on packaged Core `5a9e6b63` removed the false file +obligation while preserving the old assignment fence and no-replay evidence. +The routine returned an authenticated verification after one explicit retry of a +confirmed rejection. A new task message then produced a new offered assignment. +Grok rejected its webhook and its explicit retry with HTTP 400. A separate +revoked-binding diagnostic identified `Automation run rate limit exceeded`. +Included usage was still available; no documented cooldown or routine-start +allowance was visible. No billing setting or provider limit was changed. + +Stopping that unaccepted session created no false file obligation. Pending work +then made four failed preparation attempts with no new external assignment or +webhook. Pause stopped the churn, and every managed lease reports successful +cleanup. The admission correction checks the existing broker's readiness before +claiming fresh queued work, preserves the input for later recovery, and leaves +already-started reconciliation under its recorded execution contract. Current +runtime-selection and dispatch authority checks stay in force. The patch still +requires review, packaged deployment and a real recovery repeat. + ## Candidate and release contract The production candidate started at `origin/master` commit `833769ab557255b11dd8f650f43ed7333653f5e4` and incorporates master through `e0f30a4c71` in the managed `codex/grokbot-production` worktree. The complete earlier prototype is preserved in snapshot `e0a8150ca248c1b80c58a7e26c10ccda9951907b`; its October 2 test results are historical, not evidence for this candidate. Reuse its provider-specific work where appropriate. Do not import its old gateway baseline, migration numbers, independent session coordinator, or retired UI entry point wholesale. diff --git a/server/src/__tests__/ai-connection-router.test.ts b/server/src/__tests__/ai-connection-router.test.ts index 71186aba2cf..209bcb33df4 100644 --- a/server/src/__tests__/ai-connection-router.test.ts +++ b/server/src/__tests__/ai-connection-router.test.ts @@ -1,11 +1,11 @@ import { enableRunFailureReporting } from "./helpers/run-failure-reporting.js"; import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; import { randomUUID } from "node:crypto"; -import { mkdtemp, rm } from "node:fs/promises"; -import os from "node:os"; +import { mkdir } from "node:fs/promises"; +import { createIsolatedTestHome } from "./helpers/isolated-test-home.js"; import path from "node:path"; import { and, eq } from "drizzle-orm"; -import { createDb, closeRegisteredClients, companies, agents, companyMemberships, connectionGrants, plugins, aiConnectionRouterCursors, aiConnectionTaskPins, toolConnections, companySecrets, heartbeatRuns, issues } from "@paperclipai/db"; +import { createDb, closeRegisteredClients, companies, agents, companyMemberships, connectionGrants, plugins, aiConnectionRouterCursors, aiConnectionTaskPins, toolConnections, companySecrets, heartbeatRuns, issues, dotAgentBindings, grokBotRoutines } from "@paperclipai/db"; import { startEmbeddedPostgresTestDatabase } from "@paperclipai/db/test-embedded-postgres"; import type { AiConnectionPoolMember, AiConnectionRouterRequest, AiConnectionRouterResult, PaperclipPluginManifestV1 } from "@paperclipai/shared"; import { aiConnectionRouterService, AiConnectionPoolExhausted, poolMemberRuntimeConfig, applyAiConnectionRouterTaskSettings } from "../services/ai-connection-router.js"; @@ -26,6 +26,7 @@ import { connectionIntentService } from "../services/connection-intents.js"; import { legacyExecutionNeedsReconciliation } from "../services/legacy-execution-recovery.js"; import { getServerAdapter, registerServerAdapter, unregisterServerAdapter } from "../adapters/index.js"; import { waitForPendingRunFailureReports } from "../services/run-failure-report.js"; +import { dotRunnerBroker } from "../services/dot-runner-broker.js"; const captureRunFailure = vi.hoisted(() => vi.fn()); vi.mock("../sentry.js", async (importOriginal) => ({ @@ -37,6 +38,7 @@ let database: Awaited>; const externalTestDatabaseUrl = process.env.PAPERCLIP_TEST_DATABASE_URL?.trim(); let db: ReturnType; let home: string; +let isolatedHome: ReturnType | undefined; const companyId = randomUUID(), otherCompanyId = randomUUID(), agentId = randomUUID(), secondAgentId = randomUUID(), pluginId = randomUUID(); const pluginKey = "fixture.ai-router"; const manifest: PaperclipPluginManifestV1 = { id: pluginKey, apiVersion: 1, version: "0.1.0", displayName: "Fixture router", description: "Fixture", author: "Tests", categories: ["connector"], capabilities: ["ai.connections.route"], aiConnectionRouter: { name: "AI connection pool", description: "Use saved connections" }, entrypoints: { worker: "worker.js" } }; @@ -53,7 +55,8 @@ const service = () => aiConnectionRouterService(db, worker); const resolve = (poolId: string, taskKey: string, extra = {}) => service().resolve({ companyId, poolId, agentId, userId: "alice", adapterType: "paperclip_runner", taskKey, ...extra }); const makePool = async (config = {}) => service().save(pluginKey, { companyId, config: { name: randomUUID(), enabled: true, mode: "round_robin", thresholdPercent: 90, members, ...config } }, "alice"); beforeAll(async () => { - home = await mkdtemp(path.join(os.tmpdir(), "paperclip-router-tests-")); + isolatedHome = createIsolatedTestHome(); + home = isolatedHome.home; vi.stubEnv("PAPERCLIP_HOME", home); vi.stubEnv("PAPERCLIP_INSTANCE_ID", "router-fixture"); if (externalTestDatabaseUrl) db = createDb(externalTestDatabaseUrl); else { database = await startEmbeddedPostgresTestDatabase("paperclip-router-db-"); db = createDb(database.connectionString); } @@ -73,7 +76,7 @@ afterAll(async () => { await waitForPendingRunFailureReports(); if (externalTestDatabaseUrl) await closeRegisteredClients(externalTestDatabaseUrl); await db?.$client.end(); - await database?.cleanup(); vi.unstubAllEnvs(); if (home) await rm(home, { recursive: true, force: true }); + await database?.cleanup(); vi.unstubAllEnvs(); isolatedHome?.cleanup(); }); describe("durable, authorized connection routing", () => { @@ -435,6 +438,51 @@ describe("durable, authorized connection routing", () => { expect(resolveTaskSessionConfigFreshness({ hasTaskSession: true, configuredModel: String(selected.runtimeConfig.model), taskSessionParams: { __paperclipConfiguredModel: selected.runtimeConfig.model, __paperclipConfigFingerprint: previous.fingerprint, __paperclipConfigFingerprintVersion: previous.version, __paperclipConfigCategoryFingerprints: previous.categoryFingerprints }, configMetadata: current, compatibleConfigMetadata }).reset).toBe(false); } finally { await before.cleanup(); await after.cleanup(); } }); + it("admits a healthy pool member when the saved Grok connection is held", async () => { + const original = await instanceSettingsService(db).getExperimental(); + await instanceSettingsService(db).updateExperimental({ enableAiConnectionRouters: true, + enableNativeRunner: true, enableGrokBot: true, enablePublicMcp: true, enableWorktreeRunExecution: true }); + const pool = await makePool({ members: [members[0]!] }); + const bindingId = randomUUID(); + const workspace = path.join(home, "routed-grok-work"); + await mkdir(workspace); + const [agent] = await db.insert(agents).values({ companyId, name: "Routed Grok fixture", status: "idle", + adapterType: "paperclip_runner", adapterConfig: { provider: "grok_bot", externalBindingId: bindingId, cwd: workspace }, + runtimeConfig: { aiConnection: { mode: "router", connectionId: pool.id }, heartbeat: { enabled: false, wakeOnDemand: true } } }).returning(); + await db.insert(dotAgentBindings).values({ id: bindingId, companyId, agentId: agent.id, + operatorId: "alice", providerKind: "grok_bot", status: "connected" }); + const [grant] = await db.select().from(connectionGrants).where(eq(connectionGrants.id, members[0]!.binding.grantId)); + const ref = grant!.credentialSecretRefs.find(ref => ref.configPath === "ai.credential")!; + await db.insert(grokBotRoutines).values({ companyId, agentId: agent.id, bindingId, bindingGeneration: 1, + botName: "Unused held Bot", secretId: ref.secretId, secretVersion: 1, + recoveryRequiredAt: new Date(), recoveryReason: "work_session" }); + expect(await dotRunnerBroker(db).canStartNewWork(companyId, agent.id, bindingId)).toBe(false); + const readiness = vi.spyOn(dotRunnerBroker(db), "canStartNewWork"); + const beforeNative = vi.fn(async () => { throw new Error("fixture stopped before provider start"); }); + const heartbeat = heartbeatService(db, { pluginWorkerManager: worker, runtimeEnv: {}, + beforeNativeRuntimeSelection: beforeNative, + nativeSessionBackendFactory: () => { throw new Error("Unexpected provider start"); } }); + const [issue] = await db.insert(issues).values({ companyId, title: "Healthy routed task", status: "todo", + assigneeAgentId: agent.id, responsibleUserId: "alice", createdByUserId: "alice" }).returning(); + try { + const run = await heartbeat.wakeup(agent.id, { source: "assignment", reason: "issue_assigned", triggerDetail: "system", + payload: { issueId: issue.id }, contextSnapshot: { issueId: issue.id, responsibleUserId: "alice" } }); + expect(run).not.toBeNull(); + await heartbeat.drainActiveRunExecutions(); + const saved = await heartbeat.getRun(run!.id); + expect(saved?.contextSnapshot?.aiRouterSelection).toMatchObject({ poolId: pool.id, + memberId: members[0]!.id, runtimeConfig: { provider: "codex" } }); + expect(saved?.error).toContain("fixture stopped before provider start"); + expect(beforeNative).toHaveBeenCalledExactlyOnceWith(run!.id); + expect(readiness).not.toHaveBeenCalled(); + } finally { + await db.update(agents).set({ status: "paused" }).where(eq(agents.id, agent.id)); + await heartbeat.drainActiveRunExecutions(); + readiness.mockRestore(); + await instanceSettingsService(db).updateExperimental(original); + } + }, 30_000); + it("keeps session identity and fingerprints stable on authenticated refresh, but changes them on manual replacement", async () => { const input = { companyId, agentId, responsibleUserId: "alice", adapterType: "paperclip_runner", binding: members[0]!.binding, config: { provider: "codex", model: "gpt-5.6-sol" } }; const first = await prepareManagedAiRuntime(db, input); diff --git a/server/src/__tests__/grok-bot-routine.test.ts b/server/src/__tests__/grok-bot-routine.test.ts index c49b846a6d8..1bc93d8ff41 100644 --- a/server/src/__tests__/grok-bot-routine.test.ts +++ b/server/src/__tests__/grok-bot-routine.test.ts @@ -23,6 +23,8 @@ import { type GrokRoutineWake } from "../services/grok-bot/webhook.js"; import { remoteTerminationReceipt } from "../services/remote-execution-termination.js"; import { admitExplicitNativeContinuation } from "../services/explicit-native-continuation.js"; import { hasRequiredWorkspaceRecovery } from "../services/workspace-restore-recovery-state.js"; +import { heartbeatService } from "../services/heartbeat.js"; +import { environmentRuntimeService } from "../services/environment-runtime.js"; let temporary: Awaited>; let db: ReturnType; @@ -102,6 +104,63 @@ async function acceptance(f: Awaited>, work: typeof d } describe("durable Grok routine qualification", () => { + it("preflights current connection authority without treating database failures as readiness", async () => { + const f = await fixture(); + const broker = dotRunnerBroker(db); + const canStart = () => broker.canStartNewWork(f.company.id, f.agent.id, f.pairing.bindingId); + expect(await canStart()).toBe(false); + const row = await verification(f); + await f.service.send(f.company.id, f.agent.id, row.id); + await f.service.acknowledge(f.principal, acknowledgement(row)); + expect(await canStart()).toBe(true); + expect(await broker.canStartNewWork(randomUUID(), f.agent.id, f.pairing.bindingId)).toBe(false); + expect(await broker.canStartNewWork(f.company.id, f.agent.id, randomUUID())).toBe(false); + await db.update(grokBotRoutines).set({ recoveryRequiredAt: new Date(), recoveryReason: "work_session" }) + .where(eq(grokBotRoutines.bindingId, f.pairing.bindingId)); + expect(await canStart()).toBe(false); + await db.update(grokBotRoutines).set({ recoveryRequiredAt: null, recoveryReason: null }) + .where(eq(grokBotRoutines.bindingId, f.pairing.bindingId)); + await db.update(mcpOauthGrants).set({ revokedAt: new Date() }).where(eq(mcpOauthGrants.id, f.principal.grant.id)); + expect(await canStart()).toBe(false); + const unavailable = new Error("synthetic database unavailable"); + vi.spyOn(broker, "snapshot").mockRejectedValueOnce(unavailable); + await expect(canStart()).rejects.toBe(unavailable); + }); + + it("holds public heartbeat input before claiming a Runner while Grok recovery is required", async () => { + const f = await fixture(); + await db.update(agents).set({ status: "idle" }).where(eq(agents.id, f.agent.id)); + const row = await verification(f); + await f.service.send(f.company.id, f.agent.id, row.id); + await f.service.acknowledge(f.principal, acknowledgement(row)); + await db.update(grokBotRoutines).set({ recoveryRequiredAt: new Date(), recoveryReason: "work_session" }) + .where(eq(grokBotRoutines.bindingId, f.pairing.bindingId)); + const [task] = await db.insert(issues).values({ companyId: f.company.id, title: "Wait for Grok inspection", + status: "todo", assigneeAgentId: f.agent.id, responsibleUserId: f.principal.grant.userId }).returning(); + const runtime = environmentRuntimeService(db); + const acquire = vi.spyOn(runtime, "acquireRunLease").mockRejectedValue(new Error("Unexpected provider allocation")); + const heartbeat = heartbeatService(db, { environmentRuntime: runtime }); + try { + await heartbeat.wakeup(f.agent.id, { source: "assignment", reason: "issue_assigned", + requestedByActorType: "user", requestedByActorId: f.principal.grant.userId, + payload: { issueId: task.id }, contextSnapshot: { issueId: task.id } }); + await heartbeat.drainActiveRunExecutions(); + for (let poll = 0; poll < 3; poll++) await heartbeat.resumeQueuedRuns(); + await heartbeat.drainActiveRunExecutions(); + const runs = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.agentId, f.agent.id)); + expect(runs).toEqual([expect.objectContaining({ status: "queued", startedAt: null })]); + expect(await db.select().from(environmentLeases).where(eq(environmentLeases.companyId, f.company.id))).toEqual([]); + expect(await db.select().from(dotRunnerAssignments).where(eq(dotRunnerAssignments.agentId, f.agent.id))).toEqual([]); + expect(await db.select().from(grokBotDispatches).where(eq(grokBotDispatches.bindingId, f.pairing.bindingId))) + .toEqual([expect.objectContaining({ id: row.id, status: "acknowledged" })]); + expect(acquire).not.toHaveBeenCalled(); + expect((await db.select().from(issues).where(eq(issues.id, task.id)))[0].status).toBe("todo"); + } finally { + await db.update(agents).set({ status: "paused" }).where(eq(agents.id, f.agent.id)); + await heartbeat.drainActiveRunExecutions(); + } + }); + it("receives an authenticated Grok acknowledgement through the actual MCP transport without exposing personal or Dot event tools", async () => { const f = await fixture(); const row = await verification(f); await f.service.send(f.company.id, f.agent.id, row.id); diff --git a/server/src/services/dot-runner-broker.ts b/server/src/services/dot-runner-broker.ts index c602ee37a3e..cc2e594f15d 100644 --- a/server/src/services/dot-runner-broker.ts +++ b/server/src/services/dot-runner-broker.ts @@ -204,6 +204,15 @@ function createBroker(db: Db) { } }); }, + async canStartNewWork(companyId: string, agentId: string, bindingId: string): Promise { + try { + await this.snapshot(companyId, agentId, bindingId); + return true; + } catch (error) { + if (error instanceof McpOAuthError) return false; + throw error; + } + }, async snapshot(companyId: string, agentId: string, bindingId: string): Promise { const state = await this.bindingForAgent(companyId, agentId); if (!state || !await enabled(state.providerKind)) throw fail("External provider is disabled for new work."); diff --git a/server/src/services/heartbeat.ts b/server/src/services/heartbeat.ts index 2525be9a261..9cd1ff8635e 100644 --- a/server/src/services/heartbeat.ts +++ b/server/src/services/heartbeat.ts @@ -1,3 +1,4 @@ +import { dotRunnerBroker } from "./dot-runner-broker.js"; import { withRunExecutionTiming, measureRunPhase, measureWorkspaceSyncPhase } from "./execution-preparation-timing.js"; import { createHeartbeatPreparationTiming } from "./heartbeat/preparation-timing.js"; import { createHeartbeatInstructionCopyRelease, createHeartbeatInstructionWorkingCopy } from "./heartbeat/instruction-working-copy.js"; @@ -1203,6 +1204,13 @@ export function heartbeatService( }); const heartbeatQueue = createHeartbeatQueue(db, { + canStartNewExternalBotWork: async agent => { + if (agent.adapterType !== "paperclip_runner" || agent.adapterConfig.provider !== "grok_bot") return true; + const aiConnection = aiRuntimeConnectionBindingSchema.safeParse(agent.runtimeConfig?.aiConnection); + if (aiConnection.success && aiConnection.data.mode === "router") return true; + return dotRunnerBroker(db).canStartNewWork(agent.companyId, agent.id, + readNonEmptyString(agent.adapterConfig.externalBindingId) ?? ""); + }, applyWakeQueuePostCommitEffects, setRunStatus, setWakeupStatus, diff --git a/server/src/services/heartbeat/queue.test.ts b/server/src/services/heartbeat/queue.test.ts index 3572e68daf2..3e676e5af0a 100644 --- a/server/src/services/heartbeat/queue.test.ts +++ b/server/src/services/heartbeat/queue.test.ts @@ -48,6 +48,7 @@ function callbacks(db: Db) { runTaskKey: (run: Run) => typeof run.contextSnapshot?.issueId === "string" ? run.contextSnapshot.issueId : null, filterZombieCoalesceTarget: (run: T | null) => run, getSchedulingSuppression: vi.fn(async () => ({ suppressed: false, reason: null } as Awaited>)), + canStartNewExternalBotWork: vi.fn(async () => true), executeRun: vi.fn(async () => undefined), applyWakeQueuePostCommitEffects: vi.fn(async () => undefined), releaseIssueExecutionAndPromote: vi.fn(async () => undefined), @@ -199,6 +200,56 @@ describe.skipIf(!support.supported)("heartbeat queue database wiring", () => { expect((await db.select().from(issues).where(eq(issues.id, issue.id)))[0].executionRunId).toBe(waiter.id); }); + it.each(["direct", "batch"] as const)("keeps fresh work queued across a held external connection on the %s path", async path => { + const { run, agent, issue, wake } = await fixture("queued"); + await db.update(heartbeatRuns).set({ responsibleUserId: "queue-test-user" }).where(eq(heartbeatRuns.id, run.id)); + const deps = { ...callbacks(db), canStartNewExternalBotWork: vi.fn(async () => false) }; + deps.getAgent.mockResolvedValue({ ...agent, adapterType: "paperclip_runner", adapterConfig: { provider: "grok_bot" } }); + const queue = createHeartbeatQueue(db, deps); + for (let attempt = 0; attempt < 3; attempt++) { + if (path === "direct") expect(await queue.claimQueuedRun({ ...run, responsibleUserId: "queue-test-user" })).toBeNull(); + else expect(await queue.startNextQueuedRunForAgent(agent.id)).toEqual([]); + } + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)))[0]).toMatchObject({ status: "queued", startedAt: null }); + expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, wake.id)))[0].status).toBe(wake.status); + expect((await db.select().from(issues).where(eq(issues.id, issue.id)))[0].status).toBe("todo"); + expect(deps.executeRun).not.toHaveBeenCalled(); + expect(deps.setRunStatus).not.toHaveBeenCalled(); + }); + + it("admits the same queued input after the external connection is repaired", async () => { + const { run, agent } = await fixture("queued"); + await db.update(heartbeatRuns).set({ responsibleUserId: "queue-test-user" }).where(eq(heartbeatRuns.id, run.id)); + const deps = { ...callbacks(db), canStartNewExternalBotWork: vi.fn(async () => false) }; + deps.getAgent.mockResolvedValue(agent); + const queue = createHeartbeatQueue(db, deps); + expect(await queue.startNextQueuedRunForAgent(agent.id)).toEqual([]); + deps.canStartNewExternalBotWork.mockResolvedValue(true); + expect(await queue.startNextQueuedRunForAgent(agent.id)).toEqual([expect.objectContaining({ id: run.id, status: "running" })]); + expect(deps.executeRun).toHaveBeenCalledExactlyOnceWith(run.id); + }); + + it("rechecks the connection hold when Stop races the queue scan", async () => { + const { run, agent } = await fixture("queued"); + const deps = { ...callbacks(db), canStartNewExternalBotWork: vi.fn(async () => false) }; + deps.getAgent.mockResolvedValue(agent); + deps.canStartNewExternalBotWork.mockResolvedValueOnce(true); + expect(await createHeartbeatQueue(db, deps).startNextQueuedRunForAgent(agent.id)).toEqual([]); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)))[0]).toMatchObject({ status: "queued", startedAt: null }); + expect(deps.canStartNewExternalBotWork).toHaveBeenCalledTimes(2); + expect(deps.executeRun).not.toHaveBeenCalled(); + }); + + it("keeps admitted resumptions outside the fresh external connection gate", async () => { + const { run, agent } = await fixture("queued"); + await db.update(heartbeatRuns).set({ startedAt: new Date(), responsibleUserId: "queue-test-user" }).where(eq(heartbeatRuns.id, run.id)); + const [admitted] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)); + const deps = { ...callbacks(db), canStartNewExternalBotWork: vi.fn(async () => false) }; + deps.getAgent.mockResolvedValue(agent); + expect(await createHeartbeatQueue(db, deps).claimQueuedRun(admitted)).toMatchObject({ id: run.id, status: "running" }); + expect(deps.canStartNewExternalBotWork).not.toHaveBeenCalled(); + }); + it("cancels at the daily run cap before claiming or dispatching", async () => { const { run, agent } = await fixture("queued"); const cappedAgent = { ...agent, runtimeConfig: { heartbeat: { maxDailyRuns: 0 } } }; diff --git a/server/src/services/heartbeat/queue.ts b/server/src/services/heartbeat/queue.ts index b81906283a5..6a61871516b 100644 --- a/server/src/services/heartbeat/queue.ts +++ b/server/src/services/heartbeat/queue.ts @@ -196,6 +196,7 @@ export interface HeartbeatQueueDependencies extends Pick string | null; filterZombieCoalesceTarget: (run: T | null, live: { has(id: string): boolean }) => T | null; getSchedulingSuppression: () => Promise<{ suppressed: boolean; reason: "worktree_instance" | "database_restore_in_progress" | "task_drain" | null }>; + canStartNewExternalBotWork: (agent: typeof agents.$inferSelect) => Promise; executeRun: (runId: string) => Promise; releaseIssueExecutionAndPromote: (run: Pick, options?: { suppressImmediateRecovery?: boolean; deferredPostCommitEffects?: WakeQueuePostCommitEffect[] }) => Promise; applyWakeQueuePostCommitEffects: (effects: WakeQueuePostCommitEffect[]) => Promise; @@ -371,6 +372,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende cancelActiveForAgentInternal, listCompanyAgentOrgRows, executeRun, + canStartNewExternalBotWork, activeRunExecutionPromises, activeWakeupPromises, instanceSettings, @@ -984,6 +986,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende ); return null; } + if (!run.startedAt && !await canStartNewExternalBotWork(agent)) return null; const context = parseObject(run.contextSnapshot); const budgetBlock = await budgets.getInvocationBlock( @@ -1935,6 +1938,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende ) .orderBy(asc(heartbeatRuns.createdAt)); if (queuedRuns.length === 0) return []; + if (queuedRuns.every(run => !run.startedAt) && !await canStartNewExternalBotWork(agent)) return []; const dependencyReadiness = await listQueuedRunDependencyReadiness( agent.companyId, diff --git a/tests/e2e/composer-stop.spec.ts b/tests/e2e/composer-stop.spec.ts index 46cc882fe14..a9c747f9bf5 100644 --- a/tests/e2e/composer-stop.spec.ts +++ b/tests/e2e/composer-stop.spec.ts @@ -278,12 +278,12 @@ for (const adapter of ["process", "paperclip_runner"] as const) { await request.get(`/api/issues/${parent.id}/comments`), ); expect(JSON.stringify(comments)).toContain("Please check mobile too."); - const queue = await json( - await request.get(`/api/issues/${parent.id}/queued-comments`), - ); - expect(JSON.stringify(queue.entries)).toContain( - "Please check mobile too.", - ); + await expect.poll(async () => { + const queue = await json( + await request.get(`/api/issues/${parent.id}/queued-comments`), + ); + return JSON.stringify(queue.entries); + }).toContain("Please check mobile too."); let dispatchedAt = 0; page.on("request", (req) => { From 7a8b3583659150bcd819b40be5d6dc9d1df2ee88 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Thu, 15 Oct 2026 11:51:09 -0500 Subject: [PATCH 11/19] docs: add Honeycomb tracing skill (#16453) Document how to find, configure, and interpret Paperclip application and CI traces in Honeycomb. Co-Authored-By: Paperclip --- .agents/skills/honeycomb-tracing/SKILL.md | 169 ++++++++++++++++++ .../honeycomb-tracing/agents/openai.yaml | 3 + .../skills/honeycomb-tracing/references/ci.md | 96 ++++++++++ .../honeycomb-tracing/references/runtime.md | 90 ++++++++++ 4 files changed, 358 insertions(+) create mode 100644 .agents/skills/honeycomb-tracing/SKILL.md create mode 100644 .agents/skills/honeycomb-tracing/agents/openai.yaml create mode 100644 .agents/skills/honeycomb-tracing/references/ci.md create mode 100644 .agents/skills/honeycomb-tracing/references/runtime.md diff --git a/.agents/skills/honeycomb-tracing/SKILL.md b/.agents/skills/honeycomb-tracing/SKILL.md new file mode 100644 index 00000000000..1f48dcb584b --- /dev/null +++ b/.agents/skills/honeycomb-tracing/SKILL.md @@ -0,0 +1,169 @@ +--- +name: honeycomb-tracing +description: >- + Trace Paperclip runs, requests, and full CI suites in Honeycomb. Use to find + traces, diagnose latency, configure opt-in OpenTelemetry, or add missing spans. +--- + +# Honeycomb Tracing + +Produce an inspectable waterfall tied to the actual run or CI attempt, explain +which intervals dominate elapsed time, and identify the smallest useful next +experiment. Respect diagnosis-only requests; tracing does not itself authorize +replaying tasks, changing credentials, or deploying instrumentation. + +## Choose the data path + +| Question | Start here | +| --- | --- | +| Why is a task slow before, during, or after model execution? | Application workflow below; [runtime setup](references/runtime.md) if spans are absent | +| Why does an issue page load slowly? | Request spans plus browser timing, described below | +| Which CI jobs or test files delay the whole suite? | [CI profiling](references/ci.md) | +| Where should a new span go? | Instrumentation guidance below and [Observability contract](../../../doc/observability.md) | + +Repository-relative paths and commands assume the checkout root. The maintained +contracts are `doc/observability.md` and `doc/ci-profiling.md`; recheck them and +the owning code when a deployed version differs. + +Keep the repository's three paths distinct: + +- **Observability:** operator-configured OpenTelemetry export to Honeycomb or + another OTLP destination. With no endpoint, this tracing stays off. +- **Telemetry:** first-party events under `packages/shared/src/telemetry/`, sent + to Paperclip by default unless opted out. Adding these requires the generated + contract, Telemetry README, and privacy-review workflow in `AGENTS.md`. +- **Run log:** local `heartbeat_run_events` and measured performance events. + Useful for correlation, but a local event is not proof of Honeycomb export. + +Observability changes within the closed span-attribute allowlist do not need the +Telemetry privacy review. Preserve the endpoint gate and no-operation behavior. + +## Find the application's trace + +1. Resolve the instance, company, issue, agent, **run UUID**, and UTC interval. + Record the effective harness/model, workspace/repository, warm or fresh session, + and actual deployed revision. A task identifier is not a run UUID; one task + can have several runs, and one run can have several execution attempts. +2. Prefer existing Honeycomb tools when exposed and authorized. Otherwise use + the signed-in browser. The current project link targets team `paperclip`, + environment `test`, dataset `paperclip`, and `service.name = paperclip`. + These are defaults in `ui/src/lib/honeycomb-run-link.ts`, not proof of every + deployment's destination. Verify the target before comparing data. +3. Open **View in Honeycomb** on the run page when available. It requires + Paperclip Developer Mode and local implicit board or instance-admin access. + This link queries existing data. **Re-run with provider trace** is a different + action: it starts work and is not needed to read Honeycomb. +4. Without the link, calculate the same correlation value locally: + + ```sh + # Set RUN_ID to the exact saved run UUID first; no trailing newline is hashed. + node -e 'console.log(require("node:crypto").createHash("sha256").update(process.argv[1]).digest("hex").slice(0, 12))' "$RUN_ID" + ``` + + Filter `paperclip.task.run.run_id` to that 12-character lowercase hash and + `service.name` to the observed service. Start with the run's absolute time + window. The product link defaults to the last seven days; widen it for older + runs. Group `COUNT` by `trace.trace_id` to discover matching traces. +5. The product's root filter is `name in [task.execution_attempt, + task.provider_session, task.run]`. Use Honeycomb's `in` operator, not an + invented OR group. Remove the name filter when setup failed or preparation + spans are missing; older/standalone preparation can have a separate trace ID. +6. Open each relevant trace and its span details. A count-only query/export + cannot establish span durations, parentage, or causal attribution. Capture + the trace URL, time window, service/version, and relevant span names/timings. + +In recent investigations, `HONEYCOMB_TEST_CONFIG_API_KEY` in the operator's +`~/.secrets` could read metadata but could not query traces. Ingest permission, +metadata access, and query access are different. Verify current capabilities; +when queries are unavailable, use the signed-in UI rather than creating a new +key or upgrading permissions. Read only needed secret fields, never dump the +file, and keep values out of commands, artifacts, repository files, and reports. + +## Read the timeline correctly + +| Interval | Meaning / next inspection | +| --- | --- | +| Run creation → executor start | Admission/queue delay; check drain, capacity, and dispatch evidence | +| `task.execution_attempt` | Server executor invocation through preparation, dispatch, durable reply, and cleanup | +| `heartbeat.prepare.*`, `native.controller.*` | Controller preparation; expand workspace, configuration, backend, and runtime-selection work | +| `task.provider_session` | Native provider lifecycle; includes preparation and execution, ends before outer workspace persistence | +| `task.run` | Legacy adapter root | +| `runner.session.startup`, `environment.*`, `stage.*`, `sandbox.*` | Runner/environment acquisition, workspace transfer, and startup | +| `agent.turn` | Agent turn time; inspect actual children before calling all of it model latency | +| `execution.workspace_finalize`, `workspace_sync.*`, `completion.*`, `response.*` | Copyback, settlement, response persistence, and publication work | + +The attempt and provider lifecycle are linked roots; they need not share a trace +ID. Neither is a guarantee of one continuous trace across queueing or restarts. +Correlate with the run log and saved reply timestamps. The native local event +`task.run.measured` includes terminal elapsed time and durable copyback. + +Use start/end times to compare serial intervals and the union of parallel work. +Never add a parent to its children or sum parallel branches as elapsed time. +For example, `configuration_skills` is inside `execution_configuration`, and +`skills.prepare` overlaps the broader `heartbeat.prepare_before_environment`. +Detail spans are capped at 16 per name per attempt; missing per-item children +do not mean later items did no work. + +Distinguish first streamed text, committed reply, externally delivered message, +and browser paint. A successful `response.transaction.commit` proves durable storage; +`response.publish` is enqueueing and `execution.realtime` is server dispatch. +Neither proves that the user saw the response. Choose the boundary the user asked +to improve, then measure the whole path to that boundary. + +For slow issue pages, combine `issue.read.` with `Server-Timing` +(`paperclip_issue`, `issue_`) and the browser's request waterfall and +`issue-detail:navigate→content-paint` measure. Concurrent reads overlap. Fast +API completion does not establish fast conversation reveal or rendering. + +A long parent span without children is an attribution gap. It does not prove +SQL, network, or model time. For database hypotheses, use +[fix-slow-queries](../fix-slow-queries/SKILL.md). For CPU hypotheses, use a bounded +authorized profile covering the actual slow phase; a sample taken while queued +cannot explain preparation. Low aggregate container CPU does not exclude a +blocked Node main thread. Keep profiled diagnostics separate from unprofiled +acceptance timings. + +## Add only the missing instrumentation + +Read applicable repository rules before editing. Reuse the existing timing seams: + +- `server/src/services/execution-preparation-timing.ts`: + `withRunExecutionTiming`, `createRunPreparationTiming`, `measureRunPhase`, and + `beginRunPhase`. Use the existing attempt; do not create a new root per detail. +- `server/src/services/native-runtime/native-run-trace.ts`: native lifecycle + hierarchy and links to the owning attempt. +- `server/src/services/issue-read-timing.ts`: issue-read phases and Server-Timing. +- `packages/adapter-utils/src/acpx-engine/startup-timing.ts`: runtime parent + context and closed span attributes. Adapter utilities receive the host context + rather than constructing an independent exporter. + +Extend the closed phase union when a meaningful boundary is missing. Wrap the +actual awaited operation, preserve parent context across async/provider boundaries, +and end spans on success and failure. Keep real wall times for spans recorded +after the operation. Use existing hashed IDs, closed enums, and allowed numeric +counts; do not add prompts, tool arguments/results, paths, raw identifiers, +credentials, or exception text. Do not enable all filesystem instrumentation to +explain one slow scan. + +Observability must not retry the operation, replace its return/error, change +authorization/locks, or break execution when the tracer fails. Preserve root +sampling and the explicit endpoint gate even if another library registers an +OTel provider. Update `doc/observability.md` when the trace contract changes. +Run focused tests for endpoint-off behavior, parentage/concurrent-run isolation, +failure handling, and the changed seam. Use real exported evidence to claim that +new spans appear in Honeycomb; a unit-test tracer alone does not prove ingestion. + +## Compare and hand back useful evidence + +Choose the largest actionable serial wait or unexplained gap, then reproduce it +at the smallest faithful layer. Compare matched runs: same harness/model, +workspace size, session warmth, concurrency, and relevant revision/configuration. +Report sample count and median/range when several runs are available. Preserve +failures; a failed fast run is not a latency improvement. + +Return the run/attempt and trace links, observed revision, end-to-end timing, +dominant non-overlapping phases, uncertainty, and next action or verified gain. +Label local, synthetic, and hosted evidence precisely: a dev-host reproduction +is not a local checkout run, and a merged fix does not prove hosted improvement. +If traces are unavailable, state the missing access or coverage and what the +run log/browser measurements actually establish. diff --git a/.agents/skills/honeycomb-tracing/agents/openai.yaml b/.agents/skills/honeycomb-tracing/agents/openai.yaml new file mode 100644 index 00000000000..db9a552222f --- /dev/null +++ b/.agents/skills/honeycomb-tracing/agents/openai.yaml @@ -0,0 +1,3 @@ +interface: + display_name: "Honeycomb Tracing" + short_description: "Trace Paperclip runs, requests, and full CI waterfalls" diff --git a/.agents/skills/honeycomb-tracing/references/ci.md b/.agents/skills/honeycomb-tracing/references/ci.md new file mode 100644 index 00000000000..93cee4965ea --- /dev/null +++ b/.agents/skills/honeycomb-tracing/references/ci.md @@ -0,0 +1,96 @@ +# Profile the full PR suite + +Use [doc/ci-profiling.md](../../../../doc/ci-profiling.md) as the maintained +operator runbook. This path exports recorded CI timing through Honeycomb's Events +API; it does not require application OTLP settings or an application run UUID. + +## Select one complete attempt + +1. Confirm the intended PR and head. Profiling is opt-in through `profile-ci`. + The PR author, original workflow actor, and rerun actor must be individual + users authorized by trusted master's `.github/**` CODEOWNERS rule. Applying + the label to an outside contributor's PR does not grant profiling access. +2. When asked to profile that PR, add the label. It starts a fresh + **CI profile · PR #number** run of the main `PR` workflow. The label forces + all main lanes even for stack-middle or release-notes-only changes. Further + pushes also profile while it remains present. +3. Record run ID, attempt, tested revision, and job inventory. Keep the label + and head unchanged until the separate **CI profile** exporter completes. + Use a fresh label-triggered run for an unambiguous whole-suite baseline. + Rerunning only failures does not rerun the successful jobs. +4. Open the Honeycomb waterfall from the run/exporter's Summary. The project + destination is team `paperclip`, environment `test`, dataset `paperclip-ci`. + Verify the displayed root and representative file spans; a generated URL + or green test workflow is not proof that export succeeded. +5. Remove the label after a one-off profile unless ongoing profiling was requested. + Re-adding it starts another complete attempt. Report any remaining label so + future pushes do not unexpectedly keep profiling. + +The main suite preserves existing AWS routing (`paperclip-public-pr-x64`, with +`paperclip-ci-routing-x64` for routing) and its authorized hosted fallback. +Inspect actual runner assignments if AWS placement is part of acceptance. The +separate exporter uses `ubuntu-latest` and is outside the measured suite. + +## Read and qualify the result + +The hierarchy is **workflow → job → step → test filename → tests**. Collapse +spans at the job depth for a shared wall-clock view; expand slow jobs and search +for a known `test.file` to prove filename-level detail arrived. + +- Root elapsed time runs from workflow creation to the last job completion; + reruns start at that attempt's start. Do not sum concurrent job durations. +- Use `ci.finishes_last`, `ci.execution_ms`, and `ci.start_offset_ms` to find + late-finishing work, then inspect setup, caches, compilation, builds, and tests. + A long job that ends early may not delay the suite. +- Waiting spans start at job creation. Dependency waits before job creation + appear as gaps; GitHub does not expose a separate ready timestamp, so those + gaps are not proven runner-queue delay or a computed dependency critical path. +- Vitest, Playwright, and Node supply file/test detail with different timestamp + semantics. GitHub steps have second precision; Node test spans use reporter + receipt time. Rust and standalone scripts retain job/step coverage. +- Scope is the main PR workflow. Separate Storybook, image, release, deployment, + and live-provider eval workflows are not included. + +Inspect `ci.profile_complete`, coverage warnings, expected versus reported +invocations, file/test counts, `ci.partial_rerun`, and `ci.carried_job_count`. +Failed/cancelled attempts can omit downstream work; carried jobs completed before +the new attempt are not fresh measurements. Compare the trace inventory with +GitHub's attempt-specific jobs and retained timing artifacts. + +Honeycomb's waterfall displays at most 32,000 spans, prioritizing those closest +to the root. Missing deep test rows can be a display limit rather than missing +ingest. Check **View events** and coverage evidence before claiming data loss. +See the [provider's large-trace limit](https://docs.honeycomb.io/troubleshoot/common-issues/data-in-honeycomb/#your-very-large-trace-displays-only-32000-spans). + +## Credentials and export failures + +`HONEYCOMB_CI_API_KEY` is a dedicated ingest-only **environment secret** in +GitHub environment `ci-profiling`, restricted to `master`. Only the trusted +default-branch exporter reads it. PR tests still execute the PR branch and never +receive the key. Do not move it to repository/org secrets, put a configuration +key in CI, execute downloaded artifacts, or switch trusted workflow refs to a PR +head for convenience. Ordinary CI must work with profiling disabled and no key. + +The exporter checks each batch response, paces uploads, and retries only +transiently rejected events. If it fails, inspect its logs and saved artifacts +before retrying: partial export may already have sent spans, and the Events API +does not promise deduplication on resubmission. Timing artifacts are retained for +14 days. Preserve incomplete-export evidence even if a later upload succeeds. + +For local reconstruction without uploading, use the existing exporter: + +```sh +node scripts/ci-profile/export.mjs --run-id=RUN_ID --attempt=1 --historical --dry-run --output=/tmp/ci-trace.json +``` + +Replace `RUN_ID` and attempt with the recorded values. This needs authorized +GitHub read access. Historical reconstruction can recover job/step timings, but +cannot invent file/test timings that were never recorded. It does not prove the +live label-triggered journey. A real historical upload is a separate authorized +action using the ingest key, not a necessary part of diagnosis. + +When changing CI tracing, use the runbook's focused exporter, routing, reporter, +and workflow checks. For a whole-suite tracing acceptance, return the exact +GitHub attempt and Honeycomb links, elapsed time, coverage and export status, +runner placement if relevant, and the late-finishing jobs/file timings that +justify the next optimization. diff --git a/.agents/skills/honeycomb-tracing/references/runtime.md b/.agents/skills/honeycomb-tracing/references/runtime.md new file mode 100644 index 00000000000..a39fa5866be --- /dev/null +++ b/.agents/skills/honeycomb-tracing/references/runtime.md @@ -0,0 +1,90 @@ +# Runtime tracing setup and missing spans + +Read this when configuring an authorized runtime or diagnosing absent spans. +The authoritative implementation is +[instrumentation.ts](../../../../server/src/instrumentation.ts); package versions +and the data contract live in +[Observability](../../../../doc/observability.md#enabling-tracing). + +## Enable the existing bootstrap + +1. Identify the actual process/image and its owning package. Inspect + `server/package.json` for the **exact** peer versions of the OTel SDK, + auto-instrumentations, resources, semantic conventions, and one exporter. + Install them in the runtime that will resolve them. Avoid copying old version + pins from sessions or adding every exporter to the application manifest. +2. Configure that process before startup. A direct US Honeycomb HTTP/protobuf + setup uses these values: + + | Setting | Value | + | --- | --- | + | `OTEL_EXPORTER_OTLP_ENDPOINT` | `https://api.honeycomb.io` | + | `OTEL_EXPORTER_OTLP_PROTOCOL` | `http/protobuf` | + | `OTEL_EXPORTER_OTLP_HEADERS` | `x-honeycomb-team=` | + | `OTEL_SERVICE_NAME` | `paperclip`, or the deliberately selected service identity | + + Inject the header through the existing secret manager/process environment; + do not paste a key into shell history or a tracked `.env`. Confirm region and + environment from the operator's configuration. An existing collector may be + the intended endpoint instead of direct Honeycomb ingestion. +3. For HTTP, supply the **base URL**; the exporter appends `/v1/traces`. Paperclip + intentionally constructs HTTP exporters without an explicit URL so they read + the standard environment variables. A signal-specific variable alone does + not activate Paperclip: the bootstrap gate checks the generic + `OTEL_EXPORTER_OTLP_ENDPOINT`. +4. For gRPC, use the appropriate TLS endpoint and matching exporter; the protocol + defaults to gRPC when unset. Unknown protocol strings warn and fall back to + gRPC. Do not accidentally pair HTTP configuration with that default. +5. Check the runtime's startup diagnostics. A missing or mismatched peer package + leaves the server healthy **without tracing**. `instrumentationReady` settles + before DB connections/server construction; shutdown awaits a bounded flush. + A healthy `/api/health` alone cannot validate export. +6. Generate a small representative operation within the authorized task, allow + the export batch to arrive, and open its actual Honeycomb spans. Verify + service, revision, timestamps, parentage, and the expected operation. Record + a trace link as proof; a bootstrap log line only proves initialization. + +Honeycomb's [OTLP configuration](https://docs.honeycomb.io/send-data/opentelemetry/#using-the-honeycomb-opentelemetry-endpoint) +documents regions, headers, and the additional dataset header required for +Classic. Confirm current provider requirements when changing setup. + +## Identify the serving version + +Paperclip resolves `service.version` from the build stamp +(`dist/build-info.json`), then runtime Git, then `OTEL_SERVICE_VERSION`, then +`unknown`. The build stamp wins over a stale environment variable. Docker builds +without `.git` use `PAPERCLIP_BUILD_COMMIT` to populate the stamp. + +Match the trace's version with actual runtime/deployment evidence. Record source +revision and observation time; an opaque desired release or current local HEAD +is not evidence that the running service includes a fix. + +## Troubleshoot in this order + +| Observation | Check | +| --- | --- | +| No results | Correct team/environment/dataset/service, exact run hash, absolute UTC range, retention, and query permissions | +| Only native/provider spans | Remove root-name restriction; inspect other trace IDs for the same run hash and controller version | +| App healthy, nothing exported | Actual process env, exact optional peers, protocol/exporter match, bootstrap diagnostic, root sampling, and authorized endpoint reachability | +| Trace ends before saved reply | Provider root excludes outer persistence; inspect `task.execution_attempt`, settlement spans, and run-log timestamps | +| Missing unfinished phase | Active spans may not export until they end; abrupt exit can lose them | +| Long manual span, no SQL children | Auto-instrumentation does not guarantee children; inspect real dependency timings rather than equating the whole span with database time | +| Query API denied, authentication works | Check query permission/plan; use the signed-in UI when available, without widening key privileges | +| Export/CSV contains only COUNT | It is an aggregate result. Inspect span details or the authorized raw-events view for event-level analysis | + +The server exports traces only through this bootstrap. `fs`, `dns`, and `net` +auto-instrumentations are disabled for volume. Local performance events can +still exist with OTLP off; their presence is not an exporter health check. + +When changing the bootstrap or correlation, start with the relevant tests: + +```sh +pnpm exec vitest run server/src/__tests__/instrumentation.test.ts +pnpm exec vitest run server/src/services/execution-preparation-timing.test.ts +pnpm exec vitest run server/src/services/native-runtime/native-run-trace-context.test.ts +pnpm exec vitest run ui/src/lib/honeycomb-run-link.test.ts +``` + +Select affected checks rather than running this entire list for a read-only +diagnosis. These establish local contracts; deployment and Honeycomb ingestion +need their own evidence. From 95692b7785c794dab4d02491a3117702b90e3e4f Mon Sep 17 00:00:00 2001 From: Huanyi Xie <74809554+xiehuanyi@users.noreply.github.com> Date: Thu, 15 Oct 2026 20:03:01 +0300 Subject: [PATCH 12/19] fix(docker): forward tool action signing secret in Compose (#15202) ## Thinking Path > - Paperclip can run through Docker Compose. > - Approval-gated tools need an independent signing secret. > - The guide sets that secret before starting Compose. > - Neither Compose file passes it into the container. > - This pull request forwards the operator's configured secret. > - Documented setup can issue signed tool-action approvals. ## Linked Issues or Issue Description Fixes: #15064 ## What Changed - Forward PAPERCLIP_TOOL_ACTION_SIGNING_SECRET in both shipped Compose environments. - Document persistent deployment secrets and reuse on restart. - Explain that it is independent from the session secret. - Keep an empty default so deployments without approval-gated tools retain their existing startup behavior. ## Verification - Docker Compose config before the fix drops a supplied synthetic secret in both files. - After the fix, both preserve the supplied value. - Both also resolve successfully when the optional signing secret is absent. - Used --env-file /dev/null and synthetic values. No containers were started. - `node --test scripts/docker-compose-secrets.test.mjs`: 2 passed. - The Docker context CI lane runs this integration test. - git diff --check passes. ## Risks The value is visible to the container as intended. Operators must protect it as a secret. This does not change admission rules or require a new startup value for existing deployments. ## Model Used OpenAI Codex, based on GPT-6, assisted with source review, implementation and local test execution. The exact deployment model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (targeted local checks and full remote CI pass) - [x] I have added or updated tests where applicable (Hermes regression / real Compose interpolation checks) - [x] I have updated relevant documentation to reflect my changes (Hermes has no configuration changes; Compose documentation is updated) - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --- .github/workflows/pr-trusted.yml | 8 +++++ doc/DOCKER.md | 27 +++++++++++++--- docker/docker-compose.quickstart.yml | 6 +++- docker/docker-compose.yml | 2 ++ scripts/docker-compose-secrets.test.mjs | 42 +++++++++++++++++++++++++ 5 files changed, 79 insertions(+), 6 deletions(-) create mode 100644 scripts/docker-compose-secrets.test.mjs diff --git a/.github/workflows/pr-trusted.yml b/.github/workflows/pr-trusted.yml index e225da5cebd..d870c77d9eb 100644 --- a/.github/workflows/pr-trusted.yml +++ b/.github/workflows/pr-trusted.yml @@ -572,6 +572,14 @@ jobs: - name: Set up Docker Buildx uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e # v4 + - name: Set up Docker Compose + uses: docker/setup-compose-action@54042514f505b273907334ae2b9cdbb9a0213c1a # v2 + with: + version: v5.5.1 + + - name: Verify Compose signing secret forwarding + run: node --test scripts/docker-compose-secrets.test.mjs + # Same .dockerignore semantics as the real image builds: a # context-slimming change that strips a committed build input must # fail here, on the pull request, instead of failing every diff --git a/doc/DOCKER.md b/doc/DOCKER.md index 507814c483a..2870a742db8 100644 --- a/doc/DOCKER.md +++ b/doc/DOCKER.md @@ -96,14 +96,27 @@ All persisted under your bind mount (`./data/docker-paperclip` in the example ab ## Docker Compose +### Persistent signing secrets + +Before the first launch, generate two independent values with `openssl rand -hex 32` +and save them in `docker/.env`. Keep any existing deployment's values unchanged: + +```dotenv +BETTER_AUTH_SECRET= +PAPERCLIP_TOOL_ACTION_SIGNING_SECRET= +``` + +Restrict this file to its owner (`chmod 600 docker/.env`) and keep it out of version +control. Compose reads it for both commands below. Reuse the values on every +restart. Rotate the tool-action secret only after outstanding approvals are +resolved, because pending signatures use the previous value. + ### Quickstart (embedded SQLite) Single container, no external database. Data persists via a bind mount. ```sh -BETTER_AUTH_SECRET=$(openssl rand -hex 32) \ -PAPERCLIP_TOOL_ACTION_SIGNING_SECRET=$(openssl rand -hex 32) \ - docker compose -f docker/docker-compose.quickstart.yml up --build +docker compose -f docker/docker-compose.quickstart.yml up --build ``` Defaults: @@ -129,12 +142,16 @@ Pass `OPENAI_API_KEY` and/or `ANTHROPIC_API_KEY` to enable local adapter runs. Paperclip server + PostgreSQL 17. The database is health-checked before the server starts. ```sh -BETTER_AUTH_SECRET=$(openssl rand -hex 32) \ - docker compose -f docker/docker-compose.yml up --build +docker compose -f docker/docker-compose.yml up --build ``` PostgreSQL data persists in a named Docker volume (`pgdata`). Paperclip data persists in `paperclip-data`. +Both Compose files pass `PAPERCLIP_TOOL_ACTION_SIGNING_SECRET` to the server. +Set it to enable signed tool-action approvals, and keep it independent from +`BETTER_AUTH_SECRET`. Leaving it unset preserves startup without approval-gated +tool actions; those calls report `signing_secret_unconfigured`. + ### Untrusted PR review Isolated container for reviewing untrusted pull requests with Codex or Claude, without exposing your host machine. See `doc/UNTRUSTED-PR-REVIEW.md` for the full workflow. diff --git a/docker/docker-compose.quickstart.yml b/docker/docker-compose.quickstart.yml index 8123f5efb71..25b56f30237 100644 --- a/docker/docker-compose.quickstart.yml +++ b/docker/docker-compose.quickstart.yml @@ -2,7 +2,9 @@ # Copy this file and set the required env vars before running. # # cd docker -# export BETTER_AUTH_SECRET="$(openssl rand -hex 32)" +# Set BETTER_AUTH_SECRET and PAPERCLIP_TOOL_ACTION_SIGNING_SECRET in .env +# once, using a separate `openssl rand -hex 32` value for each. Reuse them +# on restarts; do not replace existing deployment secrets. # export ANTHROPIC_API_KEY="sk-ant-..." # docker compose -f docker-compose.quickstart.yml up --build # @@ -41,6 +43,8 @@ services: # ── Required secret for session signing (generate with: openssl rand -hex 32) ── BETTER_AUTH_SECRET: "${BETTER_AUTH_SECRET:?BETTER_AUTH_SECRET must be set}" + # Independent secret for signed approval-gated tool actions. + PAPERCLIP_TOOL_ACTION_SIGNING_SECRET: "${PAPERCLIP_TOOL_ACTION_SIGNING_SECRET:-}" volumes: # Persistent data directory (database, agent workspaces, uploads) - "${PAPERCLIP_DATA_DIR:-../data/docker-paperclip}:/paperclip" diff --git a/docker/docker-compose.yml b/docker/docker-compose.yml index 03e58dfc52e..06643657122 100644 --- a/docker/docker-compose.yml +++ b/docker/docker-compose.yml @@ -40,6 +40,8 @@ services: PAPERCLIP_DEPLOYMENT_EXPOSURE: "private" PAPERCLIP_PUBLIC_URL: "${PAPERCLIP_PUBLIC_URL:-http://localhost:3100}" BETTER_AUTH_SECRET: "${BETTER_AUTH_SECRET:?BETTER_AUTH_SECRET must be set}" + # Independent secret for signed approval-gated tool actions. + PAPERCLIP_TOOL_ACTION_SIGNING_SECRET: "${PAPERCLIP_TOOL_ACTION_SIGNING_SECRET:-}" volumes: - paperclip-data:/paperclip depends_on: diff --git a/scripts/docker-compose-secrets.test.mjs b/scripts/docker-compose-secrets.test.mjs new file mode 100644 index 00000000000..4704379f31c --- /dev/null +++ b/scripts/docker-compose-secrets.test.mjs @@ -0,0 +1,42 @@ +import assert from "node:assert/strict"; +import { spawnSync } from "node:child_process"; +import { mkdtempSync, rmSync, writeFileSync } from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import test from "node:test"; +import { fileURLToPath } from "node:url"; + +const repoRoot = fileURLToPath(new URL("../", import.meta.url)); +const baseEnv = Object.fromEntries( + ["PATH", "HOME", "SystemRoot", "ComSpec", "TEMP", "TMP", "USERPROFILE"] + .filter((key) => process.env[key] !== undefined) + .map((key) => [key, process.env[key]]), +); +const compose = spawnSync("docker", ["compose", "version"], { env: baseEnv }); +assert.equal(compose.status, 0, "Docker Compose is required for this integration check."); + +for (const [file, service] of [ + ["docker/docker-compose.quickstart.yml", "paperclip"], + ["docker/docker-compose.yml", "server"], +]) { + test(`${file} forwards the independent tool signing secret`, () => { + const temporary = mkdtempSync(path.join(os.tmpdir(), "paperclip-compose-secrets-")); + const envFile = path.join(temporary, "empty.env"); + writeFileSync(envFile, ""); + try { + for (const signingSecret of ["test-tool-signing-secret", undefined]) { + const env = { ...baseEnv, BETTER_AUTH_SECRET: "test-session-secret" }; + if (signingSecret !== undefined) env.PAPERCLIP_TOOL_ACTION_SIGNING_SECRET = signingSecret; + const result = spawnSync("docker", [ + "compose", "--env-file", envFile, "-f", file, "config", "--format", "json", + ], { cwd: repoRoot, env, encoding: "utf8" }); + assert.equal(result.status, 0, result.stderr); + const containerEnv = JSON.parse(result.stdout).services[service].environment; + assert.equal(containerEnv.PAPERCLIP_TOOL_ACTION_SIGNING_SECRET, signingSecret ?? ""); + assert.equal(containerEnv.BETTER_AUTH_SECRET, "test-session-secret"); + } + } finally { + rmSync(temporary, { recursive: true, force: true }); + } + }); +} From 86c965ce7ab1499864f74b8dd6ed0d2d8db15197 Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Thu, 15 Oct 2026 10:20:57 -0700 Subject: [PATCH 13/19] fix(skills): expire unused version caches and restore saved contents (#16433) Expire idle skill-version caches with durable runtime pins, bounded discovery, conservative retention, and database-backed restoration. Co-Authored-By: Dotta Co-Authored-By: Paperclip --- doc/DEVELOPING.md | 42 ++ doc/PRODUCT.md | 4 + doc/SPEC-implementation.md | 6 + .../2026-10-15-recoverable-disk-cleanup.md | 129 +++++ doc/recoverable-disk-cleanup.md | 47 ++ maps/agent-access-map/skills.md | 6 + maps/feature-map/skills.md | 4 + .../company-skills-empty-runtime.test.ts | 9 +- .../src/__tests__/skill-version-cache.test.ts | 267 ++++++++++ .../skill-version-retention-config.test.ts | 26 + .../__tests__/skill-version-retention.test.ts | 370 ++++++++++++++ server/src/config.ts | 21 + server/src/index.ts | 42 ++ server/src/services/company-skills.ts | 58 ++- .../heartbeat/execution-configuration.ts | 5 +- .../src/services/skill-repository-package.ts | 5 +- server/src/services/skill-version-cache.ts | 166 +++++++ .../src/services/skill-version-retention.ts | 458 ++++++++++++++++++ 18 files changed, 1641 insertions(+), 24 deletions(-) create mode 100644 doc/plans/2026-10-15-recoverable-disk-cleanup.md create mode 100644 doc/recoverable-disk-cleanup.md create mode 100644 server/src/__tests__/skill-version-cache.test.ts create mode 100644 server/src/__tests__/skill-version-retention-config.test.ts create mode 100644 server/src/__tests__/skill-version-retention.test.ts create mode 100644 server/src/services/skill-version-cache.ts create mode 100644 server/src/services/skill-version-retention.ts diff --git a/doc/DEVELOPING.md b/doc/DEVELOPING.md index 65f279c1661..09e8038a57b 100644 --- a/doc/DEVELOPING.md +++ b/doc/DEVELOPING.md @@ -1989,6 +1989,41 @@ CI runs `pnpm --filter @paperclipai/skills-catalog validate` and the package's vitest suite, so always regenerate the manifest in the same commit as the catalog change. +## Skill Version Snapshot Retention + +A run uses a skill version from a directory on disk: +`/skills//__versions__///`. The server +writes this directory from the stored files of the `company_skill_versions` row. +The directory is a cache. When the server removes it and the row stays, the next +run that selects the version writes it again. + +The server starts scanning one minute after startup, then hourly, in batches of +50 inspected directories with one-second continuations. Current versions, +explicit agent pins, active runs, and unresolved recovery pins stay. A company +with an active or unsettled run whose pins are unknown is skipped. Optional +`PAPERCLIP_SKILL_VERSION_RETENTION_KEEP` retains additional newest versions +(default `0`). Idle versions expire after `PAPERCLIP_SKILL_VERSION_RETENTION_GRACE_MINUTES` +(default `60`). Separate `__version_access__` records track use; existing caches +without records receive a full observation period before expiration. + +Preparation, durable run pins, audits, and eviction share a company-scoped +PostgreSQL lock across processes. Eligible directories are atomically renamed +to unique managed `.trash-` names. Deletion runs outside the preparation +lock and resumes after a crash. Saved version and repository snapshot rows are +retained, and normal runtime use restores missing caches without upstream access. + +The sweep also handles orphan version directories after explicit skill deletion, +preserving captured run pins until they settle. Only recognized staging/trash +names are eligible; unknown directories and symlinked paths are retained. + +Set `PAPERCLIP_SKILL_VERSION_RETENTION_DRY_RUN=true` before rollout to inspect +candidate bytes and skip reasons without deleting or changing access records. +Set `PAPERCLIP_SKILL_VERSION_RETENTION_ENABLED=false` to disable sweeps while +leaving restoration available. Internal `skillVersionRetentionService` methods +support company-scoped diagnosis and bounded instance-wide continuations. +Server logs report candidates, allocated bytes reclaimed, and skip reasons. +Compare volume usage separately because concurrent writes affect net savings. + ## App-Shipped Teams Catalog The team catalog package mirrors the skills catalog workflow for @@ -2110,6 +2145,13 @@ Environment overrides: non-numeric value falls back to the default. - `PAPERCLIP_WORKSPACE_COMPACTION_ENABLED=false` turns compaction off. Any other value keeps it on. +- `PAPERCLIP_SKILL_VERSION_RETENTION_KEEP=` sets how many of the newest + materialized versions of each skill stay on disk (see "Skill Version Snapshot + Retention" above). The default is `0`. A negative, empty, or non-numeric value + falls back to the default. +- `PAPERCLIP_SKILL_VERSION_RETENTION_GRACE_MINUTES=` sets how long a + materialized skill version stays after its last use. The default is `60`. A + negative, empty, or non-numeric value falls back to the default. Without `PAPERCLIP_DB_BACKUP_ALERT_FILE`, health checks look for `db-backup-to-s3.failure` in the backup directory, beside the backup directory, diff --git a/doc/PRODUCT.md b/doc/PRODUCT.md index 4a5277ddfe4..a3a220d22d9 100644 --- a/doc/PRODUCT.md +++ b/doc/PRODUCT.md @@ -130,6 +130,10 @@ Each execution records the immutable versions it uses; resuming keeps those vers while a new execution resolves latest again. Upstream failures and removals preserve installed snapshots. GitHub refresh does not delay task execution. +Old, unused skill-version disk caches expire automatically while current and in-use +versions remain. Saved database snapshots rebuild an expired copy on demand, including +when its upstream repository is unavailable. Recovery data has separate retention. + Disconnecting removes a repository from Sources after an explicit choice: keep independent editable copies in My Skills, or delete its imported skills. Kept copies and derived forks retain their project access restrictions. Copies preserve diff --git a/doc/SPEC-implementation.md b/doc/SPEC-implementation.md index 658e1859a17..edcdb86eb8c 100644 --- a/doc/SPEC-implementation.md +++ b/doc/SPEC-implementation.md @@ -2084,6 +2084,12 @@ unavailable. Preserve current ownership and newer-work fences. See and history. New key collisions cannot overwrite unrelated skills. Only changed package bytes or executable modes create versions. Library reads, tests, and subsequent unpinned runs use local installed snapshots; pinned/active runs retain their versions. +- Runtime version directories are disposable caches. An hourly, bounded sweep may + evict non-current copies unused for one hour, preserving active and unresolved + execution pins. Selection, materialization, and pin persistence serialize with + eviction across processes. Missing copies rebuild from immutable database bytes + without upstream access. See [recoverable disk cleanup](recoverable-disk-cleanup.md) + for rollout controls, legacy grace periods, and interrupted cleanup recovery. - Manually selected sources retain reviewed selection, with new candidates checked by default and explicit folder exclusions retained. Upstream deletion keeps installed content with removed status; moves are removal plus discovery. Deselect retains content diff --git a/doc/plans/2026-10-15-recoverable-disk-cleanup.md b/doc/plans/2026-10-15-recoverable-disk-cleanup.md new file mode 100644 index 00000000000..d8a11ad79fc --- /dev/null +++ b/doc/plans/2026-10-15-recoverable-disk-cleanup.md @@ -0,0 +1,129 @@ +# Recoverable disk cleanup — delivery state + +## Outcome and boundaries + +Automatically expire reproducible skill caches and finished isolated Git workspaces, +then restore them through normal use. Deliver two reviewed slices and verify actual +reclamation plus restoration on `paperclip.staging.paperclip.app`. Keep the current +storage architecture, unique work, and recovery data. Git restoration may depend +on upstream availability; skill restoration must use saved database bytes alone. + +Apply dotta-kickoff, goal-poteto-style-definition, shorten-feedback-loop, +paperclip-feature-to-production, and test-it-for-real. Retain one deployment owner. + +## Acceptance matrix + +| Case | Required outcome | +| --- | --- | +| Old skill version, upstream unavailable | Identical restored bytes and executable modes | +| Current/active/unsettled skill version | Preserved through concurrent preparation and cleanup | +| Finished Git workspace/base, upstream advanced | Reopen at recorded commit and perform subsequent work | +| Unique files, live consumers, pending recovery | Skip cleanup | +| Restart/concurrent reopen/interrupted deletion | Recover safely and preserve lifecycle fencing | +| Upstream failure/cross-company access | Explicit failure or skip; no fallback or access widening | + +## Delivery slices + +1. Skill `__versions__` cache: one-hour inactivity, current/in-use protection, + persistent access records, cross-process exclusion, hourly batches of 50. +2. Owned isolated Git workspace/base caches: 24-hour terminal-tree cooldown, + verified exact-commit recipe before deletion, safe on-demand restoration. + +Each slice gets focused tests, required repository checks, review, single-workspace +deployment at an exact SHA, dry-run inspection, measured reclamation, and ordinary +product acceptance. Rollback disables deletion and retains restoration support. + +## Current state + +- Checkout: `/Users/dotta/.codex/worktrees/2da2/paperclip` +- Skill PR branch: `fix/skill-version-dir-retention`, based on Devin Foley's + [PR #16433](https://github.com/paperclipai/paperclip/pull/16433). User approved + building on that PR. Added safety commits to its existing branch, preserving + Devin’s original commit; latest published revision before this fix is `6385a28ce7e0ee24bf0bc5e1dc088e7f4abb9508`. + User also authorized merging every PR from this effort after checks pass. +- Earlier proof/implementation preserved at `da754ff744` on + `codex/recoverable-skill-cache`; do not ship its competing worker. +- Staging: `paperclip.staging.paperclip.app`, stack `stack-pool-50531a9d17dd`. +- Staging read-only recheck at 16:07 UTC: 71% disk used, 103,676,784,640 bytes + used, 43,621,777,408 available. Serving `7ec6fd09725fdcd771425e31ce900f8db9aae07d`. + This task has not deployed or deleted staging data. +- A temporary Railway key named `paperclip-cleanup-acceptance-2026-10-15` is + registered under the prior explicit approval. Revoke it and remove both local + copies at acceptance completion (paths retained in private task evidence). +- Existing unresolved staging runs have missing skill-version pins. The approved + conservative policy skips that company; do not clear recovery or weaken the + guard to obtain reclamation. Qualify the skill slice with a disposable pinned + fixture and report the legacy-data gap separately. +- Integrated one worker, cross-process preparation/pin locks, separate last-use + metadata, legacy observation grace, unresolved-recovery protection, dry run, + bounded scans/continuations, and deletion outside the preparation lock. +- First integrated run: 22 focused tests passed; broader controls passed 100/101. + Removed an unnecessary explicit-deletion lock exposed by the rollback fixture; + that control passed on rerun (12.21s). The original failure remains in + `/tmp/paperclip-skill-integration-controls.log`. +- Full typecheck initially found a plugin dependency API mismatch from the + unlocked bootstrap install. Current master supplied the refreshed lockfile; + frozen install completed in 18.7s and full typecheck passed in 59.82s. +- Review at `5eb505ff5d` was 3/5: unsafe skill roots blocked the company sweep, + and recursive byte counts held the preparation lock. Both fixed locally; + 18 affected tests passed in 24.94s, including preparation during a paused file + walk and fresh pin revalidation. Server typecheck passed after those fixes. +- Full tests were stopped after approximately four minutes to address review + findings before final qualification; no full-suite pass claimed. Full build at + `d153929d70` passed. CI at that SHA passed all but the empty-runtime suite and + Greptile (4/5). The remaining review finding was repeated company-wide pin + queries; batching reads twice per sweep phase fixes it and has a query-count + regression. The empty-selection fast path now returns before locking. +- Final affected skill/heartbeat controls: 35 tests passed in 29.17s, log + `/tmp/paperclip-skill-final-controls.log`. A prior local attempt could not start + Postgres because macOS shared-memory slots were full; retry succeeded after a + slot became available, without changing global limits or unrelated services. + Full typecheck on `666c60c859` passed in 32.98s. Full tests are still running; + initial native-session cases hit the same infrastructure limit. Removed 19 + abandoned 56-byte IPC headers only after verifying zero attachments and dead + creator/last-user PIDs. Live databases and their files were untouched. +- Review round four accepted batched reads but found unbounded per-directory + discovery stats. Discovery now uses directory-entry types and applies the cursor + and budget before per-path inspection, including empty-root budget accounting. + Twenty affected tests passed in 22.05s, including filesystem-call counts across + continuation pages (`/tmp/paperclip-skill-discovery-budget.log`). Final review + and full checks remain. Review round five found an exhausted-root continuation + edge case; the cursor now skips completed roots before budgeting discovery. + Twenty-one affected tests passed in 27.98s, including a page-boundary regression + (`/tmp/paperclip-skill-continuation.log`). The server CI shard stopped with + "The operation was canceled" and no failing assertion; investigate its timeout. +- Concurrent Git work is on `codex/recoverable-git-cache` and remains uncommitted. + Actual runtime/service reopening now restores A after deleting the worktree, + branch, and base while upstream is at B. A subsequent commit survives another + reopen. The real repository/DB experiment took 6.831s; evidence is + `/tmp/paperclip-git-recovery-candidate.json`. Expiration is not implemented yet. +- First Git baseline used a real bare remote, runtime workspace creation, a + database row, and ordinary reopening: missing worktree restores A while its + branch exists; missing branch restores upstream B; missing base fails. Fixture + script `/tmp/paperclip-git-recovery-baseline.mts`, inspected output + `/tmp/paperclip-git-recovery-baseline.json`, retained Git fixture + `/tmp/paperclip-git-baseline-sbJWbo`. The test DB was cleaned up. +- Greptile cleared `0da71b2ca2` (all its findings resolved). Security review then + identified repeated full-directory enumeration. Discovery now retains bounded + directory iterators across continuations, counts root entries in its budget, + closes abandoned iterators, and bounds company discovery. Twenty-one focused + cases passed in 29.99s (`/tmp/paperclip-skill-streaming-final.log`); server + typecheck passed. A new regression inspects actual directory reads/opener reuse. +- No deployed artifact or staging acceptance yet. Next: publish review fixes, + qualify final skill SHA, then perform dry-run rollout and measured restoration. + +## Feedback-loop evidence + +| Question | Environment / loop | Elapsed / main wait | Result / next decision | +| --- | --- | --- | --- | +| Can this checkout run focused tests? | Local dependency setup | 43.2s install; later one-time Runner build took several minutes | Baseline manifest/lock drift required install without changing CI-owned lockfile. Warm builds are now reusable. | +| Can an older skill restore without upstream? | Existing service with real Postgres snapshots, pre-cleanup proof at baseline `57fa6b358d` | 4.83s warm test attempt; setup excluded as one-time above | Normal runtime path restored saved binary/script bytes and executable bits with fetch unavailable; proceed with cleanup. | +| Do the integrated safety cases hold? | Devin's `0f3adda980` plus local follow-up; real Postgres and filesystem | 49s from launch through inspected result, 47.23s runner (62% tests, 21% import) | 22 tests passed, server typecheck passed. Added deleted-skill and failure-budget review cases plus heartbeat controls. No cloud proof claimed. | + +| Does Git reopening preserve A after upstream moves? | `5eb505ff5d`, local bare remote plus real runtime/service/database path | 6.398s setup/run/inspection script, about 9s launch through result review | Missing branch returned B; missing base failed. Preserve fixture for exact-commit restoration tests. | +| Does cleanup block preparation during a slow file walk? | Same checkout plus review fixes, two DB clients and a deliberately paused filesystem read | 24.94s affected suite, about 28s through inspected result | 18 tests pass; preparation completes while scan is paused and its new pin prevents eviction. | + +## Deferred + +Skill-version creation/deduplication, recovery retention, non-Git workspace expiry, +shared/operator checkout expiry, and moving storage off the webserver. diff --git a/doc/recoverable-disk-cleanup.md b/doc/recoverable-disk-cleanup.md new file mode 100644 index 00000000000..f27de547ac7 --- /dev/null +++ b/doc/recoverable-disk-cleanup.md @@ -0,0 +1,47 @@ +# Recoverable disk cleanup + +The server materializes immutable skill versions under +`/skills//__versions__//`. These are disposable +disk copies: the company-scoped version and repository snapshot rows retain the +contents, including binary files and executable flags. Normal skill resolution +rebuilds a missing copy from the database without contacting its upstream source. + +## Skill version caches + +An hourly background sweep considers at most 50 directories per batch, yielding +between batches. It starts after server readiness, and respects the existing +worker shutdown, standby, and idle-drain controls. A non-current version becomes +eligible after one hour without use. Current versions, active-run pins and +unsettled recovery pins are retained. If an active run's version selection is +unknown, that company's caches remain. Non-versioned skill directories and +database snapshots are never removed by this worker. + +Last-use records live separately under `__version_access__`. Older installations +without those records receive a full one-hour observation period. Preparation, +pin persistence and eviction share a company-scoped database lock across server +processes. Snapshot validation and byte counting run outside that lock; cleanup +rechecks current versions, run pins, last use, and directory identity before rename. Audits hold the lock through their file reads. Eviction atomically +moves a cache into unique `.trash-` directories; trash deletion uses its own lock so normal +preparation can restore files while deletion finishes. A later sweep finishes +interrupted trash removal without deleting a newly materialized version. + +Controls: + +| Environment variable | Default | Effect | +| --- | --- | --- | +| `PAPERCLIP_SKILL_VERSION_RETENTION_ENABLED` | `true` | Set `false` to stop future sweeps; restoration stays available | +| `PAPERCLIP_SKILL_VERSION_RETENTION_DRY_RUN` | `false` | Set `true` to report candidates without changing cache or access records | + +The internal `skillVersionRetentionService` supports company-scoped diagnosis, +bounded continuations, cancellation between entries, and dry run. Explicit agent +pins stay cached. Operators can also retain newest versions with +`PAPERCLIP_SKILL_VERSION_RETENTION_KEEP` (default `0`), or change the one-hour +window with `PAPERCLIP_SKILL_VERSION_RETENTION_GRACE_MINUTES` (default `60`). Each batch reports +candidate and reclaimed allocated bytes, eviction counts, and skip reasons in +the server log. Compare volume usage before and after to verify physical savings; +concurrent writes and filesystem allocation can affect the net change. A dry run +does not start the legacy observation period. + +Roll out with dry run enabled, inspect candidates/skips, then enable removal. +Recovery archives and runner backups have separate ownership and retention; +this worker never scans or removes them. diff --git a/maps/agent-access-map/skills.md b/maps/agent-access-map/skills.md index 8f8cc177d89..7f1051511af 100644 --- a/maps/agent-access-map/skills.md +++ b/maps/agent-access-map/skills.md @@ -89,6 +89,12 @@ remaining operations before qualifying the whole capability. Gaps: Revision metadata has an API; pinned execution skill versions and installed latest are distinct. +Runtime disk-cache expiration does not delete version records or change access. Normal +execution rematerializes saved bytes after existing skill authorization; cleanup is an +internal company-scoped maintenance operation, with no new agent endpoint. Verify pinned +restoration and active-run protection with `server/src/__tests__/skill-version-cache.test.ts` +and the assembled heartbeat runtime-skill tests; live acceptance remains separate. + ## `policy` Review: `baseline`. diff --git a/maps/feature-map/skills.md b/maps/feature-map/skills.md index 9559dd6d6c4..216bc7203e4 100644 --- a/maps/feature-map/skills.md +++ b/maps/feature-map/skills.md @@ -65,6 +65,10 @@ Manual: Enable the skill for one test agent and leave another without access. Ru - Installed does not mean enabled for every agent. - Read-only GitHub/bundled skills may require a writable copy rather than an in-place edit. - Studio interaction coverage is in [questions and approvals](./questions-and-approvals.md); a passing mock is not a live harness test. +- Old version disk caches may expire; saved versions remain available. Verify a pinned + version is restored through normal execution with upstream unavailable. Current + and in-use versions stay materialized. See [cleanup controls](../../doc/recoverable-disk-cleanup.md) + and the [database-backed cache tests](../../server/src/__tests__/skill-version-cache.test.ts). ### Project repository source behavior diff --git a/server/src/__tests__/company-skills-empty-runtime.test.ts b/server/src/__tests__/company-skills-empty-runtime.test.ts index 369546e4e40..48195e1e63c 100644 --- a/server/src/__tests__/company-skills-empty-runtime.test.ts +++ b/server/src/__tests__/company-skills-empty-runtime.test.ts @@ -1,8 +1,15 @@ import { randomUUID } from "node:crypto"; -import { expect, it } from "vitest"; +import { expect, it, vi } from "vitest"; import type { Db } from "@paperclipai/db"; import { companySkillService } from "../services/company-skills.js"; +// Nonempty selection now first acquires a dedicated connection. Route that +// connection to the same fail-on-access fixture; empty selection must skip it. +vi.mock("@paperclipai/db", async importOriginal => ({ + ...await importOriginal(), + withDedicatedDbConnection: (database: Db, action: (connection: Db) => Promise) => action(database), +})); + const databaseAccess = new Error("Runtime selection touched the inventory database"); const db = new Proxy({} as Db, { get() { throw databaseAccess; } }); diff --git a/server/src/__tests__/skill-version-cache.test.ts b/server/src/__tests__/skill-version-cache.test.ts new file mode 100644 index 00000000000..bd6b1f6cd49 --- /dev/null +++ b/server/src/__tests__/skill-version-cache.test.ts @@ -0,0 +1,267 @@ +import { randomUUID } from "node:crypto"; +import { promises as fs } from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { eq } from "drizzle-orm"; +import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest"; +import { agents, heartbeatRuns, issues, nativeRunFinalizations, companies, companySkills, companySkillVersions, companySkillRepositorySnapshots, createDb } from "@paperclipai/db"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; +import { companySkillService } from "../services/company-skills.js"; +import { skillVersionCachePaths, touchSkillVersionCache, withSkillVersionCacheLock } from "../services/skill-version-cache.js"; + +import { skillVersionRetentionService } from "../services/skill-version-retention.js"; +async function sweepSkillVersionCache(db: ReturnType, options: { companyId: string; now?: number; dryRun?: boolean }) { + const result = await skillVersionRetentionService(db).pruneCompany(options.companyId, { dryRun: options.dryRun, now: options.now === undefined ? undefined : new Date(options.now) }); + return { ...result, evicted: result.removed }; +} + +const support = await getEmbeddedPostgresTestSupport(); +const describePostgres = support.supported ? describe : describe.skip; +if (!support.supported) console.warn(`Skill cache tests require Postgres: ${support.reason}`); + +describePostgres("immutable skill version cache", () => { + let database: Awaited>; + let db: ReturnType; + let home: string; + const previousHome = process.env.PAPERCLIP_HOME; + + beforeAll(async () => { + database = await startEmbeddedPostgresTestDatabase("skill-version-cache-"); + db = createDb(database.connectionString); + home = await fs.mkdtemp(path.join(os.tmpdir(), "skill-version-cache-")); + process.env.PAPERCLIP_HOME = home; + }); + afterEach(() => { vi.unstubAllGlobals(); vi.restoreAllMocks(); }); + afterAll(async () => { + if (previousHome === undefined) delete process.env.PAPERCLIP_HOME; + else process.env.PAPERCLIP_HOME = previousHome; + if (home) await fs.rm(home, { recursive: true, force: true }); + await database?.cleanup(); + }); + + async function fixture() { + const companyId = randomUUID(); + const skillId = randomUUID(); + const key = `company/${companyId}/saved-repository`; + await db.insert(companies).values({ id: companyId, name: "Skill cache", issuePrefix: `S${companyId.slice(0, 6)}` }); + await db.insert(companySkills).values({ id: skillId, companyId, key, slug: "saved-repository", name: "Saved repository", + markdown: "# Saved skill", sourceType: "github", sourceLocator: "https://github.com/example/unavailable", + metadata: { skillSourceId: randomUUID(), snapshotHash: "saved" } }); + const versions = []; + for (const revision of [1, 2]) { + const [snapshot] = await db.insert(companySkillRepositorySnapshots).values({ companyId, contentHash: `revision-${revision}`, + fileInventory: [ + { path: "skills/example/SKILL.md", kind: "skill", content: `---\nname: saved-repository\ndescription: Saved revision ${revision}\n---\nRun ../../scripts/example.sh.\n` }, + { path: "scripts/example.sh", kind: "script", content: `#!/bin/sh\necho revision-${revision}\n`, executable: true }, + { path: "marker.bin", kind: "reference", content: Buffer.from([0, revision, 255]).toString("base64"), encoding: "base64" }, + ] }).returning(); + const [version] = await db.insert(companySkillVersions).values({ companyId, companySkillId: skillId, revisionNumber: revision, + repositorySnapshotId: snapshot!.id, repositorySkillPath: "skills/example/SKILL.md" }).returning(); + versions.push(version!); + } + await db.update(companySkills).set({ currentVersionId: versions[1]!.id }).where(eq(companySkills.id, skillId)); + const service = companySkillService(db); + async function resolve(versionId: string) { + const entries = await service.listRuntimeSkillEntries(companyId, { onlyKeys: new Set([key]), versionSelections: new Map([[key, versionId]]) }); + expect(entries).toHaveLength(1); + expect(entries[0]!.sourceStatus, entries[0]!.missingDetail ?? undefined).toBe("available"); + return entries[0]!; + } + return { companyId, skillId, key, versions, resolve }; + } + + it("rebuilds an old repository-backed version from database bytes with upstream unavailable", async () => { + const upstream = vi.fn(() => Promise.reject(new Error("Upstream is unavailable"))); + vi.stubGlobal("fetch", upstream); + const { versions, resolve } = await fixture(); + const old = await resolve(versions[0]!.id); + const current = await resolve(versions[1]!.id); + await fs.rm(old.source, { recursive: true }); + const restored = await resolve(versions[0]!.id); + expect(restored.source).toBe(old.source); + const repository = path.join(restored.source, ".paperclip-repository"); + expect(await fs.readFile(path.join(repository, "marker.bin"))).toEqual(Buffer.from([0, 1, 255])); + expect(await fs.readFile(path.join(repository, "scripts/example.sh"), "utf8")).toBe("#!/bin/sh\necho revision-1\n"); + expect((await fs.stat(path.join(repository, "scripts/example.sh"))).mode & 0o111).toBe(0o111); + expect(await fs.readFile(path.join(current.source, ".paperclip-repository/marker.bin"))).toEqual(Buffer.from([0, 2, 255])); + expect(upstream).not.toHaveBeenCalled(); + }); + + it("expires only idle old versions, reports dry-run candidates, and restores the evicted version", async () => { + const f = await fixture(); + const old = await f.resolve(f.versions[0]!.id); + const current = await f.resolve(f.versions[1]!.id); + const aged = Date.now() - 2 * 60 * 60 * 1000; + await touchSkillVersionCache(f.companyId, f.skillId, f.versions[0]!.id, aged); + await touchSkillVersionCache(f.companyId, f.skillId, f.versions[1]!.id, aged); + const dry = await sweepSkillVersionCache(db, { companyId: f.companyId, dryRun: true }); + expect(dry).toMatchObject({ candidates: 1, evicted: 0, reclaimedBytes: 0 }); + expect(await fs.stat(old.source)).toBeTruthy(); + const swept = await sweepSkillVersionCache(db, { companyId: f.companyId }); + expect(swept).toMatchObject({ candidates: 1, evicted: 1 }); + expect(swept.reclaimedBytes).toBeGreaterThan(0); + await expect(fs.stat(old.source)).rejects.toMatchObject({ code: "ENOENT" }); + expect(await fs.stat(current.source)).toBeTruthy(); + vi.stubGlobal("fetch", vi.fn(() => Promise.reject(new Error("Upstream unavailable")))); + expect(await f.resolve(f.versions[0]!.id)).toMatchObject({ source: old.source }); + expect((await sweepSkillVersionCache(db, { companyId: f.companyId })).evicted).toBe(0); + }); + + it("gives existing unobserved caches a full grace period", async () => { + const f = await fixture(); + await f.resolve(f.versions[0]!.id); + const paths = skillVersionCachePaths(f.companyId, f.skillId, f.versions[0]!.id); + await fs.unlink(paths.access); + const now = Date.now(); + expect((await sweepSkillVersionCache(db, { companyId: f.companyId, now })).skipped.observation_grace).toBe(1); + expect((await sweepSkillVersionCache(db, { companyId: f.companyId, now: now + 59 * 60 * 1000 })).evicted).toBe(0); + expect((await sweepSkillVersionCache(db, { companyId: f.companyId, now: now + 61 * 60 * 1000 })).evicted).toBe(1); + }); + + async function activeRun(companyId: string, contextSnapshot: Record | null) { + const [agent] = await db.insert(agents).values({ companyId, name: "Cache reader", role: "engineer", adapterType: "process" }).returning(); + const [run] = await db.insert(heartbeatRuns).values({ companyId, agentId: agent!.id, status: "running", contextSnapshot }).returning(); + return run!; + } + + it("retains pinned versions and skips companies with unrecorded active selections", async () => { + const f = await fixture(); + await f.resolve(f.versions[0]!.id); + await touchSkillVersionCache(f.companyId, f.skillId, f.versions[0]!.id, Date.now() - 7_200_000); + const run = await activeRun(f.companyId, { skillVersions: { [f.key]: f.versions[0]!.id } }); + expect((await sweepSkillVersionCache(db, { companyId: f.companyId })).skipped.protected_version).toBe(1); + await db.update(heartbeatRuns).set({ contextSnapshot: null }).where(eq(heartbeatRuns.id, run.id)); + expect((await sweepSkillVersionCache(db, { companyId: f.companyId })).skipped.unknown_run_pins).toBe(1); + await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date() }).where(eq(heartbeatRuns.id, run.id)); + expect((await sweepSkillVersionCache(db, { companyId: f.companyId })).evicted).toBe(1); + }); + + it("serializes eviction behind preparation and durable pinning across database clients", async () => { + const f = await fixture(); + await f.resolve(f.versions[0]!.id); + const run = await activeRun(f.companyId, null); + const ready = Promise.withResolvers(); + const release = Promise.withResolvers(); + const preparing = withSkillVersionCacheLock(db, f.companyId, async () => { + await f.resolve(f.versions[0]!.id); + await touchSkillVersionCache(f.companyId, f.skillId, f.versions[0]!.id, Date.now() - 7_200_000); + ready.resolve(); + await release.promise; + await db.update(heartbeatRuns).set({ contextSnapshot: { skillVersions: { [f.key]: f.versions[0]!.id } } }) + .where(eq(heartbeatRuns.id, run.id)); + }); + await ready.promise; + const other = createDb(database.connectionString); + try { + const cleaning = await sweepSkillVersionCache(other, { companyId: f.companyId }); + expect(cleaning.evicted).toBe(0); + expect(cleaning.skipped.cache_busy).toBeGreaterThan(0); + } finally { release.resolve(); } + await preparing; + expect(await sweepSkillVersionCache(other, { companyId: f.companyId })).toMatchObject({ evicted: 0, skipped: { protected_version: 1 } }); + }); + + it("reads company-wide protection twice per batch regardless of version count", async () => { + const f = await fixture(); + const versionIds = [f.versions[0]!.id]; + for (let revision = 3; revision < 8; revision++) { + const [version] = await db.insert(companySkillVersions).values({ companyId: f.companyId, + companySkillId: f.skillId, revisionNumber: revision, repositorySnapshotId: f.versions[0]!.repositorySnapshotId, + repositorySkillPath: "skills/example/SKILL.md" }).returning(); + versionIds.push(version!.id); + } + for (const id of [...versionIds, f.versions[1]!.id]) { + await f.resolve(id); + await touchSkillVersionCache(f.companyId, f.skillId, id, Date.now() - 7_200_000); + } + const statements: string[] = []; + const debug = db.$client.options.debug; + db.$client.options.debug = (_connection, query) => { statements.push(query); }; + try { + expect(await sweepSkillVersionCache(db, { companyId: f.companyId })).toMatchObject({ evicted: 6 }); + expect(statements.filter(query => /from "agents"/.test(query))).toHaveLength(2); + expect(statements.filter(query => /from "heartbeat_runs"/.test(query))).toHaveLength(2); + } finally { db.$client.options.debug = debug; } + }); + + it("lets preparation finish during slow byte inspection and rechecks its new pin", async () => { + const f = await fixture(); + const old = await f.resolve(f.versions[0]!.id); + await touchSkillVersionCache(f.companyId, f.skillId, f.versions[0]!.id, Date.now() - 7_200_000); + const inspecting = Promise.withResolvers(); + const release = Promise.withResolvers(); + const lstat = fs.lstat.bind(fs); + let held = false; + vi.spyOn(fs, "lstat").mockImplementation((async (...args: Parameters) => { + if (!held && String(args[0]) === path.join(old.source, ".paperclip-repository", "marker.bin")) { + held = true; + inspecting.resolve(); + await release.promise; + } + return lstat(...args); + }) as typeof fs.lstat); + const sweep = sweepSkillVersionCache(db, { companyId: f.companyId }); + await inspecting.promise; + const other = createDb(database.connectionString); + let timeout: ReturnType | undefined; + const preparation = withSkillVersionCacheLock(other, f.companyId, async () => { + await companySkillService(other).listRuntimeSkillEntries(f.companyId, { + onlyKeys: new Set([f.key]), versionSelections: new Map([[f.key, f.versions[0]!.id]]), + }); + // Remove grace as an alternative explanation: the newly persisted run pin + // must prevent eviction after the original candidate snapshot was read. + await touchSkillVersionCache(f.companyId, f.skillId, f.versions[0]!.id, Date.now() - 7_200_000); + await activeRun(f.companyId, { skillVersions: { [f.key]: f.versions[0]!.id } }); + return true; + }); + try { + expect(await Promise.race([preparation, new Promise(resolve => { + timeout = setTimeout(() => resolve(false), 10_000); + })])).toBe(true); + } finally { clearTimeout(timeout); release.resolve(); } + await preparation; + expect(await sweep).toMatchObject({ evicted: 0, skipped: { protected_version: 1 } }); + expect(await fs.stat(old.source)).toBeTruthy(); + await other.$client.end({ timeout: 1 }); + }); + + it("keeps terminal run pins until native finalization has settled", async () => { + const f = await fixture(); + await f.resolve(f.versions[0]!.id); + await touchSkillVersionCache(f.companyId, f.skillId, f.versions[0]!.id, Date.now() - 7_200_000); + const run = await activeRun(f.companyId, { skillVersions: { [f.key]: f.versions[0]!.id } }); + const [issue] = await db.insert(issues).values({ companyId: f.companyId, title: "Pending recovery" }).returning(); + await db.update(heartbeatRuns).set({ runtimeMode: "native", nativeIssueId: issue!.id, status: "failed", finishedAt: new Date() }) + .where(eq(heartbeatRuns.id, run.id)); + await db.insert(nativeRunFinalizations).values({ companyId: f.companyId, issueId: issue!.id, runId: run.id, phase: "terminal_failure" }); + expect((await sweepSkillVersionCache(db, { companyId: f.companyId })).skipped.protected_version).toBe(1); + await db.update(nativeRunFinalizations).set({ phase: "committed" }).where(eq(nativeRunFinalizations.runId, run.id)); + expect((await sweepSkillVersionCache(db, { companyId: f.companyId })).evicted).toBe(1); + }); + + it("finishes interrupted trash removal without deleting a rehydrated copy", async () => { + const f = await fixture(); + const old = await f.resolve(f.versions[0]!.id); + const paths = skillVersionCachePaths(f.companyId, f.skillId, f.versions[0]!.id); + const trash = path.join(path.dirname(paths.directory), `.trash-${randomUUID()}`); + await fs.rename(old.source, trash); + await f.resolve(f.versions[0]!.id); + const result = await sweepSkillVersionCache(db, { companyId: f.companyId }); + expect(result.evicted).toBe(0); + expect(result.reclaimedBytes).toBeGreaterThan(0); + await expect(fs.stat(trash)).rejects.toMatchObject({ code: "ENOENT" }); + expect(await fs.stat(old.source)).toBeTruthy(); + }); + + it("does not traverse symlinked cache roots", async () => { + const f = await fixture(); + const old = await f.resolve(f.versions[0]!.id); + await touchSkillVersionCache(f.companyId, f.skillId, f.versions[0]!.id, Date.now() - 7_200_000); + const outside = await fs.mkdtemp(path.join(home, "outside-")); + await fs.writeFile(path.join(outside, "unique.txt"), "keep"); + await fs.rm(old.source, { recursive: true }); + await fs.symlink(outside, old.source); + expect((await sweepSkillVersionCache(db, { companyId: f.companyId })).skipped.unsafe_or_unavailable).toBe(1); + expect(await fs.readFile(path.join(outside, "unique.txt"), "utf8")).toBe("keep"); + }); +}); diff --git a/server/src/__tests__/skill-version-retention-config.test.ts b/server/src/__tests__/skill-version-retention-config.test.ts new file mode 100644 index 00000000000..6205e17d8f7 --- /dev/null +++ b/server/src/__tests__/skill-version-retention-config.test.ts @@ -0,0 +1,26 @@ +import { afterEach, describe, expect, it, vi } from "vitest"; +import { loadConfig } from "../config.ts"; + +describe("skill version retention config parsing", () => { + afterEach(() => { + vi.unstubAllEnvs(); + }); + + it("uses the defaults when the variables are not set", () => { + vi.stubEnv("PAPERCLIP_SKILL_VERSION_RETENTION_KEEP", undefined); + vi.stubEnv("PAPERCLIP_SKILL_VERSION_RETENTION_GRACE_MINUTES", undefined); + expect(loadConfig()).toMatchObject({ skillVersionRetentionKeepNewest: 0, skillVersionRetentionGraceMinutes: 60 }); + }); + + it("accepts explicit non-negative values, including 0", () => { + vi.stubEnv("PAPERCLIP_SKILL_VERSION_RETENTION_KEEP", " 0 "); + vi.stubEnv("PAPERCLIP_SKILL_VERSION_RETENTION_GRACE_MINUTES", "30"); + expect(loadConfig()).toMatchObject({ skillVersionRetentionKeepNewest: 0, skillVersionRetentionGraceMinutes: 30 }); + }); + + it.each(["", " ", "-1", "many"])("falls back to the defaults for %j", (value) => { + vi.stubEnv("PAPERCLIP_SKILL_VERSION_RETENTION_KEEP", value); + vi.stubEnv("PAPERCLIP_SKILL_VERSION_RETENTION_GRACE_MINUTES", value); + expect(loadConfig()).toMatchObject({ skillVersionRetentionKeepNewest: 0, skillVersionRetentionGraceMinutes: 60 }); + }); +}); diff --git a/server/src/__tests__/skill-version-retention.test.ts b/server/src/__tests__/skill-version-retention.test.ts new file mode 100644 index 00000000000..05d4d0fbfd7 --- /dev/null +++ b/server/src/__tests__/skill-version-retention.test.ts @@ -0,0 +1,370 @@ +import { touchSkillVersionCache } from "../services/skill-version-cache.js"; +import { randomUUID } from "node:crypto"; +import os from "node:os"; +import path from "node:path"; +import { promises as fs } from "node:fs"; +import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest"; +import { eq } from "drizzle-orm"; +import { + agents, + companies, + companySkills, + companySkillVersions, + createDb, + heartbeatRuns, +} from "@paperclipai/db"; +import { + getEmbeddedPostgresTestSupport, + startEmbeddedPostgresTestDatabase, +} from "./helpers/embedded-postgres.js"; +import { companySkillService } from "../services/company-skills.ts"; +import { + resolveCompanySkillVersionsRoot, + runVersionSnapshotFlight, + skillVersionRetentionService, +} from "../services/skill-version-retention.ts"; + +const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport(); +const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip; + +if (!embeddedPostgresSupport.supported) { + console.warn( + `Skipping embedded Postgres skill version retention tests on this host: ${embeddedPostgresSupport.reason ?? "unsupported environment"}`, + ); +} + +const HOUR_MS = 60 * 60 * 1000; + +async function exists(target: string) { + return fs.lstat(target).then(() => true, () => false); +} + +async function makeWritable(target: string): Promise { + const stat = await fs.lstat(target).catch(() => null); + if (!stat?.isDirectory()) return; + await fs.chmod(target, 0o755); + for (const entry of await fs.readdir(target)) await makeWritable(path.join(target, entry)); +} + +describeEmbeddedPostgres("skill version snapshot retention", () => { + let db!: ReturnType; + let tempDb: Awaited> | null = null; + let oldPaperclipHome: string | undefined; + let oldPaperclipInstanceId: string | undefined; + let paperclipHome: string | null = null; + + beforeAll(async () => { + tempDb = await startEmbeddedPostgresTestDatabase("paperclip-skill-version-retention-"); + oldPaperclipHome = process.env.PAPERCLIP_HOME; + oldPaperclipInstanceId = process.env.PAPERCLIP_INSTANCE_ID; + paperclipHome = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-skill-version-retention-home-")); + process.env.PAPERCLIP_HOME = paperclipHome; + process.env.PAPERCLIP_INSTANCE_ID = "default"; + db = createDb(tempDb.connectionString); + }, 20_000); + + afterEach(async () => { + vi.restoreAllMocks(); + await db.delete(heartbeatRuns); + await db.delete(agents); + await db.delete(companySkills); + await db.delete(companies); + if (paperclipHome) { + const skillsRoot = path.join(paperclipHome, "instances", "default", "skills"); + await makeWritable(skillsRoot); + await fs.rm(skillsRoot, { recursive: true, force: true }); + } + }); + + afterAll(async () => { + if (oldPaperclipHome === undefined) delete process.env.PAPERCLIP_HOME; + else process.env.PAPERCLIP_HOME = oldPaperclipHome; + if (oldPaperclipInstanceId === undefined) delete process.env.PAPERCLIP_INSTANCE_ID; + else process.env.PAPERCLIP_INSTANCE_ID = oldPaperclipInstanceId; + if (paperclipHome) await fs.rm(paperclipHome, { recursive: true, force: true }); + await tempDb?.cleanup(); + }); + + async function createCompany() { + const companyId = randomUUID(); + await db.insert(companies).values({ id: companyId, name: "Retention", issuePrefix: `R${companyId.slice(0, 6)}` }); + return companyId; + } + + async function createSkill(companyId: string, versionCount: number, currentIndex = versionCount - 1) { + const skillId = randomUUID(); + const key = `company/${companyId}/retained-${skillId.slice(0, 8)}`; + await db.insert(companySkills).values({ + id: skillId, companyId, key, slug: `retained-${skillId.slice(0, 8)}`, name: "Retained", + markdown: "# Retained", sourceType: "github", sourceLocator: "https://github.com/acme/retained", + sourceRef: "a".repeat(40), trustLevel: "markdown_only", compatibility: "compatible", + metadata: {}, fileInventory: [{ path: "SKILL.md", kind: "skill" }], + }); + const versionIds: string[] = []; + for (let index = 0; index < versionCount; index++) { + const [version] = await db.insert(companySkillVersions).values({ + companyId, companySkillId: skillId, revisionNumber: index + 1, label: `v${index + 1}`, + fileInventory: [ + { path: "SKILL.md", kind: "skill", content: `---\nname: retained\n---\n\n# Revision ${index + 1}\n` }, + { path: "references/notes.md", kind: "reference", content: `notes ${index + 1}` }, + ], + }).returning({ id: companySkillVersions.id }); + versionIds.push(version!.id); + } + if (versionCount > 0) { + await db.update(companySkills).set({ currentVersionId: versionIds[currentIndex]! }) + .where(eq(companySkills.id, skillId)); + } + return { skillId, key, versionIds }; + } + + /** Writes a read-only snapshot directory like materializeVersionSnapshot does, aged by `ageMs`. */ + async function writeSnapshot(companyId: string, skillId: string, name: string, ageMs = 3 * HOUR_MS) { + const dir = path.join(resolveCompanySkillVersionsRoot(companyId), skillId, name); + await fs.mkdir(path.join(dir, "references"), { recursive: true }); + await fs.writeFile(path.join(dir, "SKILL.md"), "# Snapshot", { mode: 0o444 }); + await fs.writeFile(path.join(dir, "references", "notes.md"), "notes", { mode: 0o444 }); + await fs.chmod(path.join(dir, "references"), 0o555); + const when = new Date(Date.now() - ageMs); + await fs.utimes(dir, when, when); + if (/^[a-f0-9-]{36}$/.test(name)) await touchSkillVersionCache(companyId, skillId, name, when.getTime()); + return dir; + } + + it("keeps current, pinned, live-run, newest, and recent versions and removes superseded ones", async () => { + const companyId = await createCompany(); + const { skillId, key, versionIds: v } = await createSkill(companyId, 7); + const [agent] = await db.insert(agents).values({ + companyId, name: "Pinned agent", + adapterConfig: { paperclipSkillSync: { desiredSkills: [{ key, versionId: v[0] }] } }, + }).returning({ id: agents.id }); + await db.insert(heartbeatRuns).values([ + { companyId, agentId: agent!.id, status: "running", contextSnapshot: { skillVersions: { [key]: v[1] } } }, + { companyId, agentId: agent!.id, status: "succeeded", finishedAt: new Date(), contextSnapshot: { skillVersions: { [key]: v[2] } } }, + ]); + const dirs = await Promise.all(v.map((id, index) => writeSnapshot(companyId, skillId, id, index === 3 ? 0 : 3 * HOUR_MS))); + + const result = await skillVersionRetentionService(db).pruneCompany(companyId, { keepNewest: 2 }); + + // v1 pinned, v2 live run, v4 recent, v6 + v7 newest (v7 current). + expect(await Promise.all(dirs.map(exists))).toEqual([true, true, false, true, false, true, true]); + expect(result).toMatchObject({ removed: 2, removedOrphans: 0, skippedRecent: 1, failed: 0, truncated: false }); + // Rows are never deleted. + expect(await db.select().from(companySkillVersions).where(eq(companySkillVersions.companySkillId, skillId))).toHaveLength(7); + }); + + it("removes orphan version folders, orphan skill folders, and stale staging folders", async () => { + const companyId = await createCompany(); + const { skillId, versionIds } = await createSkill(companyId, 1); + const root = resolveCompanySkillVersionsRoot(companyId); + const current = await writeSnapshot(companyId, skillId, versionIds[0]!); + const orphanVersion = await writeSnapshot(companyId, skillId, randomUUID()); + const orphanSkillId = randomUUID(); + const orphanSkillVersion = await writeSnapshot(companyId, orphanSkillId, randomUUID()); + const staleStaging = await writeSnapshot(companyId, skillId, `.${versionIds[0]}-abc123`); + const freshStaging = await writeSnapshot(companyId, skillId, `.${versionIds[0]}-def456`, 0); + const leftoverTrash = await writeSnapshot(companyId, skillId, `\.trash-${randomUUID()}`, 0); + const unknown = await writeSnapshot(companyId, skillId, "operator-notes"); + const hidden = await writeSnapshot(companyId, skillId, ".notes"); + + const result = await skillVersionRetentionService(db).pruneCompany(companyId); + + expect(await exists(current)).toBe(true); + expect(await exists(orphanVersion)).toBe(false); + expect(await exists(orphanSkillVersion)).toBe(false); + expect(await fs.readdir(path.join(root, orphanSkillId))).toEqual([]); + expect(await exists(staleStaging)).toBe(false); + expect(await exists(freshStaging)).toBe(true); + expect(await exists(leftoverTrash)).toBe(false); + expect(await exists(unknown)).toBe(true); + expect(await exists(hidden)).toBe(true); + expect(result).toMatchObject({ removedOrphans: 2, removedStaging: 1, failed: 0 }); + expect((await fs.readdir(path.join(root, skillId))).filter(name => name.startsWith(".trash-"))).toEqual([]); + }); + + it("re-materializes a pruned version from its row on the next resolve", async () => { + const companyId = await createCompany(); + const { key, versionIds } = await createSkill(companyId, 5); + const svc = companySkillService(db); + const selections = new Map([[key, versionIds[0]!]]); + const first = (await svc.listRuntimeSkillEntries(companyId, { versionSelections: selections })) + .find(entry => entry.key === key)!; + expect(first.sourceStatus).toBe("available"); + const old = new Date(Date.now() - 3 * HOUR_MS); + await touchSkillVersionCache(companyId, path.basename(path.dirname(first.source)), path.basename(first.source), old.getTime()); + + await skillVersionRetentionService(db).pruneCompany(companyId, { keepNewest: 1 }); + expect(await exists(first.source)).toBe(false); + + const again = (await svc.listRuntimeSkillEntries(companyId, { versionSelections: selections })) + .find(entry => entry.key === key)!; + expect(again).toMatchObject({ sourceStatus: "available", source: first.source }); + expect(await fs.readFile(path.join(again.source, "SKILL.md"), "utf8")).toContain("# Revision 1"); + }); + + it("refreshes the age of a reused snapshot so the grace window protects it", async () => { + const companyId = await createCompany(); + const { key, versionIds } = await createSkill(companyId, 5); + const svc = companySkillService(db); + const selections = new Map([[key, versionIds[0]!]]); + const first = (await svc.listRuntimeSkillEntries(companyId, { versionSelections: selections })) + .find(entry => entry.key === key)!; + const old = new Date(Date.now() - 3 * HOUR_MS); + await touchSkillVersionCache(companyId, path.basename(path.dirname(first.source)), path.basename(first.source), old.getTime()); + await svc.listRuntimeSkillEntries(companyId, { versionSelections: selections }); + + const result = await skillVersionRetentionService(db).pruneCompany(companyId, { keepNewest: 1 }); + expect(result.skippedRecent).toBe(1); + expect(await exists(first.source)).toBe(true); + }); + + it("skips a version folder while a materialization holds it", async () => { + const companyId = await createCompany(); + const { skillId, versionIds } = await createSkill(companyId, 3); + const dir = await writeSnapshot(companyId, skillId, versionIds[0]!); + let release!: () => void; + const held = runVersionSnapshotFlight(dir, () => new Promise(resolve => { release = () => resolve(dir); })); + + await skillVersionRetentionService(db).pruneCompany(companyId, { keepNewest: 1 }); + expect(await exists(dir)).toBe(true); + release(); + await held; + await skillVersionRetentionService(db).pruneCompany(companyId, { keepNewest: 1 }); + expect(await exists(dir)).toBe(false); + }); + + it("sweeps deleted skills after protecting existing runs", async () => { + const companyId = await createCompany(); + const { skillId, versionIds } = await createSkill(companyId, 2); + const first = await writeSnapshot(companyId, skillId, versionIds[0]!); + const second = await writeSnapshot(companyId, skillId, versionIds[1]!); + const [agent] = await db.insert(agents).values({ companyId, name: "Detached agent" }).returning(); + const [run] = await db.insert(heartbeatRuns).values({ companyId, agentId: agent!.id, status: "running", + contextSnapshot: { skillVersions: { retained: versionIds[0] } } }).returning(); + await companySkillService(db).deleteSkill(companyId, skillId); + expect(await exists(first)).toBe(true); + await skillVersionRetentionService(db).pruneCompany(companyId); + expect(await exists(first)).toBe(true); + expect(await exists(second)).toBe(false); + await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date() }).where(eq(heartbeatRuns.id, run!.id)); + await skillVersionRetentionService(db).pruneCompany(companyId); + expect(await exists(first)).toBe(false); + }); + + it("never removes anything outside the company's versions root", async () => { + const companyId = await createCompany(); + const otherCompanyId = await createCompany(); + const { skillId, versionIds } = await createSkill(companyId, 3); + const other = await createSkill(otherCompanyId, 3); + const otherDirs = await Promise.all(other.versionIds.map(id => writeSnapshot(otherCompanyId, other.skillId, id))); + const outside = await fs.mkdtemp(path.join(paperclipHome!, "outside-")); + await fs.writeFile(path.join(outside, "keep.txt"), "keep"); + const root = resolveCompanySkillVersionsRoot(companyId); + const victim = await writeSnapshot(companyId, skillId, versionIds[0]!); + const safe = await writeSnapshot(companyId, skillId, versionIds[1]!); + await fs.chmod(victim, 0o755); + await fs.symlink(outside, path.join(victim, "escape")); + const old = new Date(Date.now() - 3 * HOUR_MS); + await fs.utimes(victim, old, old); + await fs.mkdir(root, { recursive: true }); + await fs.symlink(outside, path.join(root, randomUUID())); + await fs.symlink(outside, path.join(root, skillId, randomUUID())); + + await skillVersionRetentionService(db).pruneCompany(companyId, { keepNewest: 1 }); + + expect(await exists(victim)).toBe(true); + expect(await exists(safe)).toBe(false); + expect(await fs.readFile(path.join(outside, "keep.txt"), "utf8")).toBe("keep"); + expect(await Promise.all(otherDirs.map(exists))).toEqual([true, true, true]); + }); + + it("bounds removals per tick and continues on the next tick", async () => { + const companyId = await createCompany(); + const { skillId, versionIds } = await createSkill(companyId, 6); + await Promise.all(versionIds.map(id => writeSnapshot(companyId, skillId, id))); + const retention = skillVersionRetentionService(db, { keepNewest: 1, maxRemovals: 2 }); + + let total = 0; + for (let index = 0; index < 5; index++) { + const result = await retention.tick(); + expect(result.checked).toBeLessThanOrEqual(2); + total += result.removed; + if (!result.truncated) break; + } + expect(total).toBe(5); + expect(await fs.readdir(path.join(resolveCompanySkillVersionsRoot(companyId), skillId))).toEqual([versionIds[5]]); + }); + it("bounds directory reads and retains the discovery cursor between pages", async () => { + const companyId = await createCompany(); + const { skillId, versionIds } = await createSkill(companyId, 100); + await Promise.all(versionIds.map(id => writeSnapshot(companyId, skillId, id))); + const parent = path.join(resolveCompanySkillVersionsRoot(companyId), skillId); + const originalOpen = fs.opendir.bind(fs); + let reads = 0; + const opened = vi.spyOn(fs, "opendir").mockImplementation(async (...args: Parameters) => { + const directory = await originalOpen(...args); + const read = directory.read.bind(directory); + vi.spyOn(directory, "read").mockImplementation(async () => { reads++; return read(); }); + return directory; + }); + const stat = vi.spyOn(fs, "lstat"); + const retention = skillVersionRetentionService(db, { maxEntries: 2, dryRun: true }); + try { + const first = await retention.tick(); + expect(first).toMatchObject({ checked: 2, truncated: true }); + expect(reads).toBeLessThanOrEqual(2); + const inspected = () => new Set(stat.mock.calls.map(call => String(call[0])) + .filter(file => path.dirname(file) === parent).map(file => path.basename(file))); + const firstPaths = inspected(); + expect(firstPaths.size).toBe(1); // Opening the skill root also consumes a slot. + stat.mockClear(); reads = 0; + const second = await retention.tick(); + expect(second).toMatchObject({ checked: 2, truncated: true }); + expect(reads).toBe(2); + expect(inspected().size).toBe(2); + expect([...inspected()].some(name => firstPaths.has(name))).toBe(false); + expect(opened.mock.calls.filter(call => call[0] === parent)).toHaveLength(1); + } finally { await retention.dispose(); } + }); + + it("continues beyond a page ending at an exhausted skill root", async () => { + const companyId = await createCompany(); + const roots = [1, 2, 3].map(value => `00000000-0000-4000-8000-${String(value).padStart(12, "0")}`); + await Promise.all([randomUUID(), randomUUID()].map(id => writeSnapshot(companyId, roots[0]!, id, 0))); + await fs.mkdir(path.join(resolveCompanySkillVersionsRoot(companyId), roots[1]!), { recursive: true }); + const versions = Array.from({ length: 5 }, () => randomUUID()).sort(); + const paths = await Promise.all(versions.map((id, index) => writeSnapshot(companyId, roots[2]!, id, index === 4 ? 3 * HOUR_MS : 0))); + const retention = skillVersionRetentionService(db, { maxEntries: 3 }); + let reclaimed = 0; + for (let page = 0; page < 5; page++) { + const result = await retention.tick(); + expect(result.checked).toBeLessThanOrEqual(3); + reclaimed += result.removed; + if (!result.truncated) break; + } + expect(reclaimed).toBe(1); + expect(await exists(paths[4]!)).toBe(false); + expect(await Promise.all(paths.slice(0, 4).map(exists))).toEqual([true, true, true, true]); + }); + + it("bounds failed attempts and continues beyond them", async () => { + const companyId = await createCompany(); + const { skillId, versionIds } = await createSkill(companyId, 6); + await Promise.all(versionIds.map(id => writeSnapshot(companyId, skillId, id))); + const rename = vi.spyOn(fs, "rename").mockRejectedValue(Object.assign(new Error("read-only fixture"), { code: "EACCES" })); + const retention = skillVersionRetentionService(db, { maxRemovals: 2 }); + let checked = 0; + let failures = 0; + for (let index = 0; index < 5; index++) { + const result = await retention.tick(); + expect(result.checked).toBeLessThanOrEqual(2); + checked += result.checked; + failures += result.failed; + if (!result.truncated) break; + } + expect(checked).toBe(7); // Six versions plus the skill-root discovery slot. + expect(failures).toBe(5); + expect(rename).toHaveBeenCalledTimes(5); + }); + +}); diff --git a/server/src/config.ts b/server/src/config.ts index 69860d95f96..8ea462b8fbe 100644 --- a/server/src/config.ts +++ b/server/src/config.ts @@ -81,6 +81,10 @@ export interface Config { databaseBackupDir: string; workspaceReaperCooldownDays: number; workspaceCompactionDelayHours: number | null; + skillVersionRetentionEnabled: boolean; + skillVersionRetentionDryRun: boolean; + skillVersionRetentionKeepNewest: number; + skillVersionRetentionGraceMinutes: number; serveUi: boolean; uiDevMiddleware: boolean; secretsProvider: SecretProvider; @@ -313,6 +317,19 @@ export function loadConfig(): Config { && workspaceCompactionDelayHoursRaw >= 0 ? workspaceCompactionDelayHoursRaw : 1; + // Skill version retention removes materialized version directories under + // `skills//__versions__/` that no current version, agent pin, or + // live run uses. It keeps this many newest versions per skill as a warm + // cache. A version directory used within the grace window also stays. A + // removed directory is written again from its version row on next use. + // Negative, empty, or non-numeric values fall back to the defaults. + const nonNegativeIntegerEnv = (name: string, fallback: number) => { + const raw = process.env[name]?.trim(); + const value = Number(raw); + return raw && Number.isFinite(value) && value >= 0 ? Math.floor(value) : fallback; + }; + const skillVersionRetentionKeepNewest = nonNegativeIntegerEnv("PAPERCLIP_SKILL_VERSION_RETENTION_KEEP", 0); + const skillVersionRetentionGraceMinutes = nonNegativeIntegerEnv("PAPERCLIP_SKILL_VERSION_RETENTION_GRACE_MINUTES", 60); const bindValidationErrors = validateConfiguredBindMode({ deploymentMode, deploymentExposure, @@ -360,6 +377,10 @@ export function loadConfig(): Config { databaseBackupDir, workspaceReaperCooldownDays, workspaceCompactionDelayHours, + skillVersionRetentionEnabled: process.env.PAPERCLIP_SKILL_VERSION_RETENTION_ENABLED !== "false", + skillVersionRetentionDryRun: process.env.PAPERCLIP_SKILL_VERSION_RETENTION_DRY_RUN === "true", + skillVersionRetentionKeepNewest, + skillVersionRetentionGraceMinutes, serveUi: process.env.SERVE_UI !== undefined ? process.env.SERVE_UI === "true" diff --git a/server/src/index.ts b/server/src/index.ts index cb8625e13b6..3221a851ffc 100644 --- a/server/src/index.ts +++ b/server/src/index.ts @@ -1,5 +1,10 @@ import { timeStartupRecovery } from "./startup-recovery-timing.js"; import { skillSourceSyncService } from "./services/skill-source-sync.js"; +import { + SKILL_VERSION_RETENTION_INTERVAL_MS, + SKILL_VERSION_RETENTION_STARTUP_DELAY_MS, + skillVersionRetentionService, +} from "./services/skill-version-retention.js"; import { subscribeAllCompanyLiveEvents } from "./services/live-events.js"; import { DELIVERY_QUEUES } from "./services/delivery-work-notifications.js"; import type { createDeliveryWorkCoordinator } from "./services/delivery-work-coordinator.js"; @@ -1225,6 +1230,39 @@ async function startServerWithDatabaseTeardown( const skillSourceSyncInterval = setInterval(sweepSkillSources, 5000); skillSourceSyncInterval.unref?.(); sweepSkillSources(); + // Materialized skill versions are disk caches of version rows. The sweep is + // single-flight, bounded per tick, and stops between removals on shutdown or + // idle drain. The first sweep runs after startup without delaying readiness. + const skillVersionRetention = skillVersionRetentionService(db as any, { + dryRun: config.skillVersionRetentionDryRun, + keepNewest: config.skillVersionRetentionKeepNewest, + graceMs: config.skillVersionRetentionGraceMinutes * 60 * 1000, + shouldContinue: () => !heartbeatSchedulerStopped && !isIdleTaskDrainActive() && !isWarmStandby(), + }); + let skillVersionRetentionInFlight = false; + let skillVersionRetentionContinuation: ReturnType | undefined; + const sweepSkillVersionSnapshots = () => { + if (!config.skillVersionRetentionEnabled || skillVersionRetentionInFlight || isIdleTaskDrainActive() || heartbeatSchedulerStopped || isWarmStandby()) return; + skillVersionRetentionInFlight = true; + trackHeartbeatSchedulerWork(skillVersionRetention.tick() + .then((result) => { + if (result.checked > 0 || result.failed > 0 || Object.keys(result.skipped).length > 0) { + logger.info({ ...result, dryRun: config.skillVersionRetentionDryRun }, "skill version snapshot retention"); + } + if (result.truncated && !heartbeatSchedulerStopped) { + skillVersionRetentionContinuation = setTimeout(sweepSkillVersionSnapshots, 1000); + skillVersionRetentionContinuation.unref?.(); + } + }) + .catch((err) => { + logger.error({ err }, "skill version snapshot retention failed"); + }) + .finally(() => { skillVersionRetentionInFlight = false; })); + }; + const skillVersionRetentionInterval = setInterval(sweepSkillVersionSnapshots, SKILL_VERSION_RETENTION_INTERVAL_MS); + skillVersionRetentionInterval.unref?.(); + const skillVersionRetentionStartup = setTimeout(sweepSkillVersionSnapshots, SKILL_VERSION_RETENTION_STARTUP_DELAY_MS); + skillVersionRetentionStartup.unref?.(); const externalObjects = externalObjectService(db as any, { pluginWorkerManager, enabled: async () => (await instanceSettingsService(db).getExperimental()).enableExternalObjects === true, @@ -1977,6 +2015,10 @@ async function startServerWithDatabaseTeardown( await systemdNotify(["--stopping", `--status=Stopping after ${signal}`]); heartbeatSchedulerStopped = true; clearInterval(skillSourceSyncInterval); + clearInterval(skillVersionRetentionInterval); + clearTimeout(skillVersionRetentionStartup); + clearTimeout(skillVersionRetentionContinuation); + await skillVersionRetention.dispose(); await skillSourceSync.stop(); unsubscribeChatCompletions(); await deliveryWork.stop(); diff --git a/server/src/services/company-skills.ts b/server/src/services/company-skills.ts index 0c4f5df645d..2637721cf6e 100644 --- a/server/src/services/company-skills.ts +++ b/server/src/services/company-skills.ts @@ -1,3 +1,4 @@ +import { ensureSkillCacheDirectory, touchSkillVersionCache, withSkillVersionCacheLock } from "./skill-version-cache.js"; import { retainProjectSkillAccess, skillAccessService } from "./skill-access.js"; import type { AuthorizationActor } from "./authorization.js"; import { updateAgentConfigurationInTransaction } from "./agent-configuration-transaction.js"; @@ -34,6 +35,7 @@ import { import { readPaperclipSkillSyncPreference, writePaperclipSkillSyncPreference, resolvePaperclipDesiredSkillNames, resolveLegacyPaperclipDesiredSkillNames } from "@paperclipai/adapter-utils/server-utils"; import { REPOSITORY_CONTENT_DIR, repositorySkillProjection } from "./skill-repository-package.js"; +import { runVersionSnapshotFlight } from "./skill-version-retention.js"; import type { PaperclipDesiredSkillEntry, PaperclipSkillEntry } from "@paperclipai/adapter-utils/server-utils"; import type { AgentDesiredSkillEntry, @@ -281,8 +283,6 @@ type ParsedSkillImportSource = { warnings: string[]; }; -const versionSnapshotFlights = new Map>(); - const EXTERNAL_SKILL_SOURCE_TYPES = new Set(["github", "skills_sh", "url"]); function isPinnedCommitRef(value: string | null | undefined) { @@ -3533,17 +3533,20 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo const skill = await getById(companyId, skillId); if (!skill) return null; if (skill.metadata?.skillSourceId && skill.metadata?.snapshotHash && skill.currentVersionId) { - const version = await getVersion(companyId, skill.id, skill.currentVersionId); - if (!version) throw unprocessable("Installed skill version is unavailable."); - const sourceLocator = await materializeVersionSnapshot(companyId, skill, version); - const audit = await auditInstalledSkillBytes({ - ...skill, - sourceType: "local_path", - sourceLocator, - fileInventory: await runtimeVersionInventory(companyId, version), + const versionId = skill.currentVersionId; + return withSkillVersionCacheLock(db, companyId, async () => { + const version = await getVersion(companyId, skill.id, versionId); + if (!version) throw unprocessable("Installed skill version is unavailable."); + const sourceLocator = await materializeVersionSnapshot(companyId, skill, version); + const audit = await auditInstalledSkillBytes({ + ...skill, + sourceType: "local_path", + sourceLocator, + fileInventory: await runtimeVersionInventory(companyId, version), + }); + await persistAuditMetadata(skill, audit); + return audit; }); - await persistAuditMetadata(skill, audit); - return audit; } if (skill.sourceType !== "catalog" && skill.sourceType !== "local_path") { throw unprocessable("Only local-path and catalog-managed company skills support audit."); @@ -5963,13 +5966,15 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo } else if (entry.isFile()) { out.push(normalizePortablePath(relativePath)); } else { - out.push(normalizePortablePath(relativePath)); + throw unprocessable("Unsafe file in materialized skill version."); } } return out; } try { + const rootStat = await fs.lstat(root); + if (!rootStat.isDirectory() || rootStat.isSymbolicLink()) throw unprocessable("Unsafe materialized skill directory."); return await walk(root, ""); } catch (error) { if ((error as NodeJS.ErrnoException).code === "ENOENT") return null; @@ -6013,14 +6018,20 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo } async function materializeVersionSnapshot(companyId: string, skill: CompanySkill, version: CompanySkillVersion): Promise { + return withSkillVersionCacheLock(db, companyId, async () => { + const directory = await materializeVersionSnapshotLocked(companyId, skill, version); + await touchSkillVersionCache(companyId, skill.id, version.id); + return directory; + }); + } + + async function materializeVersionSnapshotLocked(companyId: string, skill: CompanySkill, version: CompanySkillVersion): Promise { version = { ...version, fileInventory: await runtimeVersionInventory(companyId, version) }; const runtimeRoot = path.resolve(resolveManagedSkillsRoot(companyId), "__versions__", skill.id); const skillDir = path.join(runtimeRoot, version.id); - const existing = versionSnapshotFlights.get(skillDir); - if (existing) return existing; - const work = (async () => { + return runVersionSnapshotFlight(skillDir, async () => { + await ensureSkillCacheDirectory(runtimeRoot, true); if (await materializedVersionSnapshotMatches(skillDir, version)) return skillDir; - await fs.mkdir(runtimeRoot, { recursive: true }); const staging = await fs.mkdtemp(path.join(runtimeRoot, `.${version.id}-`)); try { let wroteSkillFile = false; @@ -6044,9 +6055,7 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo } return skillDir; } finally { await fs.rm(staging, { recursive: true, force: true }); } - })(); - versionSnapshotFlights.set(skillDir, work); - try { return await work; } finally { versionSnapshotFlights.delete(skillDir); } + }); } function resolveRuntimeSkillMaterializedPath(companyId: string, skill: Pick) { @@ -6145,6 +6154,13 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo ): Promise { if (options.onlyKeys?.size === 0 || (!options.onlyKeys && options.selectionConfig && !options.includeLegacy && readPaperclipSkillSyncPreference(options.selectionConfig).desiredSkills.length === 0)) return []; + return withSkillVersionCacheLock(db, companyId, () => listRuntimeSkillEntriesLocked(companyId, options)); + } + + async function listRuntimeSkillEntriesLocked( + companyId: string, + options: RuntimeSkillEntryOptions, + ): Promise { const skills = await listFull(companyId); const selectedKeys = options.onlyKeys ?? (options.selectionConfig ? new Set( (options.includeLegacy ? resolveLegacyPaperclipDesiredSkillNames : resolvePaperclipDesiredSkillNames)( @@ -7351,6 +7367,8 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo logger.warn({ err: error, companyId }, "Skill deleted; quarantined source cleanup failed"); }); } + // Version caches may still be linked into a captured run. The retention + // worker removes orphan versions only after their run pins have settled. return deleted; } diff --git a/server/src/services/heartbeat/execution-configuration.ts b/server/src/services/heartbeat/execution-configuration.ts index e36ee3d5f31..f55f8068ab9 100644 --- a/server/src/services/heartbeat/execution-configuration.ts +++ b/server/src/services/heartbeat/execution-configuration.ts @@ -1,4 +1,5 @@ import { withSkillActor } from "../skill-access.js"; +import { withSkillVersionCacheLock } from "../skill-version-cache.js"; import type { AuthorizationActor } from "../authorization.js"; import { eq, sql } from "drizzle-orm"; import { heartbeatRuns, type Db, type agents, type projects } from "@paperclipai/db"; @@ -208,7 +209,7 @@ export async function prepareHeartbeatExecutionConfiguration(db: Db, input: Hear const nativeRunnerPreparationSpans: NativeRunHistoricalSpan[] = []; const skillsPrepareStartedAtMs = Date.now(); let skillDeliveryConfig = effectiveResolvedConfig; - const runtimeSkillEntries = await time("configuration_skills", async () => { + const runtimeSkillEntries = await time("configuration_skills", () => withSkillVersionCacheLock(db, agent.companyId, async () => { try { const storedVersions = parseObject(run.contextSnapshot?.skillVersions); const hasStoredVersions = Object.prototype.hasOwnProperty.call(run.contextSnapshot ?? {}, "skillVersions"); @@ -263,7 +264,7 @@ export async function prepareHeartbeatExecutionConfiguration(db: Db, input: Hear } throw error; } - }); + })); nativeRunnerPreparationSpans.push({ name: "skills.prepare", parentName: "task.prepare", diff --git a/server/src/services/skill-repository-package.ts b/server/src/services/skill-repository-package.ts index 44f8aeb21f1..6db0a8c84e3 100644 --- a/server/src/services/skill-repository-package.ts +++ b/server/src/services/skill-repository-package.ts @@ -6,6 +6,9 @@ import { assertSkillSnapshotPath, skillFileBytes } from './skill-snapshot.js'; export const REPOSITORY_CONTENT_DIR = '.paperclip-repository'; export const MAX_REPOSITORY_FILES = 9_990; export const MAX_REPOSITORY_BYTES = 61 * 1024 * 1024; +export function repositorySkillFrontmatter(content: string): string | undefined { + return content.match(/^---\r?\n[\s\S]*?\r?\n---(?:\r?\n|$)/)?.[0]; +} const manifestSchema = z.object({ version: z.literal(1), skills: z.array(z.string().min(1).max(4096)).min(1).max(1000), @@ -38,7 +41,7 @@ export function repositorySkillProjection(files: CompanySkillVersionFileInventor assertSkillSnapshotPath(skillPath); const skill = files.find(file => file.path === skillPath); if (!skill || skill.encoding === 'base64') throw new Error('Repository skill entrypoint is missing or not text.'); - const frontmatter = skill.content.match(/^---\r?\n[\s\S]*?\r?\n---(?:\r?\n|$)/)?.[0]; + const frontmatter = repositorySkillFrontmatter(skill.content); if (!frontmatter) throw new Error('Repository skill entrypoint has no frontmatter.'); const target = `${REPOSITORY_CONTENT_DIR}/${skillPath}`; const wrapper = `${frontmatter.trimEnd()}\n\nRead and follow the complete skill instructions in [${skillPath}](${target.split('/').map(encodeURIComponent).join('/')}). Resolve the original skill's relative file references from its directory inside \`${REPOSITORY_CONTENT_DIR}/\`. That directory contains the included pinned repository files, including supporting skills and shared files. Repository-root paths are relative to that directory.\n`; diff --git a/server/src/services/skill-version-cache.ts b/server/src/services/skill-version-cache.ts new file mode 100644 index 00000000000..50ee27f8468 --- /dev/null +++ b/server/src/services/skill-version-cache.ts @@ -0,0 +1,166 @@ +import { AsyncLocalStorage } from "node:async_hooks"; +import { randomUUID } from "node:crypto"; +import { constants, promises as fs } from "node:fs"; +import path from "node:path"; +import { and, eq, sql } from "drizzle-orm"; +import { companySkillVersions, companySkillRepositorySnapshots, heartbeatRuns, + nativeRunFinalizations, environmentLeases, withDedicatedDbConnection, type Db } from "@paperclipai/db"; +import { resolvePaperclipInstanceRoot } from "../home-paths.js"; +import { assertSkillSnapshotPath } from "./skill-snapshot.js"; +import { repositorySkillFrontmatter } from "./skill-repository-package.js"; +import type { CompanySkillVersionFileInventoryEntry } from "@paperclipai/shared"; + +export const SKILL_VERSION_CACHE_IDLE_MS = 60 * 60 * 1000; +const locks = new AsyncLocalStorage>(); +const uuid = /^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/i; +export class SkillCacheBusy extends Error {} +// Repository snapshots are immutable and referenced with restrictive foreign +// keys. Cache only validated entrypoint names, never large file contents, so an +// hourly sweep does not transfer the same repository for each skill/version. +const verifiedSnapshots = new WeakMap>>(); + +/** Hold through publication AND durable run pinning, not just file creation. */ +export async function withSkillVersionCacheLock(db: Db, companyId: string, action: () => Promise, tryOnly = false): Promise { + const held = locks.getStore() ?? []; + if (held.some(lock => lock.db === db && lock.companyId === companyId)) return action(); + // Preparation uses the ordinary pool for version/pin writes. Borrowing a pool + // connection for this lock would deadlock installations with a one-slot pool. + return withDedicatedDbConnection(db, connection => connection.transaction(async tx => { + const key = `skill_version_cache:${companyId}`; + if (tryOnly) { + const rows = await tx.execute<{ acquired: boolean }>(sql`select pg_try_advisory_xact_lock(hashtextextended(${key}, 0)) as acquired`); + if (!rows[0]?.acquired) throw new SkillCacheBusy(); + } else await tx.execute(sql`select pg_advisory_xact_lock(hashtextextended(${key}, 0))`); + return locks.run([...held, { db, companyId }], action); + })); +} + +export function skillVersionCachePaths(companyId: string, skillId: string, versionId: string) { + if (![companyId, skillId, versionId].every(value => uuid.test(value))) throw new Error("Invalid skill cache identity"); + const root = path.join(resolvePaperclipInstanceRoot(), "skills", companyId); + return { + root, + directory: path.join(root, "__versions__", skillId, versionId), + access: path.join(root, "__version_access__", skillId, `${versionId}.json`), + }; +} + +/** Never traverse a symlink below the configured instance root. */ +export async function ensureSkillCacheDirectory(directory: string, create = false) { + const root = path.resolve(resolvePaperclipInstanceRoot()); + const relative = path.relative(root, directory); + if (!relative || relative.startsWith("..") || path.isAbsolute(relative)) throw new Error("Skill cache escaped instance root"); + if (create) await fs.mkdir(root, { recursive: true }); + let cursor = root; + for (const part of relative.split(path.sep)) { + cursor = path.join(cursor, part); + if (create) await fs.mkdir(cursor).catch((error: NodeJS.ErrnoException) => { + if (error.code !== "EEXIST") throw error; + }); + const stat = await fs.lstat(cursor); + if (!stat.isDirectory() || stat.isSymbolicLink()) throw new Error("Unsafe skill cache directory"); + } +} + +export async function touchSkillVersionCache(companyId: string, skillId: string, versionId: string, now = Date.now()) { + const { access } = skillVersionCachePaths(companyId, skillId, versionId); + await ensureSkillCacheDirectory(path.dirname(access), true); + const temporary = `${access}.${randomUUID()}.tmp`; + try { + await fs.writeFile(temporary, JSON.stringify({ version: 1, lastUsedAt: now }), { flag: "wx", mode: 0o600 }); + await fs.rename(temporary, access); + } finally { await fs.rm(temporary, { force: true }); } +} + +export async function lastUsedAt(filename: string): Promise { + try { + await ensureSkillCacheDirectory(path.dirname(filename)); + const file = await fs.open(filename, constants.O_RDONLY | constants.O_NOFOLLOW); + try { + const stat = await file.stat(); + if (!stat.isFile() || stat.size > 256) return null; + const value = JSON.parse(await file.readFile("utf8")); + return value.version === 1 && Number.isSafeInteger(value.lastUsedAt) && value.lastUsedAt > 0 ? value.lastUsedAt : null; + } finally { await file.close(); } + } catch (error) { + if (error instanceof SyntaxError || ["ENOENT", "ELOOP"].includes((error as NodeJS.ErrnoException).code ?? "")) return null; + throw error; + } +} + +// A terminal run can still own provider execution or recovery. Only fully settled +// runs release their pins; resume preparation reacquires the same cache lock. +export async function protectedVersions(db: Db, companyId: string): Promise | null> { + const rows = await db.select({ pins: sql`${heartbeatRuns.contextSnapshot}->'skillVersions'` }) + .from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, companyId), sql`( + ${heartbeatRuns.status} not in ('succeeded', 'failed', 'timed_out', 'cancelled') + or ${heartbeatRuns.finishedAt} is null + or ${heartbeatRuns.controllerLeaseExpiresAt} > now() + or exists (select 1 from ${nativeRunFinalizations} f + where f.company_id = ${heartbeatRuns.companyId} and f.run_id = ${heartbeatRuns.id} + and (f.phase <> 'committed' or f.lease_owner is not null or f.recovery_state is not null)) + or exists (select 1 from ${environmentLeases} l + where l.company_id = ${heartbeatRuns.companyId} and l.heartbeat_run_id = ${heartbeatRuns.id} + and (l.status = 'pending_cleanup' or l.cleanup_status = 'failed' + or l.metadata ? 'nativeWorkspaceExportResume' + or (l.released_at is null and not coalesce(l.status = 'retained' + and l.lease_policy = 'reuse_by_environment' and l.cleanup_status = 'success', false)))) + )`)); + const versions = new Set(); + for (const { pins } of rows) { + // Legacy/incomplete preparation is not proof that no skill is in use. + if (!pins || typeof pins !== "object" || Array.isArray(pins)) return null; + for (const version of Object.values(pins)) { + if (typeof version !== "string" || !uuid.test(version)) return null; + versions.add(version); + } + } + return versions; +} + +function validateInventory(inventory: CompanySkillVersionFileInventoryEntry[]) { + const seen = new Set(); + for (const file of inventory) { + assertSkillSnapshotPath(file.path); + if (seen.has(file.path) || typeof file.content !== "string") return false; + seen.add(file.path); + } + return true; +} + +export async function hasRestorableSnapshot(db: Db, companyId: string, versionId: string) { + const [version] = await db.select().from(companySkillVersions).where(and( + eq(companySkillVersions.companyId, companyId), eq(companySkillVersions.id, versionId))); + if (!version) return false; + if (version.repositorySnapshotId) { + if (!version.repositorySkillPath) return false; + let cache = verifiedSnapshots.get(db); + if (!cache) { cache = new Map(); verifiedSnapshots.set(db, cache); } + const key = `${companyId}:${version.repositorySnapshotId}`; + const verified = cache.get(key); + if (verified) return verified.has(version.repositorySkillPath); + const [snapshot] = await db.select({ inventory: companySkillRepositorySnapshots.fileInventory }) + .from(companySkillRepositorySnapshots).where(and(eq(companySkillRepositorySnapshots.companyId, companyId), + eq(companySkillRepositorySnapshots.id, version.repositorySnapshotId))); + if (!snapshot || !validateInventory(snapshot.inventory)) return false; + const entrypoints = new Set(snapshot.inventory.filter(file => file.encoding !== "base64" + && repositorySkillFrontmatter(file.content)).map(file => file.path)); + if (cache.size >= 128) cache.delete(cache.keys().next().value!); + cache.set(key, entrypoints); + return entrypoints.has(version.repositorySkillPath); + } + return validateInventory(version.fileInventory) && version.fileInventory.some(file => file.path === "SKILL.md"); +} + +// Counts allocated bytes without following links, matching what df can reclaim. +export async function allocatedBytes(directory: string, shouldContinue: () => boolean = () => true): Promise { + if (!shouldContinue()) throw new Error("Skill cache scan stopped"); + const stat = await fs.lstat(directory); + if (stat.isSymbolicLink() || (!stat.isDirectory() && !stat.isFile())) throw new Error("Unsafe skill cache entry"); + let bytes = stat.blocks * 512; + if (stat.isDirectory()) { + for (const entry of await fs.readdir(directory)) bytes += await allocatedBytes(path.join(directory, entry), shouldContinue); + } + return bytes; +} + diff --git a/server/src/services/skill-version-retention.ts b/server/src/services/skill-version-retention.ts new file mode 100644 index 00000000000..4dc46985d00 --- /dev/null +++ b/server/src/services/skill-version-retention.ts @@ -0,0 +1,458 @@ +import { randomUUID } from "node:crypto"; +import { promises as fs } from "node:fs"; +import path from "node:path"; +import { and, asc, eq, gt, gte, inArray, lte, sql } from "drizzle-orm"; +import { agents, companies, companySkills, companySkillVersions, type Db } from "@paperclipai/db"; +import { readPaperclipSkillSyncPreference } from "@paperclipai/adapter-utils/server-utils"; +import { resolvePaperclipInstanceRoot } from "../home-paths.js"; +import { logger } from "../middleware/logger.js"; +import { allocatedBytes, ensureSkillCacheDirectory, hasRestorableSnapshot, lastUsedAt, + protectedVersions, SkillCacheBusy, skillVersionCachePaths, touchSkillVersionCache, + withSkillVersionCacheLock } from "./skill-version-cache.js"; + +/** + * Retention for materialized skill version directories. + * + * `materializeVersionSnapshot` writes each selected skill version to + * `/skills//__versions__///` from the + * version row's stored file inventory. A directory is a cache: when it is + * removed while its row survives, the next resolve writes it again. Nothing + * removed these directories before, so they grew with every new version. + * + * The prune keeps a directory when it is the skill's current version, an agent + * pins it, a non-terminal heartbeat run captured it, it is among the newest + * versions of its skill (when configured), or it was materialized or reused within the grace + * window. It removes every other version directory, directories whose version + * or skill row is gone, and abandoned staging directories. Version rows are + * never deleted here. + */ + +export const SKILL_VERSION_RETENTION_DEFAULT_KEEP_NEWEST = 0; +export const SKILL_VERSION_RETENTION_DEFAULT_GRACE_MS = 60 * 60 * 1000; +export const SKILL_VERSION_RETENTION_DEFAULT_MAX_REMOVALS = 50; +export const SKILL_VERSION_RETENTION_INTERVAL_MS = 60 * 60 * 1000; +export const SKILL_VERSION_RETENTION_STARTUP_DELAY_MS = 60 * 1000; + +/** A run in one of these states can still read (or re-read) its captured skill versions. */ +export const SKILL_VERSION_LIVE_RUN_STATUSES = ["queued", "scheduled_retry", "running", "interrupted"] as const; + +const VERSIONS_DIR = "__versions__"; +const TRASH_PREFIX = ".trash-"; +const UUID_PATTERN = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i; +const QUERY_BATCH = 500; + +// In-process coordination between materialization and pruning, keyed by the +// absolute version directory. A materialization waits for a pending removal of +// its directory; a prune skips a directory that a materialization is writing. +const versionSnapshotFlights = new Map>(); +const versionSnapshotRemovals = new Map>(); +// Trash directories this process is deleting right now; a concurrent prune skips them. +const activeTrash = new Set(); + +export function resolveCompanySkillVersionsRoot(companyId: string) { + return path.resolve(resolvePaperclipInstanceRoot(), "skills", companyId, VERSIONS_DIR); +} + +export function resolveSkillVersionSnapshotDir(companyId: string, skillId: string, versionId: string) { + return path.join(resolveCompanySkillVersionsRoot(companyId), skillId, versionId); +} + +/** + * Runs one materialization per version directory at a time. Concurrent callers + * share the in-flight promise. A pending prune of the same directory finishes + * first, so a caller never receives a path that a prune is about to remove. + */ +export async function runVersionSnapshotFlight(skillDir: string, work: () => Promise): Promise { + for (;;) { + const removal = versionSnapshotRemovals.get(skillDir); + if (!removal) break; + await removal.catch(() => undefined); + } + const existing = versionSnapshotFlights.get(skillDir); + if (existing) return existing; + const flight = work(); + versionSnapshotFlights.set(skillDir, flight); + try { return await flight; } finally { + if (versionSnapshotFlights.get(skillDir) === flight) versionSnapshotFlights.delete(skillDir); + } +} + +function errorCode(error: unknown) { + return (error as NodeJS.ErrnoException | null)?.code; +} + +async function makeTreeWritable(target: string): Promise { + const stat = await fs.lstat(target).catch(() => null); + if (!stat || stat.isSymbolicLink()) return; + if (stat.isDirectory()) { + await fs.chmod(target, 0o700).catch(() => undefined); + const entries = await fs.readdir(target).catch(() => [] as string[]); + for (const entry of entries) await makeTreeWritable(path.join(target, entry)); + } else { + await fs.chmod(target, 0o600).catch(() => undefined); + } +} + +/** + * Removes a snapshot tree. Snapshot files are read-only, and some platforms + * refuse to unlink read-only entries, so a permission failure makes the tree + * writable and retries. Symbolic links are removed, never followed. + */ +async function removeSnapshotTree(target: string) { + if (activeTrash.has(target)) return; + activeTrash.add(target); + try { + await fs.rm(target, { recursive: true, force: true }); + } catch (error) { + const code = errorCode(error); + if (code !== "EACCES" && code !== "EPERM" && code !== "ENOTEMPTY") throw error; + await makeTreeWritable(target); + await fs.rm(target, { recursive: true, force: true }); + } finally { + activeTrash.delete(target); + } +} + +/** + * Moves a directory out of its readable name in one atomic rename under the + * shared cache lock. Its unique trash name can be deleted after releasing it. + */ +async function removeVersionDirectory(parent: string, name: string): Promise { + const target = path.join(parent, name); + if (path.dirname(target) !== parent) return null; + if (versionSnapshotFlights.has(target) || versionSnapshotRemovals.has(target)) return null; + const trash = path.join(parent, `${TRASH_PREFIX}${randomUUID()}`); + const rename = fs.rename(target, trash); + versionSnapshotRemovals.set(target, rename); + try { + await rename; + } catch (error) { + if (errorCode(error) === "ENOENT") return null; + throw error; + } finally { + versionSnapshotRemovals.delete(target); + } + return trash; +} + +type CompanyCursor = { skillId: string; name: string }; +export type SkillVersionRetentionCursor = CompanyCursor & { companyId: string }; +export type SkillVersionPruneOptions = { + keepNewest?: number; + graceMs?: number; + maxRemovals?: number; + /** Bound inspection too, including protected directories. */ + maxEntries?: number; + now?: Date; + dryRun?: boolean; + shouldContinue?: () => boolean; +}; +export type SkillVersionPruneResult = { + checked: number; candidates: number; candidateBytes: number; reclaimedBytes: number; + removed: number; removedOrphans: number; removedStaging: number; kept: number; + skippedRecent: number; failed: number; truncated: boolean; + skipped: Record; + cursor: CompanyCursor | null; +}; +function emptyResult(): SkillVersionPruneResult { + return { checked: 0, candidates: 0, candidateBytes: 0, reclaimedBytes: 0, + removed: 0, removedOrphans: 0, removedStaging: 0, kept: 0, + skippedRecent: 0, failed: 0, truncated: false, skipped: {}, cursor: null }; +} + +type Candidate = { parent: string; skillId: string; name: string; unsafe?: boolean; discoveryOnly?: boolean }; + +// Retain directory handles between pages instead of re-reading/sorting N names +// for every 50-entry continuation. Every directory entry consumes a scan slot. +async function* discoverDirectories(root: string): AsyncGenerator { + await ensureSkillCacheDirectory(root); + const directory = await fs.opendir(root, { bufferSize: 1 }); + try { + for (;;) { + const entry = await directory.read(); + if (!entry) break; + if (!UUID_PATTERN.test(entry.name)) { + yield { parent: root, skillId: "", name: entry.name }; + continue; + } + const parent = path.join(root, entry.name); + yield { parent, skillId: entry.name, name: "", discoveryOnly: true }; + let childDirectory; + try { + await ensureSkillCacheDirectory(parent); + childDirectory = await fs.opendir(parent, { bufferSize: 1 }); + } catch { + yield { parent, skillId: entry.name, name: "", unsafe: true }; + continue; + } + try { + for (;;) { + const child = await childDirectory.read(); + if (!child) break; + yield { parent, skillId: entry.name, name: child.name }; + } + } finally { await childDirectory.close(); } + } + } finally { await directory.close(); } +} + +function chunks(values: T[], size: number): T[][] { + const out: T[][] = []; + for (let index = 0; index < values.length; index += size) out.push(values.slice(index, index + size)); + return out; +} + +export function skillVersionRetentionService(db: Db, defaults: SkillVersionPruneOptions = {}) { + const scans = new Map; cursor: CompanyCursor | null; + pending: Candidate[]; expiry: ReturnType }>(); + const scanning = new Set(); + async function closeScan(companyId: string) { + const scan = scans.get(companyId); + if (!scan) return; + scans.delete(companyId); + clearTimeout(scan.expiry); + await scan.iterator.return(undefined); + } + async function dispose() { for (const id of [...scans.keys()]) await closeScan(id); } + + async function protectedVersionIds(companyId: string, skillIds: string[], keepNewest: number) { + const keep = new Set(); + if (skillIds.length === 0) return keep; + for (const batch of chunks(skillIds, QUERY_BATCH)) { + const rows = await db.select({ currentVersionId: companySkills.currentVersionId }).from(companySkills) + .where(and(eq(companySkills.companyId, companyId), inArray(companySkills.id, batch))); + for (const row of rows) if (row.currentVersionId) keep.add(row.currentVersionId); + if (keepNewest > 0) { + const ranked = db.select({ + id: companySkillVersions.id, + rank: sql`row_number() over (partition by ${companySkillVersions.companySkillId} order by ${companySkillVersions.revisionNumber} desc)`.as("rank"), + }).from(companySkillVersions) + .where(and(eq(companySkillVersions.companyId, companyId), inArray(companySkillVersions.companySkillId, batch))) + .as("ranked"); + for (const row of await db.select({ id: ranked.id }).from(ranked).where(lte(ranked.rank, keepNewest))) keep.add(row.id); + } + } + for (const agent of await db.select({ adapterConfig: agents.adapterConfig }).from(agents).where(eq(agents.companyId, companyId))) { + for (const entry of readPaperclipSkillSyncPreference(agent.adapterConfig ?? {}).desiredSkillEntries) { + if (entry.versionId) keep.add(entry.versionId); + } + } + const runVersions = await protectedVersions(db, companyId); + if (!runVersions) return null; + for (const version of runVersions) keep.add(version); + return keep; + } + + async function pruneCompanyPage(companyId: string, options: SkillVersionPruneOptions & { cursor?: CompanyCursor | null } = {}): Promise { + if (!UUID_PATTERN.test(companyId)) throw new Error("Invalid company ID for skill retention"); + const keepNewest = Math.max(0, Math.floor(options.keepNewest ?? defaults.keepNewest ?? SKILL_VERSION_RETENTION_DEFAULT_KEEP_NEWEST)); + const graceMs = Math.max(0, options.graceMs ?? defaults.graceMs ?? SKILL_VERSION_RETENTION_DEFAULT_GRACE_MS); + const maxRemovals = Math.min(50, Math.max(0, options.maxRemovals ?? defaults.maxRemovals ?? SKILL_VERSION_RETENTION_DEFAULT_MAX_REMOVALS)); + const maxEntries = Math.min(maxRemovals, Math.max(1, options.maxEntries ?? defaults.maxEntries ?? 50)); + const shouldContinue = options.shouldContinue ?? defaults.shouldContinue ?? (() => true); + const dryRun = options.dryRun ?? defaults.dryRun ?? false; + const now = (options.now ?? new Date()).getTime(); + const result = emptyResult(); + const skip = (reason: string) => { result.skipped[reason] = (result.skipped[reason] ?? 0) + 1; }; + const trash: string[] = []; + const root = resolveCompanySkillVersionsRoot(companyId); + const trashName = (name: string) => name.startsWith(TRASH_PREFIX) && UUID_PATTERN.test(name.slice(TRASH_PREFIX.length)); + let scan = scans.get(companyId); + if (!options.cursor || !scan || scan.cursor?.skillId !== options.cursor.skillId || scan.cursor?.name !== options.cursor.name) { + await closeScan(companyId); + // A service normally holds two descriptors for one company's continuation. + // Bound direct/internal callers too; abandoned cursors expire automatically. + if (scans.size >= 16) await closeScan(scans.keys().next().value!); + const expiry = setTimeout(() => { if (!scanning.has(companyId)) void closeScan(companyId).catch(() => {}); }, 5 * 60 * 1000); + expiry.unref(); + scan = { iterator: discoverDirectories(root), cursor: null, pending: [], expiry }; + scans.set(companyId, scan); + } + scan.expiry.refresh(); + const entries: Candidate[] = []; + let discoveryDone = false; + try { + while (entries.length < maxEntries && shouldContinue()) { + const next = scan.pending.length ? { done: false, value: scan.pending.shift()! } : await scan.iterator.next(); + if (next.done) { discoveryDone = true; break; } + entries.push(next.value); + } + } catch (error) { + await closeScan(companyId); + if (errorCode(error) === "ENOENT" || errorCode(error) === "ENOTDIR") return result; + result.checked++; + result.failed++; + skip("unsafe_or_unavailable"); + logger.warn({ err: error, companyId }, "skill version retention skipped unsafe company root"); + return result; + } + result.truncated = !discoveryDone; + if (entries.length === 0) { await closeScan(companyId); return result; } + const skillIds = [...new Set(entries.filter(entry => !entry.unsafe && entry.skillId).map(entry => entry.skillId))]; + let initialKeep: Set | null; + try { initialKeep = await withSkillVersionCacheLock(db, companyId, () => protectedVersionIds(companyId, skillIds, keepNewest), true); } + catch (error) { + result.checked = 1; + if (error instanceof SkillCacheBusy) skip("cache_busy"); + else { result.failed++; skip("unavailable_pins"); } + await closeScan(companyId); + result.truncated = false; + return result; + } + if (!initialKeep) { await closeScan(companyId); result.truncated = false; result.checked = 1; skip("unknown_run_pins"); return result; } + const inspected: Array<{ + entry: Candidate; isStaging: boolean; version: boolean; + before: { dev: number; ino: number }; bytes: number; + eligible: (keep: Set | null) => Promise<{ dev: number; ino: number } | null>; + }> = []; + for (const entry of entries) { + if (!shouldContinue()) { scan.pending.push(...entries.slice(result.checked)); result.truncated = true; break; } + result.checked++; + result.cursor = { skillId: entry.skillId, name: entry.name }; + if (entry.unsafe) { skip("unsafe_or_unavailable"); continue; } + if (entry.discoveryOnly) continue; + const target = path.join(entry.parent, entry.name); + const isTrash = trashName(entry.name); + const isVersion = UUID_PATTERN.test(entry.name); + const isStaging = entry.name.startsWith(".") && UUID_PATTERN.test(entry.name.slice(1, 37)) + && /^-[a-zA-Z0-9]{6}$/.test(entry.name.slice(37)); + if (!isTrash && !isVersion && !isStaging) { skip("unknown_directory"); continue; } + if (isTrash) { if (!dryRun) trash.push(target); continue; } + // Called under the shared lock both before inspection and immediately + // before rename. A preparation between these calls changes the pins/use. + const eligible = async (keep: Set | null) => { + if (!shouldContinue()) return null; + await ensureSkillCacheDirectory(target); + if (versionSnapshotFlights.has(target) || versionSnapshotRemovals.has(target)) { skip("cache_busy"); return null; } + if (!keep) { skip("unknown_run_pins"); return null; } + if (isVersion && keep.has(entry.name)) { result.kept++; skip("protected_version"); return null; } + const stat = await fs.lstat(target); + if (isVersion) { + const used = await lastUsedAt(skillVersionCachePaths(companyId, entry.skillId, entry.name).access); + if (used === null) { + if (!dryRun) await touchSkillVersionCache(companyId, entry.skillId, entry.name, now); + skip("observation_grace"); return null; + } + if (now - used < graceMs) { result.skippedRecent++; skip("recent_use"); return null; } + } else if (now - stat.mtimeMs < graceMs) { result.skippedRecent++; return null; } + return stat; + }; + try { + const before = await withSkillVersionCacheLock(db, companyId, () => eligible(initialKeep), true); + if (!before) continue; + // Inventory transfer/validation and recursive file walks must not hold + // the lock needed by agent preparation. Immutable snapshots can be + // checked here; mutable current/run pins are checked again below. + const [version] = isVersion ? await db.select({ id: companySkillVersions.id }).from(companySkillVersions).where(and( + eq(companySkillVersions.companyId, companyId), eq(companySkillVersions.companySkillId, entry.skillId), + eq(companySkillVersions.id, entry.name))) : []; + if (version && !await hasRestorableSnapshot(db, companyId, version.id)) { skip("snapshot_unavailable"); continue; } + const bytes = await allocatedBytes(target, shouldContinue); + inspected.push({ entry, isStaging, version: Boolean(version), before, bytes, eligible }); + } catch (error) { + if (!shouldContinue()) { result.truncated = true; break; } + if (error instanceof SkillCacheBusy) skip("cache_busy"); + else if (errorCode(error) === "ENOENT") skip("not_materialized"); + else { + result.failed++; + skip("unsafe_or_unavailable"); + logger.warn({ err: error, companyId }, "skill version cache candidate skipped"); + } + } + } + if (inspected.length && shouldContinue()) { + try { + // Read company-wide agent/run pins once per phase, not once per cache. + // The lock keeps this final set authoritative through every rename. + await withSkillVersionCacheLock(db, companyId, async () => { + const keep = await protectedVersionIds(companyId, skillIds, keepNewest); + for (const candidate of inspected) { + if (!shouldContinue()) { result.truncated = true; break; } + try { + const current = await candidate.eligible(keep); + if (!current) continue; + if (current.dev !== candidate.before.dev || current.ino !== candidate.before.ino) { skip("cache_changed"); continue; } + result.candidates++; + result.candidateBytes += candidate.bytes; + if (dryRun) continue; + const renamed = await removeVersionDirectory(candidate.entry.parent, candidate.entry.name); + if (!renamed) continue; + trash.push(renamed); + result.removed++; + if (candidate.isStaging) result.removedStaging++; + else if (!candidate.version) result.removedOrphans++; + } catch (error) { + if (errorCode(error) === "ENOENT") { skip("not_materialized"); continue; } + result.failed++; skip("unsafe_or_unavailable"); + logger.warn({ err: error, companyId }, "skill version cache rename skipped"); + } + } + }, true); + } catch (error) { + if (error instanceof SkillCacheBusy) skip("cache_busy"); + else { result.failed++; skip("unavailable_pins"); } + } + } + if (!result.truncated) { result.cursor = null; await closeScan(companyId); } + else scan.cursor = result.cursor; + // These unique names are never reused by materialization. A crash leaves + // them for the next scan; a returning run can rehydrate immediately. + for (const target of trash) { + if (!shouldContinue()) break; + try { + await withSkillVersionCacheLock(db, `trash:${companyId}:${target}`, async () => { + await ensureSkillCacheDirectory(target); + const bytes = await allocatedBytes(target, shouldContinue); + if (!shouldContinue()) return; + await removeSnapshotTree(target); + result.reclaimedBytes += bytes; + }, true); + } catch (error) { + if (!shouldContinue()) break; + if (error instanceof SkillCacheBusy || errorCode(error) === "ENOENT") continue; + result.failed++; + logger.warn({ err: error, companyId }, "skill version snapshot trash removal failed"); + } + } + return result; + } + + async function pruneCompany(companyId: string, options: SkillVersionPruneOptions & { cursor?: CompanyCursor | null } = {}) { + if (scanning.has(companyId)) return { ...emptyResult(), checked: 1, skipped: { scan_busy: 1 } }; + scanning.add(companyId); + try { return await pruneCompanyPage(companyId, options); } + finally { scanning.delete(companyId); } + } + + // A continuation visits later companies and protected directories too; a busy + // first company or a dry run cannot starve the remainder of the instance. + let cursor: SkillVersionRetentionCursor | null = null; + async function tick(options: SkillVersionPruneOptions = {}) { + const maxRemovals = Math.min(50, options.maxRemovals ?? defaults.maxRemovals ?? SKILL_VERSION_RETENTION_DEFAULT_MAX_REMOVALS); + const maxEntries = Math.min(maxRemovals, options.maxEntries ?? defaults.maxEntries ?? 50); + const shouldContinue = options.shouldContinue ?? defaults.shouldContinue ?? (() => true); + const totals = { ...emptyResult(), companies: 0, cursor: null as SkillVersionRetentionCursor | null }; + const companyRows = await db.select({ id: companies.id }).from(companies) + .where(cursor ? cursor.skillId === "\uffff" ? gt(companies.id, cursor.companyId) : gte(companies.id, cursor.companyId) : undefined) + .orderBy(asc(companies.id)).limit(51); + for (const company of companyRows) { + if (cursor && (company.id < cursor.companyId || (company.id === cursor.companyId && cursor.skillId === "\uffff"))) continue; + if (totals.companies >= 50 || totals.checked >= maxEntries || totals.candidates >= maxRemovals || !shouldContinue()) { totals.truncated = true; break; } + const result = await pruneCompany(company.id, { ...options, cursor: cursor?.companyId === company.id ? cursor : null, + maxEntries: maxEntries - totals.checked, maxRemovals: maxRemovals - totals.candidates, shouldContinue }); + totals.companies++; + for (const key of ["checked", "candidates", "candidateBytes", "reclaimedBytes", "removed", "removedOrphans", "removedStaging", "kept", "skippedRecent", "failed"] as const) totals[key] += result[key]; + for (const [reason, count] of Object.entries(result.skipped)) totals.skipped[reason] = (totals.skipped[reason] ?? 0) + count; + cursor = { companyId: company.id, skillId: "\uffff", name: "" }; + if (result.truncated) { + cursor = { companyId: company.id, ...(result.cursor ?? { skillId: "", name: "" }) }; + totals.truncated = true; + break; + } + } + if (!totals.truncated) cursor = null; + totals.cursor = cursor; + return totals; + } + return { pruneCompany, tick, dispose }; +} From 25b16e65fc70a617a70e3c5f50ce358e333615d8 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Thu, 15 Oct 2026 12:21:32 -0500 Subject: [PATCH 14/19] fix(observability): detail workspace and connector preparation (#16454) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The server prepares a repository, sandbox workspace, and connector access before an agent can reply. > - Existing traces measure large preparation phases but leave important work inside those phases unclear. > - Repository source materialization and Browser Use discovery can do more work than their callers suggest. > - New detail must preserve access checks, run ownership, and the existing work order. > - This pull request adds fixed OpenTelemetry spans inside those operations. > - Operators can use one deployed trace to select the next performance fix. ## Linked Issues or Issue Description Refs: #16419, #16425, #16428. **What existing behavior does this improve?** Startup traces for server workspace preparation, native sandbox synchronization, and connector assignment resolution. **Current behavior** Workspace and connector aggregate spans can take many seconds. The trace does not separate private source cloning, seed reuse, archive work, progress persistence, or each connector resolver. Browser Use resource selection builds the full authorized tool catalog. The configuration and native backend can each resolve connector assignments. **Proposed behavior** Show the awaited work as fixed child spans. Separate discovery queue wait from catalog construction. Preserve the initiating run context when another run releases a scheduler slot. Add no new lifecycle event writes for workspace detail callbacks. **Reason and benefit** Operators need evidence about the hosted critical path before another performance change. These spans distinguish filesystem work, provider transfer, authorization, transaction entry, locks, and reporting overhead. **Breaking changes** None. Authorization, query order, transaction ownership, synchronization, existing progress events, and task results retain their contracts. The change adds opt-in Observability spans under the existing OTLP endpoint gate. ## What Changed - Trace private source directory setup, Git credentials, clone, requested ref, source identity, and checkout publication. - Trace sandbox source capture, seed cache/replay, shallow clone and archive, overlay copy/archive, verification, upload, warm stamps, and descriptor persistence. - Separate existing awaited progress reporting from useful workspace work. Keep new detail callbacks trace-only. - Trace each connector resolver and Browser Use catalog/grant work. Split discovery queueing, descriptor sources, and policy evaluation. - Trace Slack authority, credential reads, catalog synchronization, transaction entry/settlement, and advisory locks. - Update the real filesystem staging trace expectations and the Honeycomb investigation guide. - Add a real scheduler regression for a queued job released by another run. Assert run ownership, parent spans, and exactly one completed queue span. ## Verification - Passed: server and adapter-utils TypeScript checks (`pnpm --filter @paperclipai/server exec tsc --noEmit` and `pnpm --filter @paperclipai/adapter-utils exec tsc --noEmit`). - Passed: 252 checks across 9 focused files. These cover real filesystem staging, native workspace synchronization, trace ownership, endpoint gating, and workspace preparation. - Passed after review: 26 trace-context, tracing-failure, and discovery-scheduler checks across 3 files (these overlap the first run). The new queued-discovery regression fails when context binding is removed and passes after restoration. An initial submillisecond timestamp assertion was replaced with explicit span-completion order to avoid SDK clock precision assumptions. - Passed on implementation commit `0e68968a4bd61001893d833f281f12bb110edbe1`: `pnpm -r typecheck` and `pnpm build`. The later commit updates only an engine test expectation. - The first CI workspace lane found an old engine trace-tree expectation. It expected the earlier span set. The updated assertion now checks the new detail spans, their parents, and single completion. Its targeted executor check passes. - The full test suite, build, release registry, and end-to-end gates pass in CI on final head `40e37d89ef679109d60a96bb5731a0616efeaa43`: 55 passing checks and 2 optional Storybook skips. The full suite was not duplicated locally after the focused run. - Greptile reviewed the final head at 5/5. No unresolved review threads remain. - After deployment, filter Honeycomb by the hashed run ID and verify `service.version`. Inspect `heartbeat.prepare.workspace`, `runtime_selection.workspace_sync`, and the connector assignment parents. The guide in `doc/observability.md` explains each new family and the nested intervals. - No hosted run has used this new instrumentation yet. Local checks cannot establish hosted latency improvement. ## Risks - Tracing adds spans when an operator configures an OTLP endpoint. Detail remains capped at 16 spans per phase per execution attempt. - Parent and child durations overlap. The documentation explains how to avoid counting them twice. - A discovery job can start in another run's scheduler continuation. The bound callback preserves the initiating async context. - New spans export fixed names and the existing hashed run ID. They add no paths, repository URLs, connector resource IDs, credentials, tool schemas, or exception text. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, tool use, and code execution. The exact served model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/observability.md | 48 ++++++++++- .../src/acpx-engine/execute.test.ts | 43 ++++++---- .../src/sandbox-managed-runtime.test.ts | 11 ++- .../src/sandbox-managed-runtime.ts | 78 ++++++++--------- .../chat-channels/slack/task-authority.ts | 12 +-- server/src/services/connector-runtime.ts | 11 ++- .../src/services/connectors/slack-catalog.ts | 17 ++-- server/src/services/connectors/slack.ts | 16 ++-- .../services/execution-preparation-timing.ts | 83 +++++++++++++++++-- .../services/execution-timing-context.test.ts | 51 +++++++++++- .../services/execution-transaction-timing.ts | 2 +- server/src/services/heartbeat.ts | 7 +- .../heartbeat/workspace-preparation.ts | 18 ++-- .../isolated-remote-workspace-source.ts | 46 +++++----- .../services/managed-repository-checkout.ts | 18 ++-- .../native-runtime/native-workspace-sync.ts | 41 +++++---- server/src/services/tool-gateway.ts | 33 ++++---- 17 files changed, 371 insertions(+), 164 deletions(-) diff --git a/doc/observability.md b/doc/observability.md index 2fcf0ade1b8..48f417cc2dc 100644 --- a/doc/observability.md +++ b/doc/observability.md @@ -254,10 +254,22 @@ attempt or owning phase: | `workspace.sources.*` | Source inventory queries, canonicalization, one filesystem alias scan, repository scan, and configured project scan. | | `workspace.readers`, `workspace.readers.*`, `workspace.access.*` | Retained-reader groups, identity, trust, project and source authorization, privacy and inventory reads. | | `workspace.provenance`, `workspace.notification`, `workspace.write` | Preserved ownership, notification statements, and record writes. | +| `workspace.materialize.*`, `workspace.checkout.*` | Private source-directory setup and identity verification, existing checkout lookup, Git credential resolution, clone, requested-ref checkout, and atomic publication. The enclosing `heartbeat.prepare.workspace_source_materialize` includes the complete source-materialization path. | | `runtime_selection.*` | Completion contracts, session/recovery reads, profiles, interaction materialization, backend discovery, bootstrap, run/coordinator writes, transaction entry/settlement, and workspace synchronization. | | `runtime_context.instructions`, `.skills`, `.mcp` | The three concurrent runtime-context branches. Their durations overlap. | | `runtime_context.asset_*` | Source reads, manifest hashing, cache lookup, cold materialization, and exact file verification on cold and warm paths. | -| `workspace_sync.*` | Existing Git and baseline snapshot, pack, seed verification, transfer, and restore operations. Pack and verification can be inside transfer; do not add nested durations. | +| `workspace_sync.prepare_host_current`, `.prepare_durable_seed`, `.prepare_adopt_remote` | Runtime preparation under the selected inbound mode, including snapshot and staging work. A warm mismatch may produce a second preparation; it is real repeated work. | +| `workspace_sync.source_root`, `.repositories`, `.ignore_rules`, `.snapshot_git`, `.snapshot_baseline` | Physical source capture, managed repositories, ignore rules, Git metadata and baseline content snapshot. | +| `workspace_sync.pack`, `.git_clone`, `.git_archive`, `.overlay_copy`, `.overlay_archive`, `.deleted_manifest` | Temporary shallow Git clone, its archive callback, selected working-tree overlay copy/archive, and the deleted-file manifest. The archive is inside the clone's lifetime; do not sum parent and child. | +| `workspace_sync.seed_generation`, `.seed_lookup`, `.seed_replay`, `.seed_persist`, `.seed_publish`, `.seed_verify`, `.seed_extract`, `.seed_snapshot`, `.seed_cleanup` | Seed identity, cache lookup, replay, durable copies, publication, extraction/re-snapshot verification and temporary cleanup. Cache hits omit cold clone and verification work. | +| `workspace_sync.inbound_barrier`, `.stage_workspace`, `.transfer` | The inbound scheduler barrier, whole workspace task, and provider `syncIn` call (including provider guards, upload and post-upload extraction). Parallel tasks overlap; provider transfer/exec children give the finer transport split when available. | +| `workspace_sync.progress` | Awaited progress/status reporting, including the existing lifecycle run-log writes. These are not file transfer work. | +| `workspace_sync.run_read`, `.descriptor_directory`, `.descriptor_read`, `.archive_hash`, `.warm_stamp`, `.durable_seed_verify`, `.manifests_persist`, `.descriptor_write`, `.reference_write` | Saved run state, recovery descriptors, archive hashing, remote warm-stamp check, manifests and final descriptor/reference persistence. | +| `connectors.browser_use.resolve`, `connectors.speko.resolve`, `connectors.slack.resolve`, `connectors.agentmail.resolve` | Each sequential connector resource resolver beneath configuration or native-controller assignment resolution. Empty/denied assignments still have measurable work. | +| `connectors.browser_use.connection`, `.catalog`, `.grant` | Browser Use candidate reads, the full authorized catalog listing currently used to discover Browser Use resources, and grant checks. The catalog can contain work for other connectors. | +| `connectors.discovery.scheduled`, `.queue`, `.build`, `.agent`, `.connections`, `.github`, `.slack`, `.speko`, `.policy`, `.on_demand_policy` | Discovery scheduler lifetime and wait, catalog construction, agent validation, descriptor sources, and policy decisions. Queued work retains the initiating run's trace context even when another run releases the scheduler slot. | +| `connectors.slack.candidates`, `.authority`, `.profile`, `.settings`, `.identity`, `.credentials`, `.revision` | Slack endpoint enumeration, authority revalidation, enabled profile, settings, accepted requester provenance, credential reads and authorization revision. | +| `connectors.slack.authority.transaction*`, `connectors.slack.catalog.transaction*`, `connectors.slack.catalog.lock`, `.entries`, `.write`, `.profile` | Slack authority and catalog transaction entry/settlement, advisory lock, catalog reads/updates and existing-profile lookup. Entry combines pool acquisition and BEGIN; it does not isolate pool wait. | | `execution.runtime` | Runtime dispatch, usage, workspace finalization and runtime cleanup; this includes work after provider execution. | | `execution.workspace_finalize` | Best-effort native workspace finalization, including restore recovery, copyback and commit. Its nested restore and commit spans measure distinct steps. | | `execution.workspace_ownership`, `.workspace_ownership_wait`, `.workspace_restore`, `.workspace_commit`, `.native_finalize` | Ownership coordination including its action, acquisition before the action, copyback, final snapshot/stamp/descriptor persistence, and native settlement. | @@ -283,6 +295,40 @@ can continue through later bookkeeping and resource cleanup. A later recovery invocation has its own attempt; this does not manufacture one continuous trace across queueing, process restarts, or an unavailable controller. +### Reading a slow startup in Honeycomb + +Filter `paperclip.task.run.run_id` to the first 12 hex characters of +`SHA256(run UUID)`, then inspect the attempt's waterfall with its +`service.version`. An agent/model setting or the repository's checked-out commit +does not establish the server revision. The detailed spans are available only +after this instrumentation is deployed; an earlier aggregate cannot establish +which child was slow. + +Start beneath `heartbeat.prepare.workspace` with source materialization versus +provisioning, repository admission, and environment acquisition/realization. +Source materialization now separates filesystem setup, credential resolution, +Git cloning and source identity checks. Under +`runtime_selection.workspace_sync`, compare snapshot work, the seed-cache path, +archive construction/verification, provider transfer and descriptor persistence. +Use span start/end times to identify serial work and uncovered gaps; summing +nested or concurrent durations overstates elapsed time. + +For connector delay, compare the four `connectors.*.resolve` spans beneath each +assignment-resolution parent. A long Browser Use `.catalog` can be discovery +queueing, connection descriptors, another connector's authority checks, or policy +evaluation. A long Slack transaction entry, lock or credential span calls for a +different investigation. Configuration and native backend setup currently resolve +assignments separately; the trace records both without caching authorization or +changing access decisions. + +New workspace detail callbacks have fixed names and remain trace-only. They do +not create additional lifecycle events or awaited run-log writes. Existing coarse +progress writes remain intact and appear under `workspace_sync.progress`, so +reporting overhead is visible alongside useful work. No per-file spans, paths, +repository URLs, connector resource IDs, tool schemas, credentials or exception +messages are added to the export. The endpoint gate and per-phase span limit +above still apply. + ## Native Runner Trace Spans diff --git a/packages/adapter-utils/src/acpx-engine/execute.test.ts b/packages/adapter-utils/src/acpx-engine/execute.test.ts index 422ad5a49e2..fe6021cc385 100644 --- a/packages/adapter-utils/src/acpx-engine/execute.test.ts +++ b/packages/adapter-utils/src/acpx-engine/execute.test.ts @@ -5275,7 +5275,7 @@ describe("ACPX engine sandbox-start spans (opt-in root + child parenting)", () = }; } - it("emits one root span with a child span per bring-up boundary, each parented to the root", async () => { + it("emits one root with startup and nested workspace boundaries", async () => { const root = await makeTempRoot(); const stateDir = path.join(root, "state"); const localCwd = path.join(root, "worktree"); @@ -5334,12 +5334,6 @@ describe("ACPX engine sandbox-start spans (opt-in root + child parenting)", () = expect(syncBackSpan!.parent).toBe(runRootSpan); expect(syncBackSpan!.ended).toBe(true); - // A codex bring-up over the remote sandbox lane crosses all 7 boundaries. - // Each boundary span parents to the sandbox bring-up span, not to the run - // root or the turn span. The `stage.sync` step also opens the two pre-task - // sub-step spans — `snapshot.git` and `snapshot.baseline` — plus the - // `stage.workspace` inbound task span, and the `pack` span nests one level - // deeper under `stage.workspace`. const childNames = spans .filter( (span) => @@ -5362,16 +5356,21 @@ describe("ACPX engine sandbox-start spans (opt-in root + child parenting)", () = "snapshot.git", "stage.sync", "stage.workspace", + "workspace.ignore_rules", + "workspace.inbound_barrier", + "workspace.overlay_archive", + "workspace.progress", + "workspace.progress", + "workspace.repositories", "workspace.resolve", + "workspace.source_root", + "workspace.transfer", ], ); - // The two pre-task sub-step spans and the `stage.workspace` task span nest - // under the `stage.sync` step span (that host work runs inside the step), not - // directly under the bring-up span. The `pack` span nests under - // `stage.workspace`, because the host builds the tarball inside the task. const stageSyncSpan = spans.find((span) => span.name === "stage.sync"); const stageWorkspaceSpan = spans.find((span) => span.name === "stage.workspace"); + const inboundSpan = spans.find((span) => span.name === "workspace.inbound_barrier"); const packSpan = spans.find((span) => span.name === "pack"); const snapshotGitSpan = spans.find((span) => span.name === "snapshot.git"); const snapshotBaselineSpan = spans.find((span) => span.name === "snapshot.baseline"); @@ -5380,15 +5379,12 @@ describe("ACPX engine sandbox-start spans (opt-in root + child parenting)", () = expect(packSpan).toBeTruthy(); expect(snapshotGitSpan).toBeTruthy(); expect(snapshotBaselineSpan).toBeTruthy(); - expect(stageWorkspaceSpan!.parent).toBe(stageSyncSpan); + expect(stageWorkspaceSpan!.parent).toBe(inboundSpan); expect(packSpan!.parent).toBe(stageWorkspaceSpan); expect(snapshotGitSpan!.parent).toBe(stageSyncSpan); expect(snapshotBaselineSpan!.parent).toBe(stageSyncSpan); expect(packSpan!.ended).toBe(true); - // Every boundary step span parents to the sandbox bring-up span and ends. - // The `stage.sync` sub-step spans and the run-parented `sandbox.syncBack` - // span are the exceptions: they parent above. const nonStartupChildren = new Set([ packSpan, stageWorkspaceSpan, @@ -5396,6 +5392,23 @@ describe("ACPX engine sandbox-start spans (opt-in root + child parenting)", () = snapshotBaselineSpan, syncBackSpan, ]); + for (const [name, parent] of [ + ["workspace.source_root", stageSyncSpan], ["workspace.repositories", stageSyncSpan], + ["workspace.ignore_rules", stageSyncSpan], ["workspace.inbound_barrier", stageSyncSpan], + ["workspace.overlay_archive", packSpan], ["workspace.transfer", stageWorkspaceSpan], + ] as const) { + const span = spans.find(span => span.name === name); + expect(span?.parent, name).toBe(parent); + expect(span?.ended, name).toBe(true); + expect(span?.endCalls, name).toBe(1); + nonStartupChildren.add(span); + } + const progress = spans.filter(span => span.name === "workspace.progress"); + expect(progress.map(span => span.parent)).toEqual([packSpan, stageWorkspaceSpan]); + for (const span of progress) { + expect(span.endCalls).toBe(1); + nonStartupChildren.add(span); + } for (const span of spans) { if (span === runRootSpan || span === startupSpan || span === turnSpan) continue; if (nonStartupChildren.has(span)) continue; diff --git a/packages/adapter-utils/src/sandbox-managed-runtime.test.ts b/packages/adapter-utils/src/sandbox-managed-runtime.test.ts index b2135d1b279..854f76e9102 100644 --- a/packages/adapter-utils/src/sandbox-managed-runtime.test.ts +++ b/packages/adapter-utils/src/sandbox-managed-runtime.test.ts @@ -2943,7 +2943,11 @@ describe("sandbox managed runtime", () => { // The workspace stage task opens its own `stage.workspace` span, and the // host tarball build opens the `pack` span inside it. The two pre-task // sub-steps stay ahead of the task. - expect(openedSpans).toEqual(["snapshot.git", "snapshot.baseline", "stage.workspace", "pack"]); + expect(openedSpans).toEqual([ + "workspace.source_root", "snapshot.git", "workspace.repositories", "workspace.ignore_rules", + "snapshot.baseline", "workspace.inbound_barrier", "stage.workspace", "pack", + "workspace.progress", "workspace.overlay_archive", "workspace.transfer", "workspace.progress", + ]); // The tarball build still lands the workspace inside the span, so the wrap // changes no staging behavior. await expect(readFile(path.join(remoteWorkspaceDir, "README.md"), "utf8")).resolves.toBe("workspace body\n"); @@ -3002,8 +3006,11 @@ describe("sandbox managed runtime", () => { expect(packSpan!.ended).toBe(true); // The workspace stage task opens its own `stage.workspace` span under // `stage.sync`, and the `pack` span nests under `stage.workspace`. - expect(workspaceSpan!.parentName).toBe("stage.sync"); + expect(workspaceSpan!.parentName).toBe("workspace.inbound_barrier"); expect(packSpan!.parentName).toBe("stage.workspace"); + expect(spans.find(span => span.name === "workspace.inbound_barrier")?.parentName).toBe("stage.sync"); + expect(spans.find(span => span.name === "workspace.overlay_archive")?.parentName).toBe("pack"); + expect(spans.find(span => span.name === "workspace.transfer")?.parentName).toBe("stage.workspace"); // The two pre-`pack` staging sub-steps nest under `stage.sync` the same way, // so the previously hidden gap at the head of the step is now attributed. diff --git a/packages/adapter-utils/src/sandbox-managed-runtime.ts b/packages/adapter-utils/src/sandbox-managed-runtime.ts index 3e591d7dffe..7458f3953e4 100644 --- a/packages/adapter-utils/src/sandbox-managed-runtime.ts +++ b/packages/adapter-utils/src/sandbox-managed-runtime.ts @@ -1265,11 +1265,13 @@ export async function prepareSandboxManagedRuntime(input: { // span, because the teardown runs the restore inside that span. const runStepSpan = (name: string, work: () => Promise): Promise => input.runtimeSpan ? input.runtimeSpan(name, work) : work(); + const reportWorkspaceStatus = (phase: RuntimeStatusPhase, message: string) => + runStepSpan("workspace.progress", () => emitRuntimeStatus(input.onRuntimeProgress, phase, message)); // Resolve an existing workspace alias once, before reading its snapshot. // All subsequent work uses that root, so retargeting the alias cannot select // another repository. Staging also verifies the captured directory identity. - const workspaceRoot = syncWorkspace ? await captureWorkspaceSourceRoot(input.workspaceLocalDir) : null; + const workspaceRoot = syncWorkspace ? await runStepSpan("workspace.source_root", () => captureWorkspaceSourceRoot(input.workspaceLocalDir)) : null; if (workspaceRoot) input = { ...input, workspaceLocalDir: workspaceRoot.sourceDir }; // The git enumeration (`git status --ignored`, the HEAD diffs, `ls-files`). @@ -1282,7 +1284,7 @@ export async function prepareSandboxManagedRuntime(input: { readGitWorkspaceSnapshot(input.workspaceLocalDir), ) : null; - const repositories = gitSnapshot ? (gitSnapshot.repositories ?? []) : input.workspaceRepositories ?? (syncWorkspace ? await readManagedWorkspaceRepositories(input.workspaceLocalDir) : []); + const repositories = gitSnapshot ? (gitSnapshot.repositories ?? []) : input.workspaceRepositories ?? (syncWorkspace ? await runStepSpan("workspace.repositories", () => readManagedWorkspaceRepositories(input.workspaceLocalDir)) : []); const repositoryGitPaths = (children: typeof repositories, prefix = ""): string[] => children.flatMap(repo => { const relative = prefix ? `${prefix}/${repo.path}` : repo.path; return [`${relative}/.git`, ...repositoryGitPaths(repo.snapshot.repositories ?? [], relative)]; @@ -1315,7 +1317,7 @@ export async function prepareSandboxManagedRuntime(input: { // repository's ignore rules still govern which files may leave the host. // Use the same bounded, path-relative resolver as referenced project trees. const directoryIgnore = syncWorkspace && !gitSnapshot && input.workspaceFileMode !== "all" - ? await resolveReferencedSourceIgnore(input.workspaceLocalDir) + ? await runStepSpan("workspace.ignore_rules", () => resolveReferencedSourceIgnore(input.workspaceLocalDir)) : null; if (directoryIgnore?.kind === "failed") { throw new Error(`Workspace ignore scan failed: ${directoryIgnore.reason}`); @@ -1368,9 +1370,9 @@ export async function prepareSandboxManagedRuntime(input: { } const seedGeneration = input.workspaceSeedCacheDirectory && baselineSnapshot && workspaceInboundMode !== "durable_seed" - ? workspaceSeedGeneration(baselineSnapshot, gitSnapshot, repositories) : null; + ? await runStepSpan("workspace.seed_generation", async () => workspaceSeedGeneration(baselineSnapshot!, gitSnapshot, repositories)) : null; const cachedSeed = seedGeneration - ? await readWorkspaceSeedGeneration(input.workspaceSeedCacheDirectory!, seedGeneration) : null; + ? await runStepSpan("workspace.seed_lookup", () => readWorkspaceSeedGeneration(input.workspaceSeedCacheDirectory!, seedGeneration)) : null; const replaySeed = workspaceInboundMode === "durable_seed" ? input.workspaceDurableSeed : cachedSeed; // Every inbound staging step delegates to the provider through `client.syncIn`: @@ -1467,7 +1469,7 @@ export async function prepareSandboxManagedRuntime(input: { params.progressLabel, { sink: input.onRuntimeProgress, phase: params.statusPhase }, ); - const syncResult = await syncIn(operations); + const syncResult = await runStepSpan("workspace.transfer", () => syncIn(operations)); // Prefer the transport's own byte total. It is the real count for a // provider that has no host tarball to stat (a referenced project rides // a `directory` mapping). Fall back to the caller-supplied count when @@ -1475,7 +1477,7 @@ export async function prepareSandboxManagedRuntime(input: { // shows the host-known workspace total. const transferredBytes = sumSyncResultBytes(syncResult); const reportedBytes = transferredBytes > 0 ? transferredBytes : params.progressBytes; - await upload.finish(reportedBytes, reportedBytes); + await runStepSpan("workspace.progress", () => upload.finish(reportedBytes, reportedBytes)); }; // Build the ordered inbound operation task list. Each task stages one inbound @@ -1522,7 +1524,7 @@ export async function prepareSandboxManagedRuntime(input: { // wipes the target tree EXCEPT `.paperclip-runtime`, so the overlay tar, // which sits under `.paperclip-runtime`, survives to run its own extract. if (gitSnapshot || repositories.length) { - await emitRuntimeStatus(input.onRuntimeProgress, "git_sync", "Syncing git history to environment"); + await reportWorkspaceStatus("git_sync", "Syncing git history to environment"); const gitTarPath = path.join(tempDir, "git-workspace.tar"); const remoteGitTar = path.posix.join( runtimeRootDir, @@ -1533,26 +1535,26 @@ export async function prepareSandboxManagedRuntime(input: { if (!durableGitArchive) { throw new Error("workspace_durable_seed_git_missing"); } - await copyDurableSeedArchive({ + await runStepSpan("workspace.seed_replay", () => copyDurableSeedArchive({ sourcePath: durableGitArchive, targetPath: gitTarPath, expectedSha256: replaySeed.gitArchiveSha256, - }); + })); } else { - const archive = async (cloneDir: string) => createTarballFromDirectory({ localDir: cloneDir, archivePath: gitTarPath, exclude: [".paperclip-runtime"] }); - if (gitSnapshot) await withShallowGitWorkspaceClone({ localDir: input.workspaceLocalDir, snapshot: gitSnapshot }, archive); - else await withTempDir("paperclip-managed-repositories-", async (cloneDir) => { + const archive = async (cloneDir: string) => runStepSpan("workspace.git_archive", () => createTarballFromDirectory({ localDir: cloneDir, archivePath: gitTarPath, exclude: [".paperclip-runtime"] })); + if (gitSnapshot) await runStepSpan("workspace.git_clone", () => withShallowGitWorkspaceClone({ localDir: input.workspaceLocalDir, snapshot: gitSnapshot }, archive)); + else await runStepSpan("workspace.git_clone", () => withTempDir("paperclip-managed-repositories-", async (cloneDir) => { for (const repo of repositories) await withShallowGitWorkspaceClone({ localDir: path.join(input.workspaceLocalDir, repo.path), snapshot: repo.snapshot }, async (child) => copyCloneTree(child, path.join(cloneDir, repo.path))); await archive(cloneDir); - }); + })); if (input.workspaceDurableSeed?.gitArchivePath) { - await persistDurableSeedArchive({ + await runStepSpan("workspace.seed_persist", () => persistDurableSeedArchive({ sourcePath: gitTarPath, - targetPath: input.workspaceDurableSeed.gitArchivePath, - }); + targetPath: input.workspaceDurableSeed!.gitArchivePath!, + })); } } - if (cachedSeed && input.workspaceDurableSeed?.gitArchivePath) await persistDurableSeedArchive({ sourcePath: gitTarPath, targetPath: input.workspaceDurableSeed.gitArchivePath }); + if (cachedSeed && input.workspaceDurableSeed?.gitArchivePath) await runStepSpan("workspace.seed_persist", () => persistDurableSeedArchive({ sourcePath: gitTarPath, targetPath: input.workspaceDurableSeed!.gitArchivePath! })); workspaceFiles.push({ sourcePath: gitTarPath, targetPath: remoteGitTar, @@ -1573,41 +1575,41 @@ export async function prepareSandboxManagedRuntime(input: { // 2. workspace-overlay tar. A git-backed overlay merges on top of the just // extracted git tree (no wipe); a plain workspace wipes every child except // the preserved names first. The extract runs AFTER the git extract. - await emitRuntimeStatus(input.onRuntimeProgress, "config_sync", "Syncing workspace to environment"); + await reportWorkspaceStatus("config_sync", "Syncing workspace to environment"); const workspaceTarPath = path.join(tempDir, "workspace.tar"); if (replaySeed) { - await copyDurableSeedArchive({ + await runStepSpan("workspace.seed_replay", () => copyDurableSeedArchive({ sourcePath: replaySeed.workspaceArchivePath, targetPath: workspaceTarPath, expectedSha256: replaySeed.workspaceArchiveSha256, - }); + })); } else { const workspaceArchiveDir = gitSnapshot || repositories.length ? path.join(tempDir, "workspace-overlay") : input.workspaceLocalDir; if (gitSnapshot || repositories.length) { - await copySelectedWorkspaceEntries({ + await runStepSpan("workspace.overlay_copy", () => copySelectedWorkspaceEntries({ sourceRoot: workspaceRoot!, targetDir: workspaceArchiveDir, relativePaths: (input.workspaceFileMode === "all" ? undefined : gitSnapshot?.overlayPaths) ?? (baselineSnapshot!.entries instanceof WorkspaceManifestMap ? baselineSnapshot!.entries.manifest : [...baselineSnapshot!.entries.keys()]), ignoredPaths, exclude: workspaceArchiveExclude, - }); + })); } - await createTarballFromDirectory({ + await runStepSpan("workspace.overlay_archive", () => createTarballFromDirectory({ localDir: workspaceArchiveDir, archivePath: workspaceTarPath, exclude: gitSnapshot || repositories.length ? undefined : workspaceArchiveExclude, - }); + })); if (input.workspaceDurableSeed) { - await persistDurableSeedArchive({ + await runStepSpan("workspace.seed_persist", () => persistDurableSeedArchive({ sourcePath: workspaceTarPath, - targetPath: input.workspaceDurableSeed.workspaceArchivePath, - }); + targetPath: input.workspaceDurableSeed!.workspaceArchivePath, + })); } } - if (cachedSeed && input.workspaceDurableSeed) await persistDurableSeedArchive({ sourcePath: workspaceTarPath, targetPath: input.workspaceDurableSeed.workspaceArchivePath }); + if (cachedSeed && input.workspaceDurableSeed) await runStepSpan("workspace.seed_persist", () => persistDurableSeedArchive({ sourcePath: workspaceTarPath, targetPath: input.workspaceDurableSeed!.workspaceArchivePath })); const remoteWorkspaceTar = path.posix.join( runtimeRootDir, "workspace-upload.tar", @@ -1631,7 +1633,7 @@ export async function prepareSandboxManagedRuntime(input: { if ((isPathManifest(deletedPaths) ? deletedPaths.count : deletedPaths.length) > 0) { const deletedManifest = path.join(tempDir, "deleted.nul"); const remoteDeletedManifest = path.posix.join(runtimeRootDir, "deleted.nul"); - await writeWorkspacePaths(deletedPaths, deletedManifest); + await runStepSpan("workspace.deleted_manifest", () => writeWorkspacePaths(deletedPaths, deletedManifest)); workspaceFiles.push({ sourcePath: deletedManifest, targetPath: remoteDeletedManifest, kind: "file", access: "rw", writablePath: runtimeRootDir }); workspacePostUploadCommands.push({ command: buildRemoveDeletedPathsCommand({ remoteDir: workspaceRemoteDir, manifestPath: remoteDeletedManifest }) }); } @@ -1640,14 +1642,14 @@ export async function prepareSandboxManagedRuntime(input: { if (seedGeneration && !cachedSeed && input.workspaceDurableSeed) { const seed = input.workspaceDurableSeed; - await publishWorkspaceSeedGeneration(input.workspaceSeedCacheDirectory!, seedGeneration, + await runStepSpan("workspace.seed_publish", () => publishWorkspaceSeedGeneration(input.workspaceSeedCacheDirectory!, seedGeneration, { ...seed, gitArchivePath: gitSnapshot || repositories.length ? seed.gitArchivePath : null }, { companyDirectory: input.workspaceSeedCacheCompanyDirectory, verify: () => runStepSpan("seed.verify", async () => { const verifiedRoot = path.join(tempDir, "verify-seed"); try { - if (gitSnapshot || repositories.length) await extractTarballToDirectory({ archivePath: path.join(tempDir, "git-workspace.tar"), localDir: verifiedRoot }); - await extractTarballToDirectory({ archivePath: path.join(tempDir, "workspace.tar"), localDir: verifiedRoot, exclude: repositoryMetadataExcludes }); + if (gitSnapshot || repositories.length) await runStepSpan("workspace.seed_extract", () => extractTarballToDirectory({ archivePath: path.join(tempDir, "git-workspace.tar"), localDir: verifiedRoot })); + await runStepSpan("workspace.seed_extract", () => extractTarballToDirectory({ archivePath: path.join(tempDir, "workspace.tar"), localDir: verifiedRoot, exclude: repositoryMetadataExcludes })); for (const relative of workspacePaths(deletedPaths)) { let parent = verifiedRoot; for (const segment of relative.split("/").slice(0, -1)) { @@ -1657,12 +1659,12 @@ export async function prepareSandboxManagedRuntime(input: { } await fs.rm(path.join(verifiedRoot, relative), { recursive: true, force: true }); } - const verified = await captureWorkspaceSnapshot(verifiedRoot, { exclude: baselineSnapshot!.exclude, ignoredPaths: baselineSnapshot!.ignoredPaths, diskBacked: true }); + const verified = await runStepSpan("workspace.seed_snapshot", () => captureWorkspaceSnapshot(verifiedRoot, { exclude: baselineSnapshot!.exclude, ignoredPaths: baselineSnapshot!.ignoredPaths, diskBacked: true })); try { return directorySnapshotSha256(verified) === directorySnapshotSha256(baselineSnapshot!); } finally { await disposeDirectorySnapshot(verified); } - } finally { await fs.rm(verifiedRoot, { recursive: true, force: true }); } + } finally { await runStepSpan("workspace.seed_cleanup", () => fs.rm(verifiedRoot, { recursive: true, force: true })); } }), - }); + })); } if (!stageWorkspace) return; @@ -1821,11 +1823,11 @@ export async function prepareSandboxManagedRuntime(input: { // tasks under the client's concurrency permission and the bound, and it // settles every started task before it returns. This is the startup barrier: // the coordinator returns only after every upload settles. - const inboundResults = await scheduleSyncOperations( + const inboundResults = await runStepSpan("workspace.inbound_barrier", () => scheduleSyncOperations( inboundTasks, input.client.allowConcurrentSyncOperations === true, SYNC_OPERATION_CONCURRENCY_LIMIT, - ); + )); // Collect the referenced-project failures in stable input order. for (const failure of additionalFailureSlots) { diff --git a/server/src/services/chat-channels/slack/task-authority.ts b/server/src/services/chat-channels/slack/task-authority.ts index 43b5673126d..8fa8f85396b 100644 --- a/server/src/services/chat-channels/slack/task-authority.ts +++ b/server/src/services/chat-channels/slack/task-authority.ts @@ -5,6 +5,8 @@ import { slackAuthorizationRevision } from "../../connectors/slack-revision.js"; import { instanceSettingsService } from "../../instance-settings.js"; import { registerSlackTaskAuthority, slackRunOrigin } from "../../connectors/slack-authority.js"; +import { measureRunPhase } from "../../execution-preparation-timing.js"; +import { measureRunTransaction } from "../../execution-transaction-timing.js"; import { createHash, @@ -51,13 +53,13 @@ export function createSlackTaskAuthorityRegistration(db: Db, options: SlackTaskA const unregisterSlackTaskAuthority = registerSlackTaskAuthority(db, async (binding) => { - if (!(await instanceSettingsService(db).getExperimental()).enableChatConnectors) + if (!(await measureRunPhase("connectors.slack.settings", () => instanceSettingsService(db).getExperimental())).enableChatConnectors) throw forbidden("Chat connectors are disabled"); - const identity = await slackRunOrigin(db, binding); + const identity = await measureRunPhase("connectors.slack.identity", () => slackRunOrigin(db, binding)); const run = identity.run; if ((run.contextSnapshot?.issueId ?? run.contextSnapshot?.taskId) !== binding.issueId || !run.responsibleUserId) throw forbidden("Slack tools require the bound task and an accepted linked user"); - const resolved = await db.transaction(async (tx) => { + const resolved = await measureRunTransaction(db, "connectors.slack.authority", async (tx) => { const [action] = await tx.select().from(chatActions).where(and( eq(chatActions.companyId, binding.companyId), eq(chatActions.kind, "inbound_wakeup"), @@ -134,7 +136,7 @@ export function createSlackTaskAuthorityRegistration(db: Db, options: SlackTaskA if (!principal) throw forbidden("Slack sender is no longer available"); return { ...source, principalId: principal.id, slackUserId: principal.externalId }; }); - const credentials = await resolveCredentials(resolved.endpoint); + const credentials = await measureRunPhase("connectors.slack.credentials", () => resolveCredentials(resolved.endpoint)); if (!credentials.botToken) throw forbidden("Slack bot credential is unavailable"); return { endpoint: resolved.endpoint, issueId: binding.issueId, conversation: resolved.conversation, @@ -145,7 +147,7 @@ export function createSlackTaskAuthorityRegistration(db: Db, options: SlackTaskA revision: createHash("sha256").update(JSON.stringify([ resolved.endpoint.status, resolved.endpoint.allowDirectMessages, resolved.conversation?.sessionGeneration ?? null, run.responsibleUserId, credentials.botToken, - await slackAuthorizationRevision(db, binding.companyId, resolved.endpoint.id, resolved.endpoint.connectionId, binding.agentId, run.responsibleUserId), + await measureRunPhase("connectors.slack.revision", () => slackAuthorizationRevision(db, binding.companyId, resolved.endpoint.id, resolved.endpoint.connectionId, binding.agentId, run.responsibleUserId!)), ])).digest("hex"), botToken: credentials.botToken, searchActionToken: resolved.delivery ? slackSearchActionToken(db, resolved.endpoint.id, resolved.endpoint.providerAccountId!, resolved.slackUserId, String((resolved.delivery.normalizedEvent.message as Record | undefined)?.providerMessageId ?? "")) : null, diff --git a/server/src/services/connector-runtime.ts b/server/src/services/connector-runtime.ts index 4ccd1bb89bb..6ab346660f8 100644 --- a/server/src/services/connector-runtime.ts +++ b/server/src/services/connector-runtime.ts @@ -22,6 +22,7 @@ import { } from "./connectors/agentmail.js"; import { materializeAsset } from "./native-runtime/runtime-context.js"; import { BROWSER_USE_CLOUD_SKILL } from "./connectors/browser-use-cloud/skill.js"; +import { measureRunPhase, type ExecutionPhase } from "./execution-preparation-timing.js"; type AgentBinding = { companyId: string; agentId: string; runId?: string; issueId?: string }; type ToolBinding = AgentBinding & { @@ -43,6 +44,7 @@ type Tool = { }; interface ConnectorDefinition { key: string; + resolvePhase: Extract; label: string; skillName: string; /** Trusted connector-owned content; never discovered through global skills/. */ @@ -62,7 +64,7 @@ interface ConnectorDefinition { // Trusted connector packages declare their contributions here. Assignments and // current access, not credential availability or agent-authored config, select them. const connectors: ConnectorDefinition[] = [ - { key: "browser-use-cloud", label: "Browser Use Cloud", skillName: "browser-use-cloud", skillMarkdown: BROWSER_USE_CLOUD_SKILL, tools: [], + { key: "browser-use-cloud", resolvePhase: "connectors.browser_use.resolve", label: "Browser Use Cloud", skillName: "browser-use-cloud", skillMarkdown: BROWSER_USE_CLOUD_SKILL, tools: [], legacySkillKeys: ["paperclipai/paperclip/browser-use"], async resolve(db, binding) { const { getAssignedMcpGateway } = await import("./native-runtime/assigned-mcp-tools.js"); @@ -70,18 +72,19 @@ const connectors: ConnectorDefinition[] = [ }, async execute() { throw forbidden("Use Browser Use Cloud through the connection tool gateway."); }, }, - { key: "speko", label: "Speko", skillName: "speko", skillMarkdown: SPEKO_SKILL, + { key: "speko", resolvePhase: "connectors.speko.resolve", label: "Speko", skillName: "speko", skillMarkdown: SPEKO_SKILL, tools: [], // Native phone tools keep their existing governed gateway transport. resolve: spekoAssignedResources, async execute() { throw forbidden("Use Speko through the connection tool gateway."); }, }, - { key: "slack", label: "Slack", skillName: "slack", + { key: "slack", resolvePhase: "connectors.slack.resolve", label: "Slack", skillName: "slack", tools: SLACK_TOOLS.map(({ name, description, inputSchema }) => ({ name, description, inputSchema })), resolve: slackAssignedResource, execute: executeGovernedSlackTool, }, { key: "agentmail", + resolvePhase: "connectors.agentmail.resolve", label: "AgentMail", skillName: "agentmail", tools: AGENTMAIL_TOOLS.map( @@ -127,7 +130,7 @@ export async function resolveConnectorAssignments( ): Promise { const assignments: ConnectorAssignment[] = []; for (const connector of connectors) { - const resources = await connector.resolve(db, binding); + const resources = await measureRunPhase(connector.resolvePhase, () => connector.resolve(db, binding)); if (resources.length) assignments.push({ key: connector.key, diff --git a/server/src/services/connectors/slack-catalog.ts b/server/src/services/connectors/slack-catalog.ts index 607c6c44e8e..184695da3b2 100644 --- a/server/src/services/connectors/slack-catalog.ts +++ b/server/src/services/connectors/slack-catalog.ts @@ -1,4 +1,5 @@ import { createHash } from "node:crypto"; +import { measureRunPhase } from "../execution-preparation-timing.js"; import { and, eq, isNull, sql } from "drizzle-orm"; import { SLACK_TOOLS } from "@paperclipai/shared"; import { @@ -35,13 +36,13 @@ export async function syncSlackBotTools( ), ); if (!connection) return; - await tx.execute( + await measureRunPhase("connectors.slack.catalog.lock", () => tx.execute( sql`select pg_advisory_xact_lock(hashtextextended(${`slack-catalog:${endpoint.id}`}, 0))`, - ); - const existingEntries = await tx + )); + const existingEntries = await measureRunPhase("connectors.slack.catalog.entries", () => tx .select() .from(toolCatalogEntries) - .where(eq(toolCatalogEntries.connectionId, connection.id)); + .where(eq(toolCatalogEntries.connectionId, connection.id))); const entriesByName = new Map( existingEntries.map((entry) => [entry.name, entry]), ); @@ -79,7 +80,7 @@ export async function syncSlackBotTools( }; const existing = entriesByName.get(values.name); if (existing?.versionHash === hash) continue; - await tx + await measureRunPhase("connectors.slack.catalog.write", () => tx .insert(toolCatalogEntries) .values(values) .onConflictDoUpdate({ @@ -94,10 +95,10 @@ export async function syncSlackBotTools( } : {}), }, - }); + })); } const profileKey = `slack-bot:${endpoint.id}`; - const [priorProfile] = await tx + const [priorProfile] = await measureRunPhase("connectors.slack.catalog.profile", () => tx .select() .from(toolProfiles) .where( @@ -105,7 +106,7 @@ export async function syncSlackBotTools( eq(toolProfiles.companyId, endpoint.companyId), eq(toolProfiles.profileKey, profileKey), ), - ); + )); if (priorProfile) return; const [profile] = await tx .insert(toolProfiles) diff --git a/server/src/services/connectors/slack.ts b/server/src/services/connectors/slack.ts index 9cbde7bc034..3053feb2867 100644 --- a/server/src/services/connectors/slack.ts +++ b/server/src/services/connectors/slack.ts @@ -4,6 +4,8 @@ import { slackMessage } from "./slack-message.js"; import { executeSlackWrite } from "./slack-writes.js"; import { syncSlackBotTools } from "./slack-catalog.js"; import { SLACK_TOOLS } from "@paperclipai/shared"; +import { measureRunPhase } from "../execution-preparation-timing.js"; +import { measureRunTransaction } from "../execution-transaction-timing.js"; import { toolProfiles, toolProfileBindings, @@ -36,10 +38,10 @@ async function slackAssignedEndpoint( if (!binding.runId || !binding.issueId) return []; let authority: Awaited>; try { - authority = await resolveSlackTaskAuthority( + authority = await measureRunPhase("connectors.slack.authority", () => resolveSlackTaskAuthority( db, binding as SlackTaskBinding, - ); + )); } catch (error) { if ( error && @@ -51,10 +53,10 @@ async function slackAssignedEndpoint( throw error; } // Catalog updates are idempotent and preserve configured policies. - await db.transaction((tx) => + await measureRunTransaction(db, "connectors.slack.catalog", (tx) => syncSlackBotTools(tx, authority.endpoint, authority.userId, true), ); - const [enabledProfile] = await db + const [enabledProfile] = await measureRunPhase("connectors.slack.profile", () => db .select({ id: toolProfiles.id }) .from(toolProfiles) .innerJoin( @@ -81,9 +83,9 @@ async function slackAssignedEndpoint( eq(toolProfiles.status, "active"), ), ) - .limit(1); + .limit(1)); if (!enabledProfile) return []; - authority = await resolveSlackTaskAuthority(db, binding as SlackTaskBinding); + authority = await measureRunPhase("connectors.slack.authority", () => resolveSlackTaskAuthority(db, binding as SlackTaskBinding)); return [ { id: authority.endpoint.id, @@ -106,7 +108,7 @@ export async function slackAssignedResource( binding: Partial & { companyId: string; agentId: string }, ) { const resources = []; - for (const endpoint of await slackEndpointCandidates(db, binding)) { + for (const endpoint of await measureRunPhase("connectors.slack.candidates", () => slackEndpointCandidates(db, binding))) { resources.push( ...(await slackAssignedEndpoint(db, { ...binding, diff --git a/server/src/services/execution-preparation-timing.ts b/server/src/services/execution-preparation-timing.ts index 58fc7a33cc5..0df68c93e30 100644 --- a/server/src/services/execution-preparation-timing.ts +++ b/server/src/services/execution-preparation-timing.ts @@ -14,6 +14,10 @@ type PreparationPhase = "environment_selection" | "execution_configuration" | "s | "workspace_provision_policy" | "workspace_base_validation" | "workspace_source_authorization" + | "workspace_source_materialize" + | "workspace_ownership_read" + | "workspace_environment_acquire" + | "workspace_environment_realize" | "workspace_provision" | "workspace_source_retention" | "workspace_binding" @@ -36,6 +40,9 @@ export type ExecutionPhase = | "workspace.sources.repository_scan" | "workspace.sources.project_scan" | "workspace.provenance_lock" | "workspace.readers" | "workspace.readers.identity" | "workspace.readers.trust" | "workspace.readers.projects" | "workspace.readers.sources" | "workspace.access" | "workspace.access.privacy" | "workspace.access.inventory" | "workspace.access.authorization" + | "workspace.materialize.directories" | "workspace.materialize.source_identity" + | "workspace.checkout.lookup" | "workspace.checkout.git_auth" | "workspace.checkout.git_clone" + | "workspace.checkout.git_ref" | "workspace.checkout.publish" | "runtime_selection.review" | "runtime_selection.contract_read" | "runtime_selection.contract_ensure" | "runtime_selection.resume" | "runtime_selection.recovery_profile" | "runtime_selection.profile" | "runtime_selection.interactions" | "runtime_selection.provider_discovery" | "runtime_selection.bootstrap" | "runtime_selection.run_write" | "runtime_selection.finalization_write" @@ -44,6 +51,26 @@ export type ExecutionPhase = | "runtime_context.instructions" | "runtime_context.skills" | "runtime_context.mcp" | "runtime_context.asset_read" | "runtime_context.asset_manifest" | "runtime_context.asset_cache" | "runtime_context.asset_materialize" | "runtime_context.asset_verify" | "workspace_sync.snapshot_git" | "workspace_sync.snapshot_baseline" | "workspace_sync.pack" | "workspace_sync.seed_verify" | "workspace_sync.stage_workspace" | "workspace_sync.restore" + | "workspace_sync.run_read" | "workspace_sync.descriptor_directory" | "workspace_sync.descriptor_read" + | "workspace_sync.prepare_host_current" | "workspace_sync.prepare_durable_seed" | "workspace_sync.prepare_adopt_remote" + | "workspace_sync.archive_hash" | "workspace_sync.warm_stamp" | "workspace_sync.durable_seed_verify" + | "workspace_sync.manifests_persist" | "workspace_sync.descriptor_write" | "workspace_sync.reference_write" + | "workspace_sync.source_root" | "workspace_sync.repositories" | "workspace_sync.ignore_rules" + | "workspace_sync.seed_generation" | "workspace_sync.seed_lookup" | "workspace_sync.seed_publish" + | "workspace_sync.git_clone" | "workspace_sync.git_archive" | "workspace_sync.seed_replay" | "workspace_sync.seed_persist" + | "workspace_sync.overlay_copy" | "workspace_sync.overlay_archive" | "workspace_sync.deleted_manifest" + | "workspace_sync.seed_extract" | "workspace_sync.seed_snapshot" | "workspace_sync.seed_cleanup" + | "workspace_sync.inbound_barrier" | "workspace_sync.transfer" | "workspace_sync.progress" + | "connectors.browser_use.resolve" | "connectors.speko.resolve" | "connectors.slack.resolve" | "connectors.agentmail.resolve" + | "connectors.browser_use.connection" | "connectors.browser_use.catalog" | "connectors.browser_use.grant" + | "connectors.discovery.scheduled" | "connectors.discovery.queue" | "connectors.discovery.build" | "connectors.discovery.agent" + | "connectors.discovery.connections" | "connectors.discovery.github" | "connectors.discovery.slack" | "connectors.discovery.speko" + | "connectors.discovery.policy" | "connectors.discovery.on_demand_policy" + | "connectors.slack.candidates" | "connectors.slack.authority" | "connectors.slack.profile" + | "connectors.slack.settings" | "connectors.slack.identity" | "connectors.slack.credentials" | "connectors.slack.revision" + | "connectors.slack.authority.transaction" | "connectors.slack.authority.transaction.entry" | "connectors.slack.authority.transaction.commit" | "connectors.slack.authority.transaction.rollback" + | "connectors.slack.catalog.transaction" | "connectors.slack.catalog.transaction.entry" | "connectors.slack.catalog.transaction.commit" | "connectors.slack.catalog.transaction.rollback" + | "connectors.slack.catalog.lock" | "connectors.slack.catalog.entries" | "connectors.slack.catalog.write" | "connectors.slack.catalog.profile" | "execution.runtime" | "execution.workspace_ownership" | "execution.workspace_finalize" | "execution.workspace_restore" | "execution.workspace_commit" | "execution.workspace_ownership_wait" | "execution.realtime" | "execution.usage" | "execution.native_finalize" @@ -138,16 +165,58 @@ export function beginRunPhase(phase: ExecutionPhase): (failed?: boolean) => void } catch { return () => {}; } } +export function measureToolDiscovery(schedule: (work: () => Promise) => Promise, work: () => Promise): Promise { + return measureRunPhase("connectors.discovery.scheduled", async () => { + const queued = beginRunPhase("connectors.discovery.queue"); + const build = AsyncLocalStorage.bind(() => { + queued(); + return measureRunPhase("connectors.discovery.build", work); + }); + let failed = false; + try { return await schedule(build); } + catch (error) { failed = true; throw error; } + finally { queued(failed); } + }); +} + export function createRunPreparationTiming(runId: string, scope: "heartbeat.prepare" | "native.controller" = "heartbeat.prepare") { return (phase: PreparationPhase, work: () => T | Promise): Promise => measure(runId, `${scope}.${phase}`, work); } +const workspaceSyncDetailPhases = { + "workspace.source_root": "workspace_sync.source_root", + "workspace.repositories": "workspace_sync.repositories", + "workspace.ignore_rules": "workspace_sync.ignore_rules", + "workspace.seed_generation": "workspace_sync.seed_generation", + "workspace.seed_lookup": "workspace_sync.seed_lookup", + "workspace.seed_publish": "workspace_sync.seed_publish", + "workspace.git_clone": "workspace_sync.git_clone", + "workspace.git_archive": "workspace_sync.git_archive", + "workspace.seed_replay": "workspace_sync.seed_replay", + "workspace.seed_persist": "workspace_sync.seed_persist", + "workspace.overlay_copy": "workspace_sync.overlay_copy", + "workspace.overlay_archive": "workspace_sync.overlay_archive", + "workspace.deleted_manifest": "workspace_sync.deleted_manifest", + "workspace.seed_extract": "workspace_sync.seed_extract", + "workspace.seed_snapshot": "workspace_sync.seed_snapshot", + "workspace.seed_cleanup": "workspace_sync.seed_cleanup", + "workspace.inbound_barrier": "workspace_sync.inbound_barrier", + "workspace.transfer": "workspace_sync.transfer", + "workspace.progress": "workspace_sync.progress", +} as const satisfies Record; + +export function isWorkspaceSyncDetailPhase(name: string): boolean { + return Object.prototype.hasOwnProperty.call(workspaceSyncDetailPhases, name); +} + +const workspaceSyncPhases: Readonly> = { + ...workspaceSyncDetailPhases, + "snapshot.git": "workspace_sync.snapshot_git", "snapshot.baseline": "workspace_sync.snapshot_baseline", + pack: "workspace_sync.pack", "seed.verify": "workspace_sync.seed_verify", + "stage.workspace": "workspace_sync.stage_workspace", "restore.workspace": "workspace_sync.restore", + "workspace.restore": "execution.workspace_restore", +}; + export function measureWorkspaceSyncPhase(name: string, work: () => Promise): Promise { - const phases: Readonly> = { - "snapshot.git": "workspace_sync.snapshot_git", "snapshot.baseline": "workspace_sync.snapshot_baseline", - pack: "workspace_sync.pack", "seed.verify": "workspace_sync.seed_verify", - "stage.workspace": "workspace_sync.stage_workspace", "restore.workspace": "workspace_sync.restore", - "workspace.restore": "execution.workspace_restore", - }; - return Object.prototype.hasOwnProperty.call(phases, name) ? measureRunPhase(phases[name]!, work) : work(); + return Object.prototype.hasOwnProperty.call(workspaceSyncPhases, name) ? measureRunPhase(workspaceSyncPhases[name]!, work) : work(); } diff --git a/server/src/services/execution-timing-context.test.ts b/server/src/services/execution-timing-context.test.ts index 864b07ccf43..7ac9ca1f6b1 100644 --- a/server/src/services/execution-timing-context.test.ts +++ b/server/src/services/execution-timing-context.test.ts @@ -7,7 +7,8 @@ import { afterEach, beforeEach, expect, it, vi } from "vitest"; import { getStartupTraceContext, getStartupTracer } from "../instrumentation.js"; import { prepareVerifiedRemoteProviderPack } from "./native-runtime/remote-provider-pack.js"; import { createNativeRunTrace } from "./native-runtime/native-run-trace.js"; -import { beginRunPhase, createRunPreparationTiming, getRunExecutionTraceContext, measureRunPhase, withRunExecutionTiming } from "./execution-preparation-timing.js"; +import { beginRunPhase, createRunPreparationTiming, getRunExecutionTraceContext, measureRunPhase, measureToolDiscovery, withRunExecutionTiming } from "./execution-preparation-timing.js"; +import { createToolDiscoveryScheduler } from "./tool-discovery-scheduler.js"; let exporter: InMemorySpanExporter; let provider: TracerProvider; @@ -92,6 +93,54 @@ it("uses a link for a native session whose queue history predates the current at expect(session.startTime[0]).toBeLessThan(attempt.startTime[0]); }); +it("keeps queued discovery under its initiating run when another run releases the slot", async () => { + const scheduler = createToolDiscoveryScheduler(1, 1); + let release!: () => void, started!: () => void, enqueued!: () => void; + const gate = new Promise(resolve => { release = resolve; }); + const ready = new Promise(resolve => { started = resolve; }); + const waiting = new Promise(resolve => { enqueued = resolve; }); + const first = withRunExecutionTiming("discovery-owner", () => measureToolDiscovery( + work => scheduler.run(work), async () => { started(); await gate; }, + )); + await ready; + const runHash = createHash("sha256").update("discovery-waiter").digest("hex").slice(0, 12); + const value = {}; + const work = vi.fn(async () => { + expect(getRunExecutionTraceContext("discovery-owner")).toBeUndefined(); + expect(getRunExecutionTraceContext("discovery-waiter")).toBeDefined(); + expect(exporter.getFinishedSpans().filter(span => span.name === "connectors.discovery.queue" + && span.attributes["paperclip.task.run.run_id"] === runHash)).toHaveLength(1); + await measureRunPhase("connectors.discovery.policy", () => undefined); + return value; + }); + const second = withRunExecutionTiming("discovery-waiter", () => measureToolDiscovery( + work => { const pending = scheduler.run(work); enqueued(); return pending; }, work, + )); + await waiting; + try { + expect(work).not.toHaveBeenCalled(); + expect(exporter.getFinishedSpans().some(span => span.attributes["paperclip.task.run.run_id"] === runHash)).toBe(false); + } finally { + release(); + await first; + } + await expect(second).resolves.toBe(value); + expect(work).toHaveBeenCalledOnce(); + const spans = exporter.getFinishedSpans(); + const owned = spans.filter(span => span.attributes["paperclip.task.run.run_id"] === runHash); + const attempt = owned.find(span => span.name === "task.execution_attempt")!; + const scheduled = owned.find(span => span.name === "connectors.discovery.scheduled")!; + const build = owned.find(span => span.name === "connectors.discovery.build")!; + const queue = owned.filter(span => span.name === "connectors.discovery.queue"); + expect(queue).toHaveLength(1); + expect(owned).toHaveLength(5); + expect(owned.every(span => span.spanContext().traceId === attempt.spanContext().traceId)).toBe(true); + expect(scheduled.parentSpanContext?.spanId).toBe(attempt.spanContext().spanId); + expect(build.parentSpanContext?.spanId).toBe(scheduled.spanContext().spanId); + expect(queue[0].parentSpanContext?.spanId).toBe(scheduled.spanContext().spanId); + expect(owned.find(span => span.name === "connectors.discovery.policy")?.parentSpanContext?.spanId).toBe(build.spanContext().spanId); +}); + it("caps repeated detail spans and ends failed scopes without content or exception text", async () => { const failure = new Error("private path and credential"); const work = vi.fn(() => undefined); diff --git a/server/src/services/execution-transaction-timing.ts b/server/src/services/execution-transaction-timing.ts index bb22c861be0..1cf4583fcd9 100644 --- a/server/src/services/execution-transaction-timing.ts +++ b/server/src/services/execution-transaction-timing.ts @@ -3,7 +3,7 @@ import { beginRunPhase, measureRunPhase } from "./execution-preparation-timing.j type Transaction = Parameters[0]>[0]; -export function measureRunTransaction(db: Db, scope: "workspace" | "runtime_selection" | "response", work: (tx: Transaction) => Promise): Promise { +export function measureRunTransaction(db: Db, scope: "workspace" | "runtime_selection" | "response" | "connectors.slack.authority" | "connectors.slack.catalog", work: (tx: Transaction) => Promise): Promise { return measureRunPhase(`${scope}.transaction`, async () => { const entry = beginRunPhase(`${scope}.transaction.entry`); let settle: ((failed?: boolean) => void) | undefined; diff --git a/server/src/services/heartbeat.ts b/server/src/services/heartbeat.ts index 9cd1ff8635e..221ed1632f5 100644 --- a/server/src/services/heartbeat.ts +++ b/server/src/services/heartbeat.ts @@ -1,5 +1,5 @@ import { dotRunnerBroker } from "./dot-runner-broker.js"; -import { withRunExecutionTiming, measureRunPhase, measureWorkspaceSyncPhase } from "./execution-preparation-timing.js"; +import { withRunExecutionTiming, measureRunPhase, measureWorkspaceSyncPhase, isWorkspaceSyncDetailPhase } from "./execution-preparation-timing.js"; import { createHeartbeatPreparationTiming } from "./heartbeat/preparation-timing.js"; import { createHeartbeatInstructionCopyRelease, createHeartbeatInstructionWorkingCopy } from "./heartbeat/instruction-working-copy.js"; import { prepareHeartbeatRuntimeServiceIntents, startHeartbeatRuntimeServices, recordHeartbeatAdapterRuntimeServices } from "./heartbeat/runtime-services.js"; @@ -3320,10 +3320,11 @@ export function heartbeatService( message: "workspace_checkpoint", payload: { phase: "workspace_checkpoint", ...metrics } }); }, onPhase: async (name, work) => { + if (isWorkspaceSyncDetailPhase(name)) return measureWorkspaceSyncPhase(name, work); const started = Date.now(); try { return await measureWorkspaceSyncPhase(name, work); } - finally { await appendRunEvent(run, { eventType: "lifecycle", stream: "system", level: "info", - message: name, payload: { phase: name, durationMs: Date.now() - started } }); } + finally { await measureRunPhase("workspace_sync.progress", () => appendRunEvent(run, { eventType: "lifecycle", stream: "system", level: "info", + message: name, payload: { phase: name, durationMs: Date.now() - started } })); } }, }, config: { diff --git a/server/src/services/heartbeat/workspace-preparation.ts b/server/src/services/heartbeat/workspace-preparation.ts index 5969afb6ffb..92cb041602a 100644 --- a/server/src/services/heartbeat/workspace-preparation.ts +++ b/server/src/services/heartbeat/workspace-preparation.ts @@ -508,8 +508,8 @@ export async function prepareHeartbeatWorkspace(db: Db, input: HeartbeatWorkspac selectedProjectWorkspaceId: sourceChoice.projectWorkspaceId ?? readNonEmptyString(context.projectWorkspaceId), selection: sourceChoice.selection, strategy: parseObject(hostExecutionWorkspaceConfig.workspaceStrategy), trustPresetKind: trustPreset.kind })) throw error; - const cwd = await materializeIsolatedRemoteWorkspaceSource({ companyId: agent.companyId, - issueId, repoUrl: executionWorkspaceBase.repoUrl!, resolveGitAuth: workspaceGitAuthProvider }); + const cwd = await timeWorkspace("workspace_source_materialize", () => materializeIsolatedRemoteWorkspaceSource({ companyId: agent.companyId, + issueId, repoUrl: executionWorkspaceBase.repoUrl!, resolveGitAuth: workspaceGitAuthProvider })); executionWorkspaceBase.baseCwd = cwd; resolvedWorkspace.cwd = cwd; isolatedRemoteSource = { schema: "paperclip.isolated-remote-source.v1", companyId: agent.companyId, @@ -762,9 +762,9 @@ export async function prepareHeartbeatWorkspace(db: Db, input: HeartbeatWorkspac try { persistedWorktreeInstanceRoot = ( - await readManagedWorktreeInstanceOwnership( - executionWorkspace.worktreePath, - ) + await timeWorkspace("workspace_ownership_read", () => readManagedWorktreeInstanceOwnership( + executionWorkspace.worktreePath!, + )) )?.instanceRoot ?? null; } catch (error) { logger.warn( @@ -1007,7 +1007,7 @@ export async function prepareHeartbeatWorkspace(db: Db, input: HeartbeatWorkspac recoveryWorkspace.remoteCwd !== remoteRecovery.remote.remoteCwd)) { throw new Error("native_remote_recovery_lease_mismatch"); } - acquiredEnvironment = await envOrchestrator.acquireForRun({ + acquiredEnvironment = await timeWorkspace("workspace_environment_acquire", () => envOrchestrator.acquireForRun({ companyId: agent.companyId, selectedEnvironmentId, localEnvironmentId: localEnvironment.id, @@ -1024,7 +1024,7 @@ export async function prepareHeartbeatWorkspace(db: Db, input: HeartbeatWorkspac providerLeaseId: remoteRecovery.remote.providerLeaseId, remoteCwd: remoteRecovery.remote.remoteCwd, } } : {}), - }); + })); await controllerLease.assertOwned(); nativeRunnerPreparationSpans.push({ name: "environment.acquire", @@ -1100,7 +1100,7 @@ export async function prepareHeartbeatWorkspace(db: Db, input: HeartbeatWorkspac ReturnType >; try { - realizationResult = await envOrchestrator.realizeForRun({ + realizationResult = await timeWorkspace("workspace_environment_realize", () => envOrchestrator.realizeForRun({ environment: selectedEnvironment, lease: activeEnvironmentLease.lease, adapterType: agent.adapterType, @@ -1111,7 +1111,7 @@ export async function prepareHeartbeatWorkspace(db: Db, input: HeartbeatWorkspac effectiveExecutionWorkspaceMode, persistedExecutionWorkspace, duplexObservabilityRecorder, - }); + })); nativeRunnerPreparationSpans.push({ name: "environment.workspace.realize", parentName: "task.provider_session", diff --git a/server/src/services/isolated-remote-workspace-source.ts b/server/src/services/isolated-remote-workspace-source.ts index 26c949fdfbc..efb299db2e1 100644 --- a/server/src/services/isolated-remote-workspace-source.ts +++ b/server/src/services/isolated-remote-workspace-source.ts @@ -6,6 +6,7 @@ import type { TaskWorkspaceIntent } from "@paperclipai/shared"; import type { GitRemoteAuthProvider } from "./git-credentials.js"; import { resolvePaperclipInstanceRoot } from "../home-paths.js"; import { materializeManagedProjectWorkspace } from "./managed-repository-checkout.js"; +import { measureRunPhase } from "./execution-preparation-timing.js"; const execFile = promisify(execFileCallback); @@ -89,32 +90,35 @@ export async function materializeIsolatedRemoteWorkspaceSource(input: { resolveGitAuth: GitRemoteAuthProvider; }): Promise { const cwd = isolatedRemoteSourcePath(input.companyId, input.issueId); - const root = resolvePaperclipInstanceRoot(); - await fs.mkdir(root, { recursive: true }); - const canonicalRoot = await fs.realpath(root); - let parent = canonicalRoot; - for (const segment of ["isolated-repository-sources", input.companyId, input.issueId]) { - parent = path.join(parent, segment); - await fs.mkdir(parent, { mode: 0o700 }).catch((error: NodeJS.ErrnoException) => { - if (error.code !== "EEXIST") throw error; - }); - const entry = await fs.lstat(parent); - if (!entry.isDirectory() || entry.isSymbolicLink() || await fs.realpath(parent) !== parent) { - throw new Error("Isolated repository source path is not a private directory"); + const canonicalCwd = await measureRunPhase("workspace.materialize.directories", async () => { + const root = resolvePaperclipInstanceRoot(); + await fs.mkdir(root, { recursive: true }); + const canonicalRoot = await fs.realpath(root); + let parent = canonicalRoot; + for (const segment of ["isolated-repository-sources", input.companyId, input.issueId]) { + parent = path.join(parent, segment); + await fs.mkdir(parent, { mode: 0o700 }).catch((error: NodeJS.ErrnoException) => { + if (error.code !== "EEXIST") throw error; + }); + const entry = await fs.lstat(parent); + if (!entry.isDirectory() || entry.isSymbolicLink() || await fs.realpath(parent) !== parent) { + throw new Error("Isolated repository source path is not a private directory"); + } } - } - const canonicalCwd = path.join(parent, "checkout"); - const entry = await fs.lstat(canonicalCwd).catch((error: NodeJS.ErrnoException) => { - if (error.code === "ENOENT") return null; - throw error; + const checkout = path.join(parent, "checkout"); + const entry = await fs.lstat(checkout).catch((error: NodeJS.ErrnoException) => { + if (error.code === "ENOENT") return null; + throw error; + }); + if (entry && (!entry.isDirectory() || entry.isSymbolicLink())) { + throw new Error("Isolated repository source cannot be a symbolic link"); + } + return checkout; }); - if (entry && (!entry.isDirectory() || entry.isSymbolicLink())) { - throw new Error("Isolated repository source cannot be a symbolic link"); - } const result = await materializeManagedProjectWorkspace(canonicalCwd, { repoUrl: input.repoUrl, resolveGitAuth: input.resolveGitAuth, }); if (result.warning) throw new Error("Isolated repository source is not a Git checkout"); - await assertRemoteSourceIdentity(cwd, input.repoUrl); + await measureRunPhase("workspace.materialize.source_identity", () => assertRemoteSourceIdentity(cwd, input.repoUrl)); return cwd; } diff --git a/server/src/services/managed-repository-checkout.ts b/server/src/services/managed-repository-checkout.ts index f8810406f06..e8316869473 100644 --- a/server/src/services/managed-repository-checkout.ts +++ b/server/src/services/managed-repository-checkout.ts @@ -8,6 +8,7 @@ import { sanitizeRuntimeServiceBaseEnv } from "./runtime-service-env.js"; import { describeGitAuthFailure, scrubGitCredentialText, type GitRemoteAuthProvider } from "./git-credentials.js"; import { classifyGitCloneFailure, GitConnectionFailureError, readGitConnectionFailure } from "./git-connection-failure.js"; import { isWorkspaceGitScanError, WorkspaceGitScanError } from "./workspace-git-operation-scheduler.js"; +import { measureRunPhase } from "./execution-preparation-timing.js"; const execFile = promisify(execFileCallback); const MANAGED_WORKSPACE_GIT_CLONE_TIMEOUT_MS = 10 * 60 * 1000; @@ -34,11 +35,11 @@ export async function materializeManagedProjectWorkspace( return { cwd, warning: null }; } - const hasAdoptableGitDir = () => + const hasAdoptableGitDir = () => measureRunPhase("workspace.checkout.lookup", () => fs .stat(path.resolve(cwd, ".git")) .then((entry) => entry.isDirectory()) - .catch(() => false); + .catch(() => false)); if (await hasAdoptableGitDir()) { return { cwd, warning: null }; } @@ -55,17 +56,18 @@ export async function materializeManagedProjectWorkspace( } const auth = input.resolveGitAuth && !input.localSource - ? await input.resolveGitAuth(input.repoUrl) + ? await measureRunPhase("workspace.checkout.git_auth", () => input.resolveGitAuth!(input.repoUrl!)) : null; const cloneTmpDir = await fs.mkdtemp(input.stagingParent ? path.join(input.stagingParent, `${path.basename(cwd)}.clone-`) : `${cwd}.clone-`); + const cloneSource = input.localSource ?? input.repoUrl; try { await input.beforeCopy?.(cloneTmpDir); try { - await execFile( + await measureRunPhase("workspace.checkout.git_clone", () => execFile( "git", - [...(auth?.configArgs ?? []), "clone", "--no-hardlinks", "--", input.localSource ?? input.repoUrl, cloneTmpDir], + [...(auth?.configArgs ?? []), "clone", "--no-hardlinks", "--", cloneSource, cloneTmpDir], { env: { ...sanitizeRuntimeServiceBaseEnv(process.env), @@ -74,7 +76,7 @@ export async function materializeManagedProjectWorkspace( }, timeout: MANAGED_WORKSPACE_GIT_CLONE_TIMEOUT_MS, }, - ); + )); } catch (error) { const connectionFailure = !input.localSource ? classifyGitCloneFailure(input.repoUrl, error) : null; if (connectionFailure) throw new GitConnectionFailureError( @@ -95,7 +97,7 @@ export async function materializeManagedProjectWorkspace( } await execFile("git", ["-C", cloneTmpDir, "remote", "set-url", "origin", input.repoUrl], { timeout: 10_000 }); } else if (input.repoRef) { - await execFile("git", ["-C", cloneTmpDir, "checkout", input.repoRef], { timeout: MANAGED_WORKSPACE_GIT_CLONE_TIMEOUT_MS }); + await measureRunPhase("workspace.checkout.git_ref", () => execFile("git", ["-C", cloneTmpDir, "checkout", input.repoRef!], { timeout: MANAGED_WORKSPACE_GIT_CLONE_TIMEOUT_MS })); } await input.beforePublish?.(cloneTmpDir); } catch (error) { @@ -117,7 +119,7 @@ export async function materializeManagedProjectWorkspace( } try { - await fs.rename(cloneTmpDir, cwd); + await measureRunPhase("workspace.checkout.publish", () => fs.rename(cloneTmpDir, cwd)); } catch (renameError) { await fs .rm(cloneTmpDir, { recursive: true, force: true }) diff --git a/server/src/services/native-runtime/native-workspace-sync.ts b/server/src/services/native-runtime/native-workspace-sync.ts index 8c1399b64fc..99fe0a61c83 100644 --- a/server/src/services/native-runtime/native-workspace-sync.ts +++ b/server/src/services/native-runtime/native-workspace-sync.ts @@ -182,13 +182,13 @@ function durableSeedPaths(runId: string): { } async function sha256File(filePath: string): Promise { - return await new Promise((resolveDigest, rejectDigest) => { + return measureRunPhase("workspace_sync.archive_hash", () => new Promise((resolveDigest, rejectDigest) => { const digest = createHash("sha256"); const stream = createReadStream(filePath); stream.on("data", (chunk) => digest.update(chunk)); stream.on("error", rejectDigest); stream.on("end", () => resolveDigest(digest.digest("hex"))); - }); + })); } async function verifiedDurableSeed(input: { @@ -795,7 +795,12 @@ async function prepareRuntime(input: { onPhase?: RuntimeSpanRunner; onCheckpoint?: (metrics: WorkspaceCheckpointMetrics) => Promise; }): Promise { - return prepareAdapterExecutionTargetRuntime({ + const phase = { + host_current: "workspace_sync.prepare_host_current", + durable_seed: "workspace_sync.prepare_durable_seed", + adopt_remote: "workspace_sync.prepare_adopt_remote", + } as const; + return measureRunPhase(phase[input.mode], () => prepareAdapterExecutionTargetRuntime({ runId: input.runId, target: input.target, adapterKey: "paperclip-runner", @@ -812,7 +817,7 @@ async function prepareRuntime(input: { workspaceSeedCacheDirectory: input.seedCacheDirectory, workspaceSeedCacheCompanyDirectory: input.seedCacheCompanyDirectory, runtimeSpan: input.onPhase, - }); + })); } async function finalizePreparedRuntime(input: { @@ -889,12 +894,12 @@ export async function prepareNativeWorkspaceSync(input: { input.restartRecovery.remote.remoteCwd !== target.remoteCwd)) { throw new Error("native_remote_recovery_lease_mismatch"); } - const run = await input.db + const run = await measureRunPhase("workspace_sync.run_read", () => input.db .select({ runnerProfileJson: heartbeatRuns.runnerProfileJson }) .from(heartbeatRuns) .where(eq(heartbeatRuns.id, input.runId)) .limit(1) - .then((rows) => rows[0] ?? null); + .then((rows) => rows[0] ?? null)); if (!run) throw new Error("native_workspace_sync_run_missing"); const existingReference = readNativeWorkspaceSyncReference( parseObject(run.runnerProfileJson).nativeWorkspaceSync, @@ -904,14 +909,14 @@ export async function prepareNativeWorkspaceSync(input: { let descriptor: NativeWorkspaceSyncDescriptor; let mode: WorkspaceInboundMode = "host_current"; const seedPaths = durableSeedPaths(input.runId); - await ensurePrivateDirectory(descriptorDirectory(input.runId)); + await measureRunPhase("workspace_sync.descriptor_directory", () => ensurePrivateDirectory(descriptorDirectory(input.runId))); try { if (existingReference) { - const existing = await readDescriptor({ + const existing = await measureRunPhase("workspace_sync.descriptor_read", () => readDescriptor({ runId: input.runId, reference: existingReference, - }); + })); if ( existing.descriptor.binding.companyId !== input.companyId || existing.descriptor.binding.workspaceId !== input.workspaceId || @@ -932,12 +937,12 @@ export async function prepareNativeWorkspaceSync(input: { const durableSeed = mode === "durable_seed" ? existing.descriptor.seed - ? await verifiedDurableSeed({ + ? await measureRunPhase("workspace_sync.durable_seed_verify", () => verifiedDurableSeed({ runId: input.runId, - seed: existing.descriptor.seed, + seed: existing.descriptor.seed!, gitSnapshot: existing.descriptor.gitSnapshot, repositories: existing.descriptor.repositories, - }) + })) : (() => { throw new Error("workspace_sync_out_unrecoverable"); })() @@ -1005,16 +1010,16 @@ export async function prepareNativeWorkspaceSync(input: { ); const verifiedWarmAdoption = priorStamp.hostSha256 === currentHostSha256 && - (await remoteStampMatches({ target, expected: priorStamp, gitSnapshot: currentSnapshot.gitSnapshot, repositories: currentSnapshot.repositories })); + (await measureRunPhase("workspace_sync.warm_stamp", () => remoteStampMatches({ target, expected: priorStamp, gitSnapshot: currentSnapshot.gitSnapshot, repositories: currentSnapshot.repositories }))); if (verifiedWarmAdoption) { mode = "adopt_remote"; } else { mode = "host_current"; let preparedSeed: WorkspaceDurableSeedPaths | undefined; try { - preparedSeed = await verifiedDurableSeed({ runId: input.runId, + preparedSeed = await measureRunPhase("workspace_sync.durable_seed_verify", () => verifiedDurableSeed({ runId: input.runId, seed: { workspaceArchiveSha256, gitArchiveSha256 }, - gitSnapshot: currentSnapshot.gitSnapshot, repositories: currentSnapshot.repositories }); + gitSnapshot: currentSnapshot.gitSnapshot, repositories: currentSnapshot.repositories })); } catch (error) { if (!(error instanceof Error) || error.message !== "workspace_sync_out_unrecoverable") throw error; } @@ -1041,7 +1046,7 @@ export async function prepareNativeWorkspaceSync(input: { } const snapshot = runtime.workspaceSyncSnapshot; if (!snapshot) throw new Error("native_workspace_sync_snapshot_missing"); - await persistSnapshotManifests(input.runId, snapshot); + await measureRunPhase("workspace_sync.manifests_persist", () => persistSnapshotManifests(input.runId, snapshot)); const now = new Date().toISOString(); descriptor = { schema: DESCRIPTOR_SCHEMA, @@ -1068,8 +1073,8 @@ export async function prepareNativeWorkspaceSync(input: { } const preparedRuntime = runtime; - let reference = await writeDescriptor(descriptor); - await persistRunReference(input.db, input.runId, reference); + let reference = await measureRunPhase("workspace_sync.descriptor_write", () => writeDescriptor(descriptor)); + await measureRunPhase("workspace_sync.reference_write", () => persistRunReference(input.db, input.runId, reference)); let restorePromise: Promise | null = null; return { mode, diff --git a/server/src/services/tool-gateway.ts b/server/src/services/tool-gateway.ts index 3c28f51db62..bb1c0e9192a 100644 --- a/server/src/services/tool-gateway.ts +++ b/server/src/services/tool-gateway.ts @@ -1,4 +1,5 @@ import { notifyDeliveryWork, DELIVERY_QUEUES } from "./delivery-work-notifications.js"; +import { measureToolDiscovery, measureRunPhase } from "./execution-preparation-timing.js"; import { composeConnectionInstructions } from "./connection-instructions.js"; import { isInsufficientConnectionScope, INSUFFICIENT_CONNECTION_SCOPE_MESSAGE } from "./connection-permission-errors.js"; import { boundedMcpToolName } from "./mcp-tool-names.js"; @@ -3026,7 +3027,7 @@ export function createToolGatewayService( signal?: AbortSignal, ): Promise { try { - return await toolDiscoveryScheduler.run(() => buildToolsForContext(session, signal), signal); + return await measureToolDiscovery(work => toolDiscoveryScheduler.run(work, signal), () => buildToolsForContext(session, signal)); } catch (error) { if (error instanceof ToolDiscoveryBusyError) { throw new ToolGatewayHttpError(503, error.message, "tool_discovery_busy"); @@ -3037,13 +3038,13 @@ export function createToolGatewayService( /** Shared connection descriptors and task restrictions; no provider calls. */ async function connectionToolsForContext(session: ToolGatewaySession): Promise { - const guestBotConnection = await githubGuestBotConnectionForSession(db, session); - const connectedTools = (await connectedMcpToolsForCompany(session.companyId)) + const guestBotConnection = await measureRunPhase("connectors.discovery.github", () => githubGuestBotConnectionForSession(db, session)); + const connectedTools = (await measureRunPhase("connectors.discovery.connections", () => connectedMcpToolsForCompany(session.companyId))) .filter(tool => !guestBotConnection || !tool.connectionId || (tool.connectionId === guestBotConnection && tool.providerType === "paperclip_github_chat")); return [ - ...await githubBotToolsForSession(db, session), - ...await slackToolsForSession(db, session), - ...await spekoToolsForSession(db, session), + ...await measureRunPhase("connectors.discovery.github", () => githubBotToolsForSession(db, session)), + ...await measureRunPhase("connectors.discovery.slack", () => slackToolsForSession(db, session)), + ...await measureRunPhase("connectors.discovery.speko", () => spekoToolsForSession(db, session)), ...connectedTools, ]; } @@ -3054,7 +3055,7 @@ export function createToolGatewayService( ): Promise { signal?.throwIfAborted(); if (session.agentId) { - await assertAgentInCompany(session.companyId, session.agentId); + await measureRunPhase("connectors.discovery.agent", () => assertAgentInCompany(session.companyId, session.agentId!)); } const connectionTools = await connectionToolsForContext(session); const onDemandTargets = connectionTools.filter(isOnDemandRemoteTool); @@ -3068,12 +3069,12 @@ export function createToolGatewayService( tool.providerType !== "paperclip_plugin"), ); const decisionCache = createToolAccessDecisionCache(); - const decisions = await decideToolsForListing( + const decisions = await measureRunPhase("connectors.discovery.policy", () => decideToolsForListing( tools, (tool) => policyInputForTool({ session, tool }), decisionCache, signal, - ); + )); const visibleTools = decisions .filter( ({ decision }) => @@ -3093,12 +3094,12 @@ export function createToolGatewayService( : tool, ); if (onDemandTargets.length > 0) { - const targetDecisions = await decideToolsForListing( + const targetDecisions = await measureRunPhase("connectors.discovery.on_demand_policy", () => decideToolsForListing( onDemandTargets, (tool) => policyInputForTool({ session, tool }), decisionCache, signal, - ); + )); if ( targetDecisions.some( ({ decision }) => @@ -9134,19 +9135,19 @@ export function createToolGatewayService( async browserUseResources(binding: { companyId: string; agentId: string; runId?: string; issueId?: string }) { if (!binding.runId || !binding.issueId) return []; - const [connection] = await db.select({ id: toolConnections.id }).from(toolConnections).where(and( + const [connection] = await measureRunPhase("connectors.browser_use.connection", () => db.select({ id: toolConnections.id }).from(toolConnections).where(and( eq(toolConnections.companyId, binding.companyId), eq(toolConnections.transport, "rest_api"), eq(toolConnections.enabled, true), eq(toolConnections.status, "active"), sql`${toolConnections.config}->>'sourceTemplateKey' = 'browser-use-cloud'`, - )).limit(1); + )).limit(1)); if (!connection) return []; const session: ToolGatewaySession = { ...binding, runId: binding.runId, issueId: binding.issueId, id: randomUUID(), token: "", projectId: null, createdAt: new Date(), expiresAt: new Date(Date.now() + 60000) }; - const tools = (await listToolsForContext(session)).filter(t => t.providerType === "provider_rest"); + const tools = (await measureRunPhase("connectors.browser_use.catalog", () => listToolsForContext(session))).filter(t => t.providerType === "provider_rest"); const resources: Array<{ id: string; label: string; connectionId: string }> = []; for (const connectionId of new Set(tools.map(t => t.connectionId!))) { - const [connection] = await db.select().from(toolConnections).where(and(eq(toolConnections.companyId, binding.companyId), eq(toolConnections.id, connectionId))); + const [connection] = await measureRunPhase("connectors.browser_use.connection", () => db.select().from(toolConnections).where(and(eq(toolConnections.companyId, binding.companyId), eq(toolConnections.id, connectionId)))); if (!connection || !isBrowserUseConnection(connection)) continue; - try { await resolveConnectionGrant(session, connection, false); resources.push({ id: connectionId, connectionId, label: "Browser Use Cloud" }); } catch { /* Unavailable grants do not contribute a runtime skill. */ } + try { await measureRunPhase("connectors.browser_use.grant", () => resolveConnectionGrant(session, connection, false)); resources.push({ id: connectionId, connectionId, label: "Browser Use Cloud" }); } catch { /* Unavailable grants do not contribute a runtime skill. */ } } return resources; }, From 68a09dcb5f50e3fb6ceaaeac2706d96db97fc9b3 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Thu, 15 Oct 2026 12:22:59 -0500 Subject: [PATCH 15/19] perf(ci): shorten the test execution critical path (#16434) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Contributors wait for the full PR workflow before they can deliver fixes. > - Server shards still execute independent files in series, and a single Runner file serializes two long real-clock waits. > - Faster work outside the critical path does not prove a faster workflow. > - Overlap independent files while preserving isolation, test cases and real production deadlines. > - Compare complete baseline and candidate workflows, separating queue delay from execution cost. ## Linked Issues or Issue Description Refs #16429. Related duration refresh: #12577. **What existing behavior does this improve?** The elapsed time from PR CI creation to the last required check. **Current behavior** General server shards use one worker. One Runner transport file takes about 333 seconds on Linux, including over 100 seconds in two real-clock regressions. These are the longest verification lanes after #16429. **Proposed behavior** Run independent general-server files in at most two isolated forks. Move the two intact Runner deadline regressions into a separate file to overlap their waits in the existing pool. Run two chat partitions per existing job, each with its own temporary home and embedded PostgreSQL. Preserve serial execution within files and every assertion and timing boundary. ## What Changed - Qualify both serial and parallel forks with real worker processes, isolated homes/configuration, timers, globals and environment. - Require both parallel workers to arrive before either exits. - Use two isolated forks for sharded general-server tests; retain one worker with an explicit shared test database. Other test groups keep their existing concurrency. - Extract the lease-expiry and Pi cold-admission regressions verbatim into a separate Runner test file; retain real waits and process/lease assertions. - Split each sharded chat job into two independent invocations; preserve serial shared-database execution. Await both children before failing the parent. Prove disjoint coverage and failure propagation. - Update the paired Linux benchmark to compare a complete general-server shard with identical case inventories. - Document concurrency and deadline-test placement. No production behavior, workflow topology or runner count changes. - Allow 15 seconds for snapshot integration cases that cold-start multiple real TypeScript workers. On the repeat, one-worker cases took 2.5–3 seconds, so queued or sequential starts could exceed the default five seconds. Preserve all cleanup, queue-release and filesystem assertions; these cases do not enforce a production latency deadline. ## Verification All 48 jobs passed on scheduling head `5d718e18` in [attempt 1](https://github.com/paperclipai/paperclip/actions/runs/38763885288/attempts/1). The [bounded repeat](https://github.com/paperclipai/paperclip/actions/runs/38763885288/attempts/2) failed two existing snapshot-worker tests at their default five-second timeout. Its shorter duration is **not** counted as a successful performance sample. The latest head `3c894462` adds a targeted test-budget repair and [passes all 48 CI jobs](https://github.com/paperclipai/paperclip/actions/runs/38768433532) in **8m06s** (workflow creation to final required workflow job). This is **70 seconds / 12.6% shorter** than the 9m16s intermediate run, but the earlier 9m23s result remains part of the evidence: a stable average improvement is not established. Both branch-protection checks (`ci / verify` and `ci / e2e`) pass. The non-required Superagent Security Scan is still running; external PR scans are separate from this workflow duration. | Measurement | Before | After | What it establishes | | --- | ---: | ---: | --- | | General-server shard, paired on one Linux host | 430.94s | 292.99s | 32.0% faster; identical 1,047 case inventory | | Chat job 1 | 460s | 211s | Same full chat inventory spread across isolated partitions | | Chat job 2 | 408s | 274s | Serial cases within each child, separate PostgreSQL instances | | Chat job 3 | 408s | 216s | Total chat runner time fell from 1,276s to 701s | | Full PR workflow, first chat candidate | 9m16s | 9m23s | Not an end-to-end win in this pair: unchanged Build rose from 365s to 465s | | Full PR workflow, final repaired head | 9m16s | 8m06s | All 48 jobs pass; 70 seconds faster observed, not a controlled causal estimate | - Latest [paired benchmark](https://github.com/paperclipai/paperclip/actions/runs/38763885223) passes. Earlier paired server measurements were 33.3% and 37.4% faster. Unchanged fixture timing noise is not attributed to this PR. - All **1,337 chat case identities and outcomes match exactly** across [before-chat](https://github.com/paperclipai/paperclip/actions/runs/38760937988) and [after-chat](https://github.com/paperclipai/paperclip/actions/runs/38763885288/attempts/1), without duplicates or removals. Whole-suite records rose from 44,563 to 44,574: three new launcher checks and eight upstream warm-owner tests. Remaining name differences were generated fixture IDs/timestamps/PIDs, not removed tests. - The final run used `m7i-flex.xlarge` for Build (408s), with all chat jobs completing in 238s / 318s / 280s. It used the same actual merge base `afe5b341` as the first chat candidate. Optional profiling was disabled after the bounded measurement experiment; the full 48-job correctness workflow still ran. - The fleet used `m7i.xlarge` for the earlier Build and `m6a.xlarge` for the first chat candidate Build. Several unchanged compile/install/probe phases slowed together. This does not prove hardware alone caused the difference, and full-run attribution must account for runner/cache variance and the upstream base change. - Original full baseline: [all 48 jobs passed](https://github.com/paperclipai/paperclip/actions/runs/38753547876/attempts/2), 12m07s, including a server job that queued over five minutes. It is not a clean basis for attributing the whole elapsed-time reduction to this PR. An interrupted baseline and two misrouted candidate attempts were excluded and preserved; routing/trust rules were not weakened. - Local: 32 sharding/launcher checks pass, including disjoint coverage, shared-database fallback and a failed child that waits for its sibling. Real paired chat partitions passed 416 cases in 217.63s on independent embedded databases. - Both extracted Runner deadline regressions pass unchanged with real waits. An initial local attempt rejected a Node executable under the controller home; the identical binary in an owned non-home path passed without weakening that guard. - Real worker-isolation proof, actionlint and diff checks pass. Latest-head Greptile is **5/5**, with no review threads or actionable findings. - The eight snapshot-worker cases pass locally with a private `PAPERCLIP_HOME`. The first direct local invocation failed to open the default-home lock database under the sandbox; no production guard was changed. ## Risks Parallel files and paired chat databases may expose shared-state assumptions or increase peak memory. Cases stay serial within each invocation, and explicit shared databases keep the serial path. Complete server coverage and the full 48-job CI workflow pass. Check external required PR scans before merge; the full-workflow measurements are observational and do not establish a steady 12.6% speedup. Queueing, cache state and runner variance must be distinguished from execution improvements. ## Model Used OpenAI GPT-6 through Codex, with repository inspection, code execution and GitHub tools. The exact serving model suffix and context capacity are not exposed to this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green (48 jobs; both required branch-protection checks pass; non-required external security scan pending) - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- .github/workflows/pr-trusted.yml | 23 +-- doc/DEVELOPING.md | 10 ++ doc/ci-profiling.md | 2 +- .../src/workspace-snapshot-thread.test.ts | 7 +- .../runnerd-codex-transport-deadlines.test.ts | 152 ++++++++++++++++++ .../src/live/runnerd-codex-transport.test.ts | 128 --------------- .../run-vitest-stable-shard.test.mjs | 69 +++++++- .../vitest-worker-isolation.test.mjs | 21 ++- scripts/benchmark-ci-work.mjs | 4 +- scripts/run-vitest-stable.mjs | 108 ++++++++----- 10 files changed, 324 insertions(+), 200 deletions(-) create mode 100644 packages/paperclip-runner/src/live/runnerd-codex-transport-deadlines.test.ts diff --git a/.github/workflows/pr-trusted.yml b/.github/workflows/pr-trusted.yml index d870c77d9eb..682fc9762f3 100644 --- a/.github/workflows/pr-trusted.yml +++ b/.github/workflows/pr-trusted.yml @@ -354,26 +354,9 @@ jobs: fail-fast: false matrix: include: - # The server suite is pinned to maxWorkers=1 (server/vitest.config.ts), - # so it can only be parallelized across runners. Shard it to keep this - # lane off the PR critical path. The suite has grown to ~3184s of - # serial vitest wall time (mean of runs 35036001734 and 35024948947, - # 2026-09-15): at five plain general-server shards with stale - # recorded durations the worst shard ran 806s of tests while the - # best ran 417s. This now uses the same shape as release-verify.yml: - # the ~429s chat integration suite splits by collected test location - # across three dedicated runners (~143s each), and the remaining - # ~2755s levels across twelve duration-balanced shards at ~230s - # each, in line with the other ~200-290s lanes. - # - # The balance only holds while the manifest is fresh. By 2026-10-10 - # the without-chat group had grown to 871 suites and ~5031s of serial - # vitest time (mean of runs 38076844891 and 38076972444), 230 of - # them unrecorded and weighted at the median, so the twelve shards - # ran 313-554s of tests (server 2/12 at 618s was the slowest check - # in the run). Re-recorded durations level them to ~415s each; the - # shard count stays at twelve because Build and workspaces-b sit - # above that already. + # Keep twelve duration-balanced server shards. Each shard overlaps + # at most two isolated fork workers; cases within a file stay serial. + # The chat fixture keeps three independent database-backed shards. - group: general-server-without-chat group_label: server (1/12) shard_index: 0 diff --git a/doc/DEVELOPING.md b/doc/DEVELOPING.md index 09e8038a57b..6fb6f0f6643 100644 --- a/doc/DEVELOPING.md +++ b/doc/DEVELOPING.md @@ -75,6 +75,16 @@ When changing the workflow path or branch, authorize the new reference before updating the caller. Retain older authorized SHA references while queued runs or supported reruns still use them. +General server shards run at most two test files at a time in separate Vitest +forks. Each file retains its own home, config and process state, and its test +cases run serially. An explicit `PAPERCLIP_TEST_DATABASE_URL` keeps those shards +serial because the database is shared. Unsharded local tests, route/authz lanes, +and native Runner lanes retain their existing concurrency. Sharded chat jobs run +two independent partitions, each with its own embedded PostgreSQL, home and +config; an explicit shared test database retains one partition per job. +The Runner transport lease-expiry and cold-admission regressions live in their +own file so those real-clock waits overlap other files in the existing pool. + For a full wall-clock profile of all main PR CI jobs, add `profile-ci` to the PR. See [CI profiling](ci-profiling.md) for the Honeycomb waterfall, coverage, operator setup, and activation requirements. diff --git a/doc/ci-profiling.md b/doc/ci-profiling.md index fe8472a7577..49aac6a4ebc 100644 --- a/doc/ci-profiling.md +++ b/doc/ci-profiling.md @@ -162,7 +162,7 @@ readiness fixtures, `profile-ci` also runs **CI work benchmark**. It uses one GitHub-hosted runner without secrets or AWS access. It compares the unchanged PR base and candidate sequentially. It records complete command times, test inventories, revisions, and failure logs in the `ci-work-benchmark` artifact. -The serialized shard must retain the same cases. The fixture comparison retains +The complete general-server shard must retain the same cases. The fixture comparison retains the database contract and splits large Git scanning from bounded staging. These command timings exclude queue time and setup; use the full Honeycomb waterfall to assess the complete CI critical path. diff --git a/packages/adapter-utils/src/workspace-snapshot-thread.test.ts b/packages/adapter-utils/src/workspace-snapshot-thread.test.ts index 5830b7039aa..53aaba4b877 100644 --- a/packages/adapter-utils/src/workspace-snapshot-thread.test.ts +++ b/packages/adapter-utils/src/workspace-snapshot-thread.test.ts @@ -81,6 +81,9 @@ describe("workspace snapshot threads", () => { await expect(fs.stat(created[0]!)).rejects.toMatchObject({ code: "ENOENT" }); }); + // Multiple real TypeScript worker cold starts can exceed the default 5s on + // CI (three queued workers use two slots). These cases verify cleanup and + // error propagation, not a worker startup deadline. Keep completion bounded. it("preserves filesystem errors, cleans failed workers and releases queued jobs", async () => { const root = await fixture(); const file = path.join(root, "file"); @@ -103,7 +106,7 @@ describe("workspace snapshot threads", () => { if (results[2].status === "fulfilled") await disposeDirectorySnapshot(results[2].value); expect(created).toHaveLength(3); for (const directory of created) await expect(fs.stat(directory)).rejects.toMatchObject({ code: "ENOENT" }); - }); + }, 15_000); it.each(["unchanged", "changed", "sparse"] as const)("isolates restore captures and cleans their manifests: %s", async mode => { const target = await fixture(), source = await fixture(); @@ -129,7 +132,7 @@ describe("workspace snapshot threads", () => { expect(await fs.readFile(path.join(target, "file"), "utf8")).toBe(mode === "unchanged" ? "before" : "after"); expect(await fs.readFile(path.join(target, "concurrent"), "utf8")).toBe("host bytes"); for (const directory of created) await expect(fs.stat(directory)).rejects.toMatchObject({ code: "ENOENT" }); - }); + }, 15_000); it("retains the in-memory snapshot contract when disk backing is not requested", async () => { const root = await fixture(); diff --git a/packages/paperclip-runner/src/live/runnerd-codex-transport-deadlines.test.ts b/packages/paperclip-runner/src/live/runnerd-codex-transport-deadlines.test.ts new file mode 100644 index 00000000000..1f0fbaef19f --- /dev/null +++ b/packages/paperclip-runner/src/live/runnerd-codex-transport-deadlines.test.ts @@ -0,0 +1,152 @@ +import { chmod, cp, mkdir, mkdtemp, readFile, rm, stat, writeFile } from "node:fs/promises"; +import { readFileSync } from "node:fs"; +import { createHash } from "node:crypto"; +import { tmpdir } from "node:os"; +import { join, resolve } from "node:path"; +import { fileURLToPath } from "node:url"; +import { expect, it, vi } from "vitest"; +import { QUALIFIED_ACPX_PROFILES } from "../drivers/acpx/qualified-profiles.js"; +import { codexSemanticToolSpecs } from "../drivers/codex/codex-app-server-driver.js"; +import { DurablePrpControlPlane } from "../control-plane/durable-prp-control-plane.js"; +import * as durableControlPlane from "../control-plane/durable-prp-control-plane.js"; +import { + createCapabilityRunnerdCodexTransport, + defaultCapabilityRunnerdBinary as qualifiedCapabilityRunnerdBinary, +} from "./runnerd-codex-transport.js"; + +// These regressions must cross real production timing boundaries. Keep their +// waits intact, but let the existing file worker pool overlap them with the +// rest of the transport suite instead of extending its serial critical path. +const defaultCapabilityRunnerdBinary = () => + process.env.PAPERCLIP_ATTACH_TRANSITION_RUNNER ?? qualifiedCapabilityRunnerdBinary(); +const fakeCodex = resolve(import.meta.dirname, "../../runner/target/debug/fake-codex-app-server"); +function fakeCodexArgs(stateDirectory: string, ...args: string[]): string[] { + return ["--state-file", join(stateDirectory, "fake-codex-state.json"), ...args]; +} + +it("keeps a quiet active Codex turn in the same process across connection lease expiry", async () => { + const stateDirectory = await mkdtemp(join(tmpdir(), "runnerd-renew-active-")); + const callsPath = join(stateDirectory, "calls.log"); + const cores: DurablePrpControlPlane[] = []; + const OriginalCore = durableControlPlane.DurablePrpControlPlane; + const coreSpy = vi.spyOn(durableControlPlane, "DurablePrpControlPlane") + .mockImplementation(function(options: ConstructorParameters[0]) { + const core = new OriginalCore({ ...options, connectionLeaseTtlMs: 60_000 }); + cores.push(core); + return core; + } as unknown as typeof OriginalCore); + const bundle = createCapabilityRunnerdCodexTransport({ + runnerBinary: defaultCapabilityRunnerdBinary(), codexCommand: fakeCodex, + codexArgs: fakeCodexArgs(stateDirectory, "--hold-turn", "--record-process-start", "--call-log", callsPath), + stateDirectory, runnerReconnectGraceMs: 5_000, + }); + try { + const opened = await bundle.transport.request("thread/start", { cwd: tmpdir(), dynamicTools: [] }); + await bundle.transport.request("turn/start", { input: [{ type: "text", text: "Keep working quietly." }] }); + const runnerPid = bundle.evidence().runnerPid; + const codexPid = bundle.evidence().codexPid; + expect(runnerPid).toBeGreaterThan(0); + expect(codexPid).toBeGreaterThan(0); + const core = cores[0]!; + const original = structuredClone(Object.values(core.store.state.leases)[0]!); + await vi.waitFor(() => { + expect(Date.now()).toBeGreaterThan(original.expiresAtUnixMs + 1_000); + }, { timeout: 75_000, interval: 1_000 }); + const current = Object.values(core.store.state.leases)[0]!; + expect(current.expiresAtUnixMs).toBeGreaterThan(original.expiresAtUnixMs); + expect(current.leaseId).toBe(original.leaseId); + expect(core.store.state.connectionCount).toBe(1); + expect(bundle.evidence().runnerPid).toBe(runnerPid); + expect(bundle.evidence().codexPid).toBe(codexPid); + process.kill(runnerPid!, 0); + process.kill(codexPid!, 0); + const calls = (await readFile(callsPath, "utf8")).trim().split(/\r?\n/); + expect(calls.filter(call => call === "process-start")).toHaveLength(1); + expect(calls.filter(call => call === "thread/start")).toHaveLength(1); + expect(calls.filter(call => call === "turn/start")).toHaveLength(1); + expect(calls).not.toContain("turn/interrupt"); + expect(calls).not.toContain("thread/resume"); + const state = JSON.parse(await readFile(join(stateDirectory, "fake-codex-state.json"), "utf8")); + expect(state.threadId).toBe(opened.thread.id); + expect(state.activeTurnId).toBe("provider-turn-1"); + } finally { + await bundle.transport.close(); + coreSpy.mockRestore(); + await rm(stateDirectory, { recursive: true, force: true }); + } +}, 90_000); + +it("reopens Pi after a completed turn within its cold admission budget during warm attachment", async () => { + const root = await mkdtemp(join(tmpdir(), "runnerd-pi-warm-admission-")); + const providerNode = process.platform === "linux" ? join(root, "node") : process.execPath; + if (process.platform === "linux") { + await cp(process.execPath, providerNode, { dereference: true }); + await chmod(providerNode, 0o500); + expect((await stat(providerNode)).mode & 0o777).toBe(0o500); + } + const cliRoot = join(root, "dist/cli"); + await mkdir(cliRoot, { recursive: true, mode: 0o700 }); + const sidecarPath = join(cliRoot, "acpx-runtime-sidecar.cjs"); + const journal = join(root, "commands.ndjson"); + const fixture = await readFile(fileURLToPath(new URL("./fixtures/fake-pi-warm-sidecar.cjs", import.meta.url)), "utf8"); + await writeFile(sidecarPath, fixture.replace("/* fixture-config */ null", JSON.stringify({ journal, resumeDelayMs: 32_000, commandDigest: QUALIFIED_ACPX_PROFILES.pi.commandDigest })), { mode: 0o600 }); + const digest = (path: string) => `sha256:${createHash("sha256").update(readFileSync(path)).digest("hex")}`; + const bundle = createCapabilityRunnerdCodexTransport({ + provider: "acpx", acpxAgent: "pi", piThinkingLevel: "low", acpxPermissionMode: "deny-all", + runnerBinary: defaultCapabilityRunnerdBinary(), stateDirectory: root, runnerFilesystemRoot: root, + providerNodeCommand: providerNode, providerNodeCommandSha256: digest(providerNode), + acpxSidecarPath: sidecarPath, acpxSidecarSha256: digest(sidecarPath), + providerPackAuthorityDigest: `sha256:${"d".repeat(64)}`, + lifecyclePolicy: { mode: "warm", idleTimeoutMs: 60_000 }, + environment: { PATH: "/usr/bin:/bin" }, + }); + bundle.transport.setServerRequestHandler(async () => ({ + success: true, + contentItems: [{ type: "inputText", text: "Completion report accepted." }], + })); + let failure: unknown; + try { + await bundle.transport.request("initialize", {}); + await bundle.transport.request("thread/start", { + cwd: root, model: "openrouter/deepseek/deepseek-v4-flash-0731", + dynamicTools: codexSemanticToolSpecs(), + completionContract: { revision: "warm-pi-contract", criterionIds: [] }, + }); + const runnerPid = bundle.evidence().runnerPid; + const notifications = bundle.transport.notifications()[Symbol.asyncIterator](); + const completeTurn = async () => { + await bundle.transport.request("turn/start", { input: [{ type: "text", text: "Complete the fixture turn." }] }); + for (let index = 0; index < 64; index += 1) { + const next = await Promise.race([ + notifications.next(), + new Promise((_, reject) => setTimeout(() => reject(new Error("Fixture completion timed out")), 5_000)), + ]); + if (next.value?.method === "turn/completed") return; + } + throw new Error("Fixture turn did not complete"); + }; + await completeTurn(); + const started = Date.now(); + await bundle.transport.attachRun!({ runId: "run-pi-warm-second", turnId: "turn-pi-warm-second", itemId: "item-pi-warm-second" }); + expect(Date.now() - started).toBeGreaterThan(30_000); + await completeTurn(); + expect(bundle.evidence()).toMatchObject({ runnerPid, runnerExited: false }); + const commands = (await readFile(journal, "utf8")).trim().split("\n").map((line) => JSON.parse(line)); + expect(commands.filter((entry) => entry.event === "spawn")).toHaveLength(2); + expect(commands.filter((entry) => entry.command === "session.open")).toHaveLength(2); + expect(commands.filter((entry) => entry.resumed)).toHaveLength(1); + expect(commands.filter((entry) => entry.command === "turn.start")).toHaveLength(2); + expect(commands.filter((entry) => entry.command === "session.close")).toHaveLength(1); + } catch (error) { + failure = error; + throw error; + } finally { + try { + await bundle.transport.close(); + } catch (cleanupError) { + if (failure) throw new AggregateError([failure, cleanupError], "Warm attachment and strict cleanup failed"); + throw cleanupError; + } + await rm(root, { recursive: true, force: true }); + } +}, 75_000); diff --git a/packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts b/packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts index a67cd97afd5..4491e8b1fc3 100644 --- a/packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts +++ b/packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts @@ -1,4 +1,3 @@ -import { QUALIFIED_ACPX_PROFILES } from "../drivers/acpx/qualified-profiles.js"; import { coldAdmissionTimeoutMs, waitForRunnerCommand } from "./runnerd-codex-transport.js"; import { chmod, @@ -3341,58 +3340,6 @@ it("expands coalesced canonical items without dropping strict bindings", () => { ]); }); -it("keeps a quiet active Codex turn in the same process across connection lease expiry", async () => { - const stateDirectory = await mkdtemp(join(tmpdir(), "runnerd-renew-active-")); - const callsPath = join(stateDirectory, "calls.log"); - const cores: DurablePrpControlPlane[] = []; - const OriginalCore = durableControlPlane.DurablePrpControlPlane; - const coreSpy = vi.spyOn(durableControlPlane, "DurablePrpControlPlane") - .mockImplementation(function(options: ConstructorParameters[0]) { - const core = new OriginalCore({ ...options, connectionLeaseTtlMs: 60_000 }); - cores.push(core); - return core; - } as unknown as typeof OriginalCore); - const bundle = createCapabilityRunnerdCodexTransport({ - runnerBinary: defaultCapabilityRunnerdBinary(), codexCommand: fakeCodex, - codexArgs: fakeCodexArgs(stateDirectory, "--hold-turn", "--record-process-start", "--call-log", callsPath), - stateDirectory, runnerReconnectGraceMs: 5_000, - }); - try { - const opened = await bundle.transport.request("thread/start", { cwd: tmpdir(), dynamicTools: [] }); - await bundle.transport.request("turn/start", { input: [{ type: "text", text: "Keep working quietly." }] }); - const runnerPid = bundle.evidence().runnerPid; - const codexPid = bundle.evidence().codexPid; - expect(runnerPid).toBeGreaterThan(0); - expect(codexPid).toBeGreaterThan(0); - const core = cores[0]!; - const original = structuredClone(Object.values(core.store.state.leases)[0]!); - await vi.waitFor(() => { - expect(Date.now()).toBeGreaterThan(original.expiresAtUnixMs + 1_000); - }, { timeout: 75_000, interval: 1_000 }); - const current = Object.values(core.store.state.leases)[0]!; - expect(current.expiresAtUnixMs).toBeGreaterThan(original.expiresAtUnixMs); - expect(current.leaseId).toBe(original.leaseId); - expect(core.store.state.connectionCount).toBe(1); - expect(bundle.evidence().runnerPid).toBe(runnerPid); - expect(bundle.evidence().codexPid).toBe(codexPid); - process.kill(runnerPid!, 0); - process.kill(codexPid!, 0); - const calls = (await readFile(callsPath, "utf8")).trim().split(/\r?\n/); - expect(calls.filter(call => call === "process-start")).toHaveLength(1); - expect(calls.filter(call => call === "thread/start")).toHaveLength(1); - expect(calls.filter(call => call === "turn/start")).toHaveLength(1); - expect(calls).not.toContain("turn/interrupt"); - expect(calls).not.toContain("thread/resume"); - const state = JSON.parse(await readFile(join(stateDirectory, "fake-codex-state.json"), "utf8")); - expect(state.threadId).toBe(opened.thread.id); - expect(state.activeTurnId).toBe("provider-turn-1"); - } finally { - await bundle.transport.close(); - coreSpy.mockRestore(); - await rm(stateDirectory, { recursive: true, force: true }); - } -}, 90_000); - it("runs the lab provider boundary through authenticated durable PRP", async () => { const stateDirectory = await mkdtemp(join(tmpdir(), "runnerd-lab-provider-")); const bundle = createCapabilityRunnerdCodexTransport({ @@ -7005,81 +6952,6 @@ it("rotates PRP authority in place for a warm cross-run attachment", async () => } }, 30_000); -it("reopens Pi after a completed turn within its cold admission budget during warm attachment", async () => { - const root = await mkdtemp(join(tmpdir(), "runnerd-pi-warm-admission-")); - const providerNode = process.platform === "linux" ? join(root, "node") : process.execPath; - if (process.platform === "linux") { - await cp(process.execPath, providerNode, { dereference: true }); - await chmod(providerNode, 0o500); - expect((await stat(providerNode)).mode & 0o777).toBe(0o500); - } - const cliRoot = join(root, "dist/cli"); - await mkdir(cliRoot, { recursive: true, mode: 0o700 }); - const sidecarPath = join(cliRoot, "acpx-runtime-sidecar.cjs"); - const journal = join(root, "commands.ndjson"); - const fixture = await readFile(fileURLToPath(new URL("./fixtures/fake-pi-warm-sidecar.cjs", import.meta.url)), "utf8"); - await writeFile(sidecarPath, fixture.replace("/* fixture-config */ null", JSON.stringify({ journal, resumeDelayMs: 32_000, commandDigest: QUALIFIED_ACPX_PROFILES.pi.commandDigest })), { mode: 0o600 }); - const digest = (path: string) => `sha256:${createHash("sha256").update(readFileSync(path)).digest("hex")}`; - const bundle = createCapabilityRunnerdCodexTransport({ - provider: "acpx", acpxAgent: "pi", piThinkingLevel: "low", acpxPermissionMode: "deny-all", - runnerBinary: defaultCapabilityRunnerdBinary(), stateDirectory: root, runnerFilesystemRoot: root, - providerNodeCommand: providerNode, providerNodeCommandSha256: digest(providerNode), - acpxSidecarPath: sidecarPath, acpxSidecarSha256: digest(sidecarPath), - providerPackAuthorityDigest: `sha256:${"d".repeat(64)}`, - lifecyclePolicy: { mode: "warm", idleTimeoutMs: 60_000 }, - environment: { PATH: "/usr/bin:/bin" }, - }); - bundle.transport.setServerRequestHandler(async () => ({ - success: true, - contentItems: [{ type: "inputText", text: "Completion report accepted." }], - })); - let failure: unknown; - try { - await bundle.transport.request("initialize", {}); - await bundle.transport.request("thread/start", { - cwd: root, model: "openrouter/deepseek/deepseek-v4-flash-0731", - dynamicTools: codexSemanticToolSpecs(), - completionContract: { revision: "warm-pi-contract", criterionIds: [] }, - }); - const runnerPid = bundle.evidence().runnerPid; - const notifications = bundle.transport.notifications()[Symbol.asyncIterator](); - const completeTurn = async () => { - await bundle.transport.request("turn/start", { input: [{ type: "text", text: "Complete the fixture turn." }] }); - for (let index = 0; index < 64; index += 1) { - const next = await Promise.race([ - notifications.next(), - new Promise((_, reject) => setTimeout(() => reject(new Error("Fixture completion timed out")), 5_000)), - ]); - if (next.value?.method === "turn/completed") return; - } - throw new Error("Fixture turn did not complete"); - }; - await completeTurn(); - const started = Date.now(); - await bundle.transport.attachRun!({ runId: "run-pi-warm-second", turnId: "turn-pi-warm-second", itemId: "item-pi-warm-second" }); - expect(Date.now() - started).toBeGreaterThan(30_000); - await completeTurn(); - expect(bundle.evidence()).toMatchObject({ runnerPid, runnerExited: false }); - const commands = (await readFile(journal, "utf8")).trim().split("\n").map((line) => JSON.parse(line)); - expect(commands.filter((entry) => entry.event === "spawn")).toHaveLength(2); - expect(commands.filter((entry) => entry.command === "session.open")).toHaveLength(2); - expect(commands.filter((entry) => entry.resumed)).toHaveLength(1); - expect(commands.filter((entry) => entry.command === "turn.start")).toHaveLength(2); - expect(commands.filter((entry) => entry.command === "session.close")).toHaveLength(1); - } catch (error) { - failure = error; - throw error; - } finally { - try { - await bundle.transport.close(); - } catch (cleanupError) { - if (failure) throw new AggregateError([failure, cleanupError], "Warm attachment and strict cleanup failed"); - throw cleanupError; - } - await rm(root, { recursive: true, force: true }); - } -}, 75_000); - it("waits for a warm runner to re-authenticate before probing attachment readiness", async () => { const stateDirectory = await mkdtemp( join(tmpdir(), "runnerd-warm-reattach-before-probe-"), diff --git a/scripts/__tests__/run-vitest-stable-shard.test.mjs b/scripts/__tests__/run-vitest-stable-shard.test.mjs index 9b92a9e174e..8c9ddedf969 100644 --- a/scripts/__tests__/run-vitest-stable-shard.test.mjs +++ b/scripts/__tests__/run-vitest-stable-shard.test.mjs @@ -1,6 +1,7 @@ import assert from "node:assert/strict"; import { spawnSync } from "node:child_process"; -import { readFileSync } from "node:fs"; +import { chmodSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import os from "node:os"; import path from "node:path"; import { fileURLToPath } from "node:url"; import test from "node:test"; @@ -519,3 +520,69 @@ test("filtered collection must match the exact assigned case identities, includi assert.throws(() => assertSelectedTests(expected, actual, lineShardFile)); } }); + + +test("parallel chat jobs form a complete disjoint partition", () => { + for (const count of [2, 3, 4]) { + const shards = Array.from({ length: count }, (_, index) => + dryRunJson(["--mode", "general", "--group", "general-chat", "--shard-index", String(index), "--shard-count", String(count)], + { PAPERCLIP_TEST_DATABASE_URL: "" }).chatVitestShards, + ); + assert.ok(shards.every((workers) => workers.length === 2)); + assert.deepEqual(shards.flat().map(({ index }) => index).sort((a, b) => a - b), + Array.from({ length: count * 2 }, (_, index) => index)); + assert.ok(shards.flat().every((shard) => shard.count === count * 2)); + } +}); + +test("shared-database and unsharded chat runs retain one serial partition", () => { + for (let index = 0; index < 3; index++) { + const result = dryRunJson(["--mode", "general", "--group", "general-chat", "--shard-index", String(index), "--shard-count", "3"], + { PAPERCLIP_TEST_DATABASE_URL: "postgresql://shared/test" }); + assert.deepEqual(result.chatVitestShards, [{ index, count: 3 }]); + } + assert.deepEqual(dryRunJson(["--mode", "general", "--group", "general-chat"], + { PAPERCLIP_TEST_DATABASE_URL: "" }).chatVitestShards, [{ index: 0, count: 1 }]); +}); + + +test("a failed chat child fails the job after its sibling finishes", { skip: process.platform === "win32" }, () => { + const directory = mkdtempSync(path.join(os.tmpdir(), "chat-launch-proof-")); + const file = path.join(repoRoot, "server/src/__tests__/chat-channels.integration.test.ts"); + const lines = readFileSync(file, "utf8").split("\n") + .flatMap((line, index) => /^\s*it\(/.test(line) ? [index + 1] : []).slice(0, 6); + assert.equal(lines.length, 6); + const tests = lines.map((line) => ({ file, projectName: "@paperclipai/server", + name: `launcher fixture ${line}`, location: { line } })); + const receipt = path.join(directory, "sibling-finished"); + const fakePnpm = path.join(directory, "pnpm"); + try { + writeFileSync(fakePnpm, `#!${process.execPath} + const fs = require('node:fs'); + const args = process.argv.slice(2); + const tests = ${JSON.stringify(tests)}; + const selected = args.flatMap(arg => { + const match = arg.match(/chat-channels\\.integration\\.test\\.ts:(\\d+)$/); + return match ? [Number(match[1])] : []; + }); + if (args.includes('list')) { + const output = args.find(arg => arg.startsWith('--json=')).slice(7); + fs.writeFileSync(output, JSON.stringify(selected.length + ? tests.filter(test => selected.includes(test.location.line)) : tests)); + } else if (selected.includes(${lines[0]})) process.exit(7); + else setTimeout(() => fs.writeFileSync(${JSON.stringify(receipt)}, 'finished'), 200); + `); + chmodSync(fakePnpm, 0o755); + const result = spawnSync(process.execPath, [script, "--mode", "general", "--group", "general-chat", + "--shard-index", "0", "--shard-count", "3"], { + cwd: repoRoot, encoding: "utf8", timeout: 30_000, + env: { ...process.env, PATH: `${directory}${path.delimiter}${process.env.PATH}`, + PAPERCLIP_TEST_DATABASE_URL: "", PAPERCLIP_CI_PROFILE: "false" }, + }); + assert.equal(result.status, 1, result.stdout + result.stderr); + assert.match(result.stderr, /failed \(7\)/); + assert.equal(readFileSync(receipt, "utf8"), "finished", "a failed worker must not abandon its sibling"); + } finally { + rmSync(directory, { recursive: true, force: true }); + } +}); diff --git a/scripts/__tests__/vitest-worker-isolation.test.mjs b/scripts/__tests__/vitest-worker-isolation.test.mjs index bd70f733df4..2c762728757 100644 --- a/scripts/__tests__/vitest-worker-isolation.test.mjs +++ b/scripts/__tests__/vitest-worker-isolation.test.mjs @@ -11,7 +11,7 @@ const vitest = path.join(root, 'node_modules/vitest/vitest.mjs'); const api = fileURLToPath(new URL('../../node_modules/vitest/dist/index.js', import.meta.url)); const setup = path.join(root, 'server/src/__tests__/setup-supertest.ts'); -test('one controller retains fresh fork, globals, environment, timers and home per serialized file', () => { +for (const parallel of [false, true]) test(`isolated forks retain fresh state with file parallelism ${parallel}`, () => { const directory = realpathSync(mkdtempSync(path.join(os.tmpdir(), 'pc-fork-proof-'))); try { const record = path.join(directory, 'workers.ndjson'); @@ -20,7 +20,7 @@ test('one controller retains fresh fork, globals, environment, timers and home p for (const file of files) writeFileSync(path.join(directory, file), ` import { expect, it, vi } from ${JSON.stringify(api)}; import { appendFileSync, existsSync, writeFileSync } from 'node:fs'; - it('starts clean and can contaminate only its own worker', () => { + it('starts clean and can contaminate only its own worker', async () => { expect(globalThis.serializedLeak).toBeUndefined(); expect(process.env.PAPERCLIP_SERIALIZED_LEAK).toBeUndefined(); expect(vi.isFakeTimers()).toBe(false); @@ -30,6 +30,16 @@ test('one controller retains fresh fork, globals, environment, timers and home p config: process.env.PAPERCLIP_CONFIG, instance: process.env.PAPERCLIP_INSTANCE_ID, }) + '\\n'); writeFileSync(process.env.PAPERCLIP_CONFIG, '{}'); + // In parallel mode both real workers must arrive before either exits. + // This fails if the controller silently falls back to serial execution. + if (${parallel}) { + const { readFileSync } = await import('node:fs'); + const deadline = Date.now() + 10_000; + while (readFileSync(${JSON.stringify(record)}, 'utf8').trim().split('\\n').length < 2) { + if (Date.now() > deadline) throw new Error('second isolated worker did not start'); + await new Promise(resolve => setTimeout(resolve, 10)); + } + } globalThis.serializedLeak = true; process.env.PAPERCLIP_SERIALIZED_LEAK = 'leaked'; vi.useFakeTimers(); @@ -40,8 +50,8 @@ test('one controller retains fresh fork, globals, environment, timers and home p writeFileSync(config, `import Sequencer from './sequencer.mjs'; export default ${JSON.stringify({ root: directory, test: { - include: order, pool: 'forks', isolate: true, fileParallelism: false, - maxWorkers: 1, setupFiles: [setup], + include: order, pool: 'forks', isolate: true, fileParallelism: parallel, + maxWorkers: parallel ? 2 : 1, setupFiles: [setup], sequence: { sequencer: '__SEQUENCER__' }, }, }).replace('"__SEQUENCER__"', 'Sequencer')};`); @@ -57,7 +67,8 @@ test('one controller retains fresh fork, globals, environment, timers and home p assert.equal(result.status, 0, `${result.error ?? ''}\n${result.stdout}\n${result.stderr}`); const workers = readFileSync(record, 'utf8').trim().split('\n').map((line) => JSON.parse(line)); assert.equal(workers.length, 2); - assert.deepEqual(workers.map((worker) => worker.name), order); + assert.deepEqual(workers.map((worker) => worker.name).sort(), [...order].sort()); + if (!parallel) assert.deepEqual(workers.map((worker) => worker.name), order); for (const key of ['pid', 'home', 'config', 'instance']) assert.equal(new Set(workers.map((worker) => worker[key])).size, 2, key); } } finally { diff --git a/scripts/benchmark-ci-work.mjs b/scripts/benchmark-ci-work.mjs index 7fa6dec7661..6bc2c8b99ec 100644 --- a/scripts/benchmark-ci-work.mjs +++ b/scripts/benchmark-ci-work.mjs @@ -11,7 +11,7 @@ const output = path.resolve(process.argv[3] ?? path.join(candidate, '.ci-benchma mkdirSync(output, { recursive: true }); const results = []; const commands = [ - ['serialized', process.execPath, ['scripts/run-vitest-stable.mjs', '--mode', 'serialized', '--shard-index', '0', '--shard-count', '9']], + ['general-server', process.execPath, ['scripts/run-vitest-stable.mjs', '--mode', 'general', '--group', 'general-server-without-chat-or-native-runner', '--shard-index', '0', '--shard-count', '12']], ['fixtures', 'pnpm', ['exec', 'vitest', 'run', '--project', '@paperclipai/server', 'server/src/services/workspace-git-snapshot-streaming.test.ts', 'server/src/__tests__/workspace-runtime.test.ts', '-t', 'streams.*all four real Git|commits fixed-port reservations']], @@ -47,7 +47,7 @@ for (const [name] of commands) { const after = results.find((result) => result.variant === 'candidate' && result.name === name); assert.ok(before.tests.length > 0, `${name}: missing test evidence`); assert.deepEqual(after.files, before.files, `${name}: file inventory changed`); - if (name === 'serialized') assert.deepEqual(after.tests, before.tests, 'serialized case inventory changed'); + if (name === 'general-server') assert.deepEqual(after.tests, before.tests, 'general server case inventory changed'); else { const contracts = (tests) => { const found = new Set(); diff --git a/scripts/run-vitest-stable.mjs b/scripts/run-vitest-stable.mjs index 847716b7067..7f283c9acab 100644 --- a/scripts/run-vitest-stable.mjs +++ b/scripts/run-vitest-stable.mjs @@ -1,5 +1,5 @@ #!/usr/bin/env node -import { spawnSync } from "node:child_process"; +import { spawn, spawnSync } from "node:child_process"; import { mkdirSync, mkdtempSync, readdirSync, readFileSync, realpathSync, statSync, writeFileSync } from "node:fs"; import os from "node:os"; import path from "node:path"; @@ -377,7 +377,7 @@ function selectSerializedSuites(routeTests, shardIndex, shardCount) { return shardFiles.map((file) => byRepoPath.get(file)); } -function runVitest(args, label, testShard = null) { +async function runVitest(args, label, testShard = null) { console.log(`\n[test:run] ${label}`); invocationIndex += 1; const tempRootParent = process.platform === "win32" ? os.tmpdir() : "/tmp"; @@ -408,7 +408,7 @@ function runVitest(args, label, testShard = null) { ...filters, "--allowOnly=false", "--includeTaskLocation", `--json=${output}`], { cwd: repoRoot, env, stdio: "inherit", }); - if (result.error || result.status !== 0) fail(`Vitest collection failed: ${result.error?.message ?? result.status}`); + if (result.error || result.status !== 0) throw new Error(`Vitest collection failed: ${result.error?.message ?? result.status}`); return JSON.parse(readFileSync(output, "utf8")); }; const allCollected = collect(args, "all"); @@ -434,27 +434,27 @@ function runVitest(args, label, testShard = null) { const runFilters = selected.lines.map((line) => `${chatSuite}:${chatSuiteRunLine(sourceLines, line)}`); args = [...args.filter((arg) => arg !== chatSuite), ...runFilters, "--allowOnly=false"]; } - const result = spawnSync("pnpm", ["exec", "vitest", "run", ...sourceOnlyVitestArgs, ...args], { - cwd: repoRoot, - env: invocationEnv("vitest", env), - stdio: "inherit", + await new Promise((resolve, reject) => { + const child = spawn("pnpm", ["exec", "vitest", "run", ...sourceOnlyVitestArgs, ...args], { + cwd: repoRoot, + env: invocationEnv("vitest", env), + stdio: "inherit", + }); + child.once("error", reject); + child.once("close", (code, signal) => { + if (code === 0) resolve(); + else reject(new Error(`${label} failed (${signal ?? code})`)); + }); }); - if (result.error) { - console.error(`[test:run] Failed to start Vitest: ${result.error.message}`); - process.exit(1); - } - if (result.status !== 0) { - process.exit(result.status ?? 1); - } } -function runGeneralSuites(routeTests) { +async function runGeneralSuites(routeTests) { for (const groupName of generalGroupNames) { - runGeneralGroup(routeTests, groupName); + await runGeneralGroup(routeTests, groupName); } } -function runProjectGroup(projects, groupName, shardIndex = null, shardCount = null) { +async function runProjectGroup(projects, groupName, shardIndex = null, shardCount = null) { // With shard args, lean on Vitest's native --shard: each matrix job runs the // same per-project invocations but only its slice of each project's test // files. Vitest's sharding is deterministic for an identical file list, so @@ -463,18 +463,32 @@ function runProjectGroup(projects, groupName, shardIndex = null, shardCount = nu shardCount !== null && shardCount > 1 ? [`--shard=${shardIndex + 1}/${shardCount}`] : []; const shardSuffix = shardArgs.length > 0 ? ` shard ${shardIndex + 1}/${shardCount}` : ""; for (const project of projects) { - runVitest(["--project", project, ...shardArgs], `${groupName} project ${project}${shardSuffix}`); + await runVitest(["--project", project, ...shardArgs], `${groupName} project ${project}${shardSuffix}`); } } -function runGeneralGroup(routeTests, groupName, shardIndex = null, shardCount = null) { +function chatShardWorkers(index, count) { + const workers = count > 1 && !process.env.PAPERCLIP_TEST_DATABASE_URL ? 2 : 1; + return Array.from({ length: workers }, (_, offset) => ({ + index: (index ?? 0) * workers + offset, + count: (count ?? 1) * workers, + })); +} + +async function runGeneralGroup(routeTests, groupName, shardIndex = null, shardCount = null) { if (groupName === generalChatGroupName) { - runVitest(["--project", "@paperclipai/server", ...serializedServerVitestArgs, chatSuite], - "chat integration test shard", { index: shardIndex ?? 0, count: shardCount ?? 1 }); + // Each invocation gets its own temporary home and embedded PostgreSQL. + // Wait for both children even on failure so no sibling is abandoned. + const results = await Promise.allSettled(chatShardWorkers(shardIndex, shardCount).map((shard) => + runVitest(["--project", "@paperclipai/server", ...serializedServerVitestArgs, chatSuite], + "chat integration test shard", shard), + )); + const failed = results.find((result) => result.status === "rejected"); + if (failed) throw failed.reason; return; } if (groupName === generalServerNativeRunnerGroupName) { - runVitest( + await runVitest( ["--project", "@paperclipai/server", ...serializedServerVitestArgs, ...nativeRunnerSuites], "native runner vertical-slice suite", ); @@ -497,11 +511,16 @@ function runGeneralGroup(routeTests, groupName, shardIndex = null, shardCount = return; } - runVitest( + await runVitest( [ "--project", "@paperclipai/server", - ...serializedServerVitestArgs, + // Files have separate fork processes and isolated homes/configs. + // Keep each file's cases serial; overlap only two independent files. + // An explicit shared test database cannot be isolated this way. + ...(process.env.PAPERCLIP_TEST_DATABASE_URL + ? serializedServerVitestArgs + : ["--pool=forks", "--isolate", "--fileParallelism", "--maxWorkers=2"]), ...shardFiles, ], `${groupName} shard ${shardIndex + 1}/${shardCount}`, @@ -513,7 +532,7 @@ function runGeneralGroup(routeTests, groupName, shardIndex = null, shardCount = for (const suite of excludedSuites) { excludeRouteArgs.push("--exclude", suite.replace(/^server\//, "")); } - runVitest( + await runVitest( [ "--project", "@paperclipai/server", @@ -530,31 +549,31 @@ function runGeneralGroup(routeTests, groupName, shardIndex = null, shardCount = // 31371439296, 2026-08-10, where workspaces-a was the slowest PR check). // Its 439 test files shard cleanly with Vitest's native --shard, so the // lane splits across runners without a duration manifest. - runProjectGroup(generalWorkspacesAProjects, groupName, shardIndex, shardCount); + await runProjectGroup(generalWorkspacesAProjects, groupName, shardIndex, shardCount); return; } if (groupName === generalWorkspacesBGroupName) { - runProjectGroup(generalWorkspacesBProjects, groupName); + await runProjectGroup(generalWorkspacesBProjects, groupName); return; } if (groupName === generalWorkspacesCGroupName) { - runProjectGroup(generalWorkspacesCProjects, groupName); + await runProjectGroup(generalWorkspacesCProjects, groupName); return; } fail(`Unknown group "${groupName}".`); } -function runSerializedSuites(routeTests, shardIndex, shardCount) { +async function runSerializedSuites(routeTests, shardIndex, shardCount) { const shardTests = selectSerializedSuites(routeTests, shardIndex, shardCount); console.log( `\n[test:run] serialized shard ${shardIndex + 1}/${shardCount} running ${shardTests.length} of ${routeTests.length} suites`, ); if (shardTests.length > 0) { - runVitest( + await runVitest( [ "--project", "@paperclipai/server", @@ -578,9 +597,9 @@ const routeTests = walk(serverTestsDir) // Every server test file that the general-server group is responsible for, // i.e. the whole server project minus the route/authz suites that run in the -// dedicated serialized shards. Sharding this list across runners is what keeps -// the general-server lane from becoming the PR critical path: the server vitest -// config pins maxWorkers to 1, so the only way to parallelize is across jobs. +// dedicated serialized shards. The default server config keeps files serial; +// the sharded entry point overlaps two isolated files per runner unless an +// explicit shared test database requires serial execution. // Suites are partitioned by recorded duration (scripts/general-server-shard.mjs) // rather than round-robin, so one slow suite cluster can't stretch a single shard. const serializedRepoPaths = new Set(routeTests.map(test => test.repoPath)); @@ -625,6 +644,8 @@ if (options.dryRun) { generalServerShardDurations, ) : null, + chatVitestShards: options.group === generalChatGroupName + ? chatShardWorkers(options.shardIndex, options.shardCount) : null, workspaceProjects: options.group === generalWorkspacesAGroupName ? generalWorkspacesAProjects @@ -647,14 +668,19 @@ if (options.dryRun) { process.exit(0); } -if (options.mode === generalModeName || options.mode === allModeName) { - if (options.group) { - runGeneralGroup(routeTests, options.group, options.shardIndex, options.shardCount); - } else { - runGeneralSuites(routeTests); +try { + if (options.mode === generalModeName || options.mode === allModeName) { + if (options.group) { + await runGeneralGroup(routeTests, options.group, options.shardIndex, options.shardCount); + } else { + await runGeneralSuites(routeTests); + } } -} -if (options.mode === serializedModeName || options.mode === allModeName) { - runSerializedSuites(routeTests, options.shardIndex ?? 0, options.shardCount ?? 1); + if (options.mode === serializedModeName || options.mode === allModeName) { + await runSerializedSuites(routeTests, options.shardIndex ?? 0, options.shardCount ?? 1); + } +} catch (error) { + console.error(`[test:run] ${error.message}`); + process.exitCode = 1; } From 59a70dc9e235d6df3d7db69b3ca170196d0c749f Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Thu, 15 Oct 2026 13:09:59 -0500 Subject: [PATCH 16/19] fix(workspaces): expire reproducible Git files and restore exact commits (#16457) ## Thinking Path > - Paperclip manages AI agents and their task workspaces. > - Isolated Git worktrees and private repository copies remain on the host after tasks finish. > - These files can fill the disk even when every commit is available upstream. > - Cleanup must retain unique files, local Git state, live work, and recovery data. > - This pull request records and verifies an exact-commit restoration recipe before expiration. > - Normal reopening restores that commit even after upstream advances. > - The benefit is lower host disk use while finished tasks remain available for later work. ## Linked Issues or Issue Description Refs #10285 and #2022. Follows #16433 for skill-cache expiration. Related #11936 provides separate host-local cleanup tooling; this change extends the existing company-scoped lifecycle worker and normal workspace reopening. ROADMAP.md does not specify a competing implementation. ## What Changed - Add opt-in expiration for verified Paperclip-owned Git worktrees after their whole task tree has been done or cancelled for 24 hours. - Preserve active and scheduled runs, previews, pending reopening, unresolved recovery, dirty files, unknown ignored files, and unsupported or unique Git state. - Verify an authenticated fetch in a fresh object store. Save repository identity, exact commit, branch, and ownership before any deletion. - Rename eligible files into recorded managed trash. Resume interrupted cleanup under source and lifecycle locks. - Expire private base clones only after consumer, file, reference, reflog, and object checks pass. Preserve shared project base clones. - Rebuild missing bases and worktrees through normal reopening at the recorded commit. Return an explicit retryable error when repository access fails. - Add independent workspace/base switches, dry-run reports, reclaimed-byte counts, and operational documentation. No table, public API, UI, or recovery-retention change. ## Verification - Real local bare remote plus Postgres: create at A, advance upstream to B, expire the worktree and private base, reopen at A, then make and retain a subsequent commit. - Final lifecycle suite: all 20 tests passed in 46.41 seconds at `d72947a2e45be5d0c912debb03c46b15863b4798`. Review fixes cover second-cycle expiration, newer-branch recovery, normal preparation after restoration, branch adoption, and Git administration path controls. Eleven existing branch reconciliation controls also passed. Covers unique files and Git state, live descendants, scheduled runs, recovery, previews, unauthorized access, upstream failure, concurrent reopening, interrupted deletion, shared bases, and nested private worktrees. - Final guard regressions: 3 affected tests passed. Existing workspace/runtime/preparation/source controls: 328 tests passed in 238.69 seconds. - Full repository typecheck and build passed locally. The full local test attempt did not pass: initial native-session cases hit macOS Postgres shared-memory exhaustion, source changes invalidated two discovery cases, and the checkout disappeared near the end. The final-code retry of the remaining skill and unrelated controls passed all 103 tests. All checks passed on this exact head, including typecheck, general and serialized tests, Runner checks, build, canary dry-run, browser E2E, and security scans. Greptile reviewed the head at 5/5 with all findings resolved. - Fixed a blocking CI policy false positive for `docker-compose-secrets.test.mjs`. The test still rejects GitHub secret expressions, including dot, bracket, and multiline forms. Its full 15-test suite passes. - Map validation and `git diff --check` passed. - Staging dry-run inspection, measured reclamation, and ordinary product acceptance remain pending. No live acceptance is claimed. ## Risks - Git restoration depends on continued upstream availability and credentials. Cleanup skips unverified commits; restoration never substitutes the default branch. - File deletion requires multiple conservative proofs. Unknown or unsupported state is retained and can limit reclamation. - Source and lifecycle locks serialize preparation, reopening, and expiration. Remote verification has a deadline but can delay work on the same source. - Start with both expiration switches disabled. When enabled, dry-run remains the default. Rollback disables further deletion while retaining restoration support. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository inspection, tool use, and code execution. The exact backend model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (focused tests; full local limitation documented above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- .github/scripts/tests/ci-profile.test.mjs | 9 +- doc/DEVELOPING.md | 31 ++ .../2026-10-15-recoverable-disk-cleanup.md | 161 ++++---- maps/agent-access-map/workspaces.md | 8 + maps/feature-map/workspaces.md | 9 + .../git-workspace-expiration.test.ts | 297 +++++++++++++++ server/src/config.ts | 6 + server/src/index.ts | 3 + server/src/services/execution-workspaces.ts | 86 ++++- .../src/services/git-workspace-expiration.ts | 230 ++++++++++++ .../src/services/git-workspace-restoration.ts | 346 ++++++++++++++++++ .../heartbeat/workspace-preparation.ts | 10 +- .../isolated-remote-workspace-source.ts | 22 ++ server/src/services/workspace-runtime.ts | 18 +- 14 files changed, 1143 insertions(+), 93 deletions(-) create mode 100644 server/src/__tests__/git-workspace-expiration.test.ts create mode 100644 server/src/services/git-workspace-expiration.ts create mode 100644 server/src/services/git-workspace-restoration.ts diff --git a/.github/scripts/tests/ci-profile.test.mjs b/.github/scripts/tests/ci-profile.test.mjs index b6a931b6e3c..f113d0ad87d 100644 --- a/.github/scripts/tests/ci-profile.test.mjs +++ b/.github/scripts/tests/ci-profile.test.mjs @@ -239,7 +239,14 @@ test("workflow exporter runs trusted code and keeps keys out of the PR jobs", () assert.match(workflow, /environment: ci-profiling/); assert.doesNotMatch(workflow, /head_sha|secrets: inherit|contents: write|pull-requests: write|pnpm install/); const pr = readFileSync(new URL("../../workflows/pr-trusted.yml", import.meta.url), "utf8"); - assert.doesNotMatch(pr, /HONEYCOMB.*KEY|secrets\./); + // A script filename such as docker-compose-secrets.test.mjs is not a + // GitHub secret lookup. Check expression context, including bracket access. + const secretReference = /HONEYCOMB.*KEY|\$\{\{[^}]*\bsecrets\s*(?:\.|\[)/; + assert.doesNotMatch("run: node --test scripts/docker-compose-secrets.test.mjs", secretReference); + for (const lookup of ["${{ secrets.PR_TOKEN }}", "${{ secrets['PR_TOKEN'] }}", "${{\n secrets.PR_TOKEN\n }}"]) { + assert.match(lookup, secretReference); + } + assert.doesNotMatch(pr, secretReference); assert.match(pr, /if: always\(\) && env.PAPERCLIP_CI_PROFILE == 'true'/); assert.doesNotMatch(pr, /PAPERCLIP_CI_PROFILE:.*labels/); assert.equal(pr.match(/PAPERCLIP_CI_PROFILE: \$\{\{ needs.gate.outputs.profile_ci \}\}/g).length, 9); diff --git a/doc/DEVELOPING.md b/doc/DEVELOPING.md index 6fb6f0f6643..833db6306ac 100644 --- a/doc/DEVELOPING.md +++ b/doc/DEVELOPING.md @@ -2471,6 +2471,37 @@ fields before disabling a connection. Verify workspace ownership, active runs, Git state, and runtime-service readiness before closing a workspace. A missing URL or old workspace timestamp alone does not prove that a row is disposable. +### Reconstructable Git cache expiration + +The terminal-workspace worker can expire Paperclip-owned isolated Git worktrees +24 hours after every task in their tree is done or cancelled, independently of +PR merge or delivery status. Enable it with +`PAPERCLIP_GIT_CACHE_EXPIRATION_ENABLED=true`. It defaults to dry run; set +`PAPERCLIP_GIT_CACHE_EXPIRATION_DRY_RUN=false` after inspecting the structured +`execution_workspace.git_cache_expiration` logs. Logs report candidate and +reclaimed bytes, base removal, and skip reasons. The existing bounded lifecycle +sweep supplies scheduling and shutdown cancellation. + +Before deletion the worker fetches the exact commit into a fresh object store +using the company's normal Git identity and commits a versioned restoration +recipe to workspace metadata. Current runs, previews, pending reopening, +unresolved recovery, dirty/untracked files, unknown ignored files, unproven +ownership, and unsupported or unique Git state prevent deletion. Only the +existing repository-ignored regenerable-output allowlist can be discarded. +Deletion first renames the worktree to recorded trash; restart and reopening +finish that cleanup under the same source/lifecycle locks. + +`PAPERCLIP_GIT_BASE_EXPIRATION_ENABLED=true` additionally permits removal of an +owned isolated remote-source base after all consumers expire and all remaining +refs and files are verified reproducible. Shared project clones remain intact. +Normal authorized reopening recreates a missing source and checks out the saved +commit, even after upstream advances. Inaccessible upstream produces an explicit +retryable restoration failure. It never substitutes the default branch. + +Both enable switches default off. Rollback disables further expiration while +leaving saved recipes and restoration support intact. Recovery archives, runner +backups, non-Git workspaces, and immutable skill snapshots are unaffected. + ### Terminal workspace compaction A finished task often keeps its worktree for a long time: the reaper archives a diff --git a/doc/plans/2026-10-15-recoverable-disk-cleanup.md b/doc/plans/2026-10-15-recoverable-disk-cleanup.md index d8a11ad79fc..58b4c376937 100644 --- a/doc/plans/2026-10-15-recoverable-disk-cleanup.md +++ b/doc/plans/2026-10-15-recoverable-disk-cleanup.md @@ -35,82 +35,93 @@ product acceptance. Rollback disables deletion and retains restoration support. ## Current state -- Checkout: `/Users/dotta/.codex/worktrees/2da2/paperclip` -- Skill PR branch: `fix/skill-version-dir-retention`, based on Devin Foley's - [PR #16433](https://github.com/paperclipai/paperclip/pull/16433). User approved - building on that PR. Added safety commits to its existing branch, preserving - Devin’s original commit; latest published revision before this fix is `6385a28ce7e0ee24bf0bc5e1dc088e7f4abb9508`. - User also authorized merging every PR from this effort after checks pass. -- Earlier proof/implementation preserved at `da754ff744` on - `codex/recoverable-skill-cache`; do not ship its competing worker. -- Staging: `paperclip.staging.paperclip.app`, stack `stack-pool-50531a9d17dd`. -- Staging read-only recheck at 16:07 UTC: 71% disk used, 103,676,784,640 bytes - used, 43,621,777,408 available. Serving `7ec6fd09725fdcd771425e31ce900f8db9aae07d`. - This task has not deployed or deleted staging data. -- A temporary Railway key named `paperclip-cleanup-acceptance-2026-10-15` is - registered under the prior explicit approval. Revoke it and remove both local - copies at acceptance completion (paths retained in private task evidence). +- User authorized merging every PR from this effort after checks pass. +- Skill [PR #16433](https://github.com/paperclipai/paperclip/pull/16433) merged + after exact-head CI, security review, and Greptile 5/5 passed. Reviewed head: + `afc159cf39a1228608c1f864648ab359a57bf357`; merge: + `86c965ce7ab1499864f74b8dd6ed0d2d8db15197`. Devin's original implementation + and authorship are retained. The earlier prototype on `codex/recoverable-skill-cache` + must not ship as a competing worker. +- Git [PR #16457](https://github.com/paperclipai/paperclip/pull/16457) is open on + `codex/recoverable-git-cache`, rebased onto that merge. Initial review head: + `bb3984d3fd13e7a5604e9d90c7d58748d4b50708`. CI and review are pending. +- Git evidence: 17 real database/filesystem cases passed in 48.04s, including + expiration/reopening at A after upstream B, subsequent commits, interrupted + trash deletion, concurrent reopening, unique files/Git state, live consumers, + upstream failure, company boundaries, shared bases, and nested private sources. + Three final guard cases passed; 328 existing controls passed in 238.69s. + Evidence: `/tmp/paperclip-git-expiration-tests4.log`, + `/tmp/paperclip-git-final-guards.log`, `/tmp/paperclip-git-existing-controls.log`. +- Skill evidence: 21 final focused cases passed in 29.99s and 35 affected + skill/heartbeat controls passed. Streaming discovery holds directory cursors + across bounded pages; a regression counts actual reads. Preparation/pinning + and eviction share company locks; filesystem inspection/deletion do not hold + the preparation lock. Immutable snapshots remain in the database. +- Full local typecheck/build passed for both slices. The full test attempt ran + for 3,547.58s and did not pass: startup IPC capacity failures, mixed-revision + discovery tests, two unrelated controls, and module failures after the checkout + disappeared. It is not evidence of one clean SHA. Exact-head CI supersedes + the mixed-revision run for the merged skill PR. Focused final-code retries + classified the remaining failures: all 103 tests in the affected final-code + retry passed in 42.25s. Do not repeat an hour-long loop without cause. +- The checkout unexpectedly disappeared at about 17:22 UTC after all code was + pushed. Restored the same branch/path from Git, reinstalled frozen dependencies + in 24.8s, and rebuilt successfully. Managed attachment recovery was unavailable + because the app could not verify ownership. All retained evidence is in `/tmp`. +- Staging target: `paperclip.staging.paperclip.app`, `stack-pool-50531a9d17dd`. + One deployment request owns this task's rollout: + `38a0a286-7a97-4cd5-b58e-eabdaa4d38bd`, source merge `86c965ce7a…`. + First prepare failed with `direct_migrator_artifact_invalid` while the exact + source artifact was still publishing. After its publisher passed, reran that + failed prepare and resumed the same request. No campaign duplication. +- Latest read-only staging baseline at 17:26 UTC: serving + `03f22f4cb9be9a34906bda0e4906262dc359e9ce`, disk 73%, 106,903,482,368 bytes + used and 40,395,079,680 available. This task has not deleted staging data. + Skill cleanup is configured enabled with dry-run true; Git cleanup is not enabled. - Existing unresolved staging runs have missing skill-version pins. The approved - conservative policy skips that company; do not clear recovery or weaken the - guard to obtain reclamation. Qualify the skill slice with a disposable pinned - fixture and report the legacy-data gap separately. -- Integrated one worker, cross-process preparation/pin locks, separate last-use - metadata, legacy observation grace, unresolved-recovery protection, dry run, - bounded scans/continuations, and deletion outside the preparation lock. -- First integrated run: 22 focused tests passed; broader controls passed 100/101. - Removed an unnecessary explicit-deletion lock exposed by the rollback fixture; - that control passed on rerun (12.21s). The original failure remains in - `/tmp/paperclip-skill-integration-controls.log`. -- Full typecheck initially found a plugin dependency API mismatch from the - unlocked bootstrap install. Current master supplied the refreshed lockfile; - frozen install completed in 18.7s and full typecheck passed in 59.82s. -- Review at `5eb505ff5d` was 3/5: unsafe skill roots blocked the company sweep, - and recursive byte counts held the preparation lock. Both fixed locally; - 18 affected tests passed in 24.94s, including preparation during a paused file - walk and fresh pin revalidation. Server typecheck passed after those fixes. -- Full tests were stopped after approximately four minutes to address review - findings before final qualification; no full-suite pass claimed. Full build at - `d153929d70` passed. CI at that SHA passed all but the empty-runtime suite and - Greptile (4/5). The remaining review finding was repeated company-wide pin - queries; batching reads twice per sweep phase fixes it and has a query-count - regression. The empty-selection fast path now returns before locking. -- Final affected skill/heartbeat controls: 35 tests passed in 29.17s, log - `/tmp/paperclip-skill-final-controls.log`. A prior local attempt could not start - Postgres because macOS shared-memory slots were full; retry succeeded after a - slot became available, without changing global limits or unrelated services. - Full typecheck on `666c60c859` passed in 32.98s. Full tests are still running; - initial native-session cases hit the same infrastructure limit. Removed 19 - abandoned 56-byte IPC headers only after verifying zero attachments and dead - creator/last-user PIDs. Live databases and their files were untouched. -- Review round four accepted batched reads but found unbounded per-directory - discovery stats. Discovery now uses directory-entry types and applies the cursor - and budget before per-path inspection, including empty-root budget accounting. - Twenty affected tests passed in 22.05s, including filesystem-call counts across - continuation pages (`/tmp/paperclip-skill-discovery-budget.log`). Final review - and full checks remain. Review round five found an exhausted-root continuation - edge case; the cursor now skips completed roots before budgeting discovery. - Twenty-one affected tests passed in 27.98s, including a page-boundary regression - (`/tmp/paperclip-skill-continuation.log`). The server CI shard stopped with - "The operation was canceled" and no failing assertion; investigate its timeout. -- Concurrent Git work is on `codex/recoverable-git-cache` and remains uncommitted. - Actual runtime/service reopening now restores A after deleting the worktree, - branch, and base while upstream is at B. A subsequent commit survives another - reopen. The real repository/DB experiment took 6.831s; evidence is - `/tmp/paperclip-git-recovery-candidate.json`. Expiration is not implemented yet. -- First Git baseline used a real bare remote, runtime workspace creation, a - database row, and ordinary reopening: missing worktree restores A while its - branch exists; missing branch restores upstream B; missing base fails. Fixture - script `/tmp/paperclip-git-recovery-baseline.mts`, inspected output - `/tmp/paperclip-git-recovery-baseline.json`, retained Git fixture - `/tmp/paperclip-git-baseline-sbJWbo`. The test DB was cleaned up. -- Greptile cleared `0da71b2ca2` (all its findings resolved). Security review then - identified repeated full-directory enumeration. Discovery now retains bounded - directory iterators across continuations, counts root entries in its budget, - closes abandoned iterators, and bounds company discovery. Twenty-one focused - cases passed in 29.99s (`/tmp/paperclip-skill-streaming-final.log`); server - typecheck passed. A new regression inspects actual directory reads/opener reuse. -- No deployed artifact or staging acceptance yet. Next: publish review fixes, - qualify final skill SHA, then perform dry-run rollout and measured restoration. + conservative policy skips that company. Do not clear recovery or fabricate pins + to obtain savings. A disposable fixture must distinguish worker proof from the + ordinary product restoration journey and disclose this legacy-data limitation. +- UI fixture created through Studio: skill `06a02b07-ee23-421c-afd1-5aa4b1dad423`, + slug `cache-restoration-acceptance-20261015`, initial saved marker + `SAVED-SKILL-V1-20261015`. No test run yet. Clean up only this task's fixtures. +- Live Git inventory: 152 active Git workspaces, 150 with runtime ownership and + 37 with isolated remote sources. Other owned task worktrees use shared managed + bases. Support both layouts while preserving shared bases. +- Temporary Railway key `paperclip-cleanup-acceptance-2026-10-15` remains + registered under explicit user approval. Revoke it and remove both local copies + when access is no longer needed; private paths are in retained task evidence. +- Git review found two second-cycle bugs. The worker now clears the consumed + reopen fence before expiration dispatch; successful reopening marks the + recipe restored so a later missing worktree can reattach its newer branch. Eighteen + cases passed; an added symlink fixture initially broke its own relative Git + commondir. After retaining a valid absolute commondir, that safety case passed + in 10.63s. The new guard rejects symlinked administration storage before deletion. + A second review caught the normal preparation refresh moving A to B after + recipe retirement. Keeping a restored recipe preserves the refresh guard while + accepting subsequent branch commits. Both joined lifecycle tests now pass in + 35.99s, including normal runtime preparation after reopening and another + missing-worktree rebuild after new work. The premature partial-edit test run + is excluded; `/tmp/paperclip-git-resumed-preparation2.log` is the completed run. + Full typecheck/build at `465e57ad52` passed; the guard change has its own server + typecheck. A CI policy test also mistook a script filename for a + GitHub secret expression; scope its check to expressions and retain explicit + dot/bracket/multiline secret controls. +- Remaining: address Git review, merge after all gates pass, verify exact serving + SHAs, dry-run both slices, measure actual reclamation, use restored skill and + reopened workspace through the product, inspect persisted results and protected + data, record final evidence, and revoke temporary access. No live acceptance yet. +- Latest branch-adoption review fix keeps the retained recipe's branch synchronized + with the row inside the existing reconciliation transaction. The joined restore, + rename, adoption, and normal-preparation case passed in 16.67s; 11 existing + reconciliation controls passed in 21.83s. Full typecheck/build passed for this + change. Evidence: `/tmp/paperclip-git-branch-adoption.log`, + `/tmp/paperclip-git-reconcile-controls.log`, and + `/tmp/paperclip-git-adoption-{typecheck,build}.log`. +- Skill deployment artifacts finished, but its deploy stage was rejected at + 17:50 UTC because another campaign already held the target. No competing + campaign was created. Inspect that operation before rerunning the same request. + UI fixture v2 is saved; neither staging fixture eviction nor restoration has run. ## Feedback-loop evidence diff --git a/maps/agent-access-map/workspaces.md b/maps/agent-access-map/workspaces.md index afcfaf3f9ca..354ea195f1d 100644 --- a/maps/agent-access-map/workspaces.md +++ b/maps/agent-access-map/workspaces.md @@ -57,6 +57,14 @@ are revalidated for reuse, cleanup and reopening. A failed save removes only the owned task worktree. The [isolated-source service](../../server/src/services/isolated-remote-workspace-source.ts) and its local regression tests establish source and lifecycle boundaries; live agent/provider qualification remains separate. +Reconstructable Git expiration adds no agent endpoint or permission. It records +company/workspace/repository identity and an exact commit before removing owned +cache files. Existing task/workspace reopening retains current company, source, +and trust checks. Missing upstream access returns a retryable error; it does not +choose a newer commit or grant access to another company's source. Shared project +bases and unresolved recovery remain protected. Local lifecycle evidence is in +[expiration tests](../../server/src/__tests__/git-workspace-expiration.test.ts). + Writable aliases must also be compatible with current and retained physical source projects' trust boundaries. Incompatible choices are rejected before dispatch; a projectless alias cannot bypass required sandbox containment. diff --git a/maps/feature-map/workspaces.md b/maps/feature-map/workspaces.md index 39a79412f21..4bda0d21c3c 100644 --- a/maps/feature-map/workspaces.md +++ b/maps/feature-map/workspaces.md @@ -57,6 +57,15 @@ and [preparation tests](../../server/src/services/heartbeat/workspace-preparatio exercise the real Git isolation and lifecycle guards. They do not qualify live remote credentials or a provider deployment. +For reconstructable Git expiration, finish a disposable task tree, run the +internal lifecycle sweep after its 24-hour grace, and inspect candidate/reclaimed +bytes. Advance its upstream first; use the ordinary task/workspace reopen control +and verify the original commit, file contents, and subsequent persisted work. +Current runs, previews, recovery, unique files and shared base clones must stay. +The [expiration tests](../../server/src/__tests__/git-workspace-expiration.test.ts) +exercise real local Git and database paths, interruption and concurrent reopening; +cloud credentials and the deployed user journey require separate live evidence. + For task workspace selection, use the authorized task workspace API to choose a task directory, project source, or existing workspace. Verify that a change made during a run stays pending until the next normal admission. Submit another diff --git a/server/src/__tests__/git-workspace-expiration.test.ts b/server/src/__tests__/git-workspace-expiration.test.ts new file mode 100644 index 00000000000..b0a3d7eb8ae --- /dev/null +++ b/server/src/__tests__/git-workspace-expiration.test.ts @@ -0,0 +1,297 @@ +import { promises as fs } from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { execFileSync } from "node:child_process"; +import { randomUUID } from "node:crypto"; +import { pathToFileURL } from "node:url"; +import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest"; +import { eq, sql } from "drizzle-orm"; +import { companies, projects, issues, executionWorkspaces, agents, heartbeatRuns, nativeRunFinalizations, workspaceRuntimeServices, createDb } from "@paperclipai/db"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; +import { resolveManagedProjectWorkspaceDir } from "../home-paths.js"; +import { materializeManagedProjectWorkspace } from "../services/managed-repository-checkout.js"; +import { materializeIsolatedRemoteWorkspaceSource } from "../services/isolated-remote-workspace-source.js"; +import { realizeExecutionWorkspace, ensurePersistedExecutionWorkspaceAvailable } from "../services/workspace-runtime.js"; +import { executionWorkspaceService } from "../services/execution-workspaces.js"; +import { expireReconstructableGitWorkspace } from "../services/git-workspace-expiration.js"; + +const gitEnv = { ...process.env, GIT_AUTHOR_NAME: "Fixture", GIT_AUTHOR_EMAIL: "fixture@example.test", + GIT_COMMITTER_NAME: "Fixture", GIT_COMMITTER_EMAIL: "fixture@example.test", GIT_TERMINAL_PROMPT: "0" }; +const git = (cwd: string, ...args: string[]) => execFileSync("git", args, { cwd, env: gitEnv, encoding: "utf8", stdio: ["ignore", "pipe", "pipe"] }).trim(); +const exists = (target: string) => fs.lstat(target).then(() => true, () => false); + +const support = await getEmbeddedPostgresTestSupport(); +const describeEmbeddedPostgres = support.supported ? describe : describe.skip; +describeEmbeddedPostgres("reconstructable Git expiration", () => { + let db: ReturnType; + let database: Awaited>; + let root: string; + const previousHome = process.env.PAPERCLIP_HOME; + const previousInstance = process.env.PAPERCLIP_INSTANCE_ID; + beforeAll(async () => { + database = await startEmbeddedPostgresTestDatabase("git-cache-expiration-"); + db = createDb(database.connectionString); + root = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-git-expiration-")); + process.env.PAPERCLIP_HOME = path.join(root, "home"); + process.env.PAPERCLIP_INSTANCE_ID = "default"; + }); + afterEach(async () => { vi.restoreAllMocks(); await db.execute(sql`TRUNCATE TABLE companies CASCADE`); }); + afterAll(async () => { + if (previousHome === undefined) delete process.env.PAPERCLIP_HOME; else process.env.PAPERCLIP_HOME = previousHome; + if (previousInstance === undefined) delete process.env.PAPERCLIP_INSTANCE_ID; else process.env.PAPERCLIP_INSTANCE_ID = previousInstance; + await db.$client.end({ timeout: 1 }); await database.cleanup(); await fs.rm(root, { recursive: true, force: true }); + }); + + async function fixture(managed = false, nested = false) { + const companyId = randomUUID(), issueId = randomUUID(), workspaceId = randomUUID(); + const location = path.join(root, companyId), source = path.join(location, "author"), remote = path.join(location, "remote.git"); + await fs.mkdir(source, { recursive: true }); + git(source, "init", "--initial-branch=main"); + await fs.writeFile(path.join(source, "marker.txt"), "commit A\n"); + await fs.writeFile(path.join(source, ".gitignore"), "node_modules/\nprivate-notes/\n"); + git(source, "add", "."); git(source, "commit", "-m", "A"); + const commitA = git(source, "rev-parse", "HEAD"); + git(location, "clone", "--bare", source, remote); git(source, "remote", "add", "origin", remote); + await db.insert(companies).values({ id: companyId, name: "Git expiration fixture", issuePrefix: `GC${companyId.slice(0, 6)}` }); + await db.insert(issues).values({ id: issueId, companyId, title: "Finished task", status: "done", completedAt: new Date(Date.now() - 48 * 3_600_000) }); + const repoUrl = pathToFileURL(remote).href; + const projectId = managed ? randomUUID() : null; + if (projectId) await db.insert(projects).values({ id: projectId, companyId, name: "Fixture project" }); + const baseCwd = managed + ? (await materializeManagedProjectWorkspace(resolveManagedProjectWorkspaceDir({ companyId, projectId: projectId!, repoName: "remote" }), { repoUrl })).cwd + : await materializeIsolatedRemoteWorkspaceSource({ companyId, issueId, repoUrl, resolveGitAuth: async () => null }); + const base = { baseCwd, source: "task_session" as const, projectId: null, workspaceId: null, repoUrl, repoRef: "origin/main" }; + const workspace = await realizeExecutionWorkspace({ base, issue: { id: issueId, identifier: "GC-1", title: "Fixture" }, agent: { id: null, name: "Fixture", companyId }, + config: { workspaceStrategy: { type: "git_worktree", branchTemplate: "fixture-task", worktreeParentDir: nested ? path.join(baseCwd, ".paperclip", "worktrees") : path.join(location, "worktrees"), baseRef: "origin/main" } }, skipHostProvision: true }); + await db.insert(executionWorkspaces).values({ id: workspaceId, companyId, projectId, sourceIssueId: issueId, mode: "isolated_workspace", strategyType: "git_worktree", name: "Fixture", status: "active", + providerType: "local_fs", cwd: workspace.cwd, providerRef: workspace.cwd, repoUrl, baseRef: "origin/main", branchName: workspace.branchName, + metadata: { createdByRuntime: workspace.branchCreatedByRuntime, gitBranchOwnershipVersion: 1, baseRefSha: workspace.baseRefSha, + ...(managed ? {} : { isolatedRemoteSource: { schema: "paperclip.isolated-remote-source.v1", companyId, issueId, repoUrl } }) } }); + await db.update(issues).set({ executionWorkspaceId: workspaceId }).where(eq(issues.id, issueId)); + const row = () => db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspaceId)).then(rows => rows[0]!); + const expire = async (options: Partial[2]> = {}) => + expireReconstructableGitWorkspace(db, await row(), { dryRun: false, expireBase: true, now: new Date(), ...options }); + const reopen = (company = companyId) => executionWorkspaceService(db).reopenClosedIsolatedExecutionWorkspaceForIssue({ workspaceId, + issue: { id: issueId, companyId: company, projectId: null }, actor: { actorType: "user", agentId: null }, + authorizationActor: { type: "board", source: "local_implicit", isInstanceAdmin: true } }); + const advance = async () => { + await fs.writeFile(path.join(source, "marker.txt"), "commit B\n"); git(source, "add", "."); git(source, "commit", "-m", "B"); git(source, "push", "origin", "main"); + return git(source, "rev-parse", "HEAD"); + }; + return { companyId, issueId, workspaceId, cwd: workspace.cwd, baseCwd, base, remote, source, commitA, row, expire, reopen, advance }; + } + + it("expires through the lifecycle worker and restores the recorded commit after upstream advances", async () => { + const f = await fixture(); + await fs.mkdir(path.join(f.cwd, "node_modules")); await fs.writeFile(path.join(f.cwd, "node_modules", "cache"), "regenerable"); + await fs.symlink("cache", path.join(f.cwd, "node_modules", "cache-link")); + expect(await f.expire({ dryRun: true })).toMatchObject({ candidate: true, expired: false, reclaimedBytes: 0, skipReason: null }); + expect(await exists(f.cwd)).toBe(true); expect((await f.row()).metadata?.gitCacheRestoration).toBeUndefined(); + await f.advance(); + const outcome = await executionWorkspaceService(db, { gitCacheExpirationEnabled: true, gitBaseExpirationEnabled: true }).sweepTerminalWorkspaces(); + expect(outcome.archived).toBe(1); + expect(await exists(f.cwd)).toBe(false); expect(await exists(f.baseCwd)).toBe(false); + expect((await f.row()).metadata?.gitCacheRestoration).toMatchObject({ commit: f.commitA }); + expect(await f.reopen()).toMatchObject({ ok: true, reopened: true }); + expect(git(f.cwd, "rev-parse", "HEAD")).toBe(f.commitA); + expect(await fs.readFile(path.join(f.cwd, "marker.txt"), "utf8")).toBe("commit A\n"); + // The next heartbeat uses this runtime path before handing files to the agent. + await ensurePersistedExecutionWorkspaceAvailable({ db, base: f.base, + workspace: (await executionWorkspaceService(db).getStoredById(f.companyId, f.workspaceId))!, + issue: { id: f.issueId, identifier: "GC-1", title: "Resumed task" }, + agent: { id: null, name: "Fixture", companyId: f.companyId }, skipHostProvision: true }); + expect(git(f.cwd, "rev-parse", "HEAD")).toBe(f.commitA); + await fs.writeFile(path.join(f.cwd, "continued.txt"), "continued\n"); git(f.cwd, "add", "."); git(f.cwd, "commit", "-m", "Continued"); + const continued = git(f.cwd, "rev-parse", "HEAD"); + expect(await f.reopen()).toMatchObject({ ok: true, reopened: false }); + expect(git(f.cwd, "rev-parse", "HEAD")).toBe(continued); + expect((await f.row()).metadata?.gitCacheRestoration).toMatchObject({ commit: f.commitA, restoredAt: expect.any(String) }); + git(f.baseCwd, "worktree", "remove", f.cwd); + await db.update(executionWorkspaces).set({ status: "archived" }).where(eq(executionWorkspaces.id, f.workspaceId)); + expect(await f.reopen()).toMatchObject({ ok: true, reopened: true }); + expect(git(f.cwd, "rev-parse", "HEAD")).toBe(continued); + }); + + it("expires again after a reopened task consumes its fence and finishes", async () => { + const f = await fixture(); + expect((await f.expire()).expired).toBe(true); + expect(await f.reopen()).toMatchObject({ ok: true, reopened: true }); + await db.update(issues).set({ status: "in_progress", completedAt: null }).where(eq(issues.id, f.issueId)); + const worker = executionWorkspaceService(db, { gitCacheExpirationEnabled: true, gitBaseExpirationEnabled: true }); + await worker.sweepTerminalWorkspaces(); + expect((await f.row()).metadata?.reopenPendingConsumption).not.toBe(true); + await fs.writeFile(path.join(f.cwd, "continued.txt"), "second task cycle\n"); + git(f.cwd, "add", "."); git(f.cwd, "commit", "-m", "Second task cycle"); + git(f.cwd, "push", "origin", "HEAD:refs/heads/continued"); + const continued = git(f.cwd, "rev-parse", "HEAD"); + await db.update(issues).set({ status: "done", completedAt: new Date(Date.now() - 48 * 3_600_000) }).where(eq(issues.id, f.issueId)); + expect((await worker.sweepTerminalWorkspaces()).archived).toBe(1); + expect((await f.row()).metadata?.gitCacheRestoration).toMatchObject({ commit: continued }); + expect(await f.reopen()).toMatchObject({ ok: true, reopened: true }); + expect(git(f.cwd, "rev-parse", "HEAD")).toBe(continued); + }); + + it("prepares a restored workspace after adopting its renamed branch", async () => { + const f = await fixture(); + await f.advance(); + expect((await f.expire()).expired).toBe(true); + expect(await f.reopen()).toMatchObject({ ok: true }); + git(f.cwd, "branch", "-m", "renamed-task"); + await db.update(executionWorkspaces).set({ status: "idle" }).where(eq(executionWorkspaces.id, f.workspaceId)); + const svc = executionWorkspaceService(db); + await svc.reconcileExecutionWorkspaceBranch(f.workspaceId, { + mode: "forward", actor: { actorType: "user", actorId: "local-board", agentId: null, runId: null }, + }); + expect((await f.row()).metadata?.gitCacheRestoration).toMatchObject({ + branchName: "renamed-task", commit: f.commitA, restoredAt: expect.any(String), + }); + await ensurePersistedExecutionWorkspaceAvailable({ db, base: f.base, + workspace: (await svc.getStoredById(f.companyId, f.workspaceId))!, + issue: { id: f.issueId, identifier: "GC-1", title: "Resumed task" }, + agent: { id: null, name: "Fixture", companyId: f.companyId }, skipHostProvision: true }); + expect(git(f.cwd, "branch", "--show-current")).toBe("renamed-task"); + expect(git(f.cwd, "rev-parse", "HEAD")).toBe(f.commitA); + }); + + it("preserves worktrees whose Git administration parent is a symlink", async () => { + const f = await fixture(); + const adminParent = path.join(f.baseCwd, ".git", "worktrees"); + const external = path.join(root, `external-admin-${randomUUID()}`); + const admin = git(f.cwd, "rev-parse", "--absolute-git-dir"); + await fs.writeFile(path.join(admin, "commondir"), `${path.join(f.baseCwd, ".git")}\n`); + await fs.rename(adminParent, external); + await fs.symlink(external, adminParent); + expect(git(f.cwd, "rev-parse", "HEAD")).toBe(f.commitA); + expect(await f.expire()).toMatchObject({ expired: false, skipReason: "unproven_worktree_admin" }); + expect(await exists(f.cwd)).toBe(true); + expect(await fs.readdir(external)).not.toEqual([]); + expect((await f.row()).metadata?.gitCacheRestoration).toBeUndefined(); + }); + + it.each(["dirty", "untracked", "ignored", "local_commit", "stash", "unknown_git_file"])("preserves %s unique state", async kind => { + const f = await fixture(); + if (kind === "dirty") await fs.writeFile(path.join(f.cwd, "marker.txt"), "unique changes"); + if (kind === "untracked") await fs.writeFile(path.join(f.cwd, "notes.txt"), "unique notes"); + if (kind === "ignored") { await fs.mkdir(path.join(f.cwd, "private-notes")); await fs.writeFile(path.join(f.cwd, "private-notes", "note"), "unique ignored"); } + if (kind === "local_commit") { await fs.writeFile(path.join(f.cwd, "new.txt"), "unique commit"); git(f.cwd, "add", "."); git(f.cwd, "commit", "-m", "Unpushed"); } + if (kind === "stash") { await fs.writeFile(path.join(f.cwd, "marker.txt"), "stash"); git(f.cwd, "stash"); } + if (kind === "unknown_git_file") await fs.writeFile(path.join(git(f.cwd, "rev-parse", "--absolute-git-dir"), "unique-note"), "unique Git data"); + expect((await f.expire()).expired).toBe(false); + expect(await exists(f.cwd)).toBe(true); expect(await exists(f.baseCwd)).toBe(true); + }); + + it("preserves live descendants and pending reopening", async () => { + const f = await fixture(); + await db.insert(issues).values({ companyId: f.companyId, title: "Active child", status: "in_progress", parentId: f.issueId }); + expect((await f.expire()).expired).toBe(false); + await db.delete(issues).where(eq(issues.parentId, f.issueId)); + const row = await f.row(); + await db.update(executionWorkspaces).set({ metadata: { ...row.metadata, reopenPendingConsumption: true } }).where(eq(executionWorkspaces.id, f.workspaceId)); + expect((await f.expire()).expired).toBe(false); + expect(await exists(f.cwd)).toBe(true); + }); + + it("retries after interruption with the restoration recipe already committed", async () => { + const f = await fixture(); + expect((await f.expire({ beforeDelete: async () => { throw new Error("interrupted_fixture"); } })).expired).toBe(false); + expect((await f.row()).metadata?.gitCacheRestoration).toMatchObject({ commit: f.commitA }); + expect(await exists(f.cwd)).toBe(true); + expect((await f.expire()).expired).toBe(true); + expect(await f.reopen()).toMatchObject({ ok: true }); + expect(git(f.cwd, "rev-parse", "HEAD")).toBe(f.commitA); + }); + + it("fails restoration explicitly on upstream failure and rejects a different company", async () => { + const f = await fixture(); + expect((await f.expire()).baseExpired).toBe(true); + await fs.rename(f.remote, `${f.remote}.offline`); + expect(await f.reopen()).toMatchObject({ ok: false, code: "rebuild_failed", message: "The saved workspace commit could not be restored. Check repository access and retry." }); + expect(await f.reopen(randomUUID())).toMatchObject({ ok: false, code: "not_reopenable" }); + expect(await exists(f.cwd)).toBe(false); + await fs.rename(`${f.remote}.offline`, f.remote); + expect(await f.reopen()).toMatchObject({ ok: true }); + expect(git(f.cwd, "rev-parse", "HEAD")).toBe(f.commitA); + }); + it("serializes concurrent reopening after deletion and publishes one transition", async () => { + const f = await fixture(); + let release!: () => void, entered!: () => void; + const paused = new Promise(resolve => { release = resolve; }); + const ready = new Promise(resolve => { entered = resolve; }); + const expiration = f.expire({ beforeDelete: async () => { entered(); await paused; } }); + await ready; + const first = f.reopen(), second = f.reopen(); + release(); + expect((await expiration).expired).toBe(true); + const results = await Promise.all([first, second]); + expect(results, JSON.stringify(results)).toEqual(expect.arrayContaining([expect.objectContaining({ ok: true, reopened: true }), expect.objectContaining({ ok: true, reopened: false })])); + expect(results.filter(result => result.ok && result.reopened)).toHaveLength(1); + expect(git(f.cwd, "rev-parse", "HEAD")).toBe(f.commitA); + }); + + it("recovers interrupted trash removal before reopening", async () => { + const f = await fixture(); + const originalRemove = fs.rm.bind(fs); + const remove = vi.spyOn(fs, "rm").mockImplementation(async (target, options) => { + if (String(target).includes(".expired-git-worktree-")) throw new Error("interrupted_trash_fixture"); + return originalRemove(target, options); + }); + expect((await f.expire()).skipReason).toBe("interrupted_trash_fixture"); + expect(await exists(f.cwd)).toBe(false); + expect((await f.row()).metadata?.gitCacheRestoration).toMatchObject({ commit: f.commitA }); + remove.mockRestore(); + expect(await f.reopen()).toMatchObject({ ok: true, reopened: true }); + expect(git(f.cwd, "rev-parse", "HEAD")).toBe(f.commitA); + }); + + it("preserves scheduled runs, unresolved recovery, and live previews", async () => { + const f = await fixture(); + const agentId = randomUUID(), runId = randomUUID(); + await db.insert(agents).values({ id: agentId, companyId: f.companyId, name: "Fixture", role: "engineer", adapterType: "process" }); + await db.insert(heartbeatRuns).values({ id: runId, companyId: f.companyId, agentId, nativeIssueId: f.issueId, status: "scheduled_retry" }); + expect((await f.expire()).expired).toBe(false); + await db.update(heartbeatRuns).set({ status: "failed", finishedAt: new Date() }).where(eq(heartbeatRuns.id, runId)); + await db.insert(nativeRunFinalizations).values({ runId, companyId: f.companyId, issueId: f.issueId, phase: "terminal_failure", recoveryState: "blocked" }); + expect((await f.expire()).expired).toBe(false); + await db.delete(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, runId)); + await db.insert(workspaceRuntimeServices).values({ id: randomUUID(), companyId: f.companyId, executionWorkspaceId: f.workspaceId, + scopeType: "execution_workspace", serviceName: "Preview", status: "running", lifecycle: "persistent", provider: "local" }); + expect((await f.expire()).expired).toBe(false); + expect(await exists(f.cwd)).toBe(true); + }); + + it("skips deletion when authenticated upstream verification fails", async () => { + const f = await fixture(); + await fs.rename(f.remote, `${f.remote}.offline`); + expect((await f.expire()).expired).toBe(false); + expect(await exists(f.cwd)).toBe(true); + expect((await f.row()).metadata?.gitCacheRestoration).toBeUndefined(); + }); + + it("keeps an unused base containing custom Git files", async () => { + const f = await fixture(); + const note = path.join(f.baseCwd, ".git", "private-note"); + await fs.writeFile(note, "unique base metadata"); + const outcome = await f.expire(); + expect(outcome).toMatchObject({ expired: true, baseExpired: false, skipReason: "unknown_base_git_files" }); + expect(await fs.readFile(note, "utf8")).toBe("unique base metadata"); + expect(await f.reopen()).toMatchObject({ ok: true }); + }); + + it("expires owned task worktrees while preserving their shared managed base", async () => { + const f = await fixture(true, true); + await f.advance(); + expect(await f.expire()).toMatchObject({ expired: true, baseExpired: false, skipReason: "shared_base_preserved" }); + expect(await exists(f.baseCwd)).toBe(true); + expect(await f.reopen()).toMatchObject({ ok: true }); + expect(git(f.cwd, "rev-parse", "HEAD")).toBe(f.commitA); + }); + + it("expires an isolated base whose worktree was nested inside it", async () => { + const f = await fixture(false, true); + expect(await f.expire()).toMatchObject({ expired: true, baseExpired: true }); + expect(await exists(f.baseCwd)).toBe(false); + expect(await f.reopen()).toMatchObject({ ok: true }); + expect(git(f.cwd, "rev-parse", "HEAD")).toBe(f.commitA); + }); + +}); diff --git a/server/src/config.ts b/server/src/config.ts index 8ea462b8fbe..4d752fbdb01 100644 --- a/server/src/config.ts +++ b/server/src/config.ts @@ -79,6 +79,9 @@ export interface Config { databaseBackupIntervalMinutes: number; databaseBackupRetentionDays: number; databaseBackupDir: string; + gitCacheExpirationEnabled: boolean; + gitBaseExpirationEnabled: boolean; + gitCacheExpirationDryRun: boolean; workspaceReaperCooldownDays: number; workspaceCompactionDelayHours: number | null; skillVersionRetentionEnabled: boolean; @@ -375,6 +378,9 @@ export function loadConfig(): Config { databaseBackupIntervalMinutes, databaseBackupRetentionDays, databaseBackupDir, + gitCacheExpirationEnabled: process.env.PAPERCLIP_GIT_CACHE_EXPIRATION_ENABLED === "true", + gitBaseExpirationEnabled: process.env.PAPERCLIP_GIT_BASE_EXPIRATION_ENABLED === "true", + gitCacheExpirationDryRun: process.env.PAPERCLIP_GIT_CACHE_EXPIRATION_DRY_RUN !== "false", workspaceReaperCooldownDays, workspaceCompactionDelayHours, skillVersionRetentionEnabled: process.env.PAPERCLIP_SKILL_VERSION_RETENTION_ENABLED !== "false", diff --git a/server/src/index.ts b/server/src/index.ts index 3221a851ffc..b76a63fb107 100644 --- a/server/src/index.ts +++ b/server/src/index.ts @@ -1412,6 +1412,9 @@ async function startServerWithDatabaseTeardown( wakeup: heartbeat.wakeup, }); const terminalWorkspaces = executionWorkspaceService(db as any, { + gitCacheExpirationEnabled: config.gitCacheExpirationEnabled, + gitBaseExpirationEnabled: config.gitBaseExpirationEnabled, + gitCacheExpirationDryRun: config.gitCacheExpirationDryRun, workspaceReaperCooldownDays: config.workspaceReaperCooldownDays, workspaceCompactionDelayMs: config.workspaceCompactionDelayHours === null ? null diff --git a/server/src/services/execution-workspaces.ts b/server/src/services/execution-workspaces.ts index d3a80e3aa94..8463c768eae 100644 --- a/server/src/services/execution-workspaces.ts +++ b/server/src/services/execution-workspaces.ts @@ -1,7 +1,9 @@ import { measureRunTransaction } from "./execution-transaction-timing.js"; import { measureRunPhase } from "./execution-preparation-timing.js"; import { assertTaskWorkspaceAccess, assertTaskWorkspaceSourceProjectAccess } from "./task-workspace-source-access.js"; -import { verifiedRecordedIsolatedRemoteSource } from "./isolated-remote-workspace-source.js"; +import { verifiedRecordedIsolatedRemoteSource, withIsolatedRepositorySourceLock } from "./isolated-remote-workspace-source.js"; +import { expireReconstructableGitWorkspace, hasPendingGitExpiration } from "./git-workspace-expiration.js"; +import { readGitCacheRestoration, restoreGitCacheBase, GitCacheRestorationFailure } from "./git-workspace-restoration.js"; import { parseObject } from "../adapters/utils.js"; import { resolveCoreTrustPreset } from "./trust-preset-resolver.js"; import { CONTROLLER_PROVISION_COPY_ONLY, workspaceRequiresCopyOnlyProvisioning } from "./workspace-host-provision.js"; @@ -286,6 +288,9 @@ export type ReopenClosedIsolatedExecutionWorkspaceResult = | { ok: false; code: "not_reopenable" | "rebuild_failed"; message: string }; export type ExecutionWorkspaceServiceOptions = { + gitCacheExpirationEnabled?: boolean; + gitBaseExpirationEnabled?: boolean; + gitCacheExpirationDryRun?: boolean; terminalWorkspaceInspectionBackoffMs?: number; resolvePullRequestDetails?: PullRequestMergeDetailsResolver; now?: () => Date; @@ -3132,9 +3137,10 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic try { if (options.compact !== false) await purgeDeferredCompactionTrash(signal); const baseCandidateFilter = and( - inArray(executionWorkspaces.status, ["active", "idle", "in_review"]), - isNull(executionWorkspaces.closedAt), sql`${executionWorkspaces.sourceIssueId} IS NOT NULL`, + or(and(inArray(executionWorkspaces.status, ["active", "idle", "in_review"]), isNull(executionWorkspaces.closedAt)), + opts.gitCacheExpirationEnabled ? and(inArray(executionWorkspaces.status, ["archived", "cleanup_failed"]), + sql`${executionWorkspaces.metadata}->'gitCacheExpiration'->>'completed' = 'false'`) : undefined), ); // Continue the scan after the previous sweep's last row. The keyset // predicate uses the same (updatedAt, id) order as the query, so each @@ -3222,6 +3228,21 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic result.skippedNonTerminalTree += 1; continue; } + if (opts.gitCacheExpirationEnabled && (hasPendingGitExpiration(workspace) + || (workspace.mode === "isolated_workspace" && workspace.strategyType === "git_worktree"))) { + const expiration = await expireReconstructableGitWorkspace(db, workspace, { + dryRun: opts.gitCacheExpirationDryRun ?? false, expireBase: opts.gitBaseExpirationEnabled ?? false, + now: now(), signal, beforeDelete: opts.beforeTerminalWorkspaceCleanup, + }); + terminalWorkspaceDeadlines.set(workspace.id, now().getTime() + 60 * 60 * 1000); + if (expiration.candidate) result.eligible += 1; + if (expiration.expired) result.archived += 1; + logger.info({ event: "execution_workspace.git_cache_expiration", executionWorkspaceId: workspace.id, + companyId: workspace.companyId, dryRun: opts.gitCacheExpirationDryRun ?? false, ...expiration }, + "Git workspace cache expiration inspected"); + // A failed reconstruction proof must not fall through to legacy deletion. + continue; + } const latestTerminalAt = Math.max(...issueTree.map(issue => issueTerminalTimestamp(issue)!.getTime())); // Compaction runs once per terminal cycle, after its own delay and // before the archive cooldown, so a workspace the reaper cannot archive @@ -3632,13 +3653,29 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic eq(issues.executionWorkspaceId, input.workspaceId), await issueReadSqlCondition(db, input.authorizationActor), )); - const snapshot = boundSnapshot ? await db.select().from(executionWorkspaces).where(and( + const authorizedSnapshot = boundSnapshot ? await db.select().from(executionWorkspaces).where(and( eq(executionWorkspaces.id, input.workspaceId), eq(executionWorkspaces.companyId, issue.companyId), eq(executionWorkspaces.mode, "isolated_workspace"), await executionWorkspaceReadSqlCondition(db, input.authorizationActor), )).then(rows => rows[0] ?? null) : null; - if (!snapshot) return { ok: false, code: "not_reopenable", message: "Execution workspace is not reopenable" }; + if (!authorizedSnapshot) return { ok: false, code: "not_reopenable", message: "Execution workspace is not reopenable" }; + await assertTaskWorkspaceAccess(db, input.authorizationActor, issue.companyId, authorizedSnapshot.id); + const sourceIssueId = authorizedSnapshot.sourceIssueId; + const reopen = async (): Promise => { + const freshSnapshot = await db.select().from(executionWorkspaces).where(and( + eq(executionWorkspaces.id, input.workspaceId), eq(executionWorkspaces.companyId, issue.companyId), + eq(executionWorkspaces.mode, "isolated_workspace"), + await executionWorkspaceReadSqlCondition(db, input.authorizationActor), + )).then(rows => rows[0] ?? null); + if (!freshSnapshot || freshSnapshot.sourceIssueId !== sourceIssueId) { + return { ok: false, code: "not_reopenable", message: "Execution workspace source changed during reopen" }; + } + const snapshot = freshSnapshot; await assertTaskWorkspaceAccess(db, input.authorizationActor, issue.companyId, snapshot.id); + let recipe: ReturnType; + try { recipe = readGitCacheRestoration(snapshot); } + catch { return { ok: false, code: "rebuild_failed", message: new GitCacheRestorationFailure().message }; } + const gitAuth = createGitRemoteAuthProvider(db, snapshot.companyId, { issueId: snapshot.sourceIssueId ?? issue.id }); const [ { ensurePersistedExecutionWorkspaceAvailable }, { workspaceOperationService }, @@ -3666,8 +3703,14 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic .then((rows) => parseProjectExecutionWorkspacePolicy(rows[0]?.executionWorkspacePolicy)), ]); let resolvedBaseCwd: string | null; - try { resolvedBaseCwd = await verifiedRecordedIsolatedRemoteSource(snapshot) ?? projectWorkspace?.cwd ?? null; } - catch { return { ok: false, code: "rebuild_failed", message: "Failed to rebuild the execution workspace" }; } + try { + resolvedBaseCwd = recipe && isClosedExecutionWorkspaceStatus(snapshot.status) + ? await restoreGitCacheBase(snapshot, recipe, gitAuth) + : await verifiedRecordedIsolatedRemoteSource(snapshot) ?? projectWorkspace?.cwd ?? null; + } catch { + return { ok: false, code: "rebuild_failed", message: recipe + ? new GitCacheRestorationFailure().message : "Failed to rebuild the execution workspace" }; + } if (isClosedExecutionWorkspaceStatus(snapshot.status) && resolvedBaseCwd == null && snapshot.strategyType === "git_worktree" && snapshot.projectId) { try { const managedWorkspace = await ensureManagedProjectWorkspace({ @@ -3689,7 +3732,7 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic }); resolvedBaseCwd = managedWorkspace.cwd; } catch { - return { ok: false, code: "rebuild_failed", message: "Failed to rebuild the execution workspace" }; + return { ok: false, code: "rebuild_failed", message: recipe ? new GitCacheRestorationFailure().message : "Failed to rebuild the execution workspace" }; } } let capturedSource: IsolatedWorkspaceSourceAccess | null = null; @@ -3709,7 +3752,7 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic actor: input.authorizationActor, captured: capturedSource, issueId: issue.id, phase: "preparing", }); } catch { - return { ok: false, code: "rebuild_failed", message: "Failed to rebuild the execution workspace" }; + return { ok: false, code: "rebuild_failed", message: recipe ? new GitCacheRestorationFailure().message : "Failed to rebuild the execution workspace" }; } } const result = await db.transaction(async (tx): Promise => { @@ -3779,6 +3822,7 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic const nextMetadata = bumpExecutionWorkspaceLifecycleGeneration( row.metadata as Record | null, ); + delete nextMetadata.gitCacheExpiration; const recorder = workspaceOperationService(db).createRecorder({ companyId: row.companyId, executionWorkspaceId: row.id, @@ -3815,6 +3859,7 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic if (copyOnlyProvisioning) nextMetadata[CONTROLLER_PROVISION_COPY_ONLY] = true; const realized = await ensurePersistedExecutionWorkspaceAvailable({ db: tx as unknown as Db, + resolveGitAuth: gitAuth, skipHostProvision: trust.kind === "low_trust_review" || copyOnlyProvisioning, beforeSourceCopy: verifyCopyIdentity, afterSourceCopy: verifyCopyIdentity, @@ -3909,7 +3954,7 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic generation: nextGeneration, }, }); - return { ok: false, code: "rebuild_failed", message: "Failed to rebuild the execution workspace" }; + return { ok: false, code: "rebuild_failed", message: recipe ? new GitCacheRestorationFailure().message : "Failed to rebuild the execution workspace" }; } // The rebuild succeeded. Publish the row as active in one write, and clear @@ -3918,6 +3963,10 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic // the issue out of the terminal state only after this reopen returns. The // flag stops the terminal reaper and the archive route from archiving and // destroying the rebuilt worktree in that window. + // Keep the refresh guard after reopening so normal preparation cannot + // advance the saved checkout to upstream HEAD. Its intact branch may now + // receive newer task commits; the next expiration records a fresh recipe. + if (recipe) nextMetadata.gitCacheRestoration = { ...recipe, restoredAt: now().toISOString() }; const activeMetadata = setMetadataReopenPendingConsumption(nextMetadata, now()); await measureRunPhase("workspace.notification", () => notifyTerminalWorkspaceWork(tx, `workspace:${row.id}`)); const activeRow = await tx @@ -3935,7 +3984,7 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic .returning() .then((rows) => rows[0] ?? null); if (!activeRow) { - return { ok: false, code: "rebuild_failed", message: "Failed to rebuild the execution workspace" }; + return { ok: false, code: "rebuild_failed", message: recipe ? new GitCacheRestorationFailure().message : "Failed to rebuild the execution workspace" }; } logger.info( { @@ -3967,7 +4016,7 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic return { ok: true, reopened: true, workspace: toExecutionWorkspace(activeRow), generation: nextGeneration }; }).catch((error): ReopenClosedIsolatedExecutionWorkspaceResult => { logger.warn({ error, executionWorkspaceId: snapshot.id }, "Execution workspace reopen failed"); - return { ok: false, code: "rebuild_failed", message: "Failed to rebuild the execution workspace" }; + return { ok: false, code: "rebuild_failed", message: recipe ? new GitCacheRestorationFailure().message : "Failed to rebuild the execution workspace" }; }); if (capturedSource && authorizedTarget && resolvedBaseCwd) { try { @@ -3981,10 +4030,14 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic workspaceId: snapshot.id, issue, actor, expectedGeneration: result.generation, }); } - return { ok: false, code: "rebuild_failed", message: "Failed to rebuild the execution workspace" }; + return { ok: false, code: "rebuild_failed", message: recipe ? new GitCacheRestorationFailure().message : "Failed to rebuild the execution workspace" }; } } return result; + }; + return authorizedSnapshot.sourceIssueId + ? withIsolatedRepositorySourceLock(db, authorizedSnapshot.companyId, authorizedSnapshot.sourceIssueId, reopen) + : reopen(); }, // Clear the reopen-pending flag after a caller failed to consume a reopened @@ -4373,6 +4426,13 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic branchName: inspection.toBranch, updatedAt: now, }; + const restoration = readGitCacheRestoration(lockedRow); + if (restoration) { + updatePatch.metadata = { + ...lockedRow.metadata, + gitCacheRestoration: { ...restoration, branchName: inspection.toBranch }, + }; + } if (lockedWorkspace.name === inspection.fromBranch) { updatePatch.name = inspection.toBranch; } diff --git a/server/src/services/git-workspace-expiration.ts b/server/src/services/git-workspace-expiration.ts new file mode 100644 index 00000000000..a6c39b670ae --- /dev/null +++ b/server/src/services/git-workspace-expiration.ts @@ -0,0 +1,230 @@ +import fs from "node:fs/promises"; +import path from "node:path"; +import { and, eq, ne, sql } from "drizzle-orm"; +import { executionWorkspaces, issues, heartbeatRuns, nativeRunFinalizations, environmentLeases, + workspaceRuntimeServices, issueRecoveryActions, projectWorkspaces, withDedicatedDbConnection, type Db } from "@paperclipai/db"; +import { createGitRemoteAuthProvider } from "./git-credentials.js"; +import { withIsolatedRepositorySourceLock, verifiedRecordedIsolatedRemoteSource } from "./isolated-remote-workspace-source.js"; +import { assertReproducibleGitFiles, assertReproducibleBaseGitStorage, inspectGitCacheRecipe, readGitCacheRestoration, + verifyRemoteGitCommit, gitCacheCommand, readGitCacheExpiration, finishGitCacheTrash, type GitCacheExpiration, GIT_CACHE_RESTORATION_KEY, GIT_CACHE_EXPIRATION_REASON } from "./git-workspace-restoration.js"; + + +async function allocatedBytes(target: string, shouldContinue: () => boolean): Promise { + if (!shouldContinue()) throw new Error("scan_stopped"); + const entry = await fs.lstat(target); + if (!entry.isDirectory() && !entry.isFile() && !entry.isSymbolicLink()) throw new Error("unsupported_filesystem_entry"); + let bytes = entry.blocks * 512; + if (entry.isDirectory()) for (const name of await fs.readdir(target)) bytes += await allocatedBytes(path.join(target, name), shouldContinue); + return bytes; +} +function skipReason(error: unknown) { + return error instanceof Error && /^[a-z][a-z_]+$/.test(error.message) ? error.message : "inspection_failed"; +} + +const EXPIRATION_KEY = "gitCacheExpiration"; +const DAY_MS = 24 * 60 * 60 * 1000; +type Workspace = typeof executionWorkspaces.$inferSelect; +type Tx = Parameters[0]>[0]; +type Expiration = GitCacheExpiration; +export type GitExpirationResult = { candidate: boolean; expired: boolean; reclaimedBytes: number; + candidateBytes: number; baseExpired: boolean; skipReason: string | null }; + +/** Re-read the whole tree and every runtime/recovery consumer immediately before deletion. */ +async function eligible(tx: Tx, workspace: Workspace, now: Date) { + if (workspace.metadata?.reopenPendingConsumption === true) return false; + const rows = await tx.execute<{ allowed: boolean }>(sql` + WITH RECURSIVE tree AS ( + SELECT id, status, completed_at, cancelled_at, updated_at FROM ${issues} + WHERE company_id = ${workspace.companyId} AND id = ${workspace.sourceIssueId} + UNION + SELECT c.id, c.status, c.completed_at, c.cancelled_at, c.updated_at FROM ${issues} c + JOIN tree p ON c.parent_id = p.id WHERE c.company_id = ${workspace.companyId} + ), relevant_runs AS ( + SELECT r.* FROM ${heartbeatRuns} r WHERE r.company_id = ${workspace.companyId} AND ( + r.issue_id IN (SELECT id FROM tree) OR r.native_issue_id IN (SELECT id FROM tree) + OR r.context_snapshot->>'issueId' IN (SELECT id::text FROM tree) + OR r.context_snapshot->>'taskId' IN (SELECT id::text FROM tree) + OR r.context_snapshot->>'executionWorkspaceId' = ${workspace.id} + OR r.id IN (SELECT unnest(ARRAY[checkout_run_id, execution_run_id]) FROM ${issues} + WHERE company_id = ${workspace.companyId} AND execution_workspace_id = ${workspace.id}) + ) + ) SELECT ( + EXISTS (SELECT 1 FROM tree) + AND NOT EXISTS (SELECT 1 FROM tree WHERE status NOT IN ('done', 'cancelled') + OR coalesce(CASE WHEN status = 'done' THEN completed_at ELSE cancelled_at END, updated_at) > ${new Date(now.getTime() - DAY_MS).toISOString()}::timestamptz) + AND NOT EXISTS (SELECT 1 FROM ${issues} WHERE company_id = ${workspace.companyId} + AND execution_workspace_id = ${workspace.id} AND status NOT IN ('done', 'cancelled')) + AND NOT EXISTS (SELECT 1 FROM relevant_runs WHERE status NOT IN ('succeeded', 'failed', 'timed_out', 'cancelled', 'interrupted') + OR finished_at IS NULL OR controller_lease_expires_at > now()) + AND NOT EXISTS (SELECT 1 FROM ${issueRecoveryActions} WHERE company_id = ${workspace.companyId} + AND source_issue_id IN (SELECT id FROM tree) AND status IN ('active', 'escalated')) + AND NOT EXISTS (SELECT 1 FROM ${nativeRunFinalizations} f WHERE f.company_id = ${workspace.companyId} + AND (f.issue_id IN (SELECT id FROM tree) OR f.run_id IN (SELECT id FROM relevant_runs)) + AND (f.phase <> 'committed' OR f.lease_owner IS NOT NULL OR f.recovery_state IS NOT NULL)) + AND NOT EXISTS (SELECT 1 FROM ${environmentLeases} l WHERE l.company_id = ${workspace.companyId} + AND (l.execution_workspace_id = ${workspace.id} OR l.heartbeat_run_id IN (SELECT id FROM relevant_runs)) + AND (l.status = 'pending_cleanup' OR l.cleanup_status = 'failed' OR l.metadata ? 'nativeWorkspaceExportResume' + OR (l.released_at IS NULL AND NOT coalesce(l.status = 'retained' AND l.lease_policy = 'reuse_by_environment' AND l.cleanup_status = 'success', false)))) + AND NOT EXISTS (SELECT 1 FROM ${workspaceRuntimeServices} WHERE company_id = ${workspace.companyId} + AND (execution_workspace_id = ${workspace.id} OR cwd = ${workspace.cwd} + OR left(cwd, length(${workspace.cwd}) + 1) = ${workspace.cwd} || '/') AND status <> 'stopped') + AND NOT EXISTS (SELECT 1 FROM ${executionWorkspaces} other WHERE other.id <> ${workspace.id} + AND (other.cwd = ${workspace.cwd} OR other.provider_ref = ${workspace.cwd} + OR left(other.cwd, length(${workspace.cwd}) + 1) = ${workspace.cwd} || '/')) + AND NOT EXISTS (SELECT 1 FROM ${projectWorkspaces} WHERE cwd = ${workspace.cwd} + OR left(cwd, length(${workspace.cwd}) + 1) = ${workspace.cwd} || '/') + ) AS allowed`); + return rows[0]?.allowed === true; +} + +export function hasPendingGitExpiration(workspace: Workspace) { + const state = workspace.metadata?.[EXPIRATION_KEY] as Expiration | undefined; + return state?.schema === "paperclip.git-cache-expiration.v1" && !state.completed; +} + +async function exists(target: string) { + return fs.lstat(target).then(() => true, (error: NodeJS.ErrnoException) => { if (error.code === "ENOENT") return false; throw error; }); +} + +/** Each ref/reflog object must be recoverable, and no other checkout may consume the base. */ +async function inspectBase(db: Db, workspace: Workspace, signal?: AbortSignal, allowCurrentWorktree = false) { + const recipe = readGitCacheRestoration(workspace)!; + if (recipe.sourceKind !== "isolated_remote") throw new Error("shared_base_preserved"); + const other = await db.select().from(executionWorkspaces).where(and(ne(executionWorkspaces.id, workspace.id), + sql`(${executionWorkspaces.metadata}->'isolatedRemoteSource'->>'companyId' = ${workspace.companyId} + AND ${executionWorkspaces.metadata}->'isolatedRemoteSource'->>'issueId' = ${workspace.sourceIssueId}) + OR ${executionWorkspaces.cwd} = ${recipe.sourcePath} OR ${executionWorkspaces.providerRef} = ${recipe.sourcePath}`)); + if (other.some(row => !["archived", "cleanup_failed"].includes(row.status) + || !readGitCacheRestoration(row) || !readGitCacheExpiration(row)?.workspaceRemoved)) throw new Error("base_has_consumers"); + const [projectConsumer] = await db.select({ id: projectWorkspaces.id }).from(projectWorkspaces).where(eq(projectWorkspaces.cwd, recipe.sourcePath)).limit(1); + if (projectConsumer) throw new Error("base_has_consumers"); + const [service] = await db.select({ id: workspaceRuntimeServices.id }).from(workspaceRuntimeServices).where(sql` + ${workspaceRuntimeServices.status} <> 'stopped' AND (${workspaceRuntimeServices.cwd} = ${recipe.sourcePath} + OR left(${workspaceRuntimeServices.cwd}, length(${recipe.sourcePath}) + 1) = ${recipe.sourcePath} || '/')`).limit(1); + if (service) throw new Error("base_has_live_services"); + await verifiedRecordedIsolatedRemoteSource(workspace); + const worktrees = (await gitCacheCommand(recipe.sourcePath, ["worktree", "list", "--porcelain"], signal)) + .split("\n").filter(line => line.startsWith("worktree ")).map(line => line.slice(9)); + const allowed = [await fs.realpath(recipe.sourcePath), ...(allowCurrentWorktree ? [await fs.realpath(recipe.worktreePath)] : [])]; + if (worktrees.some(cwd => !allowed.includes(cwd))) throw new Error("base_has_worktrees"); + await assertReproducibleGitFiles(recipe.sourcePath, signal); + await assertReproducibleBaseGitStorage(recipe.sourcePath, signal); + const refs = (await gitCacheCommand(recipe.sourcePath, ["for-each-ref", "--format=%(objectname) %(refname)"], signal)).trim().split("\n").filter(Boolean); + if (refs.some(ref => !/^[a-f0-9]{40} refs\/(heads|remotes\/origin)\//.test(ref))) throw new Error("unsupported_base_refs"); + const commits = new Set([...refs.map(ref => ref.split(" ")[0]!), + ...(await gitCacheCommand(recipe.sourcePath, ["reflog", "show", "--all", "--format=%H"], signal)).trim().split("\n").filter(Boolean)]); + if (commits.size > 50) throw new Error("base_ref_verification_budget"); + const auth = createGitRemoteAuthProvider(db, workspace.companyId, { issueId: workspace.sourceIssueId! }); + for (const commit of commits) await verifyRemoteGitCommit(recipe.repoUrl, commit, auth, signal); + // Unreachable objects may contain unique rewritten work not present in a reflog. + const unreachable = await gitCacheCommand(recipe.sourcePath, ["fsck", "--no-reflogs", "--unreachable", "--no-progress"], signal); + if (unreachable.trim()) throw new Error("unreachable_base_objects"); + return allocatedBytes(recipe.sourcePath, () => { signal?.throwIfAborted(); return true; }); +} + +export async function expireReconstructableGitWorkspace(db: Db, snapshot: Workspace, options: { + dryRun: boolean; expireBase: boolean; now: Date; signal?: AbortSignal; + beforeDelete?: (workspace: Workspace) => Promise; +}): Promise { + const result: GitExpirationResult = { candidate: false, expired: false, reclaimedBytes: 0, + candidateBytes: 0, baseExpired: false, skipReason: null }; + if (!snapshot.sourceIssueId) return { ...result, skipReason: "missing_source_issue" }; + try { + return await withIsolatedRepositorySourceLock(db, snapshot.companyId, snapshot.sourceIssueId, () => + withDedicatedDbConnection(db, connection => connection.transaction(async tx => { + const acquired = await tx.execute<{ locked: boolean }>(sql`select pg_try_advisory_xact_lock(hashtextextended(${`execution_workspace_lifecycle:${snapshot.id}`}, 0)) as locked`); + if (!acquired[0]?.locked) throw new Error("lifecycle_busy"); + let workspace = (await db.select().from(executionWorkspaces).where(and(eq(executionWorkspaces.id, snapshot.id), eq(executionWorkspaces.companyId, snapshot.companyId))))[0]; + if (!workspace || workspace.updatedAt.getTime() !== snapshot.updatedAt.getTime()) throw new Error("workspace_changed"); + if (!await eligible(tx, workspace, options.now)) throw new Error("live_or_unresolved_consumer"); + let state = ["active", "idle", "in_review"].includes(workspace.status) ? null : readGitCacheExpiration(workspace); + // A restored active workspace starts a new expiration cycle and records its new HEAD. + if (!hasPendingGitExpiration(workspace) && !["active", "idle", "in_review"].includes(workspace.status)) throw new Error("not_expirable"); + if (["active", "idle", "in_review"].includes(workspace.status)) state = null; + let recipe = state ? readGitCacheRestoration(workspace)! : await inspectGitCacheRecipe(workspace, + createGitRemoteAuthProvider(db, workspace.companyId, { issueId: workspace.sourceIssueId! }), options.signal); + const present = await exists(recipe.worktreePath); + if (!state && !present) throw new Error("missing_unverified_worktree"); + result.candidate = true; + if (present) result.candidateBytes = await allocatedBytes(recipe.worktreePath, () => { options.signal?.throwIfAborted(); return true; }); + if (options.dryRun) { + if (options.expireBase) { + try { result.candidateBytes += await inspectBase(db, { ...workspace, + metadata: { ...workspace.metadata, [GIT_CACHE_RESTORATION_KEY]: recipe } }, options.signal, true); } + catch (error) { result.skipReason = skipReason(error); } + } + return result; + } + if (!state) { + const generation = Number(workspace.metadata?.lifecycleGeneration ?? 0) + 1; + const rawAdmin = (await gitCacheCommand(recipe.worktreePath, ["rev-parse", "--absolute-git-dir"], options.signal)).trim(); + const worktreeAdmin = path.join(recipe.sourcePath, ".git", "worktrees", path.basename(rawAdmin)); + const canonicalAdmin = path.join(await fs.realpath(recipe.sourcePath), ".git", "worktrees", path.basename(rawAdmin)); + if (await fs.realpath(worktreeAdmin) !== canonicalAdmin || await fs.realpath(rawAdmin) !== canonicalAdmin) { + throw new Error("unproven_worktree_admin"); + } + state = { schema: "paperclip.git-cache-expiration.v1", generation, workspaceRemoved: false, + worktreeTrash: path.join(path.dirname(recipe.worktreePath), `.expired-git-worktree-${workspace.id}-${generation}`), + worktreeAdmin, baseTrash: null, completed: false }; + if (await exists(state.worktreeTrash)) throw new Error("unknown_worktree_trash"); + // Commit the immutable restoration recipe BEFORE touching disk. The separate + // lifecycle-lock connection remains held through deletion and final state. + [workspace] = await db.update(executionWorkspaces).set({ status: "archived", closedAt: options.now, + cleanupEligibleAt: options.now, cleanupReason: GIT_CACHE_EXPIRATION_REASON, updatedAt: options.now, + metadata: { ...workspace.metadata, lifecycleGeneration: generation, [GIT_CACHE_RESTORATION_KEY]: recipe, [EXPIRATION_KEY]: state }, + }).where(eq(executionWorkspaces.id, workspace.id)).returning(); + } + if (!workspace) throw new Error("workspace_changed"); + const persist = async () => { + [workspace] = await db.update(executionWorkspaces).set({ metadata: { ...workspace!.metadata, [EXPIRATION_KEY]: state }, updatedAt: new Date() }) + .where(eq(executionWorkspaces.id, snapshot.id)).returning(); + }; + await options.beforeDelete?.(workspace); + if (!await eligible(tx, workspace, options.now)) throw new Error("live_or_unresolved_consumer"); + if (!state.workspaceRemoved && present) { + const { acquireGitWorktreeCleanupLock } = await import("./workspace-runtime.js"); + const lock = await acquireGitWorktreeCleanupLock(recipe.worktreePath); + try { + await assertReproducibleGitFiles(recipe.worktreePath, options.signal); + if ((await gitCacheCommand(recipe.worktreePath, ["rev-parse", "HEAD"], options.signal)).trim() !== recipe.commit) throw new Error("head_changed"); + await fs.rename(recipe.worktreePath, state.worktreeTrash); + await finishGitCacheTrash(workspace); + result.reclaimedBytes += result.candidateBytes; + } finally { await lock.release(); } + } + await finishGitCacheTrash(workspace); + state.workspaceRemoved = true; + await persist(); + result.expired = true; + if (options.expireBase) { + try { + if ((!state.baseTrash || !await exists(state.baseTrash)) && await exists(recipe.sourcePath)) { + const bytes = await inspectBase(db, workspace!, options.signal); + if (!await eligible(tx, workspace!, options.now)) throw new Error("live_or_unresolved_consumer"); + const trash = path.join(path.dirname(recipe.sourcePath), `.expired-git-base-${snapshot.id}-${state.generation}`); + if (await exists(trash)) throw new Error("unknown_base_trash"); + state.baseTrash = trash; + await persist(); + await fs.rename(recipe.sourcePath, trash); + result.candidateBytes += bytes; + } + if (state.baseTrash && await exists(state.baseTrash)) { + const stat = await fs.lstat(state.baseTrash); + if (!stat.isDirectory() || stat.isSymbolicLink()) throw new Error("unsafe_base_trash"); + const bytes = await allocatedBytes(state.baseTrash, () => { options.signal?.throwIfAborted(); return true; }); + await fs.rm(state.baseTrash, { recursive: true, force: true }); + result.reclaimedBytes += bytes; + result.baseExpired = true; + } + } catch (error) { result.skipReason = skipReason(error); } + } + state.completed = recipe.sourceKind !== "isolated_remote" || !options.expireBase || (result.baseExpired || !await exists(recipe.sourcePath)) && (!state.baseTrash || !await exists(state.baseTrash)); + await persist(); + return result; + })), true); + } catch (error) { + options.signal?.throwIfAborted(); + result.skipReason = skipReason(error); + return result; + } +} diff --git a/server/src/services/git-workspace-restoration.ts b/server/src/services/git-workspace-restoration.ts new file mode 100644 index 00000000000..dd02daa52d1 --- /dev/null +++ b/server/src/services/git-workspace-restoration.ts @@ -0,0 +1,346 @@ +import fs from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; +import { execFile as execFileCallback } from "node:child_process"; +import { createHash } from "node:crypto"; +import { resolveManagedProjectWorkspaceDir, resolvePaperclipInstanceRoot } from "../home-paths.js"; +import { materializeManagedProjectWorkspace } from "./managed-repository-checkout.js"; +import { promisify } from "node:util"; +import type { GitRemoteAuthProvider } from "./git-credentials.js"; +import { isRuntimeOwnedGitBranch } from "./execution-workspace-branch-ownership.js"; +import { recordedIsolatedRemoteSource, verifiedRecordedIsolatedRemoteSource, materializeIsolatedRemoteWorkspaceSource } from "./isolated-remote-workspace-source.js"; +import { findRegenerableWorkspaceDirs } from "./workspace-compaction.js"; + +const execFile = promisify(execFileCallback); +const shaPattern = /^[a-f0-9]{40}$/; +export const GIT_CACHE_RESTORATION_KEY = "gitCacheRestoration"; +export const GIT_CACHE_EXPIRATION_REASON = "reproducible_git_cache_expired"; +export type GitCacheRestoration = { + schema: "paperclip.git-cache-restoration.v1"; + companyId: string; + workspaceId: string; + sourceIssueId: string; + repoUrl: string; + commit: string; + branchName: string; + sourceKind: "isolated_remote" | "managed_project"; + projectId: string | null; + sourcePath: string; + worktreePath: string; + gitBranchOwnershipVersion: 1; + verifiedAt: string; + restoredAt?: string; +}; +export type GitCacheExpiration = { + schema: "paperclip.git-cache-expiration.v1"; generation: number; workspaceRemoved: boolean; + worktreeTrash: string; worktreeAdmin: string; baseTrash: string | null; completed: boolean; +}; +type Workspace = { + id: string; companyId: string; projectId: string | null; sourceIssueId: string | null; repoUrl: string | null; + mode: string | null; strategyType: string | null; branchName: string | null; + cwd: string | null; providerRef: string | null; metadata: Record | null; +}; + +function managedSourcePaths(workspace: Workspace) { + if (!workspace.projectId || !workspace.repoUrl) return []; + let repoName: string | null = null; + try { repoName = new URL(workspace.repoUrl).pathname.replace(/\/+$/, "").split("/").pop()?.replace(/\.git$/i, "") || null; } catch { return []; } + const root = resolveManagedProjectWorkspaceDir({ companyId: workspace.companyId, projectId: workspace.projectId, repoName }); + return [root, `${root}-${createHash("sha256").update(workspace.repoUrl).digest("hex").slice(0, 12)}`]; +} + +async function verifiedGitCacheSource(workspace: Workspace, sourcePath: string) { + const privateSource = recordedIsolatedRemoteSource(workspace); + if (privateSource) { + if (sourcePath !== privateSource) throw new GitCacheRestorationFailure(); + return (await verifiedRecordedIsolatedRemoteSource(workspace))!; + } + if (!managedSourcePaths(workspace).includes(sourcePath)) throw new GitCacheRestorationFailure(); + const root = resolvePaperclipInstanceRoot(); + const actual = await fs.realpath(sourcePath); + if (actual !== path.join(await fs.realpath(root), path.relative(root, sourcePath))) throw new GitCacheRestorationFailure(); + const storage = await fs.lstat(path.join(actual, ".git")); + if (!storage.isDirectory() || storage.isSymbolicLink() + || (await gitCacheCommand(actual, ["rev-parse", "--path-format=absolute", "--git-common-dir"])).trim() !== path.join(actual, ".git") + || (await gitCacheCommand(actual, ["config", "--local", "--null", "--get-all", "remote.origin.url"])) !== `${workspace.repoUrl}\0`) throw new GitCacheRestorationFailure(); + return sourcePath; +} + +export class GitCacheRestorationFailure extends Error { + constructor() { super("The saved workspace commit could not be restored. Check repository access and retry."); } +} + +export function readGitCacheRestoration(workspace: Workspace): GitCacheRestoration | null { + const value = workspace.metadata?.[GIT_CACHE_RESTORATION_KEY]; + if (value === undefined) return null; + const recipe = value as GitCacheRestoration | null; + if (!recipe || recipe.schema !== "paperclip.git-cache-restoration.v1" + || !isRuntimeOwnedGitBranch(workspace.metadata) + || recipe.gitBranchOwnershipVersion !== 1 + || recipe.companyId !== workspace.companyId || recipe.workspaceId !== workspace.id + || recipe.sourceIssueId !== workspace.sourceIssueId || recipe.repoUrl !== workspace.repoUrl + || recipe.branchName !== workspace.branchName || recipe.worktreePath !== workspace.cwd + || (workspace.providerRef && recipe.worktreePath !== workspace.providerRef) + || recipe.projectId !== workspace.projectId + || (recipe.sourceKind === "isolated_remote" ? recipe.sourcePath !== recordedIsolatedRemoteSource(workspace) + : recipe.sourceKind !== "managed_project" || recordedIsolatedRemoteSource(workspace) !== null || !managedSourcePaths(workspace).includes(recipe.sourcePath)) + || !shaPattern.test(recipe.commit) || !Number.isFinite(Date.parse(recipe.verifiedAt)) + || (recipe.restoredAt !== undefined && !Number.isFinite(Date.parse(recipe.restoredAt)))) { + throw new GitCacheRestorationFailure(); + } + return recipe; +} + +export async function gitCacheCommand(cwd: string, args: string[], signal?: AbortSignal) { + const result = await execFile("git", ["-C", cwd, ...args], { + timeout: 60_000, signal, maxBuffer: 16 * 1024 * 1024, + env: { ...process.env, GIT_TERMINAL_PROMPT: "0", GIT_OPTIONAL_LOCKS: "0" }, + }); + return result.stdout; +} + +/** A fresh object store proves the object came from upstream, not our cache. */ +export async function verifyRemoteGitCommit(repoUrl: string, commit: string, resolveGitAuth: GitRemoteAuthProvider, signal?: AbortSignal) { + if (!shaPattern.test(commit)) throw new GitCacheRestorationFailure(); + const temporary = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-git-proof-")); + try { + const auth = await resolveGitAuth(repoUrl); + const env = { ...process.env, ...auth?.env, GIT_TERMINAL_PROMPT: "0", GIT_CONFIG_NOSYSTEM: "1", GIT_CONFIG_GLOBAL: os.devNull }; + const options = { timeout: 60_000, signal, maxBuffer: 1024 * 1024, env }; + await execFile("git", ["init", "--bare", temporary], options); + await execFile("git", [...auth?.configArgs ?? [], "-C", temporary, "fetch", "--no-tags", "--no-recurse-submodules", "--depth=1", "--", repoUrl, commit], options); + const result = await execFile("git", ["-C", temporary, "rev-parse", "FETCH_HEAD^{commit}"], options); + if (result.stdout.trim() !== commit) throw new GitCacheRestorationFailure(); + } catch { throw new GitCacheRestorationFailure(); } + finally { await fs.rm(temporary, { recursive: true, force: true }); } +} + +let templateFiles: Promise> | undefined; +async function stockGitFiles() { + templateFiles ??= (async () => { + const root = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-git-template-")); + const files = new Map(); + try { + await execFile("git", ["init", "--bare", root], { timeout: 10_000, + env: { ...process.env, GIT_CONFIG_NOSYSTEM: "1", GIT_CONFIG_GLOBAL: os.devNull } }); + for (const name of ["description", "info/exclude", ...(await fs.readdir(path.join(root, "hooks"))).map(name => `hooks/${name}`)]) { + files.set(name, await fs.readFile(path.join(root, name))); + } + return files; + } finally { await fs.rm(root, { recursive: true, force: true }); } + })(); + return templateFiles; +} + +export async function assertReproducibleBaseGitStorage(cwd: string, signal?: AbortSignal) { + const root = path.join(cwd, ".git"); + const allowed = new Set(["HEAD", "ORIG_HEAD", "FETCH_HEAD", "index", "config", "description", "hooks", "info", "refs", "objects", "logs", "packed-refs", "worktrees"]); + if ((await fs.readdir(root)).some(name => !allowed.has(name))) throw new Error("unknown_base_git_files"); + const templates = await stockGitFiles(); + for (const directory of ["hooks", "info"]) { + for (const name of await fs.readdir(path.join(root, directory))) { + const file = `${directory}/${name}`; + const entry = await fs.lstat(path.join(root, file)); + if (!entry.isFile() || !templates.get(file)?.equals(await fs.readFile(path.join(root, file)))) throw new Error("custom_git_metadata"); + } + } + if (!templates.get("description")?.equals(await fs.readFile(path.join(root, "description")))) throw new Error("custom_git_metadata"); + // A custom hook or metadata file inside object/ref storage must not disappear + // merely because Git's object/ref enumeration does not mention it. + async function walk(relative: string): Promise { + signal?.throwIfAborted(); + for (const entry of await fs.readdir(path.join(root, relative), { withFileTypes: true })) { + const name = `${relative}/${entry.name}`; + if (entry.isSymbolicLink() || (!entry.isDirectory() && !entry.isFile())) throw new Error("unsupported_git_storage"); + if (entry.isDirectory()) { await walk(name); continue; } + const valid = /^objects\/[0-9a-f]{2}\/[0-9a-f]{38}$/.test(name) + || /^objects\/pack\/pack-[0-9a-f]{40}\.(pack|idx|rev|bitmap)$/.test(name) + || /^(refs|logs\/refs)\/(heads|remotes\/origin)\//.test(name) + || name === "logs/HEAD"; + if (!valid) throw new Error("unknown_git_storage_file"); + } + } + for (const directory of ["objects", "refs", "logs"]) await walk(directory); +} + +/** Inspect every untracked/ignored file. Only repository-ignored regenerable directories qualify. */ +export async function assertReproducibleGitFiles(cwd: string, signal?: AbortSignal) { + if ((await gitCacheCommand(cwd, ["status", "--porcelain=v1", "--untracked-files=all", "-z"], signal)).length) throw new Error("dirty_or_untracked_files"); + if ((await gitCacheCommand(cwd, ["ls-files", "-v"], signal)).split("\n").some(line => /^[a-zS] /.test(line))) throw new Error("unsupported_index_flags"); + const localConfig = await gitCacheCommand(cwd, ["config", "--local", "--name-only", "--list"], signal); + if (localConfig.split("\n").filter(Boolean).some(key => !/^(core\.(repositoryformatversion|filemode|bare|logallrefupdates|ignorecase|precomposeunicode)|remote\.origin\.(url|fetch)|branch\.[^.]+\.(remote|merge))$/.test(key))) throw new Error("unsupported_local_git_config"); + const regenerable = await findRegenerableWorkspaceDirs(cwd, { signal }); + const ignored = (await gitCacheCommand(cwd, ["ls-files", "--others", "--ignored", "--exclude-standard", "-z"], signal)).split("\0").filter(Boolean); + if (ignored.some(file => !regenerable.some(dir => file.startsWith(`${dir}/`)))) throw new Error("unique_ignored_files"); + const entries = await gitCacheCommand(cwd, ["ls-tree", "-r", "HEAD"], signal); + if (/^160000 /m.test(entries)) throw new Error("unsupported_submodules"); + const gitDir = (await gitCacheCommand(cwd, ["rev-parse", "--absolute-git-dir"], signal)).trim(); + const common = (await gitCacheCommand(cwd, ["rev-parse", "--path-format=absolute", "--git-common-dir"], signal)).trim(); + for (const root of new Set([gitDir, common])) { + for (const name of ["shallow", "shallow.lock", "MERGE_HEAD", "CHERRY_PICK_HEAD", "REVERT_HEAD", "BISECT_LOG", "rebase-apply", "rebase-merge", "sequencer", "info/sparse-checkout", "objects/info/alternates", "refs/replace", "refs/stash", "modules", "lfs"]) { + if (await fs.lstat(path.join(root, name)).then(() => true, (e: NodeJS.ErrnoException) => { if(e.code === "ENOENT") return false; throw e; })) throw new Error("unsupported_local_git_state"); + } + } + const original = await gitCacheCommand(cwd, ["rev-parse", "--verify", "ORIG_HEAD"], signal).catch(() => null); + if (original) await gitCacheCommand(cwd, ["merge-base", "--is-ancestor", original.trim(), "HEAD"], signal) + .catch(() => { throw new Error("unique_original_head"); }); + if (gitDir !== common) { + const allowed = new Set(["HEAD", "ORIG_HEAD", "index", "commondir", "gitdir", "logs", "refs", "COMMIT_EDITMSG", "index.lock", "HEAD.lock"]); + if ((await fs.readdir(gitDir)).some(name => !allowed.has(name))) throw new Error("unknown_worktree_git_files"); + async function assertEmptyRefs(root: string): Promise { + const entries = await fs.readdir(root, { withFileTypes: true }).catch((error: NodeJS.ErrnoException) => { + if (error.code === "ENOENT") return []; throw error; + }); + for (const entry of entries) { + if (!entry.isDirectory()) throw new Error("unique_worktree_refs"); + await assertEmptyRefs(path.join(root, entry.name)); + } + } + await assertEmptyRefs(path.join(gitDir, "refs")); + const message = await fs.readFile(path.join(gitDir, "COMMIT_EDITMSG"), "utf8").catch((error: NodeJS.ErrnoException) => { + if (error.code === "ENOENT") return null; throw error; + }); + if (message !== null && message.trim() !== (await gitCacheCommand(cwd, ["log", "-1", "--format=%B"], signal)).trim()) throw new Error("uncommitted_git_message"); + const logs = await fs.readdir(path.join(gitDir, "logs")).catch((error: NodeJS.ErrnoException) => { + if (error.code === "ENOENT") return []; throw error; + }); + if (logs.some(name => name !== "HEAD")) throw new Error("unknown_worktree_git_logs"); + } + // Packed stashes/replacement refs are just as important as loose refs. + if ((await gitCacheCommand(cwd, ["for-each-ref", "--format=%(refname)", "refs/stash", "refs/replace"], signal)).trim()) throw new Error("unique_git_refs"); + // Rewrites can leave commits reachable only from this worktree's reflog. + for (const commit of new Set((await gitCacheCommand(cwd, ["reflog", "show", "--format=%H", "HEAD"], signal)).split("\n").filter(Boolean))) { + await gitCacheCommand(cwd, ["merge-base", "--is-ancestor", commit, "HEAD"], signal).catch(() => { throw new Error("unique_reflog_commits"); }); + } + return regenerable; +} + +export async function inspectGitCacheRecipe(workspace: Workspace, resolveGitAuth: GitRemoteAuthProvider, signal?: AbortSignal): Promise { + if (workspace.mode !== "isolated_workspace" || workspace.strategyType !== "git_worktree" + || !isRuntimeOwnedGitBranch(workspace.metadata) || !workspace.sourceIssueId || !workspace.repoUrl + || !workspace.cwd || !workspace.branchName || (workspace.providerRef && workspace.providerRef !== workspace.cwd)) throw new Error("unproven_workspace_ownership"); + let source = await verifiedRecordedIsolatedRemoteSource(workspace); + if (!source) { + const common = (await gitCacheCommand(workspace.cwd, ["rev-parse", "--path-format=absolute", "--git-common-dir"], signal)).trim(); + for (const candidate of managedSourcePaths(workspace)) { + if (await fs.realpath(path.join(candidate, ".git")).catch(() => null) === common) { + source = await verifiedGitCacheSource(workspace, candidate); break; + } + } + } + if (!source) throw new Error("unproven_source_ownership"); + const stat = await fs.lstat(workspace.cwd); + if (!stat.isDirectory() || stat.isSymbolicLink() || await fs.realpath(workspace.cwd) !== path.join(await fs.realpath(path.dirname(workspace.cwd)), path.basename(workspace.cwd))) throw new Error("unsafe_worktree_path"); + if (!(await fs.lstat(path.join(workspace.cwd, ".git"))).isFile()) throw new Error("not_linked_worktree"); + if ((await gitCacheCommand(workspace.cwd, ["rev-parse", "--path-format=absolute", "--git-common-dir"], signal)).trim() !== path.join(await fs.realpath(source), ".git")) throw new Error("source_identity_changed"); + if ((await gitCacheCommand(workspace.cwd, ["symbolic-ref", "--short", "HEAD"], signal)).trim() !== workspace.branchName) throw new Error("branch_identity_changed"); + await assertReproducibleGitFiles(workspace.cwd, signal); + const commit = (await gitCacheCommand(workspace.cwd, ["rev-parse", "HEAD"], signal)).trim(); + await verifyRemoteGitCommit(workspace.repoUrl, commit, resolveGitAuth, signal); + return { schema: "paperclip.git-cache-restoration.v1", companyId: workspace.companyId, workspaceId: workspace.id, + sourceIssueId: workspace.sourceIssueId, repoUrl: workspace.repoUrl, commit, branchName: workspace.branchName, + sourceKind: recordedIsolatedRemoteSource(workspace) ? "isolated_remote" : "managed_project", projectId: workspace.projectId, + sourcePath: source, worktreePath: workspace.cwd, gitBranchOwnershipVersion: 1, verifiedAt: new Date().toISOString() }; +} + +export function readGitCacheExpiration(workspace: Workspace): GitCacheExpiration | null { + const state = workspace.metadata?.gitCacheExpiration as GitCacheExpiration | undefined; + if (!state) return null; + const recipe = readGitCacheRestoration(workspace); + if (!recipe || state.schema !== "paperclip.git-cache-expiration.v1" || !Number.isSafeInteger(state.generation) + || state.generation !== workspace.metadata?.lifecycleGeneration + || state.worktreeTrash !== path.join(path.dirname(recipe.worktreePath), `.expired-git-worktree-${workspace.id}-${state.generation}`) + || path.dirname(state.worktreeAdmin) !== path.join(recipe.sourcePath, ".git", "worktrees") + || (state.baseTrash !== null && state.baseTrash !== path.join(path.dirname(recipe.sourcePath), `.expired-git-base-${workspace.id}-${state.generation}`))) { + throw new GitCacheRestorationFailure(); + } + return state; +} + +// Both restart recovery and reopening finish only recorded, uniquely named trash. +// No directory scan or age heuristic can authorize deleting an arbitrary path. +export async function finishGitCacheTrash(workspace: Workspace) { + const state = readGitCacheExpiration(workspace); + if (!state) return; + const recipe = readGitCacheRestoration(workspace)!; + const exists = (target: string) => fs.lstat(target).then(() => true, (error: NodeJS.ErrnoException) => { + if (error.code === "ENOENT") return false; throw error; + }); + for (const target of [state.worktreeTrash, state.baseTrash]) { + if (!target || !await exists(target)) continue; + const entry = await fs.lstat(target); + if (!entry.isDirectory() || entry.isSymbolicLink() + || await fs.realpath(target) !== path.join(await fs.realpath(path.dirname(target)), path.basename(target))) throw new GitCacheRestorationFailure(); + await fs.rm(target, { recursive: true, force: true }); + } + if (!await exists(recipe.worktreePath) && await exists(state.worktreeAdmin)) { + await verifiedGitCacheSource(workspace, recipe.sourcePath); + const entry = await fs.lstat(state.worktreeAdmin); + if (!entry.isDirectory() || entry.isSymbolicLink() + || await fs.realpath(state.worktreeAdmin) !== path.join(await fs.realpath(recipe.sourcePath), ".git", "worktrees", path.basename(state.worktreeAdmin))) { + throw new GitCacheRestorationFailure(); + } + const pointer = (await fs.readFile(path.join(state.worktreeAdmin, "gitdir"), "utf8")).trim(); + const expected = path.join(await fs.realpath(path.dirname(recipe.worktreePath)), path.basename(recipe.worktreePath), ".git"); + if (pointer !== expected) throw new GitCacheRestorationFailure(); + await fs.rm(state.worktreeAdmin, { recursive: true, force: true }); + } +} + +/** Called only after authorization and while holding the source/lifecycle fences. */ +export async function restoreGitCacheBase(workspace: Workspace, recipe: GitCacheRestoration, resolveGitAuth: GitRemoteAuthProvider) { + if (readGitCacheRestoration(workspace)?.commit !== recipe.commit) throw new GitCacheRestorationFailure(); + try { + await finishGitCacheTrash(workspace); + const present = await fs.lstat(recipe.sourcePath).then(() => true, (error: NodeJS.ErrnoException) => { + if (error.code === "ENOENT") return false; + throw error; + }); + if (!present && recipe.sourceKind === "managed_project") { + const root = resolvePaperclipInstanceRoot(); + let parent = await fs.realpath(root); + for (const segment of path.relative(root, path.dirname(recipe.sourcePath)).split(path.sep)) { + parent = path.join(parent, segment); + await fs.mkdir(parent, { mode: 0o700 }).catch((error: NodeJS.ErrnoException) => { if (error.code !== "EEXIST") throw error; }); + const entry = await fs.lstat(parent); + if (!entry.isDirectory() || entry.isSymbolicLink() || await fs.realpath(parent) !== parent) throw new GitCacheRestorationFailure(); + } + } + const cwd = present ? await verifiedGitCacheSource(workspace, recipe.sourcePath) + : recipe.sourceKind === "isolated_remote" + ? await materializeIsolatedRemoteWorkspaceSource({ companyId: recipe.companyId, issueId: recipe.sourceIssueId, + repoUrl: recipe.repoUrl, resolveGitAuth }) + : (await materializeManagedProjectWorkspace(recipe.sourcePath, { repoUrl: recipe.repoUrl, resolveGitAuth })).cwd; + await verifiedGitCacheSource(workspace, recipe.sourcePath); + if (!cwd) throw new GitCacheRestorationFailure(); + await ensureSavedGitCommit(cwd, recipe.repoUrl, recipe.commit, resolveGitAuth); + return cwd; + } catch { throw new GitCacheRestorationFailure(); } +} + +export async function ensureSavedGitCommit(cwd: string, repoUrl: string, commit: string, resolveGitAuth: (url: string) => Promise<{ configArgs: string[]; env: Record } | null>) { + if (!shaPattern.test(commit)) throw new GitCacheRestorationFailure(); + if (await gitCacheCommand(cwd, ["cat-file", "-e", `${commit}^{commit}`]).then(() => true, () => false)) return; + try { + const auth = await resolveGitAuth(repoUrl); + await execFile("git", [...auth?.configArgs ?? [], "-C", cwd, "fetch", "--no-tags", "--no-recurse-submodules", "--", repoUrl, commit], { + timeout: 60_000, maxBuffer: 1024 * 1024, env: { ...process.env, ...auth?.env, GIT_TERMINAL_PROMPT: "0" }, + }); + if ((await gitCacheCommand(cwd, ["rev-parse", `${commit}^{commit}`])).trim() !== commit) throw new GitCacheRestorationFailure(); + } catch { throw new GitCacheRestorationFailure(); } +} + +/** Runtime callers must bind the recipe to the selected workspace and base. */ +export function savedGitWorkspaceRestoration(input: { metadata?: Record | null; companyId: string; + workspaceId?: string | null; cwd: string; baseCwd: string; repoUrl: string | null; branchName: string | null }) { + const raw = input.metadata?.[GIT_CACHE_RESTORATION_KEY]; + if (raw === undefined) return null; + const value = raw as GitCacheRestoration | null; + if (!value || value.schema !== "paperclip.git-cache-restoration.v1" || !shaPattern.test(value.commit) + || value.companyId !== input.companyId || value.workspaceId !== input.workspaceId + || value.worktreePath !== input.cwd || value.sourcePath !== input.baseCwd + || value.repoUrl !== input.repoUrl || value.branchName !== input.branchName + || !isRuntimeOwnedGitBranch(input.metadata) + || (value.restoredAt !== undefined && !Number.isFinite(Date.parse(value.restoredAt)))) throw new GitCacheRestorationFailure(); + return { commit: value.commit, restored: value.restoredAt !== undefined }; +} diff --git a/server/src/services/heartbeat/workspace-preparation.ts b/server/src/services/heartbeat/workspace-preparation.ts index 92cb041602a..991398f1d4c 100644 --- a/server/src/services/heartbeat/workspace-preparation.ts +++ b/server/src/services/heartbeat/workspace-preparation.ts @@ -30,7 +30,7 @@ import { readNativeWorkspaceSyncReference, readNativeWorkspaceSyncLocalCwd } fro import { parseObject } from "../../adapters/utils.js"; import { materializeNativeChatTaskRoot } from "../native-runtime/native-chat-workspace.js"; import { materializeIsolatedTaskDirectory } from "../isolated-task-directory.js"; -import { canIsolateRemoteWorkspaceSource, materializeIsolatedRemoteWorkspaceSource } from "../isolated-remote-workspace-source.js"; +import { canIsolateRemoteWorkspaceSource, materializeIsolatedRemoteWorkspaceSource, withIsolatedRepositorySourceLock } from "../isolated-remote-workspace-source.js"; import { assertIsolatedWorkspaceSourceReadAccess, retainIsolatedWorkspaceSourceAccess, @@ -149,6 +149,14 @@ function readNonEmptyString(value: unknown): string | null { } export async function prepareHeartbeatWorkspace(db: Db, input: HeartbeatWorkspacePreparationInput) { + const sourceIssueId = input.reuse.reusableExistingExecutionWorkspace?.sourceIssueId + ?? input.reuse.existingExecutionWorkspace?.sourceIssueId ?? input.task.issueId; + if (!sourceIssueId || input.reuse.isDotRun) return prepareHeartbeatWorkspaceLocked(db, input); + return withIsolatedRepositorySourceLock(db, input.agent.companyId, sourceIssueId, + () => prepareHeartbeatWorkspaceLocked(db, input)); +} + +async function prepareHeartbeatWorkspaceLocked(db: Db, input: HeartbeatWorkspacePreparationInput) { const { run, agent, controllerLease, nativeRunnerPreparationSpans } = input; const timeWorkspace = createHeartbeatPreparationTiming(run.id); const { diff --git a/server/src/services/isolated-remote-workspace-source.ts b/server/src/services/isolated-remote-workspace-source.ts index efb299db2e1..02357b3ba35 100644 --- a/server/src/services/isolated-remote-workspace-source.ts +++ b/server/src/services/isolated-remote-workspace-source.ts @@ -1,3 +1,6 @@ +import { AsyncLocalStorage } from "node:async_hooks"; +import { withDedicatedDbConnection, type Db } from "@paperclipai/db"; +import { sql } from "drizzle-orm"; import path from "node:path"; import fs from "node:fs/promises"; import { execFile as execFileCallback } from "node:child_process"; @@ -10,6 +13,25 @@ import { measureRunPhase } from "./execution-preparation-timing.js"; const execFile = promisify(execFileCallback); +const sourceLocks = new AsyncLocalStorage>(); +export class IsolatedRepositorySourceBusy extends Error {} + +/** Creation, reopening and base expiration share this cross-process exclusion. */ +export async function withIsolatedRepositorySourceLock(db: Db, companyId: string, issueId: string, + action: () => Promise, tryOnly = false): Promise { + const key = `isolated_repository_source:${companyId}:${issueId}`; + const held = sourceLocks.getStore() ?? []; + if (held.some(lock => lock.db === db && lock.key === key)) return action(); + return withDedicatedDbConnection(db, connection => connection.transaction(async tx => { + if (tryOnly) { + const rows = await tx.execute<{ acquired: boolean }>(sql`select pg_try_advisory_xact_lock(hashtextextended(${key}, 0)) as acquired`); + if (!rows[0]?.acquired) throw new IsolatedRepositorySourceBusy(); + } else await tx.execute(sql`select pg_advisory_xact_lock(hashtextextended(${key}, 0))`); + return sourceLocks.run([...held, { db, key }], action); + })); +} + + export function isolatedRemoteSourcePath(companyId: string, issueId: string): string { if ([companyId, issueId].some(id => !/^[a-zA-Z0-9_-]+$/.test(id))) { throw new Error("Invalid isolated repository source identity"); diff --git a/server/src/services/workspace-runtime.ts b/server/src/services/workspace-runtime.ts index d497fd1feeb..97b393f8424 100644 --- a/server/src/services/workspace-runtime.ts +++ b/server/src/services/workspace-runtime.ts @@ -1,3 +1,4 @@ +import { savedGitWorkspaceRestoration, ensureSavedGitCommit, GitCacheRestorationFailure } from "./git-workspace-restoration.js"; import { sanitizeRuntimeServiceBaseEnv } from "./runtime-service-env.js"; import { verifiedRecordedIsolatedRemoteSource } from "./isolated-remote-workspace-source.js"; export { sanitizeRuntimeServiceBaseEnv } from "./runtime-service-env.js"; @@ -3848,6 +3849,10 @@ export async function ensurePersistedExecutionWorkspaceAvailable(input: { "Cannot rebuild the git worktree: the base project checkout directory does not exist.", ); } + const restoration = savedGitWorkspaceRestoration({ metadata: input.workspace.metadata, companyId: input.agent.companyId, + workspaceId: input.workspace.id, cwd, baseCwd, repoUrl: input.workspace.repoUrl ?? input.base.repoUrl, + branchName: input.workspace.branchName ?? null }); + const savedCommit = restoration?.commit ?? null; const repoRoot = await runGit(["rev-parse", "--show-toplevel"], baseCwd); await input.beforeSourceCopy?.(repoRoot, realized.worktreePath ?? cwd); const recordedBaseRefSha = readRecordedBaseRefSha(input.workspace.metadata); @@ -3908,7 +3913,7 @@ export async function ensurePersistedExecutionWorkspaceAvailable(input: { const currentBaseRefSha = reuseBaseRef ? await resolveBaseRefSha(repoRoot, reuseBaseRef) : null; // An unstarted-worktree refresh can fast-forward the checked-out branch. // Never run it for an attached operator-owned ref. - const refresh = realized.branchCreatedByRuntime && reuseBaseRef && currentBaseRefSha + const refresh = !savedCommit && realized.branchCreatedByRuntime && reuseBaseRef && currentBaseRefSha ? await refreshUnstartedWorktreeToBase({ repoRoot, worktreePath: reuseWorktreePath, @@ -3955,7 +3960,14 @@ export async function ensurePersistedExecutionWorkspaceAvailable(input: { await fs.mkdir(path.dirname(worktreePath), { recursive: true }); await runGit(["worktree", "prune"], repoRoot).catch(() => {}); - const restoreBaseRef = input.workspace.baseRef ?? input.base.repoRef ?? null; + if (savedCommit) { + await ensureSavedGitCommit(repoRoot, input.workspace.repoUrl ?? input.base.repoUrl!, savedCommit, input.resolveGitAuth ?? (async () => null)); + const existingTip = await resolveBaseRefSha(repoRoot, `refs/heads/${branchName}`); + // Once restored, subsequent task commits own this branch. The saved commit + // still prevents a clean resumed worktree from refreshing to upstream HEAD. + if (existingTip && existingTip !== savedCommit && !restoration?.restored) throw new GitCacheRestorationFailure(); + } + const restoreBaseRef = savedCommit ? null : input.workspace.baseRef ?? input.base.repoRef ?? null; const restoreRefreshWarnings = restoreBaseRef ? await refreshRemoteTrackingBaseRef(repoRoot, restoreBaseRef, input.resolveGitAuth) : []; @@ -3992,7 +4004,7 @@ export async function ensurePersistedExecutionWorkspaceAvailable(input: { `Execution workspace "${worktreePath}" cannot be restored because its operator-owned branch "${branchName}" no longer exists.`, ); } - const baseRef = input.workspace.baseRef ?? await detectDefaultBranch(repoRoot) ?? "HEAD"; + const baseRef = savedCommit ?? input.workspace.baseRef ?? await detectDefaultBranch(repoRoot) ?? "HEAD"; const recreatedBaseRefSha = await resolveBaseRefSha(repoRoot, baseRef); await recordGitOperation(input.recorder, { phase: "worktree_prepare", From db11d899cf68422102f328a669ab771a0eb4cce1 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Thu, 15 Oct 2026 13:26:09 -0500 Subject: [PATCH 17/19] fix(native): recover remote sessions while preserving warm runs (#16451) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native agents keep remote processes and conversation history across warm turns. > - A completed turn can leave handoff authority only in controller memory. > - Losing that controller can block a follow-up even while its sandbox survives. > - This pull request verifies durable ownership and reconnects the exact prior process before handing it to the next run. > - Healthy follow-ups stay warm. Recovery preserves saved work without replaying a completed turn. ## Linked Issues or Issue Description **What happened?** A remote warm turn can finish without suspended backup authority. A replacement controller rejects that prior owner even when its exact process and completed checkpoint remain. A single unavailable identity probe can also stop an otherwise valid reattachment. **Expected behavior** Keep healthy follow-ups warm. Recover a completed session after proving that its controller stopped and that the successor holds current authority. Reuse compatible processes; otherwise save and resume the conversation. Retry unavailable identity reads within one shared deadline. **Steps to reproduce** Use `doc/testing-native-remote-continuity.md`. Its Linux cases execute real turns through PostgreSQL, TLS ingress, and the Rust runner. They kill a separate controller process, recover old checkpoint metadata, and change process configuration. The provider is deterministic. Related: #16432 and #16441 fix live warm-owner handoffs. #16341 adds recovery diagnostics. #14695 covers provider process retention. ## What Changed - Preserve warm process reuse between healthy remote turns. - Verify a committed prior result, stopped controller, matching checkpoint and sandbox, and current task/coordinator ownership before recovering an orphaned warm session. - Reconnect through the prior run route, then use the existing authenticated run attachment to change authority. - Persist a process-configuration digest. Close and save recovered sessions when old or changed configuration cannot support reuse. - Retry unavailable state, marker, and process reads within one 30-second deadline. Keep signals and definitive identity failures single-attempt. - Record closed recovery diagnostics, including unreadable or malformed saved state. - Retire the recovery-only ownership check after verified attachment, so later warm turns use their current authority. - Add real-process Linux regressions and refusal controls. - Expose the existing source-only file-recovery form for retained native holds, with the same board authority, evidence, and exact stop-receipt requirements as legacy holds. Recording recovered files does not change task status or replay a run. ## Verification - `pnpm -r typecheck` passed; the server check was repeated for the third-turn fix. - `pnpm build` passed after the final UI change. The UI typecheck, token gates, and map checks also pass. - Four Linux continuity cases passed: warm reuse, controller loss, legacy checkpoint metadata, and changed configuration. Each creates one conversation and starts exactly three turns, including a healthy follow-up after recovery. The fixture uses expired prior leases; the third turn keeps the runner and provider processes. - Greptile caught the retained recovery gate; the third-turn regression now covers it. An initial test omitted the heartbeat’s saved input and left a test process after failure; the fixture was corrected and its disposable container reset before the passing run. - The current focused executor/restart group passes all 749 tests, including the original failure classifications. - Native file-recovery and UI tests pass (52 tests), including blocked tasks and both native and legacy holds. - After merging current master and retaining its equivalent CI expression check, the CI policy tests pass (15 tests) and all four three-turn Linux continuity cases pass again on `dec418ae55ff21b4fb067132613658cce445f181`. - `pnpm test:run` completed its general server group with 20,572 passes, 103 skips, and three failures: the skill-audit child process and two HTTP test timeouts. The same three suites then passed in isolation (89 tests) without code changes. That local command stopped before later groups; the complete final-head CI, including general, serialized, Runner, build, and browser gates, is green. - Greptile reviewed the combined head `dec418ae55ff21b4fb067132613658cce445f181` at 5/5 with no actionable findings; both earlier findings are resolved. All final-head CI gates and the security scan passed. Staging deployment and retained-incident disposition remain deployment acceptance work. ## Risks - Recovery requires exact durable evidence. Live conflicting controllers, replacement sandboxes, Stop/pause, and uncertain effects must remain blocked. - After controller loss, tool-contract or process-configuration changes can restart the provider. Normal healthy follow-ups retain their processes. - The new idle-recovery path has Linux fake-provider evidence. Live Codex/OpenCode and retained quarantined-state recovery still need staging qualification. - No database schema, telemetry, or runner protocol changes. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository inspection, code execution, and browser tools. The session does not expose a more specific model version or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (targeted checks and isolated reruns; full-run limitation recorded above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green on `dec418ae55ff21b4fb067132613658cce445f181` - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/execution-semantics.md | 12 + .../2026-10-15-native-runner-continuity.md | 122 ++++++++++ doc/run-log-events.md | 8 + doc/testing-native-remote-continuity.md | 67 ++++++ maps/agent-access-map/recovery.md | 2 +- maps/feature-map/recovery.md | 2 +- .../helpers/docker-command-runner.ts | 52 ++++ .../remote-warm-controller.test-worker.ts | 36 +++ .../native-restart-recovery.test.ts | 7 + .../native-runtime/native-restart-recovery.ts | 7 +- ...unner-restart-recovery.integration.test.ts | 225 +++++++++++++++++- .../native-session-executor.test.ts | 78 +++++- .../native-runtime/native-session-executor.ts | 218 ++++++++++++----- .../remote-runner-recovery-diagnostics.ts | 5 +- .../remote-runner-recovery-probe.test.ts | 44 ++++ .../remote-runner-recovery-probe.ts | 39 +++ .../remote-warm-session-recovery.ts | 109 +++++++++ .../components/WorkspaceFileRecovery.test.tsx | 7 +- ui/src/components/WorkspaceFileRecovery.tsx | 3 +- 19 files changed, 971 insertions(+), 72 deletions(-) create mode 100644 doc/plans/2026-10-15-native-runner-continuity.md create mode 100644 doc/testing-native-remote-continuity.md create mode 100644 server/src/__tests__/helpers/docker-command-runner.ts create mode 100644 server/src/__tests__/helpers/remote-warm-controller.test-worker.ts create mode 100644 server/src/services/native-runtime/remote-runner-recovery-probe.test.ts create mode 100644 server/src/services/native-runtime/remote-runner-recovery-probe.ts create mode 100644 server/src/services/native-runtime/remote-warm-session-recovery.ts diff --git a/doc/execution-semantics.md b/doc/execution-semantics.md index fe9ee0c9615..bbaf6f8eeda 100644 --- a/doc/execution-semantics.md +++ b/doc/execution-semantics.md @@ -1521,6 +1521,18 @@ size. Remote checkpoint archive and expanded-size limits remain 64 MiB. ### Warm sandbox continuity +Healthy native warm turns retain their runner and provider processes. After a +controller crash, continuation can recover a completed remote session from its +saved checkpoint and exact sandbox process identity. The prior controller must +be stopped, its result committed, and the successor must hold current task and +coordinator authority. Recovery reconnects through the prior run's route before +the authenticated run attachment changes authority. Unavailable identity probes +have a shared 30-second retry budget; an identity conflict never permits replay. +Saved process configuration must match before reuse. Old checkpoints, changed +configuration, and controller-owned credential or instruction resources use +verified suspension and saved-history resume. This recovery can restart a +provider; it does not suspend every healthy warm turn. + For the native runner, warm mode requests a reusable sandbox lease **before** lease acquisition. The environment's explicit runner lifecycle overrides the agent default; an inherited lifecycle uses the current agent setting on each diff --git a/doc/plans/2026-10-15-native-runner-continuity.md b/doc/plans/2026-10-15-native-runner-continuity.md new file mode 100644 index 00000000000..e5bcac355d7 --- /dev/null +++ b/doc/plans/2026-10-15-native-runner-continuity.md @@ -0,0 +1,122 @@ +# Native runner continuity through controller and process loss + +## Goal and delivery boundary + +Ship a verified fix to the selected staging instance and recover the two reported +tasks from their retained work. Normal follow-up messages, server restarts, and +recoverable runner loss must not strand a task behind +`native_continuation_requires_reconciliation` or +`runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate`. +Removing an error label, clearing a hold without evidence, or blindly starting +another provider does not meet this goal. + +The user authorized implementation, review, merge, staging deployment, and task +recovery. The supported release covers the affected native Codex and native +OpenCode remote execution paths, plus shared lifecycle regression controls. +It preserves existing company, ownership, Stop, pause, approval, budget, and +workspace boundaries. Genuine conflicting ownership or an unresolved external +effect must remain safe and actionable. This is not a promise that infrastructure +can never fail; it is a checkable promise that the specified failures do not lose +work or require users to diagnose runner internals. + +Warm process reuse is a firm requirement. The user superseded the earlier +save-and-suspend choice: healthy follow-up turns must keep the existing provider +process. Durable ownership and recovery must coexist with that behavior. The +suspend-every-turn candidate is withdrawn from delivery; PR #16451 is draft and +has not been merged or deployed. + +## Acceptance matrix + +| Journey or fault | Required result | Evidence | +| --- | --- | --- | +| Completed turn followed immediately by another user message | Next turn starts once with the saved history, workspace, and message | Real assembled-service regression and staging journey | +| Healthy remote warm follow-up | Same runner and provider process, with a verified handoff to the new run | Process-start identity and call-log assertions across real turns | +| Controller restart while a session is idle, including immediately after completion | Durable evidence suffices to resume or safely replace the runner; no process-local ownership dependency strands the task | Hard and graceful restart fault tests | +| Restart during startup or an active turn | Adopt the exact live execution or fence it before one bounded successor; preserve accepted results and pending messages | Real process/transport boundary fault tests | +| Remote process exits or identity cannot initially be read | Distinguish a transient probe failure, a dead execution, and a conflicting owner; reconcile automatically when evidence permits | Linux remote-process tests and persisted recovery timeline | +| Crash during checkpoint, recovery, or workspace copy-back | Resume the persistence obligation without replaying completed provider work; retain the sole recoverable source until saved | Restart/concurrency regression with independently checked file contents | +| Existing saved or quarantined state from the affected release | Recover eligible old state without requiring a newly created task or discarding its history | Compatibility fixture and recovery of both reported tasks | +| Duplicate recovery workers, lost acknowledgements, and newer user messages | At most one successor and one delivery of each message; no replay of a completed external action | Durable ledger and side-effect counter assertions | +| Wrong company/session, a genuinely live competing owner, Stop/pause, revoked access, or unknown external effect | No unsafe adoption, signalling, destruction, or speculative action replay; concrete remaining blocker | Negative controls | +| Deployment | Running staging application and any changed runner artifact match the tested revisions; both tasks have useful results or their genuine task-level disposition | Live revision, run, file, and task inspection | + +Use the existing recovery budgets (three provider attempts including the original, +30-second failure retry interval) and control-transition contract (60-second +deadline, 15-second reconciliation, resolution or explicit blocker within 90 seconds +with healthy dependencies). Provider work duration is not bounded by this timer. +Restarts must not reset budgets. Any necessary departure requires explicit evidence +and an updated contract, not a silent relaxation of acceptance. + +## Investigation and implementation slices + +1. Establish both baseline failure paths from persisted evidence. The remote + identity failure still needs its exact failing operation established; do not + infer process death from the generic error alone. +2. Close the completed-turn ownership/checkpoint gap in its existing lifecycle + owner, including compatibility with retained old state. Test prevention and + recovery, not only an isolated predicate. +3. Close the demonstrated remote recovery gap using existing fencing, finalization, + workspace retention, and continuation services. Keep diagnostics precise enough + to distinguish probe unavailability from conflicting identity. +4. Qualify the changed boundary locally first, then on a representative remote + Linux target and staging. Prepare reviewable changes and required repository + checks; deploy the verified revision and recover the original tasks. + +The first discriminating experiment is a completed native remote turn whose +controller is lost before a follow-up. Recreate the old persisted state through +the real lifecycle, then attempt the follow-up with an empty process-local owner +map. Preserve real persistence, transport, and process boundaries where mocks +would hide the failure. Existing recovery E2E and native restart suites are the +starting harnesses; past passing evidence does not qualify this change. + +## Current state + +- Branch: `codex/native-runner-recovery`, based on `afe5b3418d`. +- PR: https://github.com/paperclipai/paperclip/pull/16451 (draft, unmerged). +- Normal warm turns retain their processes. The previous suspend-every-turn + implementation is withdrawn. Earlier suspension tests are historical only. +- The base includes #16432 (idle expiry serialized with admission) and #16441 + (exact warm-owner identity independent of configuration changes). +- The revised executor can recover a completed remote session after its + controller dies. It checks the committed prior result, frozen scope, saved + checkpoint, exact remote identity, and current task/coordinator lease. It + reconnects through the prior route before authenticated run attachment. +- A saved process-configuration digest controls reuse after controller loss. + Legacy metadata or changed configuration uses verified close and checkpoint + followed by saved-history resume. Healthy warm follow-ups stay open. +- Four Linux container/TLS/PostgreSQL cases pass: ordinary warm reuse, actual + controller SIGKILL, older checkpoint metadata, and changed environment. + They assert one thread creation and exactly three turn starts, including a + healthy follow-up after recovery. The hard-restart + cases also reject a still-live prior controller and a replacement sandbox. + Normal follow-ups use one provider process; recovery may refresh the provider. + These are real runner/process tests with a deterministic fake Codex, not live + Daytona or live model evidence. The prior lease fixture uses the expired status + observed on staging. The third turn retains the runner and provider processes. +- Full workspace typecheck and production build pass on the warm candidate. + The server and UI typechecks and production build also pass after the final + changes. All 749 focused executor/restart tests pass; the full required test + suite is running. +- Greptile found a stale recovery ownership check retained across later warm + turns. Recovery admission now retires after verified attachment; the three-turn + regression passes. The earlier saved-state diagnostic finding is also fixed. + The third-turn fix received 5/5. Fresh review of the final changes is pending. +- The existing source-only file-recovery form now supports retained native holds + as well as legacy holds. It preserves board authority, exact stop-receipt + validation, task status, and the prohibition on replaying the original run. + All 52 UI/native recovery tests, token gates, and map checks pass. +- Fresh read-only staging records confirm both original tasks remain blocked. + The Codex failure preceded its coordinator claim. The OpenCode failure followed + controller takeover, with an active turn in the saved checkpoint. Its original + log lacks the failed probe reason; process death is not established. +- Both retained sandboxes were restored for file inspection. The completed + website commit is intact; the other workspace retains a benchmark script and + lockfile changes. Backup and task disposition remain in progress. No agent + turn, task status, or deployment has been changed. +- The selected tenant has an existing fleet rollout deferred by active work. + Do not override it. Remaining: full tests and review, staging qualification, + retained-state recovery, and useful disposition of both tasks. + +Deferred: unrelated skill-import setup failures, database query optimization, +general UI redesign, and broad provider qualification unrelated to the changed +shared lifecycle. diff --git a/doc/run-log-events.md b/doc/run-log-events.md index 47c8a4d16ab..485563e2b3e 100644 --- a/doc/run-log-events.md +++ b/doc/run-log-events.md @@ -165,6 +165,14 @@ proofs, encryption keys, environment variables, provider credentials, command arguments, or an unsanitized stderr stream. Detailed failed-attempt diagnostics remain in the bounded `native_run_finalizations.recovery_history` ledger. +`native.remote_runner_recovery_failed` records the closed `stage` and `reason` +when remote identity verification fails, including admission failures before the +execution coordinator opens. Unavailable probes are retried within a shared +30-second deadline; identity conflicts are not retried. The event distinguishes +an unreadable identity from a confirmed missing process and contains no command +output or credentials. Saved-state read failures use `state_read_failed`; malformed +saved JSON uses `state_invalid`. It does not itself authorize replacement or cleanup. + For a prepared remote workspace interrupted before its first executor claim, that ledger records `remote_execution_never_claimed` with a `bootstrap_incomplete` disposition. The provider attempt remains zero during diff --git a/doc/testing-native-remote-continuity.md b/doc/testing-native-remote-continuity.md new file mode 100644 index 00000000000..434b179e149 --- /dev/null +++ b/doc/testing-native-remote-continuity.md @@ -0,0 +1,67 @@ +# Native remote continuity regression + +The opt-in case in +`server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts` +runs the real executor, PostgreSQL, TLS ingress, Rust runner, remote file transfer, +and backup validation against a deterministic fake Codex process in Linux. It +checks the same runner process identity and one provider process across three warm +turns, provider history reuse, and verified suspension at idle close. Separate +cases kill the actual controller process and recover the next turn. They reject +a still-live controller and a replacement sandbox before reconnecting. Legacy +checkpoints and changed process configuration require verified suspension and +replacement, while preserving the conversation and starting each turn once. +Recovery can restart a provider to refresh its tool contract; process reuse is +required for ordinary healthy follow-ups. +Each recovery case includes another healthy turn after the recovered turn and +asserts that it does not start another runner or provider process. +It does not qualify a live model or a sandbox provider API. + +Use a disposable Linux container with Node.js, `sh`, `tar`, `base64`, `ps`, and +`setsid`. Build `paperclip-runnerd` and `fake-codex-app-server` from the checkout's +`packages/paperclip-runner/runner/Cargo.toml` for that container's architecture. +Keep the Linux runner binary on the host as well: the product hashes and stages +that artifact. Put the fake provider at `/opt/continuity/fake-codex-app-server` +inside the container. Do not put credentials in this container. + +Start the container with `--init` and publish container port 43127 on localhost. +Read its assigned port with `docker port CONTAINER 43127`; an automatic host port +can change after a container restart. The test opens a local TLS tunnel to this +port. Create a test-only certificate and key: + +```sh +openssl req -x509 -newkey rsa:2048 -nodes \ + -keyout /tmp/continuity-test.key -out /tmp/continuity-test.crt \ + -days 1 -subj '/CN=localhost' \ + -addext 'subjectAltName=IP:127.0.0.1,DNS:localhost' +``` + +Create an executable wrapper on the host at `/tmp/continuity-fake-codex`: + +```sh +#!/bin/sh +if [ "$1" = '--version' ]; then + echo 'codex-cli 0.999.0' + exit 0 +fi +exec /opt/continuity/fake-codex-app-server \ + --state-file-in-codex-home --durable-turn-ids --record-process-start \ + --call-log "$PWD/provider-calls.log" "$@" +``` + +Run the test with the container name, current host port, and Linux artifact path: + +```sh +NODE_EXTRA_CA_CERTS=/tmp/continuity-test.crt \ +PAPERCLIP_TEST_RUNNER_TLS_KEY=/tmp/continuity-test.key \ +PAPERCLIP_TEST_RUNNER_CONTAINER=CONTAINER \ +PAPERCLIP_TEST_RUNNER_PORT=HOST_PORT \ +PAPERCLIP_TEST_LINUX_RUNNER_BINARY=/path/to/linux/paperclip-runnerd \ +PAPERCLIP_TEST_FAKE_CODEX_WRAPPER=/tmp/continuity-fake-codex \ +pnpm exec vitest run \ + server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts \ + -t 'preserves remote warm continuity' --maxWorkers=1 +``` + +The caller owns the disposable container. Remove it after the run, including +after failure; failed cases can retain remote processes and files for inspection. +The test cleans up its database and host session directories. diff --git a/maps/agent-access-map/recovery.md b/maps/agent-access-map/recovery.md index ce2fd3161cf..016f34aef44 100644 --- a/maps/agent-access-map/recovery.md +++ b/maps/agent-access-map/recovery.md @@ -108,7 +108,7 @@ retry and some operator controls require board. Completion or cancellation of a task does not clear its exact unresolved native workspace owner hold. Physical owner release still requires the [operator stop-verification procedure](../../doc/native-workspace-finalization-recovery.md). -For a resolved legacy action that retains a file hold, the same resolution API +For a resolved legacy or native action that retains a file hold, the same resolution API supports source-only file repair. Board runtime management and recovery-resolution authority are required. The request identifies the exact source action/run and records recovered-file evidence; the server verifies the original stop receipt. diff --git a/maps/feature-map/recovery.md b/maps/feature-map/recovery.md index 76e1c5c2025..5f0b8573c84 100644 --- a/maps/feature-map/recovery.md +++ b/maps/feature-map/recovery.md @@ -160,7 +160,7 @@ The recovery-card suite covers workspace divergence, reissue, reconcile, and confirmation. [Workspace export](../../ui/src/components/WorkspaceExportRecovery.test.tsx) covers quiet restoration, legacy/operator assistance and resolved states. -For a retained legacy file hold, open the source task after its recovery action +For a retained legacy or native file hold, open the source task after its recovery action has resolved but its files still need recovery. An authorized board operator can record the verified recovered files and backup link without changing the current task status or starting another run. The server must independently validate the diff --git a/server/src/__tests__/helpers/docker-command-runner.ts b/server/src/__tests__/helpers/docker-command-runner.ts new file mode 100644 index 00000000000..957bdd70a47 --- /dev/null +++ b/server/src/__tests__/helpers/docker-command-runner.ts @@ -0,0 +1,52 @@ +import { spawn } from "node:child_process"; +import { readFile } from "node:fs/promises"; +import { createConnection, type Socket } from "node:net"; +import { createServer } from "node:tls"; +import type { CommandManagedRuntimeRunner } from "@paperclipai/adapter-utils/command-managed-runtime"; + +export function dockerCommandRunner(container: string): CommandManagedRuntimeRunner { + return { + async execute(input) { + const args = ["exec", "-i", "-u", "root"]; + if (input.cwd) args.push("-w", input.cwd); + for (const [name, value] of Object.entries(input.env ?? {})) args.push("-e", `${name}=${value}`); + args.push(container, input.command, ...(input.args ?? [])); + const child = spawn("docker", args, { stdio: ["pipe", "pipe", "pipe"] }); + const startedAt = new Date().toISOString(); + let stdout = "", stderr = "", timedOut = false; + child.stdout.setEncoding("utf8").on("data", chunk => { stdout += chunk; }); + child.stderr.setEncoding("utf8").on("data", chunk => { stderr += chunk; }); + const timer = setTimeout(() => { timedOut = true; child.kill("SIGKILL"); }, input.timeoutMs ?? 30_000); + child.stdin.on("error", () => undefined); + child.stdin.end(input.stdin); + try { + return await new Promise>>((resolve, reject) => { + child.on("error", reject); + child.on("close", (exitCode, signal) => resolve({ exitCode, signal, timedOut, stdout, stderr, pid: child.pid ?? null, startedAt })); + }); + } finally { + clearTimeout(timer); + } + }, + }; +} + +export async function createRunnerTestTlsTunnel(input: { key: string; cert: string; port: number }) { + const sockets = new Set(); + const server = createServer({ key: await readFile(input.key), cert: await readFile(input.cert) }, socket => { + const upstream = createConnection({ host: "127.0.0.1", port: input.port }); + for (const connection of [socket, upstream]) { + sockets.add(connection); + connection.on("close", () => sockets.delete(connection)); + connection.on("error", () => { socket.destroy(); upstream.destroy(); }); + } + socket.pipe(upstream).pipe(socket); + }); + await new Promise(resolve => server.listen(0, "127.0.0.1", resolve)); + const address = server.address(); + if (!address || typeof address === "string") throw new Error("Test TLS listener unavailable"); + return { port: address.port, close: async () => { + for (const socket of sockets) socket.destroy(); + await new Promise(resolve => server.close(() => resolve())); + } }; +} diff --git a/server/src/__tests__/helpers/remote-warm-controller.test-worker.ts b/server/src/__tests__/helpers/remote-warm-controller.test-worker.ts new file mode 100644 index 00000000000..c7a3301ceee --- /dev/null +++ b/server/src/__tests__/helpers/remote-warm-controller.test-worker.ts @@ -0,0 +1,36 @@ +import { createDb } from "@paperclipai/db"; +import { executePaperclipNativeSession } from "../../services/native-runtime/native-session-executor.js"; +import { dockerCommandRunner } from "./docker-command-runner.js"; + +process.once("message", async (input: { + connectionString: string; + container: string; + ingressUrl: string; + options: Parameters[0]; +}) => { + const target = input.options.runnerExecutionTarget; + if (target?.kind !== "remote") throw new Error("Expected remote worker target"); + try { + const result = await executePaperclipNativeSession({ + ...input.options, + db: createDb(input.connectionString), + runnerExecutionTarget: { + ...target, + runner: dockerCommandRunner(input.container), + getRunnerIngressEndpoint: async ({ path }) => { + const endpoint = { + kind: "authenticated_websocket" as const, + websocketUrl: `${input.ingressUrl}${path}`, secretHeaders: [], generation: input.container, + refresh: async () => endpoint, close: async () => undefined, + }; + return endpoint; + }, + }, + onLog: async (stream, text) => { process.send?.({ log: { stream, text } }); }, + }); + process.send?.({ result }); + } catch (error) { + process.send?.({ error: error instanceof Error ? error.stack : String(error) }); + } +}); +setInterval(() => undefined, 60_000); diff --git a/server/src/services/native-runtime/native-restart-recovery.test.ts b/server/src/services/native-runtime/native-restart-recovery.test.ts index ee69e3f494a..63f604c97bf 100644 --- a/server/src/services/native-runtime/native-restart-recovery.test.ts +++ b/server/src/services/native-runtime/native-restart-recovery.test.ts @@ -208,6 +208,13 @@ describe("native controller takeover fencing", () => { }); }); + it.each([true, false])("requires a stopped idle controller even after its run lease was released: alive=%s", async alive => { + await expect(evaluateNativeControllerTakeover({ + owner: owner({ leaseOwner: null, leaseExpiresAt: null }), now, requireStoppedController: true, + isProcessAlive: () => alive, readProcessStartedAt: async () => recordedStart, + })).resolves.toEqual({ allowed: !alive, reason: alive ? "controller_still_alive" : "controller_process_dead" }); + }); + it("takes over a recycled controller PID", async () => { await expect( evaluateNativeControllerTakeover({ diff --git a/server/src/services/native-runtime/native-restart-recovery.ts b/server/src/services/native-runtime/native-restart-recovery.ts index 886ab2d9849..3e4090e710f 100644 --- a/server/src/services/native-runtime/native-restart-recovery.ts +++ b/server/src/services/native-runtime/native-restart-recovery.ts @@ -177,11 +177,12 @@ export async function evaluateNativeControllerTakeover(input: { } | null; isProcessAlive?: (pid: number) => boolean; readProcessStartedAt?: (pid: number) => Promise; + requireStoppedController?: boolean; }): Promise<{ allowed: boolean; reason: string }> { const { owner, now } = input; const isAlive = input.isProcessAlive ?? processIsAlive; const readStartedAt = input.readProcessStartedAt ?? observedProcessStart; - if (!owner.leaseOwner || !owner.leaseExpiresAt) { + if ((!owner.leaseOwner || !owner.leaseExpiresAt) && !input.requireStoppedController) { return { allowed: true, reason: "unowned" }; } @@ -190,7 +191,7 @@ export async function evaluateNativeControllerTakeover(input: { return { allowed: false, reason: - owner.leaseExpiresAt <= now + owner.leaseExpiresAt !== null && owner.leaseExpiresAt <= now ? "expired_lease_without_controller_identity" : "live_legacy_lease_without_controller_identity", }; @@ -199,7 +200,7 @@ export async function evaluateNativeControllerTakeover(input: { return { allowed: true, reason: - owner.leaseExpiresAt <= now + owner.leaseExpiresAt !== null && owner.leaseExpiresAt <= now ? "expired_lease_controller_process_dead" : "controller_process_dead", }; diff --git a/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts b/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts index 76587710106..d949fba5d83 100644 --- a/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts +++ b/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts @@ -1,9 +1,10 @@ import { enableRunFailureReporting } from "../../__tests__/helpers/run-failure-reporting.js"; import { hasNativeLocalProcessStop } from "../native-local-process-stop.js"; import { randomUUID } from "node:crypto"; +import { createRequire } from "node:module"; import { spawn } from "node:child_process"; import { existsSync } from "node:fs"; -import { mkdtemp, readFile, rename, rm } from "node:fs/promises"; +import { mkdir, mkdtemp, readFile, readdir, rename, rm, writeFile } from "node:fs/promises"; import { createServer, type Server } from "node:http"; import { tmpdir } from "node:os"; import { resolve } from "node:path"; @@ -12,6 +13,9 @@ import { and, eq } from "drizzle-orm"; import { agents, companies, + completionContracts, + environmentLeases, + environments, createDb, heartbeatRuns, heartbeatRunEvents, @@ -38,10 +42,15 @@ import { setupRunnerPrpWebSocketServer, } from "../../realtime/runner-prp-ws.js"; import { readProcessStartedAt } from "../hot-restart.js"; +import { resolvePaperclipInstanceRoot } from "../../home-paths.js"; import { prepareNativeHeartbeatRun } from "./prepare-native-run.js"; import { buildNativeExecutionInput } from "./native-execution-input.js"; import { nativeRuntimeContextFixture } from "./runtime-context.test-fixture.js"; -import { executePaperclipNativeSession, NativeControllerDetachedForRestartError } from "./native-session-executor.js"; +import { materializeAsset } from "./runtime-context.js"; +import { createRunnerTestTlsTunnel, dockerCommandRunner } from "../../__tests__/helpers/docker-command-runner.js"; +import { createIsolatedTestHome } from "../../__tests__/helpers/isolated-test-home.js"; +import { finalizeNativeRun } from "./native-run-finalizer.js"; +import { closeIdleWarmNativeSessionsForRestart, executePaperclipNativeSession, NativeControllerDetachedForRestartError } from "./native-session-executor.js"; const mockCaptureRunFailure = vi.hoisted(() => vi.fn()); vi.mock("../../sentry.js", async () => { @@ -140,6 +149,7 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( let server: Server | null = null; let apiUrl: string; let originalPaperclipHome: string | undefined; + let isolatedHome: ReturnType; const companyId = randomUUID(); const agentId = randomUUID(); @@ -150,7 +160,8 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( "native-runner-restart-recovery-", ); runtimeRoot = await mkdtemp(resolve(tmpdir(), "native-restart-runtime-")); - paperclipHome = await mkdtemp(resolve(tmpdir(), "native-restart-home-")); + isolatedHome = createIsolatedTestHome(); + paperclipHome = isolatedHome.home; originalPaperclipHome = process.env.PAPERCLIP_HOME; process.env.PAPERCLIP_HOME = paperclipHome; const controllerStartedAt = await readProcessStartedAt(process.pid); @@ -196,10 +207,8 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( runnerPrpWebSocketInternals.resetForTests(); await closeServer(server); await temporary.cleanup(); - await Promise.all([ - rm(runtimeRoot, { recursive: true, force: true }), - rm(paperclipHome, { recursive: true, force: true }), - ]); + await rm(runtimeRoot, { recursive: true, force: true }); + isolatedHome.cleanup(); if (originalPaperclipHome === undefined) delete process.env.PAPERCLIP_HOME; else process.env.PAPERCLIP_HOME = originalPaperclipHome; }); @@ -293,6 +302,208 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( }; } + it.skipIf(!process.env.PAPERCLIP_TEST_RUNNER_CONTAINER).each(["warm", "hard", "legacy", "changed"])("preserves remote warm continuity across completed turns: %s", async (scenario) => { + const hardRestart = scenario !== "warm"; + const container = process.env.PAPERCLIP_TEST_RUNNER_CONTAINER!; + const tunnel = await createRunnerTestTlsTunnel({ port: Number(process.env.PAPERCLIP_TEST_RUNNER_PORT), + key: process.env.PAPERCLIP_TEST_RUNNER_TLS_KEY!, cert: process.env.NODE_EXTRA_CA_CERTS! }); + const runner = dockerCommandRunner(container); + const fixture = await seedRun(`REMOTE-WARM-CONTINUITY-${scenario}`); + const boundContract = async (runId: string) => { + const [row] = await fixture.db.select({ contract: completionContracts }).from(heartbeatRuns) + .innerJoin(completionContracts, eq(completionContracts.id, heartbeatRuns.completionContractId)) + .where(eq(heartbeatRuns.id, runId)); + return { id: row.contract.id, sha256: row.contract.canonicalSha256, + schemaVersion: row.contract.schemaVersion, contract: row.contract.contractJson as never }; + }; + const workspace = resolve(runtimeRoot, fixture.issueId); + await mkdir(workspace, { recursive: true }); + const context = nativeRuntimeContextFixture(); + context.instructions.bundle = await materializeAsset([{ path: "AGENTS.md", content: Buffer.from("Complete the assigned conversation turn.\n"), mode: 0o444 }]); + context.aggregateDigest = canonicalNativeRuntimeContextDigest(context); + const execution = buildNativeExecutionInput({ + companyId, agentId, runId: fixture.runId, + issue: { id: fixture.issueId, identifier: "NRR-REMOTE-WARM-CONTINUITY", title: "Continue the same conversation", description: null, workMode: "standard" }, + taskPrompt: "Complete the first turn.", + workspace: { id: randomUUID(), cwd: workspace, repoUrl: null, repoRef: null, branchName: null }, + normalizedSessionId: fixture.native.normalizedSessionId, + lifecyclePolicy: { mode: "warm", idleTimeoutMs: 60_000 }, + runtimeContext: context, + completionContract: await boundContract(fixture.runId), + }); + await fixture.db.update(heartbeatRuns).set({ runnerProfileJson: { + ...fixture.preparedProfile, nativeExecutionInput: execution, + } }).where(eq(heartbeatRuns.id, fixture.runId)); + const remoteCwd = `/tmp/paperclip-continuity-${fixture.issueId}`; + expect((await runner.execute({ command: "mkdir", args: ["-p", remoteCwd] })).exitCode).toBe(0); + const target = { + kind: "remote" as const, transport: "sandbox" as const, + providerKey: "test-container", providerLeaseId: container, + environmentId: randomUUID(), leaseId: randomUUID(), remoteCwd, runner, + effectiveCapabilities: { runnerWebSocketIngress: true, reusableLeases: true, + nativeSyncIn: false, nativeSyncOut: false, persistentProcessSessions: false, + independentControlCommands: true, incrementalSessionOutput: false, + concurrentSyncOperations: false, duplexCommandStream: false }, + sandboxLeaseAcquisition: { outcome: "created" as const, providerLeaseId: container }, + getRunnerIngressEndpoint: async ({ path }: { path: string }) => { + const endpoint = { kind: "authenticated_websocket" as const, + websocketUrl: `wss://127.0.0.1:${tunnel.port}${path}`, secretHeaders: [], generation: container, + refresh: async () => endpoint, close: async () => undefined }; + return endpoint; + }, + }; + const logs: string[] = []; + await fixture.db.insert(environments).values({ id: target.environmentId, name: `Continuity ${fixture.issueId}`, driver: "sandbox" }); + await fixture.db.insert(environmentLeases).values({ id: target.leaseId, companyId, environmentId: target.environmentId, + heartbeatRunId: fixture.runId, issueId: fixture.issueId, provider: target.providerKey, + providerLeaseId: container, leasePolicy: "reuse_by_environment", metadata: { remoteCwd } }); + const options = { + db: fixture.db, runnerInstanceId: fixture.native.runnerInstanceId, + useRunnerd: true, runnerIngressAuthorized: true, + runnerRemoteBinaryPath: process.env.PAPERCLIP_TEST_LINUX_RUNNER_BINARY!, + runnerRemoteCodexPath: process.env.PAPERCLIP_TEST_FAKE_CODEX_WRAPPER!, + runnerEnvironment: { PATH: "/usr/local/bin:/usr/bin:/bin", PAPERCLIP_CONFIGURED_ENV_KEYS: '["CONTINUITY_TEST_CONFIG"]', CONTINUITY_TEST_CONFIG: "first" }, + onLog: async (_stream: "stdout" | "stderr", text: string) => { logs.push(text); }, + }; + let controller: ReturnType | undefined; + try { + const firstOptions = { ...options, execution, runnerExecutionTarget: target }; + const first = hardRestart ? await new Promise<{ exitCode: number | null }>((resolveResult, reject) => { + controller = spawn(process.execPath, ["--import", createRequire(import.meta.url).resolve("tsx"), resolve("server/src/__tests__/helpers/remote-warm-controller.test-worker.ts")], + { stdio: ["ignore", "pipe", "pipe", "ipc"], env: process.env }); + const timer = setTimeout(() => { controller?.kill("SIGKILL"); reject(new Error("Warm controller startup exceeded 60 seconds")); }, 60_000); + controller.stdout?.on("data", data => logs.push(String(data))); + controller.stderr?.on("data", data => logs.push(String(data))); + controller.on("error", error => { clearTimeout(timer); reject(error); }); + controller.on("exit", (code, signal) => { clearTimeout(timer); reject(new Error(`Warm controller exited: ${code}/${signal}: ${logs.join("").slice(-3_000)}`)); }); + controller.on("message", (message: { log?: { text: string }; result?: { exitCode: number | null }; error?: string }) => { + if (message.log) logs.push(message.log.text); + if (message.error) { clearTimeout(timer); reject(new Error(message.error)); } + if (message.result) { clearTimeout(timer); resolveResult(message.result); } + }); + controller.send?.({ connectionString: temporary.connectionString, container, ingressUrl: `wss://127.0.0.1:${tunnel.port}`, + options: { ...firstOptions, db: undefined, onLog: undefined, runnerExecutionTarget: { ...target, runner: undefined } } }); + }) : await executePaperclipNativeSession(firstOptions); + expect(first.exitCode).toBe(0); + await finalizeNativeRun({ db: fixture.db, runId: fixture.runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true }); + const snapshot = await runner.execute({ command: "sh", args: ["-c", `cat '${remoteCwd}'/.paperclip-runtime/paperclip-runner/sessions/*/runner/runner-state.json`] }); + expect(snapshot.exitCode).toBe(0); + const saved = JSON.parse(snapshot.stdout); + expect(saved.lifecycle).toBe("ready"); + const processBefore = await runner.execute({ command: "sh", args: ["-c", `cat '${remoteCwd}'/.paperclip-runtime/paperclip-runner/sessions/*/runner/runner-process.identity`] }); + expect(processBefore.exitCode).toBe(0); + const historyBefore = await runner.execute({ command: "sh", args: ["-c", `cat '${remoteCwd}'/.paperclip-runtime/paperclip-runner/sessions/*/filesystem/codex-home/fake-codex-state.json`] }); + const providerBefore = JSON.parse(historyBefore.stdout); + const nextRunId = randomUUID(); + const nextLeaseId = randomUUID(); + const [nextRun] = await fixture.db.insert(heartbeatRuns).values({ + id: nextRunId, companyId, agentId, status: "running", runtimeMode: "native", nativeIssueId: fixture.issueId, + invocationSource: "assignment", triggerDetail: "system", contextSnapshot: { issueId: fixture.issueId }, + nativeSessionId: fixture.native.normalizedSessionId, + }).returning(); + await fixture.db.update(issues).set({ executionRunId: nextRunId, status: "in_progress" }).where(eq(issues.id, fixture.issueId)); + await fixture.db.update(environmentLeases).set({ status: "expired", releasedAt: new Date() }).where(eq(environmentLeases.id, target.leaseId)); + await fixture.db.insert(environmentLeases).values({ id: nextLeaseId, companyId, environmentId: target.environmentId, + heartbeatRunId: nextRunId, issueId: fixture.issueId, provider: target.providerKey, + providerLeaseId: container, leasePolicy: "reuse_by_environment", metadata: { remoteCwd } }); + const next = await prepareNativeHeartbeatRun({ db: fixture.db, run: nextRun, + issue: { id: fixture.issueId, title: "Continue the same conversation", description: null, reviewPolicy: null }, + environmentLeaseId: nextLeaseId }); + await fixture.db.insert(nativeRunFinalizations).values({ runId: nextRunId, companyId, issueId: fixture.issueId, phase: "observed" }); + const nextOptions = { ...options, runnerInstanceId: next.runnerInstanceId, + runnerEnvironment: scenario === "changed" ? { ...options.runnerEnvironment, CONTINUITY_TEST_CONFIG: "second" } : options.runnerEnvironment, + execution: { ...execution, completionContract: await boundContract(nextRunId), binding: { ...execution.binding, runId: nextRunId }, task: { ...execution.task, prompt: "Complete the second conversation turn." } }, + runnerExecutionTarget: { ...target, leaseId: nextLeaseId, sandboxLeaseAcquisition: { outcome: "resumed" as const, providerLeaseId: container } }, + }; + if (controller) { + await expect(executePaperclipNativeSession(nextOptions)).rejects.toThrow("prior_owner_terminal_state_indeterminate"); + const exited = new Promise(resolveExit => controller!.once("exit", () => resolveExit())); + controller.kill("SIGKILL"); + await exited; + controller = undefined; + await expect(executePaperclipNativeSession({ ...nextOptions, + runnerExecutionTarget: { ...nextOptions.runnerExecutionTarget, + sandboxLeaseAcquisition: { outcome: "replacement", providerLeaseId: "unrelated-sandbox" } }, + })).rejects.toThrow("prior_owner_terminal_state_indeterminate"); + } + if (scenario === "legacy") { + const checkpoints = resolve(resolvePaperclipInstanceRoot(), "runtime", "paperclip-runner", "sessions"); + let changed = 0; + for (const name of await readdir(checkpoints)) { + if (!name.endsWith(".json")) continue; + const path = resolve(checkpoints, name); + const envelope = JSON.parse(await readFile(path, "utf8")); + if (envelope.snapshot?.identity?.runId !== fixture.runId) continue; + delete envelope.processConfigDigest; + await writeFile(path, JSON.stringify(envelope), { mode: 0o600 }); + changed++; + } + expect(changed).toBe(1); + } + const [nextPrepared] = await fixture.db.select({ profile: heartbeatRuns.runnerProfileJson }).from(heartbeatRuns).where(eq(heartbeatRuns.id, nextRunId)); + await fixture.db.update(heartbeatRuns).set({ runnerProfileJson: { ...nextPrepared.profile, + nativeExecutionInput: nextOptions.execution } }).where(eq(heartbeatRuns.id, nextRunId)); + const second = await executePaperclipNativeSession(nextOptions); + expect(second.exitCode).toBe(0); + const historyAfter = await runner.execute({ command: "sh", args: ["-c", `cat '${remoteCwd}'/.paperclip-runtime/paperclip-runner/sessions/*/filesystem/codex-home/fake-codex-state.json`] }); + const providerAfter = JSON.parse(historyAfter.stdout); + expect(providerAfter.threadId).toBe(providerBefore.threadId); + expect(providerAfter.nextTurn).toBe(2); + const calls = await runner.execute({ command: "cat", args: [`${remoteCwd}/provider-calls.log`] }); + expect(calls.stdout.split("\n").filter(line => line === "turn/start")).toHaveLength(2); + expect(calls.stdout.split("\n").filter(line => line === "thread/start")).toHaveLength(1); + const providerStarts = calls.stdout.split("\n").filter(line => line === "process-start").length; + expect(providerStarts).toBeGreaterThanOrEqual(1); + expect(providerStarts).toBeLessThanOrEqual(["legacy", "changed"].includes(scenario) ? 3 : hardRestart ? 2 : 1); + const processAfter = await runner.execute({ command: "sh", args: ["-c", `cat '${remoteCwd}'/.paperclip-runtime/paperclip-runner/sessions/*/runner/runner-process.identity`] }); + expect(processAfter.exitCode).toBe(0); + if (["legacy", "changed"].includes(scenario)) expect(processAfter.stdout).not.toBe(processBefore.stdout); + else expect(processAfter.stdout).toBe(processBefore.stdout); + await finalizeNativeRun({ db: fixture.db, runId: nextRunId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true }); + const thirdRunId = randomUUID(); + const thirdLeaseId = randomUUID(); + const [thirdRun] = await fixture.db.insert(heartbeatRuns).values({ + id: thirdRunId, companyId, agentId, status: "running", runtimeMode: "native", nativeIssueId: fixture.issueId, + invocationSource: "assignment", triggerDetail: "system", contextSnapshot: { issueId: fixture.issueId }, + nativeSessionId: fixture.native.normalizedSessionId, + }).returning(); + await fixture.db.update(issues).set({ executionRunId: thirdRunId, status: "in_progress" }).where(eq(issues.id, fixture.issueId)); + await fixture.db.update(environmentLeases).set({ status: "expired", releasedAt: new Date() }).where(eq(environmentLeases.id, nextLeaseId)); + await fixture.db.insert(environmentLeases).values({ id: thirdLeaseId, companyId, environmentId: target.environmentId, + heartbeatRunId: thirdRunId, issueId: fixture.issueId, provider: target.providerKey, + providerLeaseId: container, leasePolicy: "reuse_by_environment", metadata: { remoteCwd } }); + const thirdNative = await prepareNativeHeartbeatRun({ db: fixture.db, run: thirdRun, + issue: { id: fixture.issueId, title: "Continue the same conversation", description: null, reviewPolicy: null }, + environmentLeaseId: thirdLeaseId }); + await fixture.db.insert(nativeRunFinalizations).values({ runId: thirdRunId, companyId, issueId: fixture.issueId, phase: "observed" }); + const third = await executePaperclipNativeSession({ ...nextOptions, runnerInstanceId: thirdNative.runnerInstanceId, + execution: { ...nextOptions.execution, completionContract: await boundContract(thirdRunId), + binding: { ...execution.binding, runId: thirdRunId }, task: { ...execution.task, prompt: "Complete the third conversation turn." } }, + runnerExecutionTarget: { ...nextOptions.runnerExecutionTarget, leaseId: thirdLeaseId }, + }); + expect(third.exitCode).toBe(0); + const finalHistory = await runner.execute({ command: "sh", args: ["-c", `cat '${remoteCwd}'/.paperclip-runtime/paperclip-runner/sessions/*/filesystem/codex-home/fake-codex-state.json`] }); + expect(JSON.parse(finalHistory.stdout)).toMatchObject({ threadId: providerBefore.threadId, nextTurn: 3 }); + const finalCalls = await runner.execute({ command: "cat", args: [`${remoteCwd}/provider-calls.log`] }); + expect(finalCalls.stdout.split("\n").filter(line => line === "turn/start")).toHaveLength(3); + expect(finalCalls.stdout.split("\n").filter(line => line === "thread/start")).toHaveLength(1); + expect(finalCalls.stdout.split("\n").filter(line => line === "process-start")).toHaveLength(providerStarts); + const finalProcess = await runner.execute({ command: "sh", args: ["-c", `cat '${remoteCwd}'/.paperclip-runtime/paperclip-runner/sessions/*/runner/runner-process.identity`] }); + expect(finalProcess.stdout).toBe(processAfter.stdout); + expect(await closeIdleWarmNativeSessionsForRestart()).toEqual({ closed: 1, busy: 0, failed: 0 }); + const suspended = await runner.execute({ command: "sh", args: ["-c", `cat '${remoteCwd}'/.paperclip-runtime/paperclip-runner/sessions/*/runner/runner-state.json`] }); + expect(suspended.exitCode).toBe(0); + expect(JSON.parse(suspended.stdout).lifecycle).toBe("suspended"); + } catch (error) { + await writeFile(resolve(runtimeRoot, "remote-continuity-failure.log"), logs.join("")); + throw new Error(`Remote continuity regression failed: ${logs.filter(line => line.startsWith("[paperclip-runner]")).join("").slice(-4_000)}`, { cause: error }); + } finally { + controller?.kill("SIGKILL"); + await closeIdleWarmNativeSessionsForRestart(); + await tunnel.close(); + } + }, 180_000); + async function persistRestartEvidence(input: { fixture: Awaited>; runnerPid: number; diff --git a/server/src/services/native-runtime/native-session-executor.test.ts b/server/src/services/native-runtime/native-session-executor.test.ts index 572e126883b..362aa46db98 100644 --- a/server/src/services/native-runtime/native-session-executor.test.ts +++ b/server/src/services/native-runtime/native-session-executor.test.ts @@ -587,6 +587,33 @@ describe("remote controller restart adoption", () => { ).rejects.toThrow("native_remote_recovery_lease_mismatch"); expect(execute).not.toHaveBeenCalled(); }); + it.each([ + ["runner_state", "timeout"], + ["runner_state", "transport"], + ["identity_marker", "timeout"], + ["identity_marker", "transport"], + ["process_verification", "timeout"], + ["process_verification", "transport"], + ] as const)("reattaches after a transient %s %s without starting another runner", async (stage, failure) => { + const { target, execute } = fixture(); + const original = execute.getMockImplementation()!; + const operation = stage === "identity_marker" + ? "paperclip-runner-recovery-identity" : "paperclip-runner-recovery-check"; + let probes = 0; + execute.mockImplementation(async request => { + if ((stage === "runner_state" ? request.args[1].includes("base64") : request.args[2] === operation) && probes++ === 0) { + if (failure === "transport") throw new Error("temporary transport outage"); + return { stdout: "", stderr: "", exitCode: -1, timedOut: true }; + } + return original(request); + }); + const adopted = await verifyRemoteRunnerReattachment({ + claim, target: target as never, identity, runId: "run", normalizedSessionId: "session", + }); + expect(adopted.pid).toBe(123); + expect(probes).toBe(2); + expect(execute.mock.calls.every(([request]) => !request.args.join(" ").match(/nohup|mkdir|rm -|mv -|kill -"/))).toBe(true); + }); it("rejects a marker that cannot prove its process generation", async () => { const { target, execute } = fixture(); const original = execute.getMockImplementation()!; @@ -599,6 +626,16 @@ describe("remote controller restart adoption", () => { remoteRunnerRecovery: { stage: "identity_marker", reason: "fingerprint_missing" }, }); }); + it.each(["failed read", "invalid JSON"])("records a closed recovery diagnostic for saved-state %s", async failure => { + const { target, execute } = fixture(); + execute.mockResolvedValueOnce({ stdout: Buffer.from("invalid JSON").toString("base64"), stderr: "", + exitCode: failure === "failed read" ? 1 : 0, timedOut: false }); + const error = await verifyRemoteRunnerReattachment({ claim, target: target as never, identity, + runId: "run", normalizedSessionId: "session" }).catch(error => error); + expect(readRemoteRunnerRecoveryError(error)).toEqual({ stage: "runner_state", + reason: failure === "failed read" ? "state_read_failed" : "state_invalid" }); + expect(execute).toHaveBeenCalledOnce(); + }); it.each([ [31, "marker_missing"], [32, "marker_unsafe"], [33, "marker_read_failed"], ] as const)("records the closed reason for marker read exit %s", async (exitCode, reason) => { @@ -8815,13 +8852,16 @@ describe("native warm session supervision", () => { describe("native session bounded recovery", () => { it("persists closed remote recovery evidence without replacing saved run results", async () => { const updates: Array<{ table: unknown; values: Record }> = []; + const onEvent = vi.fn(async () => undefined); const remoteRunnerRecovery = { stage: "identity_marker", reason: "fingerprint_missing" } as const; const failure = new RemoteRunnerRecoveryError(remoteRunnerRecovery); state.execute.mockReset().mockRejectedValueOnce(failure); state.upsertRecoveryAction.mockReset().mockResolvedValue({}); const service = vi.spyOn(issueServiceModule, "issueService").mockReturnValue({ update: vi.fn(async () => null) } as unknown as ReturnType); try { - await expect(executePaperclipNativeSession({ db: leaseDb(execution, {}, {}, updates), execution, runnerInstanceId: "runner" })).rejects.toBe(failure); + await expect(executePaperclipNativeSession({ db: leaseDb(execution, {}, {}, updates), execution, runnerInstanceId: "runner", onEvent })).rejects.toBe(failure); + expect(onEvent).toHaveBeenCalledWith({ eventType: "native.remote_runner_recovery_failed", level: "error", + message: "Could not verify the existing remote runner.", payload: { remoteRunnerRecovery } }); expect(updates.find(update => update.table === nativeRunFinalizations && update.values.failureDetail)?.values.failureDetail) .toMatchObject({ remoteRunnerRecovery }); const resultJson = updates.find(update => update.table === heartbeatRuns && update.values.error)?.values.resultJson as SQL; @@ -9714,6 +9754,42 @@ describe("native process ownership", () => { }); describe("runnerd provider runtime wiring", () => { + it("keeps remote warm turns open while retaining verified close semantics", async () => { + const current = scopedNativeExecution("remote-warm-settlement"); + current.session = { + ...current.session, + lifecyclePolicy: { mode: "warm", idleTimeoutMs: 60_000 }, + }; + state.execute.mockResolvedValueOnce({ + result: { summary: "completed" }, + terminal: { runTerminalState: "succeeded" }, + turnId: "turn", + normalizedSessionId: current.session.normalizedSessionId, + providerSessionId: "retained-provider-thread", + driverKind: "codex_app_server", + driverVersion: "1", + nativeEventCount: 1, + highestContiguousSourceSeq: 1, + }); + + await executePaperclipNativeSession({ + db: leaseDb(current), + execution: current, + runnerInstanceId: "runner-remote-warm-settlement", + useRunnerd: true, + runnerExecutionTarget: { + kind: "remote", transport: "sandbox", remoteCwd: "/workspace", + environmentId: "environment", leaseId: "lease", + runner: { execute: vi.fn() }, + } as never, + }); + + expect(state.execute).toHaveBeenLastCalledWith(expect.objectContaining({ + keepSessionOpen: true, + requireSessionCloseBeforeReturn: true, + })); + }); + function scopedNativeExecution(suffix: string): NativeExecutionInputV1 { return { ...execution, diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index f90d6a55060..0a48050d67b 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -161,6 +161,8 @@ import { REMOTE_RECOVERY_MARKER_READ_SCRIPT, REMOTE_RECOVERY_PROCESS_CHECK_SCRIPT, RemoteRunnerRecoveryError, attachRemoteRunnerRecoveryDiagnostic, readRemoteRunnerRecoveryError, remoteRecoveryProbeReason, } from "./remote-runner-recovery-diagnostics.js"; +import { readRemoteRunnerRecoveryProbe, REMOTE_RUNNER_RECOVERY_PROBE_BUDGET_MS } from "./remote-runner-recovery-probe.js"; +import { inspectRemoteWarmSessionRecovery, type RemoteWarmSessionRecovery } from "./remote-warm-session-recovery.js"; import { persistActivity, publishActivity } from "../activity-log.js"; import { commitNativeStatusDecision } from "./status-decision-committer.js"; import { resolvePaperclipInstanceRoot } from "../../home-paths.js"; @@ -6023,6 +6025,7 @@ function persistWarmNativeCheckpoint( execution: NativeExecutionInput, configDigest: string, snapshot: PersistedNativeSession, + processConfigDigest?: string, ): void { const path = nativeSessionCheckpointPath(execution); const temporary = `${path}.${process.pid}.${randomUUID()}.tmp`; @@ -6031,6 +6034,7 @@ function persistWarmNativeCheckpoint( JSON.stringify({ schema: "paperclip.native-session-supervisor.v1", configDigest, + processConfigDigest, updatedAt: new Date().toISOString(), snapshot, }), @@ -6040,6 +6044,16 @@ function persistWarmNativeCheckpoint( chmodSync(path, 0o600); } +function warmNativeProcessConfigDigest(input: Parameters[0]): string | undefined { + if (input.managedGitHub || input.runnerEnvironment?.PAPERCLIP_GITHUB_BROKER_TOKEN || input.instructionWorkingCopy) return undefined; + return nativeSha256({ + environment: configuredEnvironment(input.runnerEnvironment), + managedAiCredentialIdentity: input.managedAiCredentialIdentity ?? null, + githubAuthenticationMode: input.runnerEnvironment?.PAPERCLIP_GITHUB_AUTH_MODE ?? null, + networkAccess: input.runnerEnvironment?.PAPERCLIP_RUNNER_NETWORK_ACCESS === "enabled", + }); +} + function loadWarmNativeCheckpoint( execution: NativeExecutionInput, configDigest: string, @@ -7625,6 +7639,15 @@ export async function executePaperclipNativeSession(input: { return await executePaperclipNativeSessionWithinScope(preparedInput); } catch (error) { executionFailure = error; + const remoteRunnerRecovery = readRemoteRunnerRecoveryError(error); + if (remoteRunnerRecovery) { + await input.onEvent?.({ + eventType: "native.remote_runner_recovery_failed", + level: "error", + message: "Could not verify the existing remote runner.", + payload: { remoteRunnerRecovery }, + }).catch(() => undefined); + } throw error; } finally { if (executingNativeOwnerScopes.get(ownerScope) === ownerToken) { @@ -7744,16 +7767,39 @@ async function executePaperclipNativeSessionWithinScope( await trace.end(environmentScope, { endedAtMs: environmentEndedAtMs }); } let retainedTransition: VerifiedWarmTransitionBinding | undefined; + let remoteWarmRecovery: RemoteWarmSessionRecovery | undefined; + if (input.useRunnerd && !input.restartRecovery && !input.backend && + input.runnerExecutionTarget?.kind === "remote" && + input.execution.session.lifecyclePolicy.mode === "warm" && + !idleWarmNativeSessionOwnerRunId(input)) { + const root = scopedRunnerdStateRoot(input.execution); + const identity = readRunnerdDurableIdentity(root); + if (durableIdentityMatchesSession(identity, input.execution) && identity.runId !== input.execution.binding.runId && + runnerdAuthorityLifecycleWithVerifiedBackup({ root, identity, execution: input.execution, allowVerifiedBackup: true }) !== "suspended") { + remoteWarmRecovery = await inspectRemoteWarmSessionRecovery({ + db: input.db, execution: input.execution, target: input.runnerExecutionTarget, identity, + readControlState: () => record(JSON.parse(readBoundedNativeFile( + resolve(root, "control-plane", "control-plane-state.json"), NATIVE_CONTROL_PLANE_STATE_MAX_BYTES, + "runner_durable_identity_too_large").toString("utf8"))), + matchesExecution: previous => nativeSessionScopeKey(previous) === nativeSessionScopeKey(input.execution) && + nativeSessionConfigDigest(previous, "remote") === nativeSessionConfigDigest(input.execution, "remote"), + loadCheckpoint: previous => loadWarmNativeCheckpoint(previous, nativeSessionConfigDigest(previous, "remote"), "remote"), + canReuseProcess: previous => { + const expected = warmNativeProcessConfigDigest(input); + if (!expected) return false; + const envelope = record(JSON.parse(readBoundedNativeFile(nativeSessionCheckpointPath(previous), + NATIVE_WARM_CHECKPOINT_MAX_BYTES, "native_session_supervisor_checkpoint_too_large").toString("utf8"))); + return envelope.processConfigDigest === expected; + }, + }); + } + } if (input.useRunnerd) { retainedTransition = await migrateRunnerdStateRootForExecution({ db: input.db, execution: input.execution, allowVerifiedBackup: input.runnerExecutionTarget?.kind === "remote", - // A retained warm runner is deliberately still ready rather than - // suspended. Only the exact idle in-process owner may rotate that - // prior-run authority; after a hard restart the map is empty and the - // durable-state verifier continues to require a suspended runner. - retainedWarmOwnerRunId: idleWarmNativeSessionOwnerRunId(input), + retainedWarmOwnerRunId: idleWarmNativeSessionOwnerRunId(input) ?? remoteWarmRecovery?.identity.runId, allowLocalRecovery: input.runnerExecutionTarget?.kind !== "remote", onLog: input.onLog, restartRecovery: input.restartRecovery, @@ -8643,7 +8689,8 @@ async function executePaperclipNativeSessionWithinScope( input.db, input.execution.binding, )); - const runnerdBackend = + let remoteWarmRecoveryPending = remoteWarmRecovery !== undefined; + let runnerdBackend = input.useRunnerd && input.backend === undefined ? await timeController("backend_setup", () => createRunnerdBackend({ ...input, @@ -8653,10 +8700,34 @@ async function executePaperclipNativeSessionWithinScope( execution: input.execution, runnerInstanceId: effectiveRunnerInstanceId, durableEnvironmentLeaseId: durableRunnerBinding?.environmentLeaseId, + remoteWarmRecovery: remoteWarmRecovery ? { + ...remoteWarmRecovery, + revalidate: async () => { + if (remoteWarmRecoveryPending) await remoteWarmRecovery.revalidate(leaseOwner); + }, + } : undefined, trace, toolTrace, })) : null; + if (remoteWarmRecovery) { + await remoteWarmRecovery.revalidate(leaseOwner); + const recovered = await runnerdBackend?.recoverSession?.(remoteWarmRecovery.checkpoint, { + signal: new AbortController().signal, + }); + if (!recovered?.recovered || !recovered.session) throw new Error("runner_state_identity_mismatch: remote_warm_session_unproven"); + if (remoteWarmRecovery.reuseProcess) { + existingWarmSession = recovered.session; + persistedWarmSession = null; + } else { + await recovered.session.close({ reason: "recovered warm session requires current process configuration" }); + persistWarmNativeCheckpoint(remoteWarmRecovery.previousExecution, + nativeSessionConfigDigest(remoteWarmRecovery.previousExecution, "remote"), await recovered.session.snapshot()); + persistedWarmSession = loadWarmNativeCheckpoint(input.execution, warmConfigDigest!, "remote"); + runnerdBackend = await createRunnerdBackend({ ...input, runnerInstanceId: effectiveRunnerInstanceId, + durableEnvironmentLeaseId: durableRunnerBinding?.environmentLeaseId, trace, toolTrace }); + } + } const remoteCleanupLease = input.runnerExecutionTarget?.kind === "remote" && input.runnerExecutionTarget.transport === "sandbox" && input.runnerExecutionTarget.leaseId ? await input.db.select({ provider: environmentLeases.provider, providerLeaseId: environmentLeases.providerLeaseId }) @@ -8759,8 +8830,6 @@ async function executePaperclipNativeSessionWithinScope( keepSessionOpen: warmSessionId !== null, sessionGoalControl: input.sessionGoalControl, resumeSessionGoalHeartbeat: input.resumeSessionGoalHeartbeat, - // Every durable runner must finish its bounded suspension before - // the next run verifies and rotates the saved authority. requireSessionCloseBeforeReturn: runnerdBackend !== null || input.instructionWorkingCopy !== undefined, onSessionClosed: input.instructionWorkingCopy?.collectStopped, onCheckpoint: async (snapshot) => { @@ -8774,6 +8843,7 @@ async function executePaperclipNativeSessionWithinScope( input.execution, warmConfigDigest, snapshot, + warmNativeProcessConfigDigest(input), ); } if ( @@ -8824,6 +8894,7 @@ async function executePaperclipNativeSessionWithinScope( ); }, onSession: async (session) => { + if (session && session === existingWarmSession) remoteWarmRecoveryPending = false; if (session && input.managedAiCredentialHome) { managedCredentialSession = runnerdBackend?.bindManagedSession(session) ?? session; } @@ -10609,25 +10680,35 @@ export async function syncRemoteRunnerDirectoryOut(input: { export async function readRemoteRunnerState(input: { runner: CommandManagedRuntimeRunner; stateDirectory: string; + recoveryDeadline?: number; }): Promise> { const statePath = posix.join(input.stateDirectory, "runner-state.json"); const escapedPath = statePath.replaceAll("'", "'\\''"); - const result = await input.runner.execute({ + const request = { command: "sh", args: ["-c", `test -f '${escapedPath}' && base64 < '${escapedPath}'`], bypassSession: true, timeoutMs: 10_000, - }); + }; + const result = input.recoveryDeadline === undefined + ? await input.runner.execute(request) + : await readRemoteRunnerRecoveryProbe({ + runner: input.runner, request, stage: "runner_state", + deadline: input.recoveryDeadline, readOnly: true, + }); if (result.exitCode !== 0 || result.timedOut) { - throw new Error( + const error = new Error( `runner_remote_state_unavailable: exit=${result.exitCode} timedOut=${result.timedOut}${result.stderr.trim() ? ` ${redactSensitiveText(result.stderr).trim().slice(-512)}` : ""}`, ); + throw input.recoveryDeadline === undefined ? error : attachRemoteRunnerRecoveryDiagnostic(error, + { stage: "runner_state", reason: result.timedOut ? "probe_timeout" : "state_read_failed" }); + } + try { + return record(JSON.parse(Buffer.from(result.stdout.replace(/\s+/g, ""), "base64").toString("utf8"))); + } catch (error) { + throw input.recoveryDeadline === undefined ? error : attachRemoteRunnerRecoveryDiagnostic(error, + { stage: "runner_state", reason: "state_invalid" }); } - return record( - JSON.parse( - Buffer.from(result.stdout.replace(/\s+/g, ""), "base64").toString("utf8"), - ), - ); } async function readRemoteRunnerProviderState(input: { @@ -10700,14 +10781,15 @@ export function parseRemoteRunnerProcessIdentity( /** Verify remote ownership before exposing the transport's authenticated adoption path. */ export async function verifyRemoteRunnerReattachment(input: { - claim: Extract< + claim: Pick; + >, "runId" | "remote">; target: AdapterExecutionTarget | null | undefined; identity: Record; runId: string; normalizedSessionId: string; + requireIdle?: boolean; }) { const { claim, target, identity } = input; if ( @@ -10725,6 +10807,7 @@ export async function verifyRemoteRunnerReattachment(input: { throw new Error("native_remote_recovery_lease_mismatch"); } const runner = target.runner; + const probeDeadline = Date.now() + REMOTE_RUNNER_RECOVERY_PROBE_BUDGET_MS; const stateDirectory = posix.join( target.remoteCwd, ".paperclip-runtime", @@ -10733,7 +10816,11 @@ export async function verifyRemoteRunnerReattachment(input: { createHash("sha256").update(input.normalizedSessionId).digest("hex"), "runner", ); - const runnerState = await readRemoteRunnerState({ runner, stateDirectory }); + const runnerState = await readRemoteRunnerState({ runner, stateDirectory, recoveryDeadline: probeDeadline }); + if (input.requireIdle && (runnerState.lifecycle !== "ready" || + !Array.isArray(runnerState.outbox) || runnerState.outbox.length !== 0 || runnerState.pendingTerminalDelivery !== null)) { + throw new Error("runner_state_identity_mismatch: remote_warm_session_not_idle"); + } for (const field of [ "runId", "normalizedSessionId", @@ -10751,17 +10838,19 @@ export async function verifyRemoteRunnerReattachment(input: { } } const identityPath = posix.join(stateDirectory, "runner-process.identity"); - const marker = await runner.execute({ - command: "sh", - args: [ - "-c", - REMOTE_RECOVERY_MARKER_READ_SCRIPT, - "paperclip-runner-recovery-identity", - identityPath, - ], - bypassSession: true, - timeoutMs: 10_000, - }).catch(error => { throw attachRemoteRunnerRecoveryDiagnostic(error, { stage: "identity_marker", reason: "probe_transport_failed" }); }); + const marker = await readRemoteRunnerRecoveryProbe({ + runner, stage: "identity_marker", deadline: probeDeadline, readOnly: true, + request: { + command: "sh", + args: [ + "-c", + REMOTE_RECOVERY_MARKER_READ_SCRIPT, + "paperclip-runner-recovery-identity", + identityPath, + ], + bypassSession: true, + }, + }); const nonce = marker.stdout.split("\n")[0] ?? ""; const runnerInstanceId = identity.runnerInstanceId as string; const processIdentity = @@ -10778,28 +10867,30 @@ export async function verifyRemoteRunnerReattachment(input: { if (!processIdentity) throw new RemoteRunnerRecoveryError({ stage: "identity_marker", reason: "marker_invalid" }); if (!processIdentity.processStartFingerprint) throw new RemoteRunnerRecoveryError({ stage: "identity_marker", reason: "fingerprint_missing" }); const expectedStartFingerprint = processIdentity.processStartFingerprint; - const check = async (signal?: NodeJS.Signals) => { - const result = await runner.execute({ - command: "sh", - args: [ - "-c", - `${REMOTE_RECOVERY_PROCESS_CHECK_SCRIPT}; if ${REMOTE_RUNNER_PROCESS_FINGERPRINT_SCRIPT}; then :; else exit 46; fi; test "$process_fingerprint" = "$5" || exit 47` + - (signal ? '; kill -"$6" "$pid"' : ""), - "paperclip-runner-recovery-check", - identityPath, - nonce, - runnerInstanceId, - String(processIdentity.pid), - expectedStartFingerprint, - ...(signal ? [signal.slice(3)] : []), - ], - bypassSession: true, - timeoutMs: 10_000, - }).catch(error => { throw attachRemoteRunnerRecoveryDiagnostic(error, { stage: "process_verification", reason: "probe_transport_failed" }); }); + const check = async (signal?: NodeJS.Signals, deadline = Date.now() + REMOTE_RUNNER_RECOVERY_PROBE_BUDGET_MS) => { + const result = await readRemoteRunnerRecoveryProbe({ + runner, stage: "process_verification", deadline, readOnly: !signal, + request: { + command: "sh", + args: [ + "-c", + `${REMOTE_RECOVERY_PROCESS_CHECK_SCRIPT}; if ${REMOTE_RUNNER_PROCESS_FINGERPRINT_SCRIPT}; then :; else exit 46; fi; test "$process_fingerprint" = "$5" || exit 47` + + (signal ? '; kill -"$6" "$pid"' : ""), + "paperclip-runner-recovery-check", + identityPath, + nonce, + runnerInstanceId, + String(processIdentity.pid), + expectedStartFingerprint, + ...(signal ? [signal.slice(3)] : []), + ], + bypassSession: true, + }, + }); return { alive: result.exitCode === 0 && !result.timedOut, reason: remoteRecoveryProbeReason("process_verification", result.exitCode, result.timedOut) }; }; - const verified = await check(); + const verified = await check(undefined, probeDeadline); if (!verified.alive) throw new RemoteRunnerRecoveryError({ stage: "process_verification", reason: verified.reason }); // These checks authorize an attempt to reconnect, not the task itself. The // transport still requires authentication against the exact durable PRP key. @@ -11151,6 +11242,7 @@ export async function createRunnerdBackend(input: { chatAttachmentReadScope?: NativeChatAttachmentReadScope; restartRecovery?: NativeRestartRecoveryClaim; durableEnvironmentLeaseId?: string; + remoteWarmRecovery?: Omit & { revalidate: () => Promise }; onSpawn?: (meta: { pid: number; processGroupId: number | null; @@ -11744,7 +11836,7 @@ async function createRunnerdBackendWithinSessionClaim( ) return; selectedRemoteMode = requiredMode; - if (input.restartRecovery?.kind === "reattach_remote_runner") { + if (input.restartRecovery?.kind === "reattach_remote_runner" || input.remoteWarmRecovery) { await verifyRemoteRunner(requiredMode); if (requiresRemoteProviderPack && stagedRemoteProviderPackRoot) { await verifyRemoteProviderPack(stagedRemoteProviderPackRoot); @@ -12784,7 +12876,17 @@ async function createRunnerdBackendWithinSessionClaim( ? input.restartRecovery.process : null; const adoptedProcess = - input.restartRecovery?.kind === "reattach_remote_runner" + input.remoteWarmRecovery + ? await (async () => { + await input.remoteWarmRecovery!.revalidate(); + return verifyRemoteRunnerReattachment({ + claim: { runId: input.remoteWarmRecovery!.identity.runId, remote: input.remoteWarmRecovery!.remote }, + target, identity: input.remoteWarmRecovery!.identity, + runId: input.remoteWarmRecovery!.identity.runId, + normalizedSessionId: nativeSessionKey(input.execution), requireIdle: true, + }); + })() + : input.restartRecovery?.kind === "reattach_remote_runner" ? await verifyRemoteRunnerReattachment({ claim: input.restartRecovery, target, @@ -12988,7 +13090,14 @@ async function createRunnerdBackendWithinSessionClaim( input.execution.interactionResponses.length > 0 ? "allow_replacement_after_governed_wait" : undefined), - prpIdentity: { + prpIdentity: input.remoteWarmRecovery ? { + runnerInstanceId: input.remoteWarmRecovery.identity.runnerInstanceId, + environmentLeaseId: input.remoteWarmRecovery.identity.environmentLeaseId, + runId: input.remoteWarmRecovery.identity.runId, + normalizedSessionId: input.remoteWarmRecovery.identity.normalizedSessionId, + turnId: input.remoteWarmRecovery.identity.turnId, + itemId: input.remoteWarmRecovery.identity.itemId, + } : { runnerInstanceId: effectiveRunnerInstanceId, environmentLeaseId: effectiveEnvironmentLeaseId, runId: input.execution.binding.runId, @@ -13050,7 +13159,9 @@ async function createRunnerdBackendWithinSessionClaim( adoptExistingRunner: adoptedProcess, }, target, - binding: input.execution.binding, + binding: input.remoteWarmRecovery + ? { ...input.execution.binding, runId: input.remoteWarmRecovery.identity.runId } + : input.execution.binding, runnerIngressAuthorized: input.runnerIngressAuthorized, runnerPublicUrl: input.runnerPublicUrl, runnerCaBundlePath: input.runnerCaBundlePath, @@ -13066,6 +13177,7 @@ async function createRunnerdBackendWithinSessionClaim( return targetPath; }, beforeConnect: async () => { + await input.remoteWarmRecovery?.revalidate(); if (retainedTransition && recoveryPending) { const current = await verifyWarmTransitionRestart(input); if (current.stateFingerprint !== retainedTransition.stateFingerprint) { diff --git a/server/src/services/native-runtime/remote-runner-recovery-diagnostics.ts b/server/src/services/native-runtime/remote-runner-recovery-diagnostics.ts index 9eb00409025..3e9c5674009 100644 --- a/server/src/services/native-runtime/remote-runner-recovery-diagnostics.ts +++ b/server/src/services/native-runtime/remote-runner-recovery-diagnostics.ts @@ -1,11 +1,12 @@ export const REMOTE_RUNNER_RECOVERY_ERROR_CODE = "runner_remote_process_identity_unavailable"; -const stages = ["identity_marker", "process_verification"] as const; +const stages = ["runner_state", "identity_marker", "process_verification"] as const; const reasons = [ "marker_missing", "marker_unsafe", "marker_read_failed", "marker_invalid", "fingerprint_missing", "process_not_running", "process_identity_changed", "fingerprint_unavailable", "process_generation_changed", "probe_timeout", "probe_transport_failed", "probe_failed", + "state_read_failed", "state_invalid", ] as const; export interface RemoteRunnerRecoveryDiagnostic { @@ -78,7 +79,7 @@ export function remoteRecoveryProbeReason(stage: RemoteRunnerRecoveryDiagnostic[ if (exitCode === 31) return "marker_missing"; if (exitCode === 32) return "marker_unsafe"; if (exitCode === 33) return "marker_read_failed"; - } else { + } else if (stage === "process_verification") { if (exitCode === 41) return "marker_missing"; if (exitCode === 42) return "marker_unsafe"; if (exitCode === 43) return "marker_read_failed"; diff --git a/server/src/services/native-runtime/remote-runner-recovery-probe.test.ts b/server/src/services/native-runtime/remote-runner-recovery-probe.test.ts new file mode 100644 index 00000000000..fe25cf3296d --- /dev/null +++ b/server/src/services/native-runtime/remote-runner-recovery-probe.test.ts @@ -0,0 +1,44 @@ +import { afterEach, expect, it, vi } from "vitest"; +import type { CommandManagedRuntimeRunner } from "@paperclipai/adapter-utils/command-managed-runtime"; +import { readRemoteRunnerRecoveryError } from "./remote-runner-recovery-diagnostics.js"; +import { readRemoteRunnerRecoveryProbe } from "./remote-runner-recovery-probe.js"; + +afterEach(() => vi.useRealTimers()); + +const timeout = { exitCode: null, signal: null, timedOut: true, stdout: "", stderr: "", pid: null, startedAt: "2026-10-15T00:00:00Z" }; + +it("shares the remaining deadline across timed-out reads", async () => { + vi.useFakeTimers(); + const execute = vi.fn(async (request) => { + await new Promise(resolve => setTimeout(resolve, request.timeoutMs)); + return timeout; + }); + const result = readRemoteRunnerRecoveryProbe({ + runner: { execute } as CommandManagedRuntimeRunner, + request: { command: "sh" }, stage: "identity_marker", + readOnly: true, deadline: Date.now() + 15_000, + }).catch(readRemoteRunnerRecoveryError); + await vi.runAllTimersAsync(); + expect(await result).toEqual({ stage: "identity_marker", reason: "probe_timeout" }); + expect(execute.mock.calls.map(([request]) => request.timeoutMs)).toEqual([10_000, 4_750]); +}); + +it("never retries a signal when its acknowledgement is lost", async () => { + const execute = vi.fn().mockRejectedValue(new Error("connection lost")); + const result = await readRemoteRunnerRecoveryProbe({ + runner: { execute }, request: { command: "sh" }, stage: "process_verification", + readOnly: false, deadline: Date.now() + 30_000, + }).catch(readRemoteRunnerRecoveryError); + expect(result).toEqual({ stage: "process_verification", reason: "probe_transport_failed" }); + expect(execute).toHaveBeenCalledTimes(1); +}); + +it("returns a definitive ownership conflict without retrying it", async () => { + const conflict = { ...timeout, timedOut: false, exitCode: 47 }; + const execute = vi.fn().mockResolvedValue(conflict); + expect(await readRemoteRunnerRecoveryProbe({ + runner: { execute }, request: { command: "sh" }, stage: "process_verification", + readOnly: true, deadline: Date.now() + 30_000, + })).toBe(conflict); + expect(execute).toHaveBeenCalledTimes(1); +}); diff --git a/server/src/services/native-runtime/remote-runner-recovery-probe.ts b/server/src/services/native-runtime/remote-runner-recovery-probe.ts new file mode 100644 index 00000000000..7f48277f7e3 --- /dev/null +++ b/server/src/services/native-runtime/remote-runner-recovery-probe.ts @@ -0,0 +1,39 @@ +import type { CommandManagedRuntimeRunner } from "@paperclipai/adapter-utils/command-managed-runtime"; +import { + attachRemoteRunnerRecoveryDiagnostic, + RemoteRunnerRecoveryError, + type RemoteRunnerRecoveryDiagnostic, +} from "./remote-runner-recovery-diagnostics.js"; + +export const REMOTE_RUNNER_RECOVERY_PROBE_BUDGET_MS = 30_000; + +export async function readRemoteRunnerRecoveryProbe(input: { + runner: CommandManagedRuntimeRunner; + request: Parameters[0]; + stage: RemoteRunnerRecoveryDiagnostic["stage"]; + deadline: number; + readOnly: boolean; +}) { + let lastError: unknown = new RemoteRunnerRecoveryError({ + stage: input.stage, reason: "probe_timeout", + }); + for (let attempt = 0; attempt < (input.readOnly ? 3 : 1); attempt++) { + const remaining = input.deadline - Date.now(); + if (remaining <= 0) break; + try { + const result = await input.runner.execute({ + ...input.request, + timeoutMs: Math.min(10_000, remaining), + }); + if (!result.timedOut) return result; + lastError = new RemoteRunnerRecoveryError({ stage: input.stage, reason: "probe_timeout" }); + } catch (error) { + lastError = attachRemoteRunnerRecoveryDiagnostic(error, { + stage: input.stage, reason: "probe_transport_failed", + }); + } + if (!input.readOnly || attempt === 2 || input.deadline - Date.now() <= 250) break; + await new Promise(resolve => setTimeout(resolve, 250)); + } + throw lastError; +} diff --git a/server/src/services/native-runtime/remote-warm-session-recovery.ts b/server/src/services/native-runtime/remote-warm-session-recovery.ts new file mode 100644 index 00000000000..2d91c49f6c7 --- /dev/null +++ b/server/src/services/native-runtime/remote-warm-session-recovery.ts @@ -0,0 +1,109 @@ +import { and, eq } from "drizzle-orm"; +import { agents, companies, environmentLeases, heartbeatRuns, issues, nativeRunFinalizations, type Db } from "@paperclipai/db"; +import type { AdapterExecutionTarget } from "@paperclipai/adapter-utils/execution-target"; +import { parseNativeExecutionInput, type NativeExecutionInput, type PersistedNativeSession } from "../../vendor/paperclip-runner/index.js"; +import { currentNativeControllerIdentity, evaluateNativeControllerTakeover } from "./native-restart-recovery.js"; + +export type RemoteWarmSessionRecovery = { + identity: Record; + checkpoint: PersistedNativeSession; + previousExecution: NativeExecutionInput; + reuseProcess: boolean; + remote: { providerLeaseId: string; remoteCwd: string }; + revalidate: (leaseOwner: string) => Promise; +}; + +export async function inspectRemoteWarmSessionRecovery(input: { + db: Db; + execution: NativeExecutionInput; + target: Extract; + identity: Record; + readControlState: () => Record; + matchesExecution: (previous: NativeExecutionInput) => boolean; + loadCheckpoint: (previous: NativeExecutionInput) => PersistedNativeSession | null; + canReuseProcess: (previous: NativeExecutionInput) => boolean; +}): Promise { + const deny = (): never => { throw new Error("runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate"); }; + const { binding } = input.execution; + const { target } = input; + const identity: Record = {}; + for (const key of ["runId", "normalizedSessionId", "runnerInstanceId", "environmentLeaseId", "turnId", "itemId"]) { + const value = input.identity[key]; + if (typeof value !== "string" || !value) return deny(); + identity[key] = value; + } + if (target.transport !== "sandbox" || !target.leaseId || !target.environmentId || + !target.providerKey || !target.sandboxLeaseAcquisition || target.sandboxLeaseAcquisition.outcome !== "resumed" || + identity.runId === binding.runId) return deny(); + const providerLeaseId = target.sandboxLeaseAcquisition.providerLeaseId; + const [prior] = await input.db.select({ run: heartbeatRuns, owner: nativeRunFinalizations }) + .from(heartbeatRuns).innerJoin(nativeRunFinalizations, eq(nativeRunFinalizations.runId, heartbeatRuns.id)) + .where(and(eq(heartbeatRuns.id, identity.runId), eq(heartbeatRuns.companyId, binding.companyId), + eq(heartbeatRuns.agentId, binding.agentId), eq(heartbeatRuns.nativeIssueId, binding.issueId))); + if (!prior || prior.run.status !== "succeeded" || prior.owner.phase !== "committed" || + !prior.owner.resultId || prior.owner.leaseOwner || !prior.owner.controllerPid || + !prior.owner.controllerProcessStartedAt || !prior.owner.controllerBootId) return deny(); + const previous = parseNativeExecutionInput(prior.run.runnerProfileJson?.nativeExecutionInput); + if (previous.binding.runId !== identity.runId || !input.matchesExecution(previous)) return deny(); + const checkpoint = input.loadCheckpoint(previous); + if (!checkpoint || checkpoint.identity.runId !== identity.runId || + checkpoint.identity.companyId !== binding.companyId || checkpoint.identity.issueId !== binding.issueId || + checkpoint.identity.agentId !== binding.agentId || checkpoint.identity.sessionId !== identity.normalizedSessionId || + checkpoint.driverKind !== previous.session.driverKind || !checkpoint.providerSessionId || + checkpoint.activeTurnId !== null || checkpoint.pendingRuntimeRequests?.length !== 0) return deny(); + const control = input.readControlState(); + const commands = control.commands; + const events = control.committedEvents; + const terminal = Array.isArray(events) ? events.at(-1) : undefined; + if (!Array.isArray(commands) || commands.some(command => command.status !== "completed") || + terminal?.eventType !== "run.terminal" || terminal?.envelope?.runId !== identity.runId || + terminal?.envelope?.normalizedSessionId !== identity.normalizedSessionId || + Object.entries(identity).some(([key, value]) => (control.identity as Record)?.[key] !== value)) return deny(); + const assertPriorStopped = async () => { + const stopped = await evaluateNativeControllerTakeover({ owner: prior.owner, now: new Date(), requireStoppedController: true }); + if (!stopped.allowed) return deny(); + }; + await assertPriorStopped(); + const priorLeases = await input.db.select().from(environmentLeases).where(and( + eq(environmentLeases.companyId, binding.companyId), eq(environmentLeases.heartbeatRunId, identity.runId), + eq(environmentLeases.issueId, binding.issueId), eq(environmentLeases.environmentId, target.environmentId), + eq(environmentLeases.provider, target.providerKey), eq(environmentLeases.providerLeaseId, providerLeaseId), + )); + if (priorLeases.length !== 1 || priorLeases[0].metadata?.remoteCwd !== target.remoteCwd || + !["released", "expired", "active"].includes(priorLeases[0].status)) return deny(); + return { + identity, checkpoint, previousExecution: previous, reuseProcess: input.canReuseProcess(previous), + remote: { providerLeaseId, remoteCwd: target.remoteCwd }, + revalidate: async leaseOwner => { + await assertPriorStopped(); + const controller = await currentNativeControllerIdentity(); + const [current] = await input.db.select({ run: heartbeatRuns, owner: nativeRunFinalizations, issue: issues, agent: agents, company: companies }) + .from(heartbeatRuns).innerJoin(nativeRunFinalizations, eq(nativeRunFinalizations.runId, heartbeatRuns.id)) + .innerJoin(issues, eq(issues.id, heartbeatRuns.nativeIssueId)) + .innerJoin(agents, and(eq(agents.id, heartbeatRuns.agentId), eq(agents.companyId, heartbeatRuns.companyId))) + .innerJoin(companies, eq(companies.id, heartbeatRuns.companyId)) + .where(and(eq(heartbeatRuns.id, binding.runId), eq(heartbeatRuns.companyId, binding.companyId), + eq(heartbeatRuns.agentId, binding.agentId), eq(heartbeatRuns.nativeIssueId, binding.issueId))); + const [lease] = await input.db.select().from(environmentLeases).where(and( + eq(environmentLeases.id, target.leaseId!), eq(environmentLeases.companyId, binding.companyId), + eq(environmentLeases.heartbeatRunId, binding.runId), eq(environmentLeases.issueId, binding.issueId), + )); + const [priorOwner] = await input.db.select().from(nativeRunFinalizations).where(and( + eq(nativeRunFinalizations.runId, identity.runId), eq(nativeRunFinalizations.companyId, binding.companyId), + eq(nativeRunFinalizations.issueId, binding.issueId), + )); + if (!current || current.run.status !== "running" || current.run.resultJson?.nativeCancellation || + current.company.status !== "active" || current.agent.lifecycleState !== "ready" || + ["paused", "terminated", "pending_approval"].includes(current.agent.status) || + !priorOwner || priorOwner.leaseOwner || priorOwner.phase !== "committed" || + priorOwner.resultId !== prior.owner.resultId || priorOwner.controllerBootId !== prior.owner.controllerBootId || + current.owner.leaseOwner !== leaseOwner || !current.owner.leaseExpiresAt || + current.owner.leaseExpiresAt <= new Date() || current.owner.resultId || + current.owner.controllerBootId !== controller.bootId || current.owner.controllerPid !== controller.pid || + current.issue.executionRunId !== binding.runId || current.issue.assigneeAgentId !== binding.agentId || + current.issue.status !== "in_progress" || !lease || lease.status !== "active" || + lease.environmentId !== target.environmentId || lease.provider !== target.providerKey || + lease.providerLeaseId !== providerLeaseId || lease.metadata?.remoteCwd !== target.remoteCwd) return deny(); + }, + }; +} diff --git a/ui/src/components/WorkspaceFileRecovery.test.tsx b/ui/src/components/WorkspaceFileRecovery.test.tsx index 975d7b08072..013c33995e0 100644 --- a/ui/src/components/WorkspaceFileRecovery.test.tsx +++ b/ui/src/components/WorkspaceFileRecovery.test.tsx @@ -7,10 +7,11 @@ import type { ExecutionBlocker, Issue } from "@paperclipai/shared"; import { WorkspaceFileRecovery } from "./WorkspaceFileRecovery"; const resolve = vi.hoisted(() => vi.fn()); vi.mock("../api/issues", () => ({ issuesApi: { resolveRecoveryAction: resolve } })); -const blocker = { recoveryActionId: "held-source", runId: "original-run", cause: "legacy_execution_requires_reconciliation", +const baseBlocker = { recoveryActionId: "held-source", runId: "original-run", workspaceRepairRequired: true, canRecordWorkspaceRepair: true, canRetry: false, canContinue: false, nextAction: "Recover the original files." } as ExecutionBlocker; -describe("workspace file recovery", () => { +describe.each(["legacy_execution_requires_reconciliation", "native_continuation_requires_reconciliation"] as const)("workspace file recovery: %s", cause => { + const blocker = { ...baseBlocker, cause }; let root: Root, container: HTMLDivElement, client: QueryClient; const onRecorded = vi.fn(); beforeEach(() => { @@ -37,7 +38,7 @@ describe("workspace file recovery", () => { await act(async () => Array.from(container.querySelectorAll("button")).find(b => b.textContent === "Record file recovery")!.click()); await act(async () => { await new Promise(resolve => setTimeout(resolve, 10)); }); } - it.each(["in_progress", "in_review", "done", "cancelled"] as Issue["status"][])("records verified source files without changing the %s task", async issueStatus => { + it.each(["blocked", "in_progress", "in_review", "done", "cancelled"] as Issue["status"][])("records verified source files without changing the %s task", async issueStatus => { await mount({ issueStatus }); const button = Array.from(container.querySelectorAll("button")).find(b => b.textContent === "Record file recovery")!; expect(button.disabled).toBe(true); await enterEvidence(); expect(button.disabled).toBe(true); diff --git a/ui/src/components/WorkspaceFileRecovery.tsx b/ui/src/components/WorkspaceFileRecovery.tsx index 2c5f52286ea..e82f6dbad46 100644 --- a/ui/src/components/WorkspaceFileRecovery.tsx +++ b/ui/src/components/WorkspaceFileRecovery.tsx @@ -30,7 +30,8 @@ export function WorkspaceFileRecovery({ issueId, issueStatus, blocker, canManage }), onSuccess: onRecorded, }); - if (!blocker.workspaceRepairRequired || !blocker.canRecordWorkspaceRepair || blocker.cause !== "legacy_execution_requires_reconciliation" + if (!blocker.workspaceRepairRequired || !blocker.canRecordWorkspaceRepair || + !["legacy_execution_requires_reconciliation", "native_continuation_requires_reconciliation"].includes(blocker.cause) || !blocker.recoveryActionId || !blocker.runId) return null; return
{record.isSuccess ?

File recovery recorded. The task status is unchanged. Continue the task when it is ready.

From b1566ca494f4af6e8d4418fa37393c5caee02582 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Thu, 15 Oct 2026 13:35:10 -0500 Subject: [PATCH 18/19] perf(daytona): avoid log-follow delay for completed session commands (#16468) A completed Daytona session command now reads final separated logs without waiting for a new log-follow socket. Running commands retain streaming, recovery, and the existing total deadline. Verified with full repository typecheck/build, standalone provider typecheck/build, 395 passing provider cases, real immutable-image Daytona command controls, and fresh exact-head full CI/E2E and Apex 5/5. Co-Authored-By: Paperclip --- .../sandbox-providers/daytona/README.md | 1 + .../daytona/src/plugin.test.ts | 53 +++++---- .../sandbox-providers/daytona/src/plugin.ts | 16 ++- .../src/session-command-completion.test.ts | 107 ++++++++++++++++++ 4 files changed, 153 insertions(+), 24 deletions(-) create mode 100644 packages/plugins/sandbox-providers/daytona/src/session-command-completion.test.ts diff --git a/packages/plugins/sandbox-providers/daytona/README.md b/packages/plugins/sandbox-providers/daytona/README.md index b384c57549d..76dde8c5897 100644 --- a/packages/plugins/sandbox-providers/daytona/README.md +++ b/packages/plugins/sandbox-providers/daytona/README.md @@ -30,6 +30,7 @@ Notes: - Each cold create uses a unique provider name and ownership labels. If creation fails after Daytona has allocated a sandbox, the driver looks up that exact name, checks every ownership label, and waits for deletion. Failed, missing, or timed-out lookups and failed deletions report unconfirmed cleanup with the provider name; they never return a usable lease. A missing name lookup after an uncertain create is not proof that a delayed provider request cannot create a resource. - For plugin-backed sandbox lease acquisition, unresolved creation cleanup crosses the worker RPC as a validated ownership envelope. The host records `pending_cleanup` before retrying deletion. If only a creation name is known, the provider first returns an ownership-verified sandbox ID without deleting it. The host saves that observation before authorizing deletion. Its existing cleanup sweep and durable spool preserve retries across controller restarts and environment deletion. A missing observed ID confirms cleanup after a lost deletion reply or database update; a name that has never been observed remains unresolved. Provider exceptions and resolved credentials are excluded from that envelope. This requires the matching host and plugin SDK update; probe and custom-image interactive-setup calls still report unconfirmed immediate cleanup without this lease-recovery path. - Reusable leases map to Daytona stop/start semantics. Non-reusable leases are deleted on release. A provider-resolved `target` does not change the identity of an existing sandbox. Release closes the same scoped lease that a later sentinel-verified resume reopens. +- A command that has already exited returns its final separated log snapshot without opening a follow socket. The initial status check is bounded to 250ms and the caller's command deadline. An unfinished command, failed status read, or slow status read keeps the existing streaming and recovery path; the command is dispatched only once. - A session log socket can close while its command still runs. The driver requires a recorded command exit before returning a completed execution. It reconnects the log stream once, then polls status and log snapshots at most once per second. Replayed output is removed by byte offset, new output continues to reach the host, and the command is never dispatched again. A socket that delivers no output for 15 seconds switches directly to polling, even if the socket stays open or its initial connection never completes. Late callbacks from that socket are ignored. This recovers pending permission requests and cancellation results from saved output without replaying the command. Healthy initial and reconnected streams retain their existing lifetime under the caller’s RPC/run guard. After a clean close or an idle socket, each status or snapshot read gets a fresh operation timeout; successful recovery polling does not impose a new command lifetime. When both stream attempts fail without a clean close, fallback retains its original timeout budget starting after the stream attempts. - The SDK returns full log snapshots, so fallback bandwidth grows with retained output; it has no paged snapshot or stream cancellation API. A recovery-read timeout keeps partial output and explicitly reports whether command exit remains unconfirmed; it does not prove the remote command stopped. Provider/session teardown remains responsible for closing an outstanding socket. A final snapshot reconciles bytes missed by the stream before returning a recorded exit. - A sandbox record can survive the loss of its underlying container. Resume treats it as expired only when a fresh provider read confirms the exact missing-container error for that sandbox and marks it unrecoverable. Unknown errors and failed confirmation reads preserve the lease. The host still requires a verified native-runner backup before replacement. diff --git a/packages/plugins/sandbox-providers/daytona/src/plugin.test.ts b/packages/plugins/sandbox-providers/daytona/src/plugin.test.ts index 362021051dc..894a6237c93 100644 --- a/packages/plugins/sandbox-providers/daytona/src/plugin.test.ts +++ b/packages/plugins/sandbox-providers/daytona/src/plugin.test.ts @@ -2461,6 +2461,7 @@ describe("Daytona sandbox provider plugin", () => { process.env.DAYTONA_API_KEY = "host-key"; const sandbox = createMockSandbox(); sandbox.process.getSessionCommand.mockResolvedValue({ id: "cmd-1", command: "", exitCode: 7 }); + sandbox.process.getSessionCommand.mockResolvedValueOnce({ exitCode: undefined }); // A session command tries the log stream first: it delivers stdout and // stderr from the callback log form. sandbox.process.getSessionCommandLogs.mockImplementation( @@ -2561,6 +2562,12 @@ describe("Daytona sandbox provider plugin", () => { }); describe("log stream (default)", () => { + const createStreamingSandbox = () => { + const sandbox = createMockSandbox(); + sandbox.process.getSessionCommand.mockResolvedValueOnce({ exitCode: undefined }); + return sandbox; + }; + // A session command tries the log stream first. It streams stdout and // stderr from the callback log form instead of the 50-ms poll, and falls // back to the poll only when the stream fails. @@ -2577,7 +2584,7 @@ describe("Daytona sandbox provider plugin", () => { it("completes a session command beyond a synchronous gateway deadline without redispatch", async () => { process.env.DAYTONA_API_KEY = "host-key"; - const sandbox = createMockSandbox(); + const sandbox = createStreamingSandbox(); sandbox.process.executeCommand.mockRejectedValue(new MockDaytonaTimeoutError("opaque gateway timeout")); sandbox.process.getSessionCommand.mockResolvedValue({ exitCode: 0 }); sandbox.process.getSessionCommandLogs.mockImplementation( @@ -2614,7 +2621,7 @@ describe("Daytona sandbox provider plugin", () => { it.each([undefined, 0])("preserves the existing healthy session lifetime with command timeout %s", async (commandTimeoutMs) => { process.env.DAYTONA_API_KEY = "host-key"; - const sandbox = createMockSandbox(); + const sandbox = createStreamingSandbox(); sandbox.process.getSessionCommand.mockResolvedValue({ exitCode: 0 }); sandbox.process.getSessionCommandLogs.mockImplementation(async (_s: string, _c: string, stdout?: (chunk: string) => void, stderr?: (chunk: string) => void) => { if (!stdout) return { stdout: "done;", stderr: "" }; @@ -2633,7 +2640,7 @@ describe("Daytona sandbox provider plugin", () => { it("removes deadline listeners from completed reads during a quiet polling run", async () => { process.env.DAYTONA_API_KEY = "host-key"; - const sandbox = createMockSandbox(); + const sandbox = createStreamingSandbox(); sandbox.process.getSessionCommand.mockResolvedValue({ exitCode: undefined }); sandbox.process.getSessionCommandLogs.mockImplementation(async (_s: string, _c: string, stdout?: (chunk: string) => void) => { if (stdout) { stdout("partial;"); return; } @@ -2663,7 +2670,7 @@ describe("Daytona sandbox provider plugin", () => { process.env.DAYTONA_API_KEY = "host-key"; const executionLog = vi.fn(); const restore = __setDaytonaPluginContextForTest({ execution: { log: executionLog } } as unknown as PluginContext); - const sandbox = createMockSandbox(); + const sandbox = createStreamingSandbox(); let lateOutput: ((chunk: string) => void) | undefined; let rejectStream: ((error: Error) => void) | undefined; let keepAlive: ReturnType | undefined; @@ -2718,7 +2725,7 @@ describe("Daytona sandbox provider plugin", () => { const restore = __setDaytonaPluginContextForTest( { execution: { log: executionLog } } as unknown as PluginContext, ); - const sandbox = createMockSandbox(); + const sandbox = createStreamingSandbox(); let lateOutput!: (chunk: string) => void; let rejectStream!: (error: Error) => void; let snapshots = 0; @@ -2772,7 +2779,7 @@ describe("Daytona sandbox provider plugin", () => { it.each(["status", "snapshot"])("bounds a stuck %s read after an idle socket without claiming exit", async (stalledRead) => { process.env.DAYTONA_API_KEY = "host-key"; - const sandbox = createMockSandbox(); + const sandbox = createStreamingSandbox(); let releaseRead!: () => void; const stalled = new Promise((resolve) => { releaseRead = resolve; }); const executionLog = vi.fn(); @@ -2816,7 +2823,7 @@ describe("Daytona sandbox provider plugin", () => { it("streams ordered stdout and stderr from the callback log form (test_log_stream_delivers_ordered_chunks)", async () => { process.env.DAYTONA_API_KEY = "host-key"; - const sandbox = createMockSandbox(); + const sandbox = createStreamingSandbox(); sandbox.process.getSessionCommand.mockResolvedValue({ id: "cmd-1", command: "", exitCode: 0 }); // The callback form emits stdout and stderr chunks in order. The plugin // keeps each stream in its own arrival order. @@ -2853,7 +2860,7 @@ describe("Daytona sandbox provider plugin", () => { it("reads the exit code once after the stream ends (test_log_stream_reads_exit_code_once_after_stream_end)", async () => { process.env.DAYTONA_API_KEY = "host-key"; - const sandbox = createMockSandbox(); + const sandbox = createStreamingSandbox(); sandbox.process.getSessionCommand.mockResolvedValue({ id: "cmd-1", command: "", exitCode: 5 }); sandbox.process.getSessionCommandLogs.mockImplementation( async (_sid: string, _cmdId: string, onStdout?: (chunk: string) => void) => { @@ -2865,7 +2872,7 @@ describe("Daytona sandbox provider plugin", () => { const result = await plugin.definition.onEnvironmentExecute?.(streamExecParams()); // The exit code read runs one time after the stream promise resolves. - expect(sandbox.process.getSessionCommand).toHaveBeenCalledTimes(1); + expect(sandbox.process.getSessionCommand).toHaveBeenCalledTimes(2); expect(result).toMatchObject({ exitCode: 5, timedOut: false, stdout: "done" }); }); @@ -2876,7 +2883,7 @@ describe("Daytona sandbox provider plugin", () => { { execution: { log: executionLog } } as unknown as PluginContext, ); try { - const sandbox = createMockSandbox(); + const sandbox = createStreamingSandbox(); let streams = 0; sandbox.process.getSessionCommandLogs.mockImplementation( async (_sid: string, _cmdId: string, onStdout?: (chunk: string) => void) => { @@ -2910,7 +2917,7 @@ describe("Daytona sandbox provider plugin", () => { { execution: { log: executionLog } } as unknown as PluginContext, ); try { - const sandbox = createMockSandbox(); + const sandbox = createStreamingSandbox(); let snapshots = 0; sandbox.process.getSessionCommandLogs.mockImplementation( async (_sid: string, _cmdId: string, onStdout?: (chunk: string) => void) => { @@ -2949,7 +2956,7 @@ describe("Daytona sandbox provider plugin", () => { it.each(["close", "reject"])("preserves healthy stream lifetimes after an aged %s", async (ending) => { process.env.DAYTONA_API_KEY = "host-key"; - const sandbox = createMockSandbox(); + const sandbox = createStreamingSandbox(); let streams = 0; sandbox.process.getSessionCommandLogs.mockImplementation( async (_sid: string, _cmdId: string, onStdout?: (chunk: string) => void, onStderr?: (chunk: string) => void) => { @@ -2985,7 +2992,7 @@ describe("Daytona sandbox provider plugin", () => { it("polls an aged running command after repeated EOF without inventing an exit or lifetime cap", async () => { process.env.DAYTONA_API_KEY = "host-key"; - const sandbox = createMockSandbox(); + const sandbox = createStreamingSandbox(); let streams = 0; let snapshots = 0; let exited = false; @@ -3018,7 +3025,7 @@ describe("Daytona sandbox provider plugin", () => { it.each([0, 3_600_000])("retains the fallback budget and partial output after a failed stream lasting %i ms", async (streamLifetime) => { process.env.DAYTONA_API_KEY = "host-key"; - const sandbox = createMockSandbox(); + const sandbox = createStreamingSandbox(); let streams = 0; sandbox.process.getSessionCommandLogs.mockImplementation( async (_sid: string, _cmdId: string, onStdout?: (chunk: string) => void, onStderr?: (chunk: string) => void) => { @@ -3051,7 +3058,7 @@ describe("Daytona sandbox provider plugin", () => { it("preserves provider status errors after a clean stream close", async () => { process.env.DAYTONA_API_KEY = "host-key"; - const sandbox = createMockSandbox(); + const sandbox = createStreamingSandbox(); sandbox.process.getSessionCommandLogs.mockResolvedValue(undefined); sandbox.process.getSessionCommand.mockRejectedValue(new Error("session no longer exists")); mockGet.mockResolvedValue(sandbox); @@ -3063,7 +3070,7 @@ describe("Daytona sandbox provider plugin", () => { it("reads bytes written after socket close before the exit code becomes visible", async () => { process.env.DAYTONA_API_KEY = "host-key"; - const sandbox = createMockSandbox(); + const sandbox = createStreamingSandbox(); sandbox.process.getSessionCommandLogs.mockImplementation( async (_sid: string, _cmdId: string, onStdout?: (chunk: string) => void) => { if (onStdout) onStdout("prefix;"); @@ -3089,7 +3096,7 @@ describe("Daytona sandbox provider plugin", () => { ); vi.useFakeTimers(); try { - const sandbox = createMockSandbox(); + const sandbox = createStreamingSandbox(); let streams = 0; let lateOutput: ((chunk: string) => void) | undefined; sandbox.process.getSessionCommandLogs.mockImplementation( @@ -3141,7 +3148,7 @@ describe("Daytona sandbox provider plugin", () => { it.each(["status", "final snapshot", "fallback snapshot"])("preserves observation state when the SDK %s times out", async (stage) => { process.env.DAYTONA_API_KEY = "host-key"; - const sandbox = createMockSandbox(); + const sandbox = createStreamingSandbox(); sandbox.process.getSessionCommandLogs.mockImplementation( async (_sid: string, _cmdId: string, onStdout?: (chunk: string) => void, onStderr?: (chunk: string) => void) => { if (!onStdout) throw new MockDaytonaTimeoutError("provider timeout"); @@ -3178,7 +3185,7 @@ describe("Daytona sandbox provider plugin", () => { ); vi.useFakeTimers(); try { - const sandbox = createMockSandbox(); + const sandbox = createStreamingSandbox(); sandbox.process.getSessionCommandLogs.mockImplementation( async (_sid: string, _cmdId: string, onStdout?: (chunk: string) => void, onStderr?: (chunk: string) => void) => { onStdout?.("A\uFFFD"); @@ -3217,7 +3224,7 @@ describe("Daytona sandbox provider plugin", () => { { execution: { log: executionLog } } as unknown as PluginContext, ); try { - const sandbox = createMockSandbox(); + const sandbox = createStreamingSandbox(); let streams = 0; sandbox.process.getSessionCommandLogs.mockImplementation( async (_sid: string, _cmdId: string, onStdout?: (chunk: string) => void, onStderr?: (chunk: string) => void) => { @@ -3256,7 +3263,7 @@ describe("Daytona sandbox provider plugin", () => { it("falls back to the poll path when the stream promise rejects (test_log_stream_disconnect_rejects_and_falls_back_to_poll)", async () => { process.env.DAYTONA_API_KEY = "host-key"; - const sandbox = createMockSandbox(); + const sandbox = createStreamingSandbox(); // The callback form disconnects and rejects. The snapshot form (no // callbacks) serves the poll fallback read. sandbox.process.getSessionCommandLogs.mockImplementation( @@ -3290,7 +3297,7 @@ describe("Daytona sandbox provider plugin", () => { it("drops the replayed prefix by byte offset on a reconnect (test_log_stream_reconnect_drops_replayed_prefix_by_byte_offset)", async () => { process.env.DAYTONA_API_KEY = "host-key"; - const sandbox = createMockSandbox(); + const sandbox = createStreamingSandbox(); let attempt = 0; sandbox.process.getSessionCommandLogs.mockImplementation( async ( @@ -3331,7 +3338,7 @@ describe("Daytona sandbox provider plugin", () => { { execution: { log: executionLog } } as unknown as PluginContext, ); try { - const sandbox = createMockSandbox(); + const sandbox = createStreamingSandbox(); let attempt = 0; sandbox.process.getSessionCommandLogs.mockImplementation( async ( diff --git a/packages/plugins/sandbox-providers/daytona/src/plugin.ts b/packages/plugins/sandbox-providers/daytona/src/plugin.ts index b8e71c2ab8f..c9276cf7942 100644 --- a/packages/plugins/sandbox-providers/daytona/src/plugin.ts +++ b/packages/plugins/sandbox-providers/daytona/src/plugin.ts @@ -1808,6 +1808,7 @@ async function executeOneShot( // Fallback reads full log snapshots as well as status. Limit the heavier // polling path while keeping output available to interactive commands. const SESSION_LOG_POLL_INTERVAL_MS = 1000; +const SESSION_INITIAL_STATUS_TIMEOUT_MS = 250; // A socket can stay open after it stops delivering output. Switch to saved // logs after this much silence; silence never proves that the command exited. const SESSION_LOG_STREAM_IDLE_MS = 15_000; @@ -2189,6 +2190,17 @@ async function executeInSession( timeoutSeconds, )); const commandId = dispatched.cmdId; + let initialExitCode: number | null = null; + try { + const status = await observeSessionCommand( + Math.min(effectiveTimeoutMs, SESSION_INITIAL_STATUS_TIMEOUT_MS), + () => sandbox.process.getSessionCommand(sessionId, commandId), + commandDeadline, + ); + if (typeof status.exitCode === "number") initialExitCode = status.exitCode; + } catch (error) { + if (error instanceof SessionCommandDeadlineError) throw error; + } // Log-stream path. A session command always tries the stream first: it // streams stdout and stderr from the callback log form, then reads the exit @@ -2199,7 +2211,9 @@ async function executeInSession( // call. The host routes it to the runner log sink by the host-issued // invocation id. This is a no-op when no plugin context is set (a direct // test call) or when the host has no active execute route. - const streamResult = await runSessionLogStream( + const streamResult = initialExitCode !== null + ? { exitCode: initialExitCode, pollUntilExit: false, buffer: streamBuffer } + : await runSessionLogStream( sandbox, sessionId, commandId, diff --git a/packages/plugins/sandbox-providers/daytona/src/session-command-completion.test.ts b/packages/plugins/sandbox-providers/daytona/src/session-command-completion.test.ts new file mode 100644 index 00000000000..47ad191601f --- /dev/null +++ b/packages/plugins/sandbox-providers/daytona/src/session-command-completion.test.ts @@ -0,0 +1,107 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; + +const getSandbox = vi.hoisted(() => vi.fn()); +vi.mock("@daytonaio/sdk", () => ({ + Daytona: class { get = getSandbox; }, + DaytonaNotFoundError: class extends Error {}, + DaytonaTimeoutError: class extends Error {}, +})); + +import plugin, { __resetDaytonaSandboxHandleCacheForTest } from "./plugin.js"; + +function fixture() { + const process = { + createSession: vi.fn().mockResolvedValue(undefined), + executeSessionCommand: vi.fn().mockResolvedValue({ cmdId: "fixture-command" }), + getSessionCommand: vi.fn<(...args: string[]) => Promise<{ exitCode: number | undefined }>>().mockResolvedValue({ exitCode: 0 }), + getSessionCommandLogs: vi.fn<(session: string, command: string, stdout?: (chunk: string) => void, stderr?: (chunk: string) => void) => Promise<{ stdout: string; stderr: string } | void>>().mockImplementation(async (_session, _command, stdout) => { + if (stdout) return new Promise(() => {}); + return { stdout: "complete stdout\n", stderr: "complete stderr\n" }; + }), + }; + getSandbox.mockResolvedValue({ id: "fixture-sandbox", state: "started", process }); + const execute = (commandTimeoutMs = 2_000) => plugin.definition.onEnvironmentExecute!({ + driverKey: "daytona", companyId: "fixture-company", environmentId: "fixture-environment", + config: { apiKey: "fixture-not-a-key", timeoutMs: 30_000 }, + lease: { providerLeaseId: "fixture-sandbox" }, command: "printf", args: ["fixture"], + commandTimeoutMs, + }); + return { process, execute }; +} + +describe("Daytona completed session commands", () => { + beforeEach(() => { + __resetDaytonaSandboxHandleCacheForTest(); + getSandbox.mockReset(); + vi.useFakeTimers(); + }); + afterEach(() => { + vi.useRealTimers(); + __resetDaytonaSandboxHandleCacheForTest(); + }); + + it("returns a finished command's final separated logs without opening a follow socket", async () => { + const { process, execute } = fixture(); + let settled = false; + const pending = execute().then(result => { settled = true; return result; }); + await vi.advanceTimersByTimeAsync(500); + expect(settled).toBe(true); + expect(await pending).toMatchObject({ exitCode: 0, timedOut: false, stdout: "complete stdout\n", stderr: "complete stderr\n" }); + expect(process.getSessionCommandLogs).toHaveBeenCalledExactlyOnceWith(expect.any(String), "fixture-command"); + expect(process.executeSessionCommand).toHaveBeenCalledTimes(1); + }); + + it("streams a still-running command and reconciles its final snapshot", async () => { + const { process, execute } = fixture(); + process.getSessionCommand.mockResolvedValueOnce({ exitCode: undefined }); + process.getSessionCommandLogs.mockImplementation(async (_session, _command, stdout) => { + if (stdout) { stdout("prefix "); return; } + return { stdout: "prefix complete\n", stderr: "error\n" }; + }); + expect(await execute()).toMatchObject({ exitCode: 0, timedOut: false, stdout: "prefix complete\n", stderr: "error\n" }); + expect(process.getSessionCommandLogs.mock.calls.filter(call => call[2])).toHaveLength(1); + }); + + it("bounds an unavailable initial status read and retains the working stream", async () => { + const { process, execute } = fixture(); + process.getSessionCommand.mockImplementationOnce(() => new Promise(() => {})); + process.getSessionCommandLogs.mockImplementation(async (_session, _command, stdout) => { + if (stdout) { stdout("complete stdout\n"); return; } + return { stdout: "complete stdout\n", stderr: "complete stderr\n" }; + }); + const pending = execute(); + await vi.advanceTimersByTimeAsync(300); + expect(await pending).toMatchObject({ exitCode: 0, timedOut: false, stdout: "complete stdout\n", stderr: "complete stderr\n" }); + expect(process.getSessionCommandLogs.mock.calls.filter(call => call[2])).toHaveLength(1); + }); + + it("retains stream recovery after the initial status request rejects", async () => { + const { process, execute } = fixture(); + process.getSessionCommand.mockRejectedValueOnce(new Error("status transport unavailable")); + process.getSessionCommandLogs.mockImplementation(async (_session, _command, stdout) => { + if (stdout) { stdout("complete stdout\n"); return; } + return { stdout: "complete stdout\n", stderr: "complete stderr\n" }; + }); + expect(await execute()).toMatchObject({ exitCode: 0, timedOut: false, stdout: "complete stdout\n", stderr: "complete stderr\n" }); + expect(process.executeSessionCommand).toHaveBeenCalledTimes(1); + }); + + it("does not treat a visible exit as proof that final logs were read", async () => { + const { process, execute } = fixture(); + process.getSessionCommandLogs.mockImplementation(() => new Promise(() => {})); + const pending = execute(100); + await vi.advanceTimersByTimeAsync(101); + expect(await pending).toMatchObject({ exitCode: null, timedOut: true, + metadata: { timeoutScope: "session_log_observation", commandExitConfirmed: true, observedExitCode: 0 } }); + }); + + it("keeps the total command deadline while the initial status is unavailable", async () => { + const { process, execute } = fixture(); + process.getSessionCommand.mockImplementation(() => new Promise(() => {})); + const pending = execute(100); + await vi.advanceTimersByTimeAsync(101); + expect(await pending).toMatchObject({ exitCode: null, timedOut: true }); + expect(process.executeSessionCommand).toHaveBeenCalledTimes(1); + expect(process.getSessionCommandLogs).not.toHaveBeenCalled(); + }); +}); From 5ec93dae014c16ff33953fb0fb0274e4efc7819f Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Thu, 15 Oct 2026 14:08:01 -0500 Subject: [PATCH 19/19] fix(connectors): avoid redundant run preparation discovery (#16461) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Run preparation selects connector skills and tools under current permissions. > - An assignment check can build an unrelated company tool catalog and wait for a shared discovery slot. > - Native startup repeats assignment discovery to obtain static tool definitions. > - Slack also locks and reconciles an unchanged catalog, then repeats task authorization. > - This pull request removes that redundant startup work and preserves live checks at tool execution. > - The benefit is less preparation work, less contention, and unchanged permission and approval enforcement. ## Linked Issues or Issue Description Related: #16454 (detailed preparation tracing), #16425 (workspace preparation performance). **What happened?** Connector configuration added avoidable work to run preparation. Browser Use assignment authorized the whole company catalog, including other connector families. Four connector resolvers ran in series. Native controller startup repeated all assignment resolution when any connector skill was pinned. Slack reconciled an unchanged catalog under an advisory lock and checked task authority twice. **Expected behavior** Prepare only the connector data needed by the run. Reuse static tool definitions from pinned skills. Preserve current company, assignment, identity, profile, policy, grant, and approval checks when an operation executes. **Steps to reproduce** 1. Configure Browser Use and an unrelated MCP catalog with 500 tools for one company. 2. Resolve Browser Use resources for an authorized task. The old path reads and authorizes the unrelated catalog. 3. Occupy both full discovery slots. The old assignment check waits for those slots. 4. Resolve a linked Slack task repeatedly with an existing current catalog. The old path still acquires a catalog lock and repeats authority checks. **Paperclip version or commit** Base: `68a09dcb5f50e3fb6ceaaeac2706d96db97fc9b3`. **Deployment mode** Observed in hosted native runs. The local reproduction uses a source checkout and real embedded PostgreSQL. ## What Changed - Restrict Browser Use catalog reads and policy decisions to active Browser Use connections. Preserve guest restrictions, reviewed catalog filters, live grants, and deterministic resource order. Keep assignment checks out of the full discovery queue. - Resolve the four independent connector families concurrently. Wait for outstanding work to settle before propagating a failure, and retain registration order. - Register native direct connector tool definitions from pinned connector skills. Keep live assignment resolution and governed execution on every tool call. - Compare Slack catalog hashes before reconciliation. Current catalogs skip the catalog transaction and lock. Check task authority once after current profile checks. Initial setup and upgrades retain locked reconciliation and checks before and after it. - Precompute immutable bundled Slack descriptor hashes. Preserve disabled entries, profile state, removed bindings, and review metadata. - Document the startup paths and add focused contention, revocation, catalog upgrade, and failure-settlement regressions. - Repair a pre-existing CI guard false positive: the filename `docker-compose-secrets.test.mjs` matched `secrets.`. Check GitHub secret expressions instead, with dot, bracket, and whole-context controls. No workflow changes. ## Verification - Conflict resolution: merge `b1566ca494f4af6e8d4418fa37393c5caee02582` from master into head `55140b1609ae7bd21a038eb75a48c2683299e4b4`. Combine both secret-expression guards and regression cases. Preserve current workspace, native recovery, and Daytona changes. All 467 policy tests, 787 focused native/connector/Browser Use tests, and server typecheck passed locally after integration. Fresh full CI passed all 55 checks on this head, with two optional Storybook jobs skipped. Greptile reviewed this exact head at 5/5 with no unresolved threads. - `pnpm -r typecheck` — passed. - `pnpm build` — passed. - Focused native startup, connector concurrency, and AgentMail suites — 763 tests passed. - Complete Browser Use suite — 33 tests passed. Focused Slack catalog, linked task, retained provenance/revocation, and pinned native tool tests passed. The complete runner tool authority suite passed (26 tests). The 1,337-case chat suite passed 1,335 tests and had two failures: the original new transaction-count assertion and a 15-second timeout in the publication-batch case. Both cases passed an isolated rerun. The publication timeout occurred during concurrent local checks; the passing rerun does not erase it. CI must still pass the chat shards. - The first focused run found two incorrect new assertions: policy evaluation performs a second catalog lookup, and an explicit allow policy overrides a profile default. The tests now distinguish the scoped catalog query and revoke the governing policy. The focused Browser Use rerun passed all 33 cases after these assertion fixes. - Full-suite test verification used the CI shards of `pnpm test:run`. Local runs focused on the changed boundaries; the complete repository test command was not duplicated locally. - `node --test '.github/scripts/tests/*.test.mjs'` — all 467 policy tests passed. The CI failure reproduced on the unchanged base guard before this correction. Actual GitHub secret expressions still match the guard. - CI on the final head passed all chat shards, typecheck, build, Runner checks, browser shards, and other test lanes. One existing workspace-runtime case hit `EADDRINUSE` for a randomly selected test listener in server shard 1/12 (1,162 tests passed in that shard). The shard passed its single retry on the same SHA with no source changes. The first failure remains recorded here. Previous reviewed head `7191ac40691ddb1ef38b86665c95cc14c57ea390`: all 55 checks passed, two optional Storybook jobs skipped, and no pending or failed checks. Greptile is 5/5 on this head with no unresolved review threads. The external security scan passed. Local before/after measurements use the old and new implementations against the same fixture. Both paths are warmed. Seven samples alternate execution order. The database is real embedded PostgreSQL. No latency is injected. These measurements do not establish hosted speed. | Local path | Before median (range) | After median (range) | Work reduction | | --- | --- | --- | --- | | Browser Use assignment with 500 unrelated MCP tools | 44.020 ms (33.994–107.328) | 26.502 ms (20.385–99.037) | 26 → 22 statements; unrelated catalog and discovery queue removed | | Slack assignment with current catalog and linked responsible user | 49.723 ms (43.624–80.653) | 22.141 ms (19.781–30.818) | 5 → 2 database transactions; no catalog lock on the current path | The contention regression fails against the old implementation after a bounded five-second wait, then passes against the optimized implementation. It occupies the real global discovery scheduler and verifies that Browser Use assignment completes without releasing those slots. It records catalog SQL to verify connection scope. Revocation cases remove policy, catalog availability, and grants between calls. Existing Slack integration cases exercise retained provenance, removed profile bindings, identity revocation, approved writes, and current responsible-user restrictions. ## Risks - Pinned native tool definitions can remain visible after access is revoked. They confer no authority. Invocation rechecks current assignments and fails closed before an operation executes. Newly available connector families still require a newly admitted skill selection. - Concurrent resolution can increase instantaneous database reads to four connector families per configuration call. It is a fixed bound. Outstanding work settles before a failed preparation returns. - Slack skips reconciliation only when all bundled version hashes and an existing profile match. Missing or changed definitions still use the existing locked path. Operator revocations remain effective. - This PR addresses connector preparation. Workspace transfer, provider startup, copyback, and queue delay can still keep total latency above the user's goal. Hosted improvement needs a deployed before/after trace. - No schema, lockfile, workflow, telemetry event, or authorization cache changes. ## Model Used OpenAI GPT-6 (Codex), with reasoning, repository inspection, code execution, and GitHub tools. The runtime does not expose the exact serving model ID or context window. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- .github/scripts/tests/ci-profile.test.mjs | 5 +- doc/observability.md | 25 ++++++++- .../__tests__/browser-use-connection.test.ts | 54 +++++++++++++++++++ .../chat-channels.integration.test.ts | 12 ++++- server/src/services/connector-runtime.test.ts | 46 ++++++++++++++++ server/src/services/connector-runtime.ts | 30 ++++++----- .../services/connectors/slack-authority.ts | 2 +- .../src/services/connectors/slack-catalog.ts | 43 ++++++++++----- server/src/services/connectors/slack.ts | 45 ++++++++-------- .../native-runtime/native-session-executor.ts | 7 ++- .../paperclip-runner-tool-authority.test.ts | 13 +++++ .../paperclip-runner-tool-authority.ts | 2 +- server/src/services/tool-gateway.ts | 28 +++++++--- 13 files changed, 247 insertions(+), 65 deletions(-) create mode 100644 server/src/services/connector-runtime.test.ts diff --git a/.github/scripts/tests/ci-profile.test.mjs b/.github/scripts/tests/ci-profile.test.mjs index f113d0ad87d..313e8ece372 100644 --- a/.github/scripts/tests/ci-profile.test.mjs +++ b/.github/scripts/tests/ci-profile.test.mjs @@ -241,11 +241,12 @@ test("workflow exporter runs trusted code and keeps keys out of the PR jobs", () const pr = readFileSync(new URL("../../workflows/pr-trusted.yml", import.meta.url), "utf8"); // A script filename such as docker-compose-secrets.test.mjs is not a // GitHub secret lookup. Check expression context, including bracket access. - const secretReference = /HONEYCOMB.*KEY|\$\{\{[^}]*\bsecrets\s*(?:\.|\[)/; + const secretReference = /HONEYCOMB.*KEY|\$\{\{[^}]*\bsecrets\b/; assert.doesNotMatch("run: node --test scripts/docker-compose-secrets.test.mjs", secretReference); - for (const lookup of ["${{ secrets.PR_TOKEN }}", "${{ secrets['PR_TOKEN'] }}", "${{\n secrets.PR_TOKEN\n }}"]) { + for (const lookup of ["${{ secrets.PR_TOKEN }}", "${{ secrets['PR_TOKEN'] }}", "${{\n secrets.PR_TOKEN\n }}", "${{ toJSON(secrets) }}", "${{ secrets.HONEYCOMB_API_KEY }}"]) { assert.match(lookup, secretReference); } + assert.doesNotMatch(pr, secretReference); assert.match(pr, /if: always\(\) && env.PAPERCLIP_CI_PROFILE == 'true'/); assert.doesNotMatch(pr, /PAPERCLIP_CI_PROFILE:.*labels/); diff --git a/doc/observability.md b/doc/observability.md index 48f417cc2dc..bf93d0d7c9e 100644 --- a/doc/observability.md +++ b/doc/observability.md @@ -279,7 +279,8 @@ attempt or owning phase: Detail phases emit at most 16 spans of each name per attempt. Larger reader and asset inventories continue under the containing group without unbounded per-row -or per-file spans. Work, query order, authorization, and locks are unchanged. +or per-file spans. Adding detail spans does not itself change work, query order, +authorization, or locks. Manual transaction and dispatch boundaries retain their actual lifetime. Asset writes and verification may add only non-negative integer `paperclip.execution.fileCount` and `paperclip.execution.bytes` attributes. @@ -295,6 +296,28 @@ can continue through later bookkeeping and resource cleanup. A later recovery invocation has its own attempt; this does not manufacture one continuous trace across queueing, process restarts, or an unavailable controller. +### Connector preparation work + +Run configuration resolves the four connector families concurrently and preserves +registration order. Browser Use assignment reads only active Browser Use +connections and their eligible catalog entries. It applies the same tool policy, +grant, company, and restricted GitHub guest checks without joining the full tool +discovery queue or resolving unrelated Slack, GitHub, and Speko tools. + +Native startup registers direct connector tool definitions from the run's pinned +skills. It does not repeat assignment discovery. These definitions do not grant +access: each invocation still resolves current assignments and enters the existing +permission and approval gateway. A retained session may advertise a tool whose +access was revoked; the call fails under the current authorization state. + +Slack assignment checks stored catalog hashes against the bundled definitions. +A current catalog and existing profile need no catalog transaction or advisory +lock. Current profile bindings and entries still determine availability, followed +by one live task authority check. Missing or changed catalog definitions use the +existing locked reconciliation path, with authority checks before and after the +update. Disabled catalog entries and operator-removed bindings remain disabled. +No authorization result is cached across calls. + ### Reading a slow startup in Honeycomb Filter `paperclip.task.run.run_id` to the first 12 hex characters of diff --git a/server/src/__tests__/browser-use-connection.test.ts b/server/src/__tests__/browser-use-connection.test.ts index 90a227c55ad..87092b07a85 100644 --- a/server/src/__tests__/browser-use-connection.test.ts +++ b/server/src/__tests__/browser-use-connection.test.ts @@ -22,6 +22,7 @@ import { issues, toolConnections, toolApplications, + toolCatalogEntries, toolConnectionInstalls, toolProfiles, toolProfileBindings, @@ -419,6 +420,59 @@ const actor = { actorType: "user" as const, actorId: "browser-reviewer" }; expect(f.request).not.toHaveBeenCalled(); } finally { discovery.mockRestore(); } }); + it("resolves only the Browser Use catalog even while full discovery is saturated", async () => { + const f = await fixture(); + const [application] = await db.insert(toolApplications).values({ + companyId: f.company.id, name: "Unrelated MCP", type: "mcp_http", status: "active", + }).returning(); + const [connection] = await db.insert(toolConnections).values({ + companyId: f.company.id, applicationId: application.id, name: "Unrelated MCP", + uid: randomUUID(), transport: "mcp_remote", status: "active", enabled: true, healthStatus: "ok", + }).returning(); + await db.insert(toolCatalogEntries).values(Array.from({ length: 500 }, (_, i) => ({ + companyId: f.company.id, applicationId: application.id, connectionId: connection.id, + entryKind: "tool" as const, name: `unrelated_${i}`, toolName: `unrelated_${i}`, + inputSchema: { type: "object" }, riskLevel: "read" as const, status: "active" as const, + versionHash: "fixture", schemaHash: "fixture", + }))); + const { recordingDb } = await import("./helpers/tool-gateway-listing-fixture.js"); + const { toolDiscoveryScheduler } = await import("../services/tool-discovery-scheduler.js"); + const releases: Array<() => void> = []; + const blockers = [0, 1].map(() => toolDiscoveryScheduler.run(() => new Promise(resolve => releases.push(resolve)))); + await vi.waitFor(() => expect(releases).toHaveLength(2)); + const recorded = recordingDb(db); + const gateway = createToolGatewayService(recorded.db); + const pending = gateway.browserUseResources(f.binding); + let timeout: ReturnType | undefined; + try { + const resources = await Promise.race([pending, new Promise((_, reject) => { + timeout = setTimeout(() => reject(new Error("Assignment waited for unrelated discovery")), 5_000); + })]); + expect(resources).toEqual([{ id: f.connection.connectionId, connectionId: f.connection.connectionId, label: "Browser Use Cloud" }]); + const catalogReads = recorded.statements.flatMap((query, index) => + query.includes('from "tool_catalog_entries"') && query.includes('inner join "tool_connections"') ? [recorded.statementParams[index]] : []); + expect(catalogReads).toHaveLength(1); + expect(catalogReads[0]).toContain(f.connection.connectionId); + expect(catalogReads[0]).not.toContain(connection.id); + expect(recorded.statements.some(query => query.includes('from "chat_endpoints"'))).toBe(false); + } finally { + clearTimeout(timeout); + releases.forEach(release => release()); + await Promise.all(blockers); + await pending; + } + }); + it.each(["policy", "catalog", "grant"])("rechecks Browser Use %s revocation without a cached assignment", async kind => { + const f = await fixture(); + expect(await f.gateway.browserUseResources(f.binding)).toHaveLength(1); + if (kind === "policy") await db.insert(toolPolicies).values({ + companyId: f.company.id, name: "Revoke Browser Use", policyType: "block", + selectors: { connectionId: f.connection.connectionId }, priority: 100, + }); + if (kind === "catalog") await db.update(toolCatalogEntries).set({ quarantinedAt: new Date() }).where(eq(toolCatalogEntries.connectionId, f.connection.connectionId)); + if (kind === "grant") await db.update(connectionGrants).set({ status: "revoked" }).where(eq(connectionGrants.id, f.grant.id)); + expect(await f.gateway.browserUseResources(f.binding)).toEqual([]); + }); it("withholds Cloud instructions when connection tools are disabled", async () => { const f = await fixture(); await db.update(toolConnections).set({ enabled: false }).where(eq(toolConnections.id, f.connection.connectionId)); diff --git a/server/src/__tests__/chat-channels.integration.test.ts b/server/src/__tests__/chat-channels.integration.test.ts index b6d5f28c29f..231c24e3832 100644 --- a/server/src/__tests__/chat-channels.integration.test.ts +++ b/server/src/__tests__/chat-channels.integration.test.ts @@ -8096,10 +8096,12 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => { it("refreshes Slack tool guidance without restoring disabled catalog policies", async () => { const { SLACK_TOOLS } = await import("@paperclipai/shared"); - const { syncSlackBotTools } = await import("../services/connectors/slack-catalog.js"); + const { slackBotCatalogIsCurrent, syncSlackBotTools } = await import("../services/connectors/slack-catalog.js"); const fixture = await seedCompany(); const { endpoint } = await configuredSlackEndpoint(fixture); + expect(await slackBotCatalogIsCurrent(db, endpoint)).toBe(false); await db.transaction(tx => syncSlackBotTools(tx, endpoint, "owner-user", true)); + expect(await slackBotCatalogIsCurrent(db, endpoint)).toBe(true); const tool = SLACK_TOOLS.find(tool => tool.name === "slack_post_message")!; // Simulate the previous hash contract, which omitted descriptions. const oldHash = createHash("sha256").update(JSON.stringify({ name: tool.name, inputSchema: tool.inputSchema, risk: tool.risk, scopes: tool.scopes })).digest("hex"); @@ -8108,7 +8110,9 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => { description: "Old posting guidance", versionHash: oldHash, schemaHash: oldHash, status: "removed", reviewedByUserId: "owner-user", reviewedAt, }).where(and(eq(toolCatalogEntries.connectionId, endpoint.connectionId), eq(toolCatalogEntries.toolName, tool.name))); + expect(await slackBotCatalogIsCurrent(db, endpoint)).toBe(false); await db.transaction(tx => syncSlackBotTools(tx, endpoint, "different-operator", true)); + expect(await slackBotCatalogIsCurrent(db, endpoint)).toBe(true); const [updated] = await db.select().from(toolCatalogEntries).where(and(eq(toolCatalogEntries.connectionId, endpoint.connectionId), eq(toolCatalogEntries.toolName, tool.name))); expect(updated).toMatchObject({ description: tool.description, status: "removed", reviewedByUserId: "owner-user", reviewedAt }); expect(updated!.versionHash).not.toBe(oldHash); @@ -8132,6 +8136,12 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => { expect(assignments.map(item => item.key)).toEqual(["slack"]); expect(assignments[0].resources[0]).toMatchObject({ id: endpoint.id, metadata: { requesterId: "UROUTINE" } }); expect(assignments[0].tools.some(tool => tool.name === "slack_open_dm")).toBe(true); + const transactions = vi.spyOn(db, "transaction"); + try { + expect(await resolveConnectorAssignments(db, binding)).toEqual(assignments); + expect(transactions).toHaveBeenCalledTimes(2); + } finally { transactions.mockRestore(); } + await expect(resolveSlackTaskAuthority(db, binding)).resolves.toMatchObject({ conversation: null, userId: "owner-user", slackUserId: "UROUTINE" }); const upstream = vi.fn(async (input: RequestInfo | URL, init?: RequestInit) => { const method = String(input).split("/").at(-1); diff --git a/server/src/services/connector-runtime.test.ts b/server/src/services/connector-runtime.test.ts new file mode 100644 index 00000000000..a39667594b3 --- /dev/null +++ b/server/src/services/connector-runtime.test.ts @@ -0,0 +1,46 @@ +import { afterEach, describe, expect, it, vi } from "vitest"; +import type { Db } from "@paperclipai/db"; +import { resolveConnectorAssignments } from "./connector-runtime.js"; + +const resolvers = vi.hoisted(() => ({ + browser: vi.fn(), speko: vi.fn(), slack: vi.fn(), inboxes: vi.fn(), +})); +vi.mock("./native-runtime/assigned-mcp-tools.js", () => ({ getAssignedMcpGateway: () => ({ browserUseResources: resolvers.browser }) })); +vi.mock("./voice/speko-agent-tools.js", () => ({ spekoAssignedResources: resolvers.speko })); +vi.mock("./connectors/slack.js", () => ({ slackAssignedResource: resolvers.slack, executeGovernedSlackTool: vi.fn() })); +vi.mock("./email-channels.js", () => ({ emailChannelService: () => ({ assignedInboxes: resolvers.inboxes }) })); + +afterEach(() => vi.resetAllMocks()); + +function deferred() { + let resolve!: (value: T) => void; + let reject!: (error: Error) => void; + const promise = new Promise((yes, no) => { resolve = yes; reject = no; }); + return { promise, resolve, reject }; +} + +describe("connector assignment preparation", () => { + it("starts independent resolvers together and retains connector order", async () => { + const pending = Array.from({ length: 4 }, () => deferred>()); + Object.values(resolvers).forEach((resolver, i) => resolver.mockReturnValue(pending[i].promise)); + const assignments = resolveConnectorAssignments({} as Db, { companyId: "company", agentId: "agent" }); + await vi.waitFor(() => Object.values(resolvers).forEach(resolver => expect(resolver).toHaveBeenCalledOnce())); + for (const i of [3, 2, 1, 0]) pending[i].resolve([{ id: `resource-${i}`, connectionId: `connection-${i}`, label: "Resource", address: "inbox@example.com" }]); + expect((await assignments).map(assignment => assignment.key)).toEqual(["browser-use-cloud", "speko", "slack", "agentmail"]); + }); + + it("settles outstanding resolution before propagating a failed connector", async () => { + const slack = deferred<[]>(); + resolvers.browser.mockResolvedValue([]); + resolvers.speko.mockRejectedValue(new Error("Speko unavailable")); + resolvers.slack.mockReturnValue(slack.promise); + resolvers.inboxes.mockResolvedValue([]); + let settled = false; + const assignments = resolveConnectorAssignments({} as Db, { companyId: "company", agentId: "agent" }); + const result = expect(assignments).rejects.toThrow("Speko unavailable").finally(() => { settled = true; }); + await vi.waitFor(() => expect(resolvers.slack).toHaveBeenCalledOnce()); + expect(settled).toBe(false); + slack.resolve([]); + await result; + }); +}); diff --git a/server/src/services/connector-runtime.ts b/server/src/services/connector-runtime.ts index 6ab346660f8..8db760ddf13 100644 --- a/server/src/services/connector-runtime.ts +++ b/server/src/services/connector-runtime.ts @@ -128,19 +128,25 @@ export async function resolveConnectorAssignments( db: Db, binding: AgentBinding, ): Promise { - const assignments: ConnectorAssignment[] = []; - for (const connector of connectors) { + const results = await Promise.allSettled(connectors.map(async (connector) => { const resources = await measureRunPhase(connector.resolvePhase, () => connector.resolve(db, binding)); - if (resources.length) - assignments.push({ - key: connector.key, - label: connector.label, - skillKey: skillKey(connector), - resources, - tools: connector.tools, - }); - } - return assignments; + return resources.length ? [{ + key: connector.key, + label: connector.label, + skillKey: skillKey(connector), + resources, + tools: connector.tools, + }] : []; + })); + return results.flatMap(result => { + if (result.status === "rejected") throw result.reason; + return result.value; + }); +} + +export function connectorToolDefinitionsForSkills(skillKeys: ReadonlySet): Pick[] { + return connectors.filter(connector => skillKeys.has(skillKey(connector)) && connector.tools.length > 0) + .map(connector => ({ tools: connector.tools })); } export function isConnectorSkill(key: string) { diff --git a/server/src/services/connectors/slack-authority.ts b/server/src/services/connectors/slack-authority.ts index 31b125a92f3..d5a355f9aa6 100644 --- a/server/src/services/connectors/slack-authority.ts +++ b/server/src/services/connectors/slack-authority.ts @@ -187,7 +187,7 @@ export async function slackEndpointCandidates( binding: Pick, ) { return db - .select({ id: chatEndpoints.id }) + .select({ id: chatEndpoints.id, connectionId: chatEndpoints.connectionId }) .from(chatEndpoints) .where( and( diff --git a/server/src/services/connectors/slack-catalog.ts b/server/src/services/connectors/slack-catalog.ts index 184695da3b2..cd0b2d79f7d 100644 --- a/server/src/services/connectors/slack-catalog.ts +++ b/server/src/services/connectors/slack-catalog.ts @@ -20,6 +20,36 @@ import { resolveSlackTaskAuthority, slackEndpointCandidates, } from "./slack-authority.js"; +const catalogDefinitions = SLACK_TOOLS.map(tool => ({ + tool, + hash: createHash("sha256").update(JSON.stringify({ + name: tool.name, + description: tool.description, + inputSchema: tool.inputSchema, + risk: tool.risk, + scopes: tool.scopes, + })).digest("hex"), +})); + +export async function slackBotCatalogIsCurrent( + db: Db, + endpoint: { id: string; companyId: string; connectionId: string }, +): Promise { + const [entries, profiles] = await Promise.all([ + db.select({ name: toolCatalogEntries.name, versionHash: toolCatalogEntries.versionHash }) + .from(toolCatalogEntries).where(and( + eq(toolCatalogEntries.companyId, endpoint.companyId), + eq(toolCatalogEntries.connectionId, endpoint.connectionId), + )), + db.select({ id: toolProfiles.id }).from(toolProfiles).where(and( + eq(toolProfiles.companyId, endpoint.companyId), + eq(toolProfiles.profileKey, `slack-bot:${endpoint.id}`), + )).limit(1), + ]); + const hashes = new Map(entries.map(entry => [entry.name, entry.versionHash])); + return profiles.length > 0 && catalogDefinitions.every(({ tool, hash }) => hashes.get(`slack_bot:${tool.name}`) === hash); +} + export async function syncSlackBotTools( tx: Parameters[0]>[0], endpoint: typeof chatEndpoints.$inferSelect, @@ -46,18 +76,7 @@ export async function syncSlackBotTools( const entriesByName = new Map( existingEntries.map((entry) => [entry.name, entry]), ); - for (const tool of SLACK_TOOLS) { - const hash = createHash("sha256") - .update( - JSON.stringify({ - name: tool.name, - description: tool.description, - inputSchema: tool.inputSchema, - risk: tool.risk, - scopes: tool.scopes, - }), - ) - .digest("hex"); + for (const { tool, hash } of catalogDefinitions) { const values = { companyId: endpoint.companyId, applicationId: connection.applicationId, diff --git a/server/src/services/connectors/slack.ts b/server/src/services/connectors/slack.ts index 3053feb2867..0298ec6b7da 100644 --- a/server/src/services/connectors/slack.ts +++ b/server/src/services/connectors/slack.ts @@ -2,7 +2,7 @@ import { authorizeSlackDocument } from "./slack-document.js"; import { slackDelivery } from "./slack-delivery.js"; import { slackMessage } from "./slack-message.js"; import { executeSlackWrite } from "./slack-writes.js"; -import { syncSlackBotTools } from "./slack-catalog.js"; +import { slackBotCatalogIsCurrent, syncSlackBotTools } from "./slack-catalog.js"; import { SLACK_TOOLS } from "@paperclipai/shared"; import { measureRunPhase } from "../execution-preparation-timing.js"; import { measureRunTransaction } from "../execution-transaction-timing.js"; @@ -34,28 +34,24 @@ import { async function slackAssignedEndpoint( db: Db, binding: Partial & { companyId: string; agentId: string }, + endpoint: { id: string; connectionId: string }, ) { if (!binding.runId || !binding.issueId) return []; - let authority: Awaited>; - try { - authority = await measureRunPhase("connectors.slack.authority", () => resolveSlackTaskAuthority( - db, - binding as SlackTaskBinding, - )); - } catch (error) { - if ( - error && - typeof error === "object" && - "status" in error && - error.status === 403 - ) - return []; - throw error; + const authorize = async () => { + try { + return await measureRunPhase("connectors.slack.authority", () => resolveSlackTaskAuthority(db, binding as SlackTaskBinding)); + } catch (error) { + if (error && typeof error === "object" && "status" in error && error.status === 403) return null; + throw error; + } + }; + if (!await slackBotCatalogIsCurrent(db, { ...endpoint, companyId: binding.companyId })) { + const authority = await authorize(); + if (!authority || authority.endpoint.connectionId !== endpoint.connectionId) return []; + await measureRunTransaction(db, "connectors.slack.catalog", (tx) => + syncSlackBotTools(tx, authority.endpoint, authority.userId, true), + ); } - // Catalog updates are idempotent and preserve configured policies. - await measureRunTransaction(db, "connectors.slack.catalog", (tx) => - syncSlackBotTools(tx, authority.endpoint, authority.userId, true), - ); const [enabledProfile] = await measureRunPhase("connectors.slack.profile", () => db .select({ id: toolProfiles.id }) .from(toolProfiles) @@ -72,20 +68,21 @@ async function slackAssignedEndpoint( toolProfileEntries, and( eq(toolProfileEntries.profileId, toolProfiles.id), - eq(toolProfileEntries.connectionId, authority.endpoint.connectionId), + eq(toolProfileEntries.connectionId, endpoint.connectionId), eq(toolProfileEntries.effect, "include"), ), ) .where( and( eq(toolProfiles.companyId, binding.companyId), - eq(toolProfiles.profileKey, `slack-bot:${authority.endpoint.id}`), + eq(toolProfiles.profileKey, `slack-bot:${endpoint.id}`), eq(toolProfiles.status, "active"), ), ) .limit(1)); if (!enabledProfile) return []; - authority = await measureRunPhase("connectors.slack.authority", () => resolveSlackTaskAuthority(db, binding as SlackTaskBinding)); + const authority = await authorize(); + if (!authority || authority.endpoint.connectionId !== endpoint.connectionId) return []; return [ { id: authority.endpoint.id, @@ -113,7 +110,7 @@ export async function slackAssignedResource( ...(await slackAssignedEndpoint(db, { ...binding, endpointId: endpoint.id, - })), + }, endpoint)), ); } return resources; diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index 0a48050d67b..a4431a8d711 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -35,7 +35,7 @@ import { prepareVerifiedRemoteProviderPack } from "./remote-provider-pack.js"; import { selectRemotePiCompanion } from "./remote-pi-companion.js"; import { readNativeLocalProcessStop, PROCESS_START_REQUESTED } from "../native-local-process-stop.js"; import { remoteLeaseCleanupScope } from "../remote-execution-termination.js"; -import { resolveConnectorAssignments, isConnectorSkill } from "../connector-runtime.js"; +import { connectorToolDefinitionsForSkills } from "../connector-runtime.js"; import { activeNativeExecutionRunIds } from "./native-active-runs.js"; import { preserveNativeFailedWorkspace } from "./native-failed-workspace-recovery.js"; import { @@ -11447,8 +11447,7 @@ async function createRunnerdBackendWithinSessionClaim( input.execution.binding, )); const pinnedSkills = new Set("runtimeContext" in input.execution ? input.execution.runtimeContext.skills.map((skill) => skill.key) : []); - const connectorAssignments = [...pinnedSkills].some(isConnectorSkill) - ? await timeController("connector_assignments", () => resolveConnectorAssignments(input.db, input.execution.binding)) : []; + const connectorAssignments = connectorToolDefinitionsForSkills(pinnedSkills); const reviewRun = await input.db.select({ contextSnapshot: heartbeatRuns.contextSnapshot }) .from(heartbeatRuns).where(and( eq(heartbeatRuns.id, input.execution.binding.runId), @@ -11489,7 +11488,7 @@ async function createRunnerdBackendWithinSessionClaim( )).limit(1).then(rows => rows[0]) : undefined; const authority = new PaperclipRunnerToolAuthority(input.db, { ...(nativeReview ? { nativeReview } : {}), - connectorAssignments: connectorAssignments.filter((assignment) => pinnedSkills.has(assignment.skillKey)), + connectorAssignments, assignedMcpTools, companyId: input.execution.binding.companyId, issueId: input.execution.binding.issueId, diff --git a/server/src/services/native-runtime/paperclip-runner-tool-authority.test.ts b/server/src/services/native-runtime/paperclip-runner-tool-authority.test.ts index 39471e2da24..f9c88e4afa0 100644 --- a/server/src/services/native-runtime/paperclip-runner-tool-authority.test.ts +++ b/server/src/services/native-runtime/paperclip-runner-tool-authority.test.ts @@ -1,3 +1,4 @@ +import { connectorToolDefinitionsForSkills } from "../connector-runtime.js"; import * as cloudIdentity from "../cloud-runtime-identity.js"; import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; import { randomUUID } from "node:crypto"; @@ -96,6 +97,18 @@ describe("PaperclipRunnerToolAuthority", () => { await temporary?.cleanup(); }); + it("registers pinned connector definitions without discovery and rejects revoked assignments at use", async () => { + const connectorAssignments = connectorToolDefinitionsForSkills(new Set([ + "paperclipai/paperclip/slack", "paperclipai/paperclip/browser-use-cloud", "custom/agentmail", + ])); + expect(connectorAssignments).toHaveLength(1); + const authority = new PaperclipRunnerToolAuthority(db, { companyId, agentId, issueId, runId, connectorAssignments }); + expect(authority.definitions().some(tool => tool.name === "slack_history")).toBe(true); + expect(authority.definitions().some(tool => String(tool.name).startsWith("agentmail_"))).toBe(false); + await expect(authority.execute({ tool: "slack_history", callId: randomUUID(), arguments: { channel: "CEXAMPLE" } })) + .rejects.toThrow("This connector is no longer assigned or authorized"); + }); + it("advertises only real bindings and reads the bound task", async () => { const authority = new PaperclipRunnerToolAuthority(db, { companyId, diff --git a/server/src/services/native-runtime/paperclip-runner-tool-authority.ts b/server/src/services/native-runtime/paperclip-runner-tool-authority.ts index a7f69d73d4e..cbe300ec2da 100644 --- a/server/src/services/native-runtime/paperclip-runner-tool-authority.ts +++ b/server/src/services/native-runtime/paperclip-runner-tool-authority.ts @@ -139,7 +139,7 @@ type Binding = { apiUrl?: string; storage?: StorageService; /** Server-owned suppression for baseline evals; true never overrides operator opt-in. */ - connectorAssignments?: ConnectorAssignment[]; + connectorAssignments?: Pick[]; assignedMcpTools?: Awaited>; apiToolsEnabled?: boolean; workMode?: "standard" | "planning" | "ask"; diff --git a/server/src/services/tool-gateway.ts b/server/src/services/tool-gateway.ts index bb1c0e9192a..61693f5701c 100644 --- a/server/src/services/tool-gateway.ts +++ b/server/src/services/tool-gateway.ts @@ -1328,7 +1328,9 @@ export function createToolGatewayService( async function connectedMcpToolsForCompany( companyId: string, + selectedConnectionIds?: string[], ): Promise { + if (selectedConnectionIds?.length === 0) return []; // Read each connection and application row once. Selecting them in the // catalog join repeats the full connection row (config included) for // every tool, and one connection can expose hundreds of tools. Both reads @@ -1354,6 +1356,7 @@ export function createToolGatewayService( eq(toolCatalogEntries.status, "active"), isNull(toolCatalogEntries.quarantinedAt), connectedMcpConnectionFilter(companyId), + selectedConnectionIds ? inArray(toolConnections.id, selectedConnectionIds) : undefined, ), ) .orderBy(toolConnections.name, toolCatalogEntries.name); @@ -9135,19 +9138,30 @@ export function createToolGatewayService( async browserUseResources(binding: { companyId: string; agentId: string; runId?: string; issueId?: string }) { if (!binding.runId || !binding.issueId) return []; - const [connection] = await measureRunPhase("connectors.browser_use.connection", () => db.select({ id: toolConnections.id }).from(toolConnections).where(and( + const connections = await measureRunPhase("connectors.browser_use.connection", () => db.select({ id: toolConnections.id }).from(toolConnections).where(and( eq(toolConnections.companyId, binding.companyId), eq(toolConnections.transport, "rest_api"), eq(toolConnections.enabled, true), eq(toolConnections.status, "active"), sql`${toolConnections.config}->>'sourceTemplateKey' = 'browser-use-cloud'`, - )).limit(1)); - if (!connection) return []; + )).orderBy(toolConnections.name, toolConnections.id)); + if (!connections.length) return []; const session: ToolGatewaySession = { ...binding, runId: binding.runId, issueId: binding.issueId, id: randomUUID(), token: "", projectId: null, createdAt: new Date(), expiresAt: new Date(Date.now() + 60000) }; - const tools = (await measureRunPhase("connectors.browser_use.catalog", () => listToolsForContext(session))).filter(t => t.providerType === "provider_rest"); + await assertAgentInCompany(binding.companyId, binding.agentId); + if (await githubGuestBotConnectionForSession(db, session)) return []; + const tools = await measureRunPhase("connectors.browser_use.catalog", async () => { + const candidates = await connectedMcpToolsForCompany(binding.companyId, connections.map(connection => connection.id)); + const decisions = await decideToolsForListing(candidates, tool => policyInputForTool({ session, tool })); + return decisions.filter(({ decision }) => decision.allowed || decision.decision === "require_approval").map(({ tool }) => tool); + }); + const visible = new Set(tools.map(tool => tool.connectionId)); const resources: Array<{ id: string; label: string; connectionId: string }> = []; - for (const connectionId of new Set(tools.map(t => t.connectionId!))) { - const [connection] = await measureRunPhase("connectors.browser_use.connection", () => db.select().from(toolConnections).where(and(eq(toolConnections.companyId, binding.companyId), eq(toolConnections.id, connectionId)))); + for (const candidate of connections) { + if (!visible.has(candidate.id)) continue; + const [connection] = await measureRunPhase("connectors.browser_use.connection", () => db.select().from(toolConnections).where(and( + eq(toolConnections.companyId, binding.companyId), eq(toolConnections.id, candidate.id), + eq(toolConnections.enabled, true), eq(toolConnections.status, "active"), + ))); if (!connection || !isBrowserUseConnection(connection)) continue; - try { await measureRunPhase("connectors.browser_use.grant", () => resolveConnectionGrant(session, connection, false)); resources.push({ id: connectionId, connectionId, label: "Browser Use Cloud" }); } catch { /* Unavailable grants do not contribute a runtime skill. */ } + try { await measureRunPhase("connectors.browser_use.grant", () => resolveConnectionGrant(session, connection, false)); resources.push({ id: connection.id, connectionId: connection.id, label: "Browser Use Cloud" }); } catch { /* Unavailable grants do not contribute a runtime skill. */ } } return resources; },