Reconcile hosted candidate with current master runtime fixes

Preserve target-specific provider-pack discovery and upstream preparation spans.

Co-Authored-By: Paperclip <noreply@paperclip.ing>

.agents/skills/honeycomb-tracing/SKILL.md

@@ -0,0 +1,169 @@
+---
+name: honeycomb-tracing
+description: >-
+  Trace Paperclip runs, requests, and full CI suites in Honeycomb. Use to find
+  traces, diagnose latency, configure opt-in OpenTelemetry, or add missing spans.
+---
+
+# Honeycomb Tracing
+
+Produce an inspectable waterfall tied to the actual run or CI attempt, explain
+which intervals dominate elapsed time, and identify the smallest useful next
+experiment. Respect diagnosis-only requests; tracing does not itself authorize
+replaying tasks, changing credentials, or deploying instrumentation.
+
+## Choose the data path
+
+| Question | Start here |
+| --- | --- |
+| Why is a task slow before, during, or after model execution? | Application workflow below; [runtime setup](references/runtime.md) if spans are absent |
+| Why does an issue page load slowly? | Request spans plus browser timing, described below |
+| Which CI jobs or test files delay the whole suite? | [CI profiling](references/ci.md) |
+| Where should a new span go? | Instrumentation guidance below and [Observability contract](../../../doc/observability.md) |
+
+Repository-relative paths and commands assume the checkout root. The maintained
+contracts are `doc/observability.md` and `doc/ci-profiling.md`; recheck them and
+the owning code when a deployed version differs.
+
+Keep the repository's three paths distinct:
+
+- **Observability:** operator-configured OpenTelemetry export to Honeycomb or
+  another OTLP destination. With no endpoint, this tracing stays off.
+- **Telemetry:** first-party events under `packages/shared/src/telemetry/`, sent
+  to Paperclip by default unless opted out. Adding these requires the generated
+  contract, Telemetry README, and privacy-review workflow in `AGENTS.md`.
+- **Run log:** local `heartbeat_run_events` and measured performance events.
+  Useful for correlation, but a local event is not proof of Honeycomb export.
+
+Observability changes within the closed span-attribute allowlist do not need the
+Telemetry privacy review. Preserve the endpoint gate and no-operation behavior.
+
+## Find the application's trace
+
+1. Resolve the instance, company, issue, agent, **run UUID**, and UTC interval.
+   Record the effective harness/model, workspace/repository, warm or fresh session,
+   and actual deployed revision. A task identifier is not a run UUID; one task
+   can have several runs, and one run can have several execution attempts.
+2. Prefer existing Honeycomb tools when exposed and authorized. Otherwise use
+   the signed-in browser. The current project link targets team `paperclip`,
+   environment `test`, dataset `paperclip`, and `service.name = paperclip`.
+   These are defaults in `ui/src/lib/honeycomb-run-link.ts`, not proof of every
+   deployment's destination. Verify the target before comparing data.
+3. Open **View in Honeycomb** on the run page when available. It requires
+   Paperclip Developer Mode and local implicit board or instance-admin access.
+   This link queries existing data. **Re-run with provider trace** is a different
+   action: it starts work and is not needed to read Honeycomb.
+4. Without the link, calculate the same correlation value locally:
+
+   ```sh
+   # Set RUN_ID to the exact saved run UUID first; no trailing newline is hashed.
+   node -e 'console.log(require("node:crypto").createHash("sha256").update(process.argv[1]).digest("hex").slice(0, 12))' "$RUN_ID"
+   ```
+
+   Filter `paperclip.task.run.run_id` to that 12-character lowercase hash and
+   `service.name` to the observed service. Start with the run's absolute time
+   window. The product link defaults to the last seven days; widen it for older
+   runs. Group `COUNT` by `trace.trace_id` to discover matching traces.
+5. The product's root filter is `name in [task.execution_attempt,
+   task.provider_session, task.run]`. Use Honeycomb's `in` operator, not an
+   invented OR group. Remove the name filter when setup failed or preparation
+   spans are missing; older/standalone preparation can have a separate trace ID.
+6. Open each relevant trace and its span details. A count-only query/export
+   cannot establish span durations, parentage, or causal attribution. Capture
+   the trace URL, time window, service/version, and relevant span names/timings.
+
+In recent investigations, `HONEYCOMB_TEST_CONFIG_API_KEY` in the operator's
+`~/.secrets` could read metadata but could not query traces. Ingest permission,
+metadata access, and query access are different. Verify current capabilities;
+when queries are unavailable, use the signed-in UI rather than creating a new
+key or upgrading permissions. Read only needed secret fields, never dump the
+file, and keep values out of commands, artifacts, repository files, and reports.
+
+## Read the timeline correctly
+
+| Interval | Meaning / next inspection |
+| --- | --- |
+| Run creation → executor start | Admission/queue delay; check drain, capacity, and dispatch evidence |
+| `task.execution_attempt` | Server executor invocation through preparation, dispatch, durable reply, and cleanup |
+| `heartbeat.prepare.*`, `native.controller.*` | Controller preparation; expand workspace, configuration, backend, and runtime-selection work |
+| `task.provider_session` | Native provider lifecycle; includes preparation and execution, ends before outer workspace persistence |
+| `task.run` | Legacy adapter root |
+| `runner.session.startup`, `environment.*`, `stage.*`, `sandbox.*` | Runner/environment acquisition, workspace transfer, and startup |
+| `agent.turn` | Agent turn time; inspect actual children before calling all of it model latency |
+| `execution.workspace_finalize`, `workspace_sync.*`, `completion.*`, `response.*` | Copyback, settlement, response persistence, and publication work |
+
+The attempt and provider lifecycle are linked roots; they need not share a trace
+ID. Neither is a guarantee of one continuous trace across queueing or restarts.
+Correlate with the run log and saved reply timestamps. The native local event
+`task.run.measured` includes terminal elapsed time and durable copyback.
+
+Use start/end times to compare serial intervals and the union of parallel work.
+Never add a parent to its children or sum parallel branches as elapsed time.
+For example, `configuration_skills` is inside `execution_configuration`, and
+`skills.prepare` overlaps the broader `heartbeat.prepare_before_environment`.
+Detail spans are capped at 16 per name per attempt; missing per-item children
+do not mean later items did no work.
+
+Distinguish first streamed text, committed reply, externally delivered message,
+and browser paint. A successful `response.transaction.commit` proves durable storage;
+`response.publish` is enqueueing and `execution.realtime` is server dispatch.
+Neither proves that the user saw the response. Choose the boundary the user asked
+to improve, then measure the whole path to that boundary.
+
+For slow issue pages, combine `issue.read.<phase>` with `Server-Timing`
+(`paperclip_issue`, `issue_<phase>`) and the browser's request waterfall and
+`issue-detail:navigate→content-paint` measure. Concurrent reads overlap. Fast
+API completion does not establish fast conversation reveal or rendering.
+
+A long parent span without children is an attribution gap. It does not prove
+SQL, network, or model time. For database hypotheses, use
+[fix-slow-queries](../fix-slow-queries/SKILL.md). For CPU hypotheses, use a bounded
+authorized profile covering the actual slow phase; a sample taken while queued
+cannot explain preparation. Low aggregate container CPU does not exclude a
+blocked Node main thread. Keep profiled diagnostics separate from unprofiled
+acceptance timings.
+
+## Add only the missing instrumentation
+
+Read applicable repository rules before editing. Reuse the existing timing seams:
+
+- `server/src/services/execution-preparation-timing.ts`:
+  `withRunExecutionTiming`, `createRunPreparationTiming`, `measureRunPhase`, and
+  `beginRunPhase`. Use the existing attempt; do not create a new root per detail.
+- `server/src/services/native-runtime/native-run-trace.ts`: native lifecycle
+  hierarchy and links to the owning attempt.
+- `server/src/services/issue-read-timing.ts`: issue-read phases and Server-Timing.
+- `packages/adapter-utils/src/acpx-engine/startup-timing.ts`: runtime parent
+  context and closed span attributes. Adapter utilities receive the host context
+  rather than constructing an independent exporter.
+
+Extend the closed phase union when a meaningful boundary is missing. Wrap the
+actual awaited operation, preserve parent context across async/provider boundaries,
+and end spans on success and failure. Keep real wall times for spans recorded
+after the operation. Use existing hashed IDs, closed enums, and allowed numeric
+counts; do not add prompts, tool arguments/results, paths, raw identifiers,
+credentials, or exception text. Do not enable all filesystem instrumentation to
+explain one slow scan.
+
+Observability must not retry the operation, replace its return/error, change
+authorization/locks, or break execution when the tracer fails. Preserve root
+sampling and the explicit endpoint gate even if another library registers an
+OTel provider. Update `doc/observability.md` when the trace contract changes.
+Run focused tests for endpoint-off behavior, parentage/concurrent-run isolation,
+failure handling, and the changed seam. Use real exported evidence to claim that
+new spans appear in Honeycomb; a unit-test tracer alone does not prove ingestion.
+
+## Compare and hand back useful evidence
+
+Choose the largest actionable serial wait or unexplained gap, then reproduce it
+at the smallest faithful layer. Compare matched runs: same harness/model,
+workspace size, session warmth, concurrency, and relevant revision/configuration.
+Report sample count and median/range when several runs are available. Preserve
+failures; a failed fast run is not a latency improvement.
+
+Return the run/attempt and trace links, observed revision, end-to-end timing,
+dominant non-overlapping phases, uncertainty, and next action or verified gain.
+Label local, synthetic, and hosted evidence precisely: a dev-host reproduction
+is not a local checkout run, and a merged fix does not prove hosted improvement.
+If traces are unavailable, state the missing access or coverage and what the
+run log/browser measurements actually establish.

.agents/skills/honeycomb-tracing/agents/openai.yaml

@@ -0,0 +1,3 @@
+interface:
+  display_name: "Honeycomb Tracing"
+  short_description: "Trace Paperclip runs, requests, and full CI waterfalls"

.agents/skills/honeycomb-tracing/references/ci.md

@@ -0,0 +1,96 @@
+# Profile the full PR suite
+
+Use [doc/ci-profiling.md](../../../../doc/ci-profiling.md) as the maintained
+operator runbook. This path exports recorded CI timing through Honeycomb's Events
+API; it does not require application OTLP settings or an application run UUID.
+
+## Select one complete attempt
+
+1. Confirm the intended PR and head. Profiling is opt-in through `profile-ci`.
+   The PR author, original workflow actor, and rerun actor must be individual
+   users authorized by trusted master's `.github/**` CODEOWNERS rule. Applying
+   the label to an outside contributor's PR does not grant profiling access.
+2. When asked to profile that PR, add the label. It starts a fresh
+   **CI profile · PR #number** run of the main `PR` workflow. The label forces
+   all main lanes even for stack-middle or release-notes-only changes. Further
+   pushes also profile while it remains present.
+3. Record run ID, attempt, tested revision, and job inventory. Keep the label
+   and head unchanged until the separate **CI profile** exporter completes.
+   Use a fresh label-triggered run for an unambiguous whole-suite baseline.
+   Rerunning only failures does not rerun the successful jobs.
+4. Open the Honeycomb waterfall from the run/exporter's Summary. The project
+   destination is team `paperclip`, environment `test`, dataset `paperclip-ci`.
+   Verify the displayed root and representative file spans; a generated URL
+   or green test workflow is not proof that export succeeded.
+5. Remove the label after a one-off profile unless ongoing profiling was requested.
+   Re-adding it starts another complete attempt. Report any remaining label so
+   future pushes do not unexpectedly keep profiling.
+
+The main suite preserves existing AWS routing (`paperclip-public-pr-x64`, with
+`paperclip-ci-routing-x64` for routing) and its authorized hosted fallback.
+Inspect actual runner assignments if AWS placement is part of acceptance. The
+separate exporter uses `ubuntu-latest` and is outside the measured suite.
+
+## Read and qualify the result
+
+The hierarchy is **workflow → job → step → test filename → tests**. Collapse
+spans at the job depth for a shared wall-clock view; expand slow jobs and search
+for a known `test.file` to prove filename-level detail arrived.
+
+- Root elapsed time runs from workflow creation to the last job completion;
+  reruns start at that attempt's start. Do not sum concurrent job durations.
+- Use `ci.finishes_last`, `ci.execution_ms`, and `ci.start_offset_ms` to find
+  late-finishing work, then inspect setup, caches, compilation, builds, and tests.
+  A long job that ends early may not delay the suite.
+- Waiting spans start at job creation. Dependency waits before job creation
+  appear as gaps; GitHub does not expose a separate ready timestamp, so those
+  gaps are not proven runner-queue delay or a computed dependency critical path.
+- Vitest, Playwright, and Node supply file/test detail with different timestamp
+  semantics. GitHub steps have second precision; Node test spans use reporter
+  receipt time. Rust and standalone scripts retain job/step coverage.
+- Scope is the main PR workflow. Separate Storybook, image, release, deployment,
+  and live-provider eval workflows are not included.
+
+Inspect `ci.profile_complete`, coverage warnings, expected versus reported
+invocations, file/test counts, `ci.partial_rerun`, and `ci.carried_job_count`.
+Failed/cancelled attempts can omit downstream work; carried jobs completed before
+the new attempt are not fresh measurements. Compare the trace inventory with
+GitHub's attempt-specific jobs and retained timing artifacts.
+
+Honeycomb's waterfall displays at most 32,000 spans, prioritizing those closest
+to the root. Missing deep test rows can be a display limit rather than missing
+ingest. Check **View events** and coverage evidence before claiming data loss.
+See the [provider's large-trace limit](https://docs.honeycomb.io/troubleshoot/common-issues/data-in-honeycomb/#your-very-large-trace-displays-only-32000-spans).
+
+## Credentials and export failures
+
+`HONEYCOMB_CI_API_KEY` is a dedicated ingest-only **environment secret** in
+GitHub environment `ci-profiling`, restricted to `master`. Only the trusted
+default-branch exporter reads it. PR tests still execute the PR branch and never
+receive the key. Do not move it to repository/org secrets, put a configuration
+key in CI, execute downloaded artifacts, or switch trusted workflow refs to a PR
+head for convenience. Ordinary CI must work with profiling disabled and no key.
+
+The exporter checks each batch response, paces uploads, and retries only
+transiently rejected events. If it fails, inspect its logs and saved artifacts
+before retrying: partial export may already have sent spans, and the Events API
+does not promise deduplication on resubmission. Timing artifacts are retained for
+14 days. Preserve incomplete-export evidence even if a later upload succeeds.
+
+For local reconstruction without uploading, use the existing exporter:
+
+```sh
+node scripts/ci-profile/export.mjs --run-id=RUN_ID --attempt=1 --historical --dry-run --output=/tmp/ci-trace.json
+```
+
+Replace `RUN_ID` and attempt with the recorded values. This needs authorized
+GitHub read access. Historical reconstruction can recover job/step timings, but
+cannot invent file/test timings that were never recorded. It does not prove the
+live label-triggered journey. A real historical upload is a separate authorized
+action using the ingest key, not a necessary part of diagnosis.
+
+When changing CI tracing, use the runbook's focused exporter, routing, reporter,
+and workflow checks. For a whole-suite tracing acceptance, return the exact
+GitHub attempt and Honeycomb links, elapsed time, coverage and export status,
+runner placement if relevant, and the late-finishing jobs/file timings that
+justify the next optimization.

.agents/skills/honeycomb-tracing/references/runtime.md

@@ -0,0 +1,90 @@
+# Runtime tracing setup and missing spans
+
+Read this when configuring an authorized runtime or diagnosing absent spans.
+The authoritative implementation is
+[instrumentation.ts](../../../../server/src/instrumentation.ts); package versions
+and the data contract live in
+[Observability](../../../../doc/observability.md#enabling-tracing).
+
+## Enable the existing bootstrap
+
+1. Identify the actual process/image and its owning package. Inspect
+   `server/package.json` for the **exact** peer versions of the OTel SDK,
+   auto-instrumentations, resources, semantic conventions, and one exporter.
+   Install them in the runtime that will resolve them. Avoid copying old version
+   pins from sessions or adding every exporter to the application manifest.
+2. Configure that process before startup. A direct US Honeycomb HTTP/protobuf
+   setup uses these values:
+
+   | Setting | Value |
+   | --- | --- |
+   | `OTEL_EXPORTER_OTLP_ENDPOINT` | `https://api.honeycomb.io` |
+   | `OTEL_EXPORTER_OTLP_PROTOCOL` | `http/protobuf` |
+   | `OTEL_EXPORTER_OTLP_HEADERS` | `x-honeycomb-team=<authorized ingest key>` |
+   | `OTEL_SERVICE_NAME` | `paperclip`, or the deliberately selected service identity |
+
+   Inject the header through the existing secret manager/process environment;
+   do not paste a key into shell history or a tracked `.env`. Confirm region and
+   environment from the operator's configuration. An existing collector may be
+   the intended endpoint instead of direct Honeycomb ingestion.
+3. For HTTP, supply the **base URL**; the exporter appends `/v1/traces`. Paperclip
+   intentionally constructs HTTP exporters without an explicit URL so they read
+   the standard environment variables. A signal-specific variable alone does
+   not activate Paperclip: the bootstrap gate checks the generic
+   `OTEL_EXPORTER_OTLP_ENDPOINT`.
+4. For gRPC, use the appropriate TLS endpoint and matching exporter; the protocol
+   defaults to gRPC when unset. Unknown protocol strings warn and fall back to
+   gRPC. Do not accidentally pair HTTP configuration with that default.
+5. Check the runtime's startup diagnostics. A missing or mismatched peer package
+   leaves the server healthy **without tracing**. `instrumentationReady` settles
+   before DB connections/server construction; shutdown awaits a bounded flush.
+   A healthy `/api/health` alone cannot validate export.
+6. Generate a small representative operation within the authorized task, allow
+   the export batch to arrive, and open its actual Honeycomb spans. Verify
+   service, revision, timestamps, parentage, and the expected operation. Record
+   a trace link as proof; a bootstrap log line only proves initialization.
+
+Honeycomb's [OTLP configuration](https://docs.honeycomb.io/send-data/opentelemetry/#using-the-honeycomb-opentelemetry-endpoint)
+documents regions, headers, and the additional dataset header required for
+Classic. Confirm current provider requirements when changing setup.
+
+## Identify the serving version
+
+Paperclip resolves `service.version` from the build stamp
+(`dist/build-info.json`), then runtime Git, then `OTEL_SERVICE_VERSION`, then
+`unknown`. The build stamp wins over a stale environment variable. Docker builds
+without `.git` use `PAPERCLIP_BUILD_COMMIT` to populate the stamp.
+
+Match the trace's version with actual runtime/deployment evidence. Record source
+revision and observation time; an opaque desired release or current local HEAD
+is not evidence that the running service includes a fix.
+
+## Troubleshoot in this order
+
+| Observation | Check |
+| --- | --- |
+| No results | Correct team/environment/dataset/service, exact run hash, absolute UTC range, retention, and query permissions |
+| Only native/provider spans | Remove root-name restriction; inspect other trace IDs for the same run hash and controller version |
+| App healthy, nothing exported | Actual process env, exact optional peers, protocol/exporter match, bootstrap diagnostic, root sampling, and authorized endpoint reachability |
+| Trace ends before saved reply | Provider root excludes outer persistence; inspect `task.execution_attempt`, settlement spans, and run-log timestamps |
+| Missing unfinished phase | Active spans may not export until they end; abrupt exit can lose them |
+| Long manual span, no SQL children | Auto-instrumentation does not guarantee children; inspect real dependency timings rather than equating the whole span with database time |
+| Query API denied, authentication works | Check query permission/plan; use the signed-in UI when available, without widening key privileges |
+| Export/CSV contains only COUNT | It is an aggregate result. Inspect span details or the authorized raw-events view for event-level analysis |
+
+The server exports traces only through this bootstrap. `fs`, `dns`, and `net`
+auto-instrumentations are disabled for volume. Local performance events can
+still exist with OTLP off; their presence is not an exporter health check.
+
+When changing the bootstrap or correlation, start with the relevant tests:
+
+```sh
+pnpm exec vitest run server/src/__tests__/instrumentation.test.ts
+pnpm exec vitest run server/src/services/execution-preparation-timing.test.ts
+pnpm exec vitest run server/src/services/native-runtime/native-run-trace-context.test.ts
+pnpm exec vitest run ui/src/lib/honeycomb-run-link.test.ts
+```
+
+Select affected checks rather than running this entire list for a read-only
+diagnosis. These establish local contracts; deployment and Honeycomb ingestion
+need their own evidence.

.github/actions/setup-ci-dependencies/action.yml

@@ -0,0 +1,102 @@
+name: Restore materialized CI dependencies
+description: Reuse exact installed dependencies and certified shared build outputs; install on a miss.
+inputs:
+  working-directory:
+    description: Checkout directory relative to the workspace.
+    default: '.'
+  save-cache:
+    description: Allow a canonical master push to populate the caches.
+    default: 'false'
+  restore-cache:
+    description: Disable materialized restores for a controlled cold-path benchmark.
+    default: 'true'
+outputs:
+  dependencies-restored:
+    description: Whether the exact materialized dependency cache was restored.
+    value: ${{ steps.installed.outputs.cache-hit }}
+  builds-restored:
+    description: Whether the exact certified build cache was restored.
+    value: ${{ steps.builds.outputs.cache-hit }}
+runs:
+  using: composite
+  steps:
+    - name: Compute dependency inputs
+      id: inputs
+      shell: bash
+      working-directory: ${{ inputs.working-directory }}
+      run: node scripts/ci-dependency-cache.mjs
+
+    - name: Restore installed dependencies (read only)
+      if: inputs.restore-cache == 'true' && steps.inputs.outputs.cacheable == 'true'
+      id: installed
+      uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9
+      with:
+        key: ${{ steps.inputs.outputs.dependency-key }}
+        path: ${{ steps.inputs.outputs.dependency-paths }}
+
+    - name: Locate pnpm store
+      if: steps.installed.outputs.cache-hit != 'true'
+      id: pnpm_store
+      shell: bash
+      working-directory: ${{ inputs.working-directory }}
+      run: |
+        echo "path=$(pnpm store path --silent)" >> "$GITHUB_OUTPUT"
+        echo "arch=$(node -p 'process.arch')" >> "$GITHUB_OUTPUT"
+
+    - name: Restore pnpm store (read only)
+      if: steps.installed.outputs.cache-hit != 'true'
+      uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9
+      with:
+        path: ${{ steps.pnpm_store.outputs.path }}
+        key: node-cache-${{ runner.os }}-${{ steps.pnpm_store.outputs.arch }}-pnpm-${{ hashFiles(format('{0}/pnpm-lock.yaml', inputs.working-directory)) }}
+        restore-keys: node-cache-${{ runner.os }}-${{ steps.pnpm_store.outputs.arch }}-pnpm-
+
+    - name: Install dependencies
+      if: steps.installed.outputs.cache-hit != 'true'
+      id: install
+      shell: bash
+      working-directory: ${{ inputs.working-directory }}
+      run: |
+        if pnpm install --frozen-lockfile; then
+          echo 'frozen=true' >> "$GITHUB_OUTPUT"
+        else
+          echo '::notice title=Lockfile::checked-in lockfile is stale for this merge tree; resolving it inline'
+          pnpm install --resolution-only --ignore-scripts --no-frozen-lockfile
+          pnpm install --frozen-lockfile
+        fi
+
+    - name: Restore certified shared build outputs (read only)
+      if: inputs.restore-cache == 'true' && steps.inputs.outputs.cacheable == 'true'
+      id: builds
+      uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9
+      with:
+        key: ${{ steps.inputs.outputs.build-key }}
+        path: ${{ steps.inputs.outputs.build-paths }}
+
+    - name: Validate workspace links and shared build outputs
+      shell: bash
+      working-directory: ${{ inputs.working-directory }}
+      run: |
+        pnpm run preflight:workspace-links
+        node scripts/ensure-plugin-build-deps.mjs
+
+    - name: Save installed dependencies
+      if: >-
+        inputs.save-cache == 'true' && steps.inputs.outputs.cacheable == 'true' && github.repository == 'paperclipai/paperclip' &&
+        github.event_name == 'push' && github.ref == 'refs/heads/master' &&
+        steps.installed.outputs.cache-hit != 'true' && steps.install.outputs.frozen == 'true'
+      uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9
+      with:
+        key: ${{ steps.inputs.outputs.dependency-key }}
+        path: ${{ steps.inputs.outputs.dependency-paths }}
+
+    - name: Save certified shared build outputs
+      if: >-
+        inputs.save-cache == 'true' && steps.inputs.outputs.cacheable == 'true' && github.repository == 'paperclipai/paperclip' &&
+        github.event_name == 'push' && github.ref == 'refs/heads/master' &&
+        steps.builds.outputs.cache-hit != 'true' &&
+        (steps.installed.outputs.cache-hit == 'true' || steps.install.outputs.frozen == 'true')
+      uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9
+      with:
+        key: ${{ steps.inputs.outputs.build-key }}
+        path: ${{ steps.inputs.outputs.build-paths }}

.github/scripts/tests/ci-dependency-cache.test.mjs

@@ -0,0 +1,88 @@
+import assert from "node:assert/strict";
+import test from "node:test";
+import { readFileSync } from "node:fs";
+import { cachePlan } from "../../../scripts/ci-dependency-cache.mjs";
+
+const entries = {
+  "package.json": '{"scripts":{"postinstall":"node scripts/link-plugin-dev-sdk.mjs"}}',
+  "server/package.json": '{"dependencies":{"external":"1.0.0"}}',
+  "packages/shared/package.json": '{}',
+  "packages/plugins/sdk/package.json": '{}',
+  "pnpm-lock.yaml": 'lockfile',
+  "pnpm-workspace.yaml": 'workspace',
+  ".npmrc": 'auto-install-peers=false',
+  "patches/external.patch": 'patch',
+  "scripts/link-plugin-dev-sdk.mjs": 'postinstall',
+  "scripts/ci-dependency-cache.mjs": 'cache schema',
+  "scripts/ensure-plugin-build-deps.mjs": 'build verifier',
+  "tsconfig.base.json": '{}',
+  "packages/shared/src/index.ts": 'shared API',
+  "packages/plugins/sdk/src/index.ts": 'SDK API',
+  "server/src/server.ts": 'server',
+};
+const scope = { node: 'v24.19.0', root: '/work/paperclip', arch: 'x64', libc: '2.39', pnpm: '9.15.4', store: '/store' };
+const plan = (files = entries, runtime = scope) => cachePlan({
+  files: Object.keys(files), read: (file) => files[file] === null ? null : Buffer.from(files[file]), scope: runtime,
+  packageDirs: ['.', 'packages/plugins/sdk', 'packages/shared', 'server'],
+});
+
+test('dependency identity includes manifests, resolution, patches, installer and runtime boundaries', () => {
+  const original = plan();
+  for (const file of Object.keys(entries).filter((file) =>
+    /package\.json$/.test(file) || /^(pnpm-|\.npmrc|patches\/|scripts\/(link-plugin-dev-sdk|ci-dependency-cache))/.test(file))) {
+    for (const value of [file.endsWith('package.json') ? '{"changed":true}' : 'changed', null]) {
+      const changed = plan({ ...entries, [file]: value });
+      assert.notEqual(changed.dependencyKey, original.dependencyKey, file);
+      assert.notEqual(changed.buildKey, original.buildKey, file);
+    }
+  }
+  assert.notEqual(plan({ ...entries, 'packages/new/package.json': '{}' }).dependencyKey, original.dependencyKey);
+  for (const key of Object.keys(scope)) {
+    assert.notEqual(plan(entries, { ...scope, [key]: 'different' }).dependencyKey, original.dependencyKey, key);
+  }
+});
+
+test('shared and SDK source changes invalidate dependent builds without reinstalling packages', () => {
+  const original = plan();
+  for (const file of ['packages/shared/src/index.ts', 'packages/plugins/sdk/src/index.ts',
+    'tsconfig.base.json']) {
+    const changed = plan({ ...entries, [file]: 'changed' });
+    assert.equal(changed.dependencyKey, original.dependencyKey, file);
+    assert.notEqual(changed.buildKey, original.buildKey, file);
+  }
+  assert.deepEqual(plan({ ...entries, 'server/src/server.ts': 'different' }), original);
+  assert.deepEqual(plan(Object.fromEntries(Object.entries(entries).reverse())), original);
+  assert.deepEqual(original.dependencyPaths, ['node_modules', 'packages/plugins/sdk/node_modules',
+    'packages/shared/node_modules', 'server/node_modules']);
+  assert.deepEqual(plan({ ...entries, 'tests/fixtures/app/package.json': '{}' }).dependencyPaths, original.dependencyPaths);
+  for (const directory of ['/outside', '../outside', 'server/../../outside']) {
+    assert.throws(() => cachePlan({ files: [], read: () => null, scope, packageDirs: [directory] }));
+  }
+});
+
+test('unmodelled workspace lifecycle scripts disable materialized reuse', () => {
+  assert.equal(plan().cacheable, true);
+  for (const hook of ['preinstall', 'install', 'postinstall', 'prepare']) {
+    assert.equal(plan({ ...entries, 'server/package.json': JSON.stringify({ scripts: { [hook]: 'node custom.mjs' } }) }).cacheable, false);
+  }
+  assert.equal(plan({ ...entries, 'package.json': '{"scripts":{"postinstall":"node other.mjs"}}' }).cacheable, false);
+  for (const file of ['scripts/ensure-plugin-build-deps.mjs', 'scripts/new-install-helper.mjs', '.pnpmfile.cjs']) {
+    assert.notEqual(plan({ ...entries, [file]: 'changed' }).dependencyKey, plan().dependencyKey);
+  }
+});
+
+test('cache restores are exact and only a frozen canonical master push can publish entries', () => {
+  const action = readFileSync(new URL('../../actions/setup-ci-dependencies/action.yml', import.meta.url), 'utf8');
+  const installed = action.slice(action.indexOf('    - name: Restore installed'), action.indexOf('    - name: Locate pnpm'));
+  const builds = action.slice(action.indexOf('    - name: Restore certified'), action.indexOf('    - name: Validate workspace'));
+  for (const restore of [installed, builds]) assert.doesNotMatch(restore, /restore-keys:/);
+  for (const step of action.split('    - name: Save ').slice(1)) {
+    assert.match(step, /inputs\.save-cache == 'true'/);
+    assert.match(step, /github\.repository == 'paperclipai\/paperclip'/);
+    assert.match(step, /github\.event_name == 'push'/);
+    assert.match(step, /github\.ref == 'refs\/heads\/master'/);
+    assert.match(step, /steps\.install\.outputs\.frozen == 'true'/);
+  }
+  const workflow = readFileSync(new URL('../../workflows/release-verify.yml', import.meta.url), 'utf8');
+  assert.match(workflow, /save-cache: .*inputs\.ref == github\.sha/);
+});

.github/scripts/tests/ci-profile.test.mjs

@@ -239,7 +239,15 @@ test("workflow exporter runs trusted code and keeps keys out of the PR jobs", ()
   assert.match(workflow, /environment: ci-profiling/);
   assert.doesNotMatch(workflow, /head_sha|secrets: inherit|contents: write|pull-requests: write|pnpm install/);
   const pr = readFileSync(new URL("../../workflows/pr-trusted.yml", import.meta.url), "utf8");
-  assert.doesNotMatch(pr, /HONEYCOMB.*KEY|secrets\./);
+  // A script filename such as docker-compose-secrets.test.mjs is not a
+  // GitHub secret lookup. Check expression context, including bracket access.
+  const secretReference = /HONEYCOMB.*KEY|\$\{\{[^}]*\bsecrets\b/;
+  assert.doesNotMatch("run: node --test scripts/docker-compose-secrets.test.mjs", secretReference);
+  for (const lookup of ["${{ secrets.PR_TOKEN }}", "${{ secrets['PR_TOKEN'] }}", "${{\n secrets.PR_TOKEN\n }}", "${{ toJSON(secrets) }}", "${{ secrets.HONEYCOMB_API_KEY }}"]) {
+    assert.match(lookup, secretReference);
+  }
+
+  assert.doesNotMatch(pr, secretReference);
   assert.match(pr, /if: always\(\) && env.PAPERCLIP_CI_PROFILE == 'true'/);
   assert.doesNotMatch(pr, /PAPERCLIP_CI_PROFILE:.*labels/);
   assert.equal(pr.match(/PAPERCLIP_CI_PROFILE: \$\{\{ needs.gate.outputs.profile_ci \}\}/g).length, 9);

.github/workflows/ci-work-benchmark.yml

@@ -0,0 +1,139 @@
+name: CI work benchmark
+
+on:
+  pull_request:
+    types: [opened, synchronize, reopened, labeled]
+    paths:
+      - .github/actions/setup-ci-dependencies/**
+      - .github/workflows/ci-work-benchmark.yml
+      - scripts/ci-dependency-cache.mjs
+      - scripts/benchmark-ci-work.mjs
+      - scripts/run-vitest-stable.mjs
+      - server/src/__tests__/setup-supertest.ts
+      - server/src/__tests__/workspace-runtime.test.ts
+      - server/src/services/workspace-git-snapshot-streaming.test.ts
+
+permissions:
+  contents: read
+
+concurrency:
+  group: ci-work-benchmark-${{ github.event.pull_request.number }}
+  cancel-in-progress: true
+
+jobs:
+  paired:
+    if: >-
+      contains(github.event.pull_request.labels.*.name, 'profile-ci') &&
+      (github.event.action != 'labeled' || github.event.label.name == 'profile-ci')
+    runs-on: ubuntu-latest
+    timeout-minutes: 25
+    defaults:
+      run:
+        working-directory: candidate
+    steps:
+      - name: Checkout candidate
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1
+        with:
+          ref: ${{ github.sha }}
+          path: candidate
+          fetch-depth: 2
+          persist-credentials: false
+
+      - name: Setup Node.js
+        uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020
+        with:
+          node-version: 24
+          package-manager-cache: false
+
+      - name: Setup pnpm
+        uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86
+        with:
+          version: 9.15.4
+          run_install: false
+
+      - name: Start cold setup measurement
+        run: echo "CI_COLD_STARTED=$(date +%s)" >> "$GITHUB_ENV"
+
+      - name: Qualify candidate dependency setup
+        uses: ./candidate/.github/actions/setup-ci-dependencies
+        with:
+          working-directory: candidate
+          restore-cache: 'false'
+
+      - name: Record cold setup and compute qualification keys
+        id: qualification
+        run: |
+          echo "CI_COLD_SECONDS=$(($(date +%s) - CI_COLD_STARTED))" >> "$GITHUB_ENV"
+          node scripts/ci-dependency-cache.mjs
+
+      - name: Save PR-scoped qualification dependencies
+        uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9
+        with:
+          key: ${{ steps.qualification.outputs.dependency-key }}
+          path: ${{ steps.qualification.outputs.dependency-paths }}
+
+      - name: Save PR-scoped qualification builds
+        uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9
+        with:
+          key: ${{ steps.qualification.outputs.build-key }}
+          path: ${{ steps.qualification.outputs.build-paths }}
+
+      - name: Remove materialized outputs before restore
+        run: |
+          node --input-type=module <<'NODE'
+          import { execFileSync } from 'node:child_process';
+          import { rmSync } from 'node:fs';
+          const plan = JSON.parse(execFileSync(process.execPath, ['scripts/ci-dependency-cache.mjs'], {
+            env: { ...process.env, GITHUB_OUTPUT: '' }, encoding: 'utf8',
+          }));
+          for (const directory of [...plan.dependencyPaths, 'packages/shared/dist', 'packages/plugins/sdk/dist']) {
+            rmSync(directory, { recursive: true, force: true });
+          }
+          NODE
+          echo "CI_WARM_STARTED=$(date +%s)" >> "$GITHUB_ENV"
+
+      - name: Qualify a real warm restore
+        id: warm
+        uses: ./candidate/.github/actions/setup-ci-dependencies
+        with:
+          working-directory: candidate
+
+      - name: Verify exact cache hits and record setup comparison
+        env:
+          DEPS_RESTORED: ${{ steps.warm.outputs.dependencies-restored }}
+          BUILDS_RESTORED: ${{ steps.warm.outputs.builds-restored }}
+          OUTPUT: ${{ runner.temp }}/ci-work-benchmark
+        run: |
+          test "$DEPS_RESTORED" = true
+          test "$BUILDS_RESTORED" = true
+          mkdir -p "$OUTPUT"
+          warm_seconds=$(($(date +%s) - CI_WARM_STARTED))
+          printf '{"coldSeconds":%s,"warmSeconds":%s}\n' "$CI_COLD_SECONDS" "$warm_seconds" > "$OUTPUT/dependency-setup.json"
+          echo "Dependency setup: ${CI_COLD_SECONDS}s cold, ${warm_seconds}s with exact materialized restores." >> "$GITHUB_STEP_SUMMARY"
+
+      - name: Prepare unchanged baseline
+        run: |
+          base_sha=$(git rev-parse HEAD^1)
+          git worktree add --detach ../baseline "$base_sha"
+          cd ../baseline
+          if ! pnpm install --frozen-lockfile; then
+            pnpm install --resolution-only --ignore-scripts --no-frozen-lockfile
+            pnpm install --frozen-lockfile
+          fi
+          pnpm run preflight:workspace-links
+          pnpm --filter @paperclipai/plugin-sdk ensure-build-deps
+
+      - name: Compare complete commands and executed case inventories
+        env:
+          BENCHMARK_OUTPUT: ${{ runner.temp }}/ci-work-benchmark
+        run: |
+          node scripts/benchmark-ci-work.mjs ../baseline "$BENCHMARK_OUTPUT"
+          cat "$BENCHMARK_OUTPUT/summary.md" >> "$GITHUB_STEP_SUMMARY"
+
+      - name: Retain timings, inventories and failure logs
+        if: always()
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a
+        with:
+          name: ci-work-benchmark
+          path: ${{ runner.temp }}/ci-work-benchmark
+          retention-days: 14

.github/workflows/pr-trusted.yml

@@ -186,13 +186,19 @@ jobs:
 
       # Share the checked-in lockfile key with master. PR merge refs must not
       # save full copies of the store or evict the post-merge build caches.
+      - name: Restore materialized dependencies and shared builds
+        if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') != ''
+        uses: ./.github/actions/setup-ci-dependencies
+
       - name: Locate pnpm store
+        if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') == ''
         id: pnpm_store
         run: |
           echo "path=$(pnpm store path --silent)" >> "$GITHUB_OUTPUT"
           echo "arch=$(node -p 'process.arch')" >> "$GITHUB_OUTPUT"
 
       - name: Restore pnpm store (read only)
+        if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') == ''
         uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
         with:
           path: ${{ steps.pnpm_store.outputs.path }}
@@ -200,6 +206,7 @@ jobs:
           restore-keys: node-cache-${{ runner.os }}-${{ steps.pnpm_store.outputs.arch }}-pnpm-
 
       - name: Install dependencies
+        if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') == ''
         # Manifest-changing and stacked PRs can hold a pnpm-lock.yaml that is
         # stale for the merge tree. Resolve it inline instead of waiting on a
         # policy-job artifact: that dependency put the policy job's queue and
@@ -347,26 +354,9 @@ jobs:
       fail-fast: false
       matrix:
         include:
-          # The server suite is pinned to maxWorkers=1 (server/vitest.config.ts),
-          # so it can only be parallelized across runners. Shard it to keep this
-          # lane off the PR critical path. The suite has grown to ~3184s of
-          # serial vitest wall time (mean of runs 35036001734 and 35024948947,
-          # 2026-09-15): at five plain general-server shards with stale
-          # recorded durations the worst shard ran 806s of tests while the
-          # best ran 417s. This now uses the same shape as release-verify.yml:
-          # the ~429s chat integration suite splits by collected test location
-          # across three dedicated runners (~143s each), and the remaining
-          # ~2755s levels across twelve duration-balanced shards at ~230s
-          # each, in line with the other ~200-290s lanes.
-          #
-          # The balance only holds while the manifest is fresh. By 2026-10-10
-          # the without-chat group had grown to 871 suites and ~5031s of serial
-          # vitest time (mean of runs 38076844891 and 38076972444), 230 of
-          # them unrecorded and weighted at the median, so the twelve shards
-          # ran 313-554s of tests (server 2/12 at 618s was the slowest check
-          # in the run). Re-recorded durations level them to ~415s each; the
-          # shard count stays at twelve because Build and workspaces-b sit
-          # above that already.
+          # Keep twelve duration-balanced server shards. Each shard overlaps
+          # at most two isolated fork workers; cases within a file stay serial.
+          # The chat fixture keeps three independent database-backed shards.
           - group: general-server-without-chat
             group_label: server (1/12)
             shard_index: 0
@@ -484,13 +474,19 @@ jobs:
 
       # Share the checked-in lockfile key with master. PR merge refs must not
       # save full copies of the store or evict the post-merge build caches.
+      - name: Restore materialized dependencies and shared builds
+        if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') != ''
+        uses: ./.github/actions/setup-ci-dependencies
+
       - name: Locate pnpm store
+        if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') == ''
         id: pnpm_store
         run: |
           echo "path=$(pnpm store path --silent)" >> "$GITHUB_OUTPUT"
           echo "arch=$(node -p 'process.arch')" >> "$GITHUB_OUTPUT"
 
       - name: Restore pnpm store (read only)
+        if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') == ''
         uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
         with:
           path: ${{ steps.pnpm_store.outputs.path }}
@@ -498,6 +494,7 @@ jobs:
           restore-keys: node-cache-${{ runner.os }}-${{ steps.pnpm_store.outputs.arch }}-pnpm-
 
       - name: Install dependencies
+        if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') == ''
         # Manifest-changing and stacked PRs can hold a pnpm-lock.yaml that is
         # stale for the merge tree. Resolve it inline instead of waiting on a
         # policy-job artifact: that dependency put the policy job's queue and
@@ -558,6 +555,14 @@ jobs:
       - name: Set up Docker Buildx
         uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e # v4
 
+      - name: Set up Docker Compose
+        uses: docker/setup-compose-action@54042514f505b273907334ae2b9cdbb9a0213c1a # v2
+        with:
+          version: v5.5.1
+
+      - name: Verify Compose signing secret forwarding
+        run: node --test scripts/docker-compose-secrets.test.mjs
+
       # Same .dockerignore semantics as the real image builds: a
       # context-slimming change that strips a committed build input must
       # fail here, on the pull request, instead of failing every
@@ -684,13 +689,19 @@ jobs:
 
       # Share the checked-in lockfile key with master. PR merge refs must not
       # save full copies of the store or evict the post-merge build caches.
+      - name: Restore materialized dependencies and shared builds
+        if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') != ''
+        uses: ./.github/actions/setup-ci-dependencies
+
       - name: Locate pnpm store
+        if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') == ''
         id: pnpm_store
         run: |
           echo "path=$(pnpm store path --silent)" >> "$GITHUB_OUTPUT"
           echo "arch=$(node -p 'process.arch')" >> "$GITHUB_OUTPUT"
 
       - name: Restore pnpm store (read only)
+        if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') == ''
         uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
         with:
           path: ${{ steps.pnpm_store.outputs.path }}
@@ -698,6 +709,7 @@ jobs:
           restore-keys: node-cache-${{ runner.os }}-${{ steps.pnpm_store.outputs.arch }}-pnpm-
 
       - name: Install dependencies
+        if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') == ''
         # Manifest-changing and stacked PRs can hold a pnpm-lock.yaml that is
         # stale for the merge tree. Resolve it inline instead of waiting on a
         # policy-job artifact: that dependency put the policy job's queue and
@@ -885,13 +897,19 @@ jobs:
 
       # Share the checked-in lockfile key with master. PR merge refs must not
       # save full copies of the store or evict the post-merge build caches.
+      - name: Restore materialized dependencies and shared builds
+        if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') != ''
+        uses: ./.github/actions/setup-ci-dependencies
+
       - name: Locate pnpm store
+        if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') == ''
         id: pnpm_store
         run: |
           echo "path=$(pnpm store path --silent)" >> "$GITHUB_OUTPUT"
           echo "arch=$(node -p 'process.arch')" >> "$GITHUB_OUTPUT"
 
       - name: Restore pnpm store (read only)
+        if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') == ''
         uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
         with:
           path: ${{ steps.pnpm_store.outputs.path }}
@@ -899,6 +917,7 @@ jobs:
           restore-keys: node-cache-${{ runner.os }}-${{ steps.pnpm_store.outputs.arch }}-pnpm-
 
       - name: Install dependencies
+        if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') == ''
         # Manifest-changing and stacked PRs can hold a pnpm-lock.yaml that is
         # stale for the merge tree. Resolve it inline instead of waiting on a
         # policy-job artifact: that dependency put the policy job's queue and
@@ -1160,13 +1179,19 @@ jobs:
 
       # Share the checked-in lockfile key with master. PR merge refs must not
       # save full copies of the store or evict the post-merge build caches.
+      - name: Restore materialized dependencies and shared builds
+        if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') != ''
+        uses: ./.github/actions/setup-ci-dependencies
+
       - name: Locate pnpm store
+        if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') == ''
         id: pnpm_store
         run: |
           echo "path=$(pnpm store path --silent)" >> "$GITHUB_OUTPUT"
           echo "arch=$(node -p 'process.arch')" >> "$GITHUB_OUTPUT"
 
       - name: Restore pnpm store (read only)
+        if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') == ''
         uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
         with:
           path: ${{ steps.pnpm_store.outputs.path }}
@@ -1174,6 +1199,7 @@ jobs:
           restore-keys: node-cache-${{ runner.os }}-${{ steps.pnpm_store.outputs.arch }}-pnpm-
 
       - name: Install dependencies
+        if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') == ''
         # Manifest-changing and stacked PRs can hold a pnpm-lock.yaml that is
         # stale for the merge tree. Resolve it inline instead of waiting on a
         # policy-job artifact: that dependency put the policy job's queue and
@@ -1239,13 +1265,19 @@ jobs:
 
       # Share the checked-in lockfile key with master. PR merge refs must not
       # save full copies of the store or evict the post-merge build caches.
+      - name: Restore materialized dependencies and shared builds
+        if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') != ''
+        uses: ./.github/actions/setup-ci-dependencies
+
       - name: Locate pnpm store
+        if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') == ''
         id: pnpm_store
         run: |
           echo "path=$(pnpm store path --silent)" >> "$GITHUB_OUTPUT"
           echo "arch=$(node -p 'process.arch')" >> "$GITHUB_OUTPUT"
 
       - name: Restore pnpm store (read only)
+        if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') == ''
         uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
         with:
           path: ${{ steps.pnpm_store.outputs.path }}
@@ -1253,6 +1285,7 @@ jobs:
           restore-keys: node-cache-${{ runner.os }}-${{ steps.pnpm_store.outputs.arch }}-pnpm-
 
       - name: Install dependencies
+        if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') == ''
         # Manifest-changing and stacked PRs can hold a pnpm-lock.yaml that is
         # stale for the merge tree. Resolve it inline instead of waiting on a
         # policy-job artifact: that dependency put the policy job's queue and
@@ -1486,13 +1519,19 @@ jobs:
 
       # Share the checked-in lockfile key with master. PR merge refs must not
       # save full copies of the store or evict the post-merge build caches.
+      - name: Restore materialized dependencies and shared builds
+        if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') != ''
+        uses: ./.github/actions/setup-ci-dependencies
+
       - name: Locate pnpm store
+        if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') == ''
         id: pnpm_store
         run: |
           echo "path=$(pnpm store path --silent)" >> "$GITHUB_OUTPUT"
           echo "arch=$(node -p 'process.arch')" >> "$GITHUB_OUTPUT"
 
       - name: Restore pnpm store (read only)
+        if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') == ''
         uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
         with:
           path: ${{ steps.pnpm_store.outputs.path }}
@@ -1500,6 +1539,7 @@ jobs:
           restore-keys: node-cache-${{ runner.os }}-${{ steps.pnpm_store.outputs.arch }}-pnpm-
 
       - name: Install dependencies
+        if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') == ''
         # Manifest-changing and stacked PRs can hold a pnpm-lock.yaml that is
         # stale for the merge tree. Resolve it inline instead of waiting on a
         # policy-job artifact: that dependency put the policy job's queue and

.github/workflows/release-verify.yml

@@ -54,7 +54,7 @@ jobs:
         uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
         with:
           node-version: 24
-          cache: pnpm
+          package-manager-cache: false
 
       # rust-cache hashes its absolute cache paths into the entry's version,
       # and GitHub only serves an entry whose key and version both match. The
@@ -104,7 +104,14 @@ jobs:
       - name: Validate release package manifest
         run: node ./scripts/release-package-map.mjs check
 
+      - name: Restore materialized dependencies and shared builds
+        if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') != ''
+        uses: ./.github/actions/setup-ci-dependencies
+        with:
+          save-cache: ${{ github.repository == 'paperclipai/paperclip' && github.event_name == 'push' && github.ref == 'refs/heads/master' && inputs.ref == github.sha }}
+
       - name: Install dependencies
+        if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') == ''
         run: pnpm install --no-frozen-lockfile
 
       - name: Typecheck

doc/DATABASE.md

@@ -884,6 +884,8 @@ Migration [0338_absurd_vengeance.sql](../packages/db/src/migrations/0338_absurd_
 
 Repository admission loads retained workspace and task sources in bounded batches. Readers share source rows and privacy evaluation for identical principals, but each retained run receives its own trust and responsible-user decision. The snapshot lasts only for one read batch; later admission and publication reload current authority. Keep source-loading query counts bounded as both workspace history and retained reader counts grow.
 
+Each physical-root scan resolves duplicate workspace paths once, with bounded concurrency for distinct paths. Alias results are discarded after that scan so later admission and publication recheck symlink targets. Provenance identities select overlapping root records through the company/root index; source retention writes use bounded batches under the existing company lock, including deny-only ownership retained after a failed copy authorization.
+
 Git-source capture checks the actual common Git storage directory and verified
 checkout owners. Separately stored metadata does not make its parent a checkout.
 When a linked worktree cannot identify its primary checkout, the service verifies

doc/DEVELOPING.md

@@ -75,6 +75,16 @@ When changing the workflow path or branch, authorize the new reference before
 updating the caller. Retain older authorized SHA references while queued runs or
 supported reruns still use them.
 
+General server shards run at most two test files at a time in separate Vitest
+forks. Each file retains its own home, config and process state, and its test
+cases run serially. An explicit `PAPERCLIP_TEST_DATABASE_URL` keeps those shards
+serial because the database is shared. Unsharded local tests, route/authz lanes,
+and native Runner lanes retain their existing concurrency. Sharded chat jobs run
+two independent partitions, each with its own embedded PostgreSQL, home and
+config; an explicit shared test database retains one partition per job.
+The Runner transport lease-expiry and cold-admission regressions live in their
+own file so those real-clock waits overlap other files in the existing pool.
+
 For a full wall-clock profile of all main PR CI jobs, add `profile-ci` to the PR.
 See [CI profiling](ci-profiling.md) for the Honeycomb waterfall, coverage,
 operator setup, and activation requirements.
@@ -674,6 +684,29 @@ Set `SEARCH_EVAL_SCALE=1` to additionally measure a disposable 10,000-task,
 30,000-comment dataset. `SEARCH_EVAL_REPORT=/tmp/search-quality.json` saves
 per-query results and latency measurements; scale measurements are opt-in.
 
+### Repository workspace admission benchmark
+
+Measure the real repository preparation service with a disposable embedded
+PostgreSQL database and synthetic history, without invoking an agent:
+
+```sh
+pnpm perf:workspace-admission
+pnpm perf:workspace-admission -- --realpath-delay-ms 2.3
+pnpm perf:workspace-admission -- --distinct-cwds --realpath-delay-ms 2.3
+```
+
+Defaults model 8,516 workspaces, 27,971 provenance records and 28 retained
+readers on the shared root. The first command measures ordinary local I/O;
+the others inject filesystem metadata latency for sensitivity comparisons.
+Injected latency does not establish the actual latency of a hosted volume.
+`--distinct-cwds` creates separate folders to exercise bounded concurrent I/O.
+`--service /absolute/path/to/baseline/server/src/services/execution-workspace-repositories.ts`
+compares another checkout, which needs its workspace dependencies installed.
+The JSON report times only the service call, counts filesystem lookups and SQL
+statements, and verifies retained source identities. Setup and cleanup are
+excluded. Source tasks and permissions are synthetic; this is not an end-to-end
+agent latency measurement.
+
 ### Recent task ordering
 
 The streamlined sidebar keeps five recent tasks per company and account in browser
@@ -1893,6 +1926,14 @@ Only successful turns retain a live warm provider session. A structured failed
 or cancelled result must retire that session and collect its managed files before
 heartbeat stops the reusable sandbox. A later retry can resume the same sandbox
 without inheriting the stopped provider transport.
+
+An idle warm owner proves only the exact prior run in the same company, session,
+workspace and environment. Mutable configuration compatibility is checked
+separately: changed settings retire the owned provider before a replacement can
+execute. A copied older owner record, an active or missing owner, and a different
+environment cannot use this in-memory proof. Database terminal status alone is
+not proof that an unsuspended remote runner can be adopted.
+
 The shared lease-release boundary rechecks the durable run status for recovery
 and ordinary teardown. Successful workspace copy-back alone must not keep a
 failed turn's sandbox running.
@@ -1982,6 +2023,41 @@ CI runs `pnpm --filter @paperclipai/skills-catalog validate` and the package's
 vitest suite, so always regenerate the manifest in the same commit as the
 catalog change.
 
+## Skill Version Snapshot Retention
+
+A run uses a skill version from a directory on disk:
+`<instance>/skills/<companyId>/__versions__/<skillId>/<versionId>/`. The server
+writes this directory from the stored files of the `company_skill_versions` row.
+The directory is a cache. When the server removes it and the row stays, the next
+run that selects the version writes it again.
+
+The server starts scanning one minute after startup, then hourly, in batches of
+50 inspected directories with one-second continuations. Current versions,
+explicit agent pins, active runs, and unresolved recovery pins stay. A company
+with an active or unsettled run whose pins are unknown is skipped. Optional
+`PAPERCLIP_SKILL_VERSION_RETENTION_KEEP` retains additional newest versions
+(default `0`). Idle versions expire after `PAPERCLIP_SKILL_VERSION_RETENTION_GRACE_MINUTES`
+(default `60`). Separate `__version_access__` records track use; existing caches
+without records receive a full observation period before expiration.
+
+Preparation, durable run pins, audits, and eviction share a company-scoped
+PostgreSQL lock across processes. Eligible directories are atomically renamed
+to unique managed `.trash-<UUID>` names. Deletion runs outside the preparation
+lock and resumes after a crash. Saved version and repository snapshot rows are
+retained, and normal runtime use restores missing caches without upstream access.
+
+The sweep also handles orphan version directories after explicit skill deletion,
+preserving captured run pins until they settle. Only recognized staging/trash
+names are eligible; unknown directories and symlinked paths are retained.
+
+Set `PAPERCLIP_SKILL_VERSION_RETENTION_DRY_RUN=true` before rollout to inspect
+candidate bytes and skip reasons without deleting or changing access records.
+Set `PAPERCLIP_SKILL_VERSION_RETENTION_ENABLED=false` to disable sweeps while
+leaving restoration available. Internal `skillVersionRetentionService` methods
+support company-scoped diagnosis and bounded instance-wide continuations.
+Server logs report candidates, allocated bytes reclaimed, and skip reasons.
+Compare volume usage separately because concurrent writes affect net savings.
+
 ## App-Shipped Teams Catalog
 
 The team catalog package mirrors the skills catalog workflow for
@@ -2103,6 +2179,13 @@ Environment overrides:
   non-numeric value falls back to the default.
 - `PAPERCLIP_WORKSPACE_COMPACTION_ENABLED=false` turns compaction off. Any other
   value keeps it on.
+- `PAPERCLIP_SKILL_VERSION_RETENTION_KEEP=<count>` sets how many of the newest
+  materialized versions of each skill stay on disk (see "Skill Version Snapshot
+  Retention" above). The default is `0`. A negative, empty, or non-numeric value
+  falls back to the default.
+- `PAPERCLIP_SKILL_VERSION_RETENTION_GRACE_MINUTES=<minutes>` sets how long a
+  materialized skill version stays after its last use. The default is `60`. A
+  negative, empty, or non-numeric value falls back to the default.
 
 Without `PAPERCLIP_DB_BACKUP_ALERT_FILE`, health checks look for
 `db-backup-to-s3.failure` in the backup directory, beside the backup directory,
@@ -2412,6 +2495,37 @@ fields before disabling a connection. Verify workspace ownership, active runs,
 Git state, and runtime-service readiness before closing a workspace. A missing
 URL or old workspace timestamp alone does not prove that a row is disposable.
 
+### Reconstructable Git cache expiration
+
+The terminal-workspace worker can expire Paperclip-owned isolated Git worktrees
+24 hours after every task in their tree is done or cancelled, independently of
+PR merge or delivery status. Enable it with
+`PAPERCLIP_GIT_CACHE_EXPIRATION_ENABLED=true`. It defaults to dry run; set
+`PAPERCLIP_GIT_CACHE_EXPIRATION_DRY_RUN=false` after inspecting the structured
+`execution_workspace.git_cache_expiration` logs. Logs report candidate and
+reclaimed bytes, base removal, and skip reasons. The existing bounded lifecycle
+sweep supplies scheduling and shutdown cancellation.
+
+Before deletion the worker fetches the exact commit into a fresh object store
+using the company's normal Git identity and commits a versioned restoration
+recipe to workspace metadata. Current runs, previews, pending reopening,
+unresolved recovery, dirty/untracked files, unknown ignored files, unproven
+ownership, and unsupported or unique Git state prevent deletion. Only the
+existing repository-ignored regenerable-output allowlist can be discarded.
+Deletion first renames the worktree to recorded trash; restart and reopening
+finish that cleanup under the same source/lifecycle locks.
+
+`PAPERCLIP_GIT_BASE_EXPIRATION_ENABLED=true` additionally permits removal of an
+owned isolated remote-source base after all consumers expire and all remaining
+refs and files are verified reproducible. Shared project clones remain intact.
+Normal authorized reopening recreates a missing source and checks out the saved
+commit, even after upstream advances. Inaccessible upstream produces an explicit
+retryable restoration failure. It never substitutes the default branch.
+
+Both enable switches default off. Rollback disables further expiration while
+leaving saved recipes and restoration support intact. Recovery archives, runner
+backups, non-Git workspaces, and immutable skill snapshots are unaffected.
+
 ### Terminal workspace compaction
 
 A finished task often keeps its worktree for a long time: the reaper archives a

doc/DOCKER.md

@@ -96,14 +96,27 @@ All persisted under your bind mount (`./data/docker-paperclip` in the example ab
 
 ## Docker Compose
 
+### Persistent signing secrets
+
+Before the first launch, generate two independent values with `openssl rand -hex 32`
+and save them in `docker/.env`. Keep any existing deployment's values unchanged:
+
+```dotenv
+BETTER_AUTH_SECRET=<session-signing-secret>
+PAPERCLIP_TOOL_ACTION_SIGNING_SECRET=<tool-action-signing-secret>
+```
+
+Restrict this file to its owner (`chmod 600 docker/.env`) and keep it out of version
+control. Compose reads it for both commands below. Reuse the values on every
+restart. Rotate the tool-action secret only after outstanding approvals are
+resolved, because pending signatures use the previous value.
+
 ### Quickstart (embedded SQLite)
 
 Single container, no external database. Data persists via a bind mount.
 
 ```sh
-BETTER_AUTH_SECRET=$(openssl rand -hex 32) \
-PAPERCLIP_TOOL_ACTION_SIGNING_SECRET=$(openssl rand -hex 32) \
-  docker compose -f docker/docker-compose.quickstart.yml up --build
+docker compose -f docker/docker-compose.quickstart.yml up --build
 ```
 
 Defaults:
@@ -129,12 +142,16 @@ Pass `OPENAI_API_KEY` and/or `ANTHROPIC_API_KEY` to enable local adapter runs.
 Paperclip server + PostgreSQL 17. The database is health-checked before the server starts.
 
 ```sh
-BETTER_AUTH_SECRET=$(openssl rand -hex 32) \
-  docker compose -f docker/docker-compose.yml up --build
+docker compose -f docker/docker-compose.yml up --build
 ```
 
 PostgreSQL data persists in a named Docker volume (`pgdata`). Paperclip data persists in `paperclip-data`.
 
+Both Compose files pass `PAPERCLIP_TOOL_ACTION_SIGNING_SECRET` to the server.
+Set it to enable signed tool-action approvals, and keep it independent from
+`BETTER_AUTH_SECRET`. Leaving it unset preserves startup without approval-gated
+tool actions; those calls report `signing_secret_unconfigured`.
+
 ### Untrusted PR review
 
 Isolated container for reviewing untrusted pull requests with Codex or Claude, without exposing your host machine. See `doc/UNTRUSTED-PR-REVIEW.md` for the full workflow.

doc/PRODUCT.md

@@ -130,6 +130,10 @@ Each execution records the immutable versions it uses; resuming keeps those vers
 while a new execution resolves latest again. Upstream failures and removals preserve
 installed snapshots. GitHub refresh does not delay task execution.
 
+Old, unused skill-version disk caches expire automatically while current and in-use
+versions remain. Saved database snapshots rebuild an expired copy on demand, including
+when its upstream repository is unavailable. Recovery data has separate retention.
+
 Disconnecting removes a repository from Sources after an explicit choice: keep
 independent editable copies in My Skills, or delete its imported skills. Kept
 copies and derived forks retain their project access restrictions. Copies preserve

doc/SPEC-implementation.md

@@ -2084,6 +2084,12 @@ unavailable. Preserve current ownership and newer-work fences. See
   and history. New key collisions cannot overwrite unrelated skills. Only changed package
   bytes or executable modes create versions. Library reads, tests, and subsequent unpinned
   runs use local installed snapshots; pinned/active runs retain their versions.
+- Runtime version directories are disposable caches. An hourly, bounded sweep may
+  evict non-current copies unused for one hour, preserving active and unresolved
+  execution pins. Selection, materialization, and pin persistence serialize with
+  eviction across processes. Missing copies rebuild from immutable database bytes
+  without upstream access. See [recoverable disk cleanup](recoverable-disk-cleanup.md)
+  for rollout controls, legacy grace periods, and interrupted cleanup recovery.
 - Manually selected sources retain reviewed selection, with new candidates checked by default and
   explicit folder exclusions retained. Upstream deletion keeps installed content with
   removed status; moves are removal plus discovery. Deselect retains content

doc/ci-profiling.md

@@ -154,3 +154,44 @@ References: [Honeycomb trace fields](https://docs.honeycomb.io/get-started/best-
 [Events API](https://docs.honeycomb.io/api/events/create-events),
 [large trace display limit](https://docs.honeycomb.io/troubleshoot/common-issues/data-in-honeycomb/#your-very-large-trace-displays-only-32000-spans),
 [GitHub workflow_run](https://docs.github.com/en/actions/reference/workflows-and-actions/events-that-trigger-workflows#workflow_run).
+
+## Qualify reductions in CI work
+
+For changes to the stable test launcher, dependency setup, or the large Git and
+readiness fixtures, `profile-ci` also runs **CI work benchmark**. It uses one
+GitHub-hosted runner without secrets or AWS access. It compares the unchanged
+PR base and candidate sequentially. It records complete command times, test
+inventories, revisions, and failure logs in the `ci-work-benchmark` artifact.
+The complete general-server shard must retain the same cases. The fixture comparison retains
+the database contract and splits large Git scanning from bounded staging.
+These command timings exclude queue time and setup; use the full Honeycomb
+waterfall to assess the complete CI critical path.
+
+The benchmark also exercises cold dependency setup and a real materialized
+cache restore after removing the installed directories. Its two qualification
+entries are scoped to the PR merge ref and a separate checkout path. They cannot
+populate master's caches. Normal PR lanes restore only; the exact canonical
+master push in release verification is the normal cache writer.
+
+Materialized dependency keys include every tracked package manifest, lockfile,
+workspace configuration, patches, repository scripts, Node version/ABI, platform,
+architecture, libc, pnpm version, store location, and physical checkout path.
+There are no prefix restores. An input change takes the normal install path.
+A stale lockfile uses the existing resolution fallback but cannot publish an
+entry. An additional workspace install/prepare hook disables materialized reuse
+until its inputs are modeled. Shared/SDK build outputs have a separate key that also includes both
+packages' source/configuration and the build verifier. The existing verifier
+checks source and output fingerprints and rebuilds incomplete or changed output.
+Tests and their results are never cached or omitted by this mechanism.
+
+Serialized shards share one Vitest controller with `forks`, `isolate`, one
+worker, and file parallelism disabled. Each file gets a separate process and
+test home/configuration. The isolation regression runs files in both orders and
+checks process IDs, environment, globals, fake timers, and filesystem state.
+
+The readiness regression uses a real one-second PostgreSQL idle timeout and
+verifies that the backend connection changes while the parent write can commit.
+Its configured readiness window stays long. The Git regression keeps all four
+filename lanes above 32 MiB, while the staging case transfers and deletes 4,000
+files whose filename list exceeds 2 MiB. It still checks every staged file,
+unusual names, snapshot ownership, cleanup, and bounded command lengths.

doc/connections/tool-method-permission-reviews.json

@@ -651,6 +651,42 @@
       "reviewedAt": "2026-09-30",
       "liveProof": "not-run"
     },
+    {
+      "app": "gmail",
+      "method": "paperclip-archive",
+      "auth": "oauth",
+      "policy": "explicit",
+      "requestedScopes": [
+        "https://www.googleapis.com/auth/gmail.readonly",
+        "https://www.googleapis.com/auth/gmail.modify"
+      ],
+      "capability": "write",
+      "supportedActions": "Read Gmail and archive a thread by removing only the INBOX label.",
+      "evidence": [
+        "https://developers.google.com/workspace/gmail/api/reference/mcp/tools_list/unlabel_thread",
+        "https://gmailmcp.googleapis.com/.well-known/oauth-protected-resource/mcp/v1"
+      ],
+      "reviewedAt": "2026-10-14",
+      "liveProof": "not-run"
+    },
+    {
+      "app": "gmail",
+      "method": "customer-archive-oauth",
+      "auth": "oauth",
+      "policy": "explicit",
+      "requestedScopes": [
+        "https://www.googleapis.com/auth/gmail.readonly",
+        "https://www.googleapis.com/auth/gmail.modify"
+      ],
+      "capability": "write",
+      "supportedActions": "Read Gmail and archive a thread by removing only the INBOX label.",
+      "evidence": [
+        "https://developers.google.com/workspace/gmail/api/reference/mcp/tools_list/unlabel_thread",
+        "https://gmailmcp.googleapis.com/.well-known/oauth-protected-resource/mcp/v1"
+      ],
+      "reviewedAt": "2026-10-14",
+      "liveProof": "not-run"
+    },
     {
       "app": "google-calendar",
       "method": "paperclip-read",

doc/execution-semantics.md

@@ -1521,6 +1521,18 @@ size. Remote checkpoint archive and expanded-size limits remain 64 MiB.
 
 ### Warm sandbox continuity
 
+Healthy native warm turns retain their runner and provider processes. After a
+controller crash, continuation can recover a completed remote session from its
+saved checkpoint and exact sandbox process identity. The prior controller must
+be stopped, its result committed, and the successor must hold current task and
+coordinator authority. Recovery reconnects through the prior run's route before
+the authenticated run attachment changes authority. Unavailable identity probes
+have a shared 30-second retry budget; an identity conflict never permits replay.
+Saved process configuration must match before reuse. Old checkpoints, changed
+configuration, and controller-owned credential or instruction resources use
+verified suspension and saved-history resume. This recovery can restart a
+provider; it does not suspend every healthy warm turn.
+
 For the native runner, warm mode requests a reusable sandbox lease **before**
 lease acquisition. The environment's explicit runner lifecycle overrides the
 agent default; an inherited lifecycle uses the current agent setting on each

doc/grok-bot-agent.md

@@ -76,7 +76,29 @@ outcome needs inspection and explicit recovery; it is never sent again automatic
 Saved work remains on the task if execution fails. Reverify after clearing an
 inspected recovery hold before admitting another assignment.
 
+New assignments remain in the durable queue while the Grok connection is
+unverified, revoked, or held for recovery. Readiness is checked before claiming
+a new run or allocating its Runner. Already-admitted session reconciliation
+keeps its recorded authority; it does not gain permission to start new work.
+A configured AI connection pool selects its own provider. Its native tasks use
+the pool's normal admission checks and do not require an unused Grok connection.
+
 **Stop Paperclip work** immediately revokes the session's Paperclip authority and
 stops future dispatch. Remote Grok execution may continue. Open the Grok Bot app
 to inspect it. Paperclip shows reported activity, not a complete remote transcript
 or computer recording.
+
+The admitted Grok execution contract has no access to the Runner's workspace files. Its managed Runner still keeps durable recovery state, but stopping that session does not require workspace file repair. Inspect the Bot in Grok, clear the connection's recovery hold in the agent's Runtime settings, and verify the routine again before assigning new work. This records inspection without asserting global Grok termination.
+
+Historical stopped Grok runs can have an incorrect workspace-file hold from an
+earlier release. Runtime shows Recovery needed for these runs even when the
+routine is healthy. After inspecting Grok, use Clear recovery hold and verify
+the connection again. Paperclip retires the false file obligation only when the
+original admission denied workspace access, the assignment is fenced, the
+coordinator has settled, and the exact managed Runner leases have stop receipts.
+Connection readiness is invalidated before task inspection, so a partial failure
+cannot retain an old verified connection while removing a task hold.
+The inspection preserves the old result, action uncertainty, and no-replay hold.
+Send a new task message to request fresh work. Inspection does not certify global
+Grok termination or delete any previously retained allocation. Such allocations
+still use the existing environment cleanup controls.

doc/observability.md

@@ -179,13 +179,15 @@ connection instructions, and agent identity. Their names are
 `execution_configuration`; do not add them to its duration. They remain available
 when a later setup step fails before the provider lifecycle trace starts.
 
-Each phase starts a new root trace and links to the active trigger span when
-one exists. An unsampled HTTP request must not suppress diagnostics for the
-background run it starts. Root traces still follow the operator's configured
-sampler. The trigger's trace remains unchanged, and overlapping runs keep
-their own phase and child contexts.
+`task.execution_attempt` owns one invocation of the server executor, from its
+admission reads through preparation, provider dispatch, completion, reply persistence,
+and cleanup. It starts a new root trace with a link to the active trigger. It
+therefore follows the operator's root sampler even when the HTTP trigger was not
+sampled. Async-local ownership parents same-run preparation and nested phases to
+the current attempt. Concurrent runs have separate contexts. A phase invoked
+without a live matching attempt still starts an independent linked root.
 
-These spans exist before provider dispatch, including setup failures. Each has
+These spans exist before provider dispatch, including setup failures. Preparation phases have
 only the existing `paperclip.task.run.run_id` attribute: the first 12 hexadecimal
 characters of the run UUID's SHA-256 hash. Failed operations set error status
 without recording exception messages, paths, inputs, or returned content.
@@ -203,8 +205,8 @@ project repositories, and recovery-root reservation. These intervals are inside
 Do not add these intervals to the containing workspace duration. They use the
 same run hash and contain no workspace paths, commands, or returned records.
 
-The `paperclip.native-controller` tracer records separate
-`native.controller.<phase>` root spans around GitHub access setup, wake binding,
+The `paperclip.native-controller` tracer records
+`native.controller.<phase>` spans around GitHub access setup, wake binding,
 backend construction, connector assignments, assigned MCP discovery, tool
 authority, companion selection and Runner binary resolution. It also records
 GitHub shutdown, coordinator release, instruction checkpoints and collection,
@@ -240,9 +242,121 @@ once known, `paperclip.native_reconciliation.candidates` (a count). A failed
 pass marks the pass span and the phase that was running with an error status.
 Faster passes record nothing.
 
+### Detailed execution phases
+
+The `paperclip.execution` tracer adds these content-free spans beneath the current
+attempt or owning phase:
+
+| Span family | Measured work |
+| --- | --- |
+| `workspace.transaction`, `.entry`, `.commit`, `.rollback` | Transaction lifetime, connection acquisition plus BEGIN, and settlement after the callback returns. Entry cannot distinguish pool wait from BEGIN. |
+| `workspace.provenance_lock`, `workspace.row_lock` | The awaited advisory-lock statement and current workspace row read with its row lock. |
+| `workspace.sources.*` | Source inventory queries, canonicalization, one filesystem alias scan, repository scan, and configured project scan. |
+| `workspace.readers`, `workspace.readers.*`, `workspace.access.*` | Retained-reader groups, identity, trust, project and source authorization, privacy and inventory reads. |
+| `workspace.provenance`, `workspace.notification`, `workspace.write` | Preserved ownership, notification statements, and record writes. |
+| `workspace.materialize.*`, `workspace.checkout.*` | Private source-directory setup and identity verification, existing checkout lookup, Git credential resolution, clone, requested-ref checkout, and atomic publication. The enclosing `heartbeat.prepare.workspace_source_materialize` includes the complete source-materialization path. |
+| `runtime_selection.*` | Completion contracts, session/recovery reads, profiles, interaction materialization, backend discovery, bootstrap, run/coordinator writes, transaction entry/settlement, and workspace synchronization. |
+| `runtime_context.instructions`, `.skills`, `.mcp` | The three concurrent runtime-context branches. Their durations overlap. |
+| `runtime_context.asset_*` | Source reads, manifest hashing, cache lookup, cold materialization, and exact file verification on cold and warm paths. |
+| `workspace_sync.prepare_host_current`, `.prepare_durable_seed`, `.prepare_adopt_remote` | Runtime preparation under the selected inbound mode, including snapshot and staging work. A warm mismatch may produce a second preparation; it is real repeated work. |
+| `workspace_sync.source_root`, `.repositories`, `.ignore_rules`, `.snapshot_git`, `.snapshot_baseline` | Physical source capture, managed repositories, ignore rules, Git metadata and baseline content snapshot. |
+| `workspace_sync.pack`, `.git_clone`, `.git_archive`, `.overlay_copy`, `.overlay_archive`, `.deleted_manifest` | Temporary shallow Git clone, its archive callback, selected working-tree overlay copy/archive, and the deleted-file manifest. The archive is inside the clone's lifetime; do not sum parent and child. |
+| `workspace_sync.seed_generation`, `.seed_lookup`, `.seed_replay`, `.seed_persist`, `.seed_publish`, `.seed_verify`, `.seed_extract`, `.seed_snapshot`, `.seed_cleanup` | Seed identity, cache lookup, replay, durable copies, publication, extraction/re-snapshot verification and temporary cleanup. Cache hits omit cold clone and verification work. |
+| `workspace_sync.inbound_barrier`, `.stage_workspace`, `.transfer` | The inbound scheduler barrier, whole workspace task, and provider `syncIn` call (including provider guards, upload and post-upload extraction). Parallel tasks overlap; provider transfer/exec children give the finer transport split when available. |
+| `workspace_sync.progress` | Awaited progress/status reporting, including the existing lifecycle run-log writes. These are not file transfer work. |
+| `workspace_sync.run_read`, `.descriptor_directory`, `.descriptor_read`, `.archive_hash`, `.warm_stamp`, `.durable_seed_verify`, `.manifests_persist`, `.descriptor_write`, `.reference_write` | Saved run state, recovery descriptors, archive hashing, remote warm-stamp check, manifests and final descriptor/reference persistence. |
+| `connectors.browser_use.resolve`, `connectors.speko.resolve`, `connectors.slack.resolve`, `connectors.agentmail.resolve` | Each sequential connector resource resolver beneath configuration or native-controller assignment resolution. Empty/denied assignments still have measurable work. |
+| `connectors.browser_use.connection`, `.catalog`, `.grant` | Browser Use candidate reads, the full authorized catalog listing currently used to discover Browser Use resources, and grant checks. The catalog can contain work for other connectors. |
+| `connectors.discovery.scheduled`, `.queue`, `.build`, `.agent`, `.connections`, `.github`, `.slack`, `.speko`, `.policy`, `.on_demand_policy` | Discovery scheduler lifetime and wait, catalog construction, agent validation, descriptor sources, and policy decisions. Queued work retains the initiating run's trace context even when another run releases the scheduler slot. |
+| `connectors.slack.candidates`, `.authority`, `.profile`, `.settings`, `.identity`, `.credentials`, `.revision` | Slack endpoint enumeration, authority revalidation, enabled profile, settings, accepted requester provenance, credential reads and authorization revision. |
+| `connectors.slack.authority.transaction*`, `connectors.slack.catalog.transaction*`, `connectors.slack.catalog.lock`, `.entries`, `.write`, `.profile` | Slack authority and catalog transaction entry/settlement, advisory lock, catalog reads/updates and existing-profile lookup. Entry combines pool acquisition and BEGIN; it does not isolate pool wait. |
+| `execution.runtime` | Runtime dispatch, usage, workspace finalization and runtime cleanup; this includes work after provider execution. |
+| `execution.workspace_finalize` | Best-effort native workspace finalization, including restore recovery, copyback and commit. Its nested restore and commit spans measure distinct steps. |
+| `execution.workspace_ownership`, `.workspace_ownership_wait`, `.workspace_restore`, `.workspace_commit`, `.native_finalize` | Ownership coordination including its action, acquisition before the action, copyback, final snapshot/stamp/descriptor persistence, and native settlement. |
+| `completion.*` | Completion handler, run-log finalization, terminal persistence, accounting, liveness, and continuation summaries. |
+| `response.resolve`, `.authorize`, `.add_comment`, `.transaction*`, `.row_lock`, `.write`, `.publish` | Response selection, existing authorization, comment transaction including commit or rollback, row wait, insert, and external publication enqueue. |
+| `execution.realtime` | Synchronous live-event and plugin publication dispatch. It does not measure browser receipt or paint. |
+
+Detail phases emit at most 16 spans of each name per attempt. Larger reader and
+asset inventories continue under the containing group without unbounded per-row
+or per-file spans. Adding detail spans does not itself change work, query order,
+authorization, or locks.
+Manual transaction and dispatch boundaries retain their actual lifetime.
+Asset writes and verification may add only non-negative integer
+`paperclip.execution.fileCount` and `paperclip.execution.bytes` attributes.
+The closed keys are declared in `EXECUTION_PREPARATION_SPAN_ATTRS` in
+`packages/adapter-utils/src/acpx-engine/startup-timing.ts`.
+Other execution phases carry only the existing hashed run ID; manual boundaries
+have no attributes. Exceptions set status without exporting their messages.
+
+A committed comment transaction establishes durable server storage. A publication
+enqueue establishes pending delivery, and realtime dispatch establishes a server
+notification call. None establishes browser paint or external delivery. The attempt
+can continue through later bookkeeping and resource cleanup. A later recovery
+invocation has its own attempt; this does not manufacture one continuous trace
+across queueing, process restarts, or an unavailable controller.
+
+### Connector preparation work
+
+Run configuration resolves the four connector families concurrently and preserves
+registration order. Browser Use assignment reads only active Browser Use
+connections and their eligible catalog entries. It applies the same tool policy,
+grant, company, and restricted GitHub guest checks without joining the full tool
+discovery queue or resolving unrelated Slack, GitHub, and Speko tools.
+
+Native startup registers direct connector tool definitions from the run's pinned
+skills. It does not repeat assignment discovery. These definitions do not grant
+access: each invocation still resolves current assignments and enters the existing
+permission and approval gateway. A retained session may advertise a tool whose
+access was revoked; the call fails under the current authorization state.
+
+Slack assignment checks stored catalog hashes against the bundled definitions.
+A current catalog and existing profile need no catalog transaction or advisory
+lock. Current profile bindings and entries still determine availability, followed
+by one live task authority check. Missing or changed catalog definitions use the
+existing locked reconciliation path, with authority checks before and after the
+update. Disabled catalog entries and operator-removed bindings remain disabled.
+No authorization result is cached across calls.
+
+### Reading a slow startup in Honeycomb
+
+Filter `paperclip.task.run.run_id` to the first 12 hex characters of
+`SHA256(run UUID)`, then inspect the attempt's waterfall with its
+`service.version`. An agent/model setting or the repository's checked-out commit
+does not establish the server revision. The detailed spans are available only
+after this instrumentation is deployed; an earlier aggregate cannot establish
+which child was slow.
+
+Start beneath `heartbeat.prepare.workspace` with source materialization versus
+provisioning, repository admission, and environment acquisition/realization.
+Source materialization now separates filesystem setup, credential resolution,
+Git cloning and source identity checks. Under
+`runtime_selection.workspace_sync`, compare snapshot work, the seed-cache path,
+archive construction/verification, provider transfer and descriptor persistence.
+Use span start/end times to identify serial work and uncovered gaps; summing
+nested or concurrent durations overstates elapsed time.
+
+For connector delay, compare the four `connectors.*.resolve` spans beneath each
+assignment-resolution parent. A long Browser Use `.catalog` can be discovery
+queueing, connection descriptors, another connector's authority checks, or policy
+evaluation. A long Slack transaction entry, lock or credential span calls for a
+different investigation. Configuration and native backend setup currently resolve
+assignments separately; the trace records both without caching authorization or
+changing access decisions.
+
+New workspace detail callbacks have fixed names and remain trace-only. They do
+not create additional lifecycle events or awaited run-log writes. Existing coarse
+progress writes remain intact and appear under `workspace_sync.progress`, so
+reporting overhead is visible alongside useful work. No per-file spans, paths,
+repository URLs, connector resource IDs, tool schemas, credentials or exception
+messages are added to the export. The endpoint gate and per-phase span limit
+above still apply.
+
+
 ## Native Runner Trace Spans
 
-Paperclip Runner task runs emit a single foldable OpenTelemetry trace. This is
+The native provider lifecycle emits one foldable OpenTelemetry trace, linked to
+the server attempt described above. This is
 the native-run trace schema version `3`. `task.provider_session` is the native
 provider lifecycle root; it includes preparation and provider execution, but
 ends before outer workspace persistence. The heartbeat-owned local run-log
@@ -251,12 +365,15 @@ every other native span carries a real OpenTelemetry parent context rather than
 only a descriptive `parentName` field.
 
 The native lifecycle root uses the operator's root sampling policy. It links
-to a valid active trigger trace without inheriting its sampling decision.
+to the matching live server attempt (or a valid active trigger) without inheriting
+its sampling decision. This is a link because historical ingress and queue spans
+can predate this invocation. It does not reparent earlier work beneath a later span.
 An unsampled HTTP request therefore cannot suppress an otherwise sampled
 background run. An operator who disables root sampling still gets no spans.
 Without an OTLP endpoint, tracing remains silent and does not export data.
 
-The run link in Honeycomb matches both `task.provider_session` for native runs and `task.run` for legacy adapters. It retains the exact hashed run-ID and service filters and groups by trace ID. The URL builder uses Honeycomb’s supported `in` operator for the two root names; see the [query specification](https://github.com/honeycombio/honeycomb-mcp#query-specification-for-run_query). This changes only the query link. The operator endpoint gate and span attribute allowlist remain unchanged.
+The run link in Honeycomb matches `task.provider_session` for native runs, `task.run` for legacy adapters, and
+`task.execution_attempt` for server preparation and settlement. It retains the exact hashed run-ID and service filters and groups by trace ID. The URL builder uses Honeycomb’s supported `in` operator for the three root names; see the [query specification](https://github.com/honeycombio/honeycomb-mcp#query-specification-for-run_query). This changes only the query link. The operator endpoint gate and span attribute allowlist remain unchanged.
 
 The canonical lifecycle is:
 

doc/plans/2026-10-13-grok-bot-production-readiness.md

@@ -1,6 +1,6 @@
 # Grok Bot production readiness
 
-Status as of October 15: real assigned and Bot-initiated tasks reach canonical completion. The active-session wake correction is merged. A real proactive test found a delayed-admission echo; its correction is under verification. Production acceptance is incomplete. The current release goal and matrix below supersede the earlier dated campaign entries. Historical successes and failures remain evidence for the revisions actually tested.
+Status as of October 15: a real Bot-initiated repeat passed with one task, one run, seven completed operations, canonical Done, no assignment webhook and successful cleanup on packaged Core ea4aa185. The false file-recovery correction is merged and deployed as Core 5a9e6b63; historical inspection archived the incorrect obligation and fresh verification passed. Further assigned work is blocked by a provider-reported routine-start rate limit. Stop then exposed a separate queue-admission defect: pending work repeatedly allocated a Runner before discovering the connection hold. The qualification agent is paused and all affected leases are cleaned up. Production acceptance remains incomplete.
 
 ## Current production finish line — October 14
 
@@ -12,8 +12,8 @@ There is no testing call, case-count, token or spending cap. The former six-case
 | --- | --- | --- |
 | Ordinary onboarding works | Actual Grok UI, fixed agent consent, saved setup, wrong-key behavior, authenticated challenge, refresh, reconnect and revocation | Resumed real Grok authorization, saved setup Done, authenticated routine challenge and automatic refresh passed on the October 15 serving revision. Runtime reconnect copy fix is merged; verify its published UI and fresh production setup next. |
 | Assigned work reaches a usable result | One actual managed workflow: progress, question and answer, document, downloaded file/hash, canonical completion or review and cleanup | A preliminary comparison task on serving Core ff0 completed progress → question → answer → report/CSV → canonical Done, with both cleanup receipts. The cited CSV downloads with the exact publication hash. Two distinct CSV uploads accompanied an active-session result wake; correct that wake behavior and requalify the final candidate. |
-| Bot initiates governed work | Real-client task creation/claim/completion with stable receipts and no outbound echo or duplicate effects | A real idle-conversation request created one task and one managed run, saved a 100-word report, reached accepted Done and cleaned up its lease. A delayed-admission wake and two acceptance attempts were also recorded. Functional completion passed; no-echo acceptance failed. Requalify the correction. |
-| Recovery and authority are reliable | Healthy work past the prior five-minute failure window; actual controller restart preserving assignment/session/process and no duplicate wake; Stop, pause, reassignment, revocation, stale calls, queues and cleanup | The old hold was inspected and cleared. The successful comparison continuation ended below five minutes. Healthy work beyond that window, an actual restart and active Stop remain unverified; protocol tests are separate evidence. |
+| Bot initiates governed work | Real-client task creation/claim/completion with stable receipts and no outbound echo or duplicate effects | A real idle-conversation repeat on packaged Core ea4aa185 created one task and one run, read a public source marker with Grok's own tools, saved its report, reached canonical Done and cleaned up. One acceptance and seven operations completed, with no assignment webhook. The earlier duplicate-wake failure remains historical evidence; its correction passed this repeat. |
+| Recovery and authority are reliable | Healthy work past the prior five-minute failure window; actual controller restart preserving assignment/session/process and no duplicate wake; Stop, pause, reassignment, revocation, stale calls, queues and cleanup | The Core 11d diagnostic survived an actual restart and 13 minutes 44 seconds, then completed and cleaned up. Active Stop denied a real stale mutation. Packaged Core 5a9e6b63 later archived the historical false file hold and stopped an unaccepted session without another file obligation. Queued preparation then churned behind the connection recovery hold; correct that admission gate and repeat the final workflow, restart and active Stop. The original October 14 exit cause remains unproven. |
 | One identifiable reviewed candidate passes source gates | Focused protocol/auth/Runner/UI tests, full CI/typecheck/tests/build/token/Storybook evidence and exact artifacts | Active-session wake correction #16394 merged with 55 passing checks, two optional skips and review 5/5. The proactive echo correction started from master 7fc2dc8b3236e1149ae32756b6153a503623404b and was rebased onto freshly fetched master 250429942e27d568f20d3bc7dddcfc84a666480a before submission. Two regression cases reproduce the real delayed-admission wake before the fix. All 214 focused protocol/auth/question checks pass, including 80 Grok and Dot checks. Repository typecheck/build, token gates, the 2,731-story structure and Storybook build pass. Fresh CI, review, publication and the real no-echo repeat remain required. |
 | Production opt-in is delivered and smoke-tested | Staging acceptance, correct served Core/distribution/Runner, normal editable opt-in controls/defaults, actual production Bot workflow and inspectable task/output links | Cloud opt-in policy and release-guide corrections are merged. Staging Cloud provenance and minimal feature values were verified, with Dot disabled and defaults unchanged. The corrected wake candidate, remaining real-client cases, ordinary production opt-in controls and production smoke are outstanding. |
 
@@ -33,8 +33,34 @@ The proactive run also received an authenticated delayed-admission webhook while
 
 Normal staging automation has advanced beyond the older prepared candidate. No shared CI routing switch was changed. The earlier temporary fallback request is no longer needed for the already served successor. The dedicated production workspace still has no qualification agent grant or smoke task.
 
+The proactive no-echo correction merged as #16422 after 55 successful checks, two optional Storybook skips and fresh review 5/5. Canonical source verification and exact-source migrator publication passed. The first image attempt was displaced by an older rerun before any build job started; its single retry was then superseded by a newer master push. The successor contains the fix. Neither cancellation is a successful image publication, and no shared CI routing flag changed.
+
+The controlled recovery diagnostic preserved one original run for 824 seconds. The operator created the task, performed one same-image controller redeploy after checking for other work and rollouts, and sent one planned normal resume comment. The Bot read that queued comment at a safe boundary, read its checkpoint back, saved revision two and reached accepted Done. Its exact source identities and successful cleanup receipt are retained privately. The Bot corrected its own rejected document operation and reported correcting an invalid finish attachment; neither was repaired by the operator. This proves observed recovery on Core 11d, not the cause of the older failure or final-image acceptance.
+
+The active Stop diagnostic cancelled its run, fenced the assignment, kept the checkpoint at revision one and recorded successful sandbox cleanup. One planned negative probe in the real Grok client returned “Assignment authority expired or was revoked.” No late completion was accepted. Inspection found a generic workspace-repair obligation attached to an external native input that explicitly forbids workspace access. Four database regressions reproduce that obligation and its misleading recovery guidance. The correction keeps coordinator settlement, no-replay holds and native file safeguards while omitting file retention for an exact admitted workspace-less external contract. Repository typecheck and 68 focused file-recovery/cancellation checks pass; an assembled heartbeat regression exposed and preserved the coordinator-settlement requirement. Fresh review, full checks and deployed recovery remain required.
+
 Keep native providers and person-purpose MCP connections unchanged. No browser automation belongs in normal operation. Defer localhost relays, internal Grok APIs, marketplace distribution and Paperclip-connected third-party tools. Completion of a Paperclip work session does not assert remote Grok termination.
 
+## October 15 — queued admission during Grok recovery
+
+A real historical inspection on packaged Core `5a9e6b63` removed the false file
+obligation while preserving the old assignment fence and no-replay evidence.
+The routine returned an authenticated verification after one explicit retry of a
+confirmed rejection. A new task message then produced a new offered assignment.
+Grok rejected its webhook and its explicit retry with HTTP 400. A separate
+revoked-binding diagnostic identified `Automation run rate limit exceeded`.
+Included usage was still available; no documented cooldown or routine-start
+allowance was visible. No billing setting or provider limit was changed.
+
+Stopping that unaccepted session created no false file obligation. Pending work
+then made four failed preparation attempts with no new external assignment or
+webhook. Pause stopped the churn, and every managed lease reports successful
+cleanup. The admission correction checks the existing broker's readiness before
+claiming fresh queued work, preserves the input for later recovery, and leaves
+already-started reconciliation under its recorded execution contract. Current
+runtime-selection and dispatch authority checks stay in force. The patch still
+requires review, packaged deployment and a real recovery repeat.
+
 ## Candidate and release contract
 
 The production candidate started at `origin/master` commit `833769ab557255b11dd8f650f43ed7333653f5e4` and incorporates master through `e0f30a4c71` in the managed `codex/grokbot-production` worktree. The complete earlier prototype is preserved in snapshot `e0a8150ca248c1b80c58a7e26c10ccda9951907b`; its October 2 test results are historical, not evidence for this candidate. Reuse its provider-specific work where appropriate. Do not import its old gateway baseline, migration numbers, independent session coordinator, or retired UI entry point wholesale.
@@ -255,3 +281,31 @@ assignment authority. Disconnect completes while storage is pending, and the
 late upload returns 403 with no attachment or publication receipt. Person write
 checks remain in place. The full repository command was interrupted when this
 new review finding arrived; its partial output is not a passing test result.
+
+## October 15 — Stop review follow-up
+
+Review found that hiding the historical file-repair card alone left the stored
+file obligation blocking fresh task input. Runtime inspection now finds these
+holds even after routine recovery. It retires only the false file fields after
+exact run, assignment, coordinator and managed-lease cleanup checks, archives
+the previous evidence, and preserves no-replay. A real authenticated readiness
+exchange and new user input are still required. No global provider-stop claim is
+created. Dot historical recovery controls remain available.
+
+The later dispatch-failure owner also now honors the validated no-workspace
+contract before asking for sandbox retention. Both external providers and the
+writable native path pass the runtime boundary regression. The review follow-up
+passes 128 focused protocol, recovery, cancellation and runtime checks. An initial
+negative assertion saw the earlier readiness gate instead of the revoked-session
+gate; the corrected test checks both before and after re-verification. Server
+typecheck passes after narrowing the native workspace union. Final current-head
+CI, review, deployed historical recovery and fresh Stop are still release gates.
+
+A second review caught a partial-transaction ordering gap. Routine readiness now
+commits as invalid before the separate task inspection transactions. New tests
+hold the task lock to prove readiness is already gone and verification remains
+blocked; an unrelated active assignment cannot cause a task hold to disappear.
+All 43 Grok routine tests pass. The first follow-up CI attempt also timed out in
+one unchanged run-history case at 15 seconds; all 91 tests in that file pass
+locally. The timeout remains a recorded failure with cause unproven. The new
+source head requires fresh CI and review.

doc/plans/2026-10-15-native-runner-continuity.md

@@ -0,0 +1,122 @@
+# Native runner continuity through controller and process loss
+
+## Goal and delivery boundary
+
+Ship a verified fix to the selected staging instance and recover the two reported
+tasks from their retained work. Normal follow-up messages, server restarts, and
+recoverable runner loss must not strand a task behind
+`native_continuation_requires_reconciliation` or
+`runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate`.
+Removing an error label, clearing a hold without evidence, or blindly starting
+another provider does not meet this goal.
+
+The user authorized implementation, review, merge, staging deployment, and task
+recovery. The supported release covers the affected native Codex and native
+OpenCode remote execution paths, plus shared lifecycle regression controls.
+It preserves existing company, ownership, Stop, pause, approval, budget, and
+workspace boundaries. Genuine conflicting ownership or an unresolved external
+effect must remain safe and actionable. This is not a promise that infrastructure
+can never fail; it is a checkable promise that the specified failures do not lose
+work or require users to diagnose runner internals.
+
+Warm process reuse is a firm requirement. The user superseded the earlier
+save-and-suspend choice: healthy follow-up turns must keep the existing provider
+process. Durable ownership and recovery must coexist with that behavior. The
+suspend-every-turn candidate is withdrawn from delivery; PR #16451 is draft and
+has not been merged or deployed.
+
+## Acceptance matrix
+
+| Journey or fault | Required result | Evidence |
+| --- | --- | --- |
+| Completed turn followed immediately by another user message | Next turn starts once with the saved history, workspace, and message | Real assembled-service regression and staging journey |
+| Healthy remote warm follow-up | Same runner and provider process, with a verified handoff to the new run | Process-start identity and call-log assertions across real turns |
+| Controller restart while a session is idle, including immediately after completion | Durable evidence suffices to resume or safely replace the runner; no process-local ownership dependency strands the task | Hard and graceful restart fault tests |
+| Restart during startup or an active turn | Adopt the exact live execution or fence it before one bounded successor; preserve accepted results and pending messages | Real process/transport boundary fault tests |
+| Remote process exits or identity cannot initially be read | Distinguish a transient probe failure, a dead execution, and a conflicting owner; reconcile automatically when evidence permits | Linux remote-process tests and persisted recovery timeline |
+| Crash during checkpoint, recovery, or workspace copy-back | Resume the persistence obligation without replaying completed provider work; retain the sole recoverable source until saved | Restart/concurrency regression with independently checked file contents |
+| Existing saved or quarantined state from the affected release | Recover eligible old state without requiring a newly created task or discarding its history | Compatibility fixture and recovery of both reported tasks |
+| Duplicate recovery workers, lost acknowledgements, and newer user messages | At most one successor and one delivery of each message; no replay of a completed external action | Durable ledger and side-effect counter assertions |
+| Wrong company/session, a genuinely live competing owner, Stop/pause, revoked access, or unknown external effect | No unsafe adoption, signalling, destruction, or speculative action replay; concrete remaining blocker | Negative controls |
+| Deployment | Running staging application and any changed runner artifact match the tested revisions; both tasks have useful results or their genuine task-level disposition | Live revision, run, file, and task inspection |
+
+Use the existing recovery budgets (three provider attempts including the original,
+30-second failure retry interval) and control-transition contract (60-second
+deadline, 15-second reconciliation, resolution or explicit blocker within 90 seconds
+with healthy dependencies). Provider work duration is not bounded by this timer.
+Restarts must not reset budgets. Any necessary departure requires explicit evidence
+and an updated contract, not a silent relaxation of acceptance.
+
+## Investigation and implementation slices
+
+1. Establish both baseline failure paths from persisted evidence. The remote
+   identity failure still needs its exact failing operation established; do not
+   infer process death from the generic error alone.
+2. Close the completed-turn ownership/checkpoint gap in its existing lifecycle
+   owner, including compatibility with retained old state. Test prevention and
+   recovery, not only an isolated predicate.
+3. Close the demonstrated remote recovery gap using existing fencing, finalization,
+   workspace retention, and continuation services. Keep diagnostics precise enough
+   to distinguish probe unavailability from conflicting identity.
+4. Qualify the changed boundary locally first, then on a representative remote
+   Linux target and staging. Prepare reviewable changes and required repository
+   checks; deploy the verified revision and recover the original tasks.
+
+The first discriminating experiment is a completed native remote turn whose
+controller is lost before a follow-up. Recreate the old persisted state through
+the real lifecycle, then attempt the follow-up with an empty process-local owner
+map. Preserve real persistence, transport, and process boundaries where mocks
+would hide the failure. Existing recovery E2E and native restart suites are the
+starting harnesses; past passing evidence does not qualify this change.
+
+## Current state
+
+- Branch: `codex/native-runner-recovery`, based on `afe5b3418d`.
+- PR: https://github.com/paperclipai/paperclip/pull/16451 (draft, unmerged).
+- Normal warm turns retain their processes. The previous suspend-every-turn
+  implementation is withdrawn. Earlier suspension tests are historical only.
+- The base includes #16432 (idle expiry serialized with admission) and #16441
+  (exact warm-owner identity independent of configuration changes).
+- The revised executor can recover a completed remote session after its
+  controller dies. It checks the committed prior result, frozen scope, saved
+  checkpoint, exact remote identity, and current task/coordinator lease. It
+  reconnects through the prior route before authenticated run attachment.
+- A saved process-configuration digest controls reuse after controller loss.
+  Legacy metadata or changed configuration uses verified close and checkpoint
+  followed by saved-history resume. Healthy warm follow-ups stay open.
+- Four Linux container/TLS/PostgreSQL cases pass: ordinary warm reuse, actual
+  controller SIGKILL, older checkpoint metadata, and changed environment.
+  They assert one thread creation and exactly three turn starts, including a
+  healthy follow-up after recovery. The hard-restart
+  cases also reject a still-live prior controller and a replacement sandbox.
+  Normal follow-ups use one provider process; recovery may refresh the provider.
+  These are real runner/process tests with a deterministic fake Codex, not live
+  Daytona or live model evidence. The prior lease fixture uses the expired status
+  observed on staging. The third turn retains the runner and provider processes.
+- Full workspace typecheck and production build pass on the warm candidate.
+  The server and UI typechecks and production build also pass after the final
+  changes. All 749 focused executor/restart tests pass; the full required test
+  suite is running.
+- Greptile found a stale recovery ownership check retained across later warm
+  turns. Recovery admission now retires after verified attachment; the three-turn
+  regression passes. The earlier saved-state diagnostic finding is also fixed.
+  The third-turn fix received 5/5. Fresh review of the final changes is pending.
+- The existing source-only file-recovery form now supports retained native holds
+  as well as legacy holds. It preserves board authority, exact stop-receipt
+  validation, task status, and the prohibition on replaying the original run.
+  All 52 UI/native recovery tests, token gates, and map checks pass.
+- Fresh read-only staging records confirm both original tasks remain blocked.
+  The Codex failure preceded its coordinator claim. The OpenCode failure followed
+  controller takeover, with an active turn in the saved checkpoint. Its original
+  log lacks the failed probe reason; process death is not established.
+- Both retained sandboxes were restored for file inspection. The completed
+  website commit is intact; the other workspace retains a benchmark script and
+  lockfile changes. Backup and task disposition remain in progress. No agent
+  turn, task status, or deployment has been changed.
+- The selected tenant has an existing fleet rollout deferred by active work.
+  Do not override it. Remaining: full tests and review, staging qualification,
+  retained-state recovery, and useful disposition of both tasks.
+
+Deferred: unrelated skill-import setup failures, database query optimization,
+general UI redesign, and broad provider qualification unrelated to the changed
+shared lifecycle.

doc/plans/2026-10-15-native-warm-owner-proof.md

@@ -0,0 +1,38 @@
+# Exact warm-owner proof across native configuration changes
+
+A follow-up can fail with `prior_owner_terminal_state_indeterminate` before
+provider execution even when its prior heartbeat succeeded. The old predicate
+required the next configuration digest to match before recognizing the idle
+owner; this prevented the existing compatibility step from retiring a process
+whose configuration changed.
+
+Keep the session cache, durable-root admission and compatibility retirement in
+their current coordinator. They share process and instruction-copy ownership;
+this fix does not move their lifetime or weaken their gates. Cache each actual
+owner's heartbeat run ID and require it to match the durable prior-run identity.
+Keep company/session/workspace/environment checks and the terminal database
+check. Let the later compatibility step decide reuse versus retirement, and
+continue to reject unconfirmed retirement before another provider executes.
+
+Verification is through the production session entry point with scripted native
+backends and durable local owner files. The six existing selected controls pass.
+The initial three-case reproduction has one pass and two failures at the same
+prior-owner guard; after the fix all three pass. Eight focused controls now cover
+unchanged reuse through three turns, changed settings, changed runtime context,
+failed retirement, stale recorded identity, active owner, missing owner and a
+different environment. The runtime-context case uses the current v5 input and
+fails on the baseline at the same prior-owner guard before passing the repair. Replacing the exact
+run-ID comparison with an any-owner check makes the stale-owner control fail;
+the source is restored afterward.
+
+The first complete affected executor and resume suites passed 764 cases before
+the additional v5 runtime-context case. Full typecheck initially found an
+optional-argument error in the test database wrapper; after that correction,
+full typecheck and build passed. The v5 fixture expansion then exposed a legacy
+v1-only type on the shared database fixture. Its implementation only uses the
+common binding, so it now accepts the full native input union. After integrating the separate idle-expiry admission repair from #16432, the
+complete executor and resume suites passed 767 cases with zero selected skips.
+Full repository typecheck and build passed on the same frozen code and tests. Scripted-provider results do not establish real Daytona recovery,
+managed subscription behavior or hosted latency. Recheck the failed hosted
+continuation after normal deployment, preserving its existing recovery evidence.
+The broader response-latency target remains open independently of this repair.

doc/plans/2026-10-15-recoverable-disk-cleanup.md

@@ -0,0 +1,140 @@
+# Recoverable disk cleanup — delivery state
+
+## Outcome and boundaries
+
+Automatically expire reproducible skill caches and finished isolated Git workspaces,
+then restore them through normal use. Deliver two reviewed slices and verify actual
+reclamation plus restoration on `paperclip.staging.paperclip.app`. Keep the current
+storage architecture, unique work, and recovery data. Git restoration may depend
+on upstream availability; skill restoration must use saved database bytes alone.
+
+Apply dotta-kickoff, goal-poteto-style-definition, shorten-feedback-loop,
+paperclip-feature-to-production, and test-it-for-real. Retain one deployment owner.
+
+## Acceptance matrix
+
+| Case | Required outcome |
+| --- | --- |
+| Old skill version, upstream unavailable | Identical restored bytes and executable modes |
+| Current/active/unsettled skill version | Preserved through concurrent preparation and cleanup |
+| Finished Git workspace/base, upstream advanced | Reopen at recorded commit and perform subsequent work |
+| Unique files, live consumers, pending recovery | Skip cleanup |
+| Restart/concurrent reopen/interrupted deletion | Recover safely and preserve lifecycle fencing |
+| Upstream failure/cross-company access | Explicit failure or skip; no fallback or access widening |
+
+## Delivery slices
+
+1. Skill `__versions__` cache: one-hour inactivity, current/in-use protection,
+   persistent access records, cross-process exclusion, hourly batches of 50.
+2. Owned isolated Git workspace/base caches: 24-hour terminal-tree cooldown,
+   verified exact-commit recipe before deletion, safe on-demand restoration.
+
+Each slice gets focused tests, required repository checks, review, single-workspace
+deployment at an exact SHA, dry-run inspection, measured reclamation, and ordinary
+product acceptance. Rollback disables deletion and retains restoration support.
+
+## Current state
+
+- User authorized merging every PR from this effort after checks pass.
+- Skill [PR #16433](https://github.com/paperclipai/paperclip/pull/16433) merged
+  after exact-head CI, security review, and Greptile 5/5 passed. Reviewed head:
+  `afc159cf39a1228608c1f864648ab359a57bf357`; merge:
+  `86c965ce7ab1499864f74b8dd6ed0d2d8db15197`. Devin's original implementation
+  and authorship are retained. The earlier prototype on `codex/recoverable-skill-cache`
+  must not ship as a competing worker.
+- Git [PR #16457](https://github.com/paperclipai/paperclip/pull/16457) is open on
+  `codex/recoverable-git-cache`, rebased onto that merge. Initial review head:
+  `bb3984d3fd13e7a5604e9d90c7d58748d4b50708`. CI and review are pending.
+- Git evidence: 17 real database/filesystem cases passed in 48.04s, including
+  expiration/reopening at A after upstream B, subsequent commits, interrupted
+  trash deletion, concurrent reopening, unique files/Git state, live consumers,
+  upstream failure, company boundaries, shared bases, and nested private sources.
+  Three final guard cases passed; 328 existing controls passed in 238.69s.
+  Evidence: `/tmp/paperclip-git-expiration-tests4.log`,
+  `/tmp/paperclip-git-final-guards.log`, `/tmp/paperclip-git-existing-controls.log`.
+- Skill evidence: 21 final focused cases passed in 29.99s and 35 affected
+  skill/heartbeat controls passed. Streaming discovery holds directory cursors
+  across bounded pages; a regression counts actual reads. Preparation/pinning
+  and eviction share company locks; filesystem inspection/deletion do not hold
+  the preparation lock. Immutable snapshots remain in the database.
+- Full local typecheck/build passed for both slices. The full test attempt ran
+  for 3,547.58s and did not pass: startup IPC capacity failures, mixed-revision
+  discovery tests, two unrelated controls, and module failures after the checkout
+  disappeared. It is not evidence of one clean SHA. Exact-head CI supersedes
+  the mixed-revision run for the merged skill PR. Focused final-code retries
+  classified the remaining failures: all 103 tests in the affected final-code
+  retry passed in 42.25s. Do not repeat an hour-long loop without cause.
+- The checkout unexpectedly disappeared at about 17:22 UTC after all code was
+  pushed. Restored the same branch/path from Git, reinstalled frozen dependencies
+  in 24.8s, and rebuilt successfully. Managed attachment recovery was unavailable
+  because the app could not verify ownership. All retained evidence is in `/tmp`.
+- Staging target: `paperclip.staging.paperclip.app`, `stack-pool-50531a9d17dd`.
+  One deployment request owns this task's rollout:
+  `38a0a286-7a97-4cd5-b58e-eabdaa4d38bd`, source merge `86c965ce7a…`.
+  First prepare failed with `direct_migrator_artifact_invalid` while the exact
+  source artifact was still publishing. After its publisher passed, reran that
+  failed prepare and resumed the same request. No campaign duplication.
+- Latest read-only staging baseline at 17:26 UTC: serving
+  `03f22f4cb9be9a34906bda0e4906262dc359e9ce`, disk 73%, 106,903,482,368 bytes
+  used and 40,395,079,680 available. This task has not deleted staging data.
+  Skill cleanup is configured enabled with dry-run true; Git cleanup is not enabled.
+- Existing unresolved staging runs have missing skill-version pins. The approved
+  conservative policy skips that company. Do not clear recovery or fabricate pins
+  to obtain savings. A disposable fixture must distinguish worker proof from the
+  ordinary product restoration journey and disclose this legacy-data limitation.
+- UI fixture created through Studio: skill `06a02b07-ee23-421c-afd1-5aa4b1dad423`,
+  slug `cache-restoration-acceptance-20261015`, initial saved marker
+  `SAVED-SKILL-V1-20261015`. No test run yet. Clean up only this task's fixtures.
+- Live Git inventory: 152 active Git workspaces, 150 with runtime ownership and
+  37 with isolated remote sources. Other owned task worktrees use shared managed
+  bases. Support both layouts while preserving shared bases.
+- Temporary Railway key `paperclip-cleanup-acceptance-2026-10-15` remains
+  registered under explicit user approval. Revoke it and remove both local copies
+  when access is no longer needed; private paths are in retained task evidence.
+- Git review found two second-cycle bugs. The worker now clears the consumed
+  reopen fence before expiration dispatch; successful reopening marks the
+  recipe restored so a later missing worktree can reattach its newer branch. Eighteen
+  cases passed; an added symlink fixture initially broke its own relative Git
+  commondir. After retaining a valid absolute commondir, that safety case passed
+  in 10.63s. The new guard rejects symlinked administration storage before deletion.
+  A second review caught the normal preparation refresh moving A to B after
+  recipe retirement. Keeping a restored recipe preserves the refresh guard while
+  accepting subsequent branch commits. Both joined lifecycle tests now pass in
+  35.99s, including normal runtime preparation after reopening and another
+  missing-worktree rebuild after new work. The premature partial-edit test run
+  is excluded; `/tmp/paperclip-git-resumed-preparation2.log` is the completed run.
+  Full typecheck/build at `465e57ad52` passed; the guard change has its own server
+  typecheck. A CI policy test also mistook a script filename for a
+  GitHub secret expression; scope its check to expressions and retain explicit
+  dot/bracket/multiline secret controls.
+- Remaining: address Git review, merge after all gates pass, verify exact serving
+  SHAs, dry-run both slices, measure actual reclamation, use restored skill and
+  reopened workspace through the product, inspect persisted results and protected
+  data, record final evidence, and revoke temporary access. No live acceptance yet.
+- Latest branch-adoption review fix keeps the retained recipe's branch synchronized
+  with the row inside the existing reconciliation transaction. The joined restore,
+  rename, adoption, and normal-preparation case passed in 16.67s; 11 existing
+  reconciliation controls passed in 21.83s. Full typecheck/build passed for this
+  change. Evidence: `/tmp/paperclip-git-branch-adoption.log`,
+  `/tmp/paperclip-git-reconcile-controls.log`, and
+  `/tmp/paperclip-git-adoption-{typecheck,build}.log`.
+- Skill deployment artifacts finished, but its deploy stage was rejected at
+  17:50 UTC because another campaign already held the target. No competing
+  campaign was created. Inspect that operation before rerunning the same request.
+  UI fixture v2 is saved; neither staging fixture eviction nor restoration has run.
+
+## Feedback-loop evidence
+
+| Question | Environment / loop | Elapsed / main wait | Result / next decision |
+| --- | --- | --- | --- |
+| Can this checkout run focused tests? | Local dependency setup | 43.2s install; later one-time Runner build took several minutes | Baseline manifest/lock drift required install without changing CI-owned lockfile. Warm builds are now reusable. |
+| Can an older skill restore without upstream? | Existing service with real Postgres snapshots, pre-cleanup proof at baseline `57fa6b358d` | 4.83s warm test attempt; setup excluded as one-time above | Normal runtime path restored saved binary/script bytes and executable bits with fetch unavailable; proceed with cleanup. |
+| Do the integrated safety cases hold? | Devin's `0f3adda980` plus local follow-up; real Postgres and filesystem | 49s from launch through inspected result, 47.23s runner (62% tests, 21% import) | 22 tests passed, server typecheck passed. Added deleted-skill and failure-budget review cases plus heartbeat controls. No cloud proof claimed. |
+
+| Does Git reopening preserve A after upstream moves? | `5eb505ff5d`, local bare remote plus real runtime/service/database path | 6.398s setup/run/inspection script, about 9s launch through result review | Missing branch returned B; missing base failed. Preserve fixture for exact-commit restoration tests. |
+| Does cleanup block preparation during a slow file walk? | Same checkout plus review fixes, two DB clients and a deliberately paused filesystem read | 24.94s affected suite, about 28s through inspected result | 18 tests pass; preparation completes while scan is paused and its new pin prevents eviction. |
+
+## Deferred
+
+Skill-version creation/deduplication, recovery retention, non-Git workspace expiry,
+shared/operator checkout expiry, and moving storage off the webserver.

doc/recoverable-disk-cleanup.md

@@ -0,0 +1,47 @@
+# Recoverable disk cleanup
+
+The server materializes immutable skill versions under
+`<instance>/skills/<company>/__versions__/<skill>/<version>`. These are disposable
+disk copies: the company-scoped version and repository snapshot rows retain the
+contents, including binary files and executable flags. Normal skill resolution
+rebuilds a missing copy from the database without contacting its upstream source.
+
+## Skill version caches
+
+An hourly background sweep considers at most 50 directories per batch, yielding
+between batches. It starts after server readiness, and respects the existing
+worker shutdown, standby, and idle-drain controls. A non-current version becomes
+eligible after one hour without use. Current versions, active-run pins and
+unsettled recovery pins are retained. If an active run's version selection is
+unknown, that company's caches remain. Non-versioned skill directories and
+database snapshots are never removed by this worker.
+
+Last-use records live separately under `__version_access__`. Older installations
+without those records receive a full one-hour observation period. Preparation,
+pin persistence and eviction share a company-scoped database lock across server
+processes. Snapshot validation and byte counting run outside that lock; cleanup
+rechecks current versions, run pins, last use, and directory identity before rename. Audits hold the lock through their file reads. Eviction atomically
+moves a cache into unique `.trash-<UUID>` directories; trash deletion uses its own lock so normal
+preparation can restore files while deletion finishes. A later sweep finishes
+interrupted trash removal without deleting a newly materialized version.
+
+Controls:
+
+| Environment variable | Default | Effect |
+| --- | --- | --- |
+| `PAPERCLIP_SKILL_VERSION_RETENTION_ENABLED` | `true` | Set `false` to stop future sweeps; restoration stays available |
+| `PAPERCLIP_SKILL_VERSION_RETENTION_DRY_RUN` | `false` | Set `true` to report candidates without changing cache or access records |
+
+The internal `skillVersionRetentionService` supports company-scoped diagnosis,
+bounded continuations, cancellation between entries, and dry run. Explicit agent
+pins stay cached. Operators can also retain newest versions with
+`PAPERCLIP_SKILL_VERSION_RETENTION_KEEP` (default `0`), or change the one-hour
+window with `PAPERCLIP_SKILL_VERSION_RETENTION_GRACE_MINUTES` (default `60`). Each batch reports
+candidate and reclaimed allocated bytes, eviction counts, and skip reasons in
+the server log. Compare volume usage before and after to verify physical savings;
+concurrent writes and filesystem allocation can affect the net change. A dry run
+does not start the legacy observation period.
+
+Roll out with dry run enabled, inspect candidates/skips, then enable removal.
+Recovery archives and runner backups have separate ownership and retention;
+this worker never scans or removes them.

doc/run-log-events.md

@@ -5,6 +5,20 @@ Run-log events write to the `heartbeat_run_events` table
 Paperclip Telemetry events, and they are not OpenTelemetry exports. A run-log
 event needs no operator endpoint.
 
+## Provider-pack preparation timing
+
+The existing `run.performance.span` event also records `provider_pack.select`,
+`provider_pack.discover`, `provider_pack.verify`, `provider_pack.verify_preinstalled`,
+`provider_pack.link_preinstalled`, `provider_pack.clear_stale`, `provider_pack.upload`,
+and `provider_pack.verify_uploaded`. Selection records `stateSource` as `staged`,
+`preinstalled`, or `uploaded`. Its closed reason is `verified_pack_reused` or
+`cached_packs_unverified`. A verification failure describes an unusable candidate;
+it does not identify a particular compatibility mismatch or a failed run.
+Harness startup also records `harness_state.inspect`, `harness_state.path_check`,
+and `harness_state.launch_material` for remote state inspection and fresh launch
+material. These events retain the existing schema and carry no paths or exception strings.
+They also become native Observability spans only with an operator OTLP endpoint.
+
 ## Native PRP Run-Log Events
 
 `native.recovery.execution_reclaimed` records an atomic same-run recovery claim
@@ -151,6 +165,14 @@ proofs, encryption keys, environment variables, provider credentials, command
 arguments, or an unsanitized stderr stream. Detailed failed-attempt diagnostics
 remain in the bounded `native_run_finalizations.recovery_history` ledger.
 
+`native.remote_runner_recovery_failed` records the closed `stage` and `reason`
+when remote identity verification fails, including admission failures before the
+execution coordinator opens. Unavailable probes are retried within a shared
+30-second deadline; identity conflicts are not retried. The event distinguishes
+an unreadable identity from a confirmed missing process and contains no command
+output or credentials. Saved-state read failures use `state_read_failed`; malformed
+saved JSON uses `state_invalid`. It does not itself authorize replacement or cleanup.
+
 For a prepared remote workspace interrupted before its first executor claim,
 that ledger records `remote_execution_never_claimed` with a
 `bootstrap_incomplete` disposition. The provider attempt remains zero during

doc/testing-native-remote-continuity.md

@@ -0,0 +1,67 @@
+# Native remote continuity regression
+
+The opt-in case in
+`server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`
+runs the real executor, PostgreSQL, TLS ingress, Rust runner, remote file transfer,
+and backup validation against a deterministic fake Codex process in Linux. It
+checks the same runner process identity and one provider process across three warm
+turns, provider history reuse, and verified suspension at idle close. Separate
+cases kill the actual controller process and recover the next turn. They reject
+a still-live controller and a replacement sandbox before reconnecting. Legacy
+checkpoints and changed process configuration require verified suspension and
+replacement, while preserving the conversation and starting each turn once.
+Recovery can restart a provider to refresh its tool contract; process reuse is
+required for ordinary healthy follow-ups.
+Each recovery case includes another healthy turn after the recovered turn and
+asserts that it does not start another runner or provider process.
+It does not qualify a live model or a sandbox provider API.
+
+Use a disposable Linux container with Node.js, `sh`, `tar`, `base64`, `ps`, and
+`setsid`. Build `paperclip-runnerd` and `fake-codex-app-server` from the checkout's
+`packages/paperclip-runner/runner/Cargo.toml` for that container's architecture.
+Keep the Linux runner binary on the host as well: the product hashes and stages
+that artifact. Put the fake provider at `/opt/continuity/fake-codex-app-server`
+inside the container. Do not put credentials in this container.
+
+Start the container with `--init` and publish container port 43127 on localhost.
+Read its assigned port with `docker port CONTAINER 43127`; an automatic host port
+can change after a container restart. The test opens a local TLS tunnel to this
+port. Create a test-only certificate and key:
+
+```sh
+openssl req -x509 -newkey rsa:2048 -nodes \
+  -keyout /tmp/continuity-test.key -out /tmp/continuity-test.crt \
+  -days 1 -subj '/CN=localhost' \
+  -addext 'subjectAltName=IP:127.0.0.1,DNS:localhost'
+```
+
+Create an executable wrapper on the host at `/tmp/continuity-fake-codex`:
+
+```sh
+#!/bin/sh
+if [ "$1" = '--version' ]; then
+  echo 'codex-cli 0.999.0'
+  exit 0
+fi
+exec /opt/continuity/fake-codex-app-server \
+  --state-file-in-codex-home --durable-turn-ids --record-process-start \
+  --call-log "$PWD/provider-calls.log" "$@"
+```
+
+Run the test with the container name, current host port, and Linux artifact path:
+
+```sh
+NODE_EXTRA_CA_CERTS=/tmp/continuity-test.crt \
+PAPERCLIP_TEST_RUNNER_TLS_KEY=/tmp/continuity-test.key \
+PAPERCLIP_TEST_RUNNER_CONTAINER=CONTAINER \
+PAPERCLIP_TEST_RUNNER_PORT=HOST_PORT \
+PAPERCLIP_TEST_LINUX_RUNNER_BINARY=/path/to/linux/paperclip-runnerd \
+PAPERCLIP_TEST_FAKE_CODEX_WRAPPER=/tmp/continuity-fake-codex \
+pnpm exec vitest run \
+  server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts \
+  -t 'preserves remote warm continuity' --maxWorkers=1
+```
+
+The caller owns the disposable container. Remove it after the run, including
+after failure; failed cases can retain remote processes and files for inspection.
+The test cleans up its database and host session directories.

docker/docker-compose.quickstart.yml

@@ -2,7 +2,9 @@
 # Copy this file and set the required env vars before running.
 #
 #   cd docker
-#   export BETTER_AUTH_SECRET="$(openssl rand -hex 32)"
+#   Set BETTER_AUTH_SECRET and PAPERCLIP_TOOL_ACTION_SIGNING_SECRET in .env
+#   once, using a separate `openssl rand -hex 32` value for each. Reuse them
+#   on restarts; do not replace existing deployment secrets.
 #   export ANTHROPIC_API_KEY="sk-ant-..."
 #   docker compose -f docker-compose.quickstart.yml up --build
 #
@@ -41,6 +43,8 @@ services:
 
       # ── Required secret for session signing (generate with: openssl rand -hex 32) ──
       BETTER_AUTH_SECRET: "${BETTER_AUTH_SECRET:?BETTER_AUTH_SECRET must be set}"
+      # Independent secret for signed approval-gated tool actions.
+      PAPERCLIP_TOOL_ACTION_SIGNING_SECRET: "${PAPERCLIP_TOOL_ACTION_SIGNING_SECRET:-}"
     volumes:
       # Persistent data directory (database, agent workspaces, uploads)
       - "${PAPERCLIP_DATA_DIR:-../data/docker-paperclip}:/paperclip"

docker/docker-compose.yml

@@ -40,6 +40,8 @@ services:
       PAPERCLIP_DEPLOYMENT_EXPOSURE: "private"
       PAPERCLIP_PUBLIC_URL: "${PAPERCLIP_PUBLIC_URL:-http://localhost:3100}"
       BETTER_AUTH_SECRET: "${BETTER_AUTH_SECRET:?BETTER_AUTH_SECRET must be set}"
+      # Independent secret for signed approval-gated tool actions.
+      PAPERCLIP_TOOL_ACTION_SIGNING_SECRET: "${PAPERCLIP_TOOL_ACTION_SIGNING_SECRET:-}"
     volumes:
       - paperclip-data:/paperclip
     depends_on:

maps/agent-access-map/recovery.md

@@ -108,7 +108,7 @@ retry and some operator controls require board. Completion or cancellation of a
 task does not clear its exact unresolved native workspace owner hold. Physical
 owner release still requires the [operator stop-verification procedure](../../doc/native-workspace-finalization-recovery.md).
 
-For a resolved legacy action that retains a file hold, the same resolution API
+For a resolved legacy or native action that retains a file hold, the same resolution API
 supports source-only file repair. Board runtime management and recovery-resolution
 authority are required. The request identifies the exact source action/run and
 records recovered-file evidence; the server verifies the original stop receipt.

maps/agent-access-map/skills.md

@@ -89,6 +89,12 @@ remaining operations before qualifying the whole capability.
 Gaps: Revision metadata has an API; pinned execution skill versions and installed latest
 are distinct.
 
+Runtime disk-cache expiration does not delete version records or change access. Normal
+execution rematerializes saved bytes after existing skill authorization; cleanup is an
+internal company-scoped maintenance operation, with no new agent endpoint. Verify pinned
+restoration and active-run protection with `server/src/__tests__/skill-version-cache.test.ts`
+and the assembled heartbeat runtime-skill tests; live acceptance remains separate.
+
 ## `policy`
 
 Review: `baseline`.

maps/agent-access-map/workspaces.md

@@ -57,6 +57,14 @@ are revalidated for reuse, cleanup and reopening. A failed save removes only the
 owned task worktree. The [isolated-source service](../../server/src/services/isolated-remote-workspace-source.ts)
 and its local regression tests establish source and lifecycle boundaries; live
 agent/provider qualification remains separate.
+Reconstructable Git expiration adds no agent endpoint or permission. It records
+company/workspace/repository identity and an exact commit before removing owned
+cache files. Existing task/workspace reopening retains current company, source,
+and trust checks. Missing upstream access returns a retryable error; it does not
+choose a newer commit or grant access to another company's source. Shared project
+bases and unresolved recovery remain protected. Local lifecycle evidence is in
+[expiration tests](../../server/src/__tests__/git-workspace-expiration.test.ts).
+
 Writable aliases must also be compatible with current and retained physical
 source projects' trust boundaries. Incompatible choices are rejected before
 dispatch; a projectless alias cannot bypass required sandbox containment.

maps/feature-map/recovery.md

@@ -160,7 +160,7 @@ The recovery-card suite covers workspace divergence, reissue, reconcile, and
 confirmation. [Workspace export](../../ui/src/components/WorkspaceExportRecovery.test.tsx)
 covers quiet restoration, legacy/operator assistance and resolved states.
 
-For a retained legacy file hold, open the source task after its recovery action
+For a retained legacy or native file hold, open the source task after its recovery action
 has resolved but its files still need recovery. An authorized board operator can
 record the verified recovered files and backup link without changing the current
 task status or starting another run. The server must independently validate the

maps/feature-map/skills.md

@@ -65,6 +65,10 @@ Manual: Enable the skill for one test agent and leave another without access. Ru
 - Installed does not mean enabled for every agent.
 - Read-only GitHub/bundled skills may require a writable copy rather than an in-place edit.
 - Studio interaction coverage is in [questions and approvals](./questions-and-approvals.md); a passing mock is not a live harness test.
+- Old version disk caches may expire; saved versions remain available. Verify a pinned
+  version is restored through normal execution with upstream unavailable. Current
+  and in-use versions stay materialized. See [cleanup controls](../../doc/recoverable-disk-cleanup.md)
+  and the [database-backed cache tests](../../server/src/__tests__/skill-version-cache.test.ts).
 
 ### Project repository source behavior
 

maps/feature-map/workspaces.md

@@ -57,6 +57,15 @@ and [preparation tests](../../server/src/services/heartbeat/workspace-preparatio
 exercise the real Git isolation and lifecycle guards. They do not qualify live
 remote credentials or a provider deployment.
 
+For reconstructable Git expiration, finish a disposable task tree, run the
+internal lifecycle sweep after its 24-hour grace, and inspect candidate/reclaimed
+bytes. Advance its upstream first; use the ordinary task/workspace reopen control
+and verify the original commit, file contents, and subsequent persisted work.
+Current runs, previews, recovery, unique files and shared base clones must stay.
+The [expiration tests](../../server/src/__tests__/git-workspace-expiration.test.ts)
+exercise real local Git and database paths, interruption and concurrent reopening;
+cloud credentials and the deployed user journey require separate live evidence.
+
 For task workspace selection, use the authorized task workspace API to choose a
 task directory, project source, or existing workspace. Verify that a change
 made during a run stays pending until the next normal admission. Submit another

package.json

@@ -3,6 +3,7 @@
   "private": true,
   "type": "module",
   "scripts": {
+    "perf:workspace-admission": "node --import ./server/node_modules/tsx/dist/loader.mjs server/scripts/benchmark-workspace-admission.ts",
     "postinstall": "node scripts/link-plugin-dev-sdk.mjs",
     "preflight:workspace-links": "node cli/node_modules/tsx/dist/cli.mjs scripts/ensure-workspace-package-links.ts",
     "dev": "pnpm --filter @paperclipai/server exec tsx ../scripts/dev-runner.ts watch",
@@ -61,7 +62,7 @@
     "smoke:posthog-live": "node scripts/smoke/posthog-live.mjs",
     "smoke:pipelines-tutorial": "./scripts/smoke/pipelines-tutorial-smoke.sh",
     "smoke:terminal-bench-loop-skill": "node scripts/smoke/terminal-bench-loop-skill-smoke.mjs",
-    "test:release-registry": "node scripts/ci-profile/node-test.mjs scripts/ci-profile/reporters.test.mjs scripts/__tests__/canary-startup-playwright.test.mjs scripts/__tests__/cursor-public-install-sandbox.test.mjs scripts/__tests__/grok-public-install-sandbox.test.mjs scripts/__tests__/public-install-closure.test.mjs scripts/__tests__/prefetch-child.test.mjs scripts/__tests__/stage-bundled-packages.test.mjs scripts/__tests__/pack-staged-packages.test.mjs scripts/__tests__/retain-runner-qualification-packages.test.mjs scripts/__tests__/runner-release-crate-cache.test.mjs scripts/verify-release-registry-state.test.mjs scripts/release-package-map.test.mjs scripts/check-release-package-bootstrap.test.mjs scripts/check-no-git-push.test.mjs scripts/release-lib.test.mjs scripts/release-registry-versions.test.mjs scripts/link-plugin-dev-sdk.test.js scripts/acpx-patch-packaging.test.mjs scripts/service-onboard-smoke.test.mjs scripts/docker-onboard-smoke.test.mjs scripts/preview-artifacts.test.mjs scripts/cloud-migrator-artifacts.test.mjs",
+    "test:release-registry": "node scripts/ci-profile/node-test.mjs scripts/ci-profile/reporters.test.mjs scripts/__tests__/canary-startup-playwright.test.mjs scripts/__tests__/cursor-public-install-sandbox.test.mjs scripts/__tests__/grok-public-install-sandbox.test.mjs scripts/__tests__/public-install-closure.test.mjs scripts/__tests__/prefetch-child.test.mjs scripts/__tests__/stage-bundled-packages.test.mjs scripts/__tests__/pack-staged-packages.test.mjs scripts/__tests__/retain-runner-qualification-packages.test.mjs scripts/__tests__/runner-release-crate-cache.test.mjs scripts/__tests__/vitest-worker-isolation.test.mjs scripts/verify-release-registry-state.test.mjs scripts/release-package-map.test.mjs scripts/check-release-package-bootstrap.test.mjs scripts/check-no-git-push.test.mjs scripts/release-lib.test.mjs scripts/release-registry-versions.test.mjs scripts/link-plugin-dev-sdk.test.js scripts/acpx-patch-packaging.test.mjs scripts/service-onboard-smoke.test.mjs scripts/docker-onboard-smoke.test.mjs scripts/preview-artifacts.test.mjs scripts/cloud-migrator-artifacts.test.mjs",
     "storybook-visual:baseline": "node scripts/storybook-visual-baseline.mjs",
     "test:storybook-visual": "node scripts/storybook-visual-baseline.mjs download && node scripts/storybook-visual-baseline.mjs verify && pnpm build-storybook && npx playwright test --config tests/storybook-visual/playwright.config.ts",
     "test:storybook-visual:update": "node scripts/storybook-visual-baseline.mjs download && pnpm build-storybook && npx playwright test --config tests/storybook-visual/playwright.config.ts --update-snapshots && node scripts/storybook-visual-baseline.mjs pack",

packages/adapter-utils/src/acpx-engine/execute.test.ts

@@ -5275,7 +5275,7 @@ describe("ACPX engine sandbox-start spans (opt-in root + child parenting)", () =
     };
   }
 
-  it("emits one root span with a child span per bring-up boundary, each parented to the root", async () => {
+  it("emits one root with startup and nested workspace boundaries", async () => {
     const root = await makeTempRoot();
     const stateDir = path.join(root, "state");
     const localCwd = path.join(root, "worktree");
@@ -5334,12 +5334,6 @@ describe("ACPX engine sandbox-start spans (opt-in root + child parenting)", () =
     expect(syncBackSpan!.parent).toBe(runRootSpan);
     expect(syncBackSpan!.ended).toBe(true);
 
-    // A codex bring-up over the remote sandbox lane crosses all 7 boundaries.
-    // Each boundary span parents to the sandbox bring-up span, not to the run
-    // root or the turn span. The `stage.sync` step also opens the two pre-task
-    // sub-step spans — `snapshot.git` and `snapshot.baseline` — plus the
-    // `stage.workspace` inbound task span, and the `pack` span nests one level
-    // deeper under `stage.workspace`.
     const childNames = spans
       .filter(
         (span) =>
@@ -5362,16 +5356,21 @@ describe("ACPX engine sandbox-start spans (opt-in root + child parenting)", () =
         "snapshot.git",
         "stage.sync",
         "stage.workspace",
+        "workspace.ignore_rules",
+        "workspace.inbound_barrier",
+        "workspace.overlay_archive",
+        "workspace.progress",
+        "workspace.progress",
+        "workspace.repositories",
         "workspace.resolve",
+        "workspace.source_root",
+        "workspace.transfer",
       ],
     );
 
-    // The two pre-task sub-step spans and the `stage.workspace` task span nest
-    // under the `stage.sync` step span (that host work runs inside the step), not
-    // directly under the bring-up span. The `pack` span nests under
-    // `stage.workspace`, because the host builds the tarball inside the task.
     const stageSyncSpan = spans.find((span) => span.name === "stage.sync");
     const stageWorkspaceSpan = spans.find((span) => span.name === "stage.workspace");
+    const inboundSpan = spans.find((span) => span.name === "workspace.inbound_barrier");
     const packSpan = spans.find((span) => span.name === "pack");
     const snapshotGitSpan = spans.find((span) => span.name === "snapshot.git");
     const snapshotBaselineSpan = spans.find((span) => span.name === "snapshot.baseline");
@@ -5380,15 +5379,12 @@ describe("ACPX engine sandbox-start spans (opt-in root + child parenting)", () =
     expect(packSpan).toBeTruthy();
     expect(snapshotGitSpan).toBeTruthy();
     expect(snapshotBaselineSpan).toBeTruthy();
-    expect(stageWorkspaceSpan!.parent).toBe(stageSyncSpan);
+    expect(stageWorkspaceSpan!.parent).toBe(inboundSpan);
     expect(packSpan!.parent).toBe(stageWorkspaceSpan);
     expect(snapshotGitSpan!.parent).toBe(stageSyncSpan);
     expect(snapshotBaselineSpan!.parent).toBe(stageSyncSpan);
     expect(packSpan!.ended).toBe(true);
 
-    // Every boundary step span parents to the sandbox bring-up span and ends.
-    // The `stage.sync` sub-step spans and the run-parented `sandbox.syncBack`
-    // span are the exceptions: they parent above.
     const nonStartupChildren = new Set([
       packSpan,
       stageWorkspaceSpan,
@@ -5396,6 +5392,23 @@ describe("ACPX engine sandbox-start spans (opt-in root + child parenting)", () =
       snapshotBaselineSpan,
       syncBackSpan,
     ]);
+    for (const [name, parent] of [
+      ["workspace.source_root", stageSyncSpan], ["workspace.repositories", stageSyncSpan],
+      ["workspace.ignore_rules", stageSyncSpan], ["workspace.inbound_barrier", stageSyncSpan],
+      ["workspace.overlay_archive", packSpan], ["workspace.transfer", stageWorkspaceSpan],
+    ] as const) {
+      const span = spans.find(span => span.name === name);
+      expect(span?.parent, name).toBe(parent);
+      expect(span?.ended, name).toBe(true);
+      expect(span?.endCalls, name).toBe(1);
+      nonStartupChildren.add(span);
+    }
+    const progress = spans.filter(span => span.name === "workspace.progress");
+    expect(progress.map(span => span.parent)).toEqual([packSpan, stageWorkspaceSpan]);
+    for (const span of progress) {
+      expect(span.endCalls).toBe(1);
+      nonStartupChildren.add(span);
+    }
     for (const span of spans) {
       if (span === runRootSpan || span === startupSpan || span === turnSpan) continue;
       if (nonStartupChildren.has(span)) continue;

packages/adapter-utils/src/acpx-engine/startup-timing.ts

@@ -47,6 +47,12 @@ export function normalizeProviderFamily(key: string | undefined): string {
  */
 export const SANDBOX_STARTUP_SPAN_ATTR_PREFIX = "paperclip.sandbox.startup.";
 
+export const EXECUTION_PREPARATION_SPAN_ATTRS = {
+  runId: "paperclip.task.run.run_id",
+  fileCount: "paperclip.execution.fileCount",
+  bytes: "paperclip.execution.bytes",
+} as const;
+
 /**
  * The closed attribute-name contract for every sandbox-startup span. This is
  * the single source of truth for the harness span attributes. Each name uses

packages/adapter-utils/src/sandbox-managed-runtime.test.ts

@@ -2943,7 +2943,11 @@ describe("sandbox managed runtime", () => {
     // The workspace stage task opens its own `stage.workspace` span, and the
     // host tarball build opens the `pack` span inside it. The two pre-task
     // sub-steps stay ahead of the task.
-    expect(openedSpans).toEqual(["snapshot.git", "snapshot.baseline", "stage.workspace", "pack"]);
+    expect(openedSpans).toEqual([
+      "workspace.source_root", "snapshot.git", "workspace.repositories", "workspace.ignore_rules",
+      "snapshot.baseline", "workspace.inbound_barrier", "stage.workspace", "pack",
+      "workspace.progress", "workspace.overlay_archive", "workspace.transfer", "workspace.progress",
+    ]);
     // The tarball build still lands the workspace inside the span, so the wrap
     // changes no staging behavior.
     await expect(readFile(path.join(remoteWorkspaceDir, "README.md"), "utf8")).resolves.toBe("workspace body\n");
@@ -3002,8 +3006,11 @@ describe("sandbox managed runtime", () => {
     expect(packSpan!.ended).toBe(true);
     // The workspace stage task opens its own `stage.workspace` span under
     // `stage.sync`, and the `pack` span nests under `stage.workspace`.
-    expect(workspaceSpan!.parentName).toBe("stage.sync");
+    expect(workspaceSpan!.parentName).toBe("workspace.inbound_barrier");
     expect(packSpan!.parentName).toBe("stage.workspace");
+    expect(spans.find(span => span.name === "workspace.inbound_barrier")?.parentName).toBe("stage.sync");
+    expect(spans.find(span => span.name === "workspace.overlay_archive")?.parentName).toBe("pack");
+    expect(spans.find(span => span.name === "workspace.transfer")?.parentName).toBe("stage.workspace");
 
     // The two pre-`pack` staging sub-steps nest under `stage.sync` the same way,
     // so the previously hidden gap at the head of the step is now attributed.

packages/adapter-utils/src/sandbox-managed-runtime.ts

@@ -1265,11 +1265,13 @@ export async function prepareSandboxManagedRuntime(input: {
   // span, because the teardown runs the restore inside that span.
   const runStepSpan = <T>(name: string, work: () => Promise<T>): Promise<T> =>
     input.runtimeSpan ? input.runtimeSpan(name, work) : work();
+  const reportWorkspaceStatus = (phase: RuntimeStatusPhase, message: string) =>
+    runStepSpan("workspace.progress", () => emitRuntimeStatus(input.onRuntimeProgress, phase, message));
 
   // Resolve an existing workspace alias once, before reading its snapshot.
   // All subsequent work uses that root, so retargeting the alias cannot select
   // another repository. Staging also verifies the captured directory identity.
-  const workspaceRoot = syncWorkspace ? await captureWorkspaceSourceRoot(input.workspaceLocalDir) : null;
+  const workspaceRoot = syncWorkspace ? await runStepSpan("workspace.source_root", () => captureWorkspaceSourceRoot(input.workspaceLocalDir)) : null;
   if (workspaceRoot) input = { ...input, workspaceLocalDir: workspaceRoot.sourceDir };
 
   // The git enumeration (`git status --ignored`, the HEAD diffs, `ls-files`).
@@ -1282,7 +1284,7 @@ export async function prepareSandboxManagedRuntime(input: {
           readGitWorkspaceSnapshot(input.workspaceLocalDir),
         )
     : null;
-  const repositories = gitSnapshot ? (gitSnapshot.repositories ?? []) : input.workspaceRepositories ?? (syncWorkspace ? await readManagedWorkspaceRepositories(input.workspaceLocalDir) : []);
+  const repositories = gitSnapshot ? (gitSnapshot.repositories ?? []) : input.workspaceRepositories ?? (syncWorkspace ? await runStepSpan("workspace.repositories", () => readManagedWorkspaceRepositories(input.workspaceLocalDir)) : []);
   const repositoryGitPaths = (children: typeof repositories, prefix = ""): string[] => children.flatMap(repo => {
     const relative = prefix ? `${prefix}/${repo.path}` : repo.path;
     return [`${relative}/.git`, ...repositoryGitPaths(repo.snapshot.repositories ?? [], relative)];
@@ -1315,7 +1317,7 @@ export async function prepareSandboxManagedRuntime(input: {
   // repository's ignore rules still govern which files may leave the host.
   // Use the same bounded, path-relative resolver as referenced project trees.
   const directoryIgnore = syncWorkspace && !gitSnapshot && input.workspaceFileMode !== "all"
-    ? await resolveReferencedSourceIgnore(input.workspaceLocalDir)
+    ? await runStepSpan("workspace.ignore_rules", () => resolveReferencedSourceIgnore(input.workspaceLocalDir))
     : null;
   if (directoryIgnore?.kind === "failed") {
     throw new Error(`Workspace ignore scan failed: ${directoryIgnore.reason}`);
@@ -1368,9 +1370,9 @@ export async function prepareSandboxManagedRuntime(input: {
   }
 
   const seedGeneration = input.workspaceSeedCacheDirectory && baselineSnapshot && workspaceInboundMode !== "durable_seed"
-    ? workspaceSeedGeneration(baselineSnapshot, gitSnapshot, repositories) : null;
+    ? await runStepSpan("workspace.seed_generation", async () => workspaceSeedGeneration(baselineSnapshot!, gitSnapshot, repositories)) : null;
   const cachedSeed = seedGeneration
-    ? await readWorkspaceSeedGeneration(input.workspaceSeedCacheDirectory!, seedGeneration) : null;
+    ? await runStepSpan("workspace.seed_lookup", () => readWorkspaceSeedGeneration(input.workspaceSeedCacheDirectory!, seedGeneration)) : null;
   const replaySeed = workspaceInboundMode === "durable_seed" ? input.workspaceDurableSeed : cachedSeed;
 
   // Every inbound staging step delegates to the provider through `client.syncIn`:
@@ -1467,7 +1469,7 @@ export async function prepareSandboxManagedRuntime(input: {
         params.progressLabel,
         { sink: input.onRuntimeProgress, phase: params.statusPhase },
       );
-      const syncResult = await syncIn(operations);
+      const syncResult = await runStepSpan("workspace.transfer", () => syncIn(operations));
       // Prefer the transport's own byte total. It is the real count for a
       // provider that has no host tarball to stat (a referenced project rides
       // a `directory` mapping). Fall back to the caller-supplied count when
@@ -1475,7 +1477,7 @@ export async function prepareSandboxManagedRuntime(input: {
       // shows the host-known workspace total.
       const transferredBytes = sumSyncResultBytes(syncResult);
       const reportedBytes = transferredBytes > 0 ? transferredBytes : params.progressBytes;
-      await upload.finish(reportedBytes, reportedBytes);
+      await runStepSpan("workspace.progress", () => upload.finish(reportedBytes, reportedBytes));
     };
 
     // Build the ordered inbound operation task list. Each task stages one inbound
@@ -1522,7 +1524,7 @@ export async function prepareSandboxManagedRuntime(input: {
             //    wipes the target tree EXCEPT `.paperclip-runtime`, so the overlay tar,
             //    which sits under `.paperclip-runtime`, survives to run its own extract.
             if (gitSnapshot || repositories.length) {
-              await emitRuntimeStatus(input.onRuntimeProgress, "git_sync", "Syncing git history to environment");
+              await reportWorkspaceStatus("git_sync", "Syncing git history to environment");
               const gitTarPath = path.join(tempDir, "git-workspace.tar");
               const remoteGitTar = path.posix.join(
                 runtimeRootDir,
@@ -1533,26 +1535,26 @@ export async function prepareSandboxManagedRuntime(input: {
                 if (!durableGitArchive) {
                   throw new Error("workspace_durable_seed_git_missing");
                 }
-                await copyDurableSeedArchive({
+                await runStepSpan("workspace.seed_replay", () => copyDurableSeedArchive({
                   sourcePath: durableGitArchive,
                   targetPath: gitTarPath,
                   expectedSha256: replaySeed.gitArchiveSha256,
-                });
+                }));
               } else {
-                const archive = async (cloneDir: string) => createTarballFromDirectory({ localDir: cloneDir, archivePath: gitTarPath, exclude: [".paperclip-runtime"] });
-                if (gitSnapshot) await withShallowGitWorkspaceClone({ localDir: input.workspaceLocalDir, snapshot: gitSnapshot }, archive);
-                else await withTempDir("paperclip-managed-repositories-", async (cloneDir) => {
+                const archive = async (cloneDir: string) => runStepSpan("workspace.git_archive", () => createTarballFromDirectory({ localDir: cloneDir, archivePath: gitTarPath, exclude: [".paperclip-runtime"] }));
+                if (gitSnapshot) await runStepSpan("workspace.git_clone", () => withShallowGitWorkspaceClone({ localDir: input.workspaceLocalDir, snapshot: gitSnapshot }, archive));
+                else await runStepSpan("workspace.git_clone", () => withTempDir("paperclip-managed-repositories-", async (cloneDir) => {
                   for (const repo of repositories) await withShallowGitWorkspaceClone({ localDir: path.join(input.workspaceLocalDir, repo.path), snapshot: repo.snapshot }, async (child) => copyCloneTree(child, path.join(cloneDir, repo.path)));
                   await archive(cloneDir);
-                });
+                }));
                 if (input.workspaceDurableSeed?.gitArchivePath) {
-                  await persistDurableSeedArchive({
+                  await runStepSpan("workspace.seed_persist", () => persistDurableSeedArchive({
                     sourcePath: gitTarPath,
-                    targetPath: input.workspaceDurableSeed.gitArchivePath,
-                  });
+                    targetPath: input.workspaceDurableSeed!.gitArchivePath!,
+                  }));
                 }
               }
-              if (cachedSeed && input.workspaceDurableSeed?.gitArchivePath) await persistDurableSeedArchive({ sourcePath: gitTarPath, targetPath: input.workspaceDurableSeed.gitArchivePath });
+              if (cachedSeed && input.workspaceDurableSeed?.gitArchivePath) await runStepSpan("workspace.seed_persist", () => persistDurableSeedArchive({ sourcePath: gitTarPath, targetPath: input.workspaceDurableSeed!.gitArchivePath! }));
               workspaceFiles.push({
                 sourcePath: gitTarPath,
                 targetPath: remoteGitTar,
@@ -1573,41 +1575,41 @@ export async function prepareSandboxManagedRuntime(input: {
             // 2. workspace-overlay tar. A git-backed overlay merges on top of the just
             //    extracted git tree (no wipe); a plain workspace wipes every child except
             //    the preserved names first. The extract runs AFTER the git extract.
-            await emitRuntimeStatus(input.onRuntimeProgress, "config_sync", "Syncing workspace to environment");
+            await reportWorkspaceStatus("config_sync", "Syncing workspace to environment");
             const workspaceTarPath = path.join(tempDir, "workspace.tar");
             if (replaySeed) {
-              await copyDurableSeedArchive({
+              await runStepSpan("workspace.seed_replay", () => copyDurableSeedArchive({
                 sourcePath: replaySeed.workspaceArchivePath,
                 targetPath: workspaceTarPath,
                 expectedSha256:
                   replaySeed.workspaceArchiveSha256,
-              });
+              }));
             } else {
               const workspaceArchiveDir = gitSnapshot || repositories.length
                 ? path.join(tempDir, "workspace-overlay")
                 : input.workspaceLocalDir;
               if (gitSnapshot || repositories.length) {
-                await copySelectedWorkspaceEntries({
+                await runStepSpan("workspace.overlay_copy", () => copySelectedWorkspaceEntries({
                   sourceRoot: workspaceRoot!,
                   targetDir: workspaceArchiveDir,
                   relativePaths: (input.workspaceFileMode === "all" ? undefined : gitSnapshot?.overlayPaths) ?? (baselineSnapshot!.entries instanceof WorkspaceManifestMap ? baselineSnapshot!.entries.manifest : [...baselineSnapshot!.entries.keys()]),
                   ignoredPaths,
                   exclude: workspaceArchiveExclude,
-                });
+                }));
               }
-              await createTarballFromDirectory({
+              await runStepSpan("workspace.overlay_archive", () => createTarballFromDirectory({
                 localDir: workspaceArchiveDir,
                 archivePath: workspaceTarPath,
                 exclude: gitSnapshot || repositories.length ? undefined : workspaceArchiveExclude,
-              });
+              }));
               if (input.workspaceDurableSeed) {
-                await persistDurableSeedArchive({
+                await runStepSpan("workspace.seed_persist", () => persistDurableSeedArchive({
                   sourcePath: workspaceTarPath,
-                  targetPath: input.workspaceDurableSeed.workspaceArchivePath,
-                });
+                  targetPath: input.workspaceDurableSeed!.workspaceArchivePath,
+                }));
               }
             }
-            if (cachedSeed && input.workspaceDurableSeed) await persistDurableSeedArchive({ sourcePath: workspaceTarPath, targetPath: input.workspaceDurableSeed.workspaceArchivePath });
+            if (cachedSeed && input.workspaceDurableSeed) await runStepSpan("workspace.seed_persist", () => persistDurableSeedArchive({ sourcePath: workspaceTarPath, targetPath: input.workspaceDurableSeed!.workspaceArchivePath }));
             const remoteWorkspaceTar = path.posix.join(
               runtimeRootDir,
               "workspace-upload.tar",
@@ -1631,7 +1633,7 @@ export async function prepareSandboxManagedRuntime(input: {
             if ((isPathManifest(deletedPaths) ? deletedPaths.count : deletedPaths.length) > 0) {
               const deletedManifest = path.join(tempDir, "deleted.nul");
               const remoteDeletedManifest = path.posix.join(runtimeRootDir, "deleted.nul");
-              await writeWorkspacePaths(deletedPaths, deletedManifest);
+              await runStepSpan("workspace.deleted_manifest", () => writeWorkspacePaths(deletedPaths, deletedManifest));
               workspaceFiles.push({ sourcePath: deletedManifest, targetPath: remoteDeletedManifest, kind: "file", access: "rw", writablePath: runtimeRootDir });
               workspacePostUploadCommands.push({ command: buildRemoveDeletedPathsCommand({ remoteDir: workspaceRemoteDir, manifestPath: remoteDeletedManifest }) });
             }
@@ -1640,14 +1642,14 @@ export async function prepareSandboxManagedRuntime(input: {
 
           if (seedGeneration && !cachedSeed && input.workspaceDurableSeed) {
             const seed = input.workspaceDurableSeed;
-            await publishWorkspaceSeedGeneration(input.workspaceSeedCacheDirectory!, seedGeneration,
+            await runStepSpan("workspace.seed_publish", () => publishWorkspaceSeedGeneration(input.workspaceSeedCacheDirectory!, seedGeneration,
               { ...seed, gitArchivePath: gitSnapshot || repositories.length ? seed.gitArchivePath : null }, {
                 companyDirectory: input.workspaceSeedCacheCompanyDirectory,
                 verify: () => runStepSpan("seed.verify", async () => {
                   const verifiedRoot = path.join(tempDir, "verify-seed");
                   try {
-                    if (gitSnapshot || repositories.length) await extractTarballToDirectory({ archivePath: path.join(tempDir, "git-workspace.tar"), localDir: verifiedRoot });
-                    await extractTarballToDirectory({ archivePath: path.join(tempDir, "workspace.tar"), localDir: verifiedRoot, exclude: repositoryMetadataExcludes });
+                    if (gitSnapshot || repositories.length) await runStepSpan("workspace.seed_extract", () => extractTarballToDirectory({ archivePath: path.join(tempDir, "git-workspace.tar"), localDir: verifiedRoot }));
+                    await runStepSpan("workspace.seed_extract", () => extractTarballToDirectory({ archivePath: path.join(tempDir, "workspace.tar"), localDir: verifiedRoot, exclude: repositoryMetadataExcludes }));
                     for (const relative of workspacePaths(deletedPaths)) {
                       let parent = verifiedRoot;
                       for (const segment of relative.split("/").slice(0, -1)) {
@@ -1657,12 +1659,12 @@ export async function prepareSandboxManagedRuntime(input: {
                       }
                       await fs.rm(path.join(verifiedRoot, relative), { recursive: true, force: true });
                     }
-                    const verified = await captureWorkspaceSnapshot(verifiedRoot, { exclude: baselineSnapshot!.exclude, ignoredPaths: baselineSnapshot!.ignoredPaths, diskBacked: true });
+                    const verified = await runStepSpan("workspace.seed_snapshot", () => captureWorkspaceSnapshot(verifiedRoot, { exclude: baselineSnapshot!.exclude, ignoredPaths: baselineSnapshot!.ignoredPaths, diskBacked: true }));
                     try { return directorySnapshotSha256(verified) === directorySnapshotSha256(baselineSnapshot!); }
                     finally { await disposeDirectorySnapshot(verified); }
-                  } finally { await fs.rm(verifiedRoot, { recursive: true, force: true }); }
+                  } finally { await runStepSpan("workspace.seed_cleanup", () => fs.rm(verifiedRoot, { recursive: true, force: true })); }
                 }),
-              });
+              }));
           }
           if (!stageWorkspace) return;
 
@@ -1821,11 +1823,11 @@ export async function prepareSandboxManagedRuntime(input: {
     // tasks under the client's concurrency permission and the bound, and it
     // settles every started task before it returns. This is the startup barrier:
     // the coordinator returns only after every upload settles.
-    const inboundResults = await scheduleSyncOperations(
+    const inboundResults = await runStepSpan("workspace.inbound_barrier", () => scheduleSyncOperations(
       inboundTasks,
       input.client.allowConcurrentSyncOperations === true,
       SYNC_OPERATION_CONCURRENCY_LIMIT,
-    );
+    ));
 
     // Collect the referenced-project failures in stable input order.
     for (const failure of additionalFailureSlots) {

packages/adapter-utils/src/workspace-snapshot-thread.test.ts

@@ -81,6 +81,9 @@ describe("workspace snapshot threads", () => {
     await expect(fs.stat(created[0]!)).rejects.toMatchObject({ code: "ENOENT" });
   });
 
+  // Multiple real TypeScript worker cold starts can exceed the default 5s on
+  // CI (three queued workers use two slots). These cases verify cleanup and
+  // error propagation, not a worker startup deadline. Keep completion bounded.
   it("preserves filesystem errors, cleans failed workers and releases queued jobs", async () => {
     const root = await fixture();
     const file = path.join(root, "file");
@@ -103,7 +106,7 @@ describe("workspace snapshot threads", () => {
     if (results[2].status === "fulfilled") await disposeDirectorySnapshot(results[2].value);
     expect(created).toHaveLength(3);
     for (const directory of created) await expect(fs.stat(directory)).rejects.toMatchObject({ code: "ENOENT" });
-  });
+  }, 15_000);
 
   it.each(["unchanged", "changed", "sparse"] as const)("isolates restore captures and cleans their manifests: %s", async mode => {
     const target = await fixture(), source = await fixture();
@@ -129,7 +132,7 @@ describe("workspace snapshot threads", () => {
     expect(await fs.readFile(path.join(target, "file"), "utf8")).toBe(mode === "unchanged" ? "before" : "after");
     expect(await fs.readFile(path.join(target, "concurrent"), "utf8")).toBe("host bytes");
     for (const directory of created) await expect(fs.stat(directory)).rejects.toMatchObject({ code: "ENOENT" });
-  });
+  }, 15_000);
 
   it("retains the in-memory snapshot contract when disk backing is not requested", async () => {
     const root = await fixture();

packages/paperclip-runner/src/live/runnerd-codex-transport-deadlines.test.ts

@@ -0,0 +1,152 @@
+import { chmod, cp, mkdir, mkdtemp, readFile, rm, stat, writeFile } from "node:fs/promises";
+import { readFileSync } from "node:fs";
+import { createHash } from "node:crypto";
+import { tmpdir } from "node:os";
+import { join, resolve } from "node:path";
+import { fileURLToPath } from "node:url";
+import { expect, it, vi } from "vitest";
+import { QUALIFIED_ACPX_PROFILES } from "../drivers/acpx/qualified-profiles.js";
+import { codexSemanticToolSpecs } from "../drivers/codex/codex-app-server-driver.js";
+import { DurablePrpControlPlane } from "../control-plane/durable-prp-control-plane.js";
+import * as durableControlPlane from "../control-plane/durable-prp-control-plane.js";
+import {
+  createCapabilityRunnerdCodexTransport,
+  defaultCapabilityRunnerdBinary as qualifiedCapabilityRunnerdBinary,
+} from "./runnerd-codex-transport.js";
+
+// These regressions must cross real production timing boundaries. Keep their
+// waits intact, but let the existing file worker pool overlap them with the
+// rest of the transport suite instead of extending its serial critical path.
+const defaultCapabilityRunnerdBinary = () =>
+  process.env.PAPERCLIP_ATTACH_TRANSITION_RUNNER ?? qualifiedCapabilityRunnerdBinary();
+const fakeCodex = resolve(import.meta.dirname, "../../runner/target/debug/fake-codex-app-server");
+function fakeCodexArgs(stateDirectory: string, ...args: string[]): string[] {
+  return ["--state-file", join(stateDirectory, "fake-codex-state.json"), ...args];
+}
+
+it("keeps a quiet active Codex turn in the same process across connection lease expiry", async () => {
+  const stateDirectory = await mkdtemp(join(tmpdir(), "runnerd-renew-active-"));
+  const callsPath = join(stateDirectory, "calls.log");
+  const cores: DurablePrpControlPlane[] = [];
+  const OriginalCore = durableControlPlane.DurablePrpControlPlane;
+  const coreSpy = vi.spyOn(durableControlPlane, "DurablePrpControlPlane")
+    .mockImplementation(function(options: ConstructorParameters<typeof OriginalCore>[0]) {
+      const core = new OriginalCore({ ...options, connectionLeaseTtlMs: 60_000 });
+      cores.push(core);
+      return core;
+    } as unknown as typeof OriginalCore);
+  const bundle = createCapabilityRunnerdCodexTransport({
+    runnerBinary: defaultCapabilityRunnerdBinary(), codexCommand: fakeCodex,
+    codexArgs: fakeCodexArgs(stateDirectory, "--hold-turn", "--record-process-start", "--call-log", callsPath),
+    stateDirectory, runnerReconnectGraceMs: 5_000,
+  });
+  try {
+    const opened = await bundle.transport.request("thread/start", { cwd: tmpdir(), dynamicTools: [] });
+    await bundle.transport.request("turn/start", { input: [{ type: "text", text: "Keep working quietly." }] });
+    const runnerPid = bundle.evidence().runnerPid;
+    const codexPid = bundle.evidence().codexPid;
+    expect(runnerPid).toBeGreaterThan(0);
+    expect(codexPid).toBeGreaterThan(0);
+    const core = cores[0]!;
+    const original = structuredClone(Object.values(core.store.state.leases)[0]!);
+    await vi.waitFor(() => {
+      expect(Date.now()).toBeGreaterThan(original.expiresAtUnixMs + 1_000);
+    }, { timeout: 75_000, interval: 1_000 });
+    const current = Object.values(core.store.state.leases)[0]!;
+    expect(current.expiresAtUnixMs).toBeGreaterThan(original.expiresAtUnixMs);
+    expect(current.leaseId).toBe(original.leaseId);
+    expect(core.store.state.connectionCount).toBe(1);
+    expect(bundle.evidence().runnerPid).toBe(runnerPid);
+    expect(bundle.evidence().codexPid).toBe(codexPid);
+    process.kill(runnerPid!, 0);
+    process.kill(codexPid!, 0);
+    const calls = (await readFile(callsPath, "utf8")).trim().split(/\r?\n/);
+    expect(calls.filter(call => call === "process-start")).toHaveLength(1);
+    expect(calls.filter(call => call === "thread/start")).toHaveLength(1);
+    expect(calls.filter(call => call === "turn/start")).toHaveLength(1);
+    expect(calls).not.toContain("turn/interrupt");
+    expect(calls).not.toContain("thread/resume");
+    const state = JSON.parse(await readFile(join(stateDirectory, "fake-codex-state.json"), "utf8"));
+    expect(state.threadId).toBe(opened.thread.id);
+    expect(state.activeTurnId).toBe("provider-turn-1");
+  } finally {
+    await bundle.transport.close();
+    coreSpy.mockRestore();
+    await rm(stateDirectory, { recursive: true, force: true });
+  }
+}, 90_000);
+
+it("reopens Pi after a completed turn within its cold admission budget during warm attachment", async () => {
+  const root = await mkdtemp(join(tmpdir(), "runnerd-pi-warm-admission-"));
+  const providerNode = process.platform === "linux" ? join(root, "node") : process.execPath;
+  if (process.platform === "linux") {
+    await cp(process.execPath, providerNode, { dereference: true });
+    await chmod(providerNode, 0o500);
+    expect((await stat(providerNode)).mode & 0o777).toBe(0o500);
+  }
+  const cliRoot = join(root, "dist/cli");
+  await mkdir(cliRoot, { recursive: true, mode: 0o700 });
+  const sidecarPath = join(cliRoot, "acpx-runtime-sidecar.cjs");
+  const journal = join(root, "commands.ndjson");
+  const fixture = await readFile(fileURLToPath(new URL("./fixtures/fake-pi-warm-sidecar.cjs", import.meta.url)), "utf8");
+  await writeFile(sidecarPath, fixture.replace("/* fixture-config */ null", JSON.stringify({ journal, resumeDelayMs: 32_000, commandDigest: QUALIFIED_ACPX_PROFILES.pi.commandDigest })), { mode: 0o600 });
+  const digest = (path: string) => `sha256:${createHash("sha256").update(readFileSync(path)).digest("hex")}`;
+  const bundle = createCapabilityRunnerdCodexTransport({
+    provider: "acpx", acpxAgent: "pi", piThinkingLevel: "low", acpxPermissionMode: "deny-all",
+    runnerBinary: defaultCapabilityRunnerdBinary(), stateDirectory: root, runnerFilesystemRoot: root,
+    providerNodeCommand: providerNode, providerNodeCommandSha256: digest(providerNode),
+    acpxSidecarPath: sidecarPath, acpxSidecarSha256: digest(sidecarPath),
+    providerPackAuthorityDigest: `sha256:${"d".repeat(64)}`,
+    lifecyclePolicy: { mode: "warm", idleTimeoutMs: 60_000 },
+    environment: { PATH: "/usr/bin:/bin" },
+  });
+  bundle.transport.setServerRequestHandler(async () => ({
+    success: true,
+    contentItems: [{ type: "inputText", text: "Completion report accepted." }],
+  }));
+  let failure: unknown;
+  try {
+    await bundle.transport.request("initialize", {});
+    await bundle.transport.request("thread/start", {
+      cwd: root, model: "openrouter/deepseek/deepseek-v4-flash-0731",
+      dynamicTools: codexSemanticToolSpecs(),
+      completionContract: { revision: "warm-pi-contract", criterionIds: [] },
+    });
+    const runnerPid = bundle.evidence().runnerPid;
+    const notifications = bundle.transport.notifications()[Symbol.asyncIterator]();
+    const completeTurn = async () => {
+      await bundle.transport.request("turn/start", { input: [{ type: "text", text: "Complete the fixture turn." }] });
+      for (let index = 0; index < 64; index += 1) {
+        const next = await Promise.race([
+          notifications.next(),
+          new Promise<never>((_, reject) => setTimeout(() => reject(new Error("Fixture completion timed out")), 5_000)),
+        ]);
+        if (next.value?.method === "turn/completed") return;
+      }
+      throw new Error("Fixture turn did not complete");
+    };
+    await completeTurn();
+    const started = Date.now();
+    await bundle.transport.attachRun!({ runId: "run-pi-warm-second", turnId: "turn-pi-warm-second", itemId: "item-pi-warm-second" });
+    expect(Date.now() - started).toBeGreaterThan(30_000);
+    await completeTurn();
+    expect(bundle.evidence()).toMatchObject({ runnerPid, runnerExited: false });
+    const commands = (await readFile(journal, "utf8")).trim().split("\n").map((line) => JSON.parse(line));
+    expect(commands.filter((entry) => entry.event === "spawn")).toHaveLength(2);
+    expect(commands.filter((entry) => entry.command === "session.open")).toHaveLength(2);
+    expect(commands.filter((entry) => entry.resumed)).toHaveLength(1);
+    expect(commands.filter((entry) => entry.command === "turn.start")).toHaveLength(2);
+    expect(commands.filter((entry) => entry.command === "session.close")).toHaveLength(1);
+  } catch (error) {
+    failure = error;
+    throw error;
+  } finally {
+    try {
+      await bundle.transport.close();
+    } catch (cleanupError) {
+      if (failure) throw new AggregateError([failure, cleanupError], "Warm attachment and strict cleanup failed");
+      throw cleanupError;
+    }
+    await rm(root, { recursive: true, force: true });
+  }
+}, 75_000);

packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts

@@ -1,4 +1,3 @@
-import { QUALIFIED_ACPX_PROFILES } from "../drivers/acpx/qualified-profiles.js";
 import { coldAdmissionTimeoutMs, waitForRunnerCommand } from "./runnerd-codex-transport.js";
 import {
   chmod,
@@ -3341,58 +3340,6 @@ it("expands coalesced canonical items without dropping strict bindings", () => {
   ]);
 });
 
-it("keeps a quiet active Codex turn in the same process across connection lease expiry", async () => {
-  const stateDirectory = await mkdtemp(join(tmpdir(), "runnerd-renew-active-"));
-  const callsPath = join(stateDirectory, "calls.log");
-  const cores: DurablePrpControlPlane[] = [];
-  const OriginalCore = durableControlPlane.DurablePrpControlPlane;
-  const coreSpy = vi.spyOn(durableControlPlane, "DurablePrpControlPlane")
-    .mockImplementation(function(options: ConstructorParameters<typeof OriginalCore>[0]) {
-      const core = new OriginalCore({ ...options, connectionLeaseTtlMs: 60_000 });
-      cores.push(core);
-      return core;
-    } as unknown as typeof OriginalCore);
-  const bundle = createCapabilityRunnerdCodexTransport({
-    runnerBinary: defaultCapabilityRunnerdBinary(), codexCommand: fakeCodex,
-    codexArgs: fakeCodexArgs(stateDirectory, "--hold-turn", "--record-process-start", "--call-log", callsPath),
-    stateDirectory, runnerReconnectGraceMs: 5_000,
-  });
-  try {
-    const opened = await bundle.transport.request("thread/start", { cwd: tmpdir(), dynamicTools: [] });
-    await bundle.transport.request("turn/start", { input: [{ type: "text", text: "Keep working quietly." }] });
-    const runnerPid = bundle.evidence().runnerPid;
-    const codexPid = bundle.evidence().codexPid;
-    expect(runnerPid).toBeGreaterThan(0);
-    expect(codexPid).toBeGreaterThan(0);
-    const core = cores[0]!;
-    const original = structuredClone(Object.values(core.store.state.leases)[0]!);
-    await vi.waitFor(() => {
-      expect(Date.now()).toBeGreaterThan(original.expiresAtUnixMs + 1_000);
-    }, { timeout: 75_000, interval: 1_000 });
-    const current = Object.values(core.store.state.leases)[0]!;
-    expect(current.expiresAtUnixMs).toBeGreaterThan(original.expiresAtUnixMs);
-    expect(current.leaseId).toBe(original.leaseId);
-    expect(core.store.state.connectionCount).toBe(1);
-    expect(bundle.evidence().runnerPid).toBe(runnerPid);
-    expect(bundle.evidence().codexPid).toBe(codexPid);
-    process.kill(runnerPid!, 0);
-    process.kill(codexPid!, 0);
-    const calls = (await readFile(callsPath, "utf8")).trim().split(/\r?\n/);
-    expect(calls.filter(call => call === "process-start")).toHaveLength(1);
-    expect(calls.filter(call => call === "thread/start")).toHaveLength(1);
-    expect(calls.filter(call => call === "turn/start")).toHaveLength(1);
-    expect(calls).not.toContain("turn/interrupt");
-    expect(calls).not.toContain("thread/resume");
-    const state = JSON.parse(await readFile(join(stateDirectory, "fake-codex-state.json"), "utf8"));
-    expect(state.threadId).toBe(opened.thread.id);
-    expect(state.activeTurnId).toBe("provider-turn-1");
-  } finally {
-    await bundle.transport.close();
-    coreSpy.mockRestore();
-    await rm(stateDirectory, { recursive: true, force: true });
-  }
-}, 90_000);
-
 it("runs the lab provider boundary through authenticated durable PRP", async () => {
   const stateDirectory = await mkdtemp(join(tmpdir(), "runnerd-lab-provider-"));
   const bundle = createCapabilityRunnerdCodexTransport({
@@ -7005,81 +6952,6 @@ it("rotates PRP authority in place for a warm cross-run attachment", async () =>
   }
 }, 30_000);
 
-it("reopens Pi after a completed turn within its cold admission budget during warm attachment", async () => {
-  const root = await mkdtemp(join(tmpdir(), "runnerd-pi-warm-admission-"));
-  const providerNode = process.platform === "linux" ? join(root, "node") : process.execPath;
-  if (process.platform === "linux") {
-    await cp(process.execPath, providerNode, { dereference: true });
-    await chmod(providerNode, 0o500);
-    expect((await stat(providerNode)).mode & 0o777).toBe(0o500);
-  }
-  const cliRoot = join(root, "dist/cli");
-  await mkdir(cliRoot, { recursive: true, mode: 0o700 });
-  const sidecarPath = join(cliRoot, "acpx-runtime-sidecar.cjs");
-  const journal = join(root, "commands.ndjson");
-  const fixture = await readFile(fileURLToPath(new URL("./fixtures/fake-pi-warm-sidecar.cjs", import.meta.url)), "utf8");
-  await writeFile(sidecarPath, fixture.replace("/* fixture-config */ null", JSON.stringify({ journal, resumeDelayMs: 32_000, commandDigest: QUALIFIED_ACPX_PROFILES.pi.commandDigest })), { mode: 0o600 });
-  const digest = (path: string) => `sha256:${createHash("sha256").update(readFileSync(path)).digest("hex")}`;
-  const bundle = createCapabilityRunnerdCodexTransport({
-    provider: "acpx", acpxAgent: "pi", piThinkingLevel: "low", acpxPermissionMode: "deny-all",
-    runnerBinary: defaultCapabilityRunnerdBinary(), stateDirectory: root, runnerFilesystemRoot: root,
-    providerNodeCommand: providerNode, providerNodeCommandSha256: digest(providerNode),
-    acpxSidecarPath: sidecarPath, acpxSidecarSha256: digest(sidecarPath),
-    providerPackAuthorityDigest: `sha256:${"d".repeat(64)}`,
-    lifecyclePolicy: { mode: "warm", idleTimeoutMs: 60_000 },
-    environment: { PATH: "/usr/bin:/bin" },
-  });
-  bundle.transport.setServerRequestHandler(async () => ({
-    success: true,
-    contentItems: [{ type: "inputText", text: "Completion report accepted." }],
-  }));
-  let failure: unknown;
-  try {
-    await bundle.transport.request("initialize", {});
-    await bundle.transport.request("thread/start", {
-      cwd: root, model: "openrouter/deepseek/deepseek-v4-flash-0731",
-      dynamicTools: codexSemanticToolSpecs(),
-      completionContract: { revision: "warm-pi-contract", criterionIds: [] },
-    });
-    const runnerPid = bundle.evidence().runnerPid;
-    const notifications = bundle.transport.notifications()[Symbol.asyncIterator]();
-    const completeTurn = async () => {
-      await bundle.transport.request("turn/start", { input: [{ type: "text", text: "Complete the fixture turn." }] });
-      for (let index = 0; index < 64; index += 1) {
-        const next = await Promise.race([
-          notifications.next(),
-          new Promise<never>((_, reject) => setTimeout(() => reject(new Error("Fixture completion timed out")), 5_000)),
-        ]);
-        if (next.value?.method === "turn/completed") return;
-      }
-      throw new Error("Fixture turn did not complete");
-    };
-    await completeTurn();
-    const started = Date.now();
-    await bundle.transport.attachRun!({ runId: "run-pi-warm-second", turnId: "turn-pi-warm-second", itemId: "item-pi-warm-second" });
-    expect(Date.now() - started).toBeGreaterThan(30_000);
-    await completeTurn();
-    expect(bundle.evidence()).toMatchObject({ runnerPid, runnerExited: false });
-    const commands = (await readFile(journal, "utf8")).trim().split("\n").map((line) => JSON.parse(line));
-    expect(commands.filter((entry) => entry.event === "spawn")).toHaveLength(2);
-    expect(commands.filter((entry) => entry.command === "session.open")).toHaveLength(2);
-    expect(commands.filter((entry) => entry.resumed)).toHaveLength(1);
-    expect(commands.filter((entry) => entry.command === "turn.start")).toHaveLength(2);
-    expect(commands.filter((entry) => entry.command === "session.close")).toHaveLength(1);
-  } catch (error) {
-    failure = error;
-    throw error;
-  } finally {
-    try {
-      await bundle.transport.close();
-    } catch (cleanupError) {
-      if (failure) throw new AggregateError([failure, cleanupError], "Warm attachment and strict cleanup failed");
-      throw cleanupError;
-    }
-    await rm(root, { recursive: true, force: true });
-  }
-}, 75_000);
-
 it("waits for a warm runner to re-authenticate before probing attachment readiness", async () => {
   const stateDirectory = await mkdtemp(
     join(tmpdir(), "runnerd-warm-reattach-before-probe-"),

packages/plugins/sandbox-providers/daytona/README.md

@@ -30,6 +30,7 @@ Notes:
 - Each cold create uses a unique provider name and ownership labels. If creation fails after Daytona has allocated a sandbox, the driver looks up that exact name, checks every ownership label, and waits for deletion. Failed, missing, or timed-out lookups and failed deletions report unconfirmed cleanup with the provider name; they never return a usable lease. A missing name lookup after an uncertain create is not proof that a delayed provider request cannot create a resource.
 - For plugin-backed sandbox lease acquisition, unresolved creation cleanup crosses the worker RPC as a validated ownership envelope. The host records `pending_cleanup` before retrying deletion. If only a creation name is known, the provider first returns an ownership-verified sandbox ID without deleting it. The host saves that observation before authorizing deletion. Its existing cleanup sweep and durable spool preserve retries across controller restarts and environment deletion. A missing observed ID confirms cleanup after a lost deletion reply or database update; a name that has never been observed remains unresolved. Provider exceptions and resolved credentials are excluded from that envelope. This requires the matching host and plugin SDK update; probe and custom-image interactive-setup calls still report unconfirmed immediate cleanup without this lease-recovery path.
 - Reusable leases map to Daytona stop/start semantics. Non-reusable leases are deleted on release. A provider-resolved `target` does not change the identity of an existing sandbox. Release closes the same scoped lease that a later sentinel-verified resume reopens.
+- A command that has already exited returns its final separated log snapshot without opening a follow socket. The initial status check is bounded to 250ms and the caller's command deadline. An unfinished command, failed status read, or slow status read keeps the existing streaming and recovery path; the command is dispatched only once.
 - A session log socket can close while its command still runs. The driver requires a recorded command exit before returning a completed execution. It reconnects the log stream once, then polls status and log snapshots at most once per second. Replayed output is removed by byte offset, new output continues to reach the host, and the command is never dispatched again. A socket that delivers no output for 15 seconds switches directly to polling, even if the socket stays open or its initial connection never completes. Late callbacks from that socket are ignored. This recovers pending permission requests and cancellation results from saved output without replaying the command. Healthy initial and reconnected streams retain their existing lifetime under the caller’s RPC/run guard. After a clean close or an idle socket, each status or snapshot read gets a fresh operation timeout; successful recovery polling does not impose a new command lifetime. When both stream attempts fail without a clean close, fallback retains its original timeout budget starting after the stream attempts.
 - The SDK returns full log snapshots, so fallback bandwidth grows with retained output; it has no paged snapshot or stream cancellation API. A recovery-read timeout keeps partial output and explicitly reports whether command exit remains unconfirmed; it does not prove the remote command stopped. Provider/session teardown remains responsible for closing an outstanding socket. A final snapshot reconciles bytes missed by the stream before returning a recorded exit.
 - A sandbox record can survive the loss of its underlying container. Resume treats it as expired only when a fresh provider read confirms the exact missing-container error for that sandbox and marks it unrecoverable. Unknown errors and failed confirmation reads preserve the lease. The host still requires a verified native-runner backup before replacement.

packages/plugins/sandbox-providers/daytona/src/plugin.test.ts

@@ -2505,6 +2505,7 @@ describe("Daytona sandbox provider plugin", () => {
       process.env.DAYTONA_API_KEY = "host-key";
       const sandbox = createMockSandbox();
       sandbox.process.getSessionCommand.mockResolvedValue({ id: "cmd-1", command: "", exitCode: 7 });
+      sandbox.process.getSessionCommand.mockResolvedValueOnce({ exitCode: undefined });
       // A session command tries the log stream first: it delivers stdout and
       // stderr from the callback log form.
       sandbox.process.getSessionCommandLogs.mockImplementation(
@@ -2605,6 +2606,12 @@ describe("Daytona sandbox provider plugin", () => {
     });
 
     describe("log stream (default)", () => {
+      const createStreamingSandbox = () => {
+        const sandbox = createMockSandbox();
+        sandbox.process.getSessionCommand.mockResolvedValueOnce({ exitCode: undefined });
+        return sandbox;
+      };
+
       // A session command tries the log stream first. It streams stdout and
       // stderr from the callback log form instead of the 50-ms poll, and falls
       // back to the poll only when the stream fails.
@@ -2621,7 +2628,7 @@ describe("Daytona sandbox provider plugin", () => {
 
       it("completes a session command beyond a synchronous gateway deadline without redispatch", async () => {
         process.env.DAYTONA_API_KEY = "host-key";
-        const sandbox = createMockSandbox();
+        const sandbox = createStreamingSandbox();
         sandbox.process.executeCommand.mockRejectedValue(new MockDaytonaTimeoutError("opaque gateway timeout"));
         sandbox.process.getSessionCommand.mockResolvedValue({ exitCode: 0 });
         sandbox.process.getSessionCommandLogs.mockImplementation(
@@ -2658,7 +2665,7 @@ describe("Daytona sandbox provider plugin", () => {
 
       it.each([undefined, 0])("preserves the existing healthy session lifetime with command timeout %s", async (commandTimeoutMs) => {
         process.env.DAYTONA_API_KEY = "host-key";
-        const sandbox = createMockSandbox();
+        const sandbox = createStreamingSandbox();
         sandbox.process.getSessionCommand.mockResolvedValue({ exitCode: 0 });
         sandbox.process.getSessionCommandLogs.mockImplementation(async (_s: string, _c: string, stdout?: (chunk: string) => void, stderr?: (chunk: string) => void) => {
           if (!stdout) return { stdout: "done;", stderr: "" };
@@ -2677,7 +2684,7 @@ describe("Daytona sandbox provider plugin", () => {
 
       it("removes deadline listeners from completed reads during a quiet polling run", async () => {
         process.env.DAYTONA_API_KEY = "host-key";
-        const sandbox = createMockSandbox();
+        const sandbox = createStreamingSandbox();
         sandbox.process.getSessionCommand.mockResolvedValue({ exitCode: undefined });
         sandbox.process.getSessionCommandLogs.mockImplementation(async (_s: string, _c: string, stdout?: (chunk: string) => void) => {
           if (stdout) { stdout("partial;"); return; }
@@ -2707,7 +2714,7 @@ describe("Daytona sandbox provider plugin", () => {
         process.env.DAYTONA_API_KEY = "host-key";
         const executionLog = vi.fn();
         const restore = __setDaytonaPluginContextForTest({ execution: { log: executionLog } } as unknown as PluginContext);
-        const sandbox = createMockSandbox();
+        const sandbox = createStreamingSandbox();
         let lateOutput: ((chunk: string) => void) | undefined;
         let rejectStream: ((error: Error) => void) | undefined;
         let keepAlive: ReturnType<typeof setInterval> | undefined;
@@ -2762,7 +2769,7 @@ describe("Daytona sandbox provider plugin", () => {
         const restore = __setDaytonaPluginContextForTest(
           { execution: { log: executionLog } } as unknown as PluginContext,
         );
-        const sandbox = createMockSandbox();
+        const sandbox = createStreamingSandbox();
         let lateOutput!: (chunk: string) => void;
         let rejectStream!: (error: Error) => void;
         let snapshots = 0;
@@ -2816,7 +2823,7 @@ describe("Daytona sandbox provider plugin", () => {
 
       it.each(["status", "snapshot"])("bounds a stuck %s read after an idle socket without claiming exit", async (stalledRead) => {
         process.env.DAYTONA_API_KEY = "host-key";
-        const sandbox = createMockSandbox();
+        const sandbox = createStreamingSandbox();
         let releaseRead!: () => void;
         const stalled = new Promise<void>((resolve) => { releaseRead = resolve; });
         const executionLog = vi.fn();
@@ -2860,7 +2867,7 @@ describe("Daytona sandbox provider plugin", () => {
 
       it("streams ordered stdout and stderr from the callback log form (test_log_stream_delivers_ordered_chunks)", async () => {
         process.env.DAYTONA_API_KEY = "host-key";
-        const sandbox = createMockSandbox();
+        const sandbox = createStreamingSandbox();
         sandbox.process.getSessionCommand.mockResolvedValue({ id: "cmd-1", command: "", exitCode: 0 });
         // The callback form emits stdout and stderr chunks in order. The plugin
         // keeps each stream in its own arrival order.
@@ -2897,7 +2904,7 @@ describe("Daytona sandbox provider plugin", () => {
 
       it("reads the exit code once after the stream ends (test_log_stream_reads_exit_code_once_after_stream_end)", async () => {
         process.env.DAYTONA_API_KEY = "host-key";
-        const sandbox = createMockSandbox();
+        const sandbox = createStreamingSandbox();
         sandbox.process.getSessionCommand.mockResolvedValue({ id: "cmd-1", command: "", exitCode: 5 });
         sandbox.process.getSessionCommandLogs.mockImplementation(
           async (_sid: string, _cmdId: string, onStdout?: (chunk: string) => void) => {
@@ -2909,7 +2916,7 @@ describe("Daytona sandbox provider plugin", () => {
         const result = await plugin.definition.onEnvironmentExecute?.(streamExecParams());
 
         // The exit code read runs one time after the stream promise resolves.
-        expect(sandbox.process.getSessionCommand).toHaveBeenCalledTimes(1);
+        expect(sandbox.process.getSessionCommand).toHaveBeenCalledTimes(2);
         expect(result).toMatchObject({ exitCode: 5, timedOut: false, stdout: "done" });
       });
 
@@ -2920,7 +2927,7 @@ describe("Daytona sandbox provider plugin", () => {
           { execution: { log: executionLog } } as unknown as PluginContext,
         );
         try {
-          const sandbox = createMockSandbox();
+          const sandbox = createStreamingSandbox();
           let streams = 0;
           sandbox.process.getSessionCommandLogs.mockImplementation(
             async (_sid: string, _cmdId: string, onStdout?: (chunk: string) => void) => {
@@ -2954,7 +2961,7 @@ describe("Daytona sandbox provider plugin", () => {
           { execution: { log: executionLog } } as unknown as PluginContext,
         );
         try {
-          const sandbox = createMockSandbox();
+          const sandbox = createStreamingSandbox();
           let snapshots = 0;
           sandbox.process.getSessionCommandLogs.mockImplementation(
             async (_sid: string, _cmdId: string, onStdout?: (chunk: string) => void) => {
@@ -2993,7 +3000,7 @@ describe("Daytona sandbox provider plugin", () => {
 
       it.each(["close", "reject"])("preserves healthy stream lifetimes after an aged %s", async (ending) => {
         process.env.DAYTONA_API_KEY = "host-key";
-        const sandbox = createMockSandbox();
+        const sandbox = createStreamingSandbox();
         let streams = 0;
         sandbox.process.getSessionCommandLogs.mockImplementation(
           async (_sid: string, _cmdId: string, onStdout?: (chunk: string) => void, onStderr?: (chunk: string) => void) => {
@@ -3029,7 +3036,7 @@ describe("Daytona sandbox provider plugin", () => {
 
       it("polls an aged running command after repeated EOF without inventing an exit or lifetime cap", async () => {
         process.env.DAYTONA_API_KEY = "host-key";
-        const sandbox = createMockSandbox();
+        const sandbox = createStreamingSandbox();
         let streams = 0;
         let snapshots = 0;
         let exited = false;
@@ -3062,7 +3069,7 @@ describe("Daytona sandbox provider plugin", () => {
 
       it.each([0, 3_600_000])("retains the fallback budget and partial output after a failed stream lasting %i ms", async (streamLifetime) => {
         process.env.DAYTONA_API_KEY = "host-key";
-        const sandbox = createMockSandbox();
+        const sandbox = createStreamingSandbox();
         let streams = 0;
         sandbox.process.getSessionCommandLogs.mockImplementation(
           async (_sid: string, _cmdId: string, onStdout?: (chunk: string) => void, onStderr?: (chunk: string) => void) => {
@@ -3095,7 +3102,7 @@ describe("Daytona sandbox provider plugin", () => {
 
       it("preserves provider status errors after a clean stream close", async () => {
         process.env.DAYTONA_API_KEY = "host-key";
-        const sandbox = createMockSandbox();
+        const sandbox = createStreamingSandbox();
         sandbox.process.getSessionCommandLogs.mockResolvedValue(undefined);
         sandbox.process.getSessionCommand.mockRejectedValue(new Error("session no longer exists"));
         mockGet.mockResolvedValue(sandbox);
@@ -3107,7 +3114,7 @@ describe("Daytona sandbox provider plugin", () => {
 
       it("reads bytes written after socket close before the exit code becomes visible", async () => {
         process.env.DAYTONA_API_KEY = "host-key";
-        const sandbox = createMockSandbox();
+        const sandbox = createStreamingSandbox();
         sandbox.process.getSessionCommandLogs.mockImplementation(
           async (_sid: string, _cmdId: string, onStdout?: (chunk: string) => void) => {
             if (onStdout) onStdout("prefix;");
@@ -3133,7 +3140,7 @@ describe("Daytona sandbox provider plugin", () => {
         );
         vi.useFakeTimers();
         try {
-          const sandbox = createMockSandbox();
+          const sandbox = createStreamingSandbox();
           let streams = 0;
           let lateOutput: ((chunk: string) => void) | undefined;
           sandbox.process.getSessionCommandLogs.mockImplementation(
@@ -3185,7 +3192,7 @@ describe("Daytona sandbox provider plugin", () => {
 
       it.each(["status", "final snapshot", "fallback snapshot"])("preserves observation state when the SDK %s times out", async (stage) => {
         process.env.DAYTONA_API_KEY = "host-key";
-        const sandbox = createMockSandbox();
+        const sandbox = createStreamingSandbox();
         sandbox.process.getSessionCommandLogs.mockImplementation(
           async (_sid: string, _cmdId: string, onStdout?: (chunk: string) => void, onStderr?: (chunk: string) => void) => {
             if (!onStdout) throw new MockDaytonaTimeoutError("provider timeout");
@@ -3222,7 +3229,7 @@ describe("Daytona sandbox provider plugin", () => {
         );
         vi.useFakeTimers();
         try {
-          const sandbox = createMockSandbox();
+          const sandbox = createStreamingSandbox();
           sandbox.process.getSessionCommandLogs.mockImplementation(
             async (_sid: string, _cmdId: string, onStdout?: (chunk: string) => void, onStderr?: (chunk: string) => void) => {
               onStdout?.("A\uFFFD");
@@ -3261,7 +3268,7 @@ describe("Daytona sandbox provider plugin", () => {
           { execution: { log: executionLog } } as unknown as PluginContext,
         );
         try {
-          const sandbox = createMockSandbox();
+          const sandbox = createStreamingSandbox();
           let streams = 0;
           sandbox.process.getSessionCommandLogs.mockImplementation(
             async (_sid: string, _cmdId: string, onStdout?: (chunk: string) => void, onStderr?: (chunk: string) => void) => {
@@ -3300,7 +3307,7 @@ describe("Daytona sandbox provider plugin", () => {
 
       it("falls back to the poll path when the stream promise rejects (test_log_stream_disconnect_rejects_and_falls_back_to_poll)", async () => {
         process.env.DAYTONA_API_KEY = "host-key";
-        const sandbox = createMockSandbox();
+        const sandbox = createStreamingSandbox();
         // The callback form disconnects and rejects. The snapshot form (no
         // callbacks) serves the poll fallback read.
         sandbox.process.getSessionCommandLogs.mockImplementation(
@@ -3334,7 +3341,7 @@ describe("Daytona sandbox provider plugin", () => {
 
       it("drops the replayed prefix by byte offset on a reconnect (test_log_stream_reconnect_drops_replayed_prefix_by_byte_offset)", async () => {
         process.env.DAYTONA_API_KEY = "host-key";
-        const sandbox = createMockSandbox();
+        const sandbox = createStreamingSandbox();
         let attempt = 0;
         sandbox.process.getSessionCommandLogs.mockImplementation(
           async (
@@ -3375,7 +3382,7 @@ describe("Daytona sandbox provider plugin", () => {
           { execution: { log: executionLog } } as unknown as PluginContext,
         );
         try {
-          const sandbox = createMockSandbox();
+          const sandbox = createStreamingSandbox();
           let attempt = 0;
           sandbox.process.getSessionCommandLogs.mockImplementation(
             async (

packages/plugins/sandbox-providers/daytona/src/plugin.ts

@@ -1830,6 +1830,7 @@ async function executeOneShot(
 // Fallback reads full log snapshots as well as status. Limit the heavier
 // polling path while keeping output available to interactive commands.
 const SESSION_LOG_POLL_INTERVAL_MS = 1000;
+const SESSION_INITIAL_STATUS_TIMEOUT_MS = 250;
 // A socket can stay open after it stops delivering output. Switch to saved
 // logs after this much silence; silence never proves that the command exited.
 const SESSION_LOG_STREAM_IDLE_MS = 15_000;
@@ -2211,6 +2212,17 @@ async function executeInSession(
       timeoutSeconds,
     ));
     const commandId = dispatched.cmdId;
+    let initialExitCode: number | null = null;
+    try {
+      const status = await observeSessionCommand(
+        Math.min(effectiveTimeoutMs, SESSION_INITIAL_STATUS_TIMEOUT_MS),
+        () => sandbox.process.getSessionCommand(sessionId, commandId),
+        commandDeadline,
+      );
+      if (typeof status.exitCode === "number") initialExitCode = status.exitCode;
+    } catch (error) {
+      if (error instanceof SessionCommandDeadlineError) throw error;
+    }
 
     // Log-stream path. A session command always tries the stream first: it
     // streams stdout and stderr from the callback log form, then reads the exit
@@ -2221,7 +2233,9 @@ async function executeInSession(
     // call. The host routes it to the runner log sink by the host-issued
     // invocation id. This is a no-op when no plugin context is set (a direct
     // test call) or when the host has no active execute route.
-    const streamResult = await runSessionLogStream(
+    const streamResult = initialExitCode !== null
+      ? { exitCode: initialExitCode, pollUntilExit: false, buffer: streamBuffer }
+      : await runSessionLogStream(
       sandbox,
       sessionId,
       commandId,

packages/plugins/sandbox-providers/daytona/src/session-command-completion.test.ts

@@ -0,0 +1,107 @@
+import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
+
+const getSandbox = vi.hoisted(() => vi.fn());
+vi.mock("@daytonaio/sdk", () => ({
+  Daytona: class { get = getSandbox; },
+  DaytonaNotFoundError: class extends Error {},
+  DaytonaTimeoutError: class extends Error {},
+}));
+
+import plugin, { __resetDaytonaSandboxHandleCacheForTest } from "./plugin.js";
+
+function fixture() {
+  const process = {
+    createSession: vi.fn().mockResolvedValue(undefined),
+    executeSessionCommand: vi.fn().mockResolvedValue({ cmdId: "fixture-command" }),
+    getSessionCommand: vi.fn<(...args: string[]) => Promise<{ exitCode: number | undefined }>>().mockResolvedValue({ exitCode: 0 }),
+    getSessionCommandLogs: vi.fn<(session: string, command: string, stdout?: (chunk: string) => void, stderr?: (chunk: string) => void) => Promise<{ stdout: string; stderr: string } | void>>().mockImplementation(async (_session, _command, stdout) => {
+      if (stdout) return new Promise(() => {});
+      return { stdout: "complete stdout\n", stderr: "complete stderr\n" };
+    }),
+  };
+  getSandbox.mockResolvedValue({ id: "fixture-sandbox", state: "started", process });
+  const execute = (commandTimeoutMs = 2_000) => plugin.definition.onEnvironmentExecute!({
+    driverKey: "daytona", companyId: "fixture-company", environmentId: "fixture-environment",
+    config: { apiKey: "fixture-not-a-key", timeoutMs: 30_000 },
+    lease: { providerLeaseId: "fixture-sandbox" }, command: "printf", args: ["fixture"],
+    commandTimeoutMs,
+  });
+  return { process, execute };
+}
+
+describe("Daytona completed session commands", () => {
+  beforeEach(() => {
+    __resetDaytonaSandboxHandleCacheForTest();
+    getSandbox.mockReset();
+    vi.useFakeTimers();
+  });
+  afterEach(() => {
+    vi.useRealTimers();
+    __resetDaytonaSandboxHandleCacheForTest();
+  });
+
+  it("returns a finished command's final separated logs without opening a follow socket", async () => {
+    const { process, execute } = fixture();
+    let settled = false;
+    const pending = execute().then(result => { settled = true; return result; });
+    await vi.advanceTimersByTimeAsync(500);
+    expect(settled).toBe(true);
+    expect(await pending).toMatchObject({ exitCode: 0, timedOut: false, stdout: "complete stdout\n", stderr: "complete stderr\n" });
+    expect(process.getSessionCommandLogs).toHaveBeenCalledExactlyOnceWith(expect.any(String), "fixture-command");
+    expect(process.executeSessionCommand).toHaveBeenCalledTimes(1);
+  });
+
+  it("streams a still-running command and reconciles its final snapshot", async () => {
+    const { process, execute } = fixture();
+    process.getSessionCommand.mockResolvedValueOnce({ exitCode: undefined });
+    process.getSessionCommandLogs.mockImplementation(async (_session, _command, stdout) => {
+      if (stdout) { stdout("prefix "); return; }
+      return { stdout: "prefix complete\n", stderr: "error\n" };
+    });
+    expect(await execute()).toMatchObject({ exitCode: 0, timedOut: false, stdout: "prefix complete\n", stderr: "error\n" });
+    expect(process.getSessionCommandLogs.mock.calls.filter(call => call[2])).toHaveLength(1);
+  });
+
+  it("bounds an unavailable initial status read and retains the working stream", async () => {
+    const { process, execute } = fixture();
+    process.getSessionCommand.mockImplementationOnce(() => new Promise(() => {}));
+    process.getSessionCommandLogs.mockImplementation(async (_session, _command, stdout) => {
+      if (stdout) { stdout("complete stdout\n"); return; }
+      return { stdout: "complete stdout\n", stderr: "complete stderr\n" };
+    });
+    const pending = execute();
+    await vi.advanceTimersByTimeAsync(300);
+    expect(await pending).toMatchObject({ exitCode: 0, timedOut: false, stdout: "complete stdout\n", stderr: "complete stderr\n" });
+    expect(process.getSessionCommandLogs.mock.calls.filter(call => call[2])).toHaveLength(1);
+  });
+
+  it("retains stream recovery after the initial status request rejects", async () => {
+    const { process, execute } = fixture();
+    process.getSessionCommand.mockRejectedValueOnce(new Error("status transport unavailable"));
+    process.getSessionCommandLogs.mockImplementation(async (_session, _command, stdout) => {
+      if (stdout) { stdout("complete stdout\n"); return; }
+      return { stdout: "complete stdout\n", stderr: "complete stderr\n" };
+    });
+    expect(await execute()).toMatchObject({ exitCode: 0, timedOut: false, stdout: "complete stdout\n", stderr: "complete stderr\n" });
+    expect(process.executeSessionCommand).toHaveBeenCalledTimes(1);
+  });
+
+  it("does not treat a visible exit as proof that final logs were read", async () => {
+    const { process, execute } = fixture();
+    process.getSessionCommandLogs.mockImplementation(() => new Promise(() => {}));
+    const pending = execute(100);
+    await vi.advanceTimersByTimeAsync(101);
+    expect(await pending).toMatchObject({ exitCode: null, timedOut: true,
+      metadata: { timeoutScope: "session_log_observation", commandExitConfirmed: true, observedExitCode: 0 } });
+  });
+
+  it("keeps the total command deadline while the initial status is unavailable", async () => {
+    const { process, execute } = fixture();
+    process.getSessionCommand.mockImplementation(() => new Promise(() => {}));
+    const pending = execute(100);
+    await vi.advanceTimersByTimeAsync(101);
+    expect(await pending).toMatchObject({ exitCode: null, timedOut: true });
+    expect(process.executeSessionCommand).toHaveBeenCalledTimes(1);
+    expect(process.getSessionCommandLogs).not.toHaveBeenCalled();
+  });
+});

packages/shared/src/app-definitions.test.ts

@@ -51,6 +51,15 @@ const GOOGLE_WORKSPACE_PROFILE_EXPECTATIONS = [
     scopes: [googleScope("gmail.readonly"), googleScope("gmail.compose")],
     writeTools: ["create_draft"],
   },
+  {
+    profile: "gmail.archive",
+    appSlug: "gmail",
+    serverUrl: "https://gmailmcp.googleapis.com/mcp/v1",
+    capability: "archive",
+    riskTier: "S4",
+    scopes: [googleScope("gmail.readonly"), googleScope("gmail.modify")],
+    writeTools: ["unlabel_thread"],
+  },
   {
     profile: "drive.read",
     appSlug: "google-drive",
@@ -214,7 +223,7 @@ const GOOGLE_WORKSPACE_PROFILE_EXPECTATIONS = [
   profile: GoogleWorkspaceConnectorProfileId;
   appSlug: string;
   serverUrl: string;
-  capability: "read" | "write" | "draft";
+  capability: "read" | "write" | "draft" | "archive";
   riskTier: "S3" | "S4";
   scopes: readonly string[];
   writeTools: readonly string[];
@@ -763,10 +772,10 @@ describe("AppDefinition catalog", () => {
       APP_DEFINITIONS.find((app) => app.slug === "linear")?.methods[0]?.defaults
         ?.scopesHint,
     ).toEqual(["read", "write"]));
-  it("defaults every new connection action to allowed except Enterpret writes", () => {
+  it("limits Ask-first defaults to reviewed high-risk connection methods", () => {
     for (const app of APP_DEFINITIONS)
       for (const method of app.methods) {
-        if (app.slug === "enterpret") {
+        if (app.slug === "enterpret" || (app.slug === "gmail" && method.capabilityProfile?.key === "archive")) {
           expect(recommendedDefaultsForApp(app, method.key)).toEqual({
             access: "all_agents",
             askFirstRiskLevels: ["write", "destructive"],
@@ -788,6 +797,8 @@ describe("AppDefinition catalog", () => {
     expect(getAvailableConnectionMethod(gmail)?.key).toBe(
       "customer-draft-oauth",
     );
+    expect(gmail.methods.filter((method) => method.capabilityProfile?.key === "archive")
+      .map((method) => method.key)).toEqual(["paperclip-archive", "customer-archive-oauth"]);
     expect(
       getRecommendedConnectionMethod(
         drive.methods.filter((candidate) =>

packages/shared/src/app-definitions.ts

@@ -262,7 +262,15 @@ export function recommendedDefaultsForApp(app: AppDefinition, methodKey?: string
   // concrete app/method. Connection finalization remains a configure-authorized,
   // audited operation, and Ask first stays available as an operator-selected
   // policy for any action after the connection is made.
-  void methodKey;
+  const method = methodKey
+    ? app.methods.find((candidate) => candidate.key === methodKey)
+    : null;
+  if (app.slug === "gmail" && method?.capabilityProfile?.key === "archive") {
+    return {
+      access: "all_agents",
+      askFirstRiskLevels: ["write", "destructive"],
+    };
+  }
   // Enterpret's run_graph_query self-reports readOnlyHint but executes Cypher
   // and has not been established as read-only. Classify it as write (server
   // classifyRisk) and require Ask first so new token connections do not ship

packages/shared/src/app-definitions/gmail.json

@@ -177,6 +177,83 @@
         "Google Workspace MCP servers are in Developer Preview."
       ],
       "riskTier": "S4"
+    },
+    {
+      "key": "paperclip-archive",
+      "label": "Connect with Paperclip",
+      "transport": "mcp_remote",
+      "auth": "oauth",
+      "oauthStrategy": "paperclip_cloud_connector",
+      "connectorProfile": "gmail.archive",
+      "capabilityProfile": {
+        "key": "archive",
+        "label": "Read & archive",
+        "description": "Read Gmail and archive threads by removing only the Inbox label."
+      },
+      "grantKinds": [
+        "user",
+        "organization"
+      ],
+      "ownershipModes": [
+        "platform_shared"
+      ],
+      "whenToUse": "Use Paperclip-managed Google OAuth to read Gmail and archive threads.",
+      "defaults": {
+        "serverUrl": "https://gmailmcp.googleapis.com/mcp/v1",
+        "scopesHint": [
+          "https://www.googleapis.com/auth/gmail.readonly",
+          "https://www.googleapis.com/auth/gmail.modify"
+        ]
+      },
+      "guidanceMd": "Connect Gmail to search and read mail and archive threads. Archive removes only the Inbox label; sending, trashing, and other label changes remain disabled.",
+      "warnings": [
+        "Archiving requires approval and only removes the Inbox label.",
+        "Google Workspace MCP servers are in Developer Preview."
+      ],
+      "riskTier": "S4"
+    },
+    {
+      "key": "customer-archive-oauth",
+      "label": "Use your own Google OAuth app",
+      "transport": "mcp_remote",
+      "auth": "oauth",
+      "capabilityProfile": {
+        "key": "archive",
+        "label": "Read & archive",
+        "description": "Read Gmail and archive threads by removing only the Inbox label."
+      },
+      "grantKinds": [
+        "user",
+        "organization"
+      ],
+      "ownershipModes": [
+        "customer"
+      ],
+      "whenToUse": "Use a customer-owned Google OAuth client to read Gmail and archive threads.",
+      "defaults": {
+        "serverUrl": "https://gmailmcp.googleapis.com/mcp/v1",
+        "authorizationEndpoint": "https://accounts.google.com/o/oauth2/v2/auth",
+        "tokenEndpoint": "https://oauth2.googleapis.com/token",
+        "metadataUrl": "https://accounts.google.com/.well-known/openid-configuration",
+        "scopesHint": [
+          "https://www.googleapis.com/auth/gmail.readonly",
+          "https://www.googleapis.com/auth/gmail.modify"
+        ],
+        "oauthAuthorizationParams": {
+          "access_type": "offline",
+          "prompt": "consent"
+        }
+      },
+      "guidanceMd": "Enable the Gmail and Gmail MCP APIs in your Google Cloud project, then register Paperclip's callback URI. Archive removes only the Inbox label.",
+      "consoleLinks": {
+        "register": "https://console.cloud.google.com/auth/clients",
+        "docs": "https://developers.google.com/workspace/guides/configure-mcp-servers"
+      },
+      "warnings": [
+        "Archiving requires approval and only removes the Inbox label.",
+        "Google Workspace MCP servers are in Developer Preview."
+      ],
+      "riskTier": "S4"
     }
   ]
 }

packages/shared/src/google-workspace-connectors.ts

@@ -1,5 +1,5 @@
 export const GOOGLE_WORKSPACE_CONNECTOR_PROFILE_IDS = [
-  "gmail.read", "gmail.draft", "drive.read", "drive.write", "docs.read", "docs.write",
+  "gmail.read", "gmail.draft", "gmail.archive", "drive.read", "drive.write", "docs.read", "docs.write",
   "sheets.read", "sheets.write", "slides.read", "slides.write", "calendar.read",
   "calendar.write", "chat.read", "chat.write", "people.read", "workspace-search.read",
 ] as const;
@@ -16,6 +16,7 @@ export const GOOGLE_WORKSPACE_CONNECTOR_PROFILES: Readonly<Record<GoogleWorkspac
 }>> = {
   "gmail.read": def("gmail", "https://gmailmcp.googleapis.com/mcp/v1", [auth("gmail.readonly")]),
   "gmail.draft": def("gmail", "https://gmailmcp.googleapis.com/mcp/v1", [auth("gmail.readonly"), auth("gmail.compose")], ["create_draft"]),
+  "gmail.archive": def("gmail", "https://gmailmcp.googleapis.com/mcp/v1", [auth("gmail.readonly"), auth("gmail.modify")], ["unlabel_thread"]),
   "drive.read": def("google-drive", "https://drivemcp.googleapis.com/mcp/v1", [auth("drive.readonly")]),
   "drive.write": def("google-drive", "https://drivemcp.googleapis.com/mcp/v1", [auth("drive.readonly"), auth("drive.file")], ["copy_file", "create_file"]),
   "docs.read": def("google-docs", "https://docsmcp.googleapis.com/mcp/v1", [auth("documents.readonly")]),

pnpm-lock.yaml

@@ -166,6 +166,9 @@ importers:
       '@types/node':
         specifier: ^24.0.0
         version: 24.13.3
+      tsx:
+        specifier: ^4.23.15
+        version: 4.23.15
       typescript:
         specifier: ^7.0.2
         version: 7.0.2
@@ -1099,6 +1102,12 @@ importers:
         specifier: 1.0.88
         version: 1.0.88
     devDependencies:
+      '@opentelemetry/context-async-hooks':
+        specifier: 2.10.0
+        version: 2.10.0(@opentelemetry/api@1.9.1)
+      '@opentelemetry/sdk-trace':
+        specifier: 2.10.0
+        version: 2.10.0(@opentelemetry/api@1.9.1)
       '@paperclipai/paperclip-runner':
         specifier: workspace:*
         version: link:../packages/paperclip-runner
@@ -1213,6 +1222,9 @@ importers:
       '@radix-ui/react-slot':
         specifier: ^1.3.3
         version: 1.3.3(@types/react@19.2.18)(react@19.2.8)
+      '@spekoai/client':
+        specifier: 0.0.13
+        version: 0.0.13
       '@tailwindcss/typography':
         specifier: ^0.5.20
         version: 0.5.20(tailwindcss@4.3.3)
@@ -3164,6 +3176,12 @@ packages:
   '@lezer/yaml@1.0.4':
     resolution: {integrity: sha512-2lrrHqxalACEbxIbsjhqGpSW8kWpUKuY6RHgnSAFZa6qK62wvnPxA8hGOwOoDbwHcOFs5M4o27mjGu+P7TvBmw==}
 
+  '@livekit/mutex@1.1.1':
+    resolution: {integrity: sha512-EsshAucklmpuUAfkABPxJNhzj9v2sG7JuzFDL4ML1oJQSV14sqrpTYnsaOudMAw9yOaW53NU3QQTlUQoRs4czw==}
+
+  '@livekit/protocol@1.52.1':
+    resolution: {integrity: sha512-X8DxKlNbzk6Lh+v630+mjdFHSotdAFwIyzxSRSUql2DbTxCnlpa7JsvFGc6Na3mXiex2OFh0iiQtWwOBP0rIYA==}
+
   '@marijn/find-cluster-break@1.0.4':
     resolution: {integrity: sha512-Wy0V7+SGUjnF9/TkiM1hKVDPj7jKXduPNboMVtHTA8dySMURWqfg/JZ9E2Sq8JgSJmkl7k7Qe9FLeMSrSraWmQ==}
 
@@ -3384,6 +3402,30 @@ packages:
     resolution: {integrity: sha512-gLyJlPHPZYdAk1JENA9LeHejZe1Ti77/pTeFm/nMXmQH/HFZlcS/O2XJB+L8fkbrNSqhdtlvjBVjxwUYanNH5Q==}
     engines: {node: '>=8.0.0'}
 
+  '@opentelemetry/context-async-hooks@2.10.0':
+    resolution: {integrity: sha512-bvyMcgLEkozzSzpEEEo1OMoeQ97bxj6Qs2uN3mPrSdDvObMI1myffD/BPqcLlzZO9//d1SqQA/WPw7Cz2AiqhA==}
+    engines: {node: ^18.19.0 || >=20.6.0}
+    peerDependencies:
+      '@opentelemetry/api': '>=1.0.0 <1.10.0'
+
+  '@opentelemetry/core@2.10.0':
+    resolution: {integrity: sha512-/wNZ8twnEQQA4HoHu22+vcsdru6pWPWxW+7w+FlxT6Id7PE/WIbZmVKkte+PF72e0F2dnImFeHD2syyE1Mw6MQ==}
+    engines: {node: ^18.19.0 || >=20.6.0}
+    peerDependencies:
+      '@opentelemetry/api': '>=1.0.0 <1.10.0'
+
+  '@opentelemetry/resources@2.10.0':
+    resolution: {integrity: sha512-q6MMm2zhggzsHVNbabYwut+a6nbuQQe3URUoxaojM/8K1IBfwwPzvxIjNi2/lI1TFe+fMHMW9MWhrtDLEXEnkA==}
+    engines: {node: ^18.19.0 || >=20.6.0}
+    peerDependencies:
+      '@opentelemetry/api': '>=1.3.0 <1.10.0'
+
+  '@opentelemetry/sdk-trace@2.10.0':
+    resolution: {integrity: sha512-MfQGq3GRmTh5fM/y+OjaO0vj6+luCB1XO2gfXCalKCfgKw0eHL++sm75DNweC6ohlp+aFvACqeE0fYayqdRaoQ==}
+    engines: {node: ^18.19.0 || >=20.6.0}
+    peerDependencies:
+      '@opentelemetry/api': '>=1.3.0 <1.10.0'
+
   '@opentelemetry/semantic-conventions@1.43.0':
     resolution: {integrity: sha512-eSYWTm620tTk45EKSedaUL8MFYI8hW164hIXsgIHyxu3VobUB3fFCu5t0hQby6OoWRPsG1KkKUG2M5UadiLiVg==}
     engines: {node: '>=14'}
@@ -4947,6 +4989,9 @@ packages:
     resolution: {integrity: sha512-7wNWV7SugHpcMA7uzEawJNpE0GrasXM7a9E+1+Wm6NxVuDClESac/AKt+G7jMZNUz5vBLWqKlqVV7Sv7AtUr6Q==}
     engines: {node: '>=18.0.0'}
 
+  '@spekoai/client@0.0.13':
+    resolution: {integrity: sha512-Jodmw5jE9UPY0k2UrVJETFMahtSbwpImHsJ5eWrk7nWjPi0BgOOzlUVaNQsGR9153SIyeY/pNUu8RmDx4ryKaw==}
+
   '@stablelib/base64@1.0.1':
     resolution: {integrity: sha512-1bnPQqSxSuc3Ii6MhBysoWCg58j97aUjuCSZrGSmDxNqtytIi0k8utUenAwTZN4V5mXXYGsVUI9zeBqy+jBOSQ==}
 
@@ -6732,6 +6777,10 @@ packages:
   events-universal@1.0.1:
     resolution: {integrity: sha512-LUd5euvbMLpwOF8m6ivPCbhQeSiYVNb8Vs0fQ8QjXo0JTkEHpz8pxdQf0gStltaPpw0Cca8b39KxvK9cfKRiAw==}
 
+  events@3.3.0:
+    resolution: {integrity: sha512-mQw+2fkQbALzQ7V0MY0IqdnXNOeTtP4r0lN9z7AAawCXgqea7bDii20AYrIBrFd/Hx0M2Ocz6S111CaFkUcb0Q==}
+    engines: {node: '>=0.8.x'}
+
   eventsource-parser@3.1.1:
     resolution: {integrity: sha512-EKN1vKAMcZ8MlYMpaNuxN6R9yakzH6uajHcHVTqWJzvu5pWw9DyhbP35HH8MVBQ+dZjAfDxk+A8NiR9KWaXiyQ==}
     engines: {node: '>=18.0.0'}
@@ -7384,6 +7433,11 @@ packages:
   limiter@1.1.5:
     resolution: {integrity: sha512-FWWMIEOxz3GwUI4Ts/IvgVy6LPvoMPgjMdQ185nN6psJyBJ4yOpzqm695/h5umdLJg2vW3GR5iG11MAkR2AzJA==}
 
+  livekit-client@2.22.4:
+    resolution: {integrity: sha512-UJ5K8gqP8uIO+tjhGeF+s6SzgLfxZKPxQjSqzzkF298QNYMnhYmXGQRMMh3GnsfaeswnsHHMSogMeb7SocH2zw==}
+    peerDependencies:
+      '@types/dom-mediacapture-record': ^1
+
   lodash-es@4.18.1:
     resolution: {integrity: sha512-J8xewKD/Gk22OZbhpOVSwcs60zhd95ESDwezOFuA3/099925PdHJ7OFHNTGtajL3AlZkykD32HykiMo+BIBI8A==}
 
@@ -7420,6 +7474,10 @@ packages:
   lodash@4.18.1:
     resolution: {integrity: sha512-dMInicTPVE8d1e5otfwmmjlxkZoUpiVLwyeTdUsi/Caj/gfzzblBcCE5sRHV/AsjuCmxWrte2TNGSYuCeCq+0Q==}
 
+  loglevel@1.9.2:
+    resolution: {integrity: sha512-HgMmCqIJSAKqo68l0rS2AanEWfkxaZ5wNiEFb5ggm08lDs9Xl2KxBlX3PTcaD2chBM1gXAYf491/M2Rv8Jwayg==}
+    engines: {node: '>= 0.6.0'}
+
   long@5.3.2:
     resolution: {integrity: sha512-mNAgZ1GmyNhD7AuqnTG3/VQ26o760+ZYBPKjPvugO8+nLbYfX6TVpJPseBvopbdY+qpZ/lKUnmEc1LeZYS3QAA==}
 
@@ -7459,6 +7517,10 @@ packages:
     resolution: {integrity: sha512-h5bgJWpxJNswbU7qCrV0tIKQCaS3blPDrqKWx+QxzuzL1zGUzij9XCWLrSLsJPu5t+eWA/ycetzYAO5IOMcWAQ==}
     hasBin: true
 
+  machina@7.0.1:
+    resolution: {integrity: sha512-0Xkgs8NeBvO2hW6bjJT8ewLWrMfh1haMPstaghpASWjXkLjRhtDiaJG1Kb7mslaOHxZYxpH7PIjpmirRVYbMMA==}
+    engines: {node: '>=22.22'}
+
   magic-bytes.js@1.13.1:
     resolution: {integrity: sha512-x5sn4UX2k5gCWlcfmoFwG4TPie8+dctESyqOBdhB5p6MsgWXdBKGmt9nXPObj/JI50TTL928lc5Yt1WntMn1bw==}
 
@@ -8389,6 +8451,9 @@ packages:
   rw@1.3.3:
     resolution: {integrity: sha512-PdhdWy89SiZogBLaw42zdeqtRJ//zFd2PgQavcICDUgJT5oW10QCRKbJ6bg4r0/UY2M6BWd5tkxuGFRvCkgfHQ==}
 
+  rxjs@7.8.2:
+    resolution: {integrity: sha512-dhKf903U/PQZY6boNNtAGdWbG85WAbjT/1xYoZIC7FAY0yWapOBQVsVrDl58W86//e1VpMNBtRV4MaXfdMySFA==}
+
   sade@1.8.1:
     resolution: {integrity: sha512-xal3CZX1Xlo/k4ApwCFrHVACi9fBqJ7V+mwhBsuf/1IOKbBy098Fex+Wa/5QMubw09pSZ/u8EY8PWgevJsXp1A==}
     engines: {node: '>=6'}
@@ -8413,6 +8478,13 @@ packages:
   scheduler@0.27.0:
     resolution: {integrity: sha512-eNv+WrVbKu1f3vbYJT/xtiF5syA5HPIMtf9IgY/nKg0sWqzAUEvqY/xm7OcZc/qafLx/iO9FgOmeSAp4v5ti/Q==}
 
+  sdp-transform@2.15.0:
+    resolution: {integrity: sha512-KrOH82c/W+GYQ0LHqtr3caRpM3ITglq3ljGUIb8LTki7ByacJZ9z+piSGiwZDsRyhQbYBOBJgr2k6X4BZXi3Kw==}
+    hasBin: true
+
+  sdp@3.2.2:
+    resolution: {integrity: sha512-xZocWwfyp4hkbN4hLWxMjmv2Q8aNa9MhmOZ7L9aCZPT+dZsgRr6wZRrSYE3HTdyk/2pZKPSgqI7ns7Een1xMSA==}
+
   secure-json-parse@4.1.0:
     resolution: {integrity: sha512-l4KnYfEyqYJxDwlNVyRfO2E4NTHfMKAWdUuA8J0yve2Dz/E/PdBepY03RvyJpssIpRFwJoCD55wA+mEDs6ByWA==}
 
@@ -8742,6 +8814,9 @@ packages:
     resolution: {integrity: sha512-OZs6gsjF4vMp32qrCbiVSkrFmXtG/AZhY3t0iAMrMBiAZyV9oALtXO8hsrHbMXF9x6L3grlFuwW2oAz7cav+Gw==}
     engines: {node: '>= 0.6'}
 
+  typed-emitter@2.1.0:
+    resolution: {integrity: sha512-g/KzbYKbH5C2vPkaXGu8DJlHrGKHLsM25Zg9WuC9pMGfuvT+X25tZQWo5fK1BjBm8+UrVE9LDCvaY0CQk+fXDA==}
+
   typescript@7.0.2:
     resolution: {integrity: sha512-8FYau96o3NKOhbjKi/qNvG/W5jhzxkbdm5sj9AbZ/5T5sWqn3hJgLfGx27sRKZWTvyzCP8dLRBTf5tBTSRVUNA==}
     engines: {node: '>=16.20.0'}
@@ -9052,6 +9127,10 @@ packages:
   webpack-virtual-modules@0.6.2:
     resolution: {integrity: sha512-66/V2i5hQanC51vBQKPH4aI8NMAcBW59FVBs+rC7eGHupMyfn34q7rZIE+ETlJ+XTevqfUhVVBgSUNSW2flEUQ==}
 
+  webrtc-adapter@9.0.6:
+    resolution: {integrity: sha512-CHbl2ZQbxx164IgWRgzJno4hWtM4tFbRam1QfI3Yxhs3w/DvqluVxVWeXs3oL5/fbGkSNLKo0Ty5MgUWceNhog==}
+    engines: {node: '>=6.0.0', npm: '>=3.10.0'}
+
   whatwg-mimetype@5.0.0:
     resolution: {integrity: sha512-sXcNcHOC51uPGF0P/D4NVtrkjSU2fNsm9iog4ZvZJsL3rjoDAzXZhkm2MWt1y+PUdggKAYVoMAIYcs78wJ51Cw==}
     engines: {node: '>=20'}
@@ -11198,6 +11277,12 @@ snapshots:
       '@lezer/highlight': 1.2.3
       '@lezer/lr': 1.4.10
 
+  '@livekit/mutex@1.1.1': {}
+
+  '@livekit/protocol@1.52.1':
+    dependencies:
+      '@bufbuild/protobuf': 1.10.0
+
   '@marijn/find-cluster-break@1.0.4': {}
 
   '@mdx-js/react@3.1.1(@types/react@19.2.18)(react@19.2.8)':
@@ -11543,6 +11628,28 @@ snapshots:
 
   '@opentelemetry/api@1.9.1': {}
 
+  '@opentelemetry/context-async-hooks@2.10.0(@opentelemetry/api@1.9.1)':
+    dependencies:
+      '@opentelemetry/api': 1.9.1
+
+  '@opentelemetry/core@2.10.0(@opentelemetry/api@1.9.1)':
+    dependencies:
+      '@opentelemetry/api': 1.9.1
+      '@opentelemetry/semantic-conventions': 1.43.0
+
+  '@opentelemetry/resources@2.10.0(@opentelemetry/api@1.9.1)':
+    dependencies:
+      '@opentelemetry/api': 1.9.1
+      '@opentelemetry/core': 2.10.0(@opentelemetry/api@1.9.1)
+      '@opentelemetry/semantic-conventions': 1.43.0
+
+  '@opentelemetry/sdk-trace@2.10.0(@opentelemetry/api@1.9.1)':
+    dependencies:
+      '@opentelemetry/api': 1.9.1
+      '@opentelemetry/core': 2.10.0(@opentelemetry/api@1.9.1)
+      '@opentelemetry/resources': 2.10.0(@opentelemetry/api@1.9.1)
+      '@opentelemetry/semantic-conventions': 1.43.0
+
   '@opentelemetry/semantic-conventions@1.43.0': {}
 
   '@oxc-parser/binding-android-arm-eabi@0.127.0':
@@ -12962,6 +13069,13 @@ snapshots:
       '@smithy/core': 3.35.1
       tslib: 2.8.1
 
+  '@spekoai/client@0.0.13':
+    dependencies:
+      livekit-client: 2.22.4
+      tslib: 2.8.1
+    transitivePeerDependencies:
+      - '@types/dom-mediacapture-record'
+
   '@stablelib/base64@1.0.1': {}
 
   '@standard-schema/spec@1.1.0': {}
@@ -14658,6 +14772,8 @@ snapshots:
     transitivePeerDependencies:
       - bare-abort-controller
 
+  events@3.3.0: {}
+
   eventsource-parser@3.1.1: {}
 
   eventsource@3.0.7:
@@ -15324,6 +15440,19 @@ snapshots:
 
   limiter@1.1.5: {}
 
+  livekit-client@2.22.4:
+    dependencies:
+      '@livekit/mutex': 1.1.1
+      '@livekit/protocol': 1.52.1
+      events: 3.3.0
+      jose: 6.2.12
+      loglevel: 1.9.2
+      machina: 7.0.1
+      sdp-transform: 2.15.0
+      tslib: 2.8.1
+      typed-emitter: 2.1.0
+      webrtc-adapter: 9.0.6
+
   lodash-es@4.18.1: {}
 
   lodash.camelcase@4.3.0: {}
@@ -15348,6 +15477,8 @@ snapshots:
 
   lodash@4.18.1: {}
 
+  loglevel@1.9.2: {}
+
   long@5.3.2: {}
 
   longest-streak@3.1.0: {}
@@ -15381,6 +15512,8 @@ snapshots:
 
   lz-string@1.5.0: {}
 
+  machina@7.0.1: {}
+
   magic-bytes.js@1.13.1: {}
 
   magic-string@0.30.21:
@@ -16766,6 +16899,11 @@ snapshots:
 
   rw@1.3.3: {}
 
+  rxjs@7.8.2:
+    dependencies:
+      tslib: 2.8.1
+    optional: true
+
   sade@1.8.1:
     dependencies:
       mri: 1.2.0
@@ -16784,6 +16922,10 @@ snapshots:
 
   scheduler@0.27.0: {}
 
+  sdp-transform@2.15.0: {}
+
+  sdp@3.2.2: {}
+
   secure-json-parse@4.1.0: {}
 
   semver@6.3.1: {}
@@ -17191,6 +17333,10 @@ snapshots:
       media-typer: 1.1.0
       mime-types: 3.0.2
 
+  typed-emitter@2.1.0:
+    optionalDependencies:
+      rxjs: 7.8.2
+
   typescript@7.0.2:
     optionalDependencies:
       '@typescript/typescript-aix-ppc64': 7.0.2
@@ -17453,6 +17599,10 @@ snapshots:
 
   webpack-virtual-modules@0.6.2: {}
 
+  webrtc-adapter@9.0.6:
+    dependencies:
+      sdp: 3.2.2
+
   whatwg-mimetype@5.0.0: {}
 
   whatwg-url@16.0.1(@noble/hashes@2.4.0):

scripts/__tests__/run-vitest-stable-shard.test.mjs

@@ -1,6 +1,7 @@
 import assert from "node:assert/strict";
 import { spawnSync } from "node:child_process";
-import { readFileSync } from "node:fs";
+import { chmodSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
+import os from "node:os";
 import path from "node:path";
 import { fileURLToPath } from "node:url";
 import test from "node:test";
@@ -519,3 +520,69 @@ test("filtered collection must match the exact assigned case identities, includi
     assert.throws(() => assertSelectedTests(expected, actual, lineShardFile));
   }
 });
+
+
+test("parallel chat jobs form a complete disjoint partition", () => {
+  for (const count of [2, 3, 4]) {
+    const shards = Array.from({ length: count }, (_, index) =>
+      dryRunJson(["--mode", "general", "--group", "general-chat", "--shard-index", String(index), "--shard-count", String(count)],
+        { PAPERCLIP_TEST_DATABASE_URL: "" }).chatVitestShards,
+    );
+    assert.ok(shards.every((workers) => workers.length === 2));
+    assert.deepEqual(shards.flat().map(({ index }) => index).sort((a, b) => a - b),
+      Array.from({ length: count * 2 }, (_, index) => index));
+    assert.ok(shards.flat().every((shard) => shard.count === count * 2));
+  }
+});
+
+test("shared-database and unsharded chat runs retain one serial partition", () => {
+  for (let index = 0; index < 3; index++) {
+    const result = dryRunJson(["--mode", "general", "--group", "general-chat", "--shard-index", String(index), "--shard-count", "3"],
+      { PAPERCLIP_TEST_DATABASE_URL: "postgresql://shared/test" });
+    assert.deepEqual(result.chatVitestShards, [{ index, count: 3 }]);
+  }
+  assert.deepEqual(dryRunJson(["--mode", "general", "--group", "general-chat"],
+    { PAPERCLIP_TEST_DATABASE_URL: "" }).chatVitestShards, [{ index: 0, count: 1 }]);
+});
+
+
+test("a failed chat child fails the job after its sibling finishes", { skip: process.platform === "win32" }, () => {
+  const directory = mkdtempSync(path.join(os.tmpdir(), "chat-launch-proof-"));
+  const file = path.join(repoRoot, "server/src/__tests__/chat-channels.integration.test.ts");
+  const lines = readFileSync(file, "utf8").split("\n")
+    .flatMap((line, index) => /^\s*it\(/.test(line) ? [index + 1] : []).slice(0, 6);
+  assert.equal(lines.length, 6);
+  const tests = lines.map((line) => ({ file, projectName: "@paperclipai/server",
+    name: `launcher fixture ${line}`, location: { line } }));
+  const receipt = path.join(directory, "sibling-finished");
+  const fakePnpm = path.join(directory, "pnpm");
+  try {
+    writeFileSync(fakePnpm, `#!${process.execPath}
+      const fs = require('node:fs');
+      const args = process.argv.slice(2);
+      const tests = ${JSON.stringify(tests)};
+      const selected = args.flatMap(arg => {
+        const match = arg.match(/chat-channels\\.integration\\.test\\.ts:(\\d+)$/);
+        return match ? [Number(match[1])] : [];
+      });
+      if (args.includes('list')) {
+        const output = args.find(arg => arg.startsWith('--json=')).slice(7);
+        fs.writeFileSync(output, JSON.stringify(selected.length
+          ? tests.filter(test => selected.includes(test.location.line)) : tests));
+      } else if (selected.includes(${lines[0]})) process.exit(7);
+      else setTimeout(() => fs.writeFileSync(${JSON.stringify(receipt)}, 'finished'), 200);
+    `);
+    chmodSync(fakePnpm, 0o755);
+    const result = spawnSync(process.execPath, [script, "--mode", "general", "--group", "general-chat",
+      "--shard-index", "0", "--shard-count", "3"], {
+      cwd: repoRoot, encoding: "utf8", timeout: 30_000,
+      env: { ...process.env, PATH: `${directory}${path.delimiter}${process.env.PATH}`,
+        PAPERCLIP_TEST_DATABASE_URL: "", PAPERCLIP_CI_PROFILE: "false" },
+    });
+    assert.equal(result.status, 1, result.stdout + result.stderr);
+    assert.match(result.stderr, /failed \(7\)/);
+    assert.equal(readFileSync(receipt, "utf8"), "finished", "a failed worker must not abandon its sibling");
+  } finally {
+    rmSync(directory, { recursive: true, force: true });
+  }
+});

scripts/__tests__/vitest-worker-isolation.test.mjs

@@ -0,0 +1,77 @@
+import assert from "node:assert/strict";
+import { spawnSync } from "node:child_process";
+import { mkdtempSync, readFileSync, realpathSync, rmSync, writeFileSync } from "node:fs";
+import os from "node:os";
+import path from "node:path";
+import { fileURLToPath } from "node:url";
+import test from "node:test";
+
+const root = fileURLToPath(new URL('../../', import.meta.url));
+const vitest = path.join(root, 'node_modules/vitest/vitest.mjs');
+const api = fileURLToPath(new URL('../../node_modules/vitest/dist/index.js', import.meta.url));
+const setup = path.join(root, 'server/src/__tests__/setup-supertest.ts');
+
+for (const parallel of [false, true]) test(`isolated forks retain fresh state with file parallelism ${parallel}`, () => {
+  const directory = realpathSync(mkdtempSync(path.join(os.tmpdir(), 'pc-fork-proof-')));
+  try {
+    const record = path.join(directory, 'workers.ndjson');
+    const config = path.join(directory, 'vitest.config.mjs');
+    const files = ['first.test.ts', 'second.test.ts'];
+    for (const file of files) writeFileSync(path.join(directory, file), `
+      import { expect, it, vi } from ${JSON.stringify(api)};
+      import { appendFileSync, existsSync, writeFileSync } from 'node:fs';
+      it('starts clean and can contaminate only its own worker', async () => {
+        expect(globalThis.serializedLeak).toBeUndefined();
+        expect(process.env.PAPERCLIP_SERIALIZED_LEAK).toBeUndefined();
+        expect(vi.isFakeTimers()).toBe(false);
+        expect(existsSync(process.env.PAPERCLIP_CONFIG)).toBe(false);
+        appendFileSync(${JSON.stringify(record)}, JSON.stringify({
+          name: ${JSON.stringify(file)}, pid: process.pid, home: process.env.PAPERCLIP_HOME,
+          config: process.env.PAPERCLIP_CONFIG, instance: process.env.PAPERCLIP_INSTANCE_ID,
+        }) + '\\n');
+        writeFileSync(process.env.PAPERCLIP_CONFIG, '{}');
+        // In parallel mode both real workers must arrive before either exits.
+        // This fails if the controller silently falls back to serial execution.
+        if (${parallel}) {
+          const { readFileSync } = await import('node:fs');
+          const deadline = Date.now() + 10_000;
+          while (readFileSync(${JSON.stringify(record)}, 'utf8').trim().split('\\n').length < 2) {
+            if (Date.now() > deadline) throw new Error('second isolated worker did not start');
+            await new Promise(resolve => setTimeout(resolve, 10));
+          }
+        }
+        globalThis.serializedLeak = true;
+        process.env.PAPERCLIP_SERIALIZED_LEAK = 'leaked';
+        vi.useFakeTimers();
+      });
+    `);
+    for (const order of [files, [...files].reverse()]) {
+      rmSync(record, { force: true });
+      writeFileSync(config, `import Sequencer from './sequencer.mjs'; export default ${JSON.stringify({
+        root: directory,
+        test: {
+          include: order, pool: 'forks', isolate: true, fileParallelism: parallel,
+          maxWorkers: parallel ? 2 : 1, setupFiles: [setup],
+          sequence: { sequencer: '__SEQUENCER__' },
+        },
+      }).replace('"__SEQUENCER__"', 'Sequencer')};`);
+      writeFileSync(path.join(directory, 'sequencer.mjs'), `export default class {
+        shard(tests) { return tests; }
+        sort(tests) { return tests.sort((a, b) => ${JSON.stringify(order)}.indexOf(a.moduleId.split('/').pop()) - ${JSON.stringify(order)}.indexOf(b.moduleId.split('/').pop())); }
+      }`);
+      const result = spawnSync(process.execPath, [vitest, 'run', '--config', config], {
+        cwd: root, encoding: 'utf8', timeout: 60_000,
+        env: { ...process.env, PAPERCLIP_HOME: path.join(directory, 'controller-home'),
+          PAPERCLIP_CONFIG: path.join(directory, 'controller-config.json'), PAPERCLIP_INSTANCE_ID: 'controller' },
+      });
+      assert.equal(result.status, 0, `${result.error ?? ''}\n${result.stdout}\n${result.stderr}`);
+      const workers = readFileSync(record, 'utf8').trim().split('\n').map((line) => JSON.parse(line));
+      assert.equal(workers.length, 2);
+      assert.deepEqual(workers.map((worker) => worker.name).sort(), [...order].sort());
+      if (!parallel) assert.deepEqual(workers.map((worker) => worker.name), order);
+      for (const key of ['pid', 'home', 'config', 'instance']) assert.equal(new Set(workers.map((worker) => worker[key])).size, 2, key);
+    }
+  } finally {
+    rmSync(directory, { recursive: true, force: true });
+  }
+});

scripts/benchmark-ci-work.mjs

@@ -0,0 +1,76 @@
+import assert from "node:assert/strict";
+import { execFileSync, spawnSync } from "node:child_process";
+import { mkdirSync, readdirSync, readFileSync, writeFileSync } from "node:fs";
+import path from "node:path";
+import { fileURLToPath } from "node:url";
+
+const candidate = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..");
+const baseline = path.resolve(process.argv[2] ?? "");
+assert.ok(process.argv[2] && baseline !== candidate, 'Pass a separately installed unchanged baseline checkout');
+const output = path.resolve(process.argv[3] ?? path.join(candidate, '.ci-benchmark'));
+mkdirSync(output, { recursive: true });
+const results = [];
+const commands = [
+  ['general-server', process.execPath, ['scripts/run-vitest-stable.mjs', '--mode', 'general', '--group', 'general-server-without-chat-or-native-runner', '--shard-index', '0', '--shard-count', '12']],
+  ['fixtures', 'pnpm', ['exec', 'vitest', 'run', '--project', '@paperclipai/server',
+    'server/src/services/workspace-git-snapshot-streaming.test.ts', 'server/src/__tests__/workspace-runtime.test.ts',
+    '-t', 'streams.*all four real Git|commits fixed-port reservations']],
+];
+for (const [variant, cwd] of [['baseline', baseline], ['candidate', candidate]]) {
+  for (const [name, command, args] of commands) {
+    const profile = path.join(output, `${variant}-${name}`);
+    mkdirSync(profile, { recursive: true });
+    const started = performance.now();
+    const result = spawnSync(command, args, {
+      cwd, encoding: 'utf8', timeout: 15 * 60_000, maxBuffer: 64 * 1024 * 1024,
+      env: { ...process.env, GITHUB_WORKSPACE: cwd, NODE_ENV: 'test', PAPERCLIP_CI_PROFILE: 'true',
+        PAPERCLIP_CI_PROFILE_DIR: profile, PAPERCLIP_CI_PROFILE_JOB: `benchmark-${variant}-${name}` },
+    });
+    const wallMs = performance.now() - started;
+    writeFileSync(path.join(output, `${variant}-${name}.log`), `${result.stdout ?? ''}\n${result.stderr ?? ''}`);
+    const records = readdirSync(profile).filter((file) => file.endsWith('.ndjson'))
+      .flatMap((file) => readFileSync(path.join(profile, file), 'utf8').split('\n').filter(Boolean).map((line) => JSON.parse(line)));
+    results.push({ variant, name, wallMs, status: result.status,
+      sha: execFileSync('git', ['rev-parse', 'HEAD'], { cwd, encoding: 'utf8' }).trim(),
+      tests: records.filter((record) => record.kind === 'test' && record.status !== 'skipped')
+        .map((record) => [record.file, record.name, record.status]).sort(),
+      files: records.filter((record) => record.kind === 'file').map((record) => record.file).sort(),
+    });
+    writeFileSync(path.join(output, 'results.json'), JSON.stringify({ node: process.version, results }, null, 2));
+    assert.equal(result.status, 0, `${variant}/${name} failed: ${result.error ?? ''}; see ${output}`);
+    console.log(`${variant}/${name}: ${(wallMs / 1000).toFixed(2)}s`);
+  }
+}
+const rows = [];
+for (const [name] of commands) {
+  const before = results.find((result) => result.variant === 'baseline' && result.name === name);
+  const after = results.find((result) => result.variant === 'candidate' && result.name === name);
+  assert.ok(before.tests.length > 0, `${name}: missing test evidence`);
+  assert.deepEqual(after.files, before.files, `${name}: file inventory changed`);
+  if (name === 'general-server') assert.deepEqual(after.tests, before.tests, 'general server case inventory changed');
+  else {
+    const contracts = (tests) => {
+      const found = new Set();
+      for (const [file, title, status] of tests) {
+        assert.equal(status, 'passed');
+        if (file.endsWith('/workspace-runtime.test.ts') && title.includes('commits fixed-port reservations')) found.add('database');
+        if (file.endsWith('/workspace-git-snapshot-streaming.test.ts')) {
+          if (title.includes('above 32 MiB')) found.add('large Git lanes');
+          if (/stag(?:es|ing)/.test(title)) found.add('bounded staging');
+        }
+      }
+      return [...found].sort();
+    };
+    for (const result of [before, after]) assert.deepEqual(contracts(result.tests),
+      ['bounded staging', 'database', 'large Git lanes'], 'fixture contracts must all execute');
+  }
+  rows.push(`| ${name} | ${(before.wallMs / 1000).toFixed(2)}s | ${(after.wallMs / 1000).toFixed(2)}s | ${((1 - after.wallMs / before.wallMs) * 100).toFixed(1)}% | ${after.tests.length} |`);
+}
+const summary = ['| Work | Baseline | Candidate | Reduction | Executed cases |',
+  '|---|---:|---:|---:|---:|', ...rows,
+  '', 'Paired sequential runs on one host with Node ' + process.version + '. Setup, cache transfer, queue time and the full CI critical path are outside these command timings.',
+].join('\n');
+writeFileSync(path.join(output, 'summary.md'), summary + '\n');
+console.log(summary);
+const total = (variant) => results.filter((result) => result.variant === variant).reduce((sum, result) => sum + result.wallMs, 0);
+assert.ok(total('candidate') < total('baseline'), 'No combined speedup measured; inspect the paired evidence before claiming an improvement');

scripts/ci-dependency-cache.mjs

@@ -0,0 +1,83 @@
+import { execFileSync } from "node:child_process";
+import { createHash } from "node:crypto";
+import { appendFileSync, existsSync, readFileSync, realpathSync } from "node:fs";
+import path from "node:path";
+import { fileURLToPath } from "node:url";
+import { readPluginsUnder } from "./link-plugin-dev-sdk.mjs";
+
+const root = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..");
+
+export function cachePlan({ files, read, scope, packageDirs }) {
+  const dependencyInputs = files.filter((file) =>
+    /(^|\/)package\.json$/.test(file)
+    || ["pnpm-lock.yaml", "pnpm-workspace.yaml", ".npmrc", ".pnpmfile.cjs", "pnpmfile.cjs"].includes(file)
+    || file.startsWith("scripts/")
+    || file.startsWith("patches/"));
+  const digest = (inputs, salt) => {
+    const hash = createHash("sha256").update(JSON.stringify(salt));
+    for (const file of [...inputs].sort()) {
+      const bytes = read(file);
+      hash.update(JSON.stringify([file, bytes === null ? null : bytes.length]));
+      if (bytes !== null) hash.update(bytes);
+    }
+    return hash.digest("hex");
+  };
+  const dependencyKey = `ci-dependencies-v1-${digest(dependencyInputs, scope)}`;
+  const cacheable = packageDirs.every((directory) => {
+    const manifest = read(path.posix.join(directory, 'package.json'));
+    if (manifest === null) return false;
+    const scripts = JSON.parse(manifest).scripts ?? {};
+    return ['preinstall', 'install', 'postinstall', 'prepare'].every((hook) =>
+      scripts[hook] === undefined || (directory === '.' && hook === 'postinstall'
+        && scripts[hook] === 'node scripts/link-plugin-dev-sdk.mjs'));
+  });
+  const buildInputs = files.filter((file) =>
+    file.startsWith("packages/shared/") || file.startsWith("packages/plugins/sdk/")
+    || /^tsconfig[^/]*\.json$/.test(file)
+    || file === "scripts/ensure-plugin-build-deps.mjs");
+  return {
+    cacheable,
+    dependencyKey,
+    buildKey: `ci-build-dependencies-v1-${digest(buildInputs, dependencyKey)}`,
+    dependencyPaths: [...new Set(packageDirs.map((directory) => {
+      if (path.isAbsolute(directory) || directory.split('/').includes('..')) throw new Error('Package directory escapes the checkout');
+      return path.posix.join(directory, 'node_modules');
+    }))].sort(),
+  };
+}
+
+if (process.argv[1] && path.resolve(process.argv[1]) === fileURLToPath(import.meta.url)) {
+  const files = execFileSync("git", ["ls-files", "-z"], { cwd: root }).toString().split("\0").filter(Boolean);
+  const command = (args) => execFileSync("pnpm", args, { cwd: root, encoding: "utf8" }).trim();
+  const workspaces = JSON.parse(command(['-r', 'list', '--depth=-1', '--json']));
+  const packageDirs = ['.',
+    ...workspaces.map((workspace) => path.relative(root, workspace.path).split(path.sep).join('/') || '.'),
+    ...readPluginsUnder(path.join(root, 'packages/plugins/sandbox-providers')).map((directory) => path.relative(root, directory).split(path.sep).join('/')),
+    'packages/plugins/examples/plugin-orchestration-smoke-example',
+  ];
+  const plan = cachePlan({
+    files,
+    read: (file) => existsSync(path.join(root, file)) ? readFileSync(path.join(root, file)) : null,
+    packageDirs,
+    scope: {
+      root: realpathSync(root), node: process.version, abi: process.versions.modules,
+      platform: process.platform, arch: process.arch,
+      libc: process.report.getReport().header.glibcVersionRuntime ?? null,
+      pnpm: command(["--version"]), store: command(["store", "path", "--silent"]),
+    },
+  });
+  const output = [
+    `cacheable=${plan.cacheable}`,
+    `dependency-key=${plan.dependencyKey}`,
+    `build-key=${plan.buildKey}`,
+    "dependency-paths<<PAPERCLIP_CACHE_PATHS",
+    ...plan.dependencyPaths.map((file) => path.join(root, file)),
+    "PAPERCLIP_CACHE_PATHS",
+    "build-paths<<PAPERCLIP_BUILD_PATHS",
+    path.join(root, "packages/shared/dist"),
+    path.join(root, "packages/plugins/sdk/dist"),
+    "PAPERCLIP_BUILD_PATHS",
+  ].join("\n") + "\n";
+  if (process.env.GITHUB_OUTPUT) appendFileSync(process.env.GITHUB_OUTPUT, output);
+  else console.log(JSON.stringify(plan, null, 2));
+}

scripts/docker-compose-secrets.test.mjs

@@ -0,0 +1,42 @@
+import assert from "node:assert/strict";
+import { spawnSync } from "node:child_process";
+import { mkdtempSync, rmSync, writeFileSync } from "node:fs";
+import os from "node:os";
+import path from "node:path";
+import test from "node:test";
+import { fileURLToPath } from "node:url";
+
+const repoRoot = fileURLToPath(new URL("../", import.meta.url));
+const baseEnv = Object.fromEntries(
+  ["PATH", "HOME", "SystemRoot", "ComSpec", "TEMP", "TMP", "USERPROFILE"]
+    .filter((key) => process.env[key] !== undefined)
+    .map((key) => [key, process.env[key]]),
+);
+const compose = spawnSync("docker", ["compose", "version"], { env: baseEnv });
+assert.equal(compose.status, 0, "Docker Compose is required for this integration check.");
+
+for (const [file, service] of [
+  ["docker/docker-compose.quickstart.yml", "paperclip"],
+  ["docker/docker-compose.yml", "server"],
+]) {
+  test(`${file} forwards the independent tool signing secret`, () => {
+    const temporary = mkdtempSync(path.join(os.tmpdir(), "paperclip-compose-secrets-"));
+    const envFile = path.join(temporary, "empty.env");
+    writeFileSync(envFile, "");
+    try {
+      for (const signingSecret of ["test-tool-signing-secret", undefined]) {
+        const env = { ...baseEnv, BETTER_AUTH_SECRET: "test-session-secret" };
+        if (signingSecret !== undefined) env.PAPERCLIP_TOOL_ACTION_SIGNING_SECRET = signingSecret;
+        const result = spawnSync("docker", [
+          "compose", "--env-file", envFile, "-f", file, "config", "--format", "json",
+        ], { cwd: repoRoot, env, encoding: "utf8" });
+        assert.equal(result.status, 0, result.stderr);
+        const containerEnv = JSON.parse(result.stdout).services[service].environment;
+        assert.equal(containerEnv.PAPERCLIP_TOOL_ACTION_SIGNING_SECRET, signingSecret ?? "");
+        assert.equal(containerEnv.BETTER_AUTH_SECRET, "test-session-secret");
+      }
+    } finally {
+      rmSync(temporary, { recursive: true, force: true });
+    }
+  });
+}

scripts/run-vitest-stable.mjs

@@ -1,5 +1,5 @@
 #!/usr/bin/env node
-import { spawnSync } from "node:child_process";
+import { spawn, spawnSync } from "node:child_process";
 import { mkdirSync, mkdtempSync, readdirSync, readFileSync, realpathSync, statSync, writeFileSync } from "node:fs";
 import os from "node:os";
 import path from "node:path";
@@ -377,7 +377,7 @@ function selectSerializedSuites(routeTests, shardIndex, shardCount) {
   return shardFiles.map((file) => byRepoPath.get(file));
 }
 
-function runVitest(args, label, testShard = null) {
+async function runVitest(args, label, testShard = null) {
   console.log(`\n[test:run] ${label}`);
   invocationIndex += 1;
   const tempRootParent = process.platform === "win32" ? os.tmpdir() : "/tmp";
@@ -408,7 +408,7 @@ function runVitest(args, label, testShard = null) {
         ...filters, "--allowOnly=false", "--includeTaskLocation", `--json=${output}`], {
         cwd: repoRoot, env, stdio: "inherit",
       });
-      if (result.error || result.status !== 0) fail(`Vitest collection failed: ${result.error?.message ?? result.status}`);
+      if (result.error || result.status !== 0) throw new Error(`Vitest collection failed: ${result.error?.message ?? result.status}`);
       return JSON.parse(readFileSync(output, "utf8"));
     };
     const allCollected = collect(args, "all");
@@ -434,27 +434,27 @@ function runVitest(args, label, testShard = null) {
     const runFilters = selected.lines.map((line) => `${chatSuite}:${chatSuiteRunLine(sourceLines, line)}`);
     args = [...args.filter((arg) => arg !== chatSuite), ...runFilters, "--allowOnly=false"];
   }
-  const result = spawnSync("pnpm", ["exec", "vitest", "run", ...sourceOnlyVitestArgs, ...args], {
-    cwd: repoRoot,
-    env: invocationEnv("vitest", env),
-    stdio: "inherit",
+  await new Promise((resolve, reject) => {
+    const child = spawn("pnpm", ["exec", "vitest", "run", ...sourceOnlyVitestArgs, ...args], {
+      cwd: repoRoot,
+      env: invocationEnv("vitest", env),
+      stdio: "inherit",
+    });
+    child.once("error", reject);
+    child.once("close", (code, signal) => {
+      if (code === 0) resolve();
+      else reject(new Error(`${label} failed (${signal ?? code})`));
+    });
   });
-  if (result.error) {
-    console.error(`[test:run] Failed to start Vitest: ${result.error.message}`);
-    process.exit(1);
-  }
-  if (result.status !== 0) {
-    process.exit(result.status ?? 1);
-  }
 }
 
-function runGeneralSuites(routeTests) {
+async function runGeneralSuites(routeTests) {
   for (const groupName of generalGroupNames) {
-    runGeneralGroup(routeTests, groupName);
+    await runGeneralGroup(routeTests, groupName);
   }
 }
 
-function runProjectGroup(projects, groupName, shardIndex = null, shardCount = null) {
+async function runProjectGroup(projects, groupName, shardIndex = null, shardCount = null) {
   // With shard args, lean on Vitest's native --shard: each matrix job runs the
   // same per-project invocations but only its slice of each project's test
   // files. Vitest's sharding is deterministic for an identical file list, so
@@ -463,18 +463,32 @@ function runProjectGroup(projects, groupName, shardIndex = null, shardCount = nu
     shardCount !== null && shardCount > 1 ? [`--shard=${shardIndex + 1}/${shardCount}`] : [];
   const shardSuffix = shardArgs.length > 0 ? ` shard ${shardIndex + 1}/${shardCount}` : "";
   for (const project of projects) {
-    runVitest(["--project", project, ...shardArgs], `${groupName} project ${project}${shardSuffix}`);
+    await runVitest(["--project", project, ...shardArgs], `${groupName} project ${project}${shardSuffix}`);
   }
 }
 
-function runGeneralGroup(routeTests, groupName, shardIndex = null, shardCount = null) {
+function chatShardWorkers(index, count) {
+  const workers = count > 1 && !process.env.PAPERCLIP_TEST_DATABASE_URL ? 2 : 1;
+  return Array.from({ length: workers }, (_, offset) => ({
+    index: (index ?? 0) * workers + offset,
+    count: (count ?? 1) * workers,
+  }));
+}
+
+async function runGeneralGroup(routeTests, groupName, shardIndex = null, shardCount = null) {
   if (groupName === generalChatGroupName) {
-    runVitest(["--project", "@paperclipai/server", ...serializedServerVitestArgs, chatSuite],
-      "chat integration test shard", { index: shardIndex ?? 0, count: shardCount ?? 1 });
+    // Each invocation gets its own temporary home and embedded PostgreSQL.
+    // Wait for both children even on failure so no sibling is abandoned.
+    const results = await Promise.allSettled(chatShardWorkers(shardIndex, shardCount).map((shard) =>
+      runVitest(["--project", "@paperclipai/server", ...serializedServerVitestArgs, chatSuite],
+        "chat integration test shard", shard),
+    ));
+    const failed = results.find((result) => result.status === "rejected");
+    if (failed) throw failed.reason;
     return;
   }
   if (groupName === generalServerNativeRunnerGroupName) {
-    runVitest(
+    await runVitest(
       ["--project", "@paperclipai/server", ...serializedServerVitestArgs, ...nativeRunnerSuites],
       "native runner vertical-slice suite",
     );
@@ -497,11 +511,16 @@ function runGeneralGroup(routeTests, groupName, shardIndex = null, shardCount =
         return;
       }
 
-      runVitest(
+      await runVitest(
         [
           "--project",
           "@paperclipai/server",
-          ...serializedServerVitestArgs,
+          // Files have separate fork processes and isolated homes/configs.
+          // Keep each file's cases serial; overlap only two independent files.
+          // An explicit shared test database cannot be isolated this way.
+          ...(process.env.PAPERCLIP_TEST_DATABASE_URL
+            ? serializedServerVitestArgs
+            : ["--pool=forks", "--isolate", "--fileParallelism", "--maxWorkers=2"]),
           ...shardFiles,
         ],
         `${groupName} shard ${shardIndex + 1}/${shardCount}`,
@@ -513,7 +532,7 @@ function runGeneralGroup(routeTests, groupName, shardIndex = null, shardCount =
     for (const suite of excludedSuites) {
       excludeRouteArgs.push("--exclude", suite.replace(/^server\//, ""));
     }
-    runVitest(
+    await runVitest(
       [
         "--project",
         "@paperclipai/server",
@@ -530,39 +549,40 @@ function runGeneralGroup(routeTests, groupName, shardIndex = null, shardCount =
     // 31371439296, 2026-08-10, where workspaces-a was the slowest PR check).
     // Its 439 test files shard cleanly with Vitest's native --shard, so the
     // lane splits across runners without a duration manifest.
-    runProjectGroup(generalWorkspacesAProjects, groupName, shardIndex, shardCount);
+    await runProjectGroup(generalWorkspacesAProjects, groupName, shardIndex, shardCount);
     return;
   }
 
   if (groupName === generalWorkspacesBGroupName) {
-    runProjectGroup(generalWorkspacesBProjects, groupName);
+    await runProjectGroup(generalWorkspacesBProjects, groupName);
     return;
   }
 
   if (groupName === generalWorkspacesCGroupName) {
-    runProjectGroup(generalWorkspacesCProjects, groupName);
+    await runProjectGroup(generalWorkspacesCProjects, groupName);
     return;
   }
 
   fail(`Unknown group "${groupName}".`);
 }
 
-function runSerializedSuites(routeTests, shardIndex, shardCount) {
+async function runSerializedSuites(routeTests, shardIndex, shardCount) {
   const shardTests = selectSerializedSuites(routeTests, shardIndex, shardCount);
   console.log(
     `\n[test:run] serialized shard ${shardIndex + 1}/${shardCount} running ${shardTests.length} of ${routeTests.length} suites`,
   );
 
-  for (const routeTest of shardTests) {
-    runVitest(
+  if (shardTests.length > 0) {
+    await runVitest(
       [
         "--project",
         "@paperclipai/server",
-        routeTest.repoPath,
+        ...shardTests.map((test) => test.repoPath),
         "--pool=forks",
         "--isolate",
+        ...serializedServerVitestArgs,
       ],
-      routeTest.repoPath,
+      `serialized shard ${shardIndex + 1}/${shardCount}`,
     );
   }
 }
@@ -577,9 +597,9 @@ const routeTests = walk(serverTestsDir)
 
 // Every server test file that the general-server group is responsible for,
 // i.e. the whole server project minus the route/authz suites that run in the
-// dedicated serialized shards. Sharding this list across runners is what keeps
-// the general-server lane from becoming the PR critical path: the server vitest
-// config pins maxWorkers to 1, so the only way to parallelize is across jobs.
+// dedicated serialized shards. The default server config keeps files serial;
+// the sharded entry point overlaps two isolated files per runner unless an
+// explicit shared test database requires serial execution.
 // Suites are partitioned by recorded duration (scripts/general-server-shard.mjs)
 // rather than round-robin, so one slow suite cluster can't stretch a single shard.
 const serializedRepoPaths = new Set(routeTests.map(test => test.repoPath));
@@ -624,6 +644,8 @@ if (options.dryRun) {
                   generalServerShardDurations,
                 )
               : null,
+        chatVitestShards: options.group === generalChatGroupName
+          ? chatShardWorkers(options.shardIndex, options.shardCount) : null,
         workspaceProjects:
           options.group === generalWorkspacesAGroupName
             ? generalWorkspacesAProjects
@@ -646,14 +668,19 @@ if (options.dryRun) {
   process.exit(0);
 }
 
-if (options.mode === generalModeName || options.mode === allModeName) {
-  if (options.group) {
-    runGeneralGroup(routeTests, options.group, options.shardIndex, options.shardCount);
-  } else {
-    runGeneralSuites(routeTests);
+try {
+  if (options.mode === generalModeName || options.mode === allModeName) {
+    if (options.group) {
+      await runGeneralGroup(routeTests, options.group, options.shardIndex, options.shardCount);
+    } else {
+      await runGeneralSuites(routeTests);
+    }
   }
-}
 
-if (options.mode === serializedModeName || options.mode === allModeName) {
-  runSerializedSuites(routeTests, options.shardIndex ?? 0, options.shardCount ?? 1);
+  if (options.mode === serializedModeName || options.mode === allModeName) {
+    await runSerializedSuites(routeTests, options.shardIndex ?? 0, options.shardCount ?? 1);
+  }
+} catch (error) {
+  console.error(`[test:run] ${error.message}`);
+  process.exitCode = 1;
 }

server/scripts/benchmark-workspace-admission.ts

@@ -0,0 +1,282 @@
+import assert from "node:assert/strict";
+import { randomUUID, createHash } from "node:crypto";
+import { execFile as execFileCallback } from "node:child_process";
+import { promisify } from "node:util";
+import { mkdtemp, rm, mkdir } from "node:fs/promises";
+import fs from "node:fs/promises";
+import os from "node:os";
+import path from "node:path";
+import { pathToFileURL } from "node:url";
+import { and, eq, inArray } from "drizzle-orm";
+import { drizzle } from "drizzle-orm/postgres-js";
+import {
+  companies,
+  createDb,
+  executionWorkspaceRepositoryAccess,
+  executionWorkspaces,
+  heartbeatRuns,
+  issues,
+  startEmbeddedPostgresTestDatabase,
+} from "@paperclipai/db";
+
+import { createAgentLifecycle } from "../src/services/agent-lifecycle.js";
+
+const execFile = promisify(execFileCallback);
+
+function option(name: string, fallback: string) {
+  const index = process.argv.indexOf(name);
+  return index < 0 ? fallback : process.argv[index + 1] ?? fallback;
+}
+
+function latency(name: string, fallback: number) {
+  const value = Number(option(name, String(fallback)));
+  if (!Number.isFinite(value) || value < 0) throw new Error(`${name} must be a non-negative number`);
+  return value;
+}
+
+function count(name: string, fallback: number) {
+  const value = Number(option(name, String(fallback)));
+  if (!Number.isSafeInteger(value) || value < 0) throw new Error(`${name} must be a non-negative integer`);
+  return value;
+}
+
+async function insertBatches<T extends Record<string, unknown>>(
+  insert: (values: T[]) => Promise<unknown>,
+  values: T[],
+) {
+  for (let offset = 0; offset < values.length; offset += 500) {
+    await insert(values.slice(offset, offset + 500));
+  }
+}
+
+const workspaceCount = count("--workspaces", 8_516);
+const rootWorkspaceCount = count("--root-workspaces", 5_315);
+const sourceEntryCount = count("--source-entries", 17_256);
+const isolatedSourceEntryCount = count("--isolated-source-entries", 10_630);
+const readerEntryCount = count("--readers", 51);
+const rootReaderCount = count("--root-readers", 28);
+const otherAccessEntryCount = count("--other-access-entries", 34);
+const realpathDelayMs = latency("--realpath-delay-ms", 0);
+const distinctCwds = process.argv.includes("--distinct-cwds");
+if (workspaceCount < 1 || rootWorkspaceCount < 1 || rootWorkspaceCount > workspaceCount) {
+  throw new Error("Need at least one workspace; root workspaces must be between one and the workspace total");
+}
+if (readerEntryCount < rootReaderCount) {
+  throw new Error("Root reader count cannot exceed the company-wide reader total");
+}
+
+const configuredService = option("--service", "");
+const serviceUrl = configuredService
+  ? pathToFileURL(path.resolve(configuredService))
+  : new URL("../src/services/execution-workspace-repositories.ts", import.meta.url);
+
+let measuring = false;
+let realpathCalls = 0;
+let realpathIoMs = 0;
+let injectedWaitMs = 0;
+const actualRealpath = fs.realpath.bind(fs);
+fs.realpath = (async (...args: Parameters<typeof fs.realpath>) => {
+  if (!measuring) return actualRealpath(...args);
+  const started = performance.now();
+  try {
+    return await actualRealpath(...args);
+  } finally {
+    realpathCalls += 1;
+    realpathIoMs += performance.now() - started;
+    if (realpathDelayMs > 0) {
+      const waitStarted = performance.now();
+      await new Promise((resolve) => setTimeout(resolve, realpathDelayMs));
+      injectedWaitMs += performance.now() - waitStarted;
+    }
+  }
+}) as typeof fs.realpath;
+
+const { executionWorkspaceRepositoryService } = await import(serviceUrl.href);
+
+const temp = await startEmbeddedPostgresTestDatabase("paperclip-workspace-admission-bench-");
+const db = createDb(temp.connectionString);
+const scratchRoot = await fs.realpath(await mkdtemp(path.join(os.tmpdir(), "paperclip-workspace-admission-")));
+const repoRoot = path.join(scratchRoot, "shared-repository");
+const unrelatedRoot = path.join(scratchRoot, "unrelated-workspaces");
+
+try {
+  await mkdir(repoRoot, { recursive: true });
+  await mkdir(unrelatedRoot, { recursive: true });
+  await execFile("git", ["init", "--quiet", "--initial-branch=main", repoRoot]);
+
+  const companyId = randomUUID();
+  const agentId = randomUUID();
+  const issueId = randomUUID();
+  const workspaceIds = Array.from({ length: workspaceCount }, () => randomUUID());
+  const workspaceCwds = workspaceIds.map((_, index) => {
+    const root = index < rootWorkspaceCount ? repoRoot : unrelatedRoot;
+    return distinctCwds && index !== 0 ? path.join(root, `workspace-${index}`) : root;
+  });
+  if (distinctCwds) for (let offset = 0; offset < workspaceCwds.length; offset += 64) {
+    await Promise.all(workspaceCwds.slice(offset, offset + 64).map(cwd => mkdir(cwd, { recursive: true })));
+  }
+  const runIds = Array.from({ length: Math.max(1, rootReaderCount) }, () => randomUUID());
+  const now = new Date();
+  const rootKey = createHash("sha256").update(repoRoot).digest("hex");
+  const unrelatedKey = createHash("sha256").update(unrelatedRoot).digest("hex");
+
+  await db.insert(companies).values({ id: companyId, name: "Synthetic workspace-admission benchmark", issuePrefix: "WAB" });
+  await createAgentLifecycle(db).requestHire(companyId, {
+    id: agentId,
+    name: "Synthetic Codie",
+    role: "engineer",
+    status: "idle",
+    adapterType: "codex_local",
+    adapterConfig: {},
+    runtimeConfig: {},
+    permissions: {},
+  }, { runnerResolved: true });
+
+  await insertBatches((rows) => db.insert(executionWorkspaces).values(rows as never), workspaceIds.map((id, index) => ({
+    id,
+    companyId,
+    name: `Synthetic workspace ${index}`,
+    mode: "shared_workspace",
+    strategyType: "project_primary",
+    status: "active",
+    cwd: workspaceCwds[index],
+    providerType: "local_fs",
+  })));
+
+  await db.insert(issues).values({
+    id: issueId,
+    companyId,
+    title: "Synthetic assigned task",
+    status: "in_progress",
+    assigneeAgentId: agentId,
+    executionWorkspaceId: workspaceIds[0],
+    responsibleUserId: "local-board",
+  });
+  for (let offset = 0; offset < rootWorkspaceCount; offset += 500) {
+    await db.update(executionWorkspaces).set({ sourceIssueId: issueId })
+      .where(and(eq(executionWorkspaces.companyId, companyId), inArray(executionWorkspaces.id, workspaceIds.slice(offset, offset + 500))));
+  }
+
+  if (runIds.length) {
+    await db.insert(heartbeatRuns).values(runIds.map((id) => ({
+      id,
+      companyId,
+      agentId,
+      scopeKind: "issue" as const,
+      issueId,
+      nativeIssueId: issueId,
+      invocationSource: "on_demand",
+      status: "completed",
+      responsibleUserId: "local-board",
+      runtimeMode: "legacy",
+      contextSnapshot: { issueId },
+      startedAt: now,
+      finishedAt: now,
+    })));
+  }
+
+  const accessRows: Array<typeof executionWorkspaceRepositoryAccess.$inferInsert> = [];
+  for (let index = 0; index < sourceEntryCount; index++) {
+    const isRootSource = index < rootWorkspaceCount;
+    accessRows.push({
+      companyId,
+      rootKey: isRootSource ? rootKey : unrelatedKey,
+      canonicalCwd: isRootSource ? repoRoot : unrelatedRoot,
+      kind: "source",
+      scopeId: isRootSource ? workspaceIds[index] : randomUUID(),
+    });
+  }
+  for (let index = 0; index < isolatedSourceEntryCount; index++) {
+    accessRows.push({ companyId, rootKey: unrelatedKey, canonicalCwd: unrelatedRoot, kind: "isolated_source", scopeId: randomUUID() });
+  }
+  for (let index = 0; index < readerEntryCount; index++) {
+    const rootReader = index < rootReaderCount;
+    accessRows.push({
+      companyId,
+      rootKey: rootReader ? rootKey : unrelatedKey,
+      canonicalCwd: rootReader ? repoRoot : unrelatedRoot,
+      kind: "reader",
+      scopeId: rootReader ? runIds[index] : randomUUID(),
+      issueId: rootReader ? issueId : randomUUID(),
+      agentId: rootReader ? agentId : randomUUID(),
+      responsibleUserId: "local-board",
+    });
+  }
+  for (let index = 0; index < otherAccessEntryCount; index++) {
+    accessRows.push({ companyId, rootKey: unrelatedKey, canonicalCwd: unrelatedRoot, kind: "repository", scopeId: randomUUID() });
+  }
+  await insertBatches((rows) => db.insert(executionWorkspaceRepositoryAccess).values(rows as never), accessRows);
+
+  let queryCount = 0;
+  let provenanceInsertCount = 0;
+  const measuredDb = drizzle(db.$client, {
+    schema: db._.fullSchema,
+    logger: { logQuery(query: string) {
+      if (!measuring) return;
+      queryCount++;
+      if (query.startsWith('insert into "execution_workspace_repository_access"')) provenanceInsertCount++;
+    } },
+  }) as typeof db;
+  const service = executionWorkspaceRepositoryService(measuredDb);
+  realpathCalls = 0;
+  realpathIoMs = 0;
+  process.stderr.write(`Starting admission benchmark: ${workspaceCount} workspaces, ${rootReaderCount} retained readers, ${realpathDelayMs} ms injected per realpath.\n`);
+  measuring = true;
+  const startedAt = performance.now();
+  await service.prepareConfiguredRepositorySource({
+    companyId,
+    issueId,
+    workspaceId: workspaceIds[0],
+    cwd: repoRoot,
+    sourceCwd: repoRoot,
+    agentId,
+    runId: runIds[0] ?? randomUUID(),
+  });
+  const elapsedMs = Math.round(performance.now() - startedAt);
+  measuring = false;
+  process.stderr.write(`Admission service call completed in ${elapsedMs} ms.\n`);
+
+  const retained = await db.select({ kind: executionWorkspaceRepositoryAccess.kind, scopeId: executionWorkspaceRepositoryAccess.scopeId })
+    .from(executionWorkspaceRepositoryAccess).where(and(eq(executionWorkspaceRepositoryAccess.companyId, companyId),
+      eq(executionWorkspaceRepositoryAccess.rootKey, rootKey), eq(executionWorkspaceRepositoryAccess.kind, "isolated_source")));
+  const retainedIds = new Set(retained.map(row => row.scopeId));
+  assert.equal(retainedIds.size, rootWorkspaceCount, "Every source must retain isolated-copy provenance");
+  assert.ok(workspaceIds.slice(0, rootWorkspaceCount).every(id => retainedIds.has(id)), "Source provenance identities must match");
+  const metadataRoot = await fs.realpath(path.join(repoRoot, ".git"));
+  const metadataOwners = await db.select({ scopeId: executionWorkspaceRepositoryAccess.scopeId })
+    .from(executionWorkspaceRepositoryAccess).where(and(eq(executionWorkspaceRepositoryAccess.companyId, companyId),
+      eq(executionWorkspaceRepositoryAccess.rootKey, createHash("sha256").update(metadataRoot).digest("hex")),
+      eq(executionWorkspaceRepositoryAccess.kind, "source")));
+  const metadataOwnerIds = new Set(metadataOwners.map(row => row.scopeId));
+  assert.equal(metadataOwnerIds.size, rootWorkspaceCount, "Git metadata must retain every source owner");
+  assert.ok(workspaceIds.slice(0, rootWorkspaceCount).every(id => metadataOwnerIds.has(id)), "Git metadata source identities must match");
+  const report = {
+    elapsedMs,
+    database: { queries: queryCount, provenanceInserts: provenanceInsertCount },
+    realpath: {
+      calls: realpathCalls,
+      measuredIoMs: Math.round(realpathIoMs),
+      injectedDelayPerCallMs: realpathDelayMs,
+      summedInjectedWaitMs: Math.round(injectedWaitMs),
+    },
+    fixture: {
+      workspaces: workspaceCount,
+      sharedRootWorkspaces: rootWorkspaceCount,
+      distinctWorkspaceCwds: new Set(workspaceCwds).size,
+      repositoryAccessRows: accessRows.length,
+      sourceRows: sourceEntryCount,
+      isolatedSourceRows: isolatedSourceEntryCount,
+      retainedReaders: rootReaderCount,
+      allCompanyReaders: readerEntryCount,
+      otherAccessRows: otherAccessEntryCount,
+    },
+    service: serviceUrl.pathname,
+    note: "Synthetic local service benchmark only; no provider invocation.",
+  };
+  await new Promise<void>((resolve, reject) => {
+    process.stdout.write(`${JSON.stringify(report, null, 2)}\n`, (error) => error ? reject(error) : resolve());
+  });
+} finally {
+  await temp.cleanup();
+  await rm(scratchRoot, { recursive: true, force: true });
+}

server/src/__tests__/ai-connection-router.test.ts

@@ -1,11 +1,11 @@
 import { enableRunFailureReporting } from "./helpers/run-failure-reporting.js";
 import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
 import { randomUUID } from "node:crypto";
-import { mkdtemp, rm } from "node:fs/promises";
-import os from "node:os";
+import { mkdir } from "node:fs/promises";
+import { createIsolatedTestHome } from "./helpers/isolated-test-home.js";
 import path from "node:path";
 import { and, eq } from "drizzle-orm";
-import { createDb, closeRegisteredClients, companies, agents, companyMemberships, connectionGrants, plugins, aiConnectionRouterCursors, aiConnectionTaskPins, toolConnections, companySecrets, heartbeatRuns, issues } from "@paperclipai/db";
+import { createDb, closeRegisteredClients, companies, agents, companyMemberships, connectionGrants, plugins, aiConnectionRouterCursors, aiConnectionTaskPins, toolConnections, companySecrets, heartbeatRuns, issues, dotAgentBindings, grokBotRoutines } from "@paperclipai/db";
 import { startEmbeddedPostgresTestDatabase } from "@paperclipai/db/test-embedded-postgres";
 import type { AiConnectionPoolMember, AiConnectionRouterRequest, AiConnectionRouterResult, PaperclipPluginManifestV1 } from "@paperclipai/shared";
 import { aiConnectionRouterService, AiConnectionPoolExhausted, poolMemberRuntimeConfig, applyAiConnectionRouterTaskSettings } from "../services/ai-connection-router.js";
@@ -26,6 +26,7 @@ import { connectionIntentService } from "../services/connection-intents.js";
 import { legacyExecutionNeedsReconciliation } from "../services/legacy-execution-recovery.js";
 import { getServerAdapter, registerServerAdapter, unregisterServerAdapter } from "../adapters/index.js";
 import { waitForPendingRunFailureReports } from "../services/run-failure-report.js";
+import { dotRunnerBroker } from "../services/dot-runner-broker.js";
 
 const captureRunFailure = vi.hoisted(() => vi.fn());
 vi.mock("../sentry.js", async (importOriginal) => ({
@@ -37,6 +38,7 @@ let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
 const externalTestDatabaseUrl = process.env.PAPERCLIP_TEST_DATABASE_URL?.trim();
 let db: ReturnType<typeof createDb>;
 let home: string;
+let isolatedHome: ReturnType<typeof createIsolatedTestHome> | undefined;
 const companyId = randomUUID(), otherCompanyId = randomUUID(), agentId = randomUUID(), secondAgentId = randomUUID(), pluginId = randomUUID();
 const pluginKey = "fixture.ai-router";
 const manifest: PaperclipPluginManifestV1 = { id: pluginKey, apiVersion: 1, version: "0.1.0", displayName: "Fixture router", description: "Fixture", author: "Tests", categories: ["connector"], capabilities: ["ai.connections.route"], aiConnectionRouter: { name: "AI connection pool", description: "Use saved connections" }, entrypoints: { worker: "worker.js" } };
@@ -53,7 +55,8 @@ const service = () => aiConnectionRouterService(db, worker);
 const resolve = (poolId: string, taskKey: string, extra = {}) => service().resolve({ companyId, poolId, agentId, userId: "alice", adapterType: "paperclip_runner", taskKey, ...extra });
 const makePool = async (config = {}) => service().save(pluginKey, { companyId, config: { name: randomUUID(), enabled: true, mode: "round_robin", thresholdPercent: 90, members, ...config } }, "alice");
 beforeAll(async () => {
-  home = await mkdtemp(path.join(os.tmpdir(), "paperclip-router-tests-"));
+  isolatedHome = createIsolatedTestHome();
+  home = isolatedHome.home;
   vi.stubEnv("PAPERCLIP_HOME", home); vi.stubEnv("PAPERCLIP_INSTANCE_ID", "router-fixture");
   if (externalTestDatabaseUrl) db = createDb(externalTestDatabaseUrl);
   else { database = await startEmbeddedPostgresTestDatabase("paperclip-router-db-"); db = createDb(database.connectionString); }
@@ -73,7 +76,7 @@ afterAll(async () => {
   await waitForPendingRunFailureReports();
   if (externalTestDatabaseUrl) await closeRegisteredClients(externalTestDatabaseUrl);
   await db?.$client.end();
-  await database?.cleanup(); vi.unstubAllEnvs(); if (home) await rm(home, { recursive: true, force: true });
+  await database?.cleanup(); vi.unstubAllEnvs(); isolatedHome?.cleanup();
 });
 
 describe("durable, authorized connection routing", () => {
@@ -438,6 +441,51 @@ describe("durable, authorized connection routing", () => {
       expect(resolveTaskSessionConfigFreshness({ hasTaskSession: true, configuredModel: String(selected.runtimeConfig.model), taskSessionParams: { __paperclipConfiguredModel: selected.runtimeConfig.model, __paperclipConfigFingerprint: previous.fingerprint, __paperclipConfigFingerprintVersion: previous.version, __paperclipConfigCategoryFingerprints: previous.categoryFingerprints }, configMetadata: current, compatibleConfigMetadata }).reset).toBe(false);
     } finally { await before.cleanup(); await after.cleanup(); }
   });
+  it("admits a healthy pool member when the saved Grok connection is held", async () => {
+    const original = await instanceSettingsService(db).getExperimental();
+    await instanceSettingsService(db).updateExperimental({ enableAiConnectionRouters: true,
+      enableNativeRunner: true, enableGrokBot: true, enablePublicMcp: true, enableWorktreeRunExecution: true });
+    const pool = await makePool({ members: [members[0]!] });
+    const bindingId = randomUUID();
+    const workspace = path.join(home, "routed-grok-work");
+    await mkdir(workspace);
+    const [agent] = await db.insert(agents).values({ companyId, name: "Routed Grok fixture", status: "idle",
+      adapterType: "paperclip_runner", adapterConfig: { provider: "grok_bot", externalBindingId: bindingId, cwd: workspace },
+      runtimeConfig: { aiConnection: { mode: "router", connectionId: pool.id }, heartbeat: { enabled: false, wakeOnDemand: true } } }).returning();
+    await db.insert(dotAgentBindings).values({ id: bindingId, companyId, agentId: agent.id,
+      operatorId: "alice", providerKind: "grok_bot", status: "connected" });
+    const [grant] = await db.select().from(connectionGrants).where(eq(connectionGrants.id, members[0]!.binding.grantId));
+    const ref = grant!.credentialSecretRefs.find(ref => ref.configPath === "ai.credential")!;
+    await db.insert(grokBotRoutines).values({ companyId, agentId: agent.id, bindingId, bindingGeneration: 1,
+      botName: "Unused held Bot", secretId: ref.secretId, secretVersion: 1,
+      recoveryRequiredAt: new Date(), recoveryReason: "work_session" });
+    expect(await dotRunnerBroker(db).canStartNewWork(companyId, agent.id, bindingId)).toBe(false);
+    const readiness = vi.spyOn(dotRunnerBroker(db), "canStartNewWork");
+    const beforeNative = vi.fn(async () => { throw new Error("fixture stopped before provider start"); });
+    const heartbeat = heartbeatService(db, { pluginWorkerManager: worker, runtimeEnv: {},
+      beforeNativeRuntimeSelection: beforeNative,
+      nativeSessionBackendFactory: () => { throw new Error("Unexpected provider start"); } });
+    const [issue] = await db.insert(issues).values({ companyId, title: "Healthy routed task", status: "todo",
+      assigneeAgentId: agent.id, responsibleUserId: "alice", createdByUserId: "alice" }).returning();
+    try {
+      const run = await heartbeat.wakeup(agent.id, { source: "assignment", reason: "issue_assigned", triggerDetail: "system",
+        payload: { issueId: issue.id }, contextSnapshot: { issueId: issue.id, responsibleUserId: "alice" } });
+      expect(run).not.toBeNull();
+      await heartbeat.drainActiveRunExecutions();
+      const saved = await heartbeat.getRun(run!.id);
+      expect(saved?.contextSnapshot?.aiRouterSelection).toMatchObject({ poolId: pool.id,
+        memberId: members[0]!.id, runtimeConfig: { provider: "codex" } });
+      expect(saved?.error).toContain("fixture stopped before provider start");
+      expect(beforeNative).toHaveBeenCalledExactlyOnceWith(run!.id);
+      expect(readiness).not.toHaveBeenCalled();
+    } finally {
+      await db.update(agents).set({ status: "paused" }).where(eq(agents.id, agent.id));
+      await heartbeat.drainActiveRunExecutions();
+      readiness.mockRestore();
+      await instanceSettingsService(db).updateExperimental(original);
+    }
+  }, 30_000);
+
   it("keeps session identity and fingerprints stable on authenticated refresh, but changes them on manual replacement", async () => {
     const input = { companyId, agentId, responsibleUserId: "alice", adapterType: "paperclip_runner", binding: members[0]!.binding, config: { provider: "codex", model: "gpt-5.6-sol" } };
     const first = await prepareManagedAiRuntime(db, input);

server/src/__tests__/browser-use-connection.test.ts

@@ -22,6 +22,7 @@ import {
   issues,
   toolConnections,
   toolApplications,
+  toolCatalogEntries,
   toolConnectionInstalls,
   toolProfiles,
   toolProfileBindings,
@@ -419,6 +420,59 @@ const actor = { actorType: "user" as const, actorId: "browser-reviewer" };
         expect(f.request).not.toHaveBeenCalled();
       } finally { discovery.mockRestore(); }
     });
+    it("resolves only the Browser Use catalog even while full discovery is saturated", async () => {
+      const f = await fixture();
+      const [application] = await db.insert(toolApplications).values({
+        companyId: f.company.id, name: "Unrelated MCP", type: "mcp_http", status: "active",
+      }).returning();
+      const [connection] = await db.insert(toolConnections).values({
+        companyId: f.company.id, applicationId: application.id, name: "Unrelated MCP",
+        uid: randomUUID(), transport: "mcp_remote", status: "active", enabled: true, healthStatus: "ok",
+      }).returning();
+      await db.insert(toolCatalogEntries).values(Array.from({ length: 500 }, (_, i) => ({
+        companyId: f.company.id, applicationId: application.id, connectionId: connection.id,
+        entryKind: "tool" as const, name: `unrelated_${i}`, toolName: `unrelated_${i}`,
+        inputSchema: { type: "object" }, riskLevel: "read" as const, status: "active" as const,
+        versionHash: "fixture", schemaHash: "fixture",
+      })));
+      const { recordingDb } = await import("./helpers/tool-gateway-listing-fixture.js");
+      const { toolDiscoveryScheduler } = await import("../services/tool-discovery-scheduler.js");
+      const releases: Array<() => void> = [];
+      const blockers = [0, 1].map(() => toolDiscoveryScheduler.run(() => new Promise<void>(resolve => releases.push(resolve))));
+      await vi.waitFor(() => expect(releases).toHaveLength(2));
+      const recorded = recordingDb(db);
+      const gateway = createToolGatewayService(recorded.db);
+      const pending = gateway.browserUseResources(f.binding);
+      let timeout: ReturnType<typeof setTimeout> | undefined;
+      try {
+        const resources = await Promise.race([pending, new Promise<never>((_, reject) => {
+          timeout = setTimeout(() => reject(new Error("Assignment waited for unrelated discovery")), 5_000);
+        })]);
+        expect(resources).toEqual([{ id: f.connection.connectionId, connectionId: f.connection.connectionId, label: "Browser Use Cloud" }]);
+        const catalogReads = recorded.statements.flatMap((query, index) =>
+          query.includes('from "tool_catalog_entries"') && query.includes('inner join "tool_connections"') ? [recorded.statementParams[index]] : []);
+        expect(catalogReads).toHaveLength(1);
+        expect(catalogReads[0]).toContain(f.connection.connectionId);
+        expect(catalogReads[0]).not.toContain(connection.id);
+        expect(recorded.statements.some(query => query.includes('from "chat_endpoints"'))).toBe(false);
+      } finally {
+        clearTimeout(timeout);
+        releases.forEach(release => release());
+        await Promise.all(blockers);
+        await pending;
+      }
+    });
+    it.each(["policy", "catalog", "grant"])("rechecks Browser Use %s revocation without a cached assignment", async kind => {
+      const f = await fixture();
+      expect(await f.gateway.browserUseResources(f.binding)).toHaveLength(1);
+      if (kind === "policy") await db.insert(toolPolicies).values({
+        companyId: f.company.id, name: "Revoke Browser Use", policyType: "block",
+        selectors: { connectionId: f.connection.connectionId }, priority: 100,
+      });
+      if (kind === "catalog") await db.update(toolCatalogEntries).set({ quarantinedAt: new Date() }).where(eq(toolCatalogEntries.connectionId, f.connection.connectionId));
+      if (kind === "grant") await db.update(connectionGrants).set({ status: "revoked" }).where(eq(connectionGrants.id, f.grant.id));
+      expect(await f.gateway.browserUseResources(f.binding)).toEqual([]);
+    });
     it("withholds Cloud instructions when connection tools are disabled", async () => {
       const f = await fixture();
       await db.update(toolConnections).set({ enabled: false }).where(eq(toolConnections.id, f.connection.connectionId));

server/src/__tests__/chat-channels.integration.test.ts

@@ -8096,10 +8096,12 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
 
   it("refreshes Slack tool guidance without restoring disabled catalog policies", async () => {
     const { SLACK_TOOLS } = await import("@paperclipai/shared");
-    const { syncSlackBotTools } = await import("../services/connectors/slack-catalog.js");
+    const { slackBotCatalogIsCurrent, syncSlackBotTools } = await import("../services/connectors/slack-catalog.js");
     const fixture = await seedCompany();
     const { endpoint } = await configuredSlackEndpoint(fixture);
+    expect(await slackBotCatalogIsCurrent(db, endpoint)).toBe(false);
     await db.transaction(tx => syncSlackBotTools(tx, endpoint, "owner-user", true));
+    expect(await slackBotCatalogIsCurrent(db, endpoint)).toBe(true);
     const tool = SLACK_TOOLS.find(tool => tool.name === "slack_post_message")!;
     // Simulate the previous hash contract, which omitted descriptions.
     const oldHash = createHash("sha256").update(JSON.stringify({ name: tool.name, inputSchema: tool.inputSchema, risk: tool.risk, scopes: tool.scopes })).digest("hex");
@@ -8108,7 +8110,9 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
       description: "Old posting guidance", versionHash: oldHash, schemaHash: oldHash,
       status: "removed", reviewedByUserId: "owner-user", reviewedAt,
     }).where(and(eq(toolCatalogEntries.connectionId, endpoint.connectionId), eq(toolCatalogEntries.toolName, tool.name)));
+    expect(await slackBotCatalogIsCurrent(db, endpoint)).toBe(false);
     await db.transaction(tx => syncSlackBotTools(tx, endpoint, "different-operator", true));
+    expect(await slackBotCatalogIsCurrent(db, endpoint)).toBe(true);
     const [updated] = await db.select().from(toolCatalogEntries).where(and(eq(toolCatalogEntries.connectionId, endpoint.connectionId), eq(toolCatalogEntries.toolName, tool.name)));
     expect(updated).toMatchObject({ description: tool.description, status: "removed", reviewedByUserId: "owner-user", reviewedAt });
     expect(updated!.versionHash).not.toBe(oldHash);
@@ -8132,6 +8136,12 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
     expect(assignments.map(item => item.key)).toEqual(["slack"]);
     expect(assignments[0].resources[0]).toMatchObject({ id: endpoint.id, metadata: { requesterId: "UROUTINE" } });
     expect(assignments[0].tools.some(tool => tool.name === "slack_open_dm")).toBe(true);
+    const transactions = vi.spyOn(db, "transaction");
+    try {
+      expect(await resolveConnectorAssignments(db, binding)).toEqual(assignments);
+      expect(transactions).toHaveBeenCalledTimes(2);
+    } finally { transactions.mockRestore(); }
+
     await expect(resolveSlackTaskAuthority(db, binding)).resolves.toMatchObject({ conversation: null, userId: "owner-user", slackUserId: "UROUTINE" });
     const upstream = vi.fn(async (input: RequestInfo | URL, init?: RequestInit) => {
       const method = String(input).split("/").at(-1);

server/src/__tests__/company-skills-empty-runtime.test.ts

@@ -1,8 +1,15 @@
 import { randomUUID } from "node:crypto";
-import { expect, it } from "vitest";
+import { expect, it, vi } from "vitest";
 import type { Db } from "@paperclipai/db";
 import { companySkillService } from "../services/company-skills.js";
 
+// Nonempty selection now first acquires a dedicated connection. Route that
+// connection to the same fail-on-access fixture; empty selection must skip it.
+vi.mock("@paperclipai/db", async importOriginal => ({
+  ...await importOriginal<typeof import("@paperclipai/db")>(),
+  withDedicatedDbConnection: <T>(database: Db, action: (connection: Db) => Promise<T>) => action(database),
+}));
+
 const databaseAccess = new Error("Runtime selection touched the inventory database");
 const db = new Proxy({} as Db, { get() { throw databaseAccess; } });
 

server/src/__tests__/git-workspace-expiration.test.ts

@@ -0,0 +1,297 @@
+import { promises as fs } from "node:fs";
+import os from "node:os";
+import path from "node:path";
+import { execFileSync } from "node:child_process";
+import { randomUUID } from "node:crypto";
+import { pathToFileURL } from "node:url";
+import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
+import { eq, sql } from "drizzle-orm";
+import { companies, projects, issues, executionWorkspaces, agents, heartbeatRuns, nativeRunFinalizations, workspaceRuntimeServices, createDb } from "@paperclipai/db";
+import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
+import { resolveManagedProjectWorkspaceDir } from "../home-paths.js";
+import { materializeManagedProjectWorkspace } from "../services/managed-repository-checkout.js";
+import { materializeIsolatedRemoteWorkspaceSource } from "../services/isolated-remote-workspace-source.js";
+import { realizeExecutionWorkspace, ensurePersistedExecutionWorkspaceAvailable } from "../services/workspace-runtime.js";
+import { executionWorkspaceService } from "../services/execution-workspaces.js";
+import { expireReconstructableGitWorkspace } from "../services/git-workspace-expiration.js";
+
+const gitEnv = { ...process.env, GIT_AUTHOR_NAME: "Fixture", GIT_AUTHOR_EMAIL: "fixture@example.test",
+  GIT_COMMITTER_NAME: "Fixture", GIT_COMMITTER_EMAIL: "fixture@example.test", GIT_TERMINAL_PROMPT: "0" };
+const git = (cwd: string, ...args: string[]) => execFileSync("git", args, { cwd, env: gitEnv, encoding: "utf8", stdio: ["ignore", "pipe", "pipe"] }).trim();
+const exists = (target: string) => fs.lstat(target).then(() => true, () => false);
+
+const support = await getEmbeddedPostgresTestSupport();
+const describeEmbeddedPostgres = support.supported ? describe : describe.skip;
+describeEmbeddedPostgres("reconstructable Git expiration", () => {
+  let db: ReturnType<typeof createDb>;
+  let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
+  let root: string;
+  const previousHome = process.env.PAPERCLIP_HOME;
+  const previousInstance = process.env.PAPERCLIP_INSTANCE_ID;
+  beforeAll(async () => {
+    database = await startEmbeddedPostgresTestDatabase("git-cache-expiration-");
+    db = createDb(database.connectionString);
+    root = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-git-expiration-"));
+    process.env.PAPERCLIP_HOME = path.join(root, "home");
+    process.env.PAPERCLIP_INSTANCE_ID = "default";
+  });
+  afterEach(async () => { vi.restoreAllMocks(); await db.execute(sql`TRUNCATE TABLE companies CASCADE`); });
+  afterAll(async () => {
+    if (previousHome === undefined) delete process.env.PAPERCLIP_HOME; else process.env.PAPERCLIP_HOME = previousHome;
+    if (previousInstance === undefined) delete process.env.PAPERCLIP_INSTANCE_ID; else process.env.PAPERCLIP_INSTANCE_ID = previousInstance;
+    await db.$client.end({ timeout: 1 }); await database.cleanup(); await fs.rm(root, { recursive: true, force: true });
+  });
+
+  async function fixture(managed = false, nested = false) {
+    const companyId = randomUUID(), issueId = randomUUID(), workspaceId = randomUUID();
+    const location = path.join(root, companyId), source = path.join(location, "author"), remote = path.join(location, "remote.git");
+    await fs.mkdir(source, { recursive: true });
+    git(source, "init", "--initial-branch=main");
+    await fs.writeFile(path.join(source, "marker.txt"), "commit A\n");
+    await fs.writeFile(path.join(source, ".gitignore"), "node_modules/\nprivate-notes/\n");
+    git(source, "add", "."); git(source, "commit", "-m", "A");
+    const commitA = git(source, "rev-parse", "HEAD");
+    git(location, "clone", "--bare", source, remote); git(source, "remote", "add", "origin", remote);
+    await db.insert(companies).values({ id: companyId, name: "Git expiration fixture", issuePrefix: `GC${companyId.slice(0, 6)}` });
+    await db.insert(issues).values({ id: issueId, companyId, title: "Finished task", status: "done", completedAt: new Date(Date.now() - 48 * 3_600_000) });
+    const repoUrl = pathToFileURL(remote).href;
+    const projectId = managed ? randomUUID() : null;
+    if (projectId) await db.insert(projects).values({ id: projectId, companyId, name: "Fixture project" });
+    const baseCwd = managed
+      ? (await materializeManagedProjectWorkspace(resolveManagedProjectWorkspaceDir({ companyId, projectId: projectId!, repoName: "remote" }), { repoUrl })).cwd
+      : await materializeIsolatedRemoteWorkspaceSource({ companyId, issueId, repoUrl, resolveGitAuth: async () => null });
+    const base = { baseCwd, source: "task_session" as const, projectId: null, workspaceId: null, repoUrl, repoRef: "origin/main" };
+    const workspace = await realizeExecutionWorkspace({ base, issue: { id: issueId, identifier: "GC-1", title: "Fixture" }, agent: { id: null, name: "Fixture", companyId },
+      config: { workspaceStrategy: { type: "git_worktree", branchTemplate: "fixture-task", worktreeParentDir: nested ? path.join(baseCwd, ".paperclip", "worktrees") : path.join(location, "worktrees"), baseRef: "origin/main" } }, skipHostProvision: true });
+    await db.insert(executionWorkspaces).values({ id: workspaceId, companyId, projectId, sourceIssueId: issueId, mode: "isolated_workspace", strategyType: "git_worktree", name: "Fixture", status: "active",
+      providerType: "local_fs", cwd: workspace.cwd, providerRef: workspace.cwd, repoUrl, baseRef: "origin/main", branchName: workspace.branchName,
+      metadata: { createdByRuntime: workspace.branchCreatedByRuntime, gitBranchOwnershipVersion: 1, baseRefSha: workspace.baseRefSha,
+        ...(managed ? {} : { isolatedRemoteSource: { schema: "paperclip.isolated-remote-source.v1", companyId, issueId, repoUrl } }) } });
+    await db.update(issues).set({ executionWorkspaceId: workspaceId }).where(eq(issues.id, issueId));
+    const row = () => db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspaceId)).then(rows => rows[0]!);
+    const expire = async (options: Partial<Parameters<typeof expireReconstructableGitWorkspace>[2]> = {}) =>
+      expireReconstructableGitWorkspace(db, await row(), { dryRun: false, expireBase: true, now: new Date(), ...options });
+    const reopen = (company = companyId) => executionWorkspaceService(db).reopenClosedIsolatedExecutionWorkspaceForIssue({ workspaceId,
+      issue: { id: issueId, companyId: company, projectId: null }, actor: { actorType: "user", agentId: null },
+      authorizationActor: { type: "board", source: "local_implicit", isInstanceAdmin: true } });
+    const advance = async () => {
+      await fs.writeFile(path.join(source, "marker.txt"), "commit B\n"); git(source, "add", "."); git(source, "commit", "-m", "B"); git(source, "push", "origin", "main");
+      return git(source, "rev-parse", "HEAD");
+    };
+    return { companyId, issueId, workspaceId, cwd: workspace.cwd, baseCwd, base, remote, source, commitA, row, expire, reopen, advance };
+  }
+
+  it("expires through the lifecycle worker and restores the recorded commit after upstream advances", async () => {
+    const f = await fixture();
+    await fs.mkdir(path.join(f.cwd, "node_modules")); await fs.writeFile(path.join(f.cwd, "node_modules", "cache"), "regenerable");
+    await fs.symlink("cache", path.join(f.cwd, "node_modules", "cache-link"));
+    expect(await f.expire({ dryRun: true })).toMatchObject({ candidate: true, expired: false, reclaimedBytes: 0, skipReason: null });
+    expect(await exists(f.cwd)).toBe(true); expect((await f.row()).metadata?.gitCacheRestoration).toBeUndefined();
+    await f.advance();
+    const outcome = await executionWorkspaceService(db, { gitCacheExpirationEnabled: true, gitBaseExpirationEnabled: true }).sweepTerminalWorkspaces();
+    expect(outcome.archived).toBe(1);
+    expect(await exists(f.cwd)).toBe(false); expect(await exists(f.baseCwd)).toBe(false);
+    expect((await f.row()).metadata?.gitCacheRestoration).toMatchObject({ commit: f.commitA });
+    expect(await f.reopen()).toMatchObject({ ok: true, reopened: true });
+    expect(git(f.cwd, "rev-parse", "HEAD")).toBe(f.commitA);
+    expect(await fs.readFile(path.join(f.cwd, "marker.txt"), "utf8")).toBe("commit A\n");
+    // The next heartbeat uses this runtime path before handing files to the agent.
+    await ensurePersistedExecutionWorkspaceAvailable({ db, base: f.base,
+      workspace: (await executionWorkspaceService(db).getStoredById(f.companyId, f.workspaceId))!,
+      issue: { id: f.issueId, identifier: "GC-1", title: "Resumed task" },
+      agent: { id: null, name: "Fixture", companyId: f.companyId }, skipHostProvision: true });
+    expect(git(f.cwd, "rev-parse", "HEAD")).toBe(f.commitA);
+    await fs.writeFile(path.join(f.cwd, "continued.txt"), "continued\n"); git(f.cwd, "add", "."); git(f.cwd, "commit", "-m", "Continued");
+    const continued = git(f.cwd, "rev-parse", "HEAD");
+    expect(await f.reopen()).toMatchObject({ ok: true, reopened: false });
+    expect(git(f.cwd, "rev-parse", "HEAD")).toBe(continued);
+    expect((await f.row()).metadata?.gitCacheRestoration).toMatchObject({ commit: f.commitA, restoredAt: expect.any(String) });
+    git(f.baseCwd, "worktree", "remove", f.cwd);
+    await db.update(executionWorkspaces).set({ status: "archived" }).where(eq(executionWorkspaces.id, f.workspaceId));
+    expect(await f.reopen()).toMatchObject({ ok: true, reopened: true });
+    expect(git(f.cwd, "rev-parse", "HEAD")).toBe(continued);
+  });
+
+  it("expires again after a reopened task consumes its fence and finishes", async () => {
+    const f = await fixture();
+    expect((await f.expire()).expired).toBe(true);
+    expect(await f.reopen()).toMatchObject({ ok: true, reopened: true });
+    await db.update(issues).set({ status: "in_progress", completedAt: null }).where(eq(issues.id, f.issueId));
+    const worker = executionWorkspaceService(db, { gitCacheExpirationEnabled: true, gitBaseExpirationEnabled: true });
+    await worker.sweepTerminalWorkspaces();
+    expect((await f.row()).metadata?.reopenPendingConsumption).not.toBe(true);
+    await fs.writeFile(path.join(f.cwd, "continued.txt"), "second task cycle\n");
+    git(f.cwd, "add", "."); git(f.cwd, "commit", "-m", "Second task cycle");
+    git(f.cwd, "push", "origin", "HEAD:refs/heads/continued");
+    const continued = git(f.cwd, "rev-parse", "HEAD");
+    await db.update(issues).set({ status: "done", completedAt: new Date(Date.now() - 48 * 3_600_000) }).where(eq(issues.id, f.issueId));
+    expect((await worker.sweepTerminalWorkspaces()).archived).toBe(1);
+    expect((await f.row()).metadata?.gitCacheRestoration).toMatchObject({ commit: continued });
+    expect(await f.reopen()).toMatchObject({ ok: true, reopened: true });
+    expect(git(f.cwd, "rev-parse", "HEAD")).toBe(continued);
+  });
+
+  it("prepares a restored workspace after adopting its renamed branch", async () => {
+    const f = await fixture();
+    await f.advance();
+    expect((await f.expire()).expired).toBe(true);
+    expect(await f.reopen()).toMatchObject({ ok: true });
+    git(f.cwd, "branch", "-m", "renamed-task");
+    await db.update(executionWorkspaces).set({ status: "idle" }).where(eq(executionWorkspaces.id, f.workspaceId));
+    const svc = executionWorkspaceService(db);
+    await svc.reconcileExecutionWorkspaceBranch(f.workspaceId, {
+      mode: "forward", actor: { actorType: "user", actorId: "local-board", agentId: null, runId: null },
+    });
+    expect((await f.row()).metadata?.gitCacheRestoration).toMatchObject({
+      branchName: "renamed-task", commit: f.commitA, restoredAt: expect.any(String),
+    });
+    await ensurePersistedExecutionWorkspaceAvailable({ db, base: f.base,
+      workspace: (await svc.getStoredById(f.companyId, f.workspaceId))!,
+      issue: { id: f.issueId, identifier: "GC-1", title: "Resumed task" },
+      agent: { id: null, name: "Fixture", companyId: f.companyId }, skipHostProvision: true });
+    expect(git(f.cwd, "branch", "--show-current")).toBe("renamed-task");
+    expect(git(f.cwd, "rev-parse", "HEAD")).toBe(f.commitA);
+  });
+
+  it("preserves worktrees whose Git administration parent is a symlink", async () => {
+    const f = await fixture();
+    const adminParent = path.join(f.baseCwd, ".git", "worktrees");
+    const external = path.join(root, `external-admin-${randomUUID()}`);
+    const admin = git(f.cwd, "rev-parse", "--absolute-git-dir");
+    await fs.writeFile(path.join(admin, "commondir"), `${path.join(f.baseCwd, ".git")}\n`);
+    await fs.rename(adminParent, external);
+    await fs.symlink(external, adminParent);
+    expect(git(f.cwd, "rev-parse", "HEAD")).toBe(f.commitA);
+    expect(await f.expire()).toMatchObject({ expired: false, skipReason: "unproven_worktree_admin" });
+    expect(await exists(f.cwd)).toBe(true);
+    expect(await fs.readdir(external)).not.toEqual([]);
+    expect((await f.row()).metadata?.gitCacheRestoration).toBeUndefined();
+  });
+
+  it.each(["dirty", "untracked", "ignored", "local_commit", "stash", "unknown_git_file"])("preserves %s unique state", async kind => {
+    const f = await fixture();
+    if (kind === "dirty") await fs.writeFile(path.join(f.cwd, "marker.txt"), "unique changes");
+    if (kind === "untracked") await fs.writeFile(path.join(f.cwd, "notes.txt"), "unique notes");
+    if (kind === "ignored") { await fs.mkdir(path.join(f.cwd, "private-notes")); await fs.writeFile(path.join(f.cwd, "private-notes", "note"), "unique ignored"); }
+    if (kind === "local_commit") { await fs.writeFile(path.join(f.cwd, "new.txt"), "unique commit"); git(f.cwd, "add", "."); git(f.cwd, "commit", "-m", "Unpushed"); }
+    if (kind === "stash") { await fs.writeFile(path.join(f.cwd, "marker.txt"), "stash"); git(f.cwd, "stash"); }
+    if (kind === "unknown_git_file") await fs.writeFile(path.join(git(f.cwd, "rev-parse", "--absolute-git-dir"), "unique-note"), "unique Git data");
+    expect((await f.expire()).expired).toBe(false);
+    expect(await exists(f.cwd)).toBe(true); expect(await exists(f.baseCwd)).toBe(true);
+  });
+
+  it("preserves live descendants and pending reopening", async () => {
+    const f = await fixture();
+    await db.insert(issues).values({ companyId: f.companyId, title: "Active child", status: "in_progress", parentId: f.issueId });
+    expect((await f.expire()).expired).toBe(false);
+    await db.delete(issues).where(eq(issues.parentId, f.issueId));
+    const row = await f.row();
+    await db.update(executionWorkspaces).set({ metadata: { ...row.metadata, reopenPendingConsumption: true } }).where(eq(executionWorkspaces.id, f.workspaceId));
+    expect((await f.expire()).expired).toBe(false);
+    expect(await exists(f.cwd)).toBe(true);
+  });
+
+  it("retries after interruption with the restoration recipe already committed", async () => {
+    const f = await fixture();
+    expect((await f.expire({ beforeDelete: async () => { throw new Error("interrupted_fixture"); } })).expired).toBe(false);
+    expect((await f.row()).metadata?.gitCacheRestoration).toMatchObject({ commit: f.commitA });
+    expect(await exists(f.cwd)).toBe(true);
+    expect((await f.expire()).expired).toBe(true);
+    expect(await f.reopen()).toMatchObject({ ok: true });
+    expect(git(f.cwd, "rev-parse", "HEAD")).toBe(f.commitA);
+  });
+
+  it("fails restoration explicitly on upstream failure and rejects a different company", async () => {
+    const f = await fixture();
+    expect((await f.expire()).baseExpired).toBe(true);
+    await fs.rename(f.remote, `${f.remote}.offline`);
+    expect(await f.reopen()).toMatchObject({ ok: false, code: "rebuild_failed", message: "The saved workspace commit could not be restored. Check repository access and retry." });
+    expect(await f.reopen(randomUUID())).toMatchObject({ ok: false, code: "not_reopenable" });
+    expect(await exists(f.cwd)).toBe(false);
+    await fs.rename(`${f.remote}.offline`, f.remote);
+    expect(await f.reopen()).toMatchObject({ ok: true });
+    expect(git(f.cwd, "rev-parse", "HEAD")).toBe(f.commitA);
+  });
+  it("serializes concurrent reopening after deletion and publishes one transition", async () => {
+    const f = await fixture();
+    let release!: () => void, entered!: () => void;
+    const paused = new Promise<void>(resolve => { release = resolve; });
+    const ready = new Promise<void>(resolve => { entered = resolve; });
+    const expiration = f.expire({ beforeDelete: async () => { entered(); await paused; } });
+    await ready;
+    const first = f.reopen(), second = f.reopen();
+    release();
+    expect((await expiration).expired).toBe(true);
+    const results = await Promise.all([first, second]);
+    expect(results, JSON.stringify(results)).toEqual(expect.arrayContaining([expect.objectContaining({ ok: true, reopened: true }), expect.objectContaining({ ok: true, reopened: false })]));
+    expect(results.filter(result => result.ok && result.reopened)).toHaveLength(1);
+    expect(git(f.cwd, "rev-parse", "HEAD")).toBe(f.commitA);
+  });
+
+  it("recovers interrupted trash removal before reopening", async () => {
+    const f = await fixture();
+    const originalRemove = fs.rm.bind(fs);
+    const remove = vi.spyOn(fs, "rm").mockImplementation(async (target, options) => {
+      if (String(target).includes(".expired-git-worktree-")) throw new Error("interrupted_trash_fixture");
+      return originalRemove(target, options);
+    });
+    expect((await f.expire()).skipReason).toBe("interrupted_trash_fixture");
+    expect(await exists(f.cwd)).toBe(false);
+    expect((await f.row()).metadata?.gitCacheRestoration).toMatchObject({ commit: f.commitA });
+    remove.mockRestore();
+    expect(await f.reopen()).toMatchObject({ ok: true, reopened: true });
+    expect(git(f.cwd, "rev-parse", "HEAD")).toBe(f.commitA);
+  });
+
+  it("preserves scheduled runs, unresolved recovery, and live previews", async () => {
+    const f = await fixture();
+    const agentId = randomUUID(), runId = randomUUID();
+    await db.insert(agents).values({ id: agentId, companyId: f.companyId, name: "Fixture", role: "engineer", adapterType: "process" });
+    await db.insert(heartbeatRuns).values({ id: runId, companyId: f.companyId, agentId, nativeIssueId: f.issueId, status: "scheduled_retry" });
+    expect((await f.expire()).expired).toBe(false);
+    await db.update(heartbeatRuns).set({ status: "failed", finishedAt: new Date() }).where(eq(heartbeatRuns.id, runId));
+    await db.insert(nativeRunFinalizations).values({ runId, companyId: f.companyId, issueId: f.issueId, phase: "terminal_failure", recoveryState: "blocked" });
+    expect((await f.expire()).expired).toBe(false);
+    await db.delete(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, runId));
+    await db.insert(workspaceRuntimeServices).values({ id: randomUUID(), companyId: f.companyId, executionWorkspaceId: f.workspaceId,
+      scopeType: "execution_workspace", serviceName: "Preview", status: "running", lifecycle: "persistent", provider: "local" });
+    expect((await f.expire()).expired).toBe(false);
+    expect(await exists(f.cwd)).toBe(true);
+  });
+
+  it("skips deletion when authenticated upstream verification fails", async () => {
+    const f = await fixture();
+    await fs.rename(f.remote, `${f.remote}.offline`);
+    expect((await f.expire()).expired).toBe(false);
+    expect(await exists(f.cwd)).toBe(true);
+    expect((await f.row()).metadata?.gitCacheRestoration).toBeUndefined();
+  });
+
+  it("keeps an unused base containing custom Git files", async () => {
+    const f = await fixture();
+    const note = path.join(f.baseCwd, ".git", "private-note");
+    await fs.writeFile(note, "unique base metadata");
+    const outcome = await f.expire();
+    expect(outcome).toMatchObject({ expired: true, baseExpired: false, skipReason: "unknown_base_git_files" });
+    expect(await fs.readFile(note, "utf8")).toBe("unique base metadata");
+    expect(await f.reopen()).toMatchObject({ ok: true });
+  });
+
+  it("expires owned task worktrees while preserving their shared managed base", async () => {
+    const f = await fixture(true, true);
+    await f.advance();
+    expect(await f.expire()).toMatchObject({ expired: true, baseExpired: false, skipReason: "shared_base_preserved" });
+    expect(await exists(f.baseCwd)).toBe(true);
+    expect(await f.reopen()).toMatchObject({ ok: true });
+    expect(git(f.cwd, "rev-parse", "HEAD")).toBe(f.commitA);
+  });
+
+  it("expires an isolated base whose worktree was nested inside it", async () => {
+    const f = await fixture(false, true);
+    expect(await f.expire()).toMatchObject({ expired: true, baseExpired: true });
+    expect(await exists(f.baseCwd)).toBe(false);
+    expect(await f.reopen()).toMatchObject({ ok: true });
+    expect(git(f.cwd, "rev-parse", "HEAD")).toBe(f.commitA);
+  });
+
+});

server/src/__tests__/grok-bot-routine.test.ts

@@ -8,7 +8,7 @@ import { createHash, randomBytes, randomUUID } from "node:crypto";
 import { and, eq, sql } from "drizzle-orm";
 import { beforeAll, afterAll, afterEach, describe, expect, it, vi } from "vitest";
 import { agents, authUsers, companies, companyMemberships, createDb, dotAgentBindings, grokBotDispatches,
-  grokBotRoutines, mcpOauthGrants, activityLog, issues, heartbeatRuns, dotRunnerAssignments, dotRunnerOperations, dotMailboxItems, nativeRunFinalizations } from "@paperclipai/db";
+  grokBotRoutines, mcpOauthGrants, activityLog, issues, heartbeatRuns, dotRunnerAssignments, dotRunnerOperations, dotMailboxItems, nativeRunFinalizations, environmentLeases, environments, issueRecoveryActions, issueComments } from "@paperclipai/db";
 import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
 import { accessService } from "../services/access.js";
 import { instanceSettingsService } from "../services/instance-settings.js";
@@ -20,6 +20,12 @@ import * as secretModule from "../services/secrets.js";
 import * as externalAuthority from "../services/external-bot-authority.js";
 import { type GrokRoutineWake } from "../services/grok-bot/webhook.js";
 
+import { remoteTerminationReceipt } from "../services/remote-execution-termination.js";
+import { admitExplicitNativeContinuation } from "../services/explicit-native-continuation.js";
+import { hasRequiredWorkspaceRecovery } from "../services/workspace-restore-recovery-state.js";
+import { heartbeatService } from "../services/heartbeat.js";
+import { environmentRuntimeService } from "../services/environment-runtime.js";
+
 let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
 let db: ReturnType<typeof createDb>;
 beforeAll(async () => {
@@ -98,6 +104,63 @@ async function acceptance(f: Awaited<ReturnType<typeof fixture>>, work: typeof d
 }
 
 describe("durable Grok routine qualification", () => {
+  it("preflights current connection authority without treating database failures as readiness", async () => {
+    const f = await fixture();
+    const broker = dotRunnerBroker(db);
+    const canStart = () => broker.canStartNewWork(f.company.id, f.agent.id, f.pairing.bindingId);
+    expect(await canStart()).toBe(false);
+    const row = await verification(f);
+    await f.service.send(f.company.id, f.agent.id, row.id);
+    await f.service.acknowledge(f.principal, acknowledgement(row));
+    expect(await canStart()).toBe(true);
+    expect(await broker.canStartNewWork(randomUUID(), f.agent.id, f.pairing.bindingId)).toBe(false);
+    expect(await broker.canStartNewWork(f.company.id, f.agent.id, randomUUID())).toBe(false);
+    await db.update(grokBotRoutines).set({ recoveryRequiredAt: new Date(), recoveryReason: "work_session" })
+      .where(eq(grokBotRoutines.bindingId, f.pairing.bindingId));
+    expect(await canStart()).toBe(false);
+    await db.update(grokBotRoutines).set({ recoveryRequiredAt: null, recoveryReason: null })
+      .where(eq(grokBotRoutines.bindingId, f.pairing.bindingId));
+    await db.update(mcpOauthGrants).set({ revokedAt: new Date() }).where(eq(mcpOauthGrants.id, f.principal.grant.id));
+    expect(await canStart()).toBe(false);
+    const unavailable = new Error("synthetic database unavailable");
+    vi.spyOn(broker, "snapshot").mockRejectedValueOnce(unavailable);
+    await expect(canStart()).rejects.toBe(unavailable);
+  });
+
+  it("holds public heartbeat input before claiming a Runner while Grok recovery is required", async () => {
+    const f = await fixture();
+    await db.update(agents).set({ status: "idle" }).where(eq(agents.id, f.agent.id));
+    const row = await verification(f);
+    await f.service.send(f.company.id, f.agent.id, row.id);
+    await f.service.acknowledge(f.principal, acknowledgement(row));
+    await db.update(grokBotRoutines).set({ recoveryRequiredAt: new Date(), recoveryReason: "work_session" })
+      .where(eq(grokBotRoutines.bindingId, f.pairing.bindingId));
+    const [task] = await db.insert(issues).values({ companyId: f.company.id, title: "Wait for Grok inspection",
+      status: "todo", assigneeAgentId: f.agent.id, responsibleUserId: f.principal.grant.userId }).returning();
+    const runtime = environmentRuntimeService(db);
+    const acquire = vi.spyOn(runtime, "acquireRunLease").mockRejectedValue(new Error("Unexpected provider allocation"));
+    const heartbeat = heartbeatService(db, { environmentRuntime: runtime });
+    try {
+      await heartbeat.wakeup(f.agent.id, { source: "assignment", reason: "issue_assigned",
+        requestedByActorType: "user", requestedByActorId: f.principal.grant.userId,
+        payload: { issueId: task.id }, contextSnapshot: { issueId: task.id } });
+      await heartbeat.drainActiveRunExecutions();
+      for (let poll = 0; poll < 3; poll++) await heartbeat.resumeQueuedRuns();
+      await heartbeat.drainActiveRunExecutions();
+      const runs = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.agentId, f.agent.id));
+      expect(runs).toEqual([expect.objectContaining({ status: "queued", startedAt: null })]);
+      expect(await db.select().from(environmentLeases).where(eq(environmentLeases.companyId, f.company.id))).toEqual([]);
+      expect(await db.select().from(dotRunnerAssignments).where(eq(dotRunnerAssignments.agentId, f.agent.id))).toEqual([]);
+      expect(await db.select().from(grokBotDispatches).where(eq(grokBotDispatches.bindingId, f.pairing.bindingId)))
+        .toEqual([expect.objectContaining({ id: row.id, status: "acknowledged" })]);
+      expect(acquire).not.toHaveBeenCalled();
+      expect((await db.select().from(issues).where(eq(issues.id, task.id)))[0].status).toBe("todo");
+    } finally {
+      await db.update(agents).set({ status: "paused" }).where(eq(agents.id, f.agent.id));
+      await heartbeat.drainActiveRunExecutions();
+    }
+  });
+
   it("receives an authenticated Grok acknowledgement through the actual MCP transport without exposing personal or Dot event tools", async () => {
     const f = await fixture(); const row = await verification(f);
     await f.service.send(f.company.id, f.agent.id, row.id);
@@ -479,3 +542,140 @@ describe("durable Grok routine qualification", () => {
   });
 
 });
+
+// Exercise the public recovery service and the actual native admission boundary,
+// including a stale file obligation persisted before this fix.
+describe("Grok inspection of historical workspace-less Stop", () => {
+  async function historicalHold() {
+    const f = await fixture();
+    const { run, task, work } = await admittedWork(f, "accepted");
+    const finishedAt = new Date(Date.now() - 1000);
+    const fileHold = { workspaceRestoreFailure: "restore_failed", workspaceRestoreRecovery: {
+      schema: "paperclip.workspace-restore-recovery.v1", runId: run.id, leaseIds: [] as string[],
+    } };
+    const [environment] = await db.insert(environments).values({ name: `Grok ${run.id}`, driver: "sandbox" }).returning();
+    const [lease] = await db.insert(environmentLeases).values({ companyId: f.company.id, issueId: task.id,
+      heartbeatRunId: run.id, environmentId: environment!.id, provider: "daytona", providerLeaseId: randomUUID(),
+      status: "released", releasedAt: new Date(), cleanupStatus: "success" }).returning();
+    await db.update(environmentLeases).set({ metadata: { remoteExecutionTermination:
+      remoteTerminationReceipt(lease!, { providerLeaseId: lease!.providerLeaseId, state: "destroyed" }) } }).where(eq(environmentLeases.id, lease!.id));
+    fileHold.workspaceRestoreRecovery.leaseIds.push(lease!.id);
+    await db.update(heartbeatRuns).set({ status: "cancelled", driverKind: "grok_bot_mcp", finishedAt,
+      nativeIssueId: task.id, issueId: task.id, contextSnapshot: { issueId: task.id },
+      runnerProfileJson: { nativeExecutionInput: { schema: "paperclip.native-execution-input.v6",
+        binding: { runId: run.id, companyId: f.company.id, agentId: f.agent.id, issueId: task.id },
+        provider: { kind: "grok_bot" }, workspace: { access: "none", cwd: null } } },
+      resultJson: { ...fileHold, cancellation: { source: "user", reason: "Paperclip work stopped" } },
+    }).where(eq(heartbeatRuns.id, run.id));
+    await db.update(issues).set({ status: "blocked", executionRunId: null, checkoutRunId: null }).where(eq(issues.id, task.id));
+    await db.update(nativeRunFinalizations).set({ phase: "terminal_failure", leaseOwner: null,
+      leaseExpiresAt: null, recoveryState: "blocked" }).where(eq(nativeRunFinalizations.runId, run.id));
+    await db.update(dotRunnerAssignments).set({ status: "fenced" }).where(eq(dotRunnerAssignments.id, work.id));
+    const [action] = await db.insert(issueRecoveryActions).values({ companyId: f.company.id, sourceIssueId: task.id,
+      kind: "active_run_watchdog", cause: "native_continuation_requires_reconciliation", fingerprint: run.id,
+      status: "resolved", outcome: "blocked", nextAction: "Recover files.",
+      evidence: { ...fileHold, runId: run.id, automaticRecovery: { replay: "blocked", actionOutcome: "unknown" } },
+    }).returning();
+    const [comment] = await db.insert(issueComments).values({ companyId: f.company.id, issueId: task.id,
+      authorType: "user", authorUserId: f.principal.grant.userId, body: "Start a fresh task turn after inspection." }).returning();
+    const continueWork = (dryRun = true) => db.transaction(async tx => {
+      await tx.select().from(issues).where(eq(issues.id, task.id)).for("update");
+      return admitExplicitNativeContinuation({ db: tx as unknown as typeof db, companyId: f.company.id,
+        issueId: task.id, agentId: f.agent.id, actorType: "user", actorId: f.principal.grant.userId,
+        reason: "issue_commented", commentId: comment!.id, successorRunId: randomUUID(), dryRun });
+    });
+    return { ...f, run, task, work, lease: lease!, action: action!, continueWork };
+  }
+
+  it("offers inspection for an old task hold and admits only new explicit input after inspection and Runner cleanup", async () => {
+    const f = await historicalHold();
+    f.delivery.mockClear();
+    expect(await f.service.recoveryNeeded(f.company.id, f.agent.id)).toBe(true);
+    expect(await f.continueWork()).toBeNull();
+    expect(await f.service.recover(f.company.id, f.agent.id, f.principal.grant.userId)).toMatchObject({
+      requiresVerification: true, remoteTerminationConfirmed: false,
+    });
+    expect(await f.service.state(f.company.id, f.agent.id, f.pairing.bindingId)).toMatchObject({ verified: false, recoveryNeeded: false });
+    const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.run.id));
+    expect(hasRequiredWorkspaceRecovery(run!.resultJson)).toBe(false);
+    expect(run).toMatchObject({ status: "cancelled", resultJson: { externalWorkspaceInspection: {
+      inspectedBy: f.principal.grant.userId, remoteTerminationConfirmed: false,
+      previousWorkspaceRestoreFailure: "restore_failed", runnerLeaseIds: [f.lease.id],
+    } } });
+    const [action] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.action.id));
+    expect(action).toMatchObject({ status: "resolved", outcome: "blocked", evidence: {
+      automaticRecovery: { replay: "blocked", actionOutcome: "unknown" }, externalWorkspaceInspection: { workspaceAccess: "none" },
+    } });
+    expect(action!.evidence).not.toHaveProperty("workspaceRestoreRecovery");
+    expect(f.delivery).not.toHaveBeenCalled();
+    expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.company.id))).toHaveLength(1);
+    await expect(dotRunnerBroker(db).operation(f.principal, f.work.id, randomUUID(), "progress", { text: "late effect" })).rejects.toThrow("wake verification");
+    // Connection recovery still requires a real authenticated challenge exchange.
+    const row = await verification(f); await f.service.send(f.company.id, f.agent.id, row.id);
+    await f.service.acknowledge(f.principal, acknowledgement(row));
+    await expect(dotRunnerBroker(db).operation(f.principal, f.work.id, randomUUID(), "progress", { text: "late effect" })).rejects.toThrow("revoked");
+    expect(await f.continueWork(false)).toMatchObject({ previousRunId: f.run.id });
+    const [settled] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.action.id));
+    expect(settled!.evidence.automaticRecovery).toMatchObject({ replay: "explicit_user_continuation", actionOutcome: "unknown" });
+    expect((await db.select().from(dotRunnerAssignments).where(eq(dotRunnerAssignments.id, f.work.id)))[0]!.status).toBe("fenced");
+    await f.service.recover(f.company.id, f.agent.id, f.principal.grant.userId);
+    expect(await db.select().from(activityLog).where(and(eq(activityLog.companyId, f.company.id), eq(activityLog.action, "grok.workspace_obligation_inspected")))).toHaveLength(1);
+  });
+
+  it("does not retire task holds if another assignment prevents recovery", async () => {
+    const f = await historicalHold();
+    const [other] = await db.insert(heartbeatRuns).values({ companyId: f.company.id, agentId: f.agent.id,
+      status: "running", invocationSource: "assignment", triggerDetail: "system" }).returning();
+    await db.insert(dotRunnerAssignments).values({ ...f.work, id: randomUUID(), runId: other!.id,
+      normalizedSessionId: randomUUID(), turnId: randomUUID(), status: "accepted" });
+    await expect(f.service.recover(f.company.id, f.agent.id, f.principal.grant.userId)).rejects.toThrow("current Paperclip work session");
+    const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.run.id));
+    expect(hasRequiredWorkspaceRecovery(run!.resultJson)).toBe(true);
+    expect(await f.service.recoveryNeeded(f.company.id, f.agent.id)).toBe(true);
+  });
+
+  it("commits verification invalidation before waiting for historical task inspection", async () => {
+    const f = await historicalHold();
+    let recovery: Promise<unknown> | undefined;
+    try {
+      await db.transaction(async tx => {
+        await tx.select().from(issues).where(eq(issues.id, f.task.id)).for("update");
+        recovery = f.service.recover(f.company.id, f.agent.id, f.principal.grant.userId);
+        void recovery.catch(() => {});
+        await vi.waitFor(async () => {
+          expect(await f.service.state(f.company.id, f.agent.id, f.pairing.bindingId)).toMatchObject({ verified: false, recoveryNeeded: true });
+        });
+        await expect(verification(f)).rejects.toThrow("recover");
+        const [binding] = await db.select().from(dotAgentBindings).where(eq(dotAgentBindings.id, f.pairing.bindingId));
+        expect(binding).toMatchObject({ status: "connected", readyAt: null });
+        const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.run.id));
+        expect(hasRequiredWorkspaceRecovery(run!.resultJson)).toBe(true);
+      });
+    } finally { await recovery; }
+    expect(await f.service.state(f.company.id, f.agent.id, f.pairing.bindingId)).toMatchObject({ verified: false, recoveryNeeded: false });
+  });
+
+  it.each(["cleanup_pending", "receipt_mismatch", "controller_owned", "assignment_active", "running", "operator_revoked", "foreign_company"])("does not retire an old obligation when %s", async condition => {
+    const f = await historicalHold();
+    if (condition === "cleanup_pending") await db.update(environmentLeases).set({ cleanupStatus: "failed" }).where(eq(environmentLeases.id, f.lease.id));
+    if (condition === "receipt_mismatch") await db.update(environmentLeases).set({ metadata: { remoteExecutionTermination: { schema: "paperclip.remote-termination.v1", runId: randomUUID() } } }).where(eq(environmentLeases.id, f.lease.id));
+    if (condition === "controller_owned") await db.update(nativeRunFinalizations).set({ leaseOwner: "other-owner" }).where(eq(nativeRunFinalizations.runId, f.run.id));
+    if (condition === "assignment_active") await db.update(dotRunnerAssignments).set({ status: "accepted" }).where(eq(dotRunnerAssignments.id, f.work.id));
+    if (condition === "running") await db.update(heartbeatRuns).set({ status: "running" }).where(eq(heartbeatRuns.id, f.run.id));
+    if (condition === "operator_revoked") await accessService(db).setPrincipalPermission(f.company.id, "user", f.principal.grant.userId, "agents:create", false, f.principal.grant.userId);
+    if (condition === "foreign_company") {
+      const foreign = await fixture();
+      await f.service.recover(foreign.company.id, foreign.agent.id, foreign.principal.grant.userId);
+    } else if (condition === "running") {
+      // Not an eligible historical source. Its active assignment is fenced in
+      // this deliberately malformed fixture, but no file state may be removed.
+      await f.service.recover(f.company.id, f.agent.id, f.principal.grant.userId);
+    } else await expect(f.service.recover(f.company.id, f.agent.id, f.principal.grant.userId)).rejects.toThrow();
+    const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.run.id));
+    expect(hasRequiredWorkspaceRecovery(run!.resultJson)).toBe(true);
+    if (["cleanup_pending", "receipt_mismatch", "controller_owned"].includes(condition)) {
+      expect(await f.service.state(f.company.id, f.agent.id, f.pairing.bindingId)).toMatchObject({ verified: false, recoveryNeeded: true });
+    }
+    expect(await f.continueWork()).toBeNull();
+  });
+});

server/src/__tests__/heartbeat-process-recovery.test.ts

@@ -3521,7 +3521,7 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => {
     }
   }
 
-  it("retains the source when a saved native descriptor fails inside runtime selection", async () => {
+  it.each([false, true])("settles a failed saved native descriptor without inventing files when workspace access is disabled: %s", async withoutWorkspace => {
     await withTempPaperclipHome(async () => {
       const f = await seedQueuedIssueRunFixture();
       await db.update(agents).set({ adapterType: "paperclip_runner",
@@ -3547,7 +3547,12 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => {
               descriptorSha256: "a".repeat(64), baselineSha256: "b".repeat(64), finalHostSha256: null,
               workspaceId: randomUUID(), leaseId: lease!.id, providerLeaseId: lease!.providerLeaseId!, remoteCwd: "/work", resourceDisposition: "destroy" as const };
             await db.update(heartbeatRuns).set({ runtimeMode: "native", nativeIssueId: f.issueId, nativePhase: "observed",
-              runnerProfileJson: { nativeWorkspaceSync: reference },
+              driverKind: withoutWorkspace ? "grok_bot_mcp" : null,
+              runnerProfileJson: { nativeWorkspaceSync: reference, ...(withoutWorkspace ? { nativeExecutionInput: {
+                schema: "paperclip.native-execution-input.v6", provider: { kind: "grok_bot" },
+                binding: { companyId: f.companyId, agentId: f.agentId, runId, issueId: f.issueId },
+                workspace: { access: "none", cwd: null, repoUrl: null, repoRef: null, branchName: null },
+              } } : {}) },
             }).where(eq(heartbeatRuns.id, runId));
             await db.insert(nativeRunFinalizations).values({ runId, companyId: f.companyId, issueId: f.issueId, phase: "observed" });
             await nativeWorkspaceSyncInternals.readDescriptor({ companyId: f.companyId, runId, workspaceId: reference.workspaceId, reference });
@@ -3556,15 +3561,21 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => {
         await heartbeat.resumeQueuedRuns();
         await heartbeat.drainActiveRunExecutions();
         expect(leaseId).toBeDefined();
-        expect(await heartbeat.getRun(f.runId, { includeExecutionEvidence: true })).toMatchObject({ status: "failed",
-          runtimeMode: "native", nativePhase: "terminal_failure", resultJson: {
-            workspaceRestoreFailure: "restore_failed", workspaceRestoreRecovery: { leaseIds: [leaseId] },
-          },
-        });
-        expect(await db.select().from(environmentLeases).where(eq(environmentLeases.id, leaseId!)))
-          .toMatchObject([{ status: "pending_cleanup", leasePolicy: "retain_on_failure" }]);
-        expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toMatchObject({ workspaceRepairRequired: true, canRetry: false });
-        expect(release).toHaveBeenCalledWith(expect.objectContaining({ providerResourceDisposition: "stop_and_retain" }));
+        const failed = await heartbeat.getRun(f.runId, { includeExecutionEvidence: true });
+        expect(failed).toMatchObject({ status: "failed", runtimeMode: "native", nativePhase: "terminal_failure" });
+        if (withoutWorkspace) {
+          expect(failed?.resultJson?.workspaceRestoreFailure ?? null).toBeNull();
+          expect(failed?.resultJson?.workspaceRestoreRecovery ?? null).toBeNull();
+          expect((await getExecutionBlocker(db, f.companyId, f.issueId))?.workspaceRepairRequired).not.toBe(true);
+          expect(release).toHaveBeenCalled();
+          expect(release).not.toHaveBeenCalledWith(expect.objectContaining({ providerResourceDisposition: "stop_and_retain" }));
+        } else {
+          expect(failed?.resultJson).toMatchObject({ workspaceRestoreFailure: "restore_failed", workspaceRestoreRecovery: { leaseIds: [leaseId] } });
+          expect(await db.select().from(environmentLeases).where(eq(environmentLeases.id, leaseId!)))
+            .toMatchObject([{ status: "pending_cleanup", leasePolicy: "retain_on_failure" }]);
+          expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toMatchObject({ workspaceRepairRequired: true, canRetry: false });
+          expect(release).toHaveBeenCalledWith(expect.objectContaining({ providerResourceDisposition: "stop_and_retain" }));
+        }
         expect(provider).not.toHaveBeenCalled();
         expect(mockAdapterExecute).not.toHaveBeenCalled();
       } finally { factory.mockRestore(); }

server/src/__tests__/helpers/docker-command-runner.ts

@@ -0,0 +1,52 @@
+import { spawn } from "node:child_process";
+import { readFile } from "node:fs/promises";
+import { createConnection, type Socket } from "node:net";
+import { createServer } from "node:tls";
+import type { CommandManagedRuntimeRunner } from "@paperclipai/adapter-utils/command-managed-runtime";
+
+export function dockerCommandRunner(container: string): CommandManagedRuntimeRunner {
+  return {
+    async execute(input) {
+      const args = ["exec", "-i", "-u", "root"];
+      if (input.cwd) args.push("-w", input.cwd);
+      for (const [name, value] of Object.entries(input.env ?? {})) args.push("-e", `${name}=${value}`);
+      args.push(container, input.command, ...(input.args ?? []));
+      const child = spawn("docker", args, { stdio: ["pipe", "pipe", "pipe"] });
+      const startedAt = new Date().toISOString();
+      let stdout = "", stderr = "", timedOut = false;
+      child.stdout.setEncoding("utf8").on("data", chunk => { stdout += chunk; });
+      child.stderr.setEncoding("utf8").on("data", chunk => { stderr += chunk; });
+      const timer = setTimeout(() => { timedOut = true; child.kill("SIGKILL"); }, input.timeoutMs ?? 30_000);
+      child.stdin.on("error", () => undefined);
+      child.stdin.end(input.stdin);
+      try {
+        return await new Promise<Awaited<ReturnType<CommandManagedRuntimeRunner["execute"]>>>((resolve, reject) => {
+          child.on("error", reject);
+          child.on("close", (exitCode, signal) => resolve({ exitCode, signal, timedOut, stdout, stderr, pid: child.pid ?? null, startedAt }));
+        });
+      } finally {
+        clearTimeout(timer);
+      }
+    },
+  };
+}
+
+export async function createRunnerTestTlsTunnel(input: { key: string; cert: string; port: number }) {
+  const sockets = new Set<Socket>();
+  const server = createServer({ key: await readFile(input.key), cert: await readFile(input.cert) }, socket => {
+    const upstream = createConnection({ host: "127.0.0.1", port: input.port });
+    for (const connection of [socket, upstream]) {
+      sockets.add(connection);
+      connection.on("close", () => sockets.delete(connection));
+      connection.on("error", () => { socket.destroy(); upstream.destroy(); });
+    }
+    socket.pipe(upstream).pipe(socket);
+  });
+  await new Promise<void>(resolve => server.listen(0, "127.0.0.1", resolve));
+  const address = server.address();
+  if (!address || typeof address === "string") throw new Error("Test TLS listener unavailable");
+  return { port: address.port, close: async () => {
+    for (const socket of sockets) socket.destroy();
+    await new Promise<void>(resolve => server.close(() => resolve()));
+  } };
+}

server/src/__tests__/helpers/remote-warm-controller.test-worker.ts

@@ -0,0 +1,36 @@
+import { createDb } from "@paperclipai/db";
+import { executePaperclipNativeSession } from "../../services/native-runtime/native-session-executor.js";
+import { dockerCommandRunner } from "./docker-command-runner.js";
+
+process.once("message", async (input: {
+  connectionString: string;
+  container: string;
+  ingressUrl: string;
+  options: Parameters<typeof executePaperclipNativeSession>[0];
+}) => {
+  const target = input.options.runnerExecutionTarget;
+  if (target?.kind !== "remote") throw new Error("Expected remote worker target");
+  try {
+    const result = await executePaperclipNativeSession({
+      ...input.options,
+      db: createDb(input.connectionString),
+      runnerExecutionTarget: {
+        ...target,
+        runner: dockerCommandRunner(input.container),
+        getRunnerIngressEndpoint: async ({ path }) => {
+          const endpoint = {
+            kind: "authenticated_websocket" as const,
+            websocketUrl: `${input.ingressUrl}${path}`, secretHeaders: [], generation: input.container,
+            refresh: async () => endpoint, close: async () => undefined,
+          };
+          return endpoint;
+        },
+      },
+      onLog: async (stream, text) => { process.send?.({ log: { stream, text } }); },
+    });
+    process.send?.({ result });
+  } catch (error) {
+    process.send?.({ error: error instanceof Error ? error.stack : String(error) });
+  }
+});
+setInterval(() => undefined, 60_000);

server/src/__tests__/redaction.test.ts

@@ -380,6 +380,10 @@ describe("redaction", () => {
       "runner.session.bootstrap",
       "runner.turn.submit",
       "runner.session.startup",
+      "provider_pack.select",
+      "provider_pack.discover",
+      "provider_pack.verify_uploaded",
+      "provider_pack.upload",
       "provider.turn.queue",
       "question_response.to_run_created",
       "native.session.execute",

server/src/__tests__/repository-root-reader-queries.test.ts

@@ -1,5 +1,6 @@
 import { createHash, randomUUID } from "node:crypto";
 import { mkdtemp, realpath, rm } from "node:fs/promises";
+import fs from "node:fs/promises";
 import { tmpdir } from "node:os";
 import path from "node:path";
 import { and, eq, inArray } from "drizzle-orm";
@@ -113,3 +114,24 @@ it("keeps supplied membership snapshots separate from fresh database snapshots",
     await db.update(companyMemberships).set({ status: "active" }).where(and(eq(companyMemberships.companyId, companyId), eq(companyMemberships.principalId, userId)));
   }
 });
+
+it("batches source retention and duplicate path reads without losing owners", async () => {
+  const sourceIds = Array.from({ length: 501 }, () => randomUUID());
+  await db.insert(executionWorkspaces).values(sourceIds.map(id => ({ id, companyId, projectId, cwd,
+    sourceIssueId: issueId, name: "Shared source", mode: "shared_workspace", strategyType: "operator_directory" })));
+  const queries: string[] = [];
+  const recorded = drizzle(db.$client, { schema: db._.fullSchema, logger: { logQuery: query => queries.push(query) } }) as unknown as typeof db;
+  const paths = vi.spyOn(fs, "realpath");
+  try {
+    await executionWorkspaceRepositoryService(recorded).guardForPublication(input());
+    expect(queries.filter(query => query.startsWith('insert into "execution_workspace_repository_access"')).length).toBeLessThanOrEqual(2);
+    expect(paths.mock.calls.length).toBeLessThanOrEqual(8);
+  } finally {
+    paths.mockRestore();
+  }
+  const retained = await db.select({ scopeId: executionWorkspaceRepositoryAccess.scopeId }).from(executionWorkspaceRepositoryAccess)
+    .where(and(eq(executionWorkspaceRepositoryAccess.companyId, companyId),
+      eq(executionWorkspaceRepositoryAccess.rootKey, createHash("sha256").update(cwd).digest("hex")),
+      eq(executionWorkspaceRepositoryAccess.kind, "source")));
+  expect(new Set(retained.map(row => row.scopeId))).toEqual(new Set([workspaceId, ...sourceIds]));
+});

server/src/__tests__/setup-supertest.ts

@@ -29,6 +29,8 @@ process.env.PAPERCLIP_TEST_HOST_HOME ??= process.env.PAPERCLIP_HOME?.trim() || p
 // Receipt-spool recovery must never scan a developer's instance during tests.
 const { home: paperclipTestHome, cleanup: cleanupPaperclipTestHome } = createIsolatedTestHome();
 process.env.PAPERCLIP_HOME = paperclipTestHome;
+process.env.PAPERCLIP_CONFIG = path.join(paperclipTestHome, "config.json");
+process.env.PAPERCLIP_INSTANCE_ID = `vt-${process.pid}`;
 // Setup-file afterAll hooks run before suites drain asynchronous heartbeats.
 // Cleanup at worker exit, after all suite teardown, avoids deleting live state.
 process.once("exit", cleanupPaperclipTestHome);

server/src/__tests__/skill-version-cache.test.ts

@@ -0,0 +1,267 @@
+import { randomUUID } from "node:crypto";
+import { promises as fs } from "node:fs";
+import os from "node:os";
+import path from "node:path";
+import { eq } from "drizzle-orm";
+import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
+import { agents, heartbeatRuns, issues, nativeRunFinalizations, companies, companySkills, companySkillVersions, companySkillRepositorySnapshots, createDb } from "@paperclipai/db";
+import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
+import { companySkillService } from "../services/company-skills.js";
+import { skillVersionCachePaths, touchSkillVersionCache, withSkillVersionCacheLock } from "../services/skill-version-cache.js";
+
+import { skillVersionRetentionService } from "../services/skill-version-retention.js";
+async function sweepSkillVersionCache(db: ReturnType<typeof createDb>, options: { companyId: string; now?: number; dryRun?: boolean }) {
+  const result = await skillVersionRetentionService(db).pruneCompany(options.companyId, { dryRun: options.dryRun, now: options.now === undefined ? undefined : new Date(options.now) });
+  return { ...result, evicted: result.removed };
+}
+
+const support = await getEmbeddedPostgresTestSupport();
+const describePostgres = support.supported ? describe : describe.skip;
+if (!support.supported) console.warn(`Skill cache tests require Postgres: ${support.reason}`);
+
+describePostgres("immutable skill version cache", () => {
+  let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
+  let db: ReturnType<typeof createDb>;
+  let home: string;
+  const previousHome = process.env.PAPERCLIP_HOME;
+
+  beforeAll(async () => {
+    database = await startEmbeddedPostgresTestDatabase("skill-version-cache-");
+    db = createDb(database.connectionString);
+    home = await fs.mkdtemp(path.join(os.tmpdir(), "skill-version-cache-"));
+    process.env.PAPERCLIP_HOME = home;
+  });
+  afterEach(() => { vi.unstubAllGlobals(); vi.restoreAllMocks(); });
+  afterAll(async () => {
+    if (previousHome === undefined) delete process.env.PAPERCLIP_HOME;
+    else process.env.PAPERCLIP_HOME = previousHome;
+    if (home) await fs.rm(home, { recursive: true, force: true });
+    await database?.cleanup();
+  });
+
+  async function fixture() {
+    const companyId = randomUUID();
+    const skillId = randomUUID();
+    const key = `company/${companyId}/saved-repository`;
+    await db.insert(companies).values({ id: companyId, name: "Skill cache", issuePrefix: `S${companyId.slice(0, 6)}` });
+    await db.insert(companySkills).values({ id: skillId, companyId, key, slug: "saved-repository", name: "Saved repository",
+      markdown: "# Saved skill", sourceType: "github", sourceLocator: "https://github.com/example/unavailable",
+      metadata: { skillSourceId: randomUUID(), snapshotHash: "saved" } });
+    const versions = [];
+    for (const revision of [1, 2]) {
+      const [snapshot] = await db.insert(companySkillRepositorySnapshots).values({ companyId, contentHash: `revision-${revision}`,
+        fileInventory: [
+          { path: "skills/example/SKILL.md", kind: "skill", content: `---\nname: saved-repository\ndescription: Saved revision ${revision}\n---\nRun ../../scripts/example.sh.\n` },
+          { path: "scripts/example.sh", kind: "script", content: `#!/bin/sh\necho revision-${revision}\n`, executable: true },
+          { path: "marker.bin", kind: "reference", content: Buffer.from([0, revision, 255]).toString("base64"), encoding: "base64" },
+        ] }).returning();
+      const [version] = await db.insert(companySkillVersions).values({ companyId, companySkillId: skillId, revisionNumber: revision,
+        repositorySnapshotId: snapshot!.id, repositorySkillPath: "skills/example/SKILL.md" }).returning();
+      versions.push(version!);
+    }
+    await db.update(companySkills).set({ currentVersionId: versions[1]!.id }).where(eq(companySkills.id, skillId));
+    const service = companySkillService(db);
+    async function resolve(versionId: string) {
+      const entries = await service.listRuntimeSkillEntries(companyId, { onlyKeys: new Set([key]), versionSelections: new Map([[key, versionId]]) });
+      expect(entries).toHaveLength(1);
+      expect(entries[0]!.sourceStatus, entries[0]!.missingDetail ?? undefined).toBe("available");
+      return entries[0]!;
+    }
+    return { companyId, skillId, key, versions, resolve };
+  }
+
+  it("rebuilds an old repository-backed version from database bytes with upstream unavailable", async () => {
+    const upstream = vi.fn(() => Promise.reject(new Error("Upstream is unavailable")));
+    vi.stubGlobal("fetch", upstream);
+    const { versions, resolve } = await fixture();
+    const old = await resolve(versions[0]!.id);
+    const current = await resolve(versions[1]!.id);
+    await fs.rm(old.source, { recursive: true });
+    const restored = await resolve(versions[0]!.id);
+    expect(restored.source).toBe(old.source);
+    const repository = path.join(restored.source, ".paperclip-repository");
+    expect(await fs.readFile(path.join(repository, "marker.bin"))).toEqual(Buffer.from([0, 1, 255]));
+    expect(await fs.readFile(path.join(repository, "scripts/example.sh"), "utf8")).toBe("#!/bin/sh\necho revision-1\n");
+    expect((await fs.stat(path.join(repository, "scripts/example.sh"))).mode & 0o111).toBe(0o111);
+    expect(await fs.readFile(path.join(current.source, ".paperclip-repository/marker.bin"))).toEqual(Buffer.from([0, 2, 255]));
+    expect(upstream).not.toHaveBeenCalled();
+  });
+
+  it("expires only idle old versions, reports dry-run candidates, and restores the evicted version", async () => {
+    const f = await fixture();
+    const old = await f.resolve(f.versions[0]!.id);
+    const current = await f.resolve(f.versions[1]!.id);
+    const aged = Date.now() - 2 * 60 * 60 * 1000;
+    await touchSkillVersionCache(f.companyId, f.skillId, f.versions[0]!.id, aged);
+    await touchSkillVersionCache(f.companyId, f.skillId, f.versions[1]!.id, aged);
+    const dry = await sweepSkillVersionCache(db, { companyId: f.companyId, dryRun: true });
+    expect(dry).toMatchObject({ candidates: 1, evicted: 0, reclaimedBytes: 0 });
+    expect(await fs.stat(old.source)).toBeTruthy();
+    const swept = await sweepSkillVersionCache(db, { companyId: f.companyId });
+    expect(swept).toMatchObject({ candidates: 1, evicted: 1 });
+    expect(swept.reclaimedBytes).toBeGreaterThan(0);
+    await expect(fs.stat(old.source)).rejects.toMatchObject({ code: "ENOENT" });
+    expect(await fs.stat(current.source)).toBeTruthy();
+    vi.stubGlobal("fetch", vi.fn(() => Promise.reject(new Error("Upstream unavailable"))));
+    expect(await f.resolve(f.versions[0]!.id)).toMatchObject({ source: old.source });
+    expect((await sweepSkillVersionCache(db, { companyId: f.companyId })).evicted).toBe(0);
+  });
+
+  it("gives existing unobserved caches a full grace period", async () => {
+    const f = await fixture();
+    await f.resolve(f.versions[0]!.id);
+    const paths = skillVersionCachePaths(f.companyId, f.skillId, f.versions[0]!.id);
+    await fs.unlink(paths.access);
+    const now = Date.now();
+    expect((await sweepSkillVersionCache(db, { companyId: f.companyId, now })).skipped.observation_grace).toBe(1);
+    expect((await sweepSkillVersionCache(db, { companyId: f.companyId, now: now + 59 * 60 * 1000 })).evicted).toBe(0);
+    expect((await sweepSkillVersionCache(db, { companyId: f.companyId, now: now + 61 * 60 * 1000 })).evicted).toBe(1);
+  });
+
+  async function activeRun(companyId: string, contextSnapshot: Record<string, unknown> | null) {
+    const [agent] = await db.insert(agents).values({ companyId, name: "Cache reader", role: "engineer", adapterType: "process" }).returning();
+    const [run] = await db.insert(heartbeatRuns).values({ companyId, agentId: agent!.id, status: "running", contextSnapshot }).returning();
+    return run!;
+  }
+
+  it("retains pinned versions and skips companies with unrecorded active selections", async () => {
+    const f = await fixture();
+    await f.resolve(f.versions[0]!.id);
+    await touchSkillVersionCache(f.companyId, f.skillId, f.versions[0]!.id, Date.now() - 7_200_000);
+    const run = await activeRun(f.companyId, { skillVersions: { [f.key]: f.versions[0]!.id } });
+    expect((await sweepSkillVersionCache(db, { companyId: f.companyId })).skipped.protected_version).toBe(1);
+    await db.update(heartbeatRuns).set({ contextSnapshot: null }).where(eq(heartbeatRuns.id, run.id));
+    expect((await sweepSkillVersionCache(db, { companyId: f.companyId })).skipped.unknown_run_pins).toBe(1);
+    await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date() }).where(eq(heartbeatRuns.id, run.id));
+    expect((await sweepSkillVersionCache(db, { companyId: f.companyId })).evicted).toBe(1);
+  });
+
+  it("serializes eviction behind preparation and durable pinning across database clients", async () => {
+    const f = await fixture();
+    await f.resolve(f.versions[0]!.id);
+    const run = await activeRun(f.companyId, null);
+    const ready = Promise.withResolvers<void>();
+    const release = Promise.withResolvers<void>();
+    const preparing = withSkillVersionCacheLock(db, f.companyId, async () => {
+      await f.resolve(f.versions[0]!.id);
+      await touchSkillVersionCache(f.companyId, f.skillId, f.versions[0]!.id, Date.now() - 7_200_000);
+      ready.resolve();
+      await release.promise;
+      await db.update(heartbeatRuns).set({ contextSnapshot: { skillVersions: { [f.key]: f.versions[0]!.id } } })
+        .where(eq(heartbeatRuns.id, run.id));
+    });
+    await ready.promise;
+    const other = createDb(database.connectionString);
+    try {
+      const cleaning = await sweepSkillVersionCache(other, { companyId: f.companyId });
+      expect(cleaning.evicted).toBe(0);
+      expect(cleaning.skipped.cache_busy).toBeGreaterThan(0);
+    } finally { release.resolve(); }
+    await preparing;
+    expect(await sweepSkillVersionCache(other, { companyId: f.companyId })).toMatchObject({ evicted: 0, skipped: { protected_version: 1 } });
+  });
+
+  it("reads company-wide protection twice per batch regardless of version count", async () => {
+    const f = await fixture();
+    const versionIds = [f.versions[0]!.id];
+    for (let revision = 3; revision < 8; revision++) {
+      const [version] = await db.insert(companySkillVersions).values({ companyId: f.companyId,
+        companySkillId: f.skillId, revisionNumber: revision, repositorySnapshotId: f.versions[0]!.repositorySnapshotId,
+        repositorySkillPath: "skills/example/SKILL.md" }).returning();
+      versionIds.push(version!.id);
+    }
+    for (const id of [...versionIds, f.versions[1]!.id]) {
+      await f.resolve(id);
+      await touchSkillVersionCache(f.companyId, f.skillId, id, Date.now() - 7_200_000);
+    }
+    const statements: string[] = [];
+    const debug = db.$client.options.debug;
+    db.$client.options.debug = (_connection, query) => { statements.push(query); };
+    try {
+      expect(await sweepSkillVersionCache(db, { companyId: f.companyId })).toMatchObject({ evicted: 6 });
+      expect(statements.filter(query => /from "agents"/.test(query))).toHaveLength(2);
+      expect(statements.filter(query => /from "heartbeat_runs"/.test(query))).toHaveLength(2);
+    } finally { db.$client.options.debug = debug; }
+  });
+
+  it("lets preparation finish during slow byte inspection and rechecks its new pin", async () => {
+    const f = await fixture();
+    const old = await f.resolve(f.versions[0]!.id);
+    await touchSkillVersionCache(f.companyId, f.skillId, f.versions[0]!.id, Date.now() - 7_200_000);
+    const inspecting = Promise.withResolvers<void>();
+    const release = Promise.withResolvers<void>();
+    const lstat = fs.lstat.bind(fs);
+    let held = false;
+    vi.spyOn(fs, "lstat").mockImplementation((async (...args: Parameters<typeof fs.lstat>) => {
+      if (!held && String(args[0]) === path.join(old.source, ".paperclip-repository", "marker.bin")) {
+        held = true;
+        inspecting.resolve();
+        await release.promise;
+      }
+      return lstat(...args);
+    }) as typeof fs.lstat);
+    const sweep = sweepSkillVersionCache(db, { companyId: f.companyId });
+    await inspecting.promise;
+    const other = createDb(database.connectionString);
+    let timeout: ReturnType<typeof setTimeout> | undefined;
+    const preparation = withSkillVersionCacheLock(other, f.companyId, async () => {
+      await companySkillService(other).listRuntimeSkillEntries(f.companyId, {
+        onlyKeys: new Set([f.key]), versionSelections: new Map([[f.key, f.versions[0]!.id]]),
+      });
+      // Remove grace as an alternative explanation: the newly persisted run pin
+      // must prevent eviction after the original candidate snapshot was read.
+      await touchSkillVersionCache(f.companyId, f.skillId, f.versions[0]!.id, Date.now() - 7_200_000);
+      await activeRun(f.companyId, { skillVersions: { [f.key]: f.versions[0]!.id } });
+      return true;
+    });
+    try {
+      expect(await Promise.race([preparation, new Promise<boolean>(resolve => {
+        timeout = setTimeout(() => resolve(false), 10_000);
+      })])).toBe(true);
+    } finally { clearTimeout(timeout); release.resolve(); }
+    await preparation;
+    expect(await sweep).toMatchObject({ evicted: 0, skipped: { protected_version: 1 } });
+    expect(await fs.stat(old.source)).toBeTruthy();
+    await other.$client.end({ timeout: 1 });
+  });
+
+  it("keeps terminal run pins until native finalization has settled", async () => {
+    const f = await fixture();
+    await f.resolve(f.versions[0]!.id);
+    await touchSkillVersionCache(f.companyId, f.skillId, f.versions[0]!.id, Date.now() - 7_200_000);
+    const run = await activeRun(f.companyId, { skillVersions: { [f.key]: f.versions[0]!.id } });
+    const [issue] = await db.insert(issues).values({ companyId: f.companyId, title: "Pending recovery" }).returning();
+    await db.update(heartbeatRuns).set({ runtimeMode: "native", nativeIssueId: issue!.id, status: "failed", finishedAt: new Date() })
+      .where(eq(heartbeatRuns.id, run.id));
+    await db.insert(nativeRunFinalizations).values({ companyId: f.companyId, issueId: issue!.id, runId: run.id, phase: "terminal_failure" });
+    expect((await sweepSkillVersionCache(db, { companyId: f.companyId })).skipped.protected_version).toBe(1);
+    await db.update(nativeRunFinalizations).set({ phase: "committed" }).where(eq(nativeRunFinalizations.runId, run.id));
+    expect((await sweepSkillVersionCache(db, { companyId: f.companyId })).evicted).toBe(1);
+  });
+
+  it("finishes interrupted trash removal without deleting a rehydrated copy", async () => {
+    const f = await fixture();
+    const old = await f.resolve(f.versions[0]!.id);
+    const paths = skillVersionCachePaths(f.companyId, f.skillId, f.versions[0]!.id);
+    const trash = path.join(path.dirname(paths.directory), `.trash-${randomUUID()}`);
+    await fs.rename(old.source, trash);
+    await f.resolve(f.versions[0]!.id);
+    const result = await sweepSkillVersionCache(db, { companyId: f.companyId });
+    expect(result.evicted).toBe(0);
+    expect(result.reclaimedBytes).toBeGreaterThan(0);
+    await expect(fs.stat(trash)).rejects.toMatchObject({ code: "ENOENT" });
+    expect(await fs.stat(old.source)).toBeTruthy();
+  });
+
+  it("does not traverse symlinked cache roots", async () => {
+    const f = await fixture();
+    const old = await f.resolve(f.versions[0]!.id);
+    await touchSkillVersionCache(f.companyId, f.skillId, f.versions[0]!.id, Date.now() - 7_200_000);
+    const outside = await fs.mkdtemp(path.join(home, "outside-"));
+    await fs.writeFile(path.join(outside, "unique.txt"), "keep");
+    await fs.rm(old.source, { recursive: true });
+    await fs.symlink(outside, old.source);
+    expect((await sweepSkillVersionCache(db, { companyId: f.companyId })).skipped.unsafe_or_unavailable).toBe(1);
+    expect(await fs.readFile(path.join(outside, "unique.txt"), "utf8")).toBe("keep");
+  });
+});

server/src/__tests__/skill-version-retention-config.test.ts

@@ -0,0 +1,26 @@
+import { afterEach, describe, expect, it, vi } from "vitest";
+import { loadConfig } from "../config.ts";
+
+describe("skill version retention config parsing", () => {
+  afterEach(() => {
+    vi.unstubAllEnvs();
+  });
+
+  it("uses the defaults when the variables are not set", () => {
+    vi.stubEnv("PAPERCLIP_SKILL_VERSION_RETENTION_KEEP", undefined);
+    vi.stubEnv("PAPERCLIP_SKILL_VERSION_RETENTION_GRACE_MINUTES", undefined);
+    expect(loadConfig()).toMatchObject({ skillVersionRetentionKeepNewest: 0, skillVersionRetentionGraceMinutes: 60 });
+  });
+
+  it("accepts explicit non-negative values, including 0", () => {
+    vi.stubEnv("PAPERCLIP_SKILL_VERSION_RETENTION_KEEP", " 0 ");
+    vi.stubEnv("PAPERCLIP_SKILL_VERSION_RETENTION_GRACE_MINUTES", "30");
+    expect(loadConfig()).toMatchObject({ skillVersionRetentionKeepNewest: 0, skillVersionRetentionGraceMinutes: 30 });
+  });
+
+  it.each(["", "   ", "-1", "many"])("falls back to the defaults for %j", (value) => {
+    vi.stubEnv("PAPERCLIP_SKILL_VERSION_RETENTION_KEEP", value);
+    vi.stubEnv("PAPERCLIP_SKILL_VERSION_RETENTION_GRACE_MINUTES", value);
+    expect(loadConfig()).toMatchObject({ skillVersionRetentionKeepNewest: 0, skillVersionRetentionGraceMinutes: 60 });
+  });
+});

server/src/__tests__/skill-version-retention.test.ts

@@ -0,0 +1,370 @@
+import { touchSkillVersionCache } from "../services/skill-version-cache.js";
+import { randomUUID } from "node:crypto";
+import os from "node:os";
+import path from "node:path";
+import { promises as fs } from "node:fs";
+import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
+import { eq } from "drizzle-orm";
+import {
+  agents,
+  companies,
+  companySkills,
+  companySkillVersions,
+  createDb,
+  heartbeatRuns,
+} from "@paperclipai/db";
+import {
+  getEmbeddedPostgresTestSupport,
+  startEmbeddedPostgresTestDatabase,
+} from "./helpers/embedded-postgres.js";
+import { companySkillService } from "../services/company-skills.ts";
+import {
+  resolveCompanySkillVersionsRoot,
+  runVersionSnapshotFlight,
+  skillVersionRetentionService,
+} from "../services/skill-version-retention.ts";
+
+const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport();
+const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
+
+if (!embeddedPostgresSupport.supported) {
+  console.warn(
+    `Skipping embedded Postgres skill version retention tests on this host: ${embeddedPostgresSupport.reason ?? "unsupported environment"}`,
+  );
+}
+
+const HOUR_MS = 60 * 60 * 1000;
+
+async function exists(target: string) {
+  return fs.lstat(target).then(() => true, () => false);
+}
+
+async function makeWritable(target: string): Promise<void> {
+  const stat = await fs.lstat(target).catch(() => null);
+  if (!stat?.isDirectory()) return;
+  await fs.chmod(target, 0o755);
+  for (const entry of await fs.readdir(target)) await makeWritable(path.join(target, entry));
+}
+
+describeEmbeddedPostgres("skill version snapshot retention", () => {
+  let db!: ReturnType<typeof createDb>;
+  let tempDb: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | null = null;
+  let oldPaperclipHome: string | undefined;
+  let oldPaperclipInstanceId: string | undefined;
+  let paperclipHome: string | null = null;
+
+  beforeAll(async () => {
+    tempDb = await startEmbeddedPostgresTestDatabase("paperclip-skill-version-retention-");
+    oldPaperclipHome = process.env.PAPERCLIP_HOME;
+    oldPaperclipInstanceId = process.env.PAPERCLIP_INSTANCE_ID;
+    paperclipHome = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-skill-version-retention-home-"));
+    process.env.PAPERCLIP_HOME = paperclipHome;
+    process.env.PAPERCLIP_INSTANCE_ID = "default";
+    db = createDb(tempDb.connectionString);
+  }, 20_000);
+
+  afterEach(async () => {
+    vi.restoreAllMocks();
+    await db.delete(heartbeatRuns);
+    await db.delete(agents);
+    await db.delete(companySkills);
+    await db.delete(companies);
+    if (paperclipHome) {
+      const skillsRoot = path.join(paperclipHome, "instances", "default", "skills");
+      await makeWritable(skillsRoot);
+      await fs.rm(skillsRoot, { recursive: true, force: true });
+    }
+  });
+
+  afterAll(async () => {
+    if (oldPaperclipHome === undefined) delete process.env.PAPERCLIP_HOME;
+    else process.env.PAPERCLIP_HOME = oldPaperclipHome;
+    if (oldPaperclipInstanceId === undefined) delete process.env.PAPERCLIP_INSTANCE_ID;
+    else process.env.PAPERCLIP_INSTANCE_ID = oldPaperclipInstanceId;
+    if (paperclipHome) await fs.rm(paperclipHome, { recursive: true, force: true });
+    await tempDb?.cleanup();
+  });
+
+  async function createCompany() {
+    const companyId = randomUUID();
+    await db.insert(companies).values({ id: companyId, name: "Retention", issuePrefix: `R${companyId.slice(0, 6)}` });
+    return companyId;
+  }
+
+  async function createSkill(companyId: string, versionCount: number, currentIndex = versionCount - 1) {
+    const skillId = randomUUID();
+    const key = `company/${companyId}/retained-${skillId.slice(0, 8)}`;
+    await db.insert(companySkills).values({
+      id: skillId, companyId, key, slug: `retained-${skillId.slice(0, 8)}`, name: "Retained",
+      markdown: "# Retained", sourceType: "github", sourceLocator: "https://github.com/acme/retained",
+      sourceRef: "a".repeat(40), trustLevel: "markdown_only", compatibility: "compatible",
+      metadata: {}, fileInventory: [{ path: "SKILL.md", kind: "skill" }],
+    });
+    const versionIds: string[] = [];
+    for (let index = 0; index < versionCount; index++) {
+      const [version] = await db.insert(companySkillVersions).values({
+        companyId, companySkillId: skillId, revisionNumber: index + 1, label: `v${index + 1}`,
+        fileInventory: [
+          { path: "SKILL.md", kind: "skill", content: `---\nname: retained\n---\n\n# Revision ${index + 1}\n` },
+          { path: "references/notes.md", kind: "reference", content: `notes ${index + 1}` },
+        ],
+      }).returning({ id: companySkillVersions.id });
+      versionIds.push(version!.id);
+    }
+    if (versionCount > 0) {
+      await db.update(companySkills).set({ currentVersionId: versionIds[currentIndex]! })
+        .where(eq(companySkills.id, skillId));
+    }
+    return { skillId, key, versionIds };
+  }
+
+  /** Writes a read-only snapshot directory like materializeVersionSnapshot does, aged by `ageMs`. */
+  async function writeSnapshot(companyId: string, skillId: string, name: string, ageMs = 3 * HOUR_MS) {
+    const dir = path.join(resolveCompanySkillVersionsRoot(companyId), skillId, name);
+    await fs.mkdir(path.join(dir, "references"), { recursive: true });
+    await fs.writeFile(path.join(dir, "SKILL.md"), "# Snapshot", { mode: 0o444 });
+    await fs.writeFile(path.join(dir, "references", "notes.md"), "notes", { mode: 0o444 });
+    await fs.chmod(path.join(dir, "references"), 0o555);
+    const when = new Date(Date.now() - ageMs);
+    await fs.utimes(dir, when, when);
+    if (/^[a-f0-9-]{36}$/.test(name)) await touchSkillVersionCache(companyId, skillId, name, when.getTime());
+    return dir;
+  }
+
+  it("keeps current, pinned, live-run, newest, and recent versions and removes superseded ones", async () => {
+    const companyId = await createCompany();
+    const { skillId, key, versionIds: v } = await createSkill(companyId, 7);
+    const [agent] = await db.insert(agents).values({
+      companyId, name: "Pinned agent",
+      adapterConfig: { paperclipSkillSync: { desiredSkills: [{ key, versionId: v[0] }] } },
+    }).returning({ id: agents.id });
+    await db.insert(heartbeatRuns).values([
+      { companyId, agentId: agent!.id, status: "running", contextSnapshot: { skillVersions: { [key]: v[1] } } },
+      { companyId, agentId: agent!.id, status: "succeeded", finishedAt: new Date(), contextSnapshot: { skillVersions: { [key]: v[2] } } },
+    ]);
+    const dirs = await Promise.all(v.map((id, index) => writeSnapshot(companyId, skillId, id, index === 3 ? 0 : 3 * HOUR_MS)));
+
+    const result = await skillVersionRetentionService(db).pruneCompany(companyId, { keepNewest: 2 });
+
+    // v1 pinned, v2 live run, v4 recent, v6 + v7 newest (v7 current).
+    expect(await Promise.all(dirs.map(exists))).toEqual([true, true, false, true, false, true, true]);
+    expect(result).toMatchObject({ removed: 2, removedOrphans: 0, skippedRecent: 1, failed: 0, truncated: false });
+    // Rows are never deleted.
+    expect(await db.select().from(companySkillVersions).where(eq(companySkillVersions.companySkillId, skillId))).toHaveLength(7);
+  });
+
+  it("removes orphan version folders, orphan skill folders, and stale staging folders", async () => {
+    const companyId = await createCompany();
+    const { skillId, versionIds } = await createSkill(companyId, 1);
+    const root = resolveCompanySkillVersionsRoot(companyId);
+    const current = await writeSnapshot(companyId, skillId, versionIds[0]!);
+    const orphanVersion = await writeSnapshot(companyId, skillId, randomUUID());
+    const orphanSkillId = randomUUID();
+    const orphanSkillVersion = await writeSnapshot(companyId, orphanSkillId, randomUUID());
+    const staleStaging = await writeSnapshot(companyId, skillId, `.${versionIds[0]}-abc123`);
+    const freshStaging = await writeSnapshot(companyId, skillId, `.${versionIds[0]}-def456`, 0);
+    const leftoverTrash = await writeSnapshot(companyId, skillId, `\.trash-${randomUUID()}`, 0);
+    const unknown = await writeSnapshot(companyId, skillId, "operator-notes");
+    const hidden = await writeSnapshot(companyId, skillId, ".notes");
+
+    const result = await skillVersionRetentionService(db).pruneCompany(companyId);
+
+    expect(await exists(current)).toBe(true);
+    expect(await exists(orphanVersion)).toBe(false);
+    expect(await exists(orphanSkillVersion)).toBe(false);
+    expect(await fs.readdir(path.join(root, orphanSkillId))).toEqual([]);
+    expect(await exists(staleStaging)).toBe(false);
+    expect(await exists(freshStaging)).toBe(true);
+    expect(await exists(leftoverTrash)).toBe(false);
+    expect(await exists(unknown)).toBe(true);
+    expect(await exists(hidden)).toBe(true);
+    expect(result).toMatchObject({ removedOrphans: 2, removedStaging: 1, failed: 0 });
+    expect((await fs.readdir(path.join(root, skillId))).filter(name => name.startsWith(".trash-"))).toEqual([]);
+  });
+
+  it("re-materializes a pruned version from its row on the next resolve", async () => {
+    const companyId = await createCompany();
+    const { key, versionIds } = await createSkill(companyId, 5);
+    const svc = companySkillService(db);
+    const selections = new Map([[key, versionIds[0]!]]);
+    const first = (await svc.listRuntimeSkillEntries(companyId, { versionSelections: selections }))
+      .find(entry => entry.key === key)!;
+    expect(first.sourceStatus).toBe("available");
+    const old = new Date(Date.now() - 3 * HOUR_MS);
+    await touchSkillVersionCache(companyId, path.basename(path.dirname(first.source)), path.basename(first.source), old.getTime());
+
+    await skillVersionRetentionService(db).pruneCompany(companyId, { keepNewest: 1 });
+    expect(await exists(first.source)).toBe(false);
+
+    const again = (await svc.listRuntimeSkillEntries(companyId, { versionSelections: selections }))
+      .find(entry => entry.key === key)!;
+    expect(again).toMatchObject({ sourceStatus: "available", source: first.source });
+    expect(await fs.readFile(path.join(again.source, "SKILL.md"), "utf8")).toContain("# Revision 1");
+  });
+
+  it("refreshes the age of a reused snapshot so the grace window protects it", async () => {
+    const companyId = await createCompany();
+    const { key, versionIds } = await createSkill(companyId, 5);
+    const svc = companySkillService(db);
+    const selections = new Map([[key, versionIds[0]!]]);
+    const first = (await svc.listRuntimeSkillEntries(companyId, { versionSelections: selections }))
+      .find(entry => entry.key === key)!;
+    const old = new Date(Date.now() - 3 * HOUR_MS);
+    await touchSkillVersionCache(companyId, path.basename(path.dirname(first.source)), path.basename(first.source), old.getTime());
+    await svc.listRuntimeSkillEntries(companyId, { versionSelections: selections });
+
+    const result = await skillVersionRetentionService(db).pruneCompany(companyId, { keepNewest: 1 });
+    expect(result.skippedRecent).toBe(1);
+    expect(await exists(first.source)).toBe(true);
+  });
+
+  it("skips a version folder while a materialization holds it", async () => {
+    const companyId = await createCompany();
+    const { skillId, versionIds } = await createSkill(companyId, 3);
+    const dir = await writeSnapshot(companyId, skillId, versionIds[0]!);
+    let release!: () => void;
+    const held = runVersionSnapshotFlight(dir, () => new Promise<string>(resolve => { release = () => resolve(dir); }));
+
+    await skillVersionRetentionService(db).pruneCompany(companyId, { keepNewest: 1 });
+    expect(await exists(dir)).toBe(true);
+    release();
+    await held;
+    await skillVersionRetentionService(db).pruneCompany(companyId, { keepNewest: 1 });
+    expect(await exists(dir)).toBe(false);
+  });
+
+  it("sweeps deleted skills after protecting existing runs", async () => {
+    const companyId = await createCompany();
+    const { skillId, versionIds } = await createSkill(companyId, 2);
+    const first = await writeSnapshot(companyId, skillId, versionIds[0]!);
+    const second = await writeSnapshot(companyId, skillId, versionIds[1]!);
+    const [agent] = await db.insert(agents).values({ companyId, name: "Detached agent" }).returning();
+    const [run] = await db.insert(heartbeatRuns).values({ companyId, agentId: agent!.id, status: "running",
+      contextSnapshot: { skillVersions: { retained: versionIds[0] } } }).returning();
+    await companySkillService(db).deleteSkill(companyId, skillId);
+    expect(await exists(first)).toBe(true);
+    await skillVersionRetentionService(db).pruneCompany(companyId);
+    expect(await exists(first)).toBe(true);
+    expect(await exists(second)).toBe(false);
+    await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date() }).where(eq(heartbeatRuns.id, run!.id));
+    await skillVersionRetentionService(db).pruneCompany(companyId);
+    expect(await exists(first)).toBe(false);
+  });
+
+  it("never removes anything outside the company's versions root", async () => {
+    const companyId = await createCompany();
+    const otherCompanyId = await createCompany();
+    const { skillId, versionIds } = await createSkill(companyId, 3);
+    const other = await createSkill(otherCompanyId, 3);
+    const otherDirs = await Promise.all(other.versionIds.map(id => writeSnapshot(otherCompanyId, other.skillId, id)));
+    const outside = await fs.mkdtemp(path.join(paperclipHome!, "outside-"));
+    await fs.writeFile(path.join(outside, "keep.txt"), "keep");
+    const root = resolveCompanySkillVersionsRoot(companyId);
+    const victim = await writeSnapshot(companyId, skillId, versionIds[0]!);
+    const safe = await writeSnapshot(companyId, skillId, versionIds[1]!);
+    await fs.chmod(victim, 0o755);
+    await fs.symlink(outside, path.join(victim, "escape"));
+    const old = new Date(Date.now() - 3 * HOUR_MS);
+    await fs.utimes(victim, old, old);
+    await fs.mkdir(root, { recursive: true });
+    await fs.symlink(outside, path.join(root, randomUUID()));
+    await fs.symlink(outside, path.join(root, skillId, randomUUID()));
+
+    await skillVersionRetentionService(db).pruneCompany(companyId, { keepNewest: 1 });
+
+    expect(await exists(victim)).toBe(true);
+    expect(await exists(safe)).toBe(false);
+    expect(await fs.readFile(path.join(outside, "keep.txt"), "utf8")).toBe("keep");
+    expect(await Promise.all(otherDirs.map(exists))).toEqual([true, true, true]);
+  });
+
+  it("bounds removals per tick and continues on the next tick", async () => {
+    const companyId = await createCompany();
+    const { skillId, versionIds } = await createSkill(companyId, 6);
+    await Promise.all(versionIds.map(id => writeSnapshot(companyId, skillId, id)));
+    const retention = skillVersionRetentionService(db, { keepNewest: 1, maxRemovals: 2 });
+
+    let total = 0;
+    for (let index = 0; index < 5; index++) {
+      const result = await retention.tick();
+      expect(result.checked).toBeLessThanOrEqual(2);
+      total += result.removed;
+      if (!result.truncated) break;
+    }
+    expect(total).toBe(5);
+    expect(await fs.readdir(path.join(resolveCompanySkillVersionsRoot(companyId), skillId))).toEqual([versionIds[5]]);
+  });
+  it("bounds directory reads and retains the discovery cursor between pages", async () => {
+    const companyId = await createCompany();
+    const { skillId, versionIds } = await createSkill(companyId, 100);
+    await Promise.all(versionIds.map(id => writeSnapshot(companyId, skillId, id)));
+    const parent = path.join(resolveCompanySkillVersionsRoot(companyId), skillId);
+    const originalOpen = fs.opendir.bind(fs);
+    let reads = 0;
+    const opened = vi.spyOn(fs, "opendir").mockImplementation(async (...args: Parameters<typeof fs.opendir>) => {
+      const directory = await originalOpen(...args);
+      const read = directory.read.bind(directory);
+      vi.spyOn(directory, "read").mockImplementation(async () => { reads++; return read(); });
+      return directory;
+    });
+    const stat = vi.spyOn(fs, "lstat");
+    const retention = skillVersionRetentionService(db, { maxEntries: 2, dryRun: true });
+    try {
+      const first = await retention.tick();
+      expect(first).toMatchObject({ checked: 2, truncated: true });
+      expect(reads).toBeLessThanOrEqual(2);
+      const inspected = () => new Set(stat.mock.calls.map(call => String(call[0]))
+        .filter(file => path.dirname(file) === parent).map(file => path.basename(file)));
+      const firstPaths = inspected();
+      expect(firstPaths.size).toBe(1); // Opening the skill root also consumes a slot.
+      stat.mockClear(); reads = 0;
+      const second = await retention.tick();
+      expect(second).toMatchObject({ checked: 2, truncated: true });
+      expect(reads).toBe(2);
+      expect(inspected().size).toBe(2);
+      expect([...inspected()].some(name => firstPaths.has(name))).toBe(false);
+      expect(opened.mock.calls.filter(call => call[0] === parent)).toHaveLength(1);
+    } finally { await retention.dispose(); }
+  });
+
+  it("continues beyond a page ending at an exhausted skill root", async () => {
+    const companyId = await createCompany();
+    const roots = [1, 2, 3].map(value => `00000000-0000-4000-8000-${String(value).padStart(12, "0")}`);
+    await Promise.all([randomUUID(), randomUUID()].map(id => writeSnapshot(companyId, roots[0]!, id, 0)));
+    await fs.mkdir(path.join(resolveCompanySkillVersionsRoot(companyId), roots[1]!), { recursive: true });
+    const versions = Array.from({ length: 5 }, () => randomUUID()).sort();
+    const paths = await Promise.all(versions.map((id, index) => writeSnapshot(companyId, roots[2]!, id, index === 4 ? 3 * HOUR_MS : 0)));
+    const retention = skillVersionRetentionService(db, { maxEntries: 3 });
+    let reclaimed = 0;
+    for (let page = 0; page < 5; page++) {
+      const result = await retention.tick();
+      expect(result.checked).toBeLessThanOrEqual(3);
+      reclaimed += result.removed;
+      if (!result.truncated) break;
+    }
+    expect(reclaimed).toBe(1);
+    expect(await exists(paths[4]!)).toBe(false);
+    expect(await Promise.all(paths.slice(0, 4).map(exists))).toEqual([true, true, true, true]);
+  });
+
+  it("bounds failed attempts and continues beyond them", async () => {
+    const companyId = await createCompany();
+    const { skillId, versionIds } = await createSkill(companyId, 6);
+    await Promise.all(versionIds.map(id => writeSnapshot(companyId, skillId, id)));
+    const rename = vi.spyOn(fs, "rename").mockRejectedValue(Object.assign(new Error("read-only fixture"), { code: "EACCES" }));
+    const retention = skillVersionRetentionService(db, { maxRemovals: 2 });
+    let checked = 0;
+    let failures = 0;
+    for (let index = 0; index < 5; index++) {
+      const result = await retention.tick();
+      expect(result.checked).toBeLessThanOrEqual(2);
+      checked += result.checked;
+      failures += result.failed;
+      if (!result.truncated) break;
+    }
+    expect(checked).toBe(7); // Six versions plus the skill-root discovery slot.
+    expect(failures).toBe(5);
+    expect(rename).toHaveBeenCalledTimes(5);
+  });
+
+});

server/src/__tests__/task-workspace-source-copy-privacy.test.ts

@@ -691,6 +691,33 @@ exec "$PROBE_REAL_GIT" "$@"
     await expect(f.service.prepareForAdmission(f.admission)).resolves.toEqual([]);
   });
 
+  it("rechecks retargeted execution-workspace aliases at admission and publication", async () => {
+    const f = await fixture(), peer = await standardPeer(f);
+    const unrelated = path.join(root, `${f.issueId}-unrelated`); await mkdir(unrelated);
+    const alias = path.join(root, `${f.issueId}-alias`); await symlink(unrelated, alias);
+    await db.update(executionWorkspaces).set({ cwd: alias }).where(eq(executionWorkspaces.id, peer.workspace.id));
+    await db.update(issues).set({ visibility: "private", privacyRootIssueId: peer.issueId, responsibleUserId: peer.userId })
+      .where(eq(issues.id, peer.issueId));
+    await expect(f.service.prepareForAdmission(f.admission)).resolves.toEqual([]);
+    await rm(alias);
+    await symlink(f.admission.cwd, alias);
+    await expect(assertExecutionWorkspaceRootReadAccess(db, { companyId: f.companyId, cwd: f.admission.cwd, actor: f.actor }))
+      .rejects.toThrow();
+    await expect(f.service.prepareForAdmission(f.admission)).rejects.toThrow();
+    await expect(f.service.guardForPublication(f.admission)).rejects.toThrow();
+  });
+
+  it.each(["conflicting_path", "overlapping_invalid_key"] as const)("rejects retained root identity corruption (%s)", async scenario => {
+    const f = await fixture();
+    const canonicalCwd = await realpath(f.admission.cwd);
+    const unrelated = path.join(root, `${f.issueId}-unrelated`); await mkdir(unrelated);
+    await db.insert(executionWorkspaceRepositoryAccess).values({ companyId: f.companyId, kind: "source", scopeId: f.workspace.id,
+      rootKey: createHash("sha256").update(scenario === "conflicting_path" ? canonicalCwd : await realpath(unrelated)).digest("hex"),
+      canonicalCwd: scenario === "conflicting_path" ? await realpath(unrelated) : path.join(canonicalCwd, "child") });
+    await expect(f.service.prepareForAdmission(f.admission)).rejects.toThrow("Repository root identity conflicts");
+    await expect(f.service.guardForPublication(f.admission)).rejects.toThrow("Repository root identity conflicts");
+  });
+
   it.each(["revoked", "deleted_source", "deleted_owner", "retained_reader"] as const)("protects already-copied bytes when post-copy authorization fails (%s)", async failure => {
     const f = await fixture(), exec = promisify(execFileCallback);
     await exec("git", ["init", "--initial-branch=main", f.admission.cwd]);

server/src/__tests__/tool-access-service.test.ts

@@ -5363,7 +5363,11 @@ describeEmbeddedPostgres("tool access service", () => {
     ).toBe(false);
     expect(
       nonPilotGmail.methods.map((method: { key: string }) => method.key),
-    ).toEqual(["customer-read-oauth", "customer-draft-oauth"]);
+    ).toEqual([
+      "customer-read-oauth",
+      "customer-draft-oauth",
+      "customer-archive-oauth",
+    ]);
 
     const pilot = await request(
       createRouteApp(
@@ -5384,6 +5388,34 @@ describeEmbeddedPostgres("tool access service", () => {
       "paperclip-read",
       "customer-read-oauth",
       "customer-draft-oauth",
+      "customer-archive-oauth",
+    ]);
+
+    const archivePilotConnector = fakeGoogleWorkspaceConnector(
+      company.id,
+      userId,
+      "gmail.archive",
+    );
+    const archivePilot = await request(
+      createRouteApp(
+        db,
+        boardSessionActor(company.id, "owner", userId),
+        undefined,
+        { paperclipCloudConnector: archivePilotConnector },
+      ),
+    ).get(`/api/companies/${company.id}/tools/gallery`);
+    expect(archivePilot.status).toBe(200);
+    const archivePilotGmail = archivePilot.body.apps.find(
+      (app: { slug: string }) => app.slug === "gmail",
+    );
+    expect(archivePilotGmail.ownershipAvailability.platform_shared).toBe(true);
+    expect(
+      archivePilotGmail.methods.map((method: { key: string }) => method.key),
+    ).toEqual([
+      "customer-read-oauth",
+      "customer-draft-oauth",
+      "paperclip-archive",
+      "customer-archive-oauth",
     ]);
   });
 

server/src/__tests__/workspace-runtime.test.ts

@@ -7326,8 +7326,10 @@ describeEmbeddedPostgres("workspace runtime service control persistence", () =>
     }
   }, 20_000);
 
-  it("commits fixed-port reservations before readiness windows longer than 60 seconds", async () => {
+  it("commits fixed-port reservations across the configured database idle timeout", async () => {
     const fixture = await createRuntimeFixture();
+    const readinessDb = createDb(tempDb!.connectionString, { idleTimeoutSeconds: 1, maxConnections: 1 });
+    const [initialConnection] = await readinessDb.$client<{ pid: number }[]>`select pg_backend_pid() as pid`;
     const cleanupRuntimeHome = await createRuntimeHome();
     const workspace = fixture.workspaces[0]!;
     const basePort = await findFreePort();
@@ -7353,7 +7355,7 @@ describeEmbeddedPostgres("workspace runtime service control persistence", () =>
 
     try {
       startPromise = startRuntimeServicesForWorkspaceControl({
-        db,
+        db: readinessDb,
         actor: fixture.actor,
         issue: null,
         workspace: fixture.realizedWorkspace(workspace),
@@ -7377,16 +7379,18 @@ describeEmbeddedPostgres("workspace runtime service control persistence", () =>
         parentWrite.then(() => "committed" as const),
         new Promise<"locked">((resolve) => setTimeout(() => resolve("locked"), 2_000)),
       ]);
+      expect(parentWriteOutcome).toBe("committed");
 
       const readinessWaitStartedAt = Date.now();
-      await new Promise((resolve) => setTimeout(resolve, 60_250));
-      expect(Date.now() - readinessWaitStartedAt).toBeGreaterThanOrEqual(60_000);
+      await new Promise((resolve) => setTimeout(resolve, 1_250));
+      expect(Date.now() - readinessWaitStartedAt).toBeGreaterThanOrEqual(1_000);
+      const [reconnected] = await readinessDb.$client<{ pid: number }[]>`select pg_backend_pid() as pid`;
+      expect(reconnected!.pid).not.toBe(initialConnection!.pid);
 
       await fs.writeFile(releaseMarkerPath, "release");
       const started = (await startPromise)[0]!;
       await parentWrite;
 
-      expect(parentWriteOutcome).toBe("committed");
       expect(started).toMatchObject({
         port: basePort,
         url: `http://127.0.0.1:${basePort}`,
@@ -7413,6 +7417,7 @@ describeEmbeddedPostgres("workspace runtime service control persistence", () =>
       }).catch(() => undefined);
       await cleanupRuntimeHome();
       await fixture.cleanup();
+      await readinessDb.$client.end({ timeout: 1 });
     }
   }, 80_000);
 

server/src/config.ts

@@ -79,8 +79,15 @@ export interface Config {
   databaseBackupIntervalMinutes: number;
   databaseBackupRetentionDays: number;
   databaseBackupDir: string;
+  gitCacheExpirationEnabled: boolean;
+  gitBaseExpirationEnabled: boolean;
+  gitCacheExpirationDryRun: boolean;
   workspaceReaperCooldownDays: number;
   workspaceCompactionDelayHours: number | null;
+  skillVersionRetentionEnabled: boolean;
+  skillVersionRetentionDryRun: boolean;
+  skillVersionRetentionKeepNewest: number;
+  skillVersionRetentionGraceMinutes: number;
   serveUi: boolean;
   uiDevMiddleware: boolean;
   secretsProvider: SecretProvider;
@@ -313,6 +320,19 @@ export function loadConfig(): Config {
           && workspaceCompactionDelayHoursRaw >= 0
         ? workspaceCompactionDelayHoursRaw
         : 1;
+  // Skill version retention removes materialized version directories under
+  // `skills/<companyId>/__versions__/` that no current version, agent pin, or
+  // live run uses. It keeps this many newest versions per skill as a warm
+  // cache. A version directory used within the grace window also stays. A
+  // removed directory is written again from its version row on next use.
+  // Negative, empty, or non-numeric values fall back to the defaults.
+  const nonNegativeIntegerEnv = (name: string, fallback: number) => {
+    const raw = process.env[name]?.trim();
+    const value = Number(raw);
+    return raw && Number.isFinite(value) && value >= 0 ? Math.floor(value) : fallback;
+  };
+  const skillVersionRetentionKeepNewest = nonNegativeIntegerEnv("PAPERCLIP_SKILL_VERSION_RETENTION_KEEP", 0);
+  const skillVersionRetentionGraceMinutes = nonNegativeIntegerEnv("PAPERCLIP_SKILL_VERSION_RETENTION_GRACE_MINUTES", 60);
   const bindValidationErrors = validateConfiguredBindMode({
     deploymentMode,
     deploymentExposure,
@@ -358,8 +378,15 @@ export function loadConfig(): Config {
     databaseBackupIntervalMinutes,
     databaseBackupRetentionDays,
     databaseBackupDir,
+    gitCacheExpirationEnabled: process.env.PAPERCLIP_GIT_CACHE_EXPIRATION_ENABLED === "true",
+    gitBaseExpirationEnabled: process.env.PAPERCLIP_GIT_BASE_EXPIRATION_ENABLED === "true",
+    gitCacheExpirationDryRun: process.env.PAPERCLIP_GIT_CACHE_EXPIRATION_DRY_RUN !== "false",
     workspaceReaperCooldownDays,
     workspaceCompactionDelayHours,
+    skillVersionRetentionEnabled: process.env.PAPERCLIP_SKILL_VERSION_RETENTION_ENABLED !== "false",
+    skillVersionRetentionDryRun: process.env.PAPERCLIP_SKILL_VERSION_RETENTION_DRY_RUN === "true",
+    skillVersionRetentionKeepNewest,
+    skillVersionRetentionGraceMinutes,
     serveUi:
       process.env.SERVE_UI !== undefined
         ? process.env.SERVE_UI === "true"

server/src/index.ts

@@ -1,5 +1,10 @@
 import { timeStartupRecovery } from "./startup-recovery-timing.js";
 import { skillSourceSyncService } from "./services/skill-source-sync.js";
+import {
+  SKILL_VERSION_RETENTION_INTERVAL_MS,
+  SKILL_VERSION_RETENTION_STARTUP_DELAY_MS,
+  skillVersionRetentionService,
+} from "./services/skill-version-retention.js";
 import { subscribeAllCompanyLiveEvents } from "./services/live-events.js";
 import { DELIVERY_QUEUES } from "./services/delivery-work-notifications.js";
 import type { createDeliveryWorkCoordinator } from "./services/delivery-work-coordinator.js";
@@ -1225,6 +1230,39 @@ async function startServerWithDatabaseTeardown(
   const skillSourceSyncInterval = setInterval(sweepSkillSources, 5000);
   skillSourceSyncInterval.unref?.();
   sweepSkillSources();
+  // Materialized skill versions are disk caches of version rows. The sweep is
+  // single-flight, bounded per tick, and stops between removals on shutdown or
+  // idle drain. The first sweep runs after startup without delaying readiness.
+  const skillVersionRetention = skillVersionRetentionService(db as any, {
+    dryRun: config.skillVersionRetentionDryRun,
+    keepNewest: config.skillVersionRetentionKeepNewest,
+    graceMs: config.skillVersionRetentionGraceMinutes * 60 * 1000,
+    shouldContinue: () => !heartbeatSchedulerStopped && !isIdleTaskDrainActive() && !isWarmStandby(),
+  });
+  let skillVersionRetentionInFlight = false;
+  let skillVersionRetentionContinuation: ReturnType<typeof setTimeout> | undefined;
+  const sweepSkillVersionSnapshots = () => {
+    if (!config.skillVersionRetentionEnabled || skillVersionRetentionInFlight || isIdleTaskDrainActive() || heartbeatSchedulerStopped || isWarmStandby()) return;
+    skillVersionRetentionInFlight = true;
+    trackHeartbeatSchedulerWork(skillVersionRetention.tick()
+      .then((result) => {
+        if (result.checked > 0 || result.failed > 0 || Object.keys(result.skipped).length > 0) {
+          logger.info({ ...result, dryRun: config.skillVersionRetentionDryRun }, "skill version snapshot retention");
+        }
+        if (result.truncated && !heartbeatSchedulerStopped) {
+          skillVersionRetentionContinuation = setTimeout(sweepSkillVersionSnapshots, 1000);
+          skillVersionRetentionContinuation.unref?.();
+        }
+      })
+      .catch((err) => {
+        logger.error({ err }, "skill version snapshot retention failed");
+      })
+      .finally(() => { skillVersionRetentionInFlight = false; }));
+  };
+  const skillVersionRetentionInterval = setInterval(sweepSkillVersionSnapshots, SKILL_VERSION_RETENTION_INTERVAL_MS);
+  skillVersionRetentionInterval.unref?.();
+  const skillVersionRetentionStartup = setTimeout(sweepSkillVersionSnapshots, SKILL_VERSION_RETENTION_STARTUP_DELAY_MS);
+  skillVersionRetentionStartup.unref?.();
   const externalObjects = externalObjectService(db as any, {
     pluginWorkerManager,
     enabled: async () => (await instanceSettingsService(db).getExperimental()).enableExternalObjects === true,
@@ -1374,6 +1412,9 @@ async function startServerWithDatabaseTeardown(
       wakeup: heartbeat.wakeup,
     });
     const terminalWorkspaces = executionWorkspaceService(db as any, {
+      gitCacheExpirationEnabled: config.gitCacheExpirationEnabled,
+      gitBaseExpirationEnabled: config.gitBaseExpirationEnabled,
+      gitCacheExpirationDryRun: config.gitCacheExpirationDryRun,
       workspaceReaperCooldownDays: config.workspaceReaperCooldownDays,
       workspaceCompactionDelayMs: config.workspaceCompactionDelayHours === null
         ? null
@@ -1977,6 +2018,10 @@ async function startServerWithDatabaseTeardown(
     await systemdNotify(["--stopping", `--status=Stopping after ${signal}`]);
     heartbeatSchedulerStopped = true;
     clearInterval(skillSourceSyncInterval);
+    clearInterval(skillVersionRetentionInterval);
+    clearTimeout(skillVersionRetentionStartup);
+    clearTimeout(skillVersionRetentionContinuation);
+    await skillVersionRetention.dispose();
     await skillSourceSync.stop();
     unsubscribeChatCompletions();
     await deliveryWork.stop();

server/src/instrumentation.ts

@@ -39,7 +39,7 @@ import { checkExactPeerVersions } from "./peer-version-check.js";
 
 export { checkExactPeerVersions } from "./peer-version-check.js";
 
-const endpoint = process.env.OTEL_EXPORTER_OTLP_ENDPOINT;
+const endpoint = process.env.OTEL_EXPORTER_OTLP_ENDPOINT?.trim();
 
 let sdkShutdown: (() => Promise<void>) | null = null;
 let shutdownPromise: Promise<void> | null = null;
@@ -96,6 +96,7 @@ let tracerApiLoadFailed = false;
  * local no-op tracer (fail open).
  */
 export function getStartupTracer(name = "paperclip.startup"): StartupTracerHandle {
+  if (!process.env.OTEL_EXPORTER_OTLP_ENDPOINT?.trim()) return NOOP_TRACER;
   try {
     const require = createRequire(import.meta.url);
     const api = require("@opentelemetry/api") as {
@@ -151,6 +152,7 @@ let traceContextApiLoadFailed = false;
  * local no-op trace context (fail open).
  */
 export function getStartupTraceContext(name = "paperclip.startup"): StartupTraceContextHandle {
+  if (!process.env.OTEL_EXPORTER_OTLP_ENDPOINT?.trim()) return NOOP_TRACE_CONTEXT;
   try {
     const require = createRequire(import.meta.url);
     const api = require("@opentelemetry/api") as {

server/src/redaction.ts

@@ -329,6 +329,17 @@ const NATIVE_RUN_SPAN_NAMES = new Set([
   "provider.time_to_first_agent_event",
   "provider.turn.queue",
   "question_response.to_run_created",
+  "harness_state.inspect",
+  "harness_state.path_check",
+  "harness_state.launch_material",
+  "provider_pack.select",
+  "provider_pack.discover",
+  "provider_pack.verify",
+  "provider_pack.verify_preinstalled",
+  "provider_pack.verify_uploaded",
+  "provider_pack.link_preinstalled",
+  "provider_pack.clear_stale",
+  "provider_pack.upload",
   "runner.artifact.discover",
   "runner.artifact.prepare",
   "runner.prp.authenticate",

server/src/services/activity-log.ts

@@ -1,3 +1,4 @@
+import { beginRunPhase } from "./execution-preparation-timing.js";
 import type { ActivityPublication } from "../types/activity-publication.js";
 export type { ActivityPublication } from "../types/activity-publication.js";
 import { randomUUID } from "node:crypto";
@@ -146,12 +147,18 @@ export async function resolveResponsibleUserIdForActivity(db: Db, input: LogActi
 }
 
 export function publishActivity(publication: ActivityPublication) {
+  const finish = beginRunPhase("execution.realtime");
+  try {
   publishLiveEvent({
     companyId: publication.companyId,
     type: "activity.logged",
     payload: publication.payload,
   });
   if (publication.pluginEvent) publishPluginDomainEvent(publication.pluginEvent);
+  } catch (error) {
+    finish(true);
+    throw error;
+  } finally { finish(); }
 }
 
 export async function persistActivity(db: Db, input: LogActivityInput) {

server/src/services/chat-channels/slack/task-authority.ts

@@ -5,6 +5,8 @@ import { slackAuthorizationRevision } from "../../connectors/slack-revision.js";
 
 import { instanceSettingsService } from "../../instance-settings.js";
 import { registerSlackTaskAuthority, slackRunOrigin } from "../../connectors/slack-authority.js";
+import { measureRunPhase } from "../../execution-preparation-timing.js";
+import { measureRunTransaction } from "../../execution-transaction-timing.js";
 
 import {
   createHash,
@@ -51,13 +53,13 @@ export function createSlackTaskAuthorityRegistration(db: Db, options: SlackTaskA
 
 
   const unregisterSlackTaskAuthority = registerSlackTaskAuthority(db, async (binding) => {
-    if (!(await instanceSettingsService(db).getExperimental()).enableChatConnectors)
+    if (!(await measureRunPhase("connectors.slack.settings", () => instanceSettingsService(db).getExperimental())).enableChatConnectors)
       throw forbidden("Chat connectors are disabled");
-    const identity = await slackRunOrigin(db, binding);
+    const identity = await measureRunPhase("connectors.slack.identity", () => slackRunOrigin(db, binding));
     const run = identity.run;
     if ((run.contextSnapshot?.issueId ?? run.contextSnapshot?.taskId) !== binding.issueId || !run.responsibleUserId)
       throw forbidden("Slack tools require the bound task and an accepted linked user");
-    const resolved = await db.transaction(async (tx) => {
+    const resolved = await measureRunTransaction(db, "connectors.slack.authority", async (tx) => {
       const [action] = await tx.select().from(chatActions).where(and(
         eq(chatActions.companyId, binding.companyId),
         eq(chatActions.kind, "inbound_wakeup"),
@@ -134,7 +136,7 @@ export function createSlackTaskAuthorityRegistration(db: Db, options: SlackTaskA
       if (!principal) throw forbidden("Slack sender is no longer available");
       return { ...source, principalId: principal.id, slackUserId: principal.externalId };
     });
-    const credentials = await resolveCredentials(resolved.endpoint);
+    const credentials = await measureRunPhase("connectors.slack.credentials", () => resolveCredentials(resolved.endpoint));
     if (!credentials.botToken) throw forbidden("Slack bot credential is unavailable");
     return {
       endpoint: resolved.endpoint, issueId: binding.issueId, conversation: resolved.conversation,
@@ -145,7 +147,7 @@ export function createSlackTaskAuthorityRegistration(db: Db, options: SlackTaskA
       revision: createHash("sha256").update(JSON.stringify([
         resolved.endpoint.status, resolved.endpoint.allowDirectMessages, resolved.conversation?.sessionGeneration ?? null,
         run.responsibleUserId, credentials.botToken,
-        await slackAuthorizationRevision(db, binding.companyId, resolved.endpoint.id, resolved.endpoint.connectionId, binding.agentId, run.responsibleUserId),
+        await measureRunPhase("connectors.slack.revision", () => slackAuthorizationRevision(db, binding.companyId, resolved.endpoint.id, resolved.endpoint.connectionId, binding.agentId, run.responsibleUserId!)),
       ])).digest("hex"),
       botToken: credentials.botToken,
       searchActionToken: resolved.delivery ? slackSearchActionToken(db, resolved.endpoint.id, resolved.endpoint.providerAccountId!, resolved.slackUserId, String((resolved.delivery.normalizedEvent.message as Record<string, unknown> | undefined)?.providerMessageId ?? "")) : null,

server/src/services/chat-publication-reconciliation.test.ts

@@ -247,7 +247,7 @@ describe("chat publication commit signals", () => {
       'eventType: "run.presentation.resolved"',
     );
     const committedComment = completionSource.lastIndexOf(
-      "await issuesSvc.addComment",
+      "issuesSvc.addComment",
       presentationMarker,
     );
     expect(presentationMarker).toBeGreaterThanOrEqual(0);

server/src/services/company-skills.ts

@@ -1,3 +1,4 @@
+import { ensureSkillCacheDirectory, touchSkillVersionCache, withSkillVersionCacheLock } from "./skill-version-cache.js";
 import { retainProjectSkillAccess, skillAccessService } from "./skill-access.js";
 import type { AuthorizationActor } from "./authorization.js";
 import { updateAgentConfigurationInTransaction } from "./agent-configuration-transaction.js";
@@ -34,6 +35,7 @@ import {
 import { readPaperclipSkillSyncPreference, writePaperclipSkillSyncPreference, resolvePaperclipDesiredSkillNames,
   resolveLegacyPaperclipDesiredSkillNames } from "@paperclipai/adapter-utils/server-utils";
 import { REPOSITORY_CONTENT_DIR, repositorySkillProjection } from "./skill-repository-package.js";
+import { runVersionSnapshotFlight } from "./skill-version-retention.js";
 import type { PaperclipDesiredSkillEntry, PaperclipSkillEntry } from "@paperclipai/adapter-utils/server-utils";
 import type {
   AgentDesiredSkillEntry,
@@ -281,8 +283,6 @@ type ParsedSkillImportSource = {
   warnings: string[];
 };
 
-const versionSnapshotFlights = new Map<string, Promise<string>>();
-
 const EXTERNAL_SKILL_SOURCE_TYPES = new Set<CompanySkillSourceType>(["github", "skills_sh", "url"]);
 
 function isPinnedCommitRef(value: string | null | undefined) {
@@ -3533,17 +3533,20 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo
     const skill = await getById(companyId, skillId);
     if (!skill) return null;
     if (skill.metadata?.skillSourceId && skill.metadata?.snapshotHash && skill.currentVersionId) {
-      const version = await getVersion(companyId, skill.id, skill.currentVersionId);
-      if (!version) throw unprocessable("Installed skill version is unavailable.");
-      const sourceLocator = await materializeVersionSnapshot(companyId, skill, version);
-      const audit = await auditInstalledSkillBytes({
-        ...skill,
-        sourceType: "local_path",
-        sourceLocator,
-        fileInventory: await runtimeVersionInventory(companyId, version),
+      const versionId = skill.currentVersionId;
+      return withSkillVersionCacheLock(db, companyId, async () => {
+        const version = await getVersion(companyId, skill.id, versionId);
+        if (!version) throw unprocessable("Installed skill version is unavailable.");
+        const sourceLocator = await materializeVersionSnapshot(companyId, skill, version);
+        const audit = await auditInstalledSkillBytes({
+          ...skill,
+          sourceType: "local_path",
+          sourceLocator,
+          fileInventory: await runtimeVersionInventory(companyId, version),
+        });
+        await persistAuditMetadata(skill, audit);
+        return audit;
       });
-      await persistAuditMetadata(skill, audit);
-      return audit;
     }
     if (skill.sourceType !== "catalog" && skill.sourceType !== "local_path") {
       throw unprocessable("Only local-path and catalog-managed company skills support audit.");
@@ -5963,13 +5966,15 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo
         } else if (entry.isFile()) {
           out.push(normalizePortablePath(relativePath));
         } else {
-          out.push(normalizePortablePath(relativePath));
+          throw unprocessable("Unsafe file in materialized skill version.");
         }
       }
       return out;
     }
 
     try {
+      const rootStat = await fs.lstat(root);
+      if (!rootStat.isDirectory() || rootStat.isSymbolicLink()) throw unprocessable("Unsafe materialized skill directory.");
       return await walk(root, "");
     } catch (error) {
       if ((error as NodeJS.ErrnoException).code === "ENOENT") return null;
@@ -6013,14 +6018,20 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo
   }
 
   async function materializeVersionSnapshot(companyId: string, skill: CompanySkill, version: CompanySkillVersion): Promise<string> {
+    return withSkillVersionCacheLock(db, companyId, async () => {
+      const directory = await materializeVersionSnapshotLocked(companyId, skill, version);
+      await touchSkillVersionCache(companyId, skill.id, version.id);
+      return directory;
+    });
+  }
+
+  async function materializeVersionSnapshotLocked(companyId: string, skill: CompanySkill, version: CompanySkillVersion): Promise<string> {
     version = { ...version, fileInventory: await runtimeVersionInventory(companyId, version) };
     const runtimeRoot = path.resolve(resolveManagedSkillsRoot(companyId), "__versions__", skill.id);
     const skillDir = path.join(runtimeRoot, version.id);
-    const existing = versionSnapshotFlights.get(skillDir);
-    if (existing) return existing;
-    const work = (async () => {
+    return runVersionSnapshotFlight(skillDir, async () => {
+      await ensureSkillCacheDirectory(runtimeRoot, true);
       if (await materializedVersionSnapshotMatches(skillDir, version)) return skillDir;
-      await fs.mkdir(runtimeRoot, { recursive: true });
       const staging = await fs.mkdtemp(path.join(runtimeRoot, `.${version.id}-`));
       try {
         let wroteSkillFile = false;
@@ -6044,9 +6055,7 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo
         }
         return skillDir;
       } finally { await fs.rm(staging, { recursive: true, force: true }); }
-    })();
-    versionSnapshotFlights.set(skillDir, work);
-    try { return await work; } finally { versionSnapshotFlights.delete(skillDir); }
+    });
   }
 
   function resolveRuntimeSkillMaterializedPath(companyId: string, skill: Pick<CompanySkill, "key" | "slug">) {
@@ -6145,6 +6154,13 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo
   ): Promise<PaperclipSkillEntry[]> {
     if (options.onlyKeys?.size === 0 || (!options.onlyKeys && options.selectionConfig && !options.includeLegacy
       && readPaperclipSkillSyncPreference(options.selectionConfig).desiredSkills.length === 0)) return [];
+    return withSkillVersionCacheLock(db, companyId, () => listRuntimeSkillEntriesLocked(companyId, options));
+  }
+
+  async function listRuntimeSkillEntriesLocked(
+    companyId: string,
+    options: RuntimeSkillEntryOptions,
+  ): Promise<PaperclipSkillEntry[]> {
     const skills = await listFull(companyId);
     const selectedKeys = options.onlyKeys ?? (options.selectionConfig ? new Set(
       (options.includeLegacy ? resolveLegacyPaperclipDesiredSkillNames : resolvePaperclipDesiredSkillNames)(
@@ -7351,6 +7367,8 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo
         logger.warn({ err: error, companyId }, "Skill deleted; quarantined source cleanup failed");
       });
     }
+    // Version caches may still be linked into a captured run. The retention
+    // worker removes orphan versions only after their run pins have settled.
     return deleted;
   }
 

server/src/services/connector-runtime.test.ts

@@ -0,0 +1,46 @@
+import { afterEach, describe, expect, it, vi } from "vitest";
+import type { Db } from "@paperclipai/db";
+import { resolveConnectorAssignments } from "./connector-runtime.js";
+
+const resolvers = vi.hoisted(() => ({
+  browser: vi.fn(), speko: vi.fn(), slack: vi.fn(), inboxes: vi.fn(),
+}));
+vi.mock("./native-runtime/assigned-mcp-tools.js", () => ({ getAssignedMcpGateway: () => ({ browserUseResources: resolvers.browser }) }));
+vi.mock("./voice/speko-agent-tools.js", () => ({ spekoAssignedResources: resolvers.speko }));
+vi.mock("./connectors/slack.js", () => ({ slackAssignedResource: resolvers.slack, executeGovernedSlackTool: vi.fn() }));
+vi.mock("./email-channels.js", () => ({ emailChannelService: () => ({ assignedInboxes: resolvers.inboxes }) }));
+
+afterEach(() => vi.resetAllMocks());
+
+function deferred<T>() {
+  let resolve!: (value: T) => void;
+  let reject!: (error: Error) => void;
+  const promise = new Promise<T>((yes, no) => { resolve = yes; reject = no; });
+  return { promise, resolve, reject };
+}
+
+describe("connector assignment preparation", () => {
+  it("starts independent resolvers together and retains connector order", async () => {
+    const pending = Array.from({ length: 4 }, () => deferred<Array<{ id: string; label: string; connectionId: string; address: string }>>());
+    Object.values(resolvers).forEach((resolver, i) => resolver.mockReturnValue(pending[i].promise));
+    const assignments = resolveConnectorAssignments({} as Db, { companyId: "company", agentId: "agent" });
+    await vi.waitFor(() => Object.values(resolvers).forEach(resolver => expect(resolver).toHaveBeenCalledOnce()));
+    for (const i of [3, 2, 1, 0]) pending[i].resolve([{ id: `resource-${i}`, connectionId: `connection-${i}`, label: "Resource", address: "inbox@example.com" }]);
+    expect((await assignments).map(assignment => assignment.key)).toEqual(["browser-use-cloud", "speko", "slack", "agentmail"]);
+  });
+
+  it("settles outstanding resolution before propagating a failed connector", async () => {
+    const slack = deferred<[]>();
+    resolvers.browser.mockResolvedValue([]);
+    resolvers.speko.mockRejectedValue(new Error("Speko unavailable"));
+    resolvers.slack.mockReturnValue(slack.promise);
+    resolvers.inboxes.mockResolvedValue([]);
+    let settled = false;
+    const assignments = resolveConnectorAssignments({} as Db, { companyId: "company", agentId: "agent" });
+    const result = expect(assignments).rejects.toThrow("Speko unavailable").finally(() => { settled = true; });
+    await vi.waitFor(() => expect(resolvers.slack).toHaveBeenCalledOnce());
+    expect(settled).toBe(false);
+    slack.resolve([]);
+    await result;
+  });
+});

server/src/services/connector-runtime.ts

@@ -22,6 +22,7 @@ import {
 } from "./connectors/agentmail.js";
 import { materializeAsset } from "./native-runtime/runtime-context.js";
 import { BROWSER_USE_CLOUD_SKILL } from "./connectors/browser-use-cloud/skill.js";
+import { measureRunPhase, type ExecutionPhase } from "./execution-preparation-timing.js";
 
 type AgentBinding = { companyId: string; agentId: string; runId?: string; issueId?: string };
 type ToolBinding = AgentBinding & {
@@ -43,6 +44,7 @@ type Tool = {
 };
 interface ConnectorDefinition {
   key: string;
+  resolvePhase: Extract<ExecutionPhase, `connectors.${string}.resolve`>;
   label: string;
   skillName: string;
   /** Trusted connector-owned content; never discovered through global skills/. */
@@ -62,7 +64,7 @@ interface ConnectorDefinition {
 // Trusted connector packages declare their contributions here. Assignments and
 // current access, not credential availability or agent-authored config, select them.
 const connectors: ConnectorDefinition[] = [
-  { key: "browser-use-cloud", label: "Browser Use Cloud", skillName: "browser-use-cloud", skillMarkdown: BROWSER_USE_CLOUD_SKILL, tools: [],
+  { key: "browser-use-cloud", resolvePhase: "connectors.browser_use.resolve", label: "Browser Use Cloud", skillName: "browser-use-cloud", skillMarkdown: BROWSER_USE_CLOUD_SKILL, tools: [],
     legacySkillKeys: ["paperclipai/paperclip/browser-use"],
     async resolve(db, binding) {
       const { getAssignedMcpGateway } = await import("./native-runtime/assigned-mcp-tools.js");
@@ -70,18 +72,19 @@ const connectors: ConnectorDefinition[] = [
     },
     async execute() { throw forbidden("Use Browser Use Cloud through the connection tool gateway."); },
   },
-  { key: "speko", label: "Speko", skillName: "speko", skillMarkdown: SPEKO_SKILL,
+  { key: "speko", resolvePhase: "connectors.speko.resolve", label: "Speko", skillName: "speko", skillMarkdown: SPEKO_SKILL,
     tools: [], // Native phone tools keep their existing governed gateway transport.
     resolve: spekoAssignedResources,
     async execute() { throw forbidden("Use Speko through the connection tool gateway."); },
   },
-  { key: "slack", label: "Slack", skillName: "slack",
+  { key: "slack", resolvePhase: "connectors.slack.resolve", label: "Slack", skillName: "slack",
     tools: SLACK_TOOLS.map(({ name, description, inputSchema }) => ({ name, description, inputSchema })),
     resolve: slackAssignedResource,
     execute: executeGovernedSlackTool,
   },
   {
     key: "agentmail",
+    resolvePhase: "connectors.agentmail.resolve",
     label: "AgentMail",
     skillName: "agentmail",
     tools: AGENTMAIL_TOOLS.map(
@@ -125,19 +128,25 @@ export async function resolveConnectorAssignments(
   db: Db,
   binding: AgentBinding,
 ): Promise<ConnectorAssignment[]> {
-  const assignments: ConnectorAssignment[] = [];
-  for (const connector of connectors) {
-    const resources = await connector.resolve(db, binding);
-    if (resources.length)
-      assignments.push({
-        key: connector.key,
-        label: connector.label,
-        skillKey: skillKey(connector),
-        resources,
-        tools: connector.tools,
-      });
-  }
-  return assignments;
+  const results = await Promise.allSettled(connectors.map(async (connector) => {
+    const resources = await measureRunPhase(connector.resolvePhase, () => connector.resolve(db, binding));
+    return resources.length ? [{
+      key: connector.key,
+      label: connector.label,
+      skillKey: skillKey(connector),
+      resources,
+      tools: connector.tools,
+    }] : [];
+  }));
+  return results.flatMap(result => {
+    if (result.status === "rejected") throw result.reason;
+    return result.value;
+  });
+}
+
+export function connectorToolDefinitionsForSkills(skillKeys: ReadonlySet<string>): Pick<ConnectorAssignment, "tools">[] {
+  return connectors.filter(connector => skillKeys.has(skillKey(connector)) && connector.tools.length > 0)
+    .map(connector => ({ tools: connector.tools }));
 }
 
 export function isConnectorSkill(key: string) {

server/src/services/connectors/slack-authority.ts

@@ -187,7 +187,7 @@ export async function slackEndpointCandidates(
   binding: Pick<SlackTaskBinding, "companyId" | "agentId" | "endpointId">,
 ) {
   return db
-    .select({ id: chatEndpoints.id })
+    .select({ id: chatEndpoints.id, connectionId: chatEndpoints.connectionId })
     .from(chatEndpoints)
     .where(
       and(

server/src/services/connectors/slack-catalog.ts

@@ -1,4 +1,5 @@
 import { createHash } from "node:crypto";
+import { measureRunPhase } from "../execution-preparation-timing.js";
 import { and, eq, isNull, sql } from "drizzle-orm";
 import { SLACK_TOOLS } from "@paperclipai/shared";
 import {
@@ -19,6 +20,36 @@ import {
   resolveSlackTaskAuthority,
   slackEndpointCandidates,
 } from "./slack-authority.js";
+const catalogDefinitions = SLACK_TOOLS.map(tool => ({
+  tool,
+  hash: createHash("sha256").update(JSON.stringify({
+    name: tool.name,
+    description: tool.description,
+    inputSchema: tool.inputSchema,
+    risk: tool.risk,
+    scopes: tool.scopes,
+  })).digest("hex"),
+}));
+
+export async function slackBotCatalogIsCurrent(
+  db: Db,
+  endpoint: { id: string; companyId: string; connectionId: string },
+): Promise<boolean> {
+  const [entries, profiles] = await Promise.all([
+    db.select({ name: toolCatalogEntries.name, versionHash: toolCatalogEntries.versionHash })
+      .from(toolCatalogEntries).where(and(
+        eq(toolCatalogEntries.companyId, endpoint.companyId),
+        eq(toolCatalogEntries.connectionId, endpoint.connectionId),
+      )),
+    db.select({ id: toolProfiles.id }).from(toolProfiles).where(and(
+      eq(toolProfiles.companyId, endpoint.companyId),
+      eq(toolProfiles.profileKey, `slack-bot:${endpoint.id}`),
+    )).limit(1),
+  ]);
+  const hashes = new Map(entries.map(entry => [entry.name, entry.versionHash]));
+  return profiles.length > 0 && catalogDefinitions.every(({ tool, hash }) => hashes.get(`slack_bot:${tool.name}`) === hash);
+}
+
 export async function syncSlackBotTools(
   tx: Parameters<Parameters<Db["transaction"]>[0]>[0],
   endpoint: typeof chatEndpoints.$inferSelect,
@@ -35,28 +66,17 @@ export async function syncSlackBotTools(
       ),
     );
   if (!connection) return;
-  await tx.execute(
+  await measureRunPhase("connectors.slack.catalog.lock", () => tx.execute(
     sql`select pg_advisory_xact_lock(hashtextextended(${`slack-catalog:${endpoint.id}`}, 0))`,
-  );
-  const existingEntries = await tx
+  ));
+  const existingEntries = await measureRunPhase("connectors.slack.catalog.entries", () => tx
     .select()
     .from(toolCatalogEntries)
-    .where(eq(toolCatalogEntries.connectionId, connection.id));
+    .where(eq(toolCatalogEntries.connectionId, connection.id)));
   const entriesByName = new Map(
     existingEntries.map((entry) => [entry.name, entry]),
   );
-  for (const tool of SLACK_TOOLS) {
-    const hash = createHash("sha256")
-      .update(
-        JSON.stringify({
-          name: tool.name,
-          description: tool.description,
-          inputSchema: tool.inputSchema,
-          risk: tool.risk,
-          scopes: tool.scopes,
-        }),
-      )
-      .digest("hex");
+  for (const { tool, hash } of catalogDefinitions) {
     const values = {
       companyId: endpoint.companyId,
       applicationId: connection.applicationId,
@@ -79,7 +99,7 @@ export async function syncSlackBotTools(
     };
     const existing = entriesByName.get(values.name);
     if (existing?.versionHash === hash) continue;
-    await tx
+    await measureRunPhase("connectors.slack.catalog.write", () => tx
       .insert(toolCatalogEntries)
       .values(values)
       .onConflictDoUpdate({
@@ -94,10 +114,10 @@ export async function syncSlackBotTools(
               }
             : {}),
         },
-      });
+      }));
   }
   const profileKey = `slack-bot:${endpoint.id}`;
-  const [priorProfile] = await tx
+  const [priorProfile] = await measureRunPhase("connectors.slack.catalog.profile", () => tx
     .select()
     .from(toolProfiles)
     .where(
@@ -105,7 +125,7 @@ export async function syncSlackBotTools(
         eq(toolProfiles.companyId, endpoint.companyId),
         eq(toolProfiles.profileKey, profileKey),
       ),
-    );
+    ));
   if (priorProfile) return;
   const [profile] = await tx
     .insert(toolProfiles)

server/src/services/connectors/slack.ts

@@ -2,8 +2,10 @@ import { authorizeSlackDocument } from "./slack-document.js";
 import { slackDelivery } from "./slack-delivery.js";
 import { slackMessage } from "./slack-message.js";
 import { executeSlackWrite } from "./slack-writes.js";
-import { syncSlackBotTools } from "./slack-catalog.js";
+import { slackBotCatalogIsCurrent, syncSlackBotTools } from "./slack-catalog.js";
 import { SLACK_TOOLS } from "@paperclipai/shared";
+import { measureRunPhase } from "../execution-preparation-timing.js";
+import { measureRunTransaction } from "../execution-transaction-timing.js";
 import {
   toolProfiles,
   toolProfileBindings,
@@ -32,29 +34,25 @@ import {
 async function slackAssignedEndpoint(
   db: Db,
   binding: Partial<SlackTaskBinding> & { companyId: string; agentId: string },
+  endpoint: { id: string; connectionId: string },
 ) {
   if (!binding.runId || !binding.issueId) return [];
-  let authority: Awaited<ReturnType<typeof resolveSlackTaskAuthority>>;
-  try {
-    authority = await resolveSlackTaskAuthority(
-      db,
-      binding as SlackTaskBinding,
+  const authorize = async () => {
+    try {
+      return await measureRunPhase("connectors.slack.authority", () => resolveSlackTaskAuthority(db, binding as SlackTaskBinding));
+    } catch (error) {
+      if (error && typeof error === "object" && "status" in error && error.status === 403) return null;
+      throw error;
+    }
+  };
+  if (!await slackBotCatalogIsCurrent(db, { ...endpoint, companyId: binding.companyId })) {
+    const authority = await authorize();
+    if (!authority || authority.endpoint.connectionId !== endpoint.connectionId) return [];
+    await measureRunTransaction(db, "connectors.slack.catalog", (tx) =>
+      syncSlackBotTools(tx, authority.endpoint, authority.userId, true),
     );
-  } catch (error) {
-    if (
-      error &&
-      typeof error === "object" &&
-      "status" in error &&
-      error.status === 403
-    )
-      return [];
-    throw error;
   }
-  // Catalog updates are idempotent and preserve configured policies.
-  await db.transaction((tx) =>
-    syncSlackBotTools(tx, authority.endpoint, authority.userId, true),
-  );
-  const [enabledProfile] = await db
+  const [enabledProfile] = await measureRunPhase("connectors.slack.profile", () => db
     .select({ id: toolProfiles.id })
     .from(toolProfiles)
     .innerJoin(
@@ -70,20 +68,21 @@ async function slackAssignedEndpoint(
       toolProfileEntries,
       and(
         eq(toolProfileEntries.profileId, toolProfiles.id),
-        eq(toolProfileEntries.connectionId, authority.endpoint.connectionId),
+        eq(toolProfileEntries.connectionId, endpoint.connectionId),
         eq(toolProfileEntries.effect, "include"),
       ),
     )
     .where(
       and(
         eq(toolProfiles.companyId, binding.companyId),
-        eq(toolProfiles.profileKey, `slack-bot:${authority.endpoint.id}`),
+        eq(toolProfiles.profileKey, `slack-bot:${endpoint.id}`),
         eq(toolProfiles.status, "active"),
       ),
     )
-    .limit(1);
+    .limit(1));
   if (!enabledProfile) return [];
-  authority = await resolveSlackTaskAuthority(db, binding as SlackTaskBinding);
+  const authority = await authorize();
+  if (!authority || authority.endpoint.connectionId !== endpoint.connectionId) return [];
   return [
     {
       id: authority.endpoint.id,
@@ -106,12 +105,12 @@ export async function slackAssignedResource(
   binding: Partial<SlackTaskBinding> & { companyId: string; agentId: string },
 ) {
   const resources = [];
-  for (const endpoint of await slackEndpointCandidates(db, binding)) {
+  for (const endpoint of await measureRunPhase("connectors.slack.candidates", () => slackEndpointCandidates(db, binding))) {
     resources.push(
       ...(await slackAssignedEndpoint(db, {
         ...binding,
         endpointId: endpoint.id,
-      })),
+      }, endpoint)),
     );
   }
   return resources;

server/src/services/dot-runner-broker.ts

@@ -204,6 +204,15 @@ function createBroker(db: Db) {
         }
       });
     },
+    async canStartNewWork(companyId: string, agentId: string, bindingId: string): Promise<boolean> {
+      try {
+        await this.snapshot(companyId, agentId, bindingId);
+        return true;
+      } catch (error) {
+        if (error instanceof McpOAuthError) return false;
+        throw error;
+      }
+    },
     async snapshot(companyId: string, agentId: string, bindingId: string): Promise<DotBindingSnapshot> {
       const state = await this.bindingForAgent(companyId, agentId);
       if (!state || !await enabled(state.providerKind)) throw fail("External provider is disabled for new work.");

server/src/services/execution-blocker.ts

@@ -7,7 +7,7 @@ import { canContinueCancelledRun, readRunCancellation } from "./run-cancellation
 import { canRetryStoppedRun } from "./cancelled-native-startup.js";
 import { queuedCommentIdsFromWakePayload } from "./issue-queued-comment-queue.js";
 import { EXECUTION_RECONCILIATION_CAUSES, type ExecutionBlocker } from "@paperclipai/shared";
-import { hasRequiredWorkspaceRecovery, LEGACY_WORKSPACE_RECOVERY_SCHEMA } from "./workspace-restore-recovery-state.js";
+import { hasRequiredWorkspaceRecovery, LEGACY_WORKSPACE_RECOVERY_SCHEMA, nativeRunHasNoWorkspaceAccess } from "./workspace-restore-recovery-state.js";
 
 /** Resolved recovery bookkeeping can still carry an effective no-replay hold. */
 export function executionBlockerPredicate() {
@@ -65,6 +65,8 @@ export async function getExecutionBlocker(db: Db, companyId: string, issueId: st
   const [run] = runId ? await db.select().from(heartbeatRuns).where(and(
     eq(heartbeatRuns.companyId, companyId), eq(heartbeatRuns.id, runId),
   )).limit(1) : [];
+  const withoutWorkspace = Boolean(run?.driverKind === "grok_bot_mcp" && nativeRunHasNoWorkspaceAccess(run));
+  const workspaceRepairRequired = Boolean(restoreHold && !withoutWorkspace);
   const queue = await db.select({ payload: agentWakeupRequests.payload }).from(agentWakeupRequests).where(and(
     eq(agentWakeupRequests.companyId, companyId), eq(agentWakeupRequests.status, "deferred_issue_execution"),
     sql`${agentWakeupRequests.payload}->>'issueId' = ${issueId}`,
@@ -118,6 +120,16 @@ export async function getExecutionBlocker(db: Db, companyId: string, issueId: st
     nextAction += " Inspect the run before sending a new message to request continuation.";
   }
 
+  if (withoutWorkspace && (restoreHold || run?.status === "cancelled") && !run?.resultJson?.externalWorkspaceInspection) {
+    nextAction = run?.driverKind === "grok_bot_mcp"
+      ? "Inspect this Bot in Grok Bot, then clear the recovery hold and verify the connection in the agent's Runtime settings before assigning more work."
+      : "Inspect the stopped work in the external agent and its connection before requesting continuation.";
+  }
+
+  if (run?.resultJson?.externalWorkspaceInspection && !restoreHold) {
+    nextAction = "Verify the Grok connection, then send a new message to continue. The stopped work will not be replayed.";
+  }
+
   return {
     recoveryActionId: action.id,
     runId,
@@ -129,7 +141,7 @@ export async function getExecutionBlocker(db: Db, companyId: string, issueId: st
     runError: runError?.slice(0, 1024) ?? null,
     canContinue: eligibleContinuation && !chatBinding,
     canRetry: canRetry && !chatBinding,
-    ...(restoreHold ? { workspaceRepairRequired: true, canRecordWorkspaceRepair: hasRequiredWorkspaceRecovery(action.evidence) } : {}),
+    ...(restoreHold ? { workspaceRepairRequired, canRecordWorkspaceRepair: workspaceRepairRequired && hasRequiredWorkspaceRecovery(action.evidence) } : {}),
     savedMessageCount: saved.length,
   };
 }

server/src/services/execution-preparation-timing.ts

@@ -1,5 +1,7 @@
+import { AsyncLocalStorage } from "node:async_hooks";
+import { EXECUTION_PREPARATION_SPAN_ATTRS, runWithRuntimeParent } from "@paperclipai/adapter-utils/acpx-engine/startup-timing";
 import { createHash } from "node:crypto";
-import { isSpanContextValid, ROOT_CONTEXT, SpanStatusCode, trace } from "@opentelemetry/api";
+import { isSpanContextValid, ROOT_CONTEXT, SpanStatusCode, trace, type Context } from "@opentelemetry/api";
 
 type PreparationPhase = "environment_selection" | "execution_configuration" | "session"
   | "workspace" | "github_environment" | "github_launchers" | "run_output"
@@ -12,6 +14,10 @@ type PreparationPhase = "environment_selection" | "execution_configuration" | "s
   | "workspace_provision_policy"
   | "workspace_base_validation"
   | "workspace_source_authorization"
+  | "workspace_source_materialize"
+  | "workspace_ownership_read"
+  | "workspace_environment_acquire"
+  | "workspace_environment_realize"
   | "workspace_provision"
   | "workspace_source_retention"
   | "workspace_binding"
@@ -27,37 +33,190 @@ type PreparationPhase = "environment_selection" | "execution_configuration" | "s
   | "github_stop" | "coordinator_release" | "instruction_checkpoint"
   | "instruction_changes" | "warm_release" | "instruction_collection";
 
-export function createRunPreparationTiming(runId: string, scope: "heartbeat.prepare" | "native.controller" = "heartbeat.prepare") {
-  let runIdHash: string | undefined;
-  return <T>(phase: PreparationPhase, work: () => T | Promise<T>): Promise<T> => {
-    let operation: Promise<T> | undefined;
-    const run = (span?: { setStatus(status: { code: number }): void; end(): void }) => {
-      operation ??= (async () => {
-        try { return await work(); }
-        catch (error) {
-          try { span?.setStatus({ code: SpanStatusCode.ERROR }); } catch {}
-          throw error;
-        } finally {
-          try { span?.end(); } catch {}
-        }
-      })();
-      return operation;
+export type ExecutionPhase =
+  | "workspace.transaction" | "workspace.transaction.entry" | "workspace.transaction.commit" | "workspace.transaction.rollback"
+  | "workspace.identity_read" | "workspace.row_lock" | "workspace.provenance" | "workspace.notification" | "workspace.write"
+  | "workspace.sources" | "workspace.sources.canonicalize" | "workspace.sources.inventory" | "workspace.sources.alias_scan"
+  | "workspace.sources.repository_scan" | "workspace.sources.project_scan" | "workspace.provenance_lock"
+  | "workspace.readers" | "workspace.readers.identity" | "workspace.readers.trust" | "workspace.readers.projects" | "workspace.readers.sources"
+  | "workspace.access" | "workspace.access.privacy" | "workspace.access.inventory" | "workspace.access.authorization"
+  | "workspace.materialize.directories" | "workspace.materialize.source_identity"
+  | "workspace.checkout.lookup" | "workspace.checkout.git_auth" | "workspace.checkout.git_clone"
+  | "workspace.checkout.git_ref" | "workspace.checkout.publish"
+  | "runtime_selection.review" | "runtime_selection.contract_read" | "runtime_selection.contract_ensure"
+  | "runtime_selection.resume" | "runtime_selection.recovery_profile" | "runtime_selection.profile" | "runtime_selection.interactions"
+  | "runtime_selection.provider_discovery" | "runtime_selection.bootstrap" | "runtime_selection.run_write" | "runtime_selection.finalization_write"
+  | "runtime_selection.transaction" | "runtime_selection.transaction.entry" | "runtime_selection.transaction.commit" | "runtime_selection.transaction.rollback"
+  | "runtime_selection.row_lock" | "runtime_selection.workspace_sync"
+  | "runtime_context.instructions" | "runtime_context.skills" | "runtime_context.mcp"
+  | "runtime_context.asset_read" | "runtime_context.asset_manifest" | "runtime_context.asset_cache" | "runtime_context.asset_materialize" | "runtime_context.asset_verify"
+  | "workspace_sync.snapshot_git" | "workspace_sync.snapshot_baseline" | "workspace_sync.pack" | "workspace_sync.seed_verify" | "workspace_sync.stage_workspace" | "workspace_sync.restore"
+  | "workspace_sync.run_read" | "workspace_sync.descriptor_directory" | "workspace_sync.descriptor_read"
+  | "workspace_sync.prepare_host_current" | "workspace_sync.prepare_durable_seed" | "workspace_sync.prepare_adopt_remote"
+  | "workspace_sync.archive_hash" | "workspace_sync.warm_stamp" | "workspace_sync.durable_seed_verify"
+  | "workspace_sync.manifests_persist" | "workspace_sync.descriptor_write" | "workspace_sync.reference_write"
+  | "workspace_sync.source_root" | "workspace_sync.repositories" | "workspace_sync.ignore_rules"
+  | "workspace_sync.seed_generation" | "workspace_sync.seed_lookup" | "workspace_sync.seed_publish"
+  | "workspace_sync.git_clone" | "workspace_sync.git_archive" | "workspace_sync.seed_replay" | "workspace_sync.seed_persist"
+  | "workspace_sync.overlay_copy" | "workspace_sync.overlay_archive" | "workspace_sync.deleted_manifest"
+  | "workspace_sync.seed_extract" | "workspace_sync.seed_snapshot" | "workspace_sync.seed_cleanup"
+  | "workspace_sync.inbound_barrier" | "workspace_sync.transfer" | "workspace_sync.progress"
+  | "connectors.browser_use.resolve" | "connectors.speko.resolve" | "connectors.slack.resolve" | "connectors.agentmail.resolve"
+  | "connectors.browser_use.connection" | "connectors.browser_use.catalog" | "connectors.browser_use.grant"
+  | "connectors.discovery.scheduled" | "connectors.discovery.queue" | "connectors.discovery.build" | "connectors.discovery.agent"
+  | "connectors.discovery.connections" | "connectors.discovery.github" | "connectors.discovery.slack" | "connectors.discovery.speko"
+  | "connectors.discovery.policy" | "connectors.discovery.on_demand_policy"
+  | "connectors.slack.candidates" | "connectors.slack.authority" | "connectors.slack.profile"
+  | "connectors.slack.settings" | "connectors.slack.identity" | "connectors.slack.credentials" | "connectors.slack.revision"
+  | "connectors.slack.authority.transaction" | "connectors.slack.authority.transaction.entry" | "connectors.slack.authority.transaction.commit" | "connectors.slack.authority.transaction.rollback"
+  | "connectors.slack.catalog.transaction" | "connectors.slack.catalog.transaction.entry" | "connectors.slack.catalog.transaction.commit" | "connectors.slack.catalog.transaction.rollback"
+  | "connectors.slack.catalog.lock" | "connectors.slack.catalog.entries" | "connectors.slack.catalog.write" | "connectors.slack.catalog.profile"
+  | "execution.runtime" | "execution.workspace_ownership" | "execution.workspace_finalize" | "execution.workspace_restore" | "execution.workspace_commit"
+  | "execution.workspace_ownership_wait" | "execution.realtime"
+  | "execution.usage" | "execution.native_finalize"
+  | "completion.run" | "completion.output" | "completion.terminal_write" | "completion.cost" | "completion.liveness" | "completion.summary"
+  | "response.transaction" | "response.transaction.entry" | "response.transaction.commit" | "response.transaction.rollback" | "response.row_lock" | "response.write"
+  | "response.resolve" | "response.authorize" | "response.add_comment" | "response.activity" | "response.publish";
+
+type RunScope = { runId: string; parent?: Context; ended: boolean; details: Map<string, number> };
+const ownership = new AsyncLocalStorage<RunScope>();
+const endpointEnabled = () => Boolean(process.env.OTEL_EXPORTER_OTLP_ENDPOINT?.trim());
+
+export function getRunExecutionTraceContext(runId: string): Context | undefined {
+  const owned = ownership.getStore();
+  return endpointEnabled() && owned?.runId === runId && !owned.ended ? owned.parent : undefined;
+}
+
+function measure<T>(runId: string, name: string, work: () => T | Promise<T>, independent = false, counts: { fileCount?: number; bytes?: number } = {}): Promise<T> {
+  let operation: Promise<T> | undefined;
+  const run = (span?: { setStatus(status: { code: number }): void; end(): void }, parent?: Context) => {
+    if (operation) return operation;
+    const previous = ownership.getStore();
+    const owned: RunScope = { runId,
+      parent: parent ?? (!independent && previous?.runId === runId && !previous.ended ? previous.parent : undefined),
+      ended: false,
+      details: !independent && previous?.runId === runId ? previous.details : new Map<string, number>(),
     };
-    if (!process.env.OTEL_EXPORTER_OTLP_ENDPOINT?.trim()) return run();
-    try {
-      runIdHash ??= createHash("sha256").update(runId).digest("hex").slice(0, 12);
-      const trigger = trace.getActiveSpan()?.spanContext();
-      const traced = trace.getTracer(scope === "heartbeat.prepare" ? "paperclip.heartbeat-preparation" : "paperclip.native-controller").startActiveSpan(
-        `${scope}.${phase}`,
-        {
-          attributes: { "paperclip.task.run.run_id": runIdHash },
-          ...(trigger && isSpanContextValid(trigger) ? { links: [{ context: trigger }] } : {}),
-        },
-        ROOT_CONTEXT,
-        span => run(span),
-      );
-      void Promise.resolve(traced).catch(() => undefined);
-    } catch {}
-    return operation ?? run();
+    operation = Promise.resolve().then(() =>
+      ownership.run(owned, () => runWithRuntimeParent(owned.parent, work))).then(value => {
+        try { span?.end(); } catch {}
+        owned.ended = true;
+        return value;
+      }, error => {
+        try { span?.setStatus({ code: SpanStatusCode.ERROR }); } catch {}
+        try { span?.end(); } catch {}
+        owned.ended = true;
+        throw error;
+      });
+    return operation;
   };
+  if (!endpointEnabled()) return (async () => work())();
+  try {
+    const parent = independent ? undefined : getRunExecutionTraceContext(runId);
+    const trigger = parent ? undefined : trace.getActiveSpan()?.spanContext();
+    const traced = trace.getTracer(name.startsWith("heartbeat.prepare.") ? "paperclip.heartbeat-preparation"
+      : name.startsWith("native.controller.") ? "paperclip.native-controller" : "paperclip.execution").startActiveSpan(name, {
+      attributes: { [EXECUTION_PREPARATION_SPAN_ATTRS.runId]: createHash("sha256").update(runId).digest("hex").slice(0, 12),
+        ...Object.fromEntries((["fileCount", "bytes"] as const).flatMap(key =>
+          typeof counts[key] === "number" && Number.isSafeInteger(counts[key]) && counts[key]! >= 0
+            ? [[EXECUTION_PREPARATION_SPAN_ATTRS[key], counts[key]!]] : [])),
+      },
+      ...(trigger && isSpanContextValid(trigger) ? { links: [{ context: trigger }] } : {}),
+    }, parent ?? ROOT_CONTEXT, span => {
+      let childContext: Context | undefined;
+      try { childContext = trace.setSpan(parent ?? ROOT_CONTEXT, span); } catch {}
+      return run(span, childContext);
+    });
+    void Promise.resolve(traced).catch(() => undefined);
+  } catch {}
+  return operation ?? run();
+}
+
+export function withRunExecutionTiming<T>(runId: string, work: () => T | Promise<T>): Promise<T> {
+  return measure(runId, "task.execution_attempt", work, true);
+}
+
+export async function measureRunPhase<T>(phase: ExecutionPhase, work: () => T | Promise<T>, counts?: { fileCount?: number; bytes?: number }): Promise<T> {
+  const owned = ownership.getStore();
+  if (!endpointEnabled() || !owned || owned.ended) return work();
+  const count = owned.details.get(phase) ?? 0;
+  if (count >= 16) return work();
+  owned.details.set(phase, count + 1);
+  return measure(owned.runId, phase, work, false, counts);
+}
+
+export function beginRunPhase(phase: ExecutionPhase): (failed?: boolean) => void {
+  const owned = ownership.getStore();
+  if (!endpointEnabled() || !owned?.parent || owned.ended) return () => {};
+  const count = owned.details.get(phase) ?? 0;
+  if (count >= 16) return () => {};
+  owned.details.set(phase, count + 1);
+  try {
+    const span = trace.getTracer("paperclip.execution").startSpan(phase, {
+      attributes: { [EXECUTION_PREPARATION_SPAN_ATTRS.runId]: createHash("sha256").update(owned.runId).digest("hex").slice(0, 12) },
+    }, owned.parent);
+    let ended = false;
+    return failed => {
+      if (ended) return;
+      ended = true;
+      if (failed) { try { span.setStatus({ code: SpanStatusCode.ERROR }); } catch {} }
+      try { span.end(); } catch {}
+    };
+  } catch { return () => {}; }
+}
+
+export function measureToolDiscovery<T>(schedule: (work: () => Promise<T>) => Promise<T>, work: () => Promise<T>): Promise<T> {
+  return measureRunPhase("connectors.discovery.scheduled", async () => {
+    const queued = beginRunPhase("connectors.discovery.queue");
+    const build = AsyncLocalStorage.bind(() => {
+      queued();
+      return measureRunPhase("connectors.discovery.build", work);
+    });
+    let failed = false;
+    try { return await schedule(build); }
+    catch (error) { failed = true; throw error; }
+    finally { queued(failed); }
+  });
+}
+
+export function createRunPreparationTiming(runId: string, scope: "heartbeat.prepare" | "native.controller" = "heartbeat.prepare") {
+  return <T>(phase: PreparationPhase, work: () => T | Promise<T>): Promise<T> => measure(runId, `${scope}.${phase}`, work);
+}
+
+const workspaceSyncDetailPhases = {
+  "workspace.source_root": "workspace_sync.source_root",
+  "workspace.repositories": "workspace_sync.repositories",
+  "workspace.ignore_rules": "workspace_sync.ignore_rules",
+  "workspace.seed_generation": "workspace_sync.seed_generation",
+  "workspace.seed_lookup": "workspace_sync.seed_lookup",
+  "workspace.seed_publish": "workspace_sync.seed_publish",
+  "workspace.git_clone": "workspace_sync.git_clone",
+  "workspace.git_archive": "workspace_sync.git_archive",
+  "workspace.seed_replay": "workspace_sync.seed_replay",
+  "workspace.seed_persist": "workspace_sync.seed_persist",
+  "workspace.overlay_copy": "workspace_sync.overlay_copy",
+  "workspace.overlay_archive": "workspace_sync.overlay_archive",
+  "workspace.deleted_manifest": "workspace_sync.deleted_manifest",
+  "workspace.seed_extract": "workspace_sync.seed_extract",
+  "workspace.seed_snapshot": "workspace_sync.seed_snapshot",
+  "workspace.seed_cleanup": "workspace_sync.seed_cleanup",
+  "workspace.inbound_barrier": "workspace_sync.inbound_barrier",
+  "workspace.transfer": "workspace_sync.transfer",
+  "workspace.progress": "workspace_sync.progress",
+} as const satisfies Record<string, ExecutionPhase>;
+
+export function isWorkspaceSyncDetailPhase(name: string): boolean {
+  return Object.prototype.hasOwnProperty.call(workspaceSyncDetailPhases, name);
+}
+
+const workspaceSyncPhases: Readonly<Record<string, ExecutionPhase>> = {
+  ...workspaceSyncDetailPhases,
+  "snapshot.git": "workspace_sync.snapshot_git", "snapshot.baseline": "workspace_sync.snapshot_baseline",
+  pack: "workspace_sync.pack", "seed.verify": "workspace_sync.seed_verify",
+  "stage.workspace": "workspace_sync.stage_workspace", "restore.workspace": "workspace_sync.restore",
+  "workspace.restore": "execution.workspace_restore",
+};
+
+export function measureWorkspaceSyncPhase<T>(name: string, work: () => Promise<T>): Promise<T> {
+  return Object.prototype.hasOwnProperty.call(workspaceSyncPhases, name) ? measureRunPhase(workspaceSyncPhases[name]!, work) : work();
 }

server/src/services/execution-timing-context.test.ts

@@ -0,0 +1,241 @@
+import { getActiveStepContext } from "@paperclipai/adapter-utils/acpx-engine/startup-timing";
+import { createHash } from "node:crypto";
+import { context, trace, TraceFlags } from "@opentelemetry/api";
+import { AsyncLocalStorageContextManager } from "@opentelemetry/context-async-hooks";
+import { InMemorySpanExporter, SimpleSpanProcessor, TracerProvider } from "@opentelemetry/sdk-trace";
+import { afterEach, beforeEach, expect, it, vi } from "vitest";
+import { getStartupTraceContext, getStartupTracer } from "../instrumentation.js";
+import { prepareVerifiedRemoteProviderPack } from "./native-runtime/remote-provider-pack.js";
+import { createNativeRunTrace } from "./native-runtime/native-run-trace.js";
+import { beginRunPhase, createRunPreparationTiming, getRunExecutionTraceContext, measureRunPhase, measureToolDiscovery, withRunExecutionTiming } from "./execution-preparation-timing.js";
+import { createToolDiscoveryScheduler } from "./tool-discovery-scheduler.js";
+
+let exporter: InMemorySpanExporter;
+let provider: TracerProvider;
+let manager: AsyncLocalStorageContextManager;
+beforeEach(() => {
+  vi.stubEnv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://collector.test");
+  exporter = new InMemorySpanExporter();
+  provider = new TracerProvider({ spanProcessors: [new SimpleSpanProcessor({ exporter })] });
+  manager = new AsyncLocalStorageContextManager().enable();
+  expect(context.setGlobalContextManager(manager)).toBe(true);
+  expect(trace.setGlobalTracerProvider(provider)).toBe(true);
+});
+afterEach(async () => {
+  vi.restoreAllMocks();
+  vi.unstubAllEnvs();
+  trace.disable();
+  context.disable();
+  manager.disable();
+  await provider.shutdown();
+});
+
+it("parents overlapping preparation and response work to their own execution attempts independently of ingress sampling", async () => {
+  const trigger = { traceId: "1".repeat(32), spanId: "2".repeat(16), traceFlags: TraceFlags.NONE, isRemote: true };
+  await context.with(trace.setSpanContext(context.active(), trigger), () => Promise.all(["one", "two"].map(runId =>
+    withRunExecutionTiming(runId, async () => {
+      await createRunPreparationTiming(runId)("workspace", () => Promise.all([
+        measureRunPhase("workspace.sources", async () => {
+          await Promise.resolve();
+          await measureRunPhase("workspace.sources.alias_scan", () => {
+            const parent = getActiveStepContext()?.parentContext;
+            expect(parent).toBeDefined();
+            expect(trace.getSpan(parent as import("@opentelemetry/api").Context)?.spanContext()).toEqual(trace.getActiveSpan()?.spanContext());
+            expect(getRunExecutionTraceContext(runId === "one" ? "two" : "one")).toBeUndefined();
+          });
+        }),
+        measureRunPhase("workspace.readers", async () => { await Promise.resolve(); }),
+      ]));
+      await measureRunPhase("response.add_comment", () => undefined);
+    }),
+  )));
+  const spans = exporter.getFinishedSpans();
+  const attempts = spans.filter(span => span.name === "task.execution_attempt");
+  expect(attempts).toHaveLength(2);
+  expect(new Set(attempts.map(span => span.spanContext().traceId)).size).toBe(2);
+  for (const runId of ["one", "two"]) {
+    const attempt = attempts.find(span => span.attributes["paperclip.task.run.run_id"] === createHash("sha256").update(runId).digest("hex").slice(0, 12))!;
+    expect(attempt.parentSpanContext).toBeUndefined();
+    expect(attempt.links[0].context).toEqual(trigger);
+    const owned = spans.filter(span => span.spanContext().traceId === attempt.spanContext().traceId);
+    const phase = owned.find(span => span.name === "heartbeat.prepare.workspace")!;
+    const source = owned.find(span => span.name === "workspace.sources")!;
+    const aliases = owned.find(span => span.name === "workspace.sources.alias_scan")!;
+    const readers = owned.find(span => span.name === "workspace.readers")!;
+    const reply = owned.find(span => span.name === "response.add_comment")!;
+    expect(phase.parentSpanContext?.spanId).toBe(attempt.spanContext().spanId);
+    expect(source.parentSpanContext?.spanId).toBe(phase.spanContext().spanId);
+    expect(readers.parentSpanContext?.spanId).toBe(phase.spanContext().spanId);
+    expect(aliases.parentSpanContext?.spanId).toBe(source.spanContext().spanId);
+    expect(reply.parentSpanContext?.spanId).toBe(attempt.spanContext().spanId);
+    const ns = (time: readonly [number, number]) => time[0] * 1e9 + time[1];
+    expect(ns(readers.startTime)).toBeLessThanOrEqual(ns(source.endTime));
+    expect(ns(source.startTime)).toBeLessThanOrEqual(ns(readers.endTime));
+    expect(owned).toHaveLength(6);
+  }
+  expect(getRunExecutionTraceContext("one")).toBeUndefined();
+});
+
+it("uses a link for a native session whose queue history predates the current attempt", async () => {
+  await withRunExecutionTiming("native", async () => {
+    const native = createNativeRunTrace({ runId: "native", startedAtMs: Date.now() - 10_000 });
+    await native.measure("agent.turn", async () => {});
+    await native.finish("ok");
+    await measureRunPhase("response.add_comment", () => undefined);
+  });
+  const spans = exporter.getFinishedSpans();
+  const attempt = spans.find(span => span.name === "task.execution_attempt")!;
+  const session = spans.find(span => span.name === "task.provider_session")!;
+  const response = spans.find(span => span.name === "response.add_comment")!;
+  expect(session.parentSpanContext).toBeUndefined();
+  expect(session.links.map(link => link.context.spanId)).toContain(attempt.spanContext().spanId);
+  expect(response.parentSpanContext?.spanId).toBe(attempt.spanContext().spanId);
+  expect(session.startTime[0]).toBeLessThan(attempt.startTime[0]);
+});
+
+it("keeps queued discovery under its initiating run when another run releases the slot", async () => {
+  const scheduler = createToolDiscoveryScheduler(1, 1);
+  let release!: () => void, started!: () => void, enqueued!: () => void;
+  const gate = new Promise<void>(resolve => { release = resolve; });
+  const ready = new Promise<void>(resolve => { started = resolve; });
+  const waiting = new Promise<void>(resolve => { enqueued = resolve; });
+  const first = withRunExecutionTiming("discovery-owner", () => measureToolDiscovery(
+    work => scheduler.run(work), async () => { started(); await gate; },
+  ));
+  await ready;
+  const runHash = createHash("sha256").update("discovery-waiter").digest("hex").slice(0, 12);
+  const value = {};
+  const work = vi.fn(async () => {
+    expect(getRunExecutionTraceContext("discovery-owner")).toBeUndefined();
+    expect(getRunExecutionTraceContext("discovery-waiter")).toBeDefined();
+    expect(exporter.getFinishedSpans().filter(span => span.name === "connectors.discovery.queue"
+      && span.attributes["paperclip.task.run.run_id"] === runHash)).toHaveLength(1);
+    await measureRunPhase("connectors.discovery.policy", () => undefined);
+    return value;
+  });
+  const second = withRunExecutionTiming("discovery-waiter", () => measureToolDiscovery(
+    work => { const pending = scheduler.run(work); enqueued(); return pending; }, work,
+  ));
+  await waiting;
+  try {
+    expect(work).not.toHaveBeenCalled();
+    expect(exporter.getFinishedSpans().some(span => span.attributes["paperclip.task.run.run_id"] === runHash)).toBe(false);
+  } finally {
+    release();
+    await first;
+  }
+  await expect(second).resolves.toBe(value);
+  expect(work).toHaveBeenCalledOnce();
+  const spans = exporter.getFinishedSpans();
+  const owned = spans.filter(span => span.attributes["paperclip.task.run.run_id"] === runHash);
+  const attempt = owned.find(span => span.name === "task.execution_attempt")!;
+  const scheduled = owned.find(span => span.name === "connectors.discovery.scheduled")!;
+  const build = owned.find(span => span.name === "connectors.discovery.build")!;
+  const queue = owned.filter(span => span.name === "connectors.discovery.queue");
+  expect(queue).toHaveLength(1);
+  expect(owned).toHaveLength(5);
+  expect(owned.every(span => span.spanContext().traceId === attempt.spanContext().traceId)).toBe(true);
+  expect(scheduled.parentSpanContext?.spanId).toBe(attempt.spanContext().spanId);
+  expect(build.parentSpanContext?.spanId).toBe(scheduled.spanContext().spanId);
+  expect(queue[0].parentSpanContext?.spanId).toBe(scheduled.spanContext().spanId);
+  expect(owned.find(span => span.name === "connectors.discovery.policy")?.parentSpanContext?.spanId).toBe(build.spanContext().spanId);
+});
+
+it("caps repeated detail spans and ends failed scopes without content or exception text", async () => {
+  const failure = new Error("private path and credential");
+  const work = vi.fn(() => undefined);
+  await expect(withRunExecutionTiming("bounded", async () => {
+    for (let index = 0; index < 100; index++) await measureRunPhase("workspace.readers.identity", work);
+    for (let index = 0; index < 100; index++) beginRunPhase("workspace.transaction.entry")();
+    await measureRunPhase("workspace.readers.trust", () => { throw failure; });
+  })).rejects.toBe(failure);
+  expect(work).toHaveBeenCalledTimes(100);
+  const spans = exporter.getFinishedSpans();
+  expect(spans.filter(span => span.name === "workspace.readers.identity")).toHaveLength(16);
+  expect(spans.filter(span => span.name === "workspace.transaction.entry")).toHaveLength(16);
+  for (const span of spans) expect(Object.keys(span.attributes)).toEqual(["paperclip.task.run.run_id"]);
+  expect(spans.filter(span => span.status.code === 2).map(span => span.name).sort()).toEqual(["task.execution_attempt", "workspace.readers.trust"]);
+  expect(JSON.stringify(spans.map(span => ({ attributes: span.attributes, events: span.events, status: span.status })))).not.toContain(failure.message);
+  expect(getRunExecutionTraceContext("bounded")).toBeUndefined();
+});
+
+it.each([undefined, "", "  "])("does not consult tracers or warn without endpoint %s even with a registered provider", async endpoint => {
+  vi.stubEnv("OTEL_EXPORTER_OTLP_ENDPOINT", endpoint);
+  const lookup = vi.spyOn(trace, "getTracer");
+  const warnings = (["log", "info", "warn", "error", "debug"] as const).map(method => vi.spyOn(console, method));
+  await expect(withRunExecutionTiming("off", () => measureRunPhase("response.add_comment", () => 42))).resolves.toBe(42);
+  getStartupTracer().startSpan("disabled").end();
+  getStartupTraceContext().tracer.startSpan("disabled").end();
+  const native = createNativeRunTrace({ runId: "off" });
+  await native.measure("agent.turn", async () => undefined);
+  await native.finish("ok");
+  expect(lookup).not.toHaveBeenCalled();
+  expect(exporter.getFinishedSpans()).toEqual([]);
+  for (const warning of warnings) expect(warning).not.toHaveBeenCalled();
+});
+
+it.each(["staged", "preinstalled", "uploaded", "failed"] as const)("records provider-pack selection %s with closed source and fallback reasons", async source => {
+  const failure = new Error("private manifest path and hash");
+  const native = createNativeRunTrace({ runId: "pack" });
+  const select = prepareVerifiedRemoteProviderPack({
+    trace: native,
+    verifyStaged: async () => { if (source !== "staged") throw failure; },
+    usePreinstalled: async () => source === "preinstalled",
+    stageAndVerify: async () => { if (source === "failed") throw failure; },
+  });
+  if (source === "failed") await expect(select).rejects.toBe(failure);
+  else await expect(select).resolves.toBe(source);
+  await native.finish(source === "failed" ? "failed" : "ok");
+  const span = exporter.getFinishedSpans().find(span => span.name === "provider_pack.select")!;
+  expect(span.status.code).toBe(source === "failed" ? 2 : 0);
+  if (source !== "failed") expect(span.attributes["paperclip.native.span.stateSource"]).toBe(source);
+  expect(JSON.stringify({ attributes: span.attributes, events: span.events })).not.toContain(failure.message);
+});
+
+it("exports only the declared count attributes and rejects unsafe numbers or extra fields", async () => {
+  await withRunExecutionTiming("counts", async () => {
+    await measureRunPhase("runtime_context.asset_materialize", () => undefined,
+      { fileCount: 3, bytes: 99, privatePath: "/private/fixture" } as { fileCount: number; bytes: number });
+    await measureRunPhase("runtime_context.asset_verify", () => undefined, { fileCount: -1, bytes: Infinity });
+  });
+  const spans = exporter.getFinishedSpans();
+  const materialize = spans.find(span => span.name === "runtime_context.asset_materialize")!;
+  expect(materialize.attributes).toEqual({
+    "paperclip.task.run.run_id": createHash("sha256").update("counts").digest("hex").slice(0, 12),
+    "paperclip.execution.fileCount": 3,
+    "paperclip.execution.bytes": 99,
+  });
+  const verify = spans.find(span => span.name === "runtime_context.asset_verify")!;
+  expect(Object.keys(verify.attributes)).toEqual(["paperclip.task.run.run_id"]);
+});
+
+it("does not inherit another run when a new attempt's tracer lookup fails", async () => {
+  await withRunExecutionTiming("outer", async () => {
+    const lookup = vi.spyOn(trace, "getTracer").mockImplementationOnce(() => { throw new Error("tracer unavailable"); });
+    try {
+      await withRunExecutionTiming("inner", () => measureRunPhase("workspace.sources", () => 42));
+    } finally { lookup.mockRestore(); }
+  });
+  const spans = exporter.getFinishedSpans();
+  const outer = spans.find(span => span.name === "task.execution_attempt")!;
+  const inner = spans.find(span => span.name === "workspace.sources")!;
+  expect(inner.parentSpanContext).toBeUndefined();
+  expect(inner.spanContext().traceId).not.toBe(outer.spanContext().traceId);
+  expect(inner.attributes["paperclip.task.run.run_id"]).toBe(createHash("sha256").update("inner").digest("hex").slice(0, 12));
+});
+
+it("does not attach native children to an ambient run when native context construction fails", async () => {
+  await withRunExecutionTiming("ambient", async () => {
+    const native = createNativeRunTrace({ runId: "native-context-failure", traceContext: {
+      tracer: getStartupTracer("broken-native-context"),
+      contextWithSpan() { throw new Error("context unavailable"); },
+    } });
+    await native.measure("agent.turn", async () => undefined);
+    await native.finish("ok");
+  });
+  const spans = exporter.getFinishedSpans();
+  const ambient = spans.find(span => span.name === "task.execution_attempt")!;
+  const child = spans.find(span => span.name === "agent.turn")!;
+  expect(child.parentSpanContext).toBeUndefined();
+  expect(child.spanContext().traceId).not.toBe(ambient.spanContext().traceId);
+});

server/src/services/execution-transaction-timing.test.ts

@@ -0,0 +1,127 @@
+import { context, trace } from "@opentelemetry/api";
+import { AsyncLocalStorageContextManager } from "@opentelemetry/context-async-hooks";
+import { InMemorySpanExporter, SimpleSpanProcessor, TracerProvider } from "@opentelemetry/sdk-trace";
+import { companies, createDb, executionWorkspaces, issueComments, issues, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS } from "@paperclipai/db";
+import { eq, sql } from "drizzle-orm";
+import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
+import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../__tests__/helpers/embedded-postgres.js";
+import { measureRunTransaction } from "./execution-transaction-timing.js";
+import { withRunExecutionTiming } from "./execution-preparation-timing.js";
+import { issueService } from "./issues.js";
+import { logActivity } from "./activity-log.js";
+import { executionWorkspaceService } from "./execution-workspaces.js";
+import { lockExecutionWorkspaceRootProvenance } from "./execution-workspace-repositories.js";
+import { assertTaskWorkspaceAccess } from "./task-workspace-source-access.js";
+
+const support = await getEmbeddedPostgresTestSupport();
+(support.supported ? describe : describe.skip)("execution timing with real database waits", () => {
+  let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
+  let db: ReturnType<typeof createDb>;
+  let companyId: string, workspaceId: string;
+  let exporter: InMemorySpanExporter, provider: TracerProvider, manager: AsyncLocalStorageContextManager;
+  beforeAll(async () => {
+    temporary = await startEmbeddedPostgresTestDatabase("paperclip-execution-timing-");
+    db = createDb(temporary.connectionString);
+    const [company] = await db.insert(companies).values({ name: "Timing fixture", issuePrefix: "TIME" }).returning();
+    companyId = company.id;
+    const [workspace] = await db.insert(executionWorkspaces).values({ companyId, name: "Files", mode: "shared_workspace", strategyType: "task_directory" }).returning();
+    workspaceId = workspace.id;
+  }, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS);
+  beforeEach(() => {
+    vi.stubEnv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://collector.test");
+    exporter = new InMemorySpanExporter();
+    provider = new TracerProvider({ spanProcessors: [new SimpleSpanProcessor({ exporter })] });
+    manager = new AsyncLocalStorageContextManager().enable();
+    expect(context.setGlobalContextManager(manager)).toBe(true);
+    expect(trace.setGlobalTracerProvider(provider)).toBe(true);
+  });
+  afterEach(async () => {
+    vi.unstubAllEnvs();
+    trace.disable(); context.disable(); manager.disable();
+    await provider.shutdown();
+  });
+  afterAll(async () => { await temporary?.cleanup(); });
+
+  it.each(["advisory", "row"] as const)("attributes a controlled %s lock wait to its leaf and commits the original update", async kind => {
+    let release!: () => void, acquired!: () => void;
+    const released = new Promise<void>(resolve => { release = resolve; });
+    const ready = new Promise<void>(resolve => { acquired = resolve; });
+    const blocker = db.transaction(async tx => {
+      if (kind === "advisory") await lockExecutionWorkspaceRootProvenance(tx, companyId);
+      else await tx.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspaceId)).for("update");
+      acquired();
+      await released;
+    });
+    await ready;
+    const update = withRunExecutionTiming(`wait-${kind}`, () => executionWorkspaceService(db).update(workspaceId, { metadata: { verified: kind } }));
+    try {
+      await vi.waitFor(async () => {
+        const pending = await db.execute(sql`select count(*)::int as count from pg_locks where not granted and locktype = ${kind === "advisory" ? "advisory" : "transactionid"}`);
+        expect(pending[0]?.count).toBeGreaterThan(0);
+      });
+      expect(exporter.getFinishedSpans().some(span => span.name === (kind === "advisory" ? "workspace.provenance_lock" : "workspace.row_lock"))).toBe(false);
+    } finally {
+      release();
+      await blocker;
+      await update;
+    }
+    const [saved] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspaceId));
+    expect(saved.metadata).toMatchObject({ verified: kind });
+    const spans = exporter.getFinishedSpans();
+    const transaction = spans.find(span => span.name === "workspace.transaction")!;
+    const wait = spans.find(span => span.name === (kind === "advisory" ? "workspace.provenance_lock" : "workspace.row_lock"))!;
+    expect(wait.parentSpanContext?.spanId).toBe(transaction.spanContext().spanId);
+    expect(wait.duration[0] * 1e9 + wait.duration[1]).toBeGreaterThan(0);
+    expect(spans.map(span => span.name)).toEqual(expect.arrayContaining(["workspace.transaction.entry", "workspace.transaction.commit", "workspace.notification", "workspace.write"]));
+    expect(spans.some(span => span.name === "workspace.transaction.rollback")).toBe(false);
+    expect(JSON.stringify(spans.map(span => span.attributes))).not.toContain(workspaceId);
+  });
+
+  it("preserves transaction rollback and the original error without replaying writes", async () => {
+    const [before] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspaceId));
+    const failure = new Error("private rollback fixture");
+    let calls = 0;
+    await expect(withRunExecutionTiming("rollback", () => measureRunTransaction(db, "workspace", async tx => {
+      calls++;
+      await tx.update(executionWorkspaces).set({ metadata: { rolledBack: true } }).where(eq(executionWorkspaces.id, workspaceId));
+      throw failure;
+    }))).rejects.toBe(failure);
+    expect(calls).toBe(1);
+    const [after] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspaceId));
+    expect(after.metadata).toEqual(before.metadata);
+    const spans = exporter.getFinishedSpans();
+    expect(spans.find(span => span.name === "workspace.transaction.rollback")?.status.code).toBe(2);
+    expect(spans.some(span => span.name === "workspace.transaction.commit")).toBe(false);
+    expect(JSON.stringify(spans.map(span => ({ attributes: span.attributes, events: span.events })))).not.toContain(failure.message);
+  });
+
+  it("records durable response commit and server publication under the same attempt", async () => {
+    const [issue] = await db.insert(issues).values({ companyId, title: "Response timing" }).returning();
+    const body = "Private response fixture";
+    const comment = await withRunExecutionTiming("response", async () => {
+      const saved = await issueService(db).addComment(issue.id, body, { userId: "local-board" });
+      await logActivity(db, { companyId, actorType: "user", actorId: "local-board", action: "issue.comment_added", entityType: "issue", entityId: issue.id });
+      return saved;
+    });
+    const [stored] = await db.select().from(issueComments).where(eq(issueComments.id, comment.id));
+    expect(stored.body).toBe(body);
+    const spans = exporter.getFinishedSpans();
+    const attempt = spans.find(span => span.name === "task.execution_attempt")!;
+    const transaction = spans.find(span => span.name === "response.transaction")!;
+    const commit = spans.find(span => span.name === "response.transaction.commit")!;
+    expect(transaction.parentSpanContext?.spanId).toBe(attempt.spanContext().spanId);
+    expect(commit.parentSpanContext?.spanId).toBe(transaction.spanContext().spanId);
+    expect(spans.map(span => span.name)).toEqual(expect.arrayContaining(["response.row_lock", "response.write", "execution.realtime"]));
+    expect(spans.every(span => span.spanContext().traceId === attempt.spanContext().traceId)).toBe(true);
+    expect(JSON.stringify(spans.map(span => span.attributes))).not.toContain(body);
+  });
+
+  it("preserves a real access denial and exports only failed status", async () => {
+    await expect(withRunExecutionTiming("denied", () => assertTaskWorkspaceAccess(db,
+      { type: "board", userId: "not-a-company-member" }, companyId, workspaceId,
+    ))).rejects.toMatchObject({ status: 403 });
+    const spans = exporter.getFinishedSpans();
+    expect(spans.find(span => span.name === "task.execution_attempt")?.status.code).toBe(2);
+    expect(JSON.stringify(spans.map(span => ({ attributes: span.attributes, events: span.events })))).not.toContain("not-a-company-member");
+  });
+});

server/src/services/execution-transaction-timing.ts

@@ -0,0 +1,31 @@
+import type { Db } from "@paperclipai/db";
+import { beginRunPhase, measureRunPhase } from "./execution-preparation-timing.js";
+
+type Transaction = Parameters<Parameters<Db["transaction"]>[0]>[0];
+
+export function measureRunTransaction<T>(db: Db, scope: "workspace" | "runtime_selection" | "response" | "connectors.slack.authority" | "connectors.slack.catalog", work: (tx: Transaction) => Promise<T>): Promise<T> {
+  return measureRunPhase(`${scope}.transaction`, async () => {
+    const entry = beginRunPhase(`${scope}.transaction.entry`);
+    let settle: ((failed?: boolean) => void) | undefined;
+    let failed = false;
+    try {
+      return await db.transaction(async tx => {
+        entry();
+        try {
+          const value = await work(tx);
+          settle = beginRunPhase(`${scope}.transaction.commit`);
+          return value;
+        } catch (error) {
+          settle = beginRunPhase(`${scope}.transaction.rollback`);
+          throw error;
+        }
+      });
+    } catch (error) {
+      failed = true;
+      throw error;
+    } finally {
+      entry(failed);
+      settle?.(failed);
+    }
+  });
+}

server/src/services/execution-workspace-repositories.ts

@@ -1,3 +1,5 @@
+import { measureRunTransaction } from "./execution-transaction-timing.js";
+import { measureRunPhase } from "./execution-preparation-timing.js";
 import { createHash, randomUUID } from "node:crypto";
 import { execFile as execFileCallback } from "node:child_process";
 import { isDeepStrictEqual, promisify } from "node:util";
@@ -20,13 +22,34 @@ import { normalizeProjectRepositoryUrl } from "./project-repositories.js";
 import { buildGitAuthInvocation, createGitRemoteAuthProvider } from "./git-credentials.js";
 import { materializeManagedProjectWorkspace, ensureManagedRepositoriesIgnored } from "./managed-repository-checkout.js";
 
-import { canonicalRepositoryRoot, repositoryRootsOverlap, legacyRepositoryRootMayMatch, configuredWorkspaceRoot, verifiedIsolatedWorktree, unusedIsolatedCopyTarget, isolatedWorkspaceSourceRoots, type WorkspaceSourceCopyKind } from "./execution-workspace-source-roots.js";
+import { canonicalRepositoryRoot, resolveRepositoryRoots, repositoryRootsOverlap, legacyRepositoryRootMayMatch, configuredWorkspaceRoot, verifiedIsolatedWorktree, unusedIsolatedCopyTarget, isolatedWorkspaceSourceRoots, type WorkspaceSourceCopyKind } from "./execution-workspace-source-roots.js";
 export { canonicalRepositoryRoot } from "./execution-workspace-source-roots.js";
 
 const execFile = promisify(execFileCallback);
 type RepositoryRow = typeof executionWorkspaceRepositories.$inferSelect;
 type DbTransaction = Parameters<Parameters<Db["transaction"]>[0]>[0];
 
+function repositoryAccessKeys(entries: Array<Pick<typeof executionWorkspaceRepositoryAccess.$inferSelect, "kind" | "scopeId">>) {
+  return new Set(entries.map(entry => `${entry.kind}:${entry.scopeId}`));
+}
+
+async function retainRepositorySources(tx: DbTransaction,
+  root: { companyId: string; canonicalCwd: string; rootKey: string },
+  existingKeys: Set<string>,
+  groups: ReadonlyArray<readonly [typeof executionWorkspaceRepositoryAccess.$inferSelect["kind"], Iterable<string>]>,
+): Promise<void> {
+  const additions: Array<typeof executionWorkspaceRepositoryAccess.$inferInsert> = [];
+  for (const [kind, ids] of groups) for (const scopeId of ids) {
+    const key = `${kind}:${scopeId}`;
+    if (existingKeys.has(key)) continue;
+    additions.push({ ...root, kind, scopeId });
+    existingKeys.add(key);
+  }
+  for (let offset = 0; offset < additions.length; offset += 500) {
+    await measureRunPhase("workspace.write", () => tx.insert(executionWorkspaceRepositoryAccess).values(additions.slice(offset, offset + 500)));
+  }
+}
+
 const controllerAcquisitionMessage = "New controller repository acquisition requires a trusted task. Existing verified public checkouts remain available.";
 const lowTrustRepositoryMessage = "Low-trust repository reuse permits anonymous public URLs only; new acquisition is unavailable. Use an authorized configured source for private repositories.";
 const rootReaderRunColumns = {
@@ -65,11 +88,24 @@ export function assertRepositoryRetryMatches(row: Pick<RepositoryRow, "repositor
 async function resolveRepositoryRootSources(tx: Db | DbTransaction, input: {
   companyId: string; cwd: string; sourceWorkspaceId?: string; inspectCurrentAliases?: boolean; preparingCopyProjectIds?: Set<string>;
 }) {
-  const canonicalCwd = await canonicalRepositoryRoot(input.cwd);
+  const canonicalCwd = await measureRunPhase("workspace.sources.canonicalize", () => canonicalRepositoryRoot(input.cwd));
   const rootKey = createHash("sha256").update(canonicalCwd).digest("hex");
-  const companyEntries = await tx.select().from(executionWorkspaceRepositoryAccess)
-    .where(eq(executionWorkspaceRepositoryAccess.companyId, input.companyId));
+  const companyEntries = await measureRunPhase("workspace.sources.inventory", async () => {
+    const identities = await tx.selectDistinct({ rootKey: executionWorkspaceRepositoryAccess.rootKey,
+      canonicalCwd: executionWorkspaceRepositoryAccess.canonicalCwd }).from(executionWorkspaceRepositoryAccess)
+      .where(eq(executionWorkspaceRepositoryAccess.companyId, input.companyId));
+    const rootKeys = [...new Set([rootKey, ...identities.filter(identity => repositoryRootsOverlap(identity.canonicalCwd, canonicalCwd))
+      .map(identity => identity.rootKey)])];
+    const retained: Array<typeof executionWorkspaceRepositoryAccess.$inferSelect> = [];
+    for (let offset = 0; offset < rootKeys.length; offset += 500) {
+      retained.push(...await tx.select().from(executionWorkspaceRepositoryAccess).where(and(
+        eq(executionWorkspaceRepositoryAccess.companyId, input.companyId),
+        inArray(executionWorkspaceRepositoryAccess.rootKey, rootKeys.slice(offset, offset + 500)))));
+    }
+    return retained;
+  });
   const entries = companyEntries.filter(entry => entry.rootKey === rootKey);
+  const entryKeys = repositoryAccessKeys(entries);
   const overlappingEntries = companyEntries.filter(entry => repositoryRootsOverlap(entry.canonicalCwd, canonicalCwd));
   if (entries.some(entry => entry.canonicalCwd !== canonicalCwd)
     || overlappingEntries.some(entry => createHash("sha256").update(entry.canonicalCwd).digest("hex") !== entry.rootKey)) {
@@ -94,22 +130,22 @@ async function resolveRepositoryRootSources(tx: Db | DbTransaction, input: {
   const isolatedSourceIds = new Set(overlappingEntries.filter(entry => entry.kind === "isolated_source").map(entry => entry.scopeId));
   const sourceIds = new Set([...sharedSourceIds, ...isolatedSourceIds]);
   if (input.sourceWorkspaceId) { sourceIds.add(input.sourceWorkspaceId); sharedSourceIds.add(input.sourceWorkspaceId); addSharedSource(input.sourceWorkspaceId, canonicalCwd); }
-  const workspaces = await tx.select({ id: executionWorkspaces.id, cwd: executionWorkspaces.cwd }).from(executionWorkspaces)
-    .where(eq(executionWorkspaces.companyId, input.companyId));
-  for (const workspace of workspaces) {
-    if (!workspace.cwd) continue;
-    const peerRoot = await canonicalRepositoryRoot(workspace.cwd).catch((error: NodeJS.ErrnoException) => {
-      if (["ENOTDIR", "EACCES", "EPERM", "ELOOP", "ENAMETOOLONG"].includes(error.code ?? "")) return null;
-      throw error;
-    });
-    if (peerRoot && repositoryRootsOverlap(peerRoot, canonicalCwd)) {
-      sourceIds.add(workspace.id); sharedSourceIds.add(workspace.id); addSharedSource(workspace.id, peerRoot);
+  const workspaces = await measureRunPhase("workspace.sources.inventory", async () => tx.select({ id: executionWorkspaces.id, cwd: executionWorkspaces.cwd }).from(executionWorkspaces)
+    .where(eq(executionWorkspaces.companyId, input.companyId)));
+  await measureRunPhase("workspace.sources.alias_scan", async () => {
+    const workspaceRoots = await resolveRepositoryRoots(workspaces.flatMap(workspace => workspace.cwd ? [workspace.cwd] : []));
+    for (const workspace of workspaces) {
+      if (!workspace.cwd) continue;
+      const peerRoot = workspaceRoots.get(workspace.cwd);
+      if (peerRoot && repositoryRootsOverlap(peerRoot, canonicalCwd)) {
+        sourceIds.add(workspace.id); sharedSourceIds.add(workspace.id); addSharedSource(workspace.id, peerRoot);
+      }
     }
-  }
-  const inventory = await tx.select({ id: executionWorkspaceRepositories.id, workspaceId: executionWorkspaces.id, cwd: executionWorkspaces.cwd,
+  });
+  const inventory = await measureRunPhase("workspace.sources.inventory", async () => tx.select({ id: executionWorkspaceRepositories.id, workspaceId: executionWorkspaces.id, cwd: executionWorkspaces.cwd,
     identity: executionWorkspaceRepositories.repositoryIdentity }).from(executionWorkspaceRepositories)
     .innerJoin(executionWorkspaces, eq(executionWorkspaces.id, executionWorkspaceRepositories.executionWorkspaceId))
-    .where(and(eq(executionWorkspaceRepositories.companyId, input.companyId), eq(executionWorkspaces.companyId, input.companyId)));
+    .where(and(eq(executionWorkspaceRepositories.companyId, input.companyId), eq(executionWorkspaces.companyId, input.companyId))));
   const namespace = path.join(canonicalCwd, ".paperclip-repositories");
   const namespaceEntry = await fs.lstat(namespace).catch((error: NodeJS.ErrnoException) => {
     if (error.code === "ENOENT") return null;
@@ -122,37 +158,38 @@ async function resolveRepositoryRootSources(tx: Db | DbTransaction, input: {
     }
     for (const name of await fs.readdir(namespace)) if (name.startsWith("task-repo-")) publishedNames.add(name);
   }
-  const repositoryOrigins = await tx.select({ id: executionWorkspaceRepositoryAccess.scopeId }).from(executionWorkspaceRepositoryAccess)
-    .where(and(eq(executionWorkspaceRepositoryAccess.companyId, input.companyId), eq(executionWorkspaceRepositoryAccess.kind, "repository")));
+  const repositoryOrigins = inventory.length ? await measureRunPhase("workspace.sources.inventory", async () => tx.select({ id: executionWorkspaceRepositoryAccess.scopeId }).from(executionWorkspaceRepositoryAccess)
+    .where(and(eq(executionWorkspaceRepositoryAccess.companyId, input.companyId), eq(executionWorkspaceRepositoryAccess.kind, "repository")))) : [];
   const knownOrigins = new Set(repositoryOrigins.map(origin => origin.id));
   const recognizedNames = new Set<string>();
-  for (const peer of inventory) {
-    const name = path.basename(taskRepositoryRelativePath(peer.identity));
-    const published = publishedNames.has(name);
-    const peerRoot = peer.cwd ? await fs.realpath(peer.cwd).catch((error: NodeJS.ErrnoException) => {
-      if (["ENOENT", "ENOTDIR", "EACCES", "EPERM", "ELOOP", "ENAMETOOLONG"].includes(error.code ?? "")) return null;
-      throw error;
-    }) : null;
-    const matches = sourceIds.has(peer.workspaceId) || (peerRoot !== null && repositoryRootsOverlap(peerRoot, canonicalCwd))
-      || (published && peerRoot === null && !knownOrigins.has(peer.id))
-      || (peerRoot === null && !entries.some(entry => entry.kind === "reader") && peer.cwd && await legacyRepositoryRootMayMatch(peer.cwd, canonicalCwd));
-    if (matches) {
-      sourceIds.add(peer.workspaceId);
-      if (!isolatedSourceIds.has(peer.workspaceId) || (peerRoot && repositoryRootsOverlap(peerRoot, canonicalCwd))) {
-        sharedSourceIds.add(peer.workspaceId); addSharedSource(peer.workspaceId, peerRoot ?? canonicalCwd);
+  await measureRunPhase("workspace.sources.repository_scan", async () => {
+    const inventoryRoots = await resolveRepositoryRoots(inventory.flatMap(peer => peer.cwd ? [peer.cwd] : []), true);
+    const hasReader = entries.some(entry => entry.kind === "reader");
+    for (const peer of inventory) {
+      const name = path.basename(taskRepositoryRelativePath(peer.identity));
+      const published = publishedNames.has(name);
+      const peerRoot = peer.cwd ? inventoryRoots.get(peer.cwd)! : null;
+      const matches = sourceIds.has(peer.workspaceId) || (peerRoot !== null && repositoryRootsOverlap(peerRoot, canonicalCwd))
+        || (published && peerRoot === null && !knownOrigins.has(peer.id))
+        || (peerRoot === null && !hasReader && peer.cwd && await legacyRepositoryRootMayMatch(peer.cwd, canonicalCwd));
+      if (matches) {
+        sourceIds.add(peer.workspaceId);
+        if (!isolatedSourceIds.has(peer.workspaceId) || (peerRoot && repositoryRootsOverlap(peerRoot, canonicalCwd))) {
+          sharedSourceIds.add(peer.workspaceId); addSharedSource(peer.workspaceId, peerRoot ?? canonicalCwd);
+        }
+        if (published) recognizedNames.add(name);
       }
-      if (published) recognizedNames.add(name);
     }
-  }
+  });
   if ([...publishedNames].some(name => !recognizedNames.has(name))) throw forbidden("Published repository has no retained workspace provenance");
-  const configured = await tx.select({ projectId: projectWorkspaces.projectId, cwd: projectWorkspaces.cwd, repoUrl: projectWorkspaces.repoUrl })
-    .from(projectWorkspaces).where(eq(projectWorkspaces.companyId, input.companyId));
+  const configured = await measureRunPhase("workspace.sources.inventory", async () => tx.select({ projectId: projectWorkspaces.projectId, cwd: projectWorkspaces.cwd, repoUrl: projectWorkspaces.repoUrl })
+    .from(projectWorkspaces).where(eq(projectWorkspaces.companyId, input.companyId)));
   const sourceProjectIds = new Set(overlappingEntries.filter(entry => (entry.kind === "project_source" || entry.kind === "isolated_project_source")).map(entry => entry.scopeId));
   const sharedProjectIds = new Set(overlappingEntries.filter(entry => entry.kind === "shared_project_source" && !isCopiedAncestor(entry.scopeId, entry.canonicalCwd)).map(entry => entry.scopeId));
   const isolatedProjectIds = new Set(overlappingEntries.filter(entry => entry.kind === "isolated_project_source").map(entry => entry.scopeId));
   if (sourceIds.size) {
-    const owners = await tx.select({ id: executionWorkspaces.id, projectId: executionWorkspaces.projectId, strategyType: executionWorkspaces.strategyType }).from(executionWorkspaces).where(and(
-      eq(executionWorkspaces.companyId, input.companyId), or(...[...sourceIds].map(id => eq(executionWorkspaces.id, id)))));
+    const owners = await measureRunPhase("workspace.sources.inventory", async () => tx.select({ id: executionWorkspaces.id, projectId: executionWorkspaces.projectId, strategyType: executionWorkspaces.strategyType }).from(executionWorkspaces).where(and(
+      eq(executionWorkspaces.companyId, input.companyId), inArray(executionWorkspaces.id, [...sourceIds]))));
     for (const owner of owners) if (owner.projectId) {
       sourceProjectIds.add(owner.projectId);
       if (owner.strategyType === "git_worktree" || !sharedSourceIds.has(owner.id)
@@ -160,17 +197,19 @@ async function resolveRepositoryRootSources(tx: Db | DbTransaction, input: {
       else sharedProjectIds.add(owner.projectId);
     }
   }
-  for (const source of configured) {
-    if (sharedProjectIds.has(source.projectId)) continue;
-    const sourceRoot = await configuredWorkspaceRoot(source, true);
-    if (!sourceRoot || !repositoryRootsOverlap(sourceRoot, canonicalCwd)) continue;
-    sourceProjectIds.add(source.projectId);
-    if (!isCopiedAncestor(source.projectId, sourceRoot)) sharedProjectIds.add(source.projectId);
-  }
+  await measureRunPhase("workspace.sources.project_scan", async () => {
+    for (const source of configured) {
+      if (sharedProjectIds.has(source.projectId)) continue;
+      const sourceRoot = await configuredWorkspaceRoot(source, true);
+      if (!sourceRoot || !repositoryRootsOverlap(sourceRoot, canonicalCwd)) continue;
+      sourceProjectIds.add(source.projectId);
+      if (!isCopiedAncestor(source.projectId, sourceRoot)) sharedProjectIds.add(source.projectId);
+    }
+  });
   for (const entry of overlappingEntries) if (entry.kind === "project_source" && !isolatedProjectIds.has(entry.scopeId)) {
     sharedProjectIds.add(entry.scopeId);
   }
-  return { canonicalCwd, rootKey, entries, overlappingEntries, sourceIds, sharedSourceIds, isolatedSourceIds, sourceProjectIds, sharedProjectIds, isolatedProjectIds };
+  return { canonicalCwd, rootKey, entries, entryKeys, overlappingEntries, sourceIds, sharedSourceIds, isolatedSourceIds, sourceProjectIds, sharedProjectIds, isolatedProjectIds };
 }
 
 async function assertRootProjectAccess(db: Db | DbTransaction, actor: AuthorizationActor, companyId: string, projectIds: Set<string>,
@@ -195,9 +234,9 @@ async function assertRootProjectsReadAccessForActors(db: Db | DbTransaction, act
 }
 
 async function retainedRootReader(tx: DbTransaction, companyId: string, reader: Pick<typeof executionWorkspaceRepositoryAccess.$inferSelect, "scopeId" | "issueId" | "agentId" | "responsibleUserId">) {
-  const [run] = await tx.select(rootReaderRunColumns).from(heartbeatRuns).where(and(eq(heartbeatRuns.id, reader.scopeId), eq(heartbeatRuns.companyId, companyId)));
-  const [agent] = reader.agentId ? await tx.select({ id: agents.id }).from(agents).where(and(eq(agents.id, reader.agentId), eq(agents.companyId, companyId))) : [];
-  const [issue] = reader.issueId ? await tx.select().from(issues).where(and(eq(issues.id, reader.issueId), eq(issues.companyId, companyId))) : [];
+  const [run] = await measureRunPhase("workspace.sources.inventory", async () => tx.select(rootReaderRunColumns).from(heartbeatRuns).where(and(eq(heartbeatRuns.id, reader.scopeId), eq(heartbeatRuns.companyId, companyId))));
+  const [agent] = reader.agentId ? await measureRunPhase("workspace.sources.inventory", async () => tx.select({ id: agents.id }).from(agents).where(and(eq(agents.id, reader.agentId!), eq(agents.companyId, companyId)))) : [];
+  const [issue] = reader.issueId ? await measureRunPhase("workspace.sources.inventory", async () => tx.select().from(issues).where(and(eq(issues.id, reader.issueId!), eq(issues.companyId, companyId)))) : [];
   if (!run || !agent || !issue || run.agentId !== reader.agentId || run.responsibleUserId !== reader.responsibleUserId
     || (run.nativeIssueId ?? run.contextSnapshot?.issueId ?? run.contextSnapshot?.taskId ?? run.issueId) !== reader.issueId) {
     throw forbidden("Repository root retains a reader whose identity is no longer available");
@@ -208,7 +247,7 @@ async function retainedRootReader(tx: DbTransaction, companyId: string, reader:
 }
 
 export async function lockExecutionWorkspaceRootProvenance(tx: DbTransaction, companyId: string) {
-  await tx.execute(sql`select pg_advisory_xact_lock(hashtextextended(${`task-repository-roots:${companyId}`}, 0))`);
+  await measureRunPhase("workspace.provenance_lock", () => tx.execute(sql`select pg_advisory_xact_lock(hashtextextended(${`task-repository-roots:${companyId}`}, 0))`));
 }
 
 export async function preserveConfiguredProjectWorkspaceRootProvenance(tx: DbTransaction, input: {
@@ -217,12 +256,12 @@ export async function preserveConfiguredProjectWorkspaceRootProvenance(tx: DbTra
   const canonicalCwd = await configuredWorkspaceRoot(input, false);
   if (!canonicalCwd) return;
   const rootKey = createHash("sha256").update(canonicalCwd).digest("hex");
-  const entries = await tx.select().from(executionWorkspaceRepositoryAccess).where(and(
-    eq(executionWorkspaceRepositoryAccess.companyId, input.companyId), eq(executionWorkspaceRepositoryAccess.rootKey, rootKey)));
+  const entries = await measureRunPhase("workspace.sources.inventory", async () => tx.select().from(executionWorkspaceRepositoryAccess).where(and(
+    eq(executionWorkspaceRepositoryAccess.companyId, input.companyId), eq(executionWorkspaceRepositoryAccess.rootKey, rootKey))));
   if (entries.some(entry => entry.canonicalCwd !== canonicalCwd)) throw forbidden("Repository root identity conflicts with its retained access record");
   for (const kind of ["project_source", "shared_project_source"] as const) {
     if (!entries.some(entry => entry.kind === kind && entry.scopeId === input.projectId)) {
-      await tx.insert(executionWorkspaceRepositoryAccess).values({ companyId: input.companyId, canonicalCwd, rootKey, kind, scopeId: input.projectId });
+      await measureRunPhase("workspace.write", () => tx.insert(executionWorkspaceRepositoryAccess).values({ companyId: input.companyId, canonicalCwd, rootKey, kind, scopeId: input.projectId }));
     }
   }
 }
@@ -237,24 +276,19 @@ export async function preserveExecutionWorkspaceRootProvenance(tx: DbTransaction
   });
   if (!directory) return;
   if (!directory.isDirectory()) throw forbidden("Workspace files cannot be detached from unverifiable provenance");
-  const { canonicalCwd, rootKey, entries, sharedSourceIds, isolatedSourceIds, sourceProjectIds, sharedProjectIds, isolatedProjectIds } = await resolveRepositoryRootSources(tx, {
-    companyId: input.companyId, cwd: input.cwd, sourceWorkspaceId: input.workspaceId, inspectCurrentAliases: true,
-  });
-  for (const [kind, ids] of [["source", sharedSourceIds], ["isolated_source", isolatedSourceIds], ["project_source", sourceProjectIds], ["shared_project_source", sharedProjectIds], ["isolated_project_source", isolatedProjectIds]] as const) {
-    for (const scopeId of ids) {
-      if (!entries.some(entry => entry.kind === kind && entry.scopeId === scopeId)) {
-        await tx.insert(executionWorkspaceRepositoryAccess).values({ companyId: input.companyId, canonicalCwd, rootKey, kind, scopeId });
-      }
-    }
-  }
+  const { canonicalCwd, rootKey, entryKeys, sharedSourceIds, isolatedSourceIds, sourceProjectIds, sharedProjectIds, isolatedProjectIds } = await measureRunPhase("workspace.sources", () => resolveRepositoryRootSources(tx, {
+    companyId: input.companyId, cwd: input.cwd!, sourceWorkspaceId: input.workspaceId, inspectCurrentAliases: true,
+  }));
+  await retainRepositorySources(tx, { companyId: input.companyId, canonicalCwd, rootKey }, entryKeys,
+    [["source", sharedSourceIds], ["isolated_source", isolatedSourceIds], ["project_source", sourceProjectIds], ["shared_project_source", sharedProjectIds], ["isolated_project_source", isolatedProjectIds]]);
 }
 
 export async function assertExecutionWorkspaceRootReadAccess(db: Db | DbTransaction, input: {
   companyId: string; cwd: string; actor: AuthorizationActor; sourceWorkspaceId?: string;
 }): Promise<void> {
-  const { sourceIds, sourceProjectIds } = await resolveRepositoryRootSources(db, { ...input, inspectCurrentAliases: true });
-  await assertTaskWorkspacesReadAccess(db, input.actor, input.companyId, sourceIds);
-  await assertRootProjectAccess(db, input.actor, input.companyId, sourceProjectIds);
+  const { sourceIds, sourceProjectIds } = await measureRunPhase("workspace.sources", () => resolveRepositoryRootSources(db, { ...input, inspectCurrentAliases: true }));
+  await measureRunPhase("workspace.readers.sources", () => assertTaskWorkspacesReadAccess(db, input.actor, input.companyId, sourceIds));
+  await measureRunPhase("workspace.readers.projects", () => assertRootProjectAccess(db, input.actor, input.companyId, sourceProjectIds));
 }
 
 export type IsolatedWorkspaceSourceAccess = {
@@ -267,36 +301,32 @@ export type IsolatedWorkspaceSourceAccess = {
 async function retainGitMetadataOwners(tx: DbTransaction, input: {
   companyId: string; commonRoot: string; workspaceIds: Set<string>; projectIds: Set<string>;
 }): Promise<void> {
-  const closure = await resolveRepositoryRootSources(tx, { companyId: input.companyId, cwd: input.commonRoot, inspectCurrentAliases: true });
+  const closure = await measureRunPhase("workspace.sources", () => resolveRepositoryRootSources(tx, { companyId: input.companyId, cwd: input.commonRoot, inspectCurrentAliases: true }));
   const workspaceIds = new Set([...closure.sourceIds, ...input.workspaceIds]);
   const projectIds = new Set([...closure.sourceProjectIds, ...input.projectIds]);
   const readerActors: AuthorizationActor[] = [];
   for (const reader of closure.overlappingEntries.filter(entry => entry.kind === "reader")) {
-    const { actor } = await retainedRootReader(tx, input.companyId, reader);
+    const { actor } = await measureRunPhase("workspace.readers.identity", () => retainedRootReader(tx, input.companyId, reader));
     readerActors.push(actor);
   }
-  await assertRootProjectsReadAccessForActors(tx, readerActors, input.companyId, projectIds);
-  await assertTaskWorkspacesReadAccessForActors(tx, readerActors, input.companyId, workspaceIds);
-  for (const [kind, ids] of [["source", workspaceIds], ["project_source", projectIds]] as const) {
-    for (const scopeId of ids) if (!closure.entries.some(entry => entry.kind === kind && entry.scopeId === scopeId)) {
-      await tx.insert(executionWorkspaceRepositoryAccess).values({ companyId: input.companyId,
-        canonicalCwd: closure.canonicalCwd, rootKey: closure.rootKey, kind, scopeId });
-    }
-  }
+  await measureRunPhase("workspace.readers.projects", () => assertRootProjectsReadAccessForActors(tx, readerActors, input.companyId, projectIds));
+  await measureRunPhase("workspace.readers.sources", () => assertTaskWorkspacesReadAccessForActors(tx, readerActors, input.companyId, workspaceIds));
+  await retainRepositorySources(tx, { companyId: input.companyId, canonicalCwd: closure.canonicalCwd, rootKey: closure.rootKey }, closure.entryKeys,
+    [["source", workspaceIds], ["project_source", projectIds]]);
 }
 
 export async function assertIsolatedWorkspaceSourceReadAccess(db: Db, input: {
   companyId: string; sourceCwd: string; actor: AuthorizationActor; copyKind?: WorkspaceSourceCopyKind;
 }): Promise<IsolatedWorkspaceSourceAccess> {
   const { roots, commonRoot } = await isolatedWorkspaceSourceRoots(db, input.companyId, input.sourceCwd, input.copyKind);
-  return db.transaction(async tx => {
+  return measureRunTransaction(db, "workspace", async tx => {
     await lockExecutionWorkspaceRootProvenance(tx, input.companyId);
     const workspaceIds = new Set<string>(), projectIds = new Set<string>();
     for (const cwd of roots) {
-      const closure = await resolveRepositoryRootSources(tx, { companyId: input.companyId, cwd, inspectCurrentAliases: true });
-      await assertTaskWorkspacesReadAccess(tx, input.actor, input.companyId, closure.sourceIds);
+      const closure = await measureRunPhase("workspace.sources", () => resolveRepositoryRootSources(tx, { companyId: input.companyId, cwd, inspectCurrentAliases: true }));
+      await measureRunPhase("workspace.readers.sources", () => assertTaskWorkspacesReadAccess(tx, input.actor, input.companyId, closure.sourceIds));
       for (const id of closure.sourceIds) workspaceIds.add(id);
-      await assertRootProjectAccess(tx, input.actor, input.companyId, closure.sourceProjectIds);
+      await measureRunPhase("workspace.readers.projects", () => assertRootProjectAccess(tx, input.actor!, input.companyId, closure.sourceProjectIds));
       for (const id of closure.sourceProjectIds) projectIds.add(id);
     }
     await retainGitMetadataOwners(tx, { companyId: input.companyId, commonRoot, workspaceIds, projectIds });
@@ -313,33 +343,31 @@ export async function retainIsolatedWorkspaceSourceAccess(db: Db, input: {
   const roots = new Set(input.captured.roots);
   try { for (const cwd of (await isolatedWorkspaceSourceRoots(db, input.companyId, input.sourceCwd, input.copyKind)).roots) roots.add(cwd); }
   catch (error) { sourceFailure = error; }
-  const canonicalCwd = await canonicalRepositoryRoot(input.targetCwd);
+  const canonicalCwd = await measureRunPhase("workspace.sources.canonicalize", () => canonicalRepositoryRoot(input.targetCwd));
   const rootKey = createHash("sha256").update(canonicalCwd).digest("hex");
-  const denied = await db.transaction(async tx => {
+  const denied = await measureRunTransaction(db, "workspace", async tx => {
     await lockExecutionWorkspaceRootProvenance(tx, input.companyId);
-    const existing = await tx.select().from(executionWorkspaceRepositoryAccess).where(and(
-      eq(executionWorkspaceRepositoryAccess.companyId, input.companyId), eq(executionWorkspaceRepositoryAccess.rootKey, rootKey)));
+    const existing = await measureRunPhase("workspace.sources.inventory", async () => tx.select().from(executionWorkspaceRepositoryAccess).where(and(
+      eq(executionWorkspaceRepositoryAccess.companyId, input.companyId), eq(executionWorkspaceRepositoryAccess.rootKey, rootKey))));
     if (existing.some(entry => entry.canonicalCwd !== canonicalCwd)) throw forbidden("Repository root identity conflicts with its retained access record");
+    const existingKeys = repositoryAccessKeys(existing);
     const workspaceIds = new Set(input.captured.workspaceIds), projectIds = new Set(input.captured.projectIds);
     let failure = sourceFailure;
     for (const cwd of roots) {
       try {
-        const closure = await resolveRepositoryRootSources(tx, { companyId: input.companyId, cwd, inspectCurrentAliases: true });
+        const closure = await measureRunPhase("workspace.sources", () => resolveRepositoryRootSources(tx, { companyId: input.companyId, cwd, inspectCurrentAliases: true }));
         for (const id of closure.sourceIds) workspaceIds.add(id);
         for (const id of closure.sourceProjectIds) projectIds.add(id);
       } catch (error) { failure ??= error; }
     }
-    try { await assertTaskWorkspacesReadAccess(tx, input.actor, input.companyId, workspaceIds); }
+    try { await measureRunPhase("workspace.readers.sources", () => assertTaskWorkspacesReadAccess(tx, input.actor, input.companyId, workspaceIds)); }
     catch (error) { failure ??= error; }
     for (const id of projectIds) {
-      try { await assertRootProjectAccess(tx, input.actor, input.companyId, new Set([id])); }
+      try { await measureRunPhase("workspace.readers.projects", () => assertRootProjectAccess(tx, input.actor!, input.companyId, new Set([id]))); }
       catch (error) { failure ??= error; }
     }
-    for (const [kind, ids] of [["isolated_source", workspaceIds], ["isolated_project_source", projectIds]] as const) {
-      for (const scopeId of ids) if (!existing.some(entry => entry.kind === kind && entry.scopeId === scopeId)) {
-        await tx.insert(executionWorkspaceRepositoryAccess).values({ companyId: input.companyId, rootKey, canonicalCwd, kind, scopeId });
-      }
-    }
+    await retainRepositorySources(tx, { companyId: input.companyId, canonicalCwd, rootKey }, existingKeys,
+      [["isolated_source", workspaceIds], ["isolated_project_source", projectIds]]);
     const phase = input.phase ?? "preparing";
     let preparingCopyProjectIds: Set<string> | undefined;
     let verifiedCopyProjectIds: Set<string> | undefined;
@@ -360,23 +388,23 @@ export async function retainIsolatedWorkspaceSourceAccess(db: Db, input: {
       }
     } catch (error) { failure ??= error; }
     try {
-      const target = await resolveRepositoryRootSources(tx, { companyId: input.companyId, cwd: canonicalCwd, preparingCopyProjectIds });
-      await assertRootProjectTrust(tx, { companyId: input.companyId, actor: input.actor, issueId: input.issueId, projectIds: target.sourceProjectIds });
-      await assertTaskWorkspacesReadAccess(tx, input.actor, input.companyId, target.sourceIds);
-      await assertRootProjectAccess(tx, input.actor, input.companyId, target.sourceProjectIds,
-        { sharedProjectIds: target.sharedProjectIds, issueId: input.issueId });
+      const target = await measureRunPhase("workspace.sources", () => resolveRepositoryRootSources(tx, { companyId: input.companyId, cwd: canonicalCwd, preparingCopyProjectIds }));
+      await measureRunPhase("workspace.readers.trust", () => assertRootProjectTrust(tx, { companyId: input.companyId, actor: input.actor, issueId: input.issueId, projectIds: target.sourceProjectIds }));
+      await measureRunPhase("workspace.readers.sources", () => assertTaskWorkspacesReadAccess(tx, input.actor, input.companyId, target.sourceIds));
+      await measureRunPhase("workspace.readers.projects", () => assertRootProjectAccess(tx, input.actor!, input.companyId, target.sourceProjectIds,
+        { sharedProjectIds: target.sharedProjectIds, issueId: input.issueId }));
       const readerActors: AuthorizationActor[] = [];
       for (const reader of target.overlappingEntries.filter(entry => entry.kind === "reader")) {
-        const { actor, issue } = await retainedRootReader(tx, input.companyId, reader);
+        const { actor, issue } = await measureRunPhase("workspace.readers.identity", () => retainedRootReader(tx, input.companyId, reader));
         readerActors.push(actor);
-        await assertRootProjectTrust(tx, { companyId: input.companyId, actor, issueId: issue.id, projectIds: target.sourceProjectIds });
+        await measureRunPhase("workspace.readers.trust", () => assertRootProjectTrust(tx, { companyId: input.companyId, actor, issueId: issue.id, projectIds: target.sourceProjectIds }));
       }
-      await assertRootProjectsReadAccessForActors(tx, readerActors, input.companyId, target.sourceProjectIds);
-      await assertTaskWorkspacesReadAccessForActors(tx, readerActors, input.companyId, target.sourceIds);
+      await measureRunPhase("workspace.readers.projects", () => assertRootProjectsReadAccessForActors(tx, readerActors, input.companyId, target.sourceProjectIds));
+      await measureRunPhase("workspace.readers.sources", () => assertTaskWorkspacesReadAccessForActors(tx, readerActors, input.companyId, target.sourceIds));
     } catch (error) { failure ??= error; }
     if (!failure && verifiedCopyProjectIds) {
       for (const scopeId of verifiedCopyProjectIds) if (!existing.some(entry => entry.kind === "isolated_copy" && entry.scopeId === scopeId)) {
-        await tx.insert(executionWorkspaceRepositoryAccess).values({ companyId: input.companyId, rootKey, canonicalCwd, kind: "isolated_copy", scopeId });
+        await measureRunPhase("workspace.write", () => tx.insert(executionWorkspaceRepositoryAccess).values({ companyId: input.companyId, rootKey, canonicalCwd, kind: "isolated_copy", scopeId }));
       }
     }
     return failure;
@@ -464,10 +492,10 @@ async function assertRootProjectTrust(db: Db | DbTransaction, input: {
 export async function assertExecutionWorkspaceRootWriteAccess(db: Db | DbTransaction, input: {
   companyId: string; cwd: string; actor: AuthorizationActor; sourceWorkspaceId?: string; taskCreation?: TaskWorkspaceCreationTrustContext;
 } & Omit<NonNullable<Parameters<typeof assertTaskWorkspaceSourceProjectAccess>[4]>, "write">): Promise<void> {
-  const { sourceIds, sourceProjectIds, sharedProjectIds } = await resolveRepositoryRootSources(db, { ...input, inspectCurrentAliases: true });
-  await assertTaskWorkspacesReadAccess(db, input.actor, input.companyId, sourceIds);
-  await assertRootProjectAccess(db, input.actor, input.companyId, sourceProjectIds, { ...input, sharedProjectIds });
-  await assertRootProjectTrust(db, { ...input, projectIds: sourceProjectIds });
+  const { sourceIds, sourceProjectIds, sharedProjectIds } = await measureRunPhase("workspace.sources", () => resolveRepositoryRootSources(db, { ...input, inspectCurrentAliases: true }));
+  await measureRunPhase("workspace.readers.sources", () => assertTaskWorkspacesReadAccess(db, input.actor, input.companyId, sourceIds));
+  await measureRunPhase("workspace.readers.projects", () => assertRootProjectAccess(db, input.actor, input.companyId, sourceProjectIds, { ...input, sharedProjectIds }));
+  await measureRunPhase("workspace.readers.trust", () => assertRootProjectTrust(db, { ...input, projectIds: sourceProjectIds }));
 }
 
 export function executionWorkspaceRepositoryService(db: Db) {
@@ -501,10 +529,10 @@ export function executionWorkspaceRepositoryService(db: Db) {
     if (await requiresAnonymousAcquisition(db, { issue, workspace, agentId: input.agentId, runId: input.runId })) throw forbidden(controllerAcquisitionMessage);
   }
   async function reserveRootPolicy(tx: DbTransaction, companyId: string, cwd: string, mode: "anonymous_only" | "managed") {
-    const canonicalCwd = await canonicalRepositoryRoot(cwd);
+    const canonicalCwd = await measureRunPhase("workspace.sources.canonicalize", () => canonicalRepositoryRoot(cwd));
     const rootKey = createHash("sha256").update(canonicalCwd).digest("hex");
-    const reservations = await tx.select().from(executionWorkspaceRepositoryRoots)
-      .where(eq(executionWorkspaceRepositoryRoots.companyId, companyId));
+    const reservations = await measureRunPhase("workspace.sources.inventory", async () => tx.select().from(executionWorkspaceRepositoryRoots)
+      .where(eq(executionWorkspaceRepositoryRoots.companyId, companyId)));
     const existing = reservations.find(entry => entry.rootKey === rootKey);
     const overlapping = reservations.filter(entry => repositoryRootsOverlap(entry.canonicalCwd, canonicalCwd));
     if ((existing && existing.canonicalCwd !== canonicalCwd)
@@ -512,10 +540,10 @@ export function executionWorkspaceRepositoryService(db: Db) {
       throw forbidden(lowTrustRepositoryMessage);
     }
     if (mode === "anonymous_only") {
-      const managed = await tx.select({ cwd: executionWorkspaces.cwd, repositoryIdentity: executionWorkspaceRepositories.repositoryIdentity }).from(executionWorkspaceRepositories)
+      const managed = await measureRunPhase("workspace.sources.inventory", async () => tx.select({ cwd: executionWorkspaces.cwd, repositoryIdentity: executionWorkspaceRepositories.repositoryIdentity }).from(executionWorkspaceRepositories)
         .innerJoin(executionWorkspaces, eq(executionWorkspaces.id, executionWorkspaceRepositories.executionWorkspaceId))
         .where(and(eq(executionWorkspaceRepositories.companyId, companyId), eq(executionWorkspaces.companyId, companyId),
-          isNotNull(executionWorkspaceRepositories.catalogRepositoryId)));
+          isNotNull(executionWorkspaceRepositories.catalogRepositoryId))));
       if (managed.length) {
         const namespace = path.join(canonicalCwd, ".paperclip-repositories");
         const namespaceEntry = await fs.lstat(namespace).catch((error: NodeJS.ErrnoException) => {
@@ -539,79 +567,78 @@ export function executionWorkspaceRepositoryService(db: Db) {
         if (peer.cwd && await legacyRepositoryRootMayMatch(peer.cwd, canonicalCwd)) throw forbidden(lowTrustRepositoryMessage);
       }
     }
-    if (!existing) await tx.insert(executionWorkspaceRepositoryRoots).values({ companyId, canonicalCwd, rootKey, mode });
+    if (!existing) await measureRunPhase("workspace.write", () => tx.insert(executionWorkspaceRepositoryRoots).values({ companyId, canonicalCwd, rootKey, mode }));
     return canonicalCwd;
   }
   type RootReader = { runId: string; issueId: string; agentId: string };
   async function authorizeRootAccess(tx: DbTransaction, input: {
     companyId: string; cwd: string; sourceWorkspaceId?: string; reader?: RootReader; actor?: AuthorizationActor; issueId?: string; preserveOwnAdmission?: boolean;
   }) {
-    const { canonicalCwd, rootKey, entries, overlappingEntries, sourceIds, sharedSourceIds, isolatedSourceIds, sourceProjectIds, sharedProjectIds, isolatedProjectIds } = await resolveRepositoryRootSources(tx, input);
+    const { canonicalCwd, rootKey, entryKeys, overlappingEntries, sourceIds, sharedSourceIds, isolatedSourceIds, sourceProjectIds, sharedProjectIds, isolatedProjectIds } = await measureRunPhase("workspace.sources", () => resolveRepositoryRootSources(tx, input));
     const readers = overlappingEntries.filter(entry => entry.kind === "reader");
-    if (input.reader && !entries.some(entry => entry.kind === "reader" && entry.scopeId === input.reader!.runId)) {
-      const [run] = await tx.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.id, input.reader.runId), eq(heartbeatRuns.companyId, input.companyId)));
+    if (input.reader && !entryKeys.has(`reader:${input.reader.runId}`)) {
+      const [run] = await measureRunPhase("workspace.sources.inventory", async () => tx.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.id, input.reader!.runId), eq(heartbeatRuns.companyId, input.companyId))));
       readers.push({ id: randomUUID(), companyId: input.companyId, rootKey, canonicalCwd, kind: "reader", scopeId: input.reader.runId,
         issueId: input.reader.issueId, agentId: input.reader.agentId, responsibleUserId: run?.responsibleUserId ?? null, createdAt: new Date(), updatedAt: new Date() });
     }
     const ownAdmissionOnly = input.preserveOwnAdmission && input.reader && input.sourceWorkspaceId
       && sourceIds.size === 1 && sourceIds.has(input.sourceWorkspaceId)
       && readers.every(reader => reader.issueId === input.reader!.issueId);
-    const sourceProjects = sourceIds.size ? await tx.selectDistinct({ projectId: executionWorkspaces.projectId }).from(executionWorkspaces).where(and(
-      eq(executionWorkspaces.companyId, input.companyId), inArray(executionWorkspaces.id, [...sourceIds]))) : [];
+    const sourceProjects = sourceIds.size ? await measureRunPhase("workspace.sources.inventory", async () => tx.selectDistinct({ projectId: executionWorkspaces.projectId }).from(executionWorkspaces).where(and(
+      eq(executionWorkspaces.companyId, input.companyId), inArray(executionWorkspaces.id, [...sourceIds])))) : [];
     const sourceActors: AuthorizationActor[] = [];
     const projectReaders: AuthorizationActor[] = [];
-    for (const reader of readers) {
-      const { actor, issue } = await retainedRootReader(tx, input.companyId, reader);
-      await assertRootProjectTrust(tx, { companyId: input.companyId, projectIds: sourceProjectIds, actor, issueId: issue.id,
-        sourceWorkspaceId: input.reader?.runId === reader.scopeId ? input.sourceWorkspaceId : undefined });
-      if (input.reader?.runId === reader.scopeId) {
-        await assertRootProjectAccess(tx, actor, input.companyId, sourceProjectIds, { sharedProjectIds, issueId: issue.id });
-      } else projectReaders.push(actor);
-      if (ownAdmissionOnly) {
-        const [ownSource] = await tx.select({ id: executionWorkspaces.id }).from(executionWorkspaces).where(and(
-          eq(executionWorkspaces.companyId, input.companyId), eq(executionWorkspaces.id, input.sourceWorkspaceId!),
-          await executionWorkspaceReadSqlCondition(tx, { ...actor, onBehalfOfUserId: null }),
-          actor.onBehalfOfUserId ? await executionWorkspaceReadSqlCondition(tx, {
-            type: "board", userId: actor.onBehalfOfUserId, ignoreInstanceAdmin: true,
-          }) : undefined));
-        if (!ownSource) throw forbidden("Task workspace access is no longer available");
-      } else sourceActors.push(actor);
-      for (const sourceWorkspace of sourceProjects.length ? sourceProjects : [null]) {
-        if (await requiresAnonymousAcquisition(tx, { issue, workspace: sourceWorkspace, agentId: reader.agentId!, runId: reader.scopeId })) {
-          await reserveRootPolicy(tx, input.companyId, canonicalCwd, "anonymous_only");
+    await measureRunPhase("workspace.readers", async () => {
+      for (const reader of readers) {
+        const { actor, issue } = await measureRunPhase("workspace.readers.identity", () => retainedRootReader(tx, input.companyId, reader));
+        await measureRunPhase("workspace.readers.trust", () => assertRootProjectTrust(tx, { companyId: input.companyId, projectIds: sourceProjectIds, actor, issueId: issue.id,
+          sourceWorkspaceId: input.reader?.runId === reader.scopeId ? input.sourceWorkspaceId : undefined }));
+        if (input.reader?.runId === reader.scopeId) {
+          await measureRunPhase("workspace.readers.projects", () => assertRootProjectAccess(tx, actor, input.companyId, sourceProjectIds, { sharedProjectIds, issueId: issue.id }));
+        } else projectReaders.push(actor);
+        if (ownAdmissionOnly) {
+          const [ownSource] = await measureRunPhase("workspace.sources.inventory", async () => tx.select({ id: executionWorkspaces.id }).from(executionWorkspaces).where(and(
+            eq(executionWorkspaces.companyId, input.companyId), eq(executionWorkspaces.id, input.sourceWorkspaceId!),
+            await executionWorkspaceReadSqlCondition(tx, { ...actor, onBehalfOfUserId: null }),
+            actor.onBehalfOfUserId ? await executionWorkspaceReadSqlCondition(tx, {
+              type: "board", userId: actor.onBehalfOfUserId, ignoreInstanceAdmin: true,
+            }) : undefined)));
+          if (!ownSource) throw forbidden("Task workspace access is no longer available");
+        } else sourceActors.push(actor);
+        for (const sourceWorkspace of sourceProjects.length ? sourceProjects : [null]) {
+          if (await requiresAnonymousAcquisition(tx, { issue, workspace: sourceWorkspace, agentId: reader.agentId!, runId: reader.scopeId })) {
+            await reserveRootPolicy(tx, input.companyId, canonicalCwd, "anonymous_only");
+          }
         }
       }
-    }
-    await assertRootProjectsReadAccessForActors(tx, projectReaders, input.companyId, sourceProjectIds);
+      await measureRunPhase("workspace.readers.projects", () => assertRootProjectsReadAccessForActors(tx, projectReaders, input.companyId, sourceProjectIds));
+    });
     if (input.actor) {
       if (!input.issueId) throw forbidden("Workspace write authorization requires a task");
-      await assertRootProjectTrust(tx, { ...input, projectIds: sourceProjectIds, actor: input.actor });
-      await assertRootProjectAccess(tx, input.actor, input.companyId, sourceProjectIds, { sharedProjectIds, issueId: input.issueId });
+      await measureRunPhase("workspace.readers.trust", () => assertRootProjectTrust(tx, { ...input, projectIds: sourceProjectIds, actor: input.actor! }));
+      await measureRunPhase("workspace.readers.projects", () => assertRootProjectAccess(tx, input.actor!, input.companyId, sourceProjectIds, { sharedProjectIds, issueId: input.issueId }));
       sourceActors.push(input.actor);
     }
-    await assertTaskWorkspacesReadAccessForActors(tx, sourceActors, input.companyId, sourceIds);
-    for (const [kind, ids] of [["source", sharedSourceIds], ["isolated_source", isolatedSourceIds], ["project_source", sourceProjectIds],
-      ["shared_project_source", sharedProjectIds], ["isolated_project_source", isolatedProjectIds]] as const) {
-      for (const scopeId of ids) {
-        if (!entries.some(entry => entry.kind === kind && entry.scopeId === scopeId)) {
-          await tx.insert(executionWorkspaceRepositoryAccess).values({ companyId: input.companyId, rootKey, canonicalCwd, kind, scopeId });
+    await measureRunPhase("workspace.readers.sources", () => assertTaskWorkspacesReadAccessForActors(tx, sourceActors, input.companyId, sourceIds));
+    await retainRepositorySources(tx, { companyId: input.companyId, canonicalCwd, rootKey }, entryKeys,
+      [["source", sharedSourceIds], ["isolated_source", isolatedSourceIds], ["project_source", sourceProjectIds],
+      ["shared_project_source", sharedProjectIds], ["isolated_project_source", isolatedProjectIds]]);
+    await measureRunPhase("workspace.readers", async () => {
+      for (const reader of readers) {
+        if (reader.rootKey === rootKey && !entryKeys.has(`reader:${reader.scopeId}`)) {
+          await measureRunPhase("workspace.write", () => tx.insert(executionWorkspaceRepositoryAccess).values(reader));
         }
       }
-    }
-    for (const reader of readers) {
-      if (reader.rootKey === rootKey && !entries.some(entry => entry.kind === "reader" && entry.scopeId === reader.scopeId)) {
-        await tx.insert(executionWorkspaceRepositoryAccess).values(reader);
-      }
-    }
+    });
   }
   async function reserveRootAccess(input: Parameters<typeof authorizeRootAccess>[1]) {
-    return db.transaction(async tx => {
+    return measureRunTransaction(db, "workspace", async tx => {
       await lockExecutionWorkspaceRootProvenance(tx, input.companyId);
       await authorizeRootAccess(tx, input);
     });
   }
   async function reserveAdmissionRoot(companyId: string, cwd: string, mode: "anonymous_only" | "managed") {
-    return db.transaction(async tx => {
+    return measureRunTransaction(db, "workspace", async tx => {
       await lockExecutionWorkspaceRootProvenance(tx, companyId);
       return reserveRootPolicy(tx, companyId, cwd, mode);
     });
@@ -637,16 +664,16 @@ export function executionWorkspaceRepositoryService(db: Db) {
     }
     const available = await catalog(input.companyId, userId, localTrusted);
     const requestedRef = parsed.ref ?? "HEAD";
-    const receipt = await db.transaction(async tx => {
+    const receipt = await measureRunTransaction(db, "workspace", async tx => {
       await lockExecutionWorkspaceRootProvenance(tx, input.companyId);
-      const [issue] = await tx.select().from(issues).where(and(eq(issues.id, input.issueId), eq(issues.companyId, input.companyId), await issueReadSqlCondition(tx, input.actor))).for("no key update");
+      const [issue] = await measureRunPhase("workspace.sources.inventory", async () => tx.select().from(issues).where(and(eq(issues.id, input.issueId), eq(issues.companyId, input.companyId), await issueReadSqlCondition(tx, input.actor))).for("no key update"));
       if (!issue) throw notFound("Task not found");
       const currentDecision = await authorizationService(tx).decide({ actor: input.actor, action: "issue:mutate", resource: issueResource(issue) });
       if (!currentDecision.allowed) throw forbidden("Task workspace modification is not allowed");
       if (input.actor.type === "agent" && !["standard", "skill_test"].includes(issue.workMode)) throw forbidden("Repository preparation is unavailable in Ask or Plan mode");
       if (!issue.executionWorkspaceId) throw conflict("The task has no admitted workspace yet. Start it once before preparing a repository.");
       if (!(await canActorReadExecutionWorkspace(db, input.actor, issue.executionWorkspaceId))) throw notFound("Execution workspace not found");
-      const [workspace] = await tx.select().from(executionWorkspaces).where(and(eq(executionWorkspaces.id, issue.executionWorkspaceId), eq(executionWorkspaces.companyId, input.companyId), taskWorkspaceSelectableCondition())).for("no key update");
+      const [workspace] = await measureRunPhase("workspace.sources.inventory", async () => tx.select().from(executionWorkspaces).where(and(eq(executionWorkspaces.id, issue.executionWorkspaceId!), eq(executionWorkspaces.companyId, input.companyId), taskWorkspaceSelectableCondition())).for("no key update"));
       if (!workspace) throw conflict("Execution workspace is unavailable");
       await assertTaskWorkspaceAccess(tx, input.actor, input.companyId, workspace.id, { write: true, issueId: input.issueId });
       const anonymousOnly = await requiresAnonymousAcquisition(tx, { issue, workspace,
@@ -659,7 +686,7 @@ export function executionWorkspaceRepositoryService(db: Db) {
       if (!workspace.cwd) throw conflict("The task workspace has no directory for repository preparation");
       await authorizeRootAccess(tx, { companyId: input.companyId, cwd: workspace.cwd, sourceWorkspaceId: workspace.id, actor: input.actor, issueId: input.issueId });
       if (anonymousOnly) throw forbidden(controllerAcquisitionMessage);
-      const existing = await tx.select().from(executionWorkspaceRepositories).where(and(eq(executionWorkspaceRepositories.executionWorkspaceId, workspace.id), or(eq(executionWorkspaceRepositories.repositoryIdentity, source.repositoryIdentity), sql`${executionWorkspaceRepositories.requestKeys} ? ${parsed.requestKey}`)));
+      const existing = await measureRunPhase("workspace.sources.inventory", async () => tx.select().from(executionWorkspaceRepositories).where(and(eq(executionWorkspaceRepositories.executionWorkspaceId, workspace.id), or(eq(executionWorkspaceRepositories.repositoryIdentity, source.repositoryIdentity), sql`${executionWorkspaceRepositories.requestKeys} ? ${parsed.requestKey}`))));
       for (const row of existing) {
         assertAcquisitionReceipt(row, anonymousOnly);
         assertRepositoryRetryMatches(row, source.repositoryIdentity, requestedRef);
@@ -672,11 +699,11 @@ export function executionWorkspaceRepositoryService(db: Db) {
         }
         return row;
       }
-      const [row] = await tx.insert(executionWorkspaceRepositories).values({ companyId: input.companyId, executionWorkspaceId: workspace.id, requestedByIssueId: input.issueId, ...source, relativePath: taskRepositoryRelativePath(source.repositoryIdentity), requestedRef, requestKey: parsed.requestKey, requestKeys: [parsed.requestKey] }).returning();
-      const canonicalCwd = await canonicalRepositoryRoot(workspace.cwd);
-      await tx.insert(executionWorkspaceRepositoryAccess).values({ companyId: input.companyId,
-        rootKey: createHash("sha256").update(canonicalCwd).digest("hex"), canonicalCwd, kind: "repository", scopeId: row.id });
-      await tx.insert(activityLog).values({ companyId: input.companyId, actorType: input.actor.type === "agent" ? "agent" : "user", actorId: input.actor.agentId ?? input.actor.userId ?? "local-board", action: "workspace.repository.requested", entityType: "execution_workspace", entityId: workspace.id, runId: input.actor.runId ?? null, responsibleUserId: userId, details: { repositoryId: row.id, issueId: input.issueId, applies: "next_normal_admission" } });
+      const [row] = await measureRunPhase("workspace.write", () => tx.insert(executionWorkspaceRepositories).values({ companyId: input.companyId, executionWorkspaceId: workspace.id, requestedByIssueId: input.issueId, ...source, relativePath: taskRepositoryRelativePath(source.repositoryIdentity), requestedRef, requestKey: parsed.requestKey, requestKeys: [parsed.requestKey] }).returning());
+      const canonicalCwd = await measureRunPhase("workspace.sources.canonicalize", () => canonicalRepositoryRoot(workspace.cwd!));
+      await measureRunPhase("workspace.write", () => tx.insert(executionWorkspaceRepositoryAccess).values({ companyId: input.companyId,
+        rootKey: createHash("sha256").update(canonicalCwd).digest("hex"), canonicalCwd, kind: "repository", scopeId: row.id }));
+      await measureRunPhase("workspace.write", () => tx.insert(activityLog).values({ companyId: input.companyId, actorType: input.actor.type === "agent" ? "agent" : "user", actorId: input.actor.agentId ?? input.actor.userId ?? "local-board", action: "workspace.repository.requested", entityType: "execution_workspace", entityId: workspace.id, runId: input.actor.runId ?? null, responsibleUserId: userId, details: { repositoryId: row.id, issueId: input.issueId, applies: "next_normal_admission" } }));
       return row;
     });
     return { kind: "requires_next_admission" as const, applies: "next_normal_admission" as const, operationId: receipt.id, repository: receipt, reason: "immutable_admitted_workspace" };
@@ -686,24 +713,24 @@ export function executionWorkspaceRepositoryService(db: Db) {
   }
   type PublicationInput = { companyId: string; issueId: string; workspaceId: string; cwd: string; agentId: string; runId: string };
   async function publicationBinding(tx: DbTransaction, input: PublicationInput) {
-    const [issue] = await tx.select().from(issues).where(and(eq(issues.id, input.issueId), eq(issues.companyId, input.companyId)));
-    const [workspace] = await tx.select().from(executionWorkspaces).where(and(eq(executionWorkspaces.id, input.workspaceId), eq(executionWorkspaces.companyId, input.companyId)));
+    const [issue] = await measureRunPhase("workspace.sources.inventory", async () => tx.select().from(issues).where(and(eq(issues.id, input.issueId), eq(issues.companyId, input.companyId))));
+    const [workspace] = await measureRunPhase("workspace.sources.inventory", async () => tx.select().from(executionWorkspaces).where(and(eq(executionWorkspaces.id, input.workspaceId), eq(executionWorkspaces.companyId, input.companyId))));
     if (!issue || !workspace || issue.executionWorkspaceId !== workspace.id || !workspace.cwd
-      || await canonicalRepositoryRoot(workspace.cwd) !== await canonicalRepositoryRoot(input.cwd)) {
+      || await measureRunPhase("workspace.sources.canonicalize", () => canonicalRepositoryRoot(workspace.cwd!)) !== await measureRunPhase("workspace.sources.canonicalize", () => canonicalRepositoryRoot(input.cwd))) {
       throw forbidden("Task repository publication binding is unavailable");
     }
     await requiresAnonymousAcquisition(tx, { issue, workspace, agentId: input.agentId, runId: input.runId });
-    const [run] = await tx.select({ responsibleUserId: heartbeatRuns.responsibleUserId }).from(heartbeatRuns).where(and(eq(heartbeatRuns.id, input.runId), eq(heartbeatRuns.companyId, input.companyId)));
-    const { actor } = await retainedRootReader(tx, input.companyId, {
+    const [run] = await measureRunPhase("workspace.sources.inventory", async () => tx.select({ responsibleUserId: heartbeatRuns.responsibleUserId }).from(heartbeatRuns).where(and(eq(heartbeatRuns.id, input.runId), eq(heartbeatRuns.companyId, input.companyId))));
+    const { actor } = await measureRunPhase("workspace.readers.identity", () => retainedRootReader(tx, input.companyId, {
       scopeId: input.runId, issueId: input.issueId, agentId: input.agentId, responsibleUserId: run?.responsibleUserId ?? null,
-    });
+    }));
     return { workspace, actor };
   }
   async function guardForPublication(input: PublicationInput) {
-    await db.transaction(async tx => {
+    await measureRunTransaction(db, "workspace", async tx => {
       await lockExecutionWorkspaceRootProvenance(tx, input.companyId);
       const { workspace } = await publicationBinding(tx, input);
-      await authorizeRootAccess(tx, { companyId: input.companyId, cwd: input.cwd, sourceWorkspaceId: workspace.id, reader: input });
+      await authorizeRootAccess(tx, { companyId: input.companyId, cwd: input.cwd!, sourceWorkspaceId: workspace.id, reader: input });
     });
   }
   async function prepareConfiguredRepositorySource(input: PublicationInput & { sourceCwd: string }): Promise<() => Promise<void>> {
@@ -717,35 +744,33 @@ export function executionWorkspaceRepositoryService(db: Db) {
         commonRoot = source.commonRoot;
         for (const cwd of source.roots) roots.add(cwd);
       } catch (error) { sourceFailure = error; }
-      const denied = await db.transaction(async tx => {
+      const denied = await measureRunTransaction(db, "workspace", async tx => {
         await lockExecutionWorkspaceRootProvenance(tx, input.companyId);
         const { workspace, actor } = await publicationBinding(tx, input);
-        const canonicalCwd = await canonicalRepositoryRoot(input.cwd);
+        const canonicalCwd = await measureRunPhase("workspace.sources.canonicalize", () => canonicalRepositoryRoot(input.cwd));
         const rootKey = createHash("sha256").update(canonicalCwd).digest("hex");
-        const existing = await tx.select().from(executionWorkspaceRepositoryAccess).where(and(
-          eq(executionWorkspaceRepositoryAccess.companyId, input.companyId), eq(executionWorkspaceRepositoryAccess.rootKey, rootKey)));
+        const existing = await measureRunPhase("workspace.sources.inventory", async () => tx.select().from(executionWorkspaceRepositoryAccess).where(and(
+          eq(executionWorkspaceRepositoryAccess.companyId, input.companyId), eq(executionWorkspaceRepositoryAccess.rootKey, rootKey))));
         if (existing.some(entry => entry.canonicalCwd !== canonicalCwd)) throw forbidden("Repository root identity conflicts with its retained access record");
+        const existingKeys = repositoryAccessKeys(existing);
         let failure = sourceFailure;
         for (const cwd of roots) {
           try {
-            const source = await resolveRepositoryRootSources(tx, { companyId: input.companyId, cwd });
+            const source = await measureRunPhase("workspace.sources", () => resolveRepositoryRootSources(tx, { companyId: input.companyId, cwd }));
             for (const id of source.sourceIds) workspaceIds.add(id);
             for (const id of source.sourceProjectIds) projectIds.add(id);
           } catch (error) { failure ??= error; }
         }
-        try { await assertTaskWorkspacesReadAccess(tx, actor, input.companyId, workspaceIds); }
+        try { await measureRunPhase("workspace.readers.sources", () => assertTaskWorkspacesReadAccess(tx, actor, input.companyId, workspaceIds)); }
         catch (error) { failure ??= error; }
         for (const id of projectIds) {
-          try { await assertRootProjectAccess(tx, actor, input.companyId, new Set([id])); }
+          try { await measureRunPhase("workspace.readers.projects", () => assertRootProjectAccess(tx, actor, input.companyId, new Set([id]))); }
           catch (error) { failure ??= error; }
         }
-        for (const [kind, ids] of [["isolated_source", workspaceIds], ["isolated_project_source", projectIds]] as const) {
-          for (const scopeId of ids) if (!existing.some(entry => entry.kind === kind && entry.scopeId === scopeId)) {
-            await tx.insert(executionWorkspaceRepositoryAccess).values({ companyId: input.companyId, rootKey, canonicalCwd, kind, scopeId });
-          }
-        }
+        await retainRepositorySources(tx, { companyId: input.companyId, canonicalCwd, rootKey }, existingKeys,
+          [["isolated_source", workspaceIds], ["isolated_project_source", projectIds]]);
         try {
-          await authorizeRootAccess(tx, { companyId: input.companyId, cwd: input.cwd, sourceWorkspaceId: workspace.id,
+          await authorizeRootAccess(tx, { companyId: input.companyId, cwd: input.cwd!, sourceWorkspaceId: workspace.id,
             actor, issueId: input.issueId });
         } catch (error) { failure ??= error; }
         if (!failure && commonRoot) {

server/src/services/execution-workspace-source-roots.ts

@@ -21,6 +21,27 @@ export async function canonicalRepositoryRoot(cwd: string): Promise<string> {
   }
 }
 
+export async function resolveRepositoryRoots(cwds: Iterable<string>, requireExisting = false): Promise<Map<string, string | null>> {
+  const paths = [...new Set(cwds)];
+  const roots = new Map<string, string | null>();
+  let next = 0;
+  const settled = await Promise.allSettled(Array.from({ length: Math.min(16, paths.length) }, async () => {
+    while (next < paths.length) {
+      const cwd = paths[next++];
+      const root = await (requireExisting ? fs.realpath(cwd) : canonicalRepositoryRoot(cwd))
+        .catch((error: NodeJS.ErrnoException) => {
+          if (["ENOTDIR", "EACCES", "EPERM", "ELOOP", "ENAMETOOLONG"].includes(error.code ?? "")
+            || (requireExisting && error.code === "ENOENT")) return null;
+          throw error;
+        });
+      roots.set(cwd, root);
+    }
+  }));
+  const rejected = settled.find(result => result.status === "rejected");
+  if (rejected?.status === "rejected") throw rejected.reason;
+  return roots;
+}
+
 export function repositoryRootsOverlap(left: string, right: string): boolean {
   const contains = (parent: string, child: string) => {
     const relative = path.relative(parent, child);

server/src/services/execution-workspaces.ts

@@ -1,5 +1,9 @@
+import { measureRunTransaction } from "./execution-transaction-timing.js";
+import { measureRunPhase } from "./execution-preparation-timing.js";
 import { assertTaskWorkspaceAccess, assertTaskWorkspaceSourceProjectAccess } from "./task-workspace-source-access.js";
-import { verifiedRecordedIsolatedRemoteSource } from "./isolated-remote-workspace-source.js";
+import { verifiedRecordedIsolatedRemoteSource, withIsolatedRepositorySourceLock } from "./isolated-remote-workspace-source.js";
+import { expireReconstructableGitWorkspace, hasPendingGitExpiration } from "./git-workspace-expiration.js";
+import { readGitCacheRestoration, restoreGitCacheBase, GitCacheRestorationFailure } from "./git-workspace-restoration.js";
 import { parseObject } from "../adapters/utils.js";
 import { resolveCoreTrustPreset } from "./trust-preset-resolver.js";
 import { CONTROLLER_PROVISION_COPY_ONLY, workspaceRequiresCopyOnlyProvisioning } from "./workspace-host-provision.js";
@@ -251,9 +255,9 @@ async function preserveWorkspaceOwnershipForPatch(
   tx: DbTransaction, current: typeof executionWorkspaces.$inferSelect, patch: Partial<typeof executionWorkspaces.$inferInsert>,
 ) {
   if (workspaceSourceKeys.some(key => patch[key] !== undefined && patch[key] !== current[key])) {
-    await preserveExecutionWorkspaceRootProvenance(tx, {
+    await measureRunPhase("workspace.provenance", () => preserveExecutionWorkspaceRootProvenance(tx, {
       companyId: current.companyId, workspaceId: current.id, cwd: current.cwd,
-    });
+    }));
   }
   const nextPatch = { ...patch };
   const retainedSources = {
@@ -284,6 +288,9 @@ export type ReopenClosedIsolatedExecutionWorkspaceResult =
   | { ok: false; code: "not_reopenable" | "rebuild_failed"; message: string };
 
 export type ExecutionWorkspaceServiceOptions = {
+  gitCacheExpirationEnabled?: boolean;
+  gitBaseExpirationEnabled?: boolean;
+  gitCacheExpirationDryRun?: boolean;
   terminalWorkspaceInspectionBackoffMs?: number;
   resolvePullRequestDetails?: PullRequestMergeDetailsResolver;
   now?: () => Date;
@@ -1531,8 +1538,8 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic
 
   async function bindTaskWorkspace(companyId: string, issueId: string, workspaceId: string, patch: TaskWorkspaceBindingPatch = {}) {
     return db.transaction(async tx => {
-      const [task] = await tx.select().from(issues).where(and(eq(issues.id, issueId), eq(issues.companyId, companyId))).for("no key update");
-      const [workspace] = await tx.select().from(executionWorkspaces).where(and(eq(executionWorkspaces.id, workspaceId), eq(executionWorkspaces.companyId, companyId))).for("no key update");
+      const [task] = await measureRunPhase("workspace.row_lock", () => tx.select().from(issues).where(and(eq(issues.id, issueId), eq(issues.companyId, companyId))).for("no key update"));
+      const [workspace] = await measureRunPhase("workspace.row_lock", () => tx.select().from(executionWorkspaces).where(and(eq(executionWorkspaces.id, workspaceId), eq(executionWorkspaces.companyId, companyId))).for("no key update"));
       if (!task || !workspace) throw notFound("Task workspace unavailable");
       await tx.update(executionWorkspaces).set({ metadata: { ...workspace.metadata,
         _issuePrivacySources: { ...(workspace.metadata?._issuePrivacySources as Record<string, unknown> ?? {}), [task.id]: true } } }).where(eq(executionWorkspaces.id, workspaceId));
@@ -1572,9 +1579,9 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic
   async function prepareTaskWorkspaceUpdate(input: {
     companyId: string; issueId: string; patch: TaskWorkspaceBindingPatch;
   }, tx: DbTransaction) {
-    const [task] = await tx.select().from(issues).where(and(
+    const [task] = await measureRunPhase("workspace.row_lock", () => tx.select().from(issues).where(and(
       eq(issues.id, input.issueId), eq(issues.companyId, input.companyId),
-    )).for("no key update");
+    )).for("no key update"));
     if (!task) throw notFound("Task not found");
     const fields = ["executionWorkspaceId", "projectWorkspaceId", "executionWorkspacePreference", "executionWorkspaceSettings"] as const;
     if (!fields.some(field => input.patch[field] !== undefined && !isDeepStrictEqual(input.patch[field], task[field]))) return {};
@@ -1929,7 +1936,7 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic
         }
         return input.onSkip();
       }
-      await notifyTerminalWorkspaceWork(tx, `workspace:${input.workspaceId}`);
+      await measureRunPhase("workspace.notification", () => notifyTerminalWorkspaceWork(tx, `workspace:${input.workspaceId}`));
       return input.write({ tx, fresh });
     });
   }
@@ -3130,9 +3137,10 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic
       try {
       if (options.compact !== false) await purgeDeferredCompactionTrash(signal);
       const baseCandidateFilter = and(
-        inArray(executionWorkspaces.status, ["active", "idle", "in_review"]),
-        isNull(executionWorkspaces.closedAt),
         sql<boolean>`${executionWorkspaces.sourceIssueId} IS NOT NULL`,
+        or(and(inArray(executionWorkspaces.status, ["active", "idle", "in_review"]), isNull(executionWorkspaces.closedAt)),
+          opts.gitCacheExpirationEnabled ? and(inArray(executionWorkspaces.status, ["archived", "cleanup_failed"]),
+            sql`${executionWorkspaces.metadata}->'gitCacheExpiration'->>'completed' = 'false'`) : undefined),
       );
       // Continue the scan after the previous sweep's last row. The keyset
       // predicate uses the same (updatedAt, id) order as the query, so each
@@ -3220,6 +3228,21 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic
           result.skippedNonTerminalTree += 1;
           continue;
         }
+        if (opts.gitCacheExpirationEnabled && (hasPendingGitExpiration(workspace)
+          || (workspace.mode === "isolated_workspace" && workspace.strategyType === "git_worktree"))) {
+          const expiration = await expireReconstructableGitWorkspace(db, workspace, {
+            dryRun: opts.gitCacheExpirationDryRun ?? false, expireBase: opts.gitBaseExpirationEnabled ?? false,
+            now: now(), signal, beforeDelete: opts.beforeTerminalWorkspaceCleanup,
+          });
+          terminalWorkspaceDeadlines.set(workspace.id, now().getTime() + 60 * 60 * 1000);
+          if (expiration.candidate) result.eligible += 1;
+          if (expiration.expired) result.archived += 1;
+          logger.info({ event: "execution_workspace.git_cache_expiration", executionWorkspaceId: workspace.id,
+            companyId: workspace.companyId, dryRun: opts.gitCacheExpirationDryRun ?? false, ...expiration },
+          "Git workspace cache expiration inspected");
+          // A failed reconstruction proof must not fall through to legacy deletion.
+          continue;
+        }
         const latestTerminalAt = Math.max(...issueTree.map(issue => issueTerminalTimestamp(issue)!.getTime()));
         // Compaction runs once per terminal cycle, after its own delay and
         // before the archive cooldown, so a workspace the reaper cannot archive
@@ -3582,36 +3605,36 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic
     },
 
     create: async (data: typeof executionWorkspaces.$inferInsert) => {
-      const row = await db.transaction(async tx => {
-        await notifyTerminalWorkspaceWork(tx, data.id ? `workspace:${data.id}`
-          : data.sourceIssueId ? `issue:${data.sourceIssueId}` : undefined);
-        return tx.insert(executionWorkspaces).values(data).returning().then(rows => rows[0] ?? null);
+      const row = await measureRunTransaction(db, "workspace", async tx => {
+        await measureRunPhase("workspace.notification", () => notifyTerminalWorkspaceWork(tx, data.id ? `workspace:${data.id}`
+          : data.sourceIssueId ? `issue:${data.sourceIssueId}` : undefined));
+        return measureRunPhase("workspace.write", () => tx.insert(executionWorkspaces).values(data).returning().then(rows => rows[0] ?? null));
       });
       return row ? toExecutionWorkspace(row) : null;
     },
 
     update: async (id: string, patch: Partial<typeof executionWorkspaces.$inferInsert>) => {
       if (patch.metadata === undefined && !workspaceSourceKeys.some(key => patch[key] !== undefined)) {
-        const row = await db.transaction(async tx => {
-          await notifyTerminalWorkspaceWork(tx, `workspace:${id}`);
-          return tx.update(executionWorkspaces).set({ ...patch, updatedAt: new Date() })
-            .where(eq(executionWorkspaces.id, id)).returning().then(rows => rows[0] ?? null);
+        const row = await measureRunTransaction(db, "workspace", async tx => {
+          await measureRunPhase("workspace.notification", () => notifyTerminalWorkspaceWork(tx, `workspace:${id}`));
+          return measureRunPhase("workspace.write", () => tx.update(executionWorkspaces).set({ ...patch, updatedAt: new Date() })
+            .where(eq(executionWorkspaces.id, id)).returning().then(rows => rows[0] ?? null));
         });
         return row ? toExecutionWorkspace(row) : null;
       }
-      const [identity] = await db.select({ companyId: executionWorkspaces.companyId }).from(executionWorkspaces)
-        .where(eq(executionWorkspaces.id, id));
+      const [identity] = await measureRunPhase("workspace.identity_read", () => db.select({ companyId: executionWorkspaces.companyId }).from(executionWorkspaces)
+        .where(eq(executionWorkspaces.id, id)));
       if (!identity) return null;
-      return db.transaction(async tx => {
+      return measureRunTransaction(db, "workspace", async tx => {
         await lockExecutionWorkspaceRootProvenance(tx, identity.companyId);
-        const [current] = await tx.select().from(executionWorkspaces).where(and(
+        const [current] = await measureRunPhase("workspace.row_lock", () => tx.select().from(executionWorkspaces).where(and(
           eq(executionWorkspaces.id, id), eq(executionWorkspaces.companyId, identity.companyId),
-        )).for("no key update");
+        )).for("no key update"));
         if (!current) return null;
-        const nextPatch = await preserveWorkspaceOwnershipForPatch(tx, current, patch);
-        await notifyTerminalWorkspaceWork(tx, `workspace:${id}`);
-        const [row] = await tx.update(executionWorkspaces).set({ ...nextPatch, updatedAt: new Date() })
-          .where(and(eq(executionWorkspaces.id, id), eq(executionWorkspaces.companyId, current.companyId))).returning();
+        const nextPatch = await measureRunPhase("workspace.provenance", () => preserveWorkspaceOwnershipForPatch(tx, current, patch));
+        await measureRunPhase("workspace.notification", () => notifyTerminalWorkspaceWork(tx, `workspace:${id}`));
+        const [row] = await measureRunPhase("workspace.write", () => tx.update(executionWorkspaces).set({ ...nextPatch, updatedAt: new Date() })
+          .where(and(eq(executionWorkspaces.id, id), eq(executionWorkspaces.companyId, current.companyId))).returning());
         return row ? toExecutionWorkspace(row) : null;
       });
     },
@@ -3630,13 +3653,29 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic
         eq(issues.executionWorkspaceId, input.workspaceId),
         await issueReadSqlCondition(db, input.authorizationActor),
       ));
-      const snapshot = boundSnapshot ? await db.select().from(executionWorkspaces).where(and(
+      const authorizedSnapshot = boundSnapshot ? await db.select().from(executionWorkspaces).where(and(
         eq(executionWorkspaces.id, input.workspaceId), eq(executionWorkspaces.companyId, issue.companyId),
         eq(executionWorkspaces.mode, "isolated_workspace"),
         await executionWorkspaceReadSqlCondition(db, input.authorizationActor),
       )).then(rows => rows[0] ?? null) : null;
-      if (!snapshot) return { ok: false, code: "not_reopenable", message: "Execution workspace is not reopenable" };
+      if (!authorizedSnapshot) return { ok: false, code: "not_reopenable", message: "Execution workspace is not reopenable" };
+      await assertTaskWorkspaceAccess(db, input.authorizationActor, issue.companyId, authorizedSnapshot.id);
+      const sourceIssueId = authorizedSnapshot.sourceIssueId;
+      const reopen = async (): Promise<ReopenClosedIsolatedExecutionWorkspaceResult> => {
+      const freshSnapshot = await db.select().from(executionWorkspaces).where(and(
+        eq(executionWorkspaces.id, input.workspaceId), eq(executionWorkspaces.companyId, issue.companyId),
+        eq(executionWorkspaces.mode, "isolated_workspace"),
+        await executionWorkspaceReadSqlCondition(db, input.authorizationActor),
+      )).then(rows => rows[0] ?? null);
+      if (!freshSnapshot || freshSnapshot.sourceIssueId !== sourceIssueId) {
+        return { ok: false, code: "not_reopenable", message: "Execution workspace source changed during reopen" };
+      }
+      const snapshot = freshSnapshot;
       await assertTaskWorkspaceAccess(db, input.authorizationActor, issue.companyId, snapshot.id);
+      let recipe: ReturnType<typeof readGitCacheRestoration>;
+      try { recipe = readGitCacheRestoration(snapshot); }
+      catch { return { ok: false, code: "rebuild_failed", message: new GitCacheRestorationFailure().message }; }
+      const gitAuth = createGitRemoteAuthProvider(db, snapshot.companyId, { issueId: snapshot.sourceIssueId ?? issue.id });
       const [
         { ensurePersistedExecutionWorkspaceAvailable },
         { workspaceOperationService },
@@ -3664,8 +3703,14 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic
           .then((rows) => parseProjectExecutionWorkspacePolicy(rows[0]?.executionWorkspacePolicy)),
       ]);
       let resolvedBaseCwd: string | null;
-      try { resolvedBaseCwd = await verifiedRecordedIsolatedRemoteSource(snapshot) ?? projectWorkspace?.cwd ?? null; }
-      catch { return { ok: false, code: "rebuild_failed", message: "Failed to rebuild the execution workspace" }; }
+      try {
+        resolvedBaseCwd = recipe && isClosedExecutionWorkspaceStatus(snapshot.status)
+          ? await restoreGitCacheBase(snapshot, recipe, gitAuth)
+          : await verifiedRecordedIsolatedRemoteSource(snapshot) ?? projectWorkspace?.cwd ?? null;
+      } catch {
+        return { ok: false, code: "rebuild_failed", message: recipe
+          ? new GitCacheRestorationFailure().message : "Failed to rebuild the execution workspace" };
+      }
       if (isClosedExecutionWorkspaceStatus(snapshot.status) && resolvedBaseCwd == null && snapshot.strategyType === "git_worktree" && snapshot.projectId) {
         try {
           const managedWorkspace = await ensureManagedProjectWorkspace({
@@ -3687,7 +3732,7 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic
           });
           resolvedBaseCwd = managedWorkspace.cwd;
         } catch {
-          return { ok: false, code: "rebuild_failed", message: "Failed to rebuild the execution workspace" };
+          return { ok: false, code: "rebuild_failed", message: recipe ? new GitCacheRestorationFailure().message : "Failed to rebuild the execution workspace" };
         }
       }
       let capturedSource: IsolatedWorkspaceSourceAccess | null = null;
@@ -3707,7 +3752,7 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic
             actor: input.authorizationActor, captured: capturedSource, issueId: issue.id, phase: "preparing",
           });
         } catch {
-          return { ok: false, code: "rebuild_failed", message: "Failed to rebuild the execution workspace" };
+          return { ok: false, code: "rebuild_failed", message: recipe ? new GitCacheRestorationFailure().message : "Failed to rebuild the execution workspace" };
         }
       }
       const result = await db.transaction(async (tx): Promise<ReopenClosedIsolatedExecutionWorkspaceResult> => {
@@ -3777,6 +3822,7 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic
         const nextMetadata = bumpExecutionWorkspaceLifecycleGeneration(
           row.metadata as Record<string, unknown> | null,
         );
+        delete nextMetadata.gitCacheExpiration;
         const recorder = workspaceOperationService(db).createRecorder({
           companyId: row.companyId,
           executionWorkspaceId: row.id,
@@ -3813,6 +3859,7 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic
           if (copyOnlyProvisioning) nextMetadata[CONTROLLER_PROVISION_COPY_ONLY] = true;
           const realized = await ensurePersistedExecutionWorkspaceAvailable({
             db: tx as unknown as Db,
+            resolveGitAuth: gitAuth,
             skipHostProvision: trust.kind === "low_trust_review" || copyOnlyProvisioning,
             beforeSourceCopy: verifyCopyIdentity,
             afterSourceCopy: verifyCopyIdentity,
@@ -3866,7 +3913,7 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic
           // generation so a queued cleanup that captured the old generation does
           // nothing. Clear cleanupEligibleAt so the reaper does not destroy the
           // half-built worktree while a later reopen retries.
-          await notifyTerminalWorkspaceWork(tx, `workspace:${row.id}`);
+          await measureRunPhase("workspace.notification", () => notifyTerminalWorkspaceWork(tx, `workspace:${row.id}`));
           await tx
             .update(executionWorkspaces)
             .set({
@@ -3907,7 +3954,7 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic
               generation: nextGeneration,
             },
           });
-          return { ok: false, code: "rebuild_failed", message: "Failed to rebuild the execution workspace" };
+          return { ok: false, code: "rebuild_failed", message: recipe ? new GitCacheRestorationFailure().message : "Failed to rebuild the execution workspace" };
         }
 
         // The rebuild succeeded. Publish the row as active in one write, and clear
@@ -3916,8 +3963,12 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic
         // the issue out of the terminal state only after this reopen returns. The
         // flag stops the terminal reaper and the archive route from archiving and
         // destroying the rebuilt worktree in that window.
+        // Keep the refresh guard after reopening so normal preparation cannot
+        // advance the saved checkout to upstream HEAD. Its intact branch may now
+        // receive newer task commits; the next expiration records a fresh recipe.
+        if (recipe) nextMetadata.gitCacheRestoration = { ...recipe, restoredAt: now().toISOString() };
         const activeMetadata = setMetadataReopenPendingConsumption(nextMetadata, now());
-        await notifyTerminalWorkspaceWork(tx, `workspace:${row.id}`);
+        await measureRunPhase("workspace.notification", () => notifyTerminalWorkspaceWork(tx, `workspace:${row.id}`));
         const activeRow = await tx
           .update(executionWorkspaces)
           .set({
@@ -3933,7 +3984,7 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic
           .returning()
           .then((rows) => rows[0] ?? null);
         if (!activeRow) {
-          return { ok: false, code: "rebuild_failed", message: "Failed to rebuild the execution workspace" };
+          return { ok: false, code: "rebuild_failed", message: recipe ? new GitCacheRestorationFailure().message : "Failed to rebuild the execution workspace" };
         }
         logger.info(
           {
@@ -3965,7 +4016,7 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic
         return { ok: true, reopened: true, workspace: toExecutionWorkspace(activeRow), generation: nextGeneration };
       }).catch((error): ReopenClosedIsolatedExecutionWorkspaceResult => {
         logger.warn({ error, executionWorkspaceId: snapshot.id }, "Execution workspace reopen failed");
-        return { ok: false, code: "rebuild_failed", message: "Failed to rebuild the execution workspace" };
+        return { ok: false, code: "rebuild_failed", message: recipe ? new GitCacheRestorationFailure().message : "Failed to rebuild the execution workspace" };
       });
       if (capturedSource && authorizedTarget && resolvedBaseCwd) {
         try {
@@ -3979,10 +4030,14 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic
               workspaceId: snapshot.id, issue, actor, expectedGeneration: result.generation,
             });
           }
-          return { ok: false, code: "rebuild_failed", message: "Failed to rebuild the execution workspace" };
+          return { ok: false, code: "rebuild_failed", message: recipe ? new GitCacheRestorationFailure().message : "Failed to rebuild the execution workspace" };
         }
       }
       return result;
+      };
+      return authorizedSnapshot.sourceIssueId
+        ? withIsolatedRepositorySourceLock(db, authorizedSnapshot.companyId, authorizedSnapshot.sourceIssueId, reopen)
+        : reopen();
     },
 
     // Clear the reopen-pending flag after a caller failed to consume a reopened
@@ -4085,12 +4140,12 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic
       return db.transaction(async (tx) => {
         await lockExecutionWorkspaceRootProvenance(tx, identity.companyId);
         await acquireExecutionWorkspaceLifecycleLock(tx, input.id);
-        const fresh = await tx
+        const fresh = await measureRunPhase("workspace.row_lock", () => tx
           .select()
           .from(executionWorkspaces)
           .where(and(eq(executionWorkspaces.id, input.id), eq(executionWorkspaces.companyId, identity.companyId)))
           .for("no key update")
-          .then((rows) => rows[0] ?? null);
+          .then((rows) => rows[0] ?? null));
         if (!fresh || isClosedExecutionWorkspaceStatus(fresh.status)) {
           // The row is missing or already closed by a concurrent path. Do not
           // archive again.
@@ -4103,14 +4158,14 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic
           // rebuilt worktree. The route maps this to HTTP 409.
           return { outcome: "reopen_pending" };
         }
-        const nextPatch = await preserveWorkspaceOwnershipForPatch(tx, fresh, input.patch);
+        const nextPatch = await measureRunPhase("workspace.provenance", () => preserveWorkspaceOwnershipForPatch(tx, fresh, input.patch));
         const baseMetadata =
           (nextPatch.metadata as Record<string, unknown> | null | undefined)
           ?? (fresh.metadata as Record<string, unknown> | null);
         const archiveMetadata = clearMetadataReopenPendingConsumption(
           bumpExecutionWorkspaceLifecycleGeneration(baseMetadata),
         );
-        await notifyTerminalWorkspaceWork(tx, `workspace:${input.id}`);
+        await measureRunPhase("workspace.notification", () => notifyTerminalWorkspaceWork(tx, `workspace:${input.id}`));
         const archived = await tx
           .update(executionWorkspaces)
           .set({
@@ -4371,11 +4426,18 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic
             branchName: inspection.toBranch,
             updatedAt: now,
           };
+          const restoration = readGitCacheRestoration(lockedRow);
+          if (restoration) {
+            updatePatch.metadata = {
+              ...lockedRow.metadata,
+              gitCacheRestoration: { ...restoration, branchName: inspection.toBranch },
+            };
+          }
           if (lockedWorkspace.name === inspection.fromBranch) {
             updatePatch.name = inspection.toBranch;
           }
 
-          await notifyTerminalWorkspaceWork(tx, `workspace:${lockedWorkspace.id}`);
+          await measureRunPhase("workspace.notification", () => notifyTerminalWorkspaceWork(tx, `workspace:${lockedWorkspace.id}`));
           const [branchUpdatedRow] = await tx
             .update(executionWorkspaces)
             .set(updatePatch)

server/src/services/git-workspace-expiration.ts

@@ -0,0 +1,230 @@
+import fs from "node:fs/promises";
+import path from "node:path";
+import { and, eq, ne, sql } from "drizzle-orm";
+import { executionWorkspaces, issues, heartbeatRuns, nativeRunFinalizations, environmentLeases,
+  workspaceRuntimeServices, issueRecoveryActions, projectWorkspaces, withDedicatedDbConnection, type Db } from "@paperclipai/db";
+import { createGitRemoteAuthProvider } from "./git-credentials.js";
+import { withIsolatedRepositorySourceLock, verifiedRecordedIsolatedRemoteSource } from "./isolated-remote-workspace-source.js";
+import { assertReproducibleGitFiles, assertReproducibleBaseGitStorage, inspectGitCacheRecipe, readGitCacheRestoration,
+  verifyRemoteGitCommit, gitCacheCommand, readGitCacheExpiration, finishGitCacheTrash, type GitCacheExpiration, GIT_CACHE_RESTORATION_KEY, GIT_CACHE_EXPIRATION_REASON } from "./git-workspace-restoration.js";
+
+
+async function allocatedBytes(target: string, shouldContinue: () => boolean): Promise<number> {
+  if (!shouldContinue()) throw new Error("scan_stopped");
+  const entry = await fs.lstat(target);
+  if (!entry.isDirectory() && !entry.isFile() && !entry.isSymbolicLink()) throw new Error("unsupported_filesystem_entry");
+  let bytes = entry.blocks * 512;
+  if (entry.isDirectory()) for (const name of await fs.readdir(target)) bytes += await allocatedBytes(path.join(target, name), shouldContinue);
+  return bytes;
+}
+function skipReason(error: unknown) {
+  return error instanceof Error && /^[a-z][a-z_]+$/.test(error.message) ? error.message : "inspection_failed";
+}
+
+const EXPIRATION_KEY = "gitCacheExpiration";
+const DAY_MS = 24 * 60 * 60 * 1000;
+type Workspace = typeof executionWorkspaces.$inferSelect;
+type Tx = Parameters<Parameters<Db["transaction"]>[0]>[0];
+type Expiration = GitCacheExpiration;
+export type GitExpirationResult = { candidate: boolean; expired: boolean; reclaimedBytes: number;
+  candidateBytes: number; baseExpired: boolean; skipReason: string | null };
+
+/** Re-read the whole tree and every runtime/recovery consumer immediately before deletion. */
+async function eligible(tx: Tx, workspace: Workspace, now: Date) {
+  if (workspace.metadata?.reopenPendingConsumption === true) return false;
+  const rows = await tx.execute<{ allowed: boolean }>(sql`
+    WITH RECURSIVE tree AS (
+      SELECT id, status, completed_at, cancelled_at, updated_at FROM ${issues}
+      WHERE company_id = ${workspace.companyId} AND id = ${workspace.sourceIssueId}
+      UNION
+      SELECT c.id, c.status, c.completed_at, c.cancelled_at, c.updated_at FROM ${issues} c
+      JOIN tree p ON c.parent_id = p.id WHERE c.company_id = ${workspace.companyId}
+    ), relevant_runs AS (
+      SELECT r.* FROM ${heartbeatRuns} r WHERE r.company_id = ${workspace.companyId} AND (
+        r.issue_id IN (SELECT id FROM tree) OR r.native_issue_id IN (SELECT id FROM tree)
+        OR r.context_snapshot->>'issueId' IN (SELECT id::text FROM tree)
+        OR r.context_snapshot->>'taskId' IN (SELECT id::text FROM tree)
+        OR r.context_snapshot->>'executionWorkspaceId' = ${workspace.id}
+        OR r.id IN (SELECT unnest(ARRAY[checkout_run_id, execution_run_id]) FROM ${issues}
+          WHERE company_id = ${workspace.companyId} AND execution_workspace_id = ${workspace.id})
+      )
+    ) SELECT (
+      EXISTS (SELECT 1 FROM tree)
+      AND NOT EXISTS (SELECT 1 FROM tree WHERE status NOT IN ('done', 'cancelled')
+        OR coalesce(CASE WHEN status = 'done' THEN completed_at ELSE cancelled_at END, updated_at) > ${new Date(now.getTime() - DAY_MS).toISOString()}::timestamptz)
+      AND NOT EXISTS (SELECT 1 FROM ${issues} WHERE company_id = ${workspace.companyId}
+        AND execution_workspace_id = ${workspace.id} AND status NOT IN ('done', 'cancelled'))
+      AND NOT EXISTS (SELECT 1 FROM relevant_runs WHERE status NOT IN ('succeeded', 'failed', 'timed_out', 'cancelled', 'interrupted')
+        OR finished_at IS NULL OR controller_lease_expires_at > now())
+      AND NOT EXISTS (SELECT 1 FROM ${issueRecoveryActions} WHERE company_id = ${workspace.companyId}
+        AND source_issue_id IN (SELECT id FROM tree) AND status IN ('active', 'escalated'))
+      AND NOT EXISTS (SELECT 1 FROM ${nativeRunFinalizations} f WHERE f.company_id = ${workspace.companyId}
+        AND (f.issue_id IN (SELECT id FROM tree) OR f.run_id IN (SELECT id FROM relevant_runs))
+        AND (f.phase <> 'committed' OR f.lease_owner IS NOT NULL OR f.recovery_state IS NOT NULL))
+      AND NOT EXISTS (SELECT 1 FROM ${environmentLeases} l WHERE l.company_id = ${workspace.companyId}
+        AND (l.execution_workspace_id = ${workspace.id} OR l.heartbeat_run_id IN (SELECT id FROM relevant_runs))
+        AND (l.status = 'pending_cleanup' OR l.cleanup_status = 'failed' OR l.metadata ? 'nativeWorkspaceExportResume'
+          OR (l.released_at IS NULL AND NOT coalesce(l.status = 'retained' AND l.lease_policy = 'reuse_by_environment' AND l.cleanup_status = 'success', false))))
+      AND NOT EXISTS (SELECT 1 FROM ${workspaceRuntimeServices} WHERE company_id = ${workspace.companyId}
+        AND (execution_workspace_id = ${workspace.id} OR cwd = ${workspace.cwd}
+          OR left(cwd, length(${workspace.cwd}) + 1) = ${workspace.cwd} || '/') AND status <> 'stopped')
+      AND NOT EXISTS (SELECT 1 FROM ${executionWorkspaces} other WHERE other.id <> ${workspace.id}
+        AND (other.cwd = ${workspace.cwd} OR other.provider_ref = ${workspace.cwd}
+          OR left(other.cwd, length(${workspace.cwd}) + 1) = ${workspace.cwd} || '/'))
+      AND NOT EXISTS (SELECT 1 FROM ${projectWorkspaces} WHERE cwd = ${workspace.cwd}
+        OR left(cwd, length(${workspace.cwd}) + 1) = ${workspace.cwd} || '/')
+    ) AS allowed`);
+  return rows[0]?.allowed === true;
+}
+
+export function hasPendingGitExpiration(workspace: Workspace) {
+  const state = workspace.metadata?.[EXPIRATION_KEY] as Expiration | undefined;
+  return state?.schema === "paperclip.git-cache-expiration.v1" && !state.completed;
+}
+
+async function exists(target: string) {
+  return fs.lstat(target).then(() => true, (error: NodeJS.ErrnoException) => { if (error.code === "ENOENT") return false; throw error; });
+}
+
+/** Each ref/reflog object must be recoverable, and no other checkout may consume the base. */
+async function inspectBase(db: Db, workspace: Workspace, signal?: AbortSignal, allowCurrentWorktree = false) {
+  const recipe = readGitCacheRestoration(workspace)!;
+  if (recipe.sourceKind !== "isolated_remote") throw new Error("shared_base_preserved");
+  const other = await db.select().from(executionWorkspaces).where(and(ne(executionWorkspaces.id, workspace.id),
+    sql`(${executionWorkspaces.metadata}->'isolatedRemoteSource'->>'companyId' = ${workspace.companyId}
+      AND ${executionWorkspaces.metadata}->'isolatedRemoteSource'->>'issueId' = ${workspace.sourceIssueId})
+      OR ${executionWorkspaces.cwd} = ${recipe.sourcePath} OR ${executionWorkspaces.providerRef} = ${recipe.sourcePath}`));
+  if (other.some(row => !["archived", "cleanup_failed"].includes(row.status)
+    || !readGitCacheRestoration(row) || !readGitCacheExpiration(row)?.workspaceRemoved)) throw new Error("base_has_consumers");
+  const [projectConsumer] = await db.select({ id: projectWorkspaces.id }).from(projectWorkspaces).where(eq(projectWorkspaces.cwd, recipe.sourcePath)).limit(1);
+  if (projectConsumer) throw new Error("base_has_consumers");
+  const [service] = await db.select({ id: workspaceRuntimeServices.id }).from(workspaceRuntimeServices).where(sql`
+    ${workspaceRuntimeServices.status} <> 'stopped' AND (${workspaceRuntimeServices.cwd} = ${recipe.sourcePath}
+      OR left(${workspaceRuntimeServices.cwd}, length(${recipe.sourcePath}) + 1) = ${recipe.sourcePath} || '/')`).limit(1);
+  if (service) throw new Error("base_has_live_services");
+  await verifiedRecordedIsolatedRemoteSource(workspace);
+  const worktrees = (await gitCacheCommand(recipe.sourcePath, ["worktree", "list", "--porcelain"], signal))
+    .split("\n").filter(line => line.startsWith("worktree ")).map(line => line.slice(9));
+  const allowed = [await fs.realpath(recipe.sourcePath), ...(allowCurrentWorktree ? [await fs.realpath(recipe.worktreePath)] : [])];
+  if (worktrees.some(cwd => !allowed.includes(cwd))) throw new Error("base_has_worktrees");
+  await assertReproducibleGitFiles(recipe.sourcePath, signal);
+  await assertReproducibleBaseGitStorage(recipe.sourcePath, signal);
+  const refs = (await gitCacheCommand(recipe.sourcePath, ["for-each-ref", "--format=%(objectname) %(refname)"], signal)).trim().split("\n").filter(Boolean);
+  if (refs.some(ref => !/^[a-f0-9]{40} refs\/(heads|remotes\/origin)\//.test(ref))) throw new Error("unsupported_base_refs");
+  const commits = new Set([...refs.map(ref => ref.split(" ")[0]!),
+    ...(await gitCacheCommand(recipe.sourcePath, ["reflog", "show", "--all", "--format=%H"], signal)).trim().split("\n").filter(Boolean)]);
+  if (commits.size > 50) throw new Error("base_ref_verification_budget");
+  const auth = createGitRemoteAuthProvider(db, workspace.companyId, { issueId: workspace.sourceIssueId! });
+  for (const commit of commits) await verifyRemoteGitCommit(recipe.repoUrl, commit, auth, signal);
+  // Unreachable objects may contain unique rewritten work not present in a reflog.
+  const unreachable = await gitCacheCommand(recipe.sourcePath, ["fsck", "--no-reflogs", "--unreachable", "--no-progress"], signal);
+  if (unreachable.trim()) throw new Error("unreachable_base_objects");
+  return allocatedBytes(recipe.sourcePath, () => { signal?.throwIfAborted(); return true; });
+}
+
+export async function expireReconstructableGitWorkspace(db: Db, snapshot: Workspace, options: {
+  dryRun: boolean; expireBase: boolean; now: Date; signal?: AbortSignal;
+  beforeDelete?: (workspace: Workspace) => Promise<void>;
+}): Promise<GitExpirationResult> {
+  const result: GitExpirationResult = { candidate: false, expired: false, reclaimedBytes: 0,
+    candidateBytes: 0, baseExpired: false, skipReason: null };
+  if (!snapshot.sourceIssueId) return { ...result, skipReason: "missing_source_issue" };
+  try {
+    return await withIsolatedRepositorySourceLock(db, snapshot.companyId, snapshot.sourceIssueId, () =>
+      withDedicatedDbConnection(db, connection => connection.transaction(async tx => {
+        const acquired = await tx.execute<{ locked: boolean }>(sql`select pg_try_advisory_xact_lock(hashtextextended(${`execution_workspace_lifecycle:${snapshot.id}`}, 0)) as locked`);
+        if (!acquired[0]?.locked) throw new Error("lifecycle_busy");
+        let workspace = (await db.select().from(executionWorkspaces).where(and(eq(executionWorkspaces.id, snapshot.id), eq(executionWorkspaces.companyId, snapshot.companyId))))[0];
+        if (!workspace || workspace.updatedAt.getTime() !== snapshot.updatedAt.getTime()) throw new Error("workspace_changed");
+        if (!await eligible(tx, workspace, options.now)) throw new Error("live_or_unresolved_consumer");
+        let state = ["active", "idle", "in_review"].includes(workspace.status) ? null : readGitCacheExpiration(workspace);
+        // A restored active workspace starts a new expiration cycle and records its new HEAD.
+        if (!hasPendingGitExpiration(workspace) && !["active", "idle", "in_review"].includes(workspace.status)) throw new Error("not_expirable");
+        if (["active", "idle", "in_review"].includes(workspace.status)) state = null;
+        let recipe = state ? readGitCacheRestoration(workspace)! : await inspectGitCacheRecipe(workspace,
+          createGitRemoteAuthProvider(db, workspace.companyId, { issueId: workspace.sourceIssueId! }), options.signal);
+        const present = await exists(recipe.worktreePath);
+        if (!state && !present) throw new Error("missing_unverified_worktree");
+        result.candidate = true;
+        if (present) result.candidateBytes = await allocatedBytes(recipe.worktreePath, () => { options.signal?.throwIfAborted(); return true; });
+        if (options.dryRun) {
+          if (options.expireBase) {
+            try { result.candidateBytes += await inspectBase(db, { ...workspace,
+              metadata: { ...workspace.metadata, [GIT_CACHE_RESTORATION_KEY]: recipe } }, options.signal, true); }
+            catch (error) { result.skipReason = skipReason(error); }
+          }
+          return result;
+        }
+        if (!state) {
+          const generation = Number(workspace.metadata?.lifecycleGeneration ?? 0) + 1;
+          const rawAdmin = (await gitCacheCommand(recipe.worktreePath, ["rev-parse", "--absolute-git-dir"], options.signal)).trim();
+          const worktreeAdmin = path.join(recipe.sourcePath, ".git", "worktrees", path.basename(rawAdmin));
+          const canonicalAdmin = path.join(await fs.realpath(recipe.sourcePath), ".git", "worktrees", path.basename(rawAdmin));
+          if (await fs.realpath(worktreeAdmin) !== canonicalAdmin || await fs.realpath(rawAdmin) !== canonicalAdmin) {
+            throw new Error("unproven_worktree_admin");
+          }
+          state = { schema: "paperclip.git-cache-expiration.v1", generation, workspaceRemoved: false,
+            worktreeTrash: path.join(path.dirname(recipe.worktreePath), `.expired-git-worktree-${workspace.id}-${generation}`),
+            worktreeAdmin, baseTrash: null, completed: false };
+          if (await exists(state.worktreeTrash)) throw new Error("unknown_worktree_trash");
+          // Commit the immutable restoration recipe BEFORE touching disk. The separate
+          // lifecycle-lock connection remains held through deletion and final state.
+          [workspace] = await db.update(executionWorkspaces).set({ status: "archived", closedAt: options.now,
+            cleanupEligibleAt: options.now, cleanupReason: GIT_CACHE_EXPIRATION_REASON, updatedAt: options.now,
+            metadata: { ...workspace.metadata, lifecycleGeneration: generation, [GIT_CACHE_RESTORATION_KEY]: recipe, [EXPIRATION_KEY]: state },
+          }).where(eq(executionWorkspaces.id, workspace.id)).returning();
+        }
+        if (!workspace) throw new Error("workspace_changed");
+        const persist = async () => {
+          [workspace] = await db.update(executionWorkspaces).set({ metadata: { ...workspace!.metadata, [EXPIRATION_KEY]: state }, updatedAt: new Date() })
+            .where(eq(executionWorkspaces.id, snapshot.id)).returning();
+        };
+        await options.beforeDelete?.(workspace);
+        if (!await eligible(tx, workspace, options.now)) throw new Error("live_or_unresolved_consumer");
+        if (!state.workspaceRemoved && present) {
+          const { acquireGitWorktreeCleanupLock } = await import("./workspace-runtime.js");
+          const lock = await acquireGitWorktreeCleanupLock(recipe.worktreePath);
+          try {
+            await assertReproducibleGitFiles(recipe.worktreePath, options.signal);
+            if ((await gitCacheCommand(recipe.worktreePath, ["rev-parse", "HEAD"], options.signal)).trim() !== recipe.commit) throw new Error("head_changed");
+            await fs.rename(recipe.worktreePath, state.worktreeTrash);
+            await finishGitCacheTrash(workspace);
+            result.reclaimedBytes += result.candidateBytes;
+          } finally { await lock.release(); }
+        }
+        await finishGitCacheTrash(workspace);
+        state.workspaceRemoved = true;
+        await persist();
+        result.expired = true;
+        if (options.expireBase) {
+          try {
+            if ((!state.baseTrash || !await exists(state.baseTrash)) && await exists(recipe.sourcePath)) {
+              const bytes = await inspectBase(db, workspace!, options.signal);
+              if (!await eligible(tx, workspace!, options.now)) throw new Error("live_or_unresolved_consumer");
+              const trash = path.join(path.dirname(recipe.sourcePath), `.expired-git-base-${snapshot.id}-${state.generation}`);
+              if (await exists(trash)) throw new Error("unknown_base_trash");
+              state.baseTrash = trash;
+              await persist();
+              await fs.rename(recipe.sourcePath, trash);
+              result.candidateBytes += bytes;
+            }
+            if (state.baseTrash && await exists(state.baseTrash)) {
+              const stat = await fs.lstat(state.baseTrash);
+              if (!stat.isDirectory() || stat.isSymbolicLink()) throw new Error("unsafe_base_trash");
+              const bytes = await allocatedBytes(state.baseTrash, () => { options.signal?.throwIfAborted(); return true; });
+              await fs.rm(state.baseTrash, { recursive: true, force: true });
+              result.reclaimedBytes += bytes;
+              result.baseExpired = true;
+            }
+          } catch (error) { result.skipReason = skipReason(error); }
+        }
+        state.completed = recipe.sourceKind !== "isolated_remote" || !options.expireBase || (result.baseExpired || !await exists(recipe.sourcePath)) && (!state.baseTrash || !await exists(state.baseTrash));
+        await persist();
+        return result;
+      })), true);
+  } catch (error) {
+    options.signal?.throwIfAborted();
+    result.skipReason = skipReason(error);
+    return result;
+  }
+}